Minimizing component compilation times and code sizes

There are two reasons we are experiencing slowdowns and increased binary code
size during compiling:
* The regeneration of template code in every compilation unit (.o file)
* The use of boost::fusion template magic used in Operations which today's
compilers can barely handle.

The first item we can solve on the short term, the second only on the long
term.

For reducing the regeneration problem, typegen/orogen creates a Types.hpp file
which defines all 'extern template' class definitions which are available in the
typekits. When you include this header file, you also need to link to the
typekit library, but your component size might be cut in half (I tested this
with our HelloWorld component for example). I have a patch that when you
include a new file, <rtt/typekit/Types.hp

, you get this behavior for all
RTT primitive times (int, double,...). Since we don't know if you included
this header, we need to link at all times with the rtt typekit in our
UseOrocos.cmake macros.

A next step would be to include this Types.hpp header in <rtt/RTT.hp

such
that it is always included by most user code. Unfortunately, people not using
the UseOrocos.cmake macros will get hit by a bunch of undefined symbols, until
they link with the rtt typekit.

In the long run, I see no other option than to have components link with the
typekits of the types they use. I wouldn't introduce this in 2.2, but I would
consider it for 2.3. Please provide feedback for your use case.

On the boost::fusion front, I have reported earlier about the non-linear
increase in compilation times (with gcc) when using this library. We've had
some off-list discussions. We're not going to make major changes or any change
in the API of Operations. We'll probably 'optimize' for 1-in/1-out operations,
that is, a pure request/reply operation which would then be very cheap
compilation time wise. So these signatures:

   void foo( void );
   void foo( T in );
   R     foo( void );
   R     foo( T in );

Which would allow us to drop boost::fusion for that case and have 1.x-like
compilation times again. If you add more arguments to 'foo', you're back at
square #1, ie the way it is today.

We've also been looking at making lua-invoked operations hard real-time.
The conclusion here was that operations, as they are implemented now, impose a
considerable complexity on the RTT and that it wouldn't be easy to tinker with
that design to make that happen. That, and the fact that we want to invest in
tools and not so much more in low levels, makes that there is no direct
roadmap ready for that.

With the rest, we're very happy though :-)

Peter

Minimizing component compilation times and code sizes

On Wed, 1 Dec 2010, Peter Soetens wrote:

[...]
> We've also been looking at making lua-invoked operations hard real-time.
> The conclusion here was that operations, as they are implemented now, impose a
> considerable complexity on the RTT and that it wouldn't be easy to tinker with
> that design to make that happen. That, and the fact that we want to invest in
> tools and not so much more in low levels, makes that there is no direct
> roadmap ready for that.

I do not think that you should try to make everything that people can do
with Lua hard realtime safe! I am more interested in knowing which uses of
Lua _can_ result in hard realtime guarantees, and then document these and
give some good use cases for those. If developers use Lua in other ways
than what can (easily) be supported in hard realtime, then it's up to them
to decide whether that's good enough or not. But they should be informed
very well, in order (i) to make them realise that they _have_ to make a
decision, and (ii) to help them making that decision.
>
> With the rest, we're very happy though :-)

Good :-) (Euh... _is_ there any rest...?!?! :-))

> Peter

Herman

Minimizing component compilation times and code sizes

On Wed, Dec 01, 2010 at 05:36:22PM +0100, Herman Bruyninckx wrote:
> On Wed, 1 Dec 2010, Peter Soetens wrote:
>
> [...]
> > We've also been looking at making lua-invoked operations hard real-time.
> > The conclusion here was that operations, as they are implemented now, impose a
> > considerable complexity on the RTT and that it wouldn't be easy to tinker with
> > that design to make that happen. That, and the fact that we want to invest in
> > tools and not so much more in low levels, makes that there is no direct
> > roadmap ready for that.
>
> I do not think that you should try to make everything that people can do
> with Lua hard realtime safe! I am more interested in knowing which uses of
> Lua _can_ result in hard realtime guarantees, and then document these and
> give some good use cases for those. If developers use Lua in other ways
> than what can (easily) be supported in hard realtime, then it's up to them
> to decide whether that's good enough or not. But they should be informed
> very well, in order (i) to make them realise that they _have_ to make a
> decision, and (ii) to help them making that decision.

Yes, agreed! My concrete plan to achieve this is:

- Document which part of the Lua API is real-time safe. A 'R' flag
for this is forseen in the docs, but added yet.

- I have a Lua function "hrt_trace_mode(bool)" in the patch pipeline
which can be used to enable/disable a mode which traces all memory
allocations and unsafe function calls.

Markus

Minimizing component compilation times and code sizes

On Thu, 2 Dec 2010, Markus Klotzbuecher wrote:

> On Wed, Dec 01, 2010 at 05:36:22PM +0100, Herman Bruyninckx wrote:
>> On Wed, 1 Dec 2010, Peter Soetens wrote:
>>
>> [...]
>>> We've also been looking at making lua-invoked operations hard real-time.
>>> The conclusion here was that operations, as they are implemented now, impose a
>>> considerable complexity on the RTT and that it wouldn't be easy to tinker with
>>> that design to make that happen. That, and the fact that we want to invest in
>>> tools and not so much more in low levels, makes that there is no direct
>>> roadmap ready for that.
>>
>> I do not think that you should try to make everything that people can do
>> with Lua hard realtime safe! I am more interested in knowing which uses of
>> Lua _can_ result in hard realtime guarantees, and then document these and
>> give some good use cases for those. If developers use Lua in other ways
>> than what can (easily) be supported in hard realtime, then it's up to them
>> to decide whether that's good enough or not. But they should be informed
>> very well, in order (i) to make them realise that they _have_ to make a
>> decision, and (ii) to help them making that decision.
>
> Yes, agreed! My concrete plan to achieve this is:
>
> - Document which part of the Lua API is real-time safe. A 'R' flag
> for this is forseen in the docs, but added yet.

I am looking forward to this document! Why? Because I would find it of
great scientific satisfaction that you can really classify the realtime
safe and not realtime safe uses of Lua :-)

> - I have a Lua function "hrt_trace_mode(bool)" in the patch pipeline
> which can be used to enable/disable a mode which traces all memory
> allocations and unsafe function calls.

Would the outcome of this function be usable on line, in a "resource
coordinator", to adapt the ongoing system?

Herman

Minimizing component compilation times and code sizes

On Thu, Dec 02, 2010 at 01:11:19PM +0100, Herman Bruyninckx wrote:
> On Thu, 2 Dec 2010, Markus Klotzbuecher wrote:
>
> > On Wed, Dec 01, 2010 at 05:36:22PM +0100, Herman Bruyninckx wrote:
> >> On Wed, 1 Dec 2010, Peter Soetens wrote:
> >>
> >> [...]
> >>> We've also been looking at making lua-invoked operations hard real-time.
> >>> The conclusion here was that operations, as they are implemented now, impose a
> >>> considerable complexity on the RTT and that it wouldn't be easy to tinker with
> >>> that design to make that happen. That, and the fact that we want to invest in
> >>> tools and not so much more in low levels, makes that there is no direct
> >>> roadmap ready for that.
> >>
> >> I do not think that you should try to make everything that people can do
> >> with Lua hard realtime safe! I am more interested in knowing which uses of
> >> Lua _can_ result in hard realtime guarantees, and then document these and
> >> give some good use cases for those. If developers use Lua in other ways
> >> than what can (easily) be supported in hard realtime, then it's up to them
> >> to decide whether that's good enough or not. But they should be informed
> >> very well, in order (i) to make them realise that they _have_ to make a
> >> decision, and (ii) to help them making that decision.
> >
> > Yes, agreed! My concrete plan to achieve this is:
> >
> > - Document which part of the Lua API is real-time safe. A 'R' flag
> > for this is forseen in the docs, but added yet.
>
> I am looking forward to this document! Why? Because I would find it of
> great scientific satisfaction that you can really classify the realtime
> safe and not realtime safe uses of Lua :-)
>
> > - I have a Lua function "hrt_trace_mode(bool)" in the patch pipeline
> > which can be used to enable/disable a mode which traces all memory
> > allocations and unsafe function calls.
>
> Would the outcome of this function be usable on line, in a "resource
> coordinator", to adapt the ongoing system?

I don't think so, because the resource usage of standard c++
components is not traced, e.g. memory allocations go directly to the
os.

Markus