Plugin/Toolkit for Eigen2 or Eigen3 types [revisited]

Hello,

on a previous e-mail, I asked if anyone has already developed a toolkit for
Eigen types. I had two answers about that.

On on hand, Sylvain Joyeux told me that it was hard with RTT dataflow to
guarantee the alignment required by Eigen types. So, we have to create
intermediate types which are able to translate from Eigen types and so forth
(e.g. [1]).

On the other hand, Tinne De Laet just implemented a toolkit (see [2]) and,
as far I can see, without using intermediate types.

At this time, I just want to clarify the difference between both
alternatives. Could we just push Matrix elements in a property bags when
coding the Corba transport and XML marshalling, or the alignment problem
also arises when just using an Eigen type in a Property, Attribute or Ports
(without corba transport) ?

Sorry for these questions. I'm not very familiar with this alignment problem
and even with RTT dataflow internals. My only goal is to implemented as best
as possible an Eigen toolkit with Corba transport that could be usable by
everyone.

Thank you,

Philippe Hamelin

[1] http://github.com/doudou/base/blob/imoby/base/linear_algebra.h
[2]
http://svn.mech.kuleuven.be/repos/orocos/trunk/kul-ros-pkg/iTaSC/src/eig...

Plugin/Toolkit for Eigen2 or Eigen3 types [revisited]

Hi Philippe,

First of all you need to undestand that in rtt, data are encapsulated in a
class named datasource : Here a inheritance graph

http://people.mech.kuleuven.be/~orocos/pub/stable/documentation/rtt/v1.8...

>From DataSource, "getter and setter" are overloaded to act diffently in
DataObjectLocked, for example, use mutex to protect its data.

Since Properties use a member smartPointer on AssignableDataSource. We can
build a properties that will use DataObjectLocked and then become thread
safe properties...

The same principle may be applied to Attribut using this kind of syntax :
Attribute<string> myattr("name", new DataObjectLocked<string>("name") )

Back to Eigen Alignment problems:
If you look far in dataSource implementations, you will see dataSource use
the operator= to transfert data:

On Set on DataPort:
Port -> dataSource -> ( Mutex + operator=)
On Get on DataPort
Port -> DataSource -> (Mutex + operator=)

So if data of Eigen is built correctly in the dataSource there should be no
problem. (I'm still newbie with Eigen).

I don't know how to implement it but I think it's possible to make template
specialisation of DataSource that could handle eigen types. The only problem
is the amount of types ....

Simon

On Tue, Oct 19, 2010 at 9:14 AM, Philippe Hamelin <
philippe [dot] hamelin [..] ...> wrote:

> On 10/18/2010 03:29 PM, Simon Pelletier-Thibault wrote:
>>
>>
>> Property<string> myprop("name", "description",
>> AssignableDataSource<string>::shared_ptr(new DataObjectLocked<string>("name")
>> ) ) ;
>>
>> Sorry for that newbie question, but does someone can give me a brief
> explanation of how this line can make the Property thread-safe?
>
>

Plugin/Toolkit for Eigen2 or Eigen3 types [revisited]

Hi Philippe,

First of all you need to undestand that in rtt, data are encapsulated in a
class named datasource : Here a inheritance graph

http://people.mech.kuleuven.be/~orocos/pub/stable/documentation/rtt/v1.8...

>From DataSource, "getter and setter" are overloaded to act diffently in
DataObjectLocked, for example, use mutex to protect its data.

Since Properties use a member smartPointer on AssignableDataSource. We can
build a properties that will use DataObjectLocked and then become thread
safe properties...

The same principle may be applied to Attribut using this kind of syntax :
Attribute<string> myattr("name", new DataObjectLocked<string>("name") )

Back to Eigen Alignment problems:
If you look far in dataSource implementations, you will see dataSource use
the operator= to transfert data:

On Set on DataPort:
Port -> dataSource -> ( Mutex + operator=)
On Get on DataPort
Port -> DataSource -> (Mutex + operator=)

So if data of Eigen is built correctly in the dataSource there should be no
problem. (I'm still newbie with Eigen).

I don't know how to implement it but I think it's possible to make template
specialisation of DataSource that could handle eigen types. The only problem
is the amount of types ....

Simon

On Tue, Oct 19, 2010 at 9:14 AM, Philippe Hamelin <
philippe [dot] hamelin [..] ...> wrote:

> On 10/18/2010 03:29 PM, Simon Pelletier-Thibault wrote:
>>
>>
>> Property<string> myprop("name", "description",
>> AssignableDataSource<string>::shared_ptr(new DataObjectLocked<string>("name")
>> ) ) ;
>>
>> Sorry for that newbie question, but does someone can give me a brief
> explanation of how this line can make the Property thread-safe?
>
>

Plugin/Toolkit for Eigen2 or Eigen3 types [revisited]

Hi Philippe,

First of all you need to undestand that in rtt, data are encapsulated in a
class named datasource : Here a inheritance graph

http://people.mech.kuleuven.be/~orocos/pub/stable/documentation/rtt/v1.8...

>From DataSource, "getter and setter" are overloaded to act diffently in
DataObjectLocked, for example, use mutex to protect its data.

Since Properties use a member smartPointer on AssignableDataSource. We can
build a properties that will use DataObjectLocked and then become thread
safe properties...

The same principle may be applied to Attribut using this kind of syntax :
Attribute<string> myattr("name", new DataObjectLocked<string>("name") )

Back to Eigen Alignment problems:
If you look far in dataSource implementations, you will see dataSource use
the operator= to transfert data:

On Set on DataPort:
Port -> dataSource -> ( Mutex + operator=)
On Get on DataPort
Port -> DataSource -> (Mutex + operator=)

So if data of Eigen is built correctly in the dataSource there should be no
problem. (I'm still newbie with Eigen).

I don't know how to implement it but I think it's possible to make template
specialisation of DataSource that could handle eigen types. The only problem
is the amount of types ....

Simon

On Tue, Oct 19, 2010 at 9:14 AM, Philippe Hamelin <
philippe [dot] hamelin [..] ...> wrote:

> On 10/18/2010 03:29 PM, Simon Pelletier-Thibault wrote:
>>
>>
>> Property<string> myprop("name", "description",
>> AssignableDataSource<string>::shared_ptr(new DataObjectLocked<string>("name")
>> ) ) ;
>>
>> Sorry for that newbie question, but does someone can give me a brief
> explanation of how this line can make the Property thread-safe?
>
>

Plugin/Toolkit for Eigen2 or Eigen3 types [revisited]

Hi Philippe,

First of all you need to undestand that in rtt, data are encapsulated in a
class named datasource : Here a inheritance graph

http://people.mech.kuleuven.be/~orocos/pub/stable/documentation/rtt/v1.8...

>From DataSource, "getter and setter" are overloaded to act diffently in
DataObjectLocked, for example, use mutex to protect its data.

Since Properties use a member smartPointer on AssignableDataSource. We can
build a properties that will use DataObjectLocked and then become thread
safe properties...

The same principle may be applied to Attribut using this kind of syntax :
Attribute<string> myattr("name", new DataObjectLocked<string>("name") )

Back to Eigen Alignment problems:
If you look far in dataSource implementations, you will see dataSource use
the operator= to transfert data:

On Set on DataPort:
Port -> dataSource -> ( Mutex + operator=)
On Get on DataPort
Port -> DataSource -> (Mutex + operator=)

So if data of Eigen is built correctly in the dataSource there should be no
problem. (I'm still newbie with Eigen).

I don't know how to implement it but I think it's possible to make template
specialisation of DataSource that could handle eigen types. The only problem
is the amount of types ....

Simon

On Tue, Oct 19, 2010 at 9:14 AM, Philippe Hamelin <
philippe [dot] hamelin [..] ...> wrote:

> On 10/18/2010 03:29 PM, Simon Pelletier-Thibault wrote:
>>
>>
>> Property<string> myprop("name", "description",
>> AssignableDataSource<string>::shared_ptr(new DataObjectLocked<string>("name")
>> ) ) ;
>>
>> Sorry for that newbie question, but does someone can give me a brief
> explanation of how this line can make the Property thread-safe?
>
>

Plugin/Toolkit for Eigen2 or Eigen3 types [revisited]

Hi Philippe,

First of all you need to undestand that in rtt, data are encapsulated in a
class named datasource : Here a inheritance graph

http://people.mech.kuleuven.be/~orocos/pub/stable/documentation/rtt/v1.8...

>From DataSource, "getter and setter" are overloaded to act diffently in
DataObjectLocked, for example, use mutex to protect its data.

Since Properties use a member smartPointer on AssignableDataSource. We can
build a properties that will use DataObjectLocked and then become thread
safe properties...

The same principle may be applied to Attribut using this kind of syntax :
Attribute<string> myattr("name", new DataObjectLocked<string>("name") )

Back to Eigen Alignment problems:
If you look far in dataSource implementations, you will see dataSource use
the operator= to transfert data:

On Set on DataPort:
Port -> dataSource -> ( Mutex + operator=)
On Get on DataPort
Port -> DataSource -> (Mutex + operator=)

So if data of Eigen is built correctly in the dataSource there should be no
problem. (I'm still newbie with Eigen).

I don't know how to implement it but I think it's possible to make template
specialisation of DataSource that could handle eigen types. The only problem
is the amount of types ....

Simon

On Tue, Oct 19, 2010 at 9:14 AM, Philippe Hamelin <
philippe [dot] hamelin [..] ...> wrote:

> On 10/18/2010 03:29 PM, Simon Pelletier-Thibault wrote:
>>
>>
>> Property<string> myprop("name", "description",
>> AssignableDataSource<string>::shared_ptr(new DataObjectLocked<string>("name")
>> ) ) ;
>>
>> Sorry for that newbie question, but does someone can give me a brief
> explanation of how this line can make the Property thread-safe?
>
>

Plugin/Toolkit for Eigen2 or Eigen3 types [revisited]

Hi Philippe,

First of all you need to undestand that in rtt, data are encapsulated in a
class named datasource : Here a inheritance graph

http://people.mech.kuleuven.be/~orocos/pub/stable/documentation/rtt/v1.8...

>From DataSource, "getter and setter" are overloaded to act diffently in
DataObjectLocked, for example, use mutex to protect its data.

Since Properties use a member smartPointer on AssignableDataSource. We can
build a properties that will use DataObjectLocked and then become thread
safe properties...

The same principle may be applied to Attribut using this kind of syntax :
Attribute<string> myattr("name", new DataObjectLocked<string>("name") )

Back to Eigen Alignment problems:
If you look far in dataSource implementations, you will see dataSource use
the operator= to transfert data:

On Set on DataPort:
Port -> dataSource -> ( Mutex + operator=)
On Get on DataPort
Port -> DataSource -> (Mutex + operator=)

So if data of Eigen is built correctly in the dataSource there should be no
problem. (I'm still newbie with Eigen).

I don't know how to implement it but I think it's possible to make template
specialisation of DataSource that could handle eigen types. The only problem
is the amount of types ....

Simon

On Tue, Oct 19, 2010 at 9:14 AM, Philippe Hamelin <
philippe [dot] hamelin [..] ...> wrote:

> On 10/18/2010 03:29 PM, Simon Pelletier-Thibault wrote:
>>
>>
>> Property<string> myprop("name", "description",
>> AssignableDataSource<string>::shared_ptr(new DataObjectLocked<string>("name")
>> ) ) ;
>>
>> Sorry for that newbie question, but does someone can give me a brief
> explanation of how this line can make the Property thread-safe?
>
>

Plugin/Toolkit for Eigen2 or Eigen3 types [revisited]

Hello Herman,

It may be a good idea to use NetCDF as an intermediate type. However, I have
some question about that:

- Is there a large overhead? I have to send some hundreds of matrices per
second so I can't handle so much overhead compare to the base type.
- Is there already a RTT toolkit for NetCDF?

Philippe

2010/10/15 Herman Bruyninckx <Herman [dot] Bruyninckx [..] ...>

> On Fri, 15 Oct 2010, Sylvain Joyeux wrote:
>
> On 10/15/2010 03:19 PM, Philippe Hamelin wrote:
>>
>>> Hello,
>>>
>>> on a previous e-mail, I asked if anyone has already developed a toolkit
>>> for Eigen types. I had two answers about that.
>>>
>>> On on hand, Sylvain Joyeux told me that it was hard with RTT dataflow to
>>> guarantee the alignment required by Eigen types. So, we have to create
>>> intermediate types which are able to translate from Eigen types and so
>>> forth (e.g. [1]).
>>>
>>> On the other hand, Tinne De Laet just implemented a toolkit (see [2])
>>> and, as far I can see, without using intermediate types.
>>>
>>> At this time, I just want to clarify the difference between both
>>> alternatives. Could we just push Matrix elements in a property bags when
>>> coding the Corba transport and XML marshalling, or the alignment problem
>>> also arises when just using an Eigen type in a Property, Attribute or
>>> Ports (without corba transport) ?
>>>
>>> Sorry for these questions. I'm not very familiar with this alignment
>>> problem and even with RTT dataflow internals. My only goal is to
>>> implemented as best as possible an Eigen toolkit with Corba transport
>>> that could be usable by everyone.
>>>
>>
>> The problem might arise as soon as you put the types on a port,
>> property, attribute and so on as they get saved into std::vector<>. If
>> there was no such problem, orogen could handle them without any problem.
>>
>> Beware: it is not that it never works, but that it is fragile. For
>> instance, it might work for some data structures but not for others. It
>> will probably work fine with dynamic matrices. It will work less well on
>> 32bit machines than on 64bit (as the latter "naturally" align on bigger
>> multiples than the former).
>>
>> I just had an algorithm that worked fine for a year or so until one of
>> my colleagues ran it today on 32bit and an "exotic distribution" ... and
>> hit the misalignment assertion.
>>
>> I'm thinking about a way to manage that in orogen/typegen where orogen
>> would autoconvert the eigen-enabled types to valid wrappers at the port
>> level ... i.e. create a specialization of
>> RTT::OutputPort<EigenEnabledType> that is actually a
>> RTT::OutputPort<WrapperType> without asking for user intervention.
>>
>
> Please, don't reinvent the wheel! There are very mature projects around
> that solve all these problems, for years already; e.g., NetCDF:
> <http://www.unidata.ucar.edu/software/netcdf/>
> Read these as starters:
> <http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#whatisit>
> <http://en.wikipedia.org/wiki/Netcdf>
> The way they solve it is by introducing a _mathematical model_ layer, that
> is agnostic of word sizes, byte alignments, ....
> The merger with HFD5 <http://en.wikipedia.org/wiki/HDF5> has been
> realised.
>
> These things have orders of magnitude more support than we have, have been
> matured since more than a decade, so I really see no reason to start doing
> the same things again... Do you?
>
> The conversion to specific data types in specific libraries (like Eigen)
> should then _first_ (and _only_!) been done towards NetCDF; then NetCDF is
> used for transport; and then the data is unpacked from NetCDF to the
> specific library that uses that data (need not be Eigen at the other end at
> all).
>
> Herman
>

Plugin/Toolkit for Eigen2 or Eigen3 types [revisited]

On Fri, 15 Oct 2010, Philippe Hamelin wrote:

> It may be a good idea to use NetCDF as an intermediate type. However, I have
> some question about that:
>
> - Is there a large overhead? I have to send some hundreds of matrices per
> second so I can't handle so much overhead compare to the base type.

Two remarks:
- You have to decide what trade-off you want to make: performance versus
transport robustness? If you need performance, you should not use data
ports between components.
- if you design your system such that it _does_ use data ports, for maximum
portability, then the efficiency can (should) be gained automatically
by the deployment tool, that "sees" that you deploy things in a single
address space, so that the overhead can be avoided by not doing the
translations to/from NetCDF (or whatever transport protocol you use)
at both sides of the data port, and just copying the data, or even
sharing the same data structure.

> - Is there already a RTT toolkit for NetCDF?
No. Just as there isn't any _really_ ready RTT toolkit for whatever kind of
data transport...
So, it would be much more effective/useful to spend development efforts on
the second remark above, than on reinventing (poorly) NetCDF, and making
early optimizations for performance reasons only. This has been the death
of many good intention projects! Your first users _know_ about the
constraints and the corresponding optimizations that were at the basis of a
customized toolkit, but later users will without any doubt start using the
functionality way beyond these constraints, and become frustrated.

The first signs of such frustrations are already popping up on the
mailinglist...

Herman

>
> Philippe
>
> 2010/10/15 Herman Bruyninckx <Herman [dot] Bruyninckx [..] ...>
> On Fri, 15 Oct 2010, Sylvain Joyeux wrote:
>
> On 10/15/2010 03:19 PM, Philippe Hamelin wrote:
> Hello,
>
> on a previous e-mail, I asked if anyone has
> already developed a toolkit
> for Eigen types. I had two answers about that.
>
> On on hand, Sylvain Joyeux told me that it was
> hard with RTT dataflow to
> guarantee the alignment required by Eigen
> types. So, we have to create
> intermediate types which are able to translate
> from Eigen types and so
> forth (e.g. [1]).
>
> On the other hand, Tinne De Laet just
> implemented a toolkit (see [2])
> and, as far I can see, without using
> intermediate types.
>
> At this time, I just want to clarify the
> difference between both
> alternatives. Could we just push Matrix
> elements in a property bags when
> coding the Corba transport and XML
> marshalling, or the alignment problem
> also arises when just using an Eigen type in a
> Property, Attribute or
> Ports (without corba transport) ?
>
> Sorry for these questions. I'm not very
> familiar with this alignment
> problem and even with RTT dataflow internals.
> My only goal is to
> implemented as best as possible an Eigen
> toolkit with Corba transport
> that could be usable by everyone.
>
>
> The problem might arise as soon as you put the types on a
> port,
> property, attribute and so on as they get saved into
> std::vector<>. If
> there was no such problem, orogen could handle them
> without any problem.
>
> Beware: it is not that it never works, but that it is
> fragile. For
> instance, it might work for some data structures but not
> for others. It
> will probably work fine with dynamic matrices. It will
> work less well on
> 32bit machines than on 64bit (as the latter "naturally"
> align on bigger
> multiples than the former).
>
> I just had an algorithm that worked fine for a year or so
> until one of
> my colleagues ran it today on 32bit and an "exotic
> distribution" ... and
> hit the misalignment assertion.
>
> I'm thinking about a way to manage that in orogen/typegen
> where orogen
> would autoconvert the eigen-enabled types to valid
> wrappers at the port
> level ... i.e. create a specialization of
> RTT::OutputPort<EigenEnabledType> that is actually a
> RTT::OutputPort<WrapperType> without asking for user
> intervention.
>
>
> Please, don't reinvent the wheel! There are very mature projects
> around
> that solve all these problems, for years already; e.g., NetCDF:
>  <http://www.unidata.ucar.edu/software/netcdf/>
> Read these as starters:
>  <http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#whatisit>
>  <http://en.wikipedia.org/wiki/Netcdf>
> The way they solve it is by introducing a _mathematical model_ layer,
> that
> is agnostic of word sizes, byte alignments, ....
> The merger with HFD5 <http://en.wikipedia.org/wiki/HDF5> has been
> realised.
>
> These things have orders of magnitude more support than we have, have
> been
> matured since more than a decade, so I really see no reason to start
> doing
> the same things again... Do you?
>
> The conversion to specific data types in specific libraries (like
> Eigen)
> should then _first_ (and _only_!) been done towards NetCDF; then
> NetCDF is
> used for transport; and then the data is unpacked from NetCDF to the
> specific library that uses that data (need not be Eigen at the other
> end at
> all).
>
> Herman
>
>
>
>

--
K.U.Leuven, Mechanical Eng., Mechatronics & Robotics Research Group
<http://people.mech.kuleuven.be/~bruyninc> Tel: +32 16 328056
EURON Coordinator (European Robotics Research Network) <http://www.euron.org>
Open Realtime Control Services <http://www.orocos.org>
Associate Editor JOSER <http://www.joser.org>, IJRR <http://www.ijrr.org>