While I was testing my changes on orogen on typegen, I stumbled across
the problem of marshalling std::string in mqueue transports.
How am I supposed to do this ? I had a look in the RTT code and string
is commented out ... What is the problem ?
Sylvain
boost::serialize and std::string
On Wednesday 17 November 2010 09:51:38 Sylvain Joyeux wrote:
> While I was testing my changes on orogen on typegen, I stumbled across
> the problem of marshalling std::string in mqueue transports.
>
> How am I supposed to do this ? I had a look in the RTT code and string
> is commented out ... What is the problem ?
There's at least one major problem with 'string:
'string' is modelled as a Sequence in the TypeInfo world. Which means that it
would 'decompose' itself into 'chars'. The XML marshalling/demarshalling code
has hard-coded if's that check for typename == "string" and to not call
getMember (which would return a data source of char) on that type. MQueue does
not use getMember, so it must be something else.
I just enabled the serialization of string in mqueue (using the
MQSerializationProtocol class, which uses boost::serialization) and setup a
unit test in 2.1. What I'm seeing is that there's some kind of double-free
when the string sample that is used is being cleaned up. Needless to say that
looks messy. I recall that archives may optimize serialization of arrays by
accessing the internals directly as a char* (byte blob). Maybe that has
something to do with it.
If I had to solve _that_ problem, I would specialize MQTemplateProtocol for
string instead of using the boost::serialization variant.
Peter
boost::serialize and std::string
On 11/17/2010 10:41 AM, Peter Soetens wrote:
> If I had to solve _that_ problem, I would specialize MQTemplateProtocol for
> string instead of using the boost::serialization variant.
OK. I disable specific types from mqueue from now on to avoid these issues.
I think that "full" mqueue support in orogen will wait for me to switch
to a typelib-based marshalling (which means, when I can try out mqueue
on our systems, hopefully in the next 2-3 weeks).
boost::serialize and std::string
On Wednesday 17 November 2010 10:47:37 Sylvain Joyeux wrote:
> On 11/17/2010 10:41 AM, Peter Soetens wrote:
> > If I had to solve _that_ problem, I would specialize MQTemplateProtocol
> > for string instead of using the boost::serialization variant.
>
> OK. I disable specific types from mqueue from now on to avoid these issues.
>
> I think that "full" mqueue support in orogen will wait for me to switch
> to a typelib-based marshalling (which means, when I can try out mqueue
> on our systems, hopefully in the next 2-3 weeks).
I think the root of the problem is that we directly manipulate the string's
data, violating the encapsulation of the copy-on-write implementation. The
only solution here is to specialize string in our 'binary data archives' in
order to deal with it correctly (ie copy using c_str() and casts to/from
char*)
Peter
boost::serialize and std::string
On Nov 17, 2010, at 05:08 , Peter Soetens wrote:
> On Wednesday 17 November 2010 10:47:37 Sylvain Joyeux wrote:
>> On 11/17/2010 10:41 AM, Peter Soetens wrote:
>>> If I had to solve _that_ problem, I would specialize MQTemplateProtocol
>>> for string instead of using the boost::serialization variant.
>>
>> OK. I disable specific types from mqueue from now on to avoid these issues.
>>
>> I think that "full" mqueue support in orogen will wait for me to switch
>> to a typelib-based marshalling (which means, when I can try out mqueue
>> on our systems, hopefully in the next 2-3 weeks).
>
> I think the root of the problem is that we directly manipulate the string's
> data, violating the encapsulation of the copy-on-write implementation. The
> only solution here is to specialize string in our 'binary data archives' in
> order to deal with it correctly (ie copy using c_str() and casts to/from
> char*)
Does any of this affect RTT::rt_string? Any additional considerations for supporting this?
S
boost::serialize and std::string
On Wednesday 17 November 2010 13:28:55 S Roderick wrote:
> On Nov 17, 2010, at 05:08 , Peter Soetens wrote:
> > On Wednesday 17 November 2010 10:47:37 Sylvain Joyeux wrote:
> >> On 11/17/2010 10:41 AM, Peter Soetens wrote:
> >>> If I had to solve _that_ problem, I would specialize MQTemplateProtocol
> >>> for string instead of using the boost::serialization variant.
> >>
> >> OK. I disable specific types from mqueue from now on to avoid these
> >> issues.
> >>
> >> I think that "full" mqueue support in orogen will wait for me to switch
> >> to a typelib-based marshalling (which means, when I can try out mqueue
> >> on our systems, hopefully in the next 2-3 weeks).
> >
> > I think the root of the problem is that we directly manipulate the
> > string's data, violating the encapsulation of the copy-on-write
> > implementation. The only solution here is to specialize string in our
> > 'binary data archives' in order to deal with it correctly (ie copy using
> > c_str() and casts to/from char*)
>
> Does any of this affect RTT::rt_string? Any additional considerations for
> supporting this? S
The same problems apply to rt_string. There's clearly some work left to be
done to have rt_string work through mqueue.
Peter
boost::serialize and std::string
On Wed, 17 Nov 2010, Peter Soetens wrote:
> On Wednesday 17 November 2010 13:28:55 S Roderick wrote:
>> On Nov 17, 2010, at 05:08 , Peter Soetens wrote:
>>> On Wednesday 17 November 2010 10:47:37 Sylvain Joyeux wrote:
>>>> On 11/17/2010 10:41 AM, Peter Soetens wrote:
>>>>> If I had to solve _that_ problem, I would specialize MQTemplateProtocol
>>>>> for string instead of using the boost::serialization variant.
>>>>
>>>> OK. I disable specific types from mqueue from now on to avoid these
>>>> issues.
>>>>
>>>> I think that "full" mqueue support in orogen will wait for me to switch
>>>> to a typelib-based marshalling (which means, when I can try out mqueue
>>>> on our systems, hopefully in the next 2-3 weeks).
>>>
>>> I think the root of the problem is that we directly manipulate the
>>> string's data, violating the encapsulation of the copy-on-write
>>> implementation. The only solution here is to specialize string in our
>>> 'binary data archives' in order to deal with it correctly (ie copy using
>>> c_str() and casts to/from char*)
>>
>> Does any of this affect RTT::rt_string? Any additional considerations for
>> supporting this? S
>
> The same problems apply to rt_string. There's clearly some work left to be
> done to have rt_string work through mqueue.
>
> Peter
I have the impression that half of the discussions on the mailinglist are
about serialization (and the other half about build tool support). Isn't
that a big alarm sign to warn us that we should (re)do it right...? Isn't
that the right time to have a new, fresh look into what the NetCDF
community, and projects like DDS (openSplice and openDDS) and LabComm
<http://torvalds.cs.lth.se/moin/LabComm>...?
Herman
boost::serialize and std::string
On Wednesday 17 November 2010 13:50:52 Herman Bruyninckx wrote:
> On Wed, 17 Nov 2010, Peter Soetens wrote:
> > On Wednesday 17 November 2010 13:28:55 S Roderick wrote:
> >> On Nov 17, 2010, at 05:08 , Peter Soetens wrote:
> >>> On Wednesday 17 November 2010 10:47:37 Sylvain Joyeux wrote:
> >>>> On 11/17/2010 10:41 AM, Peter Soetens wrote:
> >>>>> If I had to solve _that_ problem, I would specialize
> >>>>> MQTemplateProtocol for string instead of using the
> >>>>> boost::serialization variant.
> >>>>
> >>>> OK. I disable specific types from mqueue from now on to avoid these
> >>>> issues.
> >>>>
> >>>> I think that "full" mqueue support in orogen will wait for me to
> >>>> switch to a typelib-based marshalling (which means, when I can try
> >>>> out mqueue on our systems, hopefully in the next 2-3 weeks).
> >>>
> >>> I think the root of the problem is that we directly manipulate the
> >>> string's data, violating the encapsulation of the copy-on-write
> >>> implementation. The only solution here is to specialize string in our
> >>> 'binary data archives' in order to deal with it correctly (ie copy
> >>> using c_str() and casts to/from char*)
> >>
> >> Does any of this affect RTT::rt_string? Any additional considerations
> >> for supporting this? S
> >
> > The same problems apply to rt_string. There's clearly some work left to
> > be done to have rt_string work through mqueue.
> >
> > Peter
>
> I have the impression that half of the discussions on the mailinglist are
> about serialization (and the other half about build tool support). Isn't
> that a big alarm sign to warn us that we should (re)do it right...? Isn't
> that the right time to have a new, fresh look into what the NetCDF
> community, and projects like DDS (openSplice and openDDS) and LabComm
> <http://torvalds.cs.lth.se/moin/LabComm>...?
I have some questions related to LabComm,
Is there any code available for LabComm?
Is LabComm you mentionned above in any way related to http://www.labcomm.co.uk
or http://sourceforge.net/projects/labcomm/ ?
Ruben
> Herman
boost::serialize and std::string
On Wed, 17 Nov 2010, Ruben Smits wrote:
> On Wednesday 17 November 2010 13:50:52 Herman Bruyninckx wrote:
>> On Wed, 17 Nov 2010, Peter Soetens wrote:
>>> On Wednesday 17 November 2010 13:28:55 S Roderick wrote:
>>>> On Nov 17, 2010, at 05:08 , Peter Soetens wrote:
>>>>> On Wednesday 17 November 2010 10:47:37 Sylvain Joyeux wrote:
>>>>>> On 11/17/2010 10:41 AM, Peter Soetens wrote:
>>>>>>> If I had to solve _that_ problem, I would specialize
>>>>>>> MQTemplateProtocol for string instead of using the
>>>>>>> boost::serialization variant.
>>>>>>
>>>>>> OK. I disable specific types from mqueue from now on to avoid these
>>>>>> issues.
>>>>>>
>>>>>> I think that "full" mqueue support in orogen will wait for me to
>>>>>> switch to a typelib-based marshalling (which means, when I can try
>>>>>> out mqueue on our systems, hopefully in the next 2-3 weeks).
>>>>>
>>>>> I think the root of the problem is that we directly manipulate the
>>>>> string's data, violating the encapsulation of the copy-on-write
>>>>> implementation. The only solution here is to specialize string in our
>>>>> 'binary data archives' in order to deal with it correctly (ie copy
>>>>> using c_str() and casts to/from char*)
>>>>
>>>> Does any of this affect RTT::rt_string? Any additional considerations
>>>> for supporting this? S
>>>
>>> The same problems apply to rt_string. There's clearly some work left to
>>> be done to have rt_string work through mqueue.
>>>
>>> Peter
>>
>> I have the impression that half of the discussions on the mailinglist are
>> about serialization (and the other half about build tool support). Isn't
>> that a big alarm sign to warn us that we should (re)do it right...? Isn't
>> that the right time to have a new, fresh look into what the NetCDF
>> community, and projects like DDS (openSplice and openDDS) and LabComm
>> <http://torvalds.cs.lth.se/moin/LabComm>...?
> I have some questions related to LabComm,
>
> Is there any code available for LabComm?
There was... And a new version is being promised to come out "soon"... Will
have Java and C bindings, I think.
> Is LabComm you mentionned above in any way related to http://www.labcomm.co.uk
> or http://sourceforge.net/projects/labcomm/ ?
No, fortunately :-) It is a development of Lund University.
Herman
> Ruben
>> Herman
>
boost::serialize and std::string
On Wednesday 17 November 2010 13:50:52 Herman Bruyninckx wrote:
> On Wed, 17 Nov 2010, Peter Soetens wrote:
> > On Wednesday 17 November 2010 13:28:55 S Roderick wrote:
> >> On Nov 17, 2010, at 05:08 , Peter Soetens wrote:
> >>> On Wednesday 17 November 2010 10:47:37 Sylvain Joyeux wrote:
> >>>> On 11/17/2010 10:41 AM, Peter Soetens wrote:
> >>>>> If I had to solve _that_ problem, I would specialize
> >>>>> MQTemplateProtocol for string instead of using the
> >>>>> boost::serialization variant.
> >>>>
> >>>> OK. I disable specific types from mqueue from now on to avoid these
> >>>> issues.
> >>>>
> >>>> I think that "full" mqueue support in orogen will wait for me to
> >>>> switch to a typelib-based marshalling (which means, when I can try
> >>>> out mqueue on our systems, hopefully in the next 2-3 weeks).
> >>>
> >>> I think the root of the problem is that we directly manipulate the
> >>> string's data, violating the encapsulation of the copy-on-write
> >>> implementation. The only solution here is to specialize string in our
> >>> 'binary data archives' in order to deal with it correctly (ie copy
> >>> using c_str() and casts to/from char*)
> >>
> >> Does any of this affect RTT::rt_string? Any additional considerations
> >> for supporting this? S
> >
> > The same problems apply to rt_string. There's clearly some work left to
> > be done to have rt_string work through mqueue.
> >
> > Peter
>
> I have the impression that half of the discussions on the mailinglist are
> about serialization (and the other half about build tool support). Isn't
> that a big alarm sign to warn us that we should (re)do it right...? Isn't
> that the right time to have a new, fresh look into what the NetCDF
> community, and projects like DDS (openSplice and openDDS) and LabComm
> <http://torvalds.cs.lth.se/moin/LabComm>...?
It's not serialization that is the problem. It's type introspection we're
talking about. The boost::serialization allows us to code generic type
introspection algorithms. We don't actually serialize data that much, except
in this one place, the mqueue transport.
We're complaining about the particular interface/behaviour of a library I
picked, in order to avoid implementing it ourselves. I doubt DDS/LabComm would
go with lesser flaws. Every serialization library has its 'pecularities'.
boost::serialization was choosen because of its hard real-time capabilities.
Peter