[PROPOSAL] redesigning the data flow implementation

Quite frankly, now that I looked at the RTT data flow implementation,
I'm a bit confused as to what you were trying to achieve. The
implementation is miles away from the model (or, at least, what I
assumed the model was).

The problems
============
* the current implementation is not about data connections (getting
data flowing from one port to another). It is about managing shared
memory places, where different ports read and write. That is quite
obvious for the data ports (i.e. there is a shared data sample that
anyone can read or write), and is IMO completely meaningless for
buffer ports. Buffer ports are really in need of a data flow model
(see above a more specific critic about multi-output buffers)

* Per se, this does not seem a problem. Data is getting transmitted
from one port to the other, isn't it ?

Well, actually it is a problem because it forbids a clean connection
management implementation. Why ? Because there is no way to know who
is reading and who is writing ... Thus, the completely useless
disconnect() call. Why useless ? Because if you do: (this is
pseudo-code of course)

connect(source, dest)
source.disconnect()

Then dest.isConnected() returns true, even though dest will not get
any data from anywhere (there is no writer anymore on that connection).
This is more general, as it is for instance very difficult to
implement proper connection management in the CORBA case.

* Because of this connection management issue, it is very difficult to
implement a "push" model. It leads to huge problems with the CORBA
transport when wireless is bad, because each pop or get needs a few
calls.

* It makes the whole implementation a huge mess. There is at least
twice the number of classes normally needed to implement a connection
model *and* code is not reused (DataPort is actually *not* a subclass of
both ReadDataPort and WriteDataPort, same for buffers).

* We already had a long thread about multiple-output buffered
connections. I'll summarize what for me was the most important
points:
- the current implementation allows to distribute workload seamlessly
between different task contexts.
- it does not allow to send the same set of samples to different task
contexts. Ther is a hack allowing to read buffer connections as if
they were data connections, but it is a hack given that the reader
cannot know if it is really reading a sample or reading a default
value because the buffer is empty.

IMO the first case is actually rare in robotic control (and you can
implement a generic workload-sharing component with nicer features
like for instance keeping the ordering between input and output) like
in the following example:
=> A0 A3 [PROCESSING] => A'0 A'3
A0 A1 A2 A3 => [WORK SHARING WORK SHARING] => A'0 A'1 A'2 A'3
=> A1 A2 [PROCESSING] => A'1 A'2

The second case is much more common. For instance, in my robot, I
want to have a safety component that monitors a laser scanner
(near-obstacle detection for the purpose of safety) and the same
laser scans to go to a SLAM algorithm. I cannot do that for now,
because I need a buffered connection to the SLAM algorithm. I cannot
use the aforementionned hack either because for now I plan to put
a network connection between the scanner driver and the two targets,
and therefore I cannot really guarantee which component will get
what.

Proposal
========

What I'm proposing is getting back to a good'ol data flow model, namely:
* making write ports "send and forget". If the port fails to write,
then it is the problem of the reader ! I really don't see what the writer
can do about it anyway, given that it does not know what the data
will be used for (principle of component separation). The reader can
still detect that its input buffer is full and that it did not get some
samples and do something about it.

* making write ports "connection-type less". I.e. no WRITE data ports
and WRITE buffer ports anymore, only write ports. This will allow to
connect a write port to a read port with any kind of connections.
Actually, I don't see a use case where the port designer can actually
decide what kind of connection is best for its OUTPUT ports. Some
examples:
- in the laser scanner example above, the safety component would like
a data port and the slam a buffer port
- in position filtering, some components just want the latest
positions and other components all the position stream (for
interpolation purposes for instance)
- in general, GUI vs. X. GUIs want most of the time the latest
values.
- ... I'm sure I can come up with other examples if you want them

* locating the sample on the read ports (i.e. no ConnectionInterface
and subclasses anymore). The bad: one copy of each sample per read
port. The good: you implement the point above (write ports do not
have a connection type), and you fix buffer connections once and for
all.

* removing (or deprecating) read/write ports. They really have no place
in a data flow model.

>From a coding point of view, I can do it. The bottom line being that
doing that will remove a lot of code and deeply clean up the set of
classes involved in data flow.

Unless there are compelling reasons not to do so, I'll do it anyway (git
to the rescue), because I cannot do with the current behaviour. So if
you are not sure (but not completely against), you will be able to see
before deciding.

[PROPOSAL] redesigning the data flow implementation

On Monday 20 October 2008 10:57:01 Sylvain Joyeux wrote:
> Quite frankly, now that I looked at the RTT data flow implementation,
> I'm a bit confused as to what you were trying to achieve. The
> implementation is miles away from the model (or, at least, what I
> assumed the model was).

Ok. I throw in the towel. Your analysis is painfully accurate. We win
simplicity and flexibility with your approach. The strong points are:

* a single 'send and forget' type write port
(note: should we have access to the last written value from scripting/C++?)
* Allowing mixed buffered/unbuffered reads from a data flow.
- makes components more reusable in different applications
* Putting the buffer size responsibility with the reader
- allows per receiver tuning.

I'll help integrating your solution into mainline. I'll mostly care about
preserving the real-timeness and migration path of existing applications.

Peter

[PROPOSAL] redesigning the data flow implementation

On Mon, 20 Oct 2008, Peter Soetens wrote:

> On Monday 20 October 2008 10:57:01 Sylvain Joyeux wrote:
>> Quite frankly, now that I looked at the RTT data flow implementation,
>> I'm a bit confused as to what you were trying to achieve. The
>> implementation is miles away from the model (or, at least, what I
>> assumed the model was).
>
> Ok. I throw in the towel. Your analysis is painfully accurate. We win
> simplicity and flexibility with your approach. The strong points are:
>
> * a single 'send and forget' type write port
> (note: should we have access to the last written value from scripting/C++?)

I suggest we should _not_ do this, since this is asking for a can of worms
to be opened. _And_ it is semantically not necessary: if a component really
wants to know what is in a channel, it should connect to it as reader...

> * Allowing mixed buffered/unbuffered reads from a data flow.
> - makes components more reusable in different applications
> * Putting the buffer size responsibility with the reader
> - allows per receiver tuning.

Yes! Please, do the redesign of the data ports with this client-driven
"Quality of Service" concept in mind for _all_ services in Orocos. :-)

> I'll help integrating your solution into mainline. I'll mostly care about
> preserving the real-timeness and migration path of existing applications.

Indeed...

Herman

[PROPOSAL] redesigning the data flow implementation

On Monday 20 October 2008 14:16:53 Herman Bruyninckx wrote:
> On Mon, 20 Oct 2008, Peter Soetens wrote:
> > On Monday 20 October 2008 10:57:01 Sylvain Joyeux wrote:
> >> Quite frankly, now that I looked at the RTT data flow implementation,
> >> I'm a bit confused as to what you were trying to achieve. The
> >> implementation is miles away from the model (or, at least, what I
> >> assumed the model was).
> >
> > Ok. I throw in the towel. Your analysis is painfully accurate. We win
> > simplicity and flexibility with your approach. The strong points are:
> >
> > * a single 'send and forget' type write port
> > (note: should we have access to the last written value from
> > scripting/C++?)
>
> I suggest we should _not_ do this, since this is asking for a can of worms
> to be opened. _And_ it is semantically not necessary: if a component really
> wants to know what is in a channel, it should connect to it as reader...

What you propose is entirely correct for reading the connection. Yet, for
debugging/taskbrowsing/... purposes, it is convenient to know which value the
current component has sent to its output port (apart from what is happening
at the connection level).

That is one of the reasons why the current WriteDataPort is under-/not-used:
you can only do a Set(...) and have no means to retrieve the last written
value. Hence people replace it with a DataPort which has Set and Get during
debugging and do not revert it back to WriteDataPort lateron.

Peter

Peter

[PROPOSAL] redesigning the data flow implementation

On Mon, 20 Oct 2008, Peter Soetens wrote:

> On Monday 20 October 2008 14:16:53 Herman Bruyninckx wrote:
>> On Mon, 20 Oct 2008, Peter Soetens wrote:
>>> On Monday 20 October 2008 10:57:01 Sylvain Joyeux wrote:
>>>> Quite frankly, now that I looked at the RTT data flow implementation,
>>>> I'm a bit confused as to what you were trying to achieve. The
>>>> implementation is miles away from the model (or, at least, what I
>>>> assumed the model was).
>>>
>>> Ok. I throw in the towel. Your analysis is painfully accurate. We win
>>> simplicity and flexibility with your approach. The strong points are:
>>>
>>> * a single 'send and forget' type write port
>>> (note: should we have access to the last written value from
>>> scripting/C++?)
>>
>> I suggest we should _not_ do this, since this is asking for a can of worms
>> to be opened. _And_ it is semantically not necessary: if a component really
>> wants to know what is in a channel, it should connect to it as reader...
>
> What you propose is entirely correct for reading the connection. Yet, for
> debugging/taskbrowsing/... purposes, it is convenient to know which value the
> current component has sent to its output port (apart from what is happening
> at the connection level).

Task browsing _has_ already its support such as reporting, so you should
use _that_ , in stead of adding debugging overhead to a clean concept...

> That is one of the reasons why the current WriteDataPort is under-/not-used:
> you can only do a Set(...) and have no means to retrieve the last written
> value. Hence people replace it with a DataPort which has Set and Get during
> debugging and do not revert it back to WriteDataPort lateron.

Than they have a development problem... I think it is definitely a bad idea
to change the nominal semantics of Orocos just because of debugging
reasons. This should be done in a debugging infrastructure, and the
reporting is already available for that...

Herman

[PROPOSAL] redesigning the data flow implementation

On Mon, Oct 20, 2008 at 03:59:20PM +0200, Herman Bruyninckx wrote:
> On Mon, 20 Oct 2008, Peter Soetens wrote:
>
>> On Monday 20 October 2008 14:16:53 Herman Bruyninckx wrote:
>>> On Mon, 20 Oct 2008, Peter Soetens wrote:
>>>> On Monday 20 October 2008 10:57:01 Sylvain Joyeux wrote:
>>>>> Quite frankly, now that I looked at the RTT data flow implementation,
>>>>> I'm a bit confused as to what you were trying to achieve. The
>>>>> implementation is miles away from the model (or, at least, what I
>>>>> assumed the model was).
>>>>
>>>> Ok. I throw in the towel. Your analysis is painfully accurate. We win
>>>> simplicity and flexibility with your approach. The strong points are:
>>>>
>>>> * a single 'send and forget' type write port
>>>> (note: should we have access to the last written value from
>>>> scripting/C++?)
>>>
>>> I suggest we should _not_ do this, since this is asking for a can of worms
>>> to be opened. _And_ it is semantically not necessary: if a component really
>>> wants to know what is in a channel, it should connect to it as reader...

Mmmm ... correcting myself:

I still think that from a semantic cleanliness point of view, Herman is
right

*But*

It is actually quite easy to implement (only have a data object on the
writer and you can get the last written value)

*and (more interesting)*

The question of "what to do on new connections" is still open:
* should we initialize the reader with the last written value
* or should it wait for a new value to be written ?

[PROPOSAL] redesigning the data flow implementation

On Monday 20 October 2008 16:46:11 Sylvain Joyeux wrote:
>
> Mmmm ... correcting myself:
>
> I still think that from a semantic cleanliness point of view, Herman is
> right
>
> *But*
>
> It is actually quite easy to implement (only have a data object on the
> writer and you can get the last written value)

Herman won't accept an 'easy to implement' argument (I can tell after all
these years[*]). However, to me it is more of a 'state' argument, ie storing
the state of the connection.

>
> *and (more interesting)*
>
> The question of "what to do on new connections" is still open:
> * should we initialize the reader with the last written value
> * or should it wait for a new value to be written ?

Ask a friend of mine (I think his name is 'klass') what he thinks from
practical experience, and he'll say[**]: 'initialise the reader with the last
written value'. When a connection endpoint is added, it must have a well
defined state, and for data flow connections, this is clearly the value of
the connection itself. This would also make sense from a backwards compatible
point of view because the current ports work like this.

Peter

[*] note my boldness, proclaiming what other people would say and accept...
[**] Yes, I'm doing it again !

[PROPOSAL] redesigning the data flow implementation

On Mon, 20 Oct 2008, Peter Soetens wrote:

> On Monday 20 October 2008 16:46:11 Sylvain Joyeux wrote:
>> Mmmm ... correcting myself:
>> I still think that from a semantic cleanliness point of view, Herman is
>> right
>> *But*
>> It is actually quite easy to implement (only have a data object on the
>> writer and you can get the last written value)
>
> Herman won't accept an 'easy to implement' argument (I can tell after all
> these years[*]). However, to me it is more of a 'state' argument, ie storing
> the state of the connection.

"state" is extremely difficult to keep consistent in things like data
communication, _by definition_, since two separate activities can be
involved...

>> *and (more interesting)*
>> The question of "what to do on new connections" is still open:
>> * should we initialize the reader with the last written value
>> * or should it wait for a new value to be written ?
>
> Ask a friend of mine (I think his name is 'klass') what he thinks from
> practical experience, and he'll say[**]: 'initialise the reader with the last
> written value'.
...and when that does not exist, you have to cope with the exception
anyway, so that's not a valid solution to be imposed generically! And what
if this value is "years" old? I suggest to let the connecting reader choose
the policy.

> When a connection endpoint is added, it must have a well
> defined state, and for data flow connections, this is clearly the value of
> the connection itself.
I agree.

> This would also make sense from a backwards compatible
> point of view because the current ports work like this.
>
> [*] note my boldness, proclaiming what other people would say and accept...
> [**] Yes, I'm doing it again !

Tch tch tch...! :-)

Herman

[PROPOSAL] redesigning the data flow implementation

On Tue, Oct 21, 2008 at 07:42:50AM +0200, Herman Bruyninckx wrote:
> On Mon, 20 Oct 2008, Peter Soetens wrote:
>
>> On Monday 20 October 2008 16:46:11 Sylvain Joyeux wrote:
>>> Mmmm ... correcting myself:
>>> I still think that from a semantic cleanliness point of view, Herman is
>>> right
>>> *But*
>>> It is actually quite easy to implement (only have a data object on the
>>> writer and you can get the last written value)
>>
>> Herman won't accept an 'easy to implement' argument (I can tell after all
>> these years[*]). However, to me it is more of a 'state' argument, ie storing
>> the state of the connection.
>
> "state" is extremely difficult to keep consistent in things like data
> communication, _by definition_, since two separate activities can be
> involved...
>
>>> *and (more interesting)*
>>> The question of "what to do on new connections" is still open:
>>> * should we initialize the reader with the last written value
>>> * or should it wait for a new value to be written ?
>>
>> Ask a friend of mine (I think his name is 'klass') what he thinks from
>> practical experience, and he'll say[**]: 'initialise the reader with the last
>> written value'.
> ...and when that does not exist, you have to cope with the exception
> anyway, so that's not a valid solution to be imposed generically! And what
> if this value is "years" old? I suggest to let the connecting reader choose
> the policy.
>
>
>> When a connection endpoint is added, it must have a well
>> defined state, and for data flow connections, this is clearly the value of
>> the connection itself.
> I agree.

I don't. All readers should be ready to have an unitialized connection,
because it is the initial state of a connection -- "initial values"
are too much of a hack for me.

... and they may want "new data" and not be bothered with a "first
write" that they don't want. Anyway, I'm going towards a "fully
configurable" connection interface, meaning that it will be possible to
chose the policy ...

[PROPOSAL] redesigning the data flow implementation

On Tue, 21 Oct 2008, Sylvain Joyeux wrote:

> On Tue, Oct 21, 2008 at 07:42:50AM +0200, Herman Bruyninckx wrote:
>> On Mon, 20 Oct 2008, Peter Soetens wrote:
>>
>>> On Monday 20 October 2008 16:46:11 Sylvain Joyeux wrote:
>>>> Mmmm ... correcting myself:
>>>> I still think that from a semantic cleanliness point of view, Herman is
>>>> right
>>>> *But*
>>>> It is actually quite easy to implement (only have a data object on the
>>>> writer and you can get the last written value)
>>>
>>> Herman won't accept an 'easy to implement' argument (I can tell after all
>>> these years[*]). However, to me it is more of a 'state' argument, ie storing
>>> the state of the connection.
>>
>> "state" is extremely difficult to keep consistent in things like data
>> communication, _by definition_, since two separate activities can be
>> involved...
>>
>>>> *and (more interesting)*
>>>> The question of "what to do on new connections" is still open:
>>>> * should we initialize the reader with the last written value
>>>> * or should it wait for a new value to be written ?
>>>
>>> Ask a friend of mine (I think his name is 'klass') what he thinks from
>>> practical experience, and he'll say[**]: 'initialise the reader with the last
>>> written value'.
>> ...and when that does not exist, you have to cope with the exception
>> anyway, so that's not a valid solution to be imposed generically! And what
>> if this value is "years" old? I suggest to let the connecting reader choose
>> the policy.
>>
>>
>>> When a connection endpoint is added, it must have a well
>>> defined state, and for data flow connections, this is clearly the value of
>>> the connection itself.
>> I agree.
>
> I don't. All readers should be ready to have an unitialized connection,
> because it is the initial state of a connection -- "initial values"
> are too much of a hack for me.

I think (I hope, rather :-)) that Peter meant that the only 'state' that a
connection itself can have is whether it is connected or not. He was not
talking about the state of the values in the connection.
And I agree with you that "initial values" are not a good idea...

> ... and they may want "new data" and not be bothered with a "first
> write" that they don't want. Anyway, I'm going towards a "fully
> configurable" connection interface, meaning that it will be possible to
> chose the policy ...
Good! Nevertheless, a good discussion about what policies _should_ be
supported is very welcome!

Herman

[PROPOSAL] redesigning the data flow implementation

On Tuesday 21 October 2008 07:42:50 Herman Bruyninckx wrote:
> On Mon, 20 Oct 2008, Peter Soetens wrote:
> > On Monday 20 October 2008 16:46:11 Sylvain Joyeux wrote:
> >> Mmmm ... correcting myself:
> >> I still think that from a semantic cleanliness point of view, Herman is
> >> right
> >> *But*
> >> It is actually quite easy to implement (only have a data object on the
> >> writer and you can get the last written value)
> >
> > Herman won't accept an 'easy to implement' argument (I can tell after all
> > these years[*]). However, to me it is more of a 'state' argument, ie
> > storing the state of the connection.
>
> "state" is extremely difficult to keep consistent in things like data
> communication, _by definition_, since two separate activities can be
> involved...

We solved this today by allowing each writer to specify an 'initial' output
value, with which the connection is initialised. On the other hand, Sylvain
had a proposal for a hasData() call. It wouldn't surprise me that this will
be part of the new API :-)

Peter

[PROPOSAL] redesigning the data flow implementation

On Tue, 21 Oct 2008, Peter Soetens wrote:

> On Tuesday 21 October 2008 07:42:50 Herman Bruyninckx wrote:
>> On Mon, 20 Oct 2008, Peter Soetens wrote:
>>> On Monday 20 October 2008 16:46:11 Sylvain Joyeux wrote:
>>>> Mmmm ... correcting myself:
>>>> I still think that from a semantic cleanliness point of view, Herman is
>>>> right
>>>> *But*
>>>> It is actually quite easy to implement (only have a data object on the
>>>> writer and you can get the last written value)
>>>
>>> Herman won't accept an 'easy to implement' argument (I can tell after all
>>> these years[*]). However, to me it is more of a 'state' argument, ie
>>> storing the state of the connection.
>>
>> "state" is extremely difficult to keep consistent in things like data
>> communication, _by definition_, since two separate activities can be
>> involved...
>
> We solved this today by allowing each writer to specify an 'initial' output
> value, with which the connection is initialised.
This is a good suggestion, compatible with the good design practice not to
have uninitialized objects. However, I have the following remarks:
- how can the reader know the difference between this "bogus" value, just
meant to initialize the channel, and a "real" value?
- Clients should have no problem dealing with an empty channel, since
reading from an empty channel _is_ normal semantics to be expected from
communication channels.

> On the other hand, Sylvain had a proposal for a hasData() call. It
> wouldn't surprise me that this will be part of the new API :-)
"hasData()" makes sense, locally, that is at the port that the reader sees.
The semantics should be that this call returns what the local "proxy" to
the communication channel has as data, but please don't try to give it a
semantics that _guarantees_ that there is no data in the channel anywhere,
since this is tremendously difficult to do on concurrent activities...

Herman

[PROPOSAL] redesigning the data flow implementation

>> On the other hand, Sylvain had a proposal for a hasData() call. It
>> wouldn't surprise me that this will be part of the new API :-)

Actually, I'm moving to read() calls that all return booleans, telling
wether or not there was a data sample.

and this boolean says only if there is a sample at the receiving end. If
there is data being propagated in the pipeline, it is not available and
therefore the read calls return false.

[PROPOSAL] redesigning the data flow implementation

On Tue, 21 Oct 2008, Sylvain Joyeux wrote:

>>> On the other hand, Sylvain had a proposal for a hasData() call. It
>>> wouldn't surprise me that this will be part of the new API :-)
>
> Actually, I'm moving to read() calls that all return booleans, telling
> wether or not there was a data sample.
>
> and this boolean says only if there is a sample at the receiving end. If
> there is data being propagated in the pipeline, it is not available and
> therefore the read calls return false.

The "hasData()" could be a usefull thing to have too, I guess: only
providing "read()" means that you have "consumed" the data to find out
whether there _is_ data; while having a "peek" without consumption could be
good to have, in some cases...

Herman

[PROPOSAL] redesigning the data flow implementation

> The "hasData()" could be a usefull thing to have too, I guess: only
> providing "read()" means that you have "consumed" the data to find out
> whether there _is_ data; while having a "peek" without consumption could be
> good to have, in some cases...
Yes. there is a peek() as well :P

I removed hasData() because of one good comment from Peter: it is
prone to race conditions while "bool peek(sample)" is not.

[PROPOSAL] redesigning the data flow implementation

On Tuesday 21 October 2008 11:51:29 Sylvain Joyeux wrote:
> > The "hasData()" could be a usefull thing to have too, I guess: only
> > providing "read()" means that you have "consumed" the data to find out
> > whether there _is_ data; while having a "peek" without consumption could
> > be good to have, in some cases...
>
> Yes. there is a peek() as well :P
>
> I removed hasData() because of one good comment from Peter: it is
> prone to race conditions while "bool peek(sample)" is not.

I knew it ! There had to be one !

[PROPOSAL] redesigning the data flow implementation

Just my two penny, as an ignorant and unexperienced user.

penny 1:
I expect (hope) that the proposed new implementation will resolve the problems I have with making connections. Let me try to explain.
My reference of how to make connections is Simulink, or rather 20-sim. I select two ports, typically one input and one output, and make a connection. I like to be warned or even punished if I make a connection between 2 outputs etc. In 20-sim, ports are typed, and hence also the type is checked; very useful I think. I expect that multiple input ports can be connected to a single output, that an unconnected output is ok, and that an unconnected input is an error.
Now about my problems with Orocos. I have read the section on Setting up Data Flow in the component manual quite often, and still think it is difficult. I do not really understand the utility/concept of 'connectPeers'; where can I read more about it? I try to reuse components, and therefore regularly have to connect two ports that have different names. Unfortunately, the powerful and useful connectPorts(TC* a,TC* b) does not work then, and I make connections using aPort.connectTo(&anotherPort). The bad thing is that in this way the outputport can only have one connection (or am I doing something wrong?). Many times, this has caused mistakes. It seems to me that as far as the proposal is concerned, making a new connection to a WritePort would not disconnect an already consisting connection, right?

penny 2
About buffered/unbuffered reading (what Herman calls ReadPort policies?). If I understand correctly, the proposal talks about copying writer data to a reader port, and then in the case of a buffered port storing it. I would think that a BufferedPort could be functioning like a boost::circular_buffer; the proposed ReadPort ("unbuffered") then rather is a BufferedPort with buffer size 1. I would in addition like to have a 'real' unbuffered ReadPort that just provides a reference to the latest written value at the connected WritePort. No hassle, if the connection is there it gives the most recent data, no data copying involved.

Cheers, Theo.

[PROPOSAL] redesigning the data flow implementation

Just my two penny, as an ignorant and unexperienced user.

penny 1:
I expect (hope) that the proposed new implementation will resolve the problems I have with making connections. Let me try to explain.
My reference of how to make connections is Simulink, or rather 20-sim. I select two ports, typically one input and one output, and make a connection. I like to be warned or even punished if I make a connection between 2 outputs etc. In 20-sim, ports are typed, and hence also the type is checked; very useful I think. I expect that multiple input ports can be connected to a single output, that an unconnected output is ok, and that an unconnected input is an error.
Now about my problems with Orocos. I have read the section on Setting up Data Flow in the component manual quite often, and still think it is difficult. I do not really understand the utility/concept of 'connectPeers'; where can I read more about it? I try to reuse components, and therefore regularly have to connect two ports that have different names. Unfortunately, the powerful and useful connectPorts(TC* a,TC* b) does not work then, and I make connections using aPort.connectTo(&anotherPort). The bad thing is that in this way the outputport can only have one connection (or am I doing something wrong?). Many times, this has caused mistakes. It seems to me that as far as the proposal is concerned, making a new connection to a WritePort would not disconnect an already consisting connection, right?

penny 2
About buffered/unbuffered reading (what Herman calls ReadPort policies?). If I understand correctly, the proposal talks about copying writer data to a reader port, and then in the case of a buffered port storing it. I would think that a BufferedPort could be functioning like a boost::circular_buffer; the proposed ReadPort ("unbuffered") then rather is a BufferedPort with buffer size 1. I would in addition like to have a 'real' unbuffered ReadPort that just provides a reference to the latest written value at the connected WritePort. No hassle, if the connection is there it gives the most recent data, no data copying involved.

Cheers, Theo.

[PROPOSAL] redesigning the data flow implementation

The mail below was accidentally filtered from the orocos-dev mailinglist. I'm bumping it for the record (and the good points of attention that it raises).

[

vri wrote:

Just my two penny, as an ignorant and unexperienced user.

penny 1:
I expect (hope) that the proposed new implementation will resolve the problems I have with making connections. Let me try to explain.
My reference of how to make connections is Simulink, or rather 20-sim. I select two ports, typically one input and one output, and make a connection. I like to be warned or even punished if I make a connection between 2 outputs etc. In 20-sim, ports are typed, and hence also the type is checked; very useful I think. I expect that multiple input ports can be connected to a single output, that an unconnected output is ok, and that an unconnected input is an error.
Now about my problems with Orocos. I have read the section on Setting up Data Flow in the component manual quite often, and still think it is difficult. I do not really understand the utility/concept of 'connectPeers'; where can I read more about it? I try to reuse components, and therefore regularly have to connect two ports that have different names. Unfortunately, the powerful and useful connectPorts(TC* a,TC* b) does not work then, and I make connections using aPort.connectTo(&anotherPort). The bad thing is that in this way the outputport can only have one connection (or am I doing something wrong?). Many times, this has caused mistakes. It seems to me that as far as the proposal is concerned, making a new connection to a WritePort would not disconnect an already consisting connection, right?

penny 2
About buffered/unbuffered reading (what Herman calls ReadPort policies?). If I understand correctly, the proposal talks about copying writer data to a reader port, and then in the case of a buffered port storing it. I would think that a BufferedPort could be functioning like a boost::circular_buffer; the proposed ReadPort ("unbuffered") then rather is a BufferedPort with buffer size 1. I would in addition like to have a 'real' unbuffered ReadPort that just provides a reference to the latest written value at the connected WritePort. No hassle, if the connection is there it gives the most recent data, no data copying involved.

Cheers, Theo.

[PROPOSAL] redesigning the data flow implementation

Here some news on the status

I already have some code done, but my current workload does not allow me to
work as much as I would like on this (paper and proposal to write ...)

Hopefully, January will be better in which case we have a prototype for
february.

I'll update the list as soon as I have a more-or-less definitive interface, to
discuss it.

[PROPOSAL] redesigning the data flow implementation

> Here some news on the status
>
> I already have some code done, but my current workload does not allow me
> to
> work as much as I would like on this (paper and proposal to write ...)
>
> Hopefully, January will be better in which case we have a prototype for
> february.
>
> I'll update the list as soon as I have a more-or-less definitive
> interface, to
> discuss it.

Thanks.

Do you object to already adding the modified version of your event based
data ports on trunk for RTT 1.8 ?

Related to git: I had to init git-svn from the SVN server myself and
abandon the clone of your git hub repository. I couldn't get git-svn
working with a clone from your repos. So you'll have to restart from my
repos if you want to track trunk...

Peter

[PROPOSAL] redesigning the data flow implementation

> penny 1:
> I expect (hope) that the proposed new implementation will resolve the problems
> I have with making connections. Let me try to explain.

> My reference of how to make connections is Simulink, or rather 20-sim.
> I select two ports, typically one input and one output, and make a connection.
> I like to be warned or even punished if I make a connection between 2 outputs
> etc. In 20-sim, ports are typed, and hence also the type is checked; very useful
> I think. I expect that multiple input ports can be connected to a single output,
> that an unconnected output is ok, and that an unconnected input is an error.
Unconnected inputs won't be an error.

> Now about my problems with Orocos. I have read the section on Setting up Data
> Flow in the component manual quite often, and still think it is difficult. I do
> not really understand the utility/concept of 'connectPeers'; where can I read
> more about it? I try to reuse components, and therefore regularly have to
> connect two ports that have different names. Unfortunately, the powerful and
> useful connectPorts(TC* a,TC* b) does not work then, and I make connections
> using aPort.connectTo(&anotherPort). The bad thing is that in this way the
> outputport can only have one connection (or am I doing something wrong?). Many
> times, this has caused mistakes. It seems to me that as far as the proposal is
> concerned, making a new connection to a WritePort would not disconnect an
> already consisting connection, right?
One of the goals of the proposal is to have a sane one-writer,multiple-readers
behaviour. I want to stress that the first implementation will NOT allow
multiple writers with one single reader.

> penny 2
> About buffered/unbuffered reading (what Herman calls ReadPort policies?). If
> I understand correctly, the proposal talks about copying writer data to a reader
> port, and then in the case of a buffered port storing it. I would think that
> a BufferedPort could be functioning like a boost::circular_buffer; the proposed
> ReadPort ("unbuffered") then rather is a BufferedPort with buffer size 1.
> I would in addition like to have a 'real' unbuffered ReadPort that just provides
> a reference to the latest written value at the connected WritePort. No hassle,
> if the connection is there it gives the most recent data, no data copying
> involved.
Nope. The ring buffers and buffer ports are (for now) two different behaviours.
In the first case, you lose old data and in the second case you lose new data.

Of course, if Peter wants to implement a lock-free ringbuffer, that would be yet
another communication policy (i.e. you can choose between ring buffers, queues
-- the buffers that we have now -- and data port). I agree that the current data
port is simply a ring buffer of size 1.

[PROPOSAL] redesigning the data flow implementation

On Friday 14 November 2008 16:02:46 Sylvain Joyeux wrote:
>
> Unconnected inputs won't be an error.

Ack. But detectable from within the component.

>
> One of the goals of the proposal is to have a sane
> one-writer,multiple-readers behaviour. I want to stress that the first
> implementation will NOT allow multiple writers with one single reader.

Ack. How would you solve (from an API perspective) switching writers at
runtime ?

>
> Nope. The ring buffers and buffer ports are (for now) two different
> behaviours. In the first case, you lose old data and in the second case you
> lose new data.
>
> Of course, if Peter wants to implement a lock-free ringbuffer, that would
> be yet another communication policy (i.e. you can choose between ring
> buffers, queues -- the buffers that we have now -- and data port). I agree
> that the current data port is simply a ring buffer of size 1.

Ack. The receiver needs to set the buffer (or not) policy.

Peter

[PROPOSAL] redesigning the data flow implementation

On Monday 24 November 2008 08:50:15 Peter Soetens wrote:
> On Friday 14 November 2008 16:02:46 Sylvain Joyeux wrote:
> > Unconnected inputs won't be an error.
>
> Ack. But detectable from within the component.
Of course.

> > One of the goals of the proposal is to have a sane
> > one-writer,multiple-readers behaviour. I want to stress that the first
> > implementation will NOT allow multiple writers with one single reader.
>
> Ack. How would you solve (from an API perspective) switching writers at
> runtime ?
Just right now,
Remove old writer.
Add new writer.

I guess that (if there is the need for it) it would be easy to add a
"switch()" call that replaces a current writer (with its policy) with a new
one (with its own policy as well), so that the switch is done atomically.

But I first want a simple and clean interface for just the dataflow. The more
complex stuff can come later on. Even re-adding the ability to have multiple
writers ;-).

[PROPOSAL] redesigning the data flow implementation

Simple and clean interface for just the dataflow, thread-safe, single writer-multiple reader, detect connection status from within component. Great! Looking forward to your proposition!

cheers,
Theo.

[PROPOSAL] redesigning the data flow implementation

On Fri, 14 Nov 2008, Sylvain Joyeux wrote:

>> penny 1:
>> I expect (hope) that the proposed new implementation will resolve the problems
>> I have with making connections. Let me try to explain.
>
>> My reference of how to make connections is Simulink, or rather 20-sim.
>> I select two ports, typically one input and one output, and make a connection.
>> I like to be warned or even punished if I make a connection between 2 outputs
>> etc. In 20-sim, ports are typed, and hence also the type is checked; very useful
>> I think. I expect that multiple input ports can be connected to a single output,
>> that an unconnected output is ok, and that an unconnected input is an error.
> Unconnected inputs won't be an error.
>
>> Now about my problems with Orocos. I have read the section on Setting up Data
>> Flow in the component manual quite often, and still think it is difficult. I do
>> not really understand the utility/concept of 'connectPeers'; where can I read
>> more about it? I try to reuse components, and therefore regularly have to
>> connect two ports that have different names. Unfortunately, the powerful and
>> useful connectPorts(TC* a,TC* b) does not work then, and I make connections
>> using aPort.connectTo(&anotherPort). The bad thing is that in this way the
>> outputport can only have one connection (or am I doing something wrong?). Many
>> times, this has caused mistakes. It seems to me that as far as the proposal is
>> concerned, making a new connection to a WritePort would not disconnect an
>> already consisting connection, right?
> One of the goals of the proposal is to have a sane one-writer,multiple-readers
> behaviour. I want to stress that the first implementation will NOT allow
> multiple writers with one single reader.
>
>> penny 2
>> About buffered/unbuffered reading (what Herman calls ReadPort policies?). If
>> I understand correctly, the proposal talks about copying writer data to a reader
>> port, and then in the case of a buffered port storing it. I would think that
>> a BufferedPort could be functioning like a boost::circular_buffer; the proposed
>> ReadPort ("unbuffered") then rather is a BufferedPort with buffer size 1.
>> I would in addition like to have a 'real' unbuffered ReadPort that just provides
>> a reference to the latest written value at the connected WritePort. No hassle,
>> if the connection is there it gives the most recent data, no data copying
>> involved.
> Nope. The ring buffers and buffer ports are (for now) two different behaviours.
> In the first case, you lose old data and in the second case you lose new data.
>
> Of course, if Peter wants to implement a lock-free ringbuffer, that would be yet
> another communication policy (i.e. you can choose between ring buffers, queues
> -- the buffers that we have now -- and data port). I agree that the current data
> port is simply a ring buffer of size 1.

I think your contributions can be very valuable! Just make sure that you
don't forget that they should work in a component-based setting, and not
just in a single-activity context such as 20Sim; that means that you (we!)
should first think carefully about what semantics to give to the various
users of the port connection functionality: the runtime components should
be able to read and write and not much more; the deployer should be able to
set different policies. Both things should not be mixed...

Herman

[PROPOSAL] redesigning the data flow implementation

I understand and appreciate the difference between the multi-threaded context of Orocos and the single-activity context of 20-sim. I am busy learning about the multi-threaded context, so please forgive my ignorance from time to time..

I can guess what you (Herman) mean by different policies set by the deployer, but I am not really sure if I understand correctly. For sure I do not know the arguments for your position; I would think that it is simpler (easier for the user) to design components with pre-selected port types with a particular read policy. Would you mind explaining to me, preferably by means of a simple example, where I am wrong?

Thanks!

Theo.

[PROPOSAL] redesigning the data flow implementation

On Wed, 26 Nov 2008, t [dot] j [dot] a [dot] devries [..] ... wrote:

> I understand and appreciate the difference between the multi-threaded
> context of Orocos and the single-activity context of 20-sim. I am busy
> learning about the multi-threaded context, so please forgive my ignorance
> from time to time..
Of course you are forgiven! But you should not make the mistake to think
that any of us is omniscient! :-) We are ignorant too, unfortunately, on
too many relevant aspects. Together, we will hopefully be better able to
reduce our common level of non-understanding :-)

(Oops, that was _a lot_ of gibberish management talk... sorry!)

> I can guess what you (Herman) mean by different policies set by the
> deployer, but I am not really sure if I understand correctly. For sure I
> do not know the arguments for your position; I would think that it is
> simpler (easier for the user) to design components with pre-selected port
> types with a particular read policy. Would you mind explaining to me,
> preferably by means of a simple example, where I am wrong?

To start with: one is never "wrong" with respect to policies! By
definition: policies are a way "to impose" a certain behaviour on
components, and what is the "best" behaviour (and hence policy) depends on
the application.
What I _do_ think is a good _design guideline_ is to make sure that the
configuration/selection/imposition of policies is done in the right place:
- a component should _not_ have to know about what DataPort policy is
chosen, because this makes the component too dependent on things that
happen outside of itself. (Hence, this conflicts with the "stable
subsystem" criterion that I want to reach for components.)
- hence, the simple "read" and "write" of DataPorts fits best for the
components. Whether or not the reads and writes will lead to blocking, or
to loss of data, or delays, or whatever, that's up to the _system_ designer
to decide, not the component designer. Practically speaking in Orocos
terms, this means that that is the responsibility of the Deployer.
- a similar reasoning is for the _type_ of the port, as visible to the
component internals: the component should only use its own limited set of
types; when type translation or other data operations have to be done, it's
the task of the _system_ designer to put the appropriate components into
the system.
In summary: keep components as "small scoped" as possible, and do all the
"glueing" at system level, not at component level.

Maybe this doesn't make too much sense...? If so, please keep asking, since
I have to get things straight(er) for myself too :-) (And for Sylvain...)

Herman

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

[PROPOSAL] redesigning the data flow implementation

On Friday 14 November 2008 15:19:07 peter [dot] soetens [..] ... wrote:
> The mail below was accidentally filtered from the orocos-dev mailinglist.
> I'm bumping it for the record (and the good points of attention that it
> raises).
>
> [

vri wrote:

Just my two penny, as an ignorant and unexperienced user.
>
> penny 1:
> I expect (hope) that the proposed new implementation will resolve the
> problems I have with making connections. Let me try to explain. My
> reference of how to make connections is Simulink, or rather 20-sim. I
> select two ports, typically one input and one output, and make a
> connection. I like to be warned or even punished if I make a connection
> between 2 outputs etc.

If you think in terms of 'channels', it is legal that two producers push data
into a channel. If you think in terms of 'signals' (like digital IO or
voltages) connecting 2 outputs is indeed not allowed.

> In 20-sim, ports are typed, and hence also the type
> is checked; very useful I think. I expect that multiple input ports can be
> connected to a single output, that an unconnected output is ok, and that an
> unconnected input is an error.

You should check ports with the .ready() method.

> Now about my problems with Orocos. I have
> read the section on Setting up Data Flow in the component manual quite
> often, and still think it is difficult. I do not really understand the
> utility/concept of 'connectPeers'; where can I read more about it?

'Peers' connections are for execution flow-type communications between
components. That is, if A is peer of B, B can use A's commands, events and
methods. In order to do so, B writes this->getPeer("A")->events()->.... etc.
It has nothing to do with data flow ports.

> I try to
> reuse components, and therefore regularly have to connect two ports that
> have different names. Unfortunately, the powerful and useful
> connectPorts(TC* a,TC* b) does not work then,

This is a known issue and the usefulness of this function has been much
questioned.

> and I make connections using
> aPort.connectTo(&anotherPort). The bad thing is that in this way the
> outputport can only have one connection (or am I doing something wrong?).

I'm afraid you are. You need to write conceptually:
unconnected_port.connectTo( &connected_or_unconnected_port );

So the 'direction' is relevant. Maybe we should drop this constraint ??

> Many times, this has caused mistakes. It seems to me that as far as the
> proposal is concerned, making a new connection to a WritePort would not
> disconnect an already consisting connection, right?

In the current code, any number of readers and writers may be connected.

>
> penny 2
> About buffered/unbuffered reading (what Herman calls ReadPort policies?).
> If I understand correctly, the proposal talks about copying writer data to
> a reader port, and then in the case of a buffered port storing it. I would
> think that a BufferedPort could be functioning like a
> boost::circular_buffer;

That would depend on policy 'drop oldest' or 'drop newest'. Sylvain ?

> the proposed ReadPort ("unbuffered") then rather is
> a BufferedPort with buffer size 1.

In the case of 'drop oldest'. It looks like overkill to use a buffer
implementation here instead of a data object. Also the interface could be
different imho. Do you want to consume the data item or not ? Following
Sylvain's reasoning, (data object has data or not) this could work. For
attaining a uniform interface, your proposal might be a good idea.

> I would in addition like to have a
> 'real' unbuffered ReadPort that just provides a reference to the latest
> written value at the connected WritePort. No hassle, if the connection is
> there it gives the most recent data, no data copying involved.

Good point as well. Especially right after the connection is created. Should
we return the last written value (before the connection was created) on the
write port ? Or must the write port write at least once before the read port
is ready ?

Peter

[PROPOSAL] redesigning the data flow implementation

On Tue, 21 Oct 2008, Sylvain Joyeux wrote:

>> The "hasData()" could be a usefull thing to have too, I guess: only
>> providing "read()" means that you have "consumed" the data to find out
>> whether there _is_ data; while having a "peek" without consumption could be
>> good to have, in some cases...
> Yes. there is a peek() as well :P
>
> I removed hasData() because of one good comment from Peter: it is
> prone to race conditions while "bool peek(sample)" is not.

To my non-informed mind both hasData() and peek() seem to have the same
semantics...?

Herman

[PROPOSAL] redesigning the data flow implementation

On Tue, Oct 21, 2008 at 01:57:39PM +0200, Herman Bruyninckx wrote:
> On Tue, 21 Oct 2008, Sylvain Joyeux wrote:
>
>>> The "hasData()" could be a usefull thing to have too, I guess: only
>>> providing "read()" means that you have "consumed" the data to find out
>>> whether there _is_ data; while having a "peek" without consumption could be
>>> good to have, in some cases...
>> Yes. there is a peek() as well :P
>>
>> I removed hasData() because of one good comment from Peter: it is
>> prone to race conditions while "bool peek(sample)" is not.
>
> To my non-informed mind both hasData() and peek() seem to have the same
> semantics...?

bool hasData();
bool peek(sample_type& sample);

One checks if there is a sample, the other checks and returns it if
there is one. What you could have -- in the most generic case -- is the
following displaying "WHAT".

if (hasData())
{
if (!read(sample))
cerr << "WHAT ?" << endl;
}

Therefore, I think it is actually safer to only use peek() and read()
instead of hasData().

[PROPOSAL] redesigning the data flow implementation

On Tue, 21 Oct 2008, Sylvain Joyeux wrote:

> On Tue, Oct 21, 2008 at 01:57:39PM +0200, Herman Bruyninckx wrote:
>> On Tue, 21 Oct 2008, Sylvain Joyeux wrote:
>>
>>>> The "hasData()" could be a usefull thing to have too, I guess: only
>>>> providing "read()" means that you have "consumed" the data to find out
>>>> whether there _is_ data; while having a "peek" without consumption could be
>>>> good to have, in some cases...
>>> Yes. there is a peek() as well :P
>>>
>>> I removed hasData() because of one good comment from Peter: it is
>>> prone to race conditions while "bool peek(sample)" is not.
>>
>> To my non-informed mind both hasData() and peek() seem to have the same
>> semantics...?
>
> bool hasData();
> bool peek(sample_type& sample);
>
> One checks if there is a sample, the other checks and returns it if
> there is one.

Thanks for the clarification!

> What you could have -- in the most generic case -- is the
> following displaying "WHAT".
>
> if (hasData())
> {
> if (!read(sample))
> cerr << "WHAT ?" << endl;
> }
>
> Therefore, I think it is actually safer to only use peek() and read()
> instead of hasData().
The difference is that the code sample above could be difficult to execute
atomically: the activity could have been interrupted between the two "if"
evaluations, and hence the logic of the code could have been corrupted.

Herman

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

[PROPOSAL] redesigning the data flow implementation

On Tuesday 21 October 2008 17:03:57 Sylvain Joyeux wrote:
> Therefore, I think it is actually safer to only use peek() and read()
> instead of hasData().

Needsless to say, but anyway. I agree 100%.

Peter

[PROPOSAL] redesigning the data flow implementation

On Mon, 20 Oct 2008, Sylvain Joyeux wrote:

> On Mon, Oct 20, 2008 at 03:59:20PM +0200, Herman Bruyninckx wrote:
>> On Mon, 20 Oct 2008, Peter Soetens wrote:
>>
>>> On Monday 20 October 2008 14:16:53 Herman Bruyninckx wrote:
>>>> On Mon, 20 Oct 2008, Peter Soetens wrote:
>>>>> On Monday 20 October 2008 10:57:01 Sylvain Joyeux wrote:
>>>>>> Quite frankly, now that I looked at the RTT data flow implementation,
>>>>>> I'm a bit confused as to what you were trying to achieve. The
>>>>>> implementation is miles away from the model (or, at least, what I
>>>>>> assumed the model was).
>>>>>
>>>>> Ok. I throw in the towel. Your analysis is painfully accurate. We win
>>>>> simplicity and flexibility with your approach. The strong points are:
>>>>>
>>>>> * a single 'send and forget' type write port
>>>>> (note: should we have access to the last written value from
>>>>> scripting/C++?)
>>>>
>>>> I suggest we should _not_ do this, since this is asking for a can of worms
>>>> to be opened. _And_ it is semantically not necessary: if a component really
>>>> wants to know what is in a channel, it should connect to it as reader...
>
> Mmmm ... correcting myself:
>
> I still think that from a semantic cleanliness point of view, Herman is
> right
>
> *But*
>
> It is actually quite easy to implement (only have a data object on the
> writer and you can get the last written value)
>
> *and (more interesting)*
>
> The question of "what to do on new connections" is still open:
> * should we initialize the reader with the last written value
> * or should it wait for a new value to be written ?

These (and other possibilities maybe) are _policies_ that the using
component should be able to select, at deployment time...

All these things are part of the "Quality of Service" discussion that I
think Orocos 2.0 should include, as being an inherent part of "component
based programming" :-)

Herman

[PROPOSAL] redesigning the data flow implementation

On Mon, Oct 20, 2008 at 04:50:18PM +0200, Herman Bruyninckx wrote:
> On Mon, 20 Oct 2008, Sylvain Joyeux wrote:
>
>> On Mon, Oct 20, 2008 at 03:59:20PM +0200, Herman Bruyninckx wrote:
>>> On Mon, 20 Oct 2008, Peter Soetens wrote:
>>>
>>>> On Monday 20 October 2008 14:16:53 Herman Bruyninckx wrote:
>>>>> On Mon, 20 Oct 2008, Peter Soetens wrote:
>>>>>> On Monday 20 October 2008 10:57:01 Sylvain Joyeux wrote:
>>>>>>> Quite frankly, now that I looked at the RTT data flow implementation,
>>>>>>> I'm a bit confused as to what you were trying to achieve. The
>>>>>>> implementation is miles away from the model (or, at least, what I
>>>>>>> assumed the model was).
>>>>>>
>>>>>> Ok. I throw in the towel. Your analysis is painfully accurate. We win
>>>>>> simplicity and flexibility with your approach. The strong points are:
>>>>>>
>>>>>> * a single 'send and forget' type write port
>>>>>> (note: should we have access to the last written value from
>>>>>> scripting/C++?)
>>>>>
>>>>> I suggest we should _not_ do this, since this is asking for a can of worms
>>>>> to be opened. _And_ it is semantically not necessary: if a component really
>>>>> wants to know what is in a channel, it should connect to it as reader...
>>
>> Mmmm ... correcting myself:
>>
>> I still think that from a semantic cleanliness point of view, Herman is
>> right
>>
>> *But*
>>
>> It is actually quite easy to implement (only have a data object on the
>> writer and you can get the last written value)
>>
>> *and (more interesting)*
>>
>> The question of "what to do on new connections" is still open:
>> * should we initialize the reader with the last written value
>> * or should it wait for a new value to be written ?
>
> These (and other possibilities maybe) are _policies_ that the using
> component should be able to select, at deployment time...
Fine, but then the actual port implementation should support it, meaning
it should keep the last written value somewhere.

Of course, it could be done "on-demand", but then I call that
over-engineering the framework.

[PROPOSAL] redesigning the data flow implementation

I fully agree with Herman here. Basically for the following reason:
once all data is managed by the receiving end (i.e. one buffer per
reader), it is actually not accessible at the sending end anymore.

If you want to read data from scripting, then associate a read port to
the corresponding scripting object and read this port. This is already
what is done by the reporting component btw ...

[PROPOSAL] redesigning the data flow implementation

On Monday 20 October 2008 16:12:15 Sylvain Joyeux wrote:
> I fully agree with Herman here. Basically for the following reason:
> once all data is managed by the receiving end (i.e. one buffer per
> reader), it is actually not accessible at the sending end anymore.

I'm not interested in what the receiving end sees. I'm interested in what this
specific sending end writes... even if no other component is connected.
Adding a read port for each write port sounds to much like code bloat to
me :-)

Peter

[PROPOSAL] redesigning the data flow implementation

On Oct 20, 2008, at 08:52 , Peter Soetens wrote:

> On Monday 20 October 2008 14:16:53 Herman Bruyninckx wrote:
>> On Mon, 20 Oct 2008, Peter Soetens wrote:
>>> On Monday 20 October 2008 10:57:01 Sylvain Joyeux wrote:
>>>> Quite frankly, now that I looked at the RTT data flow
>>>> implementation,
>>>> I'm a bit confused as to what you were trying to achieve. The
>>>> implementation is miles away from the model (or, at least, what I
>>>> assumed the model was).
>>>
>>> Ok. I throw in the towel. Your analysis is painfully accurate. We
>>> win
>>> simplicity and flexibility with your approach. The strong points
>>> are:
>>>
>>> * a single 'send and forget' type write port
>>> (note: should we have access to the last written value from
>>> scripting/C++?)
>>
>> I suggest we should _not_ do this, since this is asking for a can
>> of worms
>> to be opened. _And_ it is semantically not necessary: if a
>> component really
>> wants to know what is in a channel, it should connect to it as
>> reader...
>
> What you propose is entirely correct for reading the connection.
> Yet, for
> debugging/taskbrowsing/... purposes, it is convenient to know which
> value the
> current component has sent to its output port (apart from what is
> happening
> at the connection level).
>
> That is one of the reasons why the current WriteDataPort is under-/
> not-used:
> you can only do a Set(...) and have no means to retrieve the last
> written
> value. Hence people replace it with a DataPort which has Set and Get
> during
> debugging and do not revert it back to WriteDataPort lateron.

I agree. We certainly do this regularly.
S

[PROPOSAL] redesigning the data flow implementation

On Mon, Oct 20, 2008 at 01:30:53PM +0200, Peter Soetens wrote:
> On Monday 20 October 2008 10:57:01 Sylvain Joyeux wrote:
> > Quite frankly, now that I looked at the RTT data flow implementation,
> > I'm a bit confused as to what you were trying to achieve. The
> > implementation is miles away from the model (or, at least, what I
> > assumed the model was).
>
> Ok. I throw in the towel. Your analysis is painfully accurate. We win
> simplicity and flexibility with your approach. The strong points are:
>
> * a single 'send and forget' type write port
> (note: should we have access to the last written value from scripting/C++?)
> * Allowing mixed buffered/unbuffered reads from a data flow.
> - makes components more reusable in different applications
> * Putting the buffer size responsibility with the reader
> - allows per receiver tuning.
>
> I'll help integrating your solution into mainline. I'll mostly care about
> preserving the real-timeness and migration path of existing applications.

I'm glad to hear that. I'll try to be as RT-friendly as possible, but I
guess I'll have to do some bad stuff first to get it running.

Sylvain

[PROPOSAL] redesigning the data flow implementation

On Mon, 20 Oct 2008, Sylvain Joyeux wrote:

> Quite frankly, now that I looked at the RTT data flow implementation,
> I'm a bit confused as to what you were trying to achieve. The
> implementation is miles away from the model (or, at least, what I
> assumed the model was).

Thanks for your extensive feedback and suggestions!

I am sure that I do not understand all of your suggestions completely, but
I think that they are in this direction: a DataObject should be a Component
in its own, and readers and writers should indeed not have to care about
what the DataObject does with their data. It's the responsibility for the
deployment to give the DataObjects appropriate policies.

Is this a correct interpretation from my side?

Herman
>
> The problems
> ============
> * the current implementation is not about data connections (getting
> data flowing from one port to another). It is about managing shared
> memory places, where different ports read and write. That is quite
> obvious for the data ports (i.e. there is a shared data sample that
> anyone can read or write), and is IMO completely meaningless for
> buffer ports. Buffer ports are really in need of a data flow model
> (see above a more specific critic about multi-output buffers)
>
> * Per se, this does not seem a problem. Data is getting transmitted
> from one port to the other, isn't it ?
>
> Well, actually it is a problem because it forbids a clean connection
> management implementation. Why ? Because there is no way to know who
> is reading and who is writing ... Thus, the completely useless
> disconnect() call. Why useless ? Because if you do: (this is
> pseudo-code of course)
>
> connect(source, dest)
> source.disconnect()
>
> Then dest.isConnected() returns true, even though dest will not get
> any data from anywhere (there is no writer anymore on that connection).
> This is more general, as it is for instance very difficult to
> implement proper connection management in the CORBA case.
>
> * Because of this connection management issue, it is very difficult to
> implement a "push" model. It leads to huge problems with the CORBA
> transport when wireless is bad, because each pop or get needs a few
> calls.
>
> * It makes the whole implementation a huge mess. There is at least
> twice the number of classes normally needed to implement a connection
> model *and* code is not reused (DataPort is actually *not* a subclass of
> both ReadDataPort and WriteDataPort, same for buffers).
>
> * We already had a long thread about multiple-output buffered
> connections. I'll summarize what for me was the most important
> points:
> - the current implementation allows to distribute workload seamlessly
> between different task contexts.
> - it does not allow to send the same set of samples to different task
> contexts. Ther is a hack allowing to read buffer connections as if
> they were data connections, but it is a hack given that the reader
> cannot know if it is really reading a sample or reading a default
> value because the buffer is empty.
>
> IMO the first case is actually rare in robotic control (and you can
> implement a generic workload-sharing component with nicer features
> like for instance keeping the ordering between input and output) like
> in the following example:
> => A0 A3 [PROCESSING] => A'0 A'3
> A0 A1 A2 A3 => [WORK SHARING WORK SHARING] => A'0 A'1 A'2 A'3
> => A1 A2 [PROCESSING] => A'1 A'2
>
> The second case is much more common. For instance, in my robot, I
> want to have a safety component that monitors a laser scanner
> (near-obstacle detection for the purpose of safety) and the same
> laser scans to go to a SLAM algorithm. I cannot do that for now,
> because I need a buffered connection to the SLAM algorithm. I cannot
> use the aforementionned hack either because for now I plan to put
> a network connection between the scanner driver and the two targets,
> and therefore I cannot really guarantee which component will get
> what.
>
> Proposal
> ========
>
> What I'm proposing is getting back to a good'ol data flow model, namely:
> * making write ports "send and forget". If the port fails to write,
> then it is the problem of the reader ! I really don't see what the writer
> can do about it anyway, given that it does not know what the data
> will be used for (principle of component separation). The reader can
> still detect that its input buffer is full and that it did not get some
> samples and do something about it.
>
> * making write ports "connection-type less". I.e. no WRITE data ports
> and WRITE buffer ports anymore, only write ports. This will allow to
> connect a write port to a read port with any kind of connections.
> Actually, I don't see a use case where the port designer can actually
> decide what kind of connection is best for its OUTPUT ports. Some
> examples:
> - in the laser scanner example above, the safety component would like
> a data port and the slam a buffer port
> - in position filtering, some components just want the latest
> positions and other components all the position stream (for
> interpolation purposes for instance)
> - in general, GUI vs. X. GUIs want most of the time the latest
> values.
> - ... I'm sure I can come up with other examples if you want them
>
> * locating the sample on the read ports (i.e. no ConnectionInterface
> and subclasses anymore). The bad: one copy of each sample per read
> port. The good: you implement the point above (write ports do not
> have a connection type), and you fix buffer connections once and for
> all.
>
> * removing (or deprecating) read/write ports. They really have no place
> in a data flow model.
>
>> From a coding point of view, I can do it. The bottom line being that
> doing that will remove a lot of code and deeply clean up the set of
> classes involved in data flow.
>
> Unless there are compelling reasons not to do so, I'll do it anyway (git
> to the rescue), because I cannot do with the current behaviour. So if
> you are not sure (but not completely against), you will be able to see
> before deciding.
> --
> =======================================================================
> Dr. Ing. Sylvain Joyeux sylvain [dot] joyeux [..] ...
> Researcher
> DFKI Robotik Lab -- Bremen http://www.dfki.de/robotik
> Tel: 0049 421 218 64136
> =======================================================================
>
> --
> Orocos-Dev mailing list
> Orocos-Dev [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>
> Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm
>
>

[PROPOSAL] redesigning the data flow implementation

On Mon, Oct 20, 2008 at 12:44:17PM +0200, Herman Bruyninckx wrote:
> On Mon, 20 Oct 2008, Sylvain Joyeux wrote:
>
>> Quite frankly, now that I looked at the RTT data flow implementation,
>> I'm a bit confused as to what you were trying to achieve. The
>> implementation is miles away from the model (or, at least, what I
>> assumed the model was).
>
> Thanks for your extensive feedback and suggestions!
>
> I am sure that I do not understand all of your suggestions completely, but
> I think that they are in this direction: a DataObject should be a Component
> in its own, and readers and writers should indeed not have to care about
> what the DataObject does with their data. It's the responsibility for the
> deployment to give the DataObjects appropriate policies.
>
> Is this a correct interpretation from my side?
Not really:
* I do think that the type of inbound connection (buffered or not) is
defined by the TaskContext. I.e. some algorithms will want all the
samples while others only the latest one. So it is *not* to be
defined by the deployment.
* for the outbound connection, however, this is true (i.e. the
TaskContext does not even know what the data will be used for, so how
can it decide the kind of connection that is needed ?)