Redesign of the data flow interface

(Copied from http://github.com/doudou/orocos-rtt/commit/dc1947c8c1bdace90cf0a3aa2047ad248619e76b)

  • write ports are now common to all types of connections, and writing is "send and forget"
  • read ports still specify their type (data or buffer). The management of the connection type is offloaded on the port object (i.e. no more an intermediate ConnectionInterface object)
  • the ports maintain a list of "connected" ports. It is therefore possible to do some connection management, i.e. one knows who is listening to what.

Here is the mail that led to this implementation:

The problems

  • the current implementation is not about data connections (getting data flowing from one port to another). It is about managing shared memory places, where different ports read and write. That is quite obvious for the data ports (i.e. there is a shared data sample that anyone can read or write), and is IMO completely meaningless for buffer ports. Buffer ports are really in need of a data flow model (see above a more specific critic about multi-output buffers)
  • Per se, this does not seem a problem. Data is getting transmitted from one port to the other, isn't it ?
Well, actually it is a problem because it forbids a clean connection management implementation. Why ? Because there is no way to know who is reading and who is writing ... Thus, the completely useless disconnect() call. Why useless ? Because if you do: (this is pseudo-code of course)

     connect(source, dest)
     source.disconnect()
Then dest.isConnected() returns true, even though dest will not get any data from anywhere (there is no writer anymore on that connection).

This is more general, as it is for instance very difficult to implement proper connection management in the CORBA case.
  • Because of this connection management issue, it is very difficult to implement a "push" model. It leads to huge problems with the CORBA transport when wireless is bad, because each pop or get needs a few calls.
  • It makes the whole implementation a huge mess. There is at least twice the number of classes normally needed to implement a connection model *and* code is not reused (DataPort is actually *not* a subclass of both ReadDataPort and WriteDataPort, same for buffers).
  • We already had a long thread about multiple-output buffered connections. I'll summarize what for me was the most important points:
    • the current implementation allows to distribute workload seamlessly between different task contexts.
    • it does not allow to send the same set of samples to different task contexts. Ther is a hack allowing to read buffer connections as if they were data connections, but it is a hack given that the reader cannot know if it is really reading a sample or reading a default value because the buffer is empty.
IMO the first case is actually rare in robotic control (and you can implement a generic workload-sharing component with nicer features like for instance keeping the ordering between input and output) like in the following example:

                             => A0 A3 [PROCESSING] => A'0 A'3
 A0 A1 A2 A3 => [WORK SHARING                                  WORK SHARING] => A'0 A'1 A'2 A'3
                             => A1 A2 [PROCESSING] => A'1 A'2
The second case is much more common. For instance, in my robot, I want to have a safety component that monitors a laser scanner (near-obstacle detection for the purpose of safety) and the same laser scans to go to a SLAM algorithm. I cannot do that for now, because I need a buffered connection to the SLAM algorithm. I cannot use the aforementionned hack either because for now I plan to put a network connection between the scanner driver and the two targets, and therefore I cannot really guarantee which component will get what.

Proposal

What I'm proposing is getting back to a good'ol data flow model, namely:

  • making write ports "send and forget". If the port fails to write, then it is the problem of the reader ! I really don't see what the writer can do about it anyway, given that it does not know what the data will be used for (principle of component separation). The reader can still detect that its input buffer is full and that it did not get some samples and do something about it.

  • making write ports "connection-type less". I.e. no WRITE data ports and WRITE buffer ports anymore, only write ports. This will allow to connect a write port to a read port with any kind of connections. Actually, I don't see a use case where the port designer can actually decide what kind of connection is best for its OUTPUT ports. Some examples:
    • in the laser scanner example above, the safety component would like a data port and the slam a buffer port
    • in position filtering, some components just want the latest positions and other components all the position stream (for interpolation purposes for instance)
    • in general, GUI vs. X. GUIs want most of the time the latest values.
    • ... I'm sure I can come up with other examples if you want them
  • locating the sample on the read ports (i.e. no ConnectionInterface and subclasses anymore). The bad: one copy of each sample per read port. The good: you implement the point above (write ports do not have a connection type), and you fix buffer connections once and for all.
  • removing (or deprecating) read/write ports. They really have no place in a data flow model.