Updated dataflow semantics for RTT

Hi Sylvain,

thanks for your quick reply!

On Mon, Sep 21, 2015 at 8:09 PM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...>
wrote:

> Quick comment -- I don't have a lot of time on my hands right now:
>
> To me, it looks like you have decided that push is shared by default,
> in your description the concept of pull/push and private/shared. Am I
> wrong ? Is "push/shared" different than "push with single buffer by
> default" ? I would not see the rationale for that, if a use wants
> "private", give him "private", especially given that there are
> shortcomings ...
>
> Using "shared" will make the system fail if, later, it gets a
> connection request for a different policy than the original policy. To
> me, this spells that using shared MUST be a conscious choice of the
> system designer in all cases OR that there should be a way for RTT to
> degrade gracefully from shared to private (use shared as an
> optimization that is handled purely internally unless the user
> requested it explicitely).
>
> In general, the connection code will have to return an error if trying
> incompatible mixes of policies.
>
> Sylvain
>

That's exactly what he have in mind and how it is hopefully implemented.
push/pull and private/shared are orthogonal concepts. The default would be
push/private. Connections are never shared unless explicitly specified by
the user. Probably the table in section 5.1 of our design document
<https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4KaotpCtXM4E0/pub#h.yfd89u4uuk9u>
is
the easiest mean to get a fast overview of the possible combinations.

These are the shortest possible definitions I can come up with:

push: Every input port has a single input buffer and all connected writers
write into that buffer.
pull: Every output port has one buffer per connection and readers will poll
all connected writers for new data.

private: Every connection buffer belongs to exactly one input port, so all
read operations are independent from each other.
shared: Multiple input ports can read from the same buffer, so one read
operation can influence the outcome of another.

The combination shared/pull is less intuitive, but we interpreted it as a
single local output buffer and multiple polling input ports read from the
same buffer (vs. one buffer per connection in the pull/private case).
Actually this connection type would only be useful for shared remote
connections, where multiple remote input ports are supposed to consume
samples from a single output port with a shared local buffer. For local
connections, there is no reason not to use a shared/push connection.

Clearly there are limitations for the possible combinations of different
connection types for a single port, primarily that an input port can have
either only pull or only push connections and that all push connections
have to be consistent in the buffer type, size and locking policy. For
illegal combinations the connectTo(...) request (or whatever API is used to
establish new connections) will return an error. Users have to be more
careful and aware of the implications of ConnPolicy flags, but we think the
default private/push should work for most applications and not break
existing deployments, unless they already use a mixture of connection
policies for the same port for whatever reason.

Johannes

Updated dataflow semantics for RTT

On Mon, 28 Sep 2015, Stephen Roderick wrote:

>
> On Sep 28, 2015, at 08:42 AM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:
>
> Some thoughts on that executive summary (I'm like geoffrey, I had a
> hard time following the first email :P)
>  
> +1
>
> One thing I definitely want to warn you about: don't think, ever, that
> code-generating everything is a replacement for a well-designed
> library that solve most of the problems, along with a code generator
> for the one percent.
>  
> +1
>
> Also, code generators introduce an additional level of indirection and
> dependency. In some cases, it isn't worth that burden.
>
> With this premise: I actually that your points are not making RTT
> obsolete. It is actually making it even more relevant than ever.
>
> RTT *is* the best starting point towards something where
> process/thread policies is delegated to the tooling (either accessing
> "raw" OS-specific functionality or exposing them under an
> easier-to-use interface). It is also *already* provides an
> encapsulation of computation, which allows you to "inverse" the
> current tendency and actually make your middleware central to your
> communication setup -- instead of hiding it behind a common API.
>  
> +1
>
> Also, RTT is cross-platform. Linux isn't the be all and end all for everyone
> … so for us, systemd (or any other Linux-specific dependency) can't be a
> central part of any proposed replacement.

You're fooling yourself: RTT is a dependency which is a lot more risky than
dependencies that are widely supported and maintained in mainstream IT
developments... :-)

> S

Herman

Updated dataflow semantics for RTT

On Sep 28, 2015, at 10:28 AM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:

I agree with much at the above, and we have similar problems that we've
solved in similar ways. But there are also huge parts of our system that,
while "data driven", aren't at all interested in time. The system itself is
synchronized and time is (largely) irrelevant - everything operates on a
per-cycle basis. In the long run, we time stamp only a tiny portion of the
data in our system due to this. Equally valid, but different.

I don't dispute that you probably have a very good point, you usually
do, but could you provide me with a concrete example of such a setup
that uses buffers ? It would really help me grasp the situation.
 
Some classic producer/consumer examples
 - a camera captures images and sends them on to a processing pipeline (that is all data object connections). A separate connection stores the camera images in a buffer, which are dumped to disk.
- data is read (non-real-time) from a file and pushed into a buffer for processing (real-time)
- data samples coming from a remote source (non-real-time) is buffered on arrival into the deployer for later processing (real-time).

Sometimes we want to control whether the network access (and hence
delay/non-real-time part) is on the writer or reader side. That should be an
application-level choice.

Agreed, but this choice is currently bundled with having a buffer per
channel or one per input port.

I don't follow, Sylvain. Bundled in the existing implementation or the proposed changes?

If you put the buffer per-port, you end up with one policy per port.
Port-driven triggering (event ports) are governed by the data object
(i.e. per-port), data object size and type becomes per-port as well.
What's left in the connection policy is the info necessary to reach
the other side (i.e. the topic name in ROS), which means that it is
not a policy anymore but a "remote endpoint"

What if you don't use event ports?

How is that relevant ? Are we misunderstanding each other here ?
 
From your comment above "Port-driven triggering (event ports) …"

As I understand the proposal so far, we have: either you pull (hit
network on read, one buffer per channel) or push (not hit network on
read, have a one buffer per port). The middle ground of
one-buffer-per-channel and not hitting network is not covered, and
that's what *greatly* bothers me.
 
Our original intent was that pull/push would be independent of shared/private (if you mean that by "one buffer per channel" and "one buffer per port").

I know you guys probably have a good case here, I am trying very hard
for you to not break mine

I'm with you on this, Sylvain!

Note the "selectively remove" above - we still want to be able to use
v2-flow-status type setups within our system.

My understanding of the proposal is that the v2 semantic are not
available anymore. I'm not pushing against what you need, I want you
to not break what I need.
 
I'm personally against changing the default, even though it would help me. This kind of pervasive breakage isn't good for the community ...

 - what are going to be the options of network/no-network and
placement of buffer ?
- how can these options be or not be used with a single output /
single input ? Are they mutually exclusive ? Can they be used together
on the same input port ? On the same output port ?

Looking at Section 5.1 of Johannes' google doc, something looks wrong or I'm remembering previous discussions incorrectly. I was expecting push/pull vs shared/private to be truly orthogonal, but the push/private figure says differently. Johannes? Peter?

In particular, from Section 5.2:

Other than currently in v2, in the push case (pull=false) there will be only one buffer per input port, that is shared among all connections. The connected output ports “push” their data through the channel and fill the input buffer (or overwrite the data object), but every input port has its own buffer or data object and can read independently (Fig. 4)

Why doesn't every input port have a connection object per associated output port? Why isn't the push/private figure the same as the pull/private figure, except that the connection object is on the reader side (and for each input port, you'd have one connection object per output port)? Or is this to try and fix the non-determinstic behavior when an InputPort reads from multiple connection objects?

We have several hundred local connections in our deployers. We have maybe
less than 10 remote connections.

Again, how is this relevant ? If the policy is specified per-port, you
won't have any "kind-of-sane default" to configure the CORBA
dispatchers with. 
Buffers vs data connections definitely change the flavor of things. The vast
majority of our connections are data connections, not buffers.

I *know* which leads to this long discussion, because I am on the
completely other end of things: we mostly use buffered connections.
 
Which is good IMHO, because it means that you and I effectively span the spectrum of system approaches. So if we can find a way to satisfy both our system approaches, then likely we'll cater for everyone else as well.

I think that we're getting off topic as a whole.

I don't think so. So far, my concerns with the proposal are still
there: by getting closer to v1 semantics, the proposal breaks v2
semantics in a way that is fundamental to me. Peter's post actually
leads me to think that this breakage and known and desired, which is
why I am spending so much time trying to get my point across, while I
already have way too much on my plate.
 
Again, my intent was to add more v1-like semantics to the system, as well as have more control on where latency occurred. Not to replace any aspect of the v2 semantics.

a) push vs pull, or specifying whether the data/buffer object is on the
reader or writer side. This is very useful for system designers dealing with
significant time delay.
b) private vs shared, whether a connection is per input/output port pair or
whether the connection is shared between multiple input/output ports.

So far, "private" is defined one buffer *per input port* (change in
semantic) not *per input/output port pair* (current v2 semantic)
 
I think we should postpone the "changing the defaults" discussion. For a v2.9 type release that could include the proposed changes, having the default not change would be of most benefit to the community I think. In the future, if/when we decide that a different set of defaults is more appropriate, we could phase that into future releases in a way that minimizes disruption to the community.

Cheers
Stephen

Updated dataflow semantics for RTT

> Some classic producer/consumer examples
> - a camera captures images and sends them on to a processing pipeline (that
> is all data object connections). A separate connection stores the camera
> images in a buffer, which are dumped to disk.
> - data is read (non-real-time) from a file and pushed into a buffer for
> processing (real-time)
> - data samples coming from a remote source (non-real-time) is buffered on
> arrival into the deployer for later processing (real-time).

Asked the wrong question ... thanks for answering it perfectly ;-)

Anyways, what I wanted is a case where:
- an input port is connected to a bunch of output ports *using
buffers* (I got the problem with data policies)
- the overall behavior is governed by maintaining the overall
incoming sample order (i.e. that if a sample of output port O_1
arrives before a sample of output port O_2, that order needs to be
reflected on read())

I may be missing it, but the use-cases above don't cover that, do they ?

> > Agreed, but this choice is currently bundled with having a buffer per
> > channel or one per input port.
>
> I don't follow, Sylvain. Bundled in the existing implementation or the
> proposed changes?

In the proposed change (current v2 does not have the option of having
a single buffer per input port). Until now, it is clear to me that
this bundling is both Johannes and Peter's intent (see Peter's email
above). It does not seem to be yours, though.

> > If you put the buffer per-port, you end up with one policy per port.
> > Port-driven triggering (event ports) are governed by the data object
> > (i.e. per-port), data object size and type becomes per-port as well.
> > What's left in the connection policy is the info necessary to reach
> > the other side (i.e. the topic name in ROS), which means that it is
> > not a policy anymore but a "remote endpoint"

> From your comment above "Port-driven triggering (event ports) …"

Badly ordered bits of sentences. What I meant is:

> If you put the buffer per-port, you end up with one policy per port. Data object size and type becomes per-port. If using port-driven triggering (event ports), the triggering is governed by the data object and is therefore also becoming per-port.

> Looking at Section 5.1 of Johannes' google doc, something looks wrong or I'm
> remembering previous discussions incorrectly. I was expecting push/pull vs
> shared/private to be truly orthogonal, but the push/private figure says
> differently. Johannes? Peter?

They already clarified that for me. Basically, "private" does not
necessarily means "per channel". On a push connection, "private" for
them means "per-port".

> Again, my intent was to add more v1-like semantics to the system, as well as
> have more control on where latency occurred. Not to replace any aspect of
> the v2 semantics.

I got that from your previous email already, but it seems that it's
not Johannes/Peter's intent

Sylvain

2015-09-28 14:30 GMT-03:00 S Roderick <kiwi [dot] net [..] ...>:
> On Sep 28, 2015, at 10:28 AM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:
>
> I agree with much at the above, and we have similar problems that we've
>
> solved in similar ways. But there are also huge parts of our system that,
>
> while "data driven", aren't at all interested in time. The system itself is
>
> synchronized and time is (largely) irrelevant - everything operates on a
>
> per-cycle basis. In the long run, we time stamp only a tiny portion of the
>
> data in our system due to this. Equally valid, but different.
>
>
> I don't dispute that you probably have a very good point, you usually
> do, but could you provide me with a concrete example of such a setup
> that uses buffers ? It would really help me grasp the situation.
>
>
> Some classic producer/consumer examples
> - a camera captures images and sends them on to a processing pipeline (that
> is all data object connections). A separate connection stores the camera
> images in a buffer, which are dumped to disk.
> - data is read (non-real-time) from a file and pushed into a buffer for
> processing (real-time)
> - data samples coming from a remote source (non-real-time) is buffered on
> arrival into the deployer for later processing (real-time).
>
> Sometimes we want to control whether the network access (and hence
>
> delay/non-real-time part) is on the writer or reader side. That should be an
>
> application-level choice.
>
>
> Agreed, but this choice is currently bundled with having a buffer per
> channel or one per input port.
>
>
> I don't follow, Sylvain. Bundled in the existing implementation or the
> proposed changes?
>
> If you put the buffer per-port, you end up with one policy per port.
>
> Port-driven triggering (event ports) are governed by the data object
>
> (i.e. per-port), data object size and type becomes per-port as well.
>
> What's left in the connection policy is the info necessary to reach
>
> the other side (i.e. the topic name in ROS), which means that it is
>
> not a policy anymore but a "remote endpoint"
>
>
>
> What if you don't use event ports?
>
>
> How is that relevant ? Are we misunderstanding each other here ?
>
>
> From your comment above "Port-driven triggering (event ports) …"
>
> As I understand the proposal so far, we have: either you pull (hit
> network on read, one buffer per channel) or push (not hit network on
> read, have a one buffer per port). The middle ground of
> one-buffer-per-channel and not hitting network is not covered, and
> that's what *greatly* bothers me.
>
>
> Our original intent was that pull/push would be independent of
> shared/private (if you mean that by "one buffer per channel" and "one buffer
> per port").
>
> I know you guys probably have a good case here, I am trying very hard
> for you to not break mine
>
>
> I'm with you on this, Sylvain!
>
> Note the "selectively remove" above - we still want to be able to use
>
> v2-flow-status type setups within our system.
>
>
> My understanding of the proposal is that the v2 semantic are not
> available anymore. I'm not pushing against what you need, I want you
> to not break what I need.
>
>
> I'm personally against changing the default, even though it would help me.
> This kind of pervasive breakage isn't good for the community ...
>
> - what are going to be the options of network/no-network and
> placement of buffer ?
>
> - how can these options be or not be used with a single output /
> single input ? Are they mutually exclusive ? Can they be used together
> on the same input port ? On the same output port ?
>
>
> Looking at Section 5.1 of Johannes' google doc, something looks wrong or I'm
> remembering previous discussions incorrectly. I was expecting push/pull vs
> shared/private to be truly orthogonal, but the push/private figure says
> differently. Johannes? Peter?
>
> In particular, from Section 5.2:
>
> Other than currently in v2, in the push case (pull=false) there will be only
> one buffer per input port, that is shared among all connections. The
> connected output ports “push” their data through the channel and fill the
> input buffer (or overwrite the data object), but every input port has its
> own buffer or data object and can read independently (Fig. 4)
>
> Why doesn't every input port have a connection object per associated output
> port? Why isn't the push/private figure the same as the pull/private figure,
> except that the connection object is on the reader side (and for each input
> port, you'd have one connection object per output port)? Or is this to try
> and fix the non-determinstic behavior when an InputPort reads from multiple
> connection objects?
>
> We have several hundred local connections in our deployers. We have maybe
>
> less than 10 remote connections.
>
>
> Again, how is this relevant ? If the policy is specified per-port, you
> won't have any "kind-of-sane default" to configure the CORBA
> dispatchers with.
>
> Buffers vs data connections definitely change the flavor of things. The vast
>
> majority of our connections are data connections, not buffers.
>
>
> I *know* which leads to this long discussion, because I am on the
> completely other end of things: we mostly use buffered connections.
>
>
> Which is good IMHO, because it means that you and I effectively span the
> spectrum of system approaches. So if we can find a way to satisfy both our
> system approaches, then likely we'll cater for everyone else as well.
>
> I think that we're getting off topic as a whole.
>
>
> I don't think so. So far, my concerns with the proposal are still
> there: by getting closer to v1 semantics, the proposal breaks v2
> semantics in a way that is fundamental to me. Peter's post actually
> leads me to think that this breakage and known and desired, which is
> why I am spending so much time trying to get my point across, while I
> already have way too much on my plate.
>
>
> Again, my intent was to add more v1-like semantics to the system, as well as
> have more control on where latency occurred. Not to replace any aspect of
> the v2 semantics.
>
> a) push vs pull, or specifying whether the data/buffer object is on the
>
> reader or writer side. This is very useful for system designers dealing with
>
> significant time delay.
>
> b) private vs shared, whether a connection is per input/output port pair or
>
> whether the connection is shared between multiple input/output ports.
>
>
> So far, "private" is defined one buffer *per input port* (change in
> semantic) not *per input/output port pair* (current v2 semantic)
>
>
> I think we should postpone the "changing the defaults" discussion. For a
> v2.9 type release that could include the proposed changes, having the
> default not change would be of most benefit to the community I think. In the
> future, if/when we decide that a different set of defaults is more
> appropriate, we could phase that into future releases in a way that
> minimizes disruption to the community.
>
> Cheers
> Stephen

Updated dataflow semantics for RTT

On Sep 28, 2015, at 08:31 AM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:

(@Stephen: re-sending, mistakenly sent to you only instead of all the
involved parties)

Enforcing in order arrive on the reading side is one way of doing it. You
can also do it at a system level if you chose. They're equally valid use
cases.

Can you describe a use case where "order of arrival" is good enough
for you ? That would help me understand your use-case.

(For some clearer thoughts on this,
http://rock-robotics.org/master/documentation/data_processing/data.html
 
I agree with much at the above, and we have similar problems that we've solved in similar ways. But there are also huge parts of our system that, while "data driven", aren't at all interested in time. The system itself is synchronized and time is (largely) irrelevant - everything operates on a per-cycle basis. In the long run, we time stamp only a tiny portion of the data in our system due to this. Equally valid, but different. 

http://rock-robotics.org/master/documentation/data_processing/stream_ali...)

Except for small parts of our system/application, we don't need to worry about this. 

orogen only deals with single components. Syskit is the one inmpacted by
this change, and it is for me out of the picture to compute all possible
connections in all possible networks. Pull connections are not practical as
soon as you have a inter-host connection (i.e. remote connection) as they
break the reading part.

Can you explain this last part a little more Sylvain?

Reading on remote+pull CORBA connections requires the reader to do
network access (it's actually the whole use-case of pull for us, that
the writer is not at all affected by the connection)
 
Sometimes we want to control whether the network access (and hence delay/non-real-time part) is on the writer or reader side. That should be an application-level choice.

Isn't the current policy per connection, and not per port? If you have 2
output ports and 3 input ports, and you connect them all together, then you
have up to 2*3=6 possible connections/policies. Right?

If you put the buffer per-port, you end up with one policy per port.
Port-driven triggering (event ports) are governed by the data object
(i.e. per-port), data object size and type becomes per-port as well.
What's left in the connection policy is the info necessary to reach
the other side (i.e. the topic name in ROS), which means that it is
not a policy anymore but a "remote endpoint"
 
What if you don't use event ports? 

This issue for us boils down to the ability to selectively remove the connection-object-per-input-output-port-pair issue. As a system designer I want to be able to
- specify one connection object between multiple input/output ports (1-N or N-1 or N-N)
- specify where the connection object lives (e.g. to cope with time delay at the system level)

Note the "selectively remove" above - we still want to be able to use v2-flow-status type setups within our system.

Interestingly, you'll have a hard time figuring out how to configure
the CORBA dispatcher (which currently uses the same policy than the
whole connection).
 
We have several hundred local connections in our deployers. We have maybe less than 10 remote connections.

What do you mean by "dynamic" networks?

That are not established once, but are modified at runtime (i.e.
connections are created/removed at runtime). With syskit, that happens
often and computing the network necessary to run them all is in
principle feasible, but in practice useless -- at the least in the
development phases.
 
Got it. Thanks.

> Buffer connection: read() returns all samples received by the input port
> (apart from buffer size limitations).

Well, I'd phrase this differently. A buffer allows more than one sample to
exist, in an ordered form, between a writer and a reader. "All" samples can
only be "guaranteed" with very particular buffer sizing or additional
application semantics.

+1, closer to the current semantic.
 
Buffers vs data connections definitely change the flavor of things. The vast majority of our connections are data connections, not buffers. We typically only use buffers in the traditional producer/consumer type situation - usually to cope with processing (e.g. real-time-ness) differences between producer and consumer (e.g. sample something on one side and write it to disk on the other).

I think that we're getting off topic as a whole. Johannes proposed some modifications to cope with some known problems with the existing implementation that directly affect some of the community. There doesn't seem to be any dissent that something needs to be done about certain issues with using the current implementation. Does anyone have a concrete alternative to Johannes current approach? Certainly we can discuss how to best phase in (i.e. transition) the proposed changes, or something similar to them.

Johannes' proposed changes are focussed on
a) push vs pull, or specifying whether the data/buffer object is on the reader or writer side. This is very useful for system designers dealing with significant time delay. 
b) private vs shared, whether a connection is per input/output port pair or whether the connection is shared between multiple input/output ports.
c) mandatory, primarily intended to support knowing whether writes to a buffer succeeded (so that traditional producer/consumer relationships have some knowledge about the buffer state)

It appears to me that most people are worried about changing the defaults, more than they're worried about the actual proposed changes. Actual technical concerns with the proposed changes appear to be

1) whether to encode the result of a write() in existing enums or not
2) how to add the "shared" concept within the current API, or to use a "new" API
3) changing the RTT defaults, which then affect an application's semantics.

Did I miss any technical issues?

My huge personal caveat to all of this is that I do *not* think we should change the current system semantics in introducing this change. Too many Orocos changes have broken backwards compatibility and forced system designers and implementors to change their system in response. I don't think we should force that on users without an incredibly good reason. Otherwise it breeds frustration in the community.

If there is some way to introduce the proposed changes to the system, but to not change the behavior of current systems, then I think that is a win for all. There does seem to be some general agreement that there are valid use cases that the current port implementation handles poorly, and that this proposed approach does fix some of those. The proposed approach also improves the situation for systems that don't want or need to use the flow status approach. Both are equally valid approaches - IMHO it would be beneficial to the community as a whole if RTT supported both approaches.

Cheers
Stephen

Updated dataflow semantics for RTT

Hi,

thanks for all your input and ideas and I agree with Herman that the
"one-framework-for-everything" approach of RTT is probably not how you
would start a new project nowadays. But I would like to support Stephen's
call to focus this discussion on the technical aspects of how to improve
the existing implementation in as small as possible ("incremental") steps
that add support for some new use cases that are not covered yet (but have
been partially covered by v1).

Maintaining backwards compatibility is definitely the way to go and we can
conclude that the new default semantics (especially per-connection vs.
per-input buffers) have to be the same as in previous v2 releases. I hope
you find it acceptable that a new version would certainly be not
ABI-compatible and breaks some internal API calls that are typically not
used directly by a user. This might break more advanced things like custom
transport implementations. I fear that these kind of changes cannot be
avoided completely, especially for the ConnFactory and ChannelElement
classes.

We should not limit the use cases of RTT on the core API level, like
enforcing per-input storage objects for data connections and per-connection
storage for buffer connections, but only add new use cases. It is more a
question of having good documentation and choosing reasonable defaults that
cover as many applications as possible, but at the end the application
designer has to have the freedom to deploy the application with whatever
connection semantics it demands for.

In the following I try to wrap up and comment on some of the concerns that
came up during the discussion so far, and suggest compromise solutions
which have been at least partially discussed with Peter:

*1) There is definitely a need for per-input port buffers AND for
per-connection buffers - INDEPENDENT of whether the data/buffer object is
installed on the reader and writer side.*

Indeed, according to the original proposal the questions of at what side of
a remote connection the data object will be installed and whether there
will be a per-input or a per-connection buffer cannot specified separately.
Peter and I could not imagine why you would want separate buffers if the
data is already available locally anyways (in the push case) and we still
think that at least for data connections the current per-connection
implementation can be considered as broken. Even with readNewest() there is
no guarantee that you get the latest sample written. On the other hand,
Sylvain correctly stated that there are no guarantees anyway as soon as
remote connections (or any non-realtime transport) are considered and the
only true solution is proper timestamping and synchronization. But I think
we agree that for most deployments with only local connections the
assumption that the latest sample is the last written sample holds and that
timestamping is beyond the scope of RTT and can be left to user code or
more advanced tools like the Rock stream aligner.

Even with the proposed changes (let's call them v3 semantics as a working
title) the connection policy is still primarily describing the connection,
even if *some* settings limit the set of allowed policies for other
connections, making *some* other settings de-facto a per-port setting. This
would be primarily true for input ports and not at all for output ports,
unless it has a connection of the new shared type. But it is not an option
to associate the connection policy *only* with a port because not *all*
possible connections imply per-port buffers. Specifying a connection policy
on a writer/reader pair base as it is now and let certain invalid
connection attempts fail is still the most general approach.

*Proposal:* In order to not break existing applications, we introduce a new
ConnPolicy field "read_policy" that toggles between per-input and
per-connection buffers, instead of implying that by the existing "pull"
flag for remote connections. The read_policy would be an enum with two
possible values for now, "SingleInputBuffer" and "PreferCurrentConnection".
The reason why we propose an enum here is that you could also think of
other policies on how to select the next channel to read from in the
per-connection case, e.g. RoundRobin, PreferLastSignaled, etc. The current
v2 default would correspond to PreferCurrentConnection, as the channel that
was last read with new data is preferred by the input port and only then it
polls all other connections in the order they have been made. We have not
thought this idea to the end yet and the names of the constants are still
under discussion. We do not want to call the input buffers "shared",
because the read policy is independent of the new concept of a "shared
connection" where multiple input ports read from the same buffer. Or
someone can come up with a better name for this type of connection...

*2) Should the new per-input port buffers become the new default or not
(SingleInputBuffer read policy)?*

With that respect the conclusion is that backwards compatibility is more
important than the new "features" and the default parameters for the
current ConnPolicy class should not be touched. However, there are
different ways to tackle this:

2.1 Make it a compile-time option.

2.2 Only add *new* fields to the ConnPolicy class without changing the
default parameter values in the constructor and static methods, and a new
class ConnPolicy2 or ConnPolicy3 that inherits from ConnPolicy and might
have other default parameters or another C++ API (I am thinking of a
named-parameter <https://isocpp.org/wiki/faq/ctors#named-parameter-idiom>
based API) and can also be used for scripting. Internally, the connection
factory would work with const references to the base class only.

2.3 We add a DefaultConnPolicy attribute to the GlobalsRepository. This
application-wide (per process) policy instance will be copied to newly
constructed ConnPolicy instances and the constructors and static methods
only overwrite the respective parameters.

It might be obsoleted by the above proposal, but in the meantime I added a
compile-time option RTT_V2_COMPATIBILITY_MODE to my working branch (see
ef99f2b
<https://github.com/orocos-toolchain/rtt/commit/ef99f2bfa2c01f17431f5a319f117018081ffd91>),
that,
if set, reverts to the current v2 per-connection model, but lets RTT print
meaningful warning messages in the case a new connection would be invalid
with v3 semantics because of conflicts with other connections to the same
port. Such a flag could smooth the transition phase and help application
designers to find the minimal set of connections that require connection
policy changes.

There is one small exception from the backwards-compatibility rule, that I
think is worth discussing: The static method

static ConnPolicy data(int lock_policy = LOCK_FREE, bool init_connection =
true, bool pull = false);

initializes the init_connection field to true by default, which was kind of
surprising me, because it means that every new connection to an output port
with this policy will overwrite a sample previously written by another
port. Is this what we really want as a default? It is also not consistent
with the constructor called as

ConnPolicy(DATA)

which would initialize the init_connection flag to false. The static method
data(...) is only used in the
OutputPortInterface::createDataConnection(...) method from within RTT and
OCL itself.

By the way, the init_connection flag is a good example where the API offers
some degree of freedom to the application builder that semantically does
not make sense at all for certain combinations, like in combination with a
buffer connection. It is still not forbidden to set the flag, but it might
be worth a warning log message if a connection is created like this.

*3) How to add the "shared" concept within the current API, or to use a
"new" API?*

This point was brought up by Janosch, who asked for "a more explicit
interface for shared connections", with the main argument that it is
fundamentally different from a normal connection and a "connect port A to B
with policy C" kind of API does not fit well to that concept. That's true.
We considered to use the createStream(...) API first, with a name_id string
that identifies the shared buffer to connect to. This would actually fit
better to the shared connection model, but it still enforces the same
buffer type policies for all connections. We withdraw that idea because the
existing API is more general and disallowing certain policy combinations is
required anyway for private push connections with a single input buffer. At
least in this case it cannot break existing applications. I do not think
that there is a need for a completely new API only for shared connections.

My current implementation in the updated-dataflow-semantics branch
<https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflow-semantics>
is
as follows, and new/other ideas are always welcome:

- Shared connection objects (the shared data object or buffered) are added
to a process-wide registry. Every shared connection is identified by a
unique string, either set explictly in the name_id field of the ConnPolicy
or assigned automatically, in which case the name of the connection is
returned in exactly this field (name_id is and always was declared as
mutable).
- For new connections with the name_id set, the shared connection is looked
up in the registry, or created as new. As a consequence, if the same
ConnPolicy instance is used to make a second connection, even if name_id
was not set explicitly before, you will always connect to the same
connection object.
- For new connections with no name_id set, the port objects provide an API
to return a pointer to an existing shared connection if they are already
connected to an existing one, and only the non-connected port will be
added. So ports can be connected in any order to each other, as long as
there is always one of the two ports already connected. Otherwise the
name_id has to be set or a new connection instance would be created.
Perhaps this one could be dropped because it it too implicit and does not
work very well for remote connections...
- If an existing connection was found, the buffer policies (type, size and
locking policy) have to match.
- All ports will only store a single pointer to the shared connection
instance in their connection list (in internal::ConnectionManager) and
cannot know directly which other ports are connected.
- For remote input ports with a shared connection, only a proxy is created
locally and can be looked up in the shared connection repository. Other
ports will only connect to either the proxy or the real shared connection
instance, but never build a new remote channel once the proxy has been
created for the first remote connection (works in both directions).
- All connect and disconnect calls to shared connections are logged with
log level Debug to the RTT logger, so at least there is a manual way to
check whether the resulting connections have been made as expected.
Graphical tools like the rtt_dot_service
<https://github.com/orocos-toolchain/rtt_dot_service> could be patched
later to visualize shared connections correctly.

*4) Whether to encode the result of a write() in existing enums or not?*

Janosch asked for introducing a separate enum for the return value of
write(...) calls, in order to not break or trigger compiler warnings for
existing code that already uses the FlowStatus enum in a switch statement.
Another minor issue with a single enum is the default value of FlowStatus
attributes, which can be only meaningful for one use case.

That's also fine for me and I will update the document and implementation
accordingly (add a new enum WriteStatus). Note that the evaluation in
boolean context is counter-intuitive in this case, other than for
FlowStatus, because the value WriteSuccess (=0) value evaluates to false
and all error cases (WriteFailure and NotConnected) evaluate to true.

I hope I did not forget anything. This mail already became much longer than
expected. I am sorry.

I also did some performance measurements of read and write calls last week
in terms of absolute time, CPU time, number of assignments and so on of the
current, the new and even the v1 data flow implementations in different
scenarios. I plan to come up with some results during this week. The tests
revealed some bugs and performance caveats, and some of them are already
present since the early days of RTT v2, e.g. that data samples might be
lost (never read as NewData) because the read flag is not lock-protected or
stored within the lock-free data structure. I am preparing a pull request
to the current master branch, but it will also break the API of class
base::DataObjectInterface.

Best regards,
Johannes

On Mon, Sep 28, 2015 at 3:36 PM, S Roderick <kiwi [dot] net [..] ...> wrote:

>
> Johannes' proposed changes are focussed on
> a) push vs pull, or specifying whether the data/buffer object is on the
> reader or writer side. This is very useful for system designers dealing
> with significant time delay.
> b) private vs shared, whether a connection is per input/output port pair
> or whether the connection is shared between multiple input/output ports.
> c) mandatory, primarily intended to support knowing whether writes to a
> buffer succeeded (so that traditional producer/consumer relationships have
> some knowledge about the buffer state)
>
> It appears to me that most people are worried about changing the defaults,
> more than they're worried about the actual proposed changes. Actual
> technical concerns with the proposed changes appear to be
>
> 1) whether to encode the result of a write() in existing enums or not
> 2) how to add the "shared" concept within the current API, or to use a
> "new" API
> 3) changing the RTT defaults, which then affect an application's semantics.
>
> Did I miss any technical issues?
>
>
> My huge personal caveat to all of this is that I do *not* think we should
> change the current system semantics in introducing this change. Too many
> Orocos changes have broken backwards compatibility and forced system
> designers and implementors to change their system in response. I don't
> think we should force that on users without an incredibly good reason.
> Otherwise it breeds frustration in the community.
>
> If there is some way to introduce the proposed changes to the system, but
> to not change the behavior of current systems, then I think that is a win
> for all. There does seem to be some general agreement that there are valid
> use cases that the current port implementation handles poorly, and that
> this proposed approach does fix some of those. The proposed approach also
> improves the situation for systems that don't want or need to use the flow
> status approach. Both are equally valid approaches - IMHO it would be
> beneficial to the community as a whole if RTT supported both approaches.
>
> Cheers
> Stephen
>
>
>
> --
> Orocos-Dev mailing list
> Orocos-Dev [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>
>

Updated dataflow semantics for RTT

On Mon, 28 Sep 2015, Johannes Meyer wrote:

> Hi,
> thanks for all your input and ideas and I agree with Herman that the "one-framework-for-everything" approach of RTT
> is probably not how you would start a new project nowadays.

Some inputs from a discussion on the ROS mailinglist which is very much
related to this RTT-centric thread:
<http://lists.ros.org/pipermail/ros-users/2015-September/069663.html>
<http://lists.ros.org/pipermail/ros-users/2015-September/069664.html>

These are the critical remarks of two experienced people towards the
"one size fits all" approach of communication in ROS2.0; at the same time,
they also provide concrete use cases and trade-offs that are relevant for
the envisaged update of comunication in RTT...

Herman

PS I am of course more than a bit biased: both messages reflect also my
personal view on this issue of "communication middleware", and are not just
random samples from that ROS thread.

PS2 Geoff Biggs is another RTT-knowledgeable person who _did_ intervene in
that ROS thread, as well as in this RTT thread. Are there others on this
list who, like Geoff and myself, are following both threads? Because,
together, we might "mine" both threads to extract very interesting inputs
from potential users, with widely varying expectations and contexts...

> But I would like to support Stephen's call to focus this
> discussion on the technical aspects of how to improve the existing implementation in as small as possible
> ("incremental") steps that add support for some new use cases that are not covered yet (but have been partially
> covered by v1).

> Maintaining backwards compatibility is definitely the way to go and we can conclude that the new default semantics
> (especially per-connection vs. per-input buffers) have to be the same as in previous v2 releases. I hope you find it
> acceptable that a new version would certainly be not ABI-compatible and breaks some internal API calls that are
> typically not used directly by a user. This might break more advanced things like custom transport implementations.
> I fear that these kind of changes cannot be avoided completely, especially for the ConnFactory and ChannelElement
> classes.
>
> We should not limit the use cases of RTT on the core API level, like enforcing per-input storage objects for data
> connections and per-connection storage for buffer connections, but only add new use cases. It is more a question of
> having good documentation and choosing reasonable defaults that cover as many applications as possible, but at the
> end the application designer has to have the freedom to deploy the application with whatever connection semantics it
> demands for.
>
> In the following I try to wrap up and comment on some of the concerns that came up during the discussion so far, and
> suggest compromise solutions which have been at least partially discussed with Peter:
>
> 1) There is definitely a need for per-input port buffers AND for per-connection buffers - INDEPENDENT of whether the
> data/buffer object is installed on the reader and writer side.
>
> Indeed, according to the original proposal the questions of at what side of a remote connection the data object will
> be installed and whether there will be a per-input or a per-connection buffer cannot specified separately. Peter and
> I could not imagine why you would want separate buffers if the data is already available locally anyways (in the
> push case) and we still think that at least for data connections the current per-connection implementation can be
> considered as broken. Even with readNewest() there is no guarantee that you get the latest sample written. On the
> other hand, Sylvain correctly stated that there are no guarantees anyway as soon as remote connections (or any
> non-realtime transport) are considered and the only true solution is proper timestamping and synchronization. But I
> think we agree that for most deployments with only local connections the assumption that the latest sample is the
> last written sample holds and that timestamping is beyond the scope of RTT and can be left to user code or more
> advanced tools like the Rock stream aligner.
>
> Even with the proposed changes (let's call them v3 semantics as a working title) the connection policy is still
> primarily describing the connection, even if *some* settings limit the set of allowed policies for other
> connections, making *some* other settings de-facto a per-port setting. This would be primarily true for input ports
> and not at all for output ports, unless it has a connection of the new shared type. But it is not an option to
> associate the connection policy *only* with a port because not *all* possible connections imply per-port buffers.
> Specifying a connection policy on a writer/reader pair base as it is now and let certain invalid connection attempts
> fail is still the most general approach.
>
> Proposal: In order to not break existing applications, we introduce a new ConnPolicy field "read_policy" that
> toggles between per-input and per-connection buffers, instead of implying that by the existing "pull" flag for
> remote connections. The read_policy would be an enum with two possible values for now, "SingleInputBuffer" and
> "PreferCurrentConnection". The reason why we propose an enum here is that you could also think of other policies on
> how to select the next channel to read from in the per-connection case, e.g. RoundRobin, PreferLastSignaled, etc.
> The current v2 default would correspond to PreferCurrentConnection, as the channel that was last read with new data
> is preferred by the input port and only then it polls all other connections in the order they have been made.
> We have not thought this idea to the end yet and the names of the constants are still under discussion. We do not
> want to call the input buffers "shared", because the read policy is independent of the new concept of a "shared
> connection" where multiple input ports read from the same buffer. Or someone can come up with a better name for this
> type of connection...
>
>
> 2) Should the new per-input port buffers become the new default or not (SingleInputBuffer read policy)?
>
> With that respect the conclusion is that backwards compatibility is more important than the new "features" and the
> default parameters for the current ConnPolicy class should not be touched. However, there are different ways to
> tackle this:
>
> 2.1 Make it a compile-time option.
>
> 2.2 Only add *new* fields to the ConnPolicy class without changing the default parameter values in the constructor
> and static methods, and a new class ConnPolicy2 or ConnPolicy3 that inherits from ConnPolicy and might have other
> default parameters or another C++ API (I am thinking of a named-parameter based API) and can also be used for
> scripting. Internally, the connection factory would work with const references to the base class only.
>
> 2.3 We add a DefaultConnPolicy attribute to the GlobalsRepository. This application-wide (per process) policy
> instance will be copied to newly constructed ConnPolicy instances and the constructors and static methods only
> overwrite the respective parameters.
>
> It might be obsoleted by the above proposal, but in the meantime I added a compile-time option
> RTT_V2_COMPATIBILITY_MODE to my working branch (see ef99f2b), that, if set, reverts to the current v2 per-connection
> model, but lets RTT print meaningful warning messages in the case a new connection would be invalid with v3
> semantics because of conflicts with other connections to the same port. Such a flag could smooth the transition
> phase and help application designers to find the minimal set of connections that require connection policy changes.
>
> There is one small exception from the backwards-compatibility rule, that I think is worth discussing:  The static
> method
>
> static ConnPolicy data(int lock_policy = LOCK_FREE, bool init_connection = true, bool pull = false);
>
> initializes the init_connection field to true by default, which was kind of surprising me, because it means that
> every new connection to an output port with this policy will overwrite a sample previously written by another port.
> Is this what we really want as a default? It is also not consistent with the constructor called as
>
> ConnPolicy(DATA)
>
> which would initialize the init_connection flag to false. The static method data(...) is only used in the
> OutputPortInterface::createDataConnection(...) method from within RTT and OCL itself.
>
> By the way, the init_connection flag is a good example where the API offers some degree of freedom to the
> application builder that semantically does not make sense at all for certain combinations, like in combination with
> a buffer connection. It is still not forbidden to set the flag, but it might be worth a warning log message if a
> connection is created like this.
>
>
> 3) How to add the "shared" concept within the current API, or to use a "new" API?
>
> This point was brought up by Janosch, who asked for "a more explicit interface for shared connections", with the
> main argument that it is fundamentally different from a normal connection and a "connect port A to B with policy C"
> kind of API does not fit well to that concept. That's true. We considered to use the createStream(...) API first,
> with a name_id string that identifies the shared buffer to connect to. This would actually fit better to the shared
> connection model, but it still enforces the same buffer type policies for all connections. We withdraw that idea
> because the existing API is more general and disallowing certain policy combinations is required anyway for private
> push connections with a single input buffer. At least in this case it cannot break existing applications. I do not
> think that there is a need for a completely new API only for shared connections.
>
> My current implementation in the updated-dataflow-semantics branch is as follows, and new/other ideas are always
> welcome:
>
> - Shared connection objects (the shared data object or buffered) are added to a process-wide registry. Every shared
> connection is identified by a unique string, either set explictly in the name_id field of the ConnPolicy or assigned
> automatically, in which case the name of the connection is returned in exactly this field (name_id is and always was
> declared as mutable).
> - For new connections with the name_id set, the shared connection is looked up in the registry, or created as new.
> As a consequence, if the same ConnPolicy instance is used to make a second connection, even if name_id was not set
> explicitly before, you will always connect to the same connection object.
> - For new connections with no name_id set, the port objects provide an API to return a pointer to an existing shared
> connection if they are already connected to an existing one, and only the non-connected port will be added. So ports
> can be connected in any order to each other, as long as there is always one of the two ports already connected.
> Otherwise the name_id has to be set or a new connection instance would be created. Perhaps this one could be dropped
> because it it too implicit and does not work very well for remote connections...
> - If an existing connection was found, the buffer policies (type, size and locking policy) have to match.
> - All ports will only store a single pointer to the shared connection instance in their connection list (in
> internal::ConnectionManager) and cannot know directly which other ports are connected.
> - For remote input ports with a shared connection, only a proxy is created locally and can be looked up in the
> shared connection repository. Other ports will only connect to either the  proxy or the real shared connection
> instance, but never build a new remote channel once the proxy has been created for the first remote connection
> (works in both directions).
> - All connect and disconnect calls to shared connections are logged with log level Debug to the RTT logger, so at
> least there is a manual way to check whether the resulting connections have been made as expected. Graphical tools
> like the rtt_dot_service could be patched later to visualize shared connections correctly.
>
>
> 4) Whether to encode the result of a write() in existing enums or not?
>
> Janosch asked for introducing a separate enum for the return value of write(...) calls, in order to not break or
> trigger compiler warnings for existing code that already uses the FlowStatus enum in a switch statement. Another
> minor issue with a single enum is the default value of FlowStatus attributes, which can be only meaningful for one
> use case.
>
> That's also fine for me and I will update the document and implementation accordingly (add a new enum WriteStatus).
> Note that the evaluation in boolean context is counter-intuitive in this case, other than for FlowStatus, because
> the value WriteSuccess (=0) value evaluates to false and all error cases (WriteFailure and NotConnected) evaluate to
> true.
>
>
> I hope I did not forget anything. This mail already became much longer than expected. I am sorry.
>
> I also did some performance measurements of read and write calls last week in terms of absolute time, CPU time,
> number of assignments and so on of the current, the new and even the v1 data flow implementations in different
> scenarios. I plan to come up with some results during this week. The tests revealed some bugs and performance
> caveats, and some of them are already present since the early days of RTT v2, e.g. that data samples might be lost
> (never read as NewData) because the read flag is not lock-protected or stored within the lock-free data structure. I
> am preparing a pull request to the current master branch, but it will also break the API of class
> base::DataObjectInterface.
>
> Best regards,
> Johannes
>
>
>
> On Mon, Sep 28, 2015 at 3:36 PM, S Roderick <kiwi [dot] net [..] ...> wrote:
>
> Johannes' proposed changes are focussed on
> a) push vs pull, or specifying whether the data/buffer object is on the reader or writer side. This is very
> useful for system designers dealing with significant time delay. 
> b) private vs shared, whether a connection is per input/output port pair or whether the connection is shared
> between multiple input/output ports.
> c) mandatory, primarily intended to support knowing whether writes to a buffer succeeded (so that traditional
> producer/consumer relationships have some knowledge about the buffer state)
>
> It appears to me that most people are worried about changing the defaults, more than they're worried about the
> actual proposed changes. Actual technical concerns with the proposed changes appear to be
>
> 1) whether to encode the result of a write() in existing enums or not
> 2) how to add the "shared" concept within the current API, or to use a "new" API
> 3) changing the RTT defaults, which then affect an application's semantics.
>
> Did I miss any technical issues?
>
>
> My huge personal caveat to all of this is that I do *not* think we should change the current system semantics
> in introducing this change. Too many Orocos changes have broken backwards compatibility and forced system
> designers and implementors to change their system in response. I don't think we should force that on users
> without an incredibly good reason. Otherwise it breeds frustration in the community.
>
> If there is some way to introduce the proposed changes to the system, but to not change the behavior of
> current systems, then I think that is a win for all. There does seem to be some general agreement that there
> are valid use cases that the current port implementation handles poorly, and that this proposed approach does
> fix some of those. The proposed approach also improves the situation for systems that don't want or need to
> use the flow status approach. Both are equally valid approaches - IMHO it would be beneficial to the community
> as a whole if RTT supported both approaches.
>
> Cheers
> Stephen
>
>
> --
> Orocos-Dev mailing list
> Orocos-Dev [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>
>
>
>

Updated dataflow semantics for RTT

On Tue, Sep 29, 2015 at 1:37 PM, Herman Bruyninckx <
Herman [dot] Bruyninckx [..] ...> wrote:

> On Mon, 28 Sep 2015, Johannes Meyer wrote:
>
> Hi,
>> thanks for all your input and ideas and I agree with Herman that the
>> "one-framework-for-everything" approach of RTT
>> is probably not how you would start a new project nowadays.
>>
>
>
> Some inputs from a discussion on the ROS mailinglist which is very much
> related to this RTT-centric thread:
> <http://lists.ros.org/pipermail/ros-users/2015-September/069663.html>
> <http://lists.ros.org/pipermail/ros-users/2015-September/069664.html>
>
> These are the critical remarks of two experienced people towards the
> "one size fits all" approach of communication in ROS2.0; at the same time,
> they also provide concrete use cases and trade-offs that are relevant for
> the envisaged update of comunication in RTT...
>
> Herman
>
> PS I am of course more than a bit biased: both messages reflect also my
> personal view on this issue of "communication middleware", and are not just
> random samples from that ROS thread.
>
> PS2 Geoff Biggs is another RTT-knowledgeable person who _did_ intervene in
> that ROS thread, as well as in this RTT thread. Are there others on this
> list who, like Geoff and myself, are following both threads? Because,
> together, we might "mine" both threads to extract very interesting inputs
> from potential users, with widely varying expectations and contexts...

I also follow the ros-users and ros-sig-ng-ros mailing lists and the
discussions you mentioned. We will hopefully learn a lot about the current
state of ROS 2.0 development and why certain decisions have been made at
ROSCON next weekend in Hamburg. As I said, I completely agree that RTT
tries to solve too many problems at once, given that a lot of established
solutions exists nowadays for most of the building blocks that could be
integrated in a way more flexible way. However, I do not necessarily share
the opinion that dividing a complex problem into as small as possible
domains and re-use existing concepts and tools for each of them in a
modular way will *always* result in a better and less error-prone solution.
There are also a lot of pitfalls in the necessary "gluing" to adapt APIs,
to really make the domains interchangeable and to avoid performance
pitfalls, and at the end it will still be a complex solution for a complex
problem.

It was never the intention of this thread to start a discussion about "the
next big version of RTT" - even if we considered to increment the major
version number to 3 for the case the new behavior cannot be made in a
backwards-compatible way. It is all about relatively small updates and
bugfixes that hopefully will not disrupt existing applications and most
likely will not open RTT too a much broader community either. On the other
hand, there are some more patches that should be discussed and merged
sooner or later, that have the potential to break things and are well worth
a new version number:

- CORBA aliases (https://github.com/orocos-toolchain/rtt/pull/66)
- TLSF memory corruption detection (
https://github.com/orocos-toolchain/rtt/pull/77)
- Reintroduce v1 cmd syntax (https://github.com/orocos-toolchain/rtt/pull/84
)
- Include directories reordering (
https://github.com/orocos-toolchain/rtt/pull/85)
- Master update hook vs callback queue (
https://github.com/orocos-toolchain/rtt/pull/91)
- CORBA efficieny patches (no pull request yet, see
https://github.com/orocos-toolchain/rtt/compare/master...meyerj:corba-ef...
)

On Tue, Sep 29, 2015 at 1:44 AM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...>
wrote:

> Hi Johannes, thanks for the long explanation.
>
> On the implementation side, this whole discussion revived an idea I
> have had a while ago, and is pretty much in line with the discussion
> Herman started.
>
> Basically, one could get rid of (a.k.a. "deprecate" of course) the
> connection API on the ports themselves, and replace it by a separate
> object (let's call it "ConnectionManager") which would be in charge of
> managing all these buffers and create all these connections and
> implement the . In its simplest form, each process would have one of
> these objects.
>
> The basic operations it would have support are:
> - create/remove storage object (data object / buffer object)
> - connect output port to storage object
> - connect storage object to input port
>

That is already quite close to what the ConnFactory class is doing in RTT
2: building data objects, building channel input and output and finally
creating connections between ports or ports and a transport by using this
basic operations. All of those are governed by the ConnPolicy object which
describes the connection's internals. But the ConnFactory is considered as
internal API and the ports' connection methods are only simple wrappers
that call ConnFactory methods. The ConnFactory class is the one that has
been touched most in order to implement the proposed v3 dataflow semantics
and even without changing the semantics it was worth the effort to do a
cleanup here and clearly separate those basic operations.

At least with my proposed patch, once a connection is established, all
channel elements involved (buffers, remote channels, transport-specific
elements, ...) are managed by shared pointers and cleaned up automatically
if not needed anymore. The ConnectionManager instance, which currently
belongs to each port, is only required to introspect its connections, but
it completely lost its role in data processing and could also be dropped,
replaced by a global connection manager or some other tool.

RTT port implementations are already quite independent of the dataflow
model and channels could implement anything that supports write and/or read
operations from and to a reference to the sample.

>
> A "pull" connection would involve the ConnectionManager of the output
> port. A "push" connection the ConnectionManager of the receiving side.
> How one gets shared or private (or mixed) connections is obvious.
> Flags would be split between the two halves (which is good !). For
> instance, 'init' only affects the output->buffer side.
>

If you replace ConnectionManager by ConnFactory here, that basically
describes what is going on in RTT 2 internally and what I think is even
separated much better now.

>
> In principle -- but that's a principle that even I take with a big
> grain of salt given that I did not seriously touch that stuff for
> years -- the current underlying API (creation of the two channel
> halves) would be reused. In other words, only where the connections
> are created would change, not how they are created.
>
> On top of that, it's trivial to create higher-level, easier-to-use
> APIs, and also implement the current connection API so as to not break
> existing code.
>
> It would open the path to a lot of interesting things:
> - finally have a decent way to monitor the data objects (i.e. buffer
> fill rates, "data loss", data counters, that kind of thing). It would
> also make obvious how one could replace the RTT "base connection
> manager" by something else without touching the components. It would
> spell how wonderfully relevant RTT still is to those who don't see it
> ;-)
> - if we add the possibility of inter-ConnectionManager connections
> would allow us to move the keepSample functionalities of OutputPort
> out of it, which would also make the whole init functionality a lot
> more transparent. It would allow to expose (and monitor ...) the CORBA
> dispatchers.
>

I do not fully understand these last two points and why these things would
not already be possible today with RTT 2, perhaps by adding some simple,
non API-breaking helper methods to port and ConnectionManager classes. It
would be quite straight-forward to implement a store-and-forward channel
element that can be installed right after the channel input by the
ConnFactory and that replaces the current init functionality out of the
OutputPort implementation, but I do not really see why this would be
beneficial or open new possibilities for now.

>
> Now, I do understand that it might sound "blue sky" and that you might
> not have the time for that ... just sayin' ;-)
>
> Sylvain

Johannes

Updated dataflow semantics for RTT

> I do not fully understand these last two points and why these things would
> not already be possible today with RTT 2, perhaps by adding some simple, non
> API-breaking helper methods to port and ConnectionManager classes. It would
> be quite straight-forward to implement a store-and-forward channel element
> that can be installed right after the channel input by the ConnFactory and
> that replaces the current init functionality out of the OutputPort
> implementation, but I do not really see why this would be beneficial or open
> new possibilities for now.

I already knew the place of ConnFactory. The problem *is* that it is
internal and that the three operations I listed are "hidden" behind
the ConnPolicy object. It makes the logic behind ConnPolicy horrible.
By exposing these operations, and making them used by the tooling
(thus deprecating the port API), you actually in effect remove the
issue that RTT::TaskContext is trying to solve too many things. All
the possibilities that policy encodes in a very weird way become
different combinations of the three basic operations.

For instance: yes, you can already internally to ConnFactory replace
the "keep last written" and init stuff. But then you have to extend
the port API *even more*. Every single change / extension to the
connection handling right now requires to change the port API. By
exposing ConnFactory, in effect, you remove that need, and pave the
way for other to replace the connfactory by an extended version that
does more (or different) without requiring even more plugins or
whatnots.

Sylvain

Updated dataflow semantics for RTT

On Tue, 29 Sep 2015, Sylvain Joyeux wrote:

>> I do not fully understand these last two points and why these things would
>> not already be possible today with RTT 2, perhaps by adding some simple, non
>> API-breaking helper methods to port and ConnectionManager classes. It would
>> be quite straight-forward to implement a store-and-forward channel element
>> that can be installed right after the channel input by the ConnFactory and
>> that replaces the current init functionality out of the OutputPort
>> implementation, but I do not really see why this would be beneficial or open
>> new possibilities for now.
>
> I already knew the place of ConnFactory. The problem *is* that it is
> internal and that the three operations I listed are "hidden" behind
> the ConnPolicy object. It makes the logic behind ConnPolicy horrible.
> By exposing these operations, and making them used by the tooling
> (thus deprecating the port API), you actually in effect remove the
> issue that RTT::TaskContext is trying to solve too many things. All
> the possibilities that policy encodes in a very weird way become
> different combinations of the three basic operations.

"+1": this "inversion of control" makes a lot of sense! (It is already
available in the 'event stream' paradigm...)

> For instance: yes, you can already internally to ConnFactory replace
> the "keep last written" and init stuff. But then you have to extend
> the port API *even more*. Every single change / extension to the
> connection handling right now requires to change the port API. By
> exposing ConnFactory, in effect, you remove that need, and pave the
> way for other to replace the connfactory by an extended version that
> does more (or different) without requiring even more plugins or
> whatnots.
>
> Sylvain

Herman

Updated dataflow semantics for RTT

Hi Johannes, thanks for the long explanation.

On the implementation side, this whole discussion revived an idea I
have had a while ago, and is pretty much in line with the discussion
Herman started.

Basically, one could get rid of (a.k.a. "deprecate" of course) the
connection API on the ports themselves, and replace it by a separate
object (let's call it "ConnectionManager") which would be in charge of
managing all these buffers and create all these connections and
implement the . In its simplest form, each process would have one of
these objects.

The basic operations it would have support are:
- create/remove storage object (data object / buffer object)
- connect output port to storage object
- connect storage object to input port

A "pull" connection would involve the ConnectionManager of the output
port. A "push" connection the ConnectionManager of the receiving side.
How one gets shared or private (or mixed) connections is obvious.
Flags would be split between the two halves (which is good !). For
instance, 'init' only affects the output->buffer side.

In principle -- but that's a principle that even I take with a big
grain of salt given that I did not seriously touch that stuff for
years -- the current underlying API (creation of the two channel
halves) would be reused. In other words, only where the connections
are created would change, not how they are created.

On top of that, it's trivial to create higher-level, easier-to-use
APIs, and also implement the current connection API so as to not break
existing code.

It would open the path to a lot of interesting things:
- finally have a decent way to monitor the data objects (i.e. buffer
fill rates, "data loss", data counters, that kind of thing). It would
also make obvious how one could replace the RTT "base connection
manager" by something else without touching the components. It would
spell how wonderfully relevant RTT still is to those who don't see it
;-)
- if we add the possibility of inter-ConnectionManager connections
would allow us to move the keepSample functionalities of OutputPort
out of it, which would also make the whole init functionality a lot
more transparent. It would allow to expose (and monitor ...) the CORBA
dispatchers.

Now, I do understand that it might sound "blue sky" and that you might
not have the time for that ... just sayin' ;-)

Sylvain

2015-09-28 16:52 GMT-03:00 Johannes Meyer <johannes [..] ...>:
> Hi,
>
> thanks for all your input and ideas and I agree with Herman that the
> "one-framework-for-everything" approach of RTT is probably not how you would
> start a new project nowadays. But I would like to support Stephen's call to
> focus this discussion on the technical aspects of how to improve the
> existing implementation in as small as possible ("incremental") steps that
> add support for some new use cases that are not covered yet (but have been
> partially covered by v1).
>
> Maintaining backwards compatibility is definitely the way to go and we can
> conclude that the new default semantics (especially per-connection vs.
> per-input buffers) have to be the same as in previous v2 releases. I hope
> you find it acceptable that a new version would certainly be not
> ABI-compatible and breaks some internal API calls that are typically not
> used directly by a user. This might break more advanced things like custom
> transport implementations. I fear that these kind of changes cannot be
> avoided completely, especially for the ConnFactory and ChannelElement
> classes.
>
> We should not limit the use cases of RTT on the core API level, like
> enforcing per-input storage objects for data connections and per-connection
> storage for buffer connections, but only add new use cases. It is more a
> question of having good documentation and choosing reasonable defaults that
> cover as many applications as possible, but at the end the application
> designer has to have the freedom to deploy the application with whatever
> connection semantics it demands for.
>
> In the following I try to wrap up and comment on some of the concerns that
> came up during the discussion so far, and suggest compromise solutions which
> have been at least partially discussed with Peter:
>
> 1) There is definitely a need for per-input port buffers AND for
> per-connection buffers - INDEPENDENT of whether the data/buffer object is
> installed on the reader and writer side.
>
> Indeed, according to the original proposal the questions of at what side of
> a remote connection the data object will be installed and whether there will
> be a per-input or a per-connection buffer cannot specified separately. Peter
> and I could not imagine why you would want separate buffers if the data is
> already available locally anyways (in the push case) and we still think that
> at least for data connections the current per-connection implementation can
> be considered as broken. Even with readNewest() there is no guarantee that
> you get the latest sample written. On the other hand, Sylvain correctly
> stated that there are no guarantees anyway as soon as remote connections (or
> any non-realtime transport) are considered and the only true solution is
> proper timestamping and synchronization. But I think we agree that for most
> deployments with only local connections the assumption that the latest
> sample is the last written sample holds and that timestamping is beyond the
> scope of RTT and can be left to user code or more advanced tools like the
> Rock stream aligner.
>
> Even with the proposed changes (let's call them v3 semantics as a working
> title) the connection policy is still primarily describing the connection,
> even if *some* settings limit the set of allowed policies for other
> connections, making *some* other settings de-facto a per-port setting. This
> would be primarily true for input ports and not at all for output ports,
> unless it has a connection of the new shared type. But it is not an option
> to associate the connection policy *only* with a port because not *all*
> possible connections imply per-port buffers. Specifying a connection policy
> on a writer/reader pair base as it is now and let certain invalid connection
> attempts fail is still the most general approach.
>
> Proposal: In order to not break existing applications, we introduce a new
> ConnPolicy field "read_policy" that toggles between per-input and
> per-connection buffers, instead of implying that by the existing "pull" flag
> for remote connections. The read_policy would be an enum with two possible
> values for now, "SingleInputBuffer" and "PreferCurrentConnection". The
> reason why we propose an enum here is that you could also think of other
> policies on how to select the next channel to read from in the
> per-connection case, e.g. RoundRobin, PreferLastSignaled, etc. The current
> v2 default would correspond to PreferCurrentConnection, as the channel that
> was last read with new data is preferred by the input port and only then it
> polls all other connections in the order they have been made. We have not
> thought this idea to the end yet and the names of the constants are still
> under discussion. We do not want to call the input buffers "shared", because
> the read policy is independent of the new concept of a "shared connection"
> where multiple input ports read from the same buffer. Or someone can come up
> with a better name for this type of connection...
>
>
> 2) Should the new per-input port buffers become the new default or not
> (SingleInputBuffer read policy)?
>
> With that respect the conclusion is that backwards compatibility is more
> important than the new "features" and the default parameters for the current
> ConnPolicy class should not be touched. However, there are different ways to
> tackle this:
>
> 2.1 Make it a compile-time option.
>
> 2.2 Only add *new* fields to the ConnPolicy class without changing the
> default parameter values in the constructor and static methods, and a new
> class ConnPolicy2 or ConnPolicy3 that inherits from ConnPolicy and might
> have other default parameters or another C++ API (I am thinking of a
> named-parameter based API) and can also be used for scripting. Internally,
> the connection factory would work with const references to the base class
> only.
>
> 2.3 We add a DefaultConnPolicy attribute to the GlobalsRepository. This
> application-wide (per process) policy instance will be copied to newly
> constructed ConnPolicy instances and the constructors and static methods
> only overwrite the respective parameters.
>
> It might be obsoleted by the above proposal, but in the meantime I added a
> compile-time option RTT_V2_COMPATIBILITY_MODE to my working branch (see
> ef99f2b), that, if set, reverts to the current v2 per-connection model, but
> lets RTT print meaningful warning messages in the case a new connection
> would be invalid with v3 semantics because of conflicts with other
> connections to the same port. Such a flag could smooth the transition phase
> and help application designers to find the minimal set of connections that
> require connection policy changes.
>
> There is one small exception from the backwards-compatibility rule, that I
> think is worth discussing: The static method
>
> static ConnPolicy data(int lock_policy = LOCK_FREE, bool init_connection =
> true, bool pull = false);
>
> initializes the init_connection field to true by default, which was kind of
> surprising me, because it means that every new connection to an output port
> with this policy will overwrite a sample previously written by another port.
> Is this what we really want as a default? It is also not consistent with the
> constructor called as
>
> ConnPolicy(DATA)
>
> which would initialize the init_connection flag to false. The static method
> data(...) is only used in the OutputPortInterface::createDataConnection(...)
> method from within RTT and OCL itself.
>
> By the way, the init_connection flag is a good example where the API offers
> some degree of freedom to the application builder that semantically does not
> make sense at all for certain combinations, like in combination with a
> buffer connection. It is still not forbidden to set the flag, but it might
> be worth a warning log message if a connection is created like this.
>
>
> 3) How to add the "shared" concept within the current API, or to use a "new"
> API?
>
> This point was brought up by Janosch, who asked for "a more explicit
> interface for shared connections", with the main argument that it is
> fundamentally different from a normal connection and a "connect port A to B
> with policy C" kind of API does not fit well to that concept. That's true.
> We considered to use the createStream(...) API first, with a name_id string
> that identifies the shared buffer to connect to. This would actually fit
> better to the shared connection model, but it still enforces the same buffer
> type policies for all connections. We withdraw that idea because the
> existing API is more general and disallowing certain policy combinations is
> required anyway for private push connections with a single input buffer. At
> least in this case it cannot break existing applications. I do not think
> that there is a need for a completely new API only for shared connections.
>
> My current implementation in the updated-dataflow-semantics branch is as
> follows, and new/other ideas are always welcome:
>
> - Shared connection objects (the shared data object or buffered) are added
> to a process-wide registry. Every shared connection is identified by a
> unique string, either set explictly in the name_id field of the ConnPolicy
> or assigned automatically, in which case the name of the connection is
> returned in exactly this field (name_id is and always was declared as
> mutable).
> - For new connections with the name_id set, the shared connection is looked
> up in the registry, or created as new. As a consequence, if the same
> ConnPolicy instance is used to make a second connection, even if name_id was
> not set explicitly before, you will always connect to the same connection
> object.
> - For new connections with no name_id set, the port objects provide an API
> to return a pointer to an existing shared connection if they are already
> connected to an existing one, and only the non-connected port will be added.
> So ports can be connected in any order to each other, as long as there is
> always one of the two ports already connected. Otherwise the name_id has to
> be set or a new connection instance would be created. Perhaps this one could
> be dropped because it it too implicit and does not work very well for remote
> connections...
> - If an existing connection was found, the buffer policies (type, size and
> locking policy) have to match.
> - All ports will only store a single pointer to the shared connection
> instance in their connection list (in internal::ConnectionManager) and
> cannot know directly which other ports are connected.
> - For remote input ports with a shared connection, only a proxy is created
> locally and can be looked up in the shared connection repository. Other
> ports will only connect to either the proxy or the real shared connection
> instance, but never build a new remote channel once the proxy has been
> created for the first remote connection (works in both directions).
> - All connect and disconnect calls to shared connections are logged with log
> level Debug to the RTT logger, so at least there is a manual way to check
> whether the resulting connections have been made as expected. Graphical
> tools like the rtt_dot_service could be patched later to visualize shared
> connections correctly.
>
>
> 4) Whether to encode the result of a write() in existing enums or not?
>
> Janosch asked for introducing a separate enum for the return value of
> write(...) calls, in order to not break or trigger compiler warnings for
> existing code that already uses the FlowStatus enum in a switch statement.
> Another minor issue with a single enum is the default value of FlowStatus
> attributes, which can be only meaningful for one use case.
>
> That's also fine for me and I will update the document and implementation
> accordingly (add a new enum WriteStatus). Note that the evaluation in
> boolean context is counter-intuitive in this case, other than for
> FlowStatus, because the value WriteSuccess (=0) value evaluates to false and
> all error cases (WriteFailure and NotConnected) evaluate to true.
>
>
> I hope I did not forget anything. This mail already became much longer than
> expected. I am sorry.
>
> I also did some performance measurements of read and write calls last week
> in terms of absolute time, CPU time, number of assignments and so on of the
> current, the new and even the v1 data flow implementations in different
> scenarios. I plan to come up with some results during this week. The tests
> revealed some bugs and performance caveats, and some of them are already
> present since the early days of RTT v2, e.g. that data samples might be lost
> (never read as NewData) because the read flag is not lock-protected or
> stored within the lock-free data structure. I am preparing a pull request to
> the current master branch, but it will also break the API of class
> base::DataObjectInterface.
>
> Best regards,
> Johannes
>
>
>
> On Mon, Sep 28, 2015 at 3:36 PM, S Roderick <kiwi [dot] net [..] ...> wrote:
>>
>>
>> Johannes' proposed changes are focussed on
>> a) push vs pull, or specifying whether the data/buffer object is on the
>> reader or writer side. This is very useful for system designers dealing with
>> significant time delay.
>> b) private vs shared, whether a connection is per input/output port pair
>> or whether the connection is shared between multiple input/output ports.
>> c) mandatory, primarily intended to support knowing whether writes to a
>> buffer succeeded (so that traditional producer/consumer relationships have
>> some knowledge about the buffer state)
>>
>> It appears to me that most people are worried about changing the defaults,
>> more than they're worried about the actual proposed changes. Actual
>> technical concerns with the proposed changes appear to be
>>
>> 1) whether to encode the result of a write() in existing enums or not
>> 2) how to add the "shared" concept within the current API, or to use a
>> "new" API
>> 3) changing the RTT defaults, which then affect an application's
>> semantics.
>>
>> Did I miss any technical issues?
>>
>>
>> My huge personal caveat to all of this is that I do *not* think we should
>> change the current system semantics in introducing this change. Too many
>> Orocos changes have broken backwards compatibility and forced system
>> designers and implementors to change their system in response. I don't think
>> we should force that on users without an incredibly good reason. Otherwise
>> it breeds frustration in the community.
>>
>> If there is some way to introduce the proposed changes to the system, but
>> to not change the behavior of current systems, then I think that is a win
>> for all. There does seem to be some general agreement that there are valid
>> use cases that the current port implementation handles poorly, and that this
>> proposed approach does fix some of those. The proposed approach also
>> improves the situation for systems that don't want or need to use the flow
>> status approach. Both are equally valid approaches - IMHO it would be
>> beneficial to the community as a whole if RTT supported both approaches.
>>
>> Cheers
>> Stephen
>>
>>
>>
>> --
>> Orocos-Dev mailing list
>> Orocos-Dev [..] ...
>> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>>
>

Updated dataflow semantics for RTT

On Tue, 29 Sep 2015, Sylvain Joyeux wrote:

> Hi Johannes, thanks for the long explanation.
>
> On the implementation side, this whole discussion revived an idea I
> have had a while ago, and is pretty much in line with the discussion
> Herman started.
>
> Basically, one could get rid of (a.k.a. "deprecate" of course) the
> connection API on the ports themselves, and replace it by a separate
> object (let's call it "ConnectionManager") which would be in charge of
> managing all these buffers and create all these connections and
> implement the . In its simplest form, each process would have one of
> these objects.

That is exactly one of the (few) major _mechanism_ building blocks of "event streams".

The other ones are:
- every "event" has an UID that serves as a "pointer";
- every "event type" has an UID that serves to "filter";
- stream filtering;
- "long term" garbage collection.

All the rest are _policies_: how to filter, how to deploy, what to throw
away and what to compact,...

ROCK's "stream aligner"
<http://rock-robotics.org/master/documentation/data_processing/stream_aligner.html>
is indeed already moving in that direction.

> The basic operations it would have support are:
> - create/remove storage object (data object / buffer object)

The paradigm of "immutable data" would separate the creation from the
removal, big time. _The_ major problem in asynchronous systems is to keep
creation and deletion in sync, in an efficient way. Abundance of RAM (_if_
it is an option) helps out tremendously to simplify this problem.

> - connect output port to storage object
> - connect storage object to input port

These are two instances of the generic "event stream filtering"; what you
call a "storage object" is called an "event stream". (The connotation
"stream" is indeed refering to the semantically indentical concept of "data
flow".)

> A "pull" connection would involve the ConnectionManager of the output
> port.

In the "immutable data" paradigm, "pulling" is just reading, which can
always be done without locks. ("Always" = up to long-term stream
compaction.)

> A "push" connection the ConnectionManager of the receiving side.

This needs the same mechanism as for the lock-free buffers: the pusher
needs to get a place for its data/event immediately, without waiting.

The known policies here are simple (not necessarily "easy"!):
- drop every event in the next buffer, where the size of the buffer depends
on the event type;
- have a stream for each event type (_if_ that indeed results in equal-size
buffer slots), and a stream that collects the event UID together with the
pointer to the entry in the corresponding stream.

> How one gets shared or private (or mixed) connections is obvious.
> Flags would be split between the two halves (which is good !). For
> instance, 'init' only affects the output->buffer side.

> In principle -- but that's a principle that even I take with a big
> grain of salt given that I did not seriously touch that stuff for
> years -- the current underlying API (creation of the two channel
> halves) would be reused. In other words, only where the connections
> are created would change, not how they are created.

I suggest to cleanly separate the creation of connections from the stream
processing that is being realised on top of them. "Connections" are a
mechanismk that one should outsource to "socket libraries" such as ZeroMQ,
that allow to configure in-process and out-of-process deployments of the
socket concept.

(The above separation is a major reason why I think the ROS2 dedication to
DDS will keep ROS in the 20th century: DDS _couples_ mechanism and policy,
_and_ restricts the choices in both...)

> On top of that, it's trivial to create higher-level, easier-to-use
> APIs, and also implement the current connection API so as to not break
> existing code.

I agree. At least, if you mean the same thing as what I call "event
filters" in the "event stream" paradigm. Every filter represents one
particular type of "peer(s)-to-peer(s)" data communication, and they all
fit seamlessly on the same event stream mechamism.

In the context of realtime, the _implementation_ and the _deployment_ of
these mechanisms are of course important. And that is where an RTT-centric
development can add value compared to the event stream libraries that are
already out there, such as Apache's KAFKA, or RethinkDB's.

By the way, since RethinkDB is written in C++, for efficiency reasons,
there might be a lot of code ready to be reused... I have not been able to
look at that deep level already, so I might be completely wrong. Any
takers...?

> It would open the path to a lot of interesting things:
> - finally have a decent way to monitor the data objects (i.e. buffer
> fill rates, "data loss", data counters, that kind of thing). It would
> also make obvious how one could replace the RTT "base connection
> manager" by something else without touching the components. It would
> spell how wonderfully relevant RTT still is to those who don't see it
> ;-)

I don't see it anymore, indeed :-) But I do understand the importance of
backwards compatibility. Note however that I trade that off with growing
the community: if all that RTT does is to serve a handful of current
professional users, then maintenance of the code is a very heavy burden.

> - if we add the possibility of inter-ConnectionManager connections

This is an obvious feature of event streams, via the "stream filtering"
mechanism. (I hope that it has become clear by now that the event stream
paradigm is "simple", since it relies on very few concepts that work nicely
together; this does not mean that a good implementation is "easy"; or that
the full power of the mechanism can be abstracted behind an "easy"
interface. Reference: <http://www.infoq.com/presentations/Simple-Made-Easy>
and similar documents.)

> would allow us to move the keepSample functionalities of OutputPort
> out of it, which would also make the whole init functionality a lot
> more transparent. It would allow to expose (and monitor ...) the CORBA
> dispatchers.

Why would one still choose CORBA as a transport mechanism...? It _can_ be
used, but I do not see the need anymore. "Replication" of event streams is
part of the event stream paradigm, and implementations are readily
available, in KAFKA, RethinkDB and others.

> Now, I do understand that it might sound "blue sky" and that you might
> not have the time for that ... just sayin' ;-)

My team is working in that direction already, exploring what the already
existing implementations can realise, and with what performance.
Obviously, we are interested in working together. One way or another, which
means, without having "backwards compatibility" as the major development
driver...

> Sylvain

Herman

>
>
> 2015-09-28 16:52 GMT-03:00 Johannes Meyer <johannes [..] ...>:
>> Hi,
>>
>> thanks for all your input and ideas and I agree with Herman that the
>> "one-framework-for-everything" approach of RTT is probably not how you would
>> start a new project nowadays. But I would like to support Stephen's call to
>> focus this discussion on the technical aspects of how to improve the
>> existing implementation in as small as possible ("incremental") steps that
>> add support for some new use cases that are not covered yet (but have been
>> partially covered by v1).
>>
>> Maintaining backwards compatibility is definitely the way to go and we can
>> conclude that the new default semantics (especially per-connection vs.
>> per-input buffers) have to be the same as in previous v2 releases. I hope
>> you find it acceptable that a new version would certainly be not
>> ABI-compatible and breaks some internal API calls that are typically not
>> used directly by a user. This might break more advanced things like custom
>> transport implementations. I fear that these kind of changes cannot be
>> avoided completely, especially for the ConnFactory and ChannelElement
>> classes.
>>
>> We should not limit the use cases of RTT on the core API level, like
>> enforcing per-input storage objects for data connections and per-connection
>> storage for buffer connections, but only add new use cases. It is more a
>> question of having good documentation and choosing reasonable defaults that
>> cover as many applications as possible, but at the end the application
>> designer has to have the freedom to deploy the application with whatever
>> connection semantics it demands for.
>>
>> In the following I try to wrap up and comment on some of the concerns that
>> came up during the discussion so far, and suggest compromise solutions which
>> have been at least partially discussed with Peter:
>>
>> 1) There is definitely a need for per-input port buffers AND for
>> per-connection buffers - INDEPENDENT of whether the data/buffer object is
>> installed on the reader and writer side.
>>
>> Indeed, according to the original proposal the questions of at what side of
>> a remote connection the data object will be installed and whether there will
>> be a per-input or a per-connection buffer cannot specified separately. Peter
>> and I could not imagine why you would want separate buffers if the data is
>> already available locally anyways (in the push case) and we still think that
>> at least for data connections the current per-connection implementation can
>> be considered as broken. Even with readNewest() there is no guarantee that
>> you get the latest sample written. On the other hand, Sylvain correctly
>> stated that there are no guarantees anyway as soon as remote connections (or
>> any non-realtime transport) are considered and the only true solution is
>> proper timestamping and synchronization. But I think we agree that for most
>> deployments with only local connections the assumption that the latest
>> sample is the last written sample holds and that timestamping is beyond the
>> scope of RTT and can be left to user code or more advanced tools like the
>> Rock stream aligner.
>>
>> Even with the proposed changes (let's call them v3 semantics as a working
>> title) the connection policy is still primarily describing the connection,
>> even if *some* settings limit the set of allowed policies for other
>> connections, making *some* other settings de-facto a per-port setting. This
>> would be primarily true for input ports and not at all for output ports,
>> unless it has a connection of the new shared type. But it is not an option
>> to associate the connection policy *only* with a port because not *all*
>> possible connections imply per-port buffers. Specifying a connection policy
>> on a writer/reader pair base as it is now and let certain invalid connection
>> attempts fail is still the most general approach.
>>
>> Proposal: In order to not break existing applications, we introduce a new
>> ConnPolicy field "read_policy" that toggles between per-input and
>> per-connection buffers, instead of implying that by the existing "pull" flag
>> for remote connections. The read_policy would be an enum with two possible
>> values for now, "SingleInputBuffer" and "PreferCurrentConnection". The
>> reason why we propose an enum here is that you could also think of other
>> policies on how to select the next channel to read from in the
>> per-connection case, e.g. RoundRobin, PreferLastSignaled, etc. The current
>> v2 default would correspond to PreferCurrentConnection, as the channel that
>> was last read with new data is preferred by the input port and only then it
>> polls all other connections in the order they have been made. We have not
>> thought this idea to the end yet and the names of the constants are still
>> under discussion. We do not want to call the input buffers "shared", because
>> the read policy is independent of the new concept of a "shared connection"
>> where multiple input ports read from the same buffer. Or someone can come up
>> with a better name for this type of connection...
>>
>>
>> 2) Should the new per-input port buffers become the new default or not
>> (SingleInputBuffer read policy)?
>>
>> With that respect the conclusion is that backwards compatibility is more
>> important than the new "features" and the default parameters for the current
>> ConnPolicy class should not be touched. However, there are different ways to
>> tackle this:
>>
>> 2.1 Make it a compile-time option.
>>
>> 2.2 Only add *new* fields to the ConnPolicy class without changing the
>> default parameter values in the constructor and static methods, and a new
>> class ConnPolicy2 or ConnPolicy3 that inherits from ConnPolicy and might
>> have other default parameters or another C++ API (I am thinking of a
>> named-parameter based API) and can also be used for scripting. Internally,
>> the connection factory would work with const references to the base class
>> only.
>>
>> 2.3 We add a DefaultConnPolicy attribute to the GlobalsRepository. This
>> application-wide (per process) policy instance will be copied to newly
>> constructed ConnPolicy instances and the constructors and static methods
>> only overwrite the respective parameters.
>>
>> It might be obsoleted by the above proposal, but in the meantime I added a
>> compile-time option RTT_V2_COMPATIBILITY_MODE to my working branch (see
>> ef99f2b), that, if set, reverts to the current v2 per-connection model, but
>> lets RTT print meaningful warning messages in the case a new connection
>> would be invalid with v3 semantics because of conflicts with other
>> connections to the same port. Such a flag could smooth the transition phase
>> and help application designers to find the minimal set of connections that
>> require connection policy changes.
>>
>> There is one small exception from the backwards-compatibility rule, that I
>> think is worth discussing: The static method
>>
>> static ConnPolicy data(int lock_policy = LOCK_FREE, bool init_connection =
>> true, bool pull = false);
>>
>> initializes the init_connection field to true by default, which was kind of
>> surprising me, because it means that every new connection to an output port
>> with this policy will overwrite a sample previously written by another port.
>> Is this what we really want as a default? It is also not consistent with the
>> constructor called as
>>
>> ConnPolicy(DATA)
>>
>> which would initialize the init_connection flag to false. The static method
>> data(...) is only used in the OutputPortInterface::createDataConnection(...)
>> method from within RTT and OCL itself.
>>
>> By the way, the init_connection flag is a good example where the API offers
>> some degree of freedom to the application builder that semantically does not
>> make sense at all for certain combinations, like in combination with a
>> buffer connection. It is still not forbidden to set the flag, but it might
>> be worth a warning log message if a connection is created like this.
>>
>>
>> 3) How to add the "shared" concept within the current API, or to use a "new"
>> API?
>>
>> This point was brought up by Janosch, who asked for "a more explicit
>> interface for shared connections", with the main argument that it is
>> fundamentally different from a normal connection and a "connect port A to B
>> with policy C" kind of API does not fit well to that concept. That's true.
>> We considered to use the createStream(...) API first, with a name_id string
>> that identifies the shared buffer to connect to. This would actually fit
>> better to the shared connection model, but it still enforces the same buffer
>> type policies for all connections. We withdraw that idea because the
>> existing API is more general and disallowing certain policy combinations is
>> required anyway for private push connections with a single input buffer. At
>> least in this case it cannot break existing applications. I do not think
>> that there is a need for a completely new API only for shared connections.
>>
>> My current implementation in the updated-dataflow-semantics branch is as
>> follows, and new/other ideas are always welcome:
>>
>> - Shared connection objects (the shared data object or buffered) are added
>> to a process-wide registry. Every shared connection is identified by a
>> unique string, either set explictly in the name_id field of the ConnPolicy
>> or assigned automatically, in which case the name of the connection is
>> returned in exactly this field (name_id is and always was declared as
>> mutable).
>> - For new connections with the name_id set, the shared connection is looked
>> up in the registry, or created as new. As a consequence, if the same
>> ConnPolicy instance is used to make a second connection, even if name_id was
>> not set explicitly before, you will always connect to the same connection
>> object.
>> - For new connections with no name_id set, the port objects provide an API
>> to return a pointer to an existing shared connection if they are already
>> connected to an existing one, and only the non-connected port will be added.
>> So ports can be connected in any order to each other, as long as there is
>> always one of the two ports already connected. Otherwise the name_id has to
>> be set or a new connection instance would be created. Perhaps this one could
>> be dropped because it it too implicit and does not work very well for remote
>> connections...
>> - If an existing connection was found, the buffer policies (type, size and
>> locking policy) have to match.
>> - All ports will only store a single pointer to the shared connection
>> instance in their connection list (in internal::ConnectionManager) and
>> cannot know directly which other ports are connected.
>> - For remote input ports with a shared connection, only a proxy is created
>> locally and can be looked up in the shared connection repository. Other
>> ports will only connect to either the proxy or the real shared connection
>> instance, but never build a new remote channel once the proxy has been
>> created for the first remote connection (works in both directions).
>> - All connect and disconnect calls to shared connections are logged with log
>> level Debug to the RTT logger, so at least there is a manual way to check
>> whether the resulting connections have been made as expected. Graphical
>> tools like the rtt_dot_service could be patched later to visualize shared
>> connections correctly.
>>
>>
>> 4) Whether to encode the result of a write() in existing enums or not?
>>
>> Janosch asked for introducing a separate enum for the return value of
>> write(...) calls, in order to not break or trigger compiler warnings for
>> existing code that already uses the FlowStatus enum in a switch statement.
>> Another minor issue with a single enum is the default value of FlowStatus
>> attributes, which can be only meaningful for one use case.
>>
>> That's also fine for me and I will update the document and implementation
>> accordingly (add a new enum WriteStatus). Note that the evaluation in
>> boolean context is counter-intuitive in this case, other than for
>> FlowStatus, because the value WriteSuccess (=0) value evaluates to false and
>> all error cases (WriteFailure and NotConnected) evaluate to true.
>>
>>
>> I hope I did not forget anything. This mail already became much longer than
>> expected. I am sorry.
>>
>> I also did some performance measurements of read and write calls last week
>> in terms of absolute time, CPU time, number of assignments and so on of the
>> current, the new and even the v1 data flow implementations in different
>> scenarios. I plan to come up with some results during this week. The tests
>> revealed some bugs and performance caveats, and some of them are already
>> present since the early days of RTT v2, e.g. that data samples might be lost
>> (never read as NewData) because the read flag is not lock-protected or
>> stored within the lock-free data structure. I am preparing a pull request to
>> the current master branch, but it will also break the API of class
>> base::DataObjectInterface.
>>
>> Best regards,
>> Johannes
>>
>>
>>
>> On Mon, Sep 28, 2015 at 3:36 PM, S Roderick <kiwi [dot] net [..] ...> wrote:
>>>
>>>
>>> Johannes' proposed changes are focussed on
>>> a) push vs pull, or specifying whether the data/buffer object is on the
>>> reader or writer side. This is very useful for system designers dealing with
>>> significant time delay.
>>> b) private vs shared, whether a connection is per input/output port pair
>>> or whether the connection is shared between multiple input/output ports.
>>> c) mandatory, primarily intended to support knowing whether writes to a
>>> buffer succeeded (so that traditional producer/consumer relationships have
>>> some knowledge about the buffer state)
>>>
>>> It appears to me that most people are worried about changing the defaults,
>>> more than they're worried about the actual proposed changes. Actual
>>> technical concerns with the proposed changes appear to be
>>>
>>> 1) whether to encode the result of a write() in existing enums or not
>>> 2) how to add the "shared" concept within the current API, or to use a
>>> "new" API
>>> 3) changing the RTT defaults, which then affect an application's
>>> semantics.
>>>
>>> Did I miss any technical issues?
>>>
>>>
>>> My huge personal caveat to all of this is that I do *not* think we should
>>> change the current system semantics in introducing this change. Too many
>>> Orocos changes have broken backwards compatibility and forced system
>>> designers and implementors to change their system in response. I don't think
>>> we should force that on users without an incredibly good reason. Otherwise
>>> it breeds frustration in the community.
>>>
>>> If there is some way to introduce the proposed changes to the system, but
>>> to not change the behavior of current systems, then I think that is a win
>>> for all. There does seem to be some general agreement that there are valid
>>> use cases that the current port implementation handles poorly, and that this
>>> proposed approach does fix some of those. The proposed approach also
>>> improves the situation for systems that don't want or need to use the flow
>>> status approach. Both are equally valid approaches - IMHO it would be
>>> beneficial to the community as a whole if RTT supported both approaches.
>>>
>>> Cheers
>>> Stephen
>>>
>>>
>>>
>>> --
>>> Orocos-Dev mailing list
>>> Orocos-Dev [..] ...
>>> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>>>
>>

Updated dataflow semantics for RTT

On 29/9/15, 01:44, "orocos-dev-bounces [..] ... on behalf of
Sylvain Joyeux" <orocos-dev-bounces [..] ... on behalf of
sylvain [dot] joyeux [..] ...> wrote:

>Hi Johannes, thanks for the long explanation.
>
>On the implementation side, this whole discussion revived an idea I
>have had a while ago, and is pretty much in line with the discussion
>Herman started.
>
>Basically, one could get rid of (a.k.a. "deprecate" of course) the
>connection API on the ports themselves, and replace it by a separate
>object (let's call it "ConnectionManager") which would be in charge of
>managing all these buffers and create all these connections and
>implement the . In its simplest form, each process would have one of
>these objects.
>
>The basic operations it would have support are:
> - create/remove storage object (data object / buffer object)
> - connect output port to storage object
> - connect storage object to input port
>
>A "pull" connection would involve the ConnectionManager of the output
>port. A "push" connection the ConnectionManager of the receiving side.
>How one gets shared or private (or mixed) connections is obvious.
>Flags would be split between the two halves (which is good !). For
>instance, 'init' only affects the output->buffer side.
>
>In principle -- but that's a principle that even I take with a big
>grain of salt given that I did not seriously touch that stuff for
>years -- the current underlying API (creation of the two channel
>halves) would be reused. In other words, only where the connections
>are created would change, not how they are created.
>
>On top of that, it's trivial to create higher-level, easier-to-use
>APIs, and also implement the current connection API so as to not break
>existing code.
>
>It would open the path to a lot of interesting things:
> - finally have a decent way to monitor the data objects (i.e. buffer
>fill rates, "data loss", data counters, that kind of thing). It would
>also make obvious how one could replace the RTT "base connection
>manager" by something else without touching the components. It would
>spell how wonderfully relevant RTT still is to those who don't see it
>;-)
> - if we add the possibility of inter-ConnectionManager connections
>would allow us to move the keepSample functionalities of OutputPort
>out of it, which would also make the whole init functionality a lot
>more transparent. It would allow to expose (and monitor ...) the CORBA
>dispatchers.
>
>Now, I do understand that it might sound "blue sky" and that you might
>not have the time for that ... just sayin' ;-)
>
>Sylvain

What would the advantage of the above connection manager object be over
providing the same functionality (monitoring, etc.) in individual port
objects?

Geoff

Updated dataflow semantics for RTT

> I agree with much at the above, and we have similar problems that we've
> solved in similar ways. But there are also huge parts of our system that,
> while "data driven", aren't at all interested in time. The system itself is
> synchronized and time is (largely) irrelevant - everything operates on a
> per-cycle basis. In the long run, we time stamp only a tiny portion of the
> data in our system due to this. Equally valid, but different.

I don't dispute that you probably have a very good point, you usually
do, but could you provide me with a concrete example of such a setup
that uses buffers ? It would really help me grasp the situation.

> Sometimes we want to control whether the network access (and hence
> delay/non-real-time part) is on the writer or reader side. That should be an
> application-level choice.

Agreed, but this choice is currently bundled with having a buffer per
channel or one per input port.

> If you put the buffer per-port, you end up with one policy per port.
> Port-driven triggering (event ports) are governed by the data object
> (i.e. per-port), data object size and type becomes per-port as well.
> What's left in the connection policy is the info necessary to reach
> the other side (i.e. the topic name in ROS), which means that it is
> not a policy anymore but a "remote endpoint"
>
>
> What if you don't use event ports?

How is that relevant ? Are we misunderstanding each other here ?

As I understand the proposal so far, we have: either you pull (hit
network on read, one buffer per channel) or push (not hit network on
read, have a one buffer per port). The middle ground of
one-buffer-per-channel and not hitting network is not covered, and
that's what *greatly* bothers me.

I know you guys probably have a good case here, I am trying very hard
for you to not break mine

> Note the "selectively remove" above - we still want to be able to use
> v2-flow-status type setups within our system.

My understanding of the proposal is that the v2 semantic are not
available anymore. I'm not pushing against what you need, I want you
to not break what I need.

- what are going to be the options of network/no-network and
placement of buffer ?
- how can these options be or not be used with a single output /
single input ? Are they mutually exclusive ? Can they be used together
on the same input port ? On the same output port ?

> We have several hundred local connections in our deployers. We have maybe
> less than 10 remote connections.

Again, how is this relevant ? If the policy is specified per-port, you
won't have any "kind-of-sane default" to configure the CORBA
dispatchers with.

> Buffers vs data connections definitely change the flavor of things. The vast
> majority of our connections are data connections, not buffers.

I *know* which leads to this long discussion, because I am on the
completely other end of things: we mostly use buffered connections.

> I think that we're getting off topic as a whole.

I don't think so. So far, my concerns with the proposal are still
there: by getting closer to v1 semantics, the proposal breaks v2
semantics in a way that is fundamental to me. Peter's post actually
leads me to think that this breakage and known and desired, which is
why I am spending so much time trying to get my point across, while I
already have way too much on my plate.

> a) push vs pull, or specifying whether the data/buffer object is on the
> reader or writer side. This is very useful for system designers dealing with
> significant time delay.
> b) private vs shared, whether a connection is per input/output port pair or
> whether the connection is shared between multiple input/output ports.

So far, "private" is defined one buffer *per input port* (change in
semantic) not *per input/output port pair* (current v2 semantic)

Sylvain

Updated dataflow semantics for RTT

On Sep 28, 2015, at 08:55 AM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:

But I do not agree with the analysis that RTT is the best thing to hide
computations: I have seen way too many control loops implemented with one
single Orocos component (and activity) per block in a Simulink-like data
flow... Way too many.

Like ... iTasc ?

Unless you make a system too constrained to be actually useful, there
will always be room to misuse it. That does not spell that the system
itself is bad, only that "best practices" suck (or are inexistent).
This is a neverending fight, but IMO the solution is education, not
some horrible model-based environment too complex to every be used.
 
+1, nicely summarized.
S

Updated dataflow semantics for RTT

On Sep 28, 2015, at 08:44 AM, Herman Bruyninckx <Herman [dot] Bruyninckx [..] ...> wrote:

On Mon, 28 Sep 2015, S Roderick wrote:

On Sep 27, 2015, at 08:11 AM, Herman Bruyninckx
<Herman [dot] Bruyninckx [..] ...> wrote:

On Mon, 21 Sep 2015, Johannes Meyer wrote:

Dear Orocos community,
during the last few weeks we have been working on the concept and
implementation of a major redesign of the Orocos
RTT dataflow, that we would like to share and ask for your valuable feedback.
There was already a previous
discussion on the mailing list about the topic earlier this year [1], and the
following proposal tries to address
all the issues mentioned there.

With the version 2 of the Orocos toolchain some years ago, the previous data
flow architecture, which was based on a
single data object and buffer shared between all ports participating at the
same connection, was replaced by a
connection architecture based on data flow, where each pair of output and
input ports (or streams) is connected
through an individual channel, each having its own data object or buffer.
While there was a well substantiated
motivation for this update and it solved some of the shortcomings in RTT
version 1 [2], it also introduced some new,
sometimes hidden, problems and broke other use cases. One example is an input
port with multiple connections, where
due to the ambiguity in which channel to read from the result is sometimes
unexpected to the user, who expects that
each write overwrites previous samples in case of simple data connections.

We assembled a document (see attachment, or [3]) which introduces the version
1 and version 2 data flow semantics
for those who are not aware of the details, three example use cases and
presents the new dataflow architecture we
have in mind, which ideally works for a superset of use cases that are
supported by RTT version 1 and 2. 

We do not want to go into details here, so only a short summary for those who
do not want to read the full document:
We propose to add two more boolean flags to the ConnPolicy struct that
describes the type of connection between two
ports: "shared" and "mandatory". Furthermore the existing "pull" flag, which
was only meaningful for remote
connections, now has a big influence for local connections.

"push" vs. "pull"

Thanks for this nice effort. I think it is really time to consider a next
generation of Orocos RTT...

If you allow me to bring in my two cents... It's possible that you won't like
the
message, or find it off-topic, but I think I should share it with you anyway…

These are interesting ideas Herman, but you're right in that they're somewhat
off topic. For this discussion we are interested in the here and now - in
evolutionary and incremental changes to a product that has served many of us
fairly well.

The evolution from the current "data flow architecture" to a "event stream"
_can_/_could_ be done in an incremental way...
Policy by policy. (i.e., pull/push; read/write; buffer/port;...)
Separation by separation. (i.e., component/activity/thread/OS; pub/sub to
event stream;...)
 
With all due respect, I think that your idea of "incremental" and mine are two very different things (and both are equally valid). It took us two full years to convert our system from RTT v1 to RTT v2. There is no way we would even consider moving to a RTT replacement without some serious review and up front discover and prototype use of a replacement system. So incremental for us is defined in months to years …

Personally, I'm not so worried about "better" as I am worried
about "good enough".

I am mostly interested in "simple". Because that is what determines how
much of the potential mind share a project actually gets.
(Mind that I am not interested in "easy", which is the driver for ROS, and
that serves another public well.)
 
+1 on simple.
S

Updated dataflow semantics for RTT

On Mon, 28 Sep 2015, S Roderick wrote:

>
> On Sep 28, 2015, at 08:44 AM, Herman Bruyninckx
> <Herman [dot] Bruyninckx [..] ...> wrote:
>
> On Mon, 28 Sep 2015, S Roderick wrote:
>
> On Sep 27, 2015, at 08:11 AM, Herman Bruyninckx
> <Herman [dot] Bruyninckx [..] ...> wrote:
>
> On Mon, 21 Sep 2015, Johannes Meyer wrote:
>
> Dear Orocos community,
> during the last few weeks we have been working on the concept and
> implementation of a major redesign of the Orocos
> RTT dataflow, that we would like to share and ask for your valuable feedback.
> There was already a previous
> discussion on the mailing list about the topic earlier this year [1], and the
> following proposal tries to address
> all the issues mentioned there.
>
> With the version 2 of the Orocos toolchain some years ago, the previous data
> flow architecture, which was based on a
> single data object and buffer shared between all ports participating at the
> same connection, was replaced by a
> connection architecture based on data flow, where each pair of output and
> input ports (or streams) is connected
> through an individual channel, each having its own data object or buffer.
> While there was a well substantiated
> motivation for this update and it solved some of the shortcomings in RTT
> version 1 [2], it also introduced some new,
> sometimes hidden, problems and broke other use cases. One example is an input
> port with multiple connections, where
> due to the ambiguity in which channel to read from the result is sometimes
> unexpected to the user, who expects that
> each write overwrites previous samples in case of simple data connections.
>
> We assembled a document (see attachment, or [3]) which introduces the version
> 1 and version 2 data flow semantics
> for those who are not aware of the details, three example use cases and
> presents the new dataflow architecture we
> have in mind, which ideally works for a superset of use cases that are
> supported by RTT version 1 and 2. 
>
> We do not want to go into details here, so only a short summary for those who
> do not want to read the full document:
> We propose to add two more boolean flags to the ConnPolicy struct that
> describes the type of connection between two
> ports: "shared" and "mandatory". Furthermore the existing "pull" flag, which
> was only meaningful for remote
> connections, now has a big influence for local connections.
>
> "push" vs. "pull"
>
>
> Thanks for this nice effort. I think it is really time to consider a next
> generation of Orocos RTT...
>
> If you allow me to bring in my two cents... It's possible that you won't like
> the
> message, or find it off-topic, but I think I should share it with you anyway…
>
> These are interesting ideas Herman, but you're right in that they're somewhat
> off topic. For this discussion we are interested in the here and now - in
> evolutionary and incremental changes to a product that has served many of us
> fairly well.
>
> The evolution from the current "data flow architecture" to a "event stream"
> _can_/_could_ be done in an incremental way...
> Policy by policy. (i.e., pull/push; read/write; buffer/port;...)
> Separation by separation. (i.e., component/activity/thread/OS; pub/sub to
> event stream;...)
>  
> With all due respect, I think that your idea of "incremental" and mine are
> two very different things (and both are equally valid). It took us two full
> years to convert our system from RTT v1 to RTT v2. There is no way we would
> even consider moving to a RTT replacement without some serious review and up
> front discover and prototype use of a replacement system. So incremental for
> us is defined in months to years …

Again I agree. But I see no reason why introducing less strict policies can
not be done alongside keeping the existing ones...
Of course, it would result in a transition phase with even more conceptual
complexity than what is there now already, but it would allow the kind of
comparative prototyping you have in mind.

My concrete suggestion for such incremental "revolution": add a "event
stream" to complement the "lock-free buffer", as a second realtime-ready
internal IPC. This development is worth the effort anyway, and I know of no
implementation that is realtime ready... (Building further on TLSF and
MQueue might be low hanging fruit. Might be...)
This boils down to _adding_ a feature, which is always simpler than
deprecating one. And _if_ it turns out that it can cover the use cases of
the lockfree buffer (and some others) equally well, one could deprecate
those, later one.

> Personally, I'm not so worried about "better" as I am worried
> about "good enough".
>
> I am mostly interested in "simple". Because that is what determines how
> much of the potential mind share a project actually gets.
> (Mind that I am not interested in "easy", which is the driver for ROS, and
> that serves another public well.)
>  
> +1 on simple.
> S
>
>

Updated dataflow semantics for RTT

On Sep 27, 2015, at 08:11 AM, Herman Bruyninckx <Herman [dot] Bruyninckx [..] ...> wrote:

On Mon, 21 Sep 2015, Johannes Meyer wrote:

Dear Orocos community,
during the last few weeks we have been working on the concept and implementation of a major redesign of the Orocos
RTT dataflow, that we would like to share and ask for your valuable feedback. There was already a previous
discussion on the mailing list about the topic earlier this year [1], and the following proposal tries to address
all the issues mentioned there.

With the version 2 of the Orocos toolchain some years ago, the previous data flow architecture, which was based on a
single data object and buffer shared between all ports participating at the same connection, was replaced by a
connection architecture based on data flow, where each pair of output and input ports (or streams) is connected
through an individual channel, each having its own data object or buffer. While there was a well substantiated
motivation for this update and it solved some of the shortcomings in RTT version 1 [2], it also introduced some new,
sometimes hidden, problems and broke other use cases. One example is an input port with multiple connections, where
due to the ambiguity in which channel to read from the result is sometimes unexpected to the user, who expects that
each write overwrites previous samples in case of simple data connections.

We assembled a document (see attachment, or [3]) which introduces the version 1 and version 2 data flow semantics
for those who are not aware of the details, three example use cases and presents the new dataflow architecture we
have in mind, which ideally works for a superset of use cases that are supported by RTT version 1 and 2. 

We do not want to go into details here, so only a short summary for those who do not want to read the full document:
We propose to add two more boolean flags to the ConnPolicy struct that describes the type of connection between two
ports: "shared" and "mandatory". Furthermore the existing "pull" flag, which was only meaningful for remote
connections, now has a big influence for local connections.

"push" vs. "pull"

Thanks for this nice effort. I think it is really time to consider a next
generation of Orocos RTT...

If you allow me to bring in my two cents... It's possible that you won't like the
message, or find it off-topic, but I think I should share it with you anyway…

These are interesting ideas Herman, but you're right in that they're somewhat off topic. For this discussion we are interested in the here and now - in evolutionary and incremental changes to a product that has served many of us fairly well. Personally, I'm not so worried about "better" as I am worried about "good enough".

I'd be interesting in exploring some of what you've said here in another discussion.

Cheers
Stephen

Updated dataflow semantics for RTT

On Mon, 28 Sep 2015, S Roderick wrote:

> On Sep 27, 2015, at 08:11 AM, Herman Bruyninckx
> <Herman [dot] Bruyninckx [..] ...> wrote:
>
> On Mon, 21 Sep 2015, Johannes Meyer wrote:
>
> Dear Orocos community,
> during the last few weeks we have been working on the concept and
> implementation of a major redesign of the Orocos
> RTT dataflow, that we would like to share and ask for your valuable feedback.
> There was already a previous
> discussion on the mailing list about the topic earlier this year [1], and the
> following proposal tries to address
> all the issues mentioned there.
>
> With the version 2 of the Orocos toolchain some years ago, the previous data
> flow architecture, which was based on a
> single data object and buffer shared between all ports participating at the
> same connection, was replaced by a
> connection architecture based on data flow, where each pair of output and
> input ports (or streams) is connected
> through an individual channel, each having its own data object or buffer.
> While there was a well substantiated
> motivation for this update and it solved some of the shortcomings in RTT
> version 1 [2], it also introduced some new,
> sometimes hidden, problems and broke other use cases. One example is an input
> port with multiple connections, where
> due to the ambiguity in which channel to read from the result is sometimes
> unexpected to the user, who expects that
> each write overwrites previous samples in case of simple data connections.
>
> We assembled a document (see attachment, or [3]) which introduces the version
> 1 and version 2 data flow semantics
> for those who are not aware of the details, three example use cases and
> presents the new dataflow architecture we
> have in mind, which ideally works for a superset of use cases that are
> supported by RTT version 1 and 2. 
>
> We do not want to go into details here, so only a short summary for those who
> do not want to read the full document:
> We propose to add two more boolean flags to the ConnPolicy struct that
> describes the type of connection between two
> ports: "shared" and "mandatory". Furthermore the existing "pull" flag, which
> was only meaningful for remote
> connections, now has a big influence for local connections.
>
> "push" vs. "pull"
>
>
> Thanks for this nice effort. I think it is really time to consider a next
> generation of Orocos RTT...
>
> If you allow me to bring in my two cents... It's possible that you won't like
> the
> message, or find it off-topic, but I think I should share it with you anyway…
>
> These are interesting ideas Herman, but you're right in that they're somewhat
> off topic. For this discussion we are interested in the here and now - in
> evolutionary and incremental changes to a product that has served many of us
> fairly well.

The evolution from the current "data flow architecture" to a "event stream"
_can_/_could_ be done in an incremental way...
Policy by policy. (i.e., pull/push; read/write; buffer/port;...)
Separation by separation. (i.e., component/activity/thread/OS; pub/sub to
event stream;...)

> Personally, I'm not so worried about "better" as I am worried
> about "good enough".

I am mostly interested in "simple". Because that is what determines how
much of the potential mind share a project actually gets.
(Mind that I am not interested in "easy", which is the driver for ROS, and
that serves another public well.)

> I'd be interesting in exploring some of what you've said here in another
> discussion.

Fine! :-)

> Cheers
> Stephen

Herman

>
>

Updated dataflow semantics for RTT

involved parties)

> Enforcing in order arrive on the reading side is one way of doing it. You
> can also do it at a system level if you chose. They're equally valid use
> cases.

Can you describe a use case where "order of arrival" is good enough
for you ? That would help me understand your use-case.

(For some clearer thoughts on this,
http://rock-robotics.org/master/documentation/data_processing/data.html
http://rock-robotics.org/master/documentation/data_processing/stream_ali...)

>> orogen only deals with single components. Syskit is the one inmpacted by
>> this change, and it is for me out of the picture to compute all possible
>> connections in all possible networks. Pull connections are not practical as
>> soon as you have a inter-host connection (i.e. remote connection) as they
>> break the reading part.
>
> Can you explain this last part a little more Sylvain?

Reading on remote+pull CORBA connections requires the reader to do
network access (it's actually the whole use-case of pull for us, that
the writer is not at all affected by the connection)

> Isn't the current policy per connection, and not per port? If you have 2
> output ports and 3 input ports, and you connect them all together, then you
> have up to 2*3=6 possible connections/policies. Right?

If you put the buffer per-port, you end up with one policy per port.
Port-driven triggering (event ports) are governed by the data object
(i.e. per-port), data object size and type becomes per-port as well.
What's left in the connection policy is the info necessary to reach
the other side (i.e. the topic name in ROS), which means that it is
not a policy anymore but a "remote endpoint"

Interestingly, you'll have a hard time figuring out how to configure
the CORBA dispatcher (which currently uses the same policy than the
whole connection).

> What do you mean by "dynamic" networks?

That are not established once, but are modified at runtime (i.e.
connections are created/removed at runtime). With syskit, that happens
often and computing the network necessary to run them all is in
principle feasible, but in practice useless -- at the least in the
development phases.

> > Buffer connection: read() returns all samples received by the input port
> > (apart from buffer size limitations).
>
> Well, I'd phrase this differently. A buffer allows more than one sample to
> exist, in an ordered form, between a writer and a reader. "All" samples can
> only be "guaranteed" with very particular buffer sizing or additional
> application semantics.

+1, closer to the current semantic.

Sylvain

2015-09-28 8:32 GMT-03:00 Stephen Roderick <snrkiwi [..] ...>:
> On Sep 25, 2015, at 09:11 PM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:
>
>> It doesn't fix the out-of-order bug for the buffer-per-channel case.
>> You can still starve a certain channel and get old data much later.
>>
>> That's why we propose to change the default. This starving already
>>
>> happens in very simple/clean setups, unless you start to round-robin
>>
>> all the channels. But again, this does not prevent out-of-order arrivals.
>
>
> This is one thing where we have a completely different point of view. If you
> need in-order arrival, you have to enforce it on the reading side with
> something like Rock's stream aligner. The only case where the proposed
> change will allow you some in-order guarantees is the local connection case
> anyways, in all other cases you have unbounded latencies that do reordering.
> As for stale data: if you expect buffers to be used on your inputs, the
> receiving component must always read all data available (hence, no "very old
> stale data"). Otherwise, you will always have some form of buildup in the
> buffers, in a form you can't control. Finally, sharing a single buffer
> allows one high-frequency connection to starve all others by filling the
> buffer.
>
> Enforcing in order arrive on the reading side is one way of doing it. You
> can also do it at a system level if you chose. They're equally valid use
> cases.
>
> And agreed on local connections, which is primarily what we're hoping that
> this functionality fixes.
>
>>
>> I prefer the proposed solution since the default policy leads to a
>> much more predictable data flow. For appropriately sizing buffers
>>
>> for push channels in multi-rate component systems, we thought
>>
>> about it to specify on the input port level the buffer size
>> (there is already a default conn policy for each input port). A code
>>
>> generator like orogen could easily calculate the largest buffer size
>>
>> required between components and inform the input port, or stick to
>> pull connections.
>
>
> orogen only deals with single components. Syskit is the one inmpacted by
> this change, and it is for me out of the picture to compute all possible
> connections in all possible networks. Pull connections are not practical as
> soon as you have a inter-host connection (i.e. remote connection) as they
> break the reading part.
>
>
> Can you explain this last part a little more Sylvain?
>
>
> The bottom line is: the proposed change is really broken in the use case of
> networks that are dynamic, which is my very use case. Making the new (or the
> old) behaviour optional would be also hard because they really demand
> different APIs (you can't keep a policy in the connection API when truly the
> policy is per-port).
>
> Isn't the current policy per connection, and not per port? If you have 2
> output ports and 3 input ports, and you connect them all together, then you
> have up to 2*3=6 possible connections/policies. Right?
>
> What do you mean by "dynamic" networks?
>
>
>>
>> I would definately not 'imply' different behaviours depending on
>> buffer/data. After all, a data object is just a circular buffer of size 1.
>
>
> I thought the same, but start to disagree. Data and buffer have (and should
> have) different semantics. The fact that the data object is the same don't
> change that.
>
> Data connection: read() only returns the last sample received by the input
> port
>
>
> +1
>
> Buffer connection: read() returns all samples received by the input port
> (apart from buffer size limitations).
>
>
> Well, I'd phrase this differently. A buffer allows more than one sample to
> exist, in an ordered form, between a writer and a reader. "All" samples can
> only be "guaranteed" with very particular buffer sizing or additional
> application semantics.
>
> Cheers
> S
>
>

Updated dataflow semantics for RTT

> push: Every input port has a single input buffer and all connected writers
> write into that buffer.

That still sounds very much like "shared" to me:

> shared: Multiple input ports can read from the same buffer, so one read
> operation can influence the outcome of another.

Sylvain

Updated dataflow semantics for RTT

On Tue, Sep 22, 2015 at 1:48 PM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...>
wrote:

> > push: Every input port has a single input buffer and all connected
> writers
> > write into that buffer.
>
> That still sounds very much like "shared" to me:
>
> > shared: Multiple input ports can read from the same buffer, so one read
> > operation can influence the outcome of another.
>

For private connections the buffer (or buffers in pull mode) belongs to a
single input port, which does not mean that it cannot be filled by multiple
writers (in push mode).
For shared connections multiple input ports read from the same buffer. So
the "shared" here is meant from the viewpoint of the input port ("I have my
own input buffer." vs. "I have to share my input buffer with other
readers."), not for output ports. Output ports may always serve multiple
connections/channels except in the shared pull mode ("I have a new sample
available and whoever polls me first will get it.").

Johannes

Updated dataflow semantics for RTT

> For private connections the buffer (or buffers in pull mode) belongs to a
> single input port, which does not mean that it cannot be filled by multiple
> writers (in push mode).

> For shared connections multiple input ports read from the same buffer. So
> the "shared" here is meant from the viewpoint of the input port ("I have my
> own input buffer." vs. "I have to share my input buffer with other
> readers."), not for output ports. Output ports may always serve multiple
> connections/channels except in the shared pull mode ("I have a new sample
> available and whoever polls me first will get it.").

Ah. Got the difference now... However, my comment applies on the
"single buffer per input port" model: the first connection (or
whatever it is ...) will define what type of data object should be
used and what size it has. That should definitely be a conscious
system designer choice.

Sylvain

Updated dataflow semantics for RTT

2015-09-21 22:03 GMT+02:00 Johannes Meyer <johannes [..] ...>:

> Hi Sylvain,
>
> thanks for your quick reply!
>
> On Mon, Sep 21, 2015 at 8:09 PM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...>
> wrote:
>
>> Quick comment -- I don't have a lot of time on my hands right now:
>>
>> To me, it looks like you have decided that push is shared by default,
>> in your description the concept of pull/push and private/shared. Am I
>> wrong ? Is "push/shared" different than "push with single buffer by
>> default" ? I would not see the rationale for that, if a use wants
>> "private", give him "private", especially given that there are
>> shortcomings ...
>>
>> Using "shared" will make the system fail if, later, it gets a
>> connection request for a different policy than the original policy. To
>> me, this spells that using shared MUST be a conscious choice of the
>> system designer in all cases OR that there should be a way for RTT to
>> degrade gracefully from shared to private (use shared as an
>> optimization that is handled purely internally unless the user
>> requested it explicitely).
>>
>> In general, the connection code will have to return an error if trying
>> incompatible mixes of policies.
>>
>> Sylvain
>>
>
>
> That's exactly what he have in mind and how it is hopefully implemented.
> push/pull and private/shared are orthogonal concepts. The default would be
> push/private. Connections are never shared unless explicitly specified by
> the user. Probably the table in section 5.1 of our design document
> <https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4KaotpCtXM4E0/pub#h.yfd89u4uuk9u> is
> the easiest mean to get a fast overview of the possible combinations.
>
> These are the shortest possible definitions I can come up with:
>
> push: Every input port has a single input buffer and all connected writers
> write into that buffer.
> pull: Every output port has one buffer per connection and readers will
> poll all connected writers for new data.
>
> private: Every connection buffer belongs to exactly one input port, so all
> read operations are independent from each other.
> shared: Multiple input ports can read from the same buffer, so one read
> operation can influence the outcome of another.
>
> The combination shared/pull is less intuitive, but we interpreted it as a
> single local output buffer and multiple polling input ports read from the
> same buffer (vs. one buffer per connection in the pull/private case).
> Actually this connection type would only be useful for shared remote
> connections, where multiple remote input ports are supposed to consume
> samples from a single output port with a shared local buffer. For local
> connections, there is no reason not to use a shared/push connection.
>
> Clearly there are limitations for the possible combinations of different
> connection types for a single port, primarily that an input port can have
> either only pull or only push connections and that all push connections
> have to be consistent in the buffer type, size and locking policy. For
> illegal combinations the connectTo(...) request (or whatever API is used to
> establish new connections) will return an error. Users have to be more
> careful and aware of the implications of ConnPolicy flags, but we think the
> default private/push should work for most applications and not break
> existing deployments, unless they already use a mixture of connection
> policies for the same port for whatever reason.
>
> Johannes
>
>
> --
> Johannes Meyer, Senior Application Engineer
> +32 468 139828
> Intermodalics - Kapeldreef 60, 3001 Heverlee - BELGIUM
> www.intermodalics.eu
>
> --
> Orocos-Dev mailing list
> Orocos-Dev [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>
>
Do you know about RabbitMq ? It may be a source of inspiration, or if
doing the job, a good externalisation.
https://www.rabbitmq.com/getstarted.html

Updated dataflow semantics for RTT

Dear Orocos community,

during the last few weeks we have been working on the concept and
implementation of a major redesign of the Orocos RTT dataflow, that we
would like to share and ask for your valuable feedback. There was already a
previous discussion on the mailing list about the topic earlier this year
[1], and the following proposal tries to address all the issues mentioned
there.

With the version 2 of the Orocos toolchain some years ago, the previous
data flow architecture, which was based on a single data object and buffer
shared between all ports participating at the same connection, was replaced
by a connection architecture based on data flow, where each pair of output
and input ports (or streams) is connected through an individual channel,
each having its own data object or buffer. While there was a well
substantiated motivation for this update and it solved some of the
shortcomings in RTT version 1 [2], it also introduced some new, sometimes
hidden, problems and broke other use cases. One example is an input port
with multiple connections, where due to the ambiguity in which channel to
read from the result is sometimes unexpected to the user, who expects that
each write overwrites previous samples in case of simple data connections.

We assembled a document (see attachment, or [3]) which introduces the
version 1 and version 2 data flow semantics for those who are not aware of
the details, three example use cases and presents the new dataflow
architecture we have in mind, which ideally works for a superset of use
cases that are supported by RTT version 1 and 2.

We do not want to go into details here, so only a short summary for those
who do not want to read the full document:
We propose to add two more boolean flags to the ConnPolicy struct that
describes the type of connection between two ports: "shared" and
"mandatory". Furthermore the existing "pull" flag, which was only
meaningful for remote connections, now has a big influence for local
connections.

*"push" vs. "pull"*

Like it is already the case for remote connections via CORBA, the "pull"
flag will decide whether the connection's data object or buffer is
installed at the writer's side (pull) or the reader's side (push) of the
connection. The first case is equivalent to the current version 2
connection model and readers have to select the connection/output port to
read from, while for push connections there will be only one single buffer
per input port. We think that the push connection should be the default
connection type for local ports, but this might break existing applications
which rely on having one buffer per connection now.

*"private" vs. "shared"*

The shared flag reintroduces the shared data connection model that was
standard in RTT version 1. For shared connections multiple input ports read
from the same data object or consume samples from the same buffer instance,
so their read operations can influence each other. Shared connections are
especially useful for producer/consumer applications with multiple
consumers.

*"mandatory"*

The introduction of the "mandatory" flag was necessary to decide on the
return value of the write() operation on an output port. In RTT version 2
it was not possible anymore to get a feedback of whether a write operation
was successful or failed, e.g. because of a full buffer or broken remote
connection. Only connections with the mandatory flag set would be
considered for the overall result of write() operations, while failures of
non-mandatory connections will be ignored.

For those who want to take a look at the current state of the
implementation, this is the link to the working branch on GitHub:
Tree:
https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics
Diff to master:
https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflo...

The implementation is not fully working yet for non-standard transports
(like Corba and mqueue), but especially the updated test ports_test.cpp
already shows what would change from a user perspective.

Obviously, such a major refactoring cannot be done without changes in the
public API. However, we tried to keep the required changes in user code as
minimal as possible.
The new dataflow semantics are targeted for an upcoming version 2.9 release
(or even name it 3.0 to communicate this potentially breaking change?).

We are planning to also come up with some comparative performance
measurements during this week, as people have asked for that before [1].

Best regards,
Johannes and Peter

[1]
http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation
[2] http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow
[3]
https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4Kaotp...

Updated dataflow semantics for RTT

On Mon, 21 Sep 2015, Johannes Meyer wrote:

> Dear Orocos community,
> during the last few weeks we have been working on the concept and implementation of a major redesign of the Orocos
> RTT dataflow, that we would like to share and ask for your valuable feedback. There was already a previous
> discussion on the mailing list about the topic earlier this year [1], and the following proposal tries to address
> all the issues mentioned there.
>
> With the version 2 of the Orocos toolchain some years ago, the previous data flow architecture, which was based on a
> single data object and buffer shared between all ports participating at the same connection, was replaced by a
> connection architecture based on data flow, where each pair of output and input ports (or streams) is connected
> through an individual channel, each having its own data object or buffer. While there was a well substantiated
> motivation for this update and it solved some of the shortcomings in RTT version 1 [2], it also introduced some new,
> sometimes hidden, problems and broke other use cases. One example is an input port with multiple connections, where
> due to the ambiguity in which channel to read from the result is sometimes unexpected to the user, who expects that
> each write overwrites previous samples in case of simple data connections.
>
> We assembled a document (see attachment, or [3]) which introduces the version 1 and version 2 data flow semantics
> for those who are not aware of the details, three example use cases and presents the new dataflow architecture we
> have in mind, which ideally works for a superset of use cases that are supported by RTT version 1 and 2. 
>
> We do not want to go into details here, so only a short summary for those who do not want to read the full document:
> We propose to add two more boolean flags to the ConnPolicy struct that describes the type of connection between two
> ports: "shared" and "mandatory". Furthermore the existing "pull" flag, which was only meaningful for remote
> connections, now has a big influence for local connections.
>
> "push" vs. "pull"
>

Thanks for this nice effort. I think it is really time to consider a next
generation of Orocos RTT...

If you allow me to bring in my two cents... It's possible that you won't like the
message, or find it off-topic, but I think I should share it with you anyway...

The best document I have read about communication that connects
high-performance with flexibility and configurability is this one:
<https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying>

It cleanly separates the communication of "events" from (i) what they mean,
(ii) who is interested in them, and (iii) with what policy they are consumed.

In our current research at KU Leuven, this is going to be _the_ basis for
all interaction between asynchronous activities.
The other basis will be the separation between (i) functional requirements
of the application, (ii) "roles and responsibilities" of "components",
(iii) software components to implement both, and (iv) deployment of these
components on your hardware and OS platforms.

All of the above are not present in Orocos RTT, which I consider design flaws.
So solving these problems is definitely worthwhile. But, and this is a big
but, my view on solving them (which is currently being transformed into
a first set of models and code and tools) implies that we do not need
Orocos RTT at all anymore. The reasons why we started RTT more than fifteen
years ago have not disappeared, but have all been solved in more "mainstream"
software projects than Orocos. Fifteen years ago, what we where after was a
"node-port-connector" modelling-and-coding interface, where nodes could be
deployed in realtime activities on an operating system. It was the time of
single core CPUs, with only proprietary solutions to fast networking. Those
days have gone, together with the need to use RTT as a software solution.

More in particular, the "communication" aspect (which is the core of the
document you distributed) is a lot better understood and supported in
mainstream than 15 years ago. Taking the model of the event log and the
corresponding log processing around it means that all the policies that
your document wants to introduce as "API" methods can (should...) better be
realised by configurations of the event log mechanism in your _application
architecture_; there is no need to provide APIs in "middleware",
"frameworks" or "libraries".

_The_ core functionalities of RTT that still survive after 15 years are:
- its introspection capabilities;
- the lock-free data buffer.
The latter is just a special case ("configuration") of the event log
mechanism, that provides all other policies that you mention (and more!)
just through other configurations.
The introspection is a form of "model based" implementation, which
considers models as first-class citizens in the software; this is great,
and should be extended, starting with modelling all the relevant event log
policies explicity. (KU Leuven is working on that; it takes a lot of
efforts...) Lock-free data buffers are the core of event log processing,
and are currently better known under the new buzzword "immutable data".

The other "good-intuition-that-turned-out-to-be-badly-implemented" of
fifteen years ago is that of "stored procedures" (known under various
names, such as "microservices", proxy computations, etc.). RTT provides a
very complex and ambiguous API to let one component execute code for
another component. Again, that functionality should not be provided by an
"RPC API", but by models and tools of _deployment_.

The added value of Orocos as a project could shift from providing code to
providing professional quality documentation and tutorials for "best
practices" and "patterns", some specific realtime-optimized
implementations, or patches to the large amount of existing libraries that
support alul of the above.

It is up to the Orocos community to decide whether to go that direction or
not. I would be glad to cooperate in that directoion, because otherwise we
will be forced to start a "competing" project.

Note that this message does not contain threats to "fork" or whatever, but
just my insights after working with Orocos RTT for about a decade, and
having Orocos RTT work against the architectural requirements in our
robotics projects for the subsequent five years...

I frankly think it is time to reinvent the project. Big time.

The easiest way is to _deprecate_ things, to make the _core_ things simpler
for users, throwing out more and more code and start using other already
existing software (ZeroMQ, libevent, the lock-free buffer library,
RethinkDB,...) instead of trying to squeeze in their functionalities within
NIH APIs and implementations. In the end, RTT will be empty, and the
development efforts can then finally shift to give Sylvain a much bigger
hand with ROCK :-)
(Just make sure to refactor ROCK in order to decouple the _models_ from the
Ruby and RTT _implementations_ :-))

Best regards,

Herman Bruyninckx

> Like it is already the case for remote connections via CORBA, the "pull" flag will decide whether the connection's
> data object or buffer is installed at the writer's side (pull) or the reader's side (push) of the connection. The
> first case is equivalent to the current version 2 connection model and readers have to select the connection/output
> port to read from, while for push connections there will be only one single buffer per input port. We think that the
> push connection should be the default connection type for local ports, but this might break existing applications
> which rely on having one buffer per connection now.
>
> "private" vs. "shared"
>
> The shared flag reintroduces the shared data connection model that was standard in RTT version 1. For shared
> connections multiple input ports read from the same data object or consume samples from the same buffer instance, so
> their read operations can influence each other. Shared connections are especially useful for producer/consumer
> applications with multiple consumers.
>
> "mandatory"
>
> The introduction of the "mandatory" flag was necessary to decide on the return value of the write() operation on an
> output port. In RTT version 2 it was not possible anymore to get a feedback of whether a write operation was
> successful or failed, e.g. because of a full buffer or broken remote connection. Only connections with the mandatory
> flag set would be considered for the overall result of write() operations, while failures of non-mandatory
> connections will be ignored.
>
>
> For those who want to take a look at the current state of the implementation, this is the link to the working branch
> on GitHub:
> Tree: https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics
> Diff to master: https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflow-semantics
>
> The implementation is not fully working yet for non-standard transports (like Corba and mqueue), but especially the
> updated test ports_test.cpp already shows what would change from a user perspective.
>
> Obviously, such a major refactoring cannot be done without changes in the public API. However, we tried to keep the
> required changes in user code as minimal as possible.
> The new dataflow semantics are targeted for an upcoming version 2.9 release (or even name it 3.0 to communicate this
> potentially breaking change?).
>
> We are planning to also come up with some comparative performance measurements during this week, as people have
> asked for that before [1].
>
>
> Best regards,
> Johannes and Peter
>
> [1] http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation
> [2] http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow
> [3] https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4Kaotp...
>
>
>

Updated dataflow semantics for RTT

On Sep 28, 2015, at 03:44 PM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:

Some classic producer/consumer examples
- a camera captures images and sends them on to a processing pipeline (that
is all data object connections). A separate connection stores the camera
images in a buffer, which are dumped to disk.
- data is read (non-real-time) from a file and pushed into a buffer for
processing (real-time)
- data samples coming from a remote source (non-real-time) is buffered on
arrival into the deployer for later processing (real-time).

Asked the wrong question ... thanks for answering it perfectly ;-)

Anyways, what I wanted is a case where:
- an input port is connected to a bunch of output ports *using
buffers* (I got the problem with data policies)
- the overall behavior is governed by maintaining the overall
incoming sample order (i.e. that if a sample of output port O_1
arrives before a sample of output port O_2, that order needs to be
reflected on read())
 
I think that all of the buffer uses in our system are classic single producer/single consumer connections.  I can't think of any N:M set of connections using buffers.

I may be missing it, but the use-cases above don't cover that, do they ?

> Agreed, but this choice is currently bundled with having a buffer per
> channel or one per input port.

I don't follow, Sylvain. Bundled in the existing implementation or the
proposed changes?

In the proposed change (current v2 does not have the option of having
a single buffer per input port). Until now, it is clear to me that
this bundling is both Johannes and Peter's intent (see Peter's email
above). It does not seem to be yours, though.
 
Well, yes and no. I'm not going to throw Johannes and Peter under the bus. A lot of this work is at our urging, and we've been involved with formulating and evaluating possible solutions. They have put together a solution that they believe solves some of the issues that we, and others, have bought up. They have also come to this proposed solution by actually implementing (some of) it and running into quite a few corner cases, etc.

> If you put the buffer per-port, you end up with one policy per port.
> Port-driven triggering (event ports) are governed by the data object
> (i.e. per-port), data object size and type becomes per-port as well.
> What's left in the connection policy is the info necessary to reach
> the other side (i.e. the topic name in ROS), which means that it is
> not a policy anymore but a "remote endpoint"

From your comment above "Port-driven triggering (event ports) …"

Badly ordered bits of sentences. What I meant is:

If you put the buffer per-port, you end up with one policy per port. Data object size and type becomes per-port. If using port-driven triggering (event ports), the triggering is governed by the data object and is therefore also becoming per-port.

Looking at Section 5.1 of Johannes' google doc, something looks wrong or I'm
remembering previous discussions incorrectly. I was expecting push/pull vs
shared/private to be truly orthogonal, but the push/private figure says
differently. Johannes? Peter?

They already clarified that for me. Basically, "private" does not
necessarily means "per channel". On a push connection, "private" for
them means "per-port".
 
In my head, private means per "connection" (I think that is your "channel" also). The proposed solution may not have that for good reasons, which I'll let them state.

Again, my intent was to add more v1-like semantics to the system, as well as
have more control on where latency occurred. Not to replace any aspect of
the v2 semantics.

I got that from your previous email already, but it seems that it's
not Johannes/Peter's intent
 
Again, they're trying to fix multiple issues in one go (and all the issues are pretty inter-related). They're also dealing with some pretty horrific corner cases, and they're doing the best they can.

HTH
S

Updated dataflow semantics for RTT

Just to clarify one thing: it was definitely not my intent to "throw
Johannes and/or Peter under the bus". My apologies to both of them if that
is how it looked like.

Sylvain

Updated dataflow semantics for RTT

> On Sep 28, 2015, at 16:51, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:
>
> Just to clarify one thing: it was definitely not my intent to "throw Johannes and/or Peter under the bus". My apologies to both of them if that is how it looked like.
>
> Sylvain
>

LOL … that comment was meant at me Sylvain, not you. It’d be pretty crappy if I’d pushed Peter and Johannes to help with this port “problem", and then not backed them up when they came up with something slightly different and that was alarming people …
S

Updated dataflow semantics for RTT

On Mon, Sep 28, 2015 at 10:51 PM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:
> Just to clarify one thing: it was definitely not my intent to "throw
> Johannes and/or Peter under the bus". My apologies to both of them if that
> is how it looked like.

Apologies are certainly not accepted :-) We were worried that we would
gather too little feedback due to the complex nature of this proposal.
This discussion is really contributing and improving/changing our
proposals for the better, so it feels more like "on the bus" (one with
a piece symbol painted on it :-) ).

We really didn't understand Sylvain's/Rock's usecase and thought that
the current default was broken *for everyone*. Now that we see it
isn't, we of course keep them, in order to stay away from this
scenario: https://i.imgflip.com/rl3g1.jpg

Now that we're top-posting anyway: I believe Johannes' long email
needs some extra drawings to clarify the new, backwards compatible,
connpolicy we have, so that is yet to come in the design document.

Also, all items under Johannes' point "3" apply only on the 'new'
shared connections, not on the traditional connections (ie,
traditional connections don't look up name_id to re-use connection
objects).

Finally, these past weeks, the dataflow has gotten the best review it
got in years, that alone adds majorly to the overall health of the
project. We should indeed not un-do that by breaking things...

Peter

Updated dataflow semantics for RTT

On 27/9/15, 14:10, "orocos-dev-bounces [..] ... on behalf of
Herman Bruyninckx" <orocos-dev-bounces [..] ... on behalf of
herman [dot] bruyninckx [..] ...> wrote:
>
>Thanks for this nice effort. I think it is really time to consider a next
>generation of Orocos RTT...
>
>If you allow me to bring in my two cents... It's possible that you won't
>like the
>message, or find it off-topic, but I think I should share it with you
>anyway...
>
>The best document I have read about communication that connects
>high-performance with flexibility and configurability is this one:
>
><https://engineering.linkedin.com/distributed-systems/log-what-every-softw
>are-engineer-should-know-about-real-time-datas-unifying>
>
>It cleanly separates the communication of "events" from (i) what they
>mean,
>(ii) who is interested in them, and (iii) with what policy they are
>consumed.
>
>In our current research at KU Leuven, this is going to be _the_ basis for
>all interaction between asynchronous activities.
>The other basis will be the separation between (i) functional requirements
>of the application, (ii) "roles and responsibilities" of "components",
>(iii) software components to implement both, and (iv) deployment of these
>components on your hardware and OS platforms.
>
>All of the above are not present in Orocos RTT, which I consider design
>flaws.
>So solving these problems is definitely worthwhile. But, and this is a big
>but, my view on solving them (which is currently being transformed into
>a first set of models and code and tools) implies that we do not need
>Orocos RTT at all anymore. The reasons why we started RTT more than
>fifteen
>years ago have not disappeared, but have all been solved in more
>"mainstream"
>software projects than Orocos. Fifteen years ago, what we where after was
>a
>"node-port-connector" modelling-and-coding interface, where nodes could be
>deployed in realtime activities on an operating system. It was the time of
>single core CPUs, with only proprietary solutions to fast networking.
>Those
>days have gone, together with the need to use RTT as a software solution.
>
>More in particular, the "communication" aspect (which is the core of the
>document you distributed) is a lot better understood and supported in
>mainstream than 15 years ago. Taking the model of the event log and the
>corresponding log processing around it means that all the policies that
>your document wants to introduce as "API" methods can (should...) better
>be
>realised by configurations of the event log mechanism in your _application
>architecture_; there is no need to provide APIs in "middleware",
>"frameworks" or "libraries".
>
>_The_ core functionalities of RTT that still survive after 15 years are:
>- its introspection capabilities;
>- the lock-free data buffer.
>The latter is just a special case ("configuration") of the event log
>mechanism, that provides all other policies that you mention (and more!)
>just through other configurations.
>The introspection is a form of "model based" implementation, which
>considers models as first-class citizens in the software; this is great,
>and should be extended, starting with modelling all the relevant event log
>policies explicity. (KU Leuven is working on that; it takes a lot of
>efforts...) Lock-free data buffers are the core of event log processing,
>and are currently better known under the new buzzword "immutable data".
>
>The other "good-intuition-that-turned-out-to-be-badly-implemented" of
>fifteen years ago is that of "stored procedures" (known under various
>names, such as "microservices", proxy computations, etc.). RTT provides a
>very complex and ambiguous API to let one component execute code for
>another component. Again, that functionality should not be provided by an
>"RPC API", but by models and tools of _deployment_.
>
>The added value of Orocos as a project could shift from providing code to
>providing professional quality documentation and tutorials for "best
>practices" and "patterns", some specific realtime-optimized
>implementations, or patches to the large amount of existing libraries that
>support alul of the above.
>
>It is up to the Orocos community to decide whether to go that direction or
>not. I would be glad to cooperate in that directoion, because otherwise we
>will be forced to start a "competing" project.
>
>Note that this message does not contain threats to "fork" or whatever, but
>just my insights after working with Orocos RTT for about a decade, and
>having Orocos RTT work against the architectural requirements in our
>robotics projects for the subsequent five years...
>
>I frankly think it is time to reinvent the project. Big time.
>
>The easiest way is to _deprecate_ things, to make the _core_ things
>simpler
>for users, throwing out more and more code and start using other already
>existing software (ZeroMQ, libevent, the lock-free buffer library,
>RethinkDB,...) instead of trying to squeeze in their functionalities
>within
>NIH APIs and implementations. In the end, RTT will be empty, and the
>development efforts can then finally shift to give Sylvain a much bigger
>hand with ROCK :-)
>(Just make sure to refactor ROCK in order to decouple the _models_ from
>the
> Ruby and RTT _implementations_ :-))
>
>Best regards,
>
>Herman Bruyninckx

There are some potentially interesting ideas in there, Herman. But either
your email is uncharacteristically incoherent or I’m really jet lagged.
Could you explain more clearly what you currently think is the ideal? For
clarity, we can skip the comments Orocos, since we’ve already established
that you think it needs to change.

Geoff

Updated dataflow semantics for RTT

On Mon, 28 Sep 2015, BiggsGeoffrey wrote:

> On 27/9/15, 14:10, "orocos-dev-bounces [..] ... on behalf of
> Herman Bruyninckx" <orocos-dev-bounces [..] ... on behalf of
> herman [dot] bruyninckx [..] ...> wrote:
>>
>> Thanks for this nice effort. I think it is really time to consider a next
>> generation of Orocos RTT...
>>
>> If you allow me to bring in my two cents... It's possible that you won't
>> like the
>> message, or find it off-topic, but I think I should share it with you
>> anyway...
>>
>> The best document I have read about communication that connects
>> high-performance with flexibility and configurability is this one:
>>
>> <https://engineering.linkedin.com/distributed-systems/log-what-every-softw
>> are-engineer-should-know-about-real-time-datas-unifying>
>>
>> It cleanly separates the communication of "events" from (i) what they
>> mean,
>> (ii) who is interested in them, and (iii) with what policy they are
>> consumed.
>>
>> In our current research at KU Leuven, this is going to be _the_ basis for
>> all interaction between asynchronous activities.
>> The other basis will be the separation between (i) functional requirements
>> of the application, (ii) "roles and responsibilities" of "components",
>> (iii) software components to implement both, and (iv) deployment of these
>> components on your hardware and OS platforms.
>>
>> All of the above are not present in Orocos RTT, which I consider design
>> flaws.
>> So solving these problems is definitely worthwhile. But, and this is a big
>> but, my view on solving them (which is currently being transformed into
>> a first set of models and code and tools) implies that we do not need
>> Orocos RTT at all anymore. The reasons why we started RTT more than
>> fifteen
>> years ago have not disappeared, but have all been solved in more
>> "mainstream"
>> software projects than Orocos. Fifteen years ago, what we where after was
>> a
>> "node-port-connector" modelling-and-coding interface, where nodes could be
>> deployed in realtime activities on an operating system. It was the time of
>> single core CPUs, with only proprietary solutions to fast networking.
>> Those
>> days have gone, together with the need to use RTT as a software solution.
>>
>> More in particular, the "communication" aspect (which is the core of the
>> document you distributed) is a lot better understood and supported in
>> mainstream than 15 years ago. Taking the model of the event log and the
>> corresponding log processing around it means that all the policies that
>> your document wants to introduce as "API" methods can (should...) better
>> be
>> realised by configurations of the event log mechanism in your _application
>> architecture_; there is no need to provide APIs in "middleware",
>> "frameworks" or "libraries".
>>
>> _The_ core functionalities of RTT that still survive after 15 years are:
>> - its introspection capabilities;
>> - the lock-free data buffer.
>> The latter is just a special case ("configuration") of the event log
>> mechanism, that provides all other policies that you mention (and more!)
>> just through other configurations.
>> The introspection is a form of "model based" implementation, which
>> considers models as first-class citizens in the software; this is great,
>> and should be extended, starting with modelling all the relevant event log
>> policies explicity. (KU Leuven is working on that; it takes a lot of
>> efforts...) Lock-free data buffers are the core of event log processing,
>> and are currently better known under the new buzzword "immutable data".
>>
>> The other "good-intuition-that-turned-out-to-be-badly-implemented" of
>> fifteen years ago is that of "stored procedures" (known under various
>> names, such as "microservices", proxy computations, etc.). RTT provides a
>> very complex and ambiguous API to let one component execute code for
>> another component. Again, that functionality should not be provided by an
>> "RPC API", but by models and tools of _deployment_.
>>
>> The added value of Orocos as a project could shift from providing code to
>> providing professional quality documentation and tutorials for "best
>> practices" and "patterns", some specific realtime-optimized
>> implementations, or patches to the large amount of existing libraries that
>> support alul of the above.
>>
>> It is up to the Orocos community to decide whether to go that direction or
>> not. I would be glad to cooperate in that directoion, because otherwise we
>> will be forced to start a "competing" project.
>>
>> Note that this message does not contain threats to "fork" or whatever, but
>> just my insights after working with Orocos RTT for about a decade, and
>> having Orocos RTT work against the architectural requirements in our
>> robotics projects for the subsequent five years...
>>
>> I frankly think it is time to reinvent the project. Big time.
>>
>> The easiest way is to _deprecate_ things, to make the _core_ things
>> simpler
>> for users, throwing out more and more code and start using other already
>> existing software (ZeroMQ, libevent, the lock-free buffer library,
>> RethinkDB,...) instead of trying to squeeze in their functionalities
>> within
>> NIH APIs and implementations. In the end, RTT will be empty, and the
>> development efforts can then finally shift to give Sylvain a much bigger
>> hand with ROCK :-)
>> (Just make sure to refactor ROCK in order to decouple the _models_ from
>> the
>> Ruby and RTT _implementations_ :-))
>>
>> Best regards,
>>
>> Herman Bruyninckx
>
> There are some potentially interesting ideas in there, Herman. But either
> your email is uncharacteristically incoherent or I’m really jet lagged.
> Could you explain more clearly what you currently think is the ideal?

1. Complete separation of _any_ communication policy from the activity
execution policies: there are just way too many use cases that must be
supported to bring them all behind one API. (Which is also what ROS2.0
tries to do.)
2. "Event processing" is the most flexible _mechanism_ to build other
communication policies upon. The most important lesson that _I_ learned
in this context is that it allows "clients" and "servers" to choose
whatever policy they like _locally_, since all data is immutable. (Except
for some "long term" garbage collection.)
3. Executing activities in realtime currently does not require a separate
library/framework anymore, but is realised by control groups (on Linux), or
similar functionalities on other OSs.
4. Similarly, systemd has (or is in the process of) taken over all
configuration issues.
5. The expertise that the RTT community excels in is that of _designing_
realtime systems, with or without the RTT code. And that is more than
enough to warrant the continuation of this community, even when none of
the code is kept, since "best practices" are useful everywhere.

> For clarity, we can skip the comments Orocos, since we’ve already
> established that you think it needs to change.

My thoughts go further: there is no need for it anymore.

> Geoff

Herman

Updated dataflow semantics for RTT

Some thoughts on that executive summary (I'm like geoffrey, I had a
hard time following the first email :P)

One thing I definitely want to warn you about: don't think, ever, that
code-generating everything is a replacement for a well-designed
library that solve most of the problems, along with a code generator
for the one percent.

With this premise: I actually that your points are not making RTT
obsolete. It is actually making it even more relevant than ever.

RTT *is* the best starting point towards something where
process/thread policies is delegated to the tooling (either accessing
"raw" OS-specific functionality or exposing them under an
easier-to-use interface). It is also *already* provides an
encapsulation of computation, which allows you to "inverse" the
current tendency and actually make your middleware central to your
communication setup -- instead of hiding it behind a common API.

Sylvain

2015-09-28 2:28 GMT-03:00 Herman Bruyninckx
<Herman [dot] Bruyninckx [..] ...>:
> On Mon, 28 Sep 2015, BiggsGeoffrey wrote:
>
>> On 27/9/15, 14:10, "orocos-dev-bounces [..] ... on behalf of
>> Herman Bruyninckx" <orocos-dev-bounces [..] ... on behalf of
>> herman [dot] bruyninckx [..] ...> wrote:
>>>
>>>
>>> Thanks for this nice effort. I think it is really time to consider a next
>>> generation of Orocos RTT...
>>>
>>> If you allow me to bring in my two cents... It's possible that you won't
>>> like the
>>> message, or find it off-topic, but I think I should share it with you
>>> anyway...
>>>
>>> The best document I have read about communication that connects
>>> high-performance with flexibility and configurability is this one:
>>>
>>>
>>> <https://engineering.linkedin.com/distributed-systems/log-what-every-softw
>>> are-engineer-should-know-about-real-time-datas-unifying>
>>>
>>> It cleanly separates the communication of "events" from (i) what they
>>> mean,
>>> (ii) who is interested in them, and (iii) with what policy they are
>>> consumed.
>>>
>>> In our current research at KU Leuven, this is going to be _the_ basis for
>>> all interaction between asynchronous activities.
>>> The other basis will be the separation between (i) functional
>>> requirements
>>> of the application, (ii) "roles and responsibilities" of "components",
>>> (iii) software components to implement both, and (iv) deployment of these
>>> components on your hardware and OS platforms.
>>>
>>> All of the above are not present in Orocos RTT, which I consider design
>>> flaws.
>>> So solving these problems is definitely worthwhile. But, and this is a
>>> big
>>> but, my view on solving them (which is currently being transformed into
>>> a first set of models and code and tools) implies that we do not need
>>> Orocos RTT at all anymore. The reasons why we started RTT more than
>>> fifteen
>>> years ago have not disappeared, but have all been solved in more
>>> "mainstream"
>>> software projects than Orocos. Fifteen years ago, what we where after was
>>> a
>>> "node-port-connector" modelling-and-coding interface, where nodes could
>>> be
>>> deployed in realtime activities on an operating system. It was the time
>>> of
>>> single core CPUs, with only proprietary solutions to fast networking.
>>> Those
>>> days have gone, together with the need to use RTT as a software solution.
>>>
>>> More in particular, the "communication" aspect (which is the core of the
>>> document you distributed) is a lot better understood and supported in
>>> mainstream than 15 years ago. Taking the model of the event log and the
>>> corresponding log processing around it means that all the policies that
>>> your document wants to introduce as "API" methods can (should...) better
>>> be
>>> realised by configurations of the event log mechanism in your
>>> _application
>>> architecture_; there is no need to provide APIs in "middleware",
>>> "frameworks" or "libraries".
>>>
>>> _The_ core functionalities of RTT that still survive after 15 years are:
>>> - its introspection capabilities;
>>> - the lock-free data buffer.
>>> The latter is just a special case ("configuration") of the event log
>>> mechanism, that provides all other policies that you mention (and more!)
>>> just through other configurations.
>>> The introspection is a form of "model based" implementation, which
>>> considers models as first-class citizens in the software; this is great,
>>> and should be extended, starting with modelling all the relevant event
>>> log
>>> policies explicity. (KU Leuven is working on that; it takes a lot of
>>> efforts...) Lock-free data buffers are the core of event log processing,
>>> and are currently better known under the new buzzword "immutable data".
>>>
>>> The other "good-intuition-that-turned-out-to-be-badly-implemented" of
>>> fifteen years ago is that of "stored procedures" (known under various
>>> names, such as "microservices", proxy computations, etc.). RTT provides a
>>> very complex and ambiguous API to let one component execute code for
>>> another component. Again, that functionality should not be provided by an
>>> "RPC API", but by models and tools of _deployment_.
>>>
>>> The added value of Orocos as a project could shift from providing code to
>>> providing professional quality documentation and tutorials for "best
>>> practices" and "patterns", some specific realtime-optimized
>>> implementations, or patches to the large amount of existing libraries
>>> that
>>> support alul of the above.
>>>
>>> It is up to the Orocos community to decide whether to go that direction
>>> or
>>> not. I would be glad to cooperate in that directoion, because otherwise
>>> we
>>> will be forced to start a "competing" project.
>>>
>>> Note that this message does not contain threats to "fork" or whatever,
>>> but
>>> just my insights after working with Orocos RTT for about a decade, and
>>> having Orocos RTT work against the architectural requirements in our
>>> robotics projects for the subsequent five years...
>>>
>>> I frankly think it is time to reinvent the project. Big time.
>>>
>>> The easiest way is to _deprecate_ things, to make the _core_ things
>>> simpler
>>> for users, throwing out more and more code and start using other already
>>> existing software (ZeroMQ, libevent, the lock-free buffer library,
>>> RethinkDB,...) instead of trying to squeeze in their functionalities
>>> within
>>> NIH APIs and implementations. In the end, RTT will be empty, and the
>>> development efforts can then finally shift to give Sylvain a much bigger
>>> hand with ROCK :-)
>>> (Just make sure to refactor ROCK in order to decouple the _models_ from
>>> the
>>> Ruby and RTT _implementations_ :-))
>>>
>>> Best regards,
>>>
>>> Herman Bruyninckx
>>
>>
>> There are some potentially interesting ideas in there, Herman. But either
>> your email is uncharacteristically incoherent or I’m really jet lagged.
>> Could you explain more clearly what you currently think is the ideal?
>
>
> 1. Complete separation of _any_ communication policy from the activity
> execution policies: there are just way too many use cases that must be
> supported to bring them all behind one API. (Which is also what ROS2.0
> tries to do.)
> 2. "Event processing" is the most flexible _mechanism_ to build other
> communication policies upon. The most important lesson that _I_
> learned
> in this context is that it allows "clients" and "servers" to choose
> whatever policy they like _locally_, since all data is immutable. (Except
> for some "long term" garbage collection.)
> 3. Executing activities in realtime currently does not require a separate
> library/framework anymore, but is realised by control groups (on Linux),
> or
> similar functionalities on other OSs.
> 4. Similarly, systemd has (or is in the process of) taken over all
> configuration issues.
> 5. The expertise that the RTT community excels in is that of _designing_
> realtime systems, with or without the RTT code. And that is more than
> enough to warrant the continuation of this community, even when none of
> the code is kept, since "best practices" are useful everywhere.
>
>> For clarity, we can skip the comments Orocos, since we’ve already
>> established that you think it needs to change.
>
>
> My thoughts go further: there is no need for it anymore.
>
>> Geoff
>
>
> Herman
> --
> Orocos-Dev mailing list
> Orocos-Dev [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>

Updated dataflow semantics for RTT

On Mon, 28 Sep 2015, Sylvain Joyeux wrote:

> Some thoughts on that executive summary (I'm like geoffrey, I had a
> hard time following the first email :P)
>
> One thing I definitely want to warn you about: don't think, ever, that
> code-generating everything is a replacement for a well-designed
> library that solve most of the problems, along with a code generator
> for the one percent.

I am very well aware of this trap, but it cannot be repeated enough.

> With this premise: I actually that your points are not making RTT
> obsolete. It is actually making it even more relevant than ever.
>
> RTT *is* the best starting point towards something where
> process/thread policies is delegated to the tooling (either accessing
> "raw" OS-specific functionality or exposing them under an
> easier-to-use interface). It is also *already* provides an
> encapsulation of computation, which allows you to "inverse" the
> current tendency and actually make your middleware central to your
> communication setup -- instead of hiding it behind a common API.

I agree with your analysis in the context of "the current tendency" to
information hiding behind APIs :-)

But I do not agree with the analysis that RTT is the best thing to hide
computations: I have seen way too many control loops implemented with one
single Orocos component (and activity) per block in a Simulink-like data
flow... Way too many.

> Sylvain

Herman

>
> 2015-09-28 2:28 GMT-03:00 Herman Bruyninckx
> <Herman [dot] Bruyninckx [..] ...>:
>> On Mon, 28 Sep 2015, BiggsGeoffrey wrote:
>>
>>> On 27/9/15, 14:10, "orocos-dev-bounces [..] ... on behalf of
>>> Herman Bruyninckx" <orocos-dev-bounces [..] ... on behalf of
>>> herman [dot] bruyninckx [..] ...> wrote:
>>>>
>>>>
>>>> Thanks for this nice effort. I think it is really time to consider a next
>>>> generation of Orocos RTT...
>>>>
>>>> If you allow me to bring in my two cents... It's possible that you won't
>>>> like the
>>>> message, or find it off-topic, but I think I should share it with you
>>>> anyway...
>>>>
>>>> The best document I have read about communication that connects
>>>> high-performance with flexibility and configurability is this one:
>>>>
>>>>
>>>> <https://engineering.linkedin.com/distributed-systems/log-what-every-softw
>>>> are-engineer-should-know-about-real-time-datas-unifying>
>>>>
>>>> It cleanly separates the communication of "events" from (i) what they
>>>> mean,
>>>> (ii) who is interested in them, and (iii) with what policy they are
>>>> consumed.
>>>>
>>>> In our current research at KU Leuven, this is going to be _the_ basis for
>>>> all interaction between asynchronous activities.
>>>> The other basis will be the separation between (i) functional
>>>> requirements
>>>> of the application, (ii) "roles and responsibilities" of "components",
>>>> (iii) software components to implement both, and (iv) deployment of these
>>>> components on your hardware and OS platforms.
>>>>
>>>> All of the above are not present in Orocos RTT, which I consider design
>>>> flaws.
>>>> So solving these problems is definitely worthwhile. But, and this is a
>>>> big
>>>> but, my view on solving them (which is currently being transformed into
>>>> a first set of models and code and tools) implies that we do not need
>>>> Orocos RTT at all anymore. The reasons why we started RTT more than
>>>> fifteen
>>>> years ago have not disappeared, but have all been solved in more
>>>> "mainstream"
>>>> software projects than Orocos. Fifteen years ago, what we where after was
>>>> a
>>>> "node-port-connector" modelling-and-coding interface, where nodes could
>>>> be
>>>> deployed in realtime activities on an operating system. It was the time
>>>> of
>>>> single core CPUs, with only proprietary solutions to fast networking.
>>>> Those
>>>> days have gone, together with the need to use RTT as a software solution.
>>>>
>>>> More in particular, the "communication" aspect (which is the core of the
>>>> document you distributed) is a lot better understood and supported in
>>>> mainstream than 15 years ago. Taking the model of the event log and the
>>>> corresponding log processing around it means that all the policies that
>>>> your document wants to introduce as "API" methods can (should...) better
>>>> be
>>>> realised by configurations of the event log mechanism in your
>>>> _application
>>>> architecture_; there is no need to provide APIs in "middleware",
>>>> "frameworks" or "libraries".
>>>>
>>>> _The_ core functionalities of RTT that still survive after 15 years are:
>>>> - its introspection capabilities;
>>>> - the lock-free data buffer.
>>>> The latter is just a special case ("configuration") of the event log
>>>> mechanism, that provides all other policies that you mention (and more!)
>>>> just through other configurations.
>>>> The introspection is a form of "model based" implementation, which
>>>> considers models as first-class citizens in the software; this is great,
>>>> and should be extended, starting with modelling all the relevant event
>>>> log
>>>> policies explicity. (KU Leuven is working on that; it takes a lot of
>>>> efforts...) Lock-free data buffers are the core of event log processing,
>>>> and are currently better known under the new buzzword "immutable data".
>>>>
>>>> The other "good-intuition-that-turned-out-to-be-badly-implemented" of
>>>> fifteen years ago is that of "stored procedures" (known under various
>>>> names, such as "microservices", proxy computations, etc.). RTT provides a
>>>> very complex and ambiguous API to let one component execute code for
>>>> another component. Again, that functionality should not be provided by an
>>>> "RPC API", but by models and tools of _deployment_.
>>>>
>>>> The added value of Orocos as a project could shift from providing code to
>>>> providing professional quality documentation and tutorials for "best
>>>> practices" and "patterns", some specific realtime-optimized
>>>> implementations, or patches to the large amount of existing libraries
>>>> that
>>>> support alul of the above.
>>>>
>>>> It is up to the Orocos community to decide whether to go that direction
>>>> or
>>>> not. I would be glad to cooperate in that directoion, because otherwise
>>>> we
>>>> will be forced to start a "competing" project.
>>>>
>>>> Note that this message does not contain threats to "fork" or whatever,
>>>> but
>>>> just my insights after working with Orocos RTT for about a decade, and
>>>> having Orocos RTT work against the architectural requirements in our
>>>> robotics projects for the subsequent five years...
>>>>
>>>> I frankly think it is time to reinvent the project. Big time.
>>>>
>>>> The easiest way is to _deprecate_ things, to make the _core_ things
>>>> simpler
>>>> for users, throwing out more and more code and start using other already
>>>> existing software (ZeroMQ, libevent, the lock-free buffer library,
>>>> RethinkDB,...) instead of trying to squeeze in their functionalities
>>>> within
>>>> NIH APIs and implementations. In the end, RTT will be empty, and the
>>>> development efforts can then finally shift to give Sylvain a much bigger
>>>> hand with ROCK :-)
>>>> (Just make sure to refactor ROCK in order to decouple the _models_ from
>>>> the
>>>> Ruby and RTT _implementations_ :-))
>>>>
>>>> Best regards,
>>>>
>>>> Herman Bruyninckx
>>>
>>>
>>> There are some potentially interesting ideas in there, Herman. But either
>>> your email is uncharacteristically incoherent or I’m really jet lagged.
>>> Could you explain more clearly what you currently think is the ideal?
>>
>>
>> 1. Complete separation of _any_ communication policy from the activity
>> execution policies: there are just way too many use cases that must be
>> supported to bring them all behind one API. (Which is also what ROS2.0
>> tries to do.)
>> 2. "Event processing" is the most flexible _mechanism_ to build other
>> communication policies upon. The most important lesson that _I_
>> learned
>> in this context is that it allows "clients" and "servers" to choose
>> whatever policy they like _locally_, since all data is immutable. (Except
>> for some "long term" garbage collection.)
>> 3. Executing activities in realtime currently does not require a separate
>> library/framework anymore, but is realised by control groups (on Linux),
>> or
>> similar functionalities on other OSs.
>> 4. Similarly, systemd has (or is in the process of) taken over all
>> configuration issues.
>> 5. The expertise that the RTT community excels in is that of _designing_
>> realtime systems, with or without the RTT code. And that is more than
>> enough to warrant the continuation of this community, even when none of
>> the code is kept, since "best practices" are useful everywhere.
>>
>>> For clarity, we can skip the comments Orocos, since we’ve already
>>> established that you think it needs to change.
>>
>>
>> My thoughts go further: there is no need for it anymore.
>>
>>> Geoff
>>
>>
>> Herman
>> --
>> Orocos-Dev mailing list
>> Orocos-Dev [..] ...
>> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>>
>

Updated dataflow semantics for RTT

> But I do not agree with the analysis that RTT is the best thing to hide
> computations: I have seen way too many control loops implemented with one
> single Orocos component (and activity) per block in a Simulink-like data
> flow... Way too many.

Like ... iTasc ?

Unless you make a system too constrained to be actually useful, there
will always be room to misuse it. That does not spell that the system
itself is bad, only that "best practices" suck (or are inexistent).
This is a neverending fight, but IMO the solution is education, not
some horrible model-based environment too complex to every be used.

Sylvain

Updated dataflow semantics for RTT

On Mon, 28 Sep 2015, Sylvain Joyeux wrote:

>> But I do not agree with the analysis that RTT is the best thing to hide
>> computations: I have seen way too many control loops implemented with one
>> single Orocos component (and activity) per block in a Simulink-like data
>> flow... Way too many.
>
> Like ... iTasc ?

For example.

> Unless you make a system too constrained to be actually useful, there
> will always be room to misuse it. That does not spell that the system
> itself is bad, only that "best practices" suck (or are inexistent).

Agreed. Hence my comment that improving this situation is a very useful and
never-ending goal of the Orocos project :-)

> This is a neverending fight, but IMO the solution is education, not
> some horrible model-based environment too complex to every be used.

Which is exactly what RTT has become... I have met no-one the last couple
of years who "got it" when starting to use Orocos, and very few of them
changed their "legacy" to better practice designs...

> Sylvain

Herman

Updated dataflow semantics for RTT

> Which is exactly what RTT has become... I have met no-one the last couple
> of years who "got it" when starting to use Orocos, and very few of them
> changed their "legacy" to better practice designs...

Mmmm ... do I/we(Rock users) count ?

As I said, it is a matter of community, i.e. providing the guidance
and education for people to keep their system "clean". It's time
consuming, but it is part of using any form of complex system.

Sylvain

Updated dataflow semantics for RTT

On Mon, 28 Sep 2015, Sylvain Joyeux wrote:

>> Which is exactly what RTT has become... I have met no-one the last couple
>> of years who "got it" when starting to use Orocos, and very few of them
>> changed their "legacy" to better practice designs...
>
> Mmmm ... do I/we(Rock users) count ?

I used "when _starting_ to use Orocos" and "the last couple of years" in
the same sentence, so you don't count! :-)

> As I said, it is a matter of community, i.e. providing the guidance
> and education for people to keep their system "clean". It's time
> consuming, but it is part of using any form of complex system.

I partially agree: RTT has a too complex design for the complexity it helps
people deal with. And that is _not_ going to change via the currently
suggested incremental changes. So, RTT will _not_ grow its community to the
size it might deserve. All this is speculation, of course.

> Sylvain

Herman

Updated dataflow semantics for RTT

On 28/9/15, 07:28, "Herman Bruyninckx"
<Herman [dot] Bruyninckx [..] ...> wrote:

>On Mon, 28 Sep 2015, BiggsGeoffrey wrote:
>
>> On 27/9/15, 14:10, "orocos-dev-bounces [..] ... on behalf
>>of
>> Herman Bruyninckx" <orocos-dev-bounces [..] ... on behalf
>>of
>> herman [dot] bruyninckx [..] ...> wrote:
>>>
>>> Thanks for this nice effort. I think it is really time to consider a
>>>next
>>> generation of Orocos RTT...
>>>
>>> If you allow me to bring in my two cents... It's possible that you
>>>won't
>>> like the
>>> message, or find it off-topic, but I think I should share it with you
>>> anyway...
>>>
>>> The best document I have read about communication that connects
>>> high-performance with flexibility and configurability is this one:
>>>
>>>
>>><https://engineering.linkedin.com/distributed-systems/log-what-every-sof
>>>tw
>>> are-engineer-should-know-about-real-time-datas-unifying>
>>>
>>> It cleanly separates the communication of "events" from (i) what they
>>> mean,
>>> (ii) who is interested in them, and (iii) with what policy they are
>>> consumed.
>>>
>>> In our current research at KU Leuven, this is going to be _the_ basis
>>>for
>>> all interaction between asynchronous activities.
>>> The other basis will be the separation between (i) functional
>>>requirements
>>> of the application, (ii) "roles and responsibilities" of "components",
>>> (iii) software components to implement both, and (iv) deployment of
>>>these
>>> components on your hardware and OS platforms.
>>>
>>> All of the above are not present in Orocos RTT, which I consider design
>>> flaws.
>>> So solving these problems is definitely worthwhile. But, and this is a
>>>big
>>> but, my view on solving them (which is currently being transformed into
>>> a first set of models and code and tools) implies that we do not need
>>> Orocos RTT at all anymore. The reasons why we started RTT more than
>>> fifteen
>>> years ago have not disappeared, but have all been solved in more
>>> "mainstream"
>>> software projects than Orocos. Fifteen years ago, what we where after
>>>was
>>> a
>>> "node-port-connector" modelling-and-coding interface, where nodes
>>>could be
>>> deployed in realtime activities on an operating system. It was the
>>>time of
>>> single core CPUs, with only proprietary solutions to fast networking.
>>> Those
>>> days have gone, together with the need to use RTT as a software
>>>solution.
>>>
>>> More in particular, the "communication" aspect (which is the core of
>>>the
>>> document you distributed) is a lot better understood and supported in
>>> mainstream than 15 years ago. Taking the model of the event log and the
>>> corresponding log processing around it means that all the policies that
>>> your document wants to introduce as "API" methods can (should...)
>>>better
>>> be
>>> realised by configurations of the event log mechanism in your
>>>_application
>>> architecture_; there is no need to provide APIs in "middleware",
>>> "frameworks" or "libraries".
>>>
>>> _The_ core functionalities of RTT that still survive after 15 years
>>>are:
>>> - its introspection capabilities;
>>> - the lock-free data buffer.
>>> The latter is just a special case ("configuration") of the event log
>>> mechanism, that provides all other policies that you mention (and
>>>more!)
>>> just through other configurations.
>>> The introspection is a form of "model based" implementation, which
>>> considers models as first-class citizens in the software; this is
>>>great,
>>> and should be extended, starting with modelling all the relevant event
>>>log
>>> policies explicity. (KU Leuven is working on that; it takes a lot of
>>> efforts...) Lock-free data buffers are the core of event log
>>>processing,
>>> and are currently better known under the new buzzword "immutable data".
>>>
>>> The other "good-intuition-that-turned-out-to-be-badly-implemented" of
>>> fifteen years ago is that of "stored procedures" (known under various
>>> names, such as "microservices", proxy computations, etc.). RTT
>>>provides a
>>> very complex and ambiguous API to let one component execute code for
>>> another component. Again, that functionality should not be provided by
>>>an
>>> "RPC API", but by models and tools of _deployment_.
>>>
>>> The added value of Orocos as a project could shift from providing code
>>>to
>>> providing professional quality documentation and tutorials for "best
>>> practices" and "patterns", some specific realtime-optimized
>>> implementations, or patches to the large amount of existing libraries
>>>that
>>> support alul of the above.
>>>
>>> It is up to the Orocos community to decide whether to go that
>>>direction or
>>> not. I would be glad to cooperate in that directoion, because
>>>otherwise we
>>> will be forced to start a "competing" project.
>>>
>>> Note that this message does not contain threats to "fork" or whatever,
>>>but
>>> just my insights after working with Orocos RTT for about a decade, and
>>> having Orocos RTT work against the architectural requirements in our
>>> robotics projects for the subsequent five years...
>>>
>>> I frankly think it is time to reinvent the project. Big time.
>>>
>>> The easiest way is to _deprecate_ things, to make the _core_ things
>>> simpler
>>> for users, throwing out more and more code and start using other
>>>already
>>> existing software (ZeroMQ, libevent, the lock-free buffer library,
>>> RethinkDB,...) instead of trying to squeeze in their functionalities
>>> within
>>> NIH APIs and implementations. In the end, RTT will be empty, and the
>>> development efforts can then finally shift to give Sylvain a much
>>>bigger
>>> hand with ROCK :-)
>>> (Just make sure to refactor ROCK in order to decouple the _models_ from
>>> the
>>> Ruby and RTT _implementations_ :-))
>>>
>>> Best regards,
>>>
>>> Herman Bruyninckx
>>
>> There are some potentially interesting ideas in there, Herman. But
>>either
>> your email is uncharacteristically incoherent or I’m really jet lagged.
>> Could you explain more clearly what you currently think is the ideal?
>
>1. Complete separation of _any_ communication policy from the activity
> execution policies: there are just way too many use cases that must be
> supported to bring them all behind one API. (Which is also what ROS2.0
> tries to do.)
>2. "Event processing" is the most flexible _mechanism_ to build other
> communication policies upon. The most important lesson that _I_ learned
> in this context is that it allows "clients" and "servers" to choose
> whatever policy they like _locally_, since all data is immutable.
>(Except
> for some "long term" garbage collection.)

Does this imply distributing all provided data that a component asks for
to that component and having it all buffered to be used according to the
local policies?

>3. Executing activities in realtime currently does not require a separate
> library/framework anymore, but is realised by control groups (on
>Linux), or
> similar functionalities on other OSs.
>4. Similarly, systemd has (or is in the process of) taken over all
> configuration issues.

And caused many a flame war in the process. :)

>5. The expertise that the RTT community excels in is that of _designing_
> realtime systems, with or without the RTT code. And that is more than
> enough to warrant the continuation of this community, even when none
>of
> the code is kept, since "best practices" are useful everywhere.

So reading between the lines, I think that what you’re saying is that we
should use a component-based approach at the model level, and the code
generated from that doesn’t need to have a component-based framework, it
can just use all these existing technologies to provide the functionality.
Is this correct?

Geoff

>
>> For clarity, we can skip the comments Orocos, since we’ve already
>> established that you think it needs to change.
>
>My thoughts go further: there is no need for it anymore.
>
>> Geoff
>
>Herman

Updated dataflow semantics for RTT

On Mon, 28 Sep 2015, BiggsGeoffrey wrote:

>
>
> On 28/9/15, 07:28, "Herman Bruyninckx"
> <Herman [dot] Bruyninckx [..] ...> wrote:
>
>> On Mon, 28 Sep 2015, BiggsGeoffrey wrote:
>>
>>> On 27/9/15, 14:10, "orocos-dev-bounces [..] ... on behalf
>>> of
>>> Herman Bruyninckx" <orocos-dev-bounces [..] ... on behalf
>>> of
>>> herman [dot] bruyninckx [..] ...> wrote:
>>>>
>>>> Thanks for this nice effort. I think it is really time to consider a
>>>> next
>>>> generation of Orocos RTT...
>>>>
>>>> If you allow me to bring in my two cents... It's possible that you
>>>> won't
>>>> like the
>>>> message, or find it off-topic, but I think I should share it with you
>>>> anyway...
>>>>
>>>> The best document I have read about communication that connects
>>>> high-performance with flexibility and configurability is this one:
>>>>
>>>>
>>>> <https://engineering.linkedin.com/distributed-systems/log-what-every-sof
>>>> tw
>>>> are-engineer-should-know-about-real-time-datas-unifying>
>>>>
>>>> It cleanly separates the communication of "events" from (i) what they
>>>> mean,
>>>> (ii) who is interested in them, and (iii) with what policy they are
>>>> consumed.
>>>>
>>>> In our current research at KU Leuven, this is going to be _the_ basis
>>>> for
>>>> all interaction between asynchronous activities.
>>>> The other basis will be the separation between (i) functional
>>>> requirements
>>>> of the application, (ii) "roles and responsibilities" of "components",
>>>> (iii) software components to implement both, and (iv) deployment of
>>>> these
>>>> components on your hardware and OS platforms.
>>>>
>>>> All of the above are not present in Orocos RTT, which I consider design
>>>> flaws.
>>>> So solving these problems is definitely worthwhile. But, and this is a
>>>> big
>>>> but, my view on solving them (which is currently being transformed into
>>>> a first set of models and code and tools) implies that we do not need
>>>> Orocos RTT at all anymore. The reasons why we started RTT more than
>>>> fifteen
>>>> years ago have not disappeared, but have all been solved in more
>>>> "mainstream"
>>>> software projects than Orocos. Fifteen years ago, what we where after
>>>> was
>>>> a
>>>> "node-port-connector" modelling-and-coding interface, where nodes
>>>> could be
>>>> deployed in realtime activities on an operating system. It was the
>>>> time of
>>>> single core CPUs, with only proprietary solutions to fast networking.
>>>> Those
>>>> days have gone, together with the need to use RTT as a software
>>>> solution.
>>>>
>>>> More in particular, the "communication" aspect (which is the core of
>>>> the
>>>> document you distributed) is a lot better understood and supported in
>>>> mainstream than 15 years ago. Taking the model of the event log and the
>>>> corresponding log processing around it means that all the policies that
>>>> your document wants to introduce as "API" methods can (should...)
>>>> better
>>>> be
>>>> realised by configurations of the event log mechanism in your
>>>> _application
>>>> architecture_; there is no need to provide APIs in "middleware",
>>>> "frameworks" or "libraries".
>>>>
>>>> _The_ core functionalities of RTT that still survive after 15 years
>>>> are:
>>>> - its introspection capabilities;
>>>> - the lock-free data buffer.
>>>> The latter is just a special case ("configuration") of the event log
>>>> mechanism, that provides all other policies that you mention (and
>>>> more!)
>>>> just through other configurations.
>>>> The introspection is a form of "model based" implementation, which
>>>> considers models as first-class citizens in the software; this is
>>>> great,
>>>> and should be extended, starting with modelling all the relevant event
>>>> log
>>>> policies explicity. (KU Leuven is working on that; it takes a lot of
>>>> efforts...) Lock-free data buffers are the core of event log
>>>> processing,
>>>> and are currently better known under the new buzzword "immutable data".
>>>>
>>>> The other "good-intuition-that-turned-out-to-be-badly-implemented" of
>>>> fifteen years ago is that of "stored procedures" (known under various
>>>> names, such as "microservices", proxy computations, etc.). RTT
>>>> provides a
>>>> very complex and ambiguous API to let one component execute code for
>>>> another component. Again, that functionality should not be provided by
>>>> an
>>>> "RPC API", but by models and tools of _deployment_.
>>>>
>>>> The added value of Orocos as a project could shift from providing code
>>>> to
>>>> providing professional quality documentation and tutorials for "best
>>>> practices" and "patterns", some specific realtime-optimized
>>>> implementations, or patches to the large amount of existing libraries
>>>> that
>>>> support alul of the above.
>>>>
>>>> It is up to the Orocos community to decide whether to go that
>>>> direction or
>>>> not. I would be glad to cooperate in that directoion, because
>>>> otherwise we
>>>> will be forced to start a "competing" project.
>>>>
>>>> Note that this message does not contain threats to "fork" or whatever,
>>>> but
>>>> just my insights after working with Orocos RTT for about a decade, and
>>>> having Orocos RTT work against the architectural requirements in our
>>>> robotics projects for the subsequent five years...
>>>>
>>>> I frankly think it is time to reinvent the project. Big time.
>>>>
>>>> The easiest way is to _deprecate_ things, to make the _core_ things
>>>> simpler
>>>> for users, throwing out more and more code and start using other
>>>> already
>>>> existing software (ZeroMQ, libevent, the lock-free buffer library,
>>>> RethinkDB,...) instead of trying to squeeze in their functionalities
>>>> within
>>>> NIH APIs and implementations. In the end, RTT will be empty, and the
>>>> development efforts can then finally shift to give Sylvain a much
>>>> bigger
>>>> hand with ROCK :-)
>>>> (Just make sure to refactor ROCK in order to decouple the _models_ from
>>>> the
>>>> Ruby and RTT _implementations_ :-))
>>>>
>>>> Best regards,
>>>>
>>>> Herman Bruyninckx
>>>
>>> There are some potentially interesting ideas in there, Herman. But
>>> either
>>> your email is uncharacteristically incoherent or I’m really jet lagged.
>>> Could you explain more clearly what you currently think is the ideal?
>>
>> 1. Complete separation of _any_ communication policy from the activity
>> execution policies: there are just way too many use cases that must be
>> supported to bring them all behind one API. (Which is also what ROS2.0
>> tries to do.)
>> 2. "Event processing" is the most flexible _mechanism_ to build other
>> communication policies upon. The most important lesson that _I_ learned
>> in this context is that it allows "clients" and "servers" to choose
>> whatever policy they like _locally_, since all data is immutable.
>> (Except
>> for some "long term" garbage collection.)
>
> Does this imply distributing all provided data that a component asks for
> to that component and having it all buffered to be used according to the
> local policies?

Indeed. The "buffering" however becomes extremely simple and efficient,
once one accepts the "immutable data" paradigm. The main difference with
the implementation of the "lockfree buffer" in RTT is that one need not do
much effort not to waste RAM, since (or rather, under the condition that!)
such RAM is available in abundance.

The buffering is then yet another policy (or rather, a combo of policies)
to be made only(!) when _deploying_ a system, which is the phase where
components are actually put into processes and processes are connected
through several types of IPC mechanisms:

- which parts of the 'event stream' need to be 'filtered' by/for which 'process';
(note: there is only need for one such "mediator" in each process, to
distribute data between processes)

- how do 'threads' inside a process read for the filtered stream, and what
output stream do they write to, inside the process; we think of using the
"proxy" pattern to realise this part, because the filtering often depends
not just on the local functionality but also on that of a "peer" that
resides in another component somewhere in the system;

- how does each process combine the various output streams and forwards
them to the rest of the system (again via a communication "mediator").

Note that I suggest to postpone a lot of the "read/write and buffering
policies" to _deployment_ time, because they are not needed when
implementing components and ports. Note also that a "port" is reduced to
basically a "topic model", and all of its "functionalities" are taken over
by the "event stream" infrastructure, which is a deployment-level design
issue.

In yet other words, what I suggest is to revive the "blackboard
architecture" of some decades ago, by adding the best practices that the
community got from about a decade of (i) "immutable data" experiences, and
(ii) replication of data over peers. "Send and forget" is definitely the
best option _at runtime_; this should be complemented with the right set of
_policies_ at _design time_, _reconfigurable_ during runtime.

>> 3. Executing activities in realtime currently does not require a separate
>> library/framework anymore, but is realised by control groups (on
>> Linux), or
>> similar functionalities on other OSs.
>> 4. Similarly, systemd has (or is in the process of) taken over all
>> configuration issues.
>
> And caused many a flame war in the process. :)

Sure, and for a reason: good ideas cause a lot of friction, because they
force people to get their inertially moving ships steered in other
directions :-)

For me, systemd is the transition from "procedural programming" (in old
init _scripts_) towards "declarative specification" (in systemd _models_).
That is a "Good Thing", but the transition is not yet finalised. But for
our RTT context, it is,

>> 5. The expertise that the RTT community excels in is that of _designing_
>> realtime systems, with or without the RTT code. And that is more than
>> enough to warrant the continuation of this community, even when none
>> of
>> the code is kept, since "best practices" are useful everywhere.
>
> So reading between the lines, I think that what you’re saying is that we
> should use a component-based approach at the model level, and the code
> generated from that doesn’t need to have a component-based framework, it
> can just use all these existing technologies to provide the functionality.
> Is this correct?

That is correct!

A mild version of the "reading between the lines" is that the "component
framework" need not be the same at the three "levels of abstraction" that we
all use all the time (implicitly, most often....):
- models;
- implementations;
- deployments.

A stronger version is that, indeed, we are now, finally, reaching a state
in which "compiling models" is becoming possible; while robotics
developments are still about "compiling code". The code generated
("composed", rather) from the model-to-text transformation need not have
the same "level of abstraction" and "separation of concerns" as what we
would require for humans writing the same functional code...

> Geoff

Herman

>>> For clarity, we can skip the comments Orocos, since we’ve already
>>> established that you think it needs to change.
>>
>> My thoughts go further: there is no need for it anymore.
>>
>>> Geoff
>>
>> Herman
>
>

Updated dataflow semantics for RTT

2015-09-27 14:10 GMT+02:00 Herman Bruyninckx <
Herman [dot] Bruyninckx [..] ...>:

> On Mon, 21 Sep 2015, Johannes Meyer wrote:
>
> Dear Orocos community,
>> during the last few weeks we have been working on the concept and
>> implementation of a major redesign of the Orocos
>> RTT dataflow, that we would like to share and ask for your valuable
>> feedback. There was already a previous
>> discussion on the mailing list about the topic earlier this year [1], and
>> the following proposal tries to address
>> all the issues mentioned there.
>>
>> With the version 2 of the Orocos toolchain some years ago, the previous
>> data flow architecture, which was based on a
>> single data object and buffer shared between all ports participating at
>> the same connection, was replaced by a
>> connection architecture based on data flow, where each pair of output and
>> input ports (or streams) is connected
>> through an individual channel, each having its own data object or buffer.
>> While there was a well substantiated
>> motivation for this update and it solved some of the shortcomings in RTT
>> version 1 [2], it also introduced some new,
>> sometimes hidden, problems and broke other use cases. One example is an
>> input port with multiple connections, where
>> due to the ambiguity in which channel to read from the result is
>> sometimes unexpected to the user, who expects that
>> each write overwrites previous samples in case of simple data connections.
>>
>> We assembled a document (see attachment, or [3]) which introduces the
>> version 1 and version 2 data flow semantics
>> for those who are not aware of the details, three example use cases and
>> presents the new dataflow architecture we
>> have in mind, which ideally works for a superset of use cases that are
>> supported by RTT version 1 and 2.
>>
>> We do not want to go into details here, so only a short summary for those
>> who do not want to read the full document:
>> We propose to add two more boolean flags to the ConnPolicy struct that
>> describes the type of connection between two
>> ports: "shared" and "mandatory". Furthermore the existing "pull" flag,
>> which was only meaningful for remote
>> connections, now has a big influence for local connections.
>>
>> "push" vs. "pull"
>>
>>
> Thanks for this nice effort. I think it is really time to consider a next
> generation of Orocos RTT...
>
> If you allow me to bring in my two cents... It's possible that you won't
> like the
> message, or find it off-topic, but I think I should share it with you
> anyway...
>
> The best document I have read about communication that connects
> high-performance with flexibility and configurability is this one:
> <
> https://engineering.linkedin.com/distributed-systems/log-what-every-soft...
> >
>
> It cleanly separates the communication of "events" from (i) what they mean,
> (ii) who is interested in them, and (iii) with what policy they are
> consumed.
>
> In our current research at KU Leuven, this is going to be _the_ basis for
> all interaction between asynchronous activities.
> The other basis will be the separation between (i) functional requirements
> of the application, (ii) "roles and responsibilities" of "components",
> (iii) software components to implement both, and (iv) deployment of these
> components on your hardware and OS platforms.
>
> All of the above are not present in Orocos RTT, which I consider design
> flaws.
> So solving these problems is definitely worthwhile. But, and this is a big
> but, my view on solving them (which is currently being transformed into
> a first set of models and code and tools) implies that we do not need
> Orocos RTT at all anymore. The reasons why we started RTT more than fifteen
> years ago have not disappeared, but have all been solved in more
> "mainstream"
> software projects than Orocos. Fifteen years ago, what we where after was a
> "node-port-connector" modelling-and-coding interface, where nodes could be
> deployed in realtime activities on an operating system. It was the time of
> single core CPUs, with only proprietary solutions to fast networking. Those
> days have gone, together with the need to use RTT as a software solution.
>
> More in particular, the "communication" aspect (which is the core of the
> document you distributed) is a lot better understood and supported in
> mainstream than 15 years ago. Taking the model of the event log and the
> corresponding log processing around it means that all the policies that
> your document wants to introduce as "API" methods can (should...) better be
> realised by configurations of the event log mechanism in your _application
> architecture_; there is no need to provide APIs in "middleware",
> "frameworks" or "libraries".
>
> _The_ core functionalities of RTT that still survive after 15 years are:
> - its introspection capabilities;
> - the lock-free data buffer.
>

+1, this is my case.

> The latter is just a special case ("configuration") of the event log
> mechanism, that provides all other policies that you mention (and more!)
> just through other configurations. The introspection is a form of "model
> based" implementation, which
> considers models as first-class citizens in the software; this is great,
> and should be extended, starting with modelling all the relevant event log
> policies explicity. (KU Leuven is working on that; it takes a lot of
> efforts...) Lock-free data buffers are the core of event log processing,
> and are currently better known under the new buzzword "immutable data".
>
> The other "good-intuition-that-turned-out-to-be-badly-implemented" of
> fifteen years ago is that of "stored procedures" (known under various
> names, such as "microservices", proxy computations, etc.). RTT provides a
> very complex and ambiguous API to let one component execute code for
> another component. Again, that functionality should not be provided by an
> "RPC API", but by models and tools of _deployment_.
>
> The added value of Orocos as a project could shift from providing code to
> providing professional quality documentation and tutorials for "best
> practices" and "patterns", some specific realtime-optimized
> implementations, or patches to the large amount of existing libraries that
> support alul of the above.
>
> It is up to the Orocos community to decide whether to go that direction or
> not. I would be glad to cooperate in that directoion, because otherwise we
> will be forced to start a "competing" project.
>
> Note that this message does not contain threats to "fork" or whatever, but
> just my insights after working with Orocos RTT for about a decade, and
> having Orocos RTT work against the architectural requirements in our
> robotics projects for the subsequent five years...
>
> I frankly think it is time to reinvent the project. Big time.
>
> The easiest way is to _deprecate_ things, to make the _core_ things simpler
> for users, throwing out more and more code and start using other already
> existing software (ZeroMQ, libevent, the lock-free buffer library,
> RethinkDB,...) instead of trying to squeeze in their functionalities within
> NIH APIs and implementations. In the end, RTT will be empty, and the
> development efforts can then finally shift to give Sylvain a much bigger
> hand with ROCK :-)
> (Just make sure to refactor ROCK in order to decouple the _models_ from the
> Ruby and RTT _implementations_ :-))
>
> Best regards,
>
> Herman Bruyninckx

I have never been an active part of the project (testing and using apart),
so I can not comment all this, but :
_ thanks Hermann for sharing
_ today, if I had to choose a solution for my professionnal project, I
would not choose Orocos(Herman has well explained why) and the miss would
be the lock free communication and introspection tools.
_ I would also migrate from pure Orocos to Rock if my application wasn't
considered as "frozen".

I'm not sure it's the good post to chat on that.

>
>
> Like it is already the case for remote connections via CORBA, the "pull"
>> flag will decide whether the connection's
>> data object or buffer is installed at the writer's side (pull) or the
>> reader's side (push) of the connection. The
>> first case is equivalent to the current version 2 connection model and
>> readers have to select the connection/output
>> port to read from, while for push connections there will be only one
>> single buffer per input port. We think that the
>> push connection should be the default connection type for local ports,
>> but this might break existing applications
>> which rely on having one buffer per connection now.
>>
>> "private" vs. "shared"
>>
>> The shared flag reintroduces the shared data connection model that was
>> standard in RTT version 1. For shared
>> connections multiple input ports read from the same data object or
>> consume samples from the same buffer instance, so
>> their read operations can influence each other. Shared connections are
>> especially useful for producer/consumer
>> applications with multiple consumers.
>>
>> "mandatory"
>>
>> The introduction of the "mandatory" flag was necessary to decide on the
>> return value of the write() operation on an
>> output port. In RTT version 2 it was not possible anymore to get a
>> feedback of whether a write operation was
>> successful or failed, e.g. because of a full buffer or broken remote
>> connection. Only connections with the mandatory
>> flag set would be considered for the overall result of write()
>> operations, while failures of non-mandatory
>> connections will be ignored.
>>
>>
>> For those who want to take a look at the current state of the
>> implementation, this is the link to the working branch
>> on GitHub:
>> Tree:
>> https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics
>> Diff to master:
>> https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflo...
>>
>> The implementation is not fully working yet for non-standard transports
>> (like Corba and mqueue), but especially the
>> updated test ports_test.cpp already shows what would change from a user
>> perspective.
>>
>> Obviously, such a major refactoring cannot be done without changes in the
>> public API. However, we tried to keep the
>> required changes in user code as minimal as possible.
>> The new dataflow semantics are targeted for an upcoming version 2.9
>> release (or even name it 3.0 to communicate this
>> potentially breaking change?).
>>
>> We are planning to also come up with some comparative performance
>> measurements during this week, as people have
>> asked for that before [1].
>>
>>
>> Best regards,
>> Johannes and Peter
>>
>> [1]
>> http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation
>> [2] http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow
>> [3]
>> https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4Kaotp...
>>
>>
>>
> --
> Orocos-Dev mailing list
> Orocos-Dev [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>
>

Updated dataflow semantics for RTT

Hi,
here are my remarks:
First of all, I like the idea of the input buffer (push case). But I
also see some
problems with it. The restriction on the same buffer size and type in the
connection policy will be annoying (In my mind I can see the runtime
errors pop up).
It will be hard to track this in bigger systems. Do I get it correctly,
that you only
want to preserve arrival order and runtime here ?

If you are already changing the Connection implementation, I would
recommend, to
put a new thread in every connection in remote case. We experienced the
issue, that
bad Wifi connections, slowed our systems down, as it would hang on the
write call.

For the Feedback on the write I would recommend to create a second enum,
and not
to use FlowStatus. Sometimes I use switch/case to react to the read
FlowStatus. If you
Introduce new members, this will result in compile warnings, because of
unhandled
entrie.

For the creation of shared connections, I would recommend an new API.
The create call
should contain all involved members. This spares us the headache of
tracking buffer sizes
and types around in the code. Also this will simplify the code logic, as
we don't need the lookup.
e.g.
out.connectShared(b.in, c.in, d.in, policy) ;
Greetings
Janosch

Am 21.09.2015 um 15:15 schrieb Johannes Meyer:
> Dear Orocos community,
>
> during the last few weeks we have been working on the concept and
> implementation of a major redesign of the Orocos RTT dataflow, that we
> would like to share and ask for your valuable feedback. There was
> already a previous discussion on the mailing list about the topic
> earlier this year [1], and the following proposal tries to address all
> the issues mentioned there.
>
> With the version 2 of the Orocos toolchain some years ago, the
> previous data flow architecture, which was based on a single data
> object and buffer shared between all ports participating at the same
> connection, was replaced by a connection architecture based on data
> flow, where each pair of output and input ports (or streams) is
> connected through an individual channel, each having its own data
> object or buffer. While there was a well substantiated motivation for
> this update and it solved some of the shortcomings in RTT version 1
> [2], it also introduced some new, sometimes hidden, problems and broke
> other use cases. One example is an input port with multiple
> connections, where due to the ambiguity in which channel to read from
> the result is sometimes unexpected to the user, who expects that each
> write overwrites previous samples in case of simple data connections.
>
> We assembled a document (see attachment, or [3]) which introduces the
> version 1 and version 2 data flow semantics for those who are not
> aware of the details, three example use cases and presents the new
> dataflow architecture we have in mind, which ideally works for a
> superset of use cases that are supported by RTT version 1 and 2.
>
> We do not want to go into details here, so only a short summary for
> those who do not want to read the full document:
> We propose to add two more boolean flags to the ConnPolicy struct that
> describes the type of connection between two ports: "shared" and
> "mandatory". Furthermore the existing "pull" flag, which was only
> meaningful for remote connections, now has a big influence for local
> connections.
>
> *"push" vs. "pull"*
> *
> *
> Like it is already the case for remote connections via CORBA, the
> "pull" flag will decide whether the connection's data object or buffer
> is installed at the writer's side (pull) or the reader's side (push)
> of the connection. The first case is equivalent to the current version
> 2 connection model and readers have to select the connection/output
> port to read from, while for push connections there will be only one
> single buffer per input port. We think that the push connection should
> be the default connection type for local ports, but this might break
> existing applications which rely on having one buffer per connection now.
>
> *"private" vs. "shared"*
>
> The shared flag reintroduces the shared data connection model that was
> standard in RTT version 1. For shared connections multiple input ports
> read from the same data object or consume samples from the same buffer
> instance, so their read operations can influence each other. Shared
> connections are especially useful for producer/consumer applications
> with multiple consumers.
>
> *"mandatory"*
>
> The introduction of the "mandatory" flag was necessary to decide on
> the return value of the write() operation on an output port. In RTT
> version 2 it was not possible anymore to get a feedback of whether a
> write operation was successful or failed, e.g. because of a full
> buffer or broken remote connection. Only connections with the
> mandatory flag set would be considered for the overall result of
> write() operations, while failures of non-mandatory connections will
> be ignored.
>
>
> For those who want to take a look at the current state of the
> implementation, this is the link to the working branch on GitHub:
> Tree:
> https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics
> Diff to master:
> https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflo...
>
> The implementation is not fully working yet for non-standard
> transports (like Corba and mqueue), but especially the updated test
> ports_test.cpp already shows what would change from a user perspective.
>
> Obviously, such a major refactoring cannot be done without changes in
> the public API. However, we tried to keep the required changes in user
> code as minimal as possible.
> The new dataflow semantics are targeted for an upcoming version 2.9
> release (or even name it 3.0 to communicate this potentially breaking
> change?).
>
> We are planning to also come up with some comparative performance
> measurements during this week, as people have asked for that before [1].
>
>
> Best regards,
> Johannes and Peter
>
>
>
> [1]
> http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation
>
> [2] http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow
> [3]
> https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4Kaotp...
>
>
>

Updated dataflow semantics for RTT

> On Sep 22, 2015, at 09:32, Janosch Machowinski <Janosch [dot] Machowinski [..] ...> wrote:
>
> Hi,
> here are my remarks:
> First of all, I like the idea of the input buffer (push case). But I also see some
> problems with it. The restriction on the same buffer size and type in the
> connection policy will be annoying (In my mind I can see the runtime errors pop up).
> It will be hard to track this in bigger systems. Do I get it correctly, that you only
> want to preserve arrival order and runtime here ?

I’ll have to disagree here. We have several hundred connections in our systems, and we track the policy type amongst all.

> If you are already changing the Connection implementation, I would recommend, to
> put a new thread in every connection in remote case. We experienced the issue, that
> bad Wifi connections, slowed our systems down, as it would hang on the write call.

Not my first choice. I agree with Sylvain - this should be part of the transport layer.

> For the Feedback on the write I would recommend to create a second enum, and not
> to use FlowStatus. Sometimes I use switch/case to react to the read FlowStatus. If you
> Introduce new members, this will result in compile warnings, because of unhandled
> entrie.

Yep, backwards compatibility is a concern. :-(

> For the creation of shared connections, I would recommend an new API. The create call
> should contain all involved members. This spares us the headache of tracking buffer sizes
> and types around in the code. Also this will simplify the code logic, as we don't need the lookup.
> e.g.
> out.connectShared(b.in, c.in, d.in, policy) ;

In the above are “b”, “c”, and “d”, all ports? If so, what about cases were not all port connections are made at the same time (e.g. XML deployments)?

Cheers
S

> Greetings
> Janosch
>
>
>
> Am 21.09.2015 um 15:15 schrieb Johannes Meyer:
>> Dear Orocos community,
>>
>> during the last few weeks we have been working on the concept and implementation of a major redesign of the Orocos RTT dataflow, that we would like to share and ask for your valuable feedback. There was already a previous discussion on the mailing list about the topic earlier this year [1], and the following proposal tries to address all the issues mentioned there.
>>
>> With the version 2 of the Orocos toolchain some years ago, the previous data flow architecture, which was based on a single data object and buffer shared between all ports participating at the same connection, was replaced by a connection architecture based on data flow, where each pair of output and input ports (or streams) is connected through an individual channel, each having its own data object or buffer. While there was a well substantiated motivation for this update and it solved some of the shortcomings in RTT version 1 [2], it also introduced some new, sometimes hidden, problems and broke other use cases. One example is an input port with multiple connections, where due to the ambiguity in which channel to read from the result is sometimes unexpected to the user, who expects that each write overwrites previous samples in case of simple data connections.
>>
>> We assembled a document (see attachment, or [3]) which introduces the version 1 and version 2 data flow semantics for those who are not aware of the details, three example use cases and presents the new dataflow architecture we have in mind, which ideally works for a superset of use cases that are supported by RTT version 1 and 2.
>>
>> We do not want to go into details here, so only a short summary for those who do not want to read the full document:
>> We propose to add two more boolean flags to the ConnPolicy struct that describes the type of connection between two ports: "shared" and "mandatory". Furthermore the existing "pull" flag, which was only meaningful for remote connections, now has a big influence for local connections.
>>
>> "push" vs. "pull"
>>
>> Like it is already the case for remote connections via CORBA, the "pull" flag will decide whether the connection's data object or buffer is installed at the writer's side (pull) or the reader's side (push) of the connection. The first case is equivalent to the current version 2 connection model and readers have to select the connection/output port to read from, while for push connections there will be only one single buffer per input port. We think that the push connection should be the default connection type for local ports, but this might break existing applications which rely on having one buffer per connection now.
>>
>> "private" vs. "shared"
>>
>> The shared flag reintroduces the shared data connection model that was standard in RTT version 1. For shared connections multiple input ports read from the same data object or consume samples from the same buffer instance, so their read operations can influence each other. Shared connections are especially useful for producer/consumer applications with multiple consumers.
>>
>> "mandatory"
>>
>> The introduction of the "mandatory" flag was necessary to decide on the return value of the write() operation on an output port. In RTT version 2 it was not possible anymore to get a feedback of whether a write operation was successful or failed, e.g. because of a full buffer or broken remote connection. Only connections with the mandatory flag set would be considered for the overall result of write() operations, while failures of non-mandatory connections will be ignored.
>>
>>
>> For those who want to take a look at the current state of the implementation, this is the link to the working branch on GitHub:
>> Tree: https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics <https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics>
>> Diff to master: https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflo... <https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflow-semantics>
>>
>> The implementation is not fully working yet for non-standard transports (like Corba and mqueue), but especially the updated test ports_test.cpp already shows what would change from a user perspective.
>>
>> Obviously, such a major refactoring cannot be done without changes in the public API. However, we tried to keep the required changes in user code as minimal as possible.
>> The new dataflow semantics are targeted for an upcoming version 2.9 release (or even name it 3.0 to communicate this potentially breaking change?).
>>
>> We are planning to also come up with some comparative performance measurements during this week, as people have asked for that before [1].
>>
>>
>> Best regards,
>> Johannes and Peter
>>
>>
>>
>> [1] http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation <http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation>
>> [2] http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow <http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow>
>> [3] https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4Kaotp... <https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4KaotpCtXM4E0/pub>
>>
>>
>>
>
>

Updated dataflow semantics for RTT

Hi,

On Tue, Sep 22, 2015 at 3:32 PM, Janosch Machowinski <
Janosch [dot] Machowinski [..] ...> wrote:

> Hi,
> here are my remarks:
> First of all, I like the idea of the input buffer (push case). But I also
> see some
> problems with it. The restriction on the same buffer size and type in the
> connection policy will be annoying (In my mind I can see the runtime
> errors pop up).
> It will be hard to track this in bigger systems. Do I get it correctly,
> that you only
> want to preserve arrival order and runtime here ?
>

No, we think the current implementation with one buffer per connection
could really be considered as a bug, at least for certain use cases.
Consider two output ports W1 and W2 which are both connected to the same
input port R with a data connection. What I would expect is:

W1.write("foo")
W2.write("bar")
assert(R.read(x) == NewData && x == "bar")
assert(R.read(x) == OldData && x == "bar")

so each writer overwrites the data sample from previous writes (think of a
robot command). But what happens in RTT v2 is:

W1.write("foo")
W2.write("bar")
assert(R.read(x) == NewData && x == "foo")
assert(R.read(x) == NewData && x == "bar")

and I cannot even be sure if I first read "foo" or "bar" because it depends
on what connection was read last. The first result would be the new default
case (push), while the second can still be achieved by setting pull to
true, which had no effect until now for local connections.

Certainly there is a significant chance that the new restrictions break
existing deployments and that's why we would only release it with a new
minor, if not even major version number. But to fix this serious drawback
of the current dataflow implementation it is certainly worth the effort.

On Tue, Sep 22, 2015 at 4:17 PM, Janosch Machowinski <
Janosch [dot] Machowinski [..] ...> wrote:

> Am 22.09.2015 um 16:09 schrieb Sylvain Joyeux:
> >> If you are already changing the Connection implementation, I would
> >> recommend, to put a new thread in every connection in remote case. We
> experienced the
> >> issue, that bad Wifi connections, slowed our systems down, as it would
> hang on the write
> >> call.
> > This should IMO be done by the middleware ... in the CORBA case we
> > should have a way to spawn separate corba dispatchers (e.g. based on
> > the policie's name field) to isolate the domains. Moreover, there's
> > already been work done to use CORBA's "oneway" call to avoid this
> > problem completely (but I don't know what's the status of that)
> >
> > Sylvain
> This is basically a design decision, thread all remote connection, to be
> sure
> that the do not block you task, or rely on the connection to be properly
> implemented.

I agree with Sylvain here: It is the transport's responsibility to not
block on write or read operations and if the underlying implementation
cannot guarantee that, to use single worker thread, thread pool or one
thread per connection. If this would be enforced by the RTT connection
factory already, it would be impossible to use real-time capable transports
like Xenomai message queues.

We indeed also have an updated CORBA transport ready which introduces
oneway calls wherever possible and might solve some of the problems you
observed. It could be considered as stable and I could prepare another pull
request, but of course there would be plenty of conflicts with the
updated-dataflow-semantics branch, so I would prefer to postpone this until
this one is merged.

>
> For the Feedback on the write I would recommend to create a second enum,
> and not
> to use FlowStatus. Sometimes I use switch/case to react to the read
> FlowStatus. If you
> Introduce new members, this will result in compile warnings, because of
> unhandled
> entrie.
>

You are right and I first added a new enum-like class WriteStatus, but it
would also require a separate TypeInfo, to be added to the default typekit,
a corresponding struct in the CORBA IDL and so on, so at the end it seemed
easier to just add some constants to FlowStatus. The reason to have a class
instead of a simple enum is that the implicit conversion to bool could be
implemented more meaningful, more specifically that WriteSuccess evaluates
to true and all other constants to false.

>
> For the creation of shared connections, I would recommend an new API. The
> create call
> should contain all involved members. This spares us the headache of
> tracking buffer sizes
> and types around in the code. Also this will simplify the code logic, as
> we don't need the lookup.
> e.g.
> out.connectShared(b.in, c.in, d.in, policy) ;
>

A new API instead of adding the shared flag to the ConnPolicy? Or only as a
thin wrapper which implies that the shared flag is set?
It should be noted that it is not the new concept of shared connections
that adds constraints, but especially the per-input port buffers implied by
the default pull = false setting, even for private connections. What you
probably mean is to introduce a separate API for all new dataflow models
and make sure that the existing API calls behave the same as before.

A separate API would at least require additional updates in the deployer in
order to make it accessible from deployment scripts or XML configurations.

If people really feel a need for that, a possible intermediate solution
would be a preprocessor macro and/or cmake variable that enables a
compatibility mode which restores the current v2.x connection model and
basically enforces the pull connection type for local connections and
installs a per-connection buffer at the reader's side for remote push
connections. In this mode, RTT could at least print some deprecation
warnings to the Orocos log if a connection attempt would have failed
without the macro to be defined. Then at least you can postpone necessary
changes in your application until there is more time to do it and still use
other new features of an upcoming RTT release.

> Greetings
> Janosch
>

Best regards
Johannes

Updated dataflow semantics for RTT

> W1.write("foo")
> W2.write("bar")
> assert(R.read(x) == NewData && x == "bar")
> assert(R.read(x) == OldData && x == "bar")
>
> so each writer overwrites the data sample from previous writes (think of a
> robot command).

Now we're getting somewhere. This looks like a bug to me as well.
However, you proposal really breaks buffers / mixed policy ports.

To me, this calls for having different behaviours for data object and
buffer objects: one data object per input port and one buffer per
channel. In all cases. It would actually both fix the bug you describe
above *and* keep mixed-policy behaviour as well as keep buffer setup
feasible (since sizing buffers is meaningful only on a per-connection
basis).

Sylvain

Updated dataflow semantics for RTT

Hi Johannes

> To me, this calls for having different behaviours for data object and
> buffer objects: one data object per input port and one buffer per
> channel. In all cases. It would actually both fix the bug you describe
> above *and* keep mixed-policy behaviour as well as keep buffer setup
> feasible (since sizing buffers is meaningful only on a per-connection
> basis).

I did not get any comment from you about this ... It concerns me as my
current understanding is that by fixing RTT for your use case, you
would basically fundamentally break it for mine ...

Sylvain

Updated dataflow semantics for RTT

On Fri, Sep 25, 2015 at 6:24 PM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:
>
> Hi Johannes
>
> > To me, this calls for having different behaviours for data object and
> > buffer objects: one data object per input port and one buffer per
> > channel. In all cases. It would actually both fix the bug you describe
> > above *and* keep mixed-policy behaviour as well as keep buffer setup
> > feasible (since sizing buffers is meaningful only on a per-connection
> > basis).
>
> I did not get any comment from you about this ... It concerns me as my
> current understanding is that by fixing RTT for your use case, you
> would basically fundamentally break it for mine ...

It doesn't fix the out-of-order bug for the buffer-per-channel case.
You can still starve
a certain channel and get old data much later. That's why we propose
to change the
default. This starving already happens in very simple/clean setups,
unless you start
to round-robin all the channels. But again, this does not prevent
out-of-order arrivals.

I prefer the proposed solution since the default policy leads to a
much more predictable
data flow. For appropriately sizing buffers for push channels in
multi-rate component systems,
we thought about it to specify on the input port level the buffer size
(there is already a
default conn policy for each input port). A code generator like orogen
could easily calculate
the largest buffer size required between components and inform the
input port, or stick to
pull connections.

I would definately not 'imply' different behaviours depending on
buffer/data. After
all, a data object is just a circular buffer of size 1.

Peter

Updated dataflow semantics for RTT

>
> It doesn't fix the out-of-order bug for the buffer-per-channel case.
> You can still starve a certain channel and get old data much later.

That's why we propose to change the default. This starving already

happens in very simple/clean setups, unless you start to round-robin

all the channels. But again, this does not prevent out-of-order arrivals.

This is one thing where we have a completely different point of view. If
you need in-order arrival, you have to enforce it on the reading side with
something like Rock's stream aligner. The only case where the proposed
change will allow you some in-order guarantees is the local connection case
anyways, in all other cases you have unbounded latencies that do
reordering. As for stale data: if you expect buffers to be used on your
inputs, the receiving component must always read all data available (hence,
no "very old stale data"). Otherwise, you will always have some form of
buildup in the buffers, in a form you can't control. Finally, sharing a
single buffer allows one high-frequency connection to starve all others by
filling the buffer.

> I prefer the proposed solution since the default policy leads to a
> much more predictable data flow. For appropriately sizing buffers

for push channels in multi-rate component systems, we thought

about it to specify on the input port level the buffer size
> (there is already a default conn policy for each input port). A code

generator like orogen could easily calculate the largest buffer size

required between components and inform the input port, or stick to
> pull connections.
>

orogen only deals with single components. Syskit is the one inmpacted by
this change, and it is for me out of the picture to compute all possible
connections in all possible networks. Pull connections are not practical as
soon as you have a inter-host connection (i.e. remote connection) as they
break the reading part.

The bottom line is: the proposed change is really broken in the use case of
networks that are dynamic, which is my very use case. Making the new (or
the old) behaviour optional would be also hard because they really demand
different APIs (you can't keep a policy in the connection API when truly
the policy is per-port).

> I would definately not 'imply' different behaviours depending on
> buffer/data. After all, a data object is just a circular buffer of size 1.
>

I thought the same, but start to disagree. Data and buffer have (and should
have) different semantics. The fact that the data object is the same don't
change that.

Data connection: read() only returns the last sample received by the input
port
Buffer connection: read() returns all samples received by the input port
(apart from buffer size limitations).

Sylvain

Updated dataflow semantics for RTT

Am 22.09.2015 um 18:15 schrieb Johannes Meyer:
> Hi,
>
> On Tue, Sep 22, 2015 at 3:32 PM, Janosch Machowinski
> <Janosch [dot] Machowinski [..] ... Janosch [dot] Machowinski [..] ...>> wrote:
>
> Hi,
> here are my remarks:
> First of all, I like the idea of the input buffer (push case). But
> I also see some
> problems with it. The restriction on the same buffer size and type
> in the
> connection policy will be annoying (In my mind I can see the
> runtime errors pop up).
> It will be hard to track this in bigger systems. Do I get it
> correctly, that you only
> want to preserve arrival order and runtime here ?
>
>
> No, we think the current implementation with one buffer per connection
> could really be considered as a bug, at least for certain use cases.
> Consider two output ports W1 and W2 which are both connected to the
> same input port R with a data connection. What I would expect is:
>
> W1.write("foo")
> W2.write("bar")
> assert(R.read(x) == NewData && x == "bar")
> assert(R.read(x) == OldData && x == "bar")
>
> so each writer overwrites the data sample from previous writes (think
> of a robot command). But what happens in RTT v2 is:
>
> W1.write("foo")
> W2.write("bar")
> assert(R.read(x) == NewData && x == "foo")
> assert(R.read(x) == NewData && x == "bar")
>
> and I cannot even be sure if I first read "foo" or "bar" because it
> depends on what connection was read last. The first result would be
> the new default case (push), while the second can still be achieved by
> setting pull to true, which had no effect until now for local connections.
>
> Certainly there is a significant chance that the new restrictions
> break existing deployments and that's why we would only release it
> with a new minor, if not even major version number. But to fix this
> serious drawback of the current dataflow implementation it is
> certainly worth the effort.

What you actually want here is a paradigm change. The buffer of the
input port should not be defined
any more by the connection, but by the input port. Using this thought,
one could go further and
infer the connection policy (buffer size / type) automatically.

Ahh, I just realized, what is odd about your example. As a design
principle, a component does not know anything
about the connections that are attached to the port. So, if you are a
robot controller and want to always use
the latest sample, you MUST use R.readNewest(x). You just may not rely
on the fact that the outside world
used the correct connection. Actually in this case you get the expected
behaviour, if the data order is not broken.
>
> I agree with Sylvain here: It is the transport's responsibility to not
> block on write or read operations and if the underlying implementation
> cannot guarantee that, to use single worker thread, thread pool or one
> thread per connection. If this would be enforced by the RTT connection
> factory already, it would be impossible to use real-time capable
> transports like Xenomai message queues.
>
> We indeed also have an updated CORBA transport ready which introduces
> oneway calls wherever possible and might solve some of the problems
> you observed. It could be considered as stable and I could prepare
> another pull request, but of course there would be plenty of conflicts
> with the updated-dataflow-semantics branch, so I would prefer to
> postpone this until this one is merged.
Hm, this one has been bugging us for a long time... If it is stable, we
should perhaps merge it before the big
data flow change. It is basically a bugfix to the last version with the
old dataflow...
>
>
> For the creation of shared connections, I would recommend an new
> API. The create call
> should contain all involved members. This spares us the headache
> of tracking buffer sizes
> and types around in the code. Also this will simplify the code
> logic, as we don't need the lookup.
> e.g.
> out.connectShared(b.in <http://b.in>, c.in <http://c.in>, d.in
> <http://d.in>, policy) ;
>
>
> A new API instead of adding the shared flag to the ConnPolicy?
Yes
> Or only as a thin wrapper which implies that the shared flag is set?
> It should be noted that it is not the new concept of shared
> connections that adds constraints, but especially the per-input port
> buffers implied by the default pull = false setting, even for private
> connections. What you probably mean is to introduce a separate API for
> all new dataflow models and make sure that the existing API calls
> behave the same as before.
If I got I right, one could still create a shared output connection
together with a (second) normal buffered connection.
The new API would just make sure that you don't get annoying runtime
errors, like non matching policies etc in the
shared case. Using the old api for the shared case is just a bit to
implicit and error prone for my taste.
Janosch

Updated dataflow semantics for RTT

> On Sep 22, 2015, at 13:01, Janosch Machowinski <Janosch [dot] Machowinski [..] ...> wrote:
>
> Am 22.09.2015 um 18:15 schrieb Johannes Meyer:
>> Hi,
>>
>> On Tue, Sep 22, 2015 at 3:32 PM, Janosch Machowinski <Janosch [dot] Machowinski [..] ... Janosch [dot] Machowinski [..] ...>> wrote:
>> Hi,
>> here are my remarks:
>> First of all, I like the idea of the input buffer (push case). But I also see some
>> problems with it. The restriction on the same buffer size and type in the
>> connection policy will be annoying (In my mind I can see the runtime errors pop up).
>> It will be hard to track this in bigger systems. Do I get it correctly, that you only
>> want to preserve arrival order and runtime here ?
>>
>> No, we think the current implementation with one buffer per connection could really be considered as a bug, at least for certain use cases. Consider two output ports W1 and W2 which are both connected to the same input port R with a data connection. What I would expect is:
>>
>> W1.write("foo")
>> W2.write("bar")
>> assert(R.read(x) == NewData && x == "bar")
>> assert(R.read(x) == OldData && x == "bar")
>>
>> so each writer overwrites the data sample from previous writes (think of a robot command). But what happens in RTT v2 is:
>>
>> W1.write("foo")
>> W2.write("bar")
>> assert(R.read(x) == NewData && x == "foo")
>> assert(R.read(x) == NewData && x == "bar")
>>
>> and I cannot even be sure if I first read "foo" or "bar" because it depends on what connection was read last. The first result would be the new default case (push), while the second can still be achieved by setting pull to true, which had no effect until now for local connections.
>>
>> Certainly there is a significant chance that the new restrictions break existing deployments and that's why we would only release it with a new minor, if not even major version number. But to fix this serious drawback of the current dataflow implementation it is certainly worth the effort.
>
> What you actually want here is a paradigm change. The buffer of the input port should not be defined
> any more by the connection, but by the input port. Using this thought, one could go further and
> infer the connection policy (buffer size / type) automatically.

I will have to disagree here. Sylvain’s got it right - it is a system design issue to deal with connections. I do not want any thing inferring what type of buffer policy/size/type someone else thinks my application needs. No one but the system design of any given system has enough knowledge to make those decisions.

> Ahh, I just realized, what is odd about your example. As a design principle, a component does not know anything
> about the connections that are attached to the port. So, if you are a robot controller and want to always use
> the latest sample, you MUST use R.readNewest(x). You just may not rely on the fact that the outside world
> used the correct connection. Actually in this case you get the expected behaviour, if the data order is not broken.

Actually, you don’t need to use readNewest(). The problem is that systems designed with v1 port semantics in mind that use v2 ports end up coupling the component with the type of connection, as they have to work around the problem that Johannes so aptly described above (and that is similar to a problem posted on the ML some time ago).

>> I agree with Sylvain here: It is the transport's responsibility to not block on write or read operations and if the underlying implementation cannot guarantee that, to use single worker thread, thread pool or one thread per connection. If this would be enforced by the RTT connection factory already, it would be impossible to use real-time capable transports like Xenomai message queues.
>>
>> We indeed also have an updated CORBA transport ready which introduces oneway calls wherever possible and might solve some of the problems you observed. It could be considered as stable and I could prepare another pull request, but of course there would be plenty of conflicts with the updated-dataflow-semantics branch, so I would prefer to postpone this until this one is merged.
> Hm, this one has been bugging us for a long time... If it is stable, we should perhaps merge it before the big
> data flow change. It is basically a bugfix to the last version with the old dataflow...
>>
>>
>> For the creation of shared connections, I would recommend an new API. The create call
>> should contain all involved members. This spares us the headache of tracking buffer sizes
>> and types around in the code. Also this will simplify the code logic, as we don't need the lookup.
>> e.g.
>> out.connectShared(b.in <http://b.in />, c.in <http://c.in />, d.in <http://d.in />, policy) ;
>>
>> A new API instead of adding the shared flag to the ConnPolicy?
> Yes

Could you state the goal of the existing API, and this new API, in a way that they are orthogonal and not overlapping?

>> Or only as a thin wrapper which implies that the shared flag is set?
>> It should be noted that it is not the new concept of shared connections that adds constraints, but especially the per-input port buffers implied by the default pull = false setting, even for private connections. What you probably mean is to introduce a separate API for all new dataflow models and make sure that the existing API calls behave the same as before.
> If I got I right, one could still create a shared output connection together with a (second) normal buffered connection.
> The new API would just make sure that you don't get annoying runtime errors, like non matching policies etc in the
> shared case. Using the old api for the shared case is just a bit to implicit and error prone for my taste.

I think that Johannes is trying to graft a fix for some of the existing implementation issues on top of the existing implementation. Using a second API would cause two APIs to exist, which would simply confuse users. There’d also be the question of what is the future for either API. I think that having one API that covers both cases, and coping gracefully with backwards compatibility for a period of time, is a better long term solution. Jumping from one to the other (ala v2.0) or having two parallel APIs exist along side each other, is likely to lead to a lot of confusion and trouble.

YMMV
S

Updated dataflow semantics for RTT

Am 23.09.2015 um 00:20 schrieb S Roderick:
>> What you actually want here is a paradigm change. The buffer of the
>> input port should not be defined
>> any more by the connection, but by the input port. Using this
>> thought, one could go further and
>> infer the connection policy (buffer size / type) automatically.
>
> I will have to disagree here. Sylvain’s got it right - it is a system
> design issue to deal with connections. I do not want any thing
> inferring what type of buffer policy/size/type someone else thinks my
> application needs. No one but the system design of any given system
> has enough knowledge to make those decisions.
I think we have a misunderstand here. I only try to rephrase the
proposal. The proposal states, that there is only
one input buffer, and that all outgoing connections going to this input
buffer must match in type / buffer size.
The type of the input buffer is determined by the first connection. To
make life easier, one could also turn the
principle around. During deployment time, set the type and size of the
input buffer, and as all connecting ports,
going to this port must match anyway, not specify the information there.

>
>> Ahh, I just realized, what is odd about your example. As a design
>> principle, a component does not know anything
>> about the connections that are attached to the port. So, if you are a
>> robot controller and want to always use
>> the latest sample, you MUST use R.readNewest(x). You just may not
>> rely on the fact that the outside world
>> used the correct connection. Actually in this case you get the
>> expected behaviour, if the data order is not broken.
>
> Actually, you don’t need to use readNewest(). The problem is that
> systems designed with v1 port semantics in mind that use v2 ports end
> up coupling the component with the type of connection, as they have to
> work around the problem that Johannes so aptly described above (and
> that is similar to a problem posted on the ML some time ago).
Could you rephrase the sentence ' The problem is that systems designed
with v1 port semantics in mind that use v2 ports end up coupling the
component with the type of connection' ? I don't get what you mean.
>>>
>>>
>>> For the creation of shared connections, I would recommend an new
>>> API. The create call
>>> should contain all involved members. This spares us the headache
>>> of tracking buffer sizes
>>> and types around in the code. Also this will simplify the code
>>> logic, as we don't need the lookup.
>>> e.g.
>>> out.connectShared(b.in <http://b.in />, c.in <http://c.in />, d.in
>>> <http://d.in />, policy) ;
>>>
>>>
>>> A new API instead of adding the shared flag to the ConnPolicy?
>> Yes
>
> Could you state the goal of the existing API, and this new API, in a
> way that they are orthogonal and not overlapping?
The goal would just be, to have am more explicit interface. A shared
connection is fundamental different from a
normal connection. Also the proposal states, that all the policies used
to create a shared connection, must match.
This is error prone and hard to track in my eyes. Also it is a bit
implicit what happens.
E.g: I just called connect : Hm, did I create a new connection, or was I
attached to an existing buffer ?
Janosch