[RFC] Component Deployment and scripting in 2.2

We're looking at improving deployment of components, and now's a good time to
give your feedback. These features will be developed after 2.1 is released.

Most users are not happy with the XML format that specifies which components
are loaded and connected in an application. Especially the flexibility and the
fixed ordering of configuration/setup steps disturbs even modest applications.
The scripting interface on the other hand provides a more controlled, readable
and flexible environment to specify how an application must be setup.

Therefor, the 'best practice' in deployment will emphasis doing things in
scripts and leave XML for basic setup steps. For example: importing component
libraries or plugins and instantiating components. Everything after that
should be done in a script.

For this to work as painless as possible, we need to relook at the scripting
API of the DeploymentComponent and how XML and scripting integrate better.
These features would be 'primary' for the XML:

* Loading plugins, component libraries etc. -- 'Import'
* Instantiating components -- component struct

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Peter Soetens wrote:

> We're looking at improving deployment of components, and now's a good time to
> give your feedback. These features will be developed after 2.1 is released.
>
> Most users are not happy with the XML format that specifies which components
> are loaded and connected in an application. Especially the flexibility and the
> fixed ordering of configuration/setup steps disturbs even modest applications.
> The scripting interface on the other hand provides a more controlled, readable
> and flexible environment to specify how an application must be setup.
>
> Therefor, the 'best practice' in deployment will emphasis doing things in
> scripts and leave XML for basic setup steps. For example: importing component
> libraries or plugins and instantiating components. Everything after that
> should be done in a script.

Let me try to put this in the somewhat broader context of "separation of
concerns" according to the "4C" paradigm (= Computation, Communication,
Configuration and Coordination).

Traditionally, that is, frameworks driven by functionality focus on using
components for separating the Computations in the system. I would like to
see Orocos/RTT evolve towards the best professional framework to support 4C
in each of its use cases. And deployment _is_ such an important use case in
the professional context. And we have seen lots of deployment use cases
that require a full-fledged "deployment component", that is, one that needs
to have "Computations" inside (= the "behavioural"/"dynamic"/"runtime" part
of the deployment, and not just the "structural" part). This _requires_ to
make the step to introduce "Computation" ("scripting") into deployment; for
the deployment _specification_, but also for the deployment _execution_.

> For this to work as painless as possible, we need to relook at the scripting
> API of the DeploymentComponent and how XML and scripting integrate better.

The API will, by necessity, become "state full"...

Herman

> These features would be 'primary' for the XML:
>
> * Loading plugins, component libraries etc. -- 'Import'
> * Instantiating components -- component struct
> * Loading a script (which contains programs andor state machines) --
> 'LoadScript'
> * Starting a specific program or state machine 'StartScript'.
>
> (the old XML features would remain as-is but are not recommended)
>
> This would result in this deployment file which 90% of the applications uses:
>

> <?xml version="1.0" encoding="UTF-8"?>
> <!DOCTYPE properties SYSTEM "cpf.dtd">
> <properties>
> <!-- set import paths -->
> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>
> <!-- Load startup script in Deployer -->
> <struct name="Deployer" type="OCL::DeploymentComponent">
>  <!-- The name of a file: -->
>  <simple name="LoadScript" type="string"><value>startup.ops</value></simple>
>  <!-- The name of a program or state machine: -->
>  <simple name="StartScript" type="string"><value>BarProgram</value></simple>
> </struct>
> </properties>
> 

>
> A Lua-like variant for loading scripts would be very similar, also since lua
> will become a part of OCL.
>
> The idea is then that startup.ops contains 'top level' script commands,
> programs, functions and state machines in any order. For example:
>
>
> var int counter = 0; // counter is added to the attributes of the scripting
> service (global in this service).
>
> // foo program becomes a new service in the component
> program foo() {
>   var int foo_counter =1;
>   while (true) {
>      foo_counter = foo_counter + 1
>      yield
>  }
> }
>
> // after foo has been parsed, it is available:
> foo.foo_counter = 0
> foo.start()
> 

>
> As you can see, variables declared at the global scope are added to the
> scripting service's scope (all scripts share this). You can specify a program
> which is parsed and which can be used in subsequent script statements or
> programs/state machines defined in this file or in other files.
> Clearly, any of these elements are optional.
>
> Finally, I propose some slight (breaking) modifications to the
> DeploymentComponent API:
> - connectPorts: as it is now, it connects equally named ports.
> -> This really has to go. It's very bad practice.
> - connectTwoPorts: does not take a connection policy as argument
> -> would go too and be replaced by:
> + connect("comp1.portA","comp2.portB",ConnPolicy)
> -> connects two ports (possibly in services) with a policy
> + createStream("comp1.portC", ConnPolicy)
> -> setup a stream from a port (in a service) with a policy
>
> Any other suggestions or objections welcome.
>
> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days too.
> I expect 2.2.0 to be around by the end of october. There is no schedule yet
> for a 1.14.
>
> Peter

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Peter Soetens wrote:

> We're looking at improving deployment of components, and now's a good time to
> give your feedback. These features will be developed after 2.1 is released.
>
> Most users are not happy with the XML format that specifies which components
> are loaded and connected in an application. Especially the flexibility and the
> fixed ordering of configuration/setup steps disturbs even modest applications.
> The scripting interface on the other hand provides a more controlled, readable
> and flexible environment to specify how an application must be setup.
>
> Therefor, the 'best practice' in deployment will emphasis doing things in
> scripts and leave XML for basic setup steps. For example: importing component
> libraries or plugins and instantiating components. Everything after that
> should be done in a script.

Let me try to put this in the somewhat broader context of "separation of
concerns" according to the "4C" paradigm (= Computation, Communication,
Configuration and Coordination).

Traditionally, that is, frameworks driven by functionality focus on using
components for separating the Computations in the system. I would like to
see Orocos/RTT evolve towards the best professional framework to support 4C
in each of its use cases. And deployment _is_ such an important use case in
the professional context. And we have seen lots of deployment use cases
that require a full-fledged "deployment component", that is, one that needs
to have "Computations" inside (= the "behavioural"/"dynamic"/"runtime" part
of the deployment, and not just the "structural" part). This _requires_ to
make the step to introduce "Computation" ("scripting") into deployment; for
the deployment _specification_, but also for the deployment _execution_.

> For this to work as painless as possible, we need to relook at the scripting
> API of the DeploymentComponent and how XML and scripting integrate better.

The API will, by necessity, become "state full"...

Herman

> These features would be 'primary' for the XML:
>
> * Loading plugins, component libraries etc. -- 'Import'
> * Instantiating components -- component struct
> * Loading a script (which contains programs andor state machines) --
> 'LoadScript'
> * Starting a specific program or state machine 'StartScript'.
>
> (the old XML features would remain as-is but are not recommended)
>
> This would result in this deployment file which 90% of the applications uses:
>

> <?xml version="1.0" encoding="UTF-8"?>
> <!DOCTYPE properties SYSTEM "cpf.dtd">
> <properties>
> <!-- set import paths -->
> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>
> <!-- Load startup script in Deployer -->
> <struct name="Deployer" type="OCL::DeploymentComponent">
>  <!-- The name of a file: -->
>  <simple name="LoadScript" type="string"><value>startup.ops</value></simple>
>  <!-- The name of a program or state machine: -->
>  <simple name="StartScript" type="string"><value>BarProgram</value></simple>
> </struct>
> </properties>
> 

>
> A Lua-like variant for loading scripts would be very similar, also since lua
> will become a part of OCL.
>
> The idea is then that startup.ops contains 'top level' script commands,
> programs, functions and state machines in any order. For example:
>
>
> var int counter = 0; // counter is added to the attributes of the scripting
> service (global in this service).
>
> // foo program becomes a new service in the component
> program foo() {
>   var int foo_counter =1;
>   while (true) {
>      foo_counter = foo_counter + 1
>      yield
>  }
> }
>
> // after foo has been parsed, it is available:
> foo.foo_counter = 0
> foo.start()
> 

>
> As you can see, variables declared at the global scope are added to the
> scripting service's scope (all scripts share this). You can specify a program
> which is parsed and which can be used in subsequent script statements or
> programs/state machines defined in this file or in other files.
> Clearly, any of these elements are optional.
>
> Finally, I propose some slight (breaking) modifications to the
> DeploymentComponent API:
> - connectPorts: as it is now, it connects equally named ports.
> -> This really has to go. It's very bad practice.
> - connectTwoPorts: does not take a connection policy as argument
> -> would go too and be replaced by:
> + connect("comp1.portA","comp2.portB",ConnPolicy)
> -> connects two ports (possibly in services) with a policy
> + createStream("comp1.portC", ConnPolicy)
> -> setup a stream from a port (in a service) with a policy
>
> Any other suggestions or objections welcome.
>
> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days too.
> I expect 2.2.0 to be around by the end of october. There is no schedule yet
> for a 1.14.
>
> Peter

[RFC] Component Deployment and scripting in 2.2

On Thu, Sep 30, 2010 at 1:45 PM, Peter Soetens <peter [..] ...>wrote:

> We're looking at improving deployment of components, and now's a good time
> to
> give your feedback. These features will be developed after 2.1 is released.
>
> Most users are not happy with the XML format that specifies which
> components
> are loaded and connected in an application. Especially the flexibility and
> the
> fixed ordering of configuration/setup steps disturbs even modest
> applications.
> The scripting interface on the other hand provides a more controlled,
> readable
> and flexible environment to specify how an application must be setup.
>
> Therefor, the 'best practice' in deployment will emphasis doing things in
> scripts and leave XML for basic setup steps.

Aren't you going to fast here? It seems to me that the problems you are
sketching above merely saying that the cpf.dtd does not offer enough
semantics to express a typical application setup. I don't understand why
you would have to throw xml away in favour of scripting?

Klaas

[RFC] Component Deployment and scripting in 2.2

On Thu, Sep 30, 2010 at 1:45 PM, Peter Soetens <peter [..] ...>wrote:

> We're looking at improving deployment of components, and now's a good time
> to
> give your feedback. These features will be developed after 2.1 is released.
>
> Most users are not happy with the XML format that specifies which
> components
> are loaded and connected in an application. Especially the flexibility and
> the
> fixed ordering of configuration/setup steps disturbs even modest
> applications.
> The scripting interface on the other hand provides a more controlled,
> readable
> and flexible environment to specify how an application must be setup.
>
> Therefor, the 'best practice' in deployment will emphasis doing things in
> scripts and leave XML for basic setup steps.

Aren't you going to fast here? It seems to me that the problems you are
sketching above merely saying that the cpf.dtd does not offer enough
semantics to express a typical application setup. I don't understand why
you would have to throw xml away in favour of scripting?

Klaas

[RFC] Component Deployment and scripting in 2.2

On Thursday 30 September 2010 14:24:28 Klaas Gadeyne wrote:
> On Thu, Sep 30, 2010 at 1:45 PM, Peter Soetens
<peter [..] ...>wrote:
> > We're looking at improving deployment of components, and now's a good
> > time to
> > give your feedback. These features will be developed after 2.1 is
> > released.
> >
> > Most users are not happy with the XML format that specifies which
> > components
> > are loaded and connected in an application. Especially the flexibility
> > and the
> > fixed ordering of configuration/setup steps disturbs even modest
> > applications.
> > The scripting interface on the other hand provides a more controlled,
> > readable
> > and flexible environment to specify how an application must be setup.
> >
> > Therefor, the 'best practice' in deployment will emphasis doing things in
> > scripts and leave XML for basic setup steps.
>
> Aren't you going to fast here? It seems to me that the problems you are
> sketching above merely saying that the cpf.dtd does not offer enough
> semantics to express a typical application setup. I don't understand why
> you would have to throw xml away in favour of scripting?

I'm not throwing it away (cfr reply to Stephen). But users should be aware
that with the current situation, scripting instead of the XML format will save
them time. The idea is to remove the rough edges of the scripting approach,
while leaving the xml approach as-is, and not trying to add more features to
xml to aleviate it's short-commings.

Even so, the deployer is not offering a supervision framework like DFKI's
Roby[1], and probably never will be. In this regard, the deployer won't
accumulate features and should work as a slave/library to a more intelligent
supervision system.

Peter

[1] http://doudou.github.com/roby/index.html

[RFC] Component Deployment and scripting in 2.2

On Thursday 30 September 2010 14:24:28 Klaas Gadeyne wrote:
> On Thu, Sep 30, 2010 at 1:45 PM, Peter Soetens
<peter [..] ...>wrote:
> > We're looking at improving deployment of components, and now's a good
> > time to
> > give your feedback. These features will be developed after 2.1 is
> > released.
> >
> > Most users are not happy with the XML format that specifies which
> > components
> > are loaded and connected in an application. Especially the flexibility
> > and the
> > fixed ordering of configuration/setup steps disturbs even modest
> > applications.
> > The scripting interface on the other hand provides a more controlled,
> > readable
> > and flexible environment to specify how an application must be setup.
> >
> > Therefor, the 'best practice' in deployment will emphasis doing things in
> > scripts and leave XML for basic setup steps.
>
> Aren't you going to fast here? It seems to me that the problems you are
> sketching above merely saying that the cpf.dtd does not offer enough
> semantics to express a typical application setup. I don't understand why
> you would have to throw xml away in favour of scripting?

I'm not throwing it away (cfr reply to Stephen). But users should be aware
that with the current situation, scripting instead of the XML format will save
them time. The idea is to remove the rough edges of the scripting approach,
while leaving the xml approach as-is, and not trying to add more features to
xml to aleviate it's short-commings.

Even so, the deployer is not offering a supervision framework like DFKI's
Roby[1], and probably never will be. In this regard, the deployer won't
accumulate features and should work as a slave/library to a more intelligent
supervision system.

Peter

[1] http://doudou.github.com/roby/index.html

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Peter Soetens wrote:

> On Thursday 30 September 2010 14:24:28 Klaas Gadeyne wrote:
>> On Thu, Sep 30, 2010 at 1:45 PM, Peter Soetens
> <peter [..] ...>wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to
>>> give your feedback. These features will be developed after 2.1 is
>>> released.
>>>
>>> Most users are not happy with the XML format that specifies which
>>> components
>>> are loaded and connected in an application. Especially the flexibility
>>> and the
>>> fixed ordering of configuration/setup steps disturbs even modest
>>> applications.
>>> The scripting interface on the other hand provides a more controlled,
>>> readable
>>> and flexible environment to specify how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps.
>>
>> Aren't you going to fast here? It seems to me that the problems you are
>> sketching above merely saying that the cpf.dtd does not offer enough
>> semantics to express a typical application setup. I don't understand why
>> you would have to throw xml away in favour of scripting?
>
> I'm not throwing it away (cfr reply to Stephen). But users should be aware
> that with the current situation, scripting instead of the XML format will save
> them time. The idea is to remove the rough edges of the scripting approach,
> while leaving the xml approach as-is, and not trying to add more features to
> xml to aleviate it's short-commings.
>
> Even so, the deployer is not offering a supervision framework like DFKI's
> Roby[1], and probably never will be. In this regard, the deployer won't
> accumulate features and should work as a slave/library to a more intelligent
> supervision system.

Good!!!! Don't put efforts in Orocos when external frameworks are (could
be...) doing much better. Only spend the effort there where realtime
performance matters...

Herman

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Peter Soetens wrote:

> On Thursday 30 September 2010 14:24:28 Klaas Gadeyne wrote:
>> On Thu, Sep 30, 2010 at 1:45 PM, Peter Soetens
> <peter [..] ...>wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to
>>> give your feedback. These features will be developed after 2.1 is
>>> released.
>>>
>>> Most users are not happy with the XML format that specifies which
>>> components
>>> are loaded and connected in an application. Especially the flexibility
>>> and the
>>> fixed ordering of configuration/setup steps disturbs even modest
>>> applications.
>>> The scripting interface on the other hand provides a more controlled,
>>> readable
>>> and flexible environment to specify how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps.
>>
>> Aren't you going to fast here? It seems to me that the problems you are
>> sketching above merely saying that the cpf.dtd does not offer enough
>> semantics to express a typical application setup. I don't understand why
>> you would have to throw xml away in favour of scripting?
>
> I'm not throwing it away (cfr reply to Stephen). But users should be aware
> that with the current situation, scripting instead of the XML format will save
> them time. The idea is to remove the rough edges of the scripting approach,
> while leaving the xml approach as-is, and not trying to add more features to
> xml to aleviate it's short-commings.
>
> Even so, the deployer is not offering a supervision framework like DFKI's
> Roby[1], and probably never will be. In this regard, the deployer won't
> accumulate features and should work as a slave/library to a more intelligent
> supervision system.

Good!!!! Don't put efforts in Orocos when external frameworks are (could
be...) doing much better. Only spend the effort there where realtime
performance matters...

Herman

Ruben Smits's picture

[RFC] Component Deployment and scripting in 2.2

On Thursday 30 September 2010 13:45:45 Peter Soetens wrote:
> We're looking at improving deployment of components, and now's a good time
> to give your feedback. These features will be developed after 2.1 is
> released.
>
> Most users are not happy with the XML format that specifies which
> components are loaded and connected in an application. Especially the
> flexibility and the fixed ordering of configuration/setup steps disturbs
> even modest applications. The scripting interface on the other hand
> provides a more controlled, readable and flexible environment to specify
> how an application must be setup.
>
> Therefor, the 'best practice' in deployment will emphasis doing things in
> scripts and leave XML for basic setup steps. For example: importing
> component libraries or plugins and instantiating components. Everything
> after that should be done in a script.
>
> For this to work as painless as possible, we need to relook at the
> scripting API of the DeploymentComponent and how XML and scripting
> integrate better. These features would be 'primary' for the XML:
>
> * Loading plugins, component libraries etc. -- 'Import'
> * Instantiating components -- component struct
> * Loading a script (which contains programs andor state machines) --
> 'LoadScript'
> * Starting a specific program or state machine 'StartScript'.
>
> (the old XML features would remain as-is but are not recommended)
>
> This would result in this deployment file which 90% of the applications
> uses:

> <?xml version="1.0" encoding="UTF-8"?>
> <!DOCTYPE properties SYSTEM "cpf.dtd">
> <properties>
> <!-- set import paths -->
> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
> 
> <!-- Load startup script in Deployer -->
> <struct name="Deployer" type="OCL::DeploymentComponent">
>   <!-- The name of a file: -->
>   <simple name="LoadScript"
> type="string"><value>startup.ops</value></simple> <!-- The name of a
> program or state machine: -->
>   <simple name="StartScript"
> type="string"><value>BarProgram</value></simple> </struct>
> </properties>
> 

>
> A Lua-like variant for loading scripts would be very similar, also since
> lua will become a part of OCL.
>
> The idea is then that startup.ops contains 'top level' script commands,
> programs, functions and state machines in any order. For example:
>
>
> var int counter = 0; // counter is added to the attributes of the scripting
> service (global in this service).
> 
> // foo program becomes a new service in the component
> program foo() {
>    var int foo_counter =1;
>    while (true) {
>       foo_counter = foo_counter + 1
>       yield
>   }
> }
> 
> // after foo has been parsed, it is available:
> foo.foo_counter = 0
> foo.start()
> 

>
> As you can see, variables declared at the global scope are added to the
> scripting service's scope (all scripts share this). You can specify a
> program which is parsed and which can be used in subsequent script
> statements or programs/state machines defined in this file or in other
> files.
> Clearly, any of these elements are optional.

your example is not really explanatory for the deployement use case. Could you
provide an example in which a set of components is being deployed in a way
which is hard/impossible with the XML deployement.

> Finally, I propose some slight (breaking) modifications to the
> DeploymentComponent API:
> - connectPorts: as it is now, it connects equally named ports.
> -> This really has to go. It's very bad practice.
> - connectTwoPorts: does not take a connection policy as argument
> -> would go too and be replaced by:
> + connect("comp1.portA","comp2.portB",ConnPolicy)
> -> connects two ports (possibly in services) with a policy
> + createStream("comp1.portC", ConnPolicy)
> -> setup a stream from a port (in a service) with a policy
>
> Any other suggestions or objections welcome.

I would like to add the feature to be able to connect ports that reside in
(sub)services:

connectTwoPorts("comp1.serv1.subserv1.portA","comp2.serv2.subser2.subsubserv2.portB")

If you don't like dots you can always replace them with slashes.

The same would go for createStream

createStream("comp1.service2.subservice4.portC", ConnPolicy)

Ruben

> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
> too. I expect 2.2.0 to be around by the end of october. There is no
> schedule yet for a 1.14.
>
> Peter

[RFC] Component Deployment and scripting in 2.2

On Sep 30, 2010, at 07:45 , Peter Soetens wrote:

> We're looking at improving deployment of components, and now's a good time to
> give your feedback. These features will be developed after 2.1 is released.

Add the ability to deploy components in groups, so that group A is deployed completely, followed by group B, ..., and then group B is un-deployed completely, and finally group A is un-deployed completely. This is necessary for things like logging, which currently hacks this by using the site-file approach. This is really just sequencing application startup and shutdown.

> Most users are not happy with the XML format that specifies which components
> are loaded and connected in an application. Especially the flexibility and the
> fixed ordering of configuration/setup steps disturbs even modest applications.
> The scripting interface on the other hand provides a more controlled, readable
> and flexible environment to specify how an application must be setup.
>
> Therefor, the 'best practice' in deployment will emphasis doing things in
> scripts and leave XML for basic setup steps. For example: importing component
> libraries or plugins and instantiating components. Everything after that
> should be done in a script.

What exactly are you talking about changing here? In particular, what are you talking about removing? I would really like it if nothing that currently works was removed - we already have a high enough barrier getting from v1 to v2 due to "removed features" that we use. It would be nice if this barrier didn't grow further.

The only issue we have with XML is that large-scale deployments are tricky due to difficulty in configuring the deployment. The recent discussion on use of the XML standard ENTITY element may solve much of that for us.

> For this to work as painless as possible, we need to relook at the scripting
> API of the DeploymentComponent and how XML and scripting integrate better.
> These features would be 'primary' for the XML:
>
> * Loading plugins, component libraries etc. -- 'Import'
> * Instantiating components -- component struct

Ruben Smits's picture

[RFC] Component Deployment and scripting in 2.2

On Thursday 30 September 2010 13:45:45 Peter Soetens wrote:
> We're looking at improving deployment of components, and now's a good time
> to give your feedback. These features will be developed after 2.1 is
> released.
>
> Most users are not happy with the XML format that specifies which
> components are loaded and connected in an application. Especially the
> flexibility and the fixed ordering of configuration/setup steps disturbs
> even modest applications. The scripting interface on the other hand
> provides a more controlled, readable and flexible environment to specify
> how an application must be setup.
>
> Therefor, the 'best practice' in deployment will emphasis doing things in
> scripts and leave XML for basic setup steps. For example: importing
> component libraries or plugins and instantiating components. Everything
> after that should be done in a script.
>
> For this to work as painless as possible, we need to relook at the
> scripting API of the DeploymentComponent and how XML and scripting
> integrate better. These features would be 'primary' for the XML:
>
> * Loading plugins, component libraries etc. -- 'Import'
> * Instantiating components -- component struct
> * Loading a script (which contains programs andor state machines) --
> 'LoadScript'
> * Starting a specific program or state machine 'StartScript'.
>
> (the old XML features would remain as-is but are not recommended)
>
> This would result in this deployment file which 90% of the applications
> uses:

> <?xml version="1.0" encoding="UTF-8"?>
> <!DOCTYPE properties SYSTEM "cpf.dtd">
> <properties>
> <!-- set import paths -->
> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
> 
> <!-- Load startup script in Deployer -->
> <struct name="Deployer" type="OCL::DeploymentComponent">
>   <!-- The name of a file: -->
>   <simple name="LoadScript"
> type="string"><value>startup.ops</value></simple> <!-- The name of a
> program or state machine: -->
>   <simple name="StartScript"
> type="string"><value>BarProgram</value></simple> </struct>
> </properties>
> 

>
> A Lua-like variant for loading scripts would be very similar, also since
> lua will become a part of OCL.
>
> The idea is then that startup.ops contains 'top level' script commands,
> programs, functions and state machines in any order. For example:
>
>
> var int counter = 0; // counter is added to the attributes of the scripting
> service (global in this service).
> 
> // foo program becomes a new service in the component
> program foo() {
>    var int foo_counter =1;
>    while (true) {
>       foo_counter = foo_counter + 1
>       yield
>   }
> }
> 
> // after foo has been parsed, it is available:
> foo.foo_counter = 0
> foo.start()
> 

>
> As you can see, variables declared at the global scope are added to the
> scripting service's scope (all scripts share this). You can specify a
> program which is parsed and which can be used in subsequent script
> statements or programs/state machines defined in this file or in other
> files.
> Clearly, any of these elements are optional.

your example is not really explanatory for the deployement use case. Could you
provide an example in which a set of components is being deployed in a way
which is hard/impossible with the XML deployement.

> Finally, I propose some slight (breaking) modifications to the
> DeploymentComponent API:
> - connectPorts: as it is now, it connects equally named ports.
> -> This really has to go. It's very bad practice.
> - connectTwoPorts: does not take a connection policy as argument
> -> would go too and be replaced by:
> + connect("comp1.portA","comp2.portB",ConnPolicy)
> -> connects two ports (possibly in services) with a policy
> + createStream("comp1.portC", ConnPolicy)
> -> setup a stream from a port (in a service) with a policy
>
> Any other suggestions or objections welcome.

I would like to add the feature to be able to connect ports that reside in
(sub)services:

connectTwoPorts("comp1.serv1.subserv1.portA","comp2.serv2.subser2.subsubserv2.portB")

If you don't like dots you can always replace them with slashes.

The same would go for createStream

createStream("comp1.service2.subservice4.portC", ConnPolicy)

Ruben

> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
> too. I expect 2.2.0 to be around by the end of october. There is no
> schedule yet for a 1.14.
>
> Peter

[RFC] Component Deployment and scripting in 2.2

On Thursday 30 September 2010 14:11:14 Ruben Smits wrote:
> On Thursday 30 September 2010 13:45:45 Peter Soetens wrote:
> > We're looking at improving deployment of components, and now's a good
> > time to give your feedback. These features will be developed after 2.1
> > is released.
> >
> > Most users are not happy with the XML format that specifies which
> > components are loaded and connected in an application. Especially the
> > flexibility and the fixed ordering of configuration/setup steps disturbs
> > even modest applications. The scripting interface on the other hand
> > provides a more controlled, readable and flexible environment to specify
> > how an application must be setup.
> >
> > Therefor, the 'best practice' in deployment will emphasis doing things in
> > scripts and leave XML for basic setup steps. For example: importing
> > component libraries or plugins and instantiating components. Everything
> > after that should be done in a script.
> >
> > For this to work as painless as possible, we need to relook at the
> > scripting API of the DeploymentComponent and how XML and scripting
> > integrate better. These features would be 'primary' for the XML:
> >
> > * Loading plugins, component libraries etc. -- 'Import'
> > * Instantiating components -- component struct
> > * Loading a script (which contains programs andor state machines) --
> > 'LoadScript'
> > * Starting a specific program or state machine 'StartScript'.
> >
> > (the old XML features would remain as-is but are not recommended)
> >
> > This would result in this deployment file which 90% of the applications
> > uses:

> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE properties SYSTEM "cpf.dtd">
> > <properties>
> > <!-- set import paths -->
> > <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
> > 
> > <!-- Load startup script in Deployer -->
> > <struct name="Deployer" type="OCL::DeploymentComponent">
> > 
> >   <!-- The name of a file: -->
> >   <simple name="LoadScript"
> > 
> > type="string"><value>startup.ops</value></simple> <!-- The name of a
> > program or state machine: -->
> > 
> >   <simple name="StartScript"
> > 
> > type="string"><value>BarProgram</value></simple> </struct>
> > </properties>
> > 

> >
> > A Lua-like variant for loading scripts would be very similar, also since
> > lua will become a part of OCL.
> >
> > The idea is then that startup.ops contains 'top level' script commands,
> > programs, functions and state machines in any order. For example:
> >
> >
> > var int counter = 0; // counter is added to the attributes of the
> > scripting service (global in this service).
> > 
> > // foo program becomes a new service in the component
> > program foo() {
> > 
> >    var int foo_counter =1;
> >    while (true) {
> >    
> >       foo_counter = foo_counter + 1
> >       yield
> >   
> >   }
> > 
> > }
> > 
> > // after foo has been parsed, it is available:
> > foo.foo_counter = 0
> > foo.start()
> > 

> >
> > As you can see, variables declared at the global scope are added to the
> > scripting service's scope (all scripts share this). You can specify a
> > program which is parsed and which can be used in subsequent script
> > statements or programs/state machines defined in this file or in other
> > files.
> > Clearly, any of these elements are optional.
>
> your example is not really explanatory for the deployement use case. Could
> you provide an example in which a set of components is being deployed in a
> way which is hard/impossible with the XML deployement.

See Stephen's mail :-)

>
> > Finally, I propose some slight (breaking) modifications to the
> > DeploymentComponent API:
> > - connectPorts: as it is now, it connects equally named ports.
> >
> > -> This really has to go. It's very bad practice.
> >
> > - connectTwoPorts: does not take a connection policy as argument
> >
> > -> would go too and be replaced by:
> > + connect("comp1.portA","comp2.portB",ConnPolicy)
> >
> > -> connects two ports (possibly in services) with a policy
> >
> > + createStream("comp1.portC", ConnPolicy)
> >
> > -> setup a stream from a port (in a service) with a policy
> >
> > Any other suggestions or objections welcome.
>
> I would like to add the feature to be able to connect ports that reside in
> (sub)services:
>
> connectTwoPorts("comp1.serv1.subserv1.portA","comp2.serv2.subser2.subsubser
> v2.portB")
>
> If you don't like dots you can always replace them with slashes.
>
> The same would go for createStream
>
> createStream("comp1.service2.subservice4.portC", ConnPolicy)
>

This is what I intend. So we have connect("port-path-a","port-path-
b",ConnPolicy) and createStream() (or even shorter: stream("port-
path",ConnPolicy), both accepting a ConnPolicy and a path-to-port-of-service.

Peter

[RFC] Component Deployment and scripting in 2.2

On Thursday 30 September 2010 14:11:14 Ruben Smits wrote:
> On Thursday 30 September 2010 13:45:45 Peter Soetens wrote:
> > We're looking at improving deployment of components, and now's a good
> > time to give your feedback. These features will be developed after 2.1
> > is released.
> >
> > Most users are not happy with the XML format that specifies which
> > components are loaded and connected in an application. Especially the
> > flexibility and the fixed ordering of configuration/setup steps disturbs
> > even modest applications. The scripting interface on the other hand
> > provides a more controlled, readable and flexible environment to specify
> > how an application must be setup.
> >
> > Therefor, the 'best practice' in deployment will emphasis doing things in
> > scripts and leave XML for basic setup steps. For example: importing
> > component libraries or plugins and instantiating components. Everything
> > after that should be done in a script.
> >
> > For this to work as painless as possible, we need to relook at the
> > scripting API of the DeploymentComponent and how XML and scripting
> > integrate better. These features would be 'primary' for the XML:
> >
> > * Loading plugins, component libraries etc. -- 'Import'
> > * Instantiating components -- component struct
> > * Loading a script (which contains programs andor state machines) --
> > 'LoadScript'
> > * Starting a specific program or state machine 'StartScript'.
> >
> > (the old XML features would remain as-is but are not recommended)
> >
> > This would result in this deployment file which 90% of the applications
> > uses:

> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE properties SYSTEM "cpf.dtd">
> > <properties>
> > <!-- set import paths -->
> > <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
> > 
> > <!-- Load startup script in Deployer -->
> > <struct name="Deployer" type="OCL::DeploymentComponent">
> > 
> >   <!-- The name of a file: -->
> >   <simple name="LoadScript"
> > 
> > type="string"><value>startup.ops</value></simple> <!-- The name of a
> > program or state machine: -->
> > 
> >   <simple name="StartScript"
> > 
> > type="string"><value>BarProgram</value></simple> </struct>
> > </properties>
> > 

> >
> > A Lua-like variant for loading scripts would be very similar, also since
> > lua will become a part of OCL.
> >
> > The idea is then that startup.ops contains 'top level' script commands,
> > programs, functions and state machines in any order. For example:
> >
> >
> > var int counter = 0; // counter is added to the attributes of the
> > scripting service (global in this service).
> > 
> > // foo program becomes a new service in the component
> > program foo() {
> > 
> >    var int foo_counter =1;
> >    while (true) {
> >    
> >       foo_counter = foo_counter + 1
> >       yield
> >   
> >   }
> > 
> > }
> > 
> > // after foo has been parsed, it is available:
> > foo.foo_counter = 0
> > foo.start()
> > 

> >
> > As you can see, variables declared at the global scope are added to the
> > scripting service's scope (all scripts share this). You can specify a
> > program which is parsed and which can be used in subsequent script
> > statements or programs/state machines defined in this file or in other
> > files.
> > Clearly, any of these elements are optional.
>
> your example is not really explanatory for the deployement use case. Could
> you provide an example in which a set of components is being deployed in a
> way which is hard/impossible with the XML deployement.

See Stephen's mail :-)

>
> > Finally, I propose some slight (breaking) modifications to the
> > DeploymentComponent API:
> > - connectPorts: as it is now, it connects equally named ports.
> >
> > -> This really has to go. It's very bad practice.
> >
> > - connectTwoPorts: does not take a connection policy as argument
> >
> > -> would go too and be replaced by:
> > + connect("comp1.portA","comp2.portB",ConnPolicy)
> >
> > -> connects two ports (possibly in services) with a policy
> >
> > + createStream("comp1.portC", ConnPolicy)
> >
> > -> setup a stream from a port (in a service) with a policy
> >
> > Any other suggestions or objections welcome.
>
> I would like to add the feature to be able to connect ports that reside in
> (sub)services:
>
> connectTwoPorts("comp1.serv1.subserv1.portA","comp2.serv2.subser2.subsubser
> v2.portB")
>
> If you don't like dots you can always replace them with slashes.
>
> The same would go for createStream
>
> createStream("comp1.service2.subservice4.portC", ConnPolicy)
>

This is what I intend. So we have connect("port-path-a","port-path-
b",ConnPolicy) and createStream() (or even shorter: stream("port-
path",ConnPolicy), both accepting a ConnPolicy and a path-to-port-of-service.

Peter

[RFC] Component Deployment and scripting in 2.2

On Sep 30, 2010, at 07:45 , Peter Soetens wrote:

> We're looking at improving deployment of components, and now's a good time to
> give your feedback. These features will be developed after 2.1 is released.

Add the ability to deploy components in groups, so that group A is deployed completely, followed by group B, ..., and then group B is un-deployed completely, and finally group A is un-deployed completely. This is necessary for things like logging, which currently hacks this by using the site-file approach. This is really just sequencing application startup and shutdown.

> Most users are not happy with the XML format that specifies which components
> are loaded and connected in an application. Especially the flexibility and the
> fixed ordering of configuration/setup steps disturbs even modest applications.
> The scripting interface on the other hand provides a more controlled, readable
> and flexible environment to specify how an application must be setup.
>
> Therefor, the 'best practice' in deployment will emphasis doing things in
> scripts and leave XML for basic setup steps. For example: importing component
> libraries or plugins and instantiating components. Everything after that
> should be done in a script.

What exactly are you talking about changing here? In particular, what are you talking about removing? I would really like it if nothing that currently works was removed - we already have a high enough barrier getting from v1 to v2 due to "removed features" that we use. It would be nice if this barrier didn't grow further.

The only issue we have with XML is that large-scale deployments are tricky due to difficulty in configuring the deployment. The recent discussion on use of the XML standard ENTITY element may solve much of that for us.

> For this to work as painless as possible, we need to relook at the scripting
> API of the DeploymentComponent and how XML and scripting integrate better.
> These features would be 'primary' for the XML:
>
> * Loading plugins, component libraries etc. -- 'Import'
> * Instantiating components -- component struct

[RFC] Component Deployment and scripting in 2.2

On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> > We're looking at improving deployment of components, and now's a good
> > time to give your feedback. These features will be developed after 2.1
> > is released.
>
> Add the ability to deploy components in groups, so that group A is deployed
> completely, followed by group B, ..., and then group B is un-deployed
> completely, and finally group A is un-deployed completely. This is
> necessary for things like logging, which currently hacks this by using the
> site-file approach. This is really just sequencing application startup and
> shutdown.

You *can* already do this using scripting and a state machine.That's my whole
point. No XML / DeploymentComponent extensions are necessary for this.

>
> > Most users are not happy with the XML format that specifies which
> > components are loaded and connected in an application. Especially the
> > flexibility and the fixed ordering of configuration/setup steps disturbs
> > even modest applications. The scripting interface on the other hand
> > provides a more controlled, readable and flexible environment to specify
> > how an application must be setup.
> >
> > Therefor, the 'best practice' in deployment will emphasis doing things in
> > scripts and leave XML for basic setup steps. For example: importing
> > component libraries or plugins and instantiating components. Everything
> > after that should be done in a script.
>
> What exactly are you talking about changing here? In particular, what are
> you talking about removing? I would really like it if nothing that
> currently works was removed - we already have a high enough barrier
> getting from v1 to v2 due to "removed features" that we use. It would be
> nice if this barrier didn't grow further.

I'm not planning to remove any of them.

>
> The only issue we have with XML is that large-scale deployments are tricky
> due to difficulty in configuring the deployment. The recent discussion on
> use of the XML standard ENTITY element may solve much of that for us.

That's true, but it still won't solve ordering problems.

>
> > For this to work as painless as possible, we need to relook at the
> > scripting API of the DeploymentComponent and how XML and scripting
> > integrate better. These features would be 'primary' for the XML:
> >
> > * Loading plugins, component libraries etc. -- 'Import'
> > * Instantiating components -- component struct
> > * Loading a script (which contains programs andor state machines) --
> > 'LoadScript'
> > * Starting a specific program or state machine 'StartScript'.
>
> Can you please list here what existing features you intend to remove from
> the XML approach?

None.

>
> > (the old XML features would remain as-is but are not recommended)
>
> So are you not removing XML features, but deprecating them with an intent
> to not maintain in the future?

Well, 'deprecating with the intent to point users in the right direction in
the future'. I'm also saying this: 'not adding any more features to the XML
format (ie freezing it).

>
> > This would result in this deployment file which 90% of the applications
> > uses:

> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE properties SYSTEM "cpf.dtd">
> > <properties>
> > <!-- set import paths -->
> > <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
> > 
> > <!-- Load startup script in Deployer -->
> > <struct name="Deployer" type="OCL::DeploymentComponent">
> > 
> >  <!-- The name of a file: -->
> >  <simple name="LoadScript"
> >  type="string"><value>startup.ops</value></simple> <!-- The name of a
> >  program or state machine: -->
> >  <simple name="StartScript"
> >  type="string"><value>BarProgram</value></simple>
> > 
> > </struct>
> > </properties>
> > 

> >
> > A Lua-like variant for loading scripts would be very similar, also since
> > lua will become a part of OCL.
> >
> > The idea is then that startup.ops contains 'top level' script commands,
> > programs, functions and state machines in any order. For example:
> >
> >
> > var int counter = 0; // counter is added to the attributes of the
> > scripting service (global in this service).
> > 
> > // foo program becomes a new service in the component
> > program foo() {
> > 
> >   var int foo_counter =1;
> >   while (true) {
> >   
> >      foo_counter = foo_counter + 1
> >      yield
> >  
> >  }
> > 
> > }
> > 
> > // after foo has been parsed, it is available:
> > foo.foo_counter = 0
> > foo.start()
> > 

> >
> > As you can see, variables declared at the global scope are added to the
> > scripting service's scope (all scripts share this). You can specify a
> > program which is parsed and which can be used in subsequent script
> > statements or programs/state machines defined in this file or in other
> > files.
> > Clearly, any of these elements are optional.
> >
> > Finally, I propose some slight (breaking) modifications to the
> > DeploymentComponent API:
> > - connectPorts: as it is now, it connects equally named ports.
> >
> > -> This really has to go. It's very bad practice.
> >
> > - connectTwoPorts: does not take a connection policy as argument
> >
> > -> would go too and be replaced by:
> > + connect("comp1.portA","comp2.portB",ConnPolicy)
> >
> > -> connects two ports (possibly in services) with a policy
> >
> > + createStream("comp1.portC", ConnPolicy)
> >
> > -> setup a stream from a port (in a service) with a policy
> >
> > Any other suggestions or objections welcome.
>
> The above is not very clear in what changes you intend for user
> applications. Can you give a more concrete example (pseudo code, not
> details) showing a v1 XML, and what you intend the v2 version to look
> like. For instance, the program script you show above looks specific to
> just one component. How is that helping or hindering large scale or
> complex deployments?

I need some time to work that out, but I understand the question...

>
> Can you also list the specific problems in XML that you are trying to
> address. All I see above is "XML bad, scripting better".

What we can do in scripts and not (easily) in xml during deployment:
- Use data/properties in other components to make deployment decisions
- Group start/stop/configure components using a custom state machine
- connect ports/services that were dynamically created during configureHook()
- conditionally execute deployment steps
- go step-by-step through a deployment process (using pause()/step() api)
- Specify easily connection policies etc.
- have a very readable/sequential deployment scenario
- have deployment react to events, changing situations etc.

>
> > I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
> > too. I expect 2.2.0 to be around by the end of october. There is no
> > schedule yet for a 1.14.
>
> Do we have roadmap somewhere for what is going in these future releases?

We can prepare one on the wiki.

Peter

[RFC] Component Deployment and scripting in 2.2

On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> > We're looking at improving deployment of components, and now's a good
> > time to give your feedback. These features will be developed after 2.1
> > is released.
>
> Add the ability to deploy components in groups, so that group A is deployed
> completely, followed by group B, ..., and then group B is un-deployed
> completely, and finally group A is un-deployed completely. This is
> necessary for things like logging, which currently hacks this by using the
> site-file approach. This is really just sequencing application startup and
> shutdown.

You *can* already do this using scripting and a state machine.That's my whole
point. No XML / DeploymentComponent extensions are necessary for this.

>
> > Most users are not happy with the XML format that specifies which
> > components are loaded and connected in an application. Especially the
> > flexibility and the fixed ordering of configuration/setup steps disturbs
> > even modest applications. The scripting interface on the other hand
> > provides a more controlled, readable and flexible environment to specify
> > how an application must be setup.
> >
> > Therefor, the 'best practice' in deployment will emphasis doing things in
> > scripts and leave XML for basic setup steps. For example: importing
> > component libraries or plugins and instantiating components. Everything
> > after that should be done in a script.
>
> What exactly are you talking about changing here? In particular, what are
> you talking about removing? I would really like it if nothing that
> currently works was removed - we already have a high enough barrier
> getting from v1 to v2 due to "removed features" that we use. It would be
> nice if this barrier didn't grow further.

I'm not planning to remove any of them.

>
> The only issue we have with XML is that large-scale deployments are tricky
> due to difficulty in configuring the deployment. The recent discussion on
> use of the XML standard ENTITY element may solve much of that for us.

That's true, but it still won't solve ordering problems.

>
> > For this to work as painless as possible, we need to relook at the
> > scripting API of the DeploymentComponent and how XML and scripting
> > integrate better. These features would be 'primary' for the XML:
> >
> > * Loading plugins, component libraries etc. -- 'Import'
> > * Instantiating components -- component struct
> > * Loading a script (which contains programs andor state machines) --
> > 'LoadScript'
> > * Starting a specific program or state machine 'StartScript'.
>
> Can you please list here what existing features you intend to remove from
> the XML approach?

None.

>
> > (the old XML features would remain as-is but are not recommended)
>
> So are you not removing XML features, but deprecating them with an intent
> to not maintain in the future?

Well, 'deprecating with the intent to point users in the right direction in
the future'. I'm also saying this: 'not adding any more features to the XML
format (ie freezing it).

>
> > This would result in this deployment file which 90% of the applications
> > uses:

> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE properties SYSTEM "cpf.dtd">
> > <properties>
> > <!-- set import paths -->
> > <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
> > 
> > <!-- Load startup script in Deployer -->
> > <struct name="Deployer" type="OCL::DeploymentComponent">
> > 
> >  <!-- The name of a file: -->
> >  <simple name="LoadScript"
> >  type="string"><value>startup.ops</value></simple> <!-- The name of a
> >  program or state machine: -->
> >  <simple name="StartScript"
> >  type="string"><value>BarProgram</value></simple>
> > 
> > </struct>
> > </properties>
> > 

> >
> > A Lua-like variant for loading scripts would be very similar, also since
> > lua will become a part of OCL.
> >
> > The idea is then that startup.ops contains 'top level' script commands,
> > programs, functions and state machines in any order. For example:
> >
> >
> > var int counter = 0; // counter is added to the attributes of the
> > scripting service (global in this service).
> > 
> > // foo program becomes a new service in the component
> > program foo() {
> > 
> >   var int foo_counter =1;
> >   while (true) {
> >   
> >      foo_counter = foo_counter + 1
> >      yield
> >  
> >  }
> > 
> > }
> > 
> > // after foo has been parsed, it is available:
> > foo.foo_counter = 0
> > foo.start()
> > 

> >
> > As you can see, variables declared at the global scope are added to the
> > scripting service's scope (all scripts share this). You can specify a
> > program which is parsed and which can be used in subsequent script
> > statements or programs/state machines defined in this file or in other
> > files.
> > Clearly, any of these elements are optional.
> >
> > Finally, I propose some slight (breaking) modifications to the
> > DeploymentComponent API:
> > - connectPorts: as it is now, it connects equally named ports.
> >
> > -> This really has to go. It's very bad practice.
> >
> > - connectTwoPorts: does not take a connection policy as argument
> >
> > -> would go too and be replaced by:
> > + connect("comp1.portA","comp2.portB",ConnPolicy)
> >
> > -> connects two ports (possibly in services) with a policy
> >
> > + createStream("comp1.portC", ConnPolicy)
> >
> > -> setup a stream from a port (in a service) with a policy
> >
> > Any other suggestions or objections welcome.
>
> The above is not very clear in what changes you intend for user
> applications. Can you give a more concrete example (pseudo code, not
> details) showing a v1 XML, and what you intend the v2 version to look
> like. For instance, the program script you show above looks specific to
> just one component. How is that helping or hindering large scale or
> complex deployments?

I need some time to work that out, but I understand the question...

>
> Can you also list the specific problems in XML that you are trying to
> address. All I see above is "XML bad, scripting better".

What we can do in scripts and not (easily) in xml during deployment:
- Use data/properties in other components to make deployment decisions
- Group start/stop/configure components using a custom state machine
- connect ports/services that were dynamically created during configureHook()
- conditionally execute deployment steps
- go step-by-step through a deployment process (using pause()/step() api)
- Specify easily connection policies etc.
- have a very readable/sequential deployment scenario
- have deployment react to events, changing situations etc.

>
> > I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
> > too. I expect 2.2.0 to be around by the end of october. There is no
> > schedule yet for a 1.14.
>
> Do we have roadmap somewhere for what is going in these future releases?

We can prepare one on the wiki.

Peter

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Peter Soetens wrote:

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to give your feedback. These features will be developed after 2.1
>>> is released.
>>
>> Add the ability to deploy components in groups, so that group A is deployed
>> completely, followed by group B, ..., and then group B is un-deployed
>> completely, and finally group A is un-deployed completely. This is
>> necessary for things like logging, which currently hacks this by using the
>> site-file approach. This is really just sequencing application startup and
>> shutdown.
>
> You *can* already do this using scripting and a state machine.That's my whole
> point. No XML / DeploymentComponent extensions are necessary for this.

But there is something more: in many deployment scenarios you _need_ more
than a static description of how components have to be interconnected, and
the deployment, start-up and shut-down sequences require _behaviour_, as
provided by state machines.

>>> Most users are not happy with the XML format that specifies which
>>> components are loaded and connected in an application. Especially the
>>> flexibility and the fixed ordering of configuration/setup steps disturbs
>>> even modest applications. The scripting interface on the other hand
>>> provides a more controlled, readable and flexible environment to specify
>>> how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps. For example: importing
>>> component libraries or plugins and instantiating components. Everything
>>> after that should be done in a script.
>>
>> What exactly are you talking about changing here? In particular, what are
>> you talking about removing? I would really like it if nothing that
>> currently works was removed - we already have a high enough barrier
>> getting from v1 to v2 due to "removed features" that we use. It would be
>> nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
>>
>> The only issue we have with XML is that large-scale deployments are tricky
>> due to difficulty in configuring the deployment. The recent discussion on
>> use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.
>

This is the same discussion as we have within BRICS, about how to best
separate the structural and behavioural parts, more specifically of
deployment int his case. To support behaviour, one needs FSM functionality.
But both structural and behavioural parts can be described in XML form. The
concrete RTT problem is that we have no XML tags/DTD yet for FSMs.

It's a valid and timely question to ask now: what makes more sense,
extending our RTT-specific XML/scripting or adopting an existing deployment
"language" (such as OSGi, which uses a more readable file format than XML,
but that's nothing more than "just" a user-interfacing detail).

>>> For this to work as painless as possible, we need to relook at the
>>> scripting API of the DeploymentComponent and how XML and scripting
>>> integrate better. These features would be 'primary' for the XML:
>>>
>>> * Loading plugins, component libraries etc. -- 'Import'
>>> * Instantiating components -- component struct
>>> * Loading a script (which contains programs andor state machines) --
>>> 'LoadScript'
>>> * Starting a specific program or state machine 'StartScript'.
>>
>> Can you please list here what existing features you intend to remove from
>> the XML approach?
>
> None.
>
Indeed: it serves the structural parts already well. It's the behavioural
part that has to be introduced. With new RTT-XML, or with RTT-scripting, or
with an external deployment framework. I am in favour of the latter, if a
almost perfect one can be found.

>>> (the old XML features would remain as-is but are not recommended)
>>
>> So are you not removing XML features, but deprecating them with an intent
>> to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'.

"The right direction" = more strict decoupling between the 4Cs...!

> I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).
>
>>
>>> This would result in this deployment file which 90% of the applications
>>> uses:

>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!DOCTYPE properties SYSTEM "cpf.dtd">
>>> <properties>
>>> <!-- set import paths -->
>>> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>>>
>>> <!-- Load startup script in Deployer -->
>>> <struct name="Deployer" type="OCL::DeploymentComponent">
>>>
>>>  <!-- The name of a file: -->
>>>  <simple name="LoadScript"
>>>  type="string"><value>startup.ops</value></simple> <!-- The name of a
>>>  program or state machine: -->
>>>  <simple name="StartScript"
>>>  type="string"><value>BarProgram</value></simple>
>>>
>>> </struct>
>>> </properties>
>>> 

>>>
>>> A Lua-like variant for loading scripts would be very similar, also since
>>> lua will become a part of OCL.
>>>
>>> The idea is then that startup.ops contains 'top level' script commands,
>>> programs, functions and state machines in any order. For example:
>>>
>>>
>>> var int counter = 0; // counter is added to the attributes of the
>>> scripting service (global in this service).
>>>
>>> // foo program becomes a new service in the component
>>> program foo() {
>>>
>>>   var int foo_counter =1;
>>>   while (true) {
>>>
>>>      foo_counter = foo_counter + 1
>>>      yield
>>>
>>>  }
>>>
>>> }
>>>
>>> // after foo has been parsed, it is available:
>>> foo.foo_counter = 0
>>> foo.start()
>>> 

>>>
>>> As you can see, variables declared at the global scope are added to the
>>> scripting service's scope (all scripts share this). You can specify a
>>> program which is parsed and which can be used in subsequent script
>>> statements or programs/state machines defined in this file or in other
>>> files.
>>> Clearly, any of these elements are optional.
>>>
>>> Finally, I propose some slight (breaking) modifications to the
>>> DeploymentComponent API:
>>> - connectPorts: as it is now, it connects equally named ports.
>>>
>>> -> This really has to go. It's very bad practice.

Can you specify in detail why it is bad practice? (Good to have for our
BRICS project! :-))

>>> - connectTwoPorts: does not take a connection policy as argument
>>>
>>> -> would go too and be replaced by:
>>> + connect("comp1.portA","comp2.portB",ConnPolicy)
>>>
>>> -> connects two ports (possibly in services) with a policy
>>>
>>> + createStream("comp1.portC", ConnPolicy)
>>>
>>> -> setup a stream from a port (in a service) with a policy
>>>
>>> Any other suggestions or objections welcome.

These suggestions follow the _procedural_ approach (= you say which actions
have to be taken); the traditional alternative is the _declarative_
approach (= you say which deployment constraints have to be satisfied, and
there is a 'deployment run-time' available in the system that creates a
concrete set of deployment actions from the declarative information.

Declarative approaches have one big advantage: they are the most
composable! Their big disadvantages are: (i) need for a runtime, and (ii)
concrete sequence of actions can become difficult to predict. I have no
clear suggestion about which trade-off to choose here. Principally
speaking, I prefer the declarative approach, because of its composability.

>> The above is not very clear in what changes you intend for user
>> applications. Can you give a more concrete example (pseudo code, not
>> details) showing a v1 XML, and what you intend the v2 version to look
>> like. For instance, the program script you show above looks specific to
>> just one component. How is that helping or hindering large scale or
>> complex deployments?
>
> I need some time to work that out, but I understand the question...
>
>> Can you also list the specific problems in XML that you are trying to
>> address. All I see above is "XML bad, scripting better".

> What we can do in scripts and not (easily) in xml during deployment:
> - Use data/properties in other components to make deployment decisions

That's an example of a "runtime" requirement. Very valid use case!

> - Group start/stop/configure components using a custom state machine
Same remark.

> - connect ports/services that were dynamically created during configureHook()
Same remark.

> - conditionally execute deployment steps
Same remark.

> - go step-by-step through a deployment process (using pause()/step() api)
Same remark.

> - Specify easily connection policies etc.
This is the use case of separating the _Configuration_ of the Coordination,
from the _Computation_ of the Coordination to be done at runtime.
Very valid use case.

> - have a very readable/sequential deployment scenario
"readability" is a not so objective metric...

> - have deployment react to events, changing situations etc.
Again the runtime requirement remark.

Overall, I very much support these suggestions. But let's first look at
other frameworks where this has already been done successfully. I could ask
one of the BRICS PhD students to try to look into this matter...

>>> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
>>> too. I expect 2.2.0 to be around by the end of october. There is no
>>> schedule yet for a 1.14.
>>
>> Do we have roadmap somewhere for what is going in these future releases?
>
> We can prepare one on the wiki.
>
> Peter

Herman

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Peter Soetens wrote:

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to give your feedback. These features will be developed after 2.1
>>> is released.
>>
>> Add the ability to deploy components in groups, so that group A is deployed
>> completely, followed by group B, ..., and then group B is un-deployed
>> completely, and finally group A is un-deployed completely. This is
>> necessary for things like logging, which currently hacks this by using the
>> site-file approach. This is really just sequencing application startup and
>> shutdown.
>
> You *can* already do this using scripting and a state machine.That's my whole
> point. No XML / DeploymentComponent extensions are necessary for this.

But there is something more: in many deployment scenarios you _need_ more
than a static description of how components have to be interconnected, and
the deployment, start-up and shut-down sequences require _behaviour_, as
provided by state machines.

>>> Most users are not happy with the XML format that specifies which
>>> components are loaded and connected in an application. Especially the
>>> flexibility and the fixed ordering of configuration/setup steps disturbs
>>> even modest applications. The scripting interface on the other hand
>>> provides a more controlled, readable and flexible environment to specify
>>> how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps. For example: importing
>>> component libraries or plugins and instantiating components. Everything
>>> after that should be done in a script.
>>
>> What exactly are you talking about changing here? In particular, what are
>> you talking about removing? I would really like it if nothing that
>> currently works was removed - we already have a high enough barrier
>> getting from v1 to v2 due to "removed features" that we use. It would be
>> nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
>>
>> The only issue we have with XML is that large-scale deployments are tricky
>> due to difficulty in configuring the deployment. The recent discussion on
>> use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.
>

This is the same discussion as we have within BRICS, about how to best
separate the structural and behavioural parts, more specifically of
deployment int his case. To support behaviour, one needs FSM functionality.
But both structural and behavioural parts can be described in XML form. The
concrete RTT problem is that we have no XML tags/DTD yet for FSMs.

It's a valid and timely question to ask now: what makes more sense,
extending our RTT-specific XML/scripting or adopting an existing deployment
"language" (such as OSGi, which uses a more readable file format than XML,
but that's nothing more than "just" a user-interfacing detail).

>>> For this to work as painless as possible, we need to relook at the
>>> scripting API of the DeploymentComponent and how XML and scripting
>>> integrate better. These features would be 'primary' for the XML:
>>>
>>> * Loading plugins, component libraries etc. -- 'Import'
>>> * Instantiating components -- component struct
>>> * Loading a script (which contains programs andor state machines) --
>>> 'LoadScript'
>>> * Starting a specific program or state machine 'StartScript'.
>>
>> Can you please list here what existing features you intend to remove from
>> the XML approach?
>
> None.
>
Indeed: it serves the structural parts already well. It's the behavioural
part that has to be introduced. With new RTT-XML, or with RTT-scripting, or
with an external deployment framework. I am in favour of the latter, if a
almost perfect one can be found.

>>> (the old XML features would remain as-is but are not recommended)
>>
>> So are you not removing XML features, but deprecating them with an intent
>> to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'.

"The right direction" = more strict decoupling between the 4Cs...!

> I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).
>
>>
>>> This would result in this deployment file which 90% of the applications
>>> uses:

>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!DOCTYPE properties SYSTEM "cpf.dtd">
>>> <properties>
>>> <!-- set import paths -->
>>> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>>>
>>> <!-- Load startup script in Deployer -->
>>> <struct name="Deployer" type="OCL::DeploymentComponent">
>>>
>>>  <!-- The name of a file: -->
>>>  <simple name="LoadScript"
>>>  type="string"><value>startup.ops</value></simple> <!-- The name of a
>>>  program or state machine: -->
>>>  <simple name="StartScript"
>>>  type="string"><value>BarProgram</value></simple>
>>>
>>> </struct>
>>> </properties>
>>> 

>>>
>>> A Lua-like variant for loading scripts would be very similar, also since
>>> lua will become a part of OCL.
>>>
>>> The idea is then that startup.ops contains 'top level' script commands,
>>> programs, functions and state machines in any order. For example:
>>>
>>>
>>> var int counter = 0; // counter is added to the attributes of the
>>> scripting service (global in this service).
>>>
>>> // foo program becomes a new service in the component
>>> program foo() {
>>>
>>>   var int foo_counter =1;
>>>   while (true) {
>>>
>>>      foo_counter = foo_counter + 1
>>>      yield
>>>
>>>  }
>>>
>>> }
>>>
>>> // after foo has been parsed, it is available:
>>> foo.foo_counter = 0
>>> foo.start()
>>> 

>>>
>>> As you can see, variables declared at the global scope are added to the
>>> scripting service's scope (all scripts share this). You can specify a
>>> program which is parsed and which can be used in subsequent script
>>> statements or programs/state machines defined in this file or in other
>>> files.
>>> Clearly, any of these elements are optional.
>>>
>>> Finally, I propose some slight (breaking) modifications to the
>>> DeploymentComponent API:
>>> - connectPorts: as it is now, it connects equally named ports.
>>>
>>> -> This really has to go. It's very bad practice.

Can you specify in detail why it is bad practice? (Good to have for our
BRICS project! :-))

>>> - connectTwoPorts: does not take a connection policy as argument
>>>
>>> -> would go too and be replaced by:
>>> + connect("comp1.portA","comp2.portB",ConnPolicy)
>>>
>>> -> connects two ports (possibly in services) with a policy
>>>
>>> + createStream("comp1.portC", ConnPolicy)
>>>
>>> -> setup a stream from a port (in a service) with a policy
>>>
>>> Any other suggestions or objections welcome.

These suggestions follow the _procedural_ approach (= you say which actions
have to be taken); the traditional alternative is the _declarative_
approach (= you say which deployment constraints have to be satisfied, and
there is a 'deployment run-time' available in the system that creates a
concrete set of deployment actions from the declarative information.

Declarative approaches have one big advantage: they are the most
composable! Their big disadvantages are: (i) need for a runtime, and (ii)
concrete sequence of actions can become difficult to predict. I have no
clear suggestion about which trade-off to choose here. Principally
speaking, I prefer the declarative approach, because of its composability.

>> The above is not very clear in what changes you intend for user
>> applications. Can you give a more concrete example (pseudo code, not
>> details) showing a v1 XML, and what you intend the v2 version to look
>> like. For instance, the program script you show above looks specific to
>> just one component. How is that helping or hindering large scale or
>> complex deployments?
>
> I need some time to work that out, but I understand the question...
>
>> Can you also list the specific problems in XML that you are trying to
>> address. All I see above is "XML bad, scripting better".

> What we can do in scripts and not (easily) in xml during deployment:
> - Use data/properties in other components to make deployment decisions

That's an example of a "runtime" requirement. Very valid use case!

> - Group start/stop/configure components using a custom state machine
Same remark.

> - connect ports/services that were dynamically created during configureHook()
Same remark.

> - conditionally execute deployment steps
Same remark.

> - go step-by-step through a deployment process (using pause()/step() api)
Same remark.

> - Specify easily connection policies etc.
This is the use case of separating the _Configuration_ of the Coordination,
from the _Computation_ of the Coordination to be done at runtime.
Very valid use case.

> - have a very readable/sequential deployment scenario
"readability" is a not so objective metric...

> - have deployment react to events, changing situations etc.
Again the runtime requirement remark.

Overall, I very much support these suggestions. But let's first look at
other frameworks where this has already been done successfully. I could ask
one of the BRICS PhD students to try to look into this matter...

>>> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
>>> too. I expect 2.2.0 to be around by the end of october. There is no
>>> schedule yet for a 1.14.
>>
>> Do we have roadmap somewhere for what is going in these future releases?
>
> We can prepare one on the wiki.
>
> Peter

Herman

[RFC] Component Deployment and scripting in 2.2

On Thu, Sep 30, 2010 at 3:44 PM, Herman Bruyninckx <
Herman [dot] Bruyninckx [..] ...> wrote:

> On Thu, 30 Sep 2010, Peter Soetens wrote:
>
> > On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>> We're looking at improving deployment of components, and now's a good
> >>> time to give your feedback. These features will be developed after 2.1
> >>> is released.
> >>
> >> Add the ability to deploy components in groups, so that group A is
> deployed
> >> completely, followed by group B, ..., and then group B is un-deployed
> >> completely, and finally group A is un-deployed completely. This is
> >> necessary for things like logging, which currently hacks this by using
> the
> >> site-file approach. This is really just sequencing application startup
> and
> >> shutdown.
> >
> > You *can* already do this using scripting and a state machine.That's my
> whole
> > point. No XML / DeploymentComponent extensions are necessary for this.
>
> But there is something more: in many deployment scenarios you _need_ more
> than a static description of how components have to be interconnected, and
> the deployment, start-up and shut-down sequences require _behaviour_, as
> provided by state machines.
>

Where does "deployment" stop and "coordination" start? Or how are these
related?

k

[RFC] Component Deployment and scripting in 2.2

On Thu, Sep 30, 2010 at 3:44 PM, Herman Bruyninckx <
Herman [dot] Bruyninckx [..] ...> wrote:

> On Thu, 30 Sep 2010, Peter Soetens wrote:
>
> > On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>> We're looking at improving deployment of components, and now's a good
> >>> time to give your feedback. These features will be developed after 2.1
> >>> is released.
> >>
> >> Add the ability to deploy components in groups, so that group A is
> deployed
> >> completely, followed by group B, ..., and then group B is un-deployed
> >> completely, and finally group A is un-deployed completely. This is
> >> necessary for things like logging, which currently hacks this by using
> the
> >> site-file approach. This is really just sequencing application startup
> and
> >> shutdown.
> >
> > You *can* already do this using scripting and a state machine.That's my
> whole
> > point. No XML / DeploymentComponent extensions are necessary for this.
>
> But there is something more: in many deployment scenarios you _need_ more
> than a static description of how components have to be interconnected, and
> the deployment, start-up and shut-down sequences require _behaviour_, as
> provided by state machines.
>

Where does "deployment" stop and "coordination" start? Or how are these
related?

k

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Klaas Gadeyne wrote:

> On Thu, Sep 30, 2010 at 3:44 PM, Herman Bruyninckx
> <Herman [dot] Bruyninckx [..] ...> wrote:
> On Thu, 30 Sep 2010, Peter Soetens wrote:
>
> > On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>> We're looking at improving deployment of components, and
> now's a good
> >>> time to give your feedback. These features will be developed
> after 2.1
> >>> is released.
> >>
> >> Add the ability to deploy components in groups, so that group
> A is deployed
> >> completely, followed by group B, ..., and then group B is
> un-deployed
> >> completely, and finally group A is un-deployed completely.
> This is
> >> necessary for things like logging, which currently hacks this
> by using the
> >> site-file approach. This is really just sequencing
> application startup and
> >> shutdown.
> >
> > You *can* already do this using scripting and a state
> machine.That's my whole
> > point. No XML / DeploymentComponent extensions are necessary
> for this.
>
> But there is something more: in many deployment scenarios you
> _need_ more
> than a static description of how components have to be
> interconnected, and
> the deployment, start-up and shut-down sequences require
> _behaviour_, as
> provided by state machines.
>
>
> Where does "deployment" stop and "coordination" start?  Or how are these
> related?

They are related in the following way: all but the most simple deployment
scenarios need a "behaviour" (FSM) to coordinate all the deployment actions
that have to be done before your system is up and running.

In other words, RTT has until now been mostly limited to the static,
structural aspects of deployment. And that has shown its limitations, many
times already.

Deployment needs to be a full fledged component in itself, at runtime.

Herman

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Klaas Gadeyne wrote:

> On Thu, Sep 30, 2010 at 3:44 PM, Herman Bruyninckx
> <Herman [dot] Bruyninckx [..] ...> wrote:
> On Thu, 30 Sep 2010, Peter Soetens wrote:
>
> > On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>> We're looking at improving deployment of components, and
> now's a good
> >>> time to give your feedback. These features will be developed
> after 2.1
> >>> is released.
> >>
> >> Add the ability to deploy components in groups, so that group
> A is deployed
> >> completely, followed by group B, ..., and then group B is
> un-deployed
> >> completely, and finally group A is un-deployed completely.
> This is
> >> necessary for things like logging, which currently hacks this
> by using the
> >> site-file approach. This is really just sequencing
> application startup and
> >> shutdown.
> >
> > You *can* already do this using scripting and a state
> machine.That's my whole
> > point. No XML / DeploymentComponent extensions are necessary
> for this.
>
> But there is something more: in many deployment scenarios you
> _need_ more
> than a static description of how components have to be
> interconnected, and
> the deployment, start-up and shut-down sequences require
> _behaviour_, as
> provided by state machines.
>
>
> Where does "deployment" stop and "coordination" start?  Or how are these
> related?

They are related in the following way: all but the most simple deployment
scenarios need a "behaviour" (FSM) to coordinate all the deployment actions
that have to be done before your system is up and running.

In other words, RTT has until now been mostly limited to the static,
structural aspects of deployment. And that has shown its limitations, many
times already.

Deployment needs to be a full fledged component in itself, at runtime.

Herman

[RFC] Component Deployment and scripting in 2.2

On Thu, Sep 30, 2010 at 3:58 PM, Herman Bruyninckx <
Herman [dot] Bruyninckx [..] ...> wrote:

> On Thu, 30 Sep 2010, Klaas Gadeyne wrote:
>
> On Thu, Sep 30, 2010 at 3:44 PM, Herman Bruyninckx
>> <Herman [dot] Bruyninckx [..] ...> wrote:
>> On Thu, 30 Sep 2010, Peter Soetens wrote:
>>
>> > On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>> >> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>> >>> We're looking at improving deployment of components, and
>> now's a good
>> >>> time to give your feedback. These features will be developed
>> after 2.1
>> >>> is released.
>> >>
>> >> Add the ability to deploy components in groups, so that group
>> A is deployed
>> >> completely, followed by group B, ..., and then group B is
>> un-deployed
>> >> completely, and finally group A is un-deployed completely.
>> This is
>> >> necessary for things like logging, which currently hacks this
>> by using the
>> >> site-file approach. This is really just sequencing
>> application startup and
>> >> shutdown.
>> >
>> > You *can* already do this using scripting and a state
>> machine.That's my whole
>> > point. No XML / DeploymentComponent extensions are necessary
>> for this.
>>
>> But there is something more: in many deployment scenarios you
>> _need_ more
>> than a static description of how components have to be
>> interconnected, and
>> the deployment, start-up and shut-down sequences require
>> _behaviour_, as
>> provided by state machines.
>>
>>
>> Where does "deployment" stop and "coordination" start? Or how are these
>> related?
>>
>
> They are related in the following way: all but the most simple deployment
> scenarios need a "behaviour" (FSM) to coordinate all the deployment actions
> that have to be done before your system is up and running.
>
> In other words, RTT has until now been mostly limited to the static,
> structural aspects of deployment. And that has shown its limitations, many
> times already.
>
> Deployment needs to be a full fledged component in itself, at runtime.
>
>
Ok, now we're getting somewhere ;-)
With your above "definition", I also can agree that xml might (currently)
not be the format of choice for describing full deployment scenario's. Not
that it would not be possible (the whole UML spec, including FSMs
descriptions, are already serialized and standardised into xmi), but adding
and defining the "dynamic" deployment capabilities on the short term might
require too much effort compared to what's available with the (orocos)
FSM/scripting languages.

Maybe, if some of the RFC emails started with a good set of definitions,
there would be no need for discussion ;-)

Klaas

[RFC] Component Deployment and scripting in 2.2

On Thu, Sep 30, 2010 at 3:58 PM, Herman Bruyninckx <
Herman [dot] Bruyninckx [..] ...> wrote:

> On Thu, 30 Sep 2010, Klaas Gadeyne wrote:
>
> On Thu, Sep 30, 2010 at 3:44 PM, Herman Bruyninckx
>> <Herman [dot] Bruyninckx [..] ...> wrote:
>> On Thu, 30 Sep 2010, Peter Soetens wrote:
>>
>> > On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>> >> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>> >>> We're looking at improving deployment of components, and
>> now's a good
>> >>> time to give your feedback. These features will be developed
>> after 2.1
>> >>> is released.
>> >>
>> >> Add the ability to deploy components in groups, so that group
>> A is deployed
>> >> completely, followed by group B, ..., and then group B is
>> un-deployed
>> >> completely, and finally group A is un-deployed completely.
>> This is
>> >> necessary for things like logging, which currently hacks this
>> by using the
>> >> site-file approach. This is really just sequencing
>> application startup and
>> >> shutdown.
>> >
>> > You *can* already do this using scripting and a state
>> machine.That's my whole
>> > point. No XML / DeploymentComponent extensions are necessary
>> for this.
>>
>> But there is something more: in many deployment scenarios you
>> _need_ more
>> than a static description of how components have to be
>> interconnected, and
>> the deployment, start-up and shut-down sequences require
>> _behaviour_, as
>> provided by state machines.
>>
>>
>> Where does "deployment" stop and "coordination" start? Or how are these
>> related?
>>
>
> They are related in the following way: all but the most simple deployment
> scenarios need a "behaviour" (FSM) to coordinate all the deployment actions
> that have to be done before your system is up and running.
>
> In other words, RTT has until now been mostly limited to the static,
> structural aspects of deployment. And that has shown its limitations, many
> times already.
>
> Deployment needs to be a full fledged component in itself, at runtime.
>
>
Ok, now we're getting somewhere ;-)
With your above "definition", I also can agree that xml might (currently)
not be the format of choice for describing full deployment scenario's. Not
that it would not be possible (the whole UML spec, including FSMs
descriptions, are already serialized and standardised into xmi), but adding
and defining the "dynamic" deployment capabilities on the short term might
require too much effort compared to what's available with the (orocos)
FSM/scripting languages.

Maybe, if some of the RFC emails started with a good set of definitions,
there would be no need for discussion ;-)

Klaas

[RFC] Component Deployment and scripting in 2.2

On Thu, Sep 30, 2010 at 4:40 PM, Klaas Gadeyne <klaas [dot] gadeyne [..] ...> wrote:
[...]
> Ok, now we're getting somewhere ;-)
> With your above "definition", I also can agree that xml might (currently) not be the format of choice for describing full deployment scenario's.  Not that it would not be possible (the whole UML spec, including FSMs descriptions, are already serialized and standardised into xmi), but adding and defining the "dynamic" deployment capabilities on the short term might require too much effort compared to what's available with the (orocos) FSM/scripting languages.

That said, using 2 different "files" (cpf + osd) instead of a single
model for deployment might lead to conflicts too...

k
--
Orocos-Users mailing list
Orocos-Users [..] ...
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-users

[RFC] Component Deployment and scripting in 2.2

On Thu, Sep 30, 2010 at 4:40 PM, Klaas Gadeyne <klaas [dot] gadeyne [..] ...> wrote:
[...]
> Ok, now we're getting somewhere ;-)
> With your above "definition", I also can agree that xml might (currently) not be the format of choice for describing full deployment scenario's.  Not that it would not be possible (the whole UML spec, including FSMs descriptions, are already serialized and standardised into xmi), but adding and defining the "dynamic" deployment capabilities on the short term might require too much effort compared to what's available with the (orocos) FSM/scripting languages.

That said, using 2 different "files" (cpf + osd) instead of a single
model for deployment might lead to conflicts too...

k
--
Orocos-Dev mailing list
Orocos-Dev [..] ...
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev

Ruben Smits's picture

[RFC] Component Deployment and scripting in 2.2

On Thursday 30 September 2010 16:43:36 Klaas Gadeyne wrote:
> On Thu, Sep 30, 2010 at 4:40 PM, Klaas Gadeyne <klaas [dot] gadeyne [..] ...>
> wrote: [...]
>
> > Ok, now we're getting somewhere ;-)
> > With your above "definition", I also can agree that xml might (currently)
> > not be the format of choice for describing full deployment scenario's.
> > Not that it would not be possible (the whole UML spec, including FSMs
> > descriptions, are already serialized and standardised into xmi), but
> > adding and defining the "dynamic" deployment capabilities on the short
> > term might require too much effort compared to what's available with the
> > (orocos) FSM/scripting languages.
>
> That said, using 2 different "files" (cpf + osd) instead of a single
> model for deployment might lead to conflicts too...

Well, AFAIK you can do everything with only scripting, and not use xml/cpf for
deployment at all.

Ruben

Ruben Smits's picture

[RFC] Component Deployment and scripting in 2.2

On Thursday 30 September 2010 16:43:36 Klaas Gadeyne wrote:
> On Thu, Sep 30, 2010 at 4:40 PM, Klaas Gadeyne <klaas [dot] gadeyne [..] ...>
> wrote: [...]
>
> > Ok, now we're getting somewhere ;-)
> > With your above "definition", I also can agree that xml might (currently)
> > not be the format of choice for describing full deployment scenario's.
> > Not that it would not be possible (the whole UML spec, including FSMs
> > descriptions, are already serialized and standardised into xmi), but
> > adding and defining the "dynamic" deployment capabilities on the short
> > term might require too much effort compared to what's available with the
> > (orocos) FSM/scripting languages.
>
> That said, using 2 different "files" (cpf + osd) instead of a single
> model for deployment might lead to conflicts too...

Well, AFAIK you can do everything with only scripting, and not use xml/cpf for
deployment at all.

Ruben

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Ruben Smits wrote:

> On Thursday 30 September 2010 16:43:36 Klaas Gadeyne wrote:
>> On Thu, Sep 30, 2010 at 4:40 PM, Klaas Gadeyne <klaas [dot] gadeyne [..] ...>
>> wrote: [...]
>>
>>> Ok, now we're getting somewhere ;-)
>>> With your above "definition", I also can agree that xml might (currently)
>>> not be the format of choice for describing full deployment scenario's.
>>> Not that it would not be possible (the whole UML spec, including FSMs
>>> descriptions, are already serialized and standardised into xmi), but
>>> adding and defining the "dynamic" deployment capabilities on the short
>>> term might require too much effort compared to what's available with the
>>> (orocos) FSM/scripting languages.
>>
>> That said, using 2 different "files" (cpf + osd) instead of a single
>> model for deployment might lead to conflicts too...
>
> Well, AFAIK you can do everything with only scripting, and not use xml/cpf for
> deployment at all.
>
> Ruben

The argument that "you _can_ do it" in one way or another is seldom a
_good_ argument :-) There is one important reason why I would like to see
everything ending up in "XML" files: this would make the integration in one
single tool chain much easier. And I do not see where having multiple XML
files _in itself_ leads to conflicts... We _will_ have a need for different
XML files, because I want to separate all of the 4Cs, as well as different
steps in the "development process" of a whole system.

Note that "XML" might be replaced by any formal representation that a
decent toolchain ecosystem (a la Eclipse) can deal with.

Herman

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Ruben Smits wrote:

> On Thursday 30 September 2010 16:43:36 Klaas Gadeyne wrote:
>> On Thu, Sep 30, 2010 at 4:40 PM, Klaas Gadeyne <klaas [dot] gadeyne [..] ...>
>> wrote: [...]
>>
>>> Ok, now we're getting somewhere ;-)
>>> With your above "definition", I also can agree that xml might (currently)
>>> not be the format of choice for describing full deployment scenario's.
>>> Not that it would not be possible (the whole UML spec, including FSMs
>>> descriptions, are already serialized and standardised into xmi), but
>>> adding and defining the "dynamic" deployment capabilities on the short
>>> term might require too much effort compared to what's available with the
>>> (orocos) FSM/scripting languages.
>>
>> That said, using 2 different "files" (cpf + osd) instead of a single
>> model for deployment might lead to conflicts too...
>
> Well, AFAIK you can do everything with only scripting, and not use xml/cpf for
> deployment at all.
>
> Ruben

The argument that "you _can_ do it" in one way or another is seldom a
_good_ argument :-) There is one important reason why I would like to see
everything ending up in "XML" files: this would make the integration in one
single tool chain much easier. And I do not see where having multiple XML
files _in itself_ leads to conflicts... We _will_ have a need for different
XML files, because I want to separate all of the 4Cs, as well as different
steps in the "development process" of a whole system.

Note that "XML" might be replaced by any formal representation that a
decent toolchain ecosystem (a la Eclipse) can deal with.

Herman

[RFC] Component Deployment and scripting in 2.2

If I can from my little Orocos experience, XML is not designed to be
read/edit by humans. Even seperated in small parts it's boring to
write/edit/maintain. XML whould be appropriate if a GUI-tool was available
to create the deployment files.

My personnal point of view :
script deployment is more close to your [human] way of thinking when your
are thinking about deployment, whereas XML needs "mind reorganisations"
which is tiring and leads to more bugs.

2010/9/30 Peter Soetens <peter [..] ...>

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> > On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> > > We're looking at improving deployment of components, and now's a good
> > > time to give your feedback. These features will be developed after 2.1
> > > is released.
> >
> > Add the ability to deploy components in groups, so that group A is
> deployed
> > completely, followed by group B, ..., and then group B is un-deployed
> > completely, and finally group A is un-deployed completely. This is
> > necessary for things like logging, which currently hacks this by using
> the
> > site-file approach. This is really just sequencing application startup
> and
> > shutdown.
>
> You *can* already do this using scripting and a state machine.That's my
> whole
> point. No XML / DeploymentComponent extensions are necessary for this.
>
> >
> > > Most users are not happy with the XML format that specifies which
> > > components are loaded and connected in an application. Especially the
> > > flexibility and the fixed ordering of configuration/setup steps
> disturbs
> > > even modest applications. The scripting interface on the other hand
> > > provides a more controlled, readable and flexible environment to
> specify
> > > how an application must be setup.
> > >
> > > Therefor, the 'best practice' in deployment will emphasis doing things
> in
> > > scripts and leave XML for basic setup steps. For example: importing
> > > component libraries or plugins and instantiating components. Everything
> > > after that should be done in a script.
> >
> > What exactly are you talking about changing here? In particular, what are
> > you talking about removing? I would really like it if nothing that
> > currently works was removed - we already have a high enough barrier
> > getting from v1 to v2 due to "removed features" that we use. It would be
> > nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
> >
> > The only issue we have with XML is that large-scale deployments are
> tricky
> > due to difficulty in configuring the deployment. The recent discussion on
> > use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.
>
> >
> > > For this to work as painless as possible, we need to relook at the
> > > scripting API of the DeploymentComponent and how XML and scripting
> > > integrate better. These features would be 'primary' for the XML:
> > >
> > > * Loading plugins, component libraries etc. -- 'Import'
> > > * Instantiating components -- component struct
> > > * Loading a script (which contains programs andor state machines) --
> > > 'LoadScript'
> > > * Starting a specific program or state machine 'StartScript'.
> >
> > Can you please list here what existing features you intend to remove from
> > the XML approach?
>
> None.
>
> >
> > > (the old XML features would remain as-is but are not recommended)
> >
> > So are you not removing XML features, but deprecating them with an intent
> > to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'. I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).
>
> >
> > > This would result in this deployment file which 90% of the applications
> > > uses:

> > > <?xml version="1.0" encoding="UTF-8"?>
> > > <!DOCTYPE properties SYSTEM "cpf.dtd">
> > > <properties>
> > > <!-- set import paths -->
> > > <simple name="Import"
> type="string"><value>/usr/local/lib/orocos</simple>
> > >
> > > <!-- Load startup script in Deployer -->
> > > <struct name="Deployer" type="OCL::DeploymentComponent">
> > >
> > >  <!-- The name of a file: -->
> > >  <simple name="LoadScript"
> > >  type="string"><value>startup.ops</value></simple> <!-- The name of a
> > >  program or state machine: -->
> > >  <simple name="StartScript"
> > >  type="string"><value>BarProgram</value></simple>
> > >
> > > </struct>
> > > </properties>
> > > 

> > >
> > > A Lua-like variant for loading scripts would be very similar, also
> since
> > > lua will become a part of OCL.
> > >
> > > The idea is then that startup.ops contains 'top level' script commands,
> > > programs, functions and state machines in any order. For example:
> > >
> > >
> > > var int counter = 0; // counter is added to the attributes of the
> > > scripting service (global in this service).
> > >
> > > // foo program becomes a new service in the component
> > > program foo() {
> > >
> > >   var int foo_counter =1;
> > >   while (true) {
> > >
> > >      foo_counter = foo_counter + 1
> > >      yield
> > >
> > >  }
> > >
> > > }
> > >
> > > // after foo has been parsed, it is available:
> > > foo.foo_counter = 0
> > > foo.start()
> > > 

> > >
> > > As you can see, variables declared at the global scope are added to the
> > > scripting service's scope (all scripts share this). You can specify a
> > > program which is parsed and which can be used in subsequent script
> > > statements or programs/state machines defined in this file or in other
> > > files.
> > > Clearly, any of these elements are optional.
> > >
> > > Finally, I propose some slight (breaking) modifications to the
> > > DeploymentComponent API:
> > > - connectPorts: as it is now, it connects equally named ports.
> > >
> > > -> This really has to go. It's very bad practice.
> > >
> > > - connectTwoPorts: does not take a connection policy as argument
> > >
> > > -> would go too and be replaced by:
> > > + connect("comp1.portA","comp2.portB",ConnPolicy)
> > >
> > > -> connects two ports (possibly in services) with a policy
> > >
> > > + createStream("comp1.portC", ConnPolicy)
> > >
> > > -> setup a stream from a port (in a service) with a policy
> > >
> > > Any other suggestions or objections welcome.
> >
> > The above is not very clear in what changes you intend for user
> > applications. Can you give a more concrete example (pseudo code, not
> > details) showing a v1 XML, and what you intend the v2 version to look
> > like. For instance, the program script you show above looks specific to
> > just one component. How is that helping or hindering large scale or
> > complex deployments?
>
> I need some time to work that out, but I understand the question...
>
> >
> > Can you also list the specific problems in XML that you are trying to
> > address. All I see above is "XML bad, scripting better".
>
> What we can do in scripts and not (easily) in xml during deployment:
> - Use data/properties in other components to make deployment decisions
> - Group start/stop/configure components using a custom state machine
> - connect ports/services that were dynamically created during
> configureHook()
> - conditionally execute deployment steps
> - go step-by-step through a deployment process (using pause()/step() api)
> - Specify easily connection policies etc.
> - have a very readable/sequential deployment scenario
> - have deployment react to events, changing situations etc.
>
> >
> > > I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming
> days
> > > too. I expect 2.2.0 to be around by the end of october. There is no
> > > schedule yet for a 1.14.
> >
> > Do we have roadmap somewhere for what is going in these future releases?
>
> We can prepare one on the wiki.
>
> Peter
> --
> Orocos-Users mailing list
> Orocos-Users [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-users
>

[RFC] Component Deployment and scripting in 2.2

If I can from my little Orocos experience, XML is not designed to be
read/edit by humans. Even seperated in small parts it's boring to
write/edit/maintain. XML whould be appropriate if a GUI-tool was available
to create the deployment files.

My personnal point of view :
script deployment is more close to your [human] way of thinking when your
are thinking about deployment, whereas XML needs "mind reorganisations"
which is tiring and leads to more bugs.

2010/9/30 Peter Soetens <peter [..] ...>

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> > On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> > > We're looking at improving deployment of components, and now's a good
> > > time to give your feedback. These features will be developed after 2.1
> > > is released.
> >
> > Add the ability to deploy components in groups, so that group A is
> deployed
> > completely, followed by group B, ..., and then group B is un-deployed
> > completely, and finally group A is un-deployed completely. This is
> > necessary for things like logging, which currently hacks this by using
> the
> > site-file approach. This is really just sequencing application startup
> and
> > shutdown.
>
> You *can* already do this using scripting and a state machine.That's my
> whole
> point. No XML / DeploymentComponent extensions are necessary for this.
>
> >
> > > Most users are not happy with the XML format that specifies which
> > > components are loaded and connected in an application. Especially the
> > > flexibility and the fixed ordering of configuration/setup steps
> disturbs
> > > even modest applications. The scripting interface on the other hand
> > > provides a more controlled, readable and flexible environment to
> specify
> > > how an application must be setup.
> > >
> > > Therefor, the 'best practice' in deployment will emphasis doing things
> in
> > > scripts and leave XML for basic setup steps. For example: importing
> > > component libraries or plugins and instantiating components. Everything
> > > after that should be done in a script.
> >
> > What exactly are you talking about changing here? In particular, what are
> > you talking about removing? I would really like it if nothing that
> > currently works was removed - we already have a high enough barrier
> > getting from v1 to v2 due to "removed features" that we use. It would be
> > nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
> >
> > The only issue we have with XML is that large-scale deployments are
> tricky
> > due to difficulty in configuring the deployment. The recent discussion on
> > use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.
>
> >
> > > For this to work as painless as possible, we need to relook at the
> > > scripting API of the DeploymentComponent and how XML and scripting
> > > integrate better. These features would be 'primary' for the XML:
> > >
> > > * Loading plugins, component libraries etc. -- 'Import'
> > > * Instantiating components -- component struct
> > > * Loading a script (which contains programs andor state machines) --
> > > 'LoadScript'
> > > * Starting a specific program or state machine 'StartScript'.
> >
> > Can you please list here what existing features you intend to remove from
> > the XML approach?
>
> None.
>
> >
> > > (the old XML features would remain as-is but are not recommended)
> >
> > So are you not removing XML features, but deprecating them with an intent
> > to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'. I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).
>
> >
> > > This would result in this deployment file which 90% of the applications
> > > uses:

> > > <?xml version="1.0" encoding="UTF-8"?>
> > > <!DOCTYPE properties SYSTEM "cpf.dtd">
> > > <properties>
> > > <!-- set import paths -->
> > > <simple name="Import"
> type="string"><value>/usr/local/lib/orocos</simple>
> > >
> > > <!-- Load startup script in Deployer -->
> > > <struct name="Deployer" type="OCL::DeploymentComponent">
> > >
> > >  <!-- The name of a file: -->
> > >  <simple name="LoadScript"
> > >  type="string"><value>startup.ops</value></simple> <!-- The name of a
> > >  program or state machine: -->
> > >  <simple name="StartScript"
> > >  type="string"><value>BarProgram</value></simple>
> > >
> > > </struct>
> > > </properties>
> > > 

> > >
> > > A Lua-like variant for loading scripts would be very similar, also
> since
> > > lua will become a part of OCL.
> > >
> > > The idea is then that startup.ops contains 'top level' script commands,
> > > programs, functions and state machines in any order. For example:
> > >
> > >
> > > var int counter = 0; // counter is added to the attributes of the
> > > scripting service (global in this service).
> > >
> > > // foo program becomes a new service in the component
> > > program foo() {
> > >
> > >   var int foo_counter =1;
> > >   while (true) {
> > >
> > >      foo_counter = foo_counter + 1
> > >      yield
> > >
> > >  }
> > >
> > > }
> > >
> > > // after foo has been parsed, it is available:
> > > foo.foo_counter = 0
> > > foo.start()
> > > 

> > >
> > > As you can see, variables declared at the global scope are added to the
> > > scripting service's scope (all scripts share this). You can specify a
> > > program which is parsed and which can be used in subsequent script
> > > statements or programs/state machines defined in this file or in other
> > > files.
> > > Clearly, any of these elements are optional.
> > >
> > > Finally, I propose some slight (breaking) modifications to the
> > > DeploymentComponent API:
> > > - connectPorts: as it is now, it connects equally named ports.
> > >
> > > -> This really has to go. It's very bad practice.
> > >
> > > - connectTwoPorts: does not take a connection policy as argument
> > >
> > > -> would go too and be replaced by:
> > > + connect("comp1.portA","comp2.portB",ConnPolicy)
> > >
> > > -> connects two ports (possibly in services) with a policy
> > >
> > > + createStream("comp1.portC", ConnPolicy)
> > >
> > > -> setup a stream from a port (in a service) with a policy
> > >
> > > Any other suggestions or objections welcome.
> >
> > The above is not very clear in what changes you intend for user
> > applications. Can you give a more concrete example (pseudo code, not
> > details) showing a v1 XML, and what you intend the v2 version to look
> > like. For instance, the program script you show above looks specific to
> > just one component. How is that helping or hindering large scale or
> > complex deployments?
>
> I need some time to work that out, but I understand the question...
>
> >
> > Can you also list the specific problems in XML that you are trying to
> > address. All I see above is "XML bad, scripting better".
>
> What we can do in scripts and not (easily) in xml during deployment:
> - Use data/properties in other components to make deployment decisions
> - Group start/stop/configure components using a custom state machine
> - connect ports/services that were dynamically created during
> configureHook()
> - conditionally execute deployment steps
> - go step-by-step through a deployment process (using pause()/step() api)
> - Specify easily connection policies etc.
> - have a very readable/sequential deployment scenario
> - have deployment react to events, changing situations etc.
>
> >
> > > I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming
> days
> > > too. I expect 2.2.0 to be around by the end of october. There is no
> > > schedule yet for a 1.14.
> >
> > Do we have roadmap somewhere for what is going in these future releases?
>
> We can prepare one on the wiki.
>
> Peter
> --
> Orocos-Users mailing list
> Orocos-Users [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-users
>

[RFC] Component Deployment and scripting in 2.2

[please don't top post]

On Sep 30, 2010, at 08:50 , Willy Lambert wrote:

> If I can from my little Orocos experience, XML is not designed to be read/edit by humans. Even seperated in small parts it's boring to write/edit/maintain. XML whould be appropriate if a GUI-tool was available to create the deployment files.
>
> My personnal point of view :
> script deployment is more close to your [human] way of thinking when your are thinking about deployment, whereas XML needs "mind reorganisations" which is tiring and leads to more bugs.

Agreed, but XML deployment configurations come very naturally from UML component diagrams, which is often how we do it.

It would be nice to not preclude either approach, within reasonable limits.

In all honesty, we use a bare minimum of scripting in our systems. We simply find them too fragile, and we had to many bugs with them early on to be able to trust them. We do however use state machines a great deal - I think of them differently than scripts (which is interesting as they're pretty much the same underneath).

YMMV
S

[RFC] Component Deployment and scripting in 2.2

[please don't top post]

On Sep 30, 2010, at 08:50 , Willy Lambert wrote:

> If I can from my little Orocos experience, XML is not designed to be read/edit by humans. Even seperated in small parts it's boring to write/edit/maintain. XML whould be appropriate if a GUI-tool was available to create the deployment files.
>
> My personnal point of view :
> script deployment is more close to your [human] way of thinking when your are thinking about deployment, whereas XML needs "mind reorganisations" which is tiring and leads to more bugs.

Agreed, but XML deployment configurations come very naturally from UML component diagrams, which is often how we do it.

It would be nice to not preclude either approach, within reasonable limits.

In all honesty, we use a bare minimum of scripting in our systems. We simply find them too fragile, and we had to many bugs with them early on to be able to trust them. We do however use state machines a great deal - I think of them differently than scripts (which is interesting as they're pretty much the same underneath).

YMMV
S

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Stephen Roderick wrote:

> [please don't top post]
>
> On Sep 30, 2010, at 08:50 , Willy Lambert wrote:
>
>> If I can from my little Orocos experience, XML is not designed to be read/edit by humans. Even seperated in small parts it's boring to write/edit/maintain. XML whould be appropriate if a GUI-tool was available to create the deployment files.
>>
>> My personnal point of view :
>> script deployment is more close to your [human] way of thinking when your are thinking about deployment, whereas XML needs "mind reorganisations" which is tiring and leads to more bugs.
>
> Agreed, but XML deployment configurations come very naturally from UML component diagrams, which is often how we do it.
>
> It would be nice to not preclude either approach, within reasonable limits.
>
> In all honesty, we use a bare minimum of scripting in our systems. We simply find them too fragile, and we had to many bugs with them early on to be able to trust them. We do however use state machines a great deal - I think of them differently than scripts (which is interesting as they're pretty much the same underneath).
>
> YMMV

Indeed. But I think I can bring a bit of structure in this discussion:
- scripting can be used to replace the writing of C++ code;
- scripting can also be used to do only the Coordination via a state
machine;
- Stephen's use of scripting for only the FSM use case is a "good
practice", because:
(i) it requires only a small subset of the scripting syntax,
(ii) it separates the (functional) "Computation" from the "Coordination"
(= behavioural Configuration and Computation).
- the first use of scripting is a Pandora box: users will want to see all
of the features available in C++ becoming available in the scripting too.
While this is a feasible path for non-realtime use cases (such as
Blender, where it has been one systematically and successfully), it is
a maintenance and development nightmare in a hard realtime context.
- the concept of "DSL" (Domain Specific Languages) might be the
appropriate trade-off: a specific, simple scripting language for each
application domain, with good support to make such a new DSL. The Eclipse
ecosystem has already quite some support in this direction.
- there is a third important use case for scripting: the specification of
data types at the "mathematical model" level, that is, more abstract
(but still 100% concrete and unambiguous) than a programming
language-specific class representation, but from which the latter can be
_automatically_ derived. Also this case is rather well supported in
Eclipse.

Herman

[RFC] Component Deployment and scripting in 2.2

On Thu, 30 Sep 2010, Stephen Roderick wrote:

> [please don't top post]
>
> On Sep 30, 2010, at 08:50 , Willy Lambert wrote:
>
>> If I can from my little Orocos experience, XML is not designed to be read/edit by humans. Even seperated in small parts it's boring to write/edit/maintain. XML whould be appropriate if a GUI-tool was available to create the deployment files.
>>
>> My personnal point of view :
>> script deployment is more close to your [human] way of thinking when your are thinking about deployment, whereas XML needs "mind reorganisations" which is tiring and leads to more bugs.
>
> Agreed, but XML deployment configurations come very naturally from UML component diagrams, which is often how we do it.
>
> It would be nice to not preclude either approach, within reasonable limits.
>
> In all honesty, we use a bare minimum of scripting in our systems. We simply find them too fragile, and we had to many bugs with them early on to be able to trust them. We do however use state machines a great deal - I think of them differently than scripts (which is interesting as they're pretty much the same underneath).
>
> YMMV

Indeed. But I think I can bring a bit of structure in this discussion:
- scripting can be used to replace the writing of C++ code;
- scripting can also be used to do only the Coordination via a state
machine;
- Stephen's use of scripting for only the FSM use case is a "good
practice", because:
(i) it requires only a small subset of the scripting syntax,
(ii) it separates the (functional) "Computation" from the "Coordination"
(= behavioural Configuration and Computation).
- the first use of scripting is a Pandora box: users will want to see all
of the features available in C++ becoming available in the scripting too.
While this is a feasible path for non-realtime use cases (such as
Blender, where it has been one systematically and successfully), it is
a maintenance and development nightmare in a hard realtime context.
- the concept of "DSL" (Domain Specific Languages) might be the
appropriate trade-off: a specific, simple scripting language for each
application domain, with good support to make such a new DSL. The Eclipse
ecosystem has already quite some support in this direction.
- there is a third important use case for scripting: the specification of
data types at the "mathematical model" level, that is, more abstract
(but still 100% concrete and unambiguous) than a programming
language-specific class representation, but from which the latter can be
_automatically_ derived. Also this case is rather well supported in
Eclipse.

Herman

[RFC] Component Deployment and scripting in 2.2

On Sep 30, 2010, at 08:27 , Peter Soetens wrote:

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to give your feedback. These features will be developed after 2.1
>>> is released.
>>
>> Add the ability to deploy components in groups, so that group A is deployed
>> completely, followed by group B, ..., and then group B is un-deployed
>> completely, and finally group A is un-deployed completely. This is
>> necessary for things like logging, which currently hacks this by using the
>> site-file approach. This is really just sequencing application startup and
>> shutdown.
>
> You *can* already do this using scripting and a state machine.That's my whole
> point. No XML / DeploymentComponent extensions are necessary for this.

Got an example?

>>> Most users are not happy with the XML format that specifies which
>>> components are loaded and connected in an application. Especially the
>>> flexibility and the fixed ordering of configuration/setup steps disturbs
>>> even modest applications. The scripting interface on the other hand
>>> provides a more controlled, readable and flexible environment to specify
>>> how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps. For example: importing
>>> component libraries or plugins and instantiating components. Everything
>>> after that should be done in a script.
>>
>> What exactly are you talking about changing here? In particular, what are
>> you talking about removing? I would really like it if nothing that
>> currently works was removed - we already have a high enough barrier
>> getting from v1 to v2 due to "removed features" that we use. It would be
>> nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
>>
>> The only issue we have with XML is that large-scale deployments are tricky
>> due to difficulty in configuring the deployment. The recent discussion on
>> use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.

Can you give an example here. Besides master/slave ordering, we've not hit any issues like this, so I'd like to understand the problem better.

>>>
>>> For this to work as painless as possible, we need to relook at the
>>> scripting API of the DeploymentComponent and how XML and scripting
>>> integrate better. These features would be 'primary' for the XML:
>>>
>>> * Loading plugins, component libraries etc. -- 'Import'
>>> * Instantiating components -- component struct
>>> * Loading a script (which contains programs andor state machines) --
>>> 'LoadScript'
>>> * Starting a specific program or state machine 'StartScript'.
>>
>> Can you please list here what existing features you intend to remove from
>> the XML approach?
>
> None.
>
>>
>>> (the old XML features would remain as-is but are not recommended)
>>
>> So are you not removing XML features, but deprecating them with an intent
>> to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'. I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).

Read = cutting out any new features from XML users, forcing us to change to this combined XML/scripting approach? I'm ok with this, we all have to evolve, but I would really like some serious discussion on this. We've already gotten badly bitten by v2, I'd like to see such issues coming more clearly.

>>>
>>> This would result in this deployment file which 90% of the applications
>>> uses:

>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!DOCTYPE properties SYSTEM "cpf.dtd">
>>> <properties>
>>> <!-- set import paths -->
>>> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>>> 
>>> <!-- Load startup script in Deployer -->
>>> <struct name="Deployer" type="OCL::DeploymentComponent">
>>> 
>>> <!-- The name of a file: -->
>>> <simple name="LoadScript"
>>> type="string"><value>startup.ops</value></simple> <!-- The name of a
>>> program or state machine: -->
>>> <simple name="StartScript"
>>> type="string"><value>BarProgram</value></simple>
>>> 
>>> </struct>
>>> </properties>
>>> 

>>>
>>> A Lua-like variant for loading scripts would be very similar, also since
>>> lua will become a part of OCL.
>>>
>>> The idea is then that startup.ops contains 'top level' script commands,
>>> programs, functions and state machines in any order. For example:
>>>
>>>
>>> var int counter = 0; // counter is added to the attributes of the
>>> scripting service (global in this service).
>>> 
>>> // foo program becomes a new service in the component
>>> program foo() {
>>> 
>>>  var int foo_counter =1;
>>>  while (true) {
>>> 
>>>     foo_counter = foo_counter + 1
>>>     yield
>>> 
>>> }
>>> 
>>> }
>>> 
>>> // after foo has been parsed, it is available:
>>> foo.foo_counter = 0
>>> foo.start()
>>> 

>>>
>>> As you can see, variables declared at the global scope are added to the
>>> scripting service's scope (all scripts share this). You can specify a
>>> program which is parsed and which can be used in subsequent script
>>> statements or programs/state machines defined in this file or in other
>>> files.
>>> Clearly, any of these elements are optional.
>>>
>>> Finally, I propose some slight (breaking) modifications to the
>>> DeploymentComponent API:
>>> - connectPorts: as it is now, it connects equally named ports.
>>>
>>> -> This really has to go. It's very bad practice.
>>>
>>> - connectTwoPorts: does not take a connection policy as argument
>>>
>>> -> would go too and be replaced by:
>>> + connect("comp1.portA","comp2.portB",ConnPolicy)
>>>
>>> -> connects two ports (possibly in services) with a policy
>>>
>>> + createStream("comp1.portC", ConnPolicy)
>>>
>>> -> setup a stream from a port (in a service) with a policy
>>>
>>> Any other suggestions or objections welcome.
>>
>> The above is not very clear in what changes you intend for user
>> applications. Can you give a more concrete example (pseudo code, not
>> details) showing a v1 XML, and what you intend the v2 version to look
>> like. For instance, the program script you show above looks specific to
>> just one component. How is that helping or hindering large scale or
>> complex deployments?
>
> I need some time to work that out, but I understand the question...

Ack

>> Can you also list the specific problems in XML that you are trying to
>> address. All I see above is "XML bad, scripting better".
>
> What we can do in scripts and not (easily) in xml during deployment:
> - Use data/properties in other components to make deployment decisions

Ack

> - Group start/stop/configure components using a custom state machine

We already do this in SM. What are you getting at?

> - connect ports/services that were dynamically created during configureHook()

Ack

> - conditionally execute deployment steps

Ack

> - go step-by-step through a deployment process (using pause()/step() api)

Debugging, ok

> - Specify easily connection policies etc.

Instead of adding this to XML?

> - have a very readable/sequential deployment scenario

XML isn't?

> - have deployment react to events, changing situations etc.

Can you give more of an example of these two. Not sure what you are really getting at.

>>>
>>> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
>>> too. I expect 2.2.0 to be around by the end of october. There is no
>>> schedule yet for a 1.14.
>>
>> Do we have roadmap somewhere for what is going in these future releases?
>
> We can prepare one on the wiki.

Please do. I would like to have some input ... I'm sure others in the community do also.

I understand your desire to change this, and definitely agree with some of them, but without a complete example it is very hard to see what issues we are trying to solve. Orocos is flexible enough to provide many different ways of doing the same thing, and not all of us hit the same issues as you (and hence don't necessarily understand why you want to change something).
S

[RFC] Component Deployment and scripting in 2.2

On Sep 30, 2010, at 08:27 , Peter Soetens wrote:

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to give your feedback. These features will be developed after 2.1
>>> is released.
>>
>> Add the ability to deploy components in groups, so that group A is deployed
>> completely, followed by group B, ..., and then group B is un-deployed
>> completely, and finally group A is un-deployed completely. This is
>> necessary for things like logging, which currently hacks this by using the
>> site-file approach. This is really just sequencing application startup and
>> shutdown.
>
> You *can* already do this using scripting and a state machine.That's my whole
> point. No XML / DeploymentComponent extensions are necessary for this.

Got an example?

>>> Most users are not happy with the XML format that specifies which
>>> components are loaded and connected in an application. Especially the
>>> flexibility and the fixed ordering of configuration/setup steps disturbs
>>> even modest applications. The scripting interface on the other hand
>>> provides a more controlled, readable and flexible environment to specify
>>> how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps. For example: importing
>>> component libraries or plugins and instantiating components. Everything
>>> after that should be done in a script.
>>
>> What exactly are you talking about changing here? In particular, what are
>> you talking about removing? I would really like it if nothing that
>> currently works was removed - we already have a high enough barrier
>> getting from v1 to v2 due to "removed features" that we use. It would be
>> nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
>>
>> The only issue we have with XML is that large-scale deployments are tricky
>> due to difficulty in configuring the deployment. The recent discussion on
>> use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.

Can you give an example here. Besides master/slave ordering, we've not hit any issues like this, so I'd like to understand the problem better.

>>>
>>> For this to work as painless as possible, we need to relook at the
>>> scripting API of the DeploymentComponent and how XML and scripting
>>> integrate better. These features would be 'primary' for the XML:
>>>
>>> * Loading plugins, component libraries etc. -- 'Import'
>>> * Instantiating components -- component struct
>>> * Loading a script (which contains programs andor state machines) --
>>> 'LoadScript'
>>> * Starting a specific program or state machine 'StartScript'.
>>
>> Can you please list here what existing features you intend to remove from
>> the XML approach?
>
> None.
>
>>
>>> (the old XML features would remain as-is but are not recommended)
>>
>> So are you not removing XML features, but deprecating them with an intent
>> to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'. I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).

Read = cutting out any new features from XML users, forcing us to change to this combined XML/scripting approach? I'm ok with this, we all have to evolve, but I would really like some serious discussion on this. We've already gotten badly bitten by v2, I'd like to see such issues coming more clearly.

>>>
>>> This would result in this deployment file which 90% of the applications
>>> uses:

>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!DOCTYPE properties SYSTEM "cpf.dtd">
>>> <properties>
>>> <!-- set import paths -->
>>> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>>> 
>>> <!-- Load startup script in Deployer -->
>>> <struct name="Deployer" type="OCL::DeploymentComponent">
>>> 
>>> <!-- The name of a file: -->
>>> <simple name="LoadScript"
>>> type="string"><value>startup.ops</value></simple> <!-- The name of a
>>> program or state machine: -->
>>> <simple name="StartScript"
>>> type="string"><value>BarProgram</value></simple>
>>> 
>>> </struct>
>>> </properties>
>>> 

>>>
>>> A Lua-like variant for loading scripts would be very similar, also since
>>> lua will become a part of OCL.
>>>
>>> The idea is then that startup.ops contains 'top level' script commands,
>>> programs, functions and state machines in any order. For example:
>>>
>>>
>>> var int counter = 0; // counter is added to the attributes of the
>>> scripting service (global in this service).
>>> 
>>> // foo program becomes a new service in the component
>>> program foo() {
>>> 
>>>  var int foo_counter =1;
>>>  while (true) {
>>> 
>>>     foo_counter = foo_counter + 1
>>>     yield
>>> 
>>> }
>>> 
>>> }
>>> 
>>> // after foo has been parsed, it is available:
>>> foo.foo_counter = 0
>>> foo.start()
>>> 

>>>
>>> As you can see, variables declared at the global scope are added to the
>>> scripting service's scope (all scripts share this). You can specify a
>>> program which is parsed and which can be used in subsequent script
>>> statements or programs/state machines defined in this file or in other
>>> files.
>>> Clearly, any of these elements are optional.
>>>
>>> Finally, I propose some slight (breaking) modifications to the
>>> DeploymentComponent API:
>>> - connectPorts: as it is now, it connects equally named ports.
>>>
>>> -> This really has to go. It's very bad practice.
>>>
>>> - connectTwoPorts: does not take a connection policy as argument
>>>
>>> -> would go too and be replaced by:
>>> + connect("comp1.portA","comp2.portB",ConnPolicy)
>>>
>>> -> connects two ports (possibly in services) with a policy
>>>
>>> + createStream("comp1.portC", ConnPolicy)
>>>
>>> -> setup a stream from a port (in a service) with a policy
>>>
>>> Any other suggestions or objections welcome.
>>
>> The above is not very clear in what changes you intend for user
>> applications. Can you give a more concrete example (pseudo code, not
>> details) showing a v1 XML, and what you intend the v2 version to look
>> like. For instance, the program script you show above looks specific to
>> just one component. How is that helping or hindering large scale or
>> complex deployments?
>
> I need some time to work that out, but I understand the question...

Ack

>> Can you also list the specific problems in XML that you are trying to
>> address. All I see above is "XML bad, scripting better".
>
> What we can do in scripts and not (easily) in xml during deployment:
> - Use data/properties in other components to make deployment decisions

Ack

> - Group start/stop/configure components using a custom state machine

We already do this in SM. What are you getting at?

> - connect ports/services that were dynamically created during configureHook()

Ack

> - conditionally execute deployment steps

Ack

> - go step-by-step through a deployment process (using pause()/step() api)

Debugging, ok

> - Specify easily connection policies etc.

Instead of adding this to XML?

> - have a very readable/sequential deployment scenario

XML isn't?

> - have deployment react to events, changing situations etc.

Can you give more of an example of these two. Not sure what you are really getting at.

>>>
>>> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
>>> too. I expect 2.2.0 to be around by the end of october. There is no
>>> schedule yet for a 1.14.
>>
>> Do we have roadmap somewhere for what is going in these future releases?
>
> We can prepare one on the wiki.

Please do. I would like to have some input ... I'm sure others in the community do also.

I understand your desire to change this, and definitely agree with some of them, but without a complete example it is very hard to see what issues we are trying to solve. Orocos is flexible enough to provide many different ways of doing the same thing, and not all of us hit the same issues as you (and hence don't necessarily understand why you want to change something).
S

[RFC] Component Deployment and scripting in 2.2

On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...> wrote:
> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
>
>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>>> We're looking at improving deployment of components, and now's a good
>>>> time to give your feedback. These features will be developed after 2.1
>>>> is released.
>>>
>>> Add the ability to deploy components in groups, so that group A is deployed
>>> completely, followed by group B, ..., and then group B is un-deployed
>>> completely, and finally group A is un-deployed completely. This is
>>> necessary for things like logging, which currently hacks this by using the
>>> site-file approach. This is really just sequencing application startup and
>>> shutdown.
>>
>> You *can* already do this using scripting and a state machine.That's my whole
>> point. No XML / DeploymentComponent extensions are necessary for this.
>
> Got an example?

On how to do that ? With one XML file and one script:

Which components to load:

<properties>
<struct name="A1" type="XType" />
<struct name="A2" type="YType" />
<struct name="B1" type="ZType" />
<struct name="B2" type="ZType" />
</properties>

How to configure them (1.x compatible syntax)

StateMachine SetupShutdown {
    var bool do_cleanup = false;
    initial state setup {
           entry {
                // Configure A group
                do A1.configure() ; do A2.configure();
                do connectTwoPorts("A1","Port1", "A2","PortFoo");
                do A1.start() ; do A2.start();
 
                // Configure B group
                do B1.configure() ; do B2.configure();
                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
                do B1.start() ; do B2.start();
           }
           transitions { if do_cleanup then select shutdown; }
    }
 
    final state shutdown {
           entry {
                // Cleanup B group
                do B1.stop() ; do B2.stop();
                do B1.cleanup() ; do B2.cleanup();
 
                // Cleanup A group
                do A1.stop() ; do A2.stop();
                do A1.cleanup() ; do A2.cleanup();
           }
    }
}
RootMachine SetupShutdown deploygroups;

The limitation of scripting is that components must be loaded before the
script is *parsed*, so that's why I declare them in XML first, and do all
the rest in a script.

>
>>>> Most users are not happy with the XML format that specifies which
>>>> components are loaded and connected in an application. Especially the
>>>> flexibility and the fixed ordering of configuration/setup steps disturbs
>>>> even modest applications. The scripting interface on the other hand
>>>> provides a more controlled, readable and flexible environment to specify
>>>> how an application must be setup.
>>>>
>>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>>> scripts and leave XML for basic setup steps. For example: importing
>>>> component libraries or plugins and instantiating components. Everything
>>>> after that should be done in a script.
>>>
>>> What exactly are you talking about changing here? In particular, what are
>>> you talking about removing? I would really like it if nothing that
>>> currently works was removed - we already have a high enough barrier
>>> getting from v1 to v2 due to "removed features" that we use. It would be
>>> nice if this barrier didn't grow further.
>>
>> I'm not planning to remove any of them.
>>
>>>
>>> The only issue we have with XML is that large-scale deployments are tricky
>>> due to difficulty in configuring the deployment. The recent discussion on
>>> use of the XML standard ENTITY element may solve much of that for us.
>>
>> That's true, but it still won't solve ordering problems.
>
> Can you give an example here. Besides master/slave ordering, we've not hit any issues like this, so I'd like to understand the problem better.

Your own example of starting up logging services first and then the
'real' application
is such an example.

>
>>>>
>>>> For this to work as painless as possible, we need to relook at the
>>>> scripting API of the DeploymentComponent and how XML and scripting
>>>> integrate better. These features would be 'primary' for the XML:
>>>>
>>>> * Loading plugins, component libraries etc. -- 'Import'
>>>> * Instantiating components -- component struct
>>>> * Loading a script (which contains programs andor state machines)  --
>>>> 'LoadScript'
>>>> * Starting a specific program or state machine 'StartScript'.
>>>
>>> Can you please list here what existing features you intend to remove from
>>> the XML approach?
>>
>> None.
>>
>>>
>>>> (the old XML features would remain as-is but are not recommended)
>>>
>>> So are you not removing XML features, but deprecating them with an intent
>>> to not maintain in the future?
>>
>> Well, 'deprecating with the intent to point users in the right direction in
>> the future'. I'm also saying this: 'not adding any more features to the XML
>> format (ie freezing it).
>
> Read = cutting out any new features from XML users, forcing us to change to this combined XML/scripting approach? I'm ok with this, we all have to evolve, but I would really like some serious discussion on this. We've already gotten badly bitten by v2, I'd like to see such issues coming more clearly.

The current XML format has hit its limits. It's not just 'cutting out
features', it's 'not being able to provide them'.

>
>>>>
>>>> This would result in this deployment file which 90% of the applications
>>>> uses:

>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <!DOCTYPE properties SYSTEM "cpf.dtd">
>>>> <properties>
>>>> <!-- set import paths -->
>>>> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>>>>
>>>> <!-- Load startup script in Deployer -->
>>>> <struct name="Deployer" type="OCL::DeploymentComponent">
>>>>
>>>> <!-- The name of a file: -->
>>>> <simple name="LoadScript"
>>>> type="string"><value>startup.ops</value></simple> <!-- The name of a
>>>> program or state machine: -->
>>>> <simple name="StartScript"
>>>> type="string"><value>BarProgram</value></simple>
>>>>
>>>> </struct>
>>>> </properties>
>>>> 

>>>>
>>>> A Lua-like variant for loading scripts would be very similar, also since
>>>> lua will become a part of OCL.
>>>>
>>>> The idea is then that startup.ops contains 'top level' script commands,
>>>> programs, functions and state machines in any order. For example:
>>>>
>>>>
>>>> var int counter = 0; // counter is added to the attributes of the
>>>> scripting service (global in this service).
>>>>
>>>> // foo program becomes a new service in the component
>>>> program foo() {
>>>>
>>>>  var int foo_counter =1;
>>>>  while (true) {
>>>>
>>>>     foo_counter = foo_counter + 1
>>>>     yield
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> // after foo has been parsed, it is available:
>>>> foo.foo_counter = 0
>>>> foo.start()
>>>> 

>>>>
>>>> As you can see, variables declared at the global scope are added to the
>>>> scripting service's scope (all scripts share this). You can specify a
>>>> program which is parsed and which can be used in subsequent script
>>>> statements or programs/state machines defined in this file or in other
>>>> files.
>>>> Clearly, any of these elements are optional.
>>>>
>>>> Finally, I propose some slight (breaking) modifications to the
>>>> DeploymentComponent API:
>>>> - connectPorts: as it is now, it connects equally named ports.
>>>>
>>>> -> This really has to go. It's very bad practice.
>>>>
>>>> - connectTwoPorts: does not take a connection policy as argument
>>>>
>>>>  -> would go too and be replaced by:
>>>> + connect("comp1.portA","comp2.portB",ConnPolicy)
>>>>
>>>>  -> connects two ports (possibly in services) with a policy
>>>>
>>>> + createStream("comp1.portC", ConnPolicy)
>>>>
>>>>  -> setup a stream from a port (in a service) with a policy
>>>>
>>>> Any other suggestions or objections welcome.
>>>
>>> The above is not very clear in what changes you intend for user
>>> applications. Can you give a more concrete example (pseudo code, not
>>> details) showing a v1 XML, and what you intend the v2 version to look
>>> like. For instance, the program script you show above looks specific to
>>> just one component. How is that helping or hindering large scale or
>>> complex deployments?
>>
>> I need some time to work that out, but I understand the question...
>
> Ack
>
>>> Can you also list the specific problems in XML that you are trying to
>>> address. All I see above is "XML bad, scripting better".
>>
>> What we can do in scripts and not (easily) in xml during deployment:
>> - Use data/properties in other components to make deployment decisions
>
> Ack
>
>> - Group start/stop/configure components using a custom state machine
>
> We already do this in SM. What are you getting at?
>
>> - connect ports/services that were dynamically created during configureHook()
>
> Ack
>
>> - conditionally execute deployment steps
>
> Ack
>
>> - go step-by-step through a deployment process (using pause()/step() api)
>
> Debugging, ok
>
>> - Specify easily connection policies etc.
>
> Instead of adding this to XML?

We can specify connections in XML, but compare

<struct type="ConnPolicy" name="ConnectionA">
<simple name="nameid" type="string"><value>outstream</value></simple>
<simple name="transport" type="short"><value>2</value></simple>
<simple name="type" type="short"><value>1</value></simple>
<simple name="size" type="short"><value>100</value></simple>
</struct>
<struct type="ConnPolicy" name="ConnectionB">
<simple name="nameid" type="string"><value>instream</value></simple>
<simple name="transport" type="short"><value>2</value></simple>
<simple name="type" type="short"><value>1</value></simple>
<simple name="size" type="short"><value>10</value></simple>
</struct>
<struct type="AType" name="A1">
<struct type="PropertyBag" name="Ports">
   <simple type="string" name="PortZ"><value>ConnectionA</value></simple>
   <simple type="string" name="PortX"><value>ConnectionB</value></simple>
</struct>
</struct>

with :

var ConnPolicy cp;
cp.transport = ORO_MQUEUE_TRANSPORT
cp.type = Buffered
cp.size = 100
stream("A1.PortZ", cp);
cp.nameid= "instream"
cp.size = 10
stream("A1.PortX", cp);

Both are 100% equivalent, but in the XML case, you'll typically not find the
different structs near each other, while in the script case, you can group
everything logically, and if necessary in an entry script of a state.

>
>> - have a very readable/sequential deployment scenario
>
> XML isn't?

Are you kidding me ?!

>
>> - have deployment react to events, changing situations etc.
>
> Can you give more of an example of these two. Not sure what you are really getting at.

One of the problems of distributed applications is that one process can go down,
and is then restarted again. Today's deployer can not react to that,
other than to
be restarted completely (kickout all, kickstart all). This is a step
that must be done
by an external supervision system. Specifying what needs to happen in a script
might help here (unloadComponent("RemoteComponent");
loadComponent("RemoteComponent") )
Or we could have that intelligence in the TaskContextProxy itself (in case the
naming service is used). But even in the latter case, our deployer
will want to know
and needs to reconnect some ports or set some parameters. Sylvain says this
belongs in a dedicated supervision system. He's right. So I'm balancing here
between what to put in the deployer (as a library) and what to put in
supervision.

My proposal is to allow more supervision in the RTT scripting/lua languages and
adapt the deployment component for being able to deliver the necessary services
to such systems.

Peter
--
Orocos-Dev mailing list
Orocos-Dev [..] ...
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev

[RFC] Component Deployment and scripting in 2.2

On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...> wrote:
> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
>
>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>>> We're looking at improving deployment of components, and now's a good
>>>> time to give your feedback. These features will be developed after 2.1
>>>> is released.
>>>
>>> Add the ability to deploy components in groups, so that group A is deployed
>>> completely, followed by group B, ..., and then group B is un-deployed
>>> completely, and finally group A is un-deployed completely. This is
>>> necessary for things like logging, which currently hacks this by using the
>>> site-file approach. This is really just sequencing application startup and
>>> shutdown.
>>
>> You *can* already do this using scripting and a state machine.That's my whole
>> point. No XML / DeploymentComponent extensions are necessary for this.
>
> Got an example?

On how to do that ? With one XML file and one script:

Which components to load:

<properties>
<struct name="A1" type="XType" />
<struct name="A2" type="YType" />
<struct name="B1" type="ZType" />
<struct name="B2" type="ZType" />
</properties>

How to configure them (1.x compatible syntax)

StateMachine SetupShutdown {
    var bool do_cleanup = false;
    initial state setup {
           entry {
                // Configure A group
                do A1.configure() ; do A2.configure();
                do connectTwoPorts("A1","Port1", "A2","PortFoo");
                do A1.start() ; do A2.start();
 
                // Configure B group
                do B1.configure() ; do B2.configure();
                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
                do B1.start() ; do B2.start();
           }
           transitions { if do_cleanup then select shutdown; }
    }
 
    final state shutdown {
           entry {
                // Cleanup B group
                do B1.stop() ; do B2.stop();
                do B1.cleanup() ; do B2.cleanup();
 
                // Cleanup A group
                do A1.stop() ; do A2.stop();
                do A1.cleanup() ; do A2.cleanup();
           }
    }
}
RootMachine SetupShutdown deploygroups;

The limitation of scripting is that components must be loaded before the
script is *parsed*, so that's why I declare them in XML first, and do all
the rest in a script.

>
>>>> Most users are not happy with the XML format that specifies which
>>>> components are loaded and connected in an application. Especially the
>>>> flexibility and the fixed ordering of configuration/setup steps disturbs
>>>> even modest applications. The scripting interface on the other hand
>>>> provides a more controlled, readable and flexible environment to specify
>>>> how an application must be setup.
>>>>
>>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>>> scripts and leave XML for basic setup steps. For example: importing
>>>> component libraries or plugins and instantiating components. Everything
>>>> after that should be done in a script.
>>>
>>> What exactly are you talking about changing here? In particular, what are
>>> you talking about removing? I would really like it if nothing that
>>> currently works was removed - we already have a high enough barrier
>>> getting from v1 to v2 due to "removed features" that we use. It would be
>>> nice if this barrier didn't grow further.
>>
>> I'm not planning to remove any of them.
>>
>>>
>>> The only issue we have with XML is that large-scale deployments are tricky
>>> due to difficulty in configuring the deployment. The recent discussion on
>>> use of the XML standard ENTITY element may solve much of that for us.
>>
>> That's true, but it still won't solve ordering problems.
>
> Can you give an example here. Besides master/slave ordering, we've not hit any issues like this, so I'd like to understand the problem better.

Your own example of starting up logging services first and then the
'real' application
is such an example.

>
>>>>
>>>> For this to work as painless as possible, we need to relook at the
>>>> scripting API of the DeploymentComponent and how XML and scripting
>>>> integrate better. These features would be 'primary' for the XML:
>>>>
>>>> * Loading plugins, component libraries etc. -- 'Import'
>>>> * Instantiating components -- component struct
>>>> * Loading a script (which contains programs andor state machines)  --
>>>> 'LoadScript'
>>>> * Starting a specific program or state machine 'StartScript'.
>>>
>>> Can you please list here what existing features you intend to remove from
>>> the XML approach?
>>
>> None.
>>
>>>
>>>> (the old XML features would remain as-is but are not recommended)
>>>
>>> So are you not removing XML features, but deprecating them with an intent
>>> to not maintain in the future?
>>
>> Well, 'deprecating with the intent to point users in the right direction in
>> the future'. I'm also saying this: 'not adding any more features to the XML
>> format (ie freezing it).
>
> Read = cutting out any new features from XML users, forcing us to change to this combined XML/scripting approach? I'm ok with this, we all have to evolve, but I would really like some serious discussion on this. We've already gotten badly bitten by v2, I'd like to see such issues coming more clearly.

The current XML format has hit its limits. It's not just 'cutting out
features', it's 'not being able to provide them'.

>
>>>>
>>>> This would result in this deployment file which 90% of the applications
>>>> uses:

>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <!DOCTYPE properties SYSTEM "cpf.dtd">
>>>> <properties>
>>>> <!-- set import paths -->
>>>> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>>>>
>>>> <!-- Load startup script in Deployer -->
>>>> <struct name="Deployer" type="OCL::DeploymentComponent">
>>>>
>>>> <!-- The name of a file: -->
>>>> <simple name="LoadScript"
>>>> type="string"><value>startup.ops</value></simple> <!-- The name of a
>>>> program or state machine: -->
>>>> <simple name="StartScript"
>>>> type="string"><value>BarProgram</value></simple>
>>>>
>>>> </struct>
>>>> </properties>
>>>> 

>>>>
>>>> A Lua-like variant for loading scripts would be very similar, also since
>>>> lua will become a part of OCL.
>>>>
>>>> The idea is then that startup.ops contains 'top level' script commands,
>>>> programs, functions and state machines in any order. For example:
>>>>
>>>>
>>>> var int counter = 0; // counter is added to the attributes of the
>>>> scripting service (global in this service).
>>>>
>>>> // foo program becomes a new service in the component
>>>> program foo() {
>>>>
>>>>  var int foo_counter =1;
>>>>  while (true) {
>>>>
>>>>     foo_counter = foo_counter + 1
>>>>     yield
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> // after foo has been parsed, it is available:
>>>> foo.foo_counter = 0
>>>> foo.start()
>>>> 

>>>>
>>>> As you can see, variables declared at the global scope are added to the
>>>> scripting service's scope (all scripts share this). You can specify a
>>>> program which is parsed and which can be used in subsequent script
>>>> statements or programs/state machines defined in this file or in other
>>>> files.
>>>> Clearly, any of these elements are optional.
>>>>
>>>> Finally, I propose some slight (breaking) modifications to the
>>>> DeploymentComponent API:
>>>> - connectPorts: as it is now, it connects equally named ports.
>>>>
>>>> -> This really has to go. It's very bad practice.
>>>>
>>>> - connectTwoPorts: does not take a connection policy as argument
>>>>
>>>>  -> would go too and be replaced by:
>>>> + connect("comp1.portA","comp2.portB",ConnPolicy)
>>>>
>>>>  -> connects two ports (possibly in services) with a policy
>>>>
>>>> + createStream("comp1.portC", ConnPolicy)
>>>>
>>>>  -> setup a stream from a port (in a service) with a policy
>>>>
>>>> Any other suggestions or objections welcome.
>>>
>>> The above is not very clear in what changes you intend for user
>>> applications. Can you give a more concrete example (pseudo code, not
>>> details) showing a v1 XML, and what you intend the v2 version to look
>>> like. For instance, the program script you show above looks specific to
>>> just one component. How is that helping or hindering large scale or
>>> complex deployments?
>>
>> I need some time to work that out, but I understand the question...
>
> Ack
>
>>> Can you also list the specific problems in XML that you are trying to
>>> address. All I see above is "XML bad, scripting better".
>>
>> What we can do in scripts and not (easily) in xml during deployment:
>> - Use data/properties in other components to make deployment decisions
>
> Ack
>
>> - Group start/stop/configure components using a custom state machine
>
> We already do this in SM. What are you getting at?
>
>> - connect ports/services that were dynamically created during configureHook()
>
> Ack
>
>> - conditionally execute deployment steps
>
> Ack
>
>> - go step-by-step through a deployment process (using pause()/step() api)
>
> Debugging, ok
>
>> - Specify easily connection policies etc.
>
> Instead of adding this to XML?

We can specify connections in XML, but compare

<struct type="ConnPolicy" name="ConnectionA">
<simple name="nameid" type="string"><value>outstream</value></simple>
<simple name="transport" type="short"><value>2</value></simple>
<simple name="type" type="short"><value>1</value></simple>
<simple name="size" type="short"><value>100</value></simple>
</struct>
<struct type="ConnPolicy" name="ConnectionB">
<simple name="nameid" type="string"><value>instream</value></simple>
<simple name="transport" type="short"><value>2</value></simple>
<simple name="type" type="short"><value>1</value></simple>
<simple name="size" type="short"><value>10</value></simple>
</struct>
<struct type="AType" name="A1">
<struct type="PropertyBag" name="Ports">
   <simple type="string" name="PortZ"><value>ConnectionA</value></simple>
   <simple type="string" name="PortX"><value>ConnectionB</value></simple>
</struct>
</struct>

with :

var ConnPolicy cp;
cp.transport = ORO_MQUEUE_TRANSPORT
cp.type = Buffered
cp.size = 100
stream("A1.PortZ", cp);
cp.nameid= "instream"
cp.size = 10
stream("A1.PortX", cp);

Both are 100% equivalent, but in the XML case, you'll typically not find the
different structs near each other, while in the script case, you can group
everything logically, and if necessary in an entry script of a state.

>
>> - have a very readable/sequential deployment scenario
>
> XML isn't?

Are you kidding me ?!

>
>> - have deployment react to events, changing situations etc.
>
> Can you give more of an example of these two. Not sure what you are really getting at.

One of the problems of distributed applications is that one process can go down,
and is then restarted again. Today's deployer can not react to that,
other than to
be restarted completely (kickout all, kickstart all). This is a step
that must be done
by an external supervision system. Specifying what needs to happen in a script
might help here (unloadComponent("RemoteComponent");
loadComponent("RemoteComponent") )
Or we could have that intelligence in the TaskContextProxy itself (in case the
naming service is used). But even in the latter case, our deployer
will want to know
and needs to reconnect some ports or set some parameters. Sylvain says this
belongs in a dedicated supervision system. He's right. So I'm balancing here
between what to put in the deployer (as a library) and what to put in
supervision.

My proposal is to allow more supervision in the RTT scripting/lua languages and
adapt the deployment component for being able to deliver the necessary services
to such systems.

Peter
--
Orocos-Users mailing list
Orocos-Users [..] ...
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-users

[RFC] Component Deployment and scripting in 2.2

On Oct 16, 2010, at 10:08 , Peter Soetens wrote:

> On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...> wrote:
>> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
>>
>>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>>>> We're looking at improving deployment of components, and now's a good
>>>>> time to give your feedback. These features will be developed after 2.1
>>>>> is released.
>>>>
>>>> Add the ability to deploy components in groups, so that group A is deployed
>>>> completely, followed by group B, ..., and then group B is un-deployed
>>>> completely, and finally group A is un-deployed completely. This is
>>>> necessary for things like logging, which currently hacks this by using the
>>>> site-file approach. This is really just sequencing application startup and
>>>> shutdown.
>>>
>>> You *can* already do this using scripting and a state machine.That's my whole
>>> point. No XML / DeploymentComponent extensions are necessary for this.
>>
>> Got an example?
>
> On how to do that ? With one XML file and one script:
>
> Which components to load:
>

> <properties>
> <struct name="A1" type="XType" />
> <struct name="A2" type="YType" />
> <struct name="B1" type="ZType" />
> <struct name="B2" type="ZType" />
> </properties>
> 

>
> How to configure them (1.x compatible syntax)
>
> StateMachine SetupShutdown {
>    var bool do_cleanup = false;
>    initial state setup {
>           entry {
>                // Configure A group
>                do A1.configure() ; do A2.configure();
>                do connectTwoPorts("A1","Port1", "A2","PortFoo");
>                do A1.start() ; do A2.start();
> 
>                // Configure B group
>                do B1.configure() ; do B2.configure();
>                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
>                do B1.start() ; do B2.start();
>           }
>           transitions { if do_cleanup then select shutdown; }
>    }
> 
>    final state shutdown {
>           entry {
>                // Cleanup B group
>                do B1.stop() ; do B2.stop();
>                do B1.cleanup() ; do B2.cleanup();
> 
>                // Cleanup A group
>                do A1.stop() ; do A2.stop();
>                do A1.cleanup() ; do A2.cleanup();
>           }
>    }
> }
> RootMachine SetupShutdown deploygroups;
> 

>
> The limitation of scripting is that components must be loaded before the
> script is *parsed*, so that's why I declare them in XML first, and do all
> the rest in a script.

Interesting approach. One benefit of XML that I don't immediately see a way to do above, is that you can compose a system very easily from multiple XML files. If I wanted to have a scenario as above with the A and B groups, and another scenario with A, B, and C groups, I'd need two files duplicating the A and B parts. With XML or grouped loading, I can feed the deployer the group A file, then the B file, then the C file.

Logging or any other system service is a classic use case here.

>>>>> - Specify easily connection policies etc.
>>
>> Instead of adding this to XML?
>
> We can specify connections in XML, but compare
>
>

> <struct type="ConnPolicy" name="ConnectionA">
> <simple name="nameid" type="string"><value>outstream</value></simple>
> <simple name="transport" type="short"><value>2</value></simple>
> <simple name="type" type="short"><value>1</value></simple>
> <simple name="size" type="short"><value>100</value></simple>
> </struct>
> <struct type="ConnPolicy" name="ConnectionB">
> <simple name="nameid" type="string"><value>instream</value></simple>
> <simple name="transport" type="short"><value>2</value></simple>
> <simple name="type" type="short"><value>1</value></simple>
> <simple name="size" type="short"><value>10</value></simple>
> </struct>
> <struct type="AType" name="A1">
> <struct type="PropertyBag" name="Ports">
>   <simple type="string" name="PortZ"><value>ConnectionA</value></simple>
>   <simple type="string" name="PortX"><value>ConnectionB</value></simple>
> </struct>
> </struct>
> 

>
> with :
>
>
> var ConnPolicy cp;
> cp.transport = ORO_MQUEUE_TRANSPORT
> cp.type = Buffered
> cp.size = 100
> stream("A1.PortZ", cp);
> cp.nameid= "instream"
> cp.size = 10
> stream("A1.PortX", cp);
> 

>
> Both are 100% equivalent, but in the XML case, you'll typically not find the
> different structs near each other, while in the script case, you can group
> everything logically, and if necessary in an entry script of a state.

Yep, that's some truly ugly XML.

>>> - have deployment react to events, changing situations etc.
>>
>> Can you give more of an example of these two. Not sure what you are really getting at.
>
> One of the problems of distributed applications is that one process can go down,
> and is then restarted again. Today's deployer can not react to that,
> other than to
> be restarted completely (kickout all, kickstart all). This is a step
> that must be done
> by an external supervision system. Specifying what needs to happen in a script
> might help here (unloadComponent("RemoteComponent");
> loadComponent("RemoteComponent") )
> Or we could have that intelligence in the TaskContextProxy itself (in case the
> naming service is used). But even in the latter case, our deployer
> will want to know
> and needs to reconnect some ports or set some parameters. Sylvain says this
> belongs in a dedicated supervision system. He's right. So I'm balancing here
> between what to put in the deployer (as a library) and what to put in
> supervision.
>
> My proposal is to allow more supervision in the RTT scripting/lua languages and
> adapt the deployment component for being able to deliver the necessary services
> to such systems.

I like the overall approach and the direction you are trying to go. So is there a way to deal with some of the composition issues?
S

[RFC] Component Deployment and scripting in 2.2

On Oct 16, 2010, at 10:08 , Peter Soetens wrote:

> On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...> wrote:
>> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
>>
>>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>>>> We're looking at improving deployment of components, and now's a good
>>>>> time to give your feedback. These features will be developed after 2.1
>>>>> is released.
>>>>
>>>> Add the ability to deploy components in groups, so that group A is deployed
>>>> completely, followed by group B, ..., and then group B is un-deployed
>>>> completely, and finally group A is un-deployed completely. This is
>>>> necessary for things like logging, which currently hacks this by using the
>>>> site-file approach. This is really just sequencing application startup and
>>>> shutdown.
>>>
>>> You *can* already do this using scripting and a state machine.That's my whole
>>> point. No XML / DeploymentComponent extensions are necessary for this.
>>
>> Got an example?
>
> On how to do that ? With one XML file and one script:
>
> Which components to load:
>

> <properties>
> <struct name="A1" type="XType" />
> <struct name="A2" type="YType" />
> <struct name="B1" type="ZType" />
> <struct name="B2" type="ZType" />
> </properties>
> 

>
> How to configure them (1.x compatible syntax)
>
> StateMachine SetupShutdown {
>    var bool do_cleanup = false;
>    initial state setup {
>           entry {
>                // Configure A group
>                do A1.configure() ; do A2.configure();
>                do connectTwoPorts("A1","Port1", "A2","PortFoo");
>                do A1.start() ; do A2.start();
> 
>                // Configure B group
>                do B1.configure() ; do B2.configure();
>                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
>                do B1.start() ; do B2.start();
>           }
>           transitions { if do_cleanup then select shutdown; }
>    }
> 
>    final state shutdown {
>           entry {
>                // Cleanup B group
>                do B1.stop() ; do B2.stop();
>                do B1.cleanup() ; do B2.cleanup();
> 
>                // Cleanup A group
>                do A1.stop() ; do A2.stop();
>                do A1.cleanup() ; do A2.cleanup();
>           }
>    }
> }
> RootMachine SetupShutdown deploygroups;
> 

>
> The limitation of scripting is that components must be loaded before the
> script is *parsed*, so that's why I declare them in XML first, and do all
> the rest in a script.

Interesting approach. One benefit of XML that I don't immediately see a way to do above, is that you can compose a system very easily from multiple XML files. If I wanted to have a scenario as above with the A and B groups, and another scenario with A, B, and C groups, I'd need two files duplicating the A and B parts. With XML or grouped loading, I can feed the deployer the group A file, then the B file, then the C file.

Logging or any other system service is a classic use case here.

>>>>> - Specify easily connection policies etc.
>>
>> Instead of adding this to XML?
>
> We can specify connections in XML, but compare
>
>

> <struct type="ConnPolicy" name="ConnectionA">
> <simple name="nameid" type="string"><value>outstream</value></simple>
> <simple name="transport" type="short"><value>2</value></simple>
> <simple name="type" type="short"><value>1</value></simple>
> <simple name="size" type="short"><value>100</value></simple>
> </struct>
> <struct type="ConnPolicy" name="ConnectionB">
> <simple name="nameid" type="string"><value>instream</value></simple>
> <simple name="transport" type="short"><value>2</value></simple>
> <simple name="type" type="short"><value>1</value></simple>
> <simple name="size" type="short"><value>10</value></simple>
> </struct>
> <struct type="AType" name="A1">
> <struct type="PropertyBag" name="Ports">
>   <simple type="string" name="PortZ"><value>ConnectionA</value></simple>
>   <simple type="string" name="PortX"><value>ConnectionB</value></simple>
> </struct>
> </struct>
> 

>
> with :
>
>
> var ConnPolicy cp;
> cp.transport = ORO_MQUEUE_TRANSPORT
> cp.type = Buffered
> cp.size = 100
> stream("A1.PortZ", cp);
> cp.nameid= "instream"
> cp.size = 10
> stream("A1.PortX", cp);
> 

>
> Both are 100% equivalent, but in the XML case, you'll typically not find the
> different structs near each other, while in the script case, you can group
> everything logically, and if necessary in an entry script of a state.

Yep, that's some truly ugly XML.

>>> - have deployment react to events, changing situations etc.
>>
>> Can you give more of an example of these two. Not sure what you are really getting at.
>
> One of the problems of distributed applications is that one process can go down,
> and is then restarted again. Today's deployer can not react to that,
> other than to
> be restarted completely (kickout all, kickstart all). This is a step
> that must be done
> by an external supervision system. Specifying what needs to happen in a script
> might help here (unloadComponent("RemoteComponent");
> loadComponent("RemoteComponent") )
> Or we could have that intelligence in the TaskContextProxy itself (in case the
> naming service is used). But even in the latter case, our deployer
> will want to know
> and needs to reconnect some ports or set some parameters. Sylvain says this
> belongs in a dedicated supervision system. He's right. So I'm balancing here
> between what to put in the deployer (as a library) and what to put in
> supervision.
>
> My proposal is to allow more supervision in the RTT scripting/lua languages and
> adapt the deployment component for being able to deliver the necessary services
> to such systems.

I like the overall approach and the direction you are trying to go. So is there a way to deal with some of the composition issues?
S

[RFC] Component Deployment and scripting in 2.2

On Saturday 16 October 2010 18:46:34 Stephen Roderick wrote:
> On Oct 16, 2010, at 10:08 , Peter Soetens wrote:
> > On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...>
wrote:
> >> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
> >>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>>>> We're looking at improving deployment of components, and now's a good
> >>>>> time to give your feedback. These features will be developed after
> >>>>> 2.1 is released.
> >>>>
> >>>> Add the ability to deploy components in groups, so that group A is
> >>>> deployed completely, followed by group B, ..., and then group B is
> >>>> un-deployed completely, and finally group A is un-deployed
> >>>> completely. This is necessary for things like logging, which
> >>>> currently hacks this by using the site-file approach. This is really
> >>>> just sequencing application startup and shutdown.
> >>>
> >>> You *can* already do this using scripting and a state machine.That's my
> >>> whole point. No XML / DeploymentComponent extensions are necessary for
> >>> this.
> >>
> >> Got an example?
> >
> > On how to do that ? With one XML file and one script:
> >
> > Which components to load:
> >

> > <properties>
> > <struct name="A1" type="XType" />
> > <struct name="A2" type="YType" />
> > <struct name="B1" type="ZType" />
> > <struct name="B2" type="ZType" />
> > </properties>
> > 

> >
> > How to configure them (1.x compatible syntax)
> >
> > StateMachine SetupShutdown {
> > 
> >    var bool do_cleanup = false;
> >    initial state setup {
> >    
> >           entry {
> >           
> >                // Configure A group
> >                do A1.configure() ; do A2.configure();
> >                do connectTwoPorts("A1","Port1", "A2","PortFoo");
> >                do A1.start() ; do A2.start();
> >                
> >                // Configure B group
> >                do B1.configure() ; do B2.configure();
> >                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
> >                do B1.start() ; do B2.start();
> >           
> >           }
> >           transitions { if do_cleanup then select shutdown; }
> >    
> >    }
> >    
> >    final state shutdown {
> >    
> >           entry {
> >           
> >                // Cleanup B group
> >                do B1.stop() ; do B2.stop();
> >                do B1.cleanup() ; do B2.cleanup();
> >                
> >                // Cleanup A group
> >                do A1.stop() ; do A2.stop();
> >                do A1.cleanup() ; do A2.cleanup();
> >           
> >           }
> >    
> >    }
> > 
> > }
> > RootMachine SetupShutdown deploygroups;
> > 

> >
> > The limitation of scripting is that components must be loaded before the
> > script is *parsed*, so that's why I declare them in XML first, and do all
> > the rest in a script.
>
> Interesting approach. One benefit of XML that I don't immediately see a way
> to do above, is that you can compose a system very easily from multiple
> XML files. If I wanted to have a scenario as above with the A and B
> groups, and another scenario with A, B, and C groups, I'd need two files
> duplicating the A and B parts. With XML or grouped loading, I can feed the
> deployer the group A file, then the B file, then the C file.

I don't see why you couldn't do this in scripting ? Isn't it just three state
machines, one for group A, one for B and one for C ? I conveniently had A and
B in the same SM above, but that is separatable. What you do gain is the
implicit ordering of the loading/unloading groups: the later an xml is loaded,
the earlier it is unloaded again. We could(should?) have the same policy for
stopping the state machines, ie stop latest loaded first.

My poposal will in fact allow to coordinate this much better, by allowing 'top
level' coordinative instructions in your script files, ie, directly starting a
state machine etc. Exactly the same arguments are true for the lua scripting,
where XML is entirely unnecessary, since you can create and use components
directly in one script.

What we can't do is 'xacro'-like preprocessing since the scripting is not
providing any preprocessing tools. xacro is a ros extension which is getting
quite popular.

Peter

[RFC] Component Deployment and scripting in 2.2

On Saturday 16 October 2010 18:46:34 Stephen Roderick wrote:
> On Oct 16, 2010, at 10:08 , Peter Soetens wrote:
> > On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...>
wrote:
> >> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
> >>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>>>> We're looking at improving deployment of components, and now's a good
> >>>>> time to give your feedback. These features will be developed after
> >>>>> 2.1 is released.
> >>>>
> >>>> Add the ability to deploy components in groups, so that group A is
> >>>> deployed completely, followed by group B, ..., and then group B is
> >>>> un-deployed completely, and finally group A is un-deployed
> >>>> completely. This is necessary for things like logging, which
> >>>> currently hacks this by using the site-file approach. This is really
> >>>> just sequencing application startup and shutdown.
> >>>
> >>> You *can* already do this using scripting and a state machine.That's my
> >>> whole point. No XML / DeploymentComponent extensions are necessary for
> >>> this.
> >>
> >> Got an example?
> >
> > On how to do that ? With one XML file and one script:
> >
> > Which components to load:
> >

> > <properties>
> > <struct name="A1" type="XType" />
> > <struct name="A2" type="YType" />
> > <struct name="B1" type="ZType" />
> > <struct name="B2" type="ZType" />
> > </properties>
> > 

> >
> > How to configure them (1.x compatible syntax)
> >
> > StateMachine SetupShutdown {
> > 
> >    var bool do_cleanup = false;
> >    initial state setup {
> >    
> >           entry {
> >           
> >                // Configure A group
> >                do A1.configure() ; do A2.configure();
> >                do connectTwoPorts("A1","Port1", "A2","PortFoo");
> >                do A1.start() ; do A2.start();
> >                
> >                // Configure B group
> >                do B1.configure() ; do B2.configure();
> >                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
> >                do B1.start() ; do B2.start();
> >           
> >           }
> >           transitions { if do_cleanup then select shutdown; }
> >    
> >    }
> >    
> >    final state shutdown {
> >    
> >           entry {
> >           
> >                // Cleanup B group
> >                do B1.stop() ; do B2.stop();
> >                do B1.cleanup() ; do B2.cleanup();
> >                
> >                // Cleanup A group
> >                do A1.stop() ; do A2.stop();
> >                do A1.cleanup() ; do A2.cleanup();
> >           
> >           }
> >    
> >    }
> > 
> > }
> > RootMachine SetupShutdown deploygroups;
> > 

> >
> > The limitation of scripting is that components must be loaded before the
> > script is *parsed*, so that's why I declare them in XML first, and do all
> > the rest in a script.
>
> Interesting approach. One benefit of XML that I don't immediately see a way
> to do above, is that you can compose a system very easily from multiple
> XML files. If I wanted to have a scenario as above with the A and B
> groups, and another scenario with A, B, and C groups, I'd need two files
> duplicating the A and B parts. With XML or grouped loading, I can feed the
> deployer the group A file, then the B file, then the C file.

I don't see why you couldn't do this in scripting ? Isn't it just three state
machines, one for group A, one for B and one for C ? I conveniently had A and
B in the same SM above, but that is separatable. What you do gain is the
implicit ordering of the loading/unloading groups: the later an xml is loaded,
the earlier it is unloaded again. We could(should?) have the same policy for
stopping the state machines, ie stop latest loaded first.

My poposal will in fact allow to coordinate this much better, by allowing 'top
level' coordinative instructions in your script files, ie, directly starting a
state machine etc. Exactly the same arguments are true for the lua scripting,
where XML is entirely unnecessary, since you can create and use components
directly in one script.

What we can't do is 'xacro'-like preprocessing since the scripting is not
providing any preprocessing tools. xacro is a ros extension which is getting
quite popular.

Peter

[RFC] Component Deployment and scripting in 2.2

On Saturday 16 October 2010 18:46:34 Stephen Roderick wrote:
> On Oct 16, 2010, at 10:08 , Peter Soetens wrote:
> > On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...>
wrote:
> >> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
> >>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>>>> We're looking at improving deployment of components, and now's a good
> >>>>> time to give your feedback. These features will be developed after
> >>>>> 2.1 is released.
> >>>>
> >>>> Add the ability to deploy components in groups, so that group A is
> >>>> deployed completely, followed by group B, ..., and then group B is
> >>>> un-deployed completely, and finally group A is un-deployed
> >>>> completely. This is necessary for things like logging, which
> >>>> currently hacks this by using the site-file approach. This is really
> >>>> just sequencing application startup and shutdown.
> >>>
> >>> You *can* already do this using scripting and a state machine.That's my
> >>> whole point. No XML / DeploymentComponent extensions are necessary for
> >>> this.
> >>
> >> Got an example?
> >
> > On how to do that ? With one XML file and one script:
> >
> > Which components to load:
> >

> > <properties>
> > <struct name="A1" type="XType" />
> > <struct name="A2" type="YType" />
> > <struct name="B1" type="ZType" />
> > <struct name="B2" type="ZType" />
> > </properties>
> > 

> >
> > How to configure them (1.x compatible syntax)
> >
> > StateMachine SetupShutdown {
> > 
> >    var bool do_cleanup = false;
> >    initial state setup {
> >    
> >           entry {
> >           
> >                // Configure A group
> >                do A1.configure() ; do A2.configure();
> >                do connectTwoPorts("A1","Port1", "A2","PortFoo");
> >                do A1.start() ; do A2.start();
> >                
> >                // Configure B group
> >                do B1.configure() ; do B2.configure();
> >                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
> >                do B1.start() ; do B2.start();
> >           
> >           }
> >           transitions { if do_cleanup then select shutdown; }
> >    
> >    }
> >    
> >    final state shutdown {
> >    
> >           entry {
> >           
> >                // Cleanup B group
> >                do B1.stop() ; do B2.stop();
> >                do B1.cleanup() ; do B2.cleanup();
> >                
> >                // Cleanup A group
> >                do A1.stop() ; do A2.stop();
> >                do A1.cleanup() ; do A2.cleanup();
> >           
> >           }
> >    
> >    }
> > 
> > }
> > RootMachine SetupShutdown deploygroups;
> > 

> >
> > The limitation of scripting is that components must be loaded before the
> > script is *parsed*, so that's why I declare them in XML first, and do all
> > the rest in a script.
>
> Interesting approach. One benefit of XML that I don't immediately see a way
> to do above, is that you can compose a system very easily from multiple
> XML files. If I wanted to have a scenario as above with the A and B
> groups, and another scenario with A, B, and C groups, I'd need two files
> duplicating the A and B parts. With XML or grouped loading, I can feed the
> deployer the group A file, then the B file, then the C file.

I don't see why you couldn't do this in scripting ? Isn't it just three state
machines, one for group A, one for B and one for C ? I conveniently had A and
B in the same SM above, but that is separatable. What you do gain is the
implicit ordering of the loading/unloading groups: the later an xml is loaded,
the earlier it is unloaded again. We could(should?) have the same policy for
stopping the state machines, ie stop latest loaded first.

My poposal will in fact allow to coordinate this much better, by allowing 'top
level' coordinative instructions in your script files, ie, directly starting a
state machine etc. Exactly the same arguments are true for the lua scripting,
where XML is entirely unnecessary, since you can create and use components
directly in one script.

What we can't do is 'xacro'-like preprocessing since the scripting is not
providing any preprocessing tools. xacro is a ros extension which is getting
quite popular.

Peter

[RFC] Component Deployment and scripting in 2.2

On Saturday 16 October 2010 18:46:34 Stephen Roderick wrote:
> On Oct 16, 2010, at 10:08 , Peter Soetens wrote:
> > On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...>
wrote:
> >> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
> >>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>>>> We're looking at improving deployment of components, and now's a good
> >>>>> time to give your feedback. These features will be developed after
> >>>>> 2.1 is released.
> >>>>
> >>>> Add the ability to deploy components in groups, so that group A is
> >>>> deployed completely, followed by group B, ..., and then group B is
> >>>> un-deployed completely, and finally group A is un-deployed
> >>>> completely. This is necessary for things like logging, which
> >>>> currently hacks this by using the site-file approach. This is really
> >>>> just sequencing application startup and shutdown.
> >>>
> >>> You *can* already do this using scripting and a state machine.That's my
> >>> whole point. No XML / DeploymentComponent extensions are necessary for
> >>> this.
> >>
> >> Got an example?
> >
> > On how to do that ? With one XML file and one script:
> >
> > Which components to load:
> >

> > <properties>
> > <struct name="A1" type="XType" />
> > <struct name="A2" type="YType" />
> > <struct name="B1" type="ZType" />
> > <struct name="B2" type="ZType" />
> > </properties>
> > 

> >
> > How to configure them (1.x compatible syntax)
> >
> > StateMachine SetupShutdown {
> > 
> >    var bool do_cleanup = false;
> >    initial state setup {
> >    
> >           entry {
> >           
> >                // Configure A group
> >                do A1.configure() ; do A2.configure();
> >                do connectTwoPorts("A1","Port1", "A2","PortFoo");
> >                do A1.start() ; do A2.start();
> >                
> >                // Configure B group
> >                do B1.configure() ; do B2.configure();
> >                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
> >                do B1.start() ; do B2.start();
> >           
> >           }
> >           transitions { if do_cleanup then select shutdown; }
> >    
> >    }
> >    
> >    final state shutdown {
> >    
> >           entry {
> >           
> >                // Cleanup B group
> >                do B1.stop() ; do B2.stop();
> >                do B1.cleanup() ; do B2.cleanup();
> >                
> >                // Cleanup A group
> >                do A1.stop() ; do A2.stop();
> >                do A1.cleanup() ; do A2.cleanup();
> >           
> >           }
> >    
> >    }
> > 
> > }
> > RootMachine SetupShutdown deploygroups;
> > 

> >
> > The limitation of scripting is that components must be loaded before the
> > script is *parsed*, so that's why I declare them in XML first, and do all
> > the rest in a script.
>
> Interesting approach. One benefit of XML that I don't immediately see a way
> to do above, is that you can compose a system very easily from multiple
> XML files. If I wanted to have a scenario as above with the A and B
> groups, and another scenario with A, B, and C groups, I'd need two files
> duplicating the A and B parts. With XML or grouped loading, I can feed the
> deployer the group A file, then the B file, then the C file.

I don't see why you couldn't do this in scripting ? Isn't it just three state
machines, one for group A, one for B and one for C ? I conveniently had A and
B in the same SM above, but that is separatable. What you do gain is the
implicit ordering of the loading/unloading groups: the later an xml is loaded,
the earlier it is unloaded again. We could(should?) have the same policy for
stopping the state machines, ie stop latest loaded first.

My poposal will in fact allow to coordinate this much better, by allowing 'top
level' coordinative instructions in your script files, ie, directly starting a
state machine etc. Exactly the same arguments are true for the lua scripting,
where XML is entirely unnecessary, since you can create and use components
directly in one script.

What we can't do is 'xacro'-like preprocessing since the scripting is not
providing any preprocessing tools. xacro is a ros extension which is getting
quite popular.

Peter

[RFC] Component Deployment and scripting in 2.2

On Saturday 16 October 2010 18:46:34 Stephen Roderick wrote:
> On Oct 16, 2010, at 10:08 , Peter Soetens wrote:
> > On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...>
wrote:
> >> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
> >>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>>>> We're looking at improving deployment of components, and now's a good
> >>>>> time to give your feedback. These features will be developed after
> >>>>> 2.1 is released.
> >>>>
> >>>> Add the ability to deploy components in groups, so that group A is
> >>>> deployed completely, followed by group B, ..., and then group B is
> >>>> un-deployed completely, and finally group A is un-deployed
> >>>> completely. This is necessary for things like logging, which
> >>>> currently hacks this by using the site-file approach. This is really
> >>>> just sequencing application startup and shutdown.
> >>>
> >>> You *can* already do this using scripting and a state machine.That's my
> >>> whole point. No XML / DeploymentComponent extensions are necessary for
> >>> this.
> >>
> >> Got an example?
> >
> > On how to do that ? With one XML file and one script:
> >
> > Which components to load:
> >

> > <properties>
> > <struct name="A1" type="XType" />
> > <struct name="A2" type="YType" />
> > <struct name="B1" type="ZType" />
> > <struct name="B2" type="ZType" />
> > </properties>
> > 

> >
> > How to configure them (1.x compatible syntax)
> >
> > StateMachine SetupShutdown {
> > 
> >    var bool do_cleanup = false;
> >    initial state setup {
> >    
> >           entry {
> >           
> >                // Configure A group
> >                do A1.configure() ; do A2.configure();
> >                do connectTwoPorts("A1","Port1", "A2","PortFoo");
> >                do A1.start() ; do A2.start();
> >                
> >                // Configure B group
> >                do B1.configure() ; do B2.configure();
> >                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
> >                do B1.start() ; do B2.start();
> >           
> >           }
> >           transitions { if do_cleanup then select shutdown; }
> >    
> >    }
> >    
> >    final state shutdown {
> >    
> >           entry {
> >           
> >                // Cleanup B group
> >                do B1.stop() ; do B2.stop();
> >                do B1.cleanup() ; do B2.cleanup();
> >                
> >                // Cleanup A group
> >                do A1.stop() ; do A2.stop();
> >                do A1.cleanup() ; do A2.cleanup();
> >           
> >           }
> >    
> >    }
> > 
> > }
> > RootMachine SetupShutdown deploygroups;
> > 

> >
> > The limitation of scripting is that components must be loaded before the
> > script is *parsed*, so that's why I declare them in XML first, and do all
> > the rest in a script.
>
> Interesting approach. One benefit of XML that I don't immediately see a way
> to do above, is that you can compose a system very easily from multiple
> XML files. If I wanted to have a scenario as above with the A and B
> groups, and another scenario with A, B, and C groups, I'd need two files
> duplicating the A and B parts. With XML or grouped loading, I can feed the
> deployer the group A file, then the B file, then the C file.

I don't see why you couldn't do this in scripting ? Isn't it just three state
machines, one for group A, one for B and one for C ? I conveniently had A and
B in the same SM above, but that is separatable. What you do gain is the
implicit ordering of the loading/unloading groups: the later an xml is loaded,
the earlier it is unloaded again. We could(should?) have the same policy for
stopping the state machines, ie stop latest loaded first.

My poposal will in fact allow to coordinate this much better, by allowing 'top
level' coordinative instructions in your script files, ie, directly starting a
state machine etc. Exactly the same arguments are true for the lua scripting,
where XML is entirely unnecessary, since you can create and use components
directly in one script.

What we can't do is 'xacro'-like preprocessing since the scripting is not
providing any preprocessing tools. xacro is a ros extension which is getting
quite popular.

Peter

[RFC] Component Deployment and scripting in 2.2

On Saturday 16 October 2010 18:46:34 Stephen Roderick wrote:
> On Oct 16, 2010, at 10:08 , Peter Soetens wrote:
> > On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...>
wrote:
> >> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
> >>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>>>> We're looking at improving deployment of components, and now's a good
> >>>>> time to give your feedback. These features will be developed after
> >>>>> 2.1 is released.
> >>>>
> >>>> Add the ability to deploy components in groups, so that group A is
> >>>> deployed completely, followed by group B, ..., and then group B is
> >>>> un-deployed completely, and finally group A is un-deployed
> >>>> completely. This is necessary for things like logging, which
> >>>> currently hacks this by using the site-file approach. This is really
> >>>> just sequencing application startup and shutdown.
> >>>
> >>> You *can* already do this using scripting and a state machine.That's my
> >>> whole point. No XML / DeploymentComponent extensions are necessary for
> >>> this.
> >>
> >> Got an example?
> >
> > On how to do that ? With one XML file and one script:
> >
> > Which components to load:
> >

> > <properties>
> > <struct name="A1" type="XType" />
> > <struct name="A2" type="YType" />
> > <struct name="B1" type="ZType" />
> > <struct name="B2" type="ZType" />
> > </properties>
> > 

> >
> > How to configure them (1.x compatible syntax)
> >
> > StateMachine SetupShutdown {
> > 
> >    var bool do_cleanup = false;
> >    initial state setup {
> >    
> >           entry {
> >           
> >                // Configure A group
> >                do A1.configure() ; do A2.configure();
> >                do connectTwoPorts("A1","Port1", "A2","PortFoo");
> >                do A1.start() ; do A2.start();
> >                
> >                // Configure B group
> >                do B1.configure() ; do B2.configure();
> >                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
> >                do B1.start() ; do B2.start();
> >           
> >           }
> >           transitions { if do_cleanup then select shutdown; }
> >    
> >    }
> >    
> >    final state shutdown {
> >    
> >           entry {
> >           
> >                // Cleanup B group
> >                do B1.stop() ; do B2.stop();
> >                do B1.cleanup() ; do B2.cleanup();
> >                
> >                // Cleanup A group
> >                do A1.stop() ; do A2.stop();
> >                do A1.cleanup() ; do A2.cleanup();
> >           
> >           }
> >    
> >    }
> > 
> > }
> > RootMachine SetupShutdown deploygroups;
> > 

> >
> > The limitation of scripting is that components must be loaded before the
> > script is *parsed*, so that's why I declare them in XML first, and do all
> > the rest in a script.
>
> Interesting approach. One benefit of XML that I don't immediately see a way
> to do above, is that you can compose a system very easily from multiple
> XML files. If I wanted to have a scenario as above with the A and B
> groups, and another scenario with A, B, and C groups, I'd need two files
> duplicating the A and B parts. With XML or grouped loading, I can feed the
> deployer the group A file, then the B file, then the C file.

I don't see why you couldn't do this in scripting ? Isn't it just three state
machines, one for group A, one for B and one for C ? I conveniently had A and
B in the same SM above, but that is separatable. What you do gain is the
implicit ordering of the loading/unloading groups: the later an xml is loaded,
the earlier it is unloaded again. We could(should?) have the same policy for
stopping the state machines, ie stop latest loaded first.

My poposal will in fact allow to coordinate this much better, by allowing 'top
level' coordinative instructions in your script files, ie, directly starting a
state machine etc. Exactly the same arguments are true for the lua scripting,
where XML is entirely unnecessary, since you can create and use components
directly in one script.

What we can't do is 'xacro'-like preprocessing since the scripting is not
providing any preprocessing tools. xacro is a ros extension which is getting
quite popular.

Peter

[RFC] Component Deployment and scripting in 2.2

On Saturday 16 October 2010 18:46:34 Stephen Roderick wrote:
> On Oct 16, 2010, at 10:08 , Peter Soetens wrote:
> > On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...>
wrote:
> >> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
> >>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>>>> We're looking at improving deployment of components, and now's a good
> >>>>> time to give your feedback. These features will be developed after
> >>>>> 2.1 is released.
> >>>>
> >>>> Add the ability to deploy components in groups, so that group A is
> >>>> deployed completely, followed by group B, ..., and then group B is
> >>>> un-deployed completely, and finally group A is un-deployed
> >>>> completely. This is necessary for things like logging, which
> >>>> currently hacks this by using the site-file approach. This is really
> >>>> just sequencing application startup and shutdown.
> >>>
> >>> You *can* already do this using scripting and a state machine.That's my
> >>> whole point. No XML / DeploymentComponent extensions are necessary for
> >>> this.
> >>
> >> Got an example?
> >
> > On how to do that ? With one XML file and one script:
> >
> > Which components to load:
> >

> > <properties>
> > <struct name="A1" type="XType" />
> > <struct name="A2" type="YType" />
> > <struct name="B1" type="ZType" />
> > <struct name="B2" type="ZType" />
> > </properties>
> > 

> >
> > How to configure them (1.x compatible syntax)
> >
> > StateMachine SetupShutdown {
> > 
> >    var bool do_cleanup = false;
> >    initial state setup {
> >    
> >           entry {
> >           
> >                // Configure A group
> >                do A1.configure() ; do A2.configure();
> >                do connectTwoPorts("A1","Port1", "A2","PortFoo");
> >                do A1.start() ; do A2.start();
> >                
> >                // Configure B group
> >                do B1.configure() ; do B2.configure();
> >                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
> >                do B1.start() ; do B2.start();
> >           
> >           }
> >           transitions { if do_cleanup then select shutdown; }
> >    
> >    }
> >    
> >    final state shutdown {
> >    
> >           entry {
> >           
> >                // Cleanup B group
> >                do B1.stop() ; do B2.stop();
> >                do B1.cleanup() ; do B2.cleanup();
> >                
> >                // Cleanup A group
> >                do A1.stop() ; do A2.stop();
> >                do A1.cleanup() ; do A2.cleanup();
> >           
> >           }
> >    
> >    }
> > 
> > }
> > RootMachine SetupShutdown deploygroups;
> > 

> >
> > The limitation of scripting is that components must be loaded before the
> > script is *parsed*, so that's why I declare them in XML first, and do all
> > the rest in a script.
>
> Interesting approach. One benefit of XML that I don't immediately see a way
> to do above, is that you can compose a system very easily from multiple
> XML files. If I wanted to have a scenario as above with the A and B
> groups, and another scenario with A, B, and C groups, I'd need two files
> duplicating the A and B parts. With XML or grouped loading, I can feed the
> deployer the group A file, then the B file, then the C file.

I don't see why you couldn't do this in scripting ? Isn't it just three state
machines, one for group A, one for B and one for C ? I conveniently had A and
B in the same SM above, but that is separatable. What you do gain is the
implicit ordering of the loading/unloading groups: the later an xml is loaded,
the earlier it is unloaded again. We could(should?) have the same policy for
stopping the state machines, ie stop latest loaded first.

My poposal will in fact allow to coordinate this much better, by allowing 'top
level' coordinative instructions in your script files, ie, directly starting a
state machine etc. Exactly the same arguments are true for the lua scripting,
where XML is entirely unnecessary, since you can create and use components
directly in one script.

What we can't do is 'xacro'-like preprocessing since the scripting is not
providing any preprocessing tools. xacro is a ros extension which is getting
quite popular.

Peter

[RFC] Component Deployment and scripting in 2.2

On Saturday 16 October 2010 18:46:34 Stephen Roderick wrote:
> On Oct 16, 2010, at 10:08 , Peter Soetens wrote:
> > On Thu, Sep 30, 2010 at 2:37 PM, Stephen Roderick <kiwi [dot] net [..] ...>
wrote:
> >> On Sep 30, 2010, at 08:27 , Peter Soetens wrote:
> >>> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> >>>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> >>>>> We're looking at improving deployment of components, and now's a good
> >>>>> time to give your feedback. These features will be developed after
> >>>>> 2.1 is released.
> >>>>
> >>>> Add the ability to deploy components in groups, so that group A is
> >>>> deployed completely, followed by group B, ..., and then group B is
> >>>> un-deployed completely, and finally group A is un-deployed
> >>>> completely. This is necessary for things like logging, which
> >>>> currently hacks this by using the site-file approach. This is really
> >>>> just sequencing application startup and shutdown.
> >>>
> >>> You *can* already do this using scripting and a state machine.That's my
> >>> whole point. No XML / DeploymentComponent extensions are necessary for
> >>> this.
> >>
> >> Got an example?
> >
> > On how to do that ? With one XML file and one script:
> >
> > Which components to load:
> >

> > <properties>
> > <struct name="A1" type="XType" />
> > <struct name="A2" type="YType" />
> > <struct name="B1" type="ZType" />
> > <struct name="B2" type="ZType" />
> > </properties>
> > 

> >
> > How to configure them (1.x compatible syntax)
> >
> > StateMachine SetupShutdown {
> > 
> >    var bool do_cleanup = false;
> >    initial state setup {
> >    
> >           entry {
> >           
> >                // Configure A group
> >                do A1.configure() ; do A2.configure();
> >                do connectTwoPorts("A1","Port1", "A2","PortFoo");
> >                do A1.start() ; do A2.start();
> >                
> >                // Configure B group
> >                do B1.configure() ; do B2.configure();
> >                do connectTwoPorts("B1","PortBar", "A2","PortFoo");
> >                do B1.start() ; do B2.start();
> >           
> >           }
> >           transitions { if do_cleanup then select shutdown; }
> >    
> >    }
> >    
> >    final state shutdown {
> >    
> >           entry {
> >           
> >                // Cleanup B group
> >                do B1.stop() ; do B2.stop();
> >                do B1.cleanup() ; do B2.cleanup();
> >                
> >                // Cleanup A group
> >                do A1.stop() ; do A2.stop();
> >                do A1.cleanup() ; do A2.cleanup();
> >           
> >           }
> >    
> >    }
> > 
> > }
> > RootMachine SetupShutdown deploygroups;
> > 

> >
> > The limitation of scripting is that components must be loaded before the
> > script is *parsed*, so that's why I declare them in XML first, and do all
> > the rest in a script.
>
> Interesting approach. One benefit of XML that I don't immediately see a way
> to do above, is that you can compose a system very easily from multiple
> XML files. If I wanted to have a scenario as above with the A and B
> groups, and another scenario with A, B, and C groups, I'd need two files
> duplicating the A and B parts. With XML or grouped loading, I can feed the
> deployer the group A file, then the B file, then the C file.

I don't see why you couldn't do this in scripting ? Isn't it just three state
machines, one for group A, one for B and one for C ? I conveniently had A and
B in the same SM above, but that is separatable. What you do gain is the
implicit ordering of the loading/unloading groups: the later an xml is loaded,
the earlier it is unloaded again. We could(should?) have the same policy for
stopping the state machines, ie stop latest loaded first.

My poposal will in fact allow to coordinate this much better, by allowing 'top
level' coordinative instructions in your script files, ie, directly starting a
state machine etc. Exactly the same arguments are true for the lua scripting,
where XML is entirely unnecessary, since you can create and use components
directly in one script.

What we can't do is 'xacro'-like preprocessing since the scripting is not
providing any preprocessing tools. xacro is a ros extension which is getting
quite popular.

Peter