On Jan 25, 2012, at 03:11 , Steven Bellens wrote:

> 2012/1/24 <t [dot] t [dot] g [dot] clephas [..] student [dot] tue [dot] nl>:
>> Hello,
>>
>> At the TU/e we've constructed several controllers each containing several
>> components.
>> Every component acts on a triggerport.
>>
>> So the sequence goes as follows:
>>
>> ReadEncoders -> CalculateErrors -> Gain -> WriteOutput
>>
>> The first component has an update frequency of 1khz.
>>
>>
>> Now in order to identify our hardware we want to log the data send over these
>> channels without missing samples.
>> We hope to get a text file were every line represents the date send at that
>> milisecond over each channel:
>>
>>
>> TimeStamp ReadEncoders CalculateErrors Gain WriteOutput
>> 0.0000 0.000 0.000 0.000 0.000
>> 0.0010 0.300 0.100 0.300 3.000
>> 0.0020 0.400 0.104 0.390 3.330
>> etc.
>>
>>
>> However the current reporter has hardcoded buffers as I understood, so what
>> is the best way of doing this?
>> It is also important that every line represents sequential data.
>
>
> At 1kHz the main limitation will probably be the IO, with which the
> Reporter buffer can't help you. A quick solution could be to write
> only in e.g. sets of 100 samples, by storing the data inside the
> program and logging at a lower frequency.

Disagree (on a desktop-class computer, at least). We log thousands of bytes per cycle, at 500 Hz, and have no trouble whatsoever with the I/O. The O/S filesystem drivers are efficient at dealing with this. On an embedded system with a slow disk, then yes, this might become an issue. But given the output above, they're talking say 50 bytes/per cycle, or 50 kb/s, which is much less than 1% of the bandwidth of a modern HDD.
S

[list CC'd]

On Jan 25, 2012, at 02:07 , Clephas, T.T.G. wrote:

>>> Hello,
>>>
>>> At the TU/e we've constructed several controllers each containing several
>>> components.
>>> Every component acts on a triggerport.
>>>
>>> So the sequence goes as follows:
>>>
>>> ReadEncoders -> CalculateErrors -> Gain -> WriteOutput
>>>
>>> The first component has an update frequency of 1khz.
>>>
>>>
>>> Now in order to identify our hardware we want to log the data send over these
>>> channels without missing samples.
>>> We hope to get a text file were every line represents the date send at that
>>> milisecond over each channel:
>>>
>>>
>>> TimeStamp ReadEncoders CalculateErrors Gain WriteOutput
>>> 0.0000 0.000 0.000 0.000 0.000
>>> 0.0010 0.300 0.100 0.300 3.000
>>> 0.0020 0.400 0.104 0.390 3.330
>>> etc.
>>>
>>>
>>> However the current reporter has hardcoded buffers as I understood, so what
>>> is the best way of doing this?
>>> It is also important that every line represents sequential data.
>>>
>>> Thanks in advance!
>>
>> We do exactly this, at similar frequencies. Our approach has a bit of boilerplate code, but it works well. AFAIK you can't use the OCL::Reporting for this, as it has all kinds of limitations.
>>
>> 1) Create a component that samples the data of interest, and then logs it through the OCL Logging framework (ie real-time logging)
>> 2) Coordinate calling this component _after_ all your data has been computed on a cycle (at that point, your component samples everything, and then logs it)
>> 3) Connect a File/Socket/etc Appender to your logging category, and use that to store/transport your data.
>>
>> You'll have to make sure that your TLSF memory pool is of sufficient size, and that your logging buffers can absorb the difference in production rates of the real-time code, and the consumption rate of your (likely non-real-time) appending code. For desktop systems with oodles of RAM, this is no problem. For embedded units, this can take quite some tweaking.
> It's a desktop, plenty of RAM available.
>>
>> The boilerplate stuff is in all 1) for us, as it knows about the samples and data types of interest. I'm sure that more enterprising souls can generalize this solution …
>>
>> HTH
>> S
>
> Could you give me link to that code? That way I can get the idea of the implementation and how the data is written.
> Doesn't matter if it's boilerplate code, I might find a way to make it more general.

Sorry, no can do. But perhaps a better outline will help ...

Given component A with ports 1 and 2, and component B with port 3.

Reporter component
// NB custom per application, here Rock's generation facilities might help you
startHook()
reportHeader()

updateHook()
sample()
reportData()

sample()
// store in class members, the data from ports A.1, A.2, B.3

reportHeader()
// log your column headers

reportData()
// log your sample'd data

Executive/Supervisor/Master component
// just a state machine

state RUNNING {
entry {
do componentA.start();
do componentB.start();
do reporter.start();
}
run {
do componentA.update();
do componentB.update();
do reporter.update();
}
exit {
do componentA.stop();
do componentB.stop();
do reporter.stop();
}
}

Deploy
- Executive component as a periodic activity at 1 khz
- component A, componentB and the Reporter as slave's of the Executive
- a logging appender component (see OCL logging or the RTT v2 equivalent) as a periodic activity (probably at the 1khz rate also)

Size your TLSF buffer appropriately (we use 20 MB). Turn off SBRK and MMAP in RTT's TLSF configuration.
We use a smaller logging buffer size of 200 (the default in OCL v1 is 1000), so you'll be fine either way. See a previous post for the interaction of this value with the TLSF buffer size.

One of our smaller applications logs 100 kb/second to disk with this approach, with image storage on top of that. We also have a different state for each type of motion we can do, with different controllers being started, logged, and stopped, in each state.

YMMV
S

If you are a bit brave and are using typegen to generate your typekits,
you may want to give a try to rock's logger

http://rock-robotics.org/package_directory/packages/tools/tools_logger/i...

and

http://rock-robotics.org/documentation/data_analysis/reading_logfiles.html

It logs data in binary, with timestamping, and is *very* efficient. It
has never been tested in an OCL-based workflow, though. So, as I said,
you would have to be brave ...

On Jan 24, 2012, at 17:28 , Peter Soetens wrote:

> On Tue, Jan 24, 2012 at 8:00 PM, <t [dot] t [dot] g [dot] clephas [..] student [dot] tue [dot] nl> wrote:
> Hello,
>
> At the TU/e we've constructed several controllers each containing several
> components.
> Every component acts on a triggerport.
>
> So the sequence goes as follows:
>
> ReadEncoders -> CalculateErrors -> Gain -> WriteOutput
>
> The first component has an update frequency of 1khz.
>
>
> Now in order to identify our hardware we want to log the data send over these
> channels without missing samples.
> We hope to get a text file were every line represents the date send at that
> milisecond over each channel:
>
>
> TimeStamp ReadEncoders CalculateErrors Gain WriteOutput
> 0.0000 0.000 0.000 0.000 0.000
> 0.0010 0.300 0.100 0.300 3.000
> 0.0020 0.400 0.104 0.390 3.330
> etc.
>
>
> However the current reporter has hardcoded buffers as I understood, so what
> is the best way of doing this?
> It is also important that every line represents sequential data.
>
> If you make the reporter non-periodic, it will try to log each sample as you expect and use the buffers for, well, buffering when it doesn't get enough time.
>
> Logging to text format is very inefficient and you can only log a certain number of columns before the IO or your thread can't finish it.

Agreed, but it works and it's easy. And that's often good enough …

> The reporter was made to support different marshalling formats such that you could optimize this (for example, only write to a file at the end or write to a binary format). The Netcdf reporter uses a far more efficient format, but I haven't tested it in RTT 2.x, but it does build fine (tm) :)

I can only speak for the v1 OCL reporting, but that implementation was not up to this task. Not sure what improvements have been made in this regard in v2?
S

On Jan 24, 2012, at 14:00 , t [dot] t [dot] g [dot] clephas [..] student [dot] tue [dot] nl wrote:

> Hello,
>
> At the TU/e we've constructed several controllers each containing several
> components.
> Every component acts on a triggerport.
>
> So the sequence goes as follows:
>
> ReadEncoders -> CalculateErrors -> Gain -> WriteOutput
>
> The first component has an update frequency of 1khz.
>
>
> Now in order to identify our hardware we want to log the data send over these
> channels without missing samples.
> We hope to get a text file were every line represents the date send at that
> milisecond over each channel:
>
>
> TimeStamp ReadEncoders CalculateErrors Gain WriteOutput
> 0.0000 0.000 0.000 0.000 0.000
> 0.0010 0.300 0.100 0.300 3.000
> 0.0020 0.400 0.104 0.390 3.330
> etc.
>
>
> However the current reporter has hardcoded buffers as I understood, so what
> is the best way of doing this?
> It is also important that every line represents sequential data.
>
> Thanks in advance!

We do exactly this, at similar frequencies. Our approach has a bit of boilerplate code, but it works well. AFAIK you can't use the OCL::Reporting for this, as it has all kinds of limitations.

1) Create a component that samples the data of interest, and then logs it through the OCL Logging framework (ie real-time logging)
2) Coordinate calling this component _after_ all your data has been computed on a cycle (at that point, your component samples everything, and then logs it)
3) Connect a File/Socket/etc Appender to your logging category, and use that to store/transport your data.

You'll have to make sure that your TLSF memory pool is of sufficient size, and that your logging buffers can absorb the difference in production rates of the real-time code, and the consumption rate of your (likely non-real-time) appending code. For desktop systems with oodles of RAM, this is no problem. For embedded units, this can take quite some tweaking.

The boilerplate stuff is in all 1) for us, as it knows about the samples and data types of interest. I'm sure that more enterprising souls can generalize this solution …

HTH
S