Logging redux

I'm a newbie here, be gentle ;-) Bringing out an offline discussion with Peter:

>>> As you might have read, there is a patch in the pipeline to get this >>> working and to replace the current logger. >> >> Yeah, I've read that entire dev email chain twice now ;-) I think it's >> all good, but until it supports hard real-time (and *embedded* hard >> real-time) its almost a hindrance (since we'll end up with two logs, >> yours and ours).  And malloc, sigh ... well, see next. >

> A failure of an allocation in a logger would mean that a single line > of log content > would go missing. Nothing disastrous here.

Actually I come from a different world. The humble log is the starting point for any post mortem, gets emailed (with other diagnostic info), and has the potential of becoming a legal document in case of a disaster.

In the past I've had any problems in logging be a system fatal error. At the very least, when there are dropped messages, I want an error message (in the log ;-) on recovery saying how many messages were dropped.

- alexis

Logging redux

On Sat, Oct 24, 2009 at 00:35, <alexis [..] ...> wrote:
> I'm a newbie here, be gentle ;-)
> Bringing out an offline discussion with Peter:
>
>>>> As you might have read, there is a patch in the pipeline to get this
>>>> working and to replace the current logger.
>>>
>>> Yeah, I've read that entire dev email chain twice now ;-) I think it's
>>> all good, but until it supports hard real-time (and *embedded* hard
>>> real-time) its almost a hindrance (since we'll end up with two logs,
>>> yours and ours).  And malloc, sigh ... well, see next.
>>
>> A failure of an allocation in a logger would mean that a single line
>> of log content
>> would go missing. Nothing disastrous here.
>
> Actually I come from a different world.  The humble log is the starting point
> for any post mortem, gets emailed (with other diagnostic info), and has the
> potential of becoming a legal document in case of a disaster.
>
> In the past I've had any problems in logging be a system fatal error.
> At the very least, when there are dropped messages, I want an error message
> (in the log ;-) on recovery saying how many messages were dropped.

Full Ack. Thanks for your input.

Peter

Logging redux

On Oct 23, 2009, at 19:35 , alexis [..] ... wrote:

> I'm a newbie here, be gentle ;-)
> Bringing out an offline discussion with Peter:
>
>>>> As you might have read, there is a patch in the pipeline to get
>>>> this
>>>> working and to replace the current logger.
>>>
>>> Yeah, I've read that entire dev email chain twice now ;-) I think
>>> it's
>>> all good, but until it supports hard real-time (and *embedded* hard
>>> real-time) its almost a hindrance (since we'll end up with two logs,
>>> yours and ours). And malloc, sigh ... well, see next.
>>
>> A failure of an allocation in a logger would mean that a single line
>> of log content
>> would go missing. Nothing disastrous here.
>
> Actually I come from a different world. The humble log is the
> starting point
> for any post mortem, gets emailed (with other diagnostic info), and
> has the
> potential of becoming a legal document in case of a disaster.

Fair point.

> In the past I've had any problems in logging be a system fatal error.
> At the very least, when there are dropped messages, I want an error
> message
> (in the log ;-) on recovery saying how many messages were dropped.

I can think of two primary locations for trouble 1) allocating space
for the logging event, and 2) pushing onto the buffer connecting a
category to its appenders. They should both be fairly straight forward
to cope with exceptional conditions, and then output data later as to
what happened (e.g. log the number of dropped messages).

Cheers
Stephen

Logging redux

On Sat, 24 Oct 2009, alexis [..] ... wrote:

> I'm a newbie here, be gentle ;-)
> Bringing out an offline discussion with Peter:
>
>>>> As you might have read, there is a patch in the pipeline to get this
>>>> working and to replace the current logger.
>>>
>>> Yeah, I've read that entire dev email chain twice now ;-) I think it's
>>> all good, but until it supports hard real-time (and *embedded* hard
>>> real-time) its almost a hindrance (since we'll end up with two logs,
>>> yours and ours).  And malloc, sigh ... well, see next.
>>
>> A failure of an allocation in a logger would mean that a single line
>> of log content
>> would go missing. Nothing disastrous here.
>
> Actually I come from a different world. The humble log is the starting point
> for any post mortem, gets emailed (with other diagnostic info), and has the
> potential of becoming a legal document in case of a disaster.
>
> In the past I've had any problems in logging be a system fatal error.
> At the very least, when there are dropped messages, I want an error message
> (in the log ;-) on recovery saying how many messages were dropped.
>

If Orocos/RTT wants to hold up its claimed focus to support hard realtime,
I think we can only agree with what Alexis describes... Only pre-allocation
is really hard realtime.

So, non hard-realtime allocation (for messages, logging, whatever,...) _is_
a nice feature to have, but an application builder should be able to
configure it out. Or rather, the other way around: an application builder
that trusts his system with dynamic allocation should explicitly configure
the option in.

Herman

Logging redux

On Oct 25, 2009, at 05:44 , Herman Bruyninckx wrote:

> On Sat, 24 Oct 2009, alexis [..] ... wrote:
>
>> I'm a newbie here, be gentle ;-)
>> Bringing out an offline discussion with Peter:
>>
>>>>> As you might have read, there is a patch in the pipeline to get
>>>>> this
>>>>> working and to replace the current logger.
>>>>
>>>> Yeah, I've read that entire dev email chain twice now ;-) I think
>>>> it's
>>>> all good, but until it supports hard real-time (and *embedded* hard
>>>> real-time) its almost a hindrance (since we'll end up with two
>>>> logs,
>>>> yours and ours). And malloc, sigh ... well, see next.
>>>
>>> A failure of an allocation in a logger would mean that a single line
>>> of log content
>>> would go missing. Nothing disastrous here.
>>
>> Actually I come from a different world. The humble log is the
>> starting point
>> for any post mortem, gets emailed (with other diagnostic info), and
>> has the
>> potential of becoming a legal document in case of a disaster.
>>
>> In the past I've had any problems in logging be a system fatal error.
>> At the very least, when there are dropped messages, I want an error
>> message (in the log ;-) on recovery saying how many messages were
>> dropped.
>>
>
> If Orocos/RTT wants to hold up its claimed focus to support hard
> realtime,
> I think we can only agree with what Alexis describes... Only pre-
> allocation
> is really hard realtime.

Really?! So are you really saying that an O(1) algorithm can't claim
to support hard realtime?

> So, non hard-realtime allocation (for messages, logging,
> whatever,...) _is_
> a nice feature to have, but an application builder should be able to
> configure it out. Or rather, the other way around: an application
> builder
> that trusts his system with dynamic allocation should explicitly
> configure
> the option in.

Quite reasonable.
Stephen

Logging redux

On Sun, 25 Oct 2009, S Roderick wrote:

> On Oct 25, 2009, at 05:44 , Herman Bruyninckx wrote:
>
>> On Sat, 24 Oct 2009, alexis [..] ... wrote:
>>
>>> I'm a newbie here, be gentle ;-)
>>> Bringing out an offline discussion with Peter:
>>>
>>>>>> As you might have read, there is a patch in the pipeline to get
>>>>>> this
>>>>>> working and to replace the current logger.
>>>>>
>>>>> Yeah, I've read that entire dev email chain twice now ;-) I think
>>>>> it's
>>>>> all good, but until it supports hard real-time (and *embedded* hard
>>>>> real-time) its almost a hindrance (since we'll end up with two
>>>>> logs,
>>>>> yours and ours). And malloc, sigh ... well, see next.
>>>>
>>>> A failure of an allocation in a logger would mean that a single line
>>>> of log content
>>>> would go missing. Nothing disastrous here.
>>>
>>> Actually I come from a different world. The humble log is the
>>> starting point
>>> for any post mortem, gets emailed (with other diagnostic info), and
>>> has the
>>> potential of becoming a legal document in case of a disaster.
>>>
>>> In the past I've had any problems in logging be a system fatal error.
>>> At the very least, when there are dropped messages, I want an error
>>> message (in the log ;-) on recovery saying how many messages were
>>> dropped.
>>>
>>
>> If Orocos/RTT wants to hold up its claimed focus to support hard
>> realtime,
>> I think we can only agree with what Alexis describes... Only pre-
>> allocation
>> is really hard realtime.
>
> Really?! So are you really saying that an O(1) algorithm can't claim
> to support hard realtime?

It's not (only) the _speed_ of the algorithm that determines its
realtime-ness, but also the available memory... If that's not available
anymore, it's not available anymore, and there is nothing any dynamic
memory allocation algorithm can do about that.

>> So, non hard-realtime allocation (for messages, logging, whatever,...)
>> _is_ a nice feature to have, but an application builder should be able
>> to configure it out. Or rather, the other way around: an application
>> builder that trusts his system with dynamic allocation should explicitly
>> configure the option in.
>
> Quite reasonable.

And in fact I think everybody involved in the discussion agrees about this :-)

So, I don't think we really need to "fight" about this issue anymore.
(Maybe only about its implementation and the way the configuration is
presented to the application builder.)

Herman