Hacker News new | past | comments | ask | show | jobs | submit login

My favorite logging trick that walks the line between “not too much” and “not too little”: a ring buffer.

Put highly detailed log messages into an in-memory ring buffer, and when you encounter an error, log out the entire contents of the ring buffer. That way you get detailed information when things go wrong, an inspectable-while-debugging record of actions, all while not overwhelming your log aggregation system with useless information.




That's good for debugging, not so good for analytics


This article is about debug logging.

Debugging is also the primary use case for logging program internals.

Analytics should be done on non-internal logs (such as traffic logs), unless you're just logging everything to the same destination without any structure, which is probably a bad idea.


Using analytics generated from logs is also problematic, because if the operations team decides a service is emitting too many logs, they will start pruning them, and not always from the aggregations service.

Analytics instead should be a separate stream in most cases.


Yup, I have a demo app showing dumping from ringbuffer to JDBC here:

https://terse-logback-showcase.herokuapp.com/


One disadvantage of that - though it only matters in some contexts - is that you pay the cost of formatting all those log messages (or at least copying enough data to be able to format them later) even if they're never read.


That depends on where the ringbuffer is placed.

If you're placing the ringbuffer inside an appender, then the appender's encoder will format. However, if the ringbuffer itself is an appender, then you can store the logging events as is until it's time to dump them -- this is the approach in terse-logback [1]

Another option is to put the ringbuffer BEFORE sending it to the logging framework, so it's never processed by SLF4J at all. This means that it's more accessible to the application, at the cost of losing the initial timestamp of the logging event. This is the approach taken in Blindsight. [2]

Shout out to Bunyan, which is (AFAIK) the only server side logging framework that has ringbuffers out of the box. [3]

[1] https://tersesystems.github.io/terse-logback/guide/ringbuffe...

[2] https://tersesystems.github.io/blindsight/usage/buffer.html

[3] https://github.com/trentm/node-bunyan#raw--ringbuffer-stream


The cost of writing data to memory is incredibly small; you'd have to be working on something like gaming engines for it to even be noticeable.

Logging can also be done in a separate thread, reducing even that exceptionally minor cost.


Can confirm -- writing data to memory is incredibly fast. It's faster to use a lock-free queue and use CAS to insert data though:

http://psy-lob-saw.blogspot.com/2016/12/linked-array-queues-...

I've benchmarked code myself, and the flamegraphs will barely show the SpscArrayQueue in the profile.


Interesting. Never heard of this before.

Any examples you can provide?


Here's some examples I found pretty quickly:

https://www.ibm.com/developerworks/aix/library/au-buffer/ind...

http://www.exampler.com/writing/ring-buffer.pdf

A google search for "logging to a ring buffer" will provide more.


Thank you, very useful





Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: