My favorite logging trick that walks the line between “not too much” and “not too little”: a ring buffer.
Put highly detailed log messages into an in-memory ring buffer, and when you encounter an error, log out the entire contents of the ring buffer. That way you get detailed information when things go wrong, an inspectable-while-debugging record of actions, all while not overwhelming your log aggregation system with useless information.
Debugging is also the primary use case for logging program internals.
Analytics should be done on non-internal logs (such as traffic logs), unless you're just logging everything to the same destination without any structure, which is probably a bad idea.
Using analytics generated from logs is also problematic, because if the operations team decides a service is emitting too many logs, they will start pruning them, and not always from the aggregations service.
Analytics instead should be a separate stream in most cases.
One disadvantage of that - though it only matters in some contexts - is that you pay the cost of formatting all those log messages (or at least copying enough data to be able to format them later) even if they're never read.
If you're placing the ringbuffer inside an appender, then the appender's encoder will format. However, if the ringbuffer itself is an appender, then you can store the logging events as is until it's time to dump them -- this is the approach in terse-logback [1]
Another option is to put the ringbuffer BEFORE sending it to the logging framework, so it's never processed by SLF4J at all. This means that it's more accessible to the application, at the cost of losing the initial timestamp of the logging event. This is the approach taken in Blindsight. [2]
Shout out to Bunyan, which is (AFAIK) the only server side logging framework that has ringbuffers out of the box. [3]
Put highly detailed log messages into an in-memory ring buffer, and when you encounter an error, log out the entire contents of the ring buffer. That way you get detailed information when things go wrong, an inspectable-while-debugging record of actions, all while not overwhelming your log aggregation system with useless information.