D: Connecting to DB
D: Reading config file
D: Connecting to API server
D: Sending updated record
I: Updated user record for $USER
E: Failed to connect to API server; could not update user record. [IP=..., ErrNo=..]
syslog() and setlogmask() being the obvious examples.
Many (if not all) companies I have worked out filtered out everything DEBUG at compile time to improve performance in production as well.
The best option I've seen is putting everything in DEBUG or lower into a memory ring buffer per process that's dumped and collated with other logs at each ERROR/FATAL log.
Does anyone know of an implementation for this in any of the Java logging frameworks?
In my web app if nothing unexpected happens, only INFO/LOG level stuff is pushed to logs. If, however, I get an exception, I will instead print out all the possible traces. I.e. I always store all the logs in-memory and choose what should be printed out based on the success/failure of the request.
Now, of course, this is just a web API running in AWS Lambda, and I don't have to care overly much about the machine just silently dying mid-request, so this might not work for some of you, but this works great for my use-case and I'm sure it will be enough for a lot of people out there.
Or, to flip that around, if you take a program that produces a manageable amount of INFO logs, and change some of those INFOs to DEBUGs, how does that suddenly become unmanageable?
My experience has been that 1 customer-facing byte tends to generate something like ~10 DEBUG-level telemetry bytes. This level of request amplification can't be feasibly sustained at nontrivial request volumes, your logging infrastructure would dwarf your production infrastructure.
In a case where there are plenty of resources and the thing you are logging is very heavy then even verbose logging is trivial.
If you are beset by failures it is much cheaper than making your developers guess.
I've spent way too many hours adjusting log levels on a client environment and trying to replicate the issue without breaking or losing data while doing it.