> Logs are independent. When you can not store every event, you can drop them randomly. You lose a perfect view of every logged event, but you still retain a statistical view. As we have already assumed you can not log everything, this is the best you can do anyways.
I think one of the google SRE books mentioned why you don't want to simply randomly sample events. If you're do something like storing http events and logging 1 event per 1000 and you have a small burst of errors from some service failing, you'd potentially miss all or most of them. For certain types of events you want a random sample per [endpoint/status code/whatever], potentially at different rates.. .1% of 200 responses, but 1% of 500 errors.
You are correct. I was being a little lazy with my terminology.
You can choose however you want to sample, random just being the simplest and most easily automated mechanism. The key is that you degrade from a perfect record to a statistical record.
Traces should degrade from a perfect record to a statistical record of correlated events (whole traces).
Metrics are “inherently degraded” from a perfect record to a aggregate record for situations where you know you can not have a perfect record, but you want information (even if low precision) on every event.
Yeah, I used to run a service that got a pretty constant 20,000 qps and traces pretty much never showed me anything about the really weird issues. It is nice when you can delay the sampling decision to much later.
(I use zap for logs now and like the sampling algorithm; only duplicate log lines are suppressed, based on the message text but not the structured field values. So if you log "http request finished ok" and "http request errored", then you get a good sampling of both event types. Not distributed, of course, and no guarantee that the same x-request-id will be sampled in other systems.)
I think one of the google SRE books mentioned why you don't want to simply randomly sample events. If you're do something like storing http events and logging 1 event per 1000 and you have a small burst of errors from some service failing, you'd potentially miss all or most of them. For certain types of events you want a random sample per [endpoint/status code/whatever], potentially at different rates.. .1% of 200 responses, but 1% of 500 errors.