jonnylaw's comments

On: LLM Observability in the Wild – Why OpenTelemetry Should Be the Standard

[jonnylaw]

Attaching to the app is impractical to catch regressions in production. LLMs are probabilistic - this means you can have a regression without even changing the code / making a new deployment.

A metric to alert on could be task-completion rate using LLM as a judge or synthetic tests which are run on a schedule. Then the other metrics you mentioned are useful for debugging the problem.

On: The case for a pipe assignment operator in R

[jonnylaw]

You use a tee https://en.wikipedia.org/wiki/Tee_(command)

[jgalt212 → jonnylaw]

ooh, that's useful. thanks.