On: How Kubernetes Probes Work

[jaggederest]

That's a problem for circuitbreakers on these kinds of actions, not lying on health checks.

Something like healthcheck fails -> restart -> healthcheck fails -> restart -> healthcheck fails -> circuit breaker trip, alarm raised, give up until manual intervention or X minutes have passed

[deathanatos → jaggederest]

That circuitbreaker exists, by default. It is "CrashloopBackoff", here, and TFA covers it. (& it's an "until X minutes have passed" kind, by default.)

[dilyevsky → deathanatos]

backoff is only applied to individual pods/containers not across pods. the point is at scale it's easy to get into a situation where it's not possible to recover without (usually manual) full service drain

[jaggederest → dilyevsky]

Yeah that's why I have a manual intervention breaker that goes across all the pods/nodes etc. CrashLoopBackoff is great for selfhealing but when things go really pear shaped you want something that catches the global state. Saw it activate during an AWS outage one time where new nodes were unhealthy on start, for example.