
Debugging a TCP socket leak in a Kubernetes cluster - alberteinstein
https://blog.hasura.io/debugging-tcp-socket-leak-in-a-kubernetes-cluster-99171d3e654b
======
mbrumlow
I have read this near four times now. I can't really find any sustenance -- it
appears to be a advertisement for a product wrapped in a unsolved issue.

No mention of lsof, netstat, or tcpdump, the normal tools used for
troubleshooting these sort of problems. Without trying to sound to snarky I
find it highly concerning that the industry is now working with tools like
docker and Kubernties and we some how just throw out the fact that these sit
on top of Linux.

Not to mention kubelet's ability to spot one of many turntables reaching a max
still would have not solved this problem -- "Fundamentally, the node was
unhealthy" \-- is not a proper answer to the problem -- what was done to
resolve the memory issue is. That could be increasing the tcp_mem to to
support the workload, or finding a faulty user space program who is acting
faulty -- all of which we have no clue because no real tools for
troubleshooting this were used.

I mainly write this gripe because this appears to be a problemtisement, or a
blogtisement. A "helpful" but not informative blog simply to provide a way to
advertise your companies service at as the final blurb, leaving us with no
real solution, resolution or a closing to the mystery of why tcp_mem was
higher than expected.

~~~
kemcho
> kubelets ability to spot one of many turntables reaching a max...

Hm...how do you propose kubernetes / kubernetes users solve these kinds of
problems? It could be a fairly common error that’s hard to catch on a system
of large number of nodes where you’re not supposed to actively think about the
fact that you have nodes. What’s the right tooling / monitoring to have on a
system of 20nodes where one node is basically screwed?

These kinds of things make me think the entire K8s/container abstraction is
just broken.

~~~
pas
Shouldn't this be a Linux (per process or per namespace) feature to limit
resources available to userspace?

------
kronin
Seems that metrics providing visibility into the "network connectivity was
flaky", like looking at response times (particularly 95/99 percentile) and
digging into the pod, which gives you the node, would have isolated the
problem pretty quickly to a single node. If a problem is isolated to a node,
first thing to look at would be node logs. Would that pattern not have worked
in this case?

------
Thaxll
Checking the logs should be on everyone mind when dealing with issues.

~~~
alberteinstein
Indeed! And when the logs show nothing being wrong? All k8s components were
reporting that everything is fine.

~~~
Thaxll
Because not all problem are userspace related, it's very common on Linux to
check for kernel logs. Especially when you know how Kubernetes deals with
networking ( using iptables ect ... )

