
Volume Monitoring in Kubernetes with Prometheus - qubit23
https://blog.argoproj.io/volume-monitoring-in-kubernetes-with-prometheus-3a185e4c4035
======
lobster_johnson
Wouldn't it be better to run node_exporter (with only the fs metrics enabled)
in a sidecar container in the pod that mounts the volume? That way, you can
get the metrics scraped with the right pod labels. If you do it by node, you
will only get the node name as a label.

~~~
ibotty
Yes. There is a disadvantage, that when your app already exposes metrics, you
will have to tell prometheus' service discovery that you want to scrape two
ports.

~~~
lobster_johnson
Well, you can easily set up Prometheus with a rule that automatically scrapes
any discovered service (or pod) where the port name is =~
/^(exporter|.*-exporter)$/, for example. This allows you to name your port
just "exporter", or you can have multiple ports named things like "volume-
exporter".

------
peachia
kubelet v1.8 collects volume stats, the proposal linked in the summary was
forked, and the fork accepted & implemented:
[https://github.com/kubernetes/community/pull/930](https://github.com/kubernetes/community/pull/930)

------
discordianfish
Hi, node exporter maintainer here.

Yes, running the node exporter in a container is still tricky which is why we
recommend running it straight on the host. I like the nsenter idea, wondering
if it could be integrated without shelling out but I think there were some
inherit problems with Goroutines and nsenter preventing that. If your
interested in exploring this, feel free to open an github issues. Even better
though would be support of unisolated workloads in kubernetes, similar to rkt
stage1.

------
bogomipz
>" In this post, we discussed monitoring volumes on Kubernetes using
Prometheus, node-exporter, and nsenter. One disadvantage of this approach is
that node-exporter must be run as a privileged container. In the future, it
may be better to have Kubelet collect the volume stats so that we can get the
stats directly from Kubernetes instead of running node-exporter as a
privileged container."

I thought that node-exporter was meant to run in a container or on the host
itself. Are there any potential downside to the single node-exporter in a side
car approach?

------
alexnewman
Now I just need a flexdriver which receives the warning and activates the code
to create a new ebs volume and adds it to whatever lm k8s volume. EBS gp2 is
uselessly slow without just combining them into one super danger mount. As
with everything else in amazon, never use one of x. Whenever I use S3 and EBS
i put an abstraction between me and them because amazon.

