Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
[dead]
4 months ago


One of the blog post's authors here. As part of our infra work migrating from SLURM to Kubernetes to unlock online RL, we chose SkyPilot to orchestrate training jobs. That's when we started seeing model-loading OOM on some nodes, even though the same workloads had been running fine on SLURM. On all k8s idle nodes, nvidia-smi was showing 80GB of VRAM usage, with no process to kill. The investigation trail went nvidia-smi → stuck docker containers → D-state threads → wchan → the FUSE control filesystem → all this pointing to a single leaked file descriptor in SkyPilot's privileged fusermount helper.

Indeed, when you pass an fd over a Unix socket, the kernel duplicates the reference instead of simply moving it. So what really happened is that the fusermount helper handed the /dev/fuse file description to the user pod and kept its own copy open ever after the user pod died. Therefore, the connection's refcount never hit zero, fuse_abort_conn never fired, and dead training jobs stayed parked in request_wait_answer holding their CUDA contexts forever. The fix ended up being ~12 lines of Go (defer syscall.Close(fd)), which is now upstream.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: