On: Mapping Out the HPC Dependency Chaos

[MontyCarloHall]

Why not just containerize your software and run the container on the HPC cluster?

[bayindirh → MontyCarloHall]

Disclosure: I’m an HPC admin.

Accessing InfiniBand and GPUs directly become a problem.

Rootless containers are also still at infancy, and creates a lot of problems. You don’t want to give indirect root access via docker group, too.

[maxwell86 → bayindirh]

> Accessing InfiniBand and GPUs directly become a problem.

I use nvidia containers on HPC systems every day and accessing NICs, doing RDMA to GPUs, etc. "just works" and performs as well as baremetal. Every time we upgrade our container we verify the new container with a set of benchmarks against both the old one and baremetal.

> You don’t want to give indirect root access via docker group, too.

I don't know of any HPC center using docker though. It does not sound like a good idea because the docker daemon runs as root..

[throw0101c → MontyCarloHall]

> Why not just containerize your software and run the container on the HPC cluster?

Docker needs root access which is a big no-no in multi-user environments.

Singularity/Apptainer was developed (with HPC in mind) so that non-admin users could run containerized workloads, and Spack supports creating such workloads:

* https://spack.readthedocs.io/en/latest/containers.html

[bayindirh → throw0101c]

Singularity and Apptainer are not silver bullets tho. Both have their limitations and own problems to solve.

They will mature eventually, but bare metal is still the king in HPC realm.

[throw0101c → bayindirh]

We have bare metal HPC nodes, but some users want to run containerized apps for their batch jobs: it's not either-or, it's both-and.

[bayindirh → throw0101c]

Of course, it’s same for us. I just wanted to point out that these technologies cannot replace bare metal installations for every case as of today.