Sounds like your server may have been swapping? Disk IO is notoriously bad on most VPS so make sure to disable swap and setup oomkiller. In the case of a small VPS, containerization overhead is also not negligible: you may consider to setup everything natively (to your VM) to counteract that.
