On top of all that it's pretty standard (at least in the VMware world) to store the OS on an SD card. So the OS has to be read into memory and ESX is kinda slow to boot even if installed to a disk.
The reason it takes this long is that it does a bunch of self tests, and then it has to load all the ROMs for the components (NICs, HBAs, etc.) which often trigger messages like "X Loaded, press CTRL+L to configure" which stay on screen for 5-10 seconds each.
If you yield control to some kind of Broadcom controller it'll do all kinds of shit before giving up and handing you back to boot a disk.
Servers are built to be reliable. It is better for them to boot slowly and correctly than quickly with silent memory failures.