Problem is memory locality, or lack thereof. Having to chase pointers around and not having things laid out in sequence doesn't play well with the CPU cache.
I'm not arguing against value types as I definitely want those too but it's not that simple nowadays - a moving garbage collector (like all modern server jvm's have) can do things for locality that non-moving systems (gc'd or not) can't - https://shipilev.net/jvm-anatomy-park/11-moving-gc-locality/
That only applies in a few specific cases. Basically, a linked list or an array of objects containing no further references. Otherwise, those internal references get immediately followed and significantly dilute the locality.
There have been garbage collectors designed to improve locality of predictable accesses, but they are not the norm.
