I was mainly worried I'd be spending time downloading noveau/radeon drivers and associated packages on a server with no attached GPUs. I've been leaning towards languages that compile fat binaries (and running with docker regardless), so this is why I'm a little wary of Ubuntu bringing too much to the table. Also, it's been a long time since I ran Ubuntu on a personal machine, I am still a little worried about the risky the dist-upgrade process can be.
Basically all I feel I need is ufw, docker, ssh and was worried that Ubuntu brings too much along for the ride.
People also get overly bent out of shape for having Amazon search integration with the desktop at one point, which I did not like, but it had a clear, functional way to disable that function.
At least, those are my theories.