The first step of getting decent latency out of modern computers is to build a pile of cash that's taller than you then set it on fire.
They are optimized for throughput, not latency. If you want to do things in microseconds instead of anything from milliseconds to seconds you look at all the lovely code everyone has written in the last 40 years then throw it on top of the burning pile of cash.
I work in an area where microseconds matter and have to write everything from scratch in kernel mode C because nothing else cuts it.
This is my experience as well: almost no existing software is written with ultra low latency in mind (it's optimized for throughput instead).
Getting latency in the micros with low jitter is extremely difficult and requires a bunch of custom software. And getting into the nanos requires a bunch of custom hardware.
The cost of writing software isn't as horrible as the cost of colocating in the various financial exchanges. The market data feed costs are massive and you really need to make a significant amount of money trading just to cover the infrastructure costs.
Yes, the cost is very high. But the cost of an good software engineer at Google, Facebook or another top tier firm is also very high.
However, when Google buys a WAN link from NYC to CHI, they pay $XX dollars because they don't care about the latency too much. When a HFT buys that same link, they'll spend three times that much just to get a link that is a few milliseconds faster.
Also, the colocation costs at the major exchanges (NASDAQ, NYSE, CBOE) are insane. A cabinet in one of those data centers can be 3x as much as a cabinet somewhere else.
I guess what I'm saying is there are a lot things in the HFT world where you have to spend a lot more money than you would if you were doing similar amounts of work at any other tech firm.
Making things fast though is a whole other world.