From what I've heard it's the high speed/throughput low latency interconnect that's the differentiator compared to thousands of commodity machines. (Weather and bomb explosions being two big ones that are hard to do concurrently, as the results depend on each other)
Depends: if your application has strong scaling then just throw more power whichever way at it.
Tightly coupled hpc systems aim to improve execution speed (or application resolution) to weak scaling applications.
