The Linux kernel has come a long way in the last few years with improvements to SMP scaling of sockets, zero copy, and large numbers of file handles; make use of it! The level of technical skill being applied to fix kernel problems is probably more expensive than you can afford.
In general doing this "properly" is an exercise in icky compromise.
When 1U commodity x86 server built on C602 Intel chipset can be packed with 768 GB RAM..
If that's the case, then yeah, 10 million connections would require a 14.3 terabytes/sec connection, which doesn't exist for regular PCs, as far as I know.
There are many applications when most of your connections are sitting idly and checking something like memcache for status updates once a second.