I dont understand. I am saying that Terracotta, which is an open source product, handles the in memory table and synching stuff I was talking about and that you asked about. Its like you dont have to write a C compiler and you dont need to write linux because someone has done the heavy lifting for us. So as far as that specific issue my point was just that the solution is available in an excellent well tested product that lots of folks are using. By the way, I am actually working on exactly these sets of issues -- not to say there are not bigger minds than mine working on high velocity apps, but it is what I spend most of my time on.
When you talk about such an app it's a good idea to have some basic numbers in mind like .5 second average time to respond to an incoming message and 5seconds worst case. (Picked from thin air but I tend to stick with 1x average 3-10x worst case.)
Now as you scale to 50 or so servers they will start to fail with some regularity and you can't reboot and or rebuild in five seconds so we need to have redundant systems ready. Splitting a shard probably takes longer than five seconds so you need to be proactive about such things. So we need an active rebalancing between shard so they can quickly pass around their load and fast failover incase things fail. These are DB problems but they are only problems because they system need to keep responding quickly as parts of the DB fail.
But this is still DB centric how do you load balance incoming and outgoing requests? (Routing each existing user to the same system every time and letting each system lock data when they want to change their password etc.) How do we recover from data corruption? How do we notice an outgoing or incoming mail server is down? You also have n^2 number of connections between systems as you grow can we keep that many sockets open? (More of an issue is we are segregating the network.) Etc.
PS: These are not hard problems by themselves but they are linked. And there are a lot more where they came from.
Right. Just because we didn't write it doesn't mean it's magic.
There is still a lot of work to be done. It can get very complicated for high velocity apps. And it takes time.
So you're right. Bigger minds than the two of us are probably struggling with this (and lots of other stuff) as we write.