Have a look at what the financial industry use, they've been tackling high volume low-latency realtime data feed analysis problems for the last couple of decades. Time series databases (such as kx/kdb) are popular as are propriety in-memory databases (often based on something like BerkleyDB).
