On our side, we have redesigned all search algorithms and data-structures from scratch to run on a mobile devices. We are for example able to search all cities from Geonames (> 3M) on a old iPhone 3GS in less than 50ms (including prefix search & typo tolerance) with only 1KB or RAM and a flash disk. You can't do that with Lucene. So I let you imagine what we can do with a high-end server !
For the High-availability & Consensus, this is something complex and we have spent a lot of time to test it by killing machines of a cluster, and we still continue to test it all the days since every deployment kill index builder process !
I would think that each cluster has the whole index, so any cluster can be used for reads.
I would also think they have multiple such triple-clusters; one triple for each important customer.
(It is difficult to know if a (logical) node is actually a cluster or not.)
Everything is a balance between CPU/RAM/Disk. Having one of this resource that is far superior than the other would be a waste of resource. Our balance is globally to have about 16 cores/32threads per host with latest XEON >= 3.5Ghz, 256G of RAM and about 1TB of SSD (2X512G)
We perform a lot of IO, so the compromise is between price and number of IOPS we can get. Having the same IOPS than our two SSD in raid0, would require a lot of consumer disk, which leads to two problems:
1) We would have too many disk space and we will not have enough CPU to use all this space
2) It would not fit in a 1U server and would increase a lot the price of the server
1. more servers with less capacity (CPU,RAM,Disk) each
2. same number of servers, but six + Hard Drive Raid0 in each
Both of which could be better for other apps but include their own set of problems and, as you pointed out, could result in higher costs.
I think I was misunderstood. I was not questioning the architecture chosen, sorry if I sounded that way. I want to learn why was it chosen since I don't see many like it. So far you have provided me with some very satisfying answers, and for that I thank you.