- Greedy routing theory i.e. protocol design that is robust in a Nash equilibria sense. It still amazes me how many distributed systems are built on protocol designs that are provably unstable and inefficient.
- Space decomposition data structures. The only class of data structure that is efficiently distributable at massive scales due to their mathematical relationship to space-filling curves. Hash tables are often inadvertently implemented as space decomposition structures but there is a much richer and more expressive universe of data structures that exist.
- Game theoretic schedule awareness. This is a concept sometimes used in massively parallel systems for HPC but distributed systems designers seem to be unaware of it. It allows extremely high throughput by eliminating much of the need for distributed coordination with respect to accessing shared resources because every process can dynamically schedule its operations based on its modeling of the decisions of other schedulers it interacts with such that it almost never conflicts. I've never seen a paper on it but people have designed systems based on it (hell, I've designed a number of massively parallel systems based on it).
- Practical topology. On the surface it is about understanding how to maximize the throughput of a fabric of switch fabrics and most messaging patterns used in distributed systems today are naive at an elementary level; HPC has a much better understanding of packet flow optimization. There is also the more theoretical algebraic topology that hints how you efficiently do computational operations over space decomposition structures (mentioned above) with minimal data motion. Algebraic topology is enormously relevant to massively distributed and parallel databases but I rarely meet people who understand it outside of Oracle and similar (not that most of them are using it well).
A lot of distributed systems were engineered without any awareness of these things. Most of their fundamental weaknesses follow from that. The kinds of distributed systems that you can design if you really know the above topics greatly exceed the capabilities of popular distributed systems used today.
To reduce it to a simple metric: if you can show me a competitive algorithm design for the Graph500 benchmark, you actually understand massively scalable distributed systems. The efficiency at scale demonstrated in that benchmark is so far beyond common distributed systems because the designers of the top entries actually understand the above points I raised.
Don't get me wrong, it's good to point out the writing's shortcomings, but as you appear to be knownledgeable in the field, please don't force others to begin from the bottom. Share your knownledge.
* A low level stack that bypasses the OS for at least disk IO (supposed to provide a 2-3x throughput increase)
* "polymorphic" space-filling curves for distributed parallel indexing
* hyper-dimensional spatial sieves 
* Allen’s Interval Algebra to parallelise SQL statements
More discussion on some of the OP's previous hacker news posts .
 - http://spacecurve.com/
 - http://www.it-director.com/blogs/Bloor_IM_Blog/2014/7/how-do...
 - http://www.google.com/patents/US20090182837
 - https://news.ycombinator.com/item?id=7482151
Greedy routing theory does have a lot of literature around it because it is used a lot in Layer-2 and Layer-3 packet routing protocols to optimize aggregate throughput of inherently decentralized systems. A lot of the robustness of modern IP networks are explained by this. However, above Layer-3 and particularly at the application level you almost never see properly designed distributed protocols; the people that design L2 routing protocols are not the people that design distributed systems and the knowledge is not transferred. (Admittedly this is a difficult mathematical area to understand with a lot of open problems. I know just enough to make good design decision but otherwise do not understand the underlying math.)
I've stated many times that the HPC algorithm people could learn a lot from the people that design distributed databases and that distributed databases theory can learn a lot from the people that design massively parallel HPC algorithms. As far as I can tell, those two groups of people do not talk to each other. I just happen to have done considerable R&D in both fields so I see what people in both domains are missing.
I've recently been tasked with writing about some of these topics, which should be interesting. My intent was not to bash or gloat but to highlight the point that we can do much better than we (or even I) are currently doing with distributed systems.
Edit : https://www.youtube.com/playlist?list=PL700757A5D4B3F368
Also, if you're interested in cryptocurrencies or distributed systems with "greedy" participants, that's another class of problem that goes a level beyond byzantine consensus. The difference is that you cannot assume that even the "good" participants are running the officially sanctioned software, but rather you must assume that there may be colluders who attempt to game the network.
Byzantine generals problem + game theory -> ?
A question to the HN folk - For a programmer who spends his day time writing web applications and no systems experience, are there any good project ideas you guys have to better understand Distributed systems? I was thinking of implementing Paxos in Go / Scala but something more practical would be better.
Original Link: http://css.csail.mit.edu/6.824/2014/labs/lab-1.html
HN comments: https://news.ycombinator.com/item?id=5192650
Another good reference for understanding distributed databases is: http://cs-www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf
(EDIT: I guess what I'm trying to get at is that it would be nice to read a detailed post about how someone who maintains a big distributed system handled a major troubleshooting or scaling problem and how, if any, distributed systems principle or theory came handy in that exercise.)
"The Practice of Cloud System Administration: Designing and Operating Large Distributed Systems", coming out next month, provides a good viewpoint on how distributed systems design knowledge is useful to someone in a sysadmin role. (I've read a preview via Safari Online and found it an excellent resource.)