The network is not reliable, but usually the cost of manually fixing problems arising from infrequent types of instability is less than the cost of pre-emptively addressing the issue.
As a practical example, our preferred HA solution for MySQL replication has effectively no network partition safety - if a network becomes partitioned, we'll end up with split brain. However, we have not once had to deal with this specific problem in our years of operation on hundreds of servers.
That said, do make the assumption that your AWS instances will be unable to reach each other for 10+ seconds on a frequent basis. Your life will be happier if you've already planned for that.
> That said, do make the assumption that your AWS instances will be unable to reach each other for 10+ seconds on a frequent basis. Your life will be happier if you've already planned for that.
This was the biggest shock for me when I first moved an Akka cluster into production on EC2. Running with just the default settings we routinely saw our Akka nodes marked as unreachable by the rest of the cluster due to EC2 network noise. We wound up pushing our launch back in order to fix the issue because we just couldn't stay online in a predictable fashion. (The issue was compounded by some other problems though, wasn't all AWS's fault)
I think there's an important truth to learn from your comment: economics actually guides better approaches to development and design and helps to avoid premature/unnecessary optimization. There are probably many great solutions that can be invented, but unless you start thinking about economics of them, you're not working on making them truly remarkable.
Isn't NAT anti-economics? Sure, it helps reuse a limited set of IP addresses, but it also introduces unnecessary layers which doesn't seem cost-effective beyond a certain scale.
I think it does matter, but it's good to be realistic about it.
If the availability of your application over the internet determines your revenue flow, you probably do want to try your best to make it as reliable as practical. Creating a mesh network to serve a large website is overkill, cost-wise. So you do what you can, and then you implement things like continuous integration, monitoring, trending, alerting and escalation procedures to be ready for the eventual failure.
In my experience, relying on a service provider designed to increase reliability (let's say Akamai) will smooth out the great majority of problems a live website might have with availability. One really reliable datacenter will keep the big problems at bay, leaving you to get good at iterating over common minor issues like maintenance and local performance issues.
Great article. A lot of engineers don't have personal experience with these kinds of network failures, so sharing stories of their consequences means more engineers can make informed (and conscious) decisions of how much risk can be tolerated for their applications.
One thing that you could gleam for this article-and I think that this is incorrect-is that the application or operations engineer is responsible for understanding the nuances of distributed systems. In my experience the number of people who are relying on distributed systems is much larger than the number of people who understand these issues.
So what we really need are systems we can build on whose developers understand how to build (and test!) the nuances of data convergence, consensus algorithms, split-blain avoidance, etc. We need systems to gracefully-and automatically-deal with and recover from network failures.
I feel like the authors (or someone else) can do a lot more justice to their overall objective (i.e. tease out patterns) by applying some kind of a qualitative content analysis of case studies [0].
As a practical example, our preferred HA solution for MySQL replication has effectively no network partition safety - if a network becomes partitioned, we'll end up with split brain. However, we have not once had to deal with this specific problem in our years of operation on hundreds of servers.
That said, do make the assumption that your AWS instances will be unable to reach each other for 10+ seconds on a frequent basis. Your life will be happier if you've already planned for that.