The network is not reliable, but usually the cost of manually fixing problems arising from infrequent types of instability is less than the cost of pre-emptively addressing the issue.
As a practical example, our preferred HA solution for MySQL replication has effectively no network partition safety - if a network becomes partitioned, we'll end up with split brain. However, we have not once had to deal with this specific problem in our years of operation on hundreds of servers.
That said, do make the assumption that your AWS instances will be unable to reach each other for 10+ seconds on a frequent basis. Your life will be happier if you've already planned for that.
> That said, do make the assumption that your AWS instances will be unable to reach each other for 10+ seconds on a frequent basis. Your life will be happier if you've already planned for that.
This was the biggest shock for me when I first moved an Akka cluster into production on EC2. Running with just the default settings we routinely saw our Akka nodes marked as unreachable by the rest of the cluster due to EC2 network noise. We wound up pushing our launch back in order to fix the issue because we just couldn't stay online in a predictable fashion. (The issue was compounded by some other problems though, wasn't all AWS's fault)
I think there's an important truth to learn from your comment: economics actually guides better approaches to development and design and helps to avoid premature/unnecessary optimization. There are probably many great solutions that can be invented, but unless you start thinking about economics of them, you're not working on making them truly remarkable.
Isn't NAT anti-economics? Sure, it helps reuse a limited set of IP addresses, but it also introduces unnecessary layers which doesn't seem cost-effective beyond a certain scale.
As a practical example, our preferred HA solution for MySQL replication has effectively no network partition safety - if a network becomes partitioned, we'll end up with split brain. However, we have not once had to deal with this specific problem in our years of operation on hundreds of servers.
That said, do make the assumption that your AWS instances will be unable to reach each other for 10+ seconds on a frequent basis. Your life will be happier if you've already planned for that.