A few years ago I joined a company aggressively trying to reduce their AWS costs. My jaw got the floor when I realized they were spending over a million a month in AWS fees. I couldn't understand how they got there with what they were actually doing. I feel like this comment perfectly demonstrates how that happens.
AWS also purposefully makes it easy to shoot yourself in the foot. Case in point that we were burned on recently:
- set up some service that talks to a s3 bucket
- set up that bucket in the same region/datacenter
- send a decent but not insane amount of traffic through there (several hundred Gb per day)
- assume that you won’t get billed any data transfer fees since you’re talking to a bucket in the same data center
- receive massive bill under “EC2-Other” line item for NAT data transfer fees
- realize that AWS routes all traffic through NAT gateway by default even though it’s just turning around and going back into the data center it came from and billing exorbitant fees for that
- come to the conclusion that this is obviously a racket designed to extract money from unsuspecting people because there is almost no situation where you would want to do that by default and discover that hundreds to thousands of other people have been screwed in the exact same way for years (and there is a documented trail of it[1])
Developer will do the simplest thing to solve the problem.
If the solutions are:
* rewrite that part to add retention, or use better compression, or spend next month deciding which data to keep and which can be removed early
* Wiggle a thing in panel/API giving it more space
The second will win every single time unless there is pushback or it hits the 5% of the developers that actually care to make good architecture not just deliver tickets.
They're pricing hot storage at $50/TB (not per month). That is definitely not AWS or anything like it.
On a per-month basis, the grossly exaggerated number is in the single thousands. The non-exaggerated number is down in the double digits.
$50/TB is a lowball if you want much of the data to be on SSDs, but taking an analysis server and stuffing in 20TB of SSD (plus RAID, plus room for growth) is a very small cost compared to repeated debugging sessions. Especially because the SSD has to deal with about 0.01 DWPD.
