
A Surprise AWS Bill - oaf357
https://chrisshort.net/the-aws-bill-heard-around-the-world/
======
stefan_
That's a whole lot of text for "I had a 14 GiB VM image publicly linked and
people discovered it", and none of it has much to do with AWS or Cloudflare.

Presumably the author would do much better with a VM or something from OVH,
they'll just shut you off or limit you before it becomes a problem (not that
they would care about 30 TiB).

~~~
fxtentacle
I would say the main story here is that AWS overcharges on traffic to such a
ridiculous degree. My SaaS regularly consumes 50+ TB in traffic, but I pay
€150 in monthly fees for that.

For someone coming from my perspective, that would be a huge and unpleasant
surprise to be billed $2600 for a service that I know costs $180 elsewhere.

Edit: especially so because people keep repeating the mantra that one should
use the cloud to save money by not paying for what you don't use. Obviously,
my 3 Xeon servers with 64 GB RAM each are way overpowered for sending out some
GB of static files, but I wanted to have a bit of redundancy. But with my
setup, there should be plenty of obvious inefficiencies for "the cloud" to
eliminate

=> It feels like cloud should be cheaper than my dedicated servers. But it's
not, and that is the unpleasant surprise.

~~~
siscia
Always thought that a great statup idea would be to interact with dedicated
server provider and offer on top of those cheap servers AWS like services.

The most common services like S3, Kafka, RDS (pg and mysql), Redis should be
enough to cover most use cases.

With k8s and a dedicated smart team, it would be a possible adventure.

Moreover it can work great also on-premises. Several old medium business have
their own physical infrastructure and they are not yet ready to move to the
cloud.

~~~
alFReD-NSH
There's a actually a lot of companies doing that. Google 'aws managed
services'

~~~
bigiain
Some of those companies are even more eyewateringly expensive than the story
in the article.

I worked on a project a few years back where the client were paying $10k/month
as a "managed service" fee, on top of about $4k/month worth of platform at
full on-demand prices. I showed the client how they could have it all running
on reserved instances for Prod and spot instances for dev/staging for under
$2k/month - but no, somebody had signed up for $14k+ per month just to have
someone to blame/shout at 24x7 if something went wrong. (And that company was
~85% likely to call me and blame it on the app before they even bothered
looking to see if the platform was working...)

------
dahdum
From the article:

> Cloudflare was the least helpful service I could have imagined given the
> circumstances. A long term user and on and off customer thinks they were
> attacked for two days and you don’t lift a finger?

> File this under, “Things I should’ve known but didn’t.” Did you know that
> “The maximum file size Cloudflare’s CDN caches is 512MB for Free, Pro, and
> Business customers and 5GB for Enterprise customers.” That’s right,
> Cloudflare saw requests for a 13.7 GB file and sent them straight to origin
> every time BY DESIGN.

I don't really see how Cloudflare has much blame here. He's an "on and off
customer" which I'm guessing means currently "off". They only cache a limited
number of file extensions (qcow2 isn't one of them), and it's all documented.

AWS always seems pretty generous in resolving these cases at least.

~~~
ckdarby
I would not be surprised if AWS actually allocates marketing dollars towards
covering bills like this.

In the long term this is a brilliant plan because it helps prevent people from
blacklisting the provider.

Imagine someone gets hit with $3k bill on their personal, feels wronged, goes
to work and makes effort at their employer to move off AWS.

I don't know about most HN readers but I'd probably fall in this category and
past places I've done work for were +$100k/month corporate bills with AWS.

~~~
dencodev
Heroku really screwed me over once and the reps I spoke with were complete and
total assholes and acted like I was in the wrong when it was their system that
totally dropped the ball. I now advocate against them at my workplaces and I
know it's cost them at minimum tens of thousands of dollars, though I get that
isn't much money for them.

~~~
Shank
> Heroku really screwed me over once

How?

~~~
dencodev
There was a fraudulent and malicious DMCA claim against one of my sites.
Heroku without any notice took down ALL of my sites, one of which was a part
of my livelihood. They claimed they emailed me with 24 hour notice but that
absolutely did not happen. I had to pound on their door to make them reverse
it, they victim blamed me the entire time and had a super snarky and shitty
attitude, told me I should have been more proactive about the notice I never
got.

I cannot in good faith ever recommend a service that would take down a website
with zero notice and be so uncooperative in fixing what was ultimately their
fault. It cost me money for that site to be down and I had to cancel my entire
day to deal with them, and if the stakes were bigger it could easily wind up
costing my employer hundreds of thousands of dollars.

------
ta1234567890
Off topic but on the same line. One of the most annoying things about being a
consumer in the US is the ubiquitous unknown-until-last-minute pricing.

You rent a car, you don't know what the total is going to be. You go to the
hospital, you don't know how much you're going to have to pay. You book a
hotel and don't know the total until you check out. You go to a restaurant and
even if you order just one thing and saw the exact price on the menu, that's
not going to be the total. You go to the grocery store, see all the prices on
the items, add them up, and then when you go pay, surprise!

~~~
_spoonman
Have you ever bought a house here? You go through a 1-2 month (minimum)
process of providing financial documents from the past six months or more to
mortgage originators, and no one can tell you how much to bring to closing
until 24 hours before. It’s insane.

~~~
ta1234567890
It is crazy. Not a house, but friend had a baby in SF/Bay Area. Neither the
hospital, doctor or insurance would tell them how much the delivery would
cost, not even an estimate. It took a whole year after their hospital visit to
get the bill, it was $85k for a c-section and 3 days at the hospital.
Fortunately their insurance took care of most of the bill. But can you imagine
getting a surprise bill for $85k a year after the fact? Or not having
insurance? Terrifying!

~~~
fmpwizard
We have two kids (in NC, US)), both were c-sections, we had insurance on the
first one, but because she was born in Feb, we had to pay the deductible
twice, because even though it was the same pregnancy, it was "over two billing
years". And even with insurance we still ended up paying the rest of the bill
over 3 or 4 years. Second kid was with no insurance, I think we'll finish
paying that one when he turns 10 years old. I don't know about other areas in
the US, but around here you can at least setup a payment plan with the
hospital and they are 0% interest rate.

~~~
EE84M3i
Don't get me wrong, I'm not a fan of insurance companies, but they seem pretty
up front with the fact that deductibles are based on date of service rendered.

~~~
jedberg
They are, but it's still dumb. It puts extra demand on the medical system in
December as people try to squeeze in their elective procedures in a year where
they already paid their deductible, and it incentivizes you to _not_ get care
early in the year in the hopes that "it just goes away" and then sometimes it
gets a lot worse and costs a lot more than if you had just gone in the first
place.

Deductibles should be a rolling 12 month bill. If you have something major in
December you should be good until next December. This would eliminate all of
the issues with deductibles rolling over on Jan 1. It would even bring extra
profit to the insurance companies because people might decide to stick with
their provider another year since "I already made my deductible until
November".

~~~
refurb
I narrowly avoided that issue when my kid was born late in the year. But yeah,
you end up paying almost double for the same thing because of the deductible
and out of pocket max resets in the new year.

That said, I was able to take advantage of it by making sure my surgery was
scheduled before end of year. Ended up paying something like $200, rather than
$1500 if it had happened in January.

------
clarkevans
> Now that I’m aware of the 512 MB file limit at Cloudflare, I am moving other
> larger files in that bucket to archive.org for now (and will add them to my
> supported Causes).

...

> I don’t feel like archive.org should be my site’s dumping ground since it
> can turn a profit if it gets popular. archive.org is a stop-gap for two
> files for the time being.

I'm trying to understand... he has decided to burden a charity with his
distribution expenses?

~~~
throwlogon
There's some symmetry to the way the Internet Archive assumes consent to
reproduce other people's content, and people assuming consent to take
advantage of the Internet Archive's mirroring. I don't know if this is in line
with the Internet Archive's terms-of-service, or the law.

In this case they'd be getting stuck with a pretty big bandwidth hit.

------
HugThem
Summary:

He published a 14GB file and one day there were 2700 downloads resulting in
~30 Terrabyte of traffic.

He had the file behind CloudFlare, but since CloudFlare does not cache files
larger then 512MB, all the traffic went to his S3 bucket and Amazon billed him
$2700 for that.

~~~
iakh
I think the most important part was left out: > AWS employee discovered that
3655 partial GETs to the object might have actually been delivered as full
file requests

~~~
agustif
Wonder how many of those happen and Amazon charges their customers...

HAHAHA

------
RKearney
CloudFlare's TOS is clear on using the service to serve up 13.7GB files.

[https://www.cloudflare.com/terms/](https://www.cloudflare.com/terms/)

2.8 Limitation on Serving Non-HTML Content

[...] Use of the Service for serving video (unless purchased separately as a
Paid Service) or a disproportionate percentage of pictures, audio files, or
other non-HTML content, is prohibited.

So 500MB limit or not, the author is already violating CloudFlare's terms of
service.

~~~
gregmac
Even if you read the ToS, do you really expect people to remember this?
CloudFlare blends into the background when setup properly, and you don't have
to think about it. If you upload a big file to your site as part of some other
work -- especially personal -- most people are not running through the ToS of
every service that might be involved thinking about compliance.

Terns of service are also legal "CYA" documents. If CloudFlare was actually
serious about that restriction, there'd be a technical limitation in place
that would, for example, serve a 503.

~~~
markonen
Cloudflare is serious about that restriction. They use it to upsell the
Enterprise plan which doesn’t have it.

------
saddlerustle
Amazon gets away with high bandwidth pricing because almost all their
customers are businesses with high revenue per byte served. If you want to
serve large assets economically you have to look elsewhere.

Bandwidth on Oracle Cloud is $0.0085/GB with the first 10TB free each month,
so this would have cost only $170. Alternatively bandwidth on Backblaze B2
costs $0.01/GB, but is free out to Cloudflare, so this traffic would have been
completely free.

~~~
kitteh
I do recommend you spend some time looking at the quality Oracle provides.
Their regions plug into transit providers (NTT, Level3, etc.) and their
peering footprint is damn near non existent (they seem to be in the process of
trying to fix it). Only reason I bring this up is try to send traffic to an
eye ball network at peak on Oracle vs. Google/Azure/AWS at peak and you can
see the difference in terms of packet loss / throughput. This is because those
eyeball networks you have to be directly connected to since they run their
transit hot at peak.

------
schmichael
The fact that the cloud allows hobbyists, small businesses, and massive
enterprises ~equal access to services is amazing.

However it means sometimes things like this happen where a product’s
incentives (serve any content at any cost) are wildly misaligned with a huge
percentage of users needs (I’d rather my site, or preferably just the costly
resource, be down than pay $2k).

There’s endless tuning non-enterprises can do to get our ideal behavior: but
that’s the difference between pre-cloud and post-cloud computing. It used to
take monumental effort to build high scale high availability systems. Your
$5/mo Dreamhost site would just die under load instead of charging you
thousands. Now enterprise use cases are supported by default and it takes
careful tuning to opt out.

~~~
Slartie
Actually not even the typical $5/month cheap VPS offering would die under the
"load" of 30TB of static content HTTP traffic being served in 2700 requests
over a month. That's just a laughable drain on CPU resources that can easily
be handled by even the cheapest virtual server offering, because it doesn't
even get a single core of a modern CPU into the double digit utilization
percentage range.

The only thing that could happen is they cap your data transfer at some point.
But there are cheap VPS providers out there offering several TB of gigabit
speed traffic and throttling instead of a hard cap when you reach your limit.

~~~
schmichael
Ah sorry, I used Dreamhost as they were one of the biggest shared hosting
providers back in the day. No VPS, just a user account on a big host. They’ve
likely evolved considerably since I last used them 10+ years ago.

Point being their product was targeting me and designed appropriately. I
forget the details but I know there were caps that were ample for my meager
needs but would prevent this sort of accidental overage.

My point is that compute has become a commodity like electricity but without
the built in fuses. My residential box can’t pull industrial amps.

------
wonderlg
I think the main problem is that there isn’t an easy way to set a _hard_
monthly limit on these services.

I use a bunch of “freemium” services like S3 and Google Maps API and I’ve
never paid a penny. I use them _because_ they don’t cost a penny for my very
limited usage, but I’m not looking forward to the day I mistakenly and
disastrously exceed their free tier.

------
jrott
This is the nightmare scenario with a personal AWS account. AWS billing setup
makes it impossible to know that there is a giant bill until after the fact. I
wish there was some way to limit the bill and just have everything shut down
at that point for hobby projects.

~~~
ldoughty
Setting up billing alarms is easy for personal accounts, it's actually only a
nightmare if you go through a reseller (which is common in government
contacting).

The author had an incident starting June 23rd, but didn't know about it until
he got his bill July 7th, that's potentially ~14 days someone could have been
abusing his account. A billing alarm would have reduced this to hours or
minutes.

I would be interested how you suggest stopping a bill before it happens
otherwise.. should AWS disable your website because you got posted on hacker
news and now have a bill over your $10 limit? If AWS needs to stop your
billing at $10, then it might need to shut down your EC2 instance and destroy
your data...

If course they could offer the ability to configure the response, but if
someone doesn't take the 2 minutes necessary to put a $25 billing alarm in
place, what are the chances they will go through the effort of service/object
based abuse policies?

At the end of the day, the issue here was that the user posted something
online that people could abuse.. I don't think any CDNs cover 30 TB in free
tier...

~~~
ghaff
>should AWS disable your website because you got posted on hacker news and now
have a bill over your $10 limit?

Maybe that should be an option?

Probably not a good idea for a business but if you're just using AWS to learn
and otherwise fool around with, there's a good argument that it would be nice
to be able to have a hard circuit breaker for at least stateless services.

I've actually heard people argue that they consider everything they put up on
AWS is ephemeral so a hard circuit breaker should have the option to burn
everything down but that seems like it would create its own set of problems
for many people. Disabling data egress and EC2 seems as if it would go a long
way towards stopping most of the unexpected bill stories.

And, yes, I'm aware of billing alarms and even setting up Lambda functions to
take actions but, especially if you use S3 to host files, it would be nice to
cap expenses at mostly your storage costs. I was doing some research for a
very small non-profit that needs some hosted storage and I think Backblaze B2
is a better choice for them for this reason.

~~~
Turing_Machine
> Maybe that should be an option?

It should absolutely be an option.

Maybe not the default, but an option.

------
social_quotient
AWS is an amazing service and in our past been very accommodating for
surprises as long as it seems we know what we are doing and are going to
mitigate the cause. I’ve rarely seen this allegiance to the “customer” by an
enterprise company. They really do care and they figured out the recipe to
make caring scale.

What’s odd is the touch points are cold. Ticket system support, phone call
back etc. it feels like it’s going to be robotic canned replies but they
figured out a way to make the people on the other side smart enough to
understand the issue, empowered enough to do something about it, empathetic
enough to want to resolve things “fairly”.

~~~
tehlike
That's from Amazon's playbook. You have an item to return, you get a refund,
no fuss. Sometimes they tell you to keep the item too!

~~~
ghaff
The first real issue I ever had was last week though, truth be told, it was
mostly on UPS. (They didn't make a scheduled delivery pickup, then their
tracking system _eventually_ said they picked something up even though they
didn't. Amazon did issue a refund when the item was "picked up" so I assume
it's all on UPS now.)

The problem is that I was utterly unable to talk to a human at UPS. I even
went to a UPS Store but they were powerless to do anything.

The thing is that Amazon's automated chat bots and so forth just kept
referring me back to UPS.

~~~
MertsA
I had this same problem on the other end of things. UPS truck pulled up out
front with my package, package is scanned as "left at front door", walks out
with a couple packages for my neighbors, goes back to the truck and just
drives off (and no, I checked, it wasn't one of the packages he left for the
neighbor). It appears that the driver decided to save some time and scan
packages on the truck as delivered before getting out and then forgot to
actually deliver it once he got back on the truck from the neighbor's house.
Either UPS doesn't have an easy way to undo erroneous delivery notifications
or the driver just decided to dispose of the package when he realized what
happened and had a leftover package, either way, they never attempted to
deliver the package again.

As far as Amazon was considered, the package was delivered and the only option
was to return it. There's tons of options for problems with a package, but
nothing for "UPS stole my package". I guess they want to make it harder for
people to claim that a package was stolen but I can't see why they would go so
far to hide that given how prominent actual package theft is. Eventually after
going through Amazon's support chat one of the responses was just a generic
refund without returning the product but in the end I think that just took
that money straight from the vendor for a package that UPS lost/disposed of
because of a driver cutting corners to save time.

~~~
acdha
The loss rate for UPS is about 25% here (DC) and their support is in name
only. They won’t even let you file a complaint for (IIRC) a week after the
alleged delivery date because they know how often that’s completely
fictitious.

The USPS is generally rock-solid, although during the pandemic I’ve had a
couple of packages show up the morning after the stated date due to the
carriers being slammed (described as like Christmas but without the temporary
staff).

Amazon appears to track this: after a few lost packages they never use UPS for
us. They use their carriers and USPS for all sizes so I’m assuming there’s
some careful price arbitrage going on.

------
devwastaken
Cloud services need to have cost caps, plain and simple. This isn't
cloudlflares fault, it's Amazon's, and it's the authors. Cloudflare should be
detecting overall data transfer, but there's plenty of cases where terabytes
of traffic is entirely expected. We know Amazon won't fix their service, so
perhaps cloudflare could impliment bandwidth limits.

~~~
icefo
I think backblaze does it well. You can set limits for each type of API
transaction and you receive email and sms notification when you get close to
the cutoff limit (70% and 90% IIRC). You have a page with all the costs and
how close you're to the limit if any. 2FA was also straightforward to setup.
If your account is compromised the limits can still be lifted but there is
limited interest in compromising a backblaze account in the first place and
the 2FA makes it unlikely.

I'm the sysadmin for a small nonprofit so my organization qualify for the
$3500 yearly azure credits but the inability to set spending limits makes me
not use azure. If I make a mistake or worse an admin account is compromised
the azure bill is potentially infinite.

With azure I think you can mitigate the issue somewhat by having a superAdmin
account without limits and set quotas for everything else but I still don't
feel at ease with that.

My organization is a music festival so the infrastructure really has to work
and not stop one day per year. I can keep an eye on everything during the
festival and monitor spending. If things stop not during the festival people
are a bit annoyed until I can look into the issue but nothing bad happens.

In a perfect world I'd like to have a way to setup a limit where you need to
go through support to increase it. I'd really pay for that. I think it's
really too easy to receive a nasty surprise bill.

~~~
ghaff
Backblaze B2 also lets you put hard caps in place which can be nice for some
situations where money is tight and the application isn't critical--and
especially if you don't really have IT people monitoring.

------
hasenpfote
I have said it over and over and will repeat it happily:

IF your Services doesn't has a proper limit, you do make yourself suddenly
liable to a much higher risk than before and you have to be aware of this.

It is the same shit when you rent a car: Do NEVER rent a Car without proper
insurance.

I'm working with GCP professionally and i have used AWS in the prev company. I
do ask my manager if i can use it to try a few things out and its fine but i
will not put my credit card behind an account with unlimited cost risk (its
limited probably but you know what i mean).

And its not even simple; Everything costs you money. Storing data, receiving
data, pushing data, making api requests etc.

And what i find always quite surprising: How often people, even on hn, present
simple file based apis where you can upload images and edit them or upload
files and download them again or offering free services and that with AWS as a
backend.

I just might be to long in this industry to see all those pitfalls of exploits
and risks everywhere but i have the feeling that obvious respect against cloud
service billing is neglected by most.

~~~
fxtentacle
I believe it's the same effect as micro-transactions in mobile games.

It's really easy to justify paying $1 for a small upgrade while you're
playing. And only afterwards you notice that those $1 added up and have
financially ruined you.

In the same way, $0.08 per GB (the effective price in the article) sounds
really small and easy to justify. And we forget how they can accumulate...

~~~
dathinab
Yes, but it's hard for a third party to weaponize that against you ;-)

You want to destroy a small startup with a free alpha version and AWS (or
similar) backing?

Sure go ahead and send them tons of _legit_ (looking) traffic. This will first
mess up their bill for this month and then mess up their statistics for the
next month (when all the user they got disappear at once)...

~~~
fxtentacle
Agree, using cloud is kind of an invitation to your unethical competition to
bankrupt you.

------
hoppla
When I was first looking at aws, I received a billing prediction alert that
was ridiculous high. I could not find the culprit (I only had an ec2 instance
and some other random services I looked into). In the end I deleted my
account, to avoid this unexplainable billing. Next day, I got an email saying
I erroneously got the alert. Damage was already done, but at that time, I was
only exploring what the cloud had to offer, so no real damage done.... until a
couple years later when I had to do some real aws for my work. Because I
deleted/disabled my account, I cannot reopen an account with the same email
address.

------
rytrix
Everything is always crystal clear in hindsight, that being said I always tell
my clients to set up billing alarms as one of their first tasks when getting
started on AWS.
[https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html)

------
prepend
It’s always frustrated me how it’s not possible to set a quota and turn off
services at quota.

Logistically I know this is hard for water or power, but it should be feasible
for cloud computing. But I think this is an area where it’s not in AWS’
interest to set up that kind of billing control.

~~~
dragonwriter
> It’s always frustrated me how it’s not possible to set a quota and turn off
> services at quota.

Which services you want to turn off or alter use of as you cross various
thresholds vary, so what you probably want is a billing information services
that sends alerts at configurable levels that trigger programmed actions that
are more complex than a simple shutdown.

It would be easy to stop all spend at a threshold, but that means your entire
set of apps stops working and all retained data (which usually has an
associated periodic cost) vanishes irretrievably, which might be okay for an
account being used for only toy apps but is going to be a business-ending
error for any serious account

There's a good reason why cloud providers don't provide a simple “nuke your
account at a particular spend threshold” option, and the fact that people who
haven't thought things through _think_ that they want such an option is
actually a factor in not providing it.

If people can't keep track of spend with alerts to know that they need to take
some kind of controlled restriction of services, they are going to regret
unexpectedly having their whole set of services and data nuked more than an
unexpectedly large bill in most cases.

~~~
prepend
Initially I tried to set up something as simple as when I got near the limit
for free tier, it shut off to keep me at zero. Couldn’t do that with built in.
And billing data isn’t real-time so I couldn’t even figure out a script a shut
off.

I would lime quotas for individual services like “only spend $100 for EC2 and
shut everything off when I hit it.”

My point is that I would rather have my stuff nuked than get a $1000 or $10k
or $100k bill.

Setting up notifications and triggers is sort of possible but 1) that requires
a lot of work for something that should be built in, I think; and 2)
notifications aren’t in real-time so by the time I get a notification that
I’ve gone over my $100 threshold I might already be at $1000.

I can compensate for this with scripts and third party services, but this
would be so much easier if built in.

Unix has had disk quotas for decades right? Imagine if sysadmins left it up to
users to monitor and control their usage and just charged overages. It’s so
much more work for the user than when the system does it, or at least offers
it.

My Linux host offers bandwidth quotas with similar cutoffs. I would never want
an “unlimited “ quota where I got billed by the transfer and it was up to me
to turn off. Some may want that, but not me.

~~~
dathinab
> notifications aren’t in real-time

Be aware that even with build-in threshold limit application will not be real
time (due to parallelism, especially if multiple data-centers are involved).

But they can be _much_ closer to real-time.

------
gchamonlive
I think it is underestimated how complicated it is to deal with cloud
services. You can do a lot with minimal training and doc reading. But these
wholes in the formal understanding of how everything operates creates these
kinds of vulnerabilities.

Everything can have side effects in the cloud. You can set up a cheap EC2 type
T feet, and without managing your cpu usage, be charged a fair amount in
unlimited burst credits (which is the default for terraform for instance).

You can quickly setup a WordPress instance with cloudfront and a invalidation
Plug-in and be charged 6000 USD unadvertedly
([https://wordpress.org/support/topic/amazon-cloudfront-
invali...](https://wordpress.org/support/topic/amazon-cloudfront-
invalidations/))

You can set up lambda triggers and quickly do a proof of concept for an app,
but forget to correctly dimension your mem usage and be charged more than you
need.

Cloud requires careful policy and topology consideration. There are many
simple blocks that forms a complex mesh with opaque observability of potential
vulnerabilities in both access and billing. Cloud is nice but it requires time
and care. And with the shared responsibility model, you are responsible for
that.

------
ryanmarsh
Saw the mention of PTSD and running towards danger and immediately thought,
“oh nice hopefully I’ve found a kindred spirit”. This matters because I share
little in common with my peers in this field. So I read Chris’ About page
and...

Do any other (combat) veterans smell something wrong with an Air Force Tech
Controller (3C2X1) making statements like _”like back in the old days, when
something would go bang or boom, and I’d run towards it”_ in a civilian venue?
You know exactly what I mean, and we see it all the time.

If you aren’t a veteran, especially with a job even remotely related to
“running towards things that go boom” please just give us some space on this
one. Thanks.

------
namidark
AWS bills are un-auditable. I'm convinced every org is being over-charged for
bugs in their billing and tracking software. I've asked on multiple occasions
where charges randomly started appearing (despite no infra changes), which
weren't there the month before, and no one was able to answer on the AWS
support side.

------
knorker
Summary:

1\. The big cloud providers charge enormously for outgoing bandwidth. Most of
us know this, but unfortunately it bites people a lot. 2\. If you host big
files on these clouds with no limits or warnings, it's just a matter of time
before this happens to you.

This is why I don't run hobby things on these clouds. Any hobby project may
have backends and services running on them, but NEVER anything user-accessible
such a webserver, S3/GCS bucket, or similar. It's just too much of a "click
here to bankrupt me".

For a business it's a different matter. You are making money, and you're
spending money to do so. You still need to have a DDoS plan for your outgoing
traffic, but it's much easier to solve these problems if you have revenue.
Revenue buys time and people.

------
sunilkumarc
Nicely written article. Interesting and one should definitely be aware of how
certain services are charged before using them. This would be a good lesson
for all of us :)

On a different note, Recently I was looking to learn AWS concepts through
online courses. After so much of research I finally found this e-book on
Gumroad which is written by Daniel Vassallo who has worked in AWS team for 10+
years. I found this e-book very helpful as a beginner.

This book covers most of the topics that you need to learn to get started:

If someone is interested, here's the link:
[https://gumroad.com/a/238777459/MsVlG](https://gumroad.com/a/238777459/MsVlG)

I highly recommend buying this e-book if you think AWS documentation is
overwhelming.

------
Ciantic
This is the reason I liked Azure as I used it few years back. One could set
like a prepaid plan, and there was no way to overspend it accidentally.

Of course it is not ideal for companies who need their services be available
for all cost, but for home users it's a nice guarantee.

------
rubenhak
Not directly related to S3 traffic bill, but overall cloud cost management.
Maybe some are unintentional, but still very painful. My experience with AWS &
GCP.

\- AWS CloudWatch: expensive service, virtually unusable, hard to turn it off.

\- AWS overall: finding and cleaning up resources is messy. The order of
creating & cleanup is not same. Closing an account is a painful process. GCP
Project structure is way easier.

\- AWS EKS: You create a cluster, then a node group. Deleting a cluster fails
if there is a node group. You go ahead to delete a node group, it complains
because of "dependencies". While you're randomly looking for a "dependency"
the $ clock is still ticking. You should delete the network interface before
you could delete the node group, and only then the cluster. This does not
sense because if the network interface was created implicitly by the node
group, i should not be responsible for deleting the network interface. There
should be a symmetry in create/delete operations.

\- GCP GKE: You create a cluster, then delete it. Cluster gets deleted -
kudos, usability much better then with AWS EKS. But it turns out lots of
LoadBalancers and Firewall rules are left over and still appear on the cloud
bill. Those are implicitly created and should be cleaned up implicitly by GKE.

------
ricardo81
Maybe missed it, but I didn't see which plan he was using with Cloudflare, in
context of his comments about their support.

------
BrandoElFollito
I asked AWS to set a limit on my spending. They said that they did not want to
do that "not to break my business".

I want them to - I do not care if my site is offline vs. having to pay a huge
bill. That should be a choice.

So I moved away from AWS. It is crazy that companies agree to such a racket
(not the pricing - but the fact that you cannot set a limit).

I considered to use a virtual card with a limit on it - they could not grab
more than the limit and just sue me across the pound or remove my account. But
I refuse to play these games with a company who does not give a shit about
billing.

------
logicallee
I think this is a very common occurrence!

A good alternative to this ever-present risk is to use a dedicated virtual
private server that is unmetered. This would make mistakes like this (and yes,
it is a mistake - it is his fault he didn't read the cloudfare details and
publicly served a large VM image) impossible.

Here is my referral code for the one I use[1]:

[https://crm.vpscheap.net/aff.php?aff=15](https://crm.vpscheap.net/aff.php?aff=15)

This also (especially) applies to startups that might suddenly take off at any
moment (but don't expect to.) AWS is a ticking time bomb of unexpected
charges. You never know what the Internet will bring you. Go for an unmetered
VPS and have 1 single well-defined charge that doesn't change. That's what I
do on my side projects.

[1] I previously asked Dan, the moderator here, if I can share in this way and
he said it's okay. I don't have other affiliation with that company and have
found it good. The last time I posted this I got 80 visitors and no complaints
(and got upvotes), so I figure it is a good resource for people.

------
SergeAx
> Moving it back to AWS from GCP bumped the AWS bill to an average of
> $23/month. Not too bad given the site’s traffic.

I've checked the traffic, it was 2.3k users for entire June, like 75 user per
day at average. It is effectively nothing, why author thinks it's okay to pay
1 cent per user per month to hosting provider? $5/mo VPS can handle two orders
of magnitude more.

------
Artur96
This is why billing alerts are mega important to setup

~~~
tln
Totally agree. I have alerts for 50% and 120%. If I get the 50% in the first
week I'm digging in then. If I get the 120% I'm watching it like a hawk from
then on (and letting accounting know).

The article says they would not have helped but it doesn't say why... Maybe
because it's delayed by a day?

~~~
dathinab
They don't.

But let's say you are:

\- on holidays

\- sleeping

\- sick

\- on a party

\- etc.

Having setup alerts would still have reduced the bill.

Still assuming that something like 2h pass before you can react to an email is
quite reasonable which would still have been ~150$ on the big day, which is
~6x of the normal _monthly_ cost in 2h...

And that is assuming the alerts are send real-time, which they are not.

------
tgsovlerkhgsel
This is why I'm deathly afraid of using any major cloud provider.

External traffic is effectively unlimited, and a number of possible reasons
(popularity, misconfigured script pulling something in a loop, someone
intentionally generating traffic to hurt me) have the possibility to throw me
into arbitrary amounts of debt, with the only recourse being hope that the
cloud provider will be merciful.

Even if I have alerts set up: someone pulling 10 Gbit/s can generate over 100
TB per day, at $80-100 per TB. If I don't check my e-mails for weekend, I can
be $30k in the hole before I notice.

------
Edd314159
I racked up a $120K+ Google Cloud bill via an unsupervised and poorly-coded
script which used the geo coding API. It didn’t take much to get Google to
waive it. This happens all the time I’m sure

~~~
dathinab
I think so, too.

But this makes it even more scary in a certain way.

I mean what if they don't waive it in your case? You have no guarantees. Only
the potential to cause bad press can save you.

That is not a very healthy situation I believe.

------
nix23
For a private person it's much better to use something like a private vm or a
dedicated server, from Vultr or 1984hosting (Iceland) you can get a vm for
just 2.50$ (only IPv6) or 5$ (IPv4 and 6) or a dedicated server from Hetzner,
OVH, Scaleway (Arm64) for like 30$, some have unlimited traffic (mostly the
dedicated servers) with a 1Gbit connection. NEVER use stuff you have to pay
without knowing whats coming (count's for private use an small business)

------
ColdHeat
DigitalOcean's S3 offering probably would've kept this bill down a bit.
Probably about $300 versus $2000.

[https://www.digitalocean.com/docs/spaces/#bandwidth](https://www.digitalocean.com/docs/spaces/#bandwidth)

Digital Ocean may not be the best cloud platform but it's fairly cost
effective.

~~~
andialo
Yes AWS is crazy expensive, their business model is simple yet super effective
and they are swimming in money, power to them, make custom services with
custom APIs, make it very cheap/free for low volume use and then crazy
expensive when the usage goes up and the customer is vendor locked-in into
their services + push enough money into marketing to so it is everyone's first
choice.

Not only AWS is very expensive but also rather hard to use and all their forms
and services pretty difficult to navigate as well, it put me off the cloud
hype for very long time until I actually discovered reasonably-priced* cloud
providers like DigitalOcean (or linode, vultr,...) with also very easy to use
platforms.

* of course still pricier than dedicated hardware/VPS, however the premium for hourly billing and infrastructure maintenance is reasonable

------
tjoff
Can someone argue that the complexity of the cloud does not easily surpass the
time and effort of just setting it up yourself? And for once all of that time
and effort into setting it up is actually valuable and you get much better
insights into your own operations.

And the alternative is paying someone to lock you into their ecosystem.

Are we really _that_ lazy?

~~~
dathinab
It's more about:

\- Flexible scaling.

\- Using other existing services from the cloud provider.

\- Not needing to manage any hardware. (Or to hire someone to manage the
hardware for you).

\- Running your things in a data center. (Well connected, reasonable fault
tolerant.)

\- Less upfront operational cost (you are not required to pay for expensive
hardware upfront).

For larger companies, especially such with massive amounts of micro-services
it all about the flexible scaling and not needing to manage hardware.

~~~
tjoff
\- usually not even needed on real hardware. Focus on a sound architecture
(which will reward you in everything you do). But depending on situation VMs
are an excellent choice and it is easy to spin up more

\- that is an antifeature

\- seems trivial compared to keeping up with all the cloud gotchas

\- possible anyway

\- sure, but you probably don't need expensive hardware to start, or go with
VMs

~~~
andialo
Try this one:

\- AWS (and other cloud providers') kickbacks

~~~
tjoff
Not sure what that is?

Sounds like a convoluted, time consuming scheme designed to further lock you
in with the promise to recoup a fraction of AWS the tax?

~~~
andialo
Just making a fun of the fact that kickbacks paid by amazon to top management
surely pay a big role why corps choose AWS.

------
vmception
I had a $190,000 AWS bill for an account only used for static S3 hosting.

And guess what, I didn't write a blog post about it. I just went to support,
said remove the charges, they identified the services that created the issue
so I could kill them, and they removed the charge.

Look at that, no fan fare. I had no emotion about it whatsoever. Maturity.

~~~
stjo
“I had no emotion about it whatsoever. Maturity.” Doesn’t sit right with me.
Wanting to share shouldn’t be condemned as immature. Besides, some of us want
to read tech drama :)

~~~
kgraves
> Wanting to share shouldn’t be condemned as immature. Besides, some of us
> want to read tech drama

ew.

------
pavelevst
Good reminder to everyone including myself to use services that have budget
alerts or spike protection. And to stay away from aws, even 23$ for static
website hosting is a bit too much in 2020. At fairly priced hosting without
fancy name, 30tb traffic can cost <10$

------
reedwolf
Much better incident:

[https://www.reddit.com/r/aws/comments/g1ve18/i_am_charged_60...](https://www.reddit.com/r/aws/comments/g1ve18/i_am_charged_60k_on_aws_without_using_anything/)

~~~
neurostimulant
It's amazing that amazon would let him run that massive instance for months
without any successful payment. I would imagine other vendor would suspend the
account after a few days of late payment. For example, DO was notorious on
suspending and deleting data after just a few days of late payment and it
caught many people off guard while on vacation (I think their policy related
to late payment is much more relaxed now after it was blowing up a while ago).

------
Borlands
Copying a 12GB file and sharing it publicly on S3 (how does AWS make money
with S3, anyone care to answer?). I agree it’s expensive, maybe outreagously
expensive. It’s great he got a refund! Not so many would be that lucky. And we
can all learn

------
zxcvbn4038
The article shows again the importance of reaching out to AWS support for
issues like this, they would rather have a long term customer then a one time
score, they are really forgiving of one time mistakes.

------
dvfjsdhgfv
> Praise Twitter for at least its ability to draw attention to things. I am
> not sure this would’ve ended up as well as it did without it.

This is another bad aspect of these stories.

------
afterwalk
I wonder if aws would have given the same refund if the author wasn’t a cloud
evangelist with a twitter following highly relevant to aws.

------
Sphax
Am I reading this wrong or the site's traffic is almost always under 1000 page
views per day ?

Why would you need AWS or Cloudflare to serve that ?

~~~
blahbhthrow3748
Because some of the pages are 15GB!

------
chmod775
It would've literally been cheaper to burn that file to a few thousand DVDs
and mail them to individual people.

Nice pricing AWS.

------
voltagex_
So what do you do if you want to host a file on the Internet but don't have
$2000USD kicking around?

------
SeriousM
The title is a clickbait.

------
aaronchall
How's Linode on this sort of thing?

------
9nGQluzmnq3M
TL;DR: Don't leave 10+ GB VM images open to the world on S3 unless you want
pay everybody's bandwidth bills when they spin up a new instance using them.
And set up billing alerts!

~~~
curiousgal
I don't get why AWS refunded him, it was purely his mistake.

~~~
maest
Making a big deal out of this on Twitter may have something to do with it.

I loathe the day when I'll have a similar problem but, as I am not a Twitter
user, I won't have the luxury of getting my problems fixed.

~~~
nucleardog
AWS is actually one of the few organizations I’ve dealt with where the CSRs
seem to be empowered to do stuff like this _without_ you having to raise a
storm on social media first.

In my anecdotal experience, just submitting a support ticket explaining what
got screwed up and you made a mistake and how you’re going to rectify it so it
doesn’t happen again is usually enough to get the charges refunded. And even
if you _don’t_ submit all that, often they’ll just prompt you for it.

For example, when a legacy account got compromised and rang up like $20k in
charges running a bunch of servers to mine some sort of crypto, we asked for a
(totally undeserved) refund and they just had us rotate all our access keys
and refunded it.

The support is part of the reason why I prefer dealing with AWS. I know from
my time managing over a million bucks a year in AdWords spend that the reason
Google’s support is non-existent isn’t because you’re not a paying customer...

------
jabo
> The primary motivation was that Google had so intertwined GSuite and GCP IAM
> that it became overly confusing.

Glad I’m not the only one confused by this.

------
johnklos
Does Amazon not have... logging?

