
As AWS Use Soars, Companies Surprised by Cloud Bills - kaboro
https://www.theinformation.com/articles/as-aws-use-soars-companies-surprised-by-cloud-bills
======
jdblair
When I joined a particular startup 4 years ago I was surprised that they owned
their own servers mounted in racks in a rented colo. I was all, how retro! But
then the head of IT showed me the cost savings over AWS and how he
orchestrated it all using VMWare images, and I saw he had a point. He saved
more than enough money to pay for his full time salary to keep it all going,
even dealing with hardware maintenance. What was unique was that the company
had a predictable load that didn't change much. If we had needed to change
capacity level drastically over a regular cycle then AWS would have started to
make sense.

~~~
Tistel
I have worked at a startup that was paying > 20k _per month_ for AWS. I am
certain the whole system could run on one dedicated blade. There was just this
culture (made easy with docker containers) of "spin up another box" for every
silly little tool. I have met self taught dev ops people who did not
understand that you could create more than one user on a aws ubuntu machine. I
was like: "if cost is a big issue, then lets spin up one beefy machine and
create users for each dev", because thats what everyone using unix did back in
the day (it works fine). The reply was "thats not possible" so every dev got
their own dedicated instance that was 99% idle. We are paying >3k per month to
aws and the business team is complaining about waste. _sigh_. old guy rant
over.

~~~
thruhiker
Did you weigh these costs against the value of the developers not stepping on
eachother's toes using the same instance?Separate users with sudo privileges
provides zero isolation.

I'm unsure what use case was, but if it was a testing or staging environment
dedicated to each developer, having a separate right-sized instance for each
is quite sane and reasonable.

EC2 instances such as the T2/T3 series are cheap enough that there's normally
not a need to be so miserly in my experience.

~~~
CharlesColeman
>> I am certain the whole system could run on one dedicated blade. There was
just this culture (made easy with docker containers) of "spin up another box"
for every silly little tool.

> Did you weigh these costs against the value of the developers not stepping
> on each other's toes using the same instance? Separate users with sudo
> privileges provides zero isolation.

Each developer's machine _is_ a separate instance. If the whole system could
run on "one dedicated blade," then the developers could easily run a local
instance on their own machines (either directly or in a VM) without stepping
on anyone's toes.

Using AWS for local development seems wasteful and over-complicated to me. The
only good reason I can see for doing it is if you're _really_ vendor-locked
into AWS-proprietary services, and it's impractical to run your code
_anywhere_ except on AWS (but that's arguably a mistake in itself).

~~~
deathanatos
> _Each developer 's machine is a separate instance. If the whole system could
> run on "one dedicated blade," then the developers could easily run a local
> instance on their own machines (either directly or in a VM) without stepping
> on anyone's toes._

What the parent you're responding to likely meant was that, by running
everything on one system ("one dedicated blade") you are causing all of the
things on that system to share certain things. The set of system-installed
packages; if nginx needs to be installed and running, is its configuration now
shared between all of the services on that machine; and even the very OS
itself (e.g., if it needs to go down for updates, or it needs to do, say a
dist-upgrade, which means that _all_ of the associates services on that
instance need to dist-upgrade together, and one laggard can thus hold everyone
back.)

Even in the case of developers using an instance on AWS, the same applies,
really: a dev, legitimately developing some component, could have cross-talk
to others, in all of the same ways listed above. I do agree that giving devs
dedicated instances might not be appropriate in many instances, and it perhaps
wasn't appropriate in yours. But there's two points here: whether the devs
needed those instances, or could develop on local laptops, and whether putting
all instances or users on a single instance works well.

Even for development, given that most corps seem to love giving devs that code
primarily for Linux servers OS X laptops, there might be advantages to
allowing them to develop on the actual target hardware.

> _There was just this culture (made easy with docker containers) of "spin up
> another box" for every silly little tool._

Docker here should make it rather _trivial_ to combine everything onto a
single instance, since that's what isolation gets you: a well defined unit
that doesn't have endless unknown dependencies on the host system.

~~~
h1d
Sounds cheaper to just give them VMware fusion and a dev OS image which is
mostly identical to the target architecture.

I really don't see why many run their environment on macOS through Homebrew
etc which is surely going to require modifications here and there when
deploying to Linux servers.

------
sasasassy
I personally feel many things in AWS are purposely made in order to hide the
real costs from you.

For instance how you are charged for S3 traffic your buckets receive even if
you don't have any files there. Or how traffic between zones that is outside
of your control is still charged. Or how things like AZ replication cost a ton
and you have no metric to show if it even makes sense to enable them. Heck,
even usage alerts cost money.

~~~
Dunedan
S3 traffic cost is a funny topic. I remember ~3 years ago we had an issue with
AWS, where they kept charging us for web traffic to an S3 bucket we deleted.
When we complained, their response was like: Well, then stop sending traffic
to that (non-existing) bucket. That of course wasn't going to happen as the
URL was out in the wild and got requested quite often automatically by third-
party services.

~~~
toomuchtodo
Did you not control the DNS for that hostname? Not supporting AWS' stance, but
you should always maintain control of your traffic flow using DNS. Reverse
proxying is the latest hype for SEO juice
(blog.somewhere.com->somewhere.com/blog), but subdomains are the bees knees
when you need to point traffic somewhere else on demand.

~~~
Dunedan
Back then we already had the bucket for a few years and the person who set it
up probably didn't think about the implications and simply chose to use the
AWS-provided bucket URL.

And that person isn't alone. If you look around, you'll find a lot of services
on the internet which directly link to S3 buckets using thee AWS-provided
endpoints.

~~~
soared
A tool my company always pulled data from was depreciated and the web ui and
api got shutdown. They used the awe bucket url though for the web ui downloads
and those had predictable url patterns. Turns out they didn’t stop generating
the data we needed, so we still just go straight to their urls.

------
brootstrap
After years of working with AWS this is not surprising. You can have
developers with minimal sys admin / dev ops building instances and running
jobs. Back in early days of my startup I recall my team launching a big model
run and leaving for weekend. We can back on monday to a 10k bill and thousands
of compute hours... That did not go well, 10k on a few days of compute for a
small startup haha! Still to this day we've got tons of EC2 instances lying
around racking up charges. We dont have the devops support to build proper
auto-scaling, making things really fit AWS. We just needed a server to run
24/7 and this was easy.

Can only imagine the insanity that goes on at huge Pinterest level scale. How
many EC2 instances do they have? Thousands? Hundreds of thousands?

~~~
wolfi1
if you just have webservers, api gateway + lambda is the way to go Edit: to
make myself clear, I meant the backend, frontend, of course, should be served
like below

~~~
WrtCdEvrydy
Cloudfront + S3 for static sites.

I've seen so many people just create an EC2 instance for serving static
content.

If you have a banging new React app that generates static files, you don't
need anything more.

~~~
wolfi1
of course, I meant it backend-wise

------
dalbasal
Pay-for-what-you-use pricing has the ability to escalate. You don't buy a
server for $30k, and go through whatever procurement paperwork is set up for
that. You just increase average usage by here and there. There's no
procurement hurdle, so it can be a blank check.

..just like it's easier to get I to debt with a credit card than a personal
loan. Credit card debt mounts one purchase at a time. Personal loans require
you to decide to borrow €X.

~~~
dba7dba
Reminds me of reading how US Armed forces in Iraq being forced to buy NEW
printers because their procurement system didn't allow buying toners to
replace used up toners in practically new printers.

MBAs/Accountants...

------
josh_carterPDX
I just joined a new company who is doing nothing but helping companies figure
out how to save more money on their cloud costs. This is such a timely
article.

The same costs that are being used to justify the value of the cloud are now
being used to justify going back to buying servers. Which is the wrong
solution, by the way.

With anything it just comes down to education and best practices. People need
to utilize the incredible tools out there along with understanding better how
the cost structures work.

~~~
NightlyDev
Servers the wrong solution? Maybe in a few situations. AWS is just too
expensive.

Eg. 600 TB data out and 100 TB in is $37300 on AWS, and by going with
colocation a price of < $500 is very achieveable.

That's ~1/75th of the cost!!!

There is nothing to understand here, except that AWS in insanely expensive and
colocation is a way better option in this situation.

~~~
rstupek
I agree that the price is high but you're not comparing the same things. Is
the $500 a month DDOS protected bandwidth with as much redundancy as AWS is
providing? I definitely shift as much bandwidth away from AWS as I can with
any solution I make but they do provide more than a dumb pipe

~~~
JoshTriplett
> Is the $500 a month DDOS protected bandwidth with as much redundancy as AWS
> is providing?

Sometimes you don't need that.

AWS S3 provides reduced-redundancy options for less cost. It'd be nice if the
same applied to other services.

------
incomplete
i honestly cannot believe that this is still a thing! well, actually, i can.
:(

i run a large AWS consolidated billing family for a sizeable computer science
graduate research lab and tracking our AWS spends are an absolute nightmare.

if you think about it for a minute, why would it be in the best interests of
amazon/AWS to allow us granular, easy to parse and (most importantly) easy to
customize and use billing reports? they're in the business of making money,
and providing mechanisms for folks to limit their spends will limit profits.

another thing to consider is that the billing subsystems for all of these
cloud providers are literally one of the _first_ things to be engineered, and
after release, one of the _last_ to ever be updated.

for instance, it took amazon two years to release OUs (organizational units):
one year in closed beta, one in open. while these are great for organizing
accounts, you still can't see how much an OU spent w/o a bunch of python/boto
gymnastics!

i was on a call w/some of the leads of the aws billing subsystem about a year
back, and asked about what the roadmap for billing features and the response
was "2020 at the earliest".

~~~
user5994461
What are you having troubles with? I find the billing page to be pretty good,
once the tags are configured for billing.

Far better than any internal tool I had the chance to work with.

~~~
incomplete
we have a lot of projects, and i use OUs to track the N-ary tree of
lab->project->people and their spends.

since the aws billing subsystem can't map OU->$$$, i wrote a thing:
[https://github.com/ucbrise/aws-audit](https://github.com/ucbrise/aws-audit)

it'd be also super awesome to have OU-based budgets.

we use teevity ice when we really need to dig deep
([https://github.com/Teevity/ice](https://github.com/Teevity/ice)) but it's
pretty much abandonware at this point.

i've also explored a couple of commercial offerings (ronin & teevity) but they
don't work well in our AWS research credit-based model.

~~~
user5994461
I'm not sure about OU, didn't use them much.

In commercial tools, I would have a look at
[https://www.cloudhealthtech.com/](https://www.cloudhealthtech.com/) and
[https://cloudcheckr.com/](https://cloudcheckr.com/) and there was a third I
can't remember the name.

They should have good reporting since that's the only point of the tool, but
they charge an arm and a leg.

~~~
incomplete
those look just like the other services i looked in to, but w/o diving too
deeply in to their whitepapers i can guarantee they use instance/resource
tagging as the shim between AWS and their system(s).

with consolidated billing, and a couple of hundred linked accounts (with
plenty of turnover), enforcing a sane tagging is pretty much out of the
question.

thanks for the links tho! when i have a spare few minutes i'll definitely take
a closer look.

~~~
user5994461
Consider revisiting your account strategy. Hundreds of accounts is way too
many and it's unmanageable. Even a company with tens of thousands of employees
couldn't have anywhere near that.

Enforce some tags on purpose, team, business unit, dev or prod. It shouldn't
be allowed or possible to create instances without any information. Enforce
the rules, if an instance has no information it will be shutdown next week and
deleted next month. People will take tagging seriously very soon.

I've seen businesses who adopted AWS without any strategy and with revolving
contractors as employees. A few months later, it's a hundred instances without
a single name or tag, each one costing thousands of dollars a month for the
more expensive.

If the company cares about costs and resources management, it's easy to
achieve. If it doesn't care, then doesn't matter, not your money.

------
SEJeff
Just wanted to make a shoutout to Corey Quinn, who has a consultancy which
literally works on lowering your excessive AWS bills. He also runs the most
excellent Last Week in AWS newsletter
[https://lastweekinaws.com/](https://lastweekinaws.com/) and podcast Screaming
in the Cloud:
[https://www.screaminginthecloud.com](https://www.screaminginthecloud.com)

I've personally known Corey for 10+ years and he's a good dude, so if anyone
is looking to lower their AWS bills, talk to Corey.

~~~
crescentfresh
> literally works on lowering your excessive AWS bills

these companies are a dime a dozen, no personal offense intended to yourself
or this company.

~~~
lathiat
The fabulous level of trolling and entertainment he offers on his twitter
account however is priceless.

~~~
SEJeff
And both the newsletter + podcast are quite fantastic. He really is unrivaled
in the space. If you use AWS, you should be keeping tabs on those things.
Bonus points that they're free and amusing.

I know him from back when I was helping maintain the Saltstack python config
management tool[1]. He was a user and we were doing training on contributing
code to salt. It turned out that I did teach him a thing or two and he did
several contributions to salt[2]. That's literally it.

[1] [https://github.com/saltstack/salt](https://github.com/saltstack/salt)

[2]
[https://github.com/saltstack/salt/commits?author=quinnypig](https://github.com/saltstack/salt/commits?author=quinnypig)

------
jacques_chester
Where I work there's a monthly report sent out that shows what each team is
spending. Periodically teams poke around their infrastructure and remove
anything they don't need any more. A few times there have been company-wide
"spring cleaning" days.

That's all it has taken to save a _lot_ of money.

~~~
user5994461
Same. It's pretty amazing to be able to see where the money is going and
adjust anytime.

------
ham_sandwich
With the rise of lambda+managed services, I think we’ll start seeing finance
and development start to blend and merge.

Besides giving businesses more legibility into what specific parts of their
business logic cost to operate vs. the value they generate, you can start
building higher-order financial systems based on flows of capital+information
within businesses. From there you can implement all sorts of financial
engineering like insurance and options+derivatives that could allow businesses
to do things like dial up leverage against these flows. Certainly half-baked
ideas, but fun to think about the possibilities.

~~~
jacques_chester
I'm moderately hopeful that the basic idea (close association between activity
and cost) will improve, but I have some caveats.

One is that not everything will fit. Lots and lots and _lots_ of workloads
exist in their current form and are not going to switch overnight. Whatever
bookkeeping method that gets developed needs to deal with the mix of costing
models.

Another is that you still need to assign costs. In a lot of companies this is
the subject of intense corporate politics. Whether such a method is adopted
will depend on who is wielding what cudgel.

Related to which: there will be cases of premature optimisation on the
visible. Optimising for cost is fine, but costs often include estimates that
can be overlooked. It's one thing to optimise for "least dollars spent per
invocation". Another thing to optimise for "fewest pissed off customers". The
latter is harder to measure but in many cases more important.

But overall? Yes. I think it could be a step forward.

------
smallgovt
I was always incredulous at how many free credits AWS hands out to startups
(up to $100K per startup), but then every time I see an article like this I'm
reminded how large the recurring infrastructure bill can be for a successful
business. Such a great business for Amazon.

~~~
ceejayoz
It's quite clever, too, as it likely encourages setups that are pretty
inefficient on long-term AWS spend.

~~~
ams6110
It's like the apocryphal drug dealer who gives out free doses to unwitting
kids until they are hooked.

~~~
genericone
This article is one of the first shots fired in the 'War on Clouds'.

------
tyingq
Part of this is that AWS is often more expensive than on-prem.

The other part, though, is that dev teams in large companies often aren't held
accountable for on-prem costs.

The on-prem costs are often in a big baseline number with little visibility
into how much of that big number belongs to each team.

------
kevinAtStorj
I find it interesting that while the price of hard drives has decreased by
about 50 percent over the past five years, the price of cloud storage has
remained roughly flat.

As margin continues to expand, the need for alternative models and competition
in 'Cloud' is becoming increasingly apparent.

[https://storj.io/blog/2018/11/the-high-price-of-
traditional-...](https://storj.io/blog/2018/11/the-high-price-of-traditional-
cloud-storage/)

~~~
riskneutral
I think the end game is probably something resembling electrical grid markets.
Electrical power is bought and sold in real-time by traders, as the supply and
load fluctuate. There are many players in a competitive market, like nuclear
plants, gas plants, solar, wind, etc. There are even complex derivatives (like
“swing options”) traded in the electrical markets. Someday computing resources
will be bought and sold in this way too, with a large number of players in a
competitive market, which will drive down the currently inflated cloud
computing costs. We can already see an embryonic version of such a market in
the form of AWS “spot instances” and the AWS reserved instance secondary
market. AWS is fighting tooth and nail to avoid or delay such a
commoditization of computing utilities, which is why they announce 50 new
vendor lock-in software products every week instead of focusing only on the
hardware resource.

------
EGreg
“The Cloud” is a euphemism for extreme centralization of web and other
hosting.

~~~
mbrumlow
I joined this company to build out a software stack for being able to do
disaster recovery of sites off site. This resulted in building a system that
takes incremental backups of client systems and uploading them to our servers.
In a DR event we would spin up VMs backed by the images we had copied to our
servers.

Throughout the processes the word "cloud" was brought up. Who's cloud would we
use? Over and over again I tried to explain we were actually building a
"cloud" service. And by that I mean we are just offering a service that runs
on our servers.

Due to the feature set and cost we ended up settling on bare metal servers
from SoftLayer (IBM). The entire solutions has been made to run on commodity
hardware.

For the longest time the company kept marking that we were using "IMBs Cloud".

Every now and then I get a request to ask what it would take to move to AWS.
My response is always the same. More operating budget, and less features.

We spin up DR environments for many servers2 in seconds -- and can do so
because we have full control over the hypervisor, so moving to anything like
AWS, Azure, or what not means we give up the ability for near instance
restore. Owning the entire stack has its own set of problems, but in the long
run we should be able to move much faster.

"The Cloud" -> "The Internet"

------
spectre256
The two most insidious things I've found in AWS billing:

\- If you want the AWS dashboard metrics to have 1 minute instead of 5 minute
granularity, that's $2.10 per instance per month. 5 minutes is pretty useless,
so either you set up other monitoring or you pay a tax on the number of
instances you have.

\- If you use AMIs (which you probably should) to launch EC2 instances with
all your software baked in already, you will probably end up with dozens or
hundreds of old, unused AMIs. Furthermore each of these AMIs is linked to a
snapshot, which is stored on S3. S3 pricing is very cheap but it's a
significant amount of work to determine which AMIs are no longer in use and to
delete both the AMI and the corresponding snapshot. Every 100 AMIs you have at
the standard 8GB root volume size costs you $18/month.

~~~
Someone1234
> If you use AMIs (which you probably should) to launch EC2 instances with all
> your software baked in already, you will probably end up with dozens or
> hundreds of old, unused AMIs.

We previously had this issue. What we do now is instead of leaving old ones on
S3, we download and archive them. We just tag them on the dashboard and a
nightly script does the rest.

In general tagging is an extremely powerful way to organize your AMIs. This
article[0] has good examples of tagging you can follow. We use something akin
to the "Business Tag" strategy, with an Excel document. Definitely requires
some internal organisation but the cost savings speak for themselves.

[0] [https://aws.amazon.com/answers/configuration-
management/aws-...](https://aws.amazon.com/answers/configuration-
management/aws-ami-design/)

------
twunde
I think that everyone who's used AWS or a competitor knows that it's very easy
to rack up a large bill by accident. For companies, the main issue isn't the
expense but the unpredictability.

~~~
WrtCdEvrydy
The use is that most people use AWS like if it was a datacenter they don't
own.

If you're going for AWS, you should consider rewriting a lot of your services
to use AWS features well.

Auto Scaling Groups and Fargate EC2 are some common components I see few
companies using.

If your instances are the same size through the day, you are doing it wrong!*

*Exception: If you provide the same level of service and traffic 24/7.

~~~
breckenedge
I recently gave Fargate a try and was very unimpressed with the costs.

Anecdata is terrible, but my experience running 1 Fargate container 24x7 on
the lowest specs (0.25 CPU and 512 MB RAM) to handle baseline was was going to
cost as much as a no-contract T3.micro EC2 instance with much more capacity (2
CPU and 1 GB RAM). AWS Kubernetes was also a bust at $120/mo just to get
started (that's the cost before an EC2 server is provisioned).

~~~
WrtCdEvrydy
Fargate has two flavors, ECS or EC2.

EC2 is just EC2 with scaling groups while ECS is a fully non-managed solution
that's for small items.

~~~
JBReefer
The ECS one is weirdly difficult to set up. If you just want Docker Repo (ECR)
-> deploy with some config -> expose a single ingress point it sounds perfect,
but I had a TON of trouble with it recently

~~~
breckenedge
If it helps, I had the exact same experience. Setup was possible but involved
way too many decisions for the obvious use case.

------
AcerbicZero
This is one of those predictable things, that is conveniently ignored by
certain layers of a business. The more entertaining situation (from my
position anyway) is the drama that the rapid Azure adoption by otherwise very
non-cloud companies will cause in a year or two, when the free credits
Microsoft stuffed into their ELA's go away.

I love running stuff in the cloud, but without a conscious decision in an
organization to prepare for, and avoid, the various pitfalls, its just a giant
cost sink.

------
cobookman
A lot of people don't realize that Reserved Instnaces don't show up on your
monthly invoice. So once every 3 years you're hit by a large bill for RI
renewal.

~~~
neuronflux
You can choose how to setup your billing for reserved instances.

> You can choose to pay for your Reserved Instance upfront, partially upfront,
> or monthly

source:
[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts...](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts-
reserved-instances-application.html)

~~~
cobookman
You get the best discount by paying fully upfront. If you do this, the AWS
bill doesn't show the RI on your monthly bill what-so-ever. Leading to a
suprise 3 years down the road.

~~~
user5994461
And more importantly you're engaged to pay for the full year no matter which
option you pick, with (almost) no possibility to cancel.

Long story short, only play 1 year full upfront.

------
pnathan
One of the basic tenants of engineering is "why is this feature appropriate";
much of AWS can be sliced away with this razor, much of the time. Appropriate
tool for the appropriate time. A lot of the AWS systems simply are special
purpose and not good for general application.

I'm working on a Kubernetes configuration at home that provides the common &
popular features AWS in a consistent & portable manner: blob store, database,
etc.

------
dba7dba
AWS is really expensive. Even for running a small personal websites. I thought
I could gain AWS experience AND run a few personal websites on them. What a
mistake.

I've moved off all (well like 2) my personal sites to other VPS provider, that
starts at $5/month for 512MB VPS.

The worst annoyance is 'Reserved instances'. You are basically signing up for
a long term contract that you can't get out of easily.

~~~
phonon
Lightsail is even cheaper, _AND_ AWS

[https://aws.amazon.com/lightsail/pricing/](https://aws.amazon.com/lightsail/pricing/)

------
waterishail
I'm reminded of this article - [http://highscalability.com/blog/2017/12/4/the-
eternal-cost-s...](http://highscalability.com/blog/2017/12/4/the-eternal-cost-
savings-of-netflixs-internal-spot-market.html)

Netflix run a spot market for instances to drive down costs for exactly the
reasons mentioned here

------
nunez
companies that think that they can treat aws like their own colo and dont have
any semblance of budgeting will always be surprised by their bills. the cloud
can be very expensive...it can also be significantly cheaper (think: a
retailer with mostly static websites with some light javascript here and there
that peaks on black friday). doing an analysis and being responsible can help
here.

------
deepsun
Sometimes I wonder whether AWS makes more money than GCP because average GCP
users develop better optimized systems.

------
kemitchell
Missed opportunity:

As AWS Use Soars, Companies Surprised by Soaring Bills

------
jak92
Paywall, cannot read article past third paragraph.

~~~
okbake
I'm curious if all of these comments here are people who paid to read it or
just extrapolating off a the headline.

~~~
dang
This comment breaks the site guidelines, which ask you not to insinuate
astroturfing without evidence. Please don't post like that here; it's a toxic
trope that people are nearly always imagining and projecting onto one another.
Mind you, your post was far from the worst example of this—but it still needs
nipping in the bud.

In this case, the history of the users commenting is enough to make it
overwhelmingly unlikely that they were paid, and the well-known propensity of
users to comment just on a title clinches it.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

~~~
okbake
Sorry, I didn't mean to imply astroturfing. My comment is asking how many
people "paid to read the article" (because it's behind a paywall), and not
"were paid to comment here".

Still not a great comment I admit, but with the amount of discussion going on
I would be surprised if everyone is subscriber or if they are going off of the
headline alone.

------
pavelmark
Paywall.

