
Serverless: A lesson learned the hard way - V3loxy
https://sourcebox.be/blog/2017/08/07/serverless-a-lesson-learned-the-hard-way/
======
tobyjsullivan
>This is probably the most stupid thing I ever did. One missing return; ended
up costing me $206.

No, dear author. Setting up the AWS billing alarm was the smartest thing you
ever did. It probably saved you tens of thousands of dollars (or at least the
headache associated with fighting Amazon over the bill).

Developers make mistakes. It's part of the job. It's not unusual or bad in any
way. A bad developer is one who denies that fact and fails to prepare for it.
A great developer is one like the author.

~~~
lifeisstillgood
Just wanted to say that last paragraph is one of the simplest descriptions of
professionalism in the job I have read - accept and prepare for your own human
failings :-)

------
cddotdotslash
This is not a "Serverless" problem; this is a mistake a developer made that
used a pay-per-use system. If I write code that launches EC2 instances and I
accidentally set it to launch an instance every second instead of minute
because I divided wrong, that's my fault.

~~~
eridius
It is a serverless issue because if you were using your own server, a mistake
like this wouldn't have cost money, it would have just degraded your service
(or possibly brought it offline).

So I guess the question is, with a mistake like this, is it better to be
charged hundreds or thousands of dollars, or to have your service degrade or
go offline until you can fix it?

~~~
chrisco255
The developer should be writing unit tests for their code so they can avoid
small mistakes like this.

~~~
codedokode
It is impractical to cover every line of code with tests (it would get too
expensive). Futhermore, in this case the author would have to test production
config interacting with Amazon servers rather than a piece of code.

And even 100% code coverage doesn't find all possible errors.

~~~
Piskvorrr
Do you need to cover every _line_ of code, or do you need to test resulting
behavior? Also, while nothing is 100% foolproof, the example here would
probably have been caught.

~~~
wolfi1
I doubt this, a unit test wouldn't have covered the infinite triggering of the
created events

------
numbsafari
"The actual cost is now $206 and over $1000 forecasted, it makes me think
twice about using pay-per-use services in the future."

Never use a pay-per-use service that does not include a reasonable "turn off
after $X" feature and appropriate warnings. Also, never use such services
without being sure to configure such settings.

I like to think of this as a self-inflicted "DDOC" attack: Distributed Denial
of Capital.

Best not to leave yourself exposed.

~~~
autotune
If you really wanted, you could create a script that after a certain billed
amount gets reached switches your site via route53 over to a static s3 page
that says "down for maintenance" or something until you figure out why your
forecasted billing amount is so high. forcastedSpend is an object you can call
via api:
[http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/...](http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/data-
type-calculated-spend.html)

and a SDK like boto3:

[http://boto3.readthedocs.io/en/latest/reference/services/bud...](http://boto3.readthedocs.io/en/latest/reference/services/budgets.html)

~~~
davej
That wouldn't have made much of a difference in the particular case of this
article.

------
sudhirj
You did well to have billing alerts enabled. Exactly the same thing happened
to be, but I didn't notice for three months - no emails because I'd created an
account and domain for a side project. Didn't notice anything on my card
because the charge had been declined, but my bank didn't contact me. Finally
found out because I knew the local AWS rep (was the relationship manager for
the accounts we use at work). Had to apologise and explain the situation in
detail to AWS and they forgave the bill. That was tens of thousands of
dollars.

~~~
sudhirj
For those talking about this being a 'serverless' problem or not, think they
point is that it's a lot easier to shoot yourself in the foot. Great power +
responsibility, etc. On regular servers (outside of unbounded autoscaling)
mistakes cost a flat rate.

~~~
21
Not necessarily.

With a regular server you could go viral, your server dies, so you lose also
without a bound in lost business/good will/whatever.

Also need to take into account the time/effort spent on making the regular
server scale, albeit this is also a relatively flat rate.

------
matchagaucho
TLDR: Deployed buggy code with infinite trigger. Hosting costs increased.

~~~
dvfjsdhgfv
The only difference here is that if he deployed it on his own server, he
wouldn't have lost money.

------
Dolores12
"a $180 actual cost. I was left with a light headed feeling, it's a lot of
money for me"

people still play with fire. limit your losses, go with digital ocean or
something for 5$/mo flat no matter what.

~~~
godot
His previous blog post actually said that he moved away from the exact $5/mo
digital ocean plan you're talking about, to this.

I think the author meant to do this less of a "play with fire" way but more of
experimenting with new tech way. But yes, I agree that for personal sites,
running with your own money, you probably want to stick with something safer
like the $5/mo digital ocean box.

~~~
V3loxy
This is indeed the case, playing around with new tech. I've been a happy
customer with DO for years but my own website was the ideal case to try out
the whole serverless thing. As my website doesn't get much traffic so it'd
cost me next to nothing. I do agree with most of the comments here though and
the $5 DO box is the safest choice. I might have been a bit too excited with
the new things and failed to think logically :)

------
rb808
The infinitely scalable cloud services have this big problem with surprise
bills. In the old days teams often made do with what was available, now it too
easy to spin up more resources.

I'm surprised people still talk about cloud services as being cheaper esp
where developers are free to use what they want.

------
abalone
The main issue here is the budget notification emails aren't an adequate
mechanism to catch infinite loops. They are too slow and you've already racked
up big overages by the time you see it.

Idea: use API Gateway to configure a quota to match your budget projections.
That will force a hard stop. Would be nice if AWS made this easier.

------
thorum
I wrote my (small) AWS app so it can run both on AWS and my local machine.
Then you can write tests against the higher-level logic like "save this file
to S3" and run those tests locally as well.

My main challenge with serverless is using Lambda with API Gateway. Lambda has
no database connection pooling, so I end up with a ridiculous number of
connections to RDS - one for each simultaneous user. I haven't found a
solution to this yet, other than not using API Gateway.

~~~
voganmother42
One solution, use external pooler like pgbouncer or mysql-proxy running on a
small instance(s)

~~~
orthecreedence
I'm actually kind of blown away RDS doesn't have pgbouncer installed on the
database. That's how we operate...each db server has pgbouncer living right on
it. We connected to bouncer, not directly to the DB.

------
benologist
There's a lesson in there about how AWS makes a crisp $10m dollar bill for the
richest man in the world every day.

~~~
dvfjsdhgfv
I know you're being sarcastic but I feel it's partly true. They announce so
many different services, each month something new appears, but this very basic
feature - bill capping - asked by users from the very beginning, has never
been implemented. It's hard to believe they lack the skill or that it would be
much more complicated than the current alert system.

~~~
sudhirj
According to the folks I've spoken to, they don't do bill capping because they
have no way to safely shut down your workloads in any way - they'd prefer to
let you know and have you do it. And having that choice is way better than a
capping operation destroying your production database or causing downtime for
your users.

~~~
SimonPStevens
I'm sorry, I just don't buy that. It doesn't have to be a hard cap, it could
be a soft one. i.e. at £x your servers start shutting down, you'll get billed
for a few extra minutes over your cap, before things have finished shutting
down. Servers are totally capable of being shutdown without destroying
databases.

Besides, we aren't really talking about production databases at large
companies. The people who want caps are devs learning and experimenting. It
could come with dislaimers that if you enable a cap and exceed it that your
services will go offline unexpectedly, and that may leave databases in
inconsistent states. But for a large number of usage scenerios that is a
completely acceptable tradeoff.

The simple fact is, not having a cap certainly puts me off experimenting with
a service due to a fear of a mistake causing a big bill. And developers
learning and investigating a technology is what preceeds them recommending
that technology to their companies.

Last time I looked Azure allows a zero spend cap on free accounts, but you
can't change the amount to anything else, and once you remove it you can't
switch the cap back on. Thats limited, but it's perfect for a learning
environment.

If Azure can implement a zero spend cap, there is absolutly no reason that
either AWS or Azure can't implement an x spend cap in exactly the same way.

~~~
vacri
> _The people who want caps are devs learning and experimenting._

Then AWS is not focused on their use-case. You can make the argument that
they're throwing away potential business here, but AWS is already the gorilla
in the room, and people clamour for their products already. A couple of years
ago they were rated as being bigger than the next 17 VPS providers combined.

> _I 'm sorry, I just don't buy that. It doesn't have to be a hard cap, it
> could be a soft one. i.e. at £x your servers start shutting down, you'll get
> billed for a few extra minutes over your cap, before things have finished
> shutting down. Servers are totally capable of being shutdown without
> destroying databases._

The idea of a payment cap _sounds_ easy, but with something as complex as AWS,
it's incredibly difficult, and everyone would demand different behaviour at
the cap. So, you hit your cap. Turning off EC2 servers is easy. What about the
data you've stored on s3? That costs. Should it be purged? What about your
disk drives on EBS, should they be purged? How about items you have queued in
SQS, should they be purged? Are you using RDS databases? They can't be
stopped, only destroyed (you can do a final snapshot, but that's going to go
to block storage, which costs. not much, but it costs).

"Just stop anything that costs" _sounds_ easy, but it's not, not when you have
a service as complex as AWS. AWS's current model of "forgive the bill for
obvious mistakes" is way more workable.

~~~
SimonPStevens
Yeah, I totally accept that people who want caps may not be in the target
audience. That's probably true. But don't pretend it's because the problem is
too hard for them to build, which is what was originally claimed. If their big
customers wanted it it would get built.

Besides, Azure proves that it's clearly possible. Azure have a cap. MSDN gives
you free Azure credits. When you open an account via MSDN you still have to
put in payment details, but you have the option to enable a hard cap that
prevents you spending past your free credits. So Azure have clearly got a
solution for stopping all the services when the credit limit is reached.

All of what you describe as problems are just decisions to be made. S3 data..?
Delete it. Make it read only. Pretend to delete it, but make recovery possible
for x days. Doesn't really matter, just pick one when you build it, and
document what it does. People who want a cap are going to more concerned with
the overspend than any data or service integrity. They could stick up a
disclaimer... "If you enable this cap you data may be destroyed or corrupted
if your spending reaches the cap". There are solutions to the implementation
problem.

Besides, they probably already have all this code in place. If your payment
methods gets declined I'm prepared to bet Amazon don't just let all your
services continue running indefinitely because shutting them down
automatically is too hard of a problem for them to solve. So any cap could be
implemented by just triggering the payment declined function.

~~~
vacri
> _S3 data..? Delete it._

O_o

> _... Make it read only._

The primary use-case of s3 is reading objects. This would not be a deterrent
for quite a few use-cases

> _... Pretend to delete it, but make recovery possible for x days._

Still consumes the space that they're charging for in the first place

> _People who want a cap are going to more concerned with the overspend than
> any data or service integrity._

This is patently not true, and is why I think you don't really grok why
implementing a cap is difficult. It's specifically why I said "everyone would
demand different behaviour at the cap". Some would want only this or that
service to stop, for example.

A small business sets up a payment cap and hits that cap because they went
viral? BAM, all their block storage, destroyed. All their backups, their
analytics, their RDS databases, just gone. Right at the time they needed it
most.That's a _much_ harder lesson to deal with than "oops, our bill's a bit
high because we made a mistake, can you please forgive it?". Or even "ouch,
okay we'll pay it". The protection you want for hobbyists would destroy small
businesses that may not understand what is actually meant when that hard cap
is hit. It's not that caps aren't doable _at all_ , it's just that they're a
wicked problem, and the more you look at it, the more issues you can see.

As for soft caps, what is the functional difference between a soft cap and the
billing warnings they already have?

Also, their claim on s3 is "we don't lose objects". Destroying objects because
of billing would utterly undermine that claim.

> _If your payment methods gets declined I 'm prepared to bet Amazon don't
> just let all your services continue running indefinitely because shutting
> them down automatically is too hard of a problem for them to solve._

AWS does not destroy your services because of late payment. Source: we've just
been in late payment.

~~~
SimonPStevens
I think we are talking cross purposes here. I get what you are saying. A cap
that junks your services would certainly not be suitable for everyone, I get
that.

But this thread was initially about a dev running some experiments and getting
a $200 unexpected bill. They would have been very happy with a $30 cap that
just deleted everything. That functionality would be easy to build if they
wanted to. But they don't. For other reasons, not because it's hard.

The thing I dont buy is Amazon claiming its too hard to build a cap. A cap
that suits some people would be easy. What they really mean is... A simple
hard cap is only useful to customers we dont care about because they dont pay
us enough. An advanced cap with all the kinds of failover options and
thresholds that a medium sized business might want is complex and the people
who actually pay the big money (those we care about) don't actually want caps
anyway.

~~~
dvfjsdhgfv
> A simple hard cap is only useful to customers we dont care about because
> they dont pay us enough. An advanced cap with all the kinds of failover
> options and thresholds that a medium sized business might want is complex
> and the people who actually pay the big money (those we care about) don't
> actually want caps anyway.

That's an honest answer, I'd be happy if they formulated it this way.
Fortunately, there are other cloud providers with billing cap implemented
properly, and you don't hear horror stories about them (problems with spending
too much on AWS are very common though).

------
sah2ed
Charged at a flat rate like the cost of a Digital Ocean $5/mo instance, would
developers pay for such a service to provide automated notifications of
service overages for all the major cloud providers?

All a developer needs to do immediately after adding a credit card to
AWS/Azure/GCP would be to create an IAM role with permission to automatically
add and track fine-grained billing alarms and notify via email/sms for any
potential billing overages.

I think a $60/yr service like this would be useful to protect against future
events of bill shock.

~~~
autotune
Looks like it's already been done, at least for AWS, based on a few Google
results (not that I've used/tested any of these):

[https://github.com/Teevity/ice](https://github.com/Teevity/ice)
[https://billgist.com/](https://billgist.com/)
[http://cloudcheckr.com/](http://cloudcheckr.com/)

------
gleenn
He obviously has a smaller account so Amazon might be less flexible, but it's
worth contacting support and explaining the error. Sometimes they do give your
money back in my experience.

~~~
qaq
In my experience flexible starts north of few mil. per month spend so why all
the startups are running on AWS is a mystery to me.

------
amcleod
Not sure if someone has mentioned this already, but you should contact AWS
support and ask if they will forgive the bill given it was an error which led
to the high costs. I’ve had bills forgiven this way in the past (e.g. forgot
to disable an instance that wasn’t really doing anything).

------
supertramp_sid
I was about to make the same mistake yesterday , but I had written a
validation function that would check inside a folder only, and fortunately I
did not upload the file in that folder. And the next morning , I read this
article... Man I gotta be careful LOL

------
j45
The relatively low barrier to learn just a tiny bit of following a Linode or
DO vps hardening & stack setup guide to get an ubuntu server going can go a
very long way for development and prototyping environments.

It's gotten much, much easier, and is just another form of command line
management, similar to the CLI framework tools with your preferred stack.

Once that first setup is done, similar to setting up a serverless environment,
you are generally restoring backups of your base image and beginning projects
from there.

It also immensely helps to learn about how to build something to scale that
isn't completely reliant on the PaaS layer.

------
odammit
I've built a fair amount of serverless services over the past two years using
the Serverless framework, apex and straight API Gateway/Lambda.

It's nice not to have to worry about a server, but I feel like there are just
as many little things to futz with in serverless architectures especially
before "environment" variables existed in Lambda.

------
mullen
Setup spending alarms for your account. Personally, mine is $5, $10, $15, $20
and so on. At $30, my wife gets paged.

------
tapirl
The cost by using AWS is hard to foresee. For years, my s3 storage got charged
nothing. But some a month, it got charged several dollars.

I have migrated all my services to GCE. At least GCE provides free decent
quotas for every resource.

------
solidsnack9000
Serverless is going to make resource usage a focus in a way it hasn't been for
years. The quick feedback, the absence of "all you can eat" pricing and the
possibility for savings are all factors in this.

------
tomc1985
Run+know your infra, none of this is a problem. Serverless is a scam.

~~~
lostcolony
So run my static content only blog on dedicated hardware that I have to
administer rather than throw it in an S3 bucket with a Cloudfront on it? No
thank you.

Qualify your statements.

~~~
dvfjsdhgfv
You don't really need to administer it, there are plenty of hosting options,
not just bare metal. And you can always add Cloudlflare anyway.

~~~
lostcolony
You're missing my point. "Serverless" just means "Applications built on cloud
services rather than server(s) I have to administer".

The OP said I should run my own infrastructure. I -could- host my blog by
running a web server atop a server I administer, sure. I'd have to take on all
the infrastructural tasks of doing that, securing it, ensuring any
availability/scalability concerns I may have are taken care of, etc, but I
-could- do that.

Instead, S3 + Cloudfront (or, sure, any flavor of hosting and edge caching
options you care for; I was not implying "Just AWS") means I don't have to
worry about any of that. For me, the reduced level of control, increased
availability, scalability, and easy "it just works", is worth the tradeoff. As
is the pennies per month it costs me given the low utilization and pay-as-you-
go model. It's hardly a scam.

~~~
dvfjsdhgfv
I wouldn't call it a scam. You need a certain level of expertise anyway,
otherwise you not only screw up your service, but also, in the case of AWS,
lose money. Practically speaking, the amount of work needed to set up a blog
on S3 with CloudFlare is not that different from the one needed to deploy,
say, a WordPress droplet on DigitalOcean. You just click a few times and it
works, you can basically forget it. The only difference is that you are
guaranteed not to pay more than $5 or whatever your monthly plan is.

Of course scalability is incomparable in both cases, so if it's something that
really matters - and matters more than money - of course something like AWS is
a better choice.

------
emersonrsantos
Serverless still is a marketing tool for cloud providers. It will be useful
when it really offers advantages over managing your own servers, especially on
the cost and debugging.

------
cryptos
Hey, at least is was an enterprise cloud scale infinite loop.

------
brown9-2
You can host a static site on S3 + Cloudfront, so what purpose does Lambda
have in this picture?

------
archii
>Keep an eye on your logs, test everything again and again.

This is the takeaway quote from this for me.

------
illuminati1911
This has nothing to do with serverless itself but it's rather problem of AWS.

------
le-mark
Off topic, but the "serverless" moniker needs to die. I propose "adminless" as
in "server I don't have to admin, configure, or patch" as being much more
descriptive of whats really going on.

~~~
nathancahill
Eh, if I'm deploying a cloud function, the server truly doesn't exist for me.
It's more like a Web Worker running in a privileged environment. I'm ok with
the name.

~~~
sidlls
No, it's really a server running your function. You just (are told you) don't
have to worry about the server or what it's actually doing.

~~~
dvfjsdhgfv
> You just (are told you) don't have to worry about the server

...until you receive the $206 bill for the work done by the server.

------
danschumann
Fun programming story!

------
w8rbt
Endless loops are now billing issues, just like a DOS.

------
alsadi
If you are big enough and want serverless no-ops yet don't want to pay per
burger when you already breed cows then consider kubeless.io

------
fibo
I have a 5$ AWS billing alarm.

