
Show HN: List of serverless failure stories - alien_
https://github.com/cristim/serverless-failure-stories
======
bwc150
Really don't understand why anyone puts a side project on a pay-per-use
provider when 99% of the projects would be perfectly fine on a $5/mo max vps.
Although I also use a prepaid cell phone provider, so maybe I just hate the
idea of blank checks.

~~~
nostrebored
You get 1 million free lambda invocations per month, completely free of
infrastructure headaches for simple projects.

You can also make a CloudWatch alarm on billing to throttle your lambda
functions at your specified price point. It's only a blank check if you want
it to be.

~~~
geezerjay
> It's only a blank check if you want it to be.

Doesn't AWS force customers of their free tier to provide a valid paying
method (i.e., credit card) to be charged once any limit is surpassed?

~~~
TYPE_FASTER
You can throttle serverless services by invocation count. AWS will alert you
when an account exceeds a budget, Azure used to let you put a hard limit on
the spend, not sure if they still do.

------
chb
The scenarios linked to from the GitHub page are not only related because they
deal with serverless technologies. They're akin to one another because, in
every case, testing was flawed.

In the first, there was no testing, aside from the trivial. Oddly enough, a
lack of formal testing is described as being a symptom of not having "a local
version to play with." In the second, the test environment was used in
production ("So that we didn’t have to wait for the timer to fire “naturally”
when we were testing, we were using RunOnStartup = true"). And in the third,
there is no test environment.

~~~
alien_
Thanks, that's a great summary.

We can add a tl;dr with the root cause for each item in the list. Feel free to
send me a PR with this.

------
zackmorris
Coming at this from a purely computer science standpoint, I am concerned about
"manualization". I don't have a better word for it yet, but it's when humans
monitor and control services at a granular level instead of letting a compiler
partition and parallelize code internally.

So with AWS Lambda, humans have to decide which microservices should be put
into lambdas, and how much they need to be split up (based on the AWS Lambda
Limits, which increase over time).

Whereas in a language like Elixer/Erlang or Go, we write code at a high level
and the compiler chops it up into threads and spreads it across as many
processors as are available dynamically.

I feel, rather strongly actually, that the manual approach is a code smell.
Which means that serverless leans towards being more of a trendy solution than
a formal one built from first principles. Which means that given time (usually
1-3 years with web tech) we'll see something arrive that looks like a compiler
and lamda-izes code somewhat automagically.

That said, there is a window of profit here where anyone who jumps on
serverless and knows how to use it can generate cashflow.

Knowing all of this, I would look at previous evolutions of other web
technologies for pitfalls to avoid. Having never used AWS Lambda, this is what
I would do:

* Treat lambdas as single input/output executables like in the shell. Analogously to avoiding how AngularJS got mired in two-way data binding. This lets us use vast prior experience in data processing graphs rather than having to analyze complex webs of nondeterministic logic and side effects.

* Start from the beginning with development, staging and production environments. Just because it's lambda doesn't mean that we throw out everything we've learned about reliability just to run everything "serverless".

* Don't succumb to premature optimization. All logic can be reduced to functional programming, the highest expression of which is the 0th order functional programming of spreadsheets (which I saw making the rounds of HN the last few days). Write all code to not depend on global shared state. Even if it feels a little inefficient at first, try to do everything declaratively, as functional transformations on data rather than imperative steps storing temporary state (in class member variables, as an unfortunately common example). Then analyze the code near the end of the project and decide which functions might work better as lambdas. Maybe they are utility methods that don't justify having an entire container overhead around them. Maybe they are just glue code that swizzles one interface into another like GraphQL. That's something you'll have to decide.

I've seen problems with highly-partitioned code where versions of included
services get out of sync, the friction of dealing with things at the service
level outstrips any gains found by separating the code, and it becomes nearly
impossible to trace the codepath across services. It becomes obvious that
whoever wrote the code wasn't familiar with things like the Actor model. It's
fine to break the rules once one is familiar with them, but serverless has a
fair amount of brogramming thrown in and the move-fast-break-things mentality
is just not compatible with reliable software engineering IMHO. I know, I
know, Facebook and Amazon made billions with that approach, but guys like me
have to clean up the mess and it's anathema to finding career zen.

Edit: P.S. I forgot to say why monolithic code that is parallelized/optimized
internally is usually better (for both rapid-prototyping and long-term). It's
because the closer that code is to fully abstracted and statically-analyzable,
the more reliable, reusable and resalable it is. This is my rudder after many
years of programming, but I look forward to hearing any other opinion someday
that might replace it.

~~~
cpeterso
> Whereas in a language like Elixer/Erlang or Go, we write code at a high
> level and the compiler chops it up into threads and spreads it across as
> many processors as are available dynamically.

I haven't used Elixir/Erlang (though I've read a fair amount about them), but
it seems like the BEAM languages would be an ideal model for serverless
because processes don't (have to) know which thread, core, or machine they're
running on.

When people deploy Elixir/Erlang in production, do they use a single
distributed instance or do they run separate instances in their own
containers/VMs like you might with something like Node?

