
The largest benchmark of Serverless providers - kiyanwang
https://medium.com/elbstack/the-largest-benchmark-of-serverless-providers-ac19b55750f4
======
Dunedan
Last week I attend a Meetup where the author of these benchmarks gave a talk
about his experiments. While the goal is noble, the benchmark suffers from
some serious problems:

\- it's just a snapshot of the performance at one point in time

\- the number of benchmarks is way too low to be statistically significant

\- the benchmarks were invoked from a local PC, so the overhead tests did
depend on the location of the author and the network he was connected to

\- the benchmarks don't consider factors like AWS Lambda functions being
powered by different types of EC2 instances

Because of these problems I wouldn't count on the results, but as the author
himself points out, it might be a good starting point for constantly
collecting benchmarks of Serverless providers:

> To improve quality and validity of this data, I want to create a service
> that continuously collects data by requesting serverless functions, every
> hour and every day and provide the recent data on a dedicated website. This
> will cause quiet some costs for the serverless requests, so if you or your
> company (talk to your boss!) wants to support me in collecting this data, I
> plan on setting up a Patreon or OpenCollective account for that. For now if
> you’re interested, just leave your mail here and I will contact you as soon
> as the project starts.

------
arpinum
The most popular use cases for serverless are gluing different services
together, where critical metrics are network throughput, network latency, and
CPU scheduler. Fibonacci is not an interesting use case on its own, nor is
time-insensitive computational work in general. Sustained 100% CPU usage is
not affordable on these platforms.

When used in conjunction with a database, KV store, or blob storage, total
latency for the end user is dominated by the CPU scheduler for small
instances, and network quality for larger instances. Except Azure of course,
they are still trying to figure out how to bolt serverless onto App Services
without re-engineering the stack.

