
AWS Lambda and .zip is a recipe for serverless success - kiyanwang
https://medium.com/@PaulDJohnston/why-aws-lambda-and-zip-is-a-recipe-for-serverless-success-4f547a6e87c3
======
fapjacks
Cloud commoditized datacenters. People then tightly coupled their applications
to cloud providers like AWS. After some time, containers came along and
commoditized the cloud providers. Those applications no longer needed to be
tightly coupled to a specific cloud provider. This reduced both costs and the
risk of being invested in a single cloud provider.

Serverless (AWS Lambda) is just the cloud's way of trying to "de-commoditize"
the containers that commoditized them. They want you to tightly couple your
applications to their specific cloud provider again. And charge you more. How
much is that function in the zip from S3 going to cost you to run in a
container managed and run by AWS? And how long did you spend configuring the
API Gateway (and where else can you apply what you've learned doing the
configuration)? Next time you see an article fawning over serverless and
saying things like "Containers just don’t matter in the serverless world" take
a look at what the author does for a living. You'll start to see the same
pattern I've been seeing.

Meanwhile, you could be spending less money and time treating AWS like a dumb
processor with a network connection while only really configuring the open
source software you already know. But it's your time and money.

~~~
staticassertion
I just don't get the "lambda locks you in" thing. You write a server the same
way you always would, but like an extra 50LOC exists to make it work on
lambda. So if I wanted to deploy elsewhere what would the big deal be?

Lambda also provides a lot more than containerized services, as the article
mentions - I no longer have to patch my system, which is a huge operational/
security burden that many companies struggle with. The time to configure API
Gateway/ Lambda feels trivial compared to the work involved in maintaining a
patched FAAS solution.

~~~
paulgb
I agree, but in my experience, the lock-in with using Lambda is that you
become integrated with other Amazon services (like S3 and DynamoDB) that are
harder to replace if you abandon AWS entirely.

~~~
staticassertion
That really doesn't feel like it's inherent to Lambda at all. You could say
the same thing about EC2.

On top of that, many of AWS's services use standard API's. It's not hard to
move away from RDS or Elasticache, you're just using SQL/Redis anyways...

If we're talking about something like SQS, ok, well you can just as easily say
you're "locked in" when you use RabbitMQ because there is work involved to not
be using RabbitMQ.

~~~
np_tedious
There are a finite set of this to trigger the function and they become central
to

API Gateway, kinsesis, s3 change/addition...

It definitely has more lock in potential than more "vanilla" offerings like
EC2, RDS, S3.

You could name a replacement/competitor service for each of the above, but if
the Lambda is triggered by S3 changes then that's a big barrier to putting
your static files elsewhere

------
nine_k
I keep being dazzled how what's basically "cloud CGI" is such a runaway
success. Utter simplicity beats many considerations.

(For those who did not write web apps in 1990s: CGI here is
[https://en.wikipedia.org/wiki/Common_Gateway_Interface](https://en.wikipedia.org/wiki/Common_Gateway_Interface))

~~~
roywiggins
It certainly does seem a throwback to old managed PHP environments. No need to
set anything up, just provide your PHP code over FTP and off you go!

~~~
Zelphyr
I agree with you in principal but AWS and Lambda feels significantly more
complex than that. At least when I was FTP'ing my code up to some shared host
I still had a lot of control over my environment (ini overrides, htaccess,
etc...) With Lambda I feel like I have very little control. There is a black
box running this stuff that I have no insight into. Even worse; when things
break the resulting error is often unrelated to the actual problem.

~~~
benjaminjackman
one thing that would be great would be if it was setup in such a way that the
API gateway / lambda functions had really simple, file based mappings on a
virtual server that then automagically took care of deployment / scaling for
you.

cp -r / symlinks could take care of the whole staging / lambda version
aliasing stuff.

deployment is just scp'ing a zip file to a certain defined location.

all that insane complexity of configuring API gateway is replaced with some
config.json or js file export defining what lambda to call and what
permissions to use.

then to back the whole thing up you could just rsync it back to your local
machine ... hmm though realistically you'd probably want to invert things and
develop this whole hierarchy in a git repo that could be checkouted on that
virtual server, or it could just natively understand git and branches just
become the stages / alias as soon as you push them so no need to ssh in and
pull.

~~~
xGrill
Apex Up does something very similar along these lines.

------
nzoschke
Deploying .zip files is one of my favorite features of Lambda, particularly
when writing Go apps.

Docker always felt wrong with Go to me.

Why do I need a local VM, a Linux build service, an image registry and servers
with container orchestration to deploy Go software? I understand how all this
helps with "legacy" Rails or Java apps, but why can't I just throw a Go binary
somewhere to run it?

Lambda is exactly this. I upload a cross-compiled .zip to S3 and everything
else is taken care of. This was a big breakthrough for me, seeing a much
simpler solution for deploys than all the container stuff.

I've been building a boilerplate app that demonstrates just how little "stuff"
is needed to run a Go service on Lambda:

[https://github.com/nzoschke/gofaas](https://github.com/nzoschke/gofaas)

More thoughts about how .zip files make our lives easier is here:
[https://github.com/nzoschke/gofaas/blob/master/docs/dev-
pack...](https://github.com/nzoschke/gofaas/blob/master/docs/dev-package-
deploy.md)

~~~
fapjacks
You don't _need_ all of that shit, though. You absolutely _can_ just throw a
golang binary into a super minimal alpine image and run it wherever you've got
a Docker daemon installed. You literally don't need _any_ of the things you
listed! Plus, it is simply untrue that "everything else is taken care of" in
Lambda. There is plenty of one-off configuration to do ("toil").

~~~
subway
If you're just working with a golang binary, you shouldn't be using an Alpine
container, you should be putting the binary in an empty, from-scratch
container. Go binaries are typically static and don't need distro or the libs
it provides in their container.

~~~
fapjacks
Sure, I'm just saying that none of that stuff he says is necessary is needed
_at all_.

------
mirceal
Serverless has its upsides but it also has downsides.

In an ideal, pure, world it’s awesome. The problem is that your lambda
function usually needs to use some sort of external resources. (It is pretty
useless if you don’t interact with anything)

Now you have to worry about and understand how to do access control on the
said resources, how to collect the logs, how to collect metrics relevant to
your code, how do you troubleshoot and deploy the function itself, how do you
integrate it with the rest of things you have/own.

You’re just shifting complexity from an area to a another area.

It works great in some cases, but it’s far from beeing a panacea.

~~~
igammarays
Right, and one of biggest downsides to serverless: reusing database
connections/pools between requests. Right now the only "standard" solution
offered by serverless's biggest advocates is pretty much a hack: rely on AWS's
preservation of memory across some warm invocations. Basically keep it in a
global variable and if you get lucky it'll still be there the next time your
functions runs.

[1] [http://blog.rowanudell.com/database-connections-in-
lambda/](http://blog.rowanudell.com/database-connections-in-lambda/)

~~~
devonkim
I think a mixture of persistent proxies could work well with lambda functions
for this use case. You can run lambdas in your VPCs with assigned security
groups and IAM roles just like an EC2 instance and therefore lambda functions
should be able to proxy through an intermediate server that performs the
connection pooling on behalf of calling functions. Not sure what the standard
solution for something like this would be though because every time I've seen
connection pooling it's for use locally. Perhaps some nginx or haproxy hacking
could make it work.

~~~
matt_wulfeck
In this scenario you’re still paying the cold-connection cost to your proxy,
and that’s just keeping a pool of tcp connections open, not necessarily doing
anything smart/optimized in the sql driver.

------
keithwhor
Reading through this comment thread, if anybody is interested in a much more
streamlined “serverless” experience and superior API Gateway, check out StdLib
[1]. We’ve built a product for getting APIs built and shipped, full stop.

Bonus: .tar.gz format for submitting packages instead of .zip (though our CLI
handles it automatically), and a bunch of freebies - everything from
automatically generated documentation [2], SDKs, simple local development and
testing, and more.

Disclaimer: Am founder.

Disclaimer 2: At the risk of being borderline spammy, we’re hiring
aggressively. If you’re passionate about building a future that uses
serverless technologies to make software development and APIs more accessible,
e-mail me. (And thank you, pre-emptively.)

[1] [https://stdlib.com/](https://stdlib.com/)

[2]
[https://stdlib.com/@messagebird/lib/numbers/](https://stdlib.com/@messagebird/lib/numbers/)

~~~
appdrag
Check also a more visual approach to Cloud Backend including an sql cloud
database and an API builder :
[https://cloudbackend.appdrag.com](https://cloudbackend.appdrag.com)

Disclosure: I'm founder

------
nudpiedo
I feel it is like reinventing executables, the dumb way.

We can compare that with ELF binaries or remote procedure calls when someone
else maintains the OS and makes sure that the infrastructure scales. Actual
not even executables, because it is still interpreted code, instead of being
"compiled" it is packed and distributed with a spec much less precise than a
typical platform. Just because someone has a machine ready to be deployed on
and scale ad infinitum.

Am I the only one who finds that a dumb protocol for a remote call? Could
someone point on which part is this a smart badass thing? Perhaps it is just a
weird "renascence of Operating Systems"?

Edit: I still think that we got here because it was too much work to learn
about pre-existing technology or hire an expert and it was more
fashionable/easy/cool/sexy to kick ass and move on.

P.S. Sorry for this sudden rant

~~~
bamboozled
I agree with this sentiment and feel the same way. I also find this is a
frustrating trend in our industry in where a lot of engineers seem to be
spending more time learning about "services" from specific vendors than about
things like network protocols and their workings and limitations.

~~~
sedachv
> Maybe this is just inevitable evolution of the IT industry but I've noticed
> two things that are alarming.

I was at the Southern California Linux Expo[1] earlier this month, and one of
the really weird things was how many people from Microsoft were presenting and
attending (I wasn't the only one - Bryan Lunduke, ex-MSFT, did a podcast
episode about it[2]). I think the trends you point out are from a lot of
Microsoft, IBM, Oracle, Java, etc people now being forced into the Linux
world, because Linux is now the default for new IT projects. Unfortunately
they are bringing along their "practices" instead of retraining in good
software systems engineering.

[1]
[https://www.socallinuxexpo.org/scale/16x](https://www.socallinuxexpo.org/scale/16x)
[2] [http://lunduke.com/2018/03/10/microsoft-headlined-a-major-
li...](http://lunduke.com/2018/03/10/microsoft-headlined-a-major-linux-
conference/)

------
chapill
I kept waiting for the author to say, "Surprise, it's a jar file. Jars are
just zips, and lambda supports Java." Then I realized author really was
reinventing wheels for the reader who probably doesn't know what Java is.

~~~
nudpiedo
I am totally with you... I think we are going backwards in technical choices
just because some people had pain-points learning pre-existing technology or
did not know how to deal with infrastructure. But if people pays for it, it is
a market I guess.

~~~
byronmagnusson
Bingo!

------
wrmsr
It's just code in a .zip file! It's really that simple!

Hey, unrelated what does "cannot open shared object file" mean?

------
TAForObvReasons
Does anyone have experience with alternatives to AWS Lambda (like azure
functions)? How hard is it to switch between serverless providers?

~~~
faitswulff
I played with Firebase (Google Cloud) Functions recently and the developer
experience was pretty good. It's like if serverless
([https://serverless.com/](https://serverless.com/)) had first party support,
in the form of the Firebase CLI. Deploying was simple, the Firebase dashboard
is nice. That said, I only used it for a side project, so I don't know about
any pains scaling. I would definitely use it again for another side project,
though.

~~~
bigtunacan
I will second Firebase. It has really matured as a platform over the past
couple of years and is a much easier service to work with than AWS.

As to the parent’s question on switching cloud providers though I think it
would be non trivial.

------
TheDong
The comment about meltdown is wrong.

> You had to patch your containers and your instances/servers, but you didn’t
> have to patch Lambda functions.

This was right after talking about Fargate. Let's compare what you do and
don't have to patch in Fargate / Lambda:

Kernel / host OS: Lambda and Fargate patch that for you without any work. This
is the more important patch.

Container bins/libraries: some programs, like Chrome, which had JIT, could be
exploited to read memory of that same process. In the case of lambda and
fargate, you only needed to patch containers/zips that contained such
programs.

If you were using something like 'serverless-chrome'[0] in lambda, you would
have to update your zip file to get Chrome's workaround for meltdown. If you
had a fargate container with headless chrome, same deal. It's practically
identical in the cited case of meltdown.

There are many cases (like glibc or openssl vulnerabilities) where containers
need to be patched, but lambda can patch it for you ... but in the case of
kernel exploits, fargate and lambda can patch equally well.

[0]: [https://github.com/adieuadieu/serverless-
chrome](https://github.com/adieuadieu/serverless-chrome)

------
themihai
IMHO Cloud functions serve well a very small niche. However most of the people
advocating for cloud functions seem to misused them. Their use cases would be
better served using a well designed PaaS(though I admit is hard to find a good
one these days).

------
bahmboo
Don't think that building your whole service on lambda or google cloud
functions with tons of code is the best idea, but it's a great shim between
clients and backends. Tech changes constantly and you'll always be up against
a change agent. Be nimble.

------
tracker1
I use containers (Docker for Windows|Mac) to create my lambda zip for dist,
isn't that close enough? lol.

~~~
jbrambleDC
I actually do the same thing. Do the dependency install inside of a docker
container in order to build the .zip. This is absolutely essential for
compiled libraries in python or anything that involves postgresql. If you
attempt to install psycopg on mac and then deploy that on lambda it will fail.
So you either need a prebuilt version for ubuntu that you keep around, or you
need to build in a docker container. We just build inside a docker container.

~~~
tracker1
Yeah, it works really well, also allows me to configure CI/CD in VSTS/TFS via
Windows agents for AWS deploys to Lambda. In the end, I'm pretty pleased with
it. I've mostly used Node, but Python binary dependencies would have the same
need.

Not entirely sure why I got the downvote, other than the tongue in cheek
aspect and "lol" at the end.

------
alexcroox
I actually find zipping and uploading manually a PITA. Started using the
Serverless framework [1] recently and now I just run `serverless deploy
--stage dev --aws-profile profilename` from my repo (which is an npm script).

[1][https://serverless.com/](https://serverless.com/)

~~~
moondev
No doubt! It seems antiquated and repetitive

~~~
dasil003
Well that's what CI/CD is for. It's not any different than building and
pushing a new docker image.

~~~
fapjacks
I keep hearing responses like this about various aspects of serverless, but it
makes no sense. If a lot of these serverless operations are "not any
different" than various Docker operations, why should I be spending time and
money to couple my application to AWS Lambda versus simply running the
container myself? Why shouldn't I just deploy a container (pod) for my code if
I'm _already running kubernetes_?

The more I hear people defending serverless, the more it feels like they're
asking people to voluntarily increase their time and money costs, learn
hyperspecialized and nontransferable skills, and make their application far
less adaptable. All of this while accruing tons of technical debt for a
pittance of some convenience.

~~~
dasil003
To be clear, I'm not a proponent of serverless at all. I'm just pointing out
in isolation that zipping and shipping something is not antiquated, it's just
a methodology for shipping code. I guess it feels old because it's not wrapped
up in a lickable CLI and online docs with great typography, but it's not
fundamentally flawed.

To your point, I don't see any value of serverless over containers today. We
are now at a point where container orchestration is mature enough that it's a
sane default choice, but it took a while to get here. Serverless is where
containers were 5-7 years ago (depending how you squint).

Will serverless ever be a credible replacement for containers? That depends on
the vendors working out a whole lot of details. In theory, with the right
services around it, serverless can be lighter than containers because they can
fully control the baseline environment and build more into it, but that is
also less flexible than containers. How much standardization will be possible
between vendors? If the price comes down and the tradeoffs look good enough
then I could see it eating into container marketshare, but I'm really not
holding my breath. It's still 100% in the hype phase.

------
idbehold
You know what would be better than a .zip? A .tar.gz file. Zip files cannot be
unzipped without having the whole thing in memory. What's funny is that the
Lambda environments (at least for Node) do not have the `zip` executable
available which means that to zip/unzip something inside Lambda I need one
written in javascript which has to be included in the zip file I upload to AWS
instead of being able to use the native one.

~~~
skybrian
You have it backwards. Zip files have a table of contents letting you unzip
individual files without decompressing the whole thing, and you don't need to
load it all into memory.

~~~
sheetjs
The table of contents in a zip file is at the _end_ , not the beginning, so
you need random access to the full file if you want to properly jump to the
right location. Some zip files use the length feature of local file headers so
you can jump ahead in the file without processing the data, but many skip the
length so you have to work through the DEFLATE algorithm to find the end of
the first file (to find where the second file starts)

~~~
skybrian
Yes, you can and should use random file access to load things from a zip file.
For example, Go's zip package takes an io.ReaderAt interface, not an
io.Reader. Using sequential I/O for this would be inefficient.

~~~
jlouis
The problem here is that you mostly have a streaming interface In s3.

~~~
skybrian
I'm not familiar with s3, but it seems like if you want random access for a
key-value database, you should store each file under a separate key? Storing
an entire file archive as a blob (in any format) and downloading the whole
thing just to access one file seems like a weird way to do it.

Apparently there is "s3 select" in preview [1], but it only supports gzipped
CSV or JSON.

[1] [https://aws.amazon.com/blogs/aws/s3-glacier-
select/](https://aws.amazon.com/blogs/aws/s3-glacier-select/)

