
Going Serverless: AWS and Compelling Science Fiction - mojoe
http://compellingsciencefiction.com/blog/2016-11-10.html
======
falcolas
The biggest pain for me in managing a system is handling all of the service
configurations; making sure the ACLs are in place, that the configuration
files are all in the right place and in the correct format.

Serverless isn't saving me any of that pain. I still have to configure ACLs
for _everything_ , web folders in S3, ensure that my backend isn't hitting any
concurrency or timeout limits, ensure that all of my routes are set up
properly in the API gateway, that my DB queries are properly tuned and that
someone is watching that DB...

All it's really buying me is not having to write Ansible scripts, but instead
I have to write CloudFormation templates. Sure, I have to think about
maintenance and troubleshooting less, but when I do have to do
troubleshooting, I'm for a long, frustrating day.

As easy as it is to create VMs anymore, the serverless story is nowhere near
as compelling as it should be. It makes the easy things easier, and the hard
things really f'ing hard.

Concrete example: I wanted to just store the contents of a github webhook post
in S3, after verifying the hashed secret. Should be a simple case of wiring
together the API gateway to S3, right?

First bug: You can't test an API Gateway -> S3 connection if they are both in
the same region. Known issue from back in 2014.

First hurdle: You can't pass the contents of a post to an auth hook in API
gateway, just a single header value. That means I can't use the API Gateway
authentication hook for this purpose; github creates a HMAC hash of the post
contents.

Second hurdle: Finding the proper Velocity Template Language function (and its
location in Amazon's custom libraries) to escape the JSON from the webhook
body so I could pass it on to a lambda function. It's
'$util.escapeJavaScript($input.body)' by the way. You're welcome.

By this point, I was wishing I had just set up a t2.tiny server running flask.

------
ohstopitu
Let's talk about serverless/lambda/FaaS for a minute:

I'm currently working on a side project (which if successful can easily be a
product in itself) to create a service that would be the Digital Ocean of FaaS
(lots of talk - I understand).

What would be the core features that devs/users/managers are looking for?
(something you downright expect for v1 that would compel you to put the
service to use for development/production)

Currently some of the items top of my list (in no particular order) are:

* API development (like AWS's APIG)

* Support for more languages by default

* Ability for users to a custom defined language easily

* Creation of an opinionated framework / workflow for FaaS development (you don't need to do this if you don't want to...but FaaS first apps would be done better/easier this way)

* Rewriting common products (Wordpress, Ghost, Jekyll etc.) for to be served via FaaS.

Things I don't want to work on (or would prefer to not jump into):

* Storage service (S3 alternatives)

* Databases (there are so many out there...building a cloud is hard, building a database is even more so)

Who do I feel would pay for this?

* Front end & Mobile devs who don't want to complicate a backend

* Mobile devs who don't want to complicate a backend

* Enterprises maybe?

~~~
jedberg
1) Loop detection -- don't let me shoot myself in the foot an make an infinite
loop of two functions calling each other

2) Multi-datacenter -- I want to deploy to two places at least so my risk goes
down

The lack of database and/or storage would be a big showstopper. It would mean
that if I want build anything with state, I need to store it in another
service, which costs money to move data out of. Even a very basic key store
would better than nothing. You could actually build one pretty easily on top
of Postgres or even SQLLite (although you'd need one per customer w/ SQLLite).

~~~
mdergosits
Wouldn't loop detection be equivalent to solving the halting problem?
Obviously a naive approach would be disallow recursion/ mutual recursion, but
there are valid use cases for those that may not result in infinite execution

~~~
harlanji
I helped design a large scale JS execution engine. We used Babel to rewrite
code with loops that could be interrupted by our system. Eg.
`for(...){mycode();}` would be recompiled to
`for(...){if(!running()){halt();}mycode();}`. The technical reason is that
tight loops wouldn't respond to SIGINT, if I recall correctly. So while it
doesn't stop infinite loops, it can control the program. I wasn't involved in
productionizing it.

~~~
duaneb
Sorry--couldn't be interrupted by SIGINT? That's arguably the point of
signals. Does the runtime just queue signals rather than respond in the signal
handler itself?

------
vesak
Doesn't AWS constitute a total vendor lock-in?

~~~
jedberg
If you just use Lambda, S3, and Dynamo, no not really. It's pretty easy to
move to Google. If you use more of the ecosystem, yes, but so what?

Are you worried about vendor lock in with your power company?

~~~
boulos
As you know, I've had to deal with a lot of these and certainly S3 is the
easiest because GCS roughly supports the S3 XML API (we do resumable uploads
differently). The sad thing is that even when you say "oh just rewrite this
piece to be Datastore or BigTable instead of Dynamo" that can be a pretty tall
order. I think "pretty easy" depends on the organization, and how tied to a
single provider it was in the first place. That said, I think it's a mistake
to "avoid lock-in" religiously, because otherwise you're really getting a
lowest common denominator experience. You really should use BigQuery if you're
on GCP, and if you need to leave it might be tough. But the alternative is
worse (using an inferior service or hand-rolled system from the get go).

P.S. As we're both in the same region of California, we're both with PG&E and
can't do anything about it. It's kind of okay because they're so regulated,
but how do you know we're getting a good price per kWh?

Disclosure: I work on Google Cloud (and met jedberg in person once).

------
methurston
Tangentially related question. What deployment tools are people using to
manage systems with many lambda functions?

We use SWF to trigger over 50 separate lambda functions in processing. We've
got some very nice internally developed tools to identify which functions are
out of date and help with deployment. I'm just curious what else is available
to handle DevOps tasks in a Serverless environment (i.e. deploying library
updates, etc).

~~~
ak217
I'm not aware of a publicly available tool that does this (without bringing in
a big clumsy framework), but internally we just use a script that idempotently
redeploys all the lambdas by calling update_function_code on them. It ingests
package dependency changes by running "python setup.py install" and grabbing
the resulting site-packages contents. This script can be subscribed to an SNS-
SQS bus that listens to GitHub notifications for changes on a branch, etc.

This will flush hot lambdas, so if you have a lot of traffic it can be a bit
disruptive. I can imagine comparing deployment and commit timetamps make-
style, or calling update_function_configuration(Description=...) to save and
later compare version/commit metadata.

AWS recently came out with
[https://github.com/awslabs/chalice](https://github.com/awslabs/chalice),
which is pretty sweet if you're using Python (although it doesn't seem to
directly address the concern of deployment orchestration).

~~~
methurston
I've looked at chalice very shortly, and it was really nice to easily set up
an API Gateway -> Lambda connection, we haven't really looked at it for our
non api-gateway uses.

------
mojoe
Has anyone used the AWS API Gateway service with lambdas to set up a small
backend for a web service? The next thing I'm thinking about setting up is a
slush reader with a GUI, and I'm wondering how difficult it will be to pipe
binary files from S3 -> Lambda -> API Gateway -> client. It looks like I'll
probably need to encode the binaries to base64 on the lambda side, and will
need a library for that.

~~~
balls187
Yes extensively.

API-Gateway => Lambda isn't fun to trouble shoot.

That said, once you get it working, it works really well, and it's
ridiculously cheap.

~~~
mojoe
Have you written anything about this, or can you point me to any resources
that might help me avoid pitfalls? I'd appreciate it!

~~~
balls187
Sadly, no I haven't written anything. (if my comments are any indication--I
kinda suck as a writer).

The biggest pains are:

* Mapping parameters in API Gateway

* Setting the IAM Roles for API Gateway & Lambda (the latter is required to provide execution for API Gateway)

* Setting up custom domains for your API (it uses cloudfront, but must be done via the commandline, and the process kind of sucks)

I've done this work now 3 times. Feel free to email me any questions you have:
balasuar@gmail.com and I'll be happy to answer, and provide guidance.

~~~
mojoe
Thanks, I may take you up on that down the road!

------
mooneater
Any comments from people who are happy with testability of their Lambda code:
how much testing were you able to achieve, and how did you achieve that?
Indirection, mocks? Frameworks? I am thinking python, but any language would
be interesting.

------
jflowers45
Thanks for sharing. I have used SES previously but hadn't thought to use it to
route INCOMING emails ... was only using it to send ... so that concept in and
of itself made it worth the read!

~~~
toddh
It's a pain to setup, as most of these things are, but it works well after
that. You can trigger lambda functions to read email stored in S3, which is
quite convenient. And with a library parsing the email is easy as pie.

------
elevensies
I'm curious regarding the motivation behind this. You are receiving and
storing emails, looking at them, and optionally forwarding them for
processing. Why not use an email client, and only forward your chosen stories
into a linear ingest and publish lambda job? I ask because I'm considering
building something similar and I'm thinking about the rationale.

~~~
mojoe
If I was the only person in the organization, that might work fine. However,
there are two main drivers of this methodology:

1\. I need to allow other people to simultaneously evaluate stories, and a
database works well to coordinate multiple people accessing the story queue.

2\. Having metadata and stories in dynamo/S3 allows for much more flexible
querying of ALL stories, to look for patterns, model the data, etc.

~~~
twic
Some friends of mine run a science fiction site. A reasonably serious one -
they've been nominated for a Hugo several times. They have WordPress and some
email accounts. They do grumble about the website, but it's nowhere near their
biggest problem. You are definitely overthinking this.

~~~
mojoe
I'd love to talk shop with your friends, if they're interested please direct
them to the site or have them email me at joe@compellingsciencefiction.com!

------
digi_owl
Serverless? Err, yeah, right...

~~~
Can_Not
As a community we should be correcting this proliferation of useless,
misleading, and blatantly wrong terminology.

------
balls187
Shameful plug: Our developer site is serverless:
[http://developer.avalara.com/](http://developer.avalara.com/)

Content is stored in our public github repo, and we use TravisCI to build
Jekyll pages on commit/merge, which are uploaded and served from S3.

Interactive functionality is provided via React in the front end, and API-
Gateway/ AWS Lambda in the backend.

The API Gateway/Lambda provide various proxies to our various sandbox API
environments, so Developers can quickly try our APIs directly on the site.

