
Serverless dev practices questions - rsh210m
Hi guys, 
So after spending some time exploring serverless practices (Lambda in particular), a few questions popped into my mind which i’d love to get some more experienced views. The development guidelines seem to be quite straightforward for some simple stuff to do and the given sample code demonstrate it nicely. Yet - what happens when the overall application isn’t as simple and requires more than 15 lines of code? Building the design and logic of the app shouldn’t be that hard, yet making sure that it actually works is what worries me more… 
What are the given practices to test, debug, troubleshoot and iterate with the code i write in Lambda? The best i came up with is traces and log collection (most common), wrappers for the code in Lambda to collect more metrics and SAM local which should allow you to run stuff locally (but then, how to simulate the real context and the flow of the app)... are these really enough to test code in Lambda? Are there any other options around? Are there any other outstanding issues in developing for for Lambda?
======
mseebach
AWS Lambda is a very small interface. I'm only getting started, but I'm
planning to keep all my application code in a separate project with no Lambda
dependencies, test _that_ as I would any application, and then have a thin
wrapper that just wraps each endpoint in the Lambda interface.

This would also maintain as loose as possible coupling to the Lambda
ecosystem, making porting easy -- I'm in Java-land, so that probably means
Dropwizard. I'm not there yet, but I'm planning to maintain a parallel
Dropwizard wrapper, to make local dev easy.

~~~
parzivalm
Have you thought about the added execution time from JVM startup? A project I
had dropped in my lap awhile ago was figuring out a way to speed up a java
lambda function. Basically we decided the only way was to try and make the
lambda container always hot. A cloudwatch event would trigger the lambda
function every 5 minutes to make sure it was always hot.

~~~
mseebach
For now, my project isn't very sensitive to taking a couple-of-seconds warm-up
hit. But I'm aware of the issue, and it seems that keeping your classpath lean
works wonders. I'll be investigating further, but my tentative plan is to
section the endpoints by their dependency-size. The majority of the enduser-
latency-sensitive endpoints will need little more than a JDBC connector.

------
davemontorio
I recommend to use the Serverless framework
[https://serverless.com/](https://serverless.com/). It's pretty straight
forward and very useful to configure imho. In this way the entire project
(configuration + code) can be versioned (svn, git...) easily.

------
kostarelo
We use AWS Lambda exclusively on the product I am currently working on (3
peoples team).

\- We use the Serverless framework

\- Monorepo with multiple (micro) services

\- Each service exposes Lambda for particular functions (e.g. Articles service
exposes GetArticle, CreateArticle, etc..)

\- No business logic is being shared between services, only libraries

\- Three environments, Dev, Stage, Production (and every developer can also
deploy its own)

\- Tests are invoking the Lambda handlers directly and are mocking responses
from other Lambda calls. We use Localstack to stub AWS services, such as S3
and DynamoDB.

Still, a lot to learn of course, but so far everything is working great for
us.

~~~
davemontorio
What do you use to store articles? DynamoDB? If so, are you ok with querying
and searching data? What about security? How do you implement it?

~~~
kostarelo
DynamoDB for storing only. After an article is stored in DynamoDB and stream
will index it into an ES Service. Then we are quering and searching only ES.
DynamoDB can't handle searching very good.

Security wise, everything is private except one Lambda that exposes a GraphQL
server and is being forwarded to the public net through API Gateway. We will
implement subnets and security groups very soon.

~~~
davemontorio
With ES do you mean ElasticSearch from AWS? This is interesting, can you tell
me something more please?

------
hkchad
We are using microrepos, we implement all logic in libraries then import those
libs into another project that wraps it w/ the lambda logic. The lambda logic
tends to be small and all the 'work' is done in libraries. Thought here being
if we need to migrate to another serverless solution (or server solution) it's
quick/easy to wrap the tested libraries in whatever the next logic should be.

Then we use Apigee a127 Node project to invoke the lambda's. This is where
authentication is done (we use Auth0 JWT/Bearer tokens). The Apigee a127
project invokes the lambda and handles routing. We looked at serverless but
didn't want to use API Gateway since we already had apigee. We plan to look at
SAM next but so far haven't need it. We have a few grunt scripts that we can
use for running the lambda locally so far not a big deal.

For logging we use an AWS ElasticSearch cluster and Kibana.

So far we haven't run into any big issues with lambda, we see better response
times when there is a constant load on the system, I guess that helps w/
keeping the lambda's warm. Even cold start isn't a big issue b/c the $$$ saved
is well worth it.

Automate EVERYTHING. You will quickly get a bunch of lambda's and you need a
way to keep them all updated, manual SUCKS.

------
nevon
I would recommend splitting out the lambda-entrypoints from the rest of your
app, so that you can test it separately.

Personally, I've found it tough to test full integrations where I'm depending
on cron jobs, work queues and notifications. I felt like I was writing an
awful lot of code just to be able to fake all these integration points, which
meant that in reality there was a lot of behavior that wasn't really being
tested.

------
reinhardholl
I use [http://serverless.com](http://serverless.com) to manage and deploy my
Lambda functions, then I use [https://github.com/dherault/serverless-
offline](https://github.com/dherault/serverless-offline) to run a local copy
of most things needed. Not perfect yet, but really helped me getting moving
fast.

------
agentgt
> Are there any other options around? Are there any other outstanding issues
> in developing for for Lambda?

I'm not sure how AWS Lambda handles the charging of network latency (or
failure) case of downstream external service dependencies but that is one of
our biggest problems. ie pull/scrape architecture. The push portion (ie
events) of our architecture is an obvious fit but that is already taken care
of quite nicely.

We interact with hundreds of web services (REST APIS, SOAP, etc) that are
actually external from us.

Its unclear if or how Amazon charges you for a slow external service (I would
imagine its particularly bad if the code is blocking IO) or if that is even
possible. I do know spawning a whole bunch of servers to connect to a single
REST API endpoint is damn slow.

I have feeling Lambda for our case is probably not a good one? Obviously for
data processing I can see it but that is not a challenge we have.

In an ideal world our downstream (or I guess it could be called upstream...)
dependencies would contact us but... that sadly isn't the case particularly
for adhoc stuff.

------
tmclaugh
This stuff is hard and we're all still figuring it out. Just recently we
realized we built a monolith because we weren't paying enough attention to
whether all the features should be a part of a single system. We're going to
revisit that.

Don't look at your function as an app, it's an function and the best functions
do a single thing. The definition of a single thing depends on the scale of
view you're taking. But that may help you to realize your functions shouldn't
be very complex and possibly small enough to test more easily.

However, that results in increased system complexity as you start adding SNS
topics, SQS queues, DynamoDB tables, and all sorts of event triggers.
Something I've adopted is using draw.io (others use Lucid Chart) to draw out
the system logic. Maybe this helps?

[https://www.draw.io/](https://www.draw.io/)
[https://www.lucidchart.com/](https://www.lucidchart.com/)

Maybe this makes the confusion worse?

------
Jdam
For lambda, I highly recommend integrating AWS X-Ray.

------
RileyJames
I've been using Claudia.js [1] with Node / Lambda, and it's a lot easier than
going through the web ui. My colleague built an Alexa skill linking service
using Claudia, Node & Lambda. You can check out the code as an example of
building something that's got a few functions & endpoints
[https://github.com/XeroAPI/xdhax-alexa-
sample/tree/master/al...](https://github.com/XeroAPI/xdhax-alexa-
sample/tree/master/alexa-xero-linking-service)

[1] [https://claudiajs.com/](https://claudiajs.com/)

------
giancarlostoro
We use Azure Functions, mainly cause that's what our project is already on.
Seems to work just fine. Done all in C# .NET no huge complaints yet.

[https://azure.microsoft.com/en-
us/pricing/details/functions/](https://azure.microsoft.com/en-
us/pricing/details/functions/)

------
hullsean
i like the serverless framework also. x-ray integration is also key although
it hadn't come out when i wrote this...

30 questions to ask a serverless fanboy
[http://www.iheavy.com/2017/03/13/30-questions-to-ask-a-
serve...](http://www.iheavy.com/2017/03/13/30-questions-to-ask-a-serverless-
fanboy/).

------
fiokoden
I have a very large application implemented as one single lambda python
function.

Works great.

~~~
nvivo
Please define "very large". When I think large app, I think at least a few
hundred endpoints split across tens of thousands of files. How can you have a
very large app with a single endpoint? Could you elaborate?

~~~
fiokoden
Ok we have very different definitions of very large. I mean s complete full
featured SAAS workflow application.

I don't actually use the AWS HTTP API gateway. I control the front end
entirely so my UI just talks direct to lambda. It's very simple - instead of
futzing around implementing REST API mapping layers I just connect all my
front end functions to a single lambda function and specify in the params the
name of the python function I want to run. Super easy.

Even if you wanted to use API gateway you could still map all the end points
to one single application and dispatch based on the route/URL.

I can make a solid recommendation to anyone use lambda .... seriously consider
using Cognito for your user management.

There's no reason why you couldn't have tens of thousands of functions in the
same function, in fact probably a good idea.

~~~
mseebach
> seriously consider using Cognito for your user management.

Can you please expand on that recommendation? I tried and found it profoundly
opaque, and eventually gave up in favour of auth0. I specifically wanted to
use the hosted login/sign-up forms.

~~~
danesparza
I found this tutorial had some very good information on incorporating Cognito
into an app: [https://aws.amazon.com/blogs/aws/build-your-first-
serverless...](https://aws.amazon.com/blogs/aws/build-your-first-serverless-
application)

It lays out the architecture fairly well and how Cognito ties into it. Still
very fiddly, and doesn't give any indication on how to use their hosted forms
(which are a bit of a mystery to me as well).

Since it's still on point with the OP's question, I'll ask: What has your
experience been with auth0? It's another service I have considered.

~~~
mseebach
I really liked it. Easy to follow instructions and an active community where
all questions I could come up with were already answered. Took me an afternoon
after getting almost nowhere with Cognito in three days.

The thing I spend the most time on was "user_metadata" and "app_metadata", two
JSON blobs on each user, RW and RO respectively. In order to read those at all
from your application, you have to define a custom "rule" (arbitrary
Javascript that wraps responses). The reason for this is something about
standards and name-spacing, but I couldn't really follow the argument, and
it's seems like this is something that very confusing to a lot of users.

------
zn44
checkout [http://theburningmonk.com/](http://theburningmonk.com/) for good
posts on some serverless topics you've mentioned

