
Serverless Aurora - abd12
https://serverless.com/blog/serverless-aurora-future-of-data/
======
jondubois
Serverless is a bad idea, it was designed to serve big infrastructure
providers as a way to increase their lock-in factor but the benefits to
software developers is negative.

Serverless is simple but opaque; it's great for making demo apps but not ideal
for real production systems. It narrows down integration possibilities,
complicates large-scale development and deployment, and often, it makes it
more difficult to keep track of backend errors... In addition to this, it
makes you fully dependent on a single infrastructure provider; it takes away
all your leverage when negotiating hosting costs.

The control panels of some of these services have become so complex and
elaborate that you'll be wishing that you could just SSH into the server
directly.

~~~
leetbulb
I built a fairly large system using the Serverless "framework" on AWS. It's
been running for over 8 months now with zero maintenance and zero service
disruptions, with an ingress of ~50M events per day and providing a sleek
ReactJS/SemanticUI frontend that the users seem to really enjoy. Being a side
project, it's been relatively stress-free.

While the application is still very profitable, the cost is ~10x its
implementation on traditional servers. I'm not going to argue the pro's and
con's, just providing some numbers. At this point, we're working to optimize
the original stack in order to reduce costs.

I will say though...honestly, working with Serverless and the AWS stack is a
very pleasant experience.

~~~
social_quotient
Details on this would be great. We do a lot of serverless for content
publishing and 3rd party api integration work... I’d love to hear more of how
others use it at scale.

I can confirm a couple things. It seems like we spend a bit more time getting
things to work as expected.

Troubleshooting can get expensive given the system disappears.

Telemetry on the lambdas needs to improve. Finding what’s eating time gets
tough when you are looking to optimize. (A lot baked in to this statement but
getting 800ms of compute to 400ms is sometimes important whilst staying on
lambda)

~~~
leetbulb
Unfortunately I cannot provice much detail about the nature of the
application, however I can say that we collect and process various types of
events and perform automated actions based on them. The ingress begins at API
Gateway, runs through a pipeline of Lambda functions, ends up in a Kinesis
firehose, and ultimately in S3 for further processing. The primary cost (~60%)
comes from API Gateway acting as the event ingress point, which was a design
mistake...though at the time it did make things easier.

We're currently working on migrating the event collectors to a pure CloudFront
based solution (inspired by SnowPlow) where the events will be submitted to
CloudFront via signed GET requests and the CF logs will be streamed to the
same Lambda-based processing pipeline. Doing this will eliminate almost all of
the overhead of API Gateway (still required for certain events that cannot be
collected via HTTP GET).

------
boyter
This was also the biggest news of re:Invent for me. Total game changer. No
need to provision large R instance types to support the batch processing that
happens a 3AM and sits idle most of the time. All environmental tiers can have
100% the same configuration.

I know of some projects burning millions a month in database costs because
they like to replicate environments for testing branches. Switching to
serverless Aurora would reduce that cost to probably low thousands.

Of course it all depends on how quickly it can scale up to meet the load and I
reserve full judgement till I can try it out, but am very interested in this.

------
oh_hello
The idea is exciting, but the pricing looks too high for side projects. The
docs state "Aurora Serverless can scale from a minimum of 1 ACU to a maximum
of 256 ACUs." Unless I'm reading this incorrectly, you are paying for 1 ACU 24
hours a day.

$0.06 * 24 * 365 = $525.60 per year.

This is more expensive than a low-end RDS instance. It's a shame, I
immediately thought this would be the perfect solution for side projects and
prototypes, but the pricing killed it.

~~~
jajern
You pay per second of usage and the ACU starts and shuts down automatically.
If your DB is only actually accessed a occasionally then you are not paying
for 24hrs of usage. I think the auto-scaling is the draw though, not
necessarily the price. So you are only getting charged the the database is
being accessed. I think the examples on the pricing page make it a little more
clear.

~~~
jchanimal
FaunaDB Serverless Cloud has global ACID transactions with per-request
pricing. You can optimize your queries for cost by looking at the response
headers, or support your organization with an on-premise multitenant cluster.
Learn more about pricing and features here
[https://fauna.com/serverless](https://fauna.com/serverless)

------
paulgb
I wish Amazon or Google would just make a simple key/value store to give
minimal persistence for their serverless offerings. DynamoDB comes closest but
it still overcomplicates things.

~~~
eropple
I'm not super familiar with DynamoDB except as a strict key/value store. You
set a partition key, you ignore your range key, and you go--what's
particularly overcomplicated about that?

~~~
noobiemcfoob
Setting the number of write nodes and read nodes to match your workload. I was
scaling up a system and as more things came online, the traffic became more
bursty we were constantly bumping them up and wondering if we were allocating
too much (probably useless fretting regardless).

Ultimately, we moved to an S3 only solution since we have such a low
throughput environment and the simplicity is addictive.

~~~
eropple
Makes sense in the past, but with DynamoDB autoscaling that's largely gone by
the wayside.

~~~
noobiemcfoob
I hadn't heard about this feature update! Thanks for pointing me at it!

------
machbio
Am struggling to understand the concept of Serverless Aurora and AWS Lambda
integration - they highlight that AWS Lambda would scale on traffic and there
by Serverless Aurora kicks into support this scale - but there is a limitation
on AWS Lambda inside a VPC[1]. So in order to really utilize the Serverless
Aurora fully - the Proxy fleet would have special permission to access Aurora?

Also, has anyone solved the issue of AWS Lambda (in VPC) scaling beyond the
limited IP space ?

[1][http://docs.aws.amazon.com/lambda/latest/dg/vpc.html](http://docs.aws.amazon.com/lambda/latest/dg/vpc.html)

ENIs = Projected peak concurrent executions * (Memory in GB / 3GB)

~~~
bni
But would you really need to put Serverless Aurora in a VPC? Is it even
possible to do that? I expect it working like DynamoDB in this regard, the
tables/database just exists and access is determined by IAM role.

------
obiefernandez
This is super interesting to me on applications where I'm using Heroku's
$50/month Postgres plan. Almost inevitably I find that the only reason I've
upgraded past hobby level is for performance, not scaling. That it would auto-
scale under load is just the icing on the cake.

~~~
le-mark
What performance problems were you having specifically? What are your use
cases? Just curious to hear your experience.

~~~
obiefernandez
Alas, I don't have metrics. Just seemed noticeably faster on the more
expensive Postgres plan.

------
moduspwnens14
It looks really great. The only concern I have (relative to other serverless
features) is that it does have a top-end architectural limit (256 ACUs).

It's probably not worth fretting over, but it is worth mentioning when
comparing to DynamoDB. That means in building a serverless web app, you might
look at starting with Serverless Aurora, graduating to regular Aurora (as
usage increases due to pricing), and then perhaps moving off workloads better
suited to DynamoDB as you discover them.

------
sctb
Recent related discussion:
[https://news.ycombinator.com/item?id=15808271](https://news.ycombinator.com/item?id=15808271)

------
Footkerchief
Any ideas what replication model they're using under the hood to bring new
instances online so quickly?

------
andrewstuart
I wonder if this can be accessed from Lambda VPCs?

~~~
simonmales
It will be accessible via HTTP, just like all of Amazon's APIs. Doesn't matter
if you are in a VPC or not.

~~~
zwily
Aurora is accessible via TCP (mysql or Postgres), not HTTP.

