
Amazon RDS Proxy for AWS Lambda - Tehnix
https://aws.amazon.com/blogs/compute/using-amazon-rds-proxy-with-aws-lambda/
======
topicseed
Long overdue, and now waiting for the PostgreSQL's RDS Proxy as it's the main
database we use along with Lambdas... and gosh it's tedious!

------
nzoschke
I wonder how this compares to Aurora Serverless Data API. This is an HTTP API
that you POST a query to and get results data back.

The result format is a bit clunky but there's all the column information you'd
expect.

I found this golang database interface which ended up dropping in as expected
with a few tweaks.

[https://github.com/Clever/rds/tree/birthday](https://github.com/Clever/rds/tree/birthday)

My bill for Aurora Serverless Postgres last month was less than a dollar.

~~~
PetahNZ
We use Aurora Serverless and this is a sorely needed feature for us. We get
large spikes of traffic that also cause large increases in connections to our
DB, but Aurora Serverless cannot scale fast enough to cater for this.
Unfortunately it doesn't seem we can enable this proxy for serverless
clusters.

~~~
sudhirj
Aurora serverless already has a proxy by default - if it can’t scale fast
enough it’s likely because your queries are too long and interleaved to find a
logical switchover point.

This proxy is probably an offshoot of the serverless proxy.

~~~
PetahNZ
It has a hard limit on max_connections based on the "Aurora Capacity Unit",
e.g. for 16 ACU's max_connections it is 2,000. We see spikes of 10x to 100x
concurrent users in a matter of minutes (think hug of death style). And
although our queries run fast and we see limited increase of CPU on the
cluster we almost instantly hit the connection limit.

[https://www.jeremydaly.com/aurora-serverless-the-good-the-
ba...](https://www.jeremydaly.com/aurora-serverless-the-good-the-bad-and-the-
scalable/)

------
joecot
For anyone else who spent 10 minutes trying to find a link to pricing from
this page and failing, here it is:
[https://aws.amazon.com/rds/proxy/pricing/](https://aws.amazon.com/rds/proxy/pricing/)

Free til the end of 2019, 0.015 per vCPU (of the underlying RDS instance) per
hour the proxy is enabled. So it is not scaled down to 0 when not in use. For
comparison, a t3.micro (2 vCPU, 1gb ram) is 0.0104 per hour. If you're a big
project it might be worth it, but just keep in mind rolling your own proxy or
using the RDS serverless APIs are cheaper options.

------
revicon
I don't understand why this isn't launching with data api support? AWS
specifically built data apis into the Aurora Serverless product for this exact
usecase...

[https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-
api.html)

If I'm using Aurora Serverless right now and want to switch over to RDS Proxy
I'm going to have to rewrite my calls.

------
crgwbr
Does anyone know why RDS Postgres always seems to lag behind their MySQL
offering? I can imagine Postgres is that much more difficult to develop a
connection pooling service for, especially since PGBouncer and PGPool-II both
already exist and are widely used.

~~~
Aeolun
I think both for the reasons you named, and because more people use MySQL
maybe?

------
astuyvenberg
Finally, this has been a long time coming. Connecting to existing RDS
instances from lambda functions has been a pain for users looking to adopt
serverless tech into their existing applications.

------
tybit
This is really nice, we used lambda for event based ingestion into RDS and
MySQL connection limits were such a pain.

However I do wish they’d include how this will evolve alongside last years
announced data API.

------
dylanz
This is awesome. We were just looking at throwing up PgBouncer in front of RDS
in the coming year. Looking forward to PostgreSQL support!

~~~
ocdnix
I really hope it can replace PgBouncer. We've spent way too much time getting
it running in ECS, mostly due to us needing a separate PgBouncer service for
the primary and replica RDS instances. And then a third ECS service running
HAProxy in TCP mode to load balance between the PgBouncer services, to provide
a read-only endpoint that balances between the primary and replica. Add issues
like: having to patch in HAProxy TCP keep-alive parameter support (it only
allows enabling/disabling it, leaving params up to the OS, which you can't
control on ECS Fargate, _and_ the AWS NLB having a 350-second TCP flow idle
timeout - good luck with long-running queries); AWS NLBs having horrible
timing issues when handling target group changes... it was not a great
experience.

~~~
Znafon
Why did you not use ECS with EC2 instead of patching HAProxy? With the Amazon
AMI it is pretty straightforward.

~~~
ocdnix
If I were to do it again, I'd consider either that or building it directly on
EC2, preparing the AMI with Packer or something, and handling scaling with an
autoscaling group.

------
vpontis
This is really cool! I've had to do the same thing with PGBouncer for Postgres
+ serverless and it's a pain.

Would love to see this come out for Postgres and come to Google Cloud :)

------
beering
TL;DR: AWS now has a managed connection pooling service for MySQL on RDS only
(public preview right now). The article talks about using it in conjunction
with Lambda.

~~~
jakemal
It took me over half the article before I could parse "Amazon RDS Proxy for
AWS Lambda" as a coherent sentence rather than a series of semi-related words
strung together.

------
dmahanta
hahaha JDBC as a service. We really needed this, what has the world come too.

~~~
JonMR
You may already know all this, but I thought I’d add to what you said.

JDBC doesn’t provide connection pooling out of the box. There are connection
pooling libraries that layer into JDBC. Most frameworks and application
servers do bundle a connection pool out of the box.

Lambda is just a different paradigm than a single monolithic app with a
connection pool. I assume that’s what you’re getting at. :)

There are other use cases though. Tools like pgBouncer are fairly useful in a
large enterprise where a single database has multiple clients. They provide a
single point to coordinate failovers and manage resource limits. I generally
prefer a 1:1 relationship between an application and a database, but sometimes
you have to play the cards you’re dealt.

------
knodi
If you're doing this you're using lambda's wrong.

~~~
revicon
What's the right way?

~~~
PetahNZ
Not sure the op's case, but typically it's better for queue processing.

------
foo4u
The same functionality could have been accomplished by just allowing us to
specify the max number of concurrent instances of a function that can be
active at a time. With proper request queuing the problem would be solved. As
usual, AWS solves a problem they failed to design for by charging for an
overly complex solution. Flame me if you want but there are K8s serverless
frameworks that address this out of the box.

~~~
pavelrub
Specifying the max number of concurrent lambda execution has been possible for
years - it's called reserved concurrency.

This does nothing to solve the issue however. There are many use cases where
you can't delay handling requests.

