
Scaling Down Gracefully with AWS Lambda and HAProxy - realityking
https://www.contentful.com/blog/2016/04/27/aws-lambda-haproxy-scaling-down/
======
angryasian
What are the conditions that using HAProxy is preferable over using ELB or in
conjunction with ?

~~~
toomuchtodo
When you want more control over your load balancing, or you have long lived
connections that exceed the connection timeout you can configure with an ELB
(RabbitMQ connections, for example; ask if you want deets, we've tried using
ELBs multiple times with Rabbit, and always come back to HAproxy).

~~~
whalesalad
We had issues with internal ELB's for RabbitMQ (the 1 hour idle connection
timeout) and the solution ended up being a heartbeat in the clients. Most
client libraries have a parameter that will perform a heartbeat at some
interval to keep the connection open.

I can't imagine deploying our own HAProxy infrastructure. Sure, it's
relatively simple, but the set-it-and-forget-it comfort provided by ELB (not
to mention free SSL) is hard to beat.

~~~
toomuchtodo
We've seen some really weird ELB behavior in AWS (latency spiking excessively
every few hours for seconds at a time, underlying ELB instances having scale
down events occur at unexpected times, and so forth), so we use it sparingly
for internal microservices.

Running your own HAProxy infrastructure isn't terribly time consuming, but I
do devops/infrastructure all day long; I'm biased. If you can and do avoid it,
I can understand why.

------
totally
So HA only updates when chef runs (every 15 mins or so)? And when nodes are
deleted, I guess something else has to clean them from chef? You'll also have
to add these hosts to monitoring config.

I implemented something similar with puppet/mcollective/SQS years back. Poor
man's service discovery. Here are some other options that, if you can handle
the complexity of running Zookeper (or similar), are probably a bit cleaner:

[http://jasonwilder.com/blog/2014/02/04/service-discovery-
in-...](http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-
cloud/)

------
andonaeus
Interesting article. While my company's cloud infrastructure is in its final
nascent stages, I will definitely share this article and the ideas therein for
basically reloading the HAProxy context w.r.t the proxied backends.

------
atom_enger
How do you ensure that no requests are lost when you reload HAProxy?

~~~
jvehent
Soft reload takes care of that for you, sockets aren't dropped.

If you restart, however, everything's dropped.

~~~
atom_enger
Is that new in 1.6? As of 1.5.11 there was a lot of hacking required to get a
true zero downtime reload: [http://engineeringblog.yelp.com/2015/04/true-zero-
downtime-h...](http://engineeringblog.yelp.com/2015/04/true-zero-downtime-
haproxy-reloads.html)

~~~
luhn
No, he's referring to the soft reload added in 1.5, which the article you
linked addresses under "Just Reload HAProxy." As the article points out, it's
not quite zero-downtime, but pretty close.

~~~
jontro
Why not drop servers in a way described here?
[http://serverfault.com/questions/249316/how-can-i-remove-
bal...](http://serverfault.com/questions/249316/how-can-i-remove-balanced-
node-from-haproxy-via-command-line)

~~~
jolynch
You can down servers or backends over the stats socket, but you can't add them
(adding servers requires a reload).

The DNS resolution work that is going into 1.6 releases may lead to a workable
approach where you over allocate servers in a backend and then update them to
have the right host/port. Last I checked, however, you still couldn't set the
port dynamically.

