
Etcd Clustering in AWS - polvi
http://engineering.monsanto.com/2015/06/12/etcd-clustering/
======
justizin
"If this happened our cluster would become unavailable and may have trouble
re-clustering."

This was basically the repeated experience I had which caused me to abandon
etcd for the time being.

If it can barely ever heal, what the fuck good is it? And I found that it
could barely ever heal. A 3-node CoreOS cluster I ran _always_ crashed when it
attempted a coordinated update, and rarely could be repaired with the help of
#CoreOS over hours.

Because CoreOS pushes out updates with versions of etcd incompatible with
recent versions, the etcd cluster could never survive the upgrade.

Add this to the fact that the CEO of CoreOS told me in person that he expected
them to be the _only_ Operating System on the internet, and I'm generally not
along for the ride with CoreOS any longer.

Consul, Mesos, and Docker are looking good.

Anyone interested in this space should check out:

    
    
      https://github.com/CiscoCloud/microservices-infrastructure

~~~
jefe78
Have you tried using Mesos? We're doing a POC but ran into some issues that
we're going to wait out. Also, I've spoken to Mesos and they stated that they
had no intentions to make deployments easier/more stable, in favour of pushing
their commercial offering.

~~~
justizin
I pointed at an open-source project by Cisco that pretty much sidesteps them
entirely.

Obviously, CoreOS is going to start wanting your money pretty soon, as well.

Having worked for one of the earliest commercial linux distributors, I have
little faith in such an effort to get anywhere. Red Hat and Canonical can
barely make a dime.

Mesosphere isn't really an alternative to etcd, though. It relies on
Zookeeper, which isn't perfect, but is much more battle torn than either
Consul or etcd.

I have high hopes for something to replace Zookeeper, but I'm not deploying
something in infancy which inherently can't heal from outages.

~~~
vidarh
> Red Hat and Canonical can barely make a dime.

Redhat Fiscal 2015 revenue: $1.79 billion. Net income: $180 million.

Their fastest growing business areas are incidentally exactly in this space:
OpenShift, OpenStack and Ceph.

------
jefe78
... thanks Monsanto?

In all seriousness, this is really interesting. They solved some of the
problems associated with persisting a cluster and we're likely going to use
that. Feels weird thanking them for anything though.

Edit: Is anyone using CoreOS in a physical DC? We're using AWS with ~1.5k VMs
but have another 5-6k hosts in physical DCs. Trying to move us towards
containers but struggling.

~~~
mooreds
I just sent an email to some clients that I've been trying to get to blog
about technical issues (for recruiting and retention purposes)--if Monsanto
can do it, most anyone can.

~~~
justizin
They're certainly known for not always knowing the difference between 'can'
and 'should'. ;)

------
yeukhon
I think they fixed etcd cluster problem in 2.0 release (previously this is 0.5
branch).

For example, we use CF (old version), and we hit
[https://github.com/coreos/etcd/issues/863](https://github.com/coreos/etcd/issues/863).

------
KnownSubset
From my experience etcd is pretty rock solid, until you start using it across
availability zones. Then if you add in SSL into the mix, the reliability drops
even further if you are using the default configuration. At that point you
need to start tweaking the heartbeat and timeout parameters for a the cluster
to stay stable.

~~~
ideal0227
We have fixed the SSL issue in 2.1. We are also considering back-port it to
2.0 release if possible.

------
narsil
We solve the bootstrapping problem with an internal ELB instead.

Autoscaling Groups can be configured to have instances join multiple ELBs. We
have one be the regular ELB to access the instances with, and the other is an
internal ELB that only allows connections from instances in the cluster to
other instances in the cluster on the etcd port (controlled via security
groups).

When an instance comes up, it adds itself to the cluster via the internal
ELB's hostname. The hostname is set in Route 53.

The biggest issues we've been having with etcd continue to be simultaneous
reboots and/or joins to the cluster. It would also be great if the membership
timeout feature that used to exist in 0.4 made its way back in. Right now,
each member has to be explicitly removed rather than eventually timing out if
it hasn't joined back in.

Looking forward to hear any other approaches folks have taken.

------
codewithcheese
Running docker clusters on AWS seems a little foolish to me unless your trying
to save money. Instead of manage containers why not just manage instances?

~~~
vidarh
If you're trying to save money, you wouldn't be using AWS in the first place.

Here's one reason to run Docker on AWS: Putting everything in Docker
containers makes it far easier to migrate _off_ AWS if/when you want to.

~~~
brianwawok
Well unless you use sqs and sns and s3 and redshift... Then you are pretty
locked in anyway

------
gct
I can't get over how bad a name etcd is. Everytime I see it I think it's some
sort of daemon for /etc files.

~~~
phildougherty
I think what they were going for was:

/etc is where your config files go on your server etcd is where your config
goes for the distributed system

