
Basic Concepts of High Availability Linux - frenkel
http://frankgroeneveld.nl/2014/03/23/basic-concepts-of-high-availability-linux/
======
keypusher
This article is pretty thin. We are currently building out a clustered
application on corosync/pacemaker with postgres synchronous replication and
tomcat, and I have mixed feelings about Linux HA so far. It isn't too bad to
get something basic set up (cluster with virtual IP for instance), but when
things don't work it can be difficult to figure out why. If you are looking
for a distributed filesytem GFS2 in this stack isn't bad. However, it seems
like there are a lot of differences between package versions, and the
interactions between versions of heartbeat, corosync, pacemaker, crm, pcs,
your resource definition ocf files, the stonith resources, cluster-glue, along
with the linux packages makes problems hard to track down and much of the web
info you do find out of date. I've often had to resort to irc or the mailing
list to try and figure things out, and even then sometimes it seems like
nobody knows. The whole thing feels a little bit shaky at first, but it is
possible to build a solid cluster on top of it with enough effort.

~~~
jvoorhis
If you truly care about reliability, finding a reliable and fully tested
distribution of these packages is the way to go. RedHat, SUSE and LinBit offer
certified, fully tested binary distributions and support, and companies like
Percona also provide support for specific HA stacks, and their expertise can
save you a lot of guesswork.

The available documentation for heartbeat, corosync and pacemaker can be
disappointing. I found the best way to learn is by doing–if you can afford it,
cobble together a lab environment and see how many different ways you can
break a system. With pacemaker, I find this depends highly on the resources
you are managing!

------
ww520
Here's my experience in building HA in Linux. There are two key pieces:
storage replication and failure detection. Replication is so that there's a
standby system with the same persistent state ready to go, and failure
detection, well, the whole point of HA is to ensure ongoing operation to
continue in case of failure.

For storage replication, Linux has the excellent DRBD
([http://www.drbd.org/](http://www.drbd.org/)) software to replicate disk at
the block device level. This is great because any kind of disk based systems
can be supported, such as database server, mail server, file server, DNS
server, etc.

For failure detection, Linux has the Linux HA Heartbeat ( [http://www.linux-
ha.org/wiki/Heartbeat](http://www.linux-ha.org/wiki/Heartbeat)). This would
detect failure at machine level and ensure proper failover.

Within a machine, there are other tools to monitor process level failure and
propagate the failure to Linux HA Heartbeat.

BTW, STONITH is a super simple way to avoid the partition problem.

~~~
jvoorhis
My a-ha moment for Linux HA + DRBD came when I realized you could use them to
retrofit redundancy and failover onto just about any system. (Of course, this
won't protect against your replicated filesystem becoming corrupted, and
current versions of DRBD don't allow you to scale out.)

------
reader_1000
A lot of tools are mentioned in both article and in this thread. So what is
the simplest and best way to achieve a failover virtual ip assigned to cluster
members? I don't want the tool to start services, it would be enough if it
won't send the traffic to failed note by determining with simple logic like if
port 80 is not listening? Having a lot of alternatives is good but confusing
and having powerful tools is also good but when only simple things needs to be
achieved, it requires a lot of time to configure it and it is harder while
troubleshooting. I prefer "keep it simple stupid".

~~~
krig
If all you want is virtual IP failover, the simplest tool I know of is ucarp
as mentioned by pandemicsyn below. That's basically all it does.

Note that ucarp and most (all?) virtual IP tools rely on being able to send
fake ARP to update the IP, which puts restrictions on where the servers can
live on the network, switch configuration etc.

edit: If what you're using it for is HTTP traffic, HAproxy seems to be the
most mature tool out there for HTTP load balancing and failover.

~~~
y0ghur7_xxx
> _If what you 're using it for is HTTP traffic, HAproxy seems to be the most
> mature tool out there for HTTP load balancing and failover._

HAproxy is a reverse proxy, it can do load balancing, but it's not useful if
you need a HA cluster, as HAproxy becomes the single point of failure in your
architecture. You still need something like wackamole, vippy or ucarp to
activate the switch from a failed machine to a standby box.

~~~
phil21
I personally hate this model. Layer2 networking architectures like this need
to finally die off, as they are extremely complex and difficult to
troubleshoot.

For high-availability HAProxy I've found ECMP to be by far the most simple way
to achieve very reliable redundancy, with a side benefit of more or less
infinite horizontal scaling (depending on what routers you're talking to).
I've served hundreds of gigabits with this model, and it works well. You can
even scale it out on a global level by utilizing anycasting.

It works pretty simple. Run bgpd on your HAProxy boxes talking to your router.
Each HAProxy box advertises your VIP, and your router will load balance this
via ECMP. Should a HAProxy machine die, the BGP announcement gets withdrawn
and traffic flows to the remaining proxy servers still advertising the VIP.

The only thing left to do is get some sort of service monitoring going that
can automatically down bgpd should haproxy die/otherwise mess up on an
individual machine. Add a couple checks in to ensure there is always a "path
of last resort" should you have a bug in your app or monitoring code, and this
proves to be very resilient, scalable, and is something nearly anyone can
troubleshoot in a very short amount of time. It also works well in a cloud
type on-demand/devops model - as it's extremely easy to simply spin up
additional haproxy machines and have them automatically announce their
configuration via bgpd.

------
y0ghur7_xxx
No mention of Wackamole¹ in the article, and I feel really compelled to
mention it here, as it is really simple to set up a HA cluster using it. I
followed this howto² a few months ago. It was really easy to configure and it
runs stable since.

¹[http://www.backhand.org/wackamole/](http://www.backhand.org/wackamole/)

²[http://www.howtoforge.com/setting-up-a-high-availability-
loa...](http://www.howtoforge.com/setting-up-a-high-availability-load-
balancer-with-haproxy-wackamole-spread-on-debian-etch)

~~~
y0ghur7_xxx
And there is also vippy, that is written in node.js and looks really good, but
I didn't have the pleasure to try yet:

[https://github.com/postwait/vippy](https://github.com/postwait/vippy)

------
iSloth
Linux High Availability is certainly still used a lot within application
infrastructures, especially in some of the smaller ones. However what I find
more interesting are the architectures that are performing all of the
availability functions in the software layer such as in the application or
database code, as typically these are simpler and offer far better
scalability.

------
pandemicsyn
Its not mentioned but ucarp is handy for those times when you want to float a
vip between two boxes for a bit of redundancy but don't need something super
intelligent (like where a bit of flapping is ok).

[http://www.ucarp.org/](http://www.ucarp.org/)

~~~
jvoorhis
OpenVPN Access Server ships with ucarp integration out of the box, and for the
most part it just works. Keepalived is another common daemon for managing
floating vips. If I understand correctly, ucarp was originally designed to
support redundant routers and firewalls, but in my experience, they are great
for moving vips between stateless services, like load balancers and reverse
proxies.

------
jpettersson
Great overview of the basic concepts, thanks!

------
snorkel
Server 500 error. Now that's irony.

~~~
lgierth
Highly available 500 error, though! (It seems to be back up.)

------
hepek
Install 5 different things, or just run Erlang on all nodes.

