
How Lanyrd moved from AWS to SoftLayer and MySQL to PostgreSQL with no downtime - simonw
http://lanyrd.com/blog/2012/lanyrds-big-move/
======
simonw
Not exactly no downtime... but all site content stayed available throughout
the move thanks to our handy read-only mode. We added this early in the life
of the site and it's proved extremely useful for a couple of major moves. It's
a pretty simple implementation (based on our site-wide feature flags) - once
in read-only mode all login cookies are ignored (so everyone gets the signed-
out experience) and the sign-in button is disabled. We also return an error
for any POST requests just in case someone has already loaded a page with a
form before we turned read-only mode on.

The moment we've gone in to read-only mode we can create a brand new instance
of the site from a copy of the (now frozen) database, make any necessary
changes to that, then switch traffic over to the new instance once we've
tested that everything is working properly. If the new version of the site has
a problem we can turn read-only mode off and continue to run on our original
database.

If you're running a more content-oriented site it's well worth taking the time
to set this kind of thing up - it's not too hard to do, and it gives you an
enormous amount of flexibility for maintenance further down the line.

~~~
jballanc
This is _excellent_ advice! I can't tell you the number of times I've worked
on projects where I wished just such a "read-only" mode existed. As an added
benefit if you have a read-only version of your site, I know at least Akamai
(and possibly their competitors) has a service where even if your origin
server disappears, they will continue to serve the latest version of your
read-only site until the origin servers reappear.

~~~
simonw
We have varnish at the front of our stack with a one-minute cache timeout for
users without cookies - but thinking about it, there's no reason we couldn't
bump that timeout up to something much higher for the duration of read-only
mode (though since the servers aren't having to deal with writes they don't
really need an extra performance boost).

~~~
chrismsnz
Check out Varnish's configurable grace period.

It will basically instruct it to continue to serve content that it considers
"stale" (up to a point) until it is able to update it.

------
whalesalad
Did you guys consider Heroku PostgreSQL? It's inside of AWS. If so, can you
shed some light on why you opted against it? I don't want to sound like a
fanboy of Heroku (I am not) but I am just interested in whether or not you
guys considered it.

~~~
simonw
We didn't - it's a fantastic product (I'm really impressed with how easy
they've made it to set up followers and forks) but we were ready to move to
dedicated hardware. It's also a pretty expensive option.

~~~
whalesalad
Thanks for the response.

I used to agree with you on the price until I realized that with Heroku you're
essentially paying for a dedicated server with that much RAM for each plan.
Reason being, every single database is "hot" and the entire thing lives in
memory to avoid I/O.

On second glance ... you're right. It is kind of pricey.

Speaking of moving to dedicated.. I'd love to hear some of your general
feedback on how AWS has been at some point. Essentially where you'd use it
again or where you'd go directly to dedicated (or a diet option like a VPS) if
you were starting on a new project.

------
professorplumb
If you follow the link to the more detailed blog post, Andrew shows how he was
able to speed up the data transformation script by a factor of 60 by using
ALTER TABLE statements to cast column types after the fact rather than
rewriting every single INSERT. A little more dangerous perhaps, but still an
impressive improvement.

------
Alex3917
While this is technically impressive, if you had gone down in the middle of
the night for a few hours would it really have mattered? Apple.com takes down
their entire site for several hours at a time every few months, and they don't
seem to be hurting too much for it.

~~~
simonw
For us, it does matter. We have conferences running all over the world which
use us for their schedule - for them, two hours of downtime in the middle of
their event is a big problem. Since we're a global site we have to think about
events across every timezone so there isn't really an obvious time to do this
kind of thing.

We might be able to schedule downtime for a period when no active events are
going on (tricky considering the number of conferences happening around the
world) but it's much easier for us to be able to make these kinds of changes
without worrying about events that are using us to serve critical information.
As it is, we still make sure to communicate planned read-only mode periods in
advance so conference organisers have a chance to plan around them.

------
nigma
Have you considered using PostgreSQL foreign data wrapper (mysql_fdw) to pull
data directly out of MySQL instead of the dump/load process?

It would be great to know more about your new setup, i.e. do you use streaming
replication and some resource manager like Pacemaker?

~~~
simonw
No, we didn't look at mysql_fdw. We're using standard PostgreSQL 9 streaming
replication.

------
acangiano
I've been using Softlayer for years now and have only good things to say about
them and their support team.

------
mwsherman
Would like to know more about the hardware specs, and what Lanyrd is paying
over at SoftLayer.

~~~
neverm0re
The hardware Softlayer is offering looks ancient, Xeons from 2006, only a
paltry 5GB/mo bandwidth tier as initial offering -- this page looks stuck in
time, like what you could get maybe three~four years ago in dedicated hosting.

Even Cari.net, which I think prices a bit higher than others is offering more
for the money. I've had several dozen machines with them since 2006 without
issue, top notch support. I also use the really cheap and no-frills folks like
Ubservers for when I want a bunch of disposable cheap dedicated machines.
Their support is absolutely shit, but by god are they cheap and the bandwidth
real. I've been through probably dozen and some change of these guys and
that's really all that matters is cheap solid bandwidth and a vague sense of
support.

~~~
boundlessdreamz
Who are the other reliable providers offer value for money in US ?

~~~
staunch
My startup does: <https://uptano.com>

~~~
X-Istence
What virtualisation tech do you guys use? Can I run FreeBSD on top of it?

~~~
staunch
OpenVZ (because it's closer to bare metal performance), which limits it to
Linux distributions currently. We've had requests for FreeBSD though, and
we're adding Xen support soon which should allow that.

~~~
X-Istence
FreeBSD on top of Xen has limitations (such as only one virtual CPU (AFAIK))
and in my testing is really unstable. Would you consider using something like
KVM instead which has less issues?

~~~
staunch
It's definitely possible. We're not wed to any particular
container/virtualization system. Networking tends to be a bit of a tricky
issue, but I'll definitely check out FreeBSD on KVM. Thanks for the heads up.

------
knodi
What is the size of Lanyrd data? I'm curious to see what your think is
considered "big data".

~~~
rpicard
He refers to "our tens of gigabytes of data" in the post on his site. [1]

[1] <http://www.aeracode.org/2012/11/13/one-change-not-enough/>

------
bfirsh
We're looking for hosting which has both dedicated and virtual servers on the
same network. SoftLayer and Rackspace seem to be the two big players who do
this. Does anybody have any experience with them both?

~~~
misterbwong
Rackspace user here. Be prepared to pay $$$$-their hybrid environment is not
cheap, especially for servers with above normal specs. There are also
(industry standard) restrictions on the network side of things with their
VM's, which we weren't prepared for when we started. Lastly, if you're ever
the target of DDoS attacks...good luck.

~~~
druiid
Previous Rackspace user here (we got fed up with Rackspace and finally got our
own equipment for a song compared to our monthly hosting costs with them)...
Yeah, good luck if you get a very common DDoS attack while on their network.
Apparently their data-centers in general are large DDoS targets. We moved and
have not had an attack since (and it's a big deal since we do e-commerce).

That said, out of any company I've ever been with their support was really
good. Just understand you're paying like crazy for it.

------
genwin
> There's nothing wrong with AWS - indeed, we still run a staging environment
> there - but our database benefits greatly from the low latencies of physical
> disks...

Doesn't AWS have physical disks for RDS? What am I missing?

~~~
friism
AWS RDS uses EBS for storage[1]. Depending on RDS instance type, I think they
use multiple raid'ed EBS volumes. EBS relies on physical disks, of course, but
they're accessed as SAN devices over the network and that means higher latency
than accessing disks that are hardware-attached to the local instance.

EBS has historically also had somewhat variable performance (on top of the
extra network latency), but the new provisioned IOPS feature should help with
that[2].

[1]: <http://aws.amazon.com/message/65648/>

[2]: [http://aws.amazon.com/about-aws/whats-
new/2012/07/31/announc...](http://aws.amazon.com/about-aws/whats-
new/2012/07/31/announcing-provisioned-iops-for-amazon-ebs/)

------
gsibble
I have used SoftLayer for years and have nothing but great things to say about
them. Nothing but speed, stability, and excellent service. Thank you SL!

