

Linode is down again - raychancc
http://status.linode.com

======
rbranson
Remember guys & gals: this can happen to you. I am a fanatical Linode
customer, and they are, in general, an extremely rock solid service, but that
doesn't alleviate my responsibility to ensure my application is available
regardless of datacenter outage. Power outages happen to EVERY datacenter.
There are some very well documented and understood ways to alleviate this
problem.

~~~
KaeseEs
Care to enumerate a few of them?

~~~
al_james
More than one server across multiple datacenters is the only way. The use
either a load balancer (which brings its own problems) or some kind of status
aware DNS scheme to route traffic to live servers.

I wish linode would offer some kind of high availability load balancing like
amazon ec2 does, that really is a killer feature.

~~~
rbranson
I wouldn't be so sure that ELB's are actually failing whole IP addresses
across availability zones. Given the current ELB architecture is heavily
dependent on dynamic DNS resolution, I am pretty suspicious that they are
using DNS for failover. The separate set of DNS servers specifically for
elb.amazonaws.com and the 60 second TTLs on the CNAMEs they generate seem to
indicate this. Of course, there is no documentation to either confirm, or deny
this. Amazon simply states in vague terms that it's highly available, but
doesn't state how it's designed, or even possible failure outcomes.

~~~
al_james
Hmmm. Interesting point. However, if that were the case, why would ELBs be
restricted to single AWS regions?

Anyway, its a nice feature that it just works without me needing to set it up.

------
brlewis
Please change the title to indicate this is just the Fremont datacenter. The
current title needlessly scares those of us with linodes in other datacenters.

------
megamark16
And as luck would have it this happened right after one of my biggest clients
sent out a bunch of holiday coupon codes to all of their customers. Do. Not.
Want.

------
bobzimuta
Are there clever solutions for routing traffic to other linode datacenters
during one of these outages? I'm moving to AWS if I can't do something akin to
elastic ip across datacenters. Linode currently only allows you to swap ip
addresses within a dc.

~~~
nenolod
use a DNS roundrobin with low TTLs and use monitoring to remove IPs when they
go down. if you have 3 hosts in your production app's roundrobin, that makes
it much less likely that they'll hit the dead server and if the TTL is 300,
then it's only a few minute window anyway.

~~~
tropin
That's material for a great blog post. Please write an article wherever you
can and share!

~~~
paraschopra
Visual Website Optimizer setup is exactly like this. We documented it here
[http://visualwebsiteoptimizer.com/split-testing-
blog/maximum...](http://visualwebsiteoptimizer.com/split-testing-blog/maximum-
theoretical-downtime-for-a-website-30-minutes/)

------
iampims
Waiting for the mandatory "Goodbye Linode" post. </sarcasm>

------
santry
Can anyone recommend good documentation on load balancing a LAMP app across
multiple independent VPS providers?

We've had a high profile site affected by these Fremont outages so I'm
thinking the only real solution is to have VPSs at Linode, Slicehost and EC2
then load balance among them. Of course, then the question is, what do you do
when the load balancer goes down?

------
david_shaw
This is why Linode is better than many other services I've used. Every data
center has outages, but at least Linode makes a point to update the server
status promptly and (it seems) implement the fixes as quickly as possible.

rbranson is right: the responsibility for redundancy lies with you.

------
jrnkntl
Other discussion: <http://news.ycombinator.com/item?id=1934013>

------
lwhi
My London node was down briefly two days ago .. same happened about a month
ago - is this a reason to worry?

------
jread
From status page: Initial indications are pointing towards another power
outage. We are now working on bringing Linodes up as soon as possible. We will
continue to post updates as we have more information.

<http://status.linode.com>

------
schwanksta
My Linode slice is in Atlanta, not Freemont, and it went down once today and
once earlier this week.

~~~
Encosia
I don't think that was the entire Atlanta data center, for what it's worth.
Mine's in Atlanta too and I haven't had any recent downtime.

------
tlack
And to think how much time I spent last month recommending these guys,
especially the Facebook-friendly Fremont data center. Awkkkk-ward.

~~~
foljs
Why? Because there are other services with no downtime?

Even Amazon and Rackspace go down...

Even Google has had outages...

------
raychancc
This is really annoying... Down 2 times in 3 days!

~~~
mike-cardwell
Have they broken your SLA?

~~~
aberkowitz
Linode has a 99.99% uptime guarantee, which amounts to 43 minutes and 12
seconds offline in a 30 day month. While I don't have a Linode in Freemont,
their status page appears to indicate, at minimum, 1 hour of downtime. As
repayment for the inconvenience, users get the lost time added to their
balance.

~~~
WillyF
A 99.99% uptime guarantee would be 4.32 minutes per month. You calculated for
a 99.9% uptime guarantee.

~~~
aberkowitz
I'm going to take a long break from ever doing math again on HN.

