

Linode down in Newark data center - hendler
http://status.linode.com/

======
dangrossman
"We have detected a large incoming DDoS attack to our Newark facility. We
already have our administrators working to mitigate the attack. We will have
service restored as soon as possible."

    
    
        [dan@palladium]# traceroute 66.175.213.170
         1  * * *
         2  ae12.dar02.sr01.wdc01.networklayer.com (208.43.118.150)  0.349 ms  0.351 ms                                                                                                                                     0.325 ms
         3  ae9.bbr02.eq01.wdc02.networklayer.com (173.192.18.188)  0.951 ms  0.966 ms  0.975 ms
         4  * * *
         5  * * *
         6  * * *
         ...

~~~
eli
Where is that quote from?

~~~
j15e
Same here, got this reply from support :

"Linode 1 hour ago

Hello there,

We have detected a large incoming DDoS attack to our Newark facility. We
already have our administrators working to mitigate the attack. We will have
service restored as soon as possible. Please refer to the following page for
status updates:

[http://status.linode.com](http://status.linode.com)

If you have any other questions please let me know.

Thank you.

Nick"

------
j15e
New response from support :

"Based on the information we had at the time we believed it to be a DDoS
attack. We are now aware that it was a power problem affecting our Newark data
center. The backup system did not work and we are still looking for additional
information from the data center staff.

We will be updating our status page with any further information.

Thank you."

------
extrapolate
Seems like it was caused by a power outage.

    
    
      1:09AM EDT (UTC -4): The Newark datacenter has suffered from a power outage. We are working on bringing Linodes back up as soon as possible.

------
mmahemoff
As others have mentioned, we've seen reports of network and power outage, and
now I've received a notice of hardware failure.

Just when I set up HA load-balancing, the whole data center outage takes it
all down for 4 hours and counting.

    
    
        Hello,
    
        The host that your Linode resides on experienced hardware
        failure. As a result, drives are being transplanted to
        separate, known-working standby hardware which we keep
        available in case of events such as this.
    
        Again, we're very sorry for any inconvenience caused by
        this issue. Please feel free to update this ticket in the
        event that you have any additional questions or concerns.

~~~
dan_bk
Does Linode automatically check for, detect and replace faulty hardware? Or is
this something you have to explicitly ask them to carry out?

~~~
mmahemoff
Normally they just detect it and raise a ticket like this (which happened
here).

------
thaumaturgy
They previously had a message that Linodes would be returned to their previous
state. Mine was rebooted. Just FYI. Check your DBs and all that usual stuff.

~~~
someotheridiot
Mine was rebooted 2hrs ago but still cant access it.

~~~
thaumaturgy
That happened to mine, too, though not for as long.

I bet they had some host filesystem corruption and they're dealing with it
manually.

~~~
someotheridiot
Thanks for the encouraging words ;) Still not up, 5+hrs outage so far... EDIT:
it was up, but the network interface was messed up for some reason. Rebooted
again and its ok now.

------
j15e
Nodes are now being reported as stopped, but can't be started yet.

"An issue affecting the physical hardware this Linode resides on has been
detected.

We are working to resolve the issue as quickly as possible and will update you
as soon as we have more information.

Your Linode will return to its previous state once the issue is resolved.

Thank you for your patience and understanding."

------
mslider
We have tens of servers across 4 Linode data centers. We get hit with both
data-center-wide and single-server downtime pretty much on a weekly basis.

In the span of 3 months we had something like 25% of our servers go under
"emergency maintenance" downtime. Just part of the ebb and flow of "the
cloud", I guess.

~~~
kevingadd
Wow, that sounds pretty unlucky. Do certain data centers/servers have more
downtime for you than others?

------
zzzeek
update: "The Newark datacenter has suffered from a power outage. "

so, no DDoS after all?

------
jdprgm
Is it just me or does it seem like there have been tons of DDoS attacks the
past couple months.

~~~
colinbartlett
It sure feels that way, but I don't know if they are just being better
reported now. Services to mitigate DDOS have been around for ages, so I
imagine the attacks have as well.

~~~
akerl_
It doesn't help that every time a service gets hit and the result is a
frontpage HN post, more people notice that DoS attacks work for gaining
"attention".

~~~
eli
My understanding is that many recent attacks have accompanied ransom demands.

------
bengtan
My server auto-rebooted and is back up, and has been up for 30 minutes as of
writing (approx 1:00am EDT / UTC -4). Strange that mine is up and fine now,
but they haven't updated the status page yet. Maybe mine is an isolated case?

~~~
j15e
Is your node location "Newark, NJ, USA"?

Mine are still very unreachable.

~~~
bengtan
Yes, in Newark. I was also affected by this outage earlier. Looks like maybe
I'm one of the luckier ones (relatively speaking).

------
BorisMelnik
I feel like this should be included in every press release post down-time.

"As issues unfold, information changes and speculations can lead to improper
actions based on incorrect information."

So very well said.

------
davidw
Hanging out on #linode, it appears as if some people's servers are coming back
up. This is the second day this week I've woken up to unpleasant computer news
:-(

... and mine are back up.

------
j15e
Complete blackout for my nodes, can't reach them at all. Even Linode dashboard
stats for my nodes are empty starting 11:51.

Almost 30 minutes now, I hope it get fix soon!

------
ommunist
I have 12 websites there. zomg. Looks like they are alive now, after 2hrs
outage. Did someone cut the cable for scrap metal?

------
jrockway
I think this is the first time I've ever noticed my own website being down.

------
timdev2
Some of mine back up, others not. Not how I wanted to end my Saturday night.

~~~
timdev2
Evidently DC issues are fixed, since some of my servers are up again. Others
still down due to (related?) hardware issue. Linode support proactively opened
tickets about those that are still down, and upon friendly questioning,
expressed a high level of confidence that they'll be back up once drives are
migrated to stand-by hardware.

