

PayFail: PayPal And Its APIs Go Down - vaksel
http://www.techcrunch.com/2009/08/03/payfail-paypal-and-its-apis-go-down-online-shopping-grinds-to-a-halt/

======
zandorg
Yup, it took me over 30 minutes to get it to work with an Ebay item I just
bought.

I was worried it had been hacked.

Funnily enough, I checked their RSS feed for what's going on, and it told
nothing.

Then I checked twitter.com/paypal and that was garbage too.

So much for the 'real time web'.

~~~
blhack
I think they're probably more concerned with bringing it back up at the moment
than they are with informing people about what is going on. (I understand that
there are more than just engineers working there, but I can understand a PHB
screaming at somebody to stop jacking around on twitter during a crisis).

Either that or they hoped that they could rectify the problem before anybody
noticed.

~~~
potatolicious
This is where your PR department needs to kick in. It's not just engineers
that need to go mad right now, PR folk need to join in on the fun.

------
blasdel
Starting just before 10 AM it became impossible to authenticate. From about
10:40 to 11:40 they went as far as turning paypal.com into a static page with
just this on it:

    
    
      We're sprucing things up right now, but we should be done by 11:20 AM PDT.
    
      Want the details? Checkout the PayPal Announcement Board.
      http://announcements.paypal.com/us/
    

Sprucing things up? When I finally got back in I noticed some new UI with more
gradients+shit. Seems like this is a dev iteration turned cascading failure...

~~~
zandorg
I got the sprucing message too, but it wasn't clear by the time it said
(7.20).

------
pyre
This isn't news to me. I've had numerous times where I couldn't log into
Paypal, but instead would get Paypal 'This is an error, try again later'
messages for a couple of hours.

I also had an item a couple of years ago that I could _not_ pay for. Paypal
would just die in the middle of the transaction with a 500 error. I tried for
at least a week to get it to work, and Paypal customer service wasn't any
help.

------
st3fan
This must be some huge corner case failure because I can't imagine that Paypal
doesn't have some serious failover/high-availability setup.

Or some interesting human error of course :-)

~~~
lanaer
Considering the massive authorize.net outage that occured because of a fire in
1 datacenter not long ago… well, I no longer have faith that large companies
have geographic redundancy ;)

Edit: or, if they have it, I don’t trust that it is implemented in a way that
allows an immediate & transparent (or near-transparent) failover.

~~~
mbrubeck
Amazon does regular "game day" drills that involve things like unplugging an
entire datacenter to simulate an electrical failure. It's the only way to
really figure out how things will break.

