

Google Apps Services Were Down - laumac
http://www.google.com/appsstatus

======
minimax
This is the message:

"Sorry, there seems to be a problem. The service you're looking for is
temporarily unavailable. We're working hard to restore your access as soon as
possible. Please try again in a few hours. Thanks for your patience."

0900 CDT edit: works for me now.

~~~
viame
Getting the same message. However, this is happening to some of my logins
only. Same with e-mails, some of them are working fine and some of them are
timing out or asking for passwords. For a second I thought someone hacked my
e-mail. :>

~~~
g-garron
Me too, I was really scared. Now I know is Google's problem I am breading
again.

~~~
laumars
> Me too, I was really scared. Now I know is Google's problem I am _breading_
> again.

I hate it when Google get in the way of my sandwich making ;-)

~~~
Samuel_Michon
Given the naming scheme of Android releases, it seems like Google would let
you eat cake instead.

------
TranceMan
Google Apps status page: <http://www.google.com/appsstatus>

------
AndrewDucker
The Register reports that GMail went down exactly a year ago:

<http://www.theregister.co.uk/2012/04/18/google_gmail_outage/>

~~~
saurik
Huffington Post reports that Gmail also went offline on April 29th of 2010.

[http://www.huffingtonpost.com/2010/04/29/gmail-down-check-
gm...](http://www.huffingtonpost.com/2010/04/29/gmail-down-check-gmail-
ou_n_557351.html)

Google reports that Gmail also went offline on April 24th of 2009.

[http://googleblog.blogspot.com/2009/02/current-gmail-
outage....](http://googleblog.blogspot.com/2009/02/current-gmail-outage.html)

(Sadly, the partial Gmail outage of 2011 seemed to be February 27th, or this
would be an epic pattern of "don't trust Gmail during the end of April" ;P.)

[http://www.eweek.com/c/a/Messaging-and-
Collaboration/Google-...](http://www.eweek.com/c/a/Messaging-and-
Collaboration/Google-Suffers-First-Gmail-Outage-of-2011-850632/)

------
ndespres
I'm a Google Apps reseller and thought, oh no, I should call all my clients
and let them know about this outage before they call me! Where can I get a
full customer list.. the admin panel.. oh.

~~~
stedaniels
Your only copy of your customers, is in a third party reseller control panel?

~~~
ndespres
Of course not. Just the fastest way to get them! Fortunately, Confluence,
Autotask, and my memory are all still intact!

~~~
mathattack
"and my memory"

When all else fails, this is the true old school solution. :-)

------
josephb
Nearly 30 minutes since it started and the status page is it's customary blank
screen.

<http://imgur.com/OLyGf0O>

Almost as bad as AWS at getting notices up.

~~~
WimLeers
Odd, "Admin control panel/API" is marked as disrupted for me.

~~~
josephb
Yeah it is now, only took 30 minutes :-)

The outage is affecting IMAP/POP/HTTP for email plus Gchat, the admin control
panel/API is probably the least impact.

------
misnome
Well, their email services are still working, because I got a message about
our trial ending today and that we need to start paying. I hope this comes up
before we get automatically cut off..

------
JosephRedfern
Getting IMAP rejections, and Google Docs doesn't seem to be working:
<http://cl.ly/image/3W2e0X1O1i0o>

~~~
wodow
smtp.googlemail.com also failing

(edit: intermittently)

------
kyrra
Oddly enough, gmail itself works on my Google Apps domain. But accessing my
account page does not work. As well, I can't log into Gchat using one of the
generated passwords (says password rejected). I'm guessing this is the same
reason IMAP is failing for others.

------
Kristories
Already discussed here <https://news.ycombinator.com/item?id=5563949>

Service disruption

[http://www.google.com/appsstatus#hl=en&v=issue&ts=13...](http://www.google.com/appsstatus#hl=en&v=issue&ts=1366217999000&iid=bb886daf880ab16f9a8b5f8bbe2f34b0)

------
ac132
At least Google Reader still works :-)

------
logn
I don't get why apps and all sorts of other google sites go down a lot, but
search never has in my experience. Why is that? Is search just simpler? Does
it fail in non-obvious ways? Enormous redundancy/fail-over on search?

~~~
bdonlan
Search is (mostly) stateless. They can copy the search database to hundreds of
DCs around the world, and each can operate independently - so if one of them
has a problem, it's not likely to affect the others (unless it's bad data, but
that's why you shouldn't deploy index updates to all DCs simultaneously).
There's some personalization, sure, but you can just fall back to non-
personalized search if that breaks.

Things like gmail, on the other hand, are inherently stateful. When you log
into gmail, you have to eventually connect to one system that maintains your
mailbox. Sure, there might be replication - but the replicas are all talking
to each other. It's surprisingly easy to have a cascade failure in a system
like this, where one of the replicas going down triggers (directly or
indirectly) all the others failing as well. Or you can have some bad data that
gets replicated out, and then proceeds to confuse everything that's looking at
it - unlike search, you have to replicate that data immediately, and don't get
to enjoy the benefits of a staged deployment.

This also explains why not all users were affected - I'd guess that their
system is divided into some number of shards, and users are assigned to a
particular shard. That 0.07% of users affected probably represents a single
unhealthy shard.

------
freehunter
I'm wondering if this is related to the new update they did to the admin
control panel.

[http://arstechnica.com/information-
technology/2013/04/make-g...](http://arstechnica.com/information-
technology/2013/04/make-google-chrome-do-your-bidding-with-new-google-apps-
control-panel/)

~~~
corresation
There is a massively linked Boston bomber Google Docs spreadsheet making the
rounds -- the last time a viral Google doc saw that sort of exposure, the
service went down.

------
appleflaxen
Since you always worry you are going crazy when a high-uptime site is down:
mine too

FWIW: I was able to log in by going to www.gmail.com and logging in suing
myusername@mydomain.com as the username.

~~~
bkanber
Tried this -- didn't work for me unfortunately.

------
terryjsmith
I'm getting IMAP logins rejected as well.

------
runemadsen
I'm also getting errors when trying to retrieve Gmail messages in IOS Mail.
"Wrong username or password", but it works in a browser.

------
js2
It's intermittent for me. I first noticed trouble when the Android gmail app
said it couldn't connect for one of my accounts. I cannot login to my apps
domain control panel right now, but I can access its email via both IMAP and
the web interface. However, another account on that same domain cannot access
via IMAP.

------
Leszek
This explains the SMTP problems I was having, I was worried someone had hacked
my account or something...

------
yaskyj
It's interesting that I kind find news for this on google itself, but Hacker
News already has two posts.

------
glennos
The dashboard hasn't caught up. Admin Control Panel is one thing, but
GMail/Apps has a huge impact!

------
dotbill
As I said in the other google apps thread on hackernews..

I thought I'd give them a call to find out whats going on, however to call
them I need my support/admin pin - which is only accessible from the admin
portal and without it they won't answer the phone. Nice.

------
silasb
Appears Groups is down also.

------
brokentone
Still getting errors sending transactional messages to users on gmail

Apr 17 10:23:30 postfix/smtp: connect to gmail-smtp-
in.l.google.com[2607:f8b0:4002:c02::1a]:25: Network is unreachable

------
mattvot
I did wonder about this. My phone said my username or password was wrong for
my email, got a bit worried, so I tried with their webmail and it worked
there.

------
theg2
Was having problems with play developers console yesterday, suddenly it was a
404. Seems like they're planning something in my opinion.

------
X-Istence
That explains why I can't get my email =).

------
bdg
This is happening for gmail for some accounts, but mine seem to be running.
Seems the outage is per user account.

------
jlengrand
I got a 500 when I try to access our Drive too. It's been an hour now. Using
the business google services.

------
delinka
I also can't authorize an app to have access to a Gmail account. Google's auth
page presents an error.

~~~
traeregan
Same here. Intermittent though. I'm like 2 for 5.

------
arcosdev
Faaack. I thought I was just having a local connection issue. Same problem
here.

------
digitalpbk
digitalpbk$ nslookup smtp.google.com Server: 8.8.8.8 Address: 8.8.8.8#53

 __server can't find smtp.google.com: NXDOMAIN

SMTP servers are down for me. Anyone else?

~~~
brokentone
NXDOMAIN wouldn't be server down, that's no DNS record. Still won't work, but
different (much bigger) issue.

~~~
tekacs
I'm pretty sure the 'bigger' issue here is that it's smtp.gmail.com -
smtp.google.com is, I believe, _always_ down on account of not existing. :P

~~~
deno
Works fine for me:

    
    
        > dig +short @8.8.8.8 smtp.gmail.com
        gmail-smtp-msa.l.google.com.
        74.125.136.108
        74.125.136.109
    

[http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/nsl...](http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/nslookup-
flaws.html)

------
bkanber
Down here. Additionally, my iCal won't sync with goocal.

------
meerita
I cannot access my email from Sparrow app too.

------
frankydp
Webmaster tools also down with auth failure.

------
jschuur
Google Drive is down for me on Apps too.

------
varun1308
Yeah.. my pop mail stopped working too

------
calleskonto
The provision api is still fine doh.

------
ckdarby
Why do people report this kind of stuff on hacker news. I am sure everyone
knows how to google for a site's status page.

------
EvanAnderson
SAML logons are failing, too.

------
paulmolluzzo
It's back.

------
level09
not only that, seems like pop3 is also facing issues ..

------
abhi_21
Now its working..

------
Selfcommit
And We're up.

------
eggnet
down for me too

