

Gmail service disruption - jpadilla_
http://www.google.com/appsstatus#hl=en&v=issue&ts=1379995199000&sid=1&iid=043f082bc7cd18e15458318035d9bc7a

======
PaulHoule
Google services have lately seemed like something Ralph Nader warned you
about.

Lately I've found that many Google services (like Google Groups) don't work
entirely right with IE or Firefox and they only work right with chrome.

Today I logged into my Google Analytics for the first time, and Google wanted
to verify my phone by sending me an SMS. Well, my house is outside the GSM
service footprint except when there's a strong temperature inversion, so I get
in my car and drive 2 miles, park at the school, don't get the message, I
reboot my phone, the message finally comes, then I drive home, and by that
time I guess my session timed out because after I typed in my verification
code into my computer, it asked me for my password and wanted me to verify
again.

I canceled and then Google Analytics in years came up (after about three
minutes loading) and locked my browser (Google Chrome) up, until I discovered
there was some dialog box that wound up off my main screen that was locking it
up.

And then they came out with the new gmail which seems to be yet another scheme
to shove Google+ down my throat.

Try doing a search for a programming related query and instead of getting the
relevant documentation you get something in broken English that tells you what
a loser you are because you are trying to do what you're trying to do and
gives you ten incoherent reasons why you shouldn't do what you're doing.

Meanwhile, Matt Butts is making Youtube videos about how you'd better make
sure that anybody who links to your site had better make it a nofollow link
because if anybody makes you a link that's not a nofollow link you must surely
be a spammer. Therefore, you should check all your links in Google Webmaster
Tools and write a polite letter to anybody who links to your web site that
they'd better remove the links or if they can't do that, make the links
nofollow.

Speaking of Youtube, if you're in the 80% of the land area of the USA where
all you can get is bad DSL and bad satellite service, it's just unwatchable.
Amazon Prime, Vudu, Netflix and everybody else seem to be able to cope, but I
don't find anything entertaining at all on Youtube when I have to wait 15
minutes to see a 3 minute video,.

Oh yeah, but if you get 50,000 people to plus you, maybe your site rankings
will move from the third page to the second page, or maybe not.

The only organizations who can get away with being linked to today are
Wikipedia and spam factories that used to be search engines, like about.com,
because if Google bans them (as they should) the DoJ will be on them with an
anti-trust lawsuit.

What's worst about it is that if some small competitor comes up with something
better, Google buys them up, shuts them down, and then you're left with
Google's offerings which just get worse every week.

I remember when I was proud to be a Google AdSense publisher because it didn't
have the "one rule to a flat tummy" ads and you know what, they bought the
sleazy ad networks and now AdSense is as sleazy as anything. Oh, but if you
have one erotic painting from 1649 on your site out of 1,000,000 images
they'll kick you out of AdSense without warning the day you are flying back
from a conference in San Francisco after one of their agents watches you give
a talk at a conference about your web site.

I will say I'm impressed with android, both as an end user and a developer --
it's amazing. But my understanding is that their "scalable" platform for
distributed applications is getting long in the tooth and that everything
takes four times longer for Googlers to do it than anybody else. Their
recruiters never stop calling, never get the hint that you don't want to go
work at their beehive where whatever they pay you won't go very far because
unless you commute to Fresno you'll spend it all on a mortgage for a house
that will be worthless in 25 years and, of all places, you know you'll be just
a number to them, if you don't fit their mathematical model of what a coder or
a product manager or whatever it is, you just suck.

That won't stop the recruiters from calling though, because the whole point
isn't that they want your labor but they just want to kneecap their
competitors by taking anybody who can pass an IQ test and wants to live around
San Jose away.

Google used to be about a culture of excellence, but today Google is the new
Microsoft. Because they never release versions, the press never catches on to
the fact that Google's services are like Windows 8.

~~~
powera
This comment has nothing to do with the article except for "I hate Google"
sentiment, and does a pretty lousy job of it. It seems reasonable to assume
you don't need to drive 2 miles to answer your phone, and "Matt Butts" is just
juvenile.

And yet it's currently the top comment.

~~~
PaulHoule
I do need to drive 2 miles to answer my phone, really. Don't get me started on
AT&T and Verizon or the Nigeria-class broadband service in the US. Cell
service sort works in mid-tier cities like Rochester, NY, but go to Greenwich
Village or Beverly Hills and try getting a signal.

When I talk to people in China or India or Australia or Germany, I get a good
connection, but when I talk to people in Encino or Boston I have to repeat
half of what I say because their connections keep dropping out.

Calling him Matt Butts gives me once chance in ten that his team of flying
monkeys won't see a reference to his name and ban all my web sites and every
web site that links to my web sites (except for Wikipedia)

And it's not just "I hate Google", I used to love Google. I've just watched
how Google's ecosystem has wrecked the web. So many topics are dominated by
old crappy web sites (talked about in Hacker News today) and these sites have
an incentive NOT to improve because if you make any radical change in your web
site, there's a high risk that your rankings will tank.

In the meantime, if you want make a good site, there's a high risk you'll
build it and nobody will come.

If the comment above has gotten a lot of votes it's because a lot of people
feel the same way. I think tomorrow I am calling my broker and selling GOOG
short.

~~~
jfoster
Perhaps you do need to drive 2 miles to answer your phone, but I think the
point was that the vast, vast majority of Google's users do not need to do
that. (sidenote: isn't that because you have 2-step-auth turned on? how would
you like it to work, given your circumstances?)

~~~
PaulHoule
I don't have two step auth on, but it wanted to check my number anyway, which
was fine with me. If they'd set a longer timeout it would have been find with
me.

------
blhack
This is really frustrating for me.

I argued up and DOWN with people about us switching to google apps for our
email.

Up until recently, we were hosting our own mail on a server in our switch
room.

"It's more reliable!" I would insist.

Well...it _hasn 't_ been more reliable. There has been more than one longish-
term downtime that I can remember since we switched over.

When I was hosting it myself, I could give people status updates, an estimate
for when it would be fixed, some kind of explanation.

Now, all I can give them is "it's broken. I don't know why, and google isn't
telling anybody why."

Very frustrating, especially considering how much we pay every month for it.

~~~
jlgaddis
You've just described one of my biggest problems w/ outsourcing mail, etc.

Really, downtime isn't a big deal (to an extent, of course) because most
people realize that "shit happens" and there will be the occasional downtime
regardless of who's handling mail.

The problem is that you are, in effect, helpless in these situations. Like you
mentioned, you can at least keep your customers updated when it's a problem
with your own machines. When you've outsourced it to someone else, you're at
their mercy, so to speak.

 _" What's wrong with the mail system?" "I dunno." "When do you think it will
be fixed?" "I dunno." "Today? Tomorrow? Next week?" "I dunno."_

~~~
jfoster
It's true that it can be awkward in such situations, but there are two things
that still make it "better":

1\. Google will bring it back up in a timeframe that is very comparable (if
not better) than your average in-house IT team would.

2\. Given (1), if you ask people how much per year they are willing to pay for
transparency in such circumstances, most businesses will not be willing to
dedicate very much of their budget towards it unless it increases reliability.
The "helpless" feeling is just because you can't see Google's engineers
working to fix it. Surely no one doubts that they are doing their utmost to
bring it back.

------
chuckp
"The email delays are affecting less than 50% of Gmail users."

That is a HUGE change from 0.024% this has been changed from a "Service
disruption" to a "Service outage".

Also, not sure if it is dependent on your Google login, but mine are all in
EST.

~~~
benjohnson
PST is broken too - all my clients are PST and most are effected. About half
are paid, others are grandfathered free accounts and I see no appreciable
difference.

~~~
peachepe
I'm in Nicaragua and having issues with free accounts and apps accounts.

------
ltnately
I've had issues getting steadily worse all day across domain since ~9 am est
or so. I just received this from Google as part of an auto reply to support
ticket.

"Thanks for your patience.

Please read below for updates on the status of your issue:

Updates since last message:

This issue has been partially resolved for some users. We expect it to be
fully resolved within the next hour. The issue caused severe delays sending
emails, especially emails with large attachments. It also caused delays in
loading Drive files with large images.

The delays were caused because of a problem with an undersea cable. All
parties involved are actively working to: 1) Redirect traffic and add
additional capacity to resolve the immediate issue. 2) Fix the problem with
the faulty undersea cable. "

~~~
Terretta
What kind of mechanism for USA-based email service causes email delays for 50%
of Gmail users due to an undersea cable?

Not disputing, just curious about cause and effect here, and Gmail
architecture.

~~~
thrownaway2424
[http://www.google.com/about/datacenters/inside/locations/ind...](http://www.google.com/about/datacenters/inside/locations/index.html)

------
Terretta
We've been reporting delayed inbound emails to Gmail (Google Apps) since
Friday 13th.

Glad to see they're now acknowledging it, though the percentage affected
listed here seems remarkably low, considering an informal survey of employees
and clients suggests from 1 in 50 increasing to 1 in 20 affected over the past
week, with delays up to 24 hours.

------
ChikkaChiChi
Did anyone else just realize that there is NO WAY for us to be notified when
an issue is affecting your specfic account?

/appstatus isn't good enough...

------
busterc
Worse than an outage, I got this dreaded email from Google a few hours ago. I
hope there isn't some sort of attack going on:

Someone recently tried to use an application to sign in to your Google Account
- xxx@gmail.com.

We prevented the sign-in attempt in case this was a hijacker trying to access
your account.

~~~
mr_spothawk
I had this today as well. Where was your attempt coming from? Mine was in HK.

(To be fair, I think it was my own machine attempting to connect via VPN)

~~~
cheapsteak
Got one yesterday from Anhui China. I'm definitely not using any VPN services
there

------
donohoe
TIMEZONE!

I wish they (and others) would include timezone information. I always have to
second-guess if its PST, EST, GMT or something else.

~~~
onestone
Bottom of the page: "All times are shown in your local timezone unless
otherwise noted."

------
fjabre
So does this mean we've lost some emails forever or does this mean we'll get
the emails eventually just delayed?

EDIT: I have an email currently lost in limbo. 3 hours delayed and counting.
Not sure if it will ever get delivered.

~~~
blhack
I think this is a bit more complicated than normal.

You can read the RFCs if you want to, but normally what happens if an email
cannot be delivered is that it will sit in the queue on the SMTP server trying
to relay it with status: deferred.

What's different this time around is that gmail is actually reporting that
emails ARE being delivered, even though they may not be.

Earlier this morning, I did a manual SMTP transaction[1] with
aspmx.l.google.com and got an OK status when I sent the email.

This means that a server relaying mail to gmail would NOT leave a message in
its own queue (to retry delivery later), because the server would "think" that
it had been delivered.

The problem here is that my "delivered" message did not show up in my inbox
until quite a bit later.

SO! Instead of the SMTP system being able to take care of its own outages, the
responsibility is upon google to have properly queued messages for delivery
internally, and then to deliver them when the system goes back to normal.

It sounds like what is happening is that google is accepting the messages, but
then they aren't getting delivered to a mailbox. They're being queued
somewhere along the way (at google).

\--

Your emails will _probably_ be fine, but unfortunately you probably can't call
your mail administrator and ask what is up (unless you work at google).
Normally, you'd be able to look in your smtp logs and see a message status
(delivered, deferred, etc.)

------
chuckp
I can confirm our service is having issues right now with delayed inbound
emails. We are Google Apps for Business users, non edu/gov. Not all users are
effected just specific domains within the account.

~~~
jryedinak
Us too. Seems that when you attach a file, it's even less likely to go
through.

~~~
chuckp
Not sure about you, but I find these status updates a little insulting:
[http://www.google.com/appsstatus#hl=en&v=issue&ts=1379995199...](http://www.google.com/appsstatus#hl=en&v=issue&ts=1379995199000&sid=1&iid=043f082bc7cd18e15458318035d9bc7a)

At least with my cloud servers if there is an issue I at least get an idea of
what is going on or an expectation of resolution. Google... what is really
going on that an issue remains unresolved for 3-4hrs impacting paying users.

------
pfortuny
I know this is nitpicking but this datum is at least confusing:

"The average (median) delay was just 2.6 seconds, but some mail was more
severely delayed. However, this issue did not affect users' access to the
Gmail page or other functionality."

What does "some" mean in Google's order of magnitude?

A median time without the maximum and minimum is not quite useful, as a mean
time without more data.

------
zippergz
This is really killing my productivity today. I don't like being so reliant on
email, but I work remotely, so it's a necessary evil. Having worked in
operations for prominent companies, I sympathize with the gmail team right
now, but my patience is wearing thin with the problem approaching the 5 hour
mark.

------
aroch
So, at the 2012 I/O conference Google announced there were 425million gmail
users. I'll tack an extra 75million to make things an even 500mil for growth
over the last year. That's 12million affected users.

~~~
anamexis
500 million * 0.00024 = 120,000

~~~
m0th87
It's been updated to say "less than 50% of Gmail users" now. We're seeing it
across most/all of our accounts.

~~~
benjohnson
Based on my experience this morning, I'd say that 15% of our 300 or so Gmail
users over about 25 domains were effected. In my opinion, their initial
estimate was way too low and seemed dismissive.

------
bluedino
I was wondering why my emails on my phone weren't going through of a server I
took pictures of this morning, and then I was getting 'rejected by server'
emails after that.

Here I was blaming iOS 7.

------
guilhermetk
9/23/13 4:00 PM Gmail service has already been restored for some users, and we
expect a resolution for all users within the next 1 hours. Please note this
time frame is an estimate and may change.

------
ltcoleman
That announcement at least tells me something is going on, but i am not seeing
a disruption. I have a total outage. I cannot even get to the login page.

