
Gmail and Google Drive Outage - severb
https://www.google.com/appsstatus#hl=en&v=status
======
jordanthoms
There was an outage August 19th, 2019 - almost 1 year ago to the day. As I
posted at the time: "Google often has a outage or two around this time of the
year when all the US schools come back and millions of students log in at the
same time."

My pet theory wasn't too popular but I'm going to stick with it :)

1-
[https://news.ycombinator.com/item?id=20740997](https://news.ycombinator.com/item?id=20740997)

~~~
mrkramer
Millions of people are searching simultaneously at google.com or youtube.com
but servers are not crashing. Issue is not traffic overload but something
else.

~~~
klohto
These are not the same products nor infrastructure

~~~
mrkramer
But I'm sure similar infrastructure architecture was applied to gmail.com as
it was to google.com and youtube.com.

And similar concepts of maintaining by sysadmins are practiced.

~~~
londons_explore
Hah...

Press and hold the F5 key on your keyboard for 2 minutes while on gmail.com.
You will get a "service unavailable" error. About 500 other people whose data
happens to be cohosted with you will also get the same error, and all of you
will be unable to send or receive email, even by IMAP, for about 10 mins while
your particular corner of the data store is restarted and the data integrity
checked.

That doesn't happen on Google.com

~~~
jordanthoms
Ok, I definitely want to know how you discovered that... (and found one of
those 500 people to verify?)

~~~
londons_explore
Not sure if this is still the case, but if you did this a couple of times,
your account data would be permanently migrated to an instance with more CPU
and RAM allocated - you'd also be in with all the other badly behaved
accounts, so reliability goes down lots. The benefit was much quicker complex
searches, and being able to bulk label or delete emails without it taking
minutes or hours.

Don't believe me how slow it is on a regular instance? Try going to "All
mail", selecting all of your emails, and applying a label to them all. In my
experience, it can only label about 50 mails per second, so it can take hours
to do them all. It will keep going if you quit the browser, but will stop if
the gmail devs do a software update, which they seem to do on usually
tuesdays, but never fridays or the weekends.

~~~
1f60c
I find it hard to believe that Gmail will always serve certain users from the
same machines, especially in this day and age, with “cattle, not pets” and
ephemeral containers.

I’m sure they have machines that are only used to serve G Suite and Google One
customers, and maybe some other VIPs, but regular heavy users? It sounds like
an urban legend to me.

~~~
jeffbee
Gmail accounts will sometimes be automatically "hospitalized" \-- assigned
more than the usual amount of resources because for some reason they are
chronically behind or growing without bound -- or "jailed" \-- moved into
isolation along with other bozo accounts, to keep from disturbing normal
people's accounts. Not a legend.

------
cezart
The status from Google Cloud status page offers a bit more technical details
of what happened:

"We are experiencing an issue with Google Kubernetes Engine (GKE) clusters
using node auto-provisioning becoming stuck during node version upgrades. Node
auto-upgrades have been disabled temporarily."

[https://status.cloud.google.com/](https://status.cloud.google.com/)

~~~
piahoo
Does it mean, that Gmail is hosted on GKE?

~~~
brown9-2
No, it’s likely the GKE incident is caused by a dependency that Gmail also
has.

~~~
piahoo
thanks

------
naedish
I'm guessing this outage will allow GSuite customers to claim Service Credits
under the SLA - does anyone have any experience with doing so? Google's
documentation is lacking in detail[0].

[0]
[https://gsuite.google.com/intl/en/terms/sla.html](https://gsuite.google.com/intl/en/terms/sla.html)

~~~
RachelF
Good luck with that - read GSuite's terms, they, Google, define what an outage
is, not the customer.

~~~
dannyw
The linked terms say:

> "Downtime" means, for a domain, if there is more than a five percent user
> error rate. Downtime is measured based on server side error rate.

~~~
breakingcups
Which is fantastic if the error causes the server side not to be able to log
errors

------
synack
Incident says it's affecting anything that uses Cloud Storage, which I'm
guessing is most things.

[https://status.cloud.google.com/incident/zall/20007](https://status.cloud.google.com/incident/zall/20007)

Edit: New incident number
[https://status.cloud.google.com/incident/zall/20008](https://status.cloud.google.com/incident/zall/20008)

~~~
downvote-avoid
Well this happens if you rely only on 3rd party providers. My email server is
still running without any issues and without any indecent for last 3 years
(system reinstall).

Surely no one can send me emails now that gmail is down as most of world is
relying on single point of failure, but this is another story.

Moral of this story is - always own your mission critical infrastructure.

~~~
wuunderbar
> Moral of this story is - always own your mission critical infrastructure.

Sure, if you have the time and money to own it properly. How far do you need
to go to say you own it? Multi-regional servers located on properties that you
own?

------
samdung
I'm so used to Gmail/Gsuite working that i assumed the problem was with my
internet and restarted the router a couple of times.

~~~
Havoc
Yeah same w/ hotel internet.

Especially because Teams status notifications seem funky today too

------
OnWriting
Gmail is practically unusable at the moment. It's been going on for a while
too, so I'm interested in the post mortem for this.

I'm in Sydney, Australia.

~~~
hnick
Do you have a signature image or other attachment? It sends instantly without,
fails with [on the desktop web client]. The google drive server they save our
stuff on is giving a 503.

~~~
OnWriting
My email drafts can't even save. A popup saying "Oops, something went wrong.
Recent changes may not have been saved."

No images either!

~~~
hnick
Hmm odd. I had a draft from this morning (I took a photo on my phone,
attached, then went to desktop later to type the rest up). It would not send
and kept popping up that error you mentioned while typing. As soon as I
deleted the attachment, I could send it. New emails with nothing attached work
fine too in all 3 accounts.

~~~
Yeri
Yes -- remove the attachment (including in-line images) and it'll work

------
bartman
Was greeted by this just as we started work. Observations so far: Reading and
receiving emails via the Gmail web UI works, sending emails via the web UI
doesn't work, sending emails via SMTP works.

~~~
Yeri
It's mostly related to attachment. Regular emails should be fine.

------
tobtoh
Youtube is also affected. You can upload files but they won't process.

------
bleepblorp
Gmail accessible via IMAP at $undisclosed_location. SMTP send works, but sent
messages reappear in the inbox rather than in sent mail.

Haven't tested the Gmail web interface. I don't use it.

~~~
londons_explore
Sent messages appearing in the inbox has been a bug with Gmail since the dawn
of time.

------
tommykins
I have 5 tabs open with emails written ready to send at some point. Had issues
saving drafts, too.

~~~
blindm
In fairness Gmail has a good uptime record, so it's not the end of the world

------
nindalf
I'm not that interested in a dashboard, but I'd bee interested in reading the
post-mortem. Since they have paying customers for these products, they might
release that.

~~~
octvcdrc
They usually provide it on the same page after the incident (an example:
[https://www.google.com/appsstatus#hl=en&v=issue&ts=159398639...](https://www.google.com/appsstatus#hl=en&v=issue&ts=1593986399000&sid=1&iid=be4cdd50ed9bc6458ede98158a71cfd4)
)

------
kuu
I wonder how much money this kind of outages can cost to all affected
companies and institutions

~~~
danparsonson
Hopefully a lot less than they save by outsourcing this part of their
infrastructure :-)

~~~
kuu
Yeah, I imagine so :D

------
Mandatum
Right at 9am pacific too. Good reason for me to wrap up in APAC. Thanks
Google!

~~~
hnick
It's nearly the end of the workday here in Australia. Very insensitive of
them, they should be more timezone aware.

Edit: I seem to be able to send from some accounts but not others. It looks
like emails without attachments are fine, otherwise the drive error messes
them up.

~~~
throwawaywrench
>It's nearly the end of the workday here in Australia. Very insensitive of
them

It's nearly the end of the workday on the east coast of Australia. Very
insensitive of you. Sincerely, Western Australia

The above is a joke just in case that wasn't clear.

~~~
hnick
I guess I kicked an own goal there.

At least it's not daylight savings yet so the QLDers can't be annoyed at me
too!

------
maple3142
Is this only affecting G suite? I still can send and receive email nornally.

------
sushshshsh
That would explain why my ETLs are failing at 3am in the morning !!!

~~~
FridgeSeal
Your ETL processes write to google drive or are dependent on email?

Earnest question, don't mean to sound derisive.

~~~
shrikant
I'm assuming some kind of workflow which uses email notifications, and only
goes to the next step if the email notification succeeds..?

------
laurynas-s
Gmail works but I can't send any emails - so frustrating.

------
antihero
We're unable to publish apps in the Play Store console/fastlane, too.

