
Google OAuth Is Failing with 500 Error Code - adige01can
https://status.cloud.google.com/incident/developers-console/19008
======
TheIronYuppie
Disclosure: Former Googler

I didn't think I would laugh that hard coming in here and reading these
comments, but here we are. I can virtually guarantee that the following things
are not the root of the problem:

* Lack of compute/networking/storage

* Incompetence of employees

* Back to school traffic spikes

* Just about anything else here

My $0.02. these are almost always due to bad roll outs, usually configuration
changes. But I've been wrong countless times before!

~~~
VectorLock
"these are almost always due to bad roll outs, usually configuration changes."
Some would consider that falling under #2. (Not me though)

~~~
heleninboodler
Given that "87.623% of all outages are caused by changes" is generally
accepted wisdom of running large scale services, I would tend to disagree with
those people. Incompetence: no. Opportunity for improvement: yes.

Edit: made-up number has large margin of error

------
jordanthoms
Google often has a outage or two around this time of the year when all the US
schools come back and millions of students log in at the same time.

~~~
pugworthy
Sounds pretty anecdotal. Can you back up this claim?

~~~
jordanthoms
We run a app making millions of API calls to Google every day, so they show up
in our monitoring, even if they don't make the status page. It's a pattern
I've noticed from getting paged at 3am (NZ time) this time of year going back
the last 4 years, I don't have hard data at hand for it though. It's not the
same thing every year either - this time OAuth got hit hard, previously we've
mostly seen slowdowns and higher error rates on the Drive APIs.

------
herpderperator
Just as I was implementing and testing high-priority Google SSO changes at
work... it goes down :-)

~~~
ilikehurdles
You broke Google. I hope you feel good about yourself.

~~~
jastanton
Better now when you can test your error handling then at 3:30am when your high
priority service goes down :)

------
bmurphy1976
It sure feels like there have been quite a few big outages this summer (Google
in particular). I wonder if they are getting sloppy or this is just bad luck?

~~~
farisjarrah
My gut reaction is that many companies make the majority of their income/deals
during the fall/winter/spring (think back to school shopping, Christmas, re-
signing contracts for the next year, tax season, etc.) Thus, many companies
try to make major infrastructure changes during the summer, when there will be
a slightly smaller negative impact on the business's bottom line in case
something does go wrong.

~~~
FrobozzElectric
My take is that many people are out on vacation during the summer months and
sometimes things break (or break harder than usual) when certain knowledgeable
people aren't available.

~~~
jdm2212
Also, summer interns. A friend who worked at FB said outages there go up
markedly when interns start pushing to prod.

~~~
remyp
> interns start pushing to prod

hey everybody, I spotted the problem! I'll keep an eye on my mailbox for the
giant consulting fee that I assume is enroute.

------
henpa
I can't login to my business gmail account using Google Chrome, but I can
login successfully using Internet Explorer and Firefox. Duh. :-)

~~~
psetq
Chrome in incognito mode seems to work as well.

------
DevKoala
I like the majority of GCP products I work with, but judging by the amount of
issues in the past year, GCP feels amateurish compared to AWS, which we
continue to user in order to host most mission critical operations.

I cant access any of my Airflow clusters atm. :/

~~~
DevKoala
To whoever downvoted me. Censoring the fact customers have issues, doesn't
make the product better.

------
gundmc
Login from incognito works for me so I assumed one of my extensions was
causing an issue somehow and started disabling them. I'm sorry I doubted you,
extensions.

------
mrobins
Link to incident on Google Cloud status:

[https://status.cloud.google.com/incident/developers-
console/...](https://status.cloud.google.com/incident/developers-
console/19008)

------
samcheng
My users are reporting that it's back up, or at least intermittently working
now.

------
Animats
When you use Google OAuth, you give Google the power to turn you off. Think
about it.

~~~
IAmGraydon
Unless you own the complete path from server to user (and no one does) there
is always someone who can “turn you off”.

~~~
Animats
Remember when you could route around failures? Have connections to two
different backbone providers? Does anyone still do that?

------
eecsninja
Googler not on Cloud team, but using Google Cloud Platform for an internal
project. I've encountered my share of bugs and other flaws while using this
platform. I think all these platforms are just too damn complex and brittle.
It's easy for even a smart SWE or SRE to overlook one little thing that will
bring down a bigger part of system.

------
t0astbread
Is this just my perception or is GCP really down a lot this year?

------
frostyj
*Again

~~~
frostyj
Eh, I was stating a fact, don't understand why it got down voted.

