Hacker News new | past | comments | ask | show | jobs | submit | page 2 login
Google outage – resolved
2316 points by abluecloud 4 months ago | hide | past | favorite | 827 comments
various services are broken

- youtube returning error

- gmail returning 502

- docs returning 500

- drive not working

status page now reflecting outage: https://www.google.com/appsstatus

--------------

services look to be restored.




As many comments in this thread have mentioned, the crash seems to happen when when logged in; youtube and other services seem to work when don't send the session cookie. This suggests something very fundamental related to user account/sessions is failing...

Did Maria Christensen make a mistake when adding "return true;"?

(This is a joke referencing Tom Scott's 2014 parable[1] about the danger of designing a system with a single point of failure. Tom's tells the fictional tale of a high level employee at Google adding "return true;" to the global "handleLogin()" function.)

[1] https://www.youtube.com/watch?v=y4GB_NDU43Q (you might need to open this link in a private window...)


1. Phone Gmail won't load. 2. Turn off wifi and use cell network. Still won't load. 3. Check hacker news.


Yeah I couldn’t get the hours of operation for a store near me on Google maps. Thought it was wifi at first.


Don’t blame people, blame systems and processes. I assume Google has a blameless process too. If an engineer can bring down huge swaths of google that’s not a human problem. As an eng your you should heavily invest in a sane test -> deploy -> monitoring process, and reward reliability.

Give people a bonus when things didn’t break, not only when there is a superhero that fixes broken things. Then you’re rewarding fragile systems that need superheroes.


It's super interesting that all Google services that I've tried are down _except_ for Google Search. What would isolate Search from the rest of Google's products such that it wouldn't be affected by a mass outage like this?


I’d guess Search is pretty well segregated from basically everything else because of how valuable it is - I’m logged into Google on Search and it works fine (unlike everything else)


Search works for me but it thinks I’m logged out - it just prompted me with the 'please agree to our T+Cs' dialog.


It looks like it is related to authentication, landing pages seem to be working, so going incognito you can still use youtube.

I would guess the search page does not actually use your credentials for anything since fingerprinting is separate from login?


I would've thought Google remembers what you search while logged in and sells it off. Maybe they do it in a different way - never really thought about that. I'm logged into Google when I search but my profile fails to load at the top right.


They probably do, but it is not essential to the functionality so it can work even if auth fails.

Fingerprinting happens separately because then they can identify you even if you are not using your own computer or are in incognito. Login is just a convenient way for you to tell them who you are.


It is something related to authentication. Even YouTube works without being logged in


When logging into any of my gmail personal/free and corporate paid for accounts i get "Couldn't find your google account" in red writing


Google Search Images doesn't work either: https://www.google.com/search?q=random&tbm=isch&source=lnms


Google search was slow for me during the outage (5-10s per request)

I imagine google search only has optional dependencies on login and correctly falls-back if these fail


Probably something related to users authentication then


The fact it seems to be related to login reminds me of the Tom Scott doomsday fictional lecture which I would link to but you know...


https://www.youtube.com/watch?v=y4GB_NDU43Q

Works fine once you log out.


opening it with Incognito mode works too


Nice. very subtle reference to our dependability on google products. you can still link the YT lecture from a private session.


Oh yeah that talk was crazy!!


It's been ages (literally) since I felt so helplessly out of a conversation not being able to check a reference instantaneously.

Weird feeling...


Just got a text from my kids' school saying GMail is down and to use the school's direct email. Immediate reaction was "Google isn't down you idiots, the problem is on your end", go to check GMail, yep it's down.


yeah this is like saying it must be a compiler bug


I had a few of these this year, e.g. thinking I have local connectivity issues when it turned out the entire backbone is in a bad shape[0].

Or that time when 80% of the apps on my iPhone crashed on launch[1] — must be an issue with the device, right?

[0] Level 3 Global Outage - https://news.ycombinator.com/item?id=24322861

[1] Facebook iOS SDK Remotely Crashing Spotify, TikTok, Pinterest, Winno and More - https://news.ycombinator.com/item?id=23097459


Unless I'm misunderstanding something, they are going to have an immense SLA claim issue. 99.9% SLA on Workspace services, so any business paying for Google for Business (now known as Workspace) is going to have a credit claim (assuming the outage is longer than 43m 49s which feels like it will be).

Edit: As I comment it looks like things are coming back! Timing or what...


57 minutes for full Gmail restoration according to https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...


Looks like the outage was very close to that duration. Over what time period is that?


0.1% of a year is 8h45m. Or did you mean 99.99% SLA?


99.9% but calculated on a monthly basis


Gmail and Youtube are back online for me. Hangouts is beach balling. There is a warning at the top of my screen though.

"Gmail is temporarily unable to access your Contacts. You may experience issues while this persists."


Everything is affected. Its Gaia(Google Auth) outage, most probably.

Disclaimer - Googler here whose workplace chat is not working.


The nature of this downtime is quite severe. How are Googlers resolving this if their internal communications are down?


There are other ways to deliver packets... https://en.wikipedia.org/wiki/IP_over_Avian_Carriers


There is IRC as backup


I recall that their SREs use IRC?


Yep. At least some of the teams do.


not hangouts? allo? duo?


I don't think Allo and Duo are products designed for use in teams/organizations, or marketing themselves as such.


Good ole phone calls is my guess


I can't login to anything via Google Auth this morning.

It reports error that my Google Account can't be found. I thought my account had been deleted!


This is bad opsec


Atlassian's #HugOps page[1] collates messages of support for Devops as a public embedded Google Map and it really encapsulates the current feeling!

https://imgur.com/a/3RurTYf

Don't forget to #HugOps all the people who've been woken up on a monday morning with this! Hope this gets resolved soon :)

[1] https://www.atlassian.com/software/statuspage/hugops


Google is also rejecting SMTP ingress traffic.

(delivery temporarily suspended: lost connection with aspmx.l.google.com[2a00:1450:400c:c04::1b] while sending DATA command)


That is Really Bad™.

People are going to miss critical emails with no recovery.


Fortunately email was designed back in a day when mail servers could fail. Don’t worry - proper SMTP servers will retry patiently.


These are temporary deferrals, a properly configured outbound mailserver should retry for days.


Emails will be delayed but sending SMTP servers have a retry mechanism up to a few days.


First the two Gmail accounts I had in Thunderbird logged me out and asked to relogin.

Then Google said my accounts did NOT exist which made me feel very uneasy. Banned? Lost all my data? OMG.

Then I got error "502 That's all we know" after entering my passwords.

Finally I realized it couldn't be just me, so I opened HN and confirmed my suspicions.

It's all up and running now but I was really scared.

I'm really curious what happened because it surely looks like Google has a single point of failure in their authentication mechanism which is just horribly wrong. The company has over two billion users - a situation/configuration like this just shouldn't be even theoretically possible.


Google Pay contactless transaction not working in UK

news.google.com returning 500 finance.google.com returning 502


oof i didn't even think of google pay


I’m seeing poor internet performance overall, even accessing HN...too many relying on Google DNS maybe? Even ISPs?


Probably just people checking non-Google sites whether it's everyone or just them. 8.8.8.8 at least seems to be working, as does Google DNS for domains hosted on domains.google.com.


The Google outage is one of the top BBC headlines now, so it has become mainstream news: https://www.bbc.co.uk/news/technology-55299779

"The outage started shortly before noon UK time, with Google sites returning server errors when visited.

Users around the world reported problems with Gmail, Google Drive, the Android Play Store, Maps, and more.

...

Despite the widespread outage, Google's service dashboard for its services reported no errors."


"Google has been contacted for comment, but one spokesperson said they were unable to access their email."


It has become regional news in local regional newspapers in South India. It's that big.


This puts into sharp relief how outages and downtime can hit ANYONE.

Cheers to all the small SaaS businesses out there keeping their services up and running without much of a hick-up all year round.


210 points in 5 minutes. 500 in 10. Seems everyone see's the status page and is like "hey, time to check hn" ;-)

EDIT: You can permanently refresh and see the score increase by 2-3 points each second. Wow.

EDIT 2: 1800 after one hour, but a few points were probably lost due to HN downtime. Being quick to post is important, guys;-)


I feel like HN's server aren't very happy with everybody doing this though.

Stop bullying the servers :(


Yep, HN getting it's very own HN hug of death from google


Haha, the irony!


This wouldn't be a problem if HN migrated to a blockchain written in go for performance


Wait. I thought Rust is the new kid on the block


Would never work without a powerful AI engine in the middle.


This is probably like an user flood wave, crushing all platforms in its wake. Give it 10 minutes and Reddit will be down as well :)


HN was actually down for me when I first checked. I was wondering what possible infrastructure HN and youtube could have in common before it clicked


I came to HN to see if there was news to calm me down, we just started business day here, and as soon I tried to login some 30 minutes ago, Google claimed my paid business account didn't exist, so I came here to check if there was some mass banning or something like that.


I noticed Gmail and of course the google doc I was working on were unavailable. I didn't come check HN until I said to my wife "Google's down" when she complained her email didn't work and she said "check the news to see if Google's really down"


who would have thought a lot of folks here also use google products.

for real, though, hacker news is getting slower every second since this thread is up. i won't be shocked if it goes down due to the surge of traffic due to google outage


YouTube shows a "something's broken" error. Similar errors when trying to access Gmail. Tried on multiple browsers and connections, got the same result. The search engine works fine, as of now.


Youtube works when not logged in though.


You are right, the YT homepage opens up in private tabs. It seems strange, I wonder might be causing this.


Services are not restored. Some came up again, some not. My Gsuite business mail is still completely down, while youtube started working again.

I'm pretty sure there will be some internal conferences at Google after this to make sure infrastructure problems can't propagate across the entire company and world at this rate even in the event of a sysop fatfingering a console...


If you are still down try restarting app... this worked for me


Google Home and Chromecast also rendered useless. _Hey Google, how are you?_ leads to some very passive aggressive replies.


In my house, it leads to a hilarious echo of multiple devices responding simultaneously with an error message. Having only one device respond seems to be cloud-mediated too.


"Hey Google!"

"Something went wrong. Try again in a few seconds."


So happy that I didnt even notice this, because I stopped using Google's services :)


Kudos! What do you use instead of YouTube? That’s the main product keeping me on the franchise


You mean as a consumer or as a creator?


Consumer


Is Google Auth down because of the SolarWinds Orion exploit?

https://cyber.dhs.gov/ed/21-01/


Wow, I'm glad there are a few things that I do that don't require Google auth.


Pretty much anything that requires login is broken.


YouTube Music failed in an interesting way. My music has been playing, but I started hearing ads (I'm on Premium), which prompted me to check what was going on.


That supports the claim about it being an auth issue.


I am still listening to youtube bot on discord... Maybe it will crash soon.


edit: added details edit: redacted my phone number edit: big mistake to add phone number edit: I think illic is right, probably not me edit: removed details


If it was you, why the hell on earth would you bother exposing your personal phone number on the internet and asking google to call you on this post? Like, seriously...

Wouldn't you rather call them directly on the hotline...?

That's a phone number from Bulgaria, and it looks like it should be part of the Vivacom GSM network, so I guess it's his personal mobile phone number or a scam.


> why the hell on earth would you bother exposing your personal phone number on the internet

Why not? It's not like you're gonna receive death threats exactly. I've had my personal phone number on my public website and in the footer of every outgoing email for 15+ years, never had any problems, spam or otherwise.


yes personal phone number.

Had to do what it takes...

To save the company I work for and all of our customers data. Which all is in Google Cloud!


Or the more realistic scenario is that the outage happened before you did getIamPolicy which is what caused the garbage data.


That makes sense


That reminds me of the time when I plugged a network cable back into an Active Directory domain controller. At the exact same time as the RJ-45 plug would have made the little "click", a door slammed shut and a polished steel tanker truck drove by the window, shining a bright light into my eyes.

I had just plugged into a cable into the most important server in the organisation and I saw a bright flash and hear a bang.

All was well, it was just a coincidence, and a good reminder that sometimes shit happens and it's not us, it's just timing.

Relax.


This isn't true, but considering every Google outage is a one in a billion rube goldberg domino machine, it could be true. Put this comment in the post mortem!


It’s probably not you.


Although since it seems to be back up now I suppose there is one way we could find out for sure.


If you are right; congrats! you just got few googlers fired!


Google doesn't fire people who cause outages.


What exactly did you do that makes you so confident?


Exactly after my setIamPolicy API request to Google Cloud was the exact moment everything went down.


Probably dozens of other people executed comparable requests in the instant you did.


Indeed. I deleted a gcp project at the same time.

However, it would be fun if it had a UUID clash with a google service :)


I'd wager a guess that you set up some weird 'expression'? coupled with some bug in the IAM service, maybe some stale resources that you were deleting at the same time?

I'd then assume once expression is evaluated the services end up busy looping / proxies throwing out internal errors and taking out capacity.

Still, you shouldn't be able to cause downtime to more then a few servers in the extremely unlikely case I am anywhere close.

PS: - I haven't used googles IAM so guessing after a few min of reading docs.

- you are incredibly unlikely to have triggered this at google's scale.


Do you have an exact time?


What are IAM permissions?


Just permissions. The "IAM" can be safely dropped. It's exactly what you think it'd be: restrictions and privileges.

"IAM" is basically the name for a specific model of doing it.

Unless something really crazy happened, this user is unlikely to be correct. Accounts are supposed to be firewalled/sandboxed in a way that you can't contagion across to someone else's let alone systemwide.

It's possible (some sweeping script on a powerful connection that smashes just the right things or some exploit to break the sandboxing), just probably not likely - especially unintentionally.

But crazier things have happened https://books.google.com/books?id=rRp7DkTegMEC&newbks=0&prin...


It's what Amazon calls your cloud login account

Identity and Access Management (IAM)


Коле, ти ли си?


Hahahahaha this is gold


It was obviously not you.


Babashka is dangerous...


I highly doubt it.


Ummm.. no.


I am not google, but if i were to start looking for idiots to throw under a bus for this outage, you will on top of the list.

besides google won't be exactly calling you because you made that comment here.

other commenters are right that you should not expose your personal number on internet regardless of it was you or not


> Last thing you want is a conversation with google's lawyer

Google more or less want people find their weaknesses so they can patch and secure them. A person accidentally triggering a global outage is not something that would cause that person to get lawyers on them. Especially not something that only affects his or hers GCP-project.


obviously it was an accident


You say "outage", Google will probably say "sunsetting" or "deprecated."


But why is HN so slow? Pages take like 30+ seconds to load for me (German vantage point, other sites are fine). Does it timeout on some Google dependency or try to use a Google submarine cable or something? Or is everyone just posting to HN about it?


I think that is simply because all HN readers using some google service jump over here to check, because google's own statuspage is useless.


they arn't prepared for such big news, you are part of the involuntary DDOS


It is called HN hug of death (or Reddit hug of death). the traffic due to google outage is much more than usual


I guess it's the demand of everyone checking HN to see if Google is down or not.


If memory serves, HN is hosted on a relatively lightweight server configuration.


Google login also seems to be affected, meaning all services using google account login are inaccessible if you aren't logged in.


Gmail, Calendar, Google Accounts down in Germany


That's why my Thunderbird constantly popped up to enter my Google Mail credentials…


Same, my work mail does not work. I was supposed to be in a meeting right now.


As a consequence, many websites depending on Google services are stuck too. Like the nytimes.com.

Ah btw HN is being crushed by the load lol.


Seems like services does indeed work in incognito.

https://news.ycombinator.com/item?id=20132520 Maybe this beast kicked the bucket.


I happened to use Google Takeout to download everything I had last weekend (~200 Gb)... at first I thought my account was banned and that was the only solace I had, but this definitely is killing my productivity.


I really hope this is just a temporary issue and no accounts are going to be lost. If that happens, way too many people are going to feel the pain of thinking Google makes backups unnecessary - including me.


Google has now updated the status page, every single service has an outrage: https://www.google.com/appsstatus


And somehow Pokemon Go is down, too. They use GCP for hosting, but I'm not seeing other outages on GCP. Maybe they have deeper integration with Google's Cloud than publicly disclosed?


Discord was sort of down (wouldn't load), but I could log in again a few minutes later and everything was OK. My company's product is hosted on GCP and there are no problems that I can detect without administrative access (which I can't obtain because it's behind a Google login); services are returning correct results.


Google Maps for sure, maybe some GCP as well (as console is down, Discord and Playstation Network as well are having issues - which seems like some GCP service)


It works if login with facebook or Apple ID, so GCP itself is fine, but I guess admin cannot login to management


Discord is GCP and down also.


I feel kinda bad - I'm pretty sure I broke Google as I was uploading quite a few large files via Google Drive and it was chugging pretty hard.

And now because of that it seems HN is also hurting.

I'm sorry internet.


Maybe it’s time to go outside?


What is this bright thing in the sky?


let's not make panicked drastic decisions just yet


back inside we go guys:(


This outage is an idea from Calico then? (just kidding)


Let's not get ahead of ourselves :)


Seems like the auth solution that is down. Every service work in incognito/private. Shared drives in Docs/Drive does not work. Probably connected to the auth service?


Gmail service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change.


We've got about 3,000+ people a minute hitting Downforeveryoneorjustme.com, pretty crazy.


I can access youtube if I'm not logged in (Incognito)


Thanks for the tip! Several people said its related to logins/auth so this helps getting access to videos/info.

maybe this is their fingerprinting/cookies/tracking server side logic has some bug that is failing?


The moment you realize that the only way you notice Google is down is through HN.

Cheers all you selfhosted-FOSS-alternatives! Time to bump up those Patreon contributions...


Hackernews is also having some kind of stability issues, either everyone is on here complaining about the google outages, or somehow they share infra.


Word from inside google is their auth is down, so anything that doesn't require auth should work fine. Log out of youtube, works fine.


Even Google Sign In service is down. Sites like Airbnb, Duolingo, etc that uses Google Sign In can't log in their users.


How's things like Stadia? Or is anybody using more exotic Google stuff? Like, how are their selfdriving cars doing?


Stadia is down.


Stadia is down for me


HN is also down because everyone is checking whether Google is down


Turns out hacker news has an indirect dependency on google!


HN outage is soon to be the next one. The HN server is overwhelmed by the traffic due to Google’s outrage. :/


Google is scaring the shit out of us, we were already trying to move out of Google for some past bizarre outages, and now before they went down, they outright claimed business account didn't exist anymore, our business drive and all our IMAP e-mail clients got kicked out.


This is a great and interesting thread. But I believe nobody discussed here the opposite and postivie effects of this kind of golbal outages. There are funny calculations for example of having less spam mail open-clciks or things like that. So think in such massive otuage of millions of users and auatomated process what means in term to generate less CPU, RAM, I/O and as consecuence less Power/Cooling >> less CO2 and pollution... It's huge! and at the Earth's weather impacts is like charge trillions in advance that we''ll have to paid in a future not only with money... (Economy Degrowth)


Bummed to have not received an email about what's happening and why from the Google team.

Has anyone on a "platinum"/"enterprise" google workspace plan received any relevant communication about what's happening and ETA on uptime?


Looking at the other comments, it seems highly likely zanzibar[1] is down. Their auth service.

[1] PDF: https://research.google/pubs/pub48190.pdf


zanzibar is an authorization service, and we are looking auth as in authentication here

So no, not zanzibar


GCP console is not working too https://console.cloud.google.com/, however at least the services are still running :) phew


It's a mixed bag. Cloud storage is down but at least cloud logging is up to stream all the errors.


Yup, Ops Genie is BLOWING UP with alerts... but can I get into GCP to see what's going on? Nope.


Yep.

Launched YouTube app on the Roku and was prompted to sign in. Opened browser on PC, entered "activation code" from YouTube app, after entering my Google account username on the login page, it presented me with the reassuring "Couldn't find your Google Account" message.

Tried logging in to Gmail directly with the same effect.

Thanks to the Twitters, I realized that Google hadn't canceled me, specifically (apparently they decided to temporarily cancel everyone!).

FWIW, I typically get directed to their Chicago datacenter(s).

(Note: 7 A.M. on the U.S. east coast on a Monday morning; at least they have impeccable timing!)


What do you think, how much money are they losing right now every second?


I was thinking the same thing. Also this funny bit that’s relevant: https://entertainment.theonion.com/google-shuts-down-gmail-f... “Google Shuts Down Gmail For Two Hours To Show Its Immense Power”


Do tech companies post that kind of data?

When I worked in the automotive industry, the Detroit Big 3 American auto makers had figured out their cost per minute of downtime. They would even charge that to their suppliers and anyone who shut down one of their plants.

It was something like $10,000 per minute since they’re able to roll a complete vehicle off the assembly line about every minute


We use Google Cloud and Firebase at work and our application and servers are still up and running. Can't say the same about the Cloud Admin Console, tho.


Somebody threw a Spanner in the works is my guess. Global-scale database underpinning everything (or maybe the giant that is colossus is having an off-day).


"all green, which does not reflect reality for me (e.g. Gmail is down)"

This isn't true. Which make it more complicated as we try to figure out what systems are down -- since their status board is usually faked.

gcloud compute instance-groups list-instances [removed] --zone=us-central1-c | awk '{print $1}' | grep -v NAMEERROR: (gcloud.compute.instance-groups.list-instances) Some requests did not succeed: - Internal Error


I'm very interested in HN' crowd back-of-the-envelope estimations on how much this outage costs the world (when we know how long it lasted).


It's impressive how fast they are at fixing outages, barely a coffee break required. I'm almost starting to believe in this DevOps thing...


Still down for me. Must not be a "vast majority of users" bit of a premature "green" on that.


Interesting that I have been watching a long video (over an hour long) and the buffer is still loading while YouTube in general is down for me


Amazing how many products are clearly getting affected due to this. The top 18 products on downdetector.com are all Google-owned or related


is it me or I noticed that google search rarely goes down? is it somewhat in an isolated system or a fundamentally different architecture?


youtube-dl is still able to download videos, which is quite interesting ^^


Singalong: She broke the whoooole world, with her change, he broke the whole wide world, with his change, they broke the whole world with their chaaaaange!

I'm sure its stressful right now. But someday, these engineers will look back and retell the stories about how it happened and the lessons they learned. Hopefully with a laugh.


GOOGLE UPDATE YOUR GCP STATUS PAGE. Stop making us run around trying to figure out what systems are down.

Running: timeout 60 gcloud compute instance-groups list-instances [removed] --zone=us-central1-c | awk '{print $1}' | grep -v NAMEERROR: (gcloud.compute.instance-groups.list-instances) Some requests did not succeed:

- Internal Error


I’m seeing poor internet performance overall, even accessing HN...too many relying on Google DNS maybe? Even ISPs?


Well, HN's auto-hug of death. Also, probably most sites can't load either analytics nor ads...


FYI if you have cronjobs using Google apis, disable them or delay them.

a bot is flagging ips as abusive. it should clear up later.


Gcloud/GKE command line utils are also having problems.

Was deploying some test applications and kubectl started complaining about gcloud auth helper throwing not zero errors. Trying to launch cloud shell from the website and nothing happened.

The web application which is a site that does not rely on any external API is running fine.


Google Meet and YouTube also 500 in Brazil


Not just google, hackernews was also displaying: "We're having some trouble serving your request. Sorry!", has it something to do with google architecture?

Btw, that's funny you listed all the google services exactly in the order I found them unavailable: first youtube, then gmail, then docs


Can I just say I'm remarkably relieved I have an excuse to kick back for the next hour and do nothing?


I hope it lasts a bit, not just five minutes. We might finally question our dependency on Google services.


It looks like the auth service is broken. Everything works when logged out. Attempting to log in gives "Account not found".

EDIT: Looks like they logged everyone out while they fix the auth issue, presumably so people can use YouTube and other stuff that doesn't necessarily need login?


Was wondering why youtube can't show videos from the search page , it even reverted my language settings.

South Africa


Seems like the incident lasted from 04:00 AM PST to 04:30 AM PST, at least for BigQuery (sorry, image timestamps are in JST, since I'm in Japan): https://imgur.com/a/kiHED6v


Definitely started before the hour.


Can confirm Incognito Mode and Private Tabs are still working on YT. Seems to be something auth based.


All services are still down, https://www.google.com/appsstatus#hl=en&v=status

Crisis averted, was not locked out. Gotta take out all the data though when it gets back up.


Thank God! Google Cloud Print is still working! I'd be destroyed if that ever stopped working!


The worst outage I've ever seen with Gmail. I can't log in, getting "Couldn't find your Google account" message. Also, with the already logged in account, I can't browse anything but whatever is in cache. I'm definitely worried.


It's time to decentralized Gmail, Youtube, and Google Docs, maybe their Search Engine too.


That's what it was 15 years back. I was slightly concerned when Google Mail and Chat were integrated. Look where we are now.

Right from "The battle of the red cliffs", we will never learn the disadvantage of lashing too many boats together.


But our tech right now is far more advanced than 15 years ago.

We have IPFS, Blockchain, dat protocol today. I think it's possible to kill the giants.

Even, Tim Berners-Lee want to decentralized the web: https://techcrunch.com/2018/10/09/tim-berners-lee-is-on-a-mi...


Even? Tim Berners-Lee tried since day 0 to make the web decentralized. One of the initial requirements in the 1989 proposal for a global hypertext system included "Non-Centralisation" where "new system must allow existing systems to be linked together without requiring any central control or coordination". See https://www.w3.org/History/1989/proposal.html for the rest.

While it went OK-ish for the internet, we massively screwed it up for the web.


No we didn't. The web is very nicely decentralised, especially with the general malaise and decline of Facebook.

What isn't decentralised is web apps but TBL never intended the web to be used for full blown desktop app replacements in the first place so no surprise it doesn't meet its design goals for that.


Anyone experiencing lost emails? I'm not sure if it's actually me messing things up or because of the Gmail outage, I couldn't find an extremely important email and I've tried archive, trash, deleted, spam, sent boxes..


I did not realize how dependent I was to google/google services until 30 minutes ago.


This affects GCP as well as Google's consumer services. I'm here because I was paged about low traffic for a service. Turns out to just be that I can't access any metrics at all from Google Cloud Monitoring (i.e. Stackdriver).


So to easily fix such down time, i have alerts setup for website, server and API using https://simpleops.io/

After that I realised how frequently you can get down time


These rare events help make extremely visible how much of our general basic infrastructure we've farmed out to a small handful of companies, centralizing something that was supposed to avoid the problems of centralization.


Look the status page still works https://google.com/appsstatus#hl=da&v=status but it says everything is broke.


Luckily nobody is using Google services for anything critical. Or what, are you?


I feel like Google is having more outages recently. I worked there for 4 years and even in such a short time you really noticed the shift from engineering focus to business focus. But maybe it is just a coincidence.


I interviewed for Google many years ago and recently. The expectations fell through the floor.

I've see plots of number of employees too. I'm not sure how Google will dig itself out of that one.


They made you interview again even after having worked there before? That's odd. It used to be that they didn't do that.


I never worked for Google. They wanted to hire me after the first set of interviews, but I took a different opportunity at the time. Interview process was an intense, and the interviewers were sharp. I came out even more impressed with Google. This was way back in the day -- early '00s. I would totally be excited to work for that Google. It's just that, well, there are lots of awesome things to do, and I had (what seemed like at the time) a more interesting option. I sometimes had regrets and sometimes not.

Second process was maybe 2-3 years ago. It didn't get to a full onsite, since after a few conversations with the team, it was clear there wasn't a fit. The old arrogance was still there, but without the same sharpness or cleverness. I spoke to a team working on a new product (under NDA) in a field I had a lot of experience in, and:

1) There were computer scientists without a clear understanding of the target market domain.

2) They believed they were the best-and-brightest, and didn't need to consult experts in the field.

3) There was a lot of hype and salesmanship.

I like to be surrounded by smart people, and it felt like I'd be the sharpest guy in that room. I was a better computer scientist, AND better domain expert in the field the product was in than anyone on that team. That's not a position I want to find myself in.

And I have a job which pays a lot less than Google, but I'm surrounded by smart people I learn a heck of a lot from, where I'm working on meaningful things, and having fun. I'm also continuing to build my personal brand.

Now a few years later, the product they were working on never shipped, so if I worked for Google, I would have likely been on a failed project. On the other hand, my mortgage would likely be paid off.


Switzerland's down too. It works if you access it without session though.


Seems to also affect Google Cloud console to some extent. Dashboard data is not loading at the moment. Not sure what services exactly are affected there. Google Cloud SQL seems to be up and running so far.


youtube greece is down returns "oops" error but i can still playback a 2 hour video that I was already watching before it went down. Trying to open any other video or channel returns this error.


At this point Amazon and Google should just have a static page with green buttons for their status page.

They NEVER show red, or oddly even go down themselves during outages. ( Last time amazon went down springs to mind).


Yup, suffering it right now.


At times like this I am thankful of running a Google takeout periodically.

Not because I use it as a live failover for Google services, but because these outages are good reminders of the mortality of online services.


How long did your takeout take? It’s been 3 weeks and mine either says not ready yet, or when it is it will say unable to download and force me to start again...


When I request a full takeout it usually takes 2-4 weeks to generate.

This time around it was missing a .tgz which had error'd. A subsequent full takeout request took a couple of days.

The .tgz files I find can only be reliably extracted on a Linux... the Gmail export for me is 20GB which in the last few exports has been a single file in the last .tgz



Google employees can't login to update their status board on https://www.google.com/appsstatus ?


Seems to be down in a bunch of places https://istheservicedown.co.uk/status/gmail


Good. Now, at least a few people would search Google alternatives in DDG


Search is still working. It’s Drive, Meet, and other stuff that millions of people depend on for work, so maybe stow your glee for a moment.


Google still is the best when it comes to search. But its also important to know the alternatives. You will be surprised how many people dont know Google alternatives. lol


How coud all of googles websites be down this long tho. You would at least think google being the company they are would have some sort of like backup serevers or backup plan type stuff. Idk


Not a fan or even their user but kudos to Google for admitting the failure. Amazon (at least AWS) customers would have probably been shown a green dashboard and told to review their code.


It's back


Apparently Google "can't find" my account.

I was so scared for a while, then I found that all of my family members have the same issue. Then checked HN and breathed a sigh of relief :)


A good explainer video here: https://www.youtube.com/watch?v=5p8wTOr8AbU


YouTube is working in India. But can not log in to any Google Accounts. GMAIL and Drive not accessible via web. Drive sync is working though. Appstatus still showing green.


The cool thing is that newpipe is working while youtube is down.


Hey, are you updating newpipe from F-Droid? I do, and it stopped working for me weeks ago.


Strangely everything seems to be little faster in incognito mode


Now we'll have a bit of outrage, talking about how dangerous it is to have the whole world impacted by one company's processes and forget about it in 2 days


HN is faster and more accurate than status pages of major service providers themselves. Special thanks to people working at those service providers and posting here!


Waiting for the post-mortem to see it was all because someone forgot to renew an SSL cert that nobody knew was a single point of failure for literally everything.

:D


Seems that Google has no budget to pay their GCP instances


Google Search seemed to be also down: https://serpapi.com/status


Time to mention the alternative, a decentralized internet, e.g. https://zeronet.io


How much do you think Google can learn from this incident's postmortem? Writing down all the internal details probably amounts to a small book.


Gmail is currently returning "550 5.1.1 The email account that you tried to reach does not exist" for many valid gmail addresses.


In inkognito window (without Adblock) it works for me!


Google page is right: servers are up and running - but not reachable. Could be an infrastructure attack (DNS?) or someone just broke the cable :)


I wonder why services like Uber (which rely on Google Maps for everything are still working, with things like routing being completely okay).


Yeah weird it’s all green. I just noticed things down as well. Came to HN to see how quickly people posted. This is already #1 at 6:55 am EST


Yeah, that status page is really frustrating. Didn't anyone consider building it to REALLY check if services were working? not just returning a 200 Status... but actually DOING things?


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: