Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: YouTube down?
575 points by LiweiZ on Oct 17, 2018 | hide | past | favorite | 468 comments

When YouTube comes back up again, I can recommend Tom Scott with "Single Point of Failure: The (Fictional) Day Google Forgot To Check Passwords"


Yes! This video was what caused me to reconsider my use of Google services seriously.

I was wondering how to react if this happened to me while watching the video. If, like me, you use Google with your email on your own domain (eg. gsuite), you could, in an emergency, change your MX records to something non-google.

Worth noting that "In an emergency" also relies on your MX records' TTL being set to a low value

And the DNS not being managed by Google and the domain not being bought through Google.

Wow, thanks for the recommendation. Harrowing.

It's back up.

it's black!



> Clicks on link.

> 0 peers.

> Refreshes YouTube.

> 500.

> Realize the internet is a distraction from what is really fulfilling and meaningful in life.

> YouTube's back up.

> Watch "Micheal Jackson's 'Bad' but Every Time he says 'bad' the Music Speeds Up".

Only getting like 1 peer per video and <150kb download.

I had the same problem, but it was because one of my privacy extensions had disabled WebRTC.

That de-Google-ify video I linked has 26 peers currently.

Everything on there looks like a scam or those 5-minute lifehack compilations where you lose a finger if you tried them yourself.

Opened it up, saw a foot fetish video on the front page.

It's like old Youtube then.

Or like random network shares on the University dorm network.

Not until it learns your specific for fetish them shows your kids on NewPipe.

4 peers with about 320KB download

Download finished, but I only uploaded 6MB total.

Too bad because I've got a symmetric up and down.

Oh wow-- just went to the "Trending" page and it's got a bunch of thumbnails of stuff like Stallman speech videos next to thumbnails of pornography. The Stallman/Porn ratio is about 2:9. (Well, to be fair there's also Peertube related videos.)

So in keeping with the history of the internet, porn seems to be the main driver here. But then their UI solution to keep newcomers from being offended is to blur out the porn thumbnails. So either:

a) the newcomer-- like me-- isn't after porn, and the majority of the screen real estate is wasted, or

b) the newcomer or return visitor is there for porn, the thumbnails they want to view are blurred out, and Stallman's face is eating up valuable porn thumbnail real estate.

Modest UX suggestion: follow the history of nearly the entire internet and separate out your porn from your everything else.

Medium sized regional AS here: I am looking at a fairly large drop in IX traffic charts for our ports that face the IX, updated every 60s, which directly corresponds in time with the beginning of the Youtube outage. (We are not big enough to have a direct, dedicated peering session with the Google/Youtube AS).

At any given time of day 4pm-11pm a huge percent of our traffic is Youtube (or netflix, or amazon video, or hulu, or similar).

Is any of this info publicly available anywhere?

IX charts are public in Chile by law. Everything looks well except the Telefonica (TIWS) international trunk [1]. It started dropping all traffic at the same time.

[1] http://pit.grupogtd.com/default.php?id=1

TIWS is back online. I'd presume they are draining most traffic to prevent overload according to the public SRE practices. I theorize it should take a few minutes to get everyone back in.

Que bacan weon! Estoy en viña ahorita. Sabes por que Movistar es muy lento com LTE aca?

Are you on prepaid? Virtually all Chilean prepaid carriers are deathly slow, especially outside of Santiago. I had barely any service in Algarrobo on Claro, but postpaid plans ran just fine, especially Claro, even in crazy places like Farellones :)

For what it's worth, I hated every second of my life in vina and that stupid town to the north of it and also Valparaiso. God, bring me back to La Parva or Embalse El Yeso. Some of the most incredible, untouched mystical land there!

I’m currently roaming on a post paid US plan. When I arrived in Chile I had 3G on Claro. Then it switched me to LTE on Movistar. 3G on Claro was more reliable.

Interesting to know about pre vs post paid. Will keep it in mind for longer trips when I normally buy a local SIM.

We just spent the last week in Torres del Paine backpacking around ... what an amazing place! Going to spend a couple days in viña and then headed to Conce to see some old friends. I’ve not been to La Parva but will add it to the list!

Enjoy, my friend! If you have a car, the mountains to the East of Santiago are absolutely awesome. Everything is safe and cool there, nothing worse than Californian tier driving. There's even free wifi at La Parva, and a ski lift to I think 12,000 feet in elevation if you dont want to walk haha.

Enjoy the city! I just had really bad experiences there. It's probably not the norm but I just hate petty thieves and the grime when I'm paying foreigner prices. I was based in Las Condes for 3 months and really loved it.

I think I still have my little $7 claro sim card lying around here somewhere :)))

siempre pillas un chileno en los comentarios

I just see a tiny blip (or 2) on Toronto’s IX, but that’s aggregate, not port by port:


Doesn't really show much of a change right now, nor does the aggregate chart for the SIX in Seattle.

The largest downstream/eyeball networks that take Youtube traffic do it via PNIs with them so you won't see that in an IX fabric traffic chart. For example the Google/Youtube AS exchanges traffic with Comcast directly by their own dedicated ports, no IX involved.

Is this not indicative of the drop of traffic in Australia?


Indeed it's not showing the Youtube issue for the reason you mentioned. Still relevant to the question above :)

there are a number of IXes that publish aggregate traffic charts, updated reasonably frequently, a lot fewer that publish traffic charts for individual ports or peers. If you want granular data you have to know somebody who runs the core BGP stuff for a reasonably sized ISP, and you can ask them for redacted copies of charts.

Here's for Portugal, we need around 85gbps at peak.


A lot of them have stuff like total traffic over time under the statistics link.


Just one I found quickly, the UK results are t interesting but check out LINX NoVA https://portal.linx.net/

What is an AS

From the wikipedia article: https://en.wikipedia.org/wiki/Autonomous_system_(Internet)

    ...an autonomous system *(AS)* is a collection of
    connected Internet Protocol *(IP)* routing prefixes
    under the control of one or more network operators on
    behalf of a single administrative entity or domain that
    presents a common, clearly defined routing policy to the

I only wish I could currently watch a video link to that instead...

I guess it goes to show how much we take connectivity and uptime for granted these days.

It saddens me that the link you posted had to leak the OS you're using... (the fact that it happens)

Or maybe I'm using an extension like this, and am pretending to be a Ubuntu+Firefox user so that they get their commission money from Google --> Mozilla, indirectly funding them.


my nerd-profiling radar had you as a network engineer = it's probably ubuntu ;-)

Sounds like cheating.

I think clients are allowed to send whatever user agent they want - it's essentially a preference for how they would like the website rendered for them.

Also I believe it's technically a violation of the HTTP spec to serve different GET requests to different user agents.

> I believe it's technically a violation of the HTTP spec to serve different GET requests to different user agents.

Is this actually true?

For what it's worth, Google serves a completely different webpage to IE 5 (try it, they still support IE 5/6/etc) as opposed to a modern browser.

That's just presentation and layout though. Am I really not allowed to send different document content to different UAs?

Arent those two statements contradicting?

If it would be a violation to change things based on user agent, but the purpose of the user agent to specify how you want things changed, then...

if it was purely about different rendering of the exact same content, then wouldn’t it be the browsers responsibility (and thus UA is unecessary)

I have to imagine browsers are allowed to extend the spec, and UAs are either in theory or in practice a way to communicate those extensions; but because site operators were only specifically designating content for the major browser(s), any compliant but unlisted browser would never get that content.. and suddenly everyone was calling themselves netscape

Autonomous System

AS? IX? Not everyone knows your insider lingo.

But seriously, reading how the internet works is pretty amazing.

This is hacker news, I think someone can be forgiven for using hacker lingo.

See: How I feel every time someone talks about using some six week old framework to program in some 3 month old extension to some 6 month old language (except, this lingo has been around for decades).

It's ISP / Network Engineer / Bit-shifter Lingo.

AS - Autonomous System (Number, sometimes called ASN). What big networks like ISPs use to identify themselves and route to each other, usually via BGP.

It's a fancy way of saying the parent runs a "mid-sized" network big enough to have an AS, likely a mid-sized regional ISP, or part of a larger ISP.

IX -- Internet Exchange, similar to IBX or DC. Essentially the building where there are routers and switches that connect the networks to each other.

Basically parent is saying that they're an ISP and their systems in the IX that route to Youtube saw a dip in traffic, implying a network event of some sort on the Youtube side.


I'm guessing it stands for Internet eXchange?

Internet Exchange.

Do they encrypt the server side error stack trace?

500 Internal Server Error Sorry, something went wrong.

A team of highly trained monkeys has been dispatched to deal with this situation.

If you see them, send them this information as text (screenshots frighten them): APkpgMWbWQ3LvimPoFynDB0W8VeUJ9ECiPcDCm8L0Qiku1I2TbAShWp- taKn-AzOGigwq0sU4oe9mbWb2Bwv4BK37C5xOAL7qm11fHn4L0swqhLk wbcnyKH2HM3AQNf-ucVsolyigJTNKA2SSNUMVZnDPmfsFH7ecKkpQmNi VGWhtXypv0zJyz9d_mpkgMoONtIrPUA4imxK-gNnE-_WQWQZNJm0CTae slJVC-TYgnvOZ9AYp6nodeUNpCoGspWaJVXn_ZSxy-71oGdlkCqWs6AY 2wmIEKe8eeAMqwkTHZNHkbAaH-fxWE_WDPuG-q7AFbOz8jZCFD06MYgf obFUSaH6B7PUdBFwVvjEaTD34J8PVhZTIJziRK-9-wSHOI6Vwf1lTuFe X0m52abRMW1VJaZB3taHK09kFT8Lv546OPhsL0Bn70UIs2durkAAYe4Z ...

Yes, they do. Pretty clever idea, I think.

Yes I agree, encrypting the stack trace enable them to share sensitive server side error codes to the client which in return can be easily reported back without having to explain anything or try to decode cryptic custom error messages.

This is fantastic!

I actually thought it was a bit jarring. Mostly because I don't know what personal information it might be collecting and having me send them.

Well, it's information collected at the server side. So if they really wanted they could just store the data and give you a code, but that would require more storage on their end.

(sorry for the sarcasm but...)

Oh no! YouTube needs to store a few kb of text! How will they have enough space?

Well, it obviously only happens when something goes wrong. Seems valuable to keep the complexity and external dependencies of your error handing to a minimum.

Maybe its + a placebo since already stored in storage system + ability to see an outage maybe dropped in the filters cause it was assumed its not user-facing and so assumed lesser severity + assurance incase storage failed due to server-storage network unreachable

Also probably a unique hash too

I too would like to know more about this such as is it encrypted by any know method (ex: pgp)? It's rather brilliant if it's an encrypted stack trace or similar. I imagine it might even contain info about who the user is as well (if logged in or tracked via cookie) to help them debug it more.

Why does this look like a problem for https://sentry.io

Thats prob base64 and your IP might be in the decoded string.

Messing around with it for a minute it seems it's a modified base64 (+ and / are replaced by - and _) with a space every 56 characters. I couldn't get any permutation where a-z, A-Z, 0-9, and -_/_- were consistent so it's either encrypted or one of 1,268,869,321,858,841,641,034,33,389,335,161,480,802,865,516,174,545,192,198,801,894,375,214,704,230,400,000,000,000,000 permutations of the encoding alphabet (or I messed up the decoding).

The modified base64 is probably just URL encoding https://en.wikipedia.org/wiki/Base64#URL_applications

I'm imagining millions of kids in restaurants crying out because their "iPads aren't working"

My son is having "a moment"

Mine is sleeping, I was ready to enjoy the new justforfunc episode... I'm the one that is crying...

Mine too. He’s so confused and mournful.


I fear something terrible has happened.

As if a million minds were screaming in terror from the sudden free thinking time.

I have just come to realise that there is no point in life without purpose.

Edit: It's back!

My wife says: "they probably ran out of space on their server"

Probably because the logs directory filled up...

Just send her this (obligatory)


Oh she knows

Insiders everywhere.

Ahhh, not enough monitoring and/or non-existent "disk space usage" alerting? Been there.

I have never seen Youtube not work. Ever. This is a special moment. Glad you are all here to share it with me :)

In the early days it went down all the time. Same with all of their competitors. The YouTube engineers had a Python+Mysql+lighttpd stack and even had a fork of lighttp for faster performance. They spent every day eliminating bottlenecks until it was stable and then Google bought them.

The original talk about their stack was on Google Videos back in the day. Now sure if it got moved to the YouBoobs.

I totally forgot google video was a thing.

Heh so did i! That was a weird feeling...

It's still Python+MySQL for what I can tell. But also has parts in Go (see i.e. their work on vitess to scale MySQL access https://vitess.io/ )

They now run their python code using a transpiler into Go with a Go runtime and Go libraries: https://github.com/google/grumpy

This is not true. Grumpy exists, but it is mostly unmaintained now, and certainly isn't capable of transpiling all of youtube.

Reminds me about Facebook, who transpiled PHP to C++. Before creating their HHVM interpreter and their hack language.

I don't remember the exact timeline of YouTube's growth, but I think during the time period when it was the most popular site for pirated TV shows, it was really stable and has been ever since.

I am proud to say that in 2008 my country knocked this puny website off the interwebs in a bid to censor it within the country. They accidentally hijacked the global IP addresses[0]

I am also proud to say that my country left the domain for their original Internet Exchange Point[1] unregistered. Guess who scooped it up?

PS. My country is Pakistan.

[0]: https://www.cnet.com/news/how-pakistan-knocked-youtube-offli...

[1]: https://pie.net.pk

I was going to link to the video of a Dyn employee (Todd whatshisname) doing a parody of American Pie about this -- "The Day The Youtube Died" -- but the only link I have to the video is on Youtube. :/

Here are the lyrics [0] instead, you'll just have to sing it to yourself.

[0]: https://dyn.com/blog/the-day-the-youtube-died-1/

> Guess who scooped it up?

This is the best domain squatting I've seen.

I have a new hero.

Great work sir!

Well everyone here seems mighty proud of censorship. Its too bad when Google themselves want to do it in China everyone loses their shit!

I guess the sarcasm in my original post wasn't obvious enough. But, yes، that was sarcasm۔

I'm also suing the Pakistani Federal Government to hand over a list of censored internet resources as part of a FOIA request. The hearing is tomorrow [0], wish me luck!

[0]: https://link.medium.com/RkevqMKY4Q

It actually makes me appreciate how reliable youtube has always been. I never even consider a youtube link might not work perfectly when I click one.

Makes a good case for self-hosting videos as a backup

Or a good case for keeping one or two physical paper books on a bookshelf, and maybe a candle and lighter.

I don't think yt being down is a signal for the end of times

Internet sites go down all the time, though usually not for long or en masse.

Never seen Youtube go down on their end. But I have seen it stop working on dozens of workstations at a news site simultaneously when they dropped ie7 support.

Magically this enabled the outsourced infrastructure provider to roll out a new default browser quite a bit faster than they previously claimed they can...

Google sites are probably the most reliable, I don't think I've ever seen google / gmail / yt going down.

I think a Google search outage would be a spectacular train wreck.

Imagine if Google's search was compromised in such a way as to be has bad as the Sony PSN network hack of ..2011? Imagine not having Google search for several weeks. Bing/DuckDuckGo/Yahoo would struggle just to keep up and people would immediately realize their dependence on the big G.

> I think a Google search outage would be a spectacular train wreck.

I'd argue that a youtube outage is worse. If google search is out, you can just use duckduckgo.

If you want to watch a video on youtube, you're SOL.

This is speaking from experience with blacklisting both google.com and youtube.com in my hosts file. I was doing fine with blacklisting google, but couldn't handle blacklisting youtube.

Try it yourself if you don't believe me.

Youtube is mostly idle entertainment though so an hour of no youtube is not the end of the world but people rely on search engines to get stuff done.

It depends how you define "going down." Here's an old screenshot of mine of when Google Search failed spectacularly by marking every single link as malicious and therefore not taking the user to that site when you click the link [1] [2]. Perhaps that might qualify as going down, although clearly it didn't go down at the network level.

It remained broken (in that way) for quite some time (maybe longer than an hour.) But, happening in 2009, I can't accurately recall.

Here is another scenario when Google Search failed to successfully serve my query [3].

And an early time when YouTube went down [4].

[1] https://i.imgur.com/X05BCpI.png [2] https://i.imgur.com/CaDh1Y6.png [3] https://i.imgur.com/VZI4Yew.png [4] https://i.imgur.com/bPTp4bg.png

I have seen gmail go down many times years ago but I haven't used the web interface for years so I don't know how often it happens nowadays...

As someone who ran a channel with a few thousand subscribers for a year or two, I’ve seen my fair share of 500 error pages on YouTube. Granted, it was always isolated and may recover after a few refreshes (IIRC), but still.

I agree, but a special moment because these days there are people like you that never had to see "big" sites crashing or dial-up speeds.

Going on over an hour now. Someone goofed big time.

Interestingly, my youtube-dl archiving script is still chugging along with only a handful of elevated retry rates. So whatever the problem is, it's probably not at the API or CDN layers.

The static contents are still available so it must be problem in their application. Strange that is down world wide, not a region. But, then Google likes to make things as a whole, ie. Spanner database.

I was able to finish watching a video on my phone, and it definitely hadn't buffered all of it yet, so some servers must still be able to handle requests for videos.

YT generates resource links with ~couple of hours expiry dates, once generated page/app will keep reusing that link.

isn't it also strange that the shadow of their ui components are still loading?

They're generic templates to make you think the page is loading faster than it is. They don't represent the content that eventually (normally) shows up.

Not really, this is done client-side.

Both m.youtube.com & the mobile app are down.

Along with their live TV product, tv.youtube.com

SMTube (aka http://tonvid.com/) failed[0] due to "(500) Backend Error"...

[0] http://i.imgur.com/PRNMtwY.png

What are you archiving?

Ha! My time to shine.. I’ve got somewhere around 80TB of YouTube content on my NAS including thousands of videos that aren’t available anymore on YouTube or anywhere else online. The joy of hoarding data..

I hope you've got plans for what happens to that content when you're not around any more. With how ephemeral a lot of content is on YouTube there's sure to be valuable videos on people's drives.

I actually do have plans for that in place!

I’ve got two identical servers, one will go to The Internet Archive and the other one should go to some German archive but I haven’t decided which one, still looking around.

I have a script in place that deletes all private data off of those servers before and instructed two friends and some family members on how to proceed.

Of course I don’t just hoard YouTube videos.. There’s a wide variety of data on there.

About lost content; I just recently started looking up all the YouTube videos I own to see how many aren’t available anymore on YouTube (A LOT). Still trying to decide if I should re-up them with some burner accounts or leave it be..

Talk to Jason Scott at archive.org. Those taken down videos could be uploaded now for safekeeping

I just have to figure out how to check which ones already are on Archive. But I’ll absolutely do that!

I know a kevingrahl from gradeschool in leipzig. Interesting comment, but thats the only thing I have to say.

Are you a ginger by any chance? ;)

Nope but that could be someone from my extended family.

I've had a lot of luck with finding YouTube videos that disappeared via archive.org

Unfortunately YouTube also deletes the title when a video is being removed. I regularly have a couple of "deleted video" entries in my favorites list and I don't have the slightest idea what these videos were about.

Glad you mentioned that!

I just recently checked my collection to see which videos aren’t available anymore online. I should probably cross check the results with Archive to see if they have them available!

I would love to have videos verified as no longer available.

As soon as I figure out how to check them I’ll be sure to send them over to you!

Long shot here, do you have any videos from The Portland Art Museum channel? https://www.youtube.com/channel/UCXneDVRq1lLde3oREF_cFyA # they deleted (or made private) one of the talks by Richard Mosse and i'd like to see it again as it was a great talk IIRC.

On topic - I've also made a habit of downloading any videos/channels that are super interesting but I feel may also be ephemeral. Some of that ephemerality is often due to copyright claims, performances of classical music by amateurs and such.

80TB is a lot just to hoard data... I don't think I even like so many vids.

Well.. I’ve got to have a selection for every occasion!

But it’s not like I’ve watched every video I have, often I just download entire channels if I deem the content to be worthy for archiving.

I bet we'll have tons of makeup channels then, right ?

I’m a bearded, bald guy..

Not really my thing ;)

So, sharing is shining. Where can we see your warez?

Send me some drives!

Your ISP must not have a cap ... or a very reasonable one.

I don’t even have regular WiFi.. I’m mostly using my phones LTE via hotspot because it’s much faster than anything I can get (in a big city that is) here. I don’t have a cap for that.

Every cell provider (in the US) has a cap (or cap and throttle system), especially for hotspots - I'm perplexed.

T-Mobile (my provider) has a plan with unlimited LTE data and when someone in the EU advertises something as unlimited is really has to be unlimited. I managed to download as much as 2TB within 24 hours with no complains.

I pay 80€ per month (~$92) which I think is fair. I get 25GB roaming data per month to use within the EU.

But most mobile plans do throttle your speed to something unusable after you’ve used up your included high speed volume here too. This can range anywhere from a couple MB up to 50GB/month afaik.

This is going to be one good postmortem.

Mistakes were made... anyways. here's Wonderwall.

I sure hope they live stream it.

On Twitch of course

You're joking, but I like the idea of watching a live postmortem with Twitch chat.

Me too, but I imagine a tsunami of 4head spam.

No postmortem yet?

Getting errors too. First time ever tho so props to the YouTube team for being pretty damn good most of the time.

Are we going to start a betting pool as to why?

I put $20 on DDoS, but I'm hedging $5 that someone spilled their drink on something important and quickly got out of there before anyone noticed.

Wouldn't DDoS be incredibly hard to pull off given YouTubes vast resources? I mean they generate so much traffic all the time and serve so many requests a second I don't know if botnets of that size exist. Anyone here more knowledgeable?

Maybe it's a brain teaser - if you can answer how to DDoS Youtube, they'll call you up for an interview.

I could tell you a few ways, but would never work for them out of principle. They can find their own bugs.

Yeah i feel like a DDoS that big would be afecting other google services right now...

A few hundred AI learning instances were started to train on youtube videos to "understand" humanity. They have become self aware and started to consume double amount of CPU/GPU/Disk/RAM/Network resource every few seconds - a skynet/super nova moment.

Would it involve hitting whoever/whatever handles their DNS like that one attack about a year ago?

FYI: I legit am retarded to such things and my networking knowledge is...meh. So please correct me.

DNS is resolving because people can get to the site. Because of the distributed nature of YouTube's resources (at DNS as well as other layers), it's very unlikely to be DDoS related. Static content loads, but video content doesn't. Seems like an application error, or a problem with media streaming infrastructure.

I suppose if you had something new and incredibly powerful, this would be a great way to show it off.

Still more likely that YT just made an oopsie.

> I put $20 on DDoS

I'd take you on that, except I have no anonymized payment account. Otherwise, I'd bet $100 that it's not.

Only Google has the scale to DDoS Google.

The DDoS came from inside the Google servers!

Now see, that I'd believe.

Hey! Conspiracy time!

Maybe it's an inner splinter rogue group within Google that's been secretly hating on a Youtube team and decided to get the ultimate payback for like... not inviting them to a lan party...or something...aliens ate all the cheesey poofs.

It's crab people for sure.

I'm not saying it's aliens, but...

It's the sunspot thing. They took down the Hubble, that other telescope, that random lab, and now the frog people Alex Jones warned us about are coming to inject you with the gay and ship you on a raft made out of Michael Moore to Cuba.

It's the sunspot thing. They took down the Hubble, that other telescope, that random lab, and now the frog people Alex Jones warned us about are coming to inject you with the gay and ship you on a raft made out of Michael Moore to Cuba.

All to conceal the real truth about what happened on 9/11 and how it connects to the plot to obfuscate what really happened at Pearl Harbor and how the Australian mafia actually had JFK assassinated to cover up the truth about Marilyn Monroe, Tupac, and Charles Proteus Steinmetz. A clever scheme, no doubt orchestrated by the evil consortium of George Westinghouse, Thomas Edison and Guglielmo Marconi. Hopefully Elon Musk, Robert Downey Jr. and Nikola Tesla will be able to bring the truth to light!

You forgot George Soros

Conspiracy theory time.

Maybe. Netflix is on the same order of magnitude. China has many websites whose volume is not much smaller than google.

I bet it’s a core network issue since it went down globally at the same time.. they deploy new binaries slowly so that shouldn’t cause it. Maybe some config was changed without testing?

Agreed. I'd suspect it'll be something similar to Google's 2016 GCP outage:


(I'm thinking something like router firmware or BGP).

A. You're not being fun.

2. Good point.

III. One thing I did notice. I had some other videos in other tabs. They still loaded through. I could skip around and it streamed in video like nothing was wrong. When I refresh, bust.

Quatre: Five......

Is it down globally? I've has some EU people on Mastodon say it's still up for them.

i think that if this is the case, it's an fast and easy fix, but they haven't solved yet so i think this is not the issue

My vote: Bad configuration got pushed.

I thought they had all sorts of crazy canary deployment processes that will automatically stop/rollback the deployment if the failure metrics start increasing?

There was a GCP Global network outage maybe a year or two ago, where their canary of a canary (Yo dawg, I heard you like bugs, so I I put a but inside of your bug) caused a cascading network failure.

It was a bad config push -- brownouts fighting with brown outs. Consistency issues. I bet this is happening.

Ran out of canaries?

Well, could be an unforeseeable-wack-crazy-bug. Can't plan for every possible failure.

That's very true, there's always a bug somewhere lurking, even if you're Google.

The janitor needed that socket to plug the vacuum

Experimental Roomba built up a static charge and fried something important.

DNS, it is always DNS.

Metadata querying I imagine. The search was working, buffered videos and main site were loading, just not displaying videos. (so not network)

Fat fingered a script would be my guess. Not the first time that caused a 'big4' site to go down.

Somebody probably accidentally powered down the Web:


I'm betting it has to do with video indexation.

I was watching a long video just fine, then I searched for something else and clicked on one result. Error message.

Joe, the only YouTube support person in the office this evening, accidentally hit the pause button on the YouTube service in his admin console before stepping out for a long bathroom break. Hey, it could happen...

I bet on Google Global Cache problem.

YouTube was down for me through my residential ISP (which no doubt has internal caching via GGC) but through VPN was fine.

The Saudis dont like all the attention?

My bet is data loss. Some engineer deleted MySQL table accidentally and they found out their backups are not working.

$20 says it was Greg. Again!

I'll take your $20 bet and raise you to $100. Not a DDoS.

NSA was installing a wire tap and sliced some fiber.

That would not cause a global outage. YouTube has caches in all the major ISPs.

maybe nsa was installing a whole bunch of wire taps?

With the help of aliens since they have a better vantage point of where all the cables are... because, you know, the Earth is fla-... nope. Can't do it. I can't go that far. I can only say "so much" stupid. That's too far.

My bet is rm -f instead of cp -R.

I was setting up a new network and thought it was my settings! I've been driving myself mad for the last 45 minutes thinking that I had fudged up somewhere! Thank goodness for this post

It was working fine for me until the minute I hit Upload on a video. I was entirely convinced I'd had my account auto-banned or something, and was very confused/frustrated. Couldn't find any mention of it anywhere at that point, then this thread popped up.

It's pretty telling that YouTube is so reliable, and Google is so horrible to their users, that my first instinct was that I got banned for no reason by some bot, rather than the site was broken.

This is what happens when one is the user be not the customer

First thing I see on their Twitter page are fake verified Youtube accounts perpetuating some Crypto scam: https://i.imgur.com/PkEbjH5.png

Typical scam. Problem is verified accounts can change names so scammers steal details for random verified accounts and then use them to pretend to be high profile accounts.

From Ops to Ops, much respect YouTube/Goog and a tremendous pat on the back to this even being such a big deal.

I'll happily, but very-boredly, wait until you're done. :)

Kudos to the Ops team indeed, can't wait to read the PM report.

New Boston Dynamics dancing robot video launched today is probably to blame. :)

(and if you haven't seen it now you have something to look forward to when Youtube comes back)

Ghost of the dancing baby.

Woke up randomly at 3am (utc+2). Opened HN. saw this ask HN

Think I felt a great disturbance on the net and woke up, as if millions of voices suddenly cried out in terror and were suddenly presented with a black screen on their YouTubes. I fear something terrible has happened

This exact same thing happened with me. I woke up at unusual time.

Its working now though.

Very glad to see it's not just me. I was about to think they had a radical new redesign again with everything AJAX'd and decided to make it totally unusable in anything but the very very latest DRM-encumbered browser(s).

Well...this is a good excuse for me to setup a PeerTube mirror for my YouTube channel.

Let's speculate, how much money would you guys guess is being lost every minute of downtime?

> Estimates for YouTube's annual revenue, nearly all of which still comes from ads, vary a fair amount. But many of the estimates are now above $10 billion. At different points, Bank of America and Mizuho forecast that YouTube would post 2017 revenue of $13 billion and $12 billion, respectively. And in February, Baird's Colin Sebastian estimated YouTube is doing around $15 billion in annual sales. [0]

I think that works out to a bit more than $28k/minute.

[0] https://www.thestreet.com/investing/youtube-might-be-worth-o...

A metric @!#$ton? :D

Unrelated, but on a recent network outage (https://status.cloud.google.com/incident/cloud-networking/18...):

  The incident occurred while Google's network operations team was replacing
  the routers that link us-central1-c to Google's backbone that connects to
  the public internet. Google engineers paused the router replacement process
  after determining that additional cabling would be required to complete the
  process and decided to start a rollback operation. The rollout and rollback
  operations utilized a version of workflow that was only compatible with the
  newer routers. Specifically, rollback was not supported on the older routers.
And the postmortem action item is:

  Fix the automated workflows for router replacements to ensure the correct
  version of workflows are utilized for both types of routers.
The action items should have been "1) make this work for these two routers, and 2) make sure no platforms ever get left out again".

This shouldn't have happened, because it should be standard practice to test both upgrade and rollback on all your gear. Network gear vendors do this as standard practice before they ship new gear with upgrade instructions. Google can throw together end-to-end automated tests of upgrades/rollbacks and refuse to perform maintenance until tests pass.

The bigger postmortem question should be, why was the change allowed at all if the platform didn't support rollback? Additional action item: "3) don't allow changes if the platforms don't support and have successful rollback tests".

Now, did they need to test rollback? Maybe they don't mind portions of CloudSQL, Spanner, Storage, BigTable, and AppEngine being down for 41 minutes in one zone. But if they're not even testing rollback for BGP changes, what else aren't they testing?

...Also, lol, they realized in the middle of an upgrade that they didn't have enough network cable? Maybe add an extra action item: "4) count how much network cable you have before you start replacing core routers"

Spanner and Storage at least are region-wide, so a single AZ going down shouldn’t affect customers on those products.

Some analysts predict YouTube to bring in $15 billion in ad revenue in 2018. If so, they are losing around $28,000 for every minute they are down.

Could be more (people stop watching and go outside) or less (people go back later to watch what they would've watched anyway, and free advertising for everyone talking about it) depending on several things.

I think you meant $280,000?

-1 on my mental math, +1 on duckduckgo's UI for arithmetic.

nah, I think 28,000 is right

Interestingly, this article [1] mentions a 2008 YouTube outage caused by a BGP misconfig during an attempt to ban YouTube in Pakistan. Included is a screenshot similar to what we've seeing with this outage. Possibly related?

[1] https://blog.cdemi.io/beginners-guide-to-understanding-bgp/

Can Netflix handle the extra load? Haha.

And Twitch.

I spent the evening binging on Agents of SHIELD, so I assume yes. ;)

A lot of different sites are down right now. I wonder what's going on.

what are some others?

Ironically, outage.report, a website for reporting downed web services is also down.

So signs point to it not just being youtube, a number of websites seem to be having issues. May be a provider is down?

Maybe outage.report is down because so many people are hitting it after YouTube going down...

PornHub is down for me.

it just got serious

Not for me

Quora is down for me.

Quota still seems to be up for me, South West England.

Seems fine for me.

Same to me.

discourse is down for me.

GCS issues maybe?

GCP [1] and G Suite [2] appear to be green.

[1] - https://status.cloud.google.com/

[2] - https://www.google.com/appsstatus

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact