
GitHub Outage - philip1209
https://github.com/#2016-01-27
======
ikawe
Sorry, I think I caused this. =[

    
    
        bower jquery#1.11.3                       not-cached git://github.com/jquery/jquery-dist.git#1.11.3
        bower jquery#1.11.3                          resolve git://github.com/jquery/jquery-dist.git#1.11.3
        bower foundation#~5.5.2                       cached git://github.com/zurb/bower-foundation.git#5.5.3
        bower foundation#~5.5.2                     validate 5.5.3 against git://github.com/zurb/bower-foundation.git#~5.5.2
        bower ember#^2.3.0                           ECMDERR Failed to execute "git ls-remote --tags --heads git://github.com/components/ember.git", exit code of #128 
        fatal: remote error:
    

Mid bower install. Rly srry guys!!! =[

~~~
chuhnk
I once ran rm -rf in the production mysql data directory.

Shit happens.

~~~
zymhan
I did that too. Destroyed out Zabbix database. Neither that Zabbix server, nor
the other one monitoring the server I destroyed, could alert us that anything
had gone wrong for over an hour. I finally realized it when I couldn't
login...

I was able to painstakingly rebuild the server after 9 hours without anyone
noticing. To this day one of my biggest fuck ups and prouder accomplishments.

~~~
postila
Are you saying that once you break something, you should break your monitoring
as well, but do it very quickly since it may be too late? :)

~~~
zymhan
Yeah, if you're going to break your monitoring solution, you'd better shoot it
in the head and vaporize the body.

------
fpgaminer
Relavant: GitTorrent: A Decentralized GitHub
([http://blog.printf.net/articles/2015/05/29/announcing-
gittor...](http://blog.printf.net/articles/2015/05/29/announcing-gittorrent-a-
decentralized-github/))

The repo is at ...
[https://github.com/cjb/GitTorrent](https://github.com/cjb/GitTorrent), so
just clone that and ... oh ...

~~~
aylmao
From the site:

    
    
      First we connect to GitHub to find out what the latest 
      revision for this repository is, so that we know what we
      want to get. GitHub tells us it’s 5fbfea8de... Then we 
      go out to the GitTorrent network.
    

So yeah, wouldn't have saved you.

~~~
fpgaminer
The article continues and covers solutions to that.

------
Perceptes
Every time this happens people make clever remarks about how Git is
distributed but we're all depending on GitHub for so much that we defeat the
purpose. But once GitHub comes back up, everyone just gets back to work,
trusting and relying on it as much as ever. Eventually it goes down again, and
we come back to complain. Convenience is the only thing that we seem to value.
(I'm no different, which makes my comment completely hypocritical.)

~~~
niutech
Companies/people don't learn on their mistakes. Almost everything is on GitHub
nowadays. This makes it a SPOF even if Git itself is distributed. More
companies should host their projects on premises. There are good open source
alternatives to GitHub: Apache Allura, Fossil, GitBucket, GitLab, Phabricator,
and Redmine.

~~~
andytuba
... GitHub Enterprise ...

~~~
niutech
It's not open source and it's expensive.

------
fletchowns
If you listen closely you can hear the sound of continuous integration builds
around the world breaking

~~~
dennispi
As GitHub queues up webhooks, we're bracing for impact at buddybuild! :)

~~~
kdaigle
You shouldn't notice a webhook deluge because the site isn't generating
events. I'm watching our webhook services though and will let you know if that
changes.

~~~
dennispi
Hi Kyle!

It looks like webhooks are wedged.. no?

~~~
kdaigle
Everything should be A-OK now. If not, hit up github.com/contact :)

------
chewbacha
I felt a great disturbance in the Force, as if millions of voices suddenly
cried out in terror and were suddenly silenced. I fear something terrible has
happened.

~~~
edem
Nomen est omen Mr. Chewbacha?

------
tomchuk
I was going to stay up for a middle-of-the-night release (DB migrations,
bleh). Instead, posted
[https://status.github.com/](https://status.github.com/) into slack and am
packing up and going to the bar.

~~~
niutech
Can you imagine what would happen if the same happened to Slack? No
communication in thousands of companies.

------
mrdrozdov
This is new. Looks like github is still "working" when it's down.

    
    
      $ git push origin master
      Counting objects: 5, done.
      Delta compression using up to 8 threads.
      Compressing objects: 100% (5/5), done.
      Writing objects: 100% (5/5), 433 bytes | 0 bytes/s, done.
      Total 5 (delta 4), reused 0 (delta 0)
      remote: Unexpected system error after push was received.
      remote: These changes may not be reflected on github.com!
      remote: Your unique error code: 4fce1b2367b5304dd3761538b8fd0c23
      To git@github.com:myrepo/myrepo.git
         a62b7f1..e88431a  master -> master
      $ git push origin master
      Everything up-to-date
    

Note: Values are fake, but message is real.

~~~
uxp
The Git backend is different from the GitHub frontend, unsurprisingly.

~~~
mrdrozdov
I suppose not. Glad to see something like this still work:

    
    
      $ git clone git@github.com:influxdata/influxdb-ios.git
      Cloning into 'influxdb-ios'...
      remote: Counting objects: 10, done.
      remote: Compressing objects: 100% (8/8), done.
      remote: Total 10 (delta 1), reused 10 (delta 1), pack-reused 0
      Receiving objects: 100% (10/10), done.
      Resolving deltas: 100% (1/1), done.
      Checking connectivity... done.
    

Even though this doesn't:

    
    
      $ wget https://github.com/influxdata/influxdb-ios
      --2016-01-27 20:03:39--  https://github.com/influxdata/influxdb-ios
      Resolving github.com... 192.30.252.131
      Connecting to github.com|192.30.252.131|:443... connected.
      HTTP request sent, awaiting response... 503 Service Unavailable
      2016-01-27 20:03:39 ERROR 503: Service Unavailable.

~~~
shurcooL
You can also git clone over ssh. That's kinda equivalent to what the
git@github.com form does.

    
    
        $ git clone ssh://git@github.com/influxdata/influxdb-ios
        Cloning into 'influxdb-ios'...
        remote: Counting objects: 10, done.
        remote: Compressing objects: 100% (8/8), done.
        remote: Total 10 (delta 1), reused 10 (delta 1), pack-reused 0
        Receiving objects: 100% (10/10), done.
        Resolving deltas: 100% (1/1), done.
        Checking connectivity... done.

------
Analemma_
Developers: "I can't get any work done because GitHub is down!"

Linus Torvalds: [ _facepalm_ ]

~~~
mavdi
I honestly can't get much work done now, I've been looking for RabbitMQ auth
plugin sample code and every link I'm clicking now shows me a unicorn.

~~~
giancarlostoro
Google the examples, then use google cache

------
ntaylor
Thank God we're on Bitbucket! Right, guys? Anyone?

~~~
firloop
Regardless, most dependencies are on GitHub which breaks bower install for
most people. It's crazy how much infrastructure relies on this single point of
failure.

~~~
brudgers
Someone should make a distributed version control system.

~~~
adamsea
What would that look like? Can you describe it? It's not very useful to say
stuff without providing some sort of useful idea. You can't just say "someone
should write some sort of stupid content tracker, and give it some random
three-letter combination that is pronounceable," or whatever.

~~~
such_a_casual
Completely agree. But whoever makes it, they should make it free and open
source and designed to handle everything from small to very large projects
with speed and efficiency.

Additionally, it should be easy to learn and have a tiny footprint with
lightning fast performance. It should outclass SCM tools like Subversion, CVS,
Perforce, and ClearCase with features like cheap local branching, convenient
staging areas, and multiple workflows.

~~~
rodgerd
> it should be easy to learn

That would be a significant upgrade.

------
ChickeNES
"We're investigating a significant network disruption effecting all
[http://github.com](http://github.com) services." \-
[https://twitter.com/githubstatus/status/692508939792039936](https://twitter.com/githubstatus/status/692508939792039936)

~~~
nacs
> effecting all [http://github.com](http://github.com) services

Guess they're too busy working on the issue to notice the misuse of
"effecting" instead of "affecting."

~~~
WillAbides
Never too busy to correct grammar.

[http://i.imgur.com/iUbz1Lh.png](http://i.imgur.com/iUbz1Lh.png)

------
beat
Troubleshooting checklist:

1\. Commit fails

2\. Try again, see if it fails twice

3\. Check internet connectivity

4\. Try github in browser

5\. Try github in a different browser

6\. Go to HN to see if it's down for everyone

7\. Write snarky comment

8\. Go try again...

~~~
melvinmt
Why would your commit fail?

~~~
beat
First noticed it when making a wiki commit, actually. Edited the story
slightly for artistic license. :)

~~~
gnuvince
I think he meant that the commit should work fine, but the push would fail.

------
johnhenry
Luckily, git is a distributed control system and we all remembered to:

git remote add backup <bitbucket or gitlab url>

git push backup

~~~
voltagex_
I'd actually like a gitolite-like system that takes my pushes and replicates
them among Gitlab/Github/Bitbucket/repo.or.cz. I'm sure it's possible with
hooks, but every time I get around to looking into it, GitHub is back up

~~~
voltagex_
Now that GitHub is back up again, it looks like a cron version of
[https://help.github.com/articles/duplicating-a-
repository/](https://help.github.com/articles/duplicating-a-repository/) would
get me some of the way there.

------
dsl
Fortunately git is distributed and lets you work offline. [https://git-
scm.com/about/distributed](https://git-scm.com/about/distributed)

~~~
msie
I'm trying to read some webpages hosted on GitHub.

~~~
eoconnell
Try adding www instead of hitting the apex. If not github.io urls should still
work.

------
madmod
For the impatient. I would make a gist but...

    
    
      while true; do curl -s https://status.github.com/api/status.json | grep good && tput bel || echo -n .; sleep 1; done

~~~
smurpy
Here is a version which uses OSX's _say_ command for a spoken announcement of
" _github is up again_ ". It also checks once every minute instead of once a
second.

    
    
        while true; do curl -s https://status.github.com/api/status.json | egrep 'good|minor' && say -r 160 "github is up again" || echo -n .; sleep 60; done
    

Edit: accept a status of _minor_ as an indication of up-ness.

~~~
pramodliv1
say "github" does not get the pronunciation right. say "git hub" is the way to
go!

------
upbeatlinux
The status page update seems like some ironic machine rant "The status is
still red at the beginning of the day"
([http://take.ms/50Tox](http://take.ms/50Tox))

Hubot has assumed control...the singularity is upon us!!!

~~~
bobwaycott
Yeah, I am confused by the existence of a status report for a time 3.5hrs in
the future. Guess they're just trying to get ahead of the game and set some
realistic expectations.

~~~
rcthompson
Maybe it's GMT mistakenly labeled as PST?

~~~
bobwaycott
Weird. It was labeled EST for me.

------
bodecker
Status page seems to be back:
[https://status.github.com/](https://status.github.com/)

No word on twitter yet:
[https://twitter.com/githubstatus](https://twitter.com/githubstatus)

~~~
CUViper
Now investigating:
[https://twitter.com/githubstatus/status/692505376554618883](https://twitter.com/githubstatus/status/692505376554618883)

------
andhess
went to push, didn't work, tried the webpage, saw the outage, now I'm on HN

~~~
merqurio
same here

~~~
ljw1001
same

~~~
cyansmoker
sadly, same

~~~
andhess
ok this is actually really annoying now

~~~
why_advertise
What the `zsh` working with `fuck` think of your comment on OSX:

git:(master) ok this is actually really annoying now

zsh: command not found: ok

git:(master) fuck

look this is actually really annoying now [enter/↑/↓/ctrl+c]

look: is: No such file or directory

git:(master) fuck

> No fucks given

That's kind of perfect.

~~~
theperfect
Nice! And in this case the perfect is not the enemy of the good!

------
ikawe
And 100,000 people's work grinds to a halt.

------
amorphid
As a workaround, I can add all of you to my company's GitHub Enterprise
install. It's only ~$2,500/year per 10 users, so I can just expense it.

------
AYBABTME
Github is down. Post github.com on HackerNews, that'll help them. =P

~~~
iLoch
To be fair, GitHub has scaled to the point where traffic probably isn't their
concern.

------
philip1209
This is reminding me of last year's major Facebook outage. If I recall
correctly, that outage was a bug in service discovery that took down all data
centers (a CLI accepted a negative value when the Zookeeper variant treated it
as an unsigned int , then all service discovery went down). I feel like
service discovery is the biggest point of failure at large companies, and it
would explain why services across so many different domains and systems went
down.

~~~
peterkelly
Except that when Facebook is down, productivity goes _increases_ , and when
GitHub is down, productivity _decreases_ :)

------
Analemma_
_sigh_ Many moons ago when I was a starry-eyed lad just learning to use Git, I
remember all the cheerful sentences in the Git book like "Unlike SVN, with Git
you can work even when the server is down!"

The more things change, the more they stay the same.

~~~
AndyKelley
I'm still working over here. I mean I took a quick break to read HN. But no
productivity problems. Why can't you keep working?

------
softwarelimits
Fiddling around with several deployment and configuration management options I
just came to the conclusion that there is too much code between me and the
systems.

I would like to have one bash script that I could start in such worst case
scenarios that tests everything and tells me exactly what the problem is.

Just imagine everything is down and you can not trust your dashboards.

What would you put into such a script? What would you test for?

------
mordrax
Does this have something to do with the constant DoS attacks they've been
getting from a certain country for hosting certain open source projects that
defy the censorship authority of said certain country?

~~~
personjerry
Russia? China? Is it forbidden to state the country?

~~~
busterarm
You have been banned from /r/Pyongyang.

------
zwetan
exactly why I host all my repos on my own dedicated server with my own backup,
and then mirror/sync to github

and also why I do include my dependencies in my projects

having a build tool fetching dependencies dynamically on github or whatever is
not a convenience, it's a PITA

just sayin' even extremly reliable hosted services can fail, do own your
repositories, going back to write some code ;)

------
philip1209
Status page is not responding either:

[https://status.github.com/](https://status.github.com/)

~~~
fiveoak
Seemed to work for me, although it took a while to load and said "All systems
operational"

edit: Now says "19:32 Eastern Standard TimeWe're investigating connectivity
problems on github.com."

edit2: "19:47 Eastern Standard TimeWe're investigating a significant network
disruption affecting all github.com services."

~~~
veidr
Yeah me too, status page says its OK but the website look like this:

[https://www.dropbox.com/s/ap1nqogmqza2e12/Screenshot%202016-...](https://www.dropbox.com/s/ap1nqogmqza2e12/Screenshot%202016-01-28%2009.29.23.png?dl=0)

And:

    
    
        cd /[mason@IT-PC-MACPRO ~]$ cd Code/rollerball
        [mason@IT-PC-MACPRO rollerball (master)]$ git pull
        remote: Internal Server Error.
        remote: 
        fatal: unable to access 'https://masonmark@github.com/RobertReidInc/rollerball.git/': The requested URL returned error: 500
        [mason@IT-PC-MACPRO rollerball (master)]$

------
qwyxzy
Apparently they can see into the future and know for a fact that they'll still
be having issues in a few hours.

    
    
      January 28, 2016
      00:00 EST The status is still red at the beginning of the day
    
      January 27, 2016
      20:02 EST We're continuing to investigate a significant network disruption affecting all github.com services.

~~~
peterkelly
... or you're in a different timezone. It's 11am on the 28th where I am.

~~~
scoates
It says EST there. It's wrong. Looks like it's reporting times in UTC but
marking them EST (for me at least).

------
doppenhe
Under/over on how many man hours will be spent by engineering teams world wide
discussing their github single point of failure issue?

------
msie
Just the other day there was a discussion about "go get" and what would happen
in the case of a Github outage. Sigh...

------
surferbayarea
scary... "remote: Unexpected system error after push was received. remote:
These changes may not be reflected on github.com! remote: Your unique error
code: 7527a6e1bbc9fe126d51c97feac7b4e3 remote: Unexpected system error after
push was received. remote: These changes may not be reflected on github.com!
remote: Your unique error code: 7527a6e1bbc9fe126d51c97feac7b4e3"

seems pushes are going into oblivion, but the client side gets a return code
that the push was successful.

~~~
GrAfiXoNeR
Yeah I got this as well and no changes on github.com so much for creating this
release tonight. :/ I hope they will fix this otherwise this push went into
oblivion and the client thinks all is well.

------
robertrobert
Two little birdies have tweeted that there was an electrical fire at a key
datacenter.

~~~
imsofuture
We had networking issues on Rackspace earlier today...

------
rdl
They're also on Prolexic right now, so...big DDoS, likely state-sponsored?

~~~
hosay123
How can you spot Prolexic? I just checked their DNS/IPv4 and it seems to be
pointing regular hosting

~~~
antsar

        $ traceroute github.com
        traceroute to github.com (192.30.252.130), 30 hops max, 60 byte packets
        ...
         9  level3-pni.iad1.us.voxel.net (4.53.116.2)  17.609 ms  15.057 ms  10.113 ms
        10  unknown.prolexic.com (209.200.144.192)  9.186 ms  9.462 ms  9.315 ms
        11  unknown.prolexic.com (209.200.144.197)  17.753 ms  17.767 ms  18.851 ms
        12  unknown.prolexic.com (209.200.169.98)  9.922 ms  9.542 ms unknown.prolexic.com (209.200.169.96)  11.471 ms
        13  192.30.252.215 (192.30.252.215)  13.569 ms 192.30.252.207 (192.30.252.207)  9.660 ms 192.30.252.215 (192.30.252.215)  13.150 ms
        14  github.com (192.30.252.130)  9.051 ms  8.833 ms *

------
ljw1001
Reminds of the talk where a Github guy says, basically, we just push, push,
push, and if something breaks we hear about it on twitter.

~~~
sondring
That was Zach Holman. He called it TDD - twitter driven development.

------
dankohn1
And their status page says 100% operational (as updated 5 minutes ago).

------
matthuggins
Help! The site is back, but I just pushed a new branch that isn't showing on
the site, meaning I can't create a pull request.

    
    
        circuitry git:(feature/middleware) git push origin feature/middleware
        Counting objects: 18, done.
        Delta compression using up to 8 threads.
        Compressing objects: 100% (18/18), done.
        Writing objects: 100% (18/18), 5.01 KiB | 0 bytes/s, done.
        Total 18 (delta 9), reused 0 (delta 0)
        To git@github.com:kapost/circuitry
         * [new branch]      feature/middleware -> feature/middleware
    

This branch is not appearing in the repo:
[https://github.com/kapost/circuitry/branches](https://github.com/kapost/circuitry/branches)

~~~
davidcelis
We're still recovering; please give the site a bit of time to come back! Not
everything can be expected to work until we've gone green on
status.github.com. Thanks

~~~
matthuggins
Got it, thanks!

------
hackercomplex
Does anyone know if an extremely controversial new peice of software was
recently pushed ?

~~~
busterarm
No, but there's recently been a lot of talk about the Hidden Tear ransomware
source being taken off of GitHub soon. Given that the author has already been
blackmailed and all the drama surrounding that, this is a possibility.

------
franciscop
There was an error in ionic with Github's error, but I cannot create a new
issue :P

    
    
        Downloading: https://github.com/driftyco/ionic-app-base/archive/master.zip
        x Invalid response status: https://github.com/driftyco/ionic-app-base/archive/master.zip (503)
        Error Initializing app: [object Object]
        errorHandler had an error [TypeError: Cannot read property 'error' of undefined]
        TypeError: Cannot read property 'error' of undefined
        ...

------
bicolao
US west side on [http://map.norsecorp.com/](http://map.norsecorp.com/) looks
like fireworks. I wonder if that where Github servers are

------
franciscop
This was basically my train of thought:

"Damn! I'm tired of fighting installing cordova/ionic. WTF is happening now?
Oh Github is down... okay that's new"

------
bcg1
Why do they have an angry unicorn on their outage page?

~~~
gelatocar
They use unicorn[1] as a HTTP server

[1] [http://unicorn.bogomips.org/](http://unicorn.bogomips.org/)

~~~
obelisk_
Huh, TIL. I always assumed that they meant that the errors themselves are
unicorns because they rarely happen and even when they do, you usually don't
see them.

------
rdl
If people are going to depend on github being up for their CI workflows, etc.,
there should be serious effort expended at the network and cache layer to be
suitably reliable. It's probably fine to not be able to do developer-level
actions for hours, but even a 5m outage in deploying other systems is
unacceptable for most businesses.

------
rahulshiv7
Initially I was control R spamming, now I'm just pondering what a complete
apocalypse will look like.

------
lyime
Time to goto the park!

------
jordanielewski
Some "Github Pages" are down

------
AdamJacobMuller
Interestingly. Our (minimal) traffic to github shifted transit carriers when
this outage happened.

Between 2016-01-28 00:39:47+00 and 2016-01-28 00:43:26+00 there was a flurry
of BGP updates that caused that.

I'm not sure on the exact timing of the outage, this could either be a symptom
or a cause.

------
peterburkimsher
Static HTML pages are loading. Example, my personal website:
[http://peterburk.github.io](http://peterburk.github.io)

But my website loads the content from .MD files via raw.githubusercontent, and
those appear to be down.

------
human
Probably a popular page atm: [http://somatose.com/2011/10/from-github-to-
bitbucket-in-60-s...](http://somatose.com/2011/10/from-github-to-bitbucket-
in-60-seconds.html)

~~~
human
Actually, I'm pretty dumb. Can't move the repo to bitbucket since Github is
down.

~~~
oldmantaiter
If you have a local copy of the repo, you definitely could.

~~~
human
True. If it's not back in a few hours, I'll have to resort to that.

------
kelseydh
Well that's the last time I ever write a deployment that assumes Github is
working...

------
kelseydh
I heavily rely on Github # tags to document my codebase. Failures like this
make me wonder if there is any way to decouple my codebase from Github while
preserving all the comments and issues on commits I've built up.

------
human
And it's back!
[https://twitter.com/githubstatus/status/692534911593549825](https://twitter.com/githubstatus/status/692534911593549825)

------
Slippery_John
Good thing we've got mirrors, and don't depend on GitHub for releases. Still,
we interact with our customers through GitHub and use it heavily in our
workflow. Productivity is impacted if not stopped.

------
asumanth
If you were stuck like me looking up an OSS github project,
[http://archive.org/web/](http://archive.org/web/) has github pages.

------
pininja
Github is back online! Github is back online! Github is back online! Github is
back online! Github is back online! Github is back online! Github is back
online! Github is back online!

------
dragonshed
I wonder how many origin2->bitbucket are being created right now.

------
mirekrusin
Doesn't look like DDoS, maybe they deployed some dodgy code?

~~~
mirekrusin
...graph is shocking, it's like somebody pulled the plug out.

~~~
dimgl
likely a data center problem

------
mikesickler
I think I'm setting some kind of Ctrl-R land speed record.

~~~
mrdrozdov
Heh, talk about a negative feedback loop. DDOS propagating more DDOS....

~~~
mitchtbaum
That would mean a positive feedback loop.
[https://en.wikipedia.org/wiki/Positive_feedback_loop](https://en.wikipedia.org/wiki/Positive_feedback_loop)

(Positive in this sense has a similar meaning in behavioral psychology's
terminology for operant conditioning, adding punishment.. which both come off
as odd with our common use of positive as good and beneficial :/
[https://en.wikipedia.org/wiki/Operant_conditioning](https://en.wikipedia.org/wiki/Operant_conditioning))

...Carrying on with your analysis, since uninterrupted positive feedback loops
result in explosion, I guess that would mean either (for this outage) github's
fallback status page would go down or the parts of the network carrying that
traffic, which could not handle their increased load, would go down.

Probably better we would rework these control mechanisms and reroute \ rewire
\ entirely change these processes.

------
rezacks
Oops!

[Oh My Zsh] Would you like to check for updates? [Y/n]: y

Updating Oh My Zsh

Username for '[https://github.com'](https://github.com'):

------
michaelmior
> January 28, 2016

> 00:00 EST The status is still red at the beginning of the day

They seem to have some time issues as well since it's not yet January 28th in
EST.

~~~
peterkelly
Perhaps it's in a different timezone to EST

~~~
michaelmior
That would be odd since all other times are (correctly) reported in EST.

------
freebasedgirl
Always funny when a DCVS is used more like a CVS like github. They go down and
you have all the same failings as if just using cvs/svn.

------
tyrankh
Seems like as good a time as any to head home then.

~~~
iKlsR
Yup. [https://pbs.twimg.com/media/CQX-
ihDWoAA0lc7.jpg](https://pbs.twimg.com/media/CQX-ihDWoAA0lc7.jpg)

~~~
logn
Please create a new branch before emergency commits.

------
kelseydh
Weird, Github is down, but I was able to pull new code not existing on my
local repo from Github successfully...

------
halayli
I wish they say more than

> We're investigating a significant network disruption affecting all
> github.com services.

~~~
Sir_Cmpwn
I would prefer that they focus on getting back up and publish a post-mortem
after the fact.

~~~
kelseydh
The best approach to this was shown recently by Slack, where they had a huge
team of people on social media while their engineers figured out the outage.

------
dreamcompiler
OMG! PONIES!!!

------
cmcginty
It looks like they stopped updating the availability percentages on the status
page? ¯\\_(ツ)_/¯

------
nanodano
This is why I run my own GitLab server as a mirror! Or just in case GitHub
gets hit by a bus.

------
kin
If this happens often enough maybe my company will start using their
Enterprise install

------
dsavinkov
Githib is down in San Francisco

------
msie
Even getting read-only access would be nice. I just want to look at some
source.

------
awqrre
On the upside, they gained lots of Twitter followers...

------
superuser2
Welp. Remember to vendor your Go dependencies, folks.

------
joeevans1000
Too bad I'm not teaching someone to use git and/or github right now. Perfect
opportunity for a practical joke. "Oh, great, look... you broke GitHub with
that last command."

------
mirekrusin
Google code was turned off and now github?

~~~
lazyjones
Can't update to a newer Go devel version; can't fetch Go packages, _sigh_ ...

~~~
iends
Why can't you update to a new Go devel version? The code on github is just a
mirror of [https://go.googlesource.com/go](https://go.googlesource.com/go)

~~~
lazyjones
Because this is my workflow for it:

$ brew reinstall go --head

==> Reinstalling go

==> Cloning
[https://github.com/golang/go.git](https://github.com/golang/go.git)

Updating /Library/Caches/Homebrew/go--git

Username for '[https://github.com'](https://github.com'): ^C

------
wikiman
DELETED because this joke has been made

------
conwayanderson
Are they associating unicorns with crashes on purpose? Is the same illuminati
that controls our financial markets taking control of Github?

------
nodesocket
Does this look like a DDos attack?

------
piratebroadcast
Github Pages still working though.

------
philip1209
Back online as of 2:28AM UTC

------
chrome_alley
They just now managed to get "effecting" changed "affecting", so there's hope!

~~~
ivthreadp110
Grammar police are on the job- everyone can calm down... Unless the grammar
police are the REASON for the outage. Mother should I trust the government?

------
xdinomode
Is this China again? Ughhh.

------
travjones
"significant network disruption" == DDoS :( Come back, Github, please.

------
marcosnils
@githubstatus updated

------
multinglets
I just really hope they honor their code of conduct while they're
investigating this.

~~~
slapresta
That doesn't even make sense.

------
debacle
I'm kind of tired of paying for a service that has so many outages.

~~~
beat
Go to bitbucket, or self-host, or find some other shiny new competitor. The
market speaks with its feet.

~~~
debacle
We're already planning a bitbucket move. Unfortunately it's not planned for
another week or two.

~~~
beat
Having used both, I find github's tools (mostly) better. And bitbucket isn't
100% reliable either.

~~~
debacle
Self-hosted is.

~~~
beat
Self-hosted isn't 100% reliable either. Nobody is that good. I've done serious
high availability (financial system that would land on the front page of the
New York Times the way Github lands on HN), and it wasn't 100%. And the
overhead of high availability is incredible.

No way would I do self-hosted version control. I have better things to do than
babysit servers for commodity services.

