
Major Service Outage at GitHub - acconrad
https://status.github.com/?date=now
======
dagenleg
Github is down --> check out the unicorn --> refresh hacker news --> refresh
again --> shitpost in comments

~~~
karlmdavis
The coding equivalent of accident rubbernecking.

------
ComputerGuru
Obligatory Gogs plug: [https://gogs.io](https://gogs.io)

It also has an awesome automatic mirror mode that can remain sync'd to a
remkte repository. I don't care what your company does or how you run it, but
please don't just rely on Github to host your code without a mirror somewhere.
There are drawbacks to this incessant centralization.

~~~
johnnycarcin
Just this week I setup gogs on my VPS to mirror all of my repos on github.
It's part of my new year goal to avoid my dependence on 3rd party apps. Of
course I still am depending on my VPS provider so...

~~~
the_duke
Just set up a Banana Pi or whatever at home to mirror your mirror.

~~~
bisby
The specs for a beaglebone black are worse than a raspberry pi 3 in every way
except the eMMC storage. Am I missing something? (I've actually been looking
for something stronger to include in my raspberry pi cluster at home, but this
doesnt seem like it)

~~~
the_duke
Whoops you are right, I meant the Banana Pi m3([http://www.banana-
pi.org/m3.html](http://www.banana-pi.org/m3.html)).

~~~
bisby
Banana pi is probably overkill for a low-key git repo. Its double the price of
a raspberry pi.

That said, I'll actually be looking into this to possibly run arm stuff that
needs more power.

------
synthecypher
How am I supposed to do any work if I can't copy and paste code.

~~~
mkaziz
My builds are broken because they have dependencies on github repos.

~~~
mjolk
This is a teaching moment for you then. Minimize external dependencies when
you've moved from "thank god it works" to "this is production ready."

------
euph0ria
I wonder how many companies are unable to deploy new updates now because they
rely on Github?

~~~
dbg31415
Friends don't let friends deploy on Fridays... it's always a horrible idea.

~~~
efrafa
Deploying on Friday is fine if you can rollback quickly :)

~~~
pc86
I think the point was that you don't want to be messing with production when
people are going to be gone for days. Even if it's easy to roll back I'd just
as soon not deal with it. If your application is of such importance that
rolling a change out Friday v. Monday makes a different in the business's
bottom line, that business can almost certainly afford to have weekend staff
that could cover it.

------
mariusbutuc
Currently 98.9212% app server availability in the last month.

That means 7h 52m 49.7s of potential downtime/unavailability [1] in the last
month.

And the trend of that percentage doesn't look that rosy.

[1] [https://uptime.is/98.9212](https://uptime.is/98.9212)

------
escapologybb
I hope everybody on HN is doing okay with this recent outage, I'm here if
anybody wants to talk.

Seriously, Github Outage Syndrome (GOS) can be horrible. If you don't talk
about your outage feelings, they will eat you from the inside out.

Obviously anything you tell me will be kept in the strictest confidence, all
without judgement.

Stay strong HN, Github will be back soon!

Edited to Add: Why they hate guys? Didn't mean to offend, thought the post
said it all and could probably do with a little levity. My bad. Maybe.

------
vitro
I've configured one remote with two pushurls and set this remote to all my
branches. This way I always push to BitBucket and GitHub at the same time.
Just wondering, how this would work if I was not the single user contributing.

~~~
bisby
You'd have to either coordinate with the other users to have them do the same,
or officially declare which one is "official" and which one is "backup" and
set up something so the backup tracks the official (and then you only need to
push to the official as well)

------
mariusbutuc
…and this is how we remember how much we rely on GitHub, and how much value it
provides. Every day.

~~~
yeukhon
It sort of becomes your Internet and computer. No computer no work. No
Internet no VPN no work.

But in this case I guess investing in local mirror is a great investment.

------
achairapart
A little bit OT, but having colored favicons based on the status is a nice
touch!

[https://status.github.com/images/status-icon-
green.png](https://status.github.com/images/status-icon-green.png)

[https://status.github.com/images/status-icon-
orange.png](https://status.github.com/images/status-icon-orange.png)

[https://status.github.com/images/status-icon-
red.png](https://status.github.com/images/status-icon-red.png)

------
sahin-boydas
Long live Gitlab...

~~~
hackerboos
Except. Pushes of new repos are still broken in the CE. No fix for over 3
weeks.

[https://gitlab.com/gitlab-com/support-
forum/issues/207](https://gitlab.com/gitlab-com/support-forum/issues/207)

[https://gitlab.com/gitlab-org/gitlab-
ce/issues/26369](https://gitlab.com/gitlab-org/gitlab-ce/issues/26369)

[https://gitlab.com/gitlab-org/gitlab-
ce/issues/21502](https://gitlab.com/gitlab-org/gitlab-ce/issues/21502)

Only work around is to create a README on empty repos via the GUI and which
then opens up the Protected Branches page. That said the rules around who is
able to commit on the repos are broken.

~~~
Perihelion
Hey there, thanks for bringing this up. First, I want to apologize for the
inconvenience you've experienced as a result of this issue. Second, I'd like
to apologize for the delay in getting back to you on this as well.

You're right, 3+ weeks is a pretty long time for something like this to be
affecting people. Unfortunately I don't have an ETA on when this issue will be
resolved, but I'll be keeping tabs on it going forward. Please feel free to
add additional comments to the issues if you haven't already, or you can reach
out to me directly. Again, I'd like to apologize for any trouble this caused
you and others affected by this issue.

~~~
sytse
This issue should be fixed as of today in 8.16.3

------
mwpmaybe
I'm envisioning some kind of caching forward proxy for git. Point your
clones/fetches at it and it will proxy the connection to the source fork. If
the source fork is down, it will serve its cached copy of the fork.

I found this[0] but it's prerelease and looks like it hasn't been touched in
five years. Anyone know of any other solutions like this? Good idea? Bad idea?

0\.
[https://github.com/rohanpm/ngitcached](https://github.com/rohanpm/ngitcached)

------
mojitoking
GitHub adds value. They have a few engineering issues to solve given the
negative attention such a popular code versioning system attracts. Although
there are some outages due to this condition, the value added to our
organization far outweighs these minor inconveniences.

------
royka118
Been working on a down detector type app, its like half baked at the mo,
hopefully I can build up a data set that allows people to fairly compare the
reliability of different SASS products such as github and bitbucket

[https://young-forest-39406.herokuapp.com/sites](https://young-
forest-39406.herokuapp.com/sites)

------
Cafey
Has anyone heard what might be the cause?

~~~
andrewstuart2
> 09:14 MST Major service outage.

It's been 4 minutes, so I highly doubt anybody at GitHub even knows yet. Maybe
_one_ guy who's thinking a lot of "oh crap, oh crap, oh crap" right now.

~~~
alfalfasprout
Pretty sure devops at a service like Github have slack alerts and/or something
like pagerduty. They'll be drowning in alerts.

~~~
andrewstuart2
Knows the cause, that is. :-)

------
yeukhon
So there was an outage yesterday which required failover to another data
center. Wonder if both are related. Either way I want to read their
postmortem. GitHub's postmortem is usually very educational.

------
mkaziz
I use JSPM on my projects and I have several dependencies on github repos that
aren't in NPM. Anyone have any ideas on a fallback solution for when github
goes out like this?

~~~
dsp1234
Have a local git server, with an automatic process for pulling in the
repositories that are your dependencies. Then use the repos from that local
git server instead of github directly.

------
pella
Hacker News Time! :)

~~~
joatmon-snoo
I count 4 threads on /new referencing GitHub down :D

~~~
messutied
but we are keeping this one :) I like it

------
srik
Problems at quite a few places actually -
[http://downdetector.com/](http://downdetector.com/)

------
almonj
Friday the 13th...

~~~
Rooster61
The day following a full moon...

~~~
ff10
The day github had a major outage...

------
webmaven
GitHub's Twitter status post at the start of the outage:

[https://twitter.com/githubstatus/status/819940738452615168](https://twitter.com/githubstatus/status/819940738452615168)

Currently 442 retweets, 266 likes.

⋯

OK, they say it's back up:

[https://twitter.com/githubstatus/status/819947685503442945](https://twitter.com/githubstatus/status/819947685503442945)

~~~
etimberg
Still getting 503 First Byte Timeouts when trying to view content on
raw.githubusercontent.com so definitely not all back yet

------
dajohnson89
OAuth API is still down.

------
mkurz
Someone was faster:
[https://news.ycombinator.com/item?id=13391883](https://news.ycombinator.com/item?id=13391883)

------
the_duke
Again.

Affects both the website and git.

~~~
joatmon-snoo
Affects push to GitHub*

I really hope that your copy of git isn't broken because GitHub went down :p

~~~
the_duke
Haha got me. :P

No I just tried both the website and a `git pull`.

~~~
mbaha
It's back :).

------
kumarm
Seems back up now.

------
analogmemory
"Github is down" is the new compiling.

[http://xkcd.com/303/](http://xkcd.com/303/)

------
turshija
Site came back online, but ssh/git still down.

------
holtalanm
looks like it is back up now

------
mbaha
It's back. [END OF THREAD ?]

~~~
dajohnson89
Since I can't push to my repo, I'm gonna say no, it's not back.

------
philipDS
Github has been down (or very slow) a lot the last few months. They used to
have a good engineering reputation (at least with me), but it's been tough
times over there.

