
GitHub was down - bithavoc
https://status.github.com/messages?ts=mon-jul-31-2017
======
kaikai
I would love to see a chart of traffic to other sites when GitHub goes down.
My bet is that HackerNews and Twitter both get significant spikes from all
those bored developers.

~~~
tinalumfoil
> bored developers

Bored? Git's a distributed version control system, so no excuses. Get back to
work!

But in all seriousness I kind of wish GitHub provided a way to mirror things
like issues and PRs so you never have to be fully reliant on one service. Not
being able to read these really does make it impossible to get work done
offline.

~~~
syshum
But they want you to Rely on Github... if they could find away to make Git
only work with Github they would in a heart bet.

Github is not a open source company, it is not really even a supporter of free
software IMO they are a danger to free software for this very reason. They are
following the old school Microsoft model of Embrace and Extend... I am waiting
to see if they can extinguish,

~~~
heartbreak
What? Github employs members of the Rails core team including Aaron Patterson.
They literally pay for open source software development.

I'm sure they employ direct maintainers/contributors to other open source
projects as well.

~~~
hueving
Are they paying him to help make an open source alternative to github? If not,
then it's not really relevant.

They are actively working against open source tooling for software development
by developing features only for their closed platform.

Gitlab is a much better example of a company that fully embraces open source.

~~~
daxelrod
GitLab does not fully embrace Open Source.

GitLab Enterprise Edition does not have an open source license. Since
GitLab.com runs GitLab EE, their two (presumably) primary sources of revenue
(EE licenses and GitLab.com paid accounts) come from non-open source software.

But: it's still fair to call GitLab as a company way more Open Source than
GitHub. All of their development happens out in the open, the vast majority of
their codebase is Open Source, and the source code for GitLab EE is even made
available.

This is a spectrum, not black and white.

------
alexchamberlain
When I break our GitHub webhooks, I joke it's time for people to practice our
Disaster Recovery (DR) procedures. In all seriousness, this is a good
opportunity to practice work without GitHub. Any service can go down; can you
deploy a critical bug fix without it? If not, why not and what can you do to
fix it?

------
kmfrk
I had to change a username from capitalized to uncapitalized and use my
updated remote afterwards, apologies if I broke it for everyone.

~~~
cjbprime
It's kmfrk's fault! Get them!

~~~
drfuchs
"Git 'em!" FTFY.

------
_Marak_
If anyone is interested, I've been working with a git host that is actually
distributed across a p2p network using SSB.

see:

[https://github.com/clehner/git-ssb](https://github.com/clehner/git-ssb)

[https://github.com/noffle/git-ssb-intro](https://github.com/noffle/git-ssb-
intro)

It's been working fairly well so far. We are using git-ssb to manage a few
projects instead of putting them into Github.

~~~
vertex-four
Is there, at least theoretically, a way to prevent other people from pushing
to my repo? That seems like it would suck re griefing for any project that
might become even mildly politically sensitive for whatever reason.

~~~
_Marak_
Everything is key based, so only key holders push to your repo. It's all based
on the SSB protocol.

I'd suggest reading [https://github.com/noffle/git-ssb-
intro](https://github.com/noffle/git-ssb-intro) to get an idea.

~~~
vertex-four
That states: "git-ssb's permissionless model has an interesting consequence:
anybody can push to anybody else's git repository." The guide doesn't show any
key-sharing in order to do that. Are you saying that's incorrect?

~~~
_Marak_
I don't want to give you the wrong answer so I've forward this question to the
SSBC network for one of the core developers to answer better.

I'd be surprised if there was no security for pushes. The repos I've worked on
did require an invite from the creator.

~~~
cel
Marak It sounds like you are working with private repos. With git-ssb
currently a repo is either public or private. Private repos are encrypted to a
fixed set of recipients so only those keyholders can access it. Public repos
are unencrypted.

------
slap_shot
Status now shows Major Service Outage:

12:32 EDTMajor service outage.

[https://status.github.com/](https://status.github.com/)

~~~
indubitably
unicorns all around

------
luhn
Pages Builds Failure Rate spiked to over 2000%. I don't know how that's
possible, but it seems pretty bad.

~~~
adrianpike
Maybe 20+ failing retries for any single page build?

------
tambourine_man
Insert remark on why we use a centralized service for a distributed source
control system, etc. No one seems to care, unfortunately

~~~
Goladus
The reason is github did a fantastic job of implementing useful features. The
visual design is unmatched and they have done a great job implementing
developer oriented integrations and social features.

A more federated approach to this sort of thing might have been nice, but so
far nothing I have seen comes close to the value-add offered by github.

~~~
xutopia
You're sidestepping the main reason I believe it worked so well. It benefits
from network effect. It is a collaborative tool and people like to have their
work on there so others can collaborate with them.

~~~
Goladus
That's important but I think people over-estimate it. It's both. I predict
that if you analyzed the github network, you'd find many hubs are based around
companies that chose to move their workflows to github based on features other
than network effects. Or at least, the existing network was only one of many
reasons.

~~~
manigandham
As a business, we (and most other companies I know) chose github for features
and performance. Nice that other open source stuff is there but doesnt matter
for what we pay for.

------
rcthompson
Looking at the status graphs, it seems like there was some clearly anomalous
data starting around midnight, about 9 hours before the actual outage "began".
Maybe a gradual botnet ramp-up, and 9:27 AM is when it got bad enough to
overload some critical service? (Or really any other threshold-based failure
scenario.)

~~~
aksx
or a bad commit being deployed on their fleet.

------
sashk
What was happening to Github for a week or so in late June - early July? I see
"The status is still red at the beginning of the day" for a whole week.

[https://status.github.com/messages/2017-07-03](https://status.github.com/messages/2017-07-03)

~~~
nowarninglabel
They seemed to be suffering some external attacks / DDoS but I never saw a
post-mortem from them on it, hopefully one is forthcoming (or maybe is out but
I missed it)

~~~
richardwhiuk
They don't do post mortems.

~~~
daxelrod
They occasionally do, and they call them Incident Reports.

[https://github.com/blog/2106-january-28th-incident-
report](https://github.com/blog/2106-january-28th-incident-report)

[https://github.com/blog/2273-incident-report-inadvertent-
pri...](https://github.com/blog/2273-incident-report-inadvertent-private-
repository-disclosure)

------
relaxitup
Do these general Github outages affect GH Pages as well, or is that service
portion segmented to some degree?

~~~
bluehatbrit
Pages are static sites served from separate infra from the main github.com
from what I've heard, so you should be fine in almost all cases. I've never
had one go down unless I pushed a dud build.

------
leesalminen
I think it started as minor as I was receiving a unicorn once per 10 pages.
It's currently happening on almost all.

Of course, I'm trying to dig into a WebKit issue and need the issues to load!

------
pmoriarty
Where is github hosted?

Do they use AWS or another commercial cloud provider, or do they have their
own servers in data centers (hopefully scattered around the globe)?

If AWS, are their services spread among multiple availability groups? I'm just
wondering how this could happen.

~~~
sattoshi
I like how this comment has 3 replies saying different things.

I am now feeling less informed than I was after the first reply.

~~~
worldexplorer
An ex-github employee who used to give talks on Github scaling challenges
revealed they shifted from Rackspace. See yourself -
[https://github.com/holman/ama/issues/553](https://github.com/holman/ama/issues/553)

------
pramodzion
Github is back online.

------
Cafey
It has leveled up to a major outage!

------
peterwwillis
Dang. It's too bad their customers' source control files aren't distributed
and decentralized, or they could keep working and ignore this.

~~~
toyg
The problem is not the files, it's the project management stuff (issues and PR
tracking). _That_ is a SPOF. If they could decentralize that, then nobody
would care if they are down half the time.

~~~
mschuster91
I wonder why you can't export them as a read-only git repository. (Read-only
to avoid people messing around with the history)

~~~
daxelrod
You can actually export them, but the format is undocumented, and is meant
only for import into GitHub Enterprise:

[https://developer.github.com/v3/migration/migrations/](https://developer.github.com/v3/migration/migrations/)

------
loomer
Anyone have any knowledge of what specifically happened?

------
AlphaWeaver
I saw a comment earlier mentioning that GitHub allegedly doesn't release post
mortems publicly? If this is true, that's upsetting.

~~~
luhn
They have published post-mortems before, but the previous bout of downtime
went unacknowledged. Maybe it's a new policy.

Here's a postmortem they did several years ago:
[https://github.com/blog/1261-github-availability-this-
week](https://github.com/blog/1261-github-availability-this-week)

------
yellowapple
My apologies. I knew my Perl 6 wrapper for GLFW was bad, but never realized
it'd be so bad that GitHub would choke to death on it.

------
ibgib
Are there any other major sites that are down?

~~~
thepumpkin1979
All software depending on Github repo releases downloads are down, e.g:
Rancher CLI.

~~~
smudgymcscmudge
Release downloads are affected?

~~~
thepumpkin1979
yep, my Circle-CI deploy fails, the following URL is not available:
[https://github.com/rancher/cli/releases/download/v0.6.0/ranc...](https://github.com/rancher/cli/releases/download/v0.6.0/rancher-
linux-amd64-v0.6.0.tar.gz)

------
vishesh92
It just became a major service outage.

------
DevKoala
This is happening too frequently now.

------
Osiris
It's starting to work again for me. I was able to approve a PR and merge it.

------
xxkylexx
Looks like I am still able to push to/pull from my repos without issue.

~~~
DylanBohlender
I've had a few CI jobs fail when attempting to pull, but I've also had some
succeed. Seems that it isn't an all-or-nothing outage.

------
macawfish
Whatever happened to gittorrent?

------
tevonsb
Thoughts on the cause?

~~~
cjbprime
Definitely Bitcoin-related.

~~~
midnitewarrior
SegWit's first victim.

------
GrumpyNl
How does this affects all your dependencies?

~~~
kentor
Why would it?

~~~
s73ver
A lot of package manager tools download their packages from Github. If you
happened to be refreshing your dependencies at the time, or doing a clean
build, then you'd be SOL.

~~~
moosingin3space
Really? I was under the impression that most language package managers
downloaded from a CDN. I know that pip, npm, yarn, cargo, and hex do, at
least.

~~~
s73ver
Homebrew I believe goes through GitHub. Many of the Vim plugin managers also
do, or at least have the option. I think CocoaPods and Carthage, both for iOS
development, do as well.

I think it's somewhat common for a new package manager to use Github as a kind
of CDN for a while, until they get big enough they can do their own.

------
adtac
>GitHub is having a minor service outage

It's definitely not minor.

------
moomin
I knew I shouldn't have released the new version of my project yesterday. :p

Sorry everyone

------
ProAm
Githubs uptime is pretty bad. Isn't it under 95% for the year now?

~~~
valarauca1
Not really.

It is just very public when they go out.

Steam, Blizzard, Activation are down at least 6 hours a week. While Bank of
America is offline 24hours/month. Scheduled downtime is still downtime.

~~~
FireBeyond
> Scheduled downtime is still downtime.

Except in the SLA.

I've argued more than once when watching a vendor announce "emergency
scheduled maintenance"(!) as an outage is looming.

------
__s
In the face of a lack of information, HN comments begin to throw around
unfounded speculation & tongue-in-cheek jokes run rampant. I suppose that in
the absence of information, many stay silent, & the remaining see a thread
lacking comments

& now we've got this meta one in the mix

------
0xbear
How many more can we expect before they develop appreciation for testing
_before_ they push to prod?

~~~
cjbprime
Newly introduced errors in site code are only one of many sources of failure
for a site like GitHub, and probably a fairly rare one.

~~~
0xbear
It's a nearly weekly occurrence. Avoidable one, I might add. When was the last
time a major Google service had a major outage?

~~~
cjbprime
You're still missing the point, by assuming that the outages are code related
when you call them avoidable. Distributed systems at scale are terrifyingly
hard to control.

When I said "probably a fairly rare one", I didn't mean that the outages are
rare. I meant that new code is probably a rare cause of the outages that
happen. They have other causes unrelated to new code.

(I'm also skeptical that GitHub major outages happen "nearly weekly", but I
don't have data.)

~~~
0xbear
I'm not "missing" anything. I worked at Google for 7 years much of which was
spent working on, you guessed it, distributed systems infrastructure. You
guard against this by carefully canarying things and putting robust testing,
monitoring, and deployment procedures in place. A release might take a few
days, but you can be reasonably certain your users won't be your guinea pigs,
and if shit does hit the fan, rollback is easy, and you can reroute traffic
elsewhere while you roll back. Most of the time no rollback is needed: you
just flip a flag and do a rolling restart on the job in Borg. For some types
of outages (most of which users never even see) Google has bots that calculate
monetary loss. And the figures can be quite staggering and motivating, so
people do postmortems and try their best to make sure the outages don't happen
again.

~~~
paublyrne
So no Google service has ever experienced an outage? I distinctly remember
Gmail being down on several occasions.

~~~
0xbear
Gmail is several orders of magnitude larger than Github will ever be, and in
recent memory I can only recall it being down once, and for a very small
subset of users.

