
GitHub is down - tomduncalf
https://www.githubstatus.com/?
======
robotsquidward
I sometimes look forward to outages like this just so I can read the post-
downtime-resolution blog post that almost always follow. I find reading about
how companies deal with issues when shit hits the fan to be really
interesting.

~~~
beebs93
Agreed. I almost always dial into any large scale events at my company -
regardless if it concerns me directly - to listen how ppl critically evaluate
data, triage errors, and determine how to mitigate the problem(s).

Whatever skill level I have in this area, I cannot nail down if it's from
multiple "baptism by fire" situations, learned naturally via university
courses in the scientific field, strategy video games (half joking), or other
sources I cannot think of.

I've listened to some very impressive people handle serious crisis situations
and I'm both in awe and curious how they achieved that level of deductive
reasoning.

~~~
joshbetz
The same way anyone gets good at anything: practice. Knowledge of the system +
experience in high pressure situations and you get pretty good at solving
those problems over time.

------
bluejellybean
Although it's nice that Git is Git and we can all mostly still work, it still
seems foolish to rely on a single point of failure like Github. I've been
toying around with the idea of creating a tool that would map the Git api to
work with two+ hosting services at the same time. The effect would be
something like, run "$git push" and it pushes to Github, Bitbucket, and
Gitlab. I can't imagine something like this would be too difficult and would
eliminate having to twiddle your thumbs while you wait for things to come back
up.

~~~
TimWolla
You mean like?

    
    
        git remote set-url --add --push origin git@github.com:Foo/bar.git
        git remote set-url --add --push origin git@gitlab.com:Foo/bar.git
    

:-)

see: [https://git-scm.com/docs/git-remote#Documentation/git-
remote...](https://git-scm.com/docs/git-remote#Documentation/git-remote.txt-
emset-urlem)

~~~
bluejellybean
Hah, today I learned! It's a great feeling when you don't have to reinvent a
wheel, thanks.

Now I'm just annoyed that more teams I've been on haven't set this up!

~~~
bartread
Me too. That's awesome: I've just suggested it to our team. Thanks for sharing
GP!

One serious question though: how do you deal with PRs when you do this? That's
one area where it feels like things could be quite messy, especially if you
have quite a few PRs going in throughout the day.

~~~
mikepurvis
There have been various proposals over the years for how to integrate issues
and reviews in the distributed git tree itself ([http://dist-
bugs.branchable.com/software/](http://dist-bugs.branchable.com/software/)),
but I don't think any of them have really gone anywhere, certainly not in
terms of support by the hosted git vendors.

Having looked briefly into it now, git-dit does look promising in its
approach. I'd be interested to hear from someone who had actually used it and
bumped up against the limitations: [https://github.com/neithernut/git-
dit/blob/master/doc/datamo...](https://github.com/neithernut/git-
dit/blob/master/doc/datamodel.md)

~~~
MayeulC
There is git-appraise to fill that gap [1]. I am personally waiting for a
federated "forge" for federating PRs across platforms, such as the one
developed in [2]. Maybe via e-mail? [3].

[1]: [https://github.com/google/git-appraise](https://github.com/google/git-
appraise)

[2]:
[https://github.com/forgefed/forgefed](https://github.com/forgefed/forgefed)

[3]: [https://drewdevault.com/2018/07/23/Git-is-already-
distribute...](https://drewdevault.com/2018/07/23/Git-is-already-
distributed.html)

------
carapace
Github: the premier collaborative platform for FLOSS development, which itself
is closed-source, made under a closed development process. Based on a
_distributed_ CVS, it has become a massive SPOF. As if that wasn't strange
enough, it was acquired by _Microsoft_. Was that the victory of FLOSS or the
defeat? How can you tell?

WTF is going on?

Mood: "Am I going crazy, or is it the world around me?" ~
[https://fishbone.bandcamp.com/track/drunk-
skitzo](https://fishbone.bandcamp.com/track/drunk-skitzo)

~~~
tyre
This falls under very specific genre of Hacker News comment that has always
struck me as profoundly uninteresting.

Github built something that inarguably made development better.

\- They host some huge number (tens of thousands? hundreds of thousands?
millions?) of repositories for free.

\- They built Github Pages for people to ship websites and host those for
free.

\- They built an intuitive flow for managing issues and pull requests, which
is included with every repository for free.

\- They integrate freely and openly with all sorts of third-party services,
some of which compete directly with them. That's quite uncommon for a for-
profit software platform of their size.

\- They have given back immeasurably to the programming community, to the open
source community, and to developers including sponsoring conferences, donating
office space for events, etc.

\- Their developers widely share their (i.e. done at/for Github) work and
contribute upstream.

There is some argument that all of the above are self-serving for Github,
which is proven false if you talk to a single developer at Github.

Do they adhere to every idealistic principle of FLOSS? No. But, honestly, who
gives a crap?

So, to answer your question of WTF is going on: real, positive progress.

~~~
emilsedgh
* This falls under very specific genre of Hacker News comment that has always struck me as profoundly uninteresting. *

Not sure why you had to lead with an irrelevant comment like that as the rest
of your comment has interesting counter arguments.

I think Github did make development easier, but the fact that it's close
source and now owned by Microsoft gives me an uneasy feeling.

With that being said, the only thing that does make it ok is the fact that Git
is decentralized.

If Github was an svn host I definitely wouldn't have been ok with it or hosted
anything FOSS on it.

~~~
evv
> Git is decentralized. If Github was an svn host I definitely wouldn't have
> been ok with it or hosted anything FOSS on it.

The most sticky part of Github is the social network that lives in issues,
pull requests, and repository permissions. This is all entirely centralized,
and it is scary how heavily the community relies on it.

So in practice, results in just as closed of a system as SVN. And it doesn't
matter if you're "ok with it". GH is where all the people and projects are. If
you want to participate, you have no choice.

------
mielecmichal
Good that git is a distributed VCS. At least our master branch history is
available on our local PCs, and we can freely work. Yupi.

~~~
pmelendez
Do you do PRs? or have CI/CD pipelines hooked with GitHub?

~~~
wongarsu
not being able to submit or review PRs for a day is much less impact than not
being able to checkout for a day.

~~~
pmelendez
I guess it depends on the definition of done of each team.

We normally can't complete tasks until they are code reviewed, so checking out
locally would be the analogous of working offline in a CVCS

~~~
viraptor
You can finish the first task as much as you can, then start on the next
one...

------
harel
This is the perfect time to take a break. Kick back, have a coffee,
contemplate your life choices. That commit can wait, that PR (i was about to
merge) can wait too. It's not the end of the world.

~~~
mattnewton
Why not just keep working? That next commit will work just fine without a
central server.

~~~
ilikehurdles
At some point there has to be centralization. We are deploying one product,
not N products for N commits. If I can't push, I can't have CI build my
branch, and I can't submit a PR for review.

~~~
mbrumlow
But you could continue working and only have to apply minor patches forward if
you later find your starting point needed to be fixed.

You don't have to wait for CI, just create a new branch and continue your work
as though CI passed, and your PR was accepted. If that comes true you don't
need to worry much, just rebase and continue.

On the other hand if you find out you needed to make changes - do those on the
branch you made those changes, and finish your PR/CI cycle. Then go to the new
branch you continued work on and rebase, and continue.

Is there something I am missing?

~~~
zymhan
Heaven forbid we find a reason to take a break

~~~
jnbiche
In the old days at least we had compiling. Not so much anymore.

~~~
ascorbic
npm install

~~~
radicalriddler
And when you need to delete your node_modules folder for the upteenth time,
it's a nice 5 minute break.

------
joshuaturner
You've got to wonder what the regex that caused _this_ downtime is going to be

------
sirtoffski
It's funny I thought pi-hole was somehow blocking github, but dig returned
good responses.

    
    
      dig raw.githubusercontent.com +noall +answer                         0 < 12:54:48
    
      ; <<>> DiG 9.14.3 <<>> raw.githubusercontent.com +noall +answer
      ;; global options: +cmd
      raw.githubusercontent.com. 6 IN CNAME github.map.fastly.net.
      github.map.fastly.net. 19 IN A 151.101.20.133
    
    

Naturally the first thing I did was check here and nothing was posted. After
20-minutes of TSing I thought it was DNSSec screwing up as after disabling it
everything works.

And now I come back here and see this.. :D

~~~
tptacek
Thankfully, like virtually all major tech companies, neither GITHUB.COM nor
GITHUBUSERCONTENT.COM are DNSSEC-signed, nor are they ever likely to be.

------
hysan
Just came here to post this. I was reading through the css-modules docs[1] and
am now stuck. I can't see anything other than the main README. Anyone know of
a mirror?

[1]: [https://github.com/css-modules/css-modules](https://github.com/css-
modules/css-modules)

~~~
sarathyweb
Here you go:
[https://web.archive.org/web/20190701222409/https://github.co...](https://web.archive.org/web/20190701222409/https://github.com/css-
modules/css-modules)

------
corford
Kinda irritating when a binary you need is only hosted on github (...together
with the files to build from source).

~~~
rvz
Even worse if libraries / packages are only hosted on Github. Then the main
ecosystem of a language dependent on those packages can grind to a full stop
from there.

I'm looking at all of you CocoaPods, SwiftPM, npm, Cargo, vgo.

~~~
cameronbrown
Insanity. Having an emergency mirror on standby is not hard when it's critical
infrastructure. Surely npm/Cargo/etc.. could do this.

~~~
knd775
I think I'll set this up now [https://github.com/local-npm/local-
npm](https://github.com/local-npm/local-npm)

------
samcday
I'm just wrapping up the work to migrate my company away from Gitlab to Github
and this happens. I did it because I figured Github _has_ to have better
reliability / uptime than Gitlab. Someone joked that as soon as the migration
is done Github will have some major downtime.

 _sigh_

~~~
q3k
I highly recommend running at least a local, self-hosted git mirror at any
tech company, just in these cases. Gitolite + cgit are extremely low
maintenance, especially if you host them next to your other production
services.

Not to mention, if you get the self-hosted route you can use Gerrit, which is
still miles better for code review than GitHub, Gitlab, bitbucket and co.

~~~
grey-area
You don't even need gitolite, if you're going the self-hosted route:

apt install git-all

is enough to host your own git server. Put it behind a firewall to limit
access and use standard linux users with ssh keys for access control if you
don't need anything fancy. For small companies I'm not sure you need anything
else. Of course if you need different levels of access etc then you'll need
more sophisticated tools, but many people won't.

Code review I do using local tools (the editor) face to face, again not sure
you need an online service for that unless you're a larger company with lots
of developers coordinating (in which case it becomes pretty essential).

~~~
q3k
> Code review I do using local tools (the editor) face to face, again not sure
> you need an online service for that unless you're a larger company with lots
> of developers coordinating (in which case it becomes pretty essential).

I mostly like online code review services because they offer an audit trail
and semantic history that's easier to navigate than email. And of course, to
let CI automation check tests, coverage and lint. Not because I don't trust my
coworkers, but because otherwise I would forget to run tests and lint myself.

~~~
grey-area
Lots of different ways to do it, and of course github and online code review
is tremendously useful to people, particularly on large projects with lots of
collaborators, and where a history of reviews is required.

For lots of small projects though, it's perhaps not as necessary as people
think. I run tests and linting locally on save and don't really use the code
review/CI features of online hosts much. That won't suit everyone of course,
but it is one possible path.

------
gtrubetskoy
Github goes down on Monday around 9am pacific time - must be totally random.

~~~
matthewowen
The funny thing about this is the idea that west coast engineers actually
start work at or before 9am

~~~
InvaderFizz
I'm rarely in before 9AM. The 10 minute standup at 10:30 AM is the only daily
requirement for most staff.

------
primitivesuave
Brings into perspective how essential git is for my workflow - I am waiting
for `scp` to transfer my files from my laptop to work computer and can't push
anything into the CI/CD pipeline.

~~~
Twisol
For what it's worth, the "D" in "Distributed Version Control System" is useful
here. You can `git init --bare foo` on your work computer, `git remote add
workcomp username@hostname:path/to/foo` on your laptop, and `git push workcomp
master` to push everything over using pure Git. (And the first steps only have
to happen once for these two machines.)

(This creates a bare repo on your work computer, meaning there's no associated
working directory -- you'd probably want to add that same repo as a remote
from whichever existing repository on your work computer you have. The bare
repo, in this scenario, is just a means of passing commits from your laptop to
your work computer in a Git semantically-meaningful way.)

~~~
huffmsa
Your comment is probably going to inspire a lot of people to start self
hosting remote repos during the downtime today.

As with most things, it'll start great, but a lot of those people will be in
tears in a few weeks.

Great power; great responsibility.

~~~
Twisol
True! It's best to consider that kind of repo as a downstream fork, with all
the responsibilities of upstreaming changes once the canonical repository
comes back up.

------
jakejarvis
I feel like this whole year has served as one big reminder of how fragile the
internet really is...

~~~
ilaksh
It's only fragile when you centralize all of the information in giant
websites. It didn't use to be like that. And now there are a ton of
decentralization technologies that we should be taking advantage of, but
almost everyone ignores them.

For example [https://medium.com/@alexberegszaszi/mango-git-completely-
dec...](https://medium.com/@alexberegszaszi/mango-git-completely-
decentralised-7aef8bcbcfe6)

~~~
jakejarvis
Absolutely. I'm (cautiously) optimistic that some good can come from all this
downtime in that sense.

~~~
noir_lord
It certainly helps make the argument not to move some stuff to the cloud,
there is still a case for on premise as always the answer is "it depends".

------
maddyboo
Over the past 2 or so weeks, I’ve been getting 500 errors from GitHub quite
regularly. A page will take a while to load, then finally fail with an octocat
error page. Reloading the page usually succeeds with the correct page, so it
seems to be some sort of intermittent issue. In the years I’ve used GitHub, I
can’t remember this ever happening – I wonder if this incident is related? Has
their user base expanded a lot lately and they’re having difficulty scaling
their system to meet demand? I know that I have been noticing a seeming
increase in activity and new users as GitHub becomes more and more mainstream,
I wonder what their growth looks like.

~~~
ryanianian
GitHub has a really bad track-record on reliability.

I worked on our devops systems for a while. Every `git clone` had to have
multiple retries, and even then there were multi-minute outages multiple times
a month that caused things to turn red and caused distrust in our CI pipeline.

I tried hard to get my company to not rely on github as part of our CI process
(as others in these comments indicate) but that's an expensive proposition for
many - similar to relying on other third-party cdns like dockerhub, npmjs.org,
etc.

~~~
noir_lord
Bitbucket isn't really any better, they just surf the line of "right, that's
it I'll just run Gitea on a local server, I've had enough" and convenience.

------
thosakwe
Took this opportunity to set up my own Git server on a VPS I have. It was
actually extremely simple to set up the server, limit the git account (ssh
only) to only using git-shell, and adding gitweb with basic auth in an nginx
setup with letsencrypt, HTTP/2, and FastCGI. Took maybe 20 minutes, and most
of it was looking up the necessary commands.

That being said, I am, and probably will still be a massive GitHub user,
because I value its community aspect. But I also now have unlimited, truly-
private Git hosting for peanuts (cheap DigitalOcean server), which is always
nice.

~~~
gfdgfdg
You can also use gitea, which is setup very quickly as uses almost no
resources

~~~
thosakwe
I definitely considered it, but it struck as more of a "host your own Github,"
where really all I needed was "host your own Git," which of course comes out-
of-the-box.

------
nanomonkey
Time to start going back to using a decentralized git archive, like git-ssb.

[https://github.com/noffle/git-ssb-intro](https://github.com/noffle/git-ssb-
intro)

------
wyldfire
It's a hassle, for sure. If this incident cost you time, I would advocate for
you to invest a brief amount of time to hedge against GH. The next downtime
may not be as brief.

These steps will cost you < 15 minutes but you could end up with significant
savings.

1\. make an account on gitlab

2\. create a new repo

3\. configure it to mirror your existing GH repo. The mirror will periodically
pull from GH without any interaction from you, so you always have another
option.

To be clear, I don't have any opinions about the operations of Github vs
Gitlab. I have no idea which has superior procedures or equipment to avoid
downtime. But I am sure that this small effort to diversify is worthwhile.

EDIT: wowsers downvotes for practical advice? Downvoters, please chime in on
how my advice could be improved.

~~~
zegl
How will this save you from outages like this one?

The problem is usually not that people loose access to their code (you usually
have a local cache), it's that you can't open or review PRs, and that you
can't trigger CI builds.

Using _both_ GitHub and GitLab simultaniously for more advanced workflows is a
big hassle.

------
madsy
Its up n running for me now, everything works. Location Sweden.

------
TimWolla
see also:
[https://news.ycombinator.com/item?id=20498983](https://news.ycombinator.com/item?id=20498983)

------
dmonitor
Damn. Thought I was going nuts when I couldn’t browse certain repos.
Everything I checked said GitHub was up

~~~
CapricornNoble
There was a certain repo I was trying to access and I was concerned the
content had been taken down. I pulled up the repo in Google Search cache,
copied the git URL, and cloned it from my terminal. That worked just fine,
only the webpage was completely down.

------
ben_jones
Skimming this thread I found only two important takeaways, are there any more?

1) It's really, really, easy to setup a second mirror that you can switch your
CI/CD processes to in the event of downtime in order to avoid being affected

2) Unless you don't vendor your builds and some of your third party
dependencies rely on Github as their sole distribution mechanism in which case
you are SOL

------
smoyer
The incident says it was resolved but I still can't get to my profile page.

~~~
marmshallow
Could be that the fix has been deployed but it takes time to propagate thru.

~~~
noja
Then it hasn't been resolved.

------
KenanSulayman
I wonder if all of these downtimes are related to the summer holiday season...

~~~
huffmsa
They're probably still moving stuff to Azure after the acquisition. If my
memory serves, there was an uptick in outages ~6 months after the buyout.

~~~
dswalter
6 months after an IPO is often the conclusion of a typical employee lockout
period on shares, which can also be a time of more frequent team members
leaving.

I wonder what the retention incentive period was for Github employees; it's
often a year, and the Github acquisition was mid-2019...

------
8K832d7tNmiQ
That explains a sudden access error when I tried to open an issue in a repo.

------
colpabar
Can we talk about how many services have had outages recently now?

~~~
viraptor
We can talk about the Poisson distribution, yes :-)
[https://en.m.wikipedia.org/wiki/Poisson_distribution](https://en.m.wikipedia.org/wiki/Poisson_distribution)

------
dugluak
push/pull seems to be working for me. Getting 500 server error while trying to
browse repositories on the website

~~~
ilikehurdles
Push and pull doesn't work for me. For a second I thought this was my
company's subtle way of terminating my employment.

------
nwatson
Working now.

~~~
acron0
Definitely isn't.

~~~
cpncrunch
Seems to be working for me.

~~~
rhacker
As of this post it was working as of 1 minute ago.

Is this going to be a PR stunt like hotmail - they're going to say the fix was
migrating to Azure or upgraded from Linux to Windows 10.

~~~
joshypants
Can people stop with the "it's working" or "it's not" \- because there is
intermittent / unevenly distributed failures. Still have some pages not
loading for me.

~~~
carrja99
It's what makes these posts cute. Thanks to multi-region and load balancing it
will work for some and be broken for others

------
rodrigopetter
Very nice page. Some one knows a open source project similat to this
githubstatus page?

------
tus88
3 hours later still down. That's a pretty long outage by most standards. Ouch.

------
langitbiru
Yup, I can not push my commit.

------
madsy
Is my project lost now?

------
lightedman
This is why I run my own internal development repository, so I don't have to
worry about online failures.

------
jaequery
It gives me the chuckles every time I hear Github or Gitlab goes down.

Because one of the main points of Git was to provide a "distributed" version
control system.

Meaning that you are probably using it wrong if your entire company completely
halts to a stop just because Github/Gitlab/Bitbucket are down.

Maybe what we need is a service that automatically consolidates them for you
so your repos are always online?

~~~
count
Do you think Github is nothing more than a 'central git' repo? Have you even
used the site in the last few years?

~~~
dang
> Have you even used the site

That's a variant of "did you even read the article", which the guidelines ask
you not to do (see
[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)).
That's partly because it's a putdown, but mostly because there's no
information in it. Readers who know less than you do are here to learn, so a
better version of this comment would share some of what you know.

~~~
count
I disagree, and it was definitely not intended to be a put down. It's a
question, because it appears the OP was speaking of GitHub as it was, not as
it is. I don't question their intelligence or ability to read. But whatever.

------
js4ever
Gitpocalypse

------
jbverschoor
git is distributed, so no worries

------
cmarschner
Just when it was time for beergarden. Thank you!

------
0x006A
Did Microsoft switch GitHub to AGI as part of the OpenAI deal?

~~~
omarhaneef
Q: Can it automatically reject bad code or would that be impossible because of
the halting problem?

~~~
EamonnMR
it could automatically reject bad-looking code perhaps

------
black_puppydog
Nobody brought up radicle yet? HN what's up?

There you go: fully IPFS based version control and collaboration.
[https://radicle.xyz](https://radicle.xyz)

~~~
colemickens
I thought Radicle was cool too, but as I understand it (in its current state),
it has a much "larger" SPOF in that changes can only be submitted when the
single authoritative repo is online.

~~~
black_puppydog
how so? If everything is stored in IPSS, then it should _always_ be online.
The IPFS daemon is local, so you can commit things without any network
connection. Am I misunderstanding what IPFS is?

~~~
colemickens
"Currently the owner of the project must be online in order to receive any
proposed RSM updates from a contributor. Once received and processed, these
updates will be written to IPFS by the project owner, and made available to
all users who follow that project." \--
[http://radicle.xyz/docs/#faq](http://radicle.xyz/docs/#faq)

~~~
colemickens
What the hell? I get downvoted for linking an authoritative answer to the
question I was asked? I'm so god damn sick of participating here in good faith
and getting wordlessly crapped on for it. The answer supports _exactly_ what I
said, and I provided a link.

Pretty obvious someone didn't like one of my other comments and then proceeded
to downvote the others they could since I commented in multiple threads at the
same time. Why is this nonsense allowed? It would be easy to detect.

~~~
dragonwriter
If you are sick of participating, don't, and your problem is solved.

Also, if you've been here long enough to get sick of anything you should have
learned that dumb downvotes happen, they mostly get reversed over time, and
nothing good comes of reacting to them at all, much less throwing a expletive-
laden fit about them.

~~~
colemickens
Such great advice, had never thought of that. Thanks!

