
GitHub down - dustinmoris
https://github.com/#
======
pselbert
Just recently I thought to myself "it has been a long time since I saw the
Github Unicorn". As it turns out, I didn't really miss it at all.

Regarding the centralized nature of Github: it is the centralized
communication that is a problem, not the ability to share code. I can easily
send a patch to somebody on my team, but that doesn't help me review a PR,
reply to comments, trigger a CI build, or initiate a deploy.

Code sharing is only a small part of what a team relies on Github for.

~~~
Tobani
Further I think these are things that are generally harder to decentralize.
I'm not sure how much marginal value there is doing so.

~~~
kemitche
Yup. Half the comments in this thread are "oh you could do issues in git or
something." No, when my PM asks me if bug 123 is fixed, I want a single source
of truth. Whether that truth is GitHub, JIRA or something else, it's still a
single point of failure.

~~~
PurpleRamen
Maybe the best way would be to put those wiki/issue/whatever-data in the
repository itself and just build tools to throw at them. This could be github,
gitlab or something local.

------
bunderbunder
Back when hurricane Sandy took out my employer's upstream Mercurial server for
several days straight, I was pretty chuffed to be able to tell my boss, "You
know how we migrated to a new source control system a couple months ago? Well,
thanks to that, we can keep working pretty much uninterrupted. We should
double check that XXX is getting frequent offsite backups, though."

My current company uses a self-hosted option, so this doesn't affect me. But I
can't help but think that this time it's different, and we'd be hosed. The git
part would still work without too much hassle, but we are _heavily_ dependent
on a bunch of additional things that GitHub offers, such as the pull request
interface. That's slightly worrisome, I suppose.

All that said, I want to steer clear of knee-jerk assuming, "We don't have
this problem b/c we self-host." There's a sickening sense of not really being
in control of your own fate when a cloud provider goes down, but,
realistically, I wouldn't be the person in charge of getting one of our self-
hosted services back up, either. What really matters is % downtime. My
experience has been that, compared to many in-house IT departments, folks like
GitHub are generally _very_ good at keeping the lights on.

~~~
Klathmon
While I don't necessarily agree with it, the argument is that with a self
hosted solution, you have more control over when you are doing things.

So if you have a really busy time coming up, you don't deploy the new git
update that day, you wait until you would be okay with some downtime.

------
james33
Is it just me or is the new GitHub Status page a joke? It used to show
historical charts for uptime and latency for each of the different services,
which was actually useful. Now it is just a daily messages list.

~~~
anowlcalledjosh
Yes, I noticed that too. I wonder when the charts got removed.

~~~
jamessb
The Wayback Machine suggests it was 13th September 2017: [1] (from 17:50:53
GMT) has graphs, but [2] (from 20:06:33 GMT) is a 302 redirect to the messages
page.

[1]:
[https://web.archive.org/web/20170913175053/https://status.gi...](https://web.archive.org/web/20170913175053/https://status.github.com/)

[2]:
[https://web.archive.org/web/20170913200633/https://status.gi...](https://web.archive.org/web/20170913200633/https://status.github.com/)

------
d4nyll
Now the status page is updated, but the status on previous days are all saying
"The status is still red at the beginning of the day"

~~~
CoffeeDregs
I saw that, too, and was fairly concerned. A smoothly running, quality focused
engineering organization doesn't leave a status page un-updated for 3+
weeks...

Also, "reports of service unavailability"? I would expect monitoring tools to
be screaming...

~~~
sverhagen
They had an outage happening a few days ago, and I happened to be looking at
the status page. It appeared that days in the past were just showing the
current status. It went orange, all days past went orange. I refreshed again,
it went red, all days past went red.

------
willejs
It is only a matter of time before someone here suggests you use self hosted
gitlab/github enterprise. Ain't got time for that.

~~~
castis
Is it so absurd that some people would want a service they use on a daily
basis to be stable?

~~~
javamelon
That is why self hosting is not the best solution.

~~~
apple4ever
Actually, that's exactly why self hosting is the best solution.

------
dx034
From twitter:
[http://classicprogrammerpaintings.com/post/144953638470/gith...](http://classicprogrammerpaintings.com/post/144953638470/github-
major-service-outage-georges-seurat)

------
sdruskat
If all goes belly up there is always Software Heritage
([https://www.softwareheritage.org/](https://www.softwareheritage.org/)).

~~~
d4nyll
Is this like [https://web.archive.org/](https://web.archive.org/) for
software?

~~~
sdruskat
Aims to be, yes. Early stages still, but it's well-supported by a number of
institutions, funders, governments even.

------
octo_t
[https://status.github.com/messages](https://status.github.com/messages) has
more info.

------
musicmatze
Reposting this
([https://news.ycombinator.com/item?id=16124702](https://news.ycombinator.com/item?id=16124702))
as top-level comment, because I think it is worth it:

The git-dit project ([https://github.com/neithernut/git-
dit](https://github.com/neithernut/git-dit)) provides a distributed issue
tracking functionality inside git.

It does this without cluttering the repository with unneeded files, gives the
possibility for having tree-like conversation (including merging
conversations), referencing issues, linking to commits and so on. It is even
possible to host the issues in another repository as the code (if that is
wanted) or having one repository of issues for several projects / moving
issues from one repository to another.

There is only a CLI as of today. I hear that there is some effort on building
a (view-only) web frontend for it, though I don't know much about its
progress. Maybe asking the maintainer would be an idea.

What's also missing is a way to give users of the tool access to a repository
where they can submit issues (which then could also be used by a web/gui
frontend for the tool). This is not the domain of git-dit itself, but a
solution needs to be found. One idea would be a publish repo (where everyone
can push) which automatically does some sanity-verification on the issues and
forwards them to the maintainers repository... or something like that. Also
integration/mappers to/from other services (gitlab, github) are missing and so
is mail->git-dit integration (posting issues from a mailinglist automagically
into the issues repository).

Also,
[https://github.com/vitiral/artifact/](https://github.com/vitiral/artifact/)
is a really nice tool to do planning of an application or library inside a git
repository. I am currently starting using it in imag ([https://imag-
pim.org](https://imag-pim.org)) and it is really wonderful. The author
currently does a reimplementation of its core functionality to make it even
more powerful.

~~~
josephfrazier
More generally, here's one of the best discussions of distributed issue
tracking I've found (dated 2013):
[http://travisbrown.ca/blog.html#TooMuchAboutDistributedBugTr...](http://travisbrown.ca/blog.html#TooMuchAboutDistributedBugTracking2013-04-20)

------
4ver
:/ even the status page is broken

"The status is still red at the beginning of the day"

------
d4nyll
If the stars align and Stack Overflow goes down too, I can just go home.

~~~
slig
You can use the cached version from Google.

------
scoot
Apparently the status has been "red" every day since December 1st:
[https://status.github.com/messages/2017-12-07](https://status.github.com/messages/2017-12-07)

"The status is still red at the beginning of the day"

Yesterday was the first "normal" day, before this outage.

~~~
d4nyll
The status page is broken, it was mostly green 10 minutes ago.

~~~
rom1v
They should provide a status page for the status page.

------
romanovcode
Again, the GitHub is down for 5 minutes and everyone rushes the "who will post
first on HN" thing.

I don't get it.

~~~
zbentley
Fake Internet Points at stake + a lot of engineers sitting around with nothing
to do = spam HN.

------
coding123
I think it's back up.

------
rayboy1995
Status page apparently isn't automatic because it is yet to be updated.

~~~
dexterdog
Is that better or worse than stripe who updates their status page saying
something is wrong all of the time and then just changes it back without
noting any problem in the history? I get alerts on them 4-5x/week and only
maybe one of those winds up as a colored entry in the history.

...and sure enough, shortly after writing this it was 'down' for 10 minutes
and came back up with the status page saying nothing about it.

------
rplnt
While I kind of hate Atlassian products (bloated, horrible APIs), and
Bitbucket has some serious UX problems (Find), it's still better than have
your repos be offline.

There's not much that can go wrong on a single instance with a local database.
Have a failover in case the HW fails and you are good to go, and it's faster.
The pricing is not great, but perhaps that's something GitLab can solve.

But being hip and being lean (even if it costs more) is more important I
guess.

~~~
nik736
Did you actually notice that BitBucket was down yesterday for several hours
and still has huge issues?

~~~
zbentley
I think GP was referring to the fact that bitbucket.org-the-website is just
the Atlassian-hosted installation of BitBucket-the-Atlassian-product, which
you can also self-host and self-administrate.

It is payware, but reasonably affordable ($10/10 users, usually).

I've seen companies squeeze some (probably contract-breaching) huge numbers of
users out of the very small plans of Atlassian software, as well, usually via
insane multiple-people-using-same-account editing conventions.

~~~
stephenr
> to the fact that bitbucket.org-the-website is just the Atlassian-hosted
> installation of BitBucket-the-Atlassian-product, which you can also self-
> host and self-administrate.

Nope. They're completely separate things.

The hosted one they bought, and does mercurial too.

The self hosted one is what used to be called stash, and is git only afaik.

------
297mxq
Git blame --help

------
MrBingley
Should be the name of a movie.

"How will the world's hackers react when their source code goes offline?
GitHub Down, watch it now!"

~~~
JdeBP
We know how they react. It's the same reaction as when their chat rooms go
off-line. (-:

* [https://news.ycombinator.com/item?id=16109735](https://news.ycombinator.com/item?id=16109735)

------
scott_karana
Self-hosted vs SaaS is not a binary decision, and I hate seeing it argued like
that every time there's an outage.

What the hell happened to basic risk mitigation? Offsite backups, disaster
procedures, etc aren't new, or unexpected. You'd think they should be the
norm, if you're a working professional...

------
bigsassy
It's back up for me.

------
kuon
It's been only 7 minutes and I already noticed it (outside of HN), we rely too
much on github :P

~~~
clon
It was 8 minutes and the stroy was already running in HN, checked it before
status.github.com even. We rely too much on HN :P

~~~
kuon
I guess I'm going out for a walk in the park (without my phone), we rely too
much on computers :P

------
ryanpcmcquen
This is why I mirror all my GitHub repos with GitLab. Redundancy FTW.

------
lihes
That's the moment when you know why you're using a decentralized version
control system. You don't care about the central server.

~~~
neosavvy
Until you need to PR a change to get it to production.

~~~
scott_karana
If _you_ can't merge to master decentrally, that's not Git's fault.

------
d4nyll
GitHub just wants to show off their Unicorn artwork.

------
kubbity
I refreshed a page and boom ... . Check status, but that was not updated yet.
Friday is coming sooner! :)

------
nik736
Why do they even have a status page if it shows nothing and was updated "a day
ago"...

~~~
lvh
It’s been on Hacker News for two minutes. Your reply is one minute ago.
Perhaps you should check your SLA with GitHub, but presumably it doesn’t say
that 60 second delays is the same as the status page being effectively
useless.

Also, it was updated by the time you posted your reply...

------
ryanpcmcquen
Anyone know specifics? They handled that DDoS attack really well not too long
ago ...

------
cobbzilla
So happy I moved our dev team to Gitea earlier this year. Already paying
dividends!

------
TickleSteve
This is why you don't rely on third parties for your build dependencies!

------
bobcallme
This is why it pays to host your own source repositories. It is kind of
shocking that many people with the skills and means are too cheap to host
their own. I personally could not risk github deleting my repositories or (in
this case) going down for any length of time.

~~~
piaste
Though in an alternate universe where somebody's self-hosted server went down
in flames (possibly literally), one could just as easily say:

 _This is why it pays to have your own source repositories in the cloud. It is
kind of shocking that many people with the awareness and means are too cheap
to pay for a GitHub private repo. I personally could not risk a careless
sysadmin deleting my repositories or (in this case) going down for any length
of time._

The obvious answer is to have both, but smooth synchronization isn't always
easy or even available.

------
myspy
I don't need the free time right now :( Time to get a coffee.

------
curry_maker
Now we should start a "Developer-0" kind of thing.... the guy who screw up
that shit deserves a second chance... maybe a promotion...

~~~
rqwerty
what is this, facebook?!?

~~~
curry_maker
Gitlab did something like that not too long ago.

------
phyller
Looks like it is back up

------
sdruskat
Aaaand it's back.

------
pandemic_region
already looking forward to the post-mortem on this one :)

------
dx034
And we're back

------
jdmoreira
... and there are queues for all ping pong tables around the globe.

------
Narann
Is there any relation with bitbucket shutdown few days ago?

------
jacksmith21006
Where is GitHub hosted?

------
naiveai
Back up now!

------
nezza-_-
We centralized a decentralized version control system.

~~~
shawabawa3
$2 billion to the first person to draft a decentralised git coin WhitePaper

~~~
coding123
I think you just did.

~~~
drdaeman
No. We need a proper whitepaper explaining how to make GitHub decentralized
with blockchain tokenization using distributed smart-contracts. ;)

(Did I forget any important buzzword?)

~~~
zbentley
Did I forget any important buzzword?

Nah, but you could always add some Serverless and Lambada to make it more
Agile.

~~~
mcguire
" _Did I forget any important buzzword?_ "

BigDataDeepLearning.

~~~
WillReplyfFood
You got to ironically buzz to be cool today? Pretend you didnt get it, so all
those d __ __* who didnt get it can lecture you. We are deeply invested into
the idea of the Seagullarity!

------
ninjakeyboard
yep.

------
leeoniya
this PSA is quite unnecessary when a problem is universally known and
maximally severe.

~~~
jeppz
Well I didn't know, not everyone uses it on a daily basis.

~~~
leeoniya
so is it interesting for you to know that a service you don't use daily is
down for a couple hours? this random github outage happens basically every
other month, and is fixed in like 30 minutes. every time it's on HN frontpage.

people who use Github will already know.

~~~
jeppz
Yea ok I didn't know that either, I thought they were rather rare and worse
than that.

