
GitHub's down? - 9point6
Getting the unicorn on any pageload
======
brudgers
I read about Github being down and know that it sucks for many people. What I
find interesting is thinking about how quickly so many installations of
distributed version control adopted a _de facto_ central server. Github is a
platform that should not need a particularly high uptime from the point of
view of any one user.

But of course it does because so much of its utility is that it substitutes
for a lower level partition strategy for source code version control at the
level of the individual user. By which I mean that the whole point of
distributed version control is to have sufficient resources for working at
hand regardless of what is happening remotely as the default condition.

Git is designed to be available and partition tolerant. Using it with the
expectation of consistency is a mistake and Git requires manual intervention
on the part of users to even get good-enough-consistency. Github can't change
that.

None of which changes makes Github being down not suck for the people for whom
it sucks.

~~~
coherentpony
> Github is a platform that should not need a particularly high uptime from
> the point of view of any one user.

This is totally false. GitHub doesn't just provide a git server, it also
provides ticketing and project management. When GitHub is down, projects that
rely heavily on, for example, commenting on issues grind to a halt.

Yes, you don't need GitHub to be up 99.999% of the time to do a `git push`
every once in a while, but there are other projects that use other features of
GitHub besides the git server aspect.

~~~
tenfingers
And it's a shame this is the case, though.

I do publish projects with github because of the popularity of the platform,
but I really don't approve github pull requests, and I also don't like github
over-simplified issues. Together, these two features are probably what most
projects depend on.

Github pull requests are definitely inferior to the patch-by-mail approach for
a conversation within a closed group of people. It's clicky and looks
friendly, but you don't have the patch to see in the mail notification (you
have to be online), and again it's more convoluted to edit/apply using vanilla
git.

Many people don't know, but git makes patch-by-mail handling extremely
convenient to handle. In fact, it's more convenient to handle than using the
several github command-line interfaces I tried(!). You submit the patch, have
a conversation by regular mail, and then either edit the patch or merge it.
Everything is always right there. Using github you have several side effects:
people often add comments to specific lines in the patch, which will looks
like garbage in the generated github notification. People will also naturally
edit the comments, which won't generate notifications. I cannot count how many
times I have to re-read comments because I had the feeling I missed something
(and indeed, some comments were edited). This is bad. It shouldn't be allowed.
It cannot happen with email.

Github has the advantage of making the conversation visible. But if you have a
developer mailing list for your project, git and plain, regular patches _do_
work really well. It might take a bit more to get used to, but it pays back.

Issues are a different problem, since I still feel the distributed solutions
are inferior. I experimented a lot with bugs-everywhere and simple defects
(sd), two distributed bug trackers that store the issues in the repository
itself. Both are nice, and have distinctive features from each other which I'm
not going to discuss much. The main thing is that 'sd', at some point, even
supported to sync issues with github, which allowed to have best of both
worlds. Unfortunately, it's lacking maintenance at the time, but it would
solve ticketing in a distributed way. It looks like there's not much interest
in this, as _both_ projects are definitely not as active as the many available
github clients which do 1/10 of what these projects offer.

Please, look into these two projects.

~~~
coherentpony
Thanks for the distributed issue tracker mentions; I didn't know they existed.

------
runn1ng
[https://status.github.com/messages](https://status.github.com/messages)

>Today

>11:32 UTC "We're seeing high error rates on github.com and are
investigating".

>11:40 UTC "We're doing emergency maintenance to recover the site".

~~~
kjjw
Just noticed an interesting design decision on that page. An 'everything OK'
message once a day. One way to push previous downtime below the fold of the
page.

~~~
DCoder
It's also a way to show "yes, this status check is still reliably doing its
job, it hasn't crashed".

~~~
kjjw
OK I could take that as a reason.

------
jballanc
For everyone who is currently loosing productivity, perhaps your time might be
well spent reviewing `git daemon`: [http://git-scm.com/book/en/v2/Git-on-the-
Server-Git-Daemon](http://git-scm.com/book/en/v2/Git-on-the-Server-Git-Daemon)

~~~
kjjw
OK I get what you're saying, but honestly, who is really 'losing
productivity'? I suspect the ones saying they are in fact just getting on with
other tasks they can do because they are, despite github, still using
distributed version control and do have access to and the ability to commit to
code repositories. You don't need to host your own git daemon or github clone
to get that functionality.

I don't use github because it gives me a git daemon. I use it because the
interface allows me interact with non-technical people when dealing with code
and resources. I use github because, even though it is down right now, my
stuff is backed up, I don't have to deal with the nitty gritty of ensuring
backups succeed, I don't have to deal with the system when it goes down -
someone else does that for me. Hence I would never want to run a local github
clone.

~~~
kmfrk
It kinda sucks when you rely on continuous integration, though.

~~~
ad-hominem
Can't you just run your tests locally?

------
sp4ke
Funny, I switched our projects to Gitlab a week ago and told the team: Because
we should not trust a centrealized platform to keep our code decentralized.

~~~
danielsamuels
So you switched from one central platform to another singular central
platform? Am I missing something here?

~~~
gibsonje
This is what I don't get. I trust github as a central platform more than
hosting my own gitlab in the cloud.

There's a lot of people spreading a holier than thou attitude in this thread.

~~~
mrsirduke
GitHub is a much larger target, and has many, many more moving parts than your
average Gitlab on, say, a Linode virtual server.

Github will probably have more downtime which you can do noting about, than
your selfhosted gitlab, which will probably fail because you upgraded
something, or tinkered with a configuration.

It's a numbers game. They have more people monitoring, they spend way more
money than you do on uptime/services, but such a service doesn't get more
stable as it grows.

I trust myself way more than I trust GitHub or other hosted/cloud services
because I cannot affect/help them in any way when stuff happens.

Ultimately, I know this, I accept this, and I happily use GitHub, Bitbucket
and such all the time.

~~~
hn_user2
>Github will probably have more downtime which you can do noting about, than
your selfhosted gitlab, which will probably fail because you upgraded
something, or tinkered with a configuration.

Personally I don't think you will see less downtime on your own server. The
difference is, that when it goes down, there is something you can do about it,
so you get busy fixing it.

When github is down we get to all get online and talk about our woes together.

The outages are usually pretty short. Go get a coffee and come back, or grab
the team and go play a round of bowling. That has to be better than trying to
manage and keep secure our own repos.

~~~
IE6
This - the last thing I want to do is be on the line when something happens to
our git server and no one can work because our solution is my own custom one-
off.

------
AdrianRossouw
the d in github stands for distributed

~~~
elcct
how did you rotate b?

~~~
s_kilk
The joke is there is no 'd' in github, it's not distributed.

~~~
darylteo
The joke is a reference to this old gem

[http://www.bash.org/?330261](http://www.bash.org/?330261)

~~~
s_kilk
Ah, a true classic :)

------
bvrlt
That's when you notice how dependent you can be on GitHub.

You would think you can still develop an iOS app when GitHub is down, but a
simple `pod install` requires access to repos hosted on GitHub.

------
sbt
Time for a sword fight!

~~~
setrofim_
For those confused by this: [https://xkcd.com/303/](https://xkcd.com/303/)

------
mrmondo
So glad we don't rely on GitHub for our deployments anymore, we haven't had an
outage to git in the past year yet it seems GitHub goes down more often than
people realise. Obviously DDOS attacks are nearly impossible to mitigate for
anyone but it should serve as a reminder of companies dependencies on external
services.

Somewhat related I highly recommend both Gogs and Gitlab for internal git
hosting.

------
kbwt
Yes, response times seem to have spiked:
[https://status.github.com/](https://status.github.com/)

Could be another DDoS?

\---

Update 11:32 UTC "We're seeing high error rates on github.com and are
investigating".

------
infectedio
Just use [https://git.directory/](https://git.directory/) :D

~~~
sytse
Cool to see you set up a GitLab instance for others to use, I had not seed
that TLD before.

~~~
infectedio
Thanks for GitLab!

~~~
sytse
You're very welcome, thanks for spreading the word.

------
smoyer
To me, that looks like an angry unicorn ... does that mean it's another DDoS?

It would also be interesting to track loss of productivity in the software
engineering professions when GitHub is down. Is this a single point of failure
for your company?

~~~
sbarre
Not just for a company, but for developer productivity in general,
professional or not...

Think of all the package managers that pull straight from Github repos as part
of their installation processes..

~~~
smoyer
Exactly ... we've got Ansible scripts that do that. One of the current Java
rockstars (not my description), Adam Bien, recommends pulling the source code
for all third-party dependencies to build them locally and put them in a local
repository. I suppose you can push directly back upstream to avoid having an
extra step.

~~~
flipp3r
If you're using Java most of the build tools use the Maven repositories, for
which you can set up a local mirror too. The company i work for runs a
Sonatype Nexus so we're not really affected by others' downtime unless we need
stuff that we hadn't ever used before.

~~~
smoyer
Yes, for our Java projects we have a Nexus repository which mirrors everything
locally. It doesn't protect us from someone deleting a project ... then we're
stuck on the last available version without a way to address issues, but it's
worked pretty well so far.

------
ff_
Up again!

11:54 UTC "We've finished emergency maintenance and are monitoring closely".

------
Already__Taken
I was uploading a new profile picture sorry everyone.

------
colinbartlett
A few projects can help you monitor this:

Octobot iOS app: [http://octobotapp.com](http://octobotapp.com)

And my own service, StatusGator:
[https://statusgator.io](https://statusgator.io)

------
delinka
Probably people trying to clone curvytron because curvytron isn't currently
playable at its own site.

It's like an unintentional pub crawl-- 400 people show up to party, empty the
pub of booze, move on to the next pub that hasn't enough booze...

------
_1284267234
Github is down again! And Slack too. The only page of slack.com that is
available is their status page that says: "All's good under the hood, boss!
Slack is up and running normally". UTC/GMT 19:06

------
winterbe
Every time GitHub is down I'm peering at Upsource:

[https://www.jetbrains.com/upsource/](https://www.jetbrains.com/upsource/)

~~~
merb
UpSource IS not a repository. It is just a Viewer for your repository. You
still need to host everything yourself.

As you can see here:

>
> [https://www.jetbrains.com/upsource/roadmap/](https://www.jetbrains.com/upsource/roadmap/)

Git hosting is on the roadmap, not ready yet. Also Upsource takes a shitload
of memory.

------
Macha
SE yesterday, gh today. Bad week for dev productivity.

~~~
philippnagel
SE?

~~~
crylics
Stack Exchange.

------
cgabios
Seems to be partly up, hurray! (maybe)

Update: "We've finished emergency maintenance and are monitoring closely".

------
tragic
I'm just getting unicorns everywhere. Sprint review in an hour. Damn, I love
wednesdays.

------
cgabios
Hopefully the responding on-call engineers aren't tired folks but folks in
other timezones. The reason that the risk of compounding the situation is
lessened, because trying to fix important shit when tired is like driving
without a seatbelt.

Update: seems to be back, so it doesn't matter unless someone's having an
early morning.

~~~
zymhan
Probably? This is a very random concern to have, why would you think that
Github has problematic on-call procedures?

------
_ikke_
11:40 UTC "We're doing emergency maintenance to recover the site".

------
rghose
Damn you github. Just tried making my first push in my new company and boom!

 _wasn;t me_

------
mohsinhijazee
Had to deploy an important service. And github is down.

~~~
andygmb
perhaps a good time to reconsider depending on a third party for your
important services to deploy?

~~~
mohsinhijazee
Yes, but these decisions aren't under me plus setting up a github kind of
thing is way too time consuming with all the features that we're used to now.

~~~
jballanc
A GitLab droplet on Digital Ocean should run you only $10/month and literally
sets up in ~10 min.

~~~
netheril96
What makes the servers of digital ocean safer than github?

~~~
jballanc
They're not GitHub. i.e. The chance of GitHub _and_ your GitLab instance on DO
going down simultaneously is very slim. Spend an extra $10/month and you can
set up another DO server in another data-center, and reduce the odds of
simultaneous failure even further.

------
ivan_ah
The 'one' day that I wake up at 7AM to get some quality caffeinated work done,
and I can't pull.

It's all good though, I still have loco host to keep me busy until their
resolve the issue.

------
dujiulun2006
OK it's back.

------
sidcool
It's back to normal. Back to work guys!!

------
ksk
I thought git was all about decentralization?

------
ruban
IS there any opensource private github?

~~~
scottlinux
[https://about.gitlab.com/](https://about.gitlab.com/)

------
_1284267234
Slack is up. And github too.

------
jbernardo95
Back up !

------
paufr
it's weird

------
alelepore
it's up

------
semirami
it's back

------
semirami
it's back now

------
liquidmetal
Not again.

------
johannesboyne
yep

------
elcct
It wasn't me

------
hoangddt
The site is live now

------
miloc_c
Yes it is!

------
alekfeel
:)

------
liamcode
looks like its time to go for lunch

------
bwghughes
Yep. Down for maintenance.

------
erikzaadi
So the DDOS is basically all of us accessing status.github.com right?

------
davidgerard
Back now! Orange octocat, "We've finished emergency maintenance and are
monitoring closely".

