

Github major service outage - forlorn
https://status.github.com/messages
https://status.github.com/
======
PommeDeTerre
I find it somewhat odd that, given git's emphasis on distributed version
control, we see so many git users have centralized themselves quite heavily on
GitHub.

Whenever GitHub runs into problems like this, it reminds me of when a team's
CVS or Subversion servers used to go down. It could be a pretty disruptive
occurrence, if it wasn't resolved quickly. While git can theoretically handle
this better, in practice the use of GitHub, for instance, renders git just as
susceptible.

~~~
spacemanaki
Even worse is other tools that rely on Github, like Homebrew (Mac OS X package
manager) which breaks in various silly ways when Github is down.

Anyone happen to know a way around this specifically? Trying to install some
stuff and brew just bails after getting 5xx from github.com.

~~~
js2
Clone the homebrew repo. Use a cronjob or Jenkins or some such to periodically
fetch updates to a server under your control. Modify homebrew and its setup
script to use your repo (it's been a while since I did this but the Github URL
for the homebrew repo used to be hard coded in one of homebrew's ruby
scripts). Whenever setting up a new machine, use the modified setup script.

Periodically you can merge upstream into your clone to get updated recipes.
(e.g., the aforementioned cronjob can mirror master from github to
remotes/upstream/master in your repo, then separately you merge that into your
master).

Your servers are now immune to github outages[1] and you can review recipe
updates before your servers update to those recipes.

[1] unless of course the recipes you're using are hosted on github. But you
could recursively mirror those sources too and modify the recipes as needed.

edit: I'm on a mobile device but I can provide further details the next time
I'm in front of a keyboard in case this wasn't clear.

~~~
spacemanaki
> unless of course the recipes you're using are hosted on github

Yeah, the problem I was having at the time was that Homebrew was trying to
fetch a tarball hosted on Github. I was just whinging though, that was the
only time I've ever hit that particular snag.

The system you propose would be great, but more work than switching to running
headless Linux VMs on OSX for most dev work ;) This is something I have been
drifting toward for a while now, using Vagrant.

------
jstreebin
<http://www.isitdownrightnow.com/github.com.html>

<https://status.github.com/messages>

~~~
samweinberg
Or for the HN friends without Javascript, <http://isup.me/github.com>.

------
Queue29
Least reliable service that I pay for by a long shot.

~~~
knowtheory
Must not pay for very many services?
<https://status.github.com/graphs/past_month>

~~~
driverdan
To be fair 99.77% uptime isn't very good.

~~~
jcape
That isn't fair at all.

It's 99.770% for a single month, immediately following a major event. If you
sampled yesterday (or tomorrow, assuming no further issues) it would be
higher. If you just look at today, it's at the much lower at 95.871%. If you
assume no availability issues for the last 12 months (not true, but the point
remains) then it's 99.981%. During an actual outage, availability was at the
unacceptable 0%.

Unfortunately they don't provide 12mo stats, which is what you typically want
if you're going to start calculating nines of availability.

~~~
josh2600
Hey, are you seriously defending 3 9's of uptime? That's abysmal.

Github, if they're honest about their 12 month uptime levels would be lucky to
be a single 9 service. Their uptime is Terrible with a capital T. But you know
what? Until there's something better everyone is going to keep using them,
right?

Great services with values that are hard to find become damn near
irreplaceable even with terrible uptime. This is an obvious place to compete;
if you made a github clone that simply stayed online you could win market
share during every downtime. However cloning github would not be trivial.

And therein lies the problem and the answer to why we accept their terrible
uptime levels. They give us something we can't get elsewhere: social coding
and easy centralization.

~~~
jcape
Are you seriously incapable of distinguishing "hang on, you are getting
numbers that look bad using statistical chicanery" from "Github = teh
awesome"?

Pointing out that someone whose point I agree with is using bad math as
evidence is not disagreeing with the point, it's asking that people who agree
with me behave like honest, civilized, human beings---I don't care that you've
already gone through the hassle of getting your pitchforks out of storage.

Speaking of which... your accusation that they are lying means that Github has
had nearly 37 days of total outage this year---that they're down for two and a
half hours a day, every day, for a year straight. And by honest, I assume you
mean "they are lying", as opposed to "they are using a different definition of
uptime than I would like." Naturally, you have some evidence for these claims,
right?

~~~
josh2600
Lying is a bit of a strong word. I think calling anything a single 9 service
should be taken in jest. It's pretty hard to be down almost 40 days and still
be in business.

Also, technically something with 98.9999% uptime would still be a single 9
service...

------
bennyg
I didn't even know it was down haha. I've been pushing to a private GitHub
repo all morning with no problems.

------
chris_wot
Database outage causing everything to go down?

~~~
Lord_DeathMatch
That and their wording suggests a single db server :/

~~~
noir_lord
They have a typical master/slave setup with memcached on the front.
<http://www.slideshare.net/err/inside-github>

~~~
nknighthb
Since they're using MySQL, is there a technical reason (as opposed to
historical/lack of time reason) they're not using Galera Cluster?

I'm in the process of migrating an existing datastore to MariaDB+Galera, and
so far it seems like everything I could hope for in a clustered RDBMS.

~~~
leedo__
Last I investigated Galera it lacked support for query caching. Over 50% of
our queries are cache hits, so it made it hard to justify using Galera over a
normal master+slave setup. However I could see it being useful for setups
where a single server can't handle the load (we average 300 queries/sec on a
single server with lots of room to spare.)

~~~
nknighthb
They still disable the query cache, but MySQL's query cache generally isn't
considered all that great a thing anyway, so few people care. You're better
off making judicious use of Redis or memcached.

The biggest win for Galera is high-availability that actually works with
minimal effort. (I've never experienced a high-availability solution not based
on multi-master/all-nodes-hot principles that didn't cause more problems than
it solved.)

They also claim some scalability wins at the front end, but I haven't really
tested that, and am content with the performance not being terrible.

~~~
blantonl
_but MySQL's query cache generally isn't considered all that great a thing
anyway_

You've never had to prime a query cache on a MySQL server, have you? :)

~~~
nknighthb
Of course not. I use a caching layer with lower overhead that doesn't
invalidate the entire cache when a single record changes. The query cache just
isn't competitive.

------
oaksagelew
The status page says that "Code Downloads" is "Normal". How do you get to the
downloads page for a given repo? Is there some canonical form for a URL for
repos? Maybe while they're fixing it we can still download code...

~~~
vjt
try:

    
    
      git://github.com/{user}/{repo}.git
    

or

    
    
      https://github.com/{user}/{repo}.git
    

anyway, not working right now for me (Italy)

EDIT: working, give it 30 seconds to start.

~~~
oaksagelew
Nope, no joy - 30 seconds or otherwise.

------
qhoc
We mainly used github and one of their last updates, our entire repository
turned to blank. Nada, no file, no wiki, nothing... We emailed support and
tweeted their account. After one week, someone replied asking "is there
problem still there?" I thought that was funny in term of how immature this
company is. They eventually fixed it after like 2 weeks by restoring the
backup database.

Anyways, Github is nice. Use it with caution. It cannot be full blown
Enterprise ready grade service yet.

------
mfonda
Looking through the past few months of messages on
<http://status.github.com/messages>, I'm really quite surprised at how many
service outages, major or minor, are reported. It seems something goes wrong
more days than not.

------
bdcravens
A number of ticketing systems, like Unfuddle and Assembla, include free git
repos. Personally I prefer Github, but if you're already using such a system,
might as well utilize its git hosting (for a backup if nothing else)

------
valokafor
Same for me from San Diego, CA. I am unable to push and unable to access the
main page. Getting "This page is taking way too long to load."

------
misiti3780
im getting a 500 (from NYC) on the main page + i cant push to remote repos

~~~
noir_lord
From here (NE UK) _everything_ is down.

Typically I was in the middle of setting up a development environment on a new
machine which has a lot of composer dependencies....

Oh well will have to go drink coffee in the sporadic sunshine :).

------
mratzloff
It's back up for me.

------
dpweb
Why Unicorn Why!

~~~
revskill
So, what's better than Unicorn ?

~~~
terhechte
Two unicorns.

