Hacker News new | comments | show | ask | jobs | submit login
Github service outage (github.com)
44 points by manojlds 1610 days ago | hide | past | web | 34 comments | favorite

I posted this as a separate thread but maybe I'll ask here. We're dead in the water due to this latest outage. They seem to hit weekly now. I'm considering switching to another provider today.

How does Bitbucket stack up in terms of reliability? Features are sort of irrelevant compared to recurring downtime.

Since git is a distributed version control system, why not create replicas of your important repos so in the event of an outage you have alternatives? Anything with a working ssh can be used as a host, it doesn't have to be fancy. GitHub's hooks make it easy to write things that can react to push notifications.

BitBucket has made strides in improving their front-end app, but it's still far, far behind GitHub in terms of features and integration.

Yes, it's distributed

I use mr to keep my multiple repositories in sync (And they all laugh at me!) across hosts in places where git submodule seems inappropriate and there's no android 'repo' manifest.

If you're paying, "but it's distributed" does not ease the pain. You didn't buy a GitHub account because of the 'Git' -- you can get that for free! You bought it because of the 'Hub' and if you made it your hub, now you're dead in the water, scrambling to find another place to coordinate.

At least I guess that's the point of this article. Not sure. Doesn't look down to me, and I'm not a paying user.

*EDIT: I see it's only some users/repositories that are down, and it's not a general outage. I wonder if it's only paying users.

Upvote for mr. This is a great tool that I just started using more seriously (initially I used it to automate my linux box setups, but it's useful on a day to day basis as well).

Whatis 'mr'?

My path:

- Search github for 'mr'

- See: https://github.com/joeyh/mr

- joeyh is the creator of git-annex, and github says 'mr' is in Perl, which is one of joeyh's favored languages. Assume he's the creator of 'mr'.

- Go to joeyh's site: http://joeyh.name

- Browse to: http://joeyh.name/code/mr/

or more simply, google for [mr git] and go directly to http://joeyh.name/code/mr/

> How does Bitbucket stack up in terms of reliability?

Used to be very unreliable. It's gotten better, but there are still issues once in a while. There was a big downtime mid-september (a good 3h of nothing): http://blog.bitbucket.org/2012/09/19/post-mortem-on-our-avai... and there were a few issues e.g. yesterday due to a screwy deploy.

edit: checking the recent twitter stream, a few issues on the 28th, degraded performances on the 16th, broken newsfeeds on the 11th, brief network blip on the 5th, maintenance on September 29th, severe crash on September 19th, deploy issue on September 17th. Not sure you'll get much more reliability out of it compared to github. Though you could use one as a backup for the other one.

Weekly? My experience must be different. This is definitely the worst one I've noticed in two years of use.. but I think I've maybe noticed issues once every 6-8 months or so. Not weekly.

It's closer to weekly than biennial. At GitHub's size, there are minor hiccups that affect a small number of users more often than any one user will see.

My company (https://circleci.com) uses GitHub OAuth and checks out code as its core functionality. Believe me, I get emails every time GitHub has an issue for 30 seconds.

Yea, I could see that volume from a SaaS service.

As a user though, I'd say github has service issues every month. I think the last one was caused by a massive ddos or an exploded db and faulty fail over.

They're pretty good a keeping a sinking ship floating and as long as `git` doesn't go down, it's not the worst thing in the world.

It's prolly terrible for you service guys.

we're currently evaluating CI solutions and I've been meaning to give Circle a look.

Am I overlooking a tour (or just a single screenshot?) on your website?

Based on skimming your docs it seems like this could be a much more approachable solution than your competitors.. and based on the care taken in the designing your website/brand I'm guessing you invested in making a good UI. show it off! :)

edit: before someone suggests 'just sign up for the trial and look' -- as mentioned, Circle's authentication is (understandably) via GitHub -- that means authorizing them to write to my private repositories without seeing anything about the product.

Hey mh-, I'm the other founder. You're not overlooking a tour, there isn't one at the moment.

Currently, our UI is boring. We (and most CI services) do a basic list-of-builds UI. I don't think that's good enough, but we've been short staffed until recently due to fund-raising, but now we're finally getting a chance to think about how best to provide the best user experience around that.

The things that are amazing about Circle now are outside the UI. For example, we can auto-parallelize long builds to run in a fraction of the time. Your tests run blazingly fast on our boxes (we've spent months ensuring we're the fastest service by a long way). We do personalized notifications of failures (cause you don't care what your colleague does on his branch). So please don't judge us on how pretty our pictures are :)

FYI: we hate the "write to my private repos" thing too, but with GitHub's permissions model there isn't a better alternative (there are alternatives, they're all bad).

Hi Paul, thanks for the reply!

I wasn't judging you guys on the 'pictures' at all. :) Was just giving some candid feedback from my experience. I actually signed up write after I wrote that, and was going to post again once I'd had a chance to try it.

I did get to try it briefly and I actually think the UI, while simple, is better than the other (albeit much more complex) solutions I've been looking at.

>FYI: we hate the "write to my private repos" thing too, but with GitHub's permissions model there isn't a better alternative (there are alternatives, they're all bad).

IMO, this would be a big improvement and is doable: let me sign up (still via GitHub auth) without asking for the repo or user scopes. Then have a call-to-action that upgrades the token, as part of the guided welcome into Manage Repos. [I realize this could hurt your funnel, perhaps offer this path buried in the FAQ somewhere if that's a concern]

If you pass nothing for the scope param to the GitHub API, that gives you the authentication piece with read-only access to the user's public stuff. Then just add a "Hey, we can only see your public repos by default - click here [..]" that sends them into the web flow again with the additional scopes, to upgrade your auth token.

Also, while I'm giving unsolicited advice :) - it may be obvious, but letting the user know that when they follow a repo (and just doing this on the first repo they use would be fine) that you're going to add an SSH key would probably be a good addition. Since adding a deploy key sends an email to all admins of the organization, some users might want to give people a heads-up they're trying out Circle.

. . .

Anyway, Circle looks great already and I'm eager to see what other stuff you guys have in store. I'll certainly be using it for some of my projects.

Thanks for the feedback! There's definitely a lot of ways we can do this, and each has tradeoffs. One tradeoff that came up when we tried to switch to public-only authentication first is that the login button needed to have different scopes - and apparently a lot of people try to sign up via that button.

I agree that we need more warning with the SSH keys. I'll try to figure out where to put that - we used to warn in a previous iteration, but we're moving very quickly and some things get lost sometimes (especially with A/B tests). We're doing a lot of work to provide user education around what they should expect from Circle, so that's definitely a must-have. Thanks!

Agreed. This is the first major outage I've noticed as a user of github in several years. I have my grievances with GitHub, but the pull requests, issues and organization systems are really useful for me to have in a team setting, so I couldn't really see myself moving to another code management solution -- what would my alternatives be? JIRA? Lighthouse? And that doesn't even touch on the social aspect -- nothing is quite like github in terms of community vibrancy and activity. I've found so much useful information and documentation scattered through issues and forks that are all through github that I couldn't imagine finding otherwise -- or at least, it would be more difficult to find elsewhere. I really like being able to associate commits to issues via commit message. Still, this kind of outage definitely makes me thinking about having a some kind of a mirror long term -- haven't been able to use my issues all day, and that's kind of frustrating.

If you want more control I recommend the open source gitlabhq (http://gitlabhq.com/) - we run it for our repos, plus you can be in control of number of repos allowed per user. I consistently create repos for proof of concept or side projects that I wouldn't normally be able to do in github unless I wanted them all public. I don't particularly want people seeing unfinished or unpolished code

Gitlabhq has worked well for us, and it's open source.

The funny thing is, you can't install GitLab right now because their github repository is down :)

Ouch, this bit "Features are sort of irrelevant compared to recurring downtime." is key. If you're going to build a network based business, you have to invest in being operationally nimble. Just because DevOps isn't 'sexy' doesn't mean it isn't important :-) (and no I'm not saying that just because I am an operations guy, but I feel the pain when looking for DevOps engineers)

I also feel GitHub's pain. We're constantly dealing with things that come in over the transom at Blekko and getting a constant stream of sales guys and gals trying to tell us how their product will solve all our issues, and as I tell folks, if you don't know how to solve the problem how is it that you trust this solution being offered is sufficient? I've known people who will respond, "Well if it isn't we'll just beat on them until it is!" but that doesn't help the 'gurgeous's of the world who are trying to use your service while your vendor it getting beaten about the head and shoulders with a tuna.

I've advised a couple of startup founders who had under 'scaling' in their proposals "EC2/S3 totally scales" that while it was true, lacquer oil paints create masterpiece paintings, except if you don't know why a masterpiece is a masterpiece, just having the paint won't get you there.

I agree completely. I've had to deal with scaling challenges before, most memorably at Urbanspoon. I'm not judging anyone. This stuff is just hard.

As an entrepreneur/engineer my day is already jam packed. We have a tiny, full-stack engineering team here. I really just don't have time to babysit services that suffer from significant downtime. Certainly I could mitigate the issue by distributing our repos to multiple places, self-hosting, etc. but I'd prefer to just use something reliable.

I use and can recommend bitbucket. I can recall one outage in the last couple of months (and that was website only, ssh push and pull were working just fine).

Also, have in mind that bitbucket is much smaller than github so they might not have their scaling/stability issues yet. Their status page is http://status.bitbucket.org/

That's the one thing keeping me off bitbucket is the fact that nobody else is there. Admittedly their pricing is better (especially wrt. private repos, something i'm still not ready to shell out cash for), but the interface seems to be a Github ripoff.

Github is quickly becoming the social network for coding - there's no reason to use an also-ran copycat who lacks the "social" part. (And this isn't to disparage Atlassian, they're one of my favorite companies!)

They're both good. My open source repos are on Github. My private (thus client-related) repos are on BitBucket. I've been pretty happy with this configuration.

Perhaps it helps that I know a few other companies and developers locally using BitBucket for private repos.

Incidentally, reliability and price are both valid reasons to "use an also-ran copycat", in my personal opinion, if that also-ran copycat is pretty damn good.

I guess the bitbucket's UI is a ripoff... the same way that OpenOffice's UI is ripoff of MS Word. This argument has been pulled so many times before and it's not relevant in this topic which is github's often downtimes.

No, but it's hardly a point in their favor, either. Anyways, OO looks like Word because the professional-compatible text editor UX is generall pretty static Toolbars, buttons, etc.

BB on the other hand is just missing some curved buttons on their UI - you have to admit the resemblance is uncanny.

I get the same gut feeling from Bitbucket as I would from a chinese knockoff of a well known product. Sure it might even be better in some ways but it still feels like a cheap imitation of the original.

Call me crazy :)

Re: Bitbucket. I have repos at both. And have a script that runs hourly to create local mirrors of my git repos. I see connection failures often to github (perhaps very intermittent), but I never do for BitBucket.

It was my cron script erroring on a github backup today that made me notice the outage.

I realize that reliability is important; GitHub should improve here.

However, my experience was that avoiding dependencies on an external server was the natural way of working with Git-based deploys. Our deploy scripts pushed the current branch from the local repository to the servers directly. This meant that the version tested locally, whether a hotfix on a developers box or the last synced/tested version on the CI server, was the exact version deployed. Granted, the scripts could have specified the exact revision to fetch from an external server, but since the deploy source and target were already communicating, why not just `git push` and a commit hook rather than various `ssh git ...` steps? It won't scale to hundreds of boxes, but by that point you surely have a local dist source for much more than just Git repositories.

Bitbucket works quite reliably nowdays. I dont remember any outage in recent months - the service improved greatly.

> We're dead in the water due to this latest outage.

Why is that? Git is distributed, so while it might be a bit inconvenient, it shouldn't be difficult to create an alternative solution, especially if it only needs to be temporary.

We use git because I am part of a distributed and often disconnected work force.

GitHub seems to go down way too often nowadays.

Working on a school project with a friend, and we've been unable to push for several hours.

Funny part is, we were configuring his MacBook with his GitHub credentials, and initially thought we had some sort of error uploading his SSH keys. Turns out it was a service outage.

This is unfortunate. I realize that it is distributed, but when you're in a major push it really sucks to have to scramble around to setup another host to get working again.

I love their service, but they really need to get their reliability issues figured out.

I have bad luck, this happens every time I want to note an idea into a task...

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact