
GitHub and Rails: You have let us all down. - chrisacky
http://chrisacky.posterous.com/github-you-have-let-us-all-down
======
trotsky
Jesus, HN goes from zero to lynch mob faster than reddit these days.

Guy drops a zero day on a major service provider, guy gets his account
suspended (temporarily, it turns out). In what possible world is disabling an
account that has recently exploited your live product in a very visible way
not ok? Remember, you don't have a chance to call a meeting with the C level
guys and your community manager - you're one or two guys responding on a
weekend.

The rest of the "oh my god the sky is falling" drivel about how terrible a bug
it could have been and how they should never have had such a vulnerable bug in
the first place is even worse. Security bugs are fuckups by nature - nobody
sat and said well shit I was going to code this wrong but since it might allow
a lot of access I won't. In terms of OH SHIT bugs this is actually rather
small - I'm sure github's live infrastructure has been open to lower level
remote execution vulnerabilities over the years - newsflash: we all have been.
Getting user or superuser or db admin is going to almost certainly be a lot
worse than an application authentication level vulnerability.

You say none of that matters because it's such an obvious bug and people have
known not to do that kind of thing for years? Say hello to our old friends
"buffer overflow" & "use after free" - still grabbing msft aapl & goog after
all these years.

TL;DR - stop acting like children.

~~~
maratd
> and how they should never have had such a vulnerable bug in the first place
> is even worse.

Bugs happen. Even stupid oh-my-god-i-can't-believe-i-did-that bugs happen. And
they happen to the best of us.

 _However_ , when someone reports a vulnerability about my code to _me_ or I
discover a problem myself, the _very_ first thing I do is break out the grep.
I grep the shit out of my code. Because I am a human being. I am a creature of
habit. And if I screwed up in one place, I promise you, I did it in other
places too.

The problem? Github didn't do that. At least, that's the impression from the
information coming out. The guy reported the issue on Friday, they fixed that
specific instance of the issue ... and it remained a problem in other places.
That is unacceptable and unprofessional. They should have burned the midnight
oil and made sure the same problem wasn't prevalent in other parts of the
code.

Having said that, I am a loyal Github client and will remain so. Every service
provider I use gets a once-a-year-screw-up credit. Github just used theirs.
Switching because of one incident is premature and will be sure to cause
regrets.

~~~
trotsky
_They should have burned the midnight oil and made sure the same problem
wasn't prevalent in other parts of the code._

I understand that's the feeling here, but it's unrealistic. I've reported
dozens of bugs to shops that ranged in size from 1 to borg. You simply never
see a whole set of bugs fixed and pushed live over a weekend. Not even close.
Exactly what company have you seen set this standard for professional? The
smallest amount of time between notification and public disclosure I've seen
get called responsible disclosure is a month. Many, many, many enterprises
take six months or longer to push fixes (which is far too long but that's
another whole discussion)

~~~
einhverfr
I agree to a point.

However, if such a problem was reported to LedgerSMB here is how we handle it:

1) Scope out the problem. What's affected? Are other related open source
projects affected? How bad is it? This itself can take a bit of time. We do
not rush this because we don't want a full disclosure when it happens to bring
other problems into the fore.

2) Within a few days we let the reporter know what we have found and give them
a chance to offer feedback.

3) Then we get everyone on the core team together and talk solutions. Once we
implement it, we test and release a patch. Two weeks after the patch is
released, we release a full disclosure along with a hat tip to the one who
discovered it.

The whole process takes time. But the fact is that it's generally better to
get a full fix out in two weeks than a partial fix out tomorrow.

So no, don't burn the midnight oil. More speed, less haste.

And this whole thing really doesn't look good for Github.

~~~
technoweenie
We didn't have the luxury of time. We were going back and forth with him over
the weekend, and he pushed the commit 2 days after the original report. I
admit, I was having some issues understanding his vague reports and went down
a different rabbit hole early Sunday morning.

I definitely wanted to do a more careful audit and investigation, but my hands
were tied.

------
dkrich
Give me an F'in break. I understand security is not something to take lightly,
but no system is infallible. There was an oversight, plain and simple. It is
debatable whether the Github/Rails Core Team was too lax, but I for one am
tired of hearing developers whine and make a witch trial out of groups of
developers that have moved the development community forward several huge
steps just to make themselves sound smart or feel fulfilled. If you're such a
hero, why didn't you discover this loophole? Please stop writing provocative
statements and behaving as if the sky is falling on top of your head and the
very fiber of our being is at stake. An open source language and a website
written in that language were shown to have a flaw. Which has since been
fixed. I hate developers who like to sound smart at the expense of somebody
else. Get over yourself.

I also don't see how you can blast Github for its oversight of this issue but
then defend the "hundreds of thousands" of sites that use Rails. Aren't they
as culpable? Oh, I suppose Github is held to a higher standard than the rest
of the dependent apps. If this is Github's fault, then it is also every other
developer's fault who doesn't by default disable mass-assignment of
attributes.

~~~
refulgentis
You're conflating two issues here. He's arguing that the Rails team was
ignoring an important issue by noting it was an easy end-user fix, and GitHub
overreacted by suspending him after he tried several times to bring it to
their attention, and then grossly mislead their user base as to the extent of
the issue (which sounds like a really fundamental security issue that any
professional Rails developer should know how to avoid) and how they discovered
it (i.e. they didn't discover it, they had to be shown it). You're responding
as if GitHub was simply being attacked for not fixing the issue.

~~~
stock_toaster
Did he notify github directly, or did he notify the rails team in a github-
issue in the rails project?

If he didn't actually contact github directly and just assumed they would see
the rails issue on a weekend, then I wouldn't exactly call it 'notice'.

If he did submit it to github via the proper channels before taking action,
and nothing was done within a reasonable timeframe (eg. not just an hour or
two on a sunday), then what he did would make a bit more sense.

~~~
hythloday
He notified Github "last Friday", according to them:

[https://github.com/blog/1068-public-key-security-
vulnerabili...](https://github.com/blog/1068-public-key-security-
vulnerability-and-mitigation#comment-17297)

~~~
stock_toaster
> His previous report was fixed last friday

Makes it sound like a separate issue.

edit: looks like github clarified <https://github.com/blog/1069-responsible-
disclosure-policy>

~~~
hythloday
Same exploit, different endpoint. Github agree they "should have...immediately
looked for related issues":

[https://twitter.com/#!/technoweenie/status/17645071550868684...](https://twitter.com/#!/technoweenie/status/176450715508686848)

...though as they've now unbanned his account (good on them), it's somewhat
moot.

~~~
stock_toaster
Aha. Thanks for the extra info.

------
epistasis
I have lost all trust in GitHub, and not because of the vulnerability, but
because of their response. With their suspension of hamakov's account and
deceptive blog post about the extent of the hole, GitHub has guaranteed that
they won't be the first to know about the next vulnerability (and there's
always another).

I've downgraded my paid account to a free account, and won't keep any non-
public data on GitHub in the future. I had a similar response with my (non-
paid) DropBox account. I guess I didn't rationally evaluate cloud resources,
and have trusted far too many people.

~~~
technoweenie
We suspended it after fixing the bug to make sure he didn't retain access to
something he shouldn't. We rarely do this, but he wasn't upfront with
everything he was doing on the site like people that disclose vulnerabilities
responsibly.

~~~
niyazpk
The sad part is that the guy was your biggest fan:
<http://homakov.blogspot.in/2011/07/octocat-tattoo.html>

~~~
technoweenie
It's a fake.

~~~
reustle
Citation needed

~~~
openbear
Not sure if technoweenie was trolling or not, but Homakov said on Twitter
"That tattoo is kind of fake made with henna".

<http://twitter.com/#!/homakov/status/176476394455437312>

 _"Thank you all,sweethearts! For support, and shit too. One more thing to
clarify. That tattoo is kind of fake made with henna. eat vegetables"_

------
elithrar
> When the large portion of the technical world all depends on a single
> service, and that service is vulnerable to a variety of attacks, that makes
> _anyone_ who consumes these services also vulnerable.

I don't mean to diminish the severity of this exploit, and the impact it
has/could have had if left unchecked.

BUT, isn't one of the biggest perks of Git the fact that it's a _distributed_
SCM? It's not a service where you must trust all of your data with the one
provider, who might go belly up at any point and take it with them.

Yes, GitHub provides some fantastic social features and helps with community
involvement through these features, but if you are dependant on a "single
service" and you're using a DSCM/DVCS, you should probably look at a few
alternatives to reduce that dependancy.

~~~
augustl
I agree wholeheartedly. I always facepalm when people aren't able to work
because the internet is down (read: no github access). Just start an SSH
server, netcat some keys around (or use an USB stick), and get back to work
again :)

Transmitting files via Google's email servers from two computers on the same
network in the same office in the same room, especially if you're in europe
(ridicilously far away from GMail's servers) instead of directly between the
computers on the local network is one of my main pet peeves..

~~~
ktizo
LANs are broken. Why is it that we have usable tools for connecting halfway
round the world, but find it massively hard to coordinate ourselves across a
small office when the internet is down. It is total madness.

~~~
augustl
I guess it's just about familiarity. We do have the tools (mdns/avahi/bonjour,
tcp/netcat, ...). But the internet is usually up, so we don't need to learn
how to set up our own local servers, and there's no punishment for
transmitting a file across the atlantic when it could have been transferred to
another room in the same building and back again.

~~~
ktizo
The fact that there is not persistent caching of mission dependent code and
data as a fundamental commercial standard shows how badly we as a species in
general can judge risk, even when the stakes are really high.

~~~
icebraining
How so? Each git user has a local copy of the whole repository.

~~~
ktizo
Git is not a fundamental commercial standard, I was winging about the state of
industry in general.

------
ericflo
I fail to see what GitHub did wrong here. They were attacked, they suspended
the account doing the hacking, and they fixed the problem. Then, they blogged
about it, explaining in detail what happened. Apparently they weren't quite
reverent enough for the person who wrote this article.

~~~
mquander
Egor is a human whose motivations were totally obvious and whose actions were
transparently harmless (and, in fact, net helpful.) The Github team are
behaving foolishly if they can't or won't distinguish that from an "attack."

Organizations should be expected to make decisions using reasoning that
amounts to more than word association, e.g. he "attacked" us, so we'll
"suspend" him.

~~~
Fluxx
I would disagree with this, quite a lot. He brought up an issue with the Rails
team, they pointed him at the canonical, "here is where we talked about this
before, sorry." Still not satisfied, he found the same exploit in Github to
prove a point. Rather than do the sensible thing by creating a dummy account
and contacting Github showing how he messed things up, he barged into the
Rails organization and left a silly commit. Github is first and foremost a
_business_ organization where lots of companies pay them lots of money to
"take this shit seriously" and protect their data, so they did the right thing
by shutting him down.

~~~
mquander
I honestly don't see the meaningful difference between contacting Github and
leaving a silly commit, except that the former would probably get the bug
fixed quietly; in contrast, now everybody is aware that the bug existed in
Github and is aware of the potential for it to exist everywhere. He
successfully proved his point, which apparently was a pretty good point. Isn't
that a better outcome?

As for Github's responsibility: Github failed to protect people's data the
minute the bug went live. That data was open to Egor since the moment he
discovered the bug until the moment they fixed it. Making a silly commit did
not make anyone's data more or less vulnerable, so I don't believe that
"taking this shit seriously" implies flipping out over it.

~~~
Daniel_Newby
> I honestly don't see the meaningful difference between contacting Github and
> leaving a silly commit, ...

The latter violates the Computer Fraud and Abuse Act, creating huge
imprisonment and employability risks.

~~~
aroberge
... if you want to take a US-centric view of things, then that last statement
is correct I suppose... but not everyone is subject to US laws - including,
unless I am grossly mistaken, the person you refer to.

~~~
Daniel_Newby
GitHub's and its computers are governed by US laws, and many countries have
extradition treaties. Many countries also have equivalent computer abuse laws.

My comment was to discourage such spectacular glory-seeking behavior by other
people that claim to be trying to help. It's a serious crime, and the FBI does
not care that your intentions were good.

------
collypops
> Beyond any shadow of a doubt, a shit storm of epic proportions has just gone
> down...

> ...this episode has been handled is a face-palm fail of epic proportions.

I'm all for, like, the evolution of language, but can we please all, like,
agree that 'fail' isn't, like, a noun, and 'epic' is, like, totally overused.

~~~
marekmroz
And don't get me started on "should of", used twice in this article...

------
jtchang
The one "good" thing that comes out of this public exploit is that github was
the target.

Github is obscure enough to non-developers but quite well known in the
development circle.

This means if you are using github you probably have a damn good idea what the
vulnerability is. I'm not a rails developer (mostly python/django) but I get
it immediately. This is mostly an issue with the framework helping me shoot
myself in the foot.

Sure it's in the documentation. But realistically a good framework gives me
sensible defaults so I don't have to refer to the documentation. I trust the
framework does the "right thing".

------
Kiro
"If you are one of those strange coders that don't use GitHub".

Never used and never will. What's strange with that?

~~~
zalew
_"If you are one of those strange coders that don't use GitHub and think you
are in the clear because you use SVN/Mercurial"_

I can't believe he actually put SVN and Mercurial on the same side, or that he
implies that not using GitHub must mean that you don't use Git at all. This
sentence is wrong on so many levels. Sigh..

I'm _strange_ , for my actual work I use Mercurial hosted on BitBucket.

~~~
p4bl0
I stopped reading when I saw that the author can't distinguish GitHub and Git
usage…

------
raganwald
As Zed Shaw pointed out, someone appearing to be Homakov has posted a comment
dating back _eight years_. Is Posterous vulnerable as well? If so, that may
not be Homakov, of course, but his twitter comments are consistent with him
posting it.

<https://twitter.com/zedshaw/status/176497720817762304>

~~~
jsnell
Yes, one of Egor's comments in the github bugtracker had a small (but
presumably not exhaustive) list of other vulnerable sites. It included
Posterous.

------
tmcdonald
I'm not sure all the things you list as being possible are true.

    
    
      - Every GitHub Repository could be access by anyone as if they had full administrator privileges.
      - This means that anyone could commit to master. 
      - This means that anyone could reopen and close issues in issue tracker. 
      - Even the *entire* history of a project could be wiped out. Gone forever.
    

As I understand it from his explanation[1] he added his public key to the
Rails user, which has permissions to push/pull to the repository. This doesn't
mean he had web administrative access, just Git access, since you cannot log
in to the web service using your private key. I hope that's the case, at
least.

[1]: <http://homakov.blogspot.com/2012/03/how-to.html>

~~~
lrm242
The way he was able to add his key was via a web-based exploit, which
effectively gave him administrative web access. So yes, the list is correct.

~~~
tmcdonald
I thought that he added his public key to the Rails user through his _own_
account settings, which wouldn't give him access to the Rails web admin.

~~~
chrisrhoden
This is correct. People who don't understand what a mass-assignment bug is are
running with this story. It's like when we witness a DDoS and have to
tollerate people who think it means that the targeted party was infiltrated.

This bug allowed one to add their public key to another user's account, and
make changes to comments and issues.

~~~
anthonyb
What are the odds that there's a similar bug which allows changes to user
accounts? If that's the case, then altering the password or email address is
trivial.

------
endlessvoid94
> Beyond any shadow of a doubt, a shit storm of epic proportions has just gone
> down.

Breathe.

~~~
ktizo
...but don't relax, it might be messy

------
alinajaf
Though I will say that mass assignment is about as rookie a rails mistake as
you can make, I won't be leaving github.

They've delivered so much value over the past four years for me that I can
forgive them this and I'd still have (a little) goodwill towards them left
over.

------
tlianza
I'm not sure if everyone noticed the last comment on this article:
<http://screencast.com/t/Nobted7zv5z>

Posted "almost 8 years ago" ie. posterous would appear to have the same
vulnerability.

~~~
zedshaw
Ha! Alright he probably should quit doing that before he gets KimDotcom'd.

~~~
getsat
Russia has no extradition treaty with the US. He's fine.

------
tlowrimore
I'm sorry, but this feels to me like an over-dramatized heap of bullshit.

First, the statement, "Rails. You clearly messed up." is self righteous
bullshit at its finest. Rails didn't mess up; the programmer(s) at Github
messed up. No conscientious developer lets the end user mass-assign variables
carte blanche. But with that said, _every_ developer messes up every now and
then despite their best efforts; some times they mess up in a big way.

Secondly, if a user discovered a vulnerability in something I wrote, and they
handled it like homakov did, I'd ban the shit of them until I knew for sure
that they weren't a threat.

Finally, Github handled this exactly the way many companies would handle it:
it's called damage control. These guys are really good at what they do, they
provide a great service and they offer-up a lot of their tools to the FOSS
community.

------
n8agrin
There is so much bombastic talk in this post. This has to be a troll.

------
meow
For me the sad part is that there is an almost even split between people
arguing the right way to bring this issue to every ones notice. I think this
is the perfect way to show how serious the issue is and to get more sites to
adopt the fix.

Exploits like this are worth a lot on black market. They are worth even more
if you provide a precious and vulnerable target to go along (github).

------
telent
AFFECT, grrr. Not effect. To effect every coder would be to bring them all
into existence, which is clearly not what happened here.

------
latetothepatty
I used GitHub and I'm not moving my stuff off. If an app gets hacked, then not
long after, that app will likely be the most secure place. GitHub at least
keeps it up most of the time. Who you really should be mad at are the Rails
maintainers and the RoR community. I switched from Java to Ruby a few years
back, and since day one, everyone using Rails has been slack on security. The
reason is that they make things too easy to leave wide open. Don't believe me?
Read the Rails official documentation for starting off. It is all about ease
of use, not security. If you are new, you have no idea what you've really left
open even when you just generate a scaffold as they show you to do. The main
thing that Rails security has going for it is that the adoption of Rails is
still relatively low, and because a newbie isn't likely to scale their app
well, odds are you won't have an extremely popular, extremely performant Rails
app that is just asking to be hacked that easily.

------
knodi
This is the first thing in securing your rails app a developer learns, how to
properly handle mass-assignment. I don't blame rails, I blame Github.

~~~
javascriptlol
A system that is designed to rely on human diligence is inherently flawed.
It's a mercy that drills and other dangerous tools aren't designed in the same
way that a lot of software is.

~~~
unimpressive
Exactly. This is like saying that it was fine for older versions of windows to
broadcast the users directory tree over an open file server because they
should _know better_ than to leave the server up. (This happened right?
Correct me if it didn't.)

It's like no, _you should have to turn the server on in the first place._ The
fact that it took some Russian kid in his basement _breaking into the github
rails master_ for this self evident truth to be realized by the core rails
team is totally outrageous.

------
eli_gottlieb
Oh, they've let you all down? Then stop using other people's web applications
and just run your own git/hg server. With blackjack, and hookers.

You are vulnerable to someone else's fuck-ups as long as you insist on giving
up control over your data and code in exchange for the convenience of someone
else doing the "hard work" of development and administration for you.

Hell, restore the network to being peer-to-peer rather than hierarchical, and
hosting your own whatever will no longer be such a damn problem.

------
colinmarc
+1 for the usage of "github-gate."

------
edu
Posterous is down?

------
aneth
The response to this makes me feel that HackerNews is now populated by a bunch
of pretenders. This "bug" has been in Rails since Day 1, and any remotely
experienced Rails developer is aware of this functionality. You can argue for
a different default, but it's not a bug.

Github did have a bug and noone knowledgeable about Rails appears to have made
even a cursory inspection of the security of their controllers - which is
where attribute protection actually belongs, since different controllers and
different users change different attributes. Protected attributes is a blunt
tool for simple situations, which is why it's not enabled by default. Github
had a pretty terrible bug, discovered, and fixed it. They may not have handled
it perfectly, but the certainly don't deserve this sort of mon hatred - any
competitor you go to is likely to have security flaws as well, perhaps more
severe and subtle.

@homako didn't just expose the bug in github, he exploited it to make an
unauthorized commit to Rails master. His account most certainly should have
been at least temporarily suspended as GitHub had no idea what else he might
do to prove his point.

So basically, most of the comments here are glaringly wrong or ignorant
bandwagoning, and it makes me wonder about the accuracy of information here
about topics I'm less familiar with. A sad day when you realize all this
intelligent discussion you thought you'd been reading about new topics was
probably just grandstanding by eloquent fools.

~~~
gurkendoktor
> Github did have a bug and noone knowledgeable about Rails appears to have
> made even a cursory inspection of the security of their controllers > Github
> had a pretty terrible bug > but the certainly don't deserve this sort of mon
> hatred

For all the free fun you can have on github, they are in the business of
selling private repositories. What could _possibly_ have been worse than
someone finding a bunch of bugs in a matter of days just to prove a point
about Rails?

~~~
aneth
What could possibly be worse? Anything actually malicious or greedy.

For all that GitHub has given to the developer community, an innocent mistake
even of this proportion of incompetence is still should not evoke such hatred.
And those upset about someone's account being suspended who was actively
misusing security holes - well, maybe you should use a provider who looks
positively upon reporting security holes by vandalizing customer data. And
they suspended his account for only a few hours! Unreasonable? Hate inspiring?
Outrageous? I think not.

And by the way, the fact the "issue" of the default had been reported 4 days
earlier in a github issue tracker for Rails (which is certainly not followed,
let alone on weekends, by github employees) does not in any way impact whether
GitHub should have been aware of this vulnerability, and to suggest so is
intellectually dishonest.

------
ktizo
I'm starting to like homakov more after reading this article.

------
mkramlich
I'm not too worried about the ability of an attack to push a commit, or cause
a commit/object deletion in a repo's history, on GH, because most smart folks
(if not everybody, by default) will have multiple copies of a repo across
multiple machines, with backups, so anything can be undone or restored. (And
trust me, I have the imagination to understand how an unauthorized commit/push
could lead to a situation enabling remote execution on client machines who've
pulled down tainted commits. Think build scripts that have "install
rootkit/malware/keylogger" commands added to them in the mal commits.)

What would be more bad is if this vulnerability allowed an attacker to get
unauthorized _read_ access or pull/clone access to a _private_ GH repo.

Can anyone clarify for me whether this was possible?

------
ggwicz
Shut the fuck up.

How many companies get hacked regularly like this but keep it under the rug?
You think FaceBook's never been exploited? TurboTax? Mint? Stripe? PayPal?
Shopify? Tumblr? Pick your app that "so so so so many businesses" use
regularly, and I guarantee something like this has happened with all of them.

But were they open about it?

GitHub's been open the whole time.

Your post is like saying "All criminals are stupid". This is ridiculous, as
the only sample you know of and can work with are the criminals who _have been
caught_. You don't know how many other criminals are out there getting away
with their crimes, because...they haven't been caught yet.

Who knows how many other companies have had hacks like this in the past two
months alone, for example? I don't, and neither do you.

But GitHub, as an open, honest company that so so so many of use regularly
_(which means we know right away when there's a problem, especially with a
hugely popular repo like Rails/rails)_ has been in the spotlight since the
second this happened.

GitHub, in my opinion, has acted really cool about this. They addressed the
issue, explained what the issue is, patched the hole, and even reinstated the
hacker's account. DHH addressed the issue in twitter, other people in the
community have admitted they fucked up, and now we as a community can work on
fixing this.

That doesn't sound like "Letting us all down".

Someone who expects everything to work perfectly all the time and have no
vulnerabilities is someone will be let down by anything, a pessimist, and
stupid. And certainly not worthy of the front page of Hacker News.

