
Rubygems.org compromised - benmmurphy
http://rubygems.org/gems/exploit/versions/22.31.31
======
qrush
Hi everyone -

A full post-mortem is coming, but here's what happened from my perspective. I
got mentioned on twitter about this thread while I was on the bus. I asked
@evanphx to put the site into maintenance mode immediately.

Once I got on wifi, I downloaded the "exploit" gems and found that they used
the Psych hack to post config/initializers/* and config/*.yml to Pastie.
Luckily, none of our API keys were actually in those files - our config is a
bit different since you can still run RubyGems.org on Heroku, which needs to
put secret keys in the ENV.

I deleted the exploit gems permanently along with the throwaway account that
posted them...that's why this post's URL is a 404.

I've reset our S3 keys just to be sure and any other keys we had on our
production site. From what we know right now, no other changes on our S3
bucket have taken place, and we're going to check the logs to make sure. Once
we get a real fix for this issue out, pushing gems will be enabled again.

Just a general PSA -

Please, if you find an issue like this, be nice. Tell the maintainers
privately. Don't post to Reddit, HN, or a public Gist. RubyGems.org is
completely volunteer run. No one gets paid to work on it. Thanks for your
patience everyone.

~~~
tptacek
If you want security notices to go to the Rubygems team, you have to set up a
security page that tells people how to do that. Like everyone else, I
appreciate your volunteer work, but no amount of goodwill creates the ability
for people to read your mind.

Please post a security page. It literally doesn't need anything more than an
email address and a PGP public key.

By the way: if Rubygems needs security help, it looks like there's quite a
number of people who are willing to pitch in. When I published a FreeBSD crt0
bug back in 1996, I was given commit privs. I thought that was a pretty
effective way to co-opt adversarial researchers.

~~~
jasonkester
What would the PGP key be for?

~~~
tptacek
The Rubygems team (or even just the person who writes the security page) can
just generate a new one, for "security@rubygems.org" or whatever, and it can
be shared by the team.

The purpose of the key is to allow people to report security vulnerabilities
without worrying that by doing so they're giving ammunition to people snooping
emails.

~~~
willyt
Maybe I dont understand PGP and this is a stupid question, but if the site is
compromised, would it not be possible for them to just put a different key up?
One which they knew the private key for? How would you know the public key on
the compromised site has not been changed unless you could compare it to the
PGP key from before the site was compromised?

~~~
jaytaylor
@JoachimSchipper do sec people inherently know to try hitting /security.html
by convention, or should there be links to it from the main site?

~~~
tptacek
They know to look for a link to the security page.

Github's is in their site footer.

37signals' is in their site footer.

Twitter's is linked off the sidebar in their "About" page.

Google's and Facebook's are the top search result for their site and
"vulnerability" "security".

These are all fine options.

~~~
jaytaylor
Thank you!

------
evanphx
It's disappointing to see this show up on HN without contact myself or the
other rubygems.org admins. We've put rubygems.org into maintenance mode until
the issue is fixed.

~~~
tptacek
What I saw was Ben twerping to Ruby people asking who to contact about this
first, then posting to HN when he couldn't find the right person. Can I ask,
where's the security page on Rubygems.org that tells people who to alert when
stuff like this happens? I'm trying to find it on Archive.org and drawing a
blank.

~~~
benmmurphy
I looked for contact page first then panicked. I should have used whois :(

~~~
tptacek
If the whole world is pulling backdoored gems off a compromised Rubygems.org,
panic seems like a reasonable first reaction. This isn't the Rails bug; there
was nothing to be gained here from secrecy.

~~~
benmmurphy
I became aware of this problem this morning when it was discussed on a public
github issue. I think the problem has been known for a couple of weeks. I
checked latest rack and active_support gem (would seem to be good targets...)
to see if they had been backdoored but they were clean. Not sure if these gems
are still clean or if any others have been compromised.

~~~
rmc
_to see if they had been backdoored but they were clean_

You can never be too sure. There's a whole contest
(<http://underhanded.xcott.com/>) whose goal is "make something that looks
legit, but actually does something evil / has an exploit).

Nuke from orbit.

~~~
benmmurphy
i diffed the rack gem and active_resource gem from the versions built from
git. i'm fairly sure the versions i downloaded were not compromised.

------
wereHamster
When will people start signing their gems? And gem/bundler warn if the
signature is not present or invalid?

~~~
mratzloff
I'd wager that most gem developers don't understand how it works, and there
are basically no resources out there on it.

Here's the chapter on it in the RubyGems manual:

<http://docs.rubygems.org/read/chapter/21>

And that's all that I can find. Try to find something else, like a blog
tutorial.

~~~
Xylakant
Even if most gem developers would understand, there's still a problem with the
trust model: If you choose "high security", you cannot install a single gem
without signature - so things won't work unless _all_ authors sign their gems.
Gem signing would have to be a requirement, similar to packages on ubuntu's
ppa. If you choose a lower security setting, a gem may just be repackaged and
signed again by the attacker or the signature removed.

~~~
mratzloff
A solution that doesn't solve 100% of the problem is still not a bad solution.
The most popular gems would all be signed, thereby dramatically reducing
attack vectors.

~~~
Xylakant
No, as I said, this doesn't help. You would need to sign the most popular gems
and all their dependencies and the dependencies dependencies. Given that a
pretty much standard rails app can easily pull in hundred or more gems that's
quite a bit of work. It's doable, but I only see that coming if for example
rubygems requires a key and gem signatures.

Then there's still the unsolved problem of providing key trust - the current
implementation relies on self-signed certificates.

------
ruby_vuln_4eva
Where are the adults? This is absolute amateurism. Why haven't Heroku or
GitHub or some actual professionals in the Ruby community, stepped up to
ensure that this kind of amateurism doesn't damage their ecosystem?

[Throwaway as I'm a small-fish in the Ruby community. I expect I'll get to use
this account again soon though]

~~~
pifflesnort
This is exactly what I'm wondering. _Are_ there adults in the Ruby community?

The vulnerability itself is simply ridiculous:

<https://github.com/tenderlove/psych/issues/119>

The response from the Gem folks has betrayed a serious lack of experience.

Has Rails been such an insular and young community for so long that there are
truly no qualified engineers around with experience in this level of
professional software development?

~~~
dbenhur
The problem is any YAML loader which supports tags that can name arbitrary
classes and have existent classes whose initialization may `eval` the supplied
values. Ruby is very publicly hurting, but equivalent vulnerabilities exist in
other runtimes, such as Perl via
[YAML::Syck]([http://search.cpan.org/~toddr/YAML-
Syck-1.22/lib/YAML/Syck.p...](http://search.cpan.org/~toddr/YAML-
Syck-1.22/lib/YAML/Syck.pm)) or Python via
[PyYAML](<http://pyyaml.org/wiki/PyYAML>) and
[LibYAML](<http://pyyaml.org/wiki/LibYAML>). If your language can accept and
load untrusted yaml and your language has `eval`, you're probably fucked and
just don't know it yet.

~~~
batiste
The problem is that YAML is used by Rails & Co as a parser for untrusted user-
provided data. It does not matter if there is buggy implementation in
Python/Perl if nobody use them.

If other framework were using YAML in this fashion, yesterday will be the time
to look for security holes in those implementations.

------
runeb
The exploit was written by @postmodern_mod3:
<https://twitter.com/judofyr/status/296649167189725184>

Edit: At least a PoC. How someone other than him (since hes denying
compromising rubygems) found his PoC and wrote the actual payload and pushed
the exploit gem is unclear.

~~~
judofyr
Short timeline:

\- 1 week ago: blambeau reported the issue to the rubygems people:
[https://github.com/tenderlove/psych/issues/119#issuecomment-...](https://github.com/tenderlove/psych/issues/119#issuecomment-12875715)

\- Postmodern wrote a PoC - <https://gist.github.com/4674219>

\- Postmodern: "I posted it in a private chat room. Guess someone took it for
a ride." -
[https://twitter.com/postmodern_mod3/status/29665192240284057...](https://twitter.com/postmodern_mod3/status/296651922402840576)

\- A gem called `exploit-36.44.16` was pushed to RubyGems. This contains a
payload that only sends `uname -r` to pastie.

\- A gem called `exploit-22.31.31` was pushed to RubyGems. This contains a
payload that posted the config/database.yml to pastie. This gem was removed by
the rubygems admins when they discovered it -
<http://news.ycombinator.com/item?id=5140109> &
<https://gist.github.com/d891e876c53e55bf0920> (that's the payload)

Other exploit-gems that was pushed (but we don't know what did):

\- 16.17.49 - <https://twitter.com/rubygems/status/296618422702309377>

\- 20.22.1 - <https://twitter.com/rubygems/status/296617610622144512>

\- 12.5.4 - <https://twitter.com/rubygems/status/296540884781109248>

\- 7.27.42 - <https://twitter.com/rubygems/status/296537952320884736>

My conclusion based on the current facts:

1 week is too long for such a severe security issue. A proof-of-concept gem
seems to be a logical step to "force" rubygems.org to fix this issue. However,
publishing database.yml on pastie is TOTALLY 100% NOT COOL, and turns this
from "whitehat who wants to help" to "blackhat that wants to destroy (or
doesn't understand social norms)".

~~~
pifflesnort
> _However, publishing database.yml on pastie is TOTALLY 100% NOT COOL ..._

This whole thing is a joke, from the ridiculously unsafe YAML parser to the
response of the RubyGems people.

If these are your community social norms, then you quite simply deserve the
resulting shake-up. This is how you become inoculated against such incredibly
poor engineering practices.

~~~
judofyr
I don't quite get what you're trying to say. What "community social norms"?
Are you talking about how rubygems.org handled this vulnerability from the
beginning? How the community handled this incident after it happened?

And what does it have to do with my quote? I was just pointing out that in
general I'm for PoC to highlight security issues when the vendor doesn't
respond, but stealing database.yml makes you nothing more than a simple crook.

------
Titanous
Here's what the payload looked like:
<https://gist.github.com/3e4829f79dbd1be11295>

~~~
timdorr
Did you catch where was this payload injected? Did they have write access to
the rubygems.org source? It would seem if they did, they didn't need to write
this script at all and could simply read the configuration files directly.

~~~
WestCoastJustin
After reading through the github.com issue posted in the comments, I get the
impression this was targeted attack at rubygems.org (and not the user install
base). There has been a tweet, also posted in the comments, suggesting that
this was successful.

------
landr0id
Mind explaining to me how this shows that rubygems was compromised? It looks
to me like someone just uploaded a gem called "exploit" which takes advantage
of a recent exploit.

~~~
kawsper
It has been put into maintenance mode, so that is what you see now, until this
is resolved.

~~~
vinhboy
It kind of says something when people can't tell the difference between your
maintenance mode page and a hacked page...

~~~
kawsper
Should he design a custom "We have been compromised" page before deactivating
the site?

It was put in maintenance to stop pushing possibly infected gems to users.

------
jordanthoms
Why is it that all these parsing issues with yaml and xml were ignored before
it all blew up recently? Input parsing is usually the most venerable part of
any application, and it's strange that these issues were ignored for so long.
These are not super complicated exploits, they are using the _expected_
behavior of the yaml parser in terms of it constructing arbitrary objects.

~~~
blambeau
Because when you write "Rails vulnerabilities are not Rails'"
([http://www.revision-zero.org/rails-vulnerabilities-are-
not-r...](http://www.revision-zero.org/rails-vulnerabilities-are-not-rails)),
most people respond "yes it is".

~~~
jordanthoms
Well, a unsafe parser (which was designed only for parsing trusted input) was
being used by lots of people (including Rails) for parsing untrusted input as
if it was a safe parser. You can debate about whether there should be a safe
parser for YAML, but that's a separate issue.

~~~
blambeau
Is "which was designed only for parsing trusted input" written anywhere in
Psych doc? Psych is shipped with Ruby, is it written in Ruby doc maybe?

~~~
jordanthoms
Agreed, there should be a clear warning - in fact, the load method should be
renamed unsafe_load. The root cause of this is probably unclear documentation
and misunderstandings between the users and authors of psych.

------
adelevie
Is there anything gem authors should be doing?

~~~
timdorr
Sign your gems before pushing them: <http://docs.rubygems.org/read/chapter/21>

~~~
kgo
The trouble is self-signed X.509 certificates are pretty worthless. X.509
certs are built around the concept that they're issued by a trusted provider.

I tried to get together a proposal to introduce OpenPGP signed gems and wrote
some proof-of-concept code so there could be a proper Web-of-Trust for
rubygems:

<https://github.com/grant-olson/rubygems-openpgp>

But the developers were resistant (and understandably so) to having gpg as a
dependency for gem verification, and I didn't trust myself to write the crypto
and a decent keyring validation algorithm from scratch.

~~~
moe
_The trouble is self-signed X.509 certificates are pretty worthless._

I disagree. If you obtain the public key from a second location that is not
rubygems.org (e.g. from gem.homepage) then that raises the bar for a potential
attacker. He now has to either compromise _two_ sites in order to inject a
malicious gem, or trick you into obtaining the public_key from a forged
location.

I hope 'bundler' and 'gem' hear the wakeup-call and will start displaying a
visible warning on unsigned gems (please also print the homepage-url to make
it less of a pain for the user to hunt down the public key). After a grace
period the default should be switched to -P HighSecurity (reject unsigned
gems).

Yes, it's is not perfect, users may still import keys without thoroughly
checking that the source is legit. However, it'll be a huge improvement over
what we have now and greatly reduce the chance of a compromised gem to go
unnoticed.

~~~
kgo
Point taken, but my day-job rails project, which isn't even that big, has 461
gems in my Gemfile.lock. (which lists all the explicitly included gems, as
well as their dependencies, if you're not familiar.) That is a lot of self-
signed certificates to track down, attempt some form of verification, and
manually add as trusted.

My proposal would allow people who want a medium-level of security to verify
that whoever uploaded the gem has (1) valid credentials to rubygems.org, and
(2) control of the account owner's email, without requiring all that manual
installation.

And it would allow people who wanted to set higher standards of verification
to do so, basically for free, by configuring their settings in gpg.

~~~
moe
I think your approach is problematic because it hinges on the integrity of
rubygems.org (the very host that is compromised right now).

I do see your concern about having a large number of gems, but I believe that
is a bootstrapping problem which could be relatively easily mitigated.

Trusted entities (rubygems team, 37signals etc.) could publish signed versions
of their keyrings, i.e. public_keys that _they_ trust. So, if you trust
37signals, you'd just download their ring, validate their signature, and
import it. And that will probably cover most of your 400 gems.

~~~
kgo
We're basically on the same page, but coming from different directions.

Once you start having people import lists of trusted X.509 signatures from
other trusted sources (e.g. 37 Signals) what you end up doing is basically
creating the Web Of Trust that already exists in OpenPGP.

That is, instead of simulating a X.509 Certificate Authority in OpenPGP like I
want to do, you're now simulating a OpenPGP web of trust in X.509 land.

I would have the rubygems OpenPGP key be the default for people who would want
moderate security. Although a compromise could affect new users, all existing
user would have the old key when they installed rubygems. That would
immediately set off flags when gems signed by a forged key only were installed
by the millions of users with the uncompromised install.

In addition, the more security conscious could trust for example 37signals or
a co-worker and create a proper Web of Trust for validation.

This is the way OpenPGP works by default. You can even different trust levels
automatically with this approach. Simply a rubygems.org signature would make
the gem moderately trusted. Two additional signatures from two other trusted
sources (37signals and a co-worker) would make the gem show up as highly
trusted.

But rather than manually importing in an ad hoc fashion, you just sign the
37signals key locally, and gpg already takes care of enforcing all the rules,
building the chain of trust, publishing peoples signatures to keyservers
distributed world-wide, performing security calculations based on a user's
custom preferences, etc.

So if you want the web-of-trust approach, best to just start with OpenPGP,
instead of trying a NIH approach with X.509 certs.

~~~
moe
Yup, we seem to agree in principle.

Your approach is definitely more complete but also seems to require fairly
substantial code changes (OpenPGP?).

My approach should be doable with a mere ~10 lines patch; Gem and bundler
should default to '-P HighSecurity' and rubygems.org should reject pushes of
unsigned gems.

Anyway, whichever implementation strategy they choose, I hope they will do it
_soon_.

~~~
technomancy
> My approach should be doable with a mere ~10 lines patch

If you actually believe this can be accomplished in 10 lines of code you are
not qualified to be giving security advice.

~~~
moe
It's always sad to have people chime in like that.

The changes I specified are, indeed, about a 10 line patch. You change the
default for '-P' (1 line) and you inspect pushes to rubygems.org for a
signature and reject them if they lack one. I'd be very surprised if the
latter exceeds 9 lines. All you have to do is look for s.cert_chain in the
gemspec, so it would actually likely be closer to 3 lines.

You may disagree with what I proposed and demand additional changes, or you
may prefer the superior (albeit more complex) PGP approach. But you should not
go ad hominem over a basic fact when you obviously have no idea what you're
talking about.

(disclaimer: one of us two has actually read the code involved with gem
signing just a few days ago - and I'm pretty sure it wasn't you...)

~~~
technomancy
I'm not disputing that this change is a step in the right direction, but the
implication that it solves roughly the same problem as the existing OpenPGP-
based infrastructure is very misleading. [Edit: if you din't mean to imply
this by your post then I apologize; it was not clear from how I read it.]

Same with the idea that you can "just apply a 10-line patch" without
considering the background in documentation, tooling, and educating the
community that such a change would require--it implies that the current
rubygems maintainers are ignorant or lazy for not having done this easy fix
already.

(I spoke at length on a PGP-based approach to this same problem at the 2012
Clojure Conj conference if you would like to talk about credentials:
<https://www.youtube.com/watch?v=sBSUIKMdQ4w>)

~~~
moe
I didn't claim it solves "roughly the same problem", only a very relevant
subset of the problem (integrity) closely related to the unfortunate situation
rubygems is in right now.

What you call background (docs, tooling, education) is an argument _in favor_
of the simple approach, which is already implemented and just not enforced -
in contrast to whipping up new infrastructure of significantly higher
complexity from scratch.

Also I didn't imply the maintainers are ignorant or lazy, they were merely
oblivious.

So I'll repeat: The PGP approach is obviously superior, but comes with serious
complexity and implementation concerns (there is no native PGP-library in
ruby). Consequently it might make sense to add _basic_ security right now, in
a minimally intrusive fashion, while a more complete solution is developed.
This, by the way, is part of the discussion that the rubygems team is having
right now[1].

[1] <https://github.com/rubygems-trust/rubygems.org/wiki/Overview>

------
zalambar
We have a great many people relying on a volunteer run project here. It would
be great to have someone sponsored to spend time maintaining the site.

Unfortunately I don't see any way to help fund such an effort. Is there an
officially supported way to donate to RubyGems? Can we create a pool to help
cover the cost of maintaining this resource?

~~~
kawsper
There is some larger organizations using Ruby. So not hiring a guy full-time,
but they might be interested in chipping in, to get a gem-repo that they can
trust.

------
blackxored
Jeeze, bad things happen (or could) to the greatest services ;) I'm late to
the discussion, didn't have time to lurk around, nevertheless what I can agree
the most is about gem signing. A gem is a package of software, and every
single package distributor makes the author sign their work (think linux
packages, being a Debian/Ubuntu developer I can tell you that if you don't get
your package signed, it won't get uploaded, period, your key in some cases
(Debian) needs to be verified _in person_ to build a circle of trust. I think
that besides the overhead (initially, I'm sure people will contribute scripts
or integrate this so that the developer won't notice) this is really a needed
step you guys should take.

------
munin
something to think about here is, how long has this bug been in rubygems.org?
how long would someone have had to modify arbitrary gems? fun times!

------
pjungwir
I also get an "invalid SSL certificate" warning when I try to visit
rubygems.org. Here is the message in Firefox:

    
    
        rubygems.org uses an invalid security certificate.
    
        The certificate is not trusted because the issuer
        certificate is unknown.
    
        (Error code: sec_error_unknown_issuer)
    

Any chance this is related?

~~~
ilikepi
According to Certificate Patrol, I'm getting the same cert I first received on
2012-12-11:

    
    
        not valid before: 2012-03-03
        serial: 00:E6:99:C4:B4:04:ED:07:D2:E5:BC:42:29:09:B8:17:D0
        sha1 fingerprint: BD:2E:60:69:49:99:53:29:BD:E9:24:12:95:E1:5E:65:E0:64:79:2A

------
benmanns
The current title is "Rubygems.org compromised." Isn't this just a gem that
exploits the YAML bug, not an exploit on RubyGems.org? I believe this
submission needs a title change.

Edit: I am incorrect. It seems that this uploaded gem was used to compromise
RubyGems.org.

~~~
benmmurphy
it looks like someone has tried to steal rubygems.org db and other
credentials. bundle update is now dangerous because you might get trojaned
files.

~~~
benmanns
But don't applications have to explicitly install this gem? If not, then I am
incorrect.

~~~
bittersweet
Qrush, one of the guys behind Rubygems tweeted that the credentials of every
yml file ended up on pastie:
<https://twitter.com/qrush/status/296638346283868160>

> Someone posted <http://rubygems.org> 's config/*.yml to pastie. Didn't
> include the S3 bucket secret key, but going to reset everything anyway.

------
davebuster
Why aren't gems protected by a trust chain all the way up like Debian does?
Gems aren't even required to be cryptographically signed by the developer and
from what I can tell are just thrown over http without any authentication.
This seems terrible.

------
eric970
For those that don't know how this exploit worked, I think the "exploit" gem
description provides a pretty good explanation:

    
    
       'A Proof-of-Concept PoC gem that exploits a vulnerability in the Psych YAML parser, which allows the #[]= method to be called on arbitrary Objects.
       If the #[]= method later calls eval() with the given arguments, this allows for
       arbitrary execution of code.'

------
prusswan
I never knew the gems were not subjected to version control at all on the
server. Can't they put everything into a git repository somewhere?

------
reybango
Being totally new to RoR (trying to learn it), I'm trying to get my head
around the scope of this.

When did the compromise happen? Was it compromised yesterday or only found out
yesterday?

I have default gems installed on my system and haven't updated anything since
the last big Rails security issue that was reported a bit ago.

It'd be great to get some guidance on what to do.

------
sch1zo
can someone explain what happened here to those like me who are a little late.
seems like the issue is resolved already or at least temporarily fixed. All I
can figure out from the url and the comments here is that there was a gem
called exploit and that somehow the config files where posted to pasties.org

------
gravis
I'm sorry to mention some RG projects are still "potentially" dangerous:
<https://gemnasium.com/rubygems> We'll do our best to help determining if
these projects really need to be updated asap!

------
davexriess
What options do we have to work around this while the issue is being fully
diagnosed?

I'm considering re-installing my gems directly from the github repos.
Thoughts?

------
rufugee
So what source should app developers use until this is cleaned up?
gemcutter.org and rubyforge.org?

------
OffTheRails
Damn it! What a mess! This is some depressing press for the Ruby community.

------
ErikRogneby
status updates posted here: <http://status.rubygems.org/>

------
bttf
rubygems.org is back up

------
static_typed
Oh the timing. So, there was a big meeting, where some of the devs pleaded the
case that the recent wave of security issues in the Ruby world were a tipping
point, and that we were over it. They presented a good case, and I think the
architects were close to permitting the usage of Ruby and Rails in their
current cases. Then we get an interruption, as a couple of projects had
updated their gems today, and there was a concern about overall security of
the apps.

IT Security required the apps were offlined, and the green light was given to
the PHP and Python guys in house (and a few retained developers) to start the
process of rewriting the apps.

It will be painful, but, I think, the business has lost it's appetite for the
constant Ruby idea of "Let's run with Yaml", before firstly "Let's walk safely
with Yaml".

~~~
sergiotapia
I see where you're coming from, it's a shame that so many bugs are available
in Rails.

------
dantillberg
The polite thing to do first would be to notify the maintainers of
rubygems.org - not post the link to HN.

~~~
benmmurphy
I didn't know who maintainers were :( I think i've notified one now.

~~~
WestCoastJustin
$ whois Rubygems.org

Tech Email: ...

~~~
robotmay
In Ben's defence, I doubt he's the only one who wouldn't have thought to check
the whois data. I wouldn't have, for a start. There should have been an
obvious contact email linked on the site itself for issues, not just a Twitter
account.

