
Ruby deploys temporarily disabled - ujeezy
https://status.heroku.com/incidents/489
======
Pewpewarrows
This should also be a reminder to everyone that you shouldn't be reliant on a
single point of failure for your deploys. It's something that we in the Python
community have already encountered (and hopefully learned from) due to the
historical unreliability of our equivalent package repo, PyPI.

Have an internal repo that's accessible by your deploy servers, which in turn
locally caches anything that you might have previously needed to externally
fetch.

~~~
steiza
I only recently realized how easy it was to run your own PyPI - it just has to
handle a few HTTP GET / POSTs.

If you want to run your own PyPI internally, here's a very simple PyPI server
(~150 lines of Python) that I wrote: <https://github.com/steiza/simplepypi>

~~~
po
Also of interest is <http://crate.io/>

What I've personally been looking for is an easy to setup caching proxy for
PyPI. Something that is pip-compatible and serves files if it has them but
will also fetch and then store packages if it doesn't. That way you could
build up a collection of 3rd party packages over time, without having to
explicitly manage it.

It probably wouldn't be hard to roll my own with a reverse proxy but it never
gets moved to the front burner.

------
pilif
I'm surprised that rubygems.org of all places did not see fit to patch the
vulnerability that's now known for multiple weeks and which has been declared
to be incredibly dangerous and for which ready-made exploit kits exist.

rubygems.org is a central distribution platform trusted by tons and tons of
projects. As such, that site is the one site you probably do not ever want
compromized. Imagine the damage an attacker could deal by uploading backdoored
versions of various popular gems.

I know - applying security patches is time-consuming and we are all afraid of
breakage. But the moment rubygems.org stepped up to be a semi-official central
distribution point for gems, I would have hoped they also took on the
responsibility that goes along with that.

If this was some new unknown 0day exploit, I would be much more understanding,
but this was known to exist, known to be dangerous, known to be exploited.

~~~
steveklabnik
This is not that vulnerability. Their Rails is just fine. This is related to
the fact that gems use YAML to store their metadata.

~~~
1qaz2wsx3edc
Semi related CVEs aside (CVE-2013-0333 and CVE-2013-0156): YAML's security
problems have been known for years by the community. YAML aside, don't trust
user input. This is egg on rubygems's face and the ruby community. I don't
think <http://rubycentral.org/> has full-time staff for rubygems either.

Gems, specifically should be signed. They are not, this type of exploit will
continue to happen, hell, remember when github screwed up ssh keys? Who knows
what's in the ecosystem.

TL;DR; Ruby's security ecosystem is butter.

Disclaimer: I love ruby and use it daily. It was two critical problems IMO:
unsigned code and GIL. Yes, GIL. I'm looking at you ruby-core.

~~~
charliesome
The GIL is not really a problem. There are far more pressing issues with MRI,
and many core developers are working hard to solve these.

~~~
krickle
Like what? I thought the GIL and GC forcing COW were the largest problems bar
none.

------
sergiotapia
The DAY I manage to convince the big wigs where I work that we should switch
from a typical shared environment to Heroku, this happens.

Talk about luck. :(

Hopefully I can spin this and not leave a bad taste in their mouths. We
(engineers) understand what's happening, management doesn't and they don't
give a shit.

~~~
pardner
From a manager perspective, this sort of alert (as well as recent Heroku
emails telling you specifically which app(s) needed patching for the recent
Rails CVE issues) is a great example of one of the extra benefits of Heroku. A
team of engineers 'watching your back' at no extra charge is a good thing.

~~~
cobrabyte
Agreed. I was surprised when I received an email from Heroku letting me know
that a few of my apps needed to be updated after the Rails vulnerabilities
were uncovered. They also named the apps that needed to be updated, which
makes my job that much simpler.

~~~
kawsper
It was a nice service with the mail. But it was sent one day after the exploit
was out in the open, which is too late, but better than nothing.

~~~
jordanthoms
I guess they had to build the feature. With the follow-on exploit for rails
<3.1, the notification email went out very quickly and probably they will have
quick notifications going forward.

------
whalesalad
I always assumed that Heroku would have an internal proxy for gems. Seems like
80% of users would probably be fetching the same gems that another user might
have just fetched. Perhaps something like this could be versioned or
snapshotted so that in the event of something like this, you could roll your
cache back to that snapshot and let people deploy who had gems in that cache.

I'm just thinking out loud.

~~~
rhizome
Which snapshot do you roll back to?

~~~
whalesalad
That's a good question. I do know one thing though: I deploy multiple times
per day and typically none of my gems have changed.

I guess it would depend on the folks doing the investigation. If an exact
timestamp could be determined for when things could have been compromised, you
just roll back to a short while before that time.

------
thinkbohemian
Heroku engineer here: Ruby deploys are back online if you don't require any
new gems, i.e. can deploy from the existing cache. We're still working on
resolving the large problem with Ruby Gems.

------
aneth4
This is why you vendor gems, and the current accepted practice of not
vendoring gems is dead wrong.

I've been chastised before in rails irc for this. I strongly believe the
source of all depndencies possible should be in your repository.

~~~
pmahoney
> source of all dependencies possible should be in your repository

How far do you go? Do you include libxml for building nokogiri? Heck, do you
include libc and gcc for building any gem with a C extension?

Coming from Java, Maven and something like Nexus Sonatype make it easy (for
certain values of "easy") to run a proxy repository. The equivalent of all
"gem install <some_gem>" goes through the proxy, which continues to serve gems
even if the original source goes away.

I don't particularly like the inclusion of dependencies in a repository. Is
this a custom version "some guy" long gone from the company created three
years ago? Can I safely upgrade it to get security fix <X>? I suppose similar
questions arise no matter the source...

This is reason why you _cryptographically sign_ your gems before publishing
them. I (unfortunately) had not known this was supported by RubyGems, but it
is: <http://docs.rubygems.org/read/chapter/21>

But I'll bet very few gems are signed. Rails does not appear to be:

    
    
      $ gem install rails -P HighSecurity
      ERROR:  While executing gem ... (Gem::Exception)
        Unsigned gem

------
martingordon
Between this hack and the recent Rails vulnerabilities, it seems like a
perfect storm. I wonder if either the hack attempted to tamper with the Rails
gems to catch late updaters or to remove the ability to use RubyGems to update
to the latest versions and keep vulnerable sites vulnerable.

~~~
awj
It more looks like this was a natural extension of (part of) the Rails
vulnerabilities. People saw that YAML on Ruby has a giant gaping security hole
in it _and_ was commonly used to decode user-supplied data.

I would not be surprised if we see even more of this as people feel out all of
the other places that YAML is used as a user-facing data interchange format.

------
mildavw
Survey: Do you depend on RubyGems for every deploy? Or do you have your own
gem server? Or cache them at some point earlier in your pipeline?

We rely on RubyGems and had a meeting yesterday about changing that when one
of the gems we use had a version just disappear.

~~~
Corrado
I was looking for a RubyGems proxy a couple of weeks ago but was unable to
find anything suitable. What I would like to find is something similar to
Artifactory for Maven. You include the proxy in your Gemfile and if it doesn't
have the Gem it downloads it from RubyGems and caches it locally.

This type of proxy wouldn't help in this particular case but it would allow
you to keep traffic to RubyGems.org down and also give you the ability to
easily host private Gems.

~~~
pmahoney
I, too, would love something like this.

You're probably aware, but it's possible to host private gems in a simple
static webserver. We have our CI server copy gems over and run a script that
calls "gem generate_index -d /path/to/gems".

Completely featureless, of course. Lacks the newer rubygems.org api (so
bundler is stuck downloading the whole index), and obsolete versions will
stick around without manual intervention (which, if you have too many, is a
pathological case for bundler's resolver).

------
benmmurphy
This is the responsible thing to do. Going through the gems and verifying they
aren't compromised is a lot of work. We should be thankful for all the effort
the rubygems maintainers and other volunteers are putting into cleaning up
this mess.

------
akavi
A tangent, but I always thought "YAML" was pronounced /'jæm.ḷ/, however the
post's use of "an YAML" suggests it's actually pronounced /waɪ.eɪ.ɛm.ɛl/.
Weird.

~~~
saurik
You still wouldn't say "an why", as "w" isn't a vowel. If Anything, I can
understand "an yamel" as more legitimate (as "y" is at least sort of a vowel).

~~~
justsee
No, "an YAML" is grammatically incorrect [0]. It's "a YAML".

What's more annoying: the odd (and wrong) belief that "An green apple" is
grammatically correct. [1]

[0] [http://english.stackexchange.com/questions/1016/do-you-
use-a...](http://english.stackexchange.com/questions/1016/do-you-use-a-or-an-
before-acronyms/11511#11511)

[1] [http://english.stackexchange.com/questions/152/when-
should-i...](http://english.stackexchange.com/questions/152/when-should-i-use-
a-vs-an/164#164)

~~~
saurik
I didn't say that "an YAML" is correct; I was first correcting that "an YAML"
would be correct if it were "an why aih ahm ell" (it wouldn't), and then went
further into "if anything" land saying that "an yahmell" is at least something
I can bring myself to say without feeling sick inside. ;P

------
freshfruit
If you are not updating gems, does it hurt to continue deploying with a custom
buildpack? Heroku shouldn't repull gems if the gemspec hasn't been altered. Is
that logic correct?

Does anybody have another suggestion for safely working around this issue? I
don't have a clear sense for how long this will take to resolve and don't wish
to slow down our release pace too much.

~~~
kawsper
If you trust your buildpack (and its contents), and it contains all the gems
necessary, you are safe to deploy.

------
taf2
I thought at one point in time rubygems had a system in place to sign the
contents of the gem? If not this might be an interesting addition. You could
have a digest stored along side every gem file allowing you to validate the
authenticity of the gem file... I'm sure others would have something to add to
this idea...

~~~
msmith
It is possible for the publisher to sign the gems [1], but it's not common.

If rubygems.org is keeping fingerprints of each gem, then it still isn't
sufficient, since those could have been compromised as well. If there's no
other trustworthy source of fingerprints, then maybe we need to crowdsource
it. Built a tool that will md5sum all the .gem files in your local cache
directory, so that we can look for any files that were changed on rubygems.org

[1] <http://docs.rubygems.org/read/chapter/21>

~~~
rmc
Sounds like a good reason to switch to "everything must be signed"

~~~
ikawe
Just to reiterate the parent: This is only valuable if we trust the signatures
- which I wouldn't if they were, say, just held along side the "hacked" gems
server.

~~~
rmc
I'm talking g about developers signing the archive on their local machine.
Private key would be stored on developers laptop

~~~
Xylakant
You still need the public key to validate the signature. If the attacker can
change the public key, he can change the signature without you knowing -
unless you explicitly want to trust each and every key for every gem you
install.

------
zdgman
Good to see them being proactive about it. Wonder how quickly they will be
able to determine that Ruby/Gems are safe to deploy again.

------
reybango
Being totally new to RoR (trying to learn it), I'm trying to get my head
around the scope of this.

When did the compromise happen? Was it compromised yesterday or only found out
yesterday?

I have default gems installed on my system and haven't updated anything since
the big Rails security issue that was reported a bit ago.

It'd be great to get some guidance on what to do.

------
pardner
Seems odd, however, that status.heroku.com lists the issue only on the
development side, suggesting production apps are not affected?
<http://screencast.com/t/L36Hpx5dx>

~~~
pvh
Running apps are not affected, but your ability to do development is. The
threat vector affects app compilation, not app execution/scaling, which
doesn't touch the tainted rubygems.org repositories.

~~~
pardner
I don't think that is correct. AFAIK the 'development' side of the heroku
status panel is not related to developing, per se, it's the status for apps
that are not running on production-level resources... for example, single-dyno
apps, or apps not running with production flavor of database.

~~~
bgentry
You are incorrect: [https://devcenter.heroku.com/articles/heroku-
status#status-i...](https://devcenter.heroku.com/articles/heroku-
status#status-information)

I agree that the production/development split is not entirely clear without
additional explanation. We've spent a lot of time thinking about how to
communicate these things and have so far not come up with a way that we feel
better describes the issues at hand.

~~~
pardner
Thx for correcting that... have used that screen a million times without
realizing they lumped dev apps and prod+dev workflow together. 3 columns would
eliminate any confusion (Production Apps, Development Apps, Development
Workflow) but might not look at clean.

------
achalkley
Heroku a great.

------
djangolover301
lmao Ruby and RoR shame PHP in terms of security flaws. Making you unknowingly
write security holes, ridiculous flaws discovered on daily/weekly basis,
package management hacked etc I have never seen as ridiculous holes as RoR
even in the CodeIgniter framework. Where are the RoR-haters when we need them?

