

Vulnerabilities in Heroku's Build System - Titanous
http://titanous.com/posts/vulnerabilities-in-heroku-build-system

======
tptacek
_Heroku offered me a paid penetration test contract, but required that I sign
a retroactive non-disclosure agreement which would have precluded publishing
this article._

Worth pointing out: virtually _all_ paid pentesting engagements are delivered
under NDA. In fact, more often than not, they're done under the far-stricter
terms of a master agreement with detailed IP clauses.

If you're talking to any firm about having your app tested, get an NDA in
place, and don't feel bad about asking. Nobody who thinks they need it should
forestall having their app checked out because they think the firm they're
going to work with is going to try to make news out of their findings.

Obviously, if your customers find vulnerabilities themselves, all bets are
off!

~~~
Titanous
Of course there was never any question that any new penetration testing I
would do with them would be under NDA. What worried me is that the disclosure
may not be made public if I was under a NDA that prevented me from publishing
it. I didn't want to have any professional or ethical conflicts.

~~~
novalis
The retroactive part of the deal was a clear ominous alarm and you acted
accordingly. I find great joy in knowing that you showed a pristine level of
ethic behaviour in your security research. Well done.

~~~
tptacek
Nothing ominous happened here at all.

------
Titanous
Heroku's official response:
[http://blog.heroku.com/archives/2012/7/3/codon_security_issu...](http://blog.heroku.com/archives/2012/7/3/codon_security_issue_and_response/)

------
kposehn
Very glad to see this not only documented, but patched extremely quickly.
Heroku continues to impress, and Titanous is a credit to the security
profession.

------
seany
Thank you for taking the high road and exposing the issue rather than giving
them the option to hide it (not saying they would).

~~~
cjeane
This is the concept of Full Disclosure.
<http://en.wikipedia.org/wiki/Full_disclosure>

------
redslazer
This is a really interesting insight into the way someone found access to a
system but could someone explain to me why he needed PGP keys from heroku? I'm
sure there is some good reason but if someone could tell me that would be
great.

~~~
leftnode
It's standard practice to encrypt your emails when sending potential security
notifications to the party that you're notifying. Not only does it keep out
prying eyes, it ensures the person sending the email is who they say they are.

~~~
bigiain
"it ensures the person sending the email is who they say they are"

Although, if you're going to trust the pgp key you get sent when you mail
someone asking for one, you don't have any more assurance that you're
encrypting it to the person you intended, and you _almost_ may as well have
sent it in cleartext in the initial email...

Get your pgp fingerprints in some out-of-band method of communication. (When
was the last time anybody even heard of a key signing party? I attended on at
the Perl Conference in '98 or '99 - don't think I've been aware of one
happening in any of my circles since)

~~~
Titanous
In this case I was able to confirm the key by matching it with a copy they
uploaded to <https://policy.heroku.com/security>

~~~
bigiain
Good to hear.

Now the paranoid in me is wondering - if there's an attacker deeply entrenched
enough to be reading their plain-text email, either on the wire between them
and you or with sufficient root privs on their infrastructure, they could
probably have arranged their own pgp key to appear there for you as well. If I
were trying to ensure I was doing my utmost best diligence, I'd have tried to
include a completely non-internet backchannel - perhaps a call phone number
for Heroku sourced from a dead-tree phone book (if such a thing exists
anymore?) and get someone to read out the key fingerprint.

~~~
munin
if they're that owned, it's highly unlikely that you telling them about some
additional vulnerability is going to help their attackers. and you'll figure
it out soon enough when none of your proposed fixes are enacted.

~~~
bigiain
True - but if they're _not_ that owned, TLS encrypted email probably would
have been sufficient. (Though I'm not sure how easy it it to force/ensure TLS
in common email clients…)

~~~
lambda
TLS only protects a single link; from your client, to your server. It doesn't
prevent you from disclosure on that server, on any relaying servers in
between, or between their server and their client (remember, they may be
reading email on wifi in a coffee shop).

S/MIME email is another end-to-end encryption scheme, like PGP, but it isn't
as popular among a technical crowd as PGP is.

------
lawn
That's a nice way of handling it, both from heroku and you who found the
exploit. I feel content with my choice to use heroku.

------
AndyKelley
I'm happy to see both parties in this situation acting with a great deal of
professionalism. It feels good to be a part of this community.

------
danso
This is a minor point, but one of my favorite details: the use of a simple
table for the "Disclosure Timeline", which is really the clearest way to
illustrate the step-by-step sequence of events. I would love to see this be a
standard practice for any narrative in which order/visual comparison is vital.
HTML tables are so easy to include (or, even bulleted lists).

A masterful explanation, on top of being an altruistic deed.

------
illamint
Perhaps a fantastic example of why keeping detailed configuration in
environment variables is probably a bad idea. Your private SSH key isn't part
of the environment. It's configuration. Treat it as such.

~~~
bigiain
Not sure that would have helped much in this case - he used dumping the
environment as the particular avenue to get the "private" data, but he
mentioned he had source code access and the ability to run untrusted code. If
important credentials were in or available to the code, it sounds like they
would have been vulnerable anyway.

It's a hard problem trying to secure credential that code needs to work, from
other code running as the same user when someone has source code access to
"authorised" code.

------
justincormack
Well someone could have easily discovered the environment variables vuln
without getting the source code, although it is unclear why so much was
disclosed in env vars. It suggests that everything runs as the same user, or
you would not get anything.

------
sodelate
the page is well composed

