
How I gained commit access to Homebrew in 30 minutes - bgentry
https://medium.com/@vesirin/how-i-gained-commit-access-to-homebrew-in-30-minutes-2ae314df03ab
======
thecodemonkey
From the Homebrew post about the incident: “The security researcher also
recommended we consider using GPG signing for Homebrew/homebrew-core. The
Homebrew project leadership committee took a vote on this and it was rejected
non-unanimously due to workflow concerns.”

How is PGP signing _not_ a no-brainer. What kind of workflow concerns would
prevent them from signing commits?!

~~~
geofft
If you make PGP signing easy enough, at some point you end up with a Jenkins
with a trusted PGP signing key, and you haven't actually solved anything.

The problem isn't making it easy to sign things, the problem is making it
sufficiently _hard_ for unauthorized parties to sign things without affecting
any workflows you'd like to preserve - that is, the real problem is a workflow
problem. The real problem is figuring out how to secure automation so it has
the privileges to do what it needs but isn't leaking access. The Jenkins
instance was designed to do authenticated pushes - it needs automated write
access to the Homebrew repos.

Also, signing _commits_ doesn't help you if the risk is unauthorized pushes to
master. You can pick up someone's test commit and push that to master, or push
a rollback of OpenSSL to a vulnerable version, or something, and still ruin
many people's days.

~~~
josephh
Are there any OSS maintainers who use air-gapped computers to sign packages
(at least for major versions)? I would expect this level of precautions be
taken for projects that may have large-scale repercussions in the event of a
security breach.

~~~
TimWolla
While not OSS the Debian packages of Tarsnap are built and signed on an air-
gapped machine: [http://mail.tarsnap.com/tarsnap-
alphatest/msg00033.html](http://mail.tarsnap.com/tarsnap-
alphatest/msg00033.html)

~~~
geofft
How much review does 'cperciva do of the source code and build-dependencies
that are copied to the air-gapped machine, which presumably originate from
internet-connected machines?

Also, how secure is the kernel on the air-gapped machine against malicious
filesystems on the USB stick? (If it's running Linux, the answer is almost
certainly "not;" I could imagine FreeBSD is better but I don't know how much
people have explored that.)

To be clear I'm not opposed to air-gapping if the maintainer is excited about
it, I just suspect there are many much weaker links on the way to/from the
air-gapped system, and fixing those is a much harder project that almost
nobody is excited about.

~~~
cperciva
_How much review does 'cperciva do of the source code and build-dependencies
that are copied to the air-gapped machine, which presumably originate from
internet-connected machines?_

I verify that the source code being compiled is the source code which is
published in a signed tarball. Yes, someone could have tampered with the
internet-connected system where I do Tarsnap development, but their tampering
would be visible in the source code.

Build dependencies are verified to be the packages shipped by Debian. If
someone has tampered with the Debian gcc package, we've lost even without
Tarsnap binary packages.

 _Also, how secure is the kernel on the air-gapped machine against malicious
filesystems on the USB stick? (If it 's running Linux, the answer is almost
certainly "not;" I could imagine FreeBSD is better but I don't know how much
people have explored that.)_

I don't use a filesystem on the USB stick I use for sneakernet, for exactly
this reason -- I write and read files to it using tar. (Yes, you can write to
a USB stick as a raw device just like you would write to a tape drive.)

~~~
ddalex
The 'malicious file system' on a USB stick is not something to worry about -
the firmware of your USB stick is. People (ie for-fun hackers) modified
firmwares on USB sticks to make them look like HID keyboards and send commands
to target computers - it is well enough in the capabilities of a determined
adversary to own your internet machine and implant something on your USB
stick.

For secure airgapped computers I'd use one way low tech comm channels with no
side bands, maybe IR or sound? (if you trust the device drivers)

~~~
MauranKilom
What happened to CD/DVD (or floppy)? Too small?

------
raesene9
This is an interesting example of a long-standing problem which is that in
general we make use of huge amounts of software and trust our security with
it, whilst knowing very little about the security practices of the people
developing it.

The article makes a good point that it's very hard for small projects, like
the team running Homebrew, to fund their security, yet they are likely to be a
target for quite high end attackers, given the access that can be gained by
getting unauthorised access to package repositories.

As a side note it also shows that Jenkins tends to be a tempting target for
attackers as it often has access to a wide range of systems to carry out it's
functions.

~~~
mikemcquaid
> As a side note it also shows that Jenkins tends to be a tempting target for
> attackers as it often has access to a wide range of systems to carry out
> it's functions.

This. I'd really love to stop us using Jenkins but none of the hosted macOS CI
services scale to meet our needs (i.e. having jobs that run for multiple hours
on better hardware). The ideal solution for me would be for us to have some
sort of modified Travis or Circle CI setup. We can even pay for it now we've
got Patreon money coming in.

~~~
alien_
Jenkins is not the problem, we have quite a few secured instances which
wouldn't leak secrets like this, or at least not to non-admin or
unauthenticated users.

It's just often misconfigured because there are a plethora of plugins and ways
to store and use secrets, and nobody audits it enough to look at the console
output or build artifacts for leaked secrets.

The project is simply missing someone familiar enough to configure Jenkins
properly.

~~~
mmt
> The project is simply missing someone familiar enough to configure Jenkins
> properly.

I would say that this is a specific case of the far more general one.
Substitute "any small organization" for "the project" and substitute any
configuration familiarity (i.e. Ops skill) for Jenkins configuration
familiarity.

Even in "Devops" job postings for startups, when mentioning CI/CD tools like
Jenkins, the main desire seems to be to hire someone who's more Dev than Ops,
to create the code to run the CI/CD pipeline, with something like
configuration or security a mere afterthough, if that.

------
hartator
> I want to give special thanks to Mike McQuaid for his quick and professional
> handling of my report while on his paternity leave.

Kudos on that. Specially when the all team make only around $700-800 per
month. [1]

[1] [https://www.patreon.com/homebrew](https://www.patreon.com/homebrew)

~~~
mikemcquaid
Thanks! To be explicit though: that money all goes to the project and not to
individuals. I've never made a cent from working on Homebrew.

------
decasia
One would think that Apple would be a prime candidate for contributing to
Homebrew security funding, since in practice it is a project with security
implications for so very many OSX developer workstations.

~~~
sanderjd
Unfortunately, it has become increasingly clear that developer workstations
are not on their radar as a primary target market.

~~~
tambourine_man
I wouldn’t say that. Apple's hardware neglect is not a privilege only
developers experience.

~~~
dep_b
I don't think the 2018 MacBook Pro leaves much to be desired unless you want
features that would turn it into a ThinkBook-like brick. For $50 you get a
charging USB-C hub that allows you even to leave your power brick home. The
iMac Pro is pretty great.

Power Mac is a different story.....don't know how and when they're going to
fix this. That's a real shame.

~~~
kbd
> unless you want features that would turn it into a ThinkBook-like brick.

Yeah literally all many of us want is a standard keyboard with function keys
instead of the touchbar. A couple regular USB ports would be nice as well so
we can plug in a mouse and keyboard without needing dongles. Neither of those
things would turn their "pro" laptop into a brick.

~~~
dep_b
MacBook 2015: power brick (with broken irreplaceable cord) + Ethernet dongle

MacBook 2018: USB-C dock with replaceable cord

So in my use case I actually got rid of one dongle at a price lower than a
replacement MagSafe 2 adapter. I really liked MagSafe though.

Never used the F-keys really. ESC is not that hard for me to live without but
that's all personal. But the difference in performance is quite noticeable.

~~~
JBiserkov
>I really liked MagSafe though.

The Surface line by Microsoft has a very similar magnetic power+docking
connector. The only issue I have with it is that the LED doesn't change color
when the device is fully charged.

The line includes:

\- Laptop,

\- Pro (tablet + typecover)

\- Book (tablet + keyboard + GPU + battery)

------
kovrik
I think that publicly exposing your build/deployment system is a very bad
idea. It contains too much sensitive and valuable information (credentials,
commits, commit author names, paths, hostnames, scripts etc.) and it is too
hard to make it secure the right way.

Unfortunately, many people do that. For example, here in New Zealand there is
a new startup Onzo (bike sharing). When they launched, I tried to google about
them (just out of curiosity). And in 1 minute I found their Jenkins server
exposed to everyone as well. I could see how their build process works, who
commits what etc. I decided to try to login using simple credentials
(something like admin:password) and it didn't work. But there was a Register
button. "Why not?" I thought and clicked it and created my own account. And
voila, it gave admin permissions by default - I could delete their projects,
change variables etc. Emailed them about that.

Moral: never expose your build/deployment systems. If you really want to
expose some parts (for whatever reason), then use/write client/UI that has no
permissions. 'Build Status' badge is a good example -- it exposes build status
info, but doesn't show too much and doesn't give any permissions whatsoever.

~~~
falsedan
Jenkins sucks for a lot of reasons but it does have a perfectly serviceable
credentials store exactly for hiding these kinds of secrets from the
parameters page and the build output. Any release engineer with the slightest
inclination to avoid incidents would have set it up, this just looks like a
lack of experience at breaking everything for everyone.

~~~
kovrik
But that's the thing: many people don't bother configuring it properly (or
they don't know how?). So why risk it?

~~~
falsedan
sometimes the risk analysis comes out in favour of providing a service for
your users ahead of studiously avoiding making mistakes

------
sytelus
This is an excellent piece. I often wonder about adversarial security issues
in large scale OSS projects like Linux kernels. You don't even need to hack
commit access to repo. One can _intentionally_ wrap malicious code out in
plain sight in an otherwise what would appear as benign change (thanks to
undefined behaviour in C/C++). What if a black hat hacker climbs up in Linux
contributors hierarchy? What if a person who is already higher up in OSS
hierarchy decide to defect and plant a logic bomb? Given that Linux kernel now
runs majority of our world from servers in data center to mobile phones in our
pockets and hospitals to war machines, security issues like this is a huge
deal.

~~~
ejholmes
It's a pretty scary prospect, to the point that I have to imagine it's already
happening to some degree. If a nation state wants a backdoor, what better way
than to bribe the cash-strapped OSS maintainer of that little project that
every company depends on.

~~~
ddalex
The problem is that the type of engineers that work on OSS takes own integrity
very seriously, and they build their network of trust on that integrity.

------
binbag
I bet it's just what the guy wanted to find out he had to deal with while on
paternity leave!

~~~
mikemcquaid
Thank god for naps.

------
closeparen
This is a fundamental problem of doing CI for open source. Running tests for
random people’s changes means giving them RCE in your test environment.
Sandboxing rarely gets approached with the seriousness it requires.

------
jlv2
> "On Jun 31st,"

As said in _Raiders Of The Lost Ark_ , bad dates.

------
sleepybrett
Lol, always jenkins (CI).

