
What can we learn from the matrix.org compromise? - cyber
https://medium.com/@tomsparks/what-can-we-learn-from-the-matrix-org-compromise-c6ae06dcaab
======
pm90
This is such a poorly written article:

* no detailed analysis of how the attack was undertaken. Its not even clear how the attacker managed to get in (was it a publicly exposed Jenkins? vulnerable bastion? what?)

* no analysis of what the existing matrix.org security perimeter looked like or how it could be made better.

* repetition of security tropes. Use VPN. Use Github Enterprise (wait wtf? Why not private repos in Github?). Don't use Ansible, use salt.

Ridiculous. I was looking forward to a nice long read about how this breach
was undertaken. Hugely disappointed.

~~~
bifrost
If you click through to the GH Issues I linked to there are some pretty good
data points as to what happened. I didn't feel the need to copypasta.

But yes, publicly exposed jenkins and repos lead to the compromise, not an
uncommon story unfortunately.

Perimeter - I didn't see much evidence of one existing and I didn't go probing
their networks to find out.

Security tropes are real for a reason, you don't have to believe me though.

Private repos in GitHub are still publicly hosted and are orders of magnitude
easier to get into than having an in perimeter repo. They've leaked before and
they'll keep on leaking. GitHub even made it harder for people to fork private
repos to their own public accounts but it still happens.

~~~
pm90
> They've leaked before and they'll keep on leaking. GitHub even made it
> harder for people to fork private repos to their own public accounts but it
> still happens

Can you provide some actual instances of this happening? Genuinely curious, as
my org is currently migrating from enterprise to cloud.

~~~
bifrost
I've mostly seen this reported in forums and during discussion, if you Google
around you'll find some pretty useful hits.

Here's a good one from reddit:
[https://www.reddit.com/r/github/comments/9odnvw/someone_fork...](https://www.reddit.com/r/github/comments/9odnvw/someone_forked_a_private_repo_and_made_it_public/)

Its also discussed reasonably well in the infosec community. Basically GitHub
is a great place to find other people's passwords and API keys.

~~~
baroffoos
Thats unrelated to github though. It sounds like the person did a git clone
and then created a new repo and pushed it. You could do that with a self
hosted git repo as well. To stop that you would have to have your git server
block logins from non company machines and have some serious logging on all
company machines to stop anyone moving it off via usb

------
KirinDave
Why aren't people reporting the fact that Matrix.org actually lost control of
their network a second time within hours of their first all clear sounding?

I feel like this is an important part of the story for anyone looking for
teachable infosec moments.

~~~
driminicus
Because the second tine was a dns hijack, not a network compromise. I'm a
little fuzzy on the details, but it had something to do with cloudflares API
not revoking some access token.

Either way, a DNS hijack is not great, but not nearly as bad as the initial
compromise.

~~~
bifrost
It wasn't CloudFlare's API not revoking a token, they just didn't revoke all
the tokens. Basically human error.

"The API key was known compromised in the original attack, and during the
rebuild the key was theoretically replaced. However, unfortunately only
personal keys were rotated, enabling the defacement."

------
Arathorn
If it wasn’t clear, this article wasn’t written by the Matrix.org team, nor
did the author discuss any of it with us to our knowledge.

We’ll publish our own full post-mortem in the next 1-2 weeks.

~~~
Arathorn
also, reading this article more carefully, much of this just plain wrong:

> One of the more interesting pieces of this was how Ansible was used to keep
> the attacker in the system.

Fwiw the infra that was compromised was not managed by Ansible; if it had been
we would likely have spotted the malicious changes much sooner.

------
nisa
It's been a few years since I last used Saltstack but if you have access to
the master you have instant root on all minions or did that somehow change?
salt '*' cmd.run 'find / -delete' and game-over?

~~~
bifrost
Very true, however I'd rather have that problem than an ever multiplying
number of user accounts on systems that can su/sudo.

~~~
verdverm
Make golden images with packer, or something similar, and then roll your fleet
over.

You should not be running package managers on production servers. Or any of
the other things salt, ansible, chef, puppet can do.

~~~
yjftsjthsd-h
As in, no _human_ should run a package manager in prod? (But salt/ansible/etc.
running it is fine) Same idea as "if you're SSHing to prod, something is
wrong" (where provisioning tools make all changes, logs are all aggregated and
delivered in their own tool, and even debugging is built into the app or
logging system).

~~~
verdverm
More as, you should not modify images running in production. By human or
machine.

Rather build new images and roll over the fleet. If you need to debug, remove
from production (quarantine) and work on it there.

Don't run master / agent setups for ansible / salt anymore. You can still use
them for creating images, which are later turned into running VMs. Think about
it like containers. Do you update the contents of your running containers, or
log into your containers to make changes?

Better yet, use OSes that cannot be modified.

~~~
Spivak
But golden images on Linux are, well, messy. It's very annoying to make a
clean VM template without some post-provisioning like cloud-init. And for most
shops if you're running cloud-init you could do that post-provisioning with
Ansible or Salt. And since your images are built with Ansible/Salt in the
first place you might as well just build each VM fresh and use the vendor's
ISO. One less thing to maintain and update.

Plus when you're in a pinch, which never happens of course, you can make
changes without having to roll your VMs.

I feel like Atomic distributions are basically the happy medium between the
two worlds.

~~~
verdverm
It's a trade-off. The point is the matrix security lapse turned worse because
they ran this master / agent setup. You can still use ansible (or similar),
just do it localhost during the build process.

Yea it's easier to not do these things, because good security posture takes
work to set up. Once you're on the immutable train, you'll find it's not
actually harder day to day. You learn to deal with issues in the pinch another
way.

On the point of building VMs fresh each time vs building golden images, you'll
find you boot time reduced, your roll over more reliable and autoscaling more
responsive. Why build the same thing dozens or hundreds of times? What happens
if a remote package is updated in the middle of your upgrade? Does this sound
messier to you?

------
ubercow13
Why is it considered safer to expose a VPN to the internet than SSH? Is it
just that there is one exposed service for the organisation rather than one
per machine?

~~~
bifrost
SSH tunneling is handy but if you want to push anything else over it, its a
pain for the "layperson". You're not going to have a great time supporting
people with it. I've done it, it sucks. Scripts and special SSH config files
are the pits. VPNs are way easier, they can support multiple access levels and
roles, are often not blocked by other people's packet filters and firewalls
and the good ones can even validate that a host is in "compliance" before
they're allowed onto the network.

------
krupan
Can anyone explain the Jenkins vulnerability that was used to initially gain
access? Reading the CVEs didn't give me the impression that they enabled
remote exploits

~~~
bifrost
My 5 second lazy summaries of the CVEs:

CVE-2019-1003001, CVE-2019-1003002 -> Anyone with read access to Jenkins can
own the build environment.

CVE-2019-1003000 -> I didn't get a lot of the details on this but it basically
looks like "broken sandboxing, you can run bad scripts".

This is also a good resource:
[https://packetstormsecurity.com/files/152132/Jenkins-ACL-
Byp...](https://packetstormsecurity.com/files/152132/Jenkins-ACL-Bypass-
Metaprogramming-Remote-Code-Execution.html)

------
zimbatm
The attacker gained network access through Jenkins.

Don't deploy a public-facing Jenkins, especially if it has credentials
attached to it. It's really hard to secure, especially if pull-requests can
run arbitrary code on your agents.

Jenkins / CI is the sudo access to most organizations.

~~~
bifrost
I agree with you 100% here, I would not deploy any CI publicly unless its
heavily fenced off into "read only" territory.

------
r1ch
One thing I learned was where to modify the pageant source code (Windows
equivalent of ssh-agent) to make my agent prompt before signing (with the
default focus on "no"). This feels much safer and is a very minor
inconvenience. I wonder why more agents don't have this built in.

Example:
[https://twitter.com/R1CH_TL/status/1118559239084158977](https://twitter.com/R1CH_TL/status/1118559239084158977)

------
forgotmypw
I'd like to take this opportunity to plug my in-development decentralized,
distributed, completely open forum, using PGP as the "account" system, and
text files as the data store.

So any reasonably competent hacker can re-validate the entire forum's content
and votes, reasonably quickly reimplement the whole thing, and/or fork the
forum at any time.

[http://shitmyself.com/](http://shitmyself.com/)

~~~
ficklepickle
This is very interesting! I have so many questions. If you see this, kindly
send me an email. It's in my profile. I love the idea!

------
mjevans
That medium.com has a paywall and doesn't want to share content? (is what I
learned)

~~~
bifrost
This might work: [https://medium.com/@tomsparks/what-can-we-learn-from-the-
mat...](https://medium.com/@tomsparks/what-can-we-learn-from-the-matrix-org-
compromise-c6ae06dcaab?sk=0517b377c873db74a6fb1d8c29e18baa)

------
inetknght
I have gone on some long verbal rants about the dark patterns (bordering on
malicious behavior) exhibited by key agents such as SSH agent, GPG agent,
Pageant, and the like.

What can you learn from the compromise? Never use an agent. Kill it with
fire^H^H^H^H -9.

~~~
nine_k
How about using hardware tokens instead? With a right setup, private keys
never leave it.

~~~
scurvy
Smart cards? They were designed for this.

~~~
bifrost
Ever seen anyone working in a Coffee shop using one? Me neither.

Good security technology exists, the problem is that people don't want to use
it because its easier to ignore it.

~~~
andrewshadura
I have, and I have myself used one until it was stolen from me with a bag it
was in.

