
Matrix.org Security Incident - Perceptes
https://matrix.org/blog/2019/04/11/security-incident/
======
zigara
The attacker seems to have responded:

[https://github.com/matrix-
org/matrix.org/issues/357](https://github.com/matrix-
org/matrix.org/issues/357) edit: just saw the rest:
[https://github.com/matrix-
org/matrix.org/issues?utf8=%E2%9C%...](https://github.com/matrix-
org/matrix.org/issues?utf8=%E2%9C%93&q=is%3Aissue+SECURITY)

"[SECURITY] SSH Agent Forwarding

I noticed in your blog post that you were talking about doing a postmortem and
steps you need to take. As someone who is intimately familiar with your entire
infrastructure, I thought I could help you out.

Complete compromise could have been avoided if developers were prohibited from
using ForwardAgent yes or not using -A in their SSH commands. The flaws with
agent forwarding are well documented."

------
iancarroll
Did the blog get hacked (again?) in between this being posted and now? It has
what looks like password hashes and `uname -a` from every(?) server in their
infrastructure.

This is about as bad as IR can get: you realize you got hacked, you re-build
your entire infrastructure and publicly say it's fixed, and then you get
popped again...

~~~
snailmailman
i think so. here is a wayback machine link to the previous blog post
[https://web.archive.org/web/20190412000400/https://matrix.or...](https://web.archive.org/web/20190412000400/https://matrix.org/blog/2019/04/11/security-
incident/)

it stated "Having fully flushed out the attacker [...]" which i guess turned
out to be false :-/

also im getting invalid HTTPS certs on the blog now. for some reason im
getting a cert that looks like its for github.com ?

edit: now im getting a lets encrypt cert on matrix.org, but a cloudflare SSL
error page when i go to www.matrix.org ? the lets encrypt cert looks like it
was _just_ issued about an hour ago.

edit2: i guess both with and without www. are lets encrypt, but the _with_
www. cert was issued back in february (and gives a cloudflare SSL error page),
while _without_ www. was issued today. (and gives the current hacked message)

~~~
deepwell
Here is the message of the hacker, after the matrix.org admins allegedly
cleaned everything up and wrote the original blog post:
[https://web.archive.org/web/20190412055614/https://matrix.or...](https://web.archive.org/web/20190412055614/https://matrix.org/blog/2019/04/11/security-
incident/)

------
ge0rg
The most favorable reading of the current defacement page is that the
attackers still controls the DNS, but no other parts of the infrastructure.

Otherwise, the page probably wouldn't run off github.

------
irgeek
Assuming the GitHub issues are from the actual attacker -- and I see no reason
to doubt they are -- this is very troubling:

[https://github.com/matrix-
org/matrix.org/issues/363](https://github.com/matrix-
org/matrix.org/issues/363)

 _Compromise began well over a month ago_

Yikes. That's a long time for a compromise to go unnoticed.

~~~
ygjb
Not really, the average in the industry seems to be floating between 70 and
400 days depending on the source of your stats on the topic (different vendors
and reports use different stats for this)

------
mkj
> As we had to log out all users from matrix.org, if you do not have backups
> of your encryption keys you will not be able to read your encrypted
> conversation history

That seems like a fairly bad usability/security design?

------
localhostdotdev
content before it gets fixed:

    
    
        Time for actual transparency.
    
        [list of servers, uname -a for each]
    
        root@[name]:/var/lib/postgresql# df -h
        [list of partitions]
    
        $ cat users.txt | grep [name] | head -n1
        @[name]:matrix.org|[hash]
        $ wc -l users.txt
        [~6M users]
    
        See you soon.
    

(affects whole site, even [https://matrix.org](https://matrix.org), site is on
jekyll BTW)

~~~
l2dy
Here's the source of that page:
[https://github.com/matrixnotorg/matrixnotorg.github.io/blob/...](https://github.com/matrixnotorg/matrixnotorg.github.io/blob/master/blog/2019/04/11/security-
incident/index.html).

~~~
localhostdotdev
account got banned :) (it's on archive.org though)

------
arunc
Why was Jenkins running on a production server?

~~~
deepwell
it was not, read again

~~~
Perceptes
But it does seem to be the case that the same SSH key pair that was used to
access Jenkins also provided access to the production infrastructure. Unless
I'm misunderstanding the nature of the attack.

~~~
zigara
It seems the issue was developers using SSH agent forwarding which was abused
to access the production environment.

