
Matrix.org hacked - nektro
https://web.archive.org/web/20190412060115/http://matrix.org/
======
dininski
I can see a lot of people trashing on Matrix.org or the "hacker" themselves
(the hacker opened a series of issues, detailing how he managed to get in -
[https://github.com/matrix-
org/matrix.org/issues/created_by/m...](https://github.com/matrix-
org/matrix.org/issues/created_by/matrixnotorg)). However everyone seems to be
missing the point - matrix seems like a pretty cool and open project. And
someone taking over their infrastructure in such an open way is also great for
the community. Even though a little dubious on the legal side of things, I
believe it's great it was approached with transparency and a dose of humor.

Some might argue that this is harmful to matrix as a product and as a brand.
But as long as there was no actual harm done and they react appropriately by
taking infrastructure security seriously, it could play out well in the end
for them. This whole ordeal could end up actually increase trust in the
project, if they take swift steps to ensure that something like this does not
happen again.

~~~
tarruda
On the first issue opened by the hacker:

> Complete compromise could have been avoided if developers were prohibited
> from using ForwardAgent yes or not using -A in their SSH commands. The flaws
> with agent forwarding are well documented.

I use agent forwarding daily and had no idea it contained well known security
holes. If that's the case, why is the feature available by default?

~~~
XMPPwocky
SSH agent forwarding makes your ssh-agent available to (possibly some subset
of) hosts you SSH into. This is its purpose. Unfortunately, it also makes your
ssh-agent available to (possibly some subset of) hosts you SSH into.

~~~
scanr
Is there a secure alternative that achieves the same outcome?

~~~
metafunctor
Here are a few ideas that might help.

Use separate keyboard-interactive 2FA (I recommend google-authenticator) for
production ssh access.

Use a key system which requires confirmation or a PIN to authenticate (such as
a Yubikey). Use a persisting ssh connection with Ansible (ControlPersist) to
avoid unnecessary multiple authentications.

Allow connections only from whitelisted IPs, or Uuse port knocking to open
temporary holes in your firewall, or require connections to production
infrastructure to go through a VPN.

Access production infrastructure from hardware dedicated for that purpose,
never do anything else on it.

I wish there was a way in ssh to tag connections and only allow agent
forwarding to keys with the same tag. That would prevent agent forwarding
production keys from a dev host.

~~~
pas
[https://github.com/StanfordSNR/guardian-
agent](https://github.com/StanfordSNR/guardian-agent)

------
Arathorn
Project lead for Matrix.org here - you can see our initial statement on this
at [http://matrix.org/blog/2019/04/11/security-
incident/](http://matrix.org/blog/2019/04/11/security-incident/).

It will be updated shortly to reflect the DNS defacement linked here (which
was because we failed to rotate a leaked cloudflare API token; we aimed to
rotate the master API token but rotated a personal one instead). To our
knowledge the rebuilt production infrastructure itself is secure.

We've revoked the compromised GPG keys, and are obviously going to do
everything we can to improve our production security to avoid a recurrence in
future.

We can only apologise to everyone caught in the crossfire of this incident.

~~~
santamarias
do you have plans to perform an external security audit?

~~~
dvdgsng
didn't that just happen? ;)

------
azhenley
The hacker seems nice:

“Anyways, that's all for now. I hope this series of issues has given you some
good ideas for how to prevent this level of compromise in the future. Security
doesn't work retroactively, but I believe in you and I think you'll come back
from this even stronger than before.

Or at least, I hope so -- My own information is in this user table.”

[https://github.com/matrix-
org/matrix.org/issues/365](https://github.com/matrix-
org/matrix.org/issues/365)

~~~
erikbye
"Or at least, I hope so -- My own information is in this user table... jk, I
use EFNet."

~~~
user9182031
I enjoyed the shout out for EFnet.

------
m_b
For a bit of context: Matrix.org infrastructure has been hacked a second time
in 24h, after restoring everything they went down again, story developing
here:
[https://twitter.com/matrixdotorg/status/1116304867683905537](https://twitter.com/matrixdotorg/status/1116304867683905537)

~~~
m_b
The hacker is now doing a post-mortem in the GitHub issues of the project:
[https://github.com/matrix-org/matrix.org/issues](https://github.com/matrix-
org/matrix.org/issues)

~~~
nothrabannosir
This is gold...

 _> I noticed in your blog post that you were talking about doing a postmortem
and steps you need to take. As someone who is intimately familiar with your
entire infrastructure, I thought I could help you out._

> _There I was, just going about my business, looking for ways I could get
> higher levels of access and explore your network more, when I stumbled
> across GPG keys that were used for signing your debian packages. It gave me
> many nefarious ideas. I would recommend that you don 't keep any signing
> keys on production hosts, and instead do all of your signing in a secure
> environment._

~~~
lelf
Another gem:

RRREEEEEEEE> _I noticed you missed a doctype in your html page. In order for
web browsers to know what type of html to render you should include a doctype.
Thanks!_

matrixnotorg> @RRREEEEEEEE Thank you, I will consider that for the next
release

 _Edit:_ it got deleted

But see also:
[https://github.com/matrixnotorg/matrixnotorg.github.io/pull/...](https://github.com/matrixnotorg/matrixnotorg.github.io/pull/2)

~~~
justaj
Wait, did Github delete matrixnotorg's profile or did matrixnotorg?

If Github deleted that profile, I don't really see that as being very hacker-
friendly.

~~~
aeternus
Although 'hacker' is often used as a positive term on HN, breaking into a
company's production server is clearly illegal activity and should not be
condoned. If Github deleted the account, they are simply acting in accordance
to published TOS & policy.

~~~
justaj
If the attacker placed sensitive information on Github, that would indeed
warrant a deletion of the account. However from what I saw from the archives,
the attacker merely published details about Matrix.org's infrastructure and
its vulnerabilities. Is that something that's against Github's ToS?

------
wink
I think it boils down to the fact that infrastructure for projects (no matter
the size) is usually a second class citizen at best.

Either no one is eager to care for it, or the people who are actually focused
on developing the software run it because they need to, or worst case - no
contributor is trusted enough to handle infrastructure work, with access being
given even more sparsely than commit rights to the whole software. Which is
fine by itself, but there are so many (big) projects where infra is kind of
terrible because 3 out 100 people involved are doing all the work. Or don't.

~~~
proy24
It's just a problem with the industry that's like "let's just get something
out there and achieve product market first..will worry about infra and
security later" that later is just pushed into a backlog and forgotten.

~~~
dijit
As a classically trained sysadmin I've seen this trend going for the last
decade.

Some developers seem to be pushing that ops shouldn't exist any longer or
should be outsourced to google (who don't hire ops) or amazon (who do).

Managers see this trend and think that hiring only developers is a good way to
save costs and do things the "new way".

Traditional ops roles are indeed not as required but
security/process/reliability focused people should not be the same people who
write new features. They're in contradiction of each other often.

If you're a developer who thinks ops shouldn't exist any longer consider this:

I can write software and design websites as a sysadmin, does that mean I don't
need you now, or that I know everything you do?

I argue that it doesn't. A focus on automation is one thing but defenestrating
the notion of operations/SRE is going to net you a bad time.

------
growt
I'm probably really out of the loop, but what is matrix.org? Looks like an
open source slack clone? Why do they have >5 million user accounts? Is that
everybody who uses that chat tool?

~~~
kzcqt
Matrix is what happened when somebody looked at XMPP and yelled "NIH".

~~~
yjftsjthsd-h
Matrix is what happened when somebody looked at XMPP and yelled "wow, this
aged poorly and has some major usability issues".

~~~
kzcqt
And instead of fixing the issues they just went to do a completely new and
incompatible thing. That's the very definition of NIH

~~~
seba_dos1
Especially that, after some period of stagnation, XMPP is doing pretty fine
these days with stepping into the modern world.

~~~
vertex-four
Ehh... not really. I still can't find a good combination of server, desktop
client and iOS client that support things like OMEMO, history sharing between
clients, and voice/video chat. And the one iOS client (ChatSecure) looks
really dodgy and regularly fails while setting up push notifications.

~~~
kzcqt
And instead of adding those features to existing clients, let's create a brand
new protocol, server, desktop client and mobile client. Because why not?

~~~
yannovitch
maybe because, among many other reasons, it takes so much f __ __ __time for
something to change in the XMPP world, because you have to wait for the XSF to
validate any change, then all the server devs to implement it, then all the
client devs to implement it, then all the sysadmin to update their (very often
very old version of) XMPP server, then for the users to update their clients
(which, with Android fragmentation for example, is a PITA) ?

------
snvzz
4chan is circulating this picture. It shows the defaced website frontpage.

[https://i.4cdn.org/g/1555048975736.png](https://i.4cdn.org/g/1555048975736.png)

I have a hard time with the idea that they run the webserver and the matrix
server on the same computer. (Regarding users.txt)

It seems they do urgently need to hire capable infrastructure people.

~~~
josephmx
I can't access that image on my corporate network, any chance of an imgur
mirror?

~~~
couterSpell
It's just a screenshot of the same info shown on the archive.org page linked
by the title.

[https://web.archive.org/web/20190412060115/http://matrix.org...](https://web.archive.org/web/20190412060115/http://matrix.org/)

If you can't get to archive.org, just respond and I'll imgur it.

------
cjslep
I believe this is meant to show that it is a targeted attack on the project
lead:

[https://news.ycombinator.com/user?id=Arathorn](https://news.ycombinator.com/user?id=Arathorn)

Unfortunately I don't have any background context for possible reasons why
"actual transparency" on the top line is the issue chosen by the attacker, but
makes it seem ideologically driven.

~~~
JD557
I don't think that this is a targeted attack.

Seems more like a way of showing "I got access to 5493973 passwords and to
show that, instead of picking some random users, I'll pick the one responsible
for the shoddy security".

~~~
saagarjha
Or it might be a clear, concise way of showing that he has access to the
entire file without disclosing the information of random users, which also
happens to be a particularly short command.

------
rglullis
The interesting thing for me here is that none of the other homeservers were
affected. Despite the weak security on the largest servers, the ecosystem
stays alive.

Antifragility at its finest.

~~~
maxidorius
... or is it? [https://github.com/matrix-org/matrix-
doc/issues/1194](https://github.com/matrix-org/matrix-doc/issues/1194) and
[https://github.com/matrix-org/matrix-
doc/pull/1915](https://github.com/matrix-org/matrix-doc/pull/1915) and
[https://github.com/matrix-org/synapse/issues/4540](https://github.com/matrix-
org/synapse/issues/4540) would tell a different story: Potentially deleting
data on remote server just because being matrix.org (or anyone with an access)

~~~
rglullis
I might be misreading it, but it seems that the issues you are pointing out
related to 3PID, which is _still_ somewhat centralized. Sure, work on this
needs to be done yet the system is evolving to be more independent.

------
odensc
TL;DR: Looks like there was a server with an unpatched Jenkins instance
running, which allowed RCE. [0]

Someone (presumably a developer) was connected to that compromised server via
SSH, and had forwarded their SSH agent to it. [1]

Apparently that person had root access to the production servers, allowing the
attacker to login via the forwarded agent. Yikes.

[0]: [https://matrix.org/blog/2019/04/11/security-
incident/](https://matrix.org/blog/2019/04/11/security-incident/)

[1]: [https://github.com/matrix-
org/matrix.org/issues/358](https://github.com/matrix-
org/matrix.org/issues/358)

~~~
lucb1e
Thanks for that summary, the twitter thread that I read on it was not quite as
enlightening as this small summary!

------
subutux
It seems he used github for hosting his content on matrix.org
[https://github.com/matrixnotorg/matrixnotorg.github.io](https://github.com/matrixnotorg/matrixnotorg.github.io)

------
IronCoderXYZ
Looks like all issues created by the "hacker" have been removed?

[https://github.com/matrix-
org/matrix.org/issues?utf8=%E2%9C%...](https://github.com/matrix-
org/matrix.org/issues?utf8=%E2%9C%93&q=author%3Amatrixnotorg+)

~~~
gerogerke
Seems like the user itself has been deleted, which might cause Github to
remove all content created by that user.

~~~
Arathorn
we (Matrix.org) haven't deleted the issues; we were deliberately leaving them
up for reference.

------
trw999
As someone running a Matrix homeserver I take this incident as an example of
the benefits of decentralization. Unlike in more centralized services, the
security lapses of Matrix.org have had no affect on my homeserver.

------
xyproto
It's "usless use of cat". He/she should have gone:

`grep arathorn users.txt | head -1`

Instead of:

`cat users.txt | grep arathorn | head -n1`

Hackers these days.

~~~
0FDAA764
It's a "useless use of head". He/she should have gone:

`grep -m1 arathorn users`

Instead of:

`grep arathorn users.txt | head -1`

Commenters these days.

~~~
shanth
It's been a while seeing UUoC awards on a random internet discussion.

------
couterSpell
I like the fact matrixnotorg decided to alert Matrix to Elasticsearch's
existence.

But Matrix probably should first figure out how to fix the whole 'all server
management ports are open to the internet' problem detailed here:
[https://github.com/matrix-
org/matrix.org/issues/360](https://github.com/matrix-
org/matrix.org/issues/360)

The last thing we need is another Elasticsearch instance listening on a public
IP accessible to the world.

------
F30
The (presumed) attacker opened a bunch of issues in Matrix' GitHub issue
tracker, explaining the security issues leading to this compromise:
[https://github.com/matrix-
org/matrix.org/issues/created_by/m...](https://github.com/matrix-
org/matrix.org/issues/created_by/matrixnotorg)

TL;DR: A collection of inadvertences and suboptimal practices, some (like
having GPG signing keys on production systems) more worrying than others.
Something that could probably have happened to most orgs without dedicated
security resources.

------
NotOscarWilde
I lost all my messages with my girlfriend (of course).

Can anyone clarify: if I use their "server key backup" and set a passphrase, I
am now two passwords away from giving the next hacker read access to all my
messages, is that right?

------
perttir
They had root account activated in hebe? Am i reading this right? He got an
passlist of 5 million users?

~~~
beagle3
But even if they didn't, "sudo -u root /bin/bash" or similar gives it to you
unless sudo is extremely locked down (which, from audits I have done, is
"rarely if ever").

There are hundreds of ways to get root prompt even with the root account
nominally deactivated.

------
nullc
I never use SSH agent forwarding but instead use -D socks5 proxying to
directly ssh from my host to the ultimate target hosts.

------
humantiy
From reading the headline couldn't help but think to myself was it Neo?

------
Oblongoid
I've been slightly annoyed with matrix ever since they boasted at FOSDEM with
the fact that they backdoored their encryption so that the French government
could Virus scan sent files. :/

~~~
Arathorn
we didn't backdoor the encryption. instead, we specced how clients _could_
securely pass attachment keys to an AV server, if they need to. but in
practice none of them (other than the french app) do.

the whole point was to spell out that we _haven 't_ backdoored the encryption,
and instead been transparent about how content filtering could be done in the
most responsible manner, if it's really needed.

------
santamarias
in order to regain trust in matrix.org, what other options exist than an
external security audit?

~~~
ndnxhs
I'm far less concerned about this because all my messages were end to end
encrypted.

------
standinator
Stop cat abuse.

------
zaarn
Doesn't surprise me that much, Matrix doesn't seem to be too concerned with
security, more with security theatre (considering you can still not easily
disable read receipts in your client, a major privacy leak IMO, among other
issues).

~~~
meruru
Disabling read receipts is a client feature. Yes, the currently most mature
client doesn't have that feature, but nothing in Matrix precludes it.

~~~
zaarn
There is plenty of other issues with matrix and the reference clients on top
of something as simple as mandatory leaking of your presence in a chatroom.
I've run a matrix homeserver for almost 3 weeks and it as an utter pain to
maintain, despite not a single version upgrade and I was plagued with issues
that no chat platform would have if the protocol was remotely sane.

edit: That is on top of the numerous security issues this hack uncovered.
Apparently the matrix.org devs kept a users.txt file with a dump of users +
passwords on the server. Signing keys for debian packages were stored
unencrypted on the production server. People used unsafe SSH settings (SSH
Agent Forwarding), ran outdated servers with known root-priv RCEs for months
and root privileges for all users on a server. Why should I ever trust a
matrix developer with their protocol or reference implementations ever again
if they can't be trusted with the simple task of updating a service when a
critical CVE comes out?

~~~
pferde
While you bring up valid concerns about the Matrix team's security hygiene,
the point of an open standard is that anyone can (try to) spot flaws in it,
and anyone can (try to) create their own implementation.

I myself am waiting for a healthy ecosystem of servers and clients to spring
up before starting to rely on Matrix for anything non-ephemeral - even if it
takes years. Perhaps I'll even try my hand at writing a client, if I ever run
out of things to do. In the meantime, I will dick around with a throwaway
matrix.org account to play with it, and to watch progress happen.

~~~
tannhaeuser
> _myself am waiting for a healthy ecosystem of servers and clients to spring
> up before starting to rely on Matrix_

Good luck with that. Right now there's only the centralized matrix.org server,
or actually there isn't because it's down. If you want open standards and
multiple servers (or your own) use XMPP period.

It's not so much a technical question as it is the attitude of "hey we're
implementing our own chat protocol cause XML sucks". Totally not getting the
point why users and developers would want to use standard protocols - to save
their efforts becoming obsolete, taken over by a single entity, or both. It
doesn't help either that scarce development resources are needlessly
fragmented between XMPP and matrix.

That said, if the matrix protocol can actually manage to attract users and
multiple implementations some years down the road (about 30-40 years after
IRC), more power to them.

~~~
joepie91_
> It doesn't help either that scarce development resources are needlessly
> fragmented between XMPP and matrix.

In my experience, there's virtually no overlap between the two groups, and
therefore no fragmentation. And for good reason: XMPP is a nightmare to
implement, so there's a significant group of developers that just won't touch
it, but that _might_ be interested in working on Matrix.

And yes, part of the blame for that lies in the usage of XML. While XML _can_
be useful to represent complex data or documents, it's unsuitable as an over-
the-wire format because it doesn't have a directly mappable representation in
most languages, due to the combination of attributes and child nodes.

This problem doesn't exist for JSON, because pretty much every language
directly supports arrays, objects/maps and primitives. This makes a JSON-based
protocol much more pleasant to work with, as there is less data-wrangling
complexity involved.

~~~
pferde
You know, I never understood why people consider JSON better than XML. Yes,
any particular use of XML can be overengineered (namespaces, I'm looking at
you), but as long as you control the format or scheme or however you want to
call it, it's exactly the same thing as JSON, but encoded differently. In the
end, it's all just keys and values or lists of values, arranged in a tree-like
hierarchy.

And frankly, I would rather be looking at a well-designed XML format than at a
well designed JSON format, with its braces and brackets and commas.

~~~
tannhaeuser
I don't like XML namespaces either (and neither is the original authors of the
namespace spec very proud of it [1]). They're greeting you with verbose and
rather ugly pseudo-URLs (another bad and confusing concept IMO) and xmlns:xyz
boilerplate on page one when you're interested in quickly gleaning XML data.

But arguably, chat log data is actually an appropriate use cases for
namespaces, given that you would want a text format that can evolve over time
in a heterogenous client and server ecosystem, yet provide a baseline
functionality supported by all clients. It's also very helpful if you want to
keep chats for archival rather than treating chat as an ephemeral medium. OTOH
people have said the excessive use of namespaces and other XML modularization
features, and too many XEPs/RFC specs is turning them away from developing
XMPP software.

There are valid use cases for JSON though such as ad-hoc data protocols where
you own both the server and (JavaScript) client and maintain those in the same
repo, and when dealing with simple app data that doesn't benefit from using
markup constructs.

[1]:
[https://www.tbray.org/ongoing/When/201x/2010/01/17/Extensibl...](https://www.tbray.org/ongoing/When/201x/2010/01/17/Extensible-
JSON)

------
GrumpyNl
Everything gets hacked, period. They have to remove this from there slogan,
"An open standard for decentralised secure communication."

~~~
johnchristopher
Really ? That's a bold claim. Can this string be hacked ?
$6$WqQvgIwr$/BA/19FRJZf.z4buUwX5Sbls07ovv/lVZJ3V2En7VgTR4Skdpz98hFNuq3VX4nIaiSDKObO9jKbkPb5tdt7zY1

~~~
chme
I guess the point is that you cannot protect anything from the future if you
don't just destroy it completely.

~~~
meruru
On that note, the matrix.org server currently doesn't delete messages from
rooms that have become empty. It probably would be a good idea to do so.

~~~
acct1771
Does synapse allow that for self hosted?

~~~
meruru
Yes, IIRC it's a script you have to run periodically.

