
We abused Slack's TURN servers to gain access to internal services - reader_1000
https://www.rtcsec.com/2020/04/01-slack-webrtc-turn-compromise/
======
russellbeattie
So Slack's VoIP uses WebRTC, which connects via UDP/TCP to always send SRTP
packets through a TURN proxy (which extends STUN via ICE) to work around usual
NAT problems. These guys scanned the TURN and found an SSRF which allowed them
to connect to Slack's VPC on AWS using IAM temporary credentials. Interesting.

For fun, read that last paragraph out loud to a non-techy near by and watch
their eyes...

~~~
pottertheotter
Do you mean this paragraph?

"Our recommendation here is to make use of the latest coturn which by default,
no longer allows peering with 127.0.0.1 or ::1. In some older versions, you
might also want to use the no-loopback-peers."

~~~
topher200
I believe GP means this paragraph:

> So Slack's VoIP uses WebRTC, which connects via UDP/TCP to always send SRTP
> packets through a TURN proxy (which extends STUN via ICE) to work around
> usual NAT problems. These guys scanned the TURN and found an SSRF which
> allowed them to connect to Slack's VPC on AWS using IAM temporary
> credentials. Interesting.

~~~
pottertheotter
Thanks! I don't know where my head was.

~~~
keanebean86
Sounds like you're STUNned. Try not to TURN you head and maybe put some ICE on
it.

------
gfodor
Huh, I happen to be knee-deep in this stuff right now. This article noted that
Slack seemed to be running an old TURN server (pre-coturn):

[https://webrtchacks.com/slack-webrtc-
slacking/](https://webrtchacks.com/slack-webrtc-slacking/)

Given that the latest coturn has this vulnerability mitigated by default,
perhaps all this boils down to is "Slack runs outdated software, we exploited
it."?

~~~
jackewiehose
It's not really a bug in old coturn, just a feature in the protocol. According
to the article newer versions just disable routing to 127.0.0.1 by default but
there are still other network addresses you might have to consider (see
article for a recommended list of "denied-peer-ips").

~~~
jerf
You may already know this, but it's worth getting the word out. Do _not_ just
deny routing to 127.0.0.1. 127.0.0.1 is merely the conventional "localhost"
address; however, ALL 127.x.x.x is "localhost". You can check this now on your
local command line with "ping 127.1.2.3".

(This just seems to be one of those bugs that every proxy goes through at some
point, just like pretty much any attempt to write a web server that serves
files off disk _will_ have at least one directory traversal bug.)

~~~
MrStonedOne
Fun fact, windows blocks remote desktoping to localhost (because doing so
would lock the console session unless you break license and modify termserv).

Fun fact 2, this only blocks 127.0.0.1.

(side note: if you don't mind breaking license compliance on your personal pc,
you can use this to remote desktop into other user accounts on the same pc for
a nice separation of work spaces (or just move to linux))

~~~
o-__-o
You are not breaking the license if you pay per-seat rather than per-device.

Ask me about CALs, I can tell you all about them.

~~~
monsieurbanana
Let's start with an easy question, what is CAL?

~~~
o-__-o
Client access license. For Microsoft Windows they come in per-seat and per-
machine. To OPs point, they were talking about RDP. When you are licensing
per-machine then (windows server) you are limited to one desktop user and two
remote users (for windows desktop it’s one user at a time). When it is per-
seat then you can have as many concurrent connections as you have licensed
for. Per-seat is enforced by Microsoft Licensing Server

~~~
MrStonedOne
I was mainly referring to personal pc uses, not corp/server users.

It's still technically breaking the license to override TERMSERV.dll's
protections on home/professional PCs not allowing somebody to be logged in
while somebody else is already logged in.

ie, a home or professional licensed pc can not support 1 user on remote
desktop, and 1 user sitting down in front of it at the same time, you have to
pay for a server license to get that functionality. This technical limitation
is considered a DRM measure and also protected by the DMCA anti-circumvention
provision.

if you don't mind breaking the law, I was pointing out how you could use abuse
127.0.0.2 to connect remote desktop to localhost in other to do seperate user
accounts for seperate tasks, (such as putting your job hunting activities in a
seperate account so you don't get distracted by discord/chrome desktop
notifications and reddit in your new tab page)

------
BoorishBears
Timeline—

November 2017: added TURN abuse to our stunner toolset

December 2017: discovered and reported TURN vulnerability in private customer
of Enable Security

February 2018: briefly tested Slack and discovered the vulnerability

April 2018: submitted our report to Slack, helped them reproduce and address
the issue through various rounds of testing

May 2018: Slack pushed patch to live servers which was retested by Enable
Security

January 2020: asked to publish report

February 2020: disclosure delayed by HackerOne/Slack

March 2020: report published

------
jrockway
Things like this are why mTLS internally are so important. If a hole is found
in your firewall, services still don't trust each other until they have a
valid TLS certificate.

~~~
Nextgrid
I wish there was an easy way to do mutual TLS auth with pre-shared keys that
can be stored and copy/pasted just like normal passwords or API keys without
having to maintain a CA and handle certificate issuing & renewal (sure,
technically forever-lived certificates aren't as secure, but even those would
already be a major upgrade compared to the status quo).

~~~
jrockway
I personally use cert-manager to run the CA, then create a cert for each app
and have k8s inject it. It is more manual than a service mesh, but many
applications support this strategy out of the box. (I will say that creating a
certificate resource is a lot easier than the old days of some directory of
shell scripts acting as your CA, though. And the same code manages my
letsencrypt certs.)

For example, I have Grafana backed by Postgres, and they both understand this
authentication scheme out of the box. Postgres is happy to be provided with a
cert to present to connecting applications, and is happy to check the cert
that applications present against the CA cert.

The main problem with my setup is that I use a ClusterIssuer CA, so really
anyone in the cluster can get a valid certificate. This is not amazingly
secure and things like Istio do a bit more provenance checking of the
application before issuing a cert, which I like. But this is simple, and does
protect against the attack this article covers -- as long as you don't go out
of your way to present the application's cert when proxying a connection.
(Which is probably an easy mistake to make, so be careful.)

------
wrkronmiller
As a complete novice in this area, I don't understand the advantage of using a
proxy-like service such as TURN.

What is the advantage over simply routing the media streams through
application servers (i.e. user A connects to server which links to user B)
which can then perform application-specific authentication, enforce
restrictions on payloads, etc... Performance?

~~~
gfodor
If you have a centralized server, you have a SFU. SFUs typically expose a
range of UDP (and/or TCP) ports for communication. Peer connections are
allocated on a port basis. So if a user is connected to your SFU, they take up
a port, and need to be able to egress over a large UDP/TCP port range to
connect, since the port is assigned randomly.

However, many firewalls block port ranges, or even UDP entirely. What you
really want is a way to let people speak WebRTC over a common port (443 TCP is
almost never blocked.) TURN facilitates this. Sometimes it's built into SFUs,
sometimes not, and requires coturn in front of it. In Slack's case (and the
project I work on as well) they are running Janus, which does not have TURN
built in, and hence, run coturn to facilitate TURN.

Slacks's approach is particularly interesting because they _always_ push
people through TURN, instead of allowing direct connectivity to their SFU.
Hard to say why exactly, but probably it's a mix of locking down SFU onto the
private network for some reasons, being able to push TURN to edge but keep SFU
on private LAN, etc. Typically you don't do this I don't think, you run TURN
and SFU both with public IPs, and the client connects to one or the other
depending on what ICE candidates win (which is a function of their firewall
rules: your browser tries to pick the 'best' candidate it can get to, ideally
one over UDP without a TURN hop.)

~~~
Sean-Der
There is no reason an SFU couldn't run everything over one port though! Then
you can just use the 3-tuple to route stuff to the proper connection.

Someone is doing this right now for Pion, really excited to see it. I am
especially excited to see what it means for deploys, right now asking people
to expose port ranges adds so much overhead vs 1 UDP and 1 TCP for media.

~~~
0az
Can you elaborate on what this means?

~~~
Sean-Der
Right now most SFUs start up an ICE Agent [0] and listen to a random port. ICE
is used to establish the connection between two peers. Basically both sides
exchange a list of peers, and try to find the best path.

With an SFU you end up having thousands of remote peers each with their own
port on your server. However you could easily listen on a single port and then
handle the inbound packet depending on what the remote 3-tuple is (clients
ip/port/protocol). Effectively you would just be running all your ICE Agents
on one port, but doing one additional step of processing.

I need to fill out [1] more to fully explain the idea, but I think it could
make a huge difference when making it easier to deploy WebRTC SFUs.

[0] [https://github.com/pion/ice](https://github.com/pion/ice)

[1]
[https://github.com/pion/webrtc/wiki/SinglePortMode](https://github.com/pion/webrtc/wiki/SinglePortMode)

~~~
gfodor
Yup that's a great point. I'd love to see this approach explored further. Is
there any risk of tuple collisions in some bizarro NAT situation? I'd guess
not, since the remote tuple needs to route to a single destination, but
there's some weird stuff out there... eg one could imagine a router abusing
the IP protocol to somehow route packets to different destinations despite
them having the same return IP/port combo. i'm no networking wizard, but in
general i assume if its possible, someone is doing it :)

------
realchucknorris
am i wrong or security researchers aren't paid well. i mean not sure how much
this bug is wort but def. $3500 looks like a small number.

~~~
anotheraccountf
Yeah, I had the same thought. For something as big as this? Should be at least
2 more zeros imo.

~~~
imtringued
I don't understand what's so big about this. It's akin to telling someone that
they forgot to use passwords on their mongodb database. Does that really
deserve $350k compensation?

~~~
anotheraccountf
Depending on what a black hat could do with the data in your database, it
might absolutely be worth it. I understand that 350k is way more than bug
bounties usually pay, but 3.5k is taking advantage of people's ethics to
outsource your security.

Let's put it another way: The team who discovered this has skills WELL worth
350k for a year's worth a work. How many security issues would they have to
catch for it to be "worth it"? Maybe more than 1, but 100 show stopping
vulnerabilities for 350k is crazy to me.

edit: ESPECIALLY slack, if it was possible to use this to get access to any
chat logs.

~~~
tptacek
No, none of this is how vulnerability research compensation works.

------
Diggsey
Instead of sending traffic anywhere, why don't they have the destination
address first send a (slack-authenticated) request to the TURN server saying
"I'm happy to receive traffic from [SOURCE]" and then a temporary window is
opened for [SOURCE] to open a connection to that specific destination.

------
kylek
tldr-

November 2017: added TURN abuse to our stunner toolset

December 2017: discovered and reported TURN vulnerability in private customer
of Enable Security

February 2018: briefly tested Slack and discovered the vulnerability

April 2018: submitted our report to Slack, helped them reproduce and address
the issue through various rounds of testing

May 2018: Slack pushed patch to live servers which was retested by Enable
Security

January 2020: asked to publish report

February 2020: disclosure delayed by HackerOne/Slack

March 2020: report published

~~~
lonelappde
Don't use indentation for formatting linebreaks. It beaks HN layout.

Just add extra linebreaks

~~~
dang
I've fixed the formatting now.

------
ChrisArchitect
kind of important almost title-edit-worthy to note this is an exploit and
research that went on late-2017 until about mid-2018 no? Not that this is some
current thing

~~~
jackewiehose
Published March 2020. It's not about some Slack issue that is irrelevant now.
It's about misconfigured TURN-servers and at least for me it's a current thing
;-)

