
Inferring and hijacking VPN-tunneled TCP connections - jedisct1
https://seclists.org/oss-sec/2019/q4/122
======
colmmacc
Disclaimer: I work at AWS, on Amazon Linux and our VPN products; those aren't
impacted by this issue.

The attack that the researchers describe is very impressive, and using traffic
analysis and error messages to find the details of an open TCP connection is
extremely clever.

Unfortunately a similar approach can be used even more practically to target
DNS on the VPN: [https://www.openwall.com/lists/oss-
security/2019/12/05/3](https://www.openwall.com/lists/oss-
security/2019/12/05/3)

Encrypted DNS queries and replies can be profiled by traffic analysis, and the
reply "paused", making it easier to ensure that a DNS spoofing attempt will
succeed. This is a good reminder that cryptographic protections are best done
end to end; DNSSEC does not help with this attack, because it does not protect
traffic between the stub resolver and the resolver. It's also a good reminder
that traffic analysis is still the most effective threat against network
encryption.

~~~
amluto
An actual high-quality DNSSEC implementation would protect against this. Using
an untrusted stub resolver is a mistake; end-user OSes should verify DNSSEC
responses directly.

~~~
tptacek
What end-user OS does this?

~~~
amluto
Fedora has a prototype:

[https://fedoraproject.org/wiki/Changes/Default_Local_DNS_Res...](https://fedoraproject.org/wiki/Changes/Default_Local_DNS_Resolver)

I ran it for a while. It was problematic from a UI perspective (nothing to do
with DNSSEC per se — it was just immature), and it wasn’t fantastic with
captive portals.

Also, it noticed that I visited an internet connection where the ISP was
hijacking google.com. So it worked!

~~~
tptacek
So by "actual high quality DNSSEC implementation", you mean one that exists
only in prototype form for desktop Linux users.

~~~
amluto
I’m saying this should be done, not that it is done. The AD bit is garbage.

~~~
tptacek
DoH _actually_ addresses this attack, directly, regardless of whether the
affected name is in a signed zone (most queried names aren't!), and it
actually works on mainstream operating systems. Why would anyone waste even a
moment with DNSSEC?

~~~
mindslight
DoH moves the attack point to a different place - either between the recursive
resolver and the authoritative servers, or (with DNSSEC) to the recursive
resolver host itself. This is certainly a step up for security against
malicious neighbors, but doesn't address malicious infrastructure.

~~~
tptacek
DoH moves the attack we're talking about to a place where the attack can't be
carried out.

~~~
mindslight
Modifying DNS results can certainly be carried out ahead of or at the
recursive resolver.

DoH without DNSSEC means the results are modifiable by authoritative servers,
recursive resolver, and any of the links between them. DNSSEC with a local
recursive resolver prevents this. (Of course if you want privacy from your
ISP, you then have to tunnel the recursive lookups through VPN(s)).

If DoH is the best one can do, then do DoH. But it's wrong to shout down
DNSSEC until an alternative method for _authentication_ comes along.

~~~
tptacek
The same argument applies to DNSSEC: if the DNSSEC authority server is
compromised, it can fake records, the same way your DNS recursor can fake
records if it's compromised. If you do DoH across the path, at each step,
DNSSEC does literally nothing, rather than practically nothing.

Regardless: for the attack that we're discussing _on this thread_ , DNSSEC is
not relevant; or rather, this particular attack is a good illustration of how
irrelevant DNSSEC is to realistic attack scenarios.

~~~
mindslight
I wasn't aware that DoH was currently being touted for authoritative servers
as well. So, comparing the best possible adoptions of each protocol to each
other on their unique strengths:

1\. DNSSEC (with clients doing full recursive lookups, ofc) is able to have
zones signed _off of_ the authoritative server, meaning a compromise of the
authoritative server isn't an attack point.

2\. With the client performing full recursive lookups themselves, DoH provides
partial query privacy.

3\. With the client delegating to a trusted third party (eg Mozilla), DoH
provides full query privacy modulo that trusted third party (the TTP can break
both privacy and integrity)

Comparing (2) to (1) I do see your argument much better now.

However, DNSSEC still has the property of E2E validation that can be used for
more than what current software does. One could write a resolver that shipped
records _laterally_ between peers to extend its privacy properties beyond
either DoH setup. Adopting this wouldn't require all the authoritative servers
to get on board, just a community of end-users. This is where my argument is
coming from, especially with DoH meaning (3) to most users and the general
course of what happens to trusted third parties.

~~~
tptacek
In reality, DNSSEC signers are online, and need to be, because the way you
keep DNSSEC from revealing the full contents of your zone is to dynamically
inject signed "whitelies" responses to queries. That, and then just the
dramatic extra ops burden of keeping offline signing keys, means that the
realistic real-world deployment of DNSSEC is, like that of TLS, based on
online keys.

DNSSEC _does not have E2E validation_. DNSSEC is validated between the
recursive cache server and the authority server. The protocol explicitly
delegates validation away from endpoints with the AD bit. You can run a full
resolver on your laptop the same way you can run a full-feed defaultless BGP4
peer on your laptop; it'll "work fine", but that's simply not how it's
deployed in reality.

Another dramatic difference between DNSSEC and DoH is that DoH works whether
or not zones sign (a tiny portion of the zones people actually make queries on
are actually signed). Nobody needs "permission" to protect their queries with
DoH, but everyone has to cooperate to make DNSSEC work.

Since the value DNSSEC provides is made more marginal with each passing
quarter --- because of MTA-STS, because of multi-perspective CA DNS
validation, because of DoH, because of LetsEncrypt making X.509 certificates
free, because of certificate transparency --- the rationale for its continued
deployment has become extremely thin. It's 1990s cryptography --- queries
aren't even encrypted! --- that people are advocating we forklift into the
Internet to solve... it's hard to see what problem?

A better plan would be to take DNSSEC back to the drawing board and come up
with a modern alternative to it. DNSSEC itself is a failed protocol.

~~~
mindslight
> _DNSSEC does not have E2E validation_

My comment compared the strengths of each protocol with the fairest
interpretation of each. Your judgement here does not do this - it's obvious
that a stub resolver relying on a third party to do verification is braindead,
and clients doing a full recursive lookup is the correct answer. How clients
are currently setup has little bearing on discussion of a protocol's
properties.

> _Nobody needs "permission" to protect their queries with DoH_

This is also false if you compare the protocols on equal footing - if the
authoritative servers are not speaking DoH/DoT, then queries are only
partially protected. In order to do " _DoH across the path_ " as you said
above, cooperation is needed.

> _A better plan would be to take DNSSEC back to the drawing board and come up
> with a modern alternative to it_

Sure, but this becomes harder when things like DoH are touted as being a
sufficient replacement...

~~~
tptacek
It is not at all obvious that stub resolvers are "braindead" and the "correct
answer" is full recursive lookups on the desktop. One way you know this is
that no mainstream operating system works this way; another way you know it is
that the DNSSEC designers explicitly took stub resolvers into account; yet
another is that full recursive lookups eliminates caching, which the DNS
depends thoroughly on.

I'm not interested in a debate about a fictitious version of DNS that you make
up as the discussion progresses. I think we can probably just wrap up here.

~~~
mindslight
You've written off the whole protocol because of 1990's cryptography. I think
it's reasonable to just ignore the specific parts that don't require
cooperation to change.

I would be interested in any stats that the DNS system actually "relies" on
having clients share caches. Firing out UDP packets is a heck of a lot easier
than a TCP/TLS session, and modern websites take the latter for granted for
every single user.

If clients sharing a cache is actually important, that's actually a negative
point for DoH/DoT as increased resource utilization means that major
authoritative servers will be tempted to form a clique with major recursive
resolvers, rather than everyone being able to query the zones directly.

------
mondoshawan
"This attack did not work against any Linux distribution we tested until the
release of Ubuntu 19.10, and we noticed that the rp_filter settings were set
to “loose” mode. We see that the default settings in sysctl.d/50-default.conf
in the systemd repository were changed from “strict” to “loose” mode on
November 28, 2018, so distributions using a version of systemd without
modified configurations after this date are now vulnerable. Most Linux
distributions we tested which use other init systems leave the value as 0, the
default for the Linux kernel."

Anybody happen to know _why_ systemd decided this was something they should be
messing with?

~~~
arianvanp
`git blame` is your friend

sysctl.d: switch net.ipv4.conf.all.rp_filter from 1 to 2

This switches the RFC3704 Reverse Path filtering from Strict mode to Loose
mode. The Strict mode breaks some pretty common and reasonable use cases, such
as keeping connections via one default route alive after another one appears
(e.g. plugging an Ethernet cable when connected via Wi-Fi).

The strict filter also makes it impossible for NetworkManager to do
connectivity check on a newly arriving default route (it starts with a higher
metric and is bumped lower if there's connectivity).

Kernel's default is 0 (no filter), but a Loose filter is good enough. The few
use cases where a Strict mode could make sense can easily override this.

The distributions that don't care about the client use cases and prefer a
strict filter could just ship a custom configuration in /usr/lib/sysctl.d/ to
override this.

[https://github.com/systemd/systemd/commit/230450d4e4f1f5fc9f...](https://github.com/systemd/systemd/commit/230450d4e4f1f5fc9fa4295ed9185eea5b6ea16e)

There was NEWS entry for this here:
[https://github.com/systemd/systemd/blob/230450d4e4f1f5fc9fa4...](https://github.com/systemd/systemd/blob/230450d4e4f1f5fc9fa4295ed9185eea5b6ea16e/NEWS#L73-L80)

as to _why_ they would make the change? Systemd trieds to be project for
distros to have a sane base system. Introducing this change was probably
something they deemed as being useful in a base system. Especially in the
context of laptops and mobile devices this default seems sane.

~~~
peterwwillis
> Systemd trieds to be project for distros to have a sane base system

Systemd tries to force a monolithic operating system to behave like a
microkernel, and uses propaganda and manipulation to enforce whatever
preference its designers have for how all systems should run onto all the
Linux distributions it can. You can call that 'sane', I'll call it
totalitarian empire-building.

------
mirages
Sorry if I seem stupid but what are the security implications ?

As far as I understood the attacker can :

\- Detect an active VPN connection (and maybe close it/monitor it)

\- Attempt to inject packets : for this part I am skeptical of the usefulness.
The connection between the server and the client are normally encrypted,
meaning that the injected packets will be dropped or can be used to forcefully
close the connection making it a DoS attack.

~~~
tetha
> The connection between the server and the client are normally encrypted

Keyword: Normally. What about DNS? DNS without DNSSEC via an untrusted AP
cannot be trusted, that's for certain. Hence, you recommend your user to
activate VPN first to avoid such attacks because now it's encrypted. Suddenly,
that might change and your trusted-page.internal resolves to a different
hostname. This /should/ be preventable via HSTS and certs for example with
HTTP, but those are assumptions again. Or what about RDP? It's not encrypted,
so you hide it in a VPN - mostly due to the cleartext password. But suddenly
there's a vector that might be able to inject data into an RDP stream inside a
VPN connection.

> \- Detect an active VPN connection (and maybe close it/monitor it)

Not just that. They can detect TCP connections inside the VPN connection.
Hence, you can start tracking if users of an access point accept anti-gov.com
even if they go through a VPN. It might be detectable on the client side and
it should be possible to mitigate this via configuration, but that's still
plenty scary.

~~~
loeg
Or in short, the authentication premise of VPN-routed traffic can be violated
by an attacker without knowing the authentication keys, due to some
misalignment of (in some cases, valid) routing behavior and VPN integration.

------
hamandcheese
> This vulnerability works against OpenVPN, WireGuard, and IKEv2/IPSec

> It should be noted, however, that the VPN technology used does not seem to
> matter

I was wondering which VPNs, but after reading it seems that it doesn’t matter
the VPN but is instead a kernel bug? I wonder if this also affects a host that
has two network interfaces, with no VPN involved whatsoever?

Also, it’s good to see Jason Donenfeld involved, which many will likely
recognize as the author of Wireguard.

~~~
tolleydbg
Donenfeld has been great and makes a wonderful product. We've been using
WireGuard as our primary VPN solution for a long time now and don't anticipate
a change anytime soon.

I think this class of attack can best be mitigated at the kernel level, but
unfortunately, side-channels exist anywhere that maintains state, and this
might be a more general routing problem.

~~~
zx2c4
We're working around this CVE in WireGuard's wg-quick(8) with a rule something
like:

    
    
        iptables -t raw -I PREROUTING ! -i wg0 -d 10.182.12.8 -m addrtype ! --src-type LOCAL -j DROP
    

where wg0 is the WireGuard interface and 10.182.12.8 is the local IP of the
interface. This says to drop all packets that are sent to that IP address that
aren't coming from the WireGuard interface. And it's done very early in
netfilter, in the "raw" table.

I don't like having to use iptables in wg-quick(8), and so Willy Tarreau and I
have been discussing a deeper kernel-level fix, which should be posted to
netdev@ sometime soon.

~~~
hamandcheese
Won't this break the use case of using Wireguard as a gateway to a private
subnet, in the typical many-clients-one-server VPN setup?

For example, suppose I have a private physical subnet 10.12.0.0/24 (perhaps in
an AWS VPC).

I want to allow clients to access to these private hosts using a Wireguard
VPN, so I set up a VPN with all clients having IPs from the 10.34.0.0/24\.
Because I want these clients to have access to the private physical subnet, so
each client's config has

    
    
        AllowedIPs = 10.34.0.0/24, 10.12.0.0/24
    

Which adds both subnets to each client's routing table.

I add a new route for the VPC to send all packets destined for 10.34.0.0/24 to
the central Wireguard "server", thus the Wireguard server acts as a gateway
between the virtual 10.34.0.0 subnet and the physical 10.12.0.0 network.

The packets originating from the 10.12.0.0/24 hosts are not local, but I
definitely want to route them onto the virtual 10.34.0.0/24 network.

~~~
tomatocracy
I don't think the filter quoted in parent would stop this. In your example
what it would stop is clients in 10.12.0.0/24 from connecting to the IP of the
wireguard server itself (but not clients it routes to) on the 10.34.0.0/24
network (but not its IP on the 10.12.0.0/24 subnet).

------
e12e
> The access point can then determine the virtual IP of the victim by sending
> SYN-ACK packets to the victim device across the entire virtual IP space (the
> default for OpenVPN is 10.8.0.0/24).

So, would a bit of address space randomization mitigate step one for ipv6?
fd00::/8 is pretty big, right? Even just picking a random IP in a random /64
from that /8 should help? Or am I missing something?

Also interesting comments in the reply on the list regarding "policy based"
vpns. I wonder if some subset of that infrastructure could be used by
Wireguard without completing it all the way out of its current nice and
secure-by-simplicity design?

[https://seclists.org/oss-sec/2019/q4/123](https://seclists.org/oss-
sec/2019/q4/123)

> Only route based VPNs are impacted. In comparison, policy based VPNs are not
> impacted (On Linux only implementable using XFRM, which is IPsec on Linux
> specific) unless the XFRM policy's level is set to "use" instead of
> "required" (default)) because any traffic received that matches a policy
> (IPsec security policy) and that is not protected is dropped.

------
mirimir
> I am reporting a vulnerability that exists on most Linux distros, and other
> *nix operating systems which allows a network adjacent attacker to determine
> if another user is connected to a VPN, the virtual IP address they have been
> assigned by the VPN server, and whether or not there is an active connection
> to a given website.

I'm not sure that I understand exactly what "network adjacent attacker" means.
But I'm guessing that it means an attacker on the same subnet. And that it
doesn't involve actually hacking VPN encryption.

But isn't it well understood that sharing LANs with untrusted neighbors is
hugely risky? At least, I always segregate critical machines in protected
subnets.

Or am I missing something?

Edit: OK, I was missing that this focuses on using VPNs via WiFi APs. And
depends on the AP being malicious. So yeah, this is a huge issue, for that use
case.

------
exikyut
Huh, I wonder how the OpenBSD team will regard this one.

I'm unfamiliar with exactly how to tell what's in the base system and what's
in ports, but I can see openvpn* entries over at
[http://ftp.openbsd.org/pub/OpenBSD/6.6/packages/amd64/](http://ftp.openbsd.org/pub/OpenBSD/6.6/packages/amd64/)
\- does that mean there's arguably a hole in the base distro?

In any case, nice work.

Question. Besides randomizing packet lengths (preferably with minimum and
maximum tunables (per each packet type) that each site can change, to further
add entropy), what else can be done to mitigate against this family of attack?

Asking as someone interested in developing "ubiquitous secure container" type
protocols that are application- and use-case-specific but (theoretically)
high-stakes.

~~~
colmmacc
Best practice for preventing traffic analysis is to send encrypted messages of
a constant size, at a constant rate. It's better to pad to a max size, than to
randomize the packet length. Sending at a constant rate may not always be
feasible or cost-effective, but it can still be useful to send data at a
constant rate for a minimum block of time.

~~~
zzzcpan
It's just easy to implement constant rate, hence why it's best practice. But
adaptive rate might be ok, like, for example, if you only pick from 4
different rates and adapt exactly every second, you only leak 2 bits of
information per second. The question then becomes which leaking rate is safe
enough in practice?

~~~
colmmacc
constant max rate tends not to be practical in remote roaming VPNs for cost
and contention reasons. It would blast through typical mobile usage rates, for
example, or get you throttled on a shared wifi network.

------
nif2ee
I just can't understand how this attack can be implemented in real world, it
not only needs having an adjacent network access to the VPN client but most
importantly knowing the destination of a currently open TCP connection
encrypted by the VPN and then guessing the right sequence numbers. Since most
TCP connections are also carrying TLS these days, this attack is pretty
useless in the vast majority of real world situations I can think of.

------
arielb1
> nping --tcp --flags SA --source-ip 192.168.12.1 --dest-ip 10.8.0.8 -- rate 3
> -c 3 -e ap0 --dest-mac 08:00:27:9c:53:12

Why is Linux accepting packets coming from one interface into an IP address
belonging to a different interface? It feels like it is "forwarding" the
packets internally, but `ip_forward` is turned off.

Is there any case where this behavior is legitimately useful?

~~~
ajross
IP addresses don't "belong" to interfaces in the general case. It's just a
hard problem. In fact there are lots of multi-homed use cases where you want
to internally route packets across interfaces without an affirmative mapping
of what address is supposed to be used where.

For the specific case of point to point VPNs, there's a rule that makes sense.
But that's not part of the network stack per se and there's no way to enforce
it generically.

~~~
loeg
Do network stacks drop 127.0/8 packets from external interfaces today?
Superficially (I'm not an experienced TCP/IP or routing stack developer,
although I do work in the kernel) it seems like the same treatment could be
used for VPN-registered interface addresses. You just need an API to specify
"I'm a VPN interface" when the device is created or the IP assigned, no?

------
knolax
It seems for the attacker to find out about active connections to any given
website they have to already know the IP of the website and then brute force
the virtual ports. Being able to determine what website the target has an
active connection to without prior knowledge would probably take even more
brute forcing. A small solice.

~~~
loeg
This is a very noisy attack, for sure.

------
derpherpsson
Correct me if I am wrong, but...

If I just make sure that incomming packets that are destined for the VPN LAN
are dropped, this attack does not work?

 _Of course there are such rules in our firewalls??_

Is everyone walking around without any firewall filtering nowadays? How is
this a bug? Maybe I am just stupid. Did I miss something?

~~~
rasz
TCP/IP stack was dropping this by default .. until systemd decided to switch
the default
[https://news.ycombinator.com/item?id=21713479](https://news.ycombinator.com/item?id=21713479)

~~~
yrro
The default behaviour of the kernel is no rp filtering at all. Older versions
of systems enabled strict to filtering, no doubt causing the same sort of
complaints from people who like to complain about that sort of thing. Newer
versions relaxed this to loose rp filtering for the reasons explained in the
commit message.

------
Gabrielfair
Disseminating vulnerabilities like these to less savvy people should be a
critical imparitive for the next century. I imagine a team of animators at a
non-profit or foundation that illustrate how these attacks work at different
levels of technical backgrounds.

------
GhettoMaestro
I'm having trouble understanding why the randomly sprayed bits from the
attacker to the client are even accepted at all at the crypto boundary.
Shouldn't it hard-fail (not decrypt) due to an invalid key?

~~~
loeg
The bits aren't VPN-framed packets; the attack isn't VPN specific. They're
just sending ordinary TCP setup packets and ordinary TCP stream packets with
fudged parameters until they find the right ones by brute force. It's a really
noisy attack that uses weaknesses in packet routing to find existing TCP
sessions and inject into them.

It's also, like, an escalation from a relatively high privilege level. You
need both a passive network observer (compromised router or ISP) and a noisy,
active LAN device (to inject IP addresses that a router would filter as
bogons). That's not to say this is crazy hard; these are definitely within the
reach of a motivated attacker. Routers were the original the-S-in-IoT-is-for-
security, and if you've got an IoT type device on the LAN it's probably
vulnerable once you've popped the router.

------
ufov2
Is this mitigated by iptables filter firewall? For example, FORWARD policy
DROP and no other rules?

~~~
jlgaddis
Yes, if you are doing firewalling properly. The problm is that most people
aren't and are just using their distribution defaults.

------
joseluisq
Someone know if there is some kind of pronouncement by distros implicated?

------
tinix
Can we merge these? I posted this 10 hours earlier here:
[https://news.ycombinator.com/item?id=21709693](https://news.ycombinator.com/item?id=21709693)

~~~
jlgaddis
There's really nothing useful there that isn't also here.

------
3xblah
"We discovered a vulnerability in Linux, FreeBSD, OpenBSD, MacOS, iOS, and
Android..."

What about NetBSD.

This also appears to be wireless only, i.e., the need for attacker to have
control over the AP. Am I reading this wrong.

~~~
3xblah
I was reading it wrong. It is not wireless only. Technically, it only requires
that the victim is sharing a local network with the attacker.

------
jiveturkey
This doesn't seem nearly as bad as 'hijacking' implies.

> The attacker can now inject arbitrary payloads into the ongoing encrypted
> connection using the inferred ACK and next sequence number.

First, it's hard to get to this point. Then, you're injecting garbage because
you don't know the payload encryption keys. So it's just disruptive, even
though yes technically it is 'hijacking'.

Unless I missed something, of course! I found the writeup to be vague on the
payload encryption point. It should have explicitly stated the impact one way
or the other.

~~~
aaronmdjones
For your first point, this isn't hard if you're in the appropriate position
(adjacent or upstream), as the post details. There are already tools available
that can do all of this, example invocations are also there.

For your other point, they don't need to know the keys (unless the traffic
travelling /over/ the VPN is /also/ encrypted). That's the whole point. This
is about being able to trick the machine into accepting traffic for a
connection, from an interface that the connection's traffic isn't travelling
over. If it's plaintext within the VPN, this sidesteps the VPN interface and
you can indeed inject your own malicious plaintext traffic into that
connection.

~~~
jiveturkey
1\. Yes, you have to already be in position. That's limited to specific
actors. Once that is achieved, you have to probe for specific IP addresses the
victim might be connected to. IOW a list you are targetting. Then the port
probing and seqno guessing.

These facts reduce the impact because it's not just "be a guy on the
internet", eg like if there were an open database of PII sitting there for the
taking, only needing discovery to find it. In no way am I claiming the attack
isn't feasible. It's definitely a real risk, beyond the theoretical.

2\. Thanks, got it. That makes more sense.

I think I actually _like_ this vuln. It reinforces the need for defense in
depth. It reduces a takeover to an annoyance (could be critical for some apps,
yes) assuming you use TLS at the app layer.

~~~
aaronmdjones
> That's limited to specific actors.

1) Anyone who can compromise the residential gateway in your home

2) Anyone & anything connected to the same home network as you (incl. any IOT
devices; and no, they don't need Internet access)

3) Anyone who can compromise the residential gateway in whatever coffee shop
you happen to be in

4) Anyone & anything connected to the same coffee shop network as you

5) ...

... and, like I said, easily scriptable, with the tools already available to
carry it out.

It's not a doomsday scenario, but it is pretty bad. The one saving grace is
that most apps these days use some kind of application-level authentication
and/or encryption, e.g. TLS.

