
Struck by a Thunderbolt - lainon
https://www.lightbluetouchpaper.org/2019/02/26/struck-by-a-thunderbolt/
======
floatboth
Thunderbolt is a horrendous piece of garbage. Not just in the sense of "we're
making an internal bus external before ensuring security", but just in what it
actually is:

[https://twitter.com/whitequark/status/1097777102563074048](https://twitter.com/whitequark/status/1097777102563074048)

> Thunderbolt, uh, does not expose PCIe lanes directly. Thunderbolt is an
> MPLS-like packet switching network that can encapsulate PCIe TLPs over a PHY
> and MAC without a spec, chips without documentation, and software with
> barely any support. […] Ah, and let's not forget that the absurdly
> complicated MPLS-like system with topology written in EEPROMs in every node
> that, as far as I can tell, you have to bitbang via the half-configured
> packet path (what the actual fuck?), well, it's very general. You can
> describe a lot of different networks in it. But of course the Linux driver
> has two or three hardcoded ones, including indexes into the EEPROMs on the
> way, that it recognizes and loads a hardcoded configuration for. The entire
> thing is made -even worse than USB-, which is an achievement.

It pains me to hear people complaining about the lack of Thunderbolt in AMD
Ryzen Mobile laptops, and wishing for it in the next generation now that the
spec is no longer Intel-exclusive in some way. Lack of Thunderbolt is a
_feature_.

~~~
crgwbr
What was the motivation for building it this way, instead of by more directly
exposing PCIe? There must be some upside to they way it’s designed, no?

~~~
stephen_g
I think to have the flexibility to have various topologies of many devices
(daisy-chained, etc.) from a single port.

Probably also timing etc. - I would think making it packet based is probably
easier to make work over longer cables than raw PCI-e.

The bit I really don’t get is why they keep all the specs secret. What do they
think they have to gain? All that it has meant is less adoption...

~~~
wmf
PCIe is also packet-based BTW [1]. The benefit from the custom protocol is
being able to run PCIe and DisplayPort (and I guess networking which no one
cares about) over the same cable at the same time with dynamic bandwidth
allocation. USB-C has alternate modes but they're not as powerful. In theory
PCIe alternate mode over USB-C should be possible and it would be much simpler
than Thunderbolt but nobody has tried to build it.

I imagine the spec doesn't really exist because Intel has been too lazy to
properly write it (I wouldn't be surprised if Thunderbolt is basically defined
as whatever the original Ridge controller implemented) and it's probably also
gross. Adoption seems to be limited by cost, not by secrecy (thinking back to
Firewire, it was open but there were very few controller chips for it).

[1] Check out PCIe over RS-232:
[https://www.youtube.com/watch?v=QMiubC6LdTA](https://www.youtube.com/watch?v=QMiubC6LdTA)

------
jerf
"We built a fake network card that is capable of interacting with the
operating system in the same way as a real one, including announcing itself
correctly, causing drivers to attach, and sending and receiving network
packets. To do this, we extracted a software model of an Intel E1000 from the
QEMU full-system emulator and ran it on an FPGA."

I wanted to call that out; that's nifty!

~~~
pjc50
> we extracted a software model of an Intel E1000 from the QEMU full-system
> emulator and ran it on an FPGA."

I need to look up what they did for this - lots of the tools _promise_ the
ability to turn a pile of C into SystemVerilog, but require a bit of manual
massaging.

Edit: "used the Arm Cortex A9 CPU on an Intel Arria 10 FPGA" well that's
boring and much simpler.

It does not surprise me that drivers are not secure against maliciously-
constructed messages from the "hardware". Most drivers are tested in a fairly
limited fashion, and nobody's even got the infrastructure to fuzz them like
this.

(There are lots of FPGA-on-PCIe boards, but they tend to be horribly
expensive. This one is almost $5000.
[https://www.digikey.com/products/en?mpart=DK-
DEV-10AX115S-A&...](https://www.digikey.com/products/en?mpart=DK-
DEV-10AX115S-A&v=544))

~~~
theomarkettos
There was actually an interesting talk on fuzzing drivers in the session
before ours: [https://www.ndss-symposium.org/ndss-paper/periscope-an-
effec...](https://www.ndss-symposium.org/ndss-paper/periscope-an-effective-
probing-and-fuzzing-framework-for-the-hardware-os-boundary/)

We're working on porting to a cheaper FPGA-on-PCIe board, but it's still ~EUR
800.

~~~
walterbell
Have you looked at PicoEVB ($300 mini-PCIe x1) or Numato Aller ($400 M.2 x4)
with Xilinx Artix-7 FPGAs?

~~~
theomarkettos
We need to use an Intel Arria or Stratix FPGA due to the 'config bypass'
feature we use to allow QEMU to implement its own PCIe config registers.
Xilinx, Lattice and Cyclone FPGAs don't support this as far as I can tell.

~~~
walterbell
If you later move beyond QEMU, these OSS projects may be of interest.

[https://github.com/texane/vpcie](https://github.com/texane/vpcie)

 _> virtual environment consisting of QEMU, LINUX and GHDL glued alltogether
by a small TCP based protocol. It allows PCIE devices to be implemented as
standard userland processes, answering actual PCIE requests coming from QEMU.
It supports PCIE configuration headers, requests, memory read/write operations
and MSI. Different abstractions are provided to simplify the implementation of
PCIE devices._

[https://github.com/KastnerRG/riffa](https://github.com/KastnerRG/riffa) &
[http://kastner.ucsd.edu/wp-
content/uploads/2014/04/admin/fpl...](http://kastner.ucsd.edu/wp-
content/uploads/2014/04/admin/fpl-riffa2.pdf)

 _> RIFFA (Reusable Integration Framework for FPGA Accelerators) is a simple
framework for communicating data from a host CPU to a FPGA via a PCI Express
bus. The framework requires a PCIe enabled workstation and a FPGA on a board
with a PCIe connector. RIFFA supports Windows and Linux, Altera and Xilinx,
with bindings for C/C++, Python, MATLAB and Java. On the software side there
are two main functions: data send and data receive ... Users can communicate
with FPGA IP cores by writing only a few lines of code._

------
jarjoura
Hmmm, on Windows whenever I plug in a new Thunderbolt device, it requests
permission to activate it. I assumed this was for this very reason. If I hand
someone my computer logged in and unlocked, it's as good as compromised.

~~~
theomarkettos
Author here. We don't break that prompt, however there are some problems.
First is that Macs have no prompt at all, so you just need to use a device on
the whitelist and you're approved.

Second, the prompt says 'you attached a thing, do you want to allow it?'. The
name is often not descriptive, eg 'CalDigit TS3' in our example, so users
can't tell what rights are being asked for. As we've found on mobile, when
apps pop up prompts about permissions users are conditioned to agree. Also,
users can be deceived by putting a sticker on the malicious device with the
name on the prompt - the writing on the plastic agrees with the prompt, so the
device must be legit. An attacker can also play social engineering tricks -
"say 'approve' to enable fast charging" for example.

Finally, it's possible to swap out the PCIe devices behind a Thunderbolt
bridge without re-prompting. We took a commercial Thunderbolt dock, removed
the PCB with the dock's PCIe peripherals and replaced it with another PCIe
card. Windows did not notice that the device had changed and didn't prompt
again.

~~~
arghwhat
Ah, so if I may reiterate, the issues are:

1\. Users cannot tell if a device is malicious before authorizing it.

2\. Even if the user could tell, the device can be made malicious post-
authorization.

Both situations seem like quite universal and unsolvable issues, in that you
will never know the exact behavior of a device unless you tear it down atom by
atom.

(Of course, #2 could be made harder if sub-devices were made part of the
authorization, but it will always be possible to make a device malicious given
physical access to it, even if a simple PCIe swap is no longer possible.)

However, assuming proper fix for the bugs discovered, the IOMMU should make
DMA access pointless, reducing any attack to be a simple case of a malicious
device that can at most be malicious in the execution of its own functions
(e.g. eavesdropping or traffic modification). Why would any peripheral receive
device mappings outside of the buffers needed to operate it?

------
JoshTriplett
The situation hasn't improved significantly in the years since the same
vulnerability appeared with Firewire.

There are so many ways to mitigate or stop this, and keep the user in control.
Android, at least, knows to prevent privileged device connections with the
screen locked.

~~~
GeekyBear
The same vulnerability goes back even farther with nearly all 90's laptops
having one or more PCMCIA/PC Card/CardBUS slots.

~~~
stephen_g
Or when they had mini-PCI/mini-PCIe slots under a cover on the bottom for
network cards etc.

As it says in the article, it can also be done with desktops where you can
just pop off the case panel and plug a malicious device into the PCIe bus.

I think the biggest issue with Thunderbolt specifically is that it works over
the USB-C port that is also used for charging - so you could make a device
that looks like a charger but also attacks the device (maybe with a cellular
modem in it to exfiltrate data or something!), making it way easier to trick
users to plug it in themselves.

------
jhallenworld
Aside from the security vulnerability, it's very cool that you can now get a
high bandwidth connection between an FPGA and a system via a cable instead of
a card.

I have already been using Cypress FX-3 to get 5 Gbps USB connection to an
FPGA, but this is better.

------
dsr_
If I'm reading this correctly, this means that physical access to any modern
machine can be leveraged to root access. Encrypted storage that hasn't been
unlocked might continue to be secure; that's about it.

~~~
Dunedan
As long as Thunderbolt is enabled. If I'm not mistaken disabling Thunderbolt
completely should be a viable workaround. It's essentially the same situation
we had with Firewire: Just blacklist the driver and all should be good.

As DisplayPort Alternate Mode for USB-C is nowadays used a lot to connect to
external displays, instead of the approach Apple chose to tunnel DisplayPort
over Thunderbolt, Thunderbolt is probably not even relevant for most users
outside of the Apple universe.

~~~
gumby
> Thunderbolt is probably not even relevant for most users outside of the
> Apple universe.

Actually not the case -- the TB chipsets are from Intel and promoted by them.

TB drives are pretty fast and great to use if you can afford them. TB external
GPUs are useful for ML too (not sure if anyone is using them for actual
rendering; that's outside my field of experience).

~~~
xpaulbettsx
To an extent - many laptops and nearly all desktops don't have Thunderbolt at
all. TB external GPUs are probably the main use-case for the technology today
(people indeed do use them for rendering)

------
ncmncm
No statement can overestimate how critical this vulnerability is.

Taking your laptop or phone away for ten minutes allows somebody with a
malicious USB-C device to read out everything without rebooting or leaving any
mark that anything had been done, bypassing filesystem encryption and
passwords.

~~~
dylan604
How does have a computer that is turned off affect this? Take a MacPro laptop
with an encrypted SSD. If the Mac is turned off, and the attack is attempted
after a fresh reboot without knowing the password, is the laptop still
susceptible?

~~~
geerlingguy
Yeah, it would have to be someone 'borrowing' or stealing a laptop that's
running and has you logged in.

If I'm reading it correctly, any newer Mac with T2 chip and full disk
encryption would not allow data to be exfiltrated unless a user were logged
in.

~~~
hermitdev
I would think itd matter in which order devices are initialized in, for
example, if you had a compromised device connected, is it initialized before
your keyboard is so you can enter your full disk encryption password.

Id wager youd probably be safe because I think the bare USB keyboard support
is provided by the UEFI BIOS and no OS level drivers would be initialized yet
(e.g. why you can use basic USB keyboard and mouse support in the BIOS), but
this is my mildly educated speculation. This is definitely not my area of
expertise.

------
jhallenworld
Are you sure IOMMU is not enabled by default in at least some versions of
Linux? I thought RHEL 6 had it enabled.. (because I remember having to disable
it..)

------
kwccoin
May be a warning not to plug anything except eGPU and power in that slot. But
then you have apple only have that slot ... Still need eGPU in any case

------
eecc
Hmm, seems that plenty at intel haven't been thinking about security at all
for a while... sleeping at the helm?

~~~
wmf
Intel gave us the IOMMU but OSes haven't been using it.

~~~
ncmncm
I have been warned against turning on IOMMU because the code to operate it is
so little exercised, and therefore buggy. A crashed kernel is secure only in
that it stops running.

~~~
the8472
On the other hand IOMMU helps protecting system memory not just against
malicious but also faulty devices that could otherwise write anywhere.

And people who use GPU passthrough rely on IOMMUs, so it's not like those code
paths are not being exercised.

~~~
bonzini
Using IOMMU through the "userspace PCI" vfio driver, as done in pass-through
scenarios, is a different path than using it through the DMA API.

~~~
walterbell
IOMMU is also exercised by Xen PCI passthrough (e.g. QubesOS network and USB
driver VMs) and SR-IOV (e.g. AWS Nitro networking).

~~~
bonzini
The problem isn't bugs in the IOMMU, it's bugs or performance issues in using
the DMA API with an IOMMU.

------
o89dfgjkodf7_
So.. odd question for everyone. Does this mean, all of a sudden, we,
potentially, have full access pass to plain text version of everything (
including things various driver vendors do not want you to know ? )?

~~~
the8472
It doesn't grant you any more access than root on the system already does.

------
exabrial
I think it's worth pointing out, the way this firm is going about the research
and working with the vendors is commendable.

~~~
confounded
The “firm” in this case is the Computer Lab at the University of Cambridge,
UK.

------
fitzroy
> “MacOS fixed the specific vulnerability we used to get administrator access
> in macOS 10.12.4 in 2016”

USB-C Macbook Pros we’re only announced on Oct 27, 2016. Not sure what version
of 10.12 they shipped with, but seems like it was a pretty fast fix.

~~~
chrono3d
The actual paper cites using devices with Thunderbolt 2 ports (iMac, Mac Mini
both have Thunderbolt 2 via DisplayPort) to experiment on as well. It sounds
like the vulnerability is in the Thunderbolt protocol in general, not just the
USB-C implementation of it.

~~~
theomarkettos
The vulnerability applies to all DMA-capable devices (ie Thunderbolt, PCIe,
wifi and NVMe chips on mobiles, on-chip peripherals like mobile basebands).
Thunderbolt 3 makes drive-by attacks easier, but in principle the attack could
be carried out from any DMA device (subject to practical limitations like
reverse engineering of firmware). Our earlier attacks were done with PCIe and
Thunderbolt 2.

