
Demystifying the i-Device NVMe NAND - duked
http://ramtin-amin.fr/#nvmepcie
======
mi100hael

        > In order to read the NVMe, I therefor developped a PCIe card with a Zero
        > Insertion Force reader. I brought the JTAG part to 20pin header. The hard
        > pard in here is the signal integrity of the differential pairs. In order
        > to do so, I had to use multi layer PCB, and have the impedence match by
        > knowing the stackup, materials used for prepeg and so on..
    

Posts like this are very humbling. They serve as a good reminder that no
matter how far I've come and how much I've learned, there will always be
someone out there who knows vastly more than me like the back of their hand.

~~~
akuma73
Don't feel too bad.

Fully understanding the complete stack of a modern computer system is outside
the scope of almost everyone. These things are complicated and we've built
abstractions, interfaces and modules to manage the complexity.

People specialize in their own fields. I am trained in digital integrated
circuit design, but don't ask me to build a file system.

~~~
djsumdog
Exactly. This is super impressive work, but also very specialized. Even for
non-EE majors, most IT people could at least understand the basics of what he
was doing.

------
sounds
The gold is at the bottom:

    
    
      The idea here would be to see if it was possible to control the NVMe
      over jtag in order to ask it to perform a DMA read over the PCIe Bus.
      In order to do so, the PCI_COMMAND_BUS_MASTER has to be set to 1. We
      can assume that since the chip is using remote RAM, it is allowed to
      act as a master over PCIe. Here is a snippet of the probing function
      of the kernel driver.
    

(code)

    
    
      Our goal here is to force the DMA to happen just by controlling the
      ARM of the NVMe over JTAG, in order to ask it to dump the region we
      alloc'd in kernel and see if we get the data out of it.
    

In other words, full root exploit of the phone from the NVMe JTAG pins.

~~~
josephg
> In other words, full root exploit of the phone from the NVMe JTAG pins.

Sure, except the data on the device will be encrypted. The keys are kept in
the iphone's tamper resistant security enclave. Android and iOS both have full
disk encryption enabled out of the box these days. (On iOS I don't think you
can disable it.)

~~~
honkhonkpants
That isn't the point. This exploits the host, which holds plaintext data in
memory.

~~~
AceJohnny2
No, it doesn't exploit the host. The "Host" in the case of such an NAND device
is the Baseband SoC, not the ARM controller on the NAND device. You can be
pretty sure that Apple did _not_ include decryption on the NAND chip itself.
That would defeat the point.

~~~
honkhonkpants
That isn't what he's describing. He is commanding the storage controller to
initiate DMA from host memory over the PCI-Express bus.

~~~
AceJohnny2
Ah good point.

However, he does this by using the NAND as a PCIe master, which implies that
the peer (main SoC) would be switched to device mode, and the whole PCIe
handshake happens properly.

While I can't dismiss that possibility entirely, I'm not holding my breath for
it to work out.

~~~
cnvogel
No, you don't have to switch "Master" and "Device" mode (what does that even
mean in PCIe?).

You might be surprised to hear that your networking card, the AHCI-SATA
interface and even the sound-card soldered onto your computer's mainboard is
allowed to, and does, read from and write to your computers memory whenever
you are using it. {that's just where I had to dig in, in the past, almost any
modern peripheral uses DMA}

If the operating system needs a block from your harddisk to be read, and
stored in a certain memory location it will actually tell the physical memory
location to the AHCI controller and tell it to write the contents into memory,
on it's own. It will signal an interrupt when it has finished transfering the
data.

Your networking card will have "ring descriptors" which tell the card a list
of buffers (in RAM) into which to save new incoming packets. It will raise an
interrupt after it has finished writing to the first buffer.

Your sound-card will likewise have a list of buffers from which it reads, and
to which it writes just as you are listening to mp3s, or having a phone
conversation with google hangouts.

The worst offender would probably be Firewire which, _if_ _not_ _configured_
_to_ _block_ _this_ (modern OSes do), allows these reads from/writes to memory
triggered by any connected external device, and from/to addresses supplied by
this external device. It can be a huge help for debugging, though.

On typical computers, there's no safeguard against this happening. But some
have a IOMMU which can be used to limit DMA and redirect DMA accesses with
respect to physical memory.

~~~
johncolanduoni
To what degree can misbehaving PCIe devices (as in the device as a whole being
malicious, not simply manipulated) override IOMMU protections? Can they
pretend to be another device or otherwise send unexpected signals to confuse
this defense?

~~~
honkhonkpants
Violating the timing specs for PCI signalling would put the whole system into
an undefined state. It's possible you could glitch the thing on the other end
into doing whatever you want. Who knows?

------
kanwisher
Refreshing to see a deep tech article on HN. I really liked how he debugged
the code on the controller

------
iuuuuu145
>It looks like to reduce the size needed, the NVMe core uses the host DDR in
order to work. Therefor, apple is not strictly following the specification
regarding the initialisation.

Yikes.

~~~
djsumdog
Doesn't surprise me; Apple for not doing anything by standards. When you
control all the hardware, I guess it doesn't matter. You can have broken ACPI
and driver implementations all over the place.

I've started to give up on ARM embedded boards for the same reason. You need
to build out images for all the different potential ARM systems if they don't
use device trees. There are Intel Atom/AMD Geode Pi/Beagelboard clones that
will boot up just like a desktop and you can install most Linux distributions
right on them without modification.

If ARM ever decides to start selling an architecture spec instead of just a
SoC spec, I think it would go a long way at making it a better platform. I'm
pretty sure Apple would still ignore it f

~~~
StillBored
Its happening (slowly) SBSA/SBBR/etc are out there and being implemented. Its
just taking a while for EDK2/linux/etc to get to the point that it can run on
a random ARM SOC. Once that happens though, suddenly you have a machine that
can install off the shelf linux distros. Right now, There are only a couple
systems that work like you would expect (mostly AMD Seattle machines, although
there are a couple others) but over the next few years I would expect that the
firmware guys to start expanding the device compatibility.

Eventually (particularly with ACPI) it will become apparent to the ARM
ecosystem, that they can make their devices compatible with the standards, or
they can spend millions on engineering effort to build their own firmware/OS
stacks that will perpetually be behind the capabilities of the rest of the
ecosystem.

------
nimish
Apple's purchase of Anobit is paying dividends!

~~~
digi_owl
Sometimes i wonder how many companies Apple has bought that now simply exist
to make parts of Apple products.

~~~
threeseed
Somewhere in the vicinity of 20 companies.

What's interesting is that about a third of those companies we have yet to see
the output of. Expect Apple to get into AR/VR in a big way in 2017/2018.

------
mmastrac
Has anyone managed to capture the text of this article? It doesn't appear to
be in a Google cache AFAICT.

~~~
arm
[https://archive.is/FGIiC](https://archive.is/FGIiC)

------
condescendence
Definitely one of the cooler and more in depth posts this year, what a great
read.

------
athiercelin
Very good stuff!

