> In the event that they don’t match perfectly: “the startup process halts and the user is prompted to reinstall macOS”.
This feels weird. So they have a very low number of possible systems to support. The OS is immutable. Why would they go with a reinstall at that point instead of "we know which part failed, so we'll re-download that one block from Apple service"?
(I guess that one ties in with: so you've got a low number of systems and an immutable OS - why does it take 15min to apply an update)
Well, it means they have to keep track of every build, and keep every version of every OS live, and that's not something they really do. Especially in cases of security fixes, they'll take down an old version from the update servers.
As for 15min, they're actually testing fast patching right now, although it's currently limited to security updates (rapid response).
FWIW, I do think they might have changed things very recently--but only for some legal argument (maybe as a result of Corellium...) as opposed to any technical one--but they really did just leave every single variant of every single update on their server... I even made a tool for jailbroken iOS devices that would get the original update files from Apple's servers and implemented exactly this idea: it would checksum every file against the copy from the bill of materials in the update and was able to download only the files that were different (by carving them out of the remote zip file using HTTP Range requests).
> it means they have to keep track of every build, and keep every version of every OS live
That's not a lot of work. I would be extremely surprised if they were not already doing it. If someone reports an issue going back to a specific version, they need to be able to install that specific one.
I think you're overestimating the effort / underestimating Apple resources here. Ubuntu keeps all released packages, going back to 2004 online for example. (And isos, even for the effectively dead powerpc) Microsoft's symbol server is likely also a way bigger project than just preserving images of all versions.
> and that's not something they really do
Are you actually speaking for Apple engineering here? The version not being advertised on the update servers does not mean it's not available.
Also, there's the question of supply chain audit. I doubt their security would accept not being able to say "this is/isn't what we shipped at the time".
> During the boot process, unless boot security has been downgraded from Full Security, the contents of that SSV are verified against its tree of hashes. In the event that they don’t match perfectly: “the startup process halts and the user is prompted to reinstall macOS”.
This is written a bit confusingly, if you don't already knows how it works (even though a later reply to a comment tries to clarify it). Because it sounds like the disk is fully and instantly validated at every boot, and "the startup process halts" if there is a mismatch.
Instead, depending a little on the actual implementation, most hashes will very likely only be validated when the corresponding block is actually accessed (up the entire path to the root, where the higher level hashes may be validated now or may have been validated already; this again is an implementation detail).
The result is also, that some blocks may never be validated during normal operation, simply if some files are never accessed. Unless there is some explicit whole disk validation sometime, e.g. during installation, though any corruption happening after that last "full disk check" will still lay dormant until access or the next full check.
But since this is a tree of hashes, it still provides all security and integrity benefits to anything that does get accessed. What never gets accessed does not matter per definition.
I don't know. I/O error and panic both make sense to me, but I'd say the latter only in specific situations. It seems unwarranted to me to panic the system when, say, a README.txt somewhere in the fs has a non-validating block.
Yeah, that's basically what I was getting at. For things like the root hash, it probably makes sense to panic (or otherwise impede boot), in order to have an early, more descriptive failure, than to later only implicitly access the root hash by validating the first accessed page of the init process (launchd, in macOS), and panicking with a less obvious I/O error reading launchd.
In most situations, you likely just want the regular I/O error and then whatever happens downstream of that.
Are there any distros that offer similar integrity checks? I understand Chromebooks provide similar assurances using dm-verity but what about more general-purpose distros?
Anything that uses ZFS gets it by default, as well as most configurations of Btrfs. Not secured by a TPM key (unless you go to great lengths), but for accidents it's plenty.
As well as hardware control including their own controller which may implement more ECC under the hood (and I'm 95% sure I remember that part of the reasoning for lack of full checksumming in APFS was "well our hardware doesn't have those errors"), 9GB also isn't actually that much even over tens of millions of systems. ZFS is used with an absolutely massive variety of hardware from expensive to cheap, with petabyte levels of data (or more at this point) in single installs. I've seen some real weirdness with APFS on non-Apple storage devices, and corruption crept into some of the data stored on my Macs somewhere in the decade before I started using ZFS (images I know were fine in the 00s with data errors I found much later, there is probably other stuff but of course the whole problem is that I don't know and have no great automated way to find out).
This isn't dunking on Apple putting together a reliable product in their own walled garden using their own full array of vertically integrated options at all, but at the same time the ZFS team can be entirely correct given a very different problem space.
Apple system volumes are stored on the internal SSDs sold by Apple as part of the system. It's reasonable to believe that these SSDs are higher quality and more reliable than many disks used with Linux machines.
I run a decently large zfs setup (in the single digit petabyte range) and I have never actually had a spinning disk trigger a zfs checksum issue that wasn't shortly followed up by the disk failing completely in the next week or so. And this is multiple years of this pattern.
While it has to happen occasionally, it does seem to be a pretty rare event in my experience. Drive actually failing are way more likely to happen it seems.
SSV is more a verification technology rather than a data integrity one. It will detect bitflips but it’s really designed to have an immutable root image for the OS, so you can wipe the data partition easily and have a clean, untampered OS image to boot from.
Thank you for putting this all in perspective. While I am one of the more vocal users calling for Apple to implement user-data integrity checks, the numbers you’ve put forth in this article imply that the potential risks are indeed quite minimal.
Common in Mobile and some of the IoT world. Blackberry was doing this ages ago, it was why you never wanted to have to power cycle them. Took ages to boot
Merkle trees! You (simplified) hash the data at the leaves of the tree, merge adjacent leaves (concatenation for example), and hash the new leaves, until you're left with just the Merkle Root, a single hash representing the entirety of your data. Verifying the root is easier than loading the data itself into memory and hashing it there since you can verify it piecemeal and without loading it all into memory at once.
Each level of the tree contains a hash of the hashes from the level below, so to verify the top hash you need only hash all its children (which are hashes themselves). Then you can explore a child, continuing recursively. At the bottom you find a hash of an actual file (or part of a file), then hash the file to check it’s validity.
This allows you to only hash those parts of the tree you actually want to read.
If you want to verify the entire image, I don’t think you can get around reading the entire image. Because any part you didn’t read to verify the hash is a part that could contain corruption.
FTA: "This is the ingenious part: verification carries little overhead, as it runs as a rolling process once the top-level ‘seal’ has been verified. So verification is a continuous process as the SSV is mounted and accessed."
As far as I know, Apple has the NAND controller integrated into the SoC. The NAND controller needs to have ECC to keep the error rates on flash down to something acceptable.
The standard for SSDs goes far beyond the ECC used on DRAM. Every SSD controller designer will have their own proprietary scheme, often a simple and fast BCH that falls back on LDPC for higher error rates. The exact codeword sizes and other parameters of the codes are vendor-specific.
This feels weird. So they have a very low number of possible systems to support. The OS is immutable. Why would they go with a reinstall at that point instead of "we know which part failed, so we'll re-download that one block from Apple service"?
(I guess that one ties in with: so you've got a low number of systems and an immutable OS - why does it take 15min to apply an update)