A vulnerability not mentioned in the article is the normalisation of executing code that has been especially targeted to a specific user or specific device with no validation of reproducibility and no ability for anyone to verify this custom build and download service hasn't been generating backdoored builds.
One should want to ensure use of the same build of xz-utils that Andres Freund is using, or at least a build of xz-utils that other security researchers can later obtain to figure out whether supply chain implants are present in open source software[1].
There's a write up at Mozilla[2] from years ago describing an abandoned attempt by Mozilla to ensure their release builds are publicly logged in a Merkle tree. Google has written up their implementation for Pixel firmware builds but apps delivered through the Google Play Store seem to be vulnerable (unless there is another log I have been unable to find).[3] Apple is seemingly worse than Google on binary transparency with Apple's firmware and app distribution system targeting builds to individual devices with no transparency of builds.
For an example of binary transparency done well, Gentoo's ebuild repository (being a single Git repository/Merkle tree containing source checksums) possibly remains the largest and most distributed Merkle trees of open source software.
[1] Post xz-utils backdoor, some researchers (including some posting to oss-security about their efforts) undertook automated/semi-automated scans of open source software builds to check for unexplained high entropy files which could contain hidden malicious code. This is not possible to achieve with customised per-user/per-device builds unless every single build is made publicly available for later analysis and a public log (Merkle tree) accompanies those published builds.
This is a nice idea, and one I also advocate for, however it's important to keep in mind that the idea of reproducibility relies on determinism. So much of what goes into a build pipeline is inherently nondeterministic, because we're making decisions at compile time which can differ from compilation run to compilation run, setting aside flags. In fact, that's the point of an optimizing compiler, as many reproducible build projects have discovered, turning on optimizations pretty much guarantees no reproducibility.
As long as the compiler is not optimizing by "let's give this 3 seconds of solving time, then continue if no better solution is found", then optimizing is not inherently nondeterministic.
Counterpoint: Archlinux is 89% reproducible with optimizations enabled. The only thing I see which is difficult to make reproducible is optimizations with a timeout.
Instead of using a timeout, an optimization that can must be cut off if the cost is excessive can keep some kind of operation or size count, where the count is strictly a function of the input. For example, an optimization based on binary decision diagrams (BDDs) can put a ceiling on the number of nodes in the BDD.
This is defeatist: compilers do not usually use the system RNG to make decisions, so what's happening is entirely accidental introduction of difference which propagates.
There is "intentional input" (contents of the source files), and "accidental input" (source file full paths, timestamps, layout of memory given to you by the OS, and so on). A reproducible build system should give the same output for the same "intentional input".
(the only place where you do see RNG driven optimization is things like FPGA routing, which is a mess of closed toolchains anyway. It has no place in regular software compilers.)
Well, lot of things can influence here. Multithreaded build, PGO, or even the different access order of the hash table inside the code optimizer can be a factor. Things are getting probalistic and thus somewhat nondeterministic: the build itself is nondeterministic but the runtime/final execution is deterministic
"Reproducible" isn't necessary for "not modified from what everyone else gets", and that still makes some attacks FAR harder (and easier to identify, as you know what the "normal" one is). And a published Merkle tree just makes it easier to verify "none of this has changed", as opposed to SHAs on a website that could change any time.
For sure, which is one of the big benefits of git + git tagging, but the issue is even if you know you received the same binary as someone else, without reproducible and auditable builds, you have no idea if that binary originated from the same code in the case of a targeted attack.
> For sure, which is one of the big benefits of git + git tagging
That's not enough for serious security though, because git is (still) using SHA1 instead of SHA256. You would need something extra, like a signed commit.
There's also the much simpler pitfall of an attacker just creating a branch named the same as a commit, in the hopes that people will accidentally check it out instead.
It's easy to defeat right now because very few are currently thinking about secure build systems.
As an example, systemd's latest source tarball has two Web Open Font Format (WOFF) files in a documentation folder, a favicon.png, two few small screenshots and error messages that have unexplained 128bit identifiers. There are also translated strings (PO files) which could include obscure writing systems few can quickly audit, and thus could be advantageous to an attacker wanting to hide malicious code.
The problem with most build systems is the entire source tarball is extracted into a single location and any build scripts executed during build have access to everything in the source tarball. Gentoo's portage does apply sandboxing around an entire build process, just not internally between the different build stages.
Continuing the Gentoo example (being one of the better examples of sandboxed builds), ideally src_unpack could take a compressed tarball distributed from an upstream project and split files into multiple separate paths such as:
- source code files and build scripts needed to build binaries
- supporting test files needed to test the the built binaries
- supporting documentation/binary/high entropy data such as application icons, game data, documentation, etc that should be copied upon installation but aren't required to build binaries
Then src_prepare, src_configure, src_compile, src_test and src_install all have different sandbox configurations restricting which types of files separated during src_unpack each build phase can interact with.
For the systemd example above, some possible improvements could be:
1. Remove WOFF font files and use system default fonts, omit favicon.png and omit screenshots. Or only make WOFF font files, favicon.png and screenshots available for copying during the src_install phase, and ensure they are not readable by build scripts during src_configure and src_compile.
2. Generate error message identifiers using an explained approach such as hash_algorithm("systemd_error_" + name_of_error_constant) to generate nothing-up-my-sleeve identifiers.
3. Only provide access to and include in the build process any translation files of languages the user cares about. Or only make translation files available for copying during the src_install phase and possibly src_test too, and ensure they are not readable by build scripts.
These build system security techniques are obviously more work, but are generally straightforward to understand and implement. These technqiues are in the realm of possibility for smaller embedded Linux systems that may just be kernel + BusyBox + a few small scripts + a bespoke application + a handful of dynamic libraries. And for more complex Linux systems, these techniques are within the realm of possibility when targeted towards high value software, such as software requiring or often executed with root permissions and software requiring simultaneous access to the Internet and access to user files.
You can shift characters between adjacent fields without changing the hash. Maybe you cannot compromise the system directly, but you could poison the cache with a broken image, or induce a downgrade.
Not quite. HMAC helps to prevent length extensions attacks (if the underlying hash was vulnerable in the first place), and the secret prevents attackers from predicting the hash value (like OP did).
But HMAC doesn't help against ambiguously encoded inputs:
hmac(key, 'aa'+'bb') == hmac(key, 'aab'+'b')
You want a way to unambiguously join the values. Common solutions are:
- prepending the length of each field (in a fixed number of bytes);
- encoding the input as JSON or other structured format;
- padding fields to fixed lengths;
- hashing fields individually, then hashing their concatenation;
> When I saw this, I wondered why it has several inner hashes instead of using the raw string.
The inner hash constrains the alphabet on that portion of the input to the outer hash, thus easily letting you use a separator like "," or "|" without having to deal with the alphabet of the inner input, since it gets run through a hash. That is, for a very simplistic use case of two inputs a & b:
sha256(','.join(
[sha256(a), sha256(b)]
))
If one is familiar with a git tree or commit object, this shouldn't be unfamiliar.
Now … whether that's why there was an inner hash at that point in TFA's code is another question, but I don't think one should dismiss inner hashes altogether.
I could see an attack vector here based on file/directory names or the full path. Different inputs could lead to the same order of enumerated checksums.
I'm not dismissing them, inner hashes returning a hexadecimal string fulfills the "the separator should not be able to show up in the inputs" constraint.
Thanks—that makes sense. I was struggling to come up with an example that would fail but I was just unconsciously assuming the separator wasn’t showing up naturally in the individual parts instead of explicitly considering that as a prerequisite.
What do you mean by "incremental hashing"? Note that the Init-Update-Finalize API provided by many cryptography libraries doesn't protect against this - calling Update multiple times is equivalent to hashing a concatenated string.
That page includes an example that shows PHP's incremental hashing is what you describe as "dysfunctional". It hashes "The quick brown fox jumped over the lazy dog." in 1 part, and in 2 parts, and shows that the resulting hashes are equal.
For anyone curious PHP ultimately uses this definition in their introduction portion of the hash extension:
> This extension provides functions that can be used for direct or incremental processing of arbitrary length messages using a variety of hashing algorithms, including the generation of HMAC values and key derivations including HKDF and PBKDF2.
I've worked with many cryptography libraries and have never seen an Init-Update-Finalize API that works the way you think it does. It does not protect against canonicalization attacks unless you're using something like TupleHash.
Maybe make it clear you are being sarcastic here. English is not my native language, and my initial interpretation was that "they" in your post referred to the "business grade closed source stuff", and that OpenWRT is really a dangerous bet because they are guilty of all the things you listed.
To be fair, that initial confusion is the intended effect of OP's humor. Poe's law and all, but you did figure it out so the joke seems effective. Prefixing or suffixing with sarcasm warnings neuters the joke.
Just why I love OpenWrt. They even ask the people that use screen readers like me to test the web interface to make sure that all is working as it should.
Whilst this is true, it looks like OpenWRT fixed the hash truncation but not the command injection.
I hope they're planning on fixing the command injection. As the blog post says, the created images are signed. Even without the signing, it's code execution from untrusted user input. And of course vulnerabilities can be strung together (just like in this hash collision case).
I have a router that from my ISP I am forced to use that has had a few CVEs ranging from not good to really bad. Most of which are years old. I can get a replacement but it's just the same model. They don't care about security at all and don't care about patching it, even though they have exclusive access rights to the router and can remotely log in to it. It's completely ridiculous.
The one I use looks scary too. And it came by default with a dumb password too. I wouldn't be surprised if it had a few CVEs hanging too.
> I have a router that from my ISP I am forced to use...
A friend of mine did impersonate the ISP's router's MAC address and used wireshark to sniff the traffic when the modem started. He then configured the ONT (which is physically inside a SFP plug, it's tiny) to establish the handshake/send the credentials.
It's a sad state of affairs, but anyone serious about security ought to consider the common ISP WiFi router to be a potentially hostile device and class it as part of the public side of the Internet. The usual advice is to put a firewall/router of your own running your preferred software, between the ISP device and your network.
Routers supplied by AT&T here in the US for their fiber gigabit service do RADIUS authentication with the carrier gateway using certs built into the device. There used to be an older version of this router that had known vulnerabilities which made extracting those certs possible but they've since been patched and those certs have been invalidated.
Note that you can still downgrade an existing gateway, extract certs[0], then bypass the device [1]. I had to do this with OPNsense to avoid the latency buildup issue, which has been ongoing for months[2].
I believe you can set those to pass through mode and put a router/firewLl behind it without any kind of double NAT. Other than some kind of MITM, you have at least minimized the likelihood of someone using it as an entry point to your network.
Ha was very user unfriendly when I last tried it ~3 years ago.
Yaml was necessary and it required a lot of fiddling to make z-wave work. Each blind was detected as ~5 things (2 useless or no idea what for)... Checking what was position, what power, ect was rather annoying.
I made work and something broke about a year later. I just replaced it with off the shelf stuff.
HASS configuration has gotten a lot better in the past few years. Almost everything can now be done via the UI, including automation and scripting, and it's one of the smoothest scripting GUIs I've used. It even supports cut/paste for visual blocks. And for those 5% cases, there's an inline YAML editor which will open (and validate) only the pertinent block of what I'm sure is a 1000-line YAML file for editing in-browser.
Z-Wave is still dodgy, but the migration to zwavejs has been an improvement and probably is as good as things will get with the state of Z-Wave being what it is.
It's still not perfect, but HASS has become one of my user-facing open-source success stories. Most of the remaining annoyances are out of their control at this point.
First - open source tool adjusted to the task it wasn’t made for in short time, only because it is open source and written without BuilderFactoryProvider.
(already mentioned so I’m sorry, but it’s killing me every day)
Big company would take probably -1 years to fix this, because it would just sue the guy, try to arrest him ASAP and never release patch.
OpenWrt after getting information just took the insecure service offline, checked the report (while clients were already safe, because of the shutdown), made patch and released in 3 hours. Wow!
Truncated hash functions are not vulnerable to length-extension attacks. But you usually take SHA512 and truncate to 256 bits. Anything shorter than this isn't really considered safe these days.
Sometimes it’s done to fit into an existing tool/database that has a preexisting limit. Or when the hash is used only as a locator rather than for integrity.
According to @Reid's answer in [2] and @ThomasPornin's answer in [3], the idea of truncating hashes is fully supported by NIST, in fact SHA-224 is just SHA-256 truncated, SHA-384 is just SHA-512 truncated, etc.
First of all, nice writeup. I am a bit surprised that so much GPU power was needed to find such short collision but it was nice to see his implementation nevertheless.
Regarding the last section, is 40k a reasonable price for one month of security analysis? Does this mean that a good security researcher make about 500k/yr?
It means a good security research company might make $500k for a good researcher, if they could bring in enough work to keep them 100% utilised. Less actually, given paid time off.
Sick leaves, maternal leave, underutilized for sure (toilet, meetings etc).
Just for reference, I have had an audit from PwC and they were skeptical about our 65% time utilization because usually anything above 60% is fake at least partly. LOL, I thought, they were right, we ended up just about 60%.
That seems very reasonable to me. It seems like the pentest companies I have worked with in the past charge that much and just do a lazy nmap/metasploit scan and wrap it into a nice PDF.
A bit over 4 hours at 18 billion per second, but yea. Impressively fast and also a completely reasonable amount of time for an attempt - the CPU version was 10m per second, which is most of a year to search the whole space.
What's this about hashcat performance being orders of magnitude different depending on arg order? Is it scanning the argument line for target pattern with every execution?
Could it be like a lock pick process where you start from the left and see if you get further or can throw away that guess, so by having the "choices" be at the beginning you don't have to make them again and again? (and for whatever reason doesn't/can't cache the prefix)? Or could it be like when counting
most of the variation is at the left and you only rarely see changes at the right? Would be interesting to get this answer from from someone who knows hashcat and isn't just pulling answers out of the air like me :)
That just makes it even more confusing that it uses longer to vary the right than the left – is the specific hash algorithm used not "balanced" (or whatever the term of art is) the way it ideally/theoretically should be?
The title uses the term "supply chain" but it appears nowhere in the blog post. I keep seeing this term used by "cybersecurity" researchers and software developers in ways that seem to differ from the definition I learned in school. .
From Wikipedia:
"A supply chain is a complex logistics system that consists of facilities that convert raw materials into finished products and distribute them^[1] to end consumers^[2] or end customers.^[3]"
1. ^ Ganeshan, R. and Harrison, T. P., An Introduction to Supply Chain Management, updated 22 May 2005, accessed 29 June 2023
2. ^ ^a ^b Ghiani, Gianpaolo; Laporte, Gilbert; Musmanno, Roberto (2004). Introduction to Logistics Systems Planning and Control. John Wiley & Sons. pp. 3-4. ISBN 9780470849170. Retrieved 8 January 2023.
3. ^ ^a ^b ^c Harrison, A. and Godsell, J. (2003), Responsive Supply Chains: An Exploratory Study of Performance Management, Cranfield School of Management, accessed 12 May 2021
The device updates can be supplied by a supplying service. The device (and its user as the end consumer) is not attacked directly but through its update supply chain. This is why it's called supply chain attack.
When somebody intercepts your Christmas presents to add a bomb to your new pager, it is also a supply chain attack. Even if you use the pager for work and the bomb targets your business partner. If somebody throws the bomb directly at the target it is not a supply chain attack.
Supply chains are often less secured than direct attack vectors.
Hi, I'm the one who created this services within a "Google Summer of Code" some 7 years ago and been maintaining it since. It's my first "larger" project and while it started as a short Python project, I eventually became a OpenWrt project member since the build system itself required so many changes.
I'd be very happy for further audits and reviews of the code, after multiple years of low interested, it now produces and caches about 1000 individual firmwares a day. I think it's only a question of time until other issues come up...
OpenWrt is also very difficult to safely upgrade on some devices which I would also consider as a huge downside. I finally gave up and bought an old Dell off eBay and installed OpnSense and am much happier.
Some devices are hard to upgrade in large part because they were never intended to be used with things like OpenWRT.
To that end: While it can be nice that OpenWRT runs on a quirky compact all-in-one MIPS-based consumer router-box (or whatever), the software also runs just fine on used Dells from eBay.
That was an excellent report and a really decent technical explanation. Good to see how quickly OpenWrt (one of my favorite open-source projects) fixed and addressed this vulnerability!
One should want to ensure use of the same build of xz-utils that Andres Freund is using, or at least a build of xz-utils that other security researchers can later obtain to figure out whether supply chain implants are present in open source software[1].
There's a write up at Mozilla[2] from years ago describing an abandoned attempt by Mozilla to ensure their release builds are publicly logged in a Merkle tree. Google has written up their implementation for Pixel firmware builds but apps delivered through the Google Play Store seem to be vulnerable (unless there is another log I have been unable to find).[3] Apple is seemingly worse than Google on binary transparency with Apple's firmware and app distribution system targeting builds to individual devices with no transparency of builds.
For an example of binary transparency done well, Gentoo's ebuild repository (being a single Git repository/Merkle tree containing source checksums) possibly remains the largest and most distributed Merkle trees of open source software.
[1] Post xz-utils backdoor, some researchers (including some posting to oss-security about their efforts) undertook automated/semi-automated scans of open source software builds to check for unexplained high entropy files which could contain hidden malicious code. This is not possible to achieve with customised per-user/per-device builds unless every single build is made publicly available for later analysis and a public log (Merkle tree) accompanies those published builds.
[2] https://wiki.mozilla.org/Security/Binary_Transparency
[3] https://developers.google.com/android/binary_transparency/ov...