I didn't read the whole paper, but how can this even happen? Seems like the buffer overflow would be triggered for any file larger than 4 GiB, which I assume someone has tested in the 8 years since it was released.
> I didn't read the whole paper, but how can this even happen? Seems like the buffer overflow would be triggered for any file larger than 4 GiB
I skimmed the paper, and as far as I understood it:
Most cryptographic hash functions operate in fixed-size blocks (for instance, 32 bytes). Additionally, most cryptographic hash function implementations are designed to be streaming, that is, they do not receive the whole input at once. If you give them a partial input which is not a multiple of their block size, these implementations have to buffer the partial input, so that it can be combined with the next partial input (or flushed, if the next call is to finish the computation and generate the output). The arithmetic overflow which leads to the buffer overflow happens when computing how much it has to buffer, given a partial input and any previous partial input already in its internal buffer.
That is, having a file larger than 4GiB is not enough; it has to also be cut into pieces which are not a multiple of the block size (which is normally a power of two). Most users of a cryptographic hash function will either give it the input in large power-of-two pieces (for instance, 8 KiB or 64 KiB), or give it the input all at once, and thus will not hit the bug.
You'd be surprised how many of those submitted and approved crypto standards are still not tested with industry best practices.
buffer overflows or integer UB's and overflows are very common.
ubsan, asan, valgrind tests are missing. some do offer symbolic verification of the algo, but not the implementations.
Proof of constant-time execution is an interesting field, but as I understand very much less mature than the SPARK toolset. If anyone has a toolchain working over llvm to check and/or make code constant-time, I'm interested.
I mean, if the standards people want to keep writing C, they can probably use Frama-C for the standard implementation...
It was worse, it wasn't a (0,0) key it accepted. If that was all then you could blame the user for loading in a bad key etc. No the vuln was that it accepted (0,0) as being a valid signature over any text and validated using any public key! So you could forge any signature by simply using (0,0) as the sig itself!
To clarify, this only affects EdDSA as far as implementations use SHA-3 to hash a message before applying the signature. The actual elliptic curve operations code seems to be fine.
This is rather sad that one must give up the range and add the signed range just to avoid overflow bugs. Is there no way to make overflow not the default and instead trap unless one uses `add_wrap()` or `add_no_wrap()` in case it's not default?
There is overflow-checking math in GCC/Clang and standardized in C23, but I'm not sure if there's anything opt-out. If there is, I don't know how to write it.
No? The effect of that is well-defined, and the cast is a pretty strong signal that the author is deliberately converting the value. Casts to unsigned that deliberately discard the high bits are relatively common.
Yes, the loss of precision warnings. It may be these only happen if you compile as C++ and not as straight C? (I don’t do straight C much.)
Of course you’ll run into dozens of instances of it with old C code… and my experience has been that some of those instances are bugs similar to this one.
I tried to compile it as C and C++, with both Clang (14.0.0) and GCC (11.3.0):
gcc -Wall -Wextra # no warning
g++ -Wall -Wextra # no warning
clang -Wall -Wextra # no warning
clang++ -Wall -Wextra # no warning
clang -Wall -Wextra -Weverything # loss of precision warning
clang++ -Wall -Wextra -Weverything # loss of precision warning
However, the warning goes away if there's an explicit cast:
unsigned u = (unsigned)llu;
Worse, I still have no warning if I do the narrowing cast then affect it to a wider variable:
long long unsigned u = (unsigned)llu;
printf("%llu (u)\n", u);
In C++ I'm warned about the old style cast of course, but using `static_cast` makes the warning go away. And of course the code overflows just like before.
I don't have a good solution to this. Sometimes I do want to lose the precision. Bignum arithmetic for instance. In any case, I'm pretty sure the Keccak team's compiler did not issue any warning. Sorry if I implied otherwise.
Thank you, I wanted to do this last night but haven’t had the time.
I think it’s perfectly fine for an algorithm contest not to require a bugfree implementation… but I also feel that if we’re going to standardize on a cryptographic algorithm, throwing all the tools and formal methods we can at it during development of the reference implementation makes a lot of sense.
I wonder if this could be avoided by writing the canonical implementations in Rust or better yet in some system with formal verification.
This is such a critical part of the software stack, that we need a more reliable way of validation than just a bunch of people staring at the code written in C.
Rust won't help. Sure the compiled code would be bounds-checked, but nobody would notice the bug unless they gave it the crashing input. And then when they reimplemented the code in their non-bounds-checked language then that would reintroduce the bug anyway.
A formal verification implementation would catch it at authoring time, yes.
It helps. In Rust debug builds, integer overflows crash -> tests will detect them. In release builds they're not detected by default, but you can add "overflow-checks = true" to the Cargo profile to enable those checks in release builds too if you want.
I just can’t get my head round the idea that software written and reviewed by experts and submitted to the “National Institute of Standards and Technology” with a budget of 1 billion dollars can fuck up this way.
I’m no mathematician but I would have thought implementing pure number crunching code is not rocket science.
Buffer overflow, overwrite memory, run arbitrary code, seriously? LOL, WTF.
Nobody with any experience would laugh at mistakes like this. It's only easy in hindsight. Past 30+ years of collective experience in our industry shows that these classes of bugs are nearly impossible to completely stamp out in any language but especially in memory unsafe ones, even with dramatically better compile time and runtime tools that can spot many of these nowadays. During the early days of the internet and the buffer overflow attacks after Morris worm, buffer overflow bugs existed in practically all software. There were times when pretty much any servers connected to the internet could be had relatively easily.
Even with memory safe languages, there are dangers. Humanity just hasn't figured out how to produce completely bug-free code at the scale we need in general, let alone in a memory-unsafe language.
This particular mistake is all the more infuriating because it comes from a precaution. Or trying to silence a compiler warning:
partialBlock = (unsigned int)(dataByteLen - i);
Where both `dataByteLen` and `i` where actually `size_t`.
Assuming this is close enough to C, what happens is that we're converting a difference between `size_t` into a mere `unsigned`, and since they're not the same sizes on 64-bit platforms this can give `partialBlock` the wrong value, and the whole thing then snowballs into a catastrophic error that is not trivial to test because it only happens with huge buffer sizes.
The biggest mistake here is having written `(unsigned int)` instead of `(size_t)`. But the reason it happened in the first place is because they tried to do the right thing: writing the cast as a precaution, even though the following would have worked:
partialBlock = dataByteLen - i;
I really can't fault them: because it was a difference it could theoretically yield a "negative" result, and therefore intuitively the type of a difference should be signed, so we should cast it back to unsigned to be crystal clear. I knew C was dangerous, but to be honest I didn't expect such a wicked mind game.
But some people might think that it could, so putting the cast could increase readability for them. A similar reasoning applies with operator precedence: even though it's very well defined, we tend to forget it, so instead we put additional parentheses. But in a couple specific cases it hurts more than it helps.
So, (unsigned int) - (unsigned int) → (unsigned int) is guaranteed on every platform.
But relevant to the bad SHA-3 code, the resulting type of size_t - size_t can be surprising. First, size_t is not guaranteed to have any relation with unsigned int; it could be narrower, the same, or wider. For example, size_t could map to unsigned short, unsigned int, unsigned long, or unsigned long long.
If size_t is defined as unsigned short (note that size_t must be at least 16 bits), and short is 16 bits wide, and int is 32 bits wide, then the calculation of size_t - size_t will be promoted to (signed int) - (signed int), and thus the result will have the type signed int.
Agreed, people wildly underestimate how difficult it is to produce correct code. I think if people adopted the mindset that it's impossible, we'd be closer to the truth. Some of these important projects should only hire genius-level developers who code at a max speed of 1 line per day. Every line of code should be added with an unfathomable degree of thoughtfulness and care. I wish I was joking, but I'm not.
Shipping a non-trivial program which has more than a few thousand lines of code borders on impossible.
Buffer overflow in hash function is unprecedented, buffering simply doesn't work this way, it was figured out long ago and never changed. Smells like bullrun to me.
It's true that Snowden revealed the NSA had their fingers in NIST's cryptographic standards team, with Dual-EC a specific example that is considered suspicious since the revelations. So a suspicion of malice is not completely unfounded, but as far as I can tell, this code was written by the Keccak team not NIST itself, and for any claim of "they're NSA stooges", I would need evidence.
It also seems to me that the Keccak team are not stupid people. That leaves "honest mistake" as the most likely explanation.
There are lots of studies on human behaviour that show a competent and diligent human will mess up every Nth time they perform a routine task; some forms of probabilistic risk analysis take N = 10^3 as a lower bound for this.
There is a long history of railroad operation in the UK (who invented the steam train after all) where occasionally, a signaller would send an express onto a line where they'd forgotten that the local was standing, or two trains head-on down a single line in opposite directions. This led to the development of interlockings and token working systems as technological solutions to mitigate the risks of human error, and later on to today's computerised safety systems, because signallers are (almost always) neither stupid nor malicious, but still human. The same can be said for programmers.
(As I understand, the recent train disaster in Greece was on a line where there should have been interlocking in place, but it wasn't active.)
