The crypto routines are mostly in spp.c (there are hand-written C implementations of AES and SHA3 in aes.c and sha3.c, but RSA is done directly off the top of OpenSSL's BN routines).
There are (I think?) at least 3 really bad crypto bugs here, but I didn't look really hard for them. Since this is shellcode, and not something anyone would ever legitimately rely on, it might be worth treating this like an exercise. Anyone want to spot the bugs?
I wrote the PIC with overall intention of minimizing code size and while I was aware of the unauthenticated length, lack of RSA padding and potential for MitM attack, I didn't think anyone would be even remotely interested in the code for its intended purpose. For me, it's just a bit of fun.
I've since added authentication of length but I'm not suggesting it's solved any other problems that might exist. If you spot them, I'd be happy to fix.I'm not aware of anyone that bothered to test it so that tells you how much interest there is in it.
I've looked at some of the AEAD algorithms from CAESAR and will probably choose between OCB and KetJe unless the results indicate something better.
But for now, this code is really just experimental. It's merely a PoC and nothing more.
As for using OpenSSL BN/RSA library, the reasons for this are simply to avoid implementing a PRNG with random tests and the additional math routines to generate the key pair.
I intend to implement those routines later which would also be used for signature verification but for now OpenSSL gets the server component working with client (which was my main focus).
Anyway, thanks for your interest. I appreciate feedback good or bad.
Your code isn't any worse than tens of commercial systems I've looked at professionally. The difference is just that yours is shellcode, so there's no-harm-no-foul in poking holes in it publicly.
OCB is a good choice, as would be Chapoly (since the Salsa core and Poly1305 MAC are both extremely simple in code).
Since you're Windows native, you should use the system's secure random number generator rather than OpenSSL's.
I still don't understand the authentication model here, or how RSA is really helping.
Another reason for using OpenSSL was to eventually get the server component running on unix-based OS. I should have explained that in post because it does seem odd using OpenSSL on Windows when I could just as easily use Crypto API.
"I still don't understand the authentication model here"
Sorry if I misunderstand but If by that you mean the way in which packets are transported between 2 systems;Length first, then data. I just wanted a really lightweight method of transmitting data so that it would be easier to implement in assembly rather than using TLS, SSH or some other well established protocol that would require lots of code. It's possible to use some really lightweight implementation of TLS but I assume the code would grow significantly.
There is support for SSL + TLS through SChannel and I probably will write something with this in the future just to see how it works out.
If you mean I could simply embed whatever keys are required inside the payload and use those for encryption? I thought with some form of key exchange, it would make the traffic more difficult to analyze.
If session keys were inside payload then it's just a simple matter of extracting keys from this and decrypting traffic.
I looked at poly1305 for authentication briefly but must test it out, it does look more compact than using AES + SHA3.
Until you mentioned here, I wasn't aware truncated SHA2 hashes were immune to length extension attack but even without the HMAC code, SHA2 still generates more code than SHA3.
I try to avoid off-topic posts like this, but I just wanted to say your crypto knowledge is always a joy to read on HN. The fact you inspired some conversation on this topic + made it a fun exercise to boot is awesome and serves to solidify the fact you're an awesome contributor to HN.
I also really like the idea of using SHA-256 hashes for trivia. A perfect way to let people guess the problem and being able to say "I knew that one already." and being able to prove it! A very elegant solution and one I will certainly use in the future for any forum-based trivia games.
Apart from what he points out in the article ("The PIC client is susceptible to a MitM (Man In The Middle) attack because it does not verify if the public key sent by a server is from a trusted party")
1. Weird encryption which seems to be XORing the plaintext against the same encrypted IV?
2. Using random padding instead of PKCS5 or similar?
3. Using memcmp instead of CRYPTO_memcmp or similar?
2. The padding is definitely concerning; see if you can devise an attack from it.
3. That's a real issue, but I'm not counting it (for my own count) since a memcmp timing channel against a general purpose CPU over IP is extraordinarily difficult to exploit.
Well, on closer inspection it seems like the MAC isn't actually an HMAC, but just SHA3. And since the padding isn't verified (because it can't be) that leaves it open to simple extension/truncation attacks (Crytopals challenges 29 and 30)?
Nope. It's safe to make a MAC out of a keyed SHA3 hash (one of the criteria for the SHA3 contest was that none of the hashes be vulnerable to length extension). Fun fact: that's also true of the "truncated" SHA2 variants (SHA2-384 and the like).
I'm not really an expert, but it seems to me there's a bit of a problem with the message structure (length + message + HMAC). You can't check the signature without knowing the length, so the HMAC can't cover the length, so an attacker could plausibly truncate messages.
I think he's using OpenSSL's RNG (that's what BN_rand is). I think (but, if you put it to me, am not off the top of my head 100% certain) that OpenSSL's RNG will automatically initialize itself if you don't.
No padding used for keys sent by client? Public Key sent by server not verified by client? Keys sent by client are not verified by server? I suspect there are whole heap of problems wrong with this but thats just what i see atm.
Is it weird that this article attributes the idea of deriving shellcode by compiling C code to a 2010 article? That's how the shellcode in smashing the stack was generated.
The crypto routines are mostly in spp.c (there are hand-written C implementations of AES and SHA3 in aes.c and sha3.c, but RSA is done directly off the top of OpenSSL's BN routines).
There are (I think?) at least 3 really bad crypto bugs here, but I didn't look really hard for them. Since this is shellcode, and not something anyone would ever legitimately rely on, it might be worth treating this like an exercise. Anyone want to spot the bugs?
edit
Here are my three, just so I can't cheat:
e019c30c75f24758a7f0abb26397f289a331c4f26daa6412d8e131924472401c
74971f5e207fdc8db836f959e48a4bb3bfbb8701870bd2d61d6178b0c4cc5109
9a9cb1b33126408bce87021e57d3552f62c8f57200f501909a865b59f64cc995