Hacker News new | past | comments | ask | show | jobs | submit login
How I implemented my own crypto (loup-vaillant.fr)
380 points by loup-vaillant on Aug 3, 2017 | hide | past | web | favorite | 386 comments



I would like to clarify:

The thing that I said was "table stakes" for implementing cryptography was passing the algorithm test vectors, which this author's previous post claimed as a security feature.

If you're unfamiliar with the concept, a test vector is a series of strings and intermediate values used to ensure that your (say) OCB3 is the same as everyone else's OCB3.

Had he asked, I'd further claim that not having dumb C bugs is also table stakes for cryptography, since serious crypto vulnerabilities happen at a higher level of abstraction. The author grazes past (somewhat disconcertingly) one such class of bugs when he discusses carry propagation and limb scheduling in the context of Poly1305. Improper carry propagation is an example of the kind of security vulnerability for which there are no test vectors and no memory safety validation tools.

You just have to know what a carry propagation bug is, where to look for them (not Poly1305), and what the impact of one is.

Hopefully, the author of this library does.


> You just have to know what a carry propagation bug is

This is one of the things that drives me nuts about the crypto community. The natural response upon reading, "You just have to know X" if you don't already know is to go search for X. Well, if you go search for "carry propagation bug" you will find lots of examples of carry propagation bugs being found and fixed, but no explanation of what one is. The thing that "you just have to know" is not so easy to find. The courteous thing to do when mentioning something that "you just have to know" but which is not easy to find is to provide a pointer or a short in-line explanation. When you don't do that you leave your reader with no choice but to publicly expose their ignorance of this purportedly crucial knowledge by asking, which I will now proceed to do as a public service: What the hell is a carry propagation bug? Where do you look for them? And what is their impact?

And while we're at it, what are "table stakes"?


Carry propagation bugs are something that can go wrong in a bignum implementation.

Bignum is what you need when you have one number that you're splitting between multiple variables (because it's a 256-bit number, and you want your library to work on 32-bit PCs, for example).

Carry propagation is if part of your result is too large for one part of your number and needs to overflow into the next part.

Actual carry propagation bug and bugfix available here: https://news.ycombinator.com/item?id=8857363


Thank you for that constructive response!



Give me a fucking break.

Sean, Marcin, and I spent the better part of a year individually replying to tens of thousands of emails from people doing the Cryptopals challenges that we wrote and published for free. Do you know how much work that was? Here's a hint: it was a lot of work.

There are a lot of charges you can level at me to try to win a dumb message board argument, but the one where I want cryptography to be a secret priesthood known only to the anointed select is not one of them --- nor is the argument that I haven't taken the time to help make these kinds of cryptographic attacks easier for people to learn about.

Please immediately find one of those other arguments to substitute for this one, and, if you have it in you, take a moment to retract.

Thanks in advance.


I do sincerely apologize for taking my frustrations out on you, because you have indeed gone above and beyond the call to help make crypto accessible.

Nonetheless, I still stand by the substance of my comment: when you (not you specifically, but anyone) say, "You have to know X" you should check that doing a Google search for X yields some reasonable results, and if it doesn't, provide a hint on how to proceed.


> Nonetheless, I still stand by the substance of my comment: when you (not you specifically, but anyone) say, "You have to know X" you should check that doing a Google search for X yields some reasonable results, and if it doesn't, provide a hint on how to proceed.

Do you actually believe you are personally entitled to demand special research tips from domain experts?


I don't think it's too much to ask for links to resources.

If it's a common enough remedy that needs linking, you could just write a dedicated blog post describing the recommended resources or whatever and just link to that.

Sometimes the hard part of researching things is knowing what to look for or where to look for it.


You can ask for anything you want. But I wouldn't expect anyone to do work for you just because you demand it.

Give me $100 and I'll send you a link to some good crypto books.


Apology totally accepted, although I still agree with no part of any of your arguments on this thread. :)


Read charitably, his post is more an expression of frustration that this concept that is supposed to be an obvious well-known keyword was difficult to find out about in a google search. https://encrypted.google.com/search?hl=en&q=carry%20propagat...

This is more an indictment of the whole community that there aren’t clear glossaries or other introductory resources with good google juice, and doesn’t necessarily reflect much on your specific comment or intent.

He could have phrased it much better, but I think you’re also overreacting.


I thought so too, but then I read his comment downthread about how my wording "rubbed his nose" into the fact that he doesn't know what a carry propagation bug is. Sorry, I stand by what I wrote.


Carry propagation is one pixel in the 4K movie that is safe crypto implementation. One reason not to link to trivia and factoids about safe crypto implementation is that it builds a dangerous false sense of competence.

There are programs that teach this, but they take years. Nobody who's passed through that is interested in creating an easy scaffolding whose levels are unclear.


> Carry propagation is one pixel in the 4K movie that is safe crypto implementation

This is exactly the sort of thing I'm talking about. This is semantically equivalent to, "If you don't already know this, then you are too stupid, and my time is too valuable, for me to do anything beyond pointing out that you are stupid and my time is valuable." That may be true, but it's not helpful.

> There are programs that teach this

But you aren't going to tell me what they are or how to go about finding them? Again, this is not helpful.


With all due respect, some things should only be done by trained professionals.

Crypto is one of them.

Flying a commercial airliner is another.


Funny you should choose flying an airliner as your example because I happen to be a pilot. I don't fly airliners. I'm just a private pilot who flies small planes. But despite the fact that I'm not a "trained professional", I can fly a plane safely. And the reason I can do this is because the flying community does not have this attitude that flying should only be done by trained professionals. To the contrary, the flying community has the attitude that anyone can learn to fly, and if you want to to it, all you have to do is find your nearest general aviation airport where you will almost certainly find a flight school, which will put you behind the wheel (so to speak) of a plane the very same day (though, obviously, with supervision).

The crypto community's attitude seems very different, far less welcoming of newcomers. The aviation community's attitude towards newcomers is, "Come on in, we are happy to help you take your first step on this exciting journey." The crypto community's attitude feels more like, "Go away, you ignorant dweeb, we are far too busy doing Hard Stuff to bother with the likes of you."

And, BTW, I say this as someone who runs a company that sells a crypto product (https://sc4.us/hsm), so I'm not exactly a beginner any more. But I was once and I still remember what it was like, so I have sympathy for people who want to climb the learning curve and find it difficult. I think the world would be a better place if more people learned more about crypto, and so I do what I can to try to remove obstacles.


The "flying community" does not have the attitude that any random person should take the stick on a 737. Like the cryptography community, the flying community encourages newcomers to learn and get certified.

This is a genuinely weird argument you're trying to make.


As a member of both communities I can tell you that I perceive a genuine difference in attitudes. The crypto community says, "Don't write your own crypto." The aviation community says something more analogous to, "Absolutely, write your own crypto! Just don't use the results of your first effort for anything mission-critical until it has been checked out by someone who knows what they are doing. And oh by the way, here's a list of people you can contact who will be more than happy to help you." (In fact, the aviation community literally encourages people to build their own airplanes!)


Sure! Now replace flying planes with building your own planes.


Huh? Did you not read the last sentence?

"In fact, the aviation community literally encourages people to build their own airplanes!"

https://www.google.com/search?client=safari&rls=en&q=build+y...

http://www.kitplanes.com/homebuilders-portal/firsttimebuilde...


It doesn't encourage people to try and build commercial jet airliners in their back sheds. You can follow a tutorial and build a KitFox or whatever, which is the equivalent of coding, I dunno, a virtual Enigma machine or something. You can't follow a tutorial to learn how to design an Airbus A380.


I missed the last sentence. But I think this is also misleading. General aviation is obviously far more dangerous than commercial aviation, and home-built planes are, by a factor of 2-3x, more dangerous than the rest of general aviation.

If you're telling me that there's a mainstream part of general aviation advocating that random people build planes to take strangers for rides in (or, to complete the analogy to crypto, to give to random strangers to themselves fly), I'm going to push back on that.


No, not random people. Just anyone who wants to.


You're telling me that the general aviation community is, in the mainstream, of the mind that "anyone who wants to" build airplanes to sell to other people should be doing so? With no additional qualifiers?


Not to sell. But for their own use, yes, absolutely. Anybody who wants to. It turns out not very many people want to, particularly when they find out how much work it is. But almost certainly the same is true of crypto.


Not only is that not true, it's demonstrably not true: when people write code, they actively want other people to use it. By way of example, the author of the library we're discussing here, "Monocypher", has declared it "ready for production" and has a web page selling its virtues versus libsodium and NaCl.

People who write crypto code as a rule are not doing it for their own edification, which is why so many more people spend time writing libraries and encryption tools and so few people spend time writing the code to exploit crypto vulnerabilities.


> Not only is that not true,

What is the antecedent of "that"? I think you read something into what I wrote that I didn't intend.


You said that it's almost certainly the case that people building their own crypto are doing it solely for their own purposes and that people won't even bother to complete their work. Obviously: no.


I wrote "almost certainly the same is true of crypto" meaning that almost certainly very few people actually want to implement crypto (just as very few people actually want to build airplanes) for any reason, and so the desire to do so serves as a pretty effective filter to eliminate "random people" doing it.

Of those few who want to do it, some will want to do it for personal reasons and some will want to do it for commercial reasons, just as in aviation.


Yes, but home-built airplanes come with all kinds of limitations that are, to continue the analogy, more like "can't use commercially ever, and any use for non-commercial purposes requires a big disclaimer on every message sent.

http://www.odtug.com/media/tvzzjjck.jpg

My use of commercial aviation in the GP was quite intentional, btw.


Yes, but 1) those are government regulations, not a community policy. As a community aviation is much more welcoming of newcomers than crypto, and in the case of crypto this attitude is entirely community-driven, since crypto is not regulated the way aviation is. And 2) homebrew is often a stepping stone towards professionalism in many fields. But the crypto community actively discourages homebrew with its "Don't build your own crypto, period, end of story" message. The message should be, "Sure, build your own, just don't rely on your first efforts for anything mission critical."


You can't view community policy and government regulation as two unrelated things. The aviation community has a more welcoming attitude because of government regulation.

They know that someone's backyard airplane isn't ever going to be used commercially without certification, and even if it were the entire world's aviation fleet isn't going to start using that model overnight, as might happen with some homebrew crypto and a popular new app. So the situation is entirely unanalogous.


No, that's not true. The aviation community's welcoming attitude goes way back before the government started to regulate it. I'd say it's because aviation began as a homebrew activity whereas cryptography was born in the military and academia.


Every professional was an amateur once.


I sure hope that all that you have learned has been learned from trained professionals...

Like making love, like walking, like being kind to others, like cooking, etc.


I wish the meme that crypto is hard would just die.

The reason is that the statements is wrong and is probably encourage exactly what it wants to discourage. Crypto is hard is not factually incorrect. The problem is that it is trivially correct, because programming is always hard. Every algorithm and design is hard to get right.

The correct statement, and the one that should be repeated instead is that crypto's stakes are high. Getting it wrong carries bigger consequences than a normal program.

So stop propagating that crypto is hard. (Because people do hard stuff everyday and might thus think they've got the right stuff to write crypto.)

Say that crypto has too high stakes to be done off the cuff.


No, crypto is actually hard. One difference from regular programming is that there really are no 'intermediate' errors. Experience shows that getting one small detail wrong is as bad as getting the whole thing wrong.

I can't think of another programming domain where that holds.


I agree crypto is actually hard.

But there are lots of domains where details matter and small errors have big consequences. Vehicles, medicine, and manufacturing are all places where errors have killed people. Obviously severity depends on the situation, but generally speaking I'd rather my encryption get broken than die.

Software defects in the Therac-25 radiation therapy machine killed several people [1].

Software defects in Toyota's acceleration killed 37 people [2]. "We've demonstrated how as little as a single bit flip can cause the driver to lose control of the engine speed in real cars due to software malfunction that is not reliably detected by any fail-safe," Michael Barr, CTO and co-founder of Barr Group, told us in an exclusive interview. Barr served as an expert witness in this case." [3]

Self-driving autopilot has got to be pretty hard. [4]

[1] http://sunnyday.mit.edu/papers/therac.pdf

[2] https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...

[2] http://www.embedded.com/electronics-news/4423365/Toyota-Camr...

[4] https://www.youtube.com/watch?v=-2ml6sjk_8c


Crypto software also has the property that "internal implementation details matter". Nearly all other software does not have this property - you don't care how your hardware/software multiplies two numbers; only that it does it correctly in a reasonable amount of time. That property does not hold for crypto software - the nature of your multiplication may actually leak information, depending on how it is implemented. So in a way crypto software goes against the entire edifice of modern software engineering, which says that you can build software against an abstract lower-level "stack" and generally not care how the lower parts work.


Yeah, that's a great point, security requires that you understand the insides of any dependencies, and treating any part as a black box is a bad idea.

I think performance critical code sometimes shares this particular side of engineering and it's inability to fit into the easy patterns and abstractions we usually prefer.


I agree too!

The point I'm trying to make is that even in critical software, where people's lives are in the balance, there can be minor errors that don't bring down the entire edifice. Whereas in crypto code, there seems to be only one kind of error possible.


I see a few kinds of intermediate crypto errors possible. You can weaken the crypto without breaking it, you can leak metadata, you can screw up protocols without compromising the algorithm, etc..

But yes in some sense you're completely right, perhaps just because the aim of a crypto algorithm is so narrow? The number of things you can do right with a pair of encrypt/decrypt functions is so limited that the number of things you can do wrong is also pretty small.

There's certainly a lot more than can go wrong with "drive a car" than with "encrypt this message".


In the same sense that modern vulnerability research is in large part the study of leveraging the smallest possible memory corruption bug to obtain remote code execution, modern practical cryptanalysis is the study of leveraging the smallest possible unexpected behavior of any sort to full breaks in both confidentiality and integrity.

By way of example: you could "weaken" an RNG so that where a construction expects a 256 bit nonce, you generate only 255 secure bits. If that construction is ECDSA, there's a good chance you've managed to disclose to attackers your signing private key. You could "weaken" a protocol so that attackers can replace an original plaintext with 16 uniform random bits. If the protocol is using CBC mode, you've allowed attackers to recover whole plaintexts.

There aren't a lot of errors that get routinely made in crypto code that don't have devastating consequences.


> You could "weaken" a protocol so that attackers can replace an original plaintext with 16 uniform random bits. If the protocol is using CBC mode, you've allowed attackers to recover whole plaintexts.

Do you have a link explaining this?


I'm describing the CBC padding oracle attack.

I'm surprised this is the thing you want the link for, and not "1 biased bit destroys the security of a 256 bit nonce where the other 255 bits come from secure random".


> I'm describing the CBC padding oracle attack.

Ah! Wouldn't that be "attackers can replace an original ciphertext with two chosen blocks"?

> I'm surprised this is the thing you want the link for, and not "1 biased bit destroys the security of a 256 bit nonce where the other 255 bits come from secure random".

IIRC the link for that is in your hiring post!


Ariane 5 rocket crashed in 1996 because of one wrong int conversion, costing Euro Space Agency something like $500 millions... how about that for "one small detail"? :)



Distributed databases.

Medical imaging.


Experience shows that getting one small detail wrong is as bad as getting the whole thing wrong.

Then it's highly likely that there isn't a single "completely" secure crypto implementation in existence...


> I can't think of another programming domain where that holds.

Static type systems.


What is the definition of "crypto" that you are using?


> And while we're at it, what are "table stakes"?

(business) minimum entry requirement for a market or business arrangement OR (poker) limit on the amount a player can win or lose in the play of a single hand [1]

[1] https://en.wikipedia.org/wiki/Table_stakes


Being able to Google unfamiliar terms is table stakes for participating in HN :)

http://www.urbandictionary.com/define.php?term=table%20stake...


"table stakes" is an idiom that comes from poker. It's the amount you can win or lose in a single hand, that is, the smallest amount of money you need to have to be able to sit at the table and play the game.


Is it possible that you are expecting the cryptography community to be one organized around being newbie-friendly and encouraging to beginners?

The crypto community has repeatedly been burned by overeager beginners with false senses of competence making rather nasty mistakes. Consequences can be dire, from destroying companies to getting people killed. On the lighter end, you have things like Decrytocat.

In some arenas, it's appropriate and a good practice to assume that everyone reading is a beginner who needs 101-level info on everything. Webdev is a good example - CSS, or cascading style sheets, are good for handling the display logic of semantic markup. The stakes are low and the costs for screwing up trivial. With cryptography, it needs to be driven home to the idle reader that this is serious stuff.


> Is it possible that you are expecting the cryptography community to be one organized around being newbie-friendly and encouraging to beginners?

Not at all. I'm not actually expecting anything. All I'm saying is that as a matter of common courtesy (in any field) if you're going to say, "You just have to know X" then you should do at least one of the following:

1. Check that if someone doesn't know X that they can get at least a lead on how to learn about it by searching for X.

2. Provide a pointer for someone who wants to learn about X to follow as a starting point

3. Provide a brief in-line primer on X

If none of those conditions hold then it's not constructive to say, "You just have to know X." What you're really doing in that case is not providing useful information, but rubbing your audience's nose in the fact that they are ignorant and you are not, and worse, that you acquired your knowledge through some privileged channel to which they do not have ready access.

Honestly, what possible reaction could you reasonably expect other than frustration?


If someone posted a message on HN about doing DIY appendectomies with Arduino-controlled robots and I pointed out that you need to go to med school to do surgeries, you would not in fact be harping on me for not having explained why.


Not at all.

If I don't know how to sign up for med school, I can find out by searching for "How do I sign up for med school?" or even "How do I become a doctor?"

If I search for "How do I sign up for crypto school" the results are far less helpful.


I think you are arguing yourself into an absurd corner here, because I know for a fact that you can name off the top of your head five university crypto programs, and I'm sure you realize that for each of the ones you can name, there are 10 more that others can name.

And: for what it's worth, I don't think going to school is the only way to learn this stuff. I just don't think putting end-users at risk, especially since they can't possibly be expected to tell the difference between Dan Boneh and some guy who assembled a broken ECDH out of random npm libraries.


I re-read your GP comment, and I think I misread it the first time. Let me try this again.

> If someone posted a message on HN about doing DIY appendectomies with Arduino-controlled robots and I pointed out that you need to go to med school to do surgeries, you would not in fact be harping on me for not having explained why.

Actually, I might well harp on you for saying that, but that's a different topic.

My complaint is not that you're pointing out that crypto is hard. It absolutely is. My complaint is much narrower than that. It is specifically a complaint about making statements of the form, "You just have to know what X is" when X is not something that is easy to look up. "Carry-propagation bug" is such an X. "Appendectomy" is not.

And all I'm asking for is that you (not just you, but anyone who writes about things like this) add something like, "If you don't know what a carry propagation bug is, here's a pointer..." or "... it's when you're doing bignum math, you have a carry from one digit to the next, and you mess that up. These bugs are particularly nasty because they only manifest themselves very rarely and so are easy to miss in testing." Or something like that.


https://www.google.com/search?client=ubuntu&channel=fs&q=+%2...

I submit that "carry-propagation bug" is easily looked up.

You're asking for people to provide footnotes and extra information so that you can learn what they're talking about when they delve into jargon. I get it. It makes it so much easier to follow along. Then you can learn!

Two points come to mind. First, the bit of jargon you've picked on is in fact easily investigated with tools you are already familiar with. Second, you're asking people writing for an audience of their choice to redefine it to include arbitrarily lay people. Which may not be maximally reasonable - there's a reason Watson and Crick's paper didn't include a primer on what X-rays are.


I clicked your link, and the results weren't useful. They were mostly links to reports about fixes for carry-propagation bugs in various pieces of software (which did not explain what a carry-propagation bug is).

Ironically, the fourth result was a link to a different HN discussion about someone trying to figure out what a carry-propagation bug is, and failing to find any good resources through Google. (Although, to be fair, if you dig into that link a bit more, someone does attempt to explain.)

So I wouldn't say it's impossible to find on one's own, but saying it's "easy" is a bit of a stretch.


It's not especially easy! That's what worries me about laypeople writing crypto libraries. If you know to look, you stand a reasonable chance of learning enough about error oracle side channels to avoid writing something that recapitulates Bleichenbacher's RSA attack. But even if you remember someone telling you to look out for carry propagation bugs, I'm not sure you have a good chance of figuring how how to test for (or exploit) a carry propagation bug. And there are other bugs like this.


> I submit that "carry-propagation bug" is easily looked up.

Well, obviously anything is easily looked up. The problem is that in this case the results aren't very helpful if you don't already know what it is. I even said this in my OP: " if you go search for "carry propagation bug" you will find lots of examples of carry propagation bugs being found and fixed, but no explanation of what one is."


I think it's perfectly normal to be curious about a field and want to know more without necessarily wanting to charge off and do it yourself.


You're right! That's perfectly normal and extremely common.

People are skittish about this in the realm of cryptography because the costs of getting it wrong can be very high. Further, programmers have a culture where experimenting by writing something yourself is perfectly reasonable and extremely common.

When these two meet in the context of crypto, I know just enough to get worried that someone's going to get overconfident from a primer and do something they believe to be very clever. History has a nasty tendency to validate fear this as well-intentioned clever people do things that they believe are good enough.


Decryptocat isn't the lighter end of things. Snowden used it to communicate with Glenn Greenwald's associates during the same time period Decryptocat was found (there are worse flaws in that software than Decryptocat, by the way).

Now you see the issue!


"Table stakes" in competetive Poker means that all you can bet is what's on the table, in front of you.

So this old movie cliché where the protagonist has lost almost all his chips and needs a huge win, so he casually throws in his car keys and the title to his house in the middle of play is not allowed. Neither is grabbing your wallet and throwing cash in.

The idea is that (a) rich people cannot bully others around at will with their huge wealth "in the background" and also (b) players are protected from losing possessions extraneous to the game.


You’re getting confused between two different things with the same name. This use of “table stakes” means something like the minimum buy-in to join the table or the table’s minimum bet to play a hand.

When used as an analogy, the table stakes are just features expected of anyone, not something special to advertise. For example, you wouldn’t say “I’m the best driver because I have a driver’s license and car insurance” – those things are just “table stakes” for drivers.


In this context, though, it means the minimum you need to play the game.


Isn't this the point of the usual advice, "don't write your own crypto"? There are a million gotchas, and most of these you've just gotta know.


Yes, and this is kind of the point I'm trying to make. "Don't write your own crypto" is the wrong message IMHO. The message should be something more like, "Absolutely, write your own crypto, because that's the best way to learn, and here's a pointer to how to get started. Just don't use the results of your early efforts for anything mission critical, and if you want to know why, read this..."

And, taking my own advice, here's the link everyone should be pointing to:

https://cryptopals.com


I have the strongest instinct that "Don't write your own crypto" is one of the few successful barrier-to-entry phrases in the same spirit as something like "Surfing sucks, don't try it".


> Improper carry propagation is an example of the kind of security vulnerability for which there are no test vectors

I beg your pardon?

Some tests vectors from the RFC are designed specifically to handle some overflow that if handled improperly will give you the wrong results. I have checked, this hits the relevant paths. Or were you talking about something else?

By the way, I haven't attempted to implement curve25519 field arithmetic myself, that would have been suicide (compared to curve25519, Poly1305 is a piece of cake). It all comes from ref10, minus the left shift UB it had.

> which this author's previous post claimed as a security feature.

Here's my current view on security: correct software is secure, period. Vulnerable software is incorrect, period.

Which means that anything that helps assert correctness is a security feature. Test vectors, sanitisers, static analysis, Valgrind… are all security tools, and passing any one of them increases confidence in the product, thus making it more secure. Add enough of these, and confidence gets close enough to 100% that we can deem it ready for production.

So, yeah, test vectors are a security feature. Laughably insufficient by themselves, but still necessary to assert the security of a crypto suite.

> Had he asked, I'd further claim that not having dumb C bugs is also table stakes for cryptography

Yeah, sorry about that. I assumed the same, but I thought Valgrind had me covered at the time —oops. Using sanitisers on Monocypher has been quite the eye opener. I didn't realise how dangerous I used to be.

I still have my doubts, by the way. While I would gladly trust Monocypher with my data by now, I don't dare openly recommending it over Libsodium just yet. That may have to wait for a serious external audit.


I feel like I'm reading you again claiming that the test vectors for an algorithm in the standards documents constitutes a battery of security tests for the algorithm. That's not the case; I think 'lisper will tell you the same thing, despite being much more on your side of this issue than I am. Test vectors are an implementation aid and mostly intended for interop.

And yes, I believe I'm talking about a more complicated class of bugs than you are. For instance: OpenSSL's algorithm implementations pass test vectors (OpenSSL is FIPS certified, too). But OpenSSL has had recent carry propagation bugs.

I am not concerned about your use of C, or about the presence of memory corruption bugs in your library. If I was going to harp on about that, I'd also have to go after everyone who's written a Markdown library. My concern is that the kinds of things you say you've done to ensure the security of your cryptography don't address real cryptographic vulnerabilities.

You may have done other things and I just haven't read about them! I don't really care about your library one way or the other; the only thing I care about is the idea that if you pass test vectors and Valgrind is OK with your binary, you're probably OK. No!


I'll look up this carry propagation business, but again, this looks like a mundane correctness issue (with possibly tragic consequences), though it is hard to assess without looking at intermediate results (I have).

Also, didn't the carry propagation bug reports come with failing tests vectors? Or are they hard to find even if one knows of the bug?

> the only thing I care about is the idea that if you pass test vectors and Valgrind is OK with your binary, you're probably OK. No!

Definitely agree. Even test vectors + safe Rust aren't enough. Which is why I added random comparison tests with libsodium (for every possible length input between 0 and a couple times the size of the block), and property based tests.

I'm still a bit uneasy about carry propagation in Poly1305, but I really checked the hell out of my code.


No, they didn't come with failing test vectors. They're implementation specific.

I'm less worried about Poly1305 --- but then: why would you write your own Poly1305 in C? There's already a fast open Poly1305.

My advice is: go do the research about how these bugs are found and exploited, and then declare your library free of them and "production-ready".


> why would you write your own Poly1305 in C?

I originally didn't. But when I noticed it was designed to facilitate 32-bit limbs, I couldn't resist. The result is pretty simple.

I also figured it would perform well, though it currently seems to be 20% slower than Donna (32-bit version). I'll try to find why (I can think of 2 causes: sub-optimal loading code, and crappy data flow in the main carry operation. If it's the latter, I'll use the Donna implementation.)

EDIT: I tweaked the loading code again, it's even faster now. I'm now 7% than poly-donna-32 (and 5% faster than libsodium). My implementation is also simpler. I'm keeping it.

> go do the research about how these bugs are found and exploited,

I'd have to know how.

My first searches turned up no methodology —just instances of such bugs being found. And again, this is just a correctness issue. One just have to prove the whole thing works as intended (even informally). I wouldn't trust myself to do it for curve25519, but poly1305 was doable.


I am terrified that I do not consider myself competent enough to write a crypto library, and yet there isn't a single mention - in this article, nor at the time of writing the comments here on Hacker News - of many of the pitfalls I know to avoid when undertaking such an endeavour. There is even a list of "you have to do A, B, C, and that's about it" that is missing some major - well known, even! - items.

I know "don't roll your own crypto" comes across as dismissive or patronising. I'm not a huge fan of the phrasing. But as a first-order approximation, it is correct. If you want to roll your own crypto, that needs to be your _thing_. It's very unlikely you're going to be a fantastic full-stack web developer _and_ be able to do that. If it's really what you want to do then awesome! Go study it, learn it, practice it. But if you're doing it as a side-hobby, never put it into production.


Could you list the main pitfalls you are thinking about? Also, I may have mentioned some of them in this earlier article: http://loup-vaillant.fr/articles/rolling-your-own-crypto


Hey, like you I decided to dive into cryptography coming from a different background. Although this quote is not directly related to your problem it can be safely applied to it:

"Almost certainly you will get the urge to invent new cryptographic algorithms, and will believe that they are unbreakable. Don't resist the urge; this is one of the fun parts. But resist the belief; almost certainly your creations will be breakable, and almost certainly no one will spend the time breaking them for you. You can break them yourself as you get better."[1]

The problem that Schneier is referring to is that doing a careful analysis (that is required if you want to use your crypto in production) is tedious, time consuming, and requires expertise in the area to know most blank spots of the algorithms (heck a single one is hard enough). That's why he recommends you to be have enough experience breaking many algorithms before you make any serious claim about your the safety of your crypto.

So all in all it's great that you got the interest in the area, there is a lot of work needed in OSS. But be very careful about claiming that your lib is ready for production. What you got is a bunch of volunteers to glance at your code. Few cryptographers (if any) will seriously try to break it.

[1] https://www.schneier.com/crypto-gram/archives/1999/1015.html...


Note that the author is not inventing crypto algorithms, rather implementing them (although the part about XChaCha20 being a mix of ChaCha20 and XSalsa20 is IMHO dancing on the line). Still a risky business, and tricky to get right, but several orders of magnitude safer than "hey, what if we just XORed everything with a random number? Unbreakable, eh?"


> the part about XChaCha20 being a mix of ChaCha20 and XSalsa20 is IMHO dancing on the line

It was a few months ago. Then Libsodium danced on the same line, and I was able to compare the results (they behave the same, phew).

It's not inventing crypto either: it's a straightforward application of what was done to Salsa20. The security reduction that worked for XSalsa20 (guaranteed secure if Salsa20 is secure) applies to XChacha20 as well.


> Then Libsodium danced on the same line, and I was able to compare the results (they behave the same, phew).

Do they really? What about timing attacks? Does all your code run in constant time? If not, there's a whole class of vulnerability that basically will only be found when experienced attackers have a chance to poke at it. Timing attacks are a great example of why "don't roll your own crypto" is sound advice. No amount of testing your library against a reference library will uncover them.


> Do they really?

Absolutely:

> […] However, XChaCha20 is currently not widely implemented outside the libsodium library, due to the absence of formal specification.

https://download.libsodium.org/doc/advanced/xchacha20.html

Can't say for sure, but it look like they didn't have an independent implementation to compare to. Just like me a few months a go.

---

> What about timing attacks?

Monocypher is immune.

First, the primitive were chosen precisely because they are easily immunised against timing attacks.

Second, this is easily checked simply by looking at the source code: no secret-dependent branches, no secret-dependent indices, and that's about it. One doesn't need to be an expert to check that, a junior programmer could do it after being told what to look for.


What's wrong with xoring with a stream of random numbers? Isn't it how stream ciphers work? Get a good cryptographic RNG, initialize it properly with a long enough key and you should be fine.


You're right about the theory, but you're also wrong about how to implement - which is the point of telling people not to roll their own crypto.

What about reseeding or prediction resistance? How do you handle biased entropy input from that hairdryer someone is blowing on your chips? Oops, the quantum random number generator card doesn't work anymore after adding that extra GPGPU to the system.


You wouldn't want true random numbers anyways. If you're encrypting it, then you'll want to decrypt it later. That means you'll need the random data you used again. If you had a secure channel over which to send or store the random data, then you could just use that channel to send or store the plaintext itself (note that the random data will be the same length as the plaintext). Such a system could only be useful if the cryptographic key was shorter than the plaintext. So it would have to be a deterministic RNG, and you would only need the seed and the ciphertext in order to decrypt.

Also, if someone has acccess to the hardware, then you're screwed. No one would bother wasting time with a hairdryer. If the plaintext was on your harddrive, they would just grab that and run.


I think you missed the point of the comment, which wasn't about physical possession of a computer.

You also make assumptions, such as chips being inside the computer, when they can in fact be in an outside environment-based RNG. The hairdryer example is the canonical and prototypical weakness in that particular type of "rng". I'll try to be explicit about that in the future.


I didn't write anything about environment-based RNG. RNG must be deterministic and seeded by preshared encryption key. One point of using a cryptographically secure RNG is that it should not be possible to learn the seed by observing the output of the RNG without bruteforcing the key.


> How do you handle biased entropy input from that hairdryer someone is blowing on your chips?

Sounds fascinating. Can you please tell us more?


If you had a RNG that sampled some hardware noise but then you altered the environment by introducing extreme temperatures, maybe the RNG would end up with biased output. A simple example: you can use a camera as a good source of randomness, even taking a picture of a wall indoors. But if an attacker was able to flood that room with light and overwhelm your sensor, he could predict the image would be 100% white pixels or near to it and guess the random output.

But the comment is exaggerating. Encryption algorithms do not handle the case of your entropy source being compromised. Good RNG algorithms like Yarrow try to mitigate this somewhat.


Even if you need a real entropy-based RNG, e.g. for key generation, you need to gather entropy once at the beginning to seed the RNG. Then running RNG does not need any more entropy, so your hairdryer trick won't work. A common myth I hear is that "entropy" gets "depleted" during RNG operation, but in fact it does not. You only need enough bits of entropy once at the beginning, equal to the internal RNG state length.


Strong RNG designs re-seed to deal with compromise or poor seeding. That way if an attacker has temporary read access they do not permanently compromise the system going forward. Or if on boot you are in a compromised or low entropy environment the system will improve over time. Perhaps the utility of this is arguable. If an attacker is able to see your private RNG state then it is most likely game over. But it is a cheap operation with little downside and again, designs like Yarrow put considerable time into thinking it through.


Well, to be fair, a good implementation should throw some simple statistical sanity checks at RNG input during self-test and throw an error when it doesn't pass. Passing such tests doesn't indicate that the RNG is cryptographically secure, but failing them indicates that it is insecure.


Thank you, that's my point exactly.

Nothing wrong with what you wrote - but note the subtle difference between "XORing a stream of random numbers" (potentially viable, even unbreakable when using a OTP) and "XORing a random number" (essentially a Caesar cipher, kid-sister encryption).


> note the subtle difference between "XORing a stream of random numbers" (potentially viable, even unbreakable when using a OTP) and "XORing a random number" (essentially a Caesar cipher, kid-sister encryption)

No, that's not a difference. A "stream of numbers" is just one very large number. What you're worrying about is how large the numbers are, not whether there's one or more than one.


Original point was certainly to point out the most common mistake of typical homegrown cryptosystems. Even programmers who know that XORing stuff with constant byte is not viable encryption can produce systems that do essentially the same thing (usually by XORing everything with RC4 output with long lived or even constant key)


There's nothing wrong with that. You are right, that's how stream ciphers work.


>XChaCha20 being a mix of ChaCha20 and XSalsa20 is IMHO dancing on the line

No pun intended?


No pun intended, but it would have been a pretty good pun, had I noticed it before.


And instead of saying nothing those who harp on "C's pitfalls" could politely enumerate those relevant to the issue at hand, that way we actually have something discrete to discuss.


I apologize if that sounds dismissive (it's not intended to) but your post is not more than the usual vague FUD. Take a look at so-called 'professional' cryptographers and their libraries, and you'll find that practically all of their code had severe bugs. Professionals make errors like everyone else, 'professional' just means you're being paid for it. You will have a hard time finding a professional and widely used crypto library that wasn't essentially compromised in one of its functions at one time or another. Not only that, professionals from the closed-source department have a long history of coming up with compromised or bogus, sometimes even ridiculous cryptographic algorithms and implementations. Two typical examples: CMEA cell phone encryption, and the Crypto AG backdoor debacle.

Surely people who invent and implement cryptographic algorithms for a living can be expected to be better than your run-off-the-mill programmer, but as I've said elsewhere implementing crypto is also not a black art. It requires about the same level of skill as e.g. implementing a production-ready low-level network protocol or a file system. That rules out many programmers but not all. That concerns implementations. As for algorithms, their development requires extensive experience in cryptanalysis (not just applied cryptography!). Some pros have that, others don't.

Last but not least, many popular and widely used crypto libraries started as a side hobby. For example, libtomcrypt started that way. In fact, many widely respected cryptographers started as hobbyists. For example, Bruce Schneier. Of course, the ones who are widely accepted as peers are also those who are in the 'hire me as a consultant, not the other guy' business, and they will naturally advise everyone to not roll their own crypto but to rely on their expertise...


As I said I'm not a fan of the phrase "don't roll your own crypto". I don't disagree that that (obviously!) prevents new crypto libraries being written. But if you're going to do it, you damn well do it right, and these days that means an amount of work that would likely total millions of dollars. If you aren't going to do it right, don't cut a 1.0 and use it in production.

Sidenote: how we started these things historically is not relevant anymore. We have learned a lot since then. Failing to "stand on the shoulders of giants" and learn from history is one the biggest factors that makes home-rolled crypto so easy to break - they are doomed to repeat the mistakes of history. Mistakes that are nowadays well documented and well known.


Having implemented crypto algorithms for educational and enjoyment purposes, I've always looked at "don't roll your own crypto" as less having to do with the actual writing of code which, as you state, isn't really that hard. Instead, what that statement is saying is that when you write your own code, you'll almost never get enough attention from the white-hat community to have any confidence that the code can be used in production.

You're right that the code written by professionals has lots of bugs. Bug is a term that describes a problem in code that has been found. With crypto software, the danger isn't the problems that get found (and then addressed), it's the problems that aren't found. "Roll your own" solutions have problems that never get found. That's the danger and it has nothing to do with the skill of the programmer.


The thing that normally nobody says, yet most the people telling you to not roll your own crypto assume you will understand is that you should rely only on well vetted libraries.

Rolling a usable crypto library is not a single person endeavor. Your own crypto won't be peer reviewed, thus you should not trust it.


So I have two points for you:

> Take a look at so-called 'professional' cryptographers and their libraries, and you'll find that practically all of their code had severe bugs. Professionals make errors like everyone else, 'professional' just means you're being paid for it. You will have a hard time finding a professional and widely used crypto library that wasn't essentially compromised in one of its functions at one time or another. Not only that, professionals from the closed-source department have a long history of coming up with compromised or bogus, sometimes even ridiculous cryptographic algorithms and implementations. Two typical examples: CMEA cell phone encryption, and the Crypto AG backdoor debacle.

This doesn't diminish the claim that the modal individual should not attempt to "roll their own crypto." You're pointing out that professionals make mistakes, but of course they do - that indicates nothing about the propensity for amateurs to make mistakes.

> It requires about the same level of skill as e.g. implementing a production-ready low-level network protocol or a file system.

I have never implemented a low level network protocol or a file system (although at least the former sounds like a fun project) - so I cannot claim you're wrong here. However, I will point out that a network protocol is not designed to be error-proof in an intentionally antagonistic environment, which is a difficulty unique to cryptographic software. There are incentives involved in breaking crypto code that do not present themselves in other areas of software development, even if the programming difficulty itself is not entirely dissimilar.

> Last but not least, many popular and widely used crypto libraries started as a side hobby. For example, libtomcrypt started that way. In fact, many widely respected cryptographers started as hobbyists. For example, Bruce Schneier. Of course, the ones who are widely accepted as peers are also those who are in the 'hire me as a consultant, not the other guy' business, and they will naturally advise everyone to not roll their own crypto but to rely on their expertise...

They can start as side hobbies, but they are not going to remain that way for any meaningful amount of time if they are to be widely deployed and safe. You can only choose two out of those three.

More importantly, there is no cabal of cryptographers spreading FUD to line their own pockets with consulting fees. The cryptography consulting industry is very small compared to the actual security consulting industry. In the application security consulting industry I do believe there are firms that try to secure business this way, but that industry is much larger (and has firms which try to misrepresent cryptographic competency). I have had significant interactions with engineers at Riscure and NCC Crypto, which is a sizable portion of all the "real" cryptanalytic consulting work that occurs (at least in the United States) - they never struck me as being the sort to suggest what you're saying. That leaves Rambus/CR and a select few other firms doing real crypto consulting, which leads me to believe the industry is, on the whole, very legitimate.


> a network protocol is not designed to be error-proof in an intentionally antagonistic environment

Err, really? That may be the case sometimes. Also these days many things can be antagonistic: anything online is exposed to adversaries. So is anything that routinely reads untrusted input on everyone's computer (like a video player, or a pdf reader).

Thinking of the sheer amount of potentially vulnerable code out there makes me a little scared.


But is the code you're speaking of the security-specific code, or the network-specific code? If you attack a networking protocol, what are you attacking if not the associated cryptography? A networking protocol without any cryptography doesn't really need to be attacked because it's already wide open.

You introduce the confidentiality, authentication and integrity (which is really just going to be part of the authentication) through cryptography and security-specific code.

I'm not saying people don't attack networking protocols, or that there is no incentive for doing so. I'm saying that the portion that plays the defensive role is in the security software, not the overlying networking protocol.


re networking vs security-critical code

Anything that accepts data from potentially-malicious sources is security-critical if it's itself privileged or produces data/events that go to something else privileged. Such analysis is usually used to determine the "Trusted Computing Base" (TCB) of a system that encompasses everything that might be used to break the security policy. All people who do real, security engineering keep it in mind, minimize it, strengthen it, and document what's left.

Here's an example from high-assurance field of what that looks like where they prototype an architecture for VPN's that minimizes TCB of key components:

https://os.inf.tu-dresden.de/mikrosina/dach2005.pdf

Far as qualifying the TCB, the seL4 team's What is Proved and What is Assumed is a great illustration of keeping it real about the security claims:

https://sel4.systems/Info/FAQ/proof.pml

Also shows their TCB is a lot bigger than their tiny kernel. Fortunately, there's been high-assurance developments to solve most of those things. I've seen robust, networking stacks but don't think they're high-assurance yet. If they're complex and done in C, here be dragons.


I was thinking of attacking the implementation of whatever processes attacker controlled input. If sending 100 pings in a row triggers a stack overflow, this will interest the attacker.

I would say that security specific code would be any code that directly processes the untrusted inputs. Parsers and decoders, mostly. You may want to isolate them from the rest of the program (ideally a separate address space), and make sure the interface between them and the rest of the program is much simpler than the input they are processing.

At least, you want to ensure such code either produces sound output for the rest of the program or fails cleanly. The attacker will have a harder time exploiting a bug deeper in to the program, I think —hope.


While I actually mostly agree with you, I feel I need to point this out... You write: > I will point out that a network protocol is not designed to be error-proof in an intentionally antagonistic environment, which is a difficulty unique to cryptographic software.

But that's actually not true, at least not anymore. People attack protocols on the internet all the time, and it's a difficult task to harden them against that.

That said, it doesn't really take aware from your point, I think; but rather confirms it: Think twice before rolling your own network protocols (at least if they are meant to be used in large scale on the open internet), be aware of all the complicated pitfalls.


> This doesn't diminish the claim that the modal individual should not attempt to "roll their own crypto."

Just wanted to point out that "individual-should-not-attempt-to-roll-their-own-crypto" is ambiguous and unnecessarily antagonistic while the paragraph quoted above from Schneier is clear and difficult to argue with.

"As simple as possible but no simpler" applies quite well to summaries, esp. when they regard cryptography.


Okay, what exactly is the claim you'd like me to argue with? That Bruce Schneier was a hobbyist when he developed any of his cryptographic algorithms? He wasn't, and more importantly that misses the point since those algorithms were widely reviewed in competition settings.

I'm not sure what you're getting at here. Yes, someone has to develop cryptography, just like someone has to work on designing rockets. Theoretically everyone begins an aspiration as a hobby, but that's pedantic. A cryptographic design or implementation is not going to survive as a side hobby regardless of how it begins.


There is even a list of "you have to do A, B, C, and that's about it" that is missing some major - well known, even! - items.

Our "field" of programming is so weak in its preservation of institutional knowledge, that we have to occasionally fight the recurrence of bad ideas about very basic things, like using string filtering to avoid SQL injection. Yes, this happens, even in "hip" new parts of the field, like the communities around newer languages. (I have a specific example in mind, but I don't want to deal with the resulting war.)


I've found some helpful guidelines, but I know it is by no means an exhaustive list.

https://cryptocoding.net/index.php/Coding_rules


Definition question: would you say that AgileBits (1password) are "rolling their own crypto" or 'merely' implementing proven third-party crypto?


I would have to read their source code to answer that question (I know almost nothing about 1Password). Some password managers just call a PGP executable. Others are assembling crypto primitives into larger pieces and making choices like "let's use CBC mode" themselves.


Anyone who can write code can write a crypto library.

The trick is to have a second person helping you, who has all your mathematics notes, a copy of your source code, physical access to your hardware, and the ability to run your program inside a debugger. You don't even get to know whether they put cream in their coffee.

If you can keep a secret from them, you're doing okay. So then you rent a smarter person to do the same thing to make sure all the tricks and shortcuts have been attempted.

If someone manages to break it in the wild after that, you're still doing about as well as some of the highest-paid professionals in the industry.

Defense is difficult.


I generally also think that it's a bad idea to do, but I feel like it's a discussion that needs to be had with governments around the world wanting to backdoor everything.

It means that normal citizens won't be completely helpless against tyrannic governments and it helps to educate that writing crypto from scratch is most definitely something that criminals can do. The government making it illegal or backdooring it will only help against lowest-effort criminals, not against organized terrorist groups. Sure, those probably won't write unbreakable encryption, but it'll be enough to bypass a government that's expecting everything to be in plain text.


> but I feel like it's a discussion that needs to be had with governments around the world wanting to backdoor everything.

On the contrary, this is -exactly- why you don't want to roll your own crypto. Because they can and will break it. Good crypto takes serious thought and effort from people who put a lot of thought and effort into it.


You can do crypto as a side-hobby and safely put it into production. Of course, you can also do it wrong, but it is with little work possible to do it correctly.

Normally I wouldn't speak against what you said because the critique is only very minor, but I heard this too often. Your warning is too strong. People should try their own crypto and with little care it is not unsafer. Maybe giving a list of do's for that would be a good start.


> You can do crypto as a side-hobby and safely put it into production.

No, you really can't, unless you stretch the definition of "side-hobby" to the point of breaking at the seams. I'd be shocked if a single person who professionally works in security or cryptography is going to agree with you in this thread. You need far more than a "little care" to assure that a new cryptographic library is safe. It's an endeavor suited to a collaborative academic or corporate environment, with robust human capital and strong oversight.

I'm willing to accept that someone can safely implement a primitive or a construction in a new library with significant time, effort and feedback from others. But that level of effort doesn't really qualify as a side-hobby, and until the implementation was formally checked by professional cryptographers I'd consider it suspect.

This is to say nothing of designing novel primitives or constructions, which I would consider as far removed from a "side-hobby" as a local 5K is to the Olympics.

However, I absolutely think people should attempt to implement their own cryptography if they're curious about it and want to learn. Just don't use it in production and assume it's unsafe. People who work in the industry don't make these claims because we think it's fun, we do it because we've all seen the consequences.


We are talking in vague terms, but assume you want to encrypt some offline data.

- You can just use any recommended (by whom?) tool.

- Or you encrypt it with recommended tool with it's own dedicated key. Then you encrypt it again by whatever way you made up in five minutes, and you need only to ensure you didn't use the key material of the recommended tool. Then there you go, you just rolled your own not-unsafer crypto.


That is not rolling your own crypto. Rolling your own would be relying on your custom construction, not double encrypting and relying on the known construction for safety.

...Furthermore I'm confused as to why someone would do that. If you already have a safe way to encrypt your plaintext and you're the only one with both keys, why would you bother double encrypting? You gain absolutely nothing because it's redundant, and you lose performance.

What you're doing is tantamount to a weird sort of encoding that doesn't offer any benefit.


If a vulnerability is found for the outer encryption method, additional encryption of the inner message by a different method may provide some defence.

If that is the goal, though, it would be better to use two well studied encryption methods rather than something homemade.


Right, that observation is correct on the surface. But the reason why that's almost never done is because the goal of a cryptographic algorithm is to contribute enough safety margin on its own. Instead of encrypting twice, it's better to encrypt with a greater number of rounds, or to come up with a superior algorithm. In practice you sacrifice an unreasonable amount of performance double encrypting in a production environment for a threat model that is fantastically unrealistic.


You find it fantastically unrealistic that a vulnerability could be found for a well known and widely used encryption method?


Depending on the algorithm: yes. I'm not saying all algorihms that are widely used and well known are extremely unlikely to be broken.

I would happily bet $10,000 that AES will not be broken in the next ten years. I'd make the same bet for several hash functions.

And as a corollary to this point: I think it's incredibly foolish to try and combat the threat of an encryption algorithm being broken by double encrypting a plaintext, including with that algorithm.


I wouldn't take that bet either as I agree that is a very unlikely thing to happen.

Note, though, that you are using the specific word 'algorithm', where as I am talking about 'methods'. Most cryptographic failures are in how the algorithms are implemented or applied, not a problem in the underlying maths.

I would not be nearly so confident about a similar bet that applied to the actual code being widely used for encryption.


> I would happily bet $10,000 that AES will not be broken in the next ten years.

What are you talking about: http://cr.yp.to/antiforgery/cachetiming-20050414.pdf

You'd lose the bet so hard. Don't take anything for granted. Even the most supposedly secure and widely used primitives should be scrutinized and are subjects to constant attacks.


> What are you talking about: http://cr.yp.to/antiforgery/cachetiming-20050414.pdf

Read that paper again. Cache timing attacks (and more generally timing attacks) are a subset of side channel attacks, which are not a break in the fundamental design of a cryptographic algorithm. When Bernstein mentions that he considers it a design flaw, that's a misnomer in the mathematical sense: he means that AES is antagonistic to a software implementation that can resist a timing attack, not that the design of the algorithm is fundamentally unsound. This is essentially true for all encryption schemes that utilize lookup tables and S-boxes, which he even mentions in the paper.

This is similar to using acoustic analysis or fault injection for key retrieval - yes, you've done it, but it's not really fair to say you've broken AES whatsoever. That remains a pipe dream. You've merely broken an AES implementation on a particular platform that was not hardened for this threat. Just a few months ago a team came up with a way to retrieve an AES key through acoustic analysis in a paper that was published here on HN. That's not actually the same as a cryptanalysis, the best of which is this: https://eprint.iacr.org/2009/317

There are effective CPU mitigations that allow you to safely implement algorithms in order to avoid this issue - for example, AES-NI theoretically mitigates cache timing attacks. See: http://cseweb.ucsd.edu/~hovav/dist/aes_cache.pdf

So: no, I wouldn't "lose that bet so hard."


I know this is a side channel attack. I know people have used acoustic analysis to retrieve 4096 bit RSA keys, but the attack outlined in the paper is more feasible in real life situations, and systems do get compromised due to implementation details. AES might not be broken as a whole but that doesn't mean you can't attack specific usages.

AES isn't proven to be secure. I don't know about $10k, but I would not bet my life on any hash function not being broken.


That's not an (algorithmic) break, but a (side-channel) attack on some implementation. "Break" is precise jargon here; you're right that a particular real-world implementation of a strong algorithm may be weak, but that doesn't contradict anything dsacco said.

Also, please be nice.


Doing your own encryption does clearly not exclude also using other encryption. Or is implementing AES already not anymore doing your own encryption, because AES wasn't invented by you? You _always_ use work of others.

Doing your own crypto can be used as way to safeguard against unknown vulnerabilities in the tools you would use otherwise. If the recommended library is safe, then fine, you are good to go. How do we know it is safe?

My idea above is a way to increase the safety margin in a way that is orthogonal to the increase of the key space of the recommended tool. That sounds like a very good thing to me.


I'm sorry, I don't understand what you're trying to say. Can you clarify?

What I am trying to say is that you gain nothing by, say, implementing your own encryption algorithm, using it on a plaintext, then encrypting the resulting ciphertext with a known good implementation of a known good algorithm like AES. That is how I interpreted your previous comment.

So - you don't gain anything from that, if that's what you meant. If you mean just implementing a known good algorithm on your own (again, like AES) - why? You are overwhelmingly more likely to shoot yourself in the foot, and there are plenty of open source implementations that you can inspect if you don't trust the code.

Finally, if you distrust the design of a known good algorithm, you're out of luck, because (again) you're almost certainly not going to come up with something better. If you do, by all means use it, but it's simply not likely whatsoever.

And again: you're not increasing the safety margin by double encrypting, unless you know both algorithms and implementations are solid. And even if they both are, the safety margin improvement will be negligible compared to the decrease in performance.


> "If you mean just implementing a known good algorithm on your own (again, like AES) - why? You are overwhelmingly more likely to shoot yourself in the foot"

I've never understood this argument. Encryption algorithms are deterministic. If you implement a known good algorithm on your own, and it gets the same outputs given the same inputs, then you are highly unlikely to shoot yourself in the foot. This is not what people are talking about when they warn not to roll your own crypto.


Just because an AES implementation matches the test vectors does not make it correct or safe.

Here is an excellent visual depiction of several different AES implementations, many of which do not run in constant time:

https://cr.yp.to/mac/variability1.html

Implementing AES with variable time secret-dependent operations can leak the key.

There are much bigger and more terrifying subtleties to implementing crypto correctly than simply matching the test vectors.


To briefly expand on this (good) comment:

In cryptography, the output of an algorithm is far less important than how that output was generated (with very precise specificity). Crypto code requires far more heavy lifting in the "how" of the output than other software does. This is not intuitive, because ordinarily there are several ways to do things and you can compare on output, and it's probably fine to develop something in a functional but unorthodox way (minus performance consequences, maybe).

As the parent comment here mentions, there are extremely bad consequences to subtle mistakes in how the algorithm and its implementation arrives at the output, even if the output is correct.

If Alice sends Bob an encrypted message and the custom crypto provides an incorrect output, Bob can't decrypt the message. But if Alice sends Bob an encrypted message with the correct output generated in an unsafe way, there are many ways by which the secret key could theoretically be recovered, which is far worse than Bob not being able to read the message in the first place.


> "Just because an AES implementation matches the test vectors does not make it correct or safe."

Actually, this does make it correct. Whether or not it is safe depends on the application. For example, I sent my friend Bob some cyphertext that I calculated by hand with a pencil and paper using the AES algorithm. I sent it via my trusted courier Eve. It took four hours for me to do the calculation. At the last minute I started to second guess my math so I double checked it against a respected crypto library. It was fine, so I handed it off to Eve. I am pretty sure that in this application my choice to calculate the answer by hand was exactly as safe as if I had just used the library. In fact, I am so sure (given that the answers were identical), that just as Eve was walking out the door, enroute to Bob, I pulled her close and whispered in her ear: "Eve, be very careful with this cyphertext, it took me four hours to create it."



...what? Yes, if you compute by hand and compare with a known implementation, than it's likely you computation is correct. But this has nothing to do with test vectors. You could match all test vectors while still giving incorrect results for values not in the test vectors.


And how do matching test vectors ensure there are no timing attacks? Or memory leaks that disclose private keys?


But there's more to cryptography than just inputs and outputs.

Like, let's say we have two functions which validate a password.

Function A takes a user-defined parameter, compares it to a stored value byte by byte, and as soon as the strings differ, it exits.

Function B takes a user-defined parameter, compares it to a stored value byte by byte, sets a flag if the strings differ, and exits when all bytes have been compared.

Both of those functions take the same input, and both of them return the same output. Are you prepared to tell me that the two functions are equally secure?


> Encryption algorithms are deterministic. If you implement a known good algorithm on your own, and it gets the same outputs given the same inputs, then you are highly unlikely to shoot yourself in the foot. This is not what people are talking about when they warn not to roll your own crypto.

That is exactly what people mean by "don't roll your own crypto." If your belief is that you only need to match the same output per input, then your implementation might be an oracle that leaks information.

Crypto people don't say this for job security or to feel important. They say it, because they, themselves, have made mistakes, and they constantly see examples of programs that trivialize both security and cryptography while simultaneously introducing foreseeable bugs and obviously weak crypto.


They are deterministic, but timing attacks are a big deal. There were lots of practical key extraction attacks on RSA with nothing but timing, even on a network.

You can also do things like forget input validation in ways that reveal key material, or reuse a nonce or IV in a way that breaks the whole system.


That's far from what I'd call "doing crypto as a side-hobby".


Sorry, but this is not good advice. Widely-used crypto implementations have had the benefit of years of analysis by dozens of high-expertise stakeholders who have a lot to lose should the crypto fail. Even that isn't always enough to catch all weaknesses and vulnerabilities.

This cowboy-programming attitude being extended to security is no small part of why we are so vulnerable as a society to attacks on our computer systems that can compromise our core infrastructure[1], our secrets[2], and our economic security[3].

[1] Think weeks-long country-wide power outages

[2] Think juicy blackmail material on anyone with their hands on levers of power. Look at the damage that North Korea's attacks did to Sony Pictures, for example.

[3] Think small-scale attacks on services that even a single major company, or many small companies, rely on.


You are wrongly assuming I say to throw away the work of others. I never said so.


> Of course, you can also do it wrong, but it is with little work possible to do it correctly.

GCHQ loves people who think this.

Have a look at some of the game console cryptography. These are companies with strong financial incentive to get it right. They're large well funded multinational organisations. They still get it wrong.


Have any studies been done on relative security of crypto algorithms that are either:

a) Well known, well studied but also attractive targets for attackers to study

b) Unknown (aside from the developer) until an attacker encounters a specific piece of encrypted data

It is a common assumption that well-known methods are better (and it is the assumption I work under) but does empirical data on security breaches back that up? There are plenty of examples of security breaches where 'standard' methods were being used. Are there similar examples where people using previously unknown methods have been compromised?

GCHQ and others invest a huge amount of resources in finding vulnerabilities in well known encryption methods. When they find one, everyone who used that method is vulnerable.

I have no doubt that if they really wanted a piece of data that I had encrypted with a homemade method, they would be able to break it.

However, are they going to invest the resources to do that if I am not being specifically targeted? Are they going to invest the resources to crack hundreds of different people's home-made encryption methods? Thousands? Hundreds of thousands?

If am being specifically targeted by something like GCHQ, they will get what they want one way or another.


I'm not aware of academic studies on the subject (I think that would be hard to do, because it's virtually impossible to know how many proprietary algorithms exist, how many are trivially breakable and how many have been broken).

However, this point is featured as 101 material in basically every cryptographic textbook. To put it very succinctly: there are conditions in which it can be beneficial to use proprietary cryptography, especially when you require very unique interoperability constraints. However it is almost never a benefit for the safety of the algorithm.

I've come across a proprietary algorithm and successfully broken it, in a black box setting, with differential cryptanalysis. This algorithm was deployed to disguise the sequential order numbers for a very large delivery company. It took me about a month, but it was done. The challenge in proprietary algorithms is shifted to figuring out what's going on because it's unrecognizable. That is a significantly easier challenge that identifying a vulnerability in an algorithm like AES, which has never had a meaningful vulnerability in a decade and a half of cryptanalysis.

If you use a proprietary algorithm it might be safer than a known unsafe open algorithm, but it's virtually guaranteed to be worse than widely studied algorithms, and most likely in a trivially breakable way. They can be safe, but that still means you're going to be working with professional cryptographers at a company like Riscure to assure it's safe.


I am aware of the usual (and strong in my opinion) argument.

Every time this discussion does the rounds, though, I do wonder whether the hypothesis could be tested.

Most vulnerabilities do not come from breaking the core algorithm but rather from a flaw in how they are implemented or applied. Standardisation can lead to monocultures that become tempting targets for those with plenty of resources to throw at them.


> I do wonder whether the hypothesis could be tested

Data point: everyone who evaluates crypto constructions says not to roll your own.


That's what leads me to being reasonably sure the hypothesis is valid.

As a scientist, though, I'm always going to wonder whether there is a way to subject it to a proper test rather than just relying on opinion (no matter how much I respect those opinions)


Game companies are a great example because:

1) they invest 6 or 7 figures in this (piracy prevention is big money)

2) they still get it wrong

3) they not only get it wrong, they get in wrong in ways that teenagers still in high school can break, let alone the NSA


Game companies are also a great example of a situation where a large group of people with time on their hands (teenagers still in high school) are motivated to look for vulnerabilities in the method used.


The fact that piracy prevention can not be solved does not help them.


Who says they are trying to make it right?

They just need to make it hard enough to delay pirated games.


Just look at the major used crypto libraries and their CVE's. Everybody gets it wrong. I consider that a sane and sound assumption.

But then you can combine stuff as I've for example written in another comment to roll your own crypto in five minutes, which is not-unsafer.

There is cult around "Don't roll your own crypto" that is thought-policing.


It's not thought policing. Would you consider us adamantly telling you not to develop and deploy your own rockets or medical software without significant expertise and third party review to be "thought policing"?

Furthermore, your argument is fallacious. That expert professionals make mistakes does not tell you anything about the likelihood of an amateur to make a mistake. You can't draw any logical conclusion from that statement on its own.


If you're telling me I can't develop and deploy my own rockets for fun (providing I don't cause risk to others) then yes, that is thought policing.

(btw, I do develop and deploy my own rockets for fun)


That's great, but that's not what I'm telling you. I'm telling you not to develop and deploy them with human passengers, in a production environment, on a trip to orbit. If you get collaboration from professionals with significant expertise and audit the design and development extensively, then sure, go for it.

Even were that as accessible as developing and deploying your own cryptography, I'd still advise against it.


If you roll your own crypto and say, "that was a fun learning experience" and write a blog post about what you learned implementing AES in some novel way, nobody is going to show up and tell you not to roll your own crypto. If you roll your own crypto and then deploy it in a production system, then you are putting actual users at risk.


> There is cult around "Don't roll your own crypto" that is thought-policing

You know who else gets "thought policed"? The people who actually need to rely on cryptography. In their case it isn't grumpy people on Internet message boards telling them not to do something, it's agents of the state killing them or imprisoning them for many years.


It was a fun read, except when all the C bugs were listed. It's getting tiresome to hear about folks writing nominally Important software in a language where it is extraordinarily difficult to get things absolutely correct, with paltry excuses such as "it needs to be fast" or "it needs to be portable [0] to a VAX-11/750." It doesn't give me confidence that these completely usual bugs popped up early on, and it will not surprise me when more show themselves.

The author's lack of practical familiarity with the memory hierarchy, caching, and pipelining—evidenced by his surprise by the speed gained by fetching more than a byte at a time—also makes me suspicious of his understanding of the semantics of his program. (I understand that this is a form of ad hominem as it relates to arguing the correctness of software, but it's a consideration worth noting in a broader context of trust.)

It would be great if this was written in another language.

[0] Most "portable" code written in C is not portable. Look at any mature C project and see the layers and layers of macros and hacks to make things portable.


> The author's lack of practical familiarity with the memory hierarchy, caching, and pipelining

Maybe I didn't express myself clearly. I knew loading stuff word by word would be much faster, thanks to my knowledge of the memory hierarchy, caching, and pipelining.

What I didn't expect was the magnitude of the overall impact. The core round functions are much, much faster that I had anticipated. I didn't think the loading code, even in its slow version, would account for a significant percentage of the total running time.

> It would be great if this was written in another language.

Easier said than done. I aim for easy integration in projects written in basically anything: C, C++, OCaml, Rust, Python… What would you suggest? Rust? That's a possibility, but (i) it's LLVM only at the moment, and (ii) I believe there's an impedance mismatch between Rust and a C compatible ABI.

I would totally recommend to rewrite the whole thing in Rust for Rust projects, though.


One of the major Rust design goals is simple C ABI compatibility, and it delivers there. That it's LLVM only appears to me irrelevant, as the generated libraries can be linked anywhere, and the specifics of how the Rust compiler are built are out of scope. So long as you get rustc up and running, what does it matter it's architecture?

If you're curious, check this out: https://doc.rust-lang.org/1.5.0/book/ffi.html

> I would totally recommend to rewrite the whole thing in Rust for Rust projects, though.

Any time you rewrite anything, you run the risk of introducing new and unexpected bugs, and you're just doing unnecessary, complex work fraught with pitfalls (doubly so here) -- generally a Rust project would create a wrapper around the C code, invoke the methods and expose idiomatic APIs on top to avoid just that.

This is my go-to article on re-writing things and why not to: https://www.joelonsoftware.com/2000/04/06/things-you-should-...

To be clear I'm not saying you made a mis-step, just that your arguments for it don't seem at first glance to support your decision.


> One of the major Rust design goals is simple C ABI compatibility, and it delivers there.

I was just telling one has to write a wrapper either way. Idiomatic Rust is not idiomatic C, so you would need to have a "C ABI" interface in addition to the clean Rust interface.

> That it's LLVM only appears to me irrelevant […]

Not if it targets less platforms than GCC. Though that will likely matter less and less in the future.

> So long as you get rustc up and running, what does it matter it's architecture?

You have to get it up and running. You have another build step. You have to use a binary library (can't just include the source code). It's not much, but I want to maximise ease of deployment.


It's even more tiresome hearing a new generation of programmers constantly lamenting the fact that C requires a great deal of skill to develop in and that X or Y new language completely avoids C pitfalls. Not only is it dishonest but it seems to be marketing 70% of the time.

Portability does not equal 'without blemishes'. It simply means that when compiled across platforms it will perform as expected.


I like to think that I'm rather neutral, since I'm neither pro-C or against-C.

But from a rather philosophical point of view, C is basically Portability 1.0, it was one of the first truly portable "fast" languages. I'd like to think that we could do better than C with 40 years or CS research and industry experience.

Of course, there's the whole second-system syndrome so it's a wicked problem :)


> that X or Y new language completely avoids C pitfalls

Yeah, you don't even need a new, hip language to avoid C's pitfalls.

> Portability does not equal 'without blemishes'. It simply means that when compiled across platforms it will perform as expected.

By that measure, C is not portable, since it will behave differently across multiple platforms – or multiple compilers, multiple compiler versions, even multiple optimization settings.


Selection and context are _so_ important, don't you think?


The quotes I selected do not change the context, nor do they feign a different context. Unfortunately, since you replied with nothing but a snarky comment, I do not know in what way you think I misrepresented your arguments.


> Look at any mature C project and see the layers and layers of macros and hacks to make things portable.

So.. it's portable, then?


We've had to wait till C99 to get stdint.h, clearly portability is kind of an afterthought in C.

C is meant to map easily to a wide range of hardware without overhead, conceptually it's almost the opposite of portability since it precludes creating standard abstractions that would hold true across architectures as those could not map to native functionality across the board.

It's easy to compile C programs on a different architecture however, which is probably where this argument stems from. Remember when we switched from 32 to 64bits? How many codebases managed to make the jump painlessly?


I can't edit this comment anymore but I'm replying to myself to point out this little bit of C "trivia": the printf function itself, probably the most iconic standard function of all, is not portable.

The problem is that it returns an "int" which contains a negative value on errors and the number of character written on success.

Of course that means that it's possible to write a string whose size doesn't fit in an int.

For instance if you have a 64bit computer (and enough RAM) you can try the following code:

    #include <stdio.h>
    #include <stdlib.h>
    #include <assert.h>
    #include <string.h>

    int main(void) {
      size_t len = 0x80000001UL;
      char *str = malloc(len);

      assert(str);

      memset(str, 'a', len);
    
      str[len - 1] = '\0';
    
      fprintf(stderr, "\n%d\n", printf(str));

      return 0;
    }
Don't forget to redirect stdout to /dev/null. You can pipe through `wc -c` if you want to make sure the billion 'a's are printed.

On my computer the "printf" manages to print the whole string successfully but it returns... -1, signaling an error.

The size shouldn't be returned that way, it should be a "size_t" as an output parameter (since you also need a special value for errors). In Rust you'd use "Result<usize>" and you'd have a more elegant solution. Of course you can't really do that in C (except by returning a struct, which is a bit cumbersome) so I guess they said "meh, who prints that much text anyway?" and assumed an int would be good enough.


> We've had to wait till C99 to get stdint.h, clearly portability is kind of an afterthought in C.

C provided "at least N bits" guarantees for its integer types since its first standard (for the curious: char at least 8-bit, short at least 16-bit, long at least 32-bit; in C99, long long as least 64-bit). This is the most that you can get if you want absolute portability, because there are architectures out there where you simply can't address memory in, say, 8-bit chunks (see SHARC for an example).

Consequently, while stdint.h defines types like int32_t, they are "conditionally supported", and so a C program that has a hard dependency on them is not universally portable. Types like int32_least_t and int32_fast_t are unconditionally supported, but they don't give you anything that you didn't have before.


Sure but as as said in an other comment in this thread those "at least N bits" guarantees are not very practical.

Suppose you want to give a unique id to users of your application. Clearly 16bit is a bit small, you might get more than 65535 users. You'll probably want 32bits to be comfortable (unless you're google or facebook I guess).

So what type do you use? On any modern 32 or 64bit computer "unsigned int" would do the trick but if you want to be portable then you can't assume that "unsigned int" is more than 16 bits. So you have to use long instead.

But then that makes your type 64bits on modern computers. Completely overkill. If this id is used all over the place it will significantly increase the memory footprint and could hurt performance as well.

In truth when you write an application you never really care about the size of the CPU registers outside of device drivers. What you want to say is "I have this value that I estimate will max out at N, find me the fastest type where that fits".

So in the end, pre-stdint everybody would use machine-specific typedefs to do just that, portability be damned. In particular any big C library ended up doing that (glib has "gint8", "gint16", Qt has "qint8", "qint16, etc, libpng has "png_uint_16", "png_uint_32", etc...). Clearly people were not fine just using the minimum requirements of the standard.

>Types like int32_least_t and int32_fast_t are unconditionally supported, but they don't give you anything that you didn't have before.

But they do! Pre-C99 there's no way to say "I want a type that's at least 32bits and as fast as possible" or "I want a type that's at least 32bits but as small as possible". "int" could potentially be 16bits and "long" could be unnecessarily large, wasting RAM and cache. The only way to have this functionality was to use architecture-specific definitions.


> But they do! Pre-C99 there's no way to say "I want a type that's at least 32bits and as fast as possible" or "I want a type that's at least 32bits but as small as possible".

I stand corrected on that point.

> Clearly people were not fine just using the minimum requirements of the standard.

Most of these typedefs are used to interop with some other standard or file format that requires that many bits exactly.

But even then I would argue that there was an overuse of such typedefs in C. I can think of several reasons for that:

First, there's a general lack of awareness that C does in fact provide "at least N bits" guarantee for most of its types. This comes up pretty often in questions on StackOverflow, and apparently there are still enough people surprised by the fact that e.g. long can never be 16-bit.

Partly, I think, it's because of the historical mess with int. Obviously, int is what people use by default, and the fact that it was 16-bit on many popular architectures in the past, and then became 32-bit on their successors, resulted in much broken code. I think that the lesson many took away from this is that all integer types in C are similarly broken, and switched to int32_t (or equivalent typedefs) just to avoid having to think about it.

Partly it's because of 64-bit ints. Standard C didn't have a 64-bit type for a long time. Popular implementations provided it as an extension, but in different ways - long long in most Unix compilers, __int64 in 32-bit DOS and Windows compilers. So if you ever needed 64 bits, you needed a typedef for that. Since most devs work on systems where word sizes are 8/16/32/64, they did the simplest and most obvious thing, and defined something like int64_t (as opposed to int64_least_t) - it just didn't occur to them that something beyond that was needed. And that set a precedent.

Then there was the part where long was made 64-bit in LP64 model, which meant that portable C had no "at least 32 bits, but no longer than it needs to be" type until C99. With the precedent established by int64_t, the logical step was to add int32_t, and int16_t as well for good measure.

And that's how we got where we are today. I would still argue that most of those libraries that use int32_t and similar types don't strictly need it. It's just a combination of historical factors, and also the fact that platforms where such types do exist comprise the vast majority of platforms, and all platforms that people care about. So there's no particular incentive to even think about how you'd write code for something else - why deal with the extra complexity, if you can just wave it off? Can't blame anyone for that.


Good explanation

And to me this shows clearly that we shouldn't be using C then.

If an arch can't address an 8 bit sized element there are two choices for what happens if you write:

char *a = "test"; char x = a[1];

Either char takes 32bytes or the a[1] read will shuffle the bytes to get a 32bit value whose least significant digits are a[1]

And saying that int32_t are "conditionally supported" is really proof we're making C worry about 8-bit microcontrolers when we're doing big application with it, it's trying too hard


On SHARC, char is 32-bit (so is short, int and long).

I don't see why it's a problem, though? As noted earlier, C accommodates that. Sure, there's a lot of C code that's written with the assumption that chars are always 8-bit, because they usually are. But it's definitely possible to write portable C code that doesn't make such assumptions.

int32_t is required to be supported on any architecture on which one of the standard types is exactly 32-bit. Since long is required to be at least 32-bit, and any 8-bit microcontroller would have to support long, it would be 32-bit as well, and so it'd have int32_t (typedef'd to long). The only case where you wouldn't have it is either if the architecture has a word larger than 32 bits as its smallest addressable unit, or if it has non-8-bit bytes.

In practice, this is really not an issue, because you don't have to target the absolute most portable subset of C. The reason for making those features conditionally supported is so that when they are there, they behave the same. So you can say "my code runs on any platform that supports ISO C, and has int32_t in stdint.h". Not only that, but you can enforce it at compile-time with clear error messages, e.g.:

    #ifndef INT32_MAX
    #error "int32_t requrired"
    #endif
Your code is still portable. It's less portable than the mandatory subset of C, but it's a strictly defined part of the spec that behaves the same on all platforms that support that part.


Pretty sure 'char' in this case would be 32 bits.

Arguing that C shouldn't support microcontrollers seems a bit unfair. What should embedded software engineers use, if not "high level assembler"? There are plenty of modern languages that sacrifice performance in order to be fully hardware agnostic.


I agree that the jab at microcontrollers is unwarranted (after all, I do want to be able to program microcontrollers in Rust!)

But you could still have any-bit integers even on an 8 bit processor, you'd just have to emulate them.

Clearly that's not really in the spirit of C which is probably why the standard doesn't force the compiler to implement those, instead relying on the coder to make a decision.

But that just proves my original point: C is not about portability. It's about writing fast code on any architecture. If portability was the concern the standard would make fixed-width integers mandatory and would define things like signed overflow, forcing the compilers to emulate the behavior when the hardware doesn't behave in the same way.

That's what portable languages do, that's what Rust does.


> Pretty sure 'char' in this case would be 32 bits.

On ARM it is 8 bit + manipulation of value

> Arguing that C shouldn't support microcontrollers seems a bit unfair

No, I'm saying the opposite

What I want to say is that because it supports a big variety of processors it tries to do too much AND that our focus should be in more expressive languages in more powerful machines

C is great for a microcontroller with limited resources and inputs


It's actually int_least32_t and int_fast32_t.


I stand corrected!

(Ironically, this also shows just how much these types are actually used...)


Indeed; the language is portable. Programs written in the language require great effort to be themselves portable.

Also, portability isn't a binary attribute. For every way two platforms vary, there are often a subset of states and transitions that are identical, and two disjoint subsets that are different.

As long as your inputs start out in the identical subset and don't wander too far, you're good to go. For example, if you never put a value above 32767 into an int, you don't really care whether your int is 16 or 32 bits in size. If that int is e.g. line number in a file, it may mean that your program works fine on most files, but only the 32-bit version works correctly on large files. And of course the cleverer you are the closer you're likely to skate to the edge of the common intersection.


Relying on a macro processor to patch and bandage your program at compile time depending on a collection of symbolic features isn't really representative of portability of a language and your code. To me, that is brute forcing portability, and is a web of conditional inclusions/exclusions that needs to be maintained and added to per-architecture and per-OS.

Something that is, in some sense, truly portable is one that has a single specification that runs on all platforms the base implementation supports. Virtual machines and certain styles of language standard are often able to accomplish portability for a vast number of applications.


I would argue the other way. Something which requires a base implementation to be installed on a system which generates different machine code based on a single specification is less portable than a macro processor generating different versions of a program. Sure, people will have had something like python installed since a long time ago, but a new setup requires downloading 2 objects (the base implementation and the program) compared to a single one (a platform-specific copy of a program).


How is the deployment of a high-level runtime different from libc? Both can be statically-linked if required.


You still need to download two objects: the C code and the C compiler, even if the C code has already been processed.

Edit: Not to mention libc, libm, and the like.


You don't need a C compiler to run an already compiled program. You may need libraries, just like almost every language, but only the developer needs to download the compiler and not every single user.

It gives options to the developer to target multiple platforms without extra effort from the users of those platforms.


> Relying on a macro processor [...] isn't really representative of portability of a language and your code.

I would argue that the C preprocessor is, for all intents and purposes, part of the C language. C is not quite practical without its preprocessor, and the preprocessor is not quite practical with any language other than C (or a derivative like C++).


The preprocessor does not add portability. It adds a path for you, the programmer, to succinctly specify additions and deletions from your program so as to make it portable.

This is markedly different from a language that, for example, guarantees the existence of integers of unbounded size. In this case, when working with integers, we make no mention of the platform to execute on. We are provided with an abstract guarantee that one can rely on regardless of the concrete platform.


Portability is not black or white. Macros are essentially selecting different code depending on the platform. A good high-level language would not need to do this.


Just because you don't see those ugly macros to make your pure high level language portable, doesn't mean they're not there, somewhere.


Yes but this is a separation of concerns. This is precisely the point of an abstraction, especially at the level of a language and its implementations.

Writing code in Common Lisp means you're writing something with semantics promised by the Common Lisp specification. Vendors implement the spec (perhaps using nasty #ifdefs), and developers write code against the spec. The spec is a contract, and from the point of view of the programmer, it's indifferent to platform change and so on (up to what the spec says is platform-dependent, like machine integer size or maximum array length).

Just because a particular implementation may not be considered portable, doesn't mean my program (which I may choose to run on that implementation) isn't portable.


Sticking to the spec is one way to be portable. Building a portability layer is another one; this is how Common Lisp grows, by taking advantage of the work of different implementations and using standardized tools (FEATURES, reader conditionals) to build a common interface. See e.g. CFFI which has become the (defacto) standard for loading C libraries (https://common-lisp.net/project/cffi/spec/cffi-sys-spec.html...).


Unfortunately the architectural details of the machine will leak through to the program you are writing, and you will need to twist your code in order to get correct behaviour and fast execution.

Case in point is the recent article posted on HN about the platform specific details of getTimeMilliseconds() in Java.


Could you link that article? I searched and couldn't find it on hN.


I believe the article was http://pzemtsov.github.io/2017/07/23/the-slow-currenttimemil... Although I couldn't easily find it in my history or on HN. I had to resort to the Googles.


The difference is: Do I have to write the code to make it portable or does the language developers write that code.


The issue here is handling portability yourself or let the language and its toolchain do it. The odds that you are doing a better job than the community to handle portability are pretty low right from the start and they get lower and lower if the laguage is well-supported.

My "ugly macros" are tested against my own implementation only. Compiler's are tested against thousands.


It's called abstraction. It's a good thing when trying to build correct software.


> It was a fun read, except when all the C bugs were listed. It's getting tiresome to hear about folks writing nominally Important software in a language where it is extraordinarily difficult to get things absolutely correct, with paltry excuses such as "it needs to be fast" or "it needs to be portable [0] to a VAX-11/750."

Are there any practical alternatives when that kind of portability is needed?

What about transpiling to C? I know Haskell and OCaml can be made to compile their code into C code that you can then pass on to a C compiler.


Yes, C++. It's the only practical, safe alternative to C.


In C++, you can for example:

1. Manage resources with RAII (though, in C you could use non-portable attribute cleanup in similar manner).

2. Use type safe wrappers around builtin types like in [0] and [1].

3. Use containers with more extensive bounds checking and iteration validity checks [1].

This could help detect some of those bugs at runtime, or even possibly prevent them from being written in the first place. Use of uninitialized value could be prevented with types that require explicit initialization or have default one (Bug 1). Left shifting a negative value could have been caught at runtime (Bug 2). Bounds check could prevent uninitialized memory read (Bug 3) Type safe wrappers could prevent an accidental promotion from being written in the first place (Bug 5).

Though, you need to go out of your way to actually do all those things, not to mention that your code would integrate poorly with existing library ecosystem. IMHO choosing C++ alone doesn't improve safety of your programs compared to ones written in C all that much.

[0] https://github.com/foonathan/type_safe

[1] https://github.com/duneroadrunner/SaferCPlusPlus


It's easier to write safer code in C++ but there's still many pitfalls and it's still a lot of effort. There's really nothing even remotely close to being practical?


Can you clarify what you mean by "practical"?


Curious about the other language. What other language that doesn't add another layer between the code and system instructions were you thinking about?


C adds a layer between your code and the system. It abstracts away memory layout, locations, as well as function call semantics, and a many other things. For example, given a C program, you will not be able to tell me with certainty that a variable will occupy memory on the stack, any of your particular processor registers, etc. Formally, C is a language to control the C Abstract Machine, as described in the standard.

The sort of direct control one might look for is, as far as I know, only practically found in your processor's assembly language, and your operating system's collection of syscalls.

I think Rust, Common Lisp, OCaml, D, Haskell, and other languages give you a lot of benefits that are simply not worth giving up in a new crypto library, like safety.


> Haskell

I think writing timing-attack resistant code in Haskell would be very hard to impossible, at least without writing very unidiomatic code (basically "C in Haskell"). I'm happy to be proven wrong, though.


Writing unidiomatic code would still be compliant with the Haskell Language Report, compiled by an Haskell compiler and safer than plain C code.


One approach would be do design an embedded DSL for implementing cryto algorithms. And then a compiler for this DSL into C or LLVM.

In fact, I believe something along the lines of this has already been done: https://cryptol.net


Honest question: What makes you think you can't mitigate? A̶ ̶t̶y̶p̶e̶s̶a̶f̶e̶ ̶c̶r̶y̶p̶t̶o̶ ̶t̶h̶a̶t̶ ̶m̶e̶a̶s̶u̶r̶e̶s̶ ̶i̶t̶s̶ ̶e̶x̶e̶c̶u̶t̶i̶o̶n̶ ̶t̶i̶m̶e̶ ̶a̶p̶p̶e̶n̶d̶e̶d̶ ̶w̶i̶t̶h̶ ̶a̶ ̶f̶i̶n̶a̶l̶ ̶d̶e̶l̶a̶y̶ ̶t̶i̶m̶e̶ ̶r̶e̶a̶d̶ ̶f̶r̶o̶m̶ ̶/̶d̶e̶v̶/̶u̶r̶a̶n̶d̶o̶m̶ ̶c̶a̶n̶ ̶s̶t̶i̶l̶l̶ ̶b̶e̶ ̶i̶d̶i̶o̶m̶a̶t̶i̶c̶.̶ It has to be monadic, that is for sure but abstracting a crypto algorithm as IO is actually treating it as a device which sounds like a safe metaphor.

Edit: Whoah, why the hate? :(


I believe padding the timing always has been considered a bad idea. If you want to guarantee you won't exceed the timing, you probably have to take so much margin that your primitive becomes prohibitively slow. And I'm not even sure it would protect you against cache timing attacks.

Simply put, there are easier and cheaper ways to guarantee timings. Perhaps Haskell is even capable of harnessing them. (I'm thinking of the usual "no secret dependent branches", and "no secret dependent indices".)


Random delays do not protect against timing attacks. Best thing it can do, is to force the attacker to collect a few more timing samples. (To the point where the the random noise averages itself out.)


Rust doesn't even exist on a plethora of platforms which C targets. I have high hopes with D and give or take ~2 years, it'll be able to progress forward in lots of domains. Not sure about other languages.


One big long term advantage of Rust is that it leverages LLVM. Basically anywhere LLVM gets ported Rust follows.

I could run Rust code on a PlayStation and PocketStation (although I had to cheat a bit on the former because LLVM doesn't support MIPS I).

LLVM is not as widely ported as GCC (especially on older, deprecated hardware) but it's getting there.

https://svkt.org/~simias/rustation/pockestation-rust.jpg


That is a matter of tooling, though.

Until the early 90's C had hardly any meaning outside big expensive UNIX boxes.

And it was already on its way out on Windows and OS/2, if it wasn't for the rise of FOSS software and its dependency on C.


I think it might be a bit more than that. As far as I can see, all the primitive data types in Rust have sizes which are multiples of eight. However, there are still architectures out there with 9 bit bytes, or 36-bit words. Maybe it's good that all the primitive data types have an explicit size (encoded in their name), but this comes at a cost (in case a byte is 9 bits long, one of the bits will effectively not be used). On the other hand, C and C++ are less explicit about the sizes of their data types so they can accommodate these more weird architectures more easily (and, if you need an explicit size, you can explicitly ask for it).


You have a point but in general you wouldn't write "generic" C code on these exotic (by modern standards) architectures. So I guess if you wanted to port Rust to these systems you could introduce an i9 and an i36 and use that instead. I doubt you could get firefox and its dependencies to work on a CHAR_BIT == 9 system without some heavy modifications. I could be wrong though, it's not like I tried.

In my experience most modern codebases tend to target POSIX systems (explicitly or implicitly). That gives you a subset of C to work with and makes your life easier. The exceptions are things like DSPs which can have rather non-standard layouts but you generally write very specific and unportable code for these anyway.

Of course if you want to be completely portable you can't assume that CHAR_BIT == 8 or that you can convert back and forth between function and data pointers but in practice a lot of software relies on this.

Even before stdint became part of the standard it was very common for programs and libraries to typedef their own "sized" integer types. It's the only reasonable way to deal with ints in C IMO, otherwise you can't assume that your ints are greater than 16bits (which is probably too tiny for many uses) so you're tempted to put "longs" everywhere. But then longs will be 64bits on modern architectures which could be overkill and reduce performance.

So in theory those variable-size integer types are interesting but in practice they're basically useless because you end up making assumptions, one way or an other. I think it was a good idea for Rust to get rid of them.


Any language that needs evangelists is clearly not able to progress on its own merits and archievments.


What? You think other languages don't have "fan-people"? People who like Rust tend to like it very much, and it does provide a real solution to problems that C and C++ create, while providing a no-compromise performance story. So why wouldn't people talk about it?


Perl had evangelists, Ruby (on Rails) had evangelists, C++ had evangelists, Java had evangelists (some would say Java even had apostles and Messiahs...), almost every language has evangelists.


Most programming languages with compilers to native code (JIT/AOT), the myth of high level Assembler for C only applies if your computer is a PDP-11 or a basic 8-bit CPU like a 6502 or Z80.

The ANSI C and C++ standards define the concept of abstract machine for the language semantics, just like in most languages.

Additionally you have the concepts of sequence points, the new memory model semantics for multi-threaded code and the beloved UB.

UB which doesn't exist in most languages, because their rather leave it implementation defined or lose the opportunity to target some strange CPU not able to support the language semantics.

Also Assembly doesn't has UB, making it ironic that it is safer to write straight Assembly at the expense of portability than C or C derived languages.


The ARM Architecture Reference Manuals are full of "if you do X, the instruction is UNPREDICTABLE".


> Also Assembly doesn't has UB, making it ironic that it is safer to write straight Assembly at the expense of portability than C or C derived languages.

You just need to know your compiler flags/configuration to produce exact assembly output (if you want that). And about assembly not having UB, BSF and BSR are pretty good examples for that.


That just is quite ironic, because it is even version dependent across releases of the same compiler, which means anything fine tuned for version X can break on version X + 1, due to changes on the optimizer.

And I doubt anyone is regularly reviewing the produced Assembly every single time they change compilers.

Assembly doesn't have more than 200 documented cases that hardly any human is capable of remembering.


You could argue that each processor's errata is UB in assembly. I remember that programming for the XScale we had a whole manual of errata that we had to program around. Though because it's documented it's not "Undefined", it's just "Incorrect but we know how incorrect".


Aren't all undocumented instructions / opcodes actually UB? If you execute an undocumented instruction the processor has to do something.


You can't really accidentally or unintentionally arrive at undefined behaviour in assembly language though. You would have to explicitly use an undocumented instruction or opcode and the result would probably be a hardware interrupt or exception every time.


Not quite every time though: https://github.com/xoreaxeaxeax/sandsifter


Woah that's pretty neat! Thanks!


I'd say that Ada would be well-suited for this. Particularly for implementing crypto. It has a steep learning curve, though.

Edit: Sorry, wrote this too hastily and want to put it in perspective. While Ada is well-suited for cryptography and has some good implementations, it is probably not as secure as C crypto libraries in practice. The reason is that the implementations I know don't do OS-specific things like locking memory or preventing core dumps, and doing this in Ada is a bit harder than in C.


OCaml was used recently by the MirageOS folks to implement a better TLS than OpenSSL.


TLS != crypto. TLS is actually very well-suited for a higher-level language like Rust or OCaml, especially due to how state machine patterns can be expressed safely in those languages. Crypto, on the other hand, often relies on low-level intrinsics to get good performance and cache timing attack resistance.


TLS depends on crypto and the mirage team implemented it using OCaml. It supports low-level intrinsics for timing attack resistance. See: https://github.com/mirleft/ocaml-nocrypto


    found an error in the Argon2 reference implementation.
This is exactly why I like as many people writing crypto as possible[0]. Bugs like the one the writer found show up in reference implementations, and are found by absolutely no one unless someone writes an alternate implementation that behaves differently.

Edit: A further example, a lot of people write crypto as part of various CTFs. This is a case of "writing crypto" that poses no threat to the community.

[0] That doesn't mean they should use it in production.


I've always seen 'rolling your own crypto' as not being recommendation against writing your own library, but creating your own primitive.

Sure, writing your own library is very difficult, but you have a simpler set of problems, which proper testing, another set of eyes and enough tools will take care of the big problems.

Now, implementing your own primitive and recommending to use it is bad. For a primitive to be deemed secure, it has to undergo years of review and testing by seasoned cryptographers.

This isn't a recommendation to write your own library, but my opinion that it is okay to do so if you want something that none of the major libraries offer.


Even just using crypto primitives is dangerous unless you've got experience. There's an old post from matasano about it https://www.nccgroup.trust/us/about-us/newsroom-and-events/b...


Every line of code written can be a severe vulnerability, not just code dealing with crypto. In case of crypto, being well-informed is good enough to not mess up those kind of implementations (on a theoretical level). The problems mentioned can be pointed out and verified on paper. But you can't say the same thing for code.


No, you can write crypto code, even when knowing stuff about the crypto, and still mess up. Crypto is problematic because you often cannot easily check whether you're wrong (it could be a tiny edge case that isn't tested because your input space is huge).


I didn't deny messing up crypto code is a thing. I'm saying messing up code is a separate problem from messing up how one architects projects using crypto.

What you say as messing up with edge cases, etc. is a programming problem. The errors talked about in the link he posted is about using faulty crypto. Today it is common knowledge to never use ECB.

What cipher mode you use is a problem you think about before you even start writing a single line of code. But you are talking about programming errors, which one can make in all areas, not only ones involving crypto. Those errors can break security similar to messing up crypto implementation.


I strongly disagree with this sort of attitude. It scares people away from learning how cryptography works. We should be encouraging everyone to write their own crypto: it's a fantastic learning experience. Just don't use your first go in production.


I disagree. The way you get experience is by trying things. If everyone followed that advice then nobody would learn anything.


Hey, fantastic work! Really happy to see some amateur crypto development out there. I started as an amateur also, now I work in crypto product dev.

Definitely impressed if this is your first crypto work. I really like the simplicity, reminds me of OpenSSH.

I gotta agree with some of the naysayers though - this needs more time to brew. Even if it's perfect, there still more proof to show.

My suggestions:

* A long-running interoperability test between your lib and another. Test vectors don't catch everything. Run it for a week, check results and publish the results.

* Fuzzing. You need to to catch false results, random crashes, stack overflows, etc. Fuzz every input, do it for random lengths (both underflow and overflow).

* Maybe port it and test on Win, Mac, ARM, MIPS. I find a lot of bugs shake out during ports.

* Timing. You can't prevent all side-channel attacks, but bits going out are easy attack vectors. Use a high-precision timing mechanism to catch any unusual spikes. Capture a few hours of points for each functional path, and publish the results. (crypto_aead_unlock has two return paths, it might be better to maintain timing).

* Personal preferences: Rather than using int returns, use bool. Zero your ctx vars, e.g. a_ctx ctx = {0,};

I spend more time in testing products than development, honestly. Publish as much relevant data as you can.


> Definitely impressed if this is your first crypto work.

It is. :-)

> A long-running interoperability test between your lib and another.

Already have one for all primitives. Just tweak the parameters to make it run for a long time. The test code is in test/sodium.c and test/donna.c

> Fuzzing. You need to to catch false results, random crashes, stack overflows, etc. Fuzz every input, do it for random lengths (both underflow and overflow).

I'm not sure how to go about it. I already check every input size, from zero to several times the size of the internal block, in my comparison tests.

Of course, all inputs are correct. Incorrect inputs result in undefined behaviour, checking for that is out of scope. (That's why most functions return void.)

> Maybe port it and test on Win, Mac, ARM, MIPS. I find a lot of bugs shake out during ports.

Yeah, I only tried GCC and Clang on my Intel Ubuntu machine. But I did use Valgrind, ASan, MSan, UBSan, and the TIS-Interpreter. I also compiled under various standards: C99, C11, C++98, C++11, C++14, C++17.

> Timing. […] Use a high-precision timing mechanism to catch any unusual spikes

Isn't that a bit overkill? I'd rather manually review the source code for the absence of secret dependant branches or secret dependent array indices. I'm not implementing AES or RSA, the primitive I have chosen were specifically designed to easily avoid timing attacks.

> crypto_aead_unlock has two return paths, it might be better to maintain timing.

No. There are 2 cases to consider: failure, and success. Each path runs in constant time, so timing can only distinguish success from failure… which is revealed anyway by the return code.

> Rather than using int returns, use bool.

I'm afraid bool is a tiny bit less portable than int. I also went with the flow about error codes, which are traditionally int.

> Zero your ctx vars, e.g. a_ctx ctx = {0,};

I'm not sure how to do it portably, with zero dependency.


Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: