
"OpenSSL has exploit mitigation countermeasures to make sure it's exploitable" - anon1385
http://article.gmane.org/gmane.os.openbsd.misc/211963
======
stiff
It is surprising that a project that is quite mission critical is completely
at the bottom of the scale when it comes to how much the development process
is oriented toward reliability. There are no systematic unit tests, no
systematic documentation, the best you get is a bunch of disorganized
integration tests, so it is not even at the level you would expect for a
decently maintained business project:
[https://github.com/openssl/openssl](https://github.com/openssl/openssl)

Perhaps the open source model of development is just not very good for
software of this kind. Of course it's good that the source is open for
everyone to look and potentially contribute, but without funding and without
having a real process and a full time team it seems to me it is hard to get
the level of quality required.

I also wonder how much in the end the big institutions care about this stuff.
Intel hires a bunch of guys to do formal models of their processors to ensure
bugs aren't shipped to millions of customers, why is nobody funding a formally
specified version of SSL? For other mission critical systems, like what goes
into spacecrafts, or hospitals, or gets developed in the military there are
rigorous processes in use to prevent stupid mistakes, so it's somewhat
disappointing that the major infrastructure pieces don't receive this kind of
treatment.

~~~
motters
My attitude towards complaints about open source projects is that if you think
the errors are rudimentary then just submit some patches. If there is not
enough test coverage then add one. This applies especially if you believe that
the project is critically important.

I think what might be happening here is expert syndrome. People may be told
that if they're not experts then they shouldn't be reviewing or changing the
code.

~~~
conformal
^ this!

back in 2010, my business partner, marco peereboom, submitted a patch to
openssl to add support for aes-xts. it was coded by joel sing, now a google
employee and golang dev. they _didn't even respond to the mailing list email_
and after marco nagged for a reply the response was "we have a different plan
for how to implement XTS" (i'm paraphrasing). _2 years later_, they added XTS
support.

the openssl dev team is not responsive, doesn't accept contributions and
generally speaking suffers from "you're not an expert" syndrome. look how
expertly they've managed their project!

~~~
TwoBit
Well maybe his patch really wasn't in line with how it needed to be. I've
always hated openssl but I'm not sure this is a valid case.

------
bhouston
Has anyone started a rumor yet that the NSA infiltrated the OpenSSL
development team to make OpenSSL ineffective and full of holes?

The convoluted code of OpenSSL alone (from yesterday's Hackernews post) seems
like a great way to add all sorts of "bugs" inadvertent or not.

Unfortunately with the Snowden disclosures, there isn't much that I rule out
of bounds for the NSA when it comes to things critical to internet security.
OpenSSL is so widely used and critical, it would be silly to think that it
would escape scrutiny by the NSA.

Remember that the Snowden docs says that the NSA can break SSL/TLS/https/VPN,
etc., but we do not know the full details:
[http://blog.cryptographyengineering.com/2013/12/how-does-
nsa...](http://blog.cryptographyengineering.com/2013/12/how-does-nsa-break-
ssl.html) But the one thing all of these technologies (SSL,TLS,https,VPNs)
have in common is usually OpenSSL.

~~~
TacticalCoder
There's a talk that was given in Belgium / Brussels at FOSDEM2014 two months
ago or so by Poul-Henning Kamp (FreeBSD) regarding the NSA and how he'd do it
if he had to create holes in software:

[http://www.youtube.com/watch?v=fwcl17Q0bpk](http://www.youtube.com/watch?v=fwcl17Q0bpk)

He's talking specifically about OpenSSL quite a lot (basically saying it's too
complex to ever be secure and probably received many "security patches" from
NSA employees).

The entire talk is an eye opener. He explains how NSA shills are reading
reddit / HN and poisoning communities / standards / protocols / etc. How
everything is made, on purpose, needlessly complex to prevent honest
developers from working on important things.

He talks about shills submitting a few correct patches over the months /
years, slowly gaining reputation among the community and then misusing that
trust to submit (not so) subtle patches introducing security holes on purpose.

He mentions a few of the "common mantra" repeated often (including here) by
people who have an interest in the status quo.

He explains why SSL/TLS is broken and says that the "SEC" part of "DNSSEC" is
not going to be that secure ; )

I think that the problem is much worse than most people think and that Poul-
Henning Kamp is closer to the truth than the ones constantly repeating "bug
happens" as if nothing malicious was ever going on.

~~~
pvnick
I'm watching this right now, it's brilliant! This is exactly the kind of stuff
I love to think about, but PHK has done a wonderful job thinking it out to a
lot of detail. I suggest everyone drop what you're doing and watch it asap.
It's exciting and really holds your interest.

~~~
ableal
I did watch, thanks, but anyone in a hurry can get just the slides here:
[http://phk.freebsd.dk/_downloads/FOSDEM_2014.pdf](http://phk.freebsd.dk/_downloads/FOSDEM_2014.pdf)

I particularly appreciated the "QUEEN success example: SSC" (self-signed
certificates, starts at slide 23), which also got a good laugh from the
audience ...

(if you're just watching the video, this is the text of the second slide in
the PDF, not shown at FOSDEM:

 _" "" What is this ?

This is a ficticious NSA briefing I gave as the closing keynote at FOSDEM 2014

The intent was to make people laugh and think, but I challenge anybody to
prove it untrue.

Playlist spoiler:

Electric Light ORCHESTRA: Confusion

ABBA: Money, Money, Money

Backman Turner OVERDRIVE: Taking Care Of Business

QUEEN: Bicycle Race

Beastie BOYS: Sabotage

PASADENA Roof Orchestra: Pennies From Heaven """_ )

------
josteink
I may be biased in that I have not written much C-code in years (decades?),
but whenever I find a codebase covered in IFDEFs, I start assuming that every
single new IFDEF introduces a new condition into the system which has not been
properly nor recently tested, and that the software is horribly broken.

For me, fixing software, is just as often removing IFDEFs together with
unmaintained and broken code. It's IMO much better to be honest about "this
isn't supported", than pretend you support something you don't.

And this seems to be another one of those stories, coupled with a (bad?) case
of NIH.

------
btown
This should be a case study about why people developing system-critical
software shouldn't write their own memory allocators.

~~~
jwr
At a first approximation, no one should write their own memory allocator. Just
as no one should write their own garbage collector, and no one should write
their own filesystem.

All these subsystems have one thing in common: they are incredibly complex and
very hard to get right, but seem easy on the outside.

~~~
pwang
I disagree.

These are complex to make robust, but they are not hard concepts. I think
_everyone_ should write their own GC, write a filesystem, and handle their own
memory layer, at least once.

Treating lower levels of the stack as "too complex, there be dragons, just
write some Javascript that hits RESTful API for JSON" is a great way to ensure
you never progress as a software dev or an engineer.

I agree that reinventing the wheel poorly - without understanding geometry or
physics - is a terrible idea. But I also think that we don't have enough
people nowadays making toy wheels to _learn_ about geometry and physics.

For each of the things you mentioned, there are concrete and good reasons why
they grow complexity: filesystems need to be robust to media corruption and
power faults; memory allocators need to be highly tuned to OS and architecture
memory layouts; GCs need to be optimized for the code patterns of the
particular language they serve, as well as understand the same OS and hardware
memory constraints as allocators.

But much of the complexity of these layers is not _intrinsic_ to the problem,
and not related to the above. In fact, most of the complexity and bugs of
production code at these layers have to do with legacy compatibility, or code
inheritance. (If you look at what Theo says about this specific OpenSSL bug,
it arises _because_ the OpenSSL team wrote their own allocator that doesn't
use malloc() protection bits because those are not compatible across some
platforms.)

As the old saying goes: in science, we advance by standing on each others'
shoulders; in software, we stand on each others' toes.

~~~
LaGrange
> I think everyone should write their own GC, write a filesystem, and handle
> their own memory layer, at least once.

Of course. But not _in production_. Which is a detail missed by the snarky
one-liner, maybe because one-liners suck. Still, the point isn't that people
shouldn't ever write them, just that they shouldn't actually _use_ the ones
they developed while not being a part of a team of experts specializing in the
issue at hand.

------
robin_reala
So what are the options then if OpenSSL isn’t fit for purpose? Is it possible
to move wholesale to a different project? Are any of them trying to ease
migration over from OpenSSL to themselves?

~~~
antocv
GnuTLS exists. And NSS in mozilla. Probably others too.

~~~
reidrac
cURL makes a nice comparison: [http://curl.haxx.se/docs/ssl-
compared.html](http://curl.haxx.se/docs/ssl-compared.html)

GnuTLS had its own problems recently (see:
[http://www.gnutls.org/security.html](http://www.gnutls.org/security.html)),
and regarding NSS... I couldn't find a proper public "security" page for it
(eg, release notes for 3.16 point to CVE-2014-1492; but the link to the
bugzilla issue is not public), so I don't know.

Obviously a good security record doesn't imply things like a good code base,
good practices, etc; (although it could).

~~~
recentdarkness
Regarding the NSS CVE-2014-1492:
[https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-14...](https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-1492)

Quote: "The cert_TestHostName function in lib/certdb/certdb.c in the
certificate-checking implementation in Mozilla Network Security Services (NSS)
before 3.16 accepts a wildcard character that is embedded in an
internationalized domain name's U-label, which might allow man-in-the-middle
attackers to spoof SSL servers via a crafted certificate."

Edit: Added Quote

~~~
reidrac
Thanks, I can look for a CVE myself ;)

My point is that having a proper security page is a good practice and I really
appreciate it as user.

------
jimrandomh
Here is the patch Theo is talking about:
[https://github.com/openssl/openssl/commit/8671b898609777c95a...](https://github.com/openssl/openssl/commit/8671b898609777c95aedf33743419a523874e6e8)

------
bananas
Great analysis. Theo is always up front on this sort of stuff which is
commendable.

~~~
simias
Could have done without the final line though, but I guess it's his equivalent
of a signature...

~~~
MrBuddyCasino
To quote Ted Dziuba:

"Anyone who ever told you that swear words have no place in technical
discussion is right. They're right, and sadly, they're part of the problem
because they miss the point. The sterile word placement that's supposed to
support an argument makes any true motivation indistinguishable from all the
hired bullshit.

[...]

 _However, when someone starts swearing in technical discussion, showing
emotion, that 's a strong indicator that I'm about to receive wisdom._ Wisdom
is earned the hard way, and it is permanent, not like some statistically shaky
performance benchmark that we'll all forget about next week."

I feel the same - I'd rather have people feeling passionate about their work
and use swear words, than work with polite non-personalities that don't give a
fuck.

~~~
Tyrannosaurs
Swearing isn't required for passion. Martin Luthor King didn't have an
(expletive of choice) dream, he just had a dream. There are plenty of people
who care about their work and manage to express that without swearing.
Similarly there are plenty of people who swear about stuff who may well be
committed but it's highly questionable about what they're committed to and
whether it's to the benefit of the project / team.

If someone does swear (and as it goes, I'm fine with swearing for emphasis and
swear way more than I feel I should) they need to be aware that some people
find it offensive and that if they don't moderate their behaviour it may be
taken very personally which can result in barriers between colleagues and / or
low morale.

You can choose to say that that's the person reacting's problem but if you do
so you need to be aware of the price that comes with (that they're working in
an environment with which they're not completely comfortable which is unlikely
to get the best out of them). The alternative is that the swearer moderates
themselves (with which they in turn may not be comfortable).

Obviously the better you know your colleagues the more leeway you have. If
you've built up trust and togetherness overtime you can get away with anything
but even then you need to consider what happens when new people join the team.

But ultimately this is about people and where it's about people there are no
absolutes - what works for one group will be kryptonite for another.

~~~
Sanddancer
Theo has a twenty year record of being a brash, non-calm[1] asshole. The pesky
problem is that he has this very bad habit of being right, especially when he
is angry, brash and profane. So yes, he may indeed may need to enhance his
calm, but at the same time, other software developers need to realize that the
best course of action in avoiding his ire is to not write software that sucks.

[1] [http://www.theos.com/dispute.html](http://www.theos.com/dispute.html)

~~~
Tyrannosaurs
It's the Steve Jobs conundrum - where someone is an asshole but clearly also
very talented.

For me two thoughts come out of this:

1) Yes they've been productive the way they work, but have they been more or
less productive than if they'd been just as right but a little less
confrontational?

Research generally doesn't show confrontation to be the most productive
approach for most people. Is their style productive, or is it unproductive but
they get away with it because they're so good that people will put up with it?

You say that the best course of action if you want to avoid Theo's ire is not
to write bad code, but how many people have taken another approach? The one
that springs to mind is not working with him and how much talent does the
project miss out on because people just steer clear? I know plenty of very
smart programmers who simply don't go near projects where the culture and
personalities are like that. Bad behaviour towards others (even when you're
right) limits the available pool of skills.

2) I can live with the arsehole geniuses but others who aren't as able often
use the likes of Jobs or Theo to justify their own poor, destructive
behaviour. For me at least, that doesn't work - if you're going to be as big a
dick but you're not as good as them you're basically just a dick and if the
day eventually arrives when the IT industry doesn't have a massive skills
shortage, those people will cease to be tolerated.

------
Orangeair
I echo what someone said earlier on another OpenSSL thread: security
applications should not be handling their own memory at a low level like this.
It's just too easy to mess up, and it often leads to the worst
vulnerabilities.

------
mannykannot
The 'many eyes' hypothesis has now been empirically tested.

~~~
xuu
Many eyes worked in this case.. It just took a long time to find. The question
now is what will it find next and when?

~~~
mannykannot
It worked in the sense that the lifeboats on the Titanic worked - they did,
after all, save hundreds of lives.

The combination of time and severity in this case should mean that we can move
on from naive 'all bugs are shallow' dogma towards developing a more evidence-
based approach to the verification of critical software.

------
maxbucknell
If there are so many problems with OpenSSL, why are there no alternatives that
are readily available and anywhere near as functional?

The whole internet runs OpenSSL, but why hasn't anyone tried to do something
different? I know it's complicated, but if a few big companies really chose to
put some muscle behind it, it could happen, right?

~~~
davidw
This sums up some of the difficulties with the production of open source
software:

[https://en.wikipedia.org/wiki/Public_good](https://en.wikipedia.org/wiki/Public_good)

~~~
ef4
That really doesn't illuminate anything, because you'd also need to explain
why open source has been spectacularly successful generating other public
goods (linux and others).

The economics of open source are pretty clear at this point. The software
industry spends a lot of money supporting open source, because it's in their
own interest to do so -- it's cheaper to share the costs than to build your
own infrastructure from scratch every time, when the infrastructure is not
part of your competitive advantage.

This particular bug was found by people that Google pays to audit open source
code all day, in an effort to improve said code.

~~~
davidw
> open source has been spectacularly successful generating other public goods
> (linux and others).

No one doubts that some open source software has been very successful. What
I'm not sure of is whether levels of open source provisioning are _optimal_ :
maybe there should be 10X what there is now. Maybe Linux should dominate the
desktop world, but does not due to lack of funding. This is Bastiat's "what is
not seen" \- what we have now is good, but perhaps it could be better. Maybe a
lot better, under different circumstances.

Also, that link mentions Coasian solutions, and privileged goods, which
between them explain a lot about open source software, no?

------
ScottBurson
So what if someone tried to crowdfund a new implementation (or a thorough
rewrite of OpenSSL, if that makes more sense). Could they raise something on
the order of $1M? It seems like it would cost that much for, I'm guessing,
three absolutely top developers for two years. Unlike a lot of crowdfunded
projects, this one would not launch a business -- there's no additional
revenue opportunity for the developers once it's completed -- so the amount
would have to compensate them not just for their time but also for their
opportunity cost.

On the other hand, given the importance of TLS to the Internet, $1M seems less
than trivial -- literally pocket change, if the cost were well distributed
among the millions of websites using SSL.

What do you think? Could it be done?

------
gpvos
OpenSSL was derived from SSLeay. Was SSLeay already that bad?

~~~
rdl
Worse.

------
keithgabryelski
the lesson: general and up-to-date understanding of larger systems is
sometimes required to make the most innocuous changes.

be a generalist keep up on current technologies

------
jokoon
yeah, why not rename this to NSASSL

------
renox
I find this remark quite funny from someone who is using C to write a "secure"
OS..

~~~
Sanddancer
Yes, C gives you a machine gun and no guard against your foot. However, the
OpenBSD team also has a philosophy, and design measures, to ensure that said
machine gun never gets anywhere near your foot. Given their security track
record, I think they've done a pretty good job of such.

Can you give an example of a "secure" OS not written in C so I can see what
this mythical beast looks like?

~~~
tokenrove
SPIN is probably the most well-known example:
[http://en.wikipedia.org/wiki/SPIN_%28operating_system%29](http://en.wikipedia.org/wiki/SPIN_%28operating_system%29)

------
supersnap
This is pretty much proof that open source fails to maintain security software
in a responsible way.

Look kids ssl is not that hard to implement.

Time to let the cargo cult go have a campfire and sing songs.

This is too important to leave to lazy paid for nothing programmers who want
to write lazy ass c code after too many beers.

Grow up

~~~
goldenkey
I guess we should just let you code it in Enterprise java and pay you
dividends.. Get a hold of yourself - most open source is actually funded by
large companies in the first place. And being drunk has nothing to do with bad
code, I for one code better when inebriated. Your straw man banter doesn't
need to be on HN.

