
Of Money, Responsibility, and Pride - silenteh
http://veridicalsystems.com/blog/of-money-responsibility-and-pride/
======
AaronFriel
This is a mistake. The Apache foundation doesn't sell $250/hour consulting
gigs for its primary source of revenue. Neither does the Linux Foundation, the
SQLite Consortium, or other massive, mission-critical open source products.

This is the wrong funding model. It keeps money in OpenSSL developer's
pockets, but there is no financial incentive for any OpenSSL developer to work
on foundational improvements to OpenSSL. He said himself: there is over
$100,000 in open contracts for competent developers to work on non-
foundational improvements to the project. If you are an enterprising developer
with good C skills and a knack for crypto projects and you apply to work for
the OpenSSL foundation, are you going to start servicing that $100,000 pool of
contracts or are you going to pretend that money doesn't exist and live on
ramen?

If nearly all of OpenSSL's revenue comes from clients that want OpenSSL to
meet their particular needs, then none of that money is going to developers to
strengthen OpenSSL's foundation. This is why OpenSSL looks like a hodgepodge
of hacks upon hacks in order to accomplish narrow goals with limited impact
testing. It should be no surprise to anyone else: clients are literally paying
OpenSSL developers for this, and nothing else.

Who is paying OpenSSL for developers to clean up the code base and remove
ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths
and do case analysis? Who is paying OpenSSL for developers to write unit tests
or even have a test harness at all?

No one will pay an hourly rate to accomplish these tasks. Google is not going
to pay by the hour for a developer to stare at a function until they grok it;
they want a feature. Joe Company will not pay for developers to write unit
tests, they want OpenSSL to handle $QUIRK from a vendor's system, or to know
how to make their code handle it.

This model needs to go away. Competent OpenSSL developers time is too valuable
to waste on client asks. Their project is too important, and as long as the
money is flowing only for novel features and not structural improvement, then
that money will dictate that only new features are developed.

~~~
uuid_to_string
This is one of the better comments I have seen on OpenSSL in the past week.
Well said.

"This is why OpenSSL looks like a hodgepodge of hacks upon hacks in order to
accomplish narrow goals with limited impact testing."

It doesn't just look like a hodgepodege of accumulated hacks, it is a
hodgepodge of accumulated hacks. :)

"It should be no surprise to anyone else: clients are literally paying OpenSSL
developers for this, and nothing else."

One could say this with respect to many popular open source projects,
including ones with corporate sponsorship. The complexity just keeps building
over time and there is no such thing as "finished, accepting bug fixes only".

"Who is paying OpenSSL for developers to clean up the code base and remove
ancient #IFDEFs? Who is paying OpenSSL for developers to analyze code paths
and do case analysis? Who is paying OpenSSL for developers to write unit tests
or even have a test harness at all?"

Those are rhetorical questions. We know the answers. Alas, when the people who
pay for (open source) software and consulting pay to have "features" removed
instead of added, "pigs will fly".

Doug McIllroy is quoted as saying, "The hero is the negative coder".

(Just in case this need explanation: Prof. McIllroy is the mind behind UNIX
pipes and one of computer science's most prominant contributors. "Negative
coder" means someone who removes code instead of constantly adding, or
"committing", new code.)

We could really use some more heros. And as we switch away from OpenSSL there
will be a lot of links to libssl to remove.

Meanwhile some people have been writing and testing small, auditable and
usable open source crypto, more or less for "free".

[http://tweetnacl.cr.yp.to](http://tweetnacl.cr.yp.to)

My guess (and hope) is that pathological requests for "features" to be added
would be met with heavy scrutiny. The authors already have day jobs in
academia.

~~~
dfc
> Meanwhile some people have been writing and testing small, auditable and
> usable open source crypto, more or less for "free".

With all due respect that is complete bullshit. I do not care that you put
quotes around free. Writing "free" will never be considered to include sums in
the hundreds of thousands of dollars. More importantly blatant lies like this
muddy the debate and set outrageous expectations. The Nacl project gives the
following description of funding:

    
    
      NaCl was initiated by the CACE (Computer Aided Cryptography Engineering)
      project funded by the European Commission\'s Seventh Framework Programme
      (FP7), contract number ICT-  2008-216499. CACE activities were organized
      into several  Work Packages (WPs). NaCl was  the main task of  CACE WP2,
      \"Accelerating Secure  Networking,\" led  by Tanja Lange  (at Technische
      Universiteit Eindhoven)  and Daniel  J. Bernstein (at the  University of
      Illinois at  Chicago, currently visiting Eindhoven). CACE  nished at the
      end of 2010 but NaCl is a continuing project.
    
      ...Many  of  the  algorithms  used  in  NaCl  were  developed as part of
      Daniel  J. Bernstein\'s   High-Speed  Cryptography  project   funded  by 
      the  U.S. National  Science  Foundation, grant  number  ITR-0716498. 
    

I found the funding information for ITR-0716498. djb is listed as the PI for
the project.[^1] I could only find the high level funding of
ICT-2008-216499.[^2] (wtf EU?) CACE WP2 is only one component of the project.
I would love it if someone with better knowledge of EU funding can find the
funding for the WP2 line item. The figures are:

    
    
      NSF ITR-0716498 funding: (USD)     400,000.00
      EU  2008-216499 funding: (EUR)   4,733,078.00 ***NEED WP2 line item***
    

The tweetnacl implementation lists two more funding sources. As above it was
easy to locate the NSF funding but I totally struck out for the nwo funding:

    
    
      NSF 1018836 funding: (USD)        $436,203.00[^3] 
      NWO grant 639.073.005 funding:    ???????????
    

Don't get me wrong, I have a lot of respect for djb and I think he and his
coworkers deserve every fractional euro/dollar of funding that they received
but they did not work for free. Most importantly they should not be expected
to work for free.

[^1]:
[http://www.nsf.gov/awardsearch/showAward?AWD_ID=0716498](http://www.nsf.gov/awardsearch/showAward?AWD_ID=0716498)

[^2]:
[http://cordis.europa.eu/projects/rcn/85344_en.html](http://cordis.europa.eu/projects/rcn/85344_en.html)

[^3]:
[http://www.nsf.gov/awardsearch/showAward?AWD_ID=1018836](http://www.nsf.gov/awardsearch/showAward?AWD_ID=1018836)

NB: This is the nwo funding site:
[http://www.nwo.nl/en/funding](http://www.nwo.nl/en/funding) I think the
english version may have a reduced set of features. I can not find the this
grant information on the site.

~~~
uuid_to_string
Wow. I guess "more or less" was not strong enough wording for you?

The point is that using something like NaCl costs you, the developer/user,
nothing more than if you are using OpenSSL.

Do you agree?

~~~
dfc
No, "more or less for free" is not close to hundreds of thousands of dollars
plus whatever funds came from the EU and NWO.

I have to say I am confused about your reply in the first sentence you seem to
acknowledge that the wordingwas related to the cost of "writing and testing"
crypto software. However in the second sentence you seem to indicate that your
thesis was about the switching costs users face. Which is it? You did not say
I get to use nacl "more or less for free" you said that "people have been
writing and testing small, auditable and usable open source crypto, more or
less for 'free'." That quote seems to be about the cost of creation not the
switching costs.

Do you think djb et al produced nacl "more or less for free?"

~~~
uuid_to_string
I think you misunderstood what I meant.

I mentioned "free" only to point out that there is no financial cost to
switching to it. I guess I did not type the sentence with enough care; words
are missing. My apologies.

I imagine people would be willing (and are accustomed) to paying for software
of similar quality.

But I'm also wondering why this bothered you so much.

Does it make a difference that grants were received?

Is the funding not transparent enough?

The blog article on OpenSSL mentions payments for consulting and "features" to
be added to OpenSSL.

Should I be concerned about what those features are, and who is paying for
them? Are you concerned?

I'm just nterested in cleaner code than OpenSSL's. NaCl looks cleaner to me.

Maybe I'm wrong. But I'd rather be compiling programs that use libnacl or some
other simpler alternative than ones that use libssl.

We all have to make decisions about what software we choose to use, even if we
are not cryptographers.

I see nothing wrong with discussing alternatives to OpenSSL. This bug has been
a real PITA.

~~~
dfc

      > I mentioned "free" only to point out that there is no financial cost
      > to switching to it. I guess I did not type the sentence with enough
      > care; words are missing. My apologies.
    

It speaks highly of your character that you say this to the jerk on the
internet said you were full of shit.

    
    
      > But I'm also wondering why this bothered you so much.
    

Because crypto is important. A lot of harmful attitudes/mindsets are
reinforced if people think NaCl was created in the authors spare time and did
not require funding:

\- "Why should I donate to GnuPG/OpenSSL/Tor/Mozilla(NSS)? Those NaCl devs
wrote NaCl for free."

\- "How hard could it be to implement a crypto library? Nacl was a side
project. The Nacl devs 'have day jobs in academia' and created nacl in their
spare time. They did it for free, so they obviously didn't need to spend money
on testing environment, research material or hire/consult experts. On the
other hand look at SelfiesMadeEa.sy they raised serious cash and had to quit
their jobs because they _tackle hard problems_."

\- "Obama and the rest of gubmint are taxing me to death. Government should be
pay for the military and maybe some roads; not waste money on liberal
academics in ivory towers, maplethorpe and those pinkos from NEA or some
stupid robot/telescope that cant do metric conversions."

\- "OMG NSA is evil. USA does nothing but invade countries and privacy."

    
    
      > Does it make a difference that grants were received?
    

No it does not make a (negative/harmful) difference that grants were received.
I think it is a shining example of modern civil society; you have the US, NL
and the EU teaming up to fund strong crypto by top notch folks from a number
of countries. Governments should fund research, applied and basic, and they
should be encouraged to fund more of it.

Somewhat tangential: Knowledge of the grants also seeks to eliminate the
idiocy in the latter two examples above. People need to be reminded that big
government is not always an evil force, governments can be a force for good. I
do not know if you saw my other comment about tor funding but tor had revenue
of \$2+ million in 2012 and 60% came from US government. I don't know where
you are from but I bet you have met a simple minded moron wearing a tea party
costume or trendy European threads that will not stop complaining about the
evil Obama surveillance administration. Blow their minds and ask them to wrap
their heads around the:

\- $800k from DoD for "Basic and Applied Research and Development in Areas
Relating to the Navy Command, Control, Communications, Computers,
Intelligence, Surveillance, and Reconnaissance"

or

\- $350k from State for "Programs to Support Democracy, Human Rights and
Labor" and "New America Foundation: International Programs to Support
Democracy, Human Rights"

    
    
      > Is the funding not transparent enough?
    

If this is in regards to the lack of numbers from NWO or the EU I am sure that
I am at fault. (I also think one of djb's EU grant numbers might have a digit
transposed) I imagine that the dutch version of nwo.nl is easier to use.

    
    
      > The blog article on OpenSSL mentions payments for consulting and
      > "features" to be added to OpenSSL.
    
      > Should I be concerned about what those features are, and who is paying
      > for them? Are you concerned?
    

I think we should be concerned that OSF is not doing a better job highlighting
sponsors and attracting new ones. It should be easier for someone with check
writing authority at big.corp.com to stumble across the sponsors information
and think to themselves "hey, we should drop some petty cash on these folks.
We use the product and I bet the marketing folks would appreciate the bump in
visibility for a fraction of the cost of our latest failed social network
branding efforts." If I was OSF I would look at the \$2 million tor brought in
and ask myself "maybe we could do a better job of sponsor outreach? Tor is
important to these people that wrote checks and tor uses libssl-dev, I wonder
if there is an opportunity?"

------
tptacek
Could someone involved with the OpenSSL Foundation and the OpenSSL project
maybe pitch in with a quick description of how the project is managed?

* Who owns which subsystems?

* Is there a board of governors or a BDFL or something like that effectively overseeing the whole project?

* What is the process for screening commits from people new to the project?

This whole post seems to be tinged with a bit of defensiveness on behalf of
the most active committers to the project. But it wasn't the active committers
who introduced this most recent bug.

------
dfc
The tor project is a great example of how open source software (OSS) projects
can work with sponsors. Trying to find more information on Qualys and PSW
Groups sponsorship of openssl is a nightmare compared to tor project
sponsors.[^1][^2] Without the tor project's emphasis on transparency and
professionalism I doubt they could post numbers like this:

    
    
      Since meeting the revenue milestones of $1,253,241 in 2009,
      $1,574,119 in 2010 and $1,681,101 in 2011, Tor has reached new
      heights in 2012 with over $2 million in revenue (unaudited).[^3]
    

My comment should not be read as a criticism of OpenSSL, it should be
interpreted as cause for optimism. The tor project has demonstrated that OSS
projects can get sponsored to solve complicated security problems that are
difficult to explain to the general public.

[^1]: List of sponsors/projects:
[https://trac.torproject.org/projects/tor/wiki/org/sponsors](https://trac.torproject.org/projects/tor/wiki/org/sponsors)

[^2]: Example monthly report for SponsorF's project:
[https://lists.torproject.org/pipermail/tor-
reports/2014-Marc...](https://lists.torproject.org/pipermail/tor-
reports/2014-March/000484.html)

[^3]: Tor Project Annual Report 2012, pg 8,
[https://www.torproject.org/about/findoc/2012-TorProject-
Annu...](https://www.torproject.org/about/findoc/2012-TorProject-Annual-
Report.pdf)

------
x0x0
It's well and fine that Stephen lives very cheaply, but all of this is an
attempt to distract from the OpenSSL project's very real issues by wearing a
cilice then bitching about it.

The fundamental facts are these: openssl contains a large quantity of code
that, if I where to check into my company's repo, I would have at best a rough
conversation with the cto and at worst I'd get fired. Plus a lack of good
tests. These combine to create more than hypothetical problems; we've seen
some severe security holes and there's almost certainly more to come.

The question that should be discussed is if openssl is, ala sendmail,
unsuitable for purpose and, if so, what should it be replaced with.

------
runn1ng
Google Cache version (site is 404ing for me)

[http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://veridicalsystems.com/blog/of-
money-responsibility-and-pride/&strip=1)

------
wbl
OpenBSD has had two holes in a heck of a long time. By contrast OpenSSL has
had a remote execute in 2010, and another 4 in 2002, and is regularly patching
DOS's resulting from memory corruption that turns out not to be exploitable.

It is 453,000 or so lines, more than five times the size of xv6. It is ten
times as big as PolarSSL. Documentation and internal structuring is wildly
inconsistent. Features that make static analysis annoying are widely used. The
API is far too low level.

Do you believe this is acceptable in a security library? Do believe that
aspiring to the security of qmail or OpenSSH is a reasonable goal, even at the
cost of features? Why should I use OpenSSL for TLS termination when formally
validated alternatives exist?

~~~
wfn
> Why should I use OpenSSL for TLS termination when formally validated
> alternatives exist?

Oh please do share! (spoiler: alternatives which enable side channels because
the pretty compiled-optimized-code (that is generated from source code that
itself may feature immutable data etc. with no side effects) is ripe with
leaks via cpu branching, caching, etc etc do not count.) This is not a
spiteful rhetorical inquiry, by the way!

~~~
AaronFriel
Those side-channel attacks are largely theoretical. OpenSSL and RSA in general
are vulnerable to side channel attacks, because many of the fundamental
operations are not constant-time. That can change eventually, but RSA is
difficult to write in a constant-time manner. OpenSSL certainly has had its
fair of lab-demonstrated side-channel attacks, but I don't think anyone has
been able to demonstrate their use in virtualized hosting environments against
arbitrary other tenants.

Side channel attacks once you've already got code running on the same
operating system as the target are much easier. But if you can get arbitrary
code to execute on the same machine running OpenSSL, you probably already have
their keys.

So, I think the criticism to OpenSSL is valid. Why use it when it seems there
are less bad alternatives? A lot of it comes down to network effects, inertia,
and licensing. That's a much better answer than side-channel attack surface
area, which all RSA implementations share :)

~~~
wbl
Nope: you can exploit the timing attacks from across a network link in AES.
DJB did this long, long ago. Furthermore, I believe OpenSSL has constant-time
RSA, but you can always use ECDSA for which constant time implementations in C
exist. (I wrote one, but I still need to switch the hash function to SHA256
and add the unnecessary encoding steps to the output, so consider it a PoC).

~~~
AaronFriel
Are you sure? Searching for "side channel OpenSSL" reveals a majority of the
attacks are against ECDSA. Of course, searches aren't the best measure of
vulnerability, it's just an indication of popularity.

~~~
wbl
[http://cr.yp.to/antiforgery/cachetiming-20050414.pdf](http://cr.yp.to/antiforgery/cachetiming-20050414.pdf)
ECDSA is nice because you can use partial nonce recovery+lattice reduction.
Furthermore recent Intel chips have AES instructions.

------
piercebot
Reading about Dr. Stephen Henson reminded me of the article written about Tarn
Adams, the creator of Dwarf Fortress.

[http://www.nytimes.com/2011/07/24/magazine/the-brilliance-
of...](http://www.nytimes.com/2011/07/24/magazine/the-brilliance-of-dwarf-
fortress.html?pagewanted=all&_r=0)

------
conductr
Not hard. Just change the license and require commercial use to be paid

