
Ruby Bug: SecureRandom should try /dev/urandom first - JBiserkov
https://bugs.ruby-lang.org/issues/9569
======
Perceptes
This is sad and embarrassing for every involved party. The people leaving
rude, entitled, hyperbolic comments in the thread, the Ruby developers who
refuse to look into the best practice suggested by experts in the field, and
the man page maintainer who refuses to update the man page in accordance with
similar information.

~~~
educar
Agreed that tensions are running high. But I think the ruby devs are in the
right here. They are just following the man page. What else is the
authoritarian source?

How is one supposed to know who wrote those blog posts? Just because it is
linuxexpert.com does not mean they are linux experts. It's funny, if people
changed it randomly following blogs then people will claim this is some NSA
conspiracy :-) How does one verify the person behind the blogs?

node has similar
[https://github.com/nodejs/node/issues/5798](https://github.com/nodejs/node/issues/5798)

~~~
binkert
The authoritative source in Linux is the code and always has been.

~~~
disposeofnick9
Code is useless if no one knows how to use it properly and it's not
communicated clearly. Users certainly can't be expected to read every line of
code. That's like shipping a car with no user manuals and saying "take the
engine apart and see how it works." Arrogance.

Clearly document system behavior or code is essentially useless.

~~~
azet
Users of languages can't be expected to. Therefore the language designers and
maintainers themselves, especially if they're working on the stdlib, should do
so, IMO. It's not only education for a proficient programmer, it helps to
understand the underlying system you're building on and it's security
assumptions.

The random char device code isn't that hard to understand, and if you're not a
strong C programmer (the Ruby-core people are good C programmers, I suppose) -
there's a paper explaining how it works:
[https://eprint.iacr.org/2012/251.pdf](https://eprint.iacr.org/2012/251.pdf)

Aaron

------
viraptor
So once again, the man page for urandom creates more problems than solutions.
([https://bugzilla.kernel.org/show_bug.cgi?id=71211](https://bugzilla.kernel.org/show_bug.cgi?id=71211))

~~~
ryuuchin
Obligatory "How To Safely Generate A Random Number" link[1] that I always wind
up posting in threads like these. There's also the getrandom syscall[2] which
uses the /dev/urandom pool.

[1] [http://sockpuppet.org/blog/2014/02/25/safely-generate-
random...](http://sockpuppet.org/blog/2014/02/25/safely-generate-random-
numbers/)

[2] [http://man7.org/linux/man-
pages/man2/getrandom.2.html](http://man7.org/linux/man-
pages/man2/getrandom.2.html)

~~~
duaneb
Why are the man pages not so clear?

~~~
ryuuchin
I'm not sure. I think part of the problem is that Linux still uses entropy
estimation. This is also one of the issues I have with the proposed
replacement[1][2] for Linux's current CSPRNG implementation since it still
retains entropy estimations.

[1] [https://lwn.net/Articles/684568/](https://lwn.net/Articles/684568/)

[2]
[https://news.ycombinator.com/item?id=11561340](https://news.ycombinator.com/item?id=11561340)
(HN thread for [1])

~~~
disposeofnick9
Btw Linux / Fortuna

[http://lkml.iu.edu/hypermail/linux/kernel/0504.1/1964.html](http://lkml.iu.edu/hypermail/linux/kernel/0504.1/1964.html)

------
Orangeair
I really don't like the sentiment of "We shouldn't change it unless the man
page says we can". That's exactly the kind of senseless bureaucracy that the
open source community should be avoiding.

~~~
derefr
The manpages for syscalls and device nodes are basically their API
specifications. They tell you what behaviours can be guaranteed (i.e. what
properties are a part of the interface contract) and what behaviours cannot
(i.e. what properties are implementation details of the current
implementation, and could change at any time.)

Just because the implementation of /dev/urandom has good properties that make
usable as a system-wide CSPRNG, doesn't mean that it's _supposed_ to have
those good properties. Only the docs can make that assertion. Without such an
assertion, the kernel is free to make /dev/urandom have less-sound properties
in a future release—because there's no spec saying they shouldn't. (They try
not to break userland code, yes, but userland code only relies on things the
docs say it can, so...)

~~~
ben0x539
> They try not to break userland code, yes, but userland code only relies on
> things the docs say it can, so...

Judging from the odd Torvalds-yelling-at-people mail getting linked on HN, it
doesn't sound like "it's ok to break userland here because clearly they didn't
read the docs" would fly.

~~~
tptacek
The proposed change --- unify urandom/random and permanently lose blocking
behavior --- can't possibly break userland. There is no way to write a program
that depends on /dev/random blocking, because it does so at random.

~~~
JoachimSchipper
I've seen you post that a few times, so two minor nitpicks.

In the mental model in which the /dev/random manpage lives, it makes (some)
sense to read /dev/random until you get a short read, then make a userland
pool from all the data you've read, "to get as much entropy as possible". That
reasoning doesn't _actually_ make sense, of course, but I would be surprised
if _nobody_ thought that such a construction would be a good idea.

Separately, there's quite a lot of code - and I've written some - that reads a
byte from /dev/random at daemon start, specifically to block until the system
has built up enough entropy. FreeBSD blocks reads, even from /dev/urandom,
until the [EDIT: estimate of how much entropy has ever gone into the pool]
gets high enough, which makes some sense; but there needs to be _some_ way to
block-until-random.

~~~
tptacek
FreeBSD blocks until the generator is seeded. I think Linux urandom should do
the same thing. But once it's seeded, there is no reason for the generator
ever to block again.

~~~
JoachimSchipper
Yes, that's what I meant. Edited for clarity, thanks!

------
wrs
To make this discussion really useful: what's the easiest way for a Ruby
application to make SecureRandom use /dev/urandom, without waiting for Ruby-
core to be convinced?

~~~
infraruby

      require "securerandom"
    
      module SecureRandom
        F = File.open("/dev/urandom")
    
        def self.random_bytes(n = nil)
          F.read(n || 16)
        end
      end

~~~
goldbrick
Why not just make the argument default to 16? Is something non-obvious going
on?

~~~
infraruby
The caller may pass nil explicitly.

~~~
goldbrick
I don't understand. You may still pass nil explicitly if the argument defaults
to 16?

~~~
infraruby
Yes:

    
    
      def f(n = 16)
        n
      end
    
      p f # => 16
      p f(nil) # => nil

------
jrochkind1
Why haven't the urandom man pages been fixed yet? Is there, like, no process
for fixing man pages at all, they just wind up set in stone forever?

~~~
zurn
If you look at the bug filed against the man page, it doesn't exactly make a
strong case. It just suggests some weasel words, about qualifying "large
amount of data" etc.
([https://bugzilla.kernel.org/show_bug.cgi?id=71211](https://bugzilla.kernel.org/show_bug.cgi?id=71211))

So bug report understates the issue, nobody has gotten around to writing a
good quality patch, everyone is just loudly complaining elsewhere.

~~~
jrochkind1
Actually, I'd say that issue/bug filed two years ago _does_ make a strong case
that the current man page is insufficient, that it's actually the current man
page that's full of ambiguous language and "weasel words".

But you may make a good point that perhaps nobody has submitted a good patch
yet -- you're right that issue isn't a good patch, just an invitation to enter
into a discussion toward one (an invitation that doesn't seem to have been
taken up, at least on the issue tracker).

One would hope that the actual maintainers of the kernel subsystems would feel
responsibility to improve a clearly insufficient man page. Which makes me
suspect there are some underlying politics or personality conflicts going on.
(Or just burn-out?)

But if all it needs is someone to write some good text -- is it really the
case that none of the cryptographers who have been writing extensive blog
posts about this for years care to submit a doc patch? If so, I wonder why?

Perhaps, just guessing, another potential issue is that nobody really wants to
_take responsibility_ for such text, in case they make a mistake. So the
existing clearly insufficient text remains. Tragedy of the open source
commons?

~~~
zurn
> it's actually the current man page that's full of ambiguous language and
> "weasel words".

The report suggests "clarifying" what the man page means by the (completely
incorrect) statement that "Users should be very economical in the amount of
seed material that they read from /dev/urandom"

Probably the reporter was just being polite, but in the absence of other
comments in the bug or any kernel developers weighing in, it just sounds like
an editorial suggestion coming from a single Linux user. Remember that the
man-pages project is separate from kernel development.

~~~
jrochkind1
Yes, it reads to me like they were just being polite. The first couple
sentences make it clear they think /dev/urandom is the right tool for "daily
tasks" and that the man page is misleading people into thinking otherwise.

They begin with the most indisputable ways the man page is insufficient, as a
way of starting the conversation -- which was never taken up.

Perhaps an actual patch would have been just accepted? Maybe the problem is
the issue-submitter assumed there was someone on the other end who understood
the kernel features and was interested in discussing making the man pages
better.

------
CiPHPerCoder
You know your language is bad when PHP -- possibly the most hated web
scripting language on the planet -- does it better.

Come on Ruby, get your act together. You can do better, right?

Also,
[https://github.com/openssl/openssl/issues/898](https://github.com/openssl/openssl/issues/898)

------
tacos
"securerandom.rb will consume too much entoropy if /dev/urandom is used
directry."

This is a quote for the ages.

------
decentrality
Regardless, whether it be OpenSSL first or /dev/urandom first, "haveged"
entropy generation is still needed for virtual machines, which is the most
common use case. In my experience both ways are a no-go without special
measures like haveged being put in place to super-charge the available
entropy. I've had processes hang up for MINUTES or just under TWO HOURS
waiting for more entropy to be available.

~~~
viraptor
That's one of the solutions, but not the only one. QEMU's solution is another
virtio device: [http://wiki.qemu.org/Features-
Done/VirtIORNG](http://wiki.qemu.org/Features-Done/VirtIORNG) I expect XEN and
others to follow soon.

Also RDRAND can be executed in a VM, but I'm not sure if it is handled
properly yet.

And like tptacek mentioned - why did you use blocking random in the app?

~~~
matt_wulfeck
Actually it's still kind of an issue. The qemu virtio device reads only from
the hosts /dev/random device (not urandom) so you can still starve the
hypervisor. Also the guest -- even though is properly seeded -- can entropy
starve because who knows what is installed on it. And clearly there's still
confusion about using /dev/random.

I don't even think the solution is to point everything to /dev/urandom. Why
maintain two? Why constantly have to explain to people the difference? The BSD
developers merged both devices into /dev/urandom and I think that's the right
approach.

~~~
viraptor
It's configurable. You can even forward host's /dev/urandom as guest's
/dev/random.

~~~
matt_wulfeck
Actually it's not supported with virtio-rng. Here[0] is a proposed patch to
add support.

[0] [http://www.redhat.com/archives/libvir-
list/2016-March/msg010...](http://www.redhat.com/archives/libvir-
list/2016-March/msg01062.html)

~~~
viraptor
I knew you can change the source (it accepts egd after all), but didn't know
they actually prevent you from choosing urandom. This is sad :(

------
technion
This whole situation has annoyed me enough to start writing this gem:

[https://github.com/technion/use_urandom](https://github.com/technion/use_urandom)

Note I said "start", so I'm aware it needs work. Feedback is appreciated
however.

~~~
matthewrudy
I'd suggest you just use the `rescue` definition from SecureRandom.

Then you don't have to do any work yourself.

    
    
        def SecureRandom.gen_random(n)
          ret = Random.raw_seed(n)
          unless ret
            raise NotImplementedError, "No random device"
          end
          unless ret.length == n
            raise NotImplementedError, "Unexpected partial read from random device: only #{ret.length} for #{n} bytes"
          end
          ret
        end

------
astrodust
Is there a good random library that's _not_ the giant ball of death that's
OpenSSL?

~~~
yarou
I've used Mersenne Twister[0] in the past.

[0]
[http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html](http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html)

If you're talking about an RNG library for cryptographic purposes, then it
depends on your use case.

~~~
tptacek
It _does not_ depend on your use case. The answer, in all cases, is just to
use /dev/urandom.

~~~
sibrahim
I had a rather specialized case where it was the pragmatic choice (note, not
technically required): running a lottery with potentially litigious losers.

If you used a CSPRNG with a seed space smaller than the set of possible
lottery outcomes, losers could argue (misleadingly, since we still couldn't
feasibly bias the result) that not all outcomes were equally probable and try
to get the results thrown out. That is, the fact that there are widespread
misconceptions about /dev/random can very rarely be a reason to use it :P

However, I agree that the rule is that you should just use /dev/urandom.

~~~
Thiez
Why not use a hardware rng in that case? Seems a lot safer if you have to deal
with litigious people.

~~~
sibrahim
We did, in a way. One of the sources used was random.org (uses radio receivers
tuned to static from atmospheric noise: hardware RNG as a service). I also had
less than 3 weeks to take it from proposal to production.

Combining two independent sources obtained by different people and using a
cryptographic commitment scheme ensured that 1) no one person could fix the
results or make it nonrandom (protection against Eddie Tipton-style attacks),
2) if at least one of the independent sources was random, the result would be.

------
Freaky
JRuby uses Java's java.security.SecureRandom, which (by default on OpenJDK 8)
uses /dev/random, mixed with its own SHA1PRNG (this so setSeed() can be
guaranteed to actually do something).

Rubinius copies MRI circa 2014 via rubysl - [https://github.com/rubysl/rubysl-
securerandom](https://github.com/rubysl/rubysl-securerandom)

~~~
viraptor
Does it really? From
[https://bugs.openjdk.java.net/browse/JDK-4705093](https://bugs.openjdk.java.net/browse/JDK-4705093)
:

> If you call: new SecureRandom() on Linux and the default values are used, it
> will read from /dev/urandom and not block. (By default on Solaris, the
> PKCS11 SecureRandom is used, and also calls into /dev/urandom.)

~~~
jcranmer
I ran strace on both OpenJDK 7 and 8. It appears to open both /dev/random and
/dev/urandom, but it only read from /dev/urandom. From reading the source
code, it appears that /dev/random is used if you call
SecureRandom::generateSeed, but /dev/urandom is used for SecureRandom::next.

Of course, you can change a whole heck of a lot of stuff by setting properties
on the command line or changing $JAVA_HOME/lib/security/java.security.

~~~
Freaky
Yep, shouldn't have second-guessed myself. Teach me to comment at 4am.

[http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/s...](http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/solaris/classes/sun/security/provider/NativePRNG.java)

------
paloaltokid
I was recently told about a method to generate a random number using something
called "quadratic residue". There was too much math for me to understand it.
Is anyone familiar with this?

------
jasonjei
[deleted/retracted]

~~~
jrochkind1
Yes, it's just you.

The core ruby team is mostly Japanese-speaking, and Japanese developers are
less likely to speak English well even than people in a lot of other countries
where English is not a common first language. I think that does serve as a
barrier to communication and transparency (with/towards English-speakers!). I
think that's all there is, nothing "about Asian culture", or "decisions are
made as a group rather than on an individual basis," or cliquishness.

We English speakers have the luxury of seeing almost all open source
development happen in our first language. Most of the world doesn't. Ruby is
an exception, with a mostly Japanese core team. Just how it is.

~~~
jasonjei
Thanks for the enlightenment, I didn't know this aspect of the Ruby core team.

~~~
jrochkind1
Yeah, I too find it hard to figure out exactly who the ruby committers are, or
what goes on amongst them. I believe this is simply because of that language
barrier. From what I do know, the way MRI is developed is not at all atypical
or unusual compared to most other large open source (primarily English-
language) projects.

Here's a useful (to those curious to learn more about ruby committers)
interview from 2013 with 15 ruby committers, note the questions and answers
have been translated between English and Japanese.
[http://www.sitepoint.com/meet-fifteen-ruby-core-
committers/](http://www.sitepoint.com/meet-fifteen-ruby-core-committers/)

My impression is that a lot of discussion on MRI happens in Japanese on
Japanese listservs. Of course, it doesn't take a language barrier to make
things seem not so transparent -- for comparison a few years ago, Rails core
team started taking the majority of their discussion to private core-team-only
listservs. (I gather because they were finding dealing with the peanut gallery
made it too hard to get things done/decided). At least there's still lots of
open (and English-language) communication on the github issues or open
listserv, but there are often times when I'm not really sure who or how or on
what basis architectural decisions are made for Rails too, a lot is done in
private, is my impression.

But I suspect to curious Japanese-speakers, MRI development may even be _more_
transparent than Rails is to English-speakers! I think (but not entirely sure)
all of the listservs MRI devs use to collaborate are actually open -- you just
have to read/write Japanese to understand what's there or engage effectively.
:)

------
meshko
Well, I dunno what the man pages say and what some dudes on the internet say,
but I know that on CentOS release 6.7 (Final) I need to run with
-Djava.security.egd=file:///dev/urandom or else I can't connect to Oracle half
the time because Java runs out of entropy. So there must still be some
difference between /dev/random and /dev/urandom.

~~~
ascorbic
Nobody's claiming there's no difference between /dev/random and /dev/urandom.
They're just saying you should always use /dev/urandom.

~~~
meshko
Not sure what up with all the downvotes. I'm just saying that Java defaults to
/dev/random and Java is kind of a big and important and all that, so not sure
why everyone is giving crap to poor little Ruby about this.

