
Npm's Self-Signed Certificate is No More - rcconf
http://blog.npmjs.org/post/78085451721
======
diminoten
There's a comment on the blog post that I thought was very good, it was from
Rob, and it ends with:

"Please, try harder or get forked. Not sure how else to say that."

~~~
cjbprime
Seems to have been moderated away by the blog owners. Classy. Here's the
comment under discussion:

==

Hey. Crazy kids. This probably needs to be that one event where you sort of
realize: "Oh. Shit. Other people...like...use this & stuff. We need a damn
road map and a release schedule. Stop smoking dabs all day, breh."

Now is also a great time to learn how to think about the potential
ramifications a production push will have prior to making said production
push. And, if your change might impact some or perhaps even all of the other
people who use your technology, then some degree of coordination - perhaps an
email? - would be nice. It's one of those things that will help make you look
professional. I suck at professionalism. You have no idea. But, even I know
this much.

Because, right now, I sort of feel like I'm asking some very rightfully
fearful people to consider entrusting perhaps their actual career into the
development of technology they need to succeed and thrive. And, I just started
recommending Node.js - with a caveat - that npm basically sucks. I hate having
to do that and it needs to stop.

So, here we are.

Your words continue to be one thing, and your actions continue to be quite
another. If it is even possible to break a tool like this, that tool is not
enterprise grade. If there is nothing that can be done to successfully
insulate a tool from unexpected behavior like this, then that tool scores less
in evaluations that consider the _risk of using it_.

npm, at this point, has more going against it in the discussion than going for
it right now. Events like this are, in the grand context, very significant and
telling. They are also ill-timed. Because big, important decisions are trying
to happen right now regarding the use of Node.js. It is literally on the cusp
of going mainstream. And, that seems to be generating some pressure that at
least one team (npm) doesn't seem to be equipped to handle.

So, before you find yourself facing a community that forks instead of trying
to work with you, I would like to just make a simple recommendation. In the
future, you seriously need to sit and think about the potential ramifications
of a production push. And...this is the important part...if those changes are
going to have a wide impact on your users - send some sort of email WELL IN
ADVANCE. A flippant blog post the day of is not Doing It Right™.

Because, and I feel like I might not only be speaking for myself, I'm not
going to allow the promise of Node.js to be voided by the lackluster and
problematic performance of its weird bolt-on archive service. Someone, perhaps
even me (as in: today), will simply replace you with a workable, decentralized
solution that enterprise can specialize to purpose and communities can use to
grow and thrive.

If you have any questions, ask somebody. Anybody. If you're struggling with
some concept of enterprise grade operations, what people expect of you and how
you can succeed with events like this in the future, I'm positive every
capable person here would provide some level of guidance and support. We want
you to succeed.

Please, try harder or get forked. Not sure how else to say that.

Best regards, -Rob

~~~
IsaacSchlueter
We didn't moderate away anything. I am literally the only person who CAN
moderate those comments, and I was at a conference all day. 100% of my online
time was spent working with my team to figure out the fastest path to a fix.
We didn't realize the extent until way too late, and that's bad on us. I
apologize.

I didn't delete your comment. I'll look at the moderation queue and see if
maybe disqus is set to auto-hide after some time or something. I'm sorry for
the confusion there.

------
emily37
If anyone is wondering what the actual change was:

It looks like the npm registry used to have a certificate signed by npm's own
CA, and existing npm clients only trust that CA by default, not the normal
list of verisign, digicert, etc. (Trusting versign et al would defeat the
point of using their own CA.) The signing key for that CA is pretty darn
important, and maybe there are entities other than npm, inc who might know it
(i.e. nodejitsu).

So npm, inc rolled out a new cert that looks to be signed by digicert, but
existing clients don't trust Digicert until you explicitly configure them to.

I was thrown off by the SELF_SIGNED_CERT_IN_CHAIN error; I expected some error
about an untrusted root CA if the problem was that npm clients didn't trust
digicert, but apparently SELF_SIGNED_CERT_IN_CHAIN is what OpenSSL returns
when the root CA isn't loaded.

~~~
mivok
If the clients trust the npm CA, can't they just sign the digicert CA with
that CA and include it in the certificate chain provided by the server? That
way the chain would be:

    
    
        npm CA -> digicert CA -> any other intermediates -> server cert
    

Clients that only trust the digicert CA (and other standard CAs) will see that
and accept it because they trust the digicert CA, and clients that trust the
npm CA will trust the cert also, allowing both old and new clients to work.
Once (almost) everyone has upgraded, the npm root CA can be removed from the
chain presented by the server. Am I missing something here?

Edit: It looks like what I'm missing is that you'd need the private key of the
digicert CA to generate the request to sign with the npm CA. I was thinking
about how CAs have been migrated in the past (e.g. equifax to geotrust global
CA). It looks like it won't work in this case.

Edit2: Actually, it appears to work after all. I just tested with the openssl
ca command, and you give it -ss_cert instead of -in for the certificate to
sign a certificate instead of a request.

~~~
teacup50
> _Am I missing something here?_

A sense of arrogance that precludes understanding x509 infrastructure before
you roll out a world-breaking change.

~~~
akerl_
Lets apply a bit of sense here: this was a failure of judgement, not
arrogance. It's perhaps easier to picture the npm developers as maniacal
villains, cackling as they wield destruction among us. But that's not the case
with them, just as it is pretty much never the case with project developers.

~~~
iends
I just picture them as cowboy coders not really aware of what it takes to
build and maintain software for large enterprises, which is, unfortunately,
their stated mission.

~~~
akerl_
Let any developer who has never pushed an update with unintended side effects
raise their hand.

This mistake was, in hindsight, a clear error in judgement. It highlights
missing steps in their change deployment process. And I expect them to learn
from it, as the larger Node community has shown they can learn from mistakes.

Part of joining the ranks of "enterprise"-grade projects is first being an
aspiring project, and part of that is learning a lot. Anyone who expects that
to happen without a few bumps is naive.

~~~
darklrd
I agree with you. Everyone makes a mistake once in a while. To be successful,
you should learn from it.

------
rcconf
This has also broken all of our own deploys on Jenkins. We had to use solution
2, and then upgrade, because solution 1 also produces the certificate error.

It was surprising that npm status didn't have any sort of warning. I had to
find out about this via twitter. I think it's extremely irresponsible and I'm
glad we've started to move away from node.js and started to use Java.

~~~
benajnim
That's not a loaded comment.. Changing your programming environment over an
SSL certificate? Tell us all about how awesome it is building apps in Java!

~~~
awj
You want to talk loaded comments? You're reducing "I'm glad to move away from
a project that breaks functionality for all users with no notice" to a quibble
over an SSL certificate change.

That npm as a project thought this was the correct way to handle this kind of
transition is _very_ unsettling.

~~~
1stop
Yeah, because no one ever had any problems with Maven, that thing is like
package heaven, it's all rainbows and unicorns!

~~~
Pacabel
It's really disappointing to see this sort of "reasoning" used so often these
days.

Just because somebody points out serious flaws or problems with one technology
does not automatically mean that he or she thinks that other, unrelated-yet-
similar technologies aren't flawed or are somehow perfect.

~~~
1stop
> I'm glad we've started to move away from node.js and started to use Java.

That is what I was responding to directly. The implication from the statement
was that npm flaws (i.e certificate changes that break everything) is a good
reason to switch to Java.

It isn't at all a good reason to switch to java, as java's equivalent would
easily waste more time than even this rather embarrassing certificate problem
with npm.

> unrelated-yet-similar technologies

It is related, and similar, it is a like-for-like comparison between nodejs
package management and java package management.

~~~
teacup50
> _It isn 't at all a good reason to switch to java, as java's equivalent
> would easily waste more time than even this rather embarrassing certificate
> problem with npm._

Works fine for me. Has a healthy ecosystem of 3rd-party artifact repository
implementations.

------
RyanZAG
Probably goes down as one of the worst large scale blunders of this type given
the sheer amount of people affected. It's actually fairly insane that node.js
relies on npm like this - isn't it only a matter of time before one of the
core node packages gets compromised and someone gets root access to thousands
of servers and dev boxes?

~~~
jdlshore
Although npm ships with node, the problems aren't because of that.

People have (foolishly, in my opinion) chosen to make npm an integral part of
their deployment process, which is why this change has broken a lot of
people's deployments. They're going against the official npm recommendation
[1], which is to check your dependencies into your source repository and _not_
use npm in deployment scripts. (A good idea with any package manager, imo.
[2])

Not that I'm excusing npm; a change like this seems like something they should
have taken more carefully.

[1] Npm recommends checking product dependencies into your repo:
[https://npmjs.org/doc/faq.html#Should-I-check-my-
node_module...](https://npmjs.org/doc/faq.html#Should-I-check-my-node_modules-
folder-into-git)

[2] I explain why checking dependencies into your repo is a good idea:
[http://www.letscodejavascript.com/v3/comments/live/2#comment...](http://www.letscodejavascript.com/v3/comments/live/2#comment-819812506)

~~~
garthk
As an alternative to checking in your dependencies: have your build server
'npm install' your module for testing, then archive the directory tree. Deploy
from the archive.

------
atonse
I'm surprised that this wasn't one of those "hey in 60 days we'll be making
this change and it will break x, y, and z. Be prepared." sort of things.

------
IgorPartola
Doesn't matter. When you download Node.js itself, you are doing so over HTTP.
[https://nodejs.org/download/](https://nodejs.org/download/) (note the httpS)
does not work. Hope you enjoy being a part of that botnet :)

Edit: I used to email project owners about issues like this, but never seem to
get any response, so I stopped. The worst one, in terms of losing $$ was
Pandora. I keep trying to sign up for the paid service, but the form to put in
your credit card details is served over HTTP. I emailed them several times
with no response.

~~~
chill1
If you are so concerned about downloading anything via HTTP, why not always
use a VPN? At least that way you aren't susceptible to local MITM. Just make
sure you trust your VPN :)

Edit: Oh, plus you can always use a checksum to verify your download packages
[1] :)

[1]
[http://nodejs.org/dist/v0.10.26/SHASUMS.txt](http://nodejs.org/dist/v0.10.26/SHASUMS.txt)

~~~
handsomeransoms
> why not always use a VPN?

Because then you're susceptible to remote MITM? VPNs are useful but do not
replace end-to-end encryption.

~~~
chill1
That's why I qualified it with local. Have anything to say about checksums
too?

~~~
lmm
A checksum that's published via HTTP doesn't protect you against MITM.

------
randunel
When package managers misbehave, the whole software bundle suffers. Maybe the
Node.JS project should have more (decision) power over its package manager.
Node maintainers properly announce api version changes sometimes months in
advance, so you don't need to read through the pr/issues on github or mailing
lists to stay up to date.

I really don't think an advance notice would have hurt. So much for
automation.

~~~
agilebyte
Well, nodejitsu are trademarking NPM so let's see how that plays out.

------
pi-rat
Go home NPM, you're drunk!

(add all the latest drama, and you're basically just asking to be forked..)

------
parris
Solution 1: Upgrading npm actually does nothing and also complains about a
cert error.

Solution 2: is a terrible idea for people who have their own private
registries or proxy caches. Also a security issue in general...

This leaves you with needing to upgrade node, which may or may not be possible
due to operating system/platform constraints. Luckily we were using this
[https://launchpad.net/~chris-
lea/+archive/node.js/](https://launchpad.net/~chris-lea/+archive/node.js/)
which makes life on ubuntu ok.

We set up our own proxy cache, at this point im thinking about increasing the
cache time to something like 2 weeks.

------
yogo
The inmates are running the asylum :)

But it does provide a case study for the rest of us on what or what not to do.

------
thirsteh
Did they previously have client-side validation of the self-signed
certificate, and now they changed it to "accept any cert signed by a root CA"?
If so, why in the world would they do that?

Maybe I'm being hopelessly optimistic by not jumping to the conclusion that
they did no validation whatsoever before?

~~~
awj
If running "npm config get ca" on my not-yet-updated copy of npm is any
indication, they were using the self signed certificate.

------
ibash
As per Joe Grund: [http://blog.npmjs.org/post/78085451721/npms-self-signed-
cert...](http://blog.npmjs.org/post/78085451721/npms-self-signed-certificate-
is-no-more#comment-1264542895)

npm config set strict-ssl false npm update npm -g npm config set strict-ssl
true

~~~
awj
That's ... still a dangerous way to accomplish the update. You're disabling a
key safeguard that ensures you're actually talking to official npm servers,
then re-enabling the safeguard after running the code that whoever-you-got-it-
from provided.

I'm sure the likelihood of experiencing a MITM attack here is low, but the
security consideration needs to be as widely advertised as the solution.

------
imslavko
Can someone explain what they actually did? What was the technical change that
was made?

I can't figure out from the blog post whether the change was in the npm server
or client.

------
iLoch
FYI I was able to bypass this last night by temporarily switching to the EU
NPM mirror.

    
    
        npm --registry http://registry.npmjs.eu/ install blah

------
purephase
I was bitten by this yesterday and I didn't know WTH was up. This is good to
know. I had two or three package installs work flawlessly and then they
started failing randomly across servers (all staging, so it wasn't that big of
an issue).

I think Rob's comment sums up my feelings. A bit more aggressively, but the
same gist.

------
2mur
Basically don't ever deploy from npm. Tarballs

~~~
darkarmani
This is exactly why I don't understand why it is called "package management."
What good is NPM if you just need to deploy tarballs anyway? It's good for
bloating dependencies and installing 10 of the same library. It completely
punts on actually managing packages, by just copying everything --
recursively.

~~~
ahallock
npm prune

------
andrewmunsell
This is also causing an issue with Dokku-- I'm trying to deploy a new Node app
and it fails because of the certificate issue. Anyone know how to update the
Buildpack to the latest version? Heroku already implemented a fix.

~~~
agilebyte
Yes I do :)

You can see on the following step which image is being used when deploying an
app:

[https://github.com/progrium/dokku/blob/master/dokku#L33](https://github.com/progrium/dokku/blob/master/dokku#L33)

So what I did was change that to an image of my own based on the
progrium/buildstep one:

    
    
      sudo docker run -i -t progrium/buildstep /bin/bash
    

Now you can make changes to it and save them into your new image (while you
are still logged in to progrium/buildstep):

    
    
      sudo docker commit <id> myownimage
    

OK, and inside this image I have referenced my own buildstep instead of the
default Heroku one. The relevant txt document is in /build/.

And to get to the point I have changed the npm install line to run pointing to
the http EU repo.

    
    
      npm install --registry "http://registry.npmjs.eu" --userconfig $build_dir/.npmrc --production 2>&1 | indent
    

That is one way to go about it.

~~~
andrewmunsell
Great, thanks. I also figured out that I could put the following line into a
file called `.env` (I was originally trying to use a file named `ENV`):

    
    
        export BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-nodejs
    

This will use Heroku's version of the build pack, which already has a fix for
the NPM problem.

------
STRML
FYI, npm is rolling back to an older cert that they used a few years back.
It's a GlobalSign cert, and the GlobalSign CA has been in npm since Aug 2012.
When it propagates their global CDN, old clients will be fixed without
breaking new ones and they can focus on a more permanent solution.

1\.
[https://news.ycombinator.com/item?id=7323093](https://news.ycombinator.com/item?id=7323093)

------
justinfreitag
WTH "We are rolling back to the older cert now, but since the registry is
distributed by a global CDN this process is slower than we’d like, and we
don’t want to break things (further) by rushing the process."

[http://blog.npmjs.org/post/78165272245/more-help-with-
self-s...](http://blog.npmjs.org/post/78165272245/more-help-with-self-signed-
cert-in-chain-and-npm)

------
jevinskie
What broke? Running the npm command? Running any nodejs program? The blog post
gives few details about what people were doing when they hit the issue.

~~~
mmorris
You can't _npm install <foo>_ without getting an error message about the self-
signed cert. At least until you update npm, which you usually do with _npm
update -g npm_ , but that will also fail with the same error. Fun!

The blog post doesn't mention deleting the config change after updating, which
(I'm not an expert) might be important. I think this is a full fix:
[http://stackoverflow.com/a/22099006/2708274](http://stackoverflow.com/a/22099006/2708274)

~~~
jevinskie
Did the npm servers previously serve up packages with the self-signed cert?

~~~
mmorris
Yes.

------
janantala
Broken builds, broken builds everywhere...

~~~
tracker1
Not here... cache proxy internally.

------
automatthew
The npm source is crazy. Here's my favorite bugfix (bug being that the
prepublish script simply wasn't running):

[https://github.com/npm/npm/commit/2389525a1df6d20e780cca5887...](https://github.com/npm/npm/commit/2389525a1df6d20e780cca5887f1b73ff583ce8d)

------
stevepike
Wow, this explains so much. I had to download the coffeescript compiler last
night to fix a bug in a library I'm using and hit this error running npm
install. I didn't even think to consider that npm itself was broken. I guess
it's not always user error!

------
deelowe
Welp that didn't take long. I sure hope someone (who is more skilled than I)
forks NPM so we can all move on from this craziness. It's getting pretty sad.

------
novaleaf
still broken. following the instructions on windows, I am getting:

D:\repos\js-master\phantomjscloud-backend>npm install npm -g --ca=""

npm http GET [https://registry.npmjs.org/npm](https://registry.npmjs.org/npm)

npm http 200 [https://registry.npmjs.org/npm](https://registry.npmjs.org/npm)

npm http GET
[https://registry.npmjs.org/npm/-/npm-1.4.4.tgz](https://registry.npmjs.org/npm/-/npm-1.4.4.tgz)

npm ERR! fetch failed
[https://registry.npmjs.org/npm/-/npm-1.4.4.tgz](https://registry.npmjs.org/npm/-/npm-1.4.4.tgz)

npm http GET
[https://registry.npmjs.org/npm/-/npm-1.4.4.tgz](https://registry.npmjs.org/npm/-/npm-1.4.4.tgz)

npm ERR! fetch failed
[https://registry.npmjs.org/npm/-/npm-1.4.4.tgz](https://registry.npmjs.org/npm/-/npm-1.4.4.tgz)

npm http GET
[https://registry.npmjs.org/npm/-/npm-1.4.4.tgz](https://registry.npmjs.org/npm/-/npm-1.4.4.tgz)

npm ERR! fetch failed
[https://registry.npmjs.org/npm/-/npm-1.4.4.tgz](https://registry.npmjs.org/npm/-/npm-1.4.4.tgz)

npm ERR! Error: CERT_UNTRUSTED

npm ERR! at SecurePair.<anonymous> (tls.js:1349:32)

npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)

npm ERR! at SecurePair.maybeInitFinished (tls.js:962:10)

npm ERR! at CleartextStream.read [as _read] (tls.js:463:15)

npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)

npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)

npm ERR! at doWrite (_stream_writable.js:219:10)

npm ERR! at writeOrBuffer (_stream_writable.js:209:5)

npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)

npm ERR! at write (_stream_readable.js:573:24)

npm ERR! If you need help, you may report this log at:

npm ERR!
<[http://github.com/isaacs/npm/issues>](http://github.com/isaacs/npm/issues>)

npm ERR! or email it to:

npm ERR! <npm-@googlegroups.com>

~~~
novaleaf
also tried on my debian 7 server. broken there also.

------
AdrianRossouw
i really don't understand why they would make a server change that would
orphan all the existing clients.

If anything, roll out a new version and gracefully drop support for other
clients.

