
Npm operational incident, 6 Jan 2018 - bradleyboy
http://blog.npmjs.org/post/169432444640/npm-operational-incident-6-jan-2018
======
tedivm
> Unfortunately, the process was complicated by well-meaning members of the
> npm community who believed that a malicious actor or security breach was to
> blame and independently attempted to publish their own replacements for
> these packages. Ensuring the integrity of the affected packages required
> additional steps and time.

That is such a bad response to this.

The problem isn't that "well-meaning members of the community" decided to
upload packages. The problem is that when their system decides that a package
shouldn't be up it completely removes the package, as if it never existed, and
allows the namespace to be reused immediately. Those "well-meaning members"
should not even be able to hijack packages this way, as it means the people
who aren't "well-meaning" can also do it.

What should happen is that they block downloads of the package while they
investigate. That way people who attempt to download the packages get a
meaningful error and people are unable to hijack the package name.

~~~
micaksica
It's been literally years since node-forward got its talk about signing
packages [1] with a lot of pushback from the npm team. Every time a new
typosquatting article shows up, there's some more waffling by npm. left-pad
happened to much consternation. Now this.

I used to really care about trying to harden the Node ecosystem, and last year
it was one of my main goals. I tried to send multiple vulnerability reports,
do mass static analysis of npm packages, and wanted to contribute more to the
ecosystem, but the consistent ambivalent reactions of much of the community
that I talked to turned me off of the project entirely. If npm wants to
continue to be a security dumpster fire, let it burn. Node is a waste of
security researchers' time and an honest goldmine for black hats looking to
compromise relatively powerful novice webdev hardware.

I don't see it changing anytime soon. npm is a business that isn't focused on
security. These things keep coming up, and yet npm install metrics I'm sure
aren't decreasing. Until they face meaningful competition and/or the rest of
the Node community begins to give even half a care to security outside of this
forum, there will be no incentive for anyone to do anything about it. It's
easier to play PR, give a little lip service to it and dodge the problem than
it is to add any friction to their potential growth.

[1] [https://github.com/node-
forward/discussions/issues/29](https://github.com/node-
forward/discussions/issues/29)

~~~
ubernostrum
This is your occasional reminder that package signing is not a panacea, and as
typically proposed for community package repositories like npm, PyPI, etc.
would likely do absolutely nothing.

For example, people often insist in the Python world that PyPI should support
package signing. But _it already does_ \-- you can generate a signature for a
package and upload the signature with the package. Django does this, and has
been doing it for years. You can also get package download/install tools that
will check the signature. But then what?

What people really _mean_ when they say there should be "signed packages" is
that there should be a whole bunch of invisible infrastructure (set up by...
who, exactly? Maintained by... who, exactly?) to decide which PGP keys are
authorized to sign releases of which packages. And that's close to an
intractable problem for an anyone-can-contribute community repository like npm
or PyPI.

~~~
tragic
This is a very important point. I work for a company that publishes client
libs for many different package indexes (although not npm). This is a fairly
well automated process, but it takes minutes (if that) to push a new version
to pypi, rubygems etc, but at least a few hours of fiddling about to get
something on maven, which of course has this security infrastructure.

An analogy might be drawn with the app stores. We all know it is massively
easier to get stuff in the play store than the iOS store. We all know there is
a shit ton of spam, malware etc on the play store and not really in the other.
But it's also much easier to contribute to. It's a trade off. Security is
important, but sometimes I feel that people are unwilling to treat it as an
input in a basic cost benefit analysis, instead turning it into a kind of
absolute value. I accept that it is not treated seriously enough by many in
the community, but overcorrection is not the answer.

Of course, other relatively 'open' package indexes exist that do not have
npm's typo squatting issues, so there are other design issues at work in this
particular case.

------
Lazare
Very displeased about this response.

> In yesterday’s case, we got it wrong, which prevented a publisher’s
> legitimate code from being distributed to developers whose projects depend
> on it. We identified the error within five minutes and followed defined
> processes to reverse this block. Unfortunately, the process was complicated
> by well-meaning members of the npm community who believed that a malicious
> actor or security breach was to blame and independently attempted to publish
> their own replacements for these packages.

No. Assuming everything in that excerpt is true (and I happen to know it's
not, but that's not even relevant here), that _wasn 't the problem_.

The problem is that NPM allowed packages to be re-uploaded by new authors
after the initial versions had been spam filtered. Especially since allowing
packages to be re-uploaded by new authors was the core issue of the left-pad
debacle, and the _one thing_ NPM said they'd fixed in response.

Let's summarise here:

1\. NPM has a big issue

2\. They claimed they had fixed it

3\. They had not

4\. In their post mortem they're pretending the issue doesn't exist

This guts any remaining trust I had in npm. Even if I wanted to trust them,
they're not even admitting the problem exists; how am I meant to believe
they're finally going to fix it? They've stopped even _promising_ to fix this,
and moved on to lies and denial.

Unacceptable. Literally. This is pushing me away from the node ecosystem
because I am not prepared to accept this sort of weaponized incompetence from
the primary package repo for node.

~~~
pwaai
With all that money they've raised, it's puzzling why they don't take PR
seriously. I guess they have a mentality of "where the fuck else are you gonna
go, the node community is tightly coupled to our service"

~~~
noobermin
It's almost as if the tech community has to learn once again centralized
systems with a trusted authority at the top can't be trusted when that trusted
authority makes mistakes.

~~~
jonny_eh
Honest question: what's the alternative?

~~~
Lazare
I've mentioned this elsewhere in this discussion, but:
[https://github.com/ramasilveyra/gitpkg](https://github.com/ramasilveyra/gitpkg)

I've been using it a bit, and it's really nice. Yarn (and the npm command line
tool) already provide nice methods to install packages _from_ arbitrary
locations, including git repos; this provides a nice mechanism for publishing
_to_ git repos. It's a super simple solution, but then, it's not really a hard
problem.

------
jchw
Very disheartening to see that NPM has not grown from the kik or left-pad
incidents. Users should not be able to republish on top of old package names
without some kind of intervention.

Of course, NPM's response to the kik/left-pad problem was also pretty awful.
Make it so users can't delete packages. Cool. For those of you using NPM's
private offerings, this also applies to you, so hope you don't care about
cleaning up your private NPM registry content.

Getting pretty tired of this. Their open source operation seems to suffer from
poor handling of community and technical issues, at least from a high level.
Their private registry operation is very lean on features, and also suffers
from very confusing limitations. I'm surprised at how long it took there to be
read-only API keys. Until last year, you literally had to give your CI keys
that could publish to your organization if you wanted access to private
packages. And you had to pay for an extra user for the privilege of doing it
wrong. It's fixed now but it still blows my mind that it took so long. Aside
from faster installation, it's actually a lot better to just use private Git
URLs instead of NPM's private offerings.

NPM will never properly take the fall. They will take just enough blame to
seem responsible and then shift the majority of the poor decision making onto
other actors in the community in their explanations. Third time in recent
memory.

------
Shank
> no malicious actors were involved in yesterday’s incident, and the security
> of npm users’ accounts and the integrity of these 106 packages were never
> jeopardized

Maybe not in the incident itself, but the sheer fact that many of the packages
were replaced by other people constitutes a jeopardization of _applications_
that depend on NPM. The only reason why some big package didn't get replaced
with code that exfiltrated data from production or subtly backdoored it is
sheer luck.

~~~
no29
it's beyond disingenuous and flat out dishonest for npm to say the integrity
of the packages was never jeopardized. within minutes there were reports that
packages had been replaced with questionable and malicious content.

[https://news.ycombinator.com/item?id=16087079](https://news.ycombinator.com/item?id=16087079)

this is probably a good indication of how a much more serious security event
would be treated by the organization.. just sayin

~~~
mannykannot
At the very least, Npm should explain why we can be sure the packages now
offered are exactly the same as they were before the incident, if, in fact, it
is in a position to make that claim.

------
0x0
How can they claim no malicious actors were involved when packages such as
duplexer3 were apparently replaced with undesirable code as reported in
[https://news.ycombinator.com/item?id=16087126](https://news.ycombinator.com/item?id=16087126)
?

~~~
akerro
This is not a claim or a fact, that's probably an uneducated guess, npm is
broken on so many levels they can't say anything for sure. [1]

npm is full of fundamental bugs in their software, workflows, architecture,
and community. We learnt this in March 2016 when one developers got a
copyright claim and had to remove their package. [2] Why is it even possible
to remove a package? Why is it possible to create new package with the same
name? Almost no software published there have unit tests, asking developers
how they tested it in a GitHub issue leaves the issue open with no response.
Everything is centralized on GitHub, which was proven to fail many times when
GitHub was down in a region or globally due to a DDoS. Why is Node allowed to
exist? We're not in 1998 anymore.

[1]
[https://news.ycombinator.com/item?id=16092584](https://news.ycombinator.com/item?id=16092584)
(comment above)

[2]
[https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/](https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/)

~~~
detaro
At least get the company name right if you have to make guesses about what
they know internally and what they don't. (lol, and edited it silently so my
comment looks dumb now)

~~~
akerro
>We don’t discuss all of our security processes and technologies in specific
detail for what should be obvious reasons, but here is a high-level overview

> they know internally and what they don't.

They know internally that they do a lot of things badly and this will result
in more disasters for npm and JS communities, that's for sure. Check out how
GitLab fixed their backup failure. They made new software for testing backups,
they live-streamed their work on youtube, they made at least 2 write ups on
this case. They were totally transparent about their mistakes and fixes,
everyone applauded them for this. What did we get from npm?

------
cjbprime
The last paragraph is good! The other paragraphs are bad. The two main ways
they are bad are:

1) A system that detects "spam" and then allows for the complete removal of
packages as if they never existed, allowing anyone to replace them, should
never be described in the neutral terms used in this post. This system appears
to be an existential threat to the company and project, and one of its largest
mistakes. It won't take a long investigation to figure that out; it should be
obvious today.

2) The claim that the security and integrity of these hijacked npm package
names was not jeopardized appears to be 100% pants-on-fire false. If it is not
false, I think npm's users are owed an explanation today of why it's false,
rather than a bare assertion.

I know it's hard to be in the hot seat. No animosity to any of the humans
involved.

------
flavio81
It is really another embarrasment for NPM, considering that this is not the
first time something similar happens (see: left-pad issue, 23 march 2016)

So they learnt zero from it.

Then, _WHEN ARE THEY GOING TO SIGN THOSE PACKAGES?_ Zero, nothing. They didn't
learn from mistakes, they don't listen. NPM is still open to all sorts of
malicious use.

Besides NPM problems, yesterday many packages won't work because package
"pinkie-promise" wasn't available. This is the full effective source code of
pinkie-promise:

    
    
        module.exports = typeof Promise === 'function' ? Promise : require('pinkie');
    

Not just a NPM fault, but a fault of the js community as a whole for accepting
systems made from hundreds of one-line packages, a sort of spaghetti code for
the modern era.

~~~
yoodenvranx
Someone should grab the 100 most used packages which contain less than 30
lines of code, merge it into one package and call it 'npm standard library'.

~~~
krapp
Guess I'll be that guy: [https://xkcd.com/927/](https://xkcd.com/927/)

~~~
i_am_nomad
I’ve gotten to the point where I recognize certain XKCD urls and no longer
have to click on them to know which one it is. The ol’ 927, I see.

------
Sephr
> who believed that a malicious actor or security breach was to blame

This _was_ a security breach. Their anti-spam system should block repos by
freezing the module name and returning blank files, not by deleting the entire
module and subsequently allowing anyone to upload new modules. This is leftpad
all over again.

Update: Woah, so I was checking out my old NPM namespaces and apparently
someone took control over
[https://www.npmjs.com/package/filesaver.js](https://www.npmjs.com/package/filesaver.js)

~~~
djsumdog
It's not a "breach" technically. No malicious person caused the packages to be
removed by circumventing authentication/authorization systems. It is a
security issue though. A bug and serious security issue, yes, but not
technically a breach.

~~~
jeremyjh
A breach didn't cause the issue, but there absolutely was a breach in response
to NPM's incompetence.
[https://news.ycombinator.com/item?id=16087126](https://news.ycombinator.com/item?id=16087126)

------
alexkavon
I think it's time to replace npm in default Node installs. They've shown a
history of negligence for package handling and procedure. Why should Node
continue putting the trust of such a substantial part of not only their
ecosystem but the JS ecosystem as well into such unreliable hands?

~~~
sirius87
the official node.js Docker image comes with yarn as well as npm. but this is
an issue with the npm registry, not the package manager shipped with node.
both yarn and npm use the npm registry.

EDIT: it may be worth debating why npm (a binary produced by npm, Inc) is
shipped with node.js at all, but that's upto the node foundation

~~~
tatersolid
I’m pretty sure npm is shipped as Javascript source by node.js

~~~
sirius87
ah yes, you are correct. it isn't a binary

------
tzs
From what I've read (I've not yet worked on anything that used npm), it
appears that npm uses a single shared flat namespace for all packages. If I
publish a package named "order_pizza" nobody else can name a package
"order_pizza", unless mine gets deleted first, and then they can...and anyone
who was using my "order_pizza" will now silently get the other developer's
"order_pizza" the next time they build.

Anyone know why they did it that way instead of giving each developer a
subspace of the namespace that only that developer can publish under?

I'm thinking of the way it works on Java, where my package would be
net.tzs.order_pizza, or GitHub (and numerous others) where it would be
tzs/order_pizza.

~~~
always_good
I think it's annoying but also toxic to an ecosystem. You have to name your
packages the same way you look for a domain name, people can squat on names,
and it causes drama like how the 'kik' package was stolen from a developer and
given to a company who never used it.

Nobody would've cared it the package was "tzs/kik". For example, the Elm
package system does this right.

Another example of this failure is the Reddit subreddit system where a
subreddit like /r/bitcoin makes it seem like the authority when it's a
carefully moderated and biased subreddit. And nobody can really compete with
it without getting a worse subreddit name like truebitcoin and bitcoin2 or
something. Would be much more clear if it was /r/@theymos/bitcoin and
/r/@spez/politics.

~~~
MikkoFinell
The failure is yours alone if you actually consider Reddit an authority on
anything.

~~~
MikkoFinell
Downvoters: I of course meant an authority on anything ELSE than memes and
karmawhoring. Sorry for not being clear about that.

~~~
always_good
Okay.

And you might say the failure is yours alone when you overeat, yet 30%+ of the
USA is overweight. This little platitude only makes you feel better about
yourself but isn't a good basis for designing something.

~~~
MikkoFinell
That's interesting. Unless I'm being forcefed, then who else is to blame? I
have free will after all.

------
FunnyLookinHat
"We don’t discuss all of our security processes and technologies in specific
detail for what should be obvious reasons..." \- Security by obfuscation?
That's not security. Your protocols and processes should stand up even if made
public.

~~~
platz
What is your home address and phone number?

~~~
daveFNbuck
If you think that your address and phone number are secret information,
consider how easy it would be to find yours from the information in your HN
profile. I'm pretty sure I found your entry on whitepages within a few minutes
and I'm barely motivated to find it.

~~~
platz
I think you are confused as to my intention

~~~
daveFNbuck
I thought you were asking a rhetorical question to show that obfuscation is a
valid security measure that we all use. What was your intention?

------
emerged
I left an Ethereum related project precisely because of the house of cards
feel of it being built off JS and npm modules. They're a great example of how
a project can be exploited by a malicious module which could proceed to
extract all the tokens.

If my hat is black, I'm writing a daemon which monitors relevant npm modules
and uploads subtlely modified versions if and when the possibility to do so
occurs again. Particularly, but not only, targeting crypto currencies.

~~~
strictnein
> If my hat is black, I'm writing a daemon which monitors relevant npm modules
> and uploads subtlely modified versions ...

I started sketching out one yesterday, and then thought "What am I going to do
with this? Probably just get in trouble" so I stopped.

------
Jare
If I was managing npm-using systems, this press release would not put my fears
to rest. They need to publish a full review of the contents of packages
replacements that were uploaded (even if just for 5 minutes), and publish
those replacements in a safe form so they can be reviewed personally by any
concerned npm user (any user - not just those who downloaded the replacements
through npm itself).

------
dboreham
Lesson : in a name space, never delete records. Mark them as unused, invalid,
inactive, "deleted" but never actually delete.

------
fencepost
Independent of npm's issues now and in the past, how vulnerable are other
package managers to similar problems? There were typosquatting issues with
pip, replacement issues with Rubygems, probably others that I don't recall.
What's the current state of things for the more commonly used languages?

~~~
_pmf_
Maven/Gradle don't allow un-namespaced packages; nobody knows for sure how
deep up their arse npm's designers have had their ass when they made non-
namespaced packages possible (even the only way in the beginning; I don't know
if it's still the default).

Additionally, there's the problem of npm's fine-grained dependencies (that
have become a laughing stock in itself), in my Java large projects, while I
don't know every single dependency, I know all the "vendors", i.e. trusted
upstream namespaces I use, and their transitive dependencies. There's the
occasional non-Eclipse, non-Google, non-Apache dependency (say, XStream), but
I am aware of it and fix it to a specific version. npm with its thousands of
dependencies by hundreds of pseudonymous developers for simple projects is an
abomination.

~~~
vbrandl
`Crates.io` (the Rust package ecosystem) doesn't have namespaces and operates
after the first come first serve principle. They won't resolve namesquatting
issues so something like the left-pad debacle won't happen. I think their
policy[0] is quite sane. At least I'm not aware of any issues of that kind
regarding crates.io.

For typosquatting: I don't think that problem can be easily solved by package
registries. They could warn you if there is a package with a similar name (and
more votes, downloads, what ever) but aside from that, the developer should
double check the dependency names and that's it.

[0]: [https://crates.io/policies](https://crates.io/policies)

------
a2tech
Npm isn't ready for production use. If you're going to use it you either need
to ship the modules you need or run it against a private repo of modules.

~~~
N_A_T_E
I agree in principal about not relying on npm during deployment. For a novice,
what is the best way to ship modules?

~~~
md224
My first thought was “What about including node_modules in your version
control?” Downside is that the repo would be much larger, but if it’s going to
expand to that size on the client after running “npm install”, maybe that’s
not a big deal? But it feels kinda wrong to do that...

~~~
acemarke
Committing your `node_modules` is a bad idea. The best answer is to either use
an offline cache of package tarballs that have been committed to your repo, or
an NPM caching proxy.

Yarn has an "offline mirror" feature built in, and Shrinkpack is a tool that
can create an offline mirror based on an npm-shrinkwrap.json file.

I wrote about using Yarn's offline mirror here:
[http://blog.isquaredsoftware.com/2017/07/practical-redux-
par...](http://blog.isquaredsoftware.com/2017/07/practical-redux-
part-9-managing-dependencies/#managing-dependency-packages-for-offline-
installation)

~~~
djsumdog
I know you can use JFrog/Artifactory to cache pulls of Jar files from maven
repos, as well as Docker images. Does JFrog support node?

~~~
Qerub
Yes, JFrog Artifactory supports npm and so does Sonatype Nexus. More info:

[https://www.jfrog.com/confluence/display/RTF/Npm+Registry](https://www.jfrog.com/confluence/display/RTF/Npm+Registry)
[https://help.sonatype.com/display/NXRM3/Node+Packaged+Module...](https://help.sonatype.com/display/NXRM3/Node+Packaged+Modules+and+npm+Registries)

------
smt88
As others have said countless times on the original thread: new packages from
different users should never have been allowed to replace the missing
packages.

Even if a package is removed as malware, a user should never unexpectedly
download the work of a totally new author that they haven't vetted.

Some of the protections should be done on the user side, but that's no reason
for NPM to have dangerous policies.

~~~
allover
> but that's no reason for NPM to have dangerous policies.

Oversight != 'dangerous policies'. Seems like they thought they had this
fixed.

------
simooooo
They need to pull their socks up and start signing these packages

~~~
regecks
What would signed packages have achieved in this incident?

Who needs to sign the packages? The developers of the packages, right?
Otherwise developer credential compromise subverts the entire point.

Which keys are authorized to sign for which packages? How to prevent
credential compromise from affecting those authorizations?

What's the difference between a signed legitimate package and a signed
malicious package?

If they introduced package signing, would packages would adopt it fast enough
for users of packages to only use signed packages?

What happens when signing keys are lost or compromised? Do we need to use
countersignatures from timestamping services as with other forms of code
signing, so that CI systems do not break if a key is pulled?

I think this is very much not straightforward. npm+pgp may be well intentioned
but seems grossly inadequate.

~~~
0x0
Android and the Google Play store works pretty well. The systems "locks on to"
a public key signature so further updates for a given app must be signed with
the same key. That would fix the problem with reproducible builds here. Only
adding new top level packages to a project for the first time is at risk then.

------
kaikai
A bunch of folks here are complaining that the explanation isn't good enough,
doesn't go into enough detail, etc.

Literally the second sentence is:

> Early this coming week, we will share a full analysis and technical
> explanation of the incident.

I'm looking forward to reading the final post-mortem, but in the mean time I
really appreciate their update.

------
wakeywakeywakey
The only thing missing from this post is 'we take security very seriously'

------
incadenza
Am I missing something in thinking that a reasonable solution is to
temporarily block downloads of potentially malicious packages? Why remove
them?

EDIT: or better yet, don’t allow people to download the update, just keep
users on the old, ostensibly safe version until everything gets sorted?

------
kevinmannix
I'm curious of the manual review process. Is this synchronous, i.e.
immediately when publishing a package? Or is it after the fact, where
suspicious code code have already been distributed? There are plenty UX trade-
offs in either direction, of course.

------
gpsx
I've used NPM only a little but this scares me. What are NPM alternatives?

~~~
allover
NPM shouldn't scare you. Simple fact is you should not be relying on ANY
package registry at the time of deployment.

~~~
reificator
> _NPM shouldn 't scare you._

It absolutely should, just like any dependency on any other third party code
or servers. Especially when they regularly have incidents like this.

> _Simple fact is you should not be relying on ANY package registry at the
> time of deployment._

This is true, but doesn't prove the previous claim.

~~~
allover
> NPM shouldn't scare you.

>> It absolutely should, just like any dependency on any other third party
code or servers

There's no need to be scared of any of those things if you understand the
trade-offs and risks.

>> Especially when they regularly have incidents like this.

They don't.

>> Simple fact is you should not be relying on ANY package registry at the
time of deployment.

> This is true, but doesn't prove the previous claim.

The 'previous claim' follows from it. If you don't treat a package registry
like an essential part of your own infrastructure, there's no need to be
scared of it.

------
glenjamin
Lots of people are piling onto npm here. This reads to me like a fairly simple
unintended consequence of what seemed like a good approach.

Automated spam filters help to avoid dodgy packages. Spam filtering operates
on heuristics so it’s sensible to not publicise how they work.

The automated spam filter kills dodgy uploads, as these mostly happen on
previously unused names a decision is made to not have the spam packages’ name
remain taken. Among other things this stops the spammers from leaving all the
good names blocked.

The spam code gets a false positive and the above logic kicks in, leaving
previously used names now available. This is noticed and corrected within a
few hours.

The npm team will likely improve their spam filtering heuristics and also
ensure that formerly good packages that get spam flagged do not release their
names - they have indicated roughly this on Twitter.

This only just happened, and it’s the weekend, so I’d expect a full write up
will be released during the week when they’ve had time to do a post-mortem and
work through the salient points.

~~~
Lazare
> This reads to me like a fairly simple unintended consequence of what seemed
> like a good approach.

Yes, but you need to keep in mind:

1\. This stuff is hideously important because getting it wrong is a incredibly
serious security hole. Letting people publish a package with a name that other
people are already requiring is a defcon 1 emergency.

2\. They already had a major issue with this and said they had fixed it. You
get, at most, _one_ time when you can say "whups, we didn't realise how big an
issue this was!".

3\. It's their job to get it right. This is, literally, npm inc. It's not some
hobbyists, or a service being provided by people in their spare time.

So when you say:

> The automated spam filter kills dodgy uploads, as these mostly happen on
> previously unused names a decision is made to not have the spam packages’
> name remain taken.

I have to disagree. They needed to check how many times the flagged package
had been downloaded, and after the left-pad debacle, they knew (or should have
known) that.

They're not taking this seriously.

------
swang
maybe everyone should wait for the full report before jumping to conclusions?

from what it appears, the reason someone else was allowed to register the same
package name was because their database didn't pull in the entries for that
package. an error? yes. an issue? yes. but it is not a repeat of the same
incidence of left-pad.

i'm not going to pass judgement until i read the full report. i think at the
very least, whatever mechanism they were using to copy/backup the database
seems to have an issue with deleting entries and needs to be addressed

------
dpwm
Others may be interested to note this blog post from about a week ago by the
CTO relating to npm's outage handling [0]. I'm not sure if I would classify
this as an outage, but this seems to be the official narrative [1]. Given that
status page was updated relatively quickly, I'm guessing that at least
something _like_ this procedure kicked in.

I am a little shocked at the brevity of the procedure. I am guessing by the
blog post pointing out the brevity that I am meant to be shocked, but if I
turned out that 310 word procedure for downtime I'd be pretty confident it was
not good enough. I'm not confident that I would be able to read that document
and understand what to do if I was the one on pager duty. Maybe it would be
different if I worked there and perhaps the procedures are more granular and
each of the stages well explained in other procedures. Maybe.

I'm finding it hard to understand what npm does that is so different to other
languages' package managers that it can have so many problems and such bad
public relations. Linux distros have repositories that handle packages with
far greater sizes and quite a few are purely voluntary efforts. They also
manage to include package signing as well. I've known of a few hiccups, but
none that been handled quite so bad as to completely undermine my trust in the
tool altogether.

Is there any compelling technical reason why we couldn't have a tool that
works more like go get for node? For those that haven't used it, you just
point it at a git repository.

[0] [http://ceejbot.tumblr.com/post/169045932039/how-to-handle-
an...](http://ceejbot.tumblr.com/post/169045932039/how-to-handle-an-outage)

[1]
[https://status.npmjs.org/incidents/41zfb8qpvrdj](https://status.npmjs.org/incidents/41zfb8qpvrdj)

------
andersriutta
I'm trying to figure out whether node2nix [0] could help avoid package
name/version hijacking vulnerabilities. Node2nix can convert a package-
lock.json file to something called a Nix expression [1], like this [2], where
each dependency has a checksum. The purpose is to make the Nix expression
deterministic so that each Nix package is reproducible.

[0]
[https://github.com/svanderburg/node2nix](https://github.com/svanderburg/node2nix)

[1] [https://nixos.org/nix/about.html](https://nixos.org/nix/about.html)

[2]
[https://github.com/svanderburg/node2nix/blob/22d0c19575ecd19...](https://github.com/svanderburg/node2nix/blob/22d0c19575ecd19208cbdfbb54e8c2228c553bf7/node-
packages.nix#L13)

Edit: fix link formatting

------
bennofs
Why does a spam system delete packages that have already been in use for quite
some time? I could understand if it blocked some newly updated ones, but it
seems like it has deleted already existing packages that also were used by
other packages?

------
noobermin
With this and leftpad etc., I think it's time for the tech community to
remember Java and Oracle, Sourceforge, etc., or look it up if you are too
young to remember. It is beneficial to learn from history instead of making
the same mistakes all over again.

Centralized trusted authorities cannot be trusted if they are private
companies who don't answer to you. If you aren't paying them and you don't
have a management position in said company, things like this will happen.

------
swsieber
Everybody is mad/disappointed at/by the NPM for their handling...

Does this have any real impact on the NPM package adoption though? There is no
alternative node module package directory, is there?

It seems like any action they could have taken would have had exactly the same
effect (well, maybe some prospective products don't adopt JS backends, but you
have all webpack projects locked in pretty tight, right?)

~~~
Lazare
I've been using gitpkg
([https://github.com/ramasilveyra/gitpkg](https://github.com/ramasilveyra/gitpkg))
lately; it works pretty well. Basically lets you use a git repo as a package
repo, instead of npm.

Very little about how I use npm requires it to be centralized; I don't _ever_
search npm looking for a package to install; I find a package I want and then
look to install it.

~~~
dbbk
Couldn't you just use github IDs in your package.json directly? npm and yarn
both have support for it.

~~~
Lazare
Short answer: Sure, but it works much better with gitpkg.

Long answer: Sometimes you can do that, yes. But you'll run into issues with
"modern" frontned toolchains and complicated packages where you want to write
ES2017 code, transpile it with babel, bundle it with webpack, make a universal
lib, etc., and push the result to your package repo, but not commit it to your
git repo.

Gitpkg handles all the wierd corner cases, and lets you keep build artifacts
out of your git repo, and raw source out of your package repos. And if you use
github you can even use the github IDs in your package.json, but the result is
going to be a lot better. If you're just writing a plain one file package for
use in node that doesn't need any processing, eh, no need for gitpkg.

------
vl-y
Such explanation looks misleading to me. Even having the module detected as
potentially malicious, why was it put into a state that such module doesn't
exist, letting others publish package with the same name. For me it looks like
there is something wrong at the npm side internally with the processes.

------
cobbzilla
How difficult is it to run your own private npm repo? Looks like that's what
security-conscious folks should be doing, given this response. Any
pointers/gotchas/battle stories much appreciated.

~~~
gedy
It's doable, and one very good option is Sonatype Nexus
[https://www.sonatype.com/download-oss-
sonatype](https://www.sonatype.com/download-oss-sonatype)

The one issue I've seen over the years is it seems npm Inc does not officially
sanction on site npm repos (presumably to help monetize their private hosted
repos), and finding info on how to set up a private repo is lacking.

------
whatever_dude
Just when I started thinking NPM had turned the ship around after the previous
incidents. Given the amount of people who depend on their service, this is
extremely amateur.

------
knodi
The pleasures of working with Node.js

------
julienfr112
Is it related to a package that write logs in colors and send visa card
numbers to hackers ?

~~~
slig
No, that article was a warning that a package like that could exist in the
wild and nobody would notice. It doesn't actually exists, as the author states
in the blog post.

------
INTPenis
Since everything uses HTTP these days I think they could use one of the 20x
codes to signal when a package has been affected by spam management or
suspension.

That way the official clients would support it but you could easily ignore it
if you wanted to.

------
yunong
Has the community discussed adding cryptographic signatures to npm packages?
It's generally a good idea in any case, but this outage further highlights the
security risks of a public registry where packages are not signed.

------
indigodaddy
Looks like they don't allow comments on their blog posts??... I guess with
good reason.

~~~
pvdebbe
They use Tumblr as their blog platform, and Tumblr AFAIK doesn't use comments.

------
sumitgt
> We don’t discuss all of our security processes and technologies in specific
> detail for what should be obvious reasons, but here is a high-level
> overview.

Isn't that the opposite of good practice? You shouldn't rely on obscurity.
It's better to have the security processes out in the open so that it can be
audited and flaws pointed out.

~~~
siler
Yeah, I found this an important aspect of the article. I don't know anything
about the culture at NPM, but this feels related to the mismanagement of
communication with consumers/the public about the incident itself. i.e., being
secretive for no good reason.

------
lend000
I think a major security improvement for npm and other package managers using
almost exclusively open-source/ non-compiled code is to require the source
code be linked from a popular open-source platform like Github and then to
take the package directly from there, to ensure the code can be audited and
nothing else can be snuck in.

------
indigodaddy
I've been meaning to set up a blog with hexojs for a few weeks now, and was
seriously all set to install node and npm (don't do too much with node...),
and start tinkering yesterday, and then I checked hacker news, and was like
'well sheeyit'...

~~~
indigodaddy
I guess this would be safe enough probably:

[https://github.com/billryan/docker-hexo](https://github.com/billryan/docker-
hexo)

------
throwaway66666
Pay attention to the tags they themselves added to their report. Namely,
"#SPAM IS WHY WE CAN'T HAVE NICE THINGS". Oh npm...

------
dzonga
noob question : would using yarn have mitigated any of the risks of
missing/broken packages ?

~~~
therealasdf
No, both Yarn and npm use the npm registry.

------
z3t4
idea: Peer reviewing packages with the ability to suggest a better name.

------
perlgeek
A manual review misjudged 106 packages at the same time? That seems...
dubious. Unless they have a bulk removal based on meta data, instead of
careful analysis. Which would also be pretty dubious.

