
Post-mortem of this weekend's NPM incident - txmjs
http://blog.npmjs.org/post/169582189317/incident-report-npm-inc-operations-incident-of
======
bhuga
This is a good post-mortem with clear, policy-based remediations. Nicely done.

I wonder why they are only preventing republishing for 24 hours. Is there a
good reason to allow a package namespace to be recycled with less than, say, a
week? Is it based on the assumption that the only case where it comes up is
during an incident, and 24 hours is enough time to assume an incident will be
resolved? I'm curious what went in to that number.

~~~
smt88
Why allow namespace recycling at all? The potential harm is high and the
potential benefit is some slight convenience.

If npm packages used a Github-style "author/package" format, name collision
would never be an issue again.

~~~
Klathmon
>If npm packages used a Github-style "author/package" format, name collision
would never be an issue again.

They have that, and many are finally starting to take advantage of it (with
babel being the most prominent with their latest version)

But this doesn't completely "fix" the problem, since the exact same conflicts
can still happen with the "author" name (if someone takes "google\" there are
going to be some very upset californians)

~~~
slig
> (if someone takes "google\" there are going to be some very upset
> californians)

Not a problem at all. They will just get the name by force. It already
happened before, google "npm kik".

~~~
Klathmon
Yeah, but it's still a "problem" in that it _technically_ doesn't change
anything from the current system.

------
josephorjoe
So, a spammer uploaded something containing copied data from a legitimate user
and npm deleted everything from that user. Oy.

Seems like npm might want to review the policy that allows stuff like that to
happen.

Even if a user violates the spam policy (which, to be clear, it seems the
affected user in this case did NOT do), that hardly seems to be appropriate
grounds for deleting everything the user has ever published on npm.

That is a policy that is just begging for griefing.

~~~
DanBC
Are "joe jobs" still a thing?

[https://en.wikipedia.org/wiki/Joe_job](https://en.wikipedia.org/wiki/Joe_job)

> A joe job is a spamming technique that sends out unsolicited e-mails using
> spoofed sender data. Early joe jobs aimed at tarnishing the reputation of
> the apparent sender or inducing the recipients to take action against them
> [...]

~~~
daurnimator
Yep. I had one against me mid last year.

------
cremp
> we have policies against ever running SQL by hand against production
> databases—but in this case we were forced to do so

Uh... Add in the fact that staff are now trigger happy, since a single button
can do a lot of damage.

------
dumbmatter
_Our first action, which began immediately after the incident concluded, was
to implement a 24-hour cooldown on republication of any deleted package name._

Why not infinity hours? I don't get it.

~~~
Vinnl
If it's a spam package that gets deleted, that would mean you'd quickly run
out of available names.

~~~
h1d
Why can't they just reuse when it is apparent the case is harmful (as in,
people complain and check number of downloads and dependent packages) by
blocking the name and disallow reuse for any other cases?

~~~
Vinnl
Hmm, I'm not sure if I can follow your question, but I'm guessing that they're
already planning to do what you want?

------
kylemuir
> Our first action, which began immediately after the incident concluded, was
> to implement a 24-hour cooldown on republication of any deleted package name

I don't understand this. Why hard delete packages at all? Soft deleting feels
like it would be easier and would stop people republishing with the same name.

They could also bake their warning process for dependent libraries (i.e. "this
package is gone!") into the soft delete process.

------
carsonreinke
I feel like a project that could help with this to identify package importance
by the dependents and downloads.

~~~
carsonreinke
I think I might actually try this out.

------
kodablah
> At the time of Saturday’s incident, however, we did not have a policy to
> publish placeholders for packages that were deleted if they were spam.

I see this acknowledgement, but I cannot find where they will remedy this by
putting placeholders in place of spam removals. As a concession, maybe only
placeholders for spam removals of packages that are older than X days or
depended on (explicitly or transitively) by X packages. Did I miss where the
remedy for this spam-removed-package-reuse was in the blog post?

~~~
avianlyric
They have added a 24hour re-publishing cooldown for all package removals
regardless of reason. Exceptions are made for the original publisher and npm
staff.

Explained somewhere near the bottom of the post, basic rational is that it
gives them time to notice fuckups and fix them.

~~~
kodablah
This does not alleviate the issue where you can reuse package names. I suppose
they believe what they mark as spam packages won't be used enough or is
already bad enough that reusing the name is harmless. And they probably also
believe that they can catch fuckups in a day. I don't think either are
necessarily true and are only true in this case because it hit popular dep
trees. But what happens when something is erroneously marked as spam that's
not as popular and the downstream dependents don't realize in 24 hours? If the
problem is that "placeholders" are too heavy, then they could be made lighter
weight or put some rules around when they will add them and when they won't.

~~~
testplzignore
My guess is that the cost of the placeholders is indeed what is driving their
decision, though perhaps it is a premature optimization.

Maybe they've had situations where a spammer has created a very large number
(millions, billions?) of packages. It's possible that the majority of user
submissions are automated spam from botnets. I would assume npm has some
mitigations in place to prevent this abuse in the first place, such as rate
limiting and captchas, though maybe that's not enough to stem the tide.

Though, given that they say they have humans doing the package deletion, that
makes me think that the number of spam packages created can't be that high.
Certainly not high enough to outweigh the risks of package name reuse.
Increase your prices a few pennies a month so you can afford to store the
placeholders forever.

