
NPM: 429 Too Many Requests - kerpele
https://github.com/npm/cli/issues/836
======
gmemstr
This seems to be slowly clearing up.

Regardless, can we talk about the conduct in this GitHub thread? I know every
community is different but is it common to have memes and jokes posted this
quickly and often in a GitHub issue? It makes it really hard to follow and
discourages genuinely useful discussion of workarounds or progress.

~~~
lm28469
Some people getting in the workforce in the last few years have troubles
making the distinction between work and play contexts. It's extremely visible
on github, slack, &c. which are more and more looking like discord / reddit
(gifs, memes, random jokes in the middle of serious discussions)

~~~
matsemann
You say it like it's a bad thing? It's possible to be both professional and
have some fun (not talking about the linked GH).

~~~
easytiger
I mean there is a difference between having fun and conducting yourself in a
professional manner where appropriate.

With the increase in adoption of certain technologies (node etc) I have
noticed a relationship with the people employed to work with them and their
poor level of professionalism in the workplace with regard to their work and
how they deal with other people (in the UK) and even how they dress. Similar
age groups in other areas don't display these traits.

A number of these people work as contractors as well, so at least it was easy
to get rid of them.

Go post memes on the LKML and see what happens

~~~
Shacklz
> [..] and even how they dress.

While there are many points in your post that I agree with, we really should
stop worrying about how people dress.

If it's a developer sitting behind a computer screen all day long with zero
customer contact whatsoever, he really shouldn't be forced to wear a certain
dress code just to satisfy someone's standard of professionalism. Such
feelings are in similar spirit as "woman should not wear trousers" a few
decades ago - there is just no rationale behind it other than conditioning by
society.

~~~
scoutt
> conditioning by society.

There is a limit. I'm sure you have one too if you really think about it. Is
it tattoos all over the face? What about sandals? Or shorts? Or going bare-
feet around the office.

I don't know... It's about people giving me money for work. I don't feel like
I can go to work dressed like any given Sunday.

~~~
ElFitz
> I don't feel like I can go to work dressed like any given Sunday.

And why not? As you said: they are giving us money... for work. Not for
playing dressing up.

If they expect otherwise, they should be clear about it up front and give a
good reason to give up on that freedom while we work for them.

~~~
scoutt
> And why not?

To me (and of course, it's a personal opinion) it shows that someone cares.
It's like when you do your resume, you want it clean and presentable, without
typos, to show that you care.

We all like good user interfaces and experiences in our programs. To me, the
_look_ is like the presentation layer of our own UX. Is it required for our
work? Most of the times it is not, because our work depends on the _logic
layers_ , but it doesn't hurt to have a nice presentation layer.

And I'm not telling about going to work in a D&G suit.

EDIT: and I'm not telling that I would discriminate or think right away that
someone doesn't care! I don't do that. My attitude about good looking is mono-
directional; from me towards others.

~~~
ElFitz
Well, to me, people care by showing up on time.

They care by solving problems.

They care by being there when the team needs them.

They care by being honest, authentic and candid.

They care by making things simpler and smoother for themselves and those
around them.

They care by striving to avoid office politics and back stabbing.

They care by being the team mate we all wish we had and strive to be.

They care making our common working space a place we are all eager to come to
in the morning.

And I would pick a reliable, self-aware, well-rounded, face tattooed, green
haired and barefoot colleague that's comfortable in their own skin any day,
over any well-dressed, high maintenance, drama king / queen.

What's interesting though, is that even knowing, and thinking that, I couldn't
help but be spontaneously biased towards he or she who made the so-called
effort of dressing to look the part. And that's a problem.

Also, I am nowhere near being that ideal team mate. More like a work in
progress >.<'

~~~
scoutt
> biased towards he or she who made the so-called effort of dressing to look
> the part

Anecdote: my wife is a psychologist. They are taught to observe how a patient
presents him/herself at an appointment. Is her/him presentable? What about
personal hygiene? Does he/she take care of him/herself? These are
characteristics that say a lot, and should be taken into account when meeting
someone. There is a difference between taking care of oneself and trying to
appear like something else by wearing branded clothes, or to appear like
something else by dressing like a 60's hippie (or whatever).

> And I would pick a reliable, self-aware, well-rounded, face tattooed, green
> haired and barefoot colleague

Me too. But the first 3 are characteristics you wouldn't know for a while. In
the meantime you have to work the hiring with what you see. Unfortunately,
it's like that.

I also believe that the worst and most dangerous people in the world walk
around in expensive suits, behave correctly in public and never say a bad
word.

------
imtringued
Honestly, don't depend on central repositories for daily availability.
Especially if you are doing CI that redownloads everything from scratch. Use
something like artifactory to cache the repository you are using:
[https://www.jfrog.com/confluence/display/RTF/npm+Registry](https://www.jfrog.com/confluence/display/RTF/npm+Registry)

~~~
krab
I think that's the issue of cost/reward. The cost is

\- N developers can't work for X hours

\- or the company can't release new versions due to CI dependency on the
registry.

\- or the registry removes a package you were using

\- or the existing package contents changes to something malicious

BUT you pay this price very occasionally and if you're a small shop, the cost
is often negligible.

On the other hand, maintaining your own mirror has very real costs even though
they can be small. One time setup, hardware, sometimes license or hosted
service fee, security upgrades. When there's a sponsor maintaining the central
repository, having very good uptime and offering it for free, the marginal
utility of a local mirror is quite small.

~~~
naniwaduni
There's no reward. It's a risk/cost tradeoff.

------
warpech
If this happens in future, you can switch an NPM mirror, such as:

\- [https://open-registry.dev/#get-started](https://open-registry.dev/#get-
started)

\- [https://npm.taobao.org/](https://npm.taobao.org/)

WARNING: Research who runs the mirror before putting your trust in it.

How to turn on:

    
    
      npm config set registry https://npm.open-registry.dev
    

How to turn off:

    
    
      npm config delete registry

~~~
yellow_lead
I would not recommend to use the Taobao registry though. This is operated by a
Chinese company. Aside from the cybersecurity concerns, if it's hosted in
China you'll be getting bad latency.

~~~
lqs469
It's just a mirror service address, the NPM package is the same. Worrying too
much about cybersecurity is a bit of a storm in a teacup.

~~~
warpech
Well, how do you know it's "just" a mirror service, and it is not using a
zero-day to exploit your system, by installing a root-kit or copying your code
to their servers?

I agree that it's a valid concern.

~~~
gpmcadam
If you're concerned about injection into a third-party package, you should be
using `package-lock.json` (or equiv) and integrity hashing your dependencies
at install time.

------
chrissnell
I dealt with this problem at a previous job where we had a build pipeline and
apps that were very dependent on NPM. To fix it, I used nginx to build a set
of two-tiered caching servers within our CI Kubernetes cluster. One used a
ramdisk to provide a low latency cache for the NPM client fetches. The second
was a disk-backed cache to provide persistence to the ramdisk-backed cache. I
had a script that would alter the package deps so that they were pulled from
the ramdisk-backed service, which used the disk-backed service as it's
backend. The disk-backed service uses actual NPM URLs for its backend.

The result was lightning-fast fetches and no rate limiting.

We needed the two-tiered system because this was Kube and occasionally we
would have to rebuild/restart nodes and we didn't want to completely lose the
cache when that happened.

You can easily extend this system to handle any package artifacts used in your
build process: .deb's, .rpm's, etc.

------
nonbirithm
Here's an explanation from CloudFlare as to the root cause. [0]

> I am the engineering manager for the DDoS protection team and this morning
> at 11:06 UTC we tweaked a rule that affected one of our signals. The signal
> relates to the HTTP referer header, and we have a piece of code that looks
> at invalid referer headers. In this case we tweaked it to include not just
> "obvious garbage" but "anything that does not conform to the HTTP
> specification"... i.e. is the referer a URI? If not then it contributes to
> knowledge about bad traffic.

> So... why did this impact npmjs.org? It turns out that a lot of NPM traffic
> sends the referer as "install" which is invalid according to the HTTP
> specification. As NPM is also a heavily trafficked site this resulted in the
> DDoS systems picking this up and treating the traffic as a HTTP flood and
> determining that a rate-limit should be applied.

> When we noticed that NPM was seeing an increase in HTTP 429s (as seen on
> Twitter) we contacted NPM and started an internal investigation. As soon as
> we identified the root cause we reverted the change, which was at 13:00 UTC.

> We'll note that NPM and 1 other site use the referer for purposes outside
> the HTTP spec and we'll update our systems to ensure that this does not
> happen again. Additionally we'll improve our monitoring around changes of
> this nature so that we can discover impact sooner and roll back
> automatically.

[0]
[https://github.com/npm/cli/issues/836#issuecomment-587019096](https://github.com/npm/cli/issues/836#issuecomment-587019096)

------
rickspencer3
I came here to see if anyone had insights about what happened to see if I
could apply any lessons learned. But all the comments seem to be "get off my
lawn" style comments annoyed about people posting jokes in the issue.

Unfortunately, I am unable to resist the urge to add to the noise by
complaining about people complaining.

------
colonwqbang
Is it common to rely on a free service like npm for your company's core
business? It seems like you would be taking a huge risk by not mirroring
anything you need internally.

~~~
dividedbyzero
I believe (based on lots of anecdata) that it's not just common, it's
absolutely overwhelmingly often the case at companies of pretty much every
size, be it a data scientist using stuff from CRAN for mission-critical
modelling or an OS package repo or the like. It appears that few shops have
this fully under control.

------
Waterluvian
I wonder how many requests to npm are an utter waste given that often
dependencies don't change due to the lockfile.

In Travis there's some (not too obvious) caching mechanisms that in many cases
avoid this and speed builds up a ton.

I wonder if we would win from a review on popularity of configuring the cache
and to educate people further on its use. I'm sure other CI systems have
similar capabilities.

------
SirensOfTitan
I’ve been considering checking node_modules into source control for some time
now, has anyone else done that successfully? There would be a variety of
benefits:

1\. Eliminate redownload of packages on every CI build 2\. Reduce the amount
of gigantic IO operations from unpacking the tens-of-thousands of files
sitting in node_modules. 3\. Better security: code checked in can be audited
better if not downloaded every single CI build.

yarn’s PnP system is promising for the zero-install paradigm, but it doesn’t
seem quite ready yet (so many packages don’t seem to get their dependencies
right).

~~~
joeyrobert
Helps if you only have one platform you're developing on and deploying to
(e.g. x86-64 Linux). If developing on macOS there can be Mac specific binaries
installed, depending on the package.

~~~
jackewiehose
And if you do have more platforms, why not just check in one node_modules-
directory for each?

This idea to redownload all packages all the time from external sources (and
not even having a fallback-plan) seems completely brain-dead to me. Didn't the
people learn from leftpad-gate?

~~~
LeonidasXIV
> And if you do have more platforms, why not just check in one node_modules-
> directory for each?

Now you have to sync it or risk running into unreproducible build failures.
Also, if you update the binary dependencies on say, macOS, then you still need
some x86-64 Linux to build the dependency.

Not saying it is not possible but without a proper process (e.g. a build
server being the only place that updates dependencies) this is going to be
painful.

------
dijit
I was (incidentally) just looking for a way of running a npm repository.
Apparently it is not so simple as running a web server with a manifest (which
is the case for basically every other package manager out there). Is there a
reason for that? Is npms approach somehow better?

The thing with using web technology for distribution is that it’s easily
accessible and, crucially, that it’s cachable in-line.

~~~
giaour
NPM started off as a CouchDB app, and you used to be to keep a local mirror of
the full repository by running a copy of CouchDB and setting up one-way
replication (not sure if this is still possible).

------
sethammons
This is one of the reasons why, in Go, my team vendors our dependencies. For
any service that seeks stability and the ability to deploy any time, removing
networked build dependencies is important.

~~~
ratata
> vendors our dependencies

Is that like a local cache?

~~~
sethammons
Vendoring is checking in your dependencies with the source. You can def
consider that a local cache. The next step up is to run your own proxy server
(pretty much a package server / mirror). Next is to use a service like
artifacory that does similar.

~~~
erik_seaberg
We should call it a backup. Calling it a cache puts people in mind of size/hit
ratio tradeoffs; what's needed is zero tolerance for loss of mission critical
code.

------
mxschumacher
In the last company I worked for, we had a pretty standard build pipeline for
a web application. I was a bit shocked to see how many different packages
repositories we depend on for every build: NPM, Pypi, Alpine packages, docker
hub ...

a single failure in any of those centralized systems that we don't pay for and
builds fail.

------
zovin
Here's a postmortem comment from a Cloudflare engineer:
[https://github.com/npm/cli/issues/836#issuecomment-587019096](https://github.com/npm/cli/issues/836#issuecomment-587019096)

------
Philomath
If this is causing production CI/CD pipelines to fail, this might be a bigger
issue than it seems. Has this already happened in the past with npm?

~~~
gmemstr
> if

It 100% is, unfortunately. I don't recall this happening in recent history,
but it has been the case that 3rd party services have broken CI/CD pipelines
and production pushes (e.g pip broke a few weeks ago, and their own pipeline
for deploying changes was blocked by the bug).

~~~
VBprogrammer
It's very easy for these kind of dependancies to creep into the build process.
If the worst case cost of not being able to create a new build out-weights the
cost of rearchitecting your build process then it's something you should
seriously consider. On the plus side it also brings additional benefits like
faster builds and resilience against packages being unilaterally removed.

------
tablethnuser
I'm always surprised that npm, a for-profit company with a lame business
model, is graciously serving redundant package requests millions of times per
day to everyone's CI/CD flows.

One day this is going to happen for real and it will be because npm org
decided to charge for API requests by `npm ci`.

------
kuon
Isn't mirroring still a thing? I am not saying this as the old creepy guy
(well a bit), but seriously wondering why a registry like npm doesn't have
tons of geographicaly spreaded mirrors. Package can easily be signed and
mirrored, that shouldn't be complicated.

------
darekkay
Check their status page [1] to see the progress:

> _Investigating - We are currently investigating 403 / 429 errors received by
> some users._

[1] [https://status.npmjs.org/](https://status.npmjs.org/)

~~~
ORioN63
It seems the issue is resolved at the moment. From the same link

> Monitoring - Our content delivery partner has informed us they've
> implemented a fix. We are monitoring. Feb 17, 13:07 UTC

Edit: And in the OP GH thread, people are also mentioning it's back up.

------
hkchad
1) Glad we have a Nexus proxy in front of npm... [a] 2) Cloudflare strikes
again, this one company is at the same time making the internet better and
worse. I'm constantly blocked by cloudflare for something that should not be,
it's extremely frustrating. Then when you complain to them they throw their
hands up and say "owner hasn't configured their site for that". Ugh.

[a] [https://blog.sonatype.com/using-nexus-3-as-your-
repository-p...](https://blog.sonatype.com/using-nexus-3-as-your-repository-
part-2-npm-packages)

~~~
hoppla
Cloudflare must value my ability to identify fire hydrants and crosswalks,
because I sure get a lot of these challenges...

------
dgellow
Yearly reminder to vendor your dependencies.

------
keanzu
Reliance on npm is one of the things covered in Ryan Dahl's "10 Things I
Regret About Node.js" talk at JSConf EU 2018 and is fixed in Deno.

 _Includes a built-in package manager for resource fetching, thus no need for
NPM._

[https://en.wikipedia.org/wiki/Deno_(software)](https://en.wikipedia.org/wiki/Deno_\(software\))

------
hannob
I guess the solution here is for the larger CI providers to host a mirror of
popular package repositories locally and provide an easy way to use them.

It really doesn't make a lot of sense to re-download tons of packages for
every minor commit where you run your CI.

------
29athrowaway
Most of the responses to the ticket are completely irrelevant.

------
cpclermont
Some users in the issue are reporting it's fixed.

------
lqs469
That's why we need a decentralized network.

------
clownpenis_fart
and the js/npm clown show marches on

~~~
wraithy
Looking at your username, I'm unsure if this is a positive or negative thing.

