
Visual Studio Code 1.7 overloaded npmjs.org, release reverted - eiopa
https://code.visualstudio.com/updates#_17-rollback
======
seldo
I'd just like to say on behalf of npm that Microsoft's handling of this
incident was A+. As soon as we alerted them to the issue they were all hands
on deck and did a rollback.

We've been really pleased that Microsoft chose to put their @types packages
into the npm registry rather than a separate, closed system, and in general
happy with Microsoft's support of node and npm. We're confident we can make
the new features of VSCode work, we just need to work with Microsoft to tweak
the implementation a little.

This was an honest mistake on their part, and we caught it in time that there
was very little impact visible to any npm users.

Fun fact: at its peak, VSCode users around the world were sending roughly as
many requests to the registry as the entire nation of India.

~~~
ec109685
"Many requests to the registry as the entire nation of India" per what time
unit?

~~~
oridecon
Approximately 3 new JS frameworks per hour.

~~~
jug
Funny. Sounds like the expansion rate of the Javascript ecosystem.

~~~
young_greedo
It's not a real HN thread until someone makes this joke...

------
BenjaminCoe
As one of the folks on the front-lines helping patch this, I certainly have no
hard feelings; and I'm excited to be able to support this feature properly

... also ... not going to lie, this was the first time we've gotten to test
several of the checks and balances we have in the npm registry which I was
jazzed about :)

~~~
raisedadead
Thanks, Benjamin, Laurie and everyone else for mitigating this, it feels great
to know when the community chimes in together for such highly unanticipated
scenarios.

On that note, however, respectfully I believe that features which have the
potential of hitting the registry so bad should first be beta tested on a
private registry and moved on to the high traffic serving CDNs of npm.

And 10% of the daily traffic is from India??? Whoa, every day is a school day.

~~~
poizan42
> And 10% of the daily traffic is from India??? Whoa, every day is a school
> day.

Well, 17% of the world's population lives in India, so doesn't seems
surprising.

~~~
raisedadead
lol. agreed.

------
mavsman
If I were on the Azure team I'd be offering tons of free credit to npmjs.org
to get them to use Azure. Azure coming to the rescue would be the perfect
ending to this story for Microsoft.

~~~
CaveTech
I don't think most organizations could relaunch their infrastructure on a
totally different stack at the drop of a dime. And if it was really a "throw
more servers at it" problem then it wouldn't really matter who was hosting
them, would it?

~~~
azinman2
Depends on who's paying

------
Arnavion
[https://github.com/Microsoft/vscode/issues/14889](https://github.com/Microsoft/vscode/issues/14889)

------
sync
Shouldn't all these requests be cached by a CDN? What exactly is overloading?

~~~
seldo
CDNs don't usually cache 404s. VSCode was looking for @types packages for any
and every npm package its users were using. Packages that had a type
description caused no issue, but most packages don't, so we had a > 1000%
spike in 404s. Our workaround before MS did the rollback was to cache 404s for
@types packages specifically, and it was effective enough that the registry
never really went down.

~~~
natuac
"a > 1000% spike in 404s" overloaded your servers? Such are your generation
times? Can I bring the entire NPM ecosystem down from my ADSL line using some
silly threaded code to make requests to randomly named packages?

~~~
seldo
99.9% of our requests are handled by the CDN. The CDN doesn't cache 404s, so
404s are handled by our origin servers, which are much fewer in number and
therefore quite easy to overwhelm.

You're right that our handling of 404s was naive, and that's definitely
something we'll be improving as a result of what we've learned from this
incident.

------
markatkinson
It is a real foreign feeling being exposed to such an actively and well run
project. Every time I see a new release on HN I get a little "wow, that time
of the month again." Even this rollback was indicative of how fast they move.

------
akfish
Which is more possible? A bug or they just underestimated the volume of
traffic that could be caused by ATA in real life?

~~~
hyperliner
The latter.

------
manojlds
Would yarn help here? (since FB have their own CDN and registry for it?)

~~~
eugeneionesco
They do? yarn uses the npm registry not something else.

~~~
PudgePacket
It does, but it also goes though cloudflare as far as I know (which does
caching).

~~~
ohitsdom
This issue though was with excessive 404s, which aren't cached.

------
PudgePacket
I wonder if any warning was given to npm that they would be getting this
potentially huge new source of traffic. It doesn't seem to be mentioned
anywhere.

~~~
vonklaus
Eh, NPM is a pretty core service and both sides probably should have done
things a bit differently. I don't neccessarily think vscode needed to reach
out to NPM to let them know they were going to be consuming their public API.
Both teams appear to be in communication as a result however-- which is good.

This will likely lead to more fault tolerant systems on both projects and
hopefully more collaboration & features in the future.

~~~
CapacitorSet
>I don't neccessarily think vscode needed to reach out to NPM to let them know
they were going to be consuming their public API.

VSCode is used by a non-negligible number of users, and seems to rely on npm
to operate at its best. It would have been good etiquette to let npm know,
even though they couldn't forecast this exact situation.

~~~
vonklaus
I am not the architect of any large scale system-- that said, I wouldn't
expect developers to reach out to GitHub.

However, it isn't bad etiquette and I'm sure Microsoft could get in touch with
the devs. Interesting thought.

------
antrion
This is a really cool feature! Is there a similar extension for Atom /
Sublime?

~~~
eugeneionesco
Not really, they don't have the user number for this.

------
andyburke
oops

------
_pmf_
But they told us that pulling in 1.6 GB for "Hello World" is normal and no big
deal.

------
gremlinsinc
I'd love to use VSCode but can't until they or someone else rolls out a
dockblockr extension that works for php as I'm mostly tied to Laravel right
now, and my company requires docblocks and they are not fun to write by hand.

~~~
winsome
They have a great extension ecosystem. Why not give writing the extensions a
shot yourself?

~~~
gremlinsinc
I've never really written much software or extension type things... I guess I
could take a look at the code and compare it with DockBlockr on Sublime. I'm
more of a web app guy.

------
z3t4
They are probably trolling for a debate about NPM ... I can smell politics.

------
hiou
_> The feature was so great that we started to overload the npmjs.org
service._

I'm not sure I would call my feature "great" if it could have brought down
npm.

~~~
cerebellum42
I thought that sentence sounded very trump-ish.

The feature was so great that npmjs couldn't keep up with it, it was yuuuuuge!

~~~
jessaustin
...and they made npm Inc. pay for it!

~~~
Aldo_MX
it was a tremendous overload

