
Disabling npm's progress bar yields a 2x npm install speed improvement - ishtu
https://twitter.com/gavinjoyce/status/691773956144119808
======
ClayFerguson
Doing GUI calls is a notorious way to slow down apps. I sped up an app (major
software from major large corporation) by a factor of 10x a few years back by
removing GUI updates. What you have to do is create a thread that refreshes
the GUI periodically and not let your main worker thread ever work on the GUI.
This particular app was an Eclipse Plugin (a source control app) and the
Eclipse SWT to update the console log window was being called after every file
operation (of which there were 10s of thousands). All I did was move the GUI
refreshing into it's own thread, and call it only every 250millis or so. I was
hailed a hero, and deservedly so, for speeding up our app by 10X and was one
of my most satisfying accomplishments in my software career.

~~~
nicklo
This strategy is also encouraged by best iOS development practices (main
thread never works on GUI).

Edit: 'main' as in the one doing the one doing the work, in the same spirit of
the original post. In truth, the true main thread that runs in main() is where
all the iOS GUI work happens. Apologies for the confusing wording.

~~~
eridius
This is not only completely wrong, it's dangerously wrong. It is in fact
wildly unsafe to touch the GUI from anything _except_ the main thread. It's
all the non-UI stuff that you should shove onto another thread.

~~~
moonchrome
Depends on what you call "main thread" \- maybe he's saying worker thread
that's doing the "main code" shouldn't be doing GUI

~~~
cellularmitosis
In other languages / frameworks, "main thread" may have flexible semantics,
but in the land of iOS, UIKit defines the term "main thread" to have one
specific meaning.

------
untog
Progress bar or not, npm has gotten extremely slow with v3, to the extent that
someone is working on a performance-centered alternative called ied:

[http://gugel.io/ied/](http://gugel.io/ied/)

installs are literally the #1 thing npm needs to do well, and I really hope we
see some improvement soon.

~~~
yunong
Even more important, npm doesn't reliably reproduce builds. Different runs of
the same shrinkwrapped build could either fail or succeed -- which is odd
given that builds should be deterministic. Additionally, shrinkwrap is also
broken in the same way.

This is the first time I've heard of ied, a little competition could go a long
way!

~~~
krisdol
Could you expand on that? What causes it to be nondeterministic? I haven't had
that experience but it's the second time I've read someone reference this
behavior

~~~
yunong
I have a project that's shrinkwrapped. I check out a new copy of the project
and run `npm install`. My co-worker does the exact same thing, his fails, yet
mine successfully builds. An install of a shrink-wrapped should be
deterinistic, i.e. either succeed or fail, but not both.

~~~
Wintamute
From that doc page it sounds like that _should_ be deterministic? Even without
shrinkwrapping, a fresh install from package.json with empty node_modules
should be deterministic.

> The npm install command, when used exclusively to install packages from a
> package.json, will always produce the same tree. This is because install
> order from a package.json is always alphabetical. Same install order means
> that you will get the same tree.

> You can reliably get the same dependency tree by removing your node_modules
> directory and running npm install whenever you make a change to your
> package.json.

Maybe you're experiencing a bug, rather than some in-grained non-determinism
in npm?

~~~
madeofpalk
My understanding is that even when you fix your dependencies to exact
versions, your dependencies probably haven't so without shrink-wrap you'll
never know _exactly_ what gets installed.

------
ghjnut
Here's a breakdown
[https://github.com/npm/npm/issues/11283#issuecomment-1752468...](https://github.com/npm/npm/issues/11283#issuecomment-175246823)

~~~
jessaustin
I guess we're supposed to assume that V8 is smart enough to inline that sort
of function definition, but it seems deeply weird to me that _drawBar()_ is
defined inside the "hot" method like that. Also, I would have expected that
streams would have been used for this. Logging is a very common "simple
example" on node streams how-to pages.

------
nijiko
I'd also like to take the time to mention that the caching system when
installing / other actions is extremely inefficient (for my times I have the
progress bar turned off already and this is a project with around 80 modules
shared between dev / prod):

[https://github.com/npm/npm/issues/10890](https://github.com/npm/npm/issues/10890)

I started on a very (I would like to emphasize very a thousand times over)
basic proof-of-concept to show how much faster it could be in the order of
magnitudes:

[https://gist.github.com/nijikokun/2f1f16325f8ffe14b1b3](https://gist.github.com/nijikokun/2f1f16325f8ffe14b1b3)

All this does is build a json of every package you currently have installed,
and utilizes that as a lookup store the next time instead of rebuilding it
every install; this was targeted towards installing / uninstalling existing
packages. Not fresh installs.

Fresh installs would benefit from bulk lookups via the API imo.

~~~
voltagex_
Looks good. I wonder if it'd be too much scope creep to add extension points
for LAN/proxy caching. I know of one small dev team who had a single ADSL
connection shared between ~10 people - NPM downloads would have been painful.

~~~
vvoyer
I made an npm package that does this [https://github.com/vvo/npm-
pkgr](https://github.com/vvo/npm-pkgr)

------
gavinjoyce
Some further details on the issue:
[https://github.com/npm/npm/issues/11283](https://github.com/npm/npm/issues/11283)

~~~
VeilEm
Please refrain from adding unhelpful comments to the issue.

~~~
striking
How is this unhelpful exactly? It's a link to another discussion that readers
might be interested in.

~~~
X-Istence
Op was stating you shouldn't follow that link and go "Hey, me too" or "+1" on
the Github Issue.

Not that the link to the issue was unhelpful.

~~~
baudehlo
GitHub badly needs an "approve" system that is external to comments, like
bitbucket has.

------
hharnisch
It's all good, Laurie fixed it -
[https://twitter.com/seldo/status/692192238445711360](https://twitter.com/seldo/status/692192238445711360)

~~~
gruez
Link to relevant commit?

~~~
iends
[https://github.com/iarna/gauge/commit/a7ab9c906bb72aa0ed8996...](https://github.com/iarna/gauge/commit/a7ab9c906bb72aa0ed8996a00db2cd35a22d5992)

------
TazeTSchnitzel
Two years ago, PHP's package manager, Composer, was sped up by disabling the
garbage collector:

[http://blog.ircmaxell.com/2014/12/what-about-
garbage.html](http://blog.ircmaxell.com/2014/12/what-about-garbage.html)

(Or more accurately the cycle collector. You can't turn off reference
counting.)

------
freshyill
I posted this below, but I'll post again here for visibility. What the heck is
up with npm install on iTerm 2? I had no idea something was weird until
recently, when I did an install with Termial.

Here's a side-by-side video:
[https://youtu.be/odoVfHHBYVM](https://youtu.be/odoVfHHBYVM)

~~~
Syrup-tan
I had the same problem (and figured it was normal for a while), after some
looking-into it I was able to fix it by unchecking "Treat ambiguous-width
characters as double width" [0]

[0]
[https://denpa.moe/~syrup/Screen%20Shot%202016-01-26%20at%206...](https://denpa.moe/~syrup/Screen%20Shot%202016-01-26%20at%206.39.12%20PM.png)

~~~
freshyill
That did the trick, thanks!

------
Too
The funky progress bar is about the coolest thing with npm. They could let it
take a minute if they wanted too and i would keep watching it with amazement.

~~~
alanh
npm 2.x user here. Does anybody want to share a video of the 3.x progress bar?
(Easy with QuickTime or CloudApp, btw)

~~~
jffry
[https://asciinema.org/a/9gt0i1kxbto41gj4796tjkdnt](https://asciinema.org/a/9gt0i1kxbto41gj4796tjkdnt)

~~~
wasd
What's the color scheme you're using for terminal? I'm also rocking agnoster.

------
teknologist
I work in China. Downloading from the outer Internet will never match the
speed of writing to the console. Not too worried

~~~
xiaq
Not to mention that when the binary you are downloading happens match
particular keywords.

~~~
teknologist
Oh, that's fine. You can just download it from one of those mirror sites that
host their own copies of the files. You can trust that it's genuine because of
that big green checkmark on the page.

------
maaaats
Could it be linked with the terminal being used?

Edit, reminds me of this, which is why I'm wondering
[http://stackoverflow.com/q/21947452/923847](http://stackoverflow.com/q/21947452/923847)

------
libeclipse
This makes sense. It's like when I'm solving a project euler problem, I don't
use verbose output because it runs much faster without having to print
everything to the screen. (Python)

~~~
jakub_g
That's exactly my experience. Once I wrote some naive PE solution in nodejs
and added some logging at each step. After few minutes and far from a solution
I stopped it and disabled logging. I got the solution within seconds. The diff
was few orders of magnitude, which was pretty surprising for me.

~~~
arjie
Sometimes writing to the terminal can be the slowest part because it's
unbuffered by default in an interactive prompt, so even just redirecting
stdout to /dev/null (or even a file) can be faster. Had that happen some ten
years ago, so my explanation could be out of date or misremembered but the
effect was real.

~~~
btilly
Your explanation is correct. That is the semantics of Unix stdio buffering as
implemented by glibc. So across a wide variety of languages, you'll see a
significant performance difference if you are doing lots of writes to the
terminal.

On the other hand if you _didn 't_ do this, then interactive terminal programs
would be entirely unusable.

------
BuckRogers
I don't know the exact cause here, but I can say I'm thankful for my systems
background that I've gleamed for years before moving into programming. Most
sysadmin types know that displaying output on a filecopy slows down transfers
greatly because the screen vsync holds up the operation.

When I write utilities today that do similar things, I only display output
when writing and testing the program. Otherwise I'll write to memory then dump
the results in a logfile at the end.

In this case since there is probably a less intensive way of providing status
updates, and can probably be resolved by doing intermittent checks.

------
bechampion
Someone with more UI development can comment here , but i always thought
progress bars where more like a signal to the user that "something is being
done" , and not a representation of how much is done and how much is left.

~~~
masklinn
The whole point of a progressbar is to show work progress (as a fraction of
total work). The issue is it's often difficult to map "work done" to a linear
percentage, so either you do it on the cheap and your progressbar is just a
souped-up spinner with no relation to wall-clock work time or you do lots of
extra work to get more precise estimates but you end up taking more time
overall.

------
swang
I don't see this at all, what's more worrying is that npm 3.* is apparently
way slower than npm 2.*

I did this test on a npm library I wrote, installs some test tools and also
has to compile some gyp stuff

2.14.2

progress=false real 0m26.589s user 0m12.086s sys 0m3.362s

progress=true real 0m29.553s user 0m12.455s sys 0m3.486s

3.5.2

progress=false real 0m57.489s user 0m13.298s sys 0m3.421s

progress=true real 1m1.084s user 0m15.690s sys 0m3.644s

While I don't see any significant difference between progress bars or none for
3.x, what is 3.x doing that causes almost 30 seconds more churning?

~~~
rhinoceraptor
It uses a very different dependency resolution algorithm, the biggest
difference as I understand it is it will install dependencies maximally flat
whereas npm 2 was much more nested.

I believe the reason it is slower is it can't do as much parallelism as npm 2.
Also, npm 2 was susceptible to a lot of race conditions and non-determinism,
especially with compiled dependencies.

------
mannykannot
I used to joke, in the context of virus scanning, that I could give you a more
accurate progress bar, but the scan would take twice as long...

------
aidenn0
sbcl's compilation output is so verbose that the compilation instructions
mention that it builds faster on xterm than gnome-terminal

------
dannypgh
Yes, but, which download feels longer?

------
ferdamravenec
Another product bitten by the dark side of progress indicators.

One of the most often underestimated areas of product development, though
certainly not among the most important underestimates.

Easy to underestimate its cost and impact. After all, it's just a progress
indicator.

------
bitwarrior
...has he controlled for network speed? Unless he's set up a local NPM caching
proxy, he's going over the 'net for these packages. It'll take more than 1
datum before I believe this hype.

~~~
draw_down
Also the second run probably used stuff that was put in the npm cache during
the first run. Not sure how good a test this is.

A good read: "Programmers need to learn statistics or I will kill them all"
[http://zedshaw.com/archive/programmers-need-to-learn-
statist...](http://zedshaw.com/archive/programmers-need-to-learn-statistics-
or-i-will-kill-them-all/)

~~~
gavinjoyce
It yields the same results in reverse order and also when the cache is primed
for both and cleared for both

------
ww520
Screen IO is slow. A trick to speed up unzip speed, do unzip -qq. That skips
all the screen output and make it flies.

------
meric
That's the biggest issue I have with the new npm. I wonder how many developer-
days it has wasted already.

------
morebetterer
Upon first seeing npm 3.x I immediately added `--progress false --color false`
to my npm installs and never looked back. Color and terminal graphics are the
work of the devil.

~~~
coldtea
> _Color and terminal graphics are the work of the devil._

Or you know, it's 2016, and it's npm who does it badly.

~~~
morebetterer
I don't even want color and terminal graphics done goodly. I code with two
monochrome terminals on the screen using vim and the command line.

~~~
untog
Not everyone is you. While I agree that flashy graphics are silly in the
command line, I find color-based highlighting to be quite useful.

------
dschiptsov
Well-designed server platform.

~~~
mrmondo
Don't even get me started on the number of reasons why nodejs should not be
running on servers, I'll be here all day. We're currently deploying a NodeJS
app and we'll never do it again - the amount of problems with that ecosystem
is quite amazing. As I tweeted earlier today "JS ppl tend to have marketing &
UI/UX experience so JS crapcode gets popularised, becomes widespread, then we
have to deal with the fallout".

~~~
pixelHD
I'm genuinely interested. I've been using nodejs for hobby projects and
scripts, so haven't seen too many issues. But I'd like to know how things are
and what issues you've been seeing.

~~~
mrmondo
Howdy, we've had lots of issues with single threaded performance, poor memory
management especially around garbage collection, we had to modify our SELinux
policies as it likes to execute memory off the stack, npm has been very
unreliable and often unreasonably slow, error handling of the node index app
server seems to have some interesting behaviour but I haven't dug into that
one too deeply yet and the list goes on. As an experienced systems engineer
it's very clear that this is browser technology that was never designed not
fit to run on servers.

