
Flat Tree Dependency Resolution in Npm v3 - funerr
https://docs.npmjs.com/how-npm-works/npm3
======
kylecordes
I understand that this is a hairy, messy problem they are solving, and that
they traded off one aspect of the problem for another (Install order
dependency! as a new "feature" in 2015!).

But I wish they had aimed higher. The better goal would be that the entire
state of the node modules directory is a pure function of the contents of the
package.json file plus the platform details (compiler used for native
modules).

While they're at it, the ecosystem could be improved considerably if there was
some sort of obvious "penalty" applied to any package that compiles native
code, because such things cause considerable extra trouble for Windows users.
A visible penalty, transitively carried up the dependency tree, would
discourage use of such modules with native code; projects would use them
(depend on them) only if absolutely necessary instead of accidentally, all
over, all the time.

~~~
city41
Why do dev communities have to continually pay the Windows tax? If Microsoft
keeps insisting on being an outlier in the world of OSes, shouldn't they be
the ones burdened?

I gotta say, as someone who never ever uses Windows but maintains a couple of
open source packages, I'm really sick of the Windows only problems that crop
up.

~~~
tvanantwerp
As someone who has no choice but to use Windows at work, I'm really sick of
all the problems that crop up because so many tools use were written by people
who won't use Windows and haven't tested anything on it.

~~~
tsuraan
That's a funny point of view. Windows is made by a multi-billion dollar
company, rented by countless wealthy software development shops, but it's
volunteer open-source programmers who are supposed to take on the burden of
supporting it? Seriously, I can't even comprehend that thought process.

I write software for fun, I release it on github, and some people find it
useful. If there are other people who don't, they can improve it for their own
purposes, or they can find or write something better. I don't think anybody
has a right to expect me to support Microsoft's product. That's insane.

~~~
Silhouette
_I don 't think anybody has a right to expect me to support Microsoft's
product. That's insane._

True enough, though it's also fair to consider a language/platform/ecosystem
inferior for some work if it relies on that kind of individual and often less
portable contribution for its effectiveness. A lot of FOSS advocates will
promote community support and a broad contributor base as advantages of that
style of development, but there is another side to that coin, which is that
sometimes you get just what you (didn't) pay for and you're on your own in
terms of support or working around any problems.

------
tlrobinson
_" dependency resolution depends on install order"_

Does this sound insane to anyone else?

EDIT: I understand not wanting to modify node's `require` semantics, but this
is an unacceptable sacrifice of consistency for efficiency. Surely it would
have been possible for an `npm install --save x` to put the `node_modules`
directory in a state identical to `npm install --save x && rm -rf node_modules
&& npm install`. It might take a little longer to shuffle some directories
around, but certainly not longer than a full `npm install`.

~~~
frank-weindel
What's more is that standard `npm install` install order is ALWAYS alphabetic.
Which causes some packages, purely arbitrarily, to influence directory
structure.

------
theGimp
That's a great improvement over the old behavior, but I'm wondering why the
team didn't go a different route.

Why not always store packages at the top level, and create a directory for
each version that's required?

For example: directory "A" contains two subdirectories: "v0.1.0" and "v0.1.1"

~~~
dc2
When your actual Node .js file runs `require('a')`, it has no context to
determine which version it is requiring. To expect Node to look for a
package.json file at runtime is absurd. This presents a problem.

~~~
theGimp
I'm guessing the current behavior for require is to bubble up the directory
structure and look for "node_packages".

So resolving things at runtime is already happening.

I agree using package.json at runtime is not the best solution though.

A way to keep things clean and avoid using package.json is to use symlinks
instead of downloading a fresh copy, which would make putting everything at
the top level a possibility.

I'm sure that option was considered, but I'm curious about why it was not
taken.

I suppose I should look at the discussion notes. Hope I'll remember to do that
when I'm home!

~~~
nawitus
Symlinks would make copying and archiving the node_modules structure more
problematic.

------
vec
So, assuming I have two libraries, A and B, that both require the same version
of library C, do those libraries still get their own separate in-memory copies
of C, or do they share a singleton?

It's terrible practice, but it's not unheard of for an NPM module to monkey
patch its dependencies, since before this the library could assume it had sole
ownership of its whole subtree.

~~~
Killswitch
If depend on A and B and both A and B depend on the same version range as C, C
is now a top-level dependency.

Your node_modules will look like this:

    
    
        - Package_A
        - Package_B
        - Package_C
    

It's only when A and B depend on different versions of C that cannot be
resolved via semver as safe.

    
    
        - Package_A
        -- node_modules
        --- Package_C
        - Package_B
        -- node_modules
        --- Package_C
    

I am pretty certain that monkey patching your dependencies is frowned upon in
the Node world. It's best to fork the repo make your changes, and then depend
on that.

~~~
randallsquared
Sadly, this is the result of the second situation:

    
    
        - Package_A
        - Package_C_vX
        - Package_B
        -- node_modules
        --- Package_C_vY

------
vfc1
Jspm solves this by installing the module in a directory appending the package
version. There is no maximally flat tree, the tree is 100% flat. At most there
are several versions of the same package side by side, but no nesting.

It even supports circular dependencies.

~~~
nmjohn
Are there any trade-offs to this approach? This seems so obvious to me I'm
confused why it is more widely used.

~~~
zebracanevra
In node, require()'ing a dependency is stateless - it searches ./node_modules/
for the module, then ../n_m/ then ../../n_m/, etc, until it finds an
appropriately named module.

In JSPM/SystemJS, require()'ing/importing a module is still by name (as it
supports NPM modules), but the package.json file has to be parsed in order to
map module names to an installed module version. Note, this mapping is only
done in the developer environment - once you build a bundle all the mapping is
statically compiled into one file.

------
amasad
This is good news for client side bundlers like Webpack and Browserify --
where size really matters -- they don't have to end up with multiple copies of
the same module.

I would also assume for very large apps it may improve startup time because
you don't have to initialize and retain multiple copies of the same module.

~~~
frank-weindel
This has been around for a number of months. The sad thing is because of this
unpredictable (or rather arbitrarily alphabetical) `npm install` order
different dependency trees can result which can still lead to a very common
module being bundled multiple times. I was a fan of bower's strictly flat
model because it prevents such duplication and even notifies you when
incompatibilities occur. However bower seems to be losing the battle with NPM
as the defacto web/javascript module repo.

The allowed/unpredictable duplication can even cause very hard to identify
bugs when a peer dependency relies on "instanceof" checks and there are
multiple versions of this dependency. I've seen it happen with React and
Backbone to name a couple.

If the `npm install` allowed control over install order (instead of just being
alphabetical) and there was a way to be notified of incompatibilities that
would cause potentially unnecessary duplication that would be at least
something that could prevent problems like this from occurring.

------
ag_dubs
as the person who wrote these docs-- if you have questions or things you'd
like to see addressed, i'd really love if you filed issues on the repo.
[https://github.com/npm/docs/issues](https://github.com/npm/docs/issues)

------
ruffrey
If this works it might reduce the slug size of a built node app by a lot of a
little.

Though, if there are problems, I wonder - can the flat dep resolution be
disabled using some CLI flag? Or when installing deps, or in .npmrc, or during
a shrinkwrap?

------
tolmasky
I just filed a bug:
[https://github.com/npm/npm/issues/10999](https://github.com/npm/npm/issues/10999)

I guess I'm not sure what level of non-determinism they expect, but on this
page: [https://docs.npmjs.com/how-npm-
works/npm3-nondet](https://docs.npmjs.com/how-npm-works/npm3-nondet) it
appears to make the claim that the only effect is on tree structure, not the
actual versions of packages that are picked up. And in fact in their example
this IS the case. I think this is fine btw.

However, I have found edge cases where install order actually changes the
versions of packages that are picked up, and in ways that make it very very
difficult to work around (basically you will be forced to manually edit a
shrink-wrap file -- so it is necessarily on the end user not the package
writer).

Basically, if any package lists and absolute dependency (vs a semver range),
it will affect ALL the packages alphabetically later than it and FORCE them to
take the same dependency.

------
bhouston
I do not understand why they just didn't have a two level directory structure:

node_modules/[module_name]/[version]

Then it would be flat and support multiple versions of the same module in a
way that is completely deterministic and also fully deduplicated.

This new system is unnecessarily complex.

~~~
JaRail
The file system layout is constrained by how dependency resolution works. For
example, a require() call has a very (computationally) simple algorithm.

I certainly agree that your suggestion simplifies file-system layouts. The
tradeoff is that the complexity shifts to other parts of the system.

That said, I'm not a fan of the v3 approach. I'd have preferred a central
package cache with a structure similar to your suggestion. I'd add that each
package in the cache should have all of its dependencies resolved in its own
/node_modules/ dir with symlinks. Unfortunately, I still can't see a nice way
to handle peer dependencies. Peer dependencies require the ability to walk up
the file system to resolve, which you can't do with symlinks.

------
hoodoof
Sorry to be negative but one of the things that I hate more than anything in
my software development work is typing "npm install blah" and almost always
being hit with wave after wave of errors typically related to dependencies. I
don't know why it happens and I don't care I just wish they'd fix it. So many,
many errors.

Go on, try installing X packages at random using npm - did they install
cleanly?

The baseline outcome for using a package installer should not be reams of
errors, it should be a cleanly installed package. Installing packages works
fine with other language ecosystems, why not with npm?

~~~
Touche
NPM has some dumb legacy features like optionalDependencies.
optionalDependencies are often native code dependencies that might fail but
the package is still usable, maybe just some specific feature isn't. That's
the most common reason I come across for errors.

~~~
hoodoof
Whatever the reason, they should fix it - I spend all day fighting through
vast numbers of problems and errors in all sorts of software systems but
nothing ranks as high as "npm install" for bad user experience. It's the one
thing that browns me off more than anything - I dread having to type "npm
install" and its my number one gripe in a world of broken software which is
really saying something when so much software is broken.

------
shill
This will help Windows users who have filesystem and zip compression issues
with deeply nested dependency trees.

~~~
frik
Windows to this day still has the 260 chars path limit.

[http://msdn.microsoft.com/en-
us/library/aa365247(VS.85).aspx...](http://msdn.microsoft.com/en-
us/library/aa365247\(VS.85\).aspx#maxpath)

[https://en.wikipedia.org/wiki/Long_filename](https://en.wikipedia.org/wiki/Long_filename)

Microsoft should finally fix such old limitations (-> update WinAPI), instead
adding work arounds to third party projects like Nodejs.

[https://github.com/Microsoft/nodejstools/issues/69](https://github.com/Microsoft/nodejstools/issues/69)

They could also improve the command line shell (cmd.exe) that also PowerShell
relies on.

~~~
bcoates
That's a serious breaking change to existing windows programs and can safely
be put in the "never going to happen" pile.

------
brianbarker
V2's deep nesting recently caused trouble for us at work. We had errors where
our node_modules folder was too long for Windows, due to the deep folder
structure. Updating to v3 resolved the problem. Hopefully other flaws being
discussed here can be worked out as npm evolves.

------
justinsaccount
Will this prevent node_modules/ from having 17 copies of the the same file in
17 different places?

~~~
ksherlock
yes, as long as there aren't version conflicts.

~~~
baudehlo
Which there will be because everyone uses --save so each module has the
version around when they added the module.

~~~
pauly
\--save adds the fairly loose ^ version number, so any major version should
match, so less conflicts than if you'd added an exact version match, or a ~
match

~~~
Touche
A good percentage of npm modules are <1.0.0 so you're very likely to get
dupes.

------
z3t4
I think they are trying to be too smart about it ... I rather waste several
GB's of HDD space then having my production crash _once_ because of
dependencies.

