
Node's nested node_modules approach is basically incompatible with Windows - tsing
https://github.com/joyent/node/issues/6960
======
girvo
Know the biggest issue I've had with Node (well not me, but my colleagues)?
That modules have a habit of using symlinks, which break horridly across
Vagrant/VirtualBox shared folders. We ended up fixing it somewhat by using the
--no-bin-links option on `npm install` and swapping over to SSHFS, but I
really wish module writers would pay more attention to Windows... one of
Node's strengths is that it works pretty darned well on Windows (just like how
XAMPP made PHP deal with windows well enough for a large developer base to
grow)

~~~
rmrfrmrf
Again, this is a Windows issue with its horribly broken symlink handling.

~~~
Argorak
So, let's go fix it.

Sorry, but complaining about the base platform behavior doesn't help much if
you want to build software on it.

------
addisonj
I am somewhat surprised at this topic. Of all the the issues Node.js has on
windows, this seems like a fairly minor one. In my eyes, the biggest issue is
simply this:

A lot of module writers don't care about Windows.

A big part of the node ecosystem is that there is a module for practically
anything. While node core has great support for windows, and also helps aid
developers in writing code that is windows compatible, from what I have seen,
its not often followed.

Any number of simple things break modules on windows, using a '/' as a path
separator instead of path.join, native modules, bash or shell scripts used in
pre- or post-install scripts and probably a hundred other small ways to screw
it up. Even trying to make your own module Windows compatible is often
thwarted by a dependency that doesn't. Like Java's promise of "Write once, run
anywhere", its a great idea, but hasn't completely panned out.

I really hope that as the node community matures, there can be more of a focus
on making sure things work on Windows. Node is a great platform partially
because of all those modules and getting them all working will make writing
cross platform apps and services a heck of a lot easier.

~~~
brokenparser
Windows actually supports using / as a path delimiter. If a program is
(following the robustness principle) liberal in what it accepts (both / and
\\\\) and strict in what it sends (just /), it'll work (barring any other
issues).

------
lucisferre
I actually find it surprising how many people feel recursive node_module
dependency folders is justified. It seems absolutely the worst possible
solution to me. Am I misunderstanding something about this. Is there any
reason Bundler's approach doesn't work here?

It seems ridiculous to me how many folders are created when I just need a
couple of node modules to do something.

------
lucisferre
This is going on a bit of a tangent, but even Visual Studio used to run up
against this problem all the time using some of Microsoft's own libraries. It
was laughable at the time, but some of Microsofts teams ( _cough_ Patterns &
Practices _cough_ ) loved to created DLLs with huge long namespaces. That
coupled with the fact that Visual Studio projects were created in
/Users/User.Name/Visual Studio/Visual Studio ##/Projects/ to begin with
created a lot of problems.

As a result I almost always put all my projects at the root something like
/src to reduce the odds of running into these problems.

------
dbbolton
This definitely seems more like a Windows issue than a Node issue.

~~~
nopar8
At first I agreed with you; but then I read one of the comments on github that
swayed me.

"You are the one playing games - calling core parts of Windows like Explorer
"3rd party tools", and suggesting that not supporting long paths is a bug.
Microsoft have made it clear repeatedly that non-support for long paths is not
a bug, and not something that will change.

A package manager creating paths that do not work with the majority of the
software written for an OS, then claiming compatibility with the OS, is
playing games, at your users expense."

At the end of the day Windows wont be the thing changing. haha.

~~~
nevi-me
I don't follow, should we then rename everything to cryptic, short characters
like some people do in MongoDB (as someone suggested using n_m instead of
node_modules)? If it works well in other platforms, it's an issue with that
specific platform

~~~
rando289
windows has path length limits. Figure out how to live in them, whatever way
is best. If that means using cryptic names, then do it. cryptic folder names,
while not a good thing, are extremely common in windows.

~~~
tlrobinson
Or, don't support Windows.

~~~
rando289
Yes, fine by me.

------
rtpg
Was the recursive "every package include its dependencies" done just because
it was easier? Or was it a conscious design choice.

~~~
aikah
Well at least there is no possibility of conflicts(unlike some other pm). The
problem is people writing modules that have like 1 function while requiring 10
other modules that have just 1 function themself that require 10*10 other
modules ... Of course it doesnt scale,and of course the package manager wont
scale either...

But it's not a NodeJS problem it is a NPM problem.NodeJS has other problems
though(the lack of governance by a dedicated foundation is the most important
one,the obvious second one is nodejs dependency on what google does/doesnt
do,since google maintains V8).

~~~
jeswin
> second one is nodejs dependency on what google does/doesnt do,since google
> maintains V8

Google maintaining V8 is one of the big reasons why Node has been successful.
There is no way an independent Node.JS foundation would have the resources to
optimize and test JS like the Chrome team. So far, the problems have been
minor and V8 has done a decent job keeping up with upcoming JS standards.

So this isn't a problem.

> 1 function while requiring 10 other modules that have just 1 function
> themself that require 10*10 other modules

As of now NPM is working (in spite of these problems) because those modules
are tiny js files. I agree it would be beneficial to have better package
management.

In fact, that's exactly what's giving me problems right now. My browserify
task breaks due to the NoYieldInGenerator error in parsing. The dependency
(esprima parser) has been patched upstream, but now that has to make its way
into 4 dependencies inside browserify, and then browserify has to update
itself. My alternative is to make 5 forks just to solve this. :/

It would have been great if they all used a single shared lib, and even better
if I could manually override dependency versions of my dependencies.

~~~
aikah
I wonder why Mozilla didnt/couldnt jump in the ship and propose a viable
alternative to node/V8,I know there is a node/spidermonkey project somewhere
but it isnt maintained.

V8 api changes made a lot of binary modules break and it will happen again,so
it is a problem on the long run.Of course NodeJS team cant maintain V8.

But please dont throw that problem under the carpet,It's a real issue.

> Google maintaining V8 is one of the big reasons why Node has been
> successful.

Sure,but it doesnt guarantee Nodejs survival on the long run.Look, nodejs is
an awesome project,i'm not here to question that fact.I use nodejs everyday I
love it.I just cant trust Google on anything on the long run.

Do V8 team consult with nodejs team,listen to their suggestion ? (that a
question).Is it a bidirectional relationship or a unilateral one?

~~~
brokenparser
There exists "gjs" and "js" ("smjs" in Debian), both of which are being
maintained.

------
abhididdigi
The title looks exaggerated. I thought NodeJS cannot just run on Windows
anymore. It's not the case though, it does - it's just it creates folders that
are nested.

~~~
malandrew
Totally. This is actually a trivially solvable problem (I've been working on a
meta package manager compatible with npm, bower and component).

The strategy I took with node_modules/, components/ and bower_modules/ was to
just follow whatever folder structure it found as far as necessary and look
for manifest files (package.json, bower.json and component.json) and register
modules at the path of wherever one of those three file types is found. This
approach allows any folder organization whatsoever. This information was then
saved to a key-value store and the store is only updated when the mtime of the
folder changes. This makes startup relatively fast since folders are only re-
indexed if something has genuinely changed.

Writing an algorithm that is dependent on the folder structure to me is a
fundamentally bad idea most of the time, especially when you have a manifest
file that better identifies a module. Additionally, for modules with semver
ranges or installed from unusual locations (i.e. URIs: git:// git+ssh://,
[https://](https://), etc.), you can write an additional metadata file with
install information and version locking information (like gemfile.lock)

If people don't rely on the file system organization as an API, npm and node's
require() algorithm can trivially reorganize or even use different folder
organization schemes on different operating systems if there were a good
reason to do so.

~~~
jonpacker
I respect where you're coming from, but I disagree. One of the huge advantages
of npm for me is that it's so transparent and I can see exactly how it's
mapping to the file system. I know exactly which code is being run, from
where—because it's incredibly easy to traverse the dependency tree. If you
abstract away the fs level and flatten the dependencies into a single folder,
you take this away, and you make the dependency tree essentially a big
intimidating blob of code that requires slogging through package lists to
determine what is actually being used.

~~~
malandrew
Nothing in what I'm doing prevents what you like about npm. In fact npm
already abstracts away what you want in a little known command called `npm
explore`.

Let's say you want to explore the dependency `foo` in your project. At the
root of your project type `npm explore foo` and press enter and you'll be in a
shell exploring the foo module. Then if you want to return, type `exit` and
return and you'll be back in your project. Or if you want to go down the
rabbit hole to a dependency of foo called bar, type `npm explore bar` and
press enter.

`npm explore` is a great way to explore things and navigate to the
dependencies used by the dependency you want to investigate. It basically uses
the information in the current package.json to find the dependency you want.

------
derefr
I'm surprised the obvious solution isn't already implemented universally by
package managers: checkout the modules into a single directory, named as
"[name]-[commit SHA]". You get all the benefits of the current system, with
the added benefit that if A and B both turn out to want to check out
C#1a029fb, then you only get one copy.

~~~
jkrems
That benefit would be a breaking change. Node modules are stateful, so two
packages suddenly sharing their dependency could have nasty side effects.

~~~
derefr
Eek, I didn't realize anyone thought it was a good idea for modules to be
holding state inside themselves.

Simple-enough fix, though: drop an extra directory in there, call it "state",
and have each state-container in that dir be named after the the hash of the
dependency path you would have traversed to load it. Virtualize modifications
to packages into those state-containers.

(This is also pretty much how Windows protects itself from programs that think
keeping their state in the Program Files directory is a good idea.)

~~~
jkrems
I'm not sure we are talking about the same thing: I was talking about runtime
state (think: singleton instances). A directory wouldn't solve that problem.

------
frik
The Win32API was initially designed to make it very easy to port from Win16API
in the early 1990s. Microsoft neglected to improve the Win32API starting with
the dotNet adventures. Instead of an improved Win64API we have the identical
API for 32 and 64 bit (and WinRT API). The shell still doesn't support UNC
path. The Windows NT series operating system would support different
subsystems other than Win32.

 _Naming Files, Paths, and Namespaces_ on MSDN explains the details:
[http://msdn.microsoft.com/en-
us/library/windows/desktop/aa36...](http://msdn.microsoft.com/en-
us/library/windows/desktop/aa365247\(v=vs.85\).aspx)

------
linhmtran168
Have the same problem. Although I'm a Mac user, but I'm using OneDrive to
store some toy projects. OneDrive stopped working and complained about the
long folder path when I tried to install a NPM package. After some researches,
I know this problem will not be fixed soon and move all of my Node projects to
DropBox. For Windows user, I think the only solution now is to use a VM.

------
wolfgke
Windows does support long paths (up to about 32k characters): Here is a good
explanation from the .net perspective (but the same holds for non-managed
APIs): [http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-
path...](http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-
part-1-of-3-kim-hamilton.aspx)

Long paths are just most ugly to use: "Long paths with the \\\?\ prefix can be
used in most of the file-related Windows APIs, but not all Windows APIs. For
example, LoadLibrary, which maps a module into the address of the calling
process, fails if the file name is longer than MAX_PATH. So this means
MoveFile will let you move a DLL to a location such that its path is longer
than 260 characters, but when you try to load the DLL, it would fail. There
are similar examples throughout the Windows APIs; some workarounds exist, but
they are on a case-by-case basis."

------
pingec
Long paths are a PITA on Windows sometimes :(. A while ago I had a problem
with long node module paths and msysgit. Luckily, this was fixed in v1.9.0

[http://pingec.si/blog/articles/msysgit-
longpath/](http://pingec.si/blog/articles/msysgit-longpath/)

------
asavi
The easiest solution I've found is to just run a VM and not sync whatever
folder you install node_modules in. I just had this problem like a week ago
with gulp because I was syncing the folder I was building in. It's a good time
to learn how to use Vagrant.

~~~
Argorak
There are people using node as part of their desktop software for scripting.
Atom for example. Vagrant that.

------
pickettd
So far I've been able to get by in Windows (for my projects at least) if I use
Cygwin for interacting with paths that are longer than MAX_PATH. For example -
if a project has node_modules committed in git then I have to clone the repo
using Cygwin. This doesn't help with the problem related to install scripts on
deeply nested components though because npm doesn't support Cygwin. In Windows
I still run npm and node commands in a regular command prompt.

------
doverton
I ran into this yesterday, having tried to install yeoman into a folder shared
between a windows host and a linux vm. Ended up with files which could not be
deleted by any other way than from the windows command prompt using the 8.3
filenames.

An hour of swearing was enough to convince me to stop using windows for this
entirely.

------
rgawdzik
What percent of the node.js community will this effect?

What's the design reason for Windows tools to only have a max of 260 file path
chars? If it's not the OS, but the tools he is using, why not use tools that
work? Isn't this the tools bug?

Honestly, if he is doing web development, his life will be easier with a linux
distro.

~~~
mappu
_> What's the design reason for Windows tools to only have a max of 260 file
path chars?_

So in C you can just say `char buff[MAX_PATH] = {0};` and not worry about heap
allocation.

That's also the reason why it can't ever be extended - because MAX_PATH is
compiled into every binary and you would create instant buffer overflow
vulnerabilities in a lot of software.

It was a reasonably good idea at the time the decision was made.

~~~
rgawdzik
Fair enough, makes sense from the C perspective.

------
aikah
isnt it the reason why python ,ruby and co install themself in /c/ by default?
I always wondered why they did that. PHP and Java dont have this problem(but
"hello" package conflicts...).

------
tsmash
Entering my serious vote to end node development on windows

------
rmrfrmrf
I can't stand hostile issue submissions like this.

------
panarky
This looks like a Docker container on Digital Ocean.

I would use this a lot, but many errors trying to apt-get install packages.

