
Ride down into JavaScript dependency hell - nikolalsvk
https://blog.appsignal.com/2020/04/09/ride-down-the-javascript-dependency-hell.html
======
tmcw
How to write a silly post about JavaScript dependencies:

\- Don't mention other languages, lest you reveal that most of them have
similar dependency bloat problems.

\- Talk about devDependencies and dependencies without considering how they
might be entirely different, say… one kind gets bundled, the other doesn't.

\- Always use npm, so you can highlight duplicates. Don't use yarn.

\- Adopt a narrow definition of 'dependency hell' so you can focus on the
thing that JavaScript does imperfectly (minimizing number of dependencies) and
avoid talking about the things that it does well (handling conflicting deps,
solving diamond deps, resolving dependency chains)

\- Don't worry about explaining how any of the issues you're discussing have
real-world implications. Have companies been failing because they run out of
disk space because of node_modules?

\- Take automated security audits extremely seriously. A backtracking bug in
the argument parser for the web framework's local CLI tool? Of course it's not
exploitable, when… it's a web app, and the CLI tool is something you run
locally. But add it to the count as if it is!

~~~
ng12
They also strategically avoid mentioning the payoff which is that your project
is entirely self-contained. `node_modules` is at once your complete
development kit, build tool, compiler, and collection of runtime dependencies.

Once you check out a node project with a package.json you can reasonably
assume you can install it and run it anywhere. A docker image for a node
project should just be "install node && npm build". You're side-stepping the
entire dev ops nightmare. Find me another language with JavaScript's
popularity and that level of simplicity. I'd go as far to argue that this is
what happens to any massively popular language that has a functional package
manager from day one.

~~~
stickfigure
I'm totally baffled by this statement.

Find me a mainstream language that's more complicated than this? "Checkout and
build" works pretty much everywhere? I mean, you need the right version of mvn
or pip or whatnot... just like you need the right version of npm.

If anything, node/npm deserves special mention for NOT having reliable
repeatable builds. Part of this is that a lot of packages rely on native code.
But most of it is that the ecosystem culture defaults to "always grab the
latest versions of everything".

And npm only got package-lock.json a few years ago, not "from day one". Prior
to package-lock.json, builds were _wildly_ unpredictable - like, expect
breaking changes week to week even if you don't change anything at all in your
code.

If you want an "entirely self contained" payoff, languages that produce static
binaries are pretty hard to beat. Node is not that.

~~~
aaomidi
Yes and now lock files exist?

Are you actually arguing over the state of JS development from 4 years ago?

~~~
stickfigure
I have pointed out that, contrary to the parent, Node is not special, and
certainly wasn't "from day one" as claimed.

------
dwheeler
Part of the issue is that JavaScript packages are often far far smaller than
packages in other ecosystems.

"small packages are extremely common in the JavaScript npm package system. For
example, in npm, 47% of the packages have 0 or 1 functions, and the average
npm package has 112 physical lines of code. In contrast, the average Python
module in the PyPI repository has 2,232 physical lines of code."

Source: "Vulnerabilities in the Core: Preliminary Report and Census II of Open
Source Software", The Linux Foundation & The Laboratory for Innovation Science
at Harvard, by Frank Nagle, Jessica Wilkerson, James Dana, Jennifer L.
Hoffman. [https://www.coreinfrastructure.org/programs/census-
program-i...](https://www.coreinfrastructure.org/programs/census-program-ii/)

(This is a contrast of citations from other works; you can see the specific
citations in the report, but I'm trying to do this from a phone. In any case,
the issue is the massive difference between JavaScript and everything else,
and I think this quotation shows it nicely.)

~~~
dep_b
iOS dependencies in Cocoapods are including less than 1 other packages on
average (1). The most I see are Alamofire (a networking package people keep
using for reasons that I do not fully understand) and Starscream (a Websocket
implementation, that's pretty hard to write from scratch to be honest). Having
unnecessary dependencies is pretty much frowned upon.

One of the reasons is that the iOS SDK is really complete by itself while
Javascript doesn't even ship with a usable Date implementation. If somebody
could just come up with a standard library for JavaScript that is really
widely adopted I think the amount of packages can be cut back dramatically.

(1) totally based on gut feeling, not based on hard data

~~~
hn_throwaway_99
Java didn't have a decent date implementation until Java 8.

> If somebody could just come up with a standard library for JavaScript that
> is really widely adopted I think the amount of packages can be cut back
> dramatically.

As this article points out, I think that lodash is that for many people.

~~~
toyg
Java has had _exact_ datetime implementations since what, 1.3? 1.2? Sometime
in the ‘90s anyway. They weren’t fun to use but they worked fine, had the
concept of multiple calendars etc...

JS’s version of Date in comparison has always been a toy.

~~~
hn_throwaway_99
JS's version of Date is pretty much exactly the same as java.util.Date: just a
thin veneer around a long representing ms since the epoch. Yes, Java has had
Calendar and DateFormat for years, but the API is generally recognized as
being pretty horrible (how many bugs were caused by Date and SimpleDateFormat
not being thread-safe?)

So the Java community coalesced around Joda time, not unlike how the JS
community coalesced around moment.js. The big difference is that yes, Java
eventually rolled these lessons learned into the java.time API, but now you're
effectively stuck with 2 standards, Joda and java.time. Not sure if that's
much of an improvement.

------
dgellow
I think that we are in a situation where the number of dependency doesn't have
any meaning anymore for people working on web frontend projects. Recently
while revamping parts of the CI/CD at $WORK, I found out that running npm
install for one of our frontend project downloads around 1 million
dependencies. And npm audit reports a completely ridiculous number of security
issues with them.

It's just so absurd and nonsensical, nobody can understand what that even
mean.

Seems to me to be a crazy huge liability, but quite a lot of people seem to be
fine with that.

I don't know what to do from that information, but trying to create some
awareness around the issue feels like talking about the number of operations
per second a CPU does with someone who has zero idea what a realistic range
is. It's seen as a high number, but doesn't have any meaning attached to it.

Edit: I don't have access to the project anymore to check more details (such
as installation time), but I checked some personal notes I took when I found
this out, the redacted output from npm audit is the following

    
    
        $ npm audit
        <REDACTED> vulnerabilities found - Packages audited: 927016
        Severity: <REDACTED> Low | <REDACTED> Moderate
        Done in 41.62s.
    
    

So slightly below the 1 million mark.

Edit 2: Now I'm wondering ... maybe the "packages audited" from npm audit
doesn't mean "you have that number of packages, and I audited them". But if
that's the case, that's a terrible UX, and I have zero idea what that number
mean, which kind of support my point.

~~~
jfkebwjsbx
How is this even possible?

How are there 1 million interesting packages?!

In my job we use like, I don't know, 20 libraries total, out of, say, 50
alternatives that could do the job. Including dependencies of the dependencies
I don't think we reach anywhere near 100. Every dependency is discussed with
the entire team when we add it.

How do you even have time to manage 1 million?!

~~~
BiteCode_dev
To quote the article:

"Luckily, most of them are using the same version of lodash, which just needs
one lodash to install inside node_modules. This is often not the case with
real-world production projects. Sometimes, different packages require
different versions of other packages."

Npm lets you install several versions of the same package. This has the
advantage of saving the dev the testing of their lib with a wide range of
dependencies versions: it almost always works. Unfortunately, that means most
devs just choose one version and call it a day. They build libs like one would
build a project: as you were alone in the world.

------
Waterluvian
I have a project with a total of eight dependencies. I walked away for six
months as it was stable. I come back to add a few features and npm tells me I
have over 38 000 vulnerabilities of different severity levels. So many that
'npm audit' just freezes up.

So there's that too.

~~~
samsquire
My stuff breaks whenever I try upgrade it. People refactoring stuff.

~~~
brobdingnagians
These types of issues are the biggest reason why I avoid javascript/npm
projects. I came back to a project after a few months and it was broken, had
to rewrite some parts and upgrade other parts just to get it to run again.

~~~
williamdclt
I'm sorry but for all the flaws of the NPM ecosystem, version locking is not
one (both `npm` and `yarn` do much much better than anything in Python).
There's no reason a correctly set-up project would magically break with time.

I don't love NPM anymore than the next guy, but my blame will go to the dev in
this case.

~~~
Waterluvian
Python dependency management is a disaster. But I find myself using far fewer
dependencies with Python projects so I suffer the pain less.

~~~
jessaustin
This is more of that "wet streets cause rain" logic we hear about...

------
tluyben2
I cannot understand how people work with this. I work with many different
technologies and try to avoid JS, but some times I have to. The past weeks I
have worked on a React Native project that was written by someone else; what a
horror show. I mean it wasn't the worst code by the previous guy but even in a
few months a lot is simply broken and not 'best practice' anymore. Compared to
most other environments I work with, iteration is somewhat faster when you
finally have everything working, but the dependencies and ecosystem is
horrible imho.

A bigger issue with all of it is that in my line of work, for the backend of
firmware I cannot just depend on 3rd party libs as they will get audited, so I
need to audit them before. Ofcourse most people don't have that issue.

~~~
williamdclt
I don't agree with you on this. The JS (or NPM) way makes it a pain to audit a
project due to thousands of dependencies, but it's far easier to "have
everything working".

`npm i && npm start` is as good as it gets in terms of "getting started"
friction (with version locking): python is worse, ruby is worse, PHP is worse,
Java is worse.

React-Native has more moving pieces indeed, but it's not really JS's (or
NPM's) fault: you get all problems of iOS and Android at once. I will
definitely agree with you that working on React-Native is a pain, to be able
to work with modern tools need to know JS, optionally TS, Obj-C, Swift, Java,
Kotlin, Ruby (for Fastlane) and Gradle's custom language. Insane.

~~~
dgellow
I've done 3 years of Golang as my main language, my experience is:

\- go build ./... builds everything from the current project

\- go test ./... tests everything from the current project

\- go install ./... installs everything from the current project

That's one of the best thing ever. I can go to any project, I know how to
build, test, install, and can directly be productive.

On the other hand, I started learning C++ one year ago, and it's the exact
opposite, each project has its own homemade build system that isn't supported
by one platform or the other.

~~~
diggan
Golang does a lot better than other languages (but probably the language that
allowed me to use the most unmaintained/oldest projects is Clojure, I can't
remember/find out when the oldest breaking change to the language was), but
the situation in Golang was piss poor not too long ago, when Golang libraries
were all fetched from GitHub/Git sources without any way of specifying
specific versions, you basically lived on the edge all the time. There was
third party tools to work around this (and the ever present vendoring
technique) but seems weird they didn't realize this until gomod started being
a thought.

------
montroser
It's not really hell though, is it? Hell is infinite pain and agony. This is
more like an occasional bother.

Yes, we get comically large numbers when we look through node_modules. But,
real "dependency hell" is when you have situations that take unbounded manual
effort to resolve.

How often do we end up with impossible circular dependencies? Or conflicting
versions clobbering each other? Or non-deterministic builds introducing random
behavior?

That all is commonplace even today with other platforms like python, and
rarely an issue with node. I'd much rather occasionally sneer at the size of
node_modules than any of that actual hell.

~~~
uhoh-itsmaciek
Yes, exactly. I used to think isolating dependencies like npm/yarn do, where
each package has its own copy of its own dependency tree (ignoring deduping),
was crazy, and potential conflicts should be a forcing function to encourage
you to minimize dependencies in the first place. But then I started using
these systems and it's by far the least time I've ever spent worrying about
deps. There are downsides to this and how it encourages dependency
proliferation, and I still believe in minimizing your dependencies, but it's
the least bad system I've seen.

------
parhamn
What’s worse is that the whole installing multiple versions of the same
package for different sub dependencies doesn’t make any sense when when the
language supports singletons and module level variables with has no way of
specifying which version to import.

I was shocked this behavior exists when a site broke by upgrading a sub
dependency. Turned out they both installed react so they used a different
“creatContext” and now there were two context instances instead of one.

~~~
tristanstcyr
React would normally be a peerDependency and so you shouldn't run into this
problem if dependencies are specified properly.

If not, NPM module flattening should take care of it.

Then worst case, you can also tell webpack to bundle a specific installation.

There are some ways to solve these problems...

~~~
parhamn
Sure (upvote) and we dealt with it accordingly after some confused digging. On
a more general level I’ve never seen something like that in a package manager.
It seems as though some of the language features (like the singleton I
mentioned) would make it obvious that arbitrarily duplicating packages and
scoping them to sub packages is a major no-no, as in most languages I’ve dealt
with.

~~~
jessaustin
It seems like a weird thing about React, or maybe how you're using it, that
this is a problem. Most packages, and the language itself, handle multiple
simultaneous versions of dependencies without issue. If packages are doing the
old-fashioned "stick a property on window" thing instead of just getting
required or imported browserify-style, they can step on each others' toes (and
it's totally possible that React is dumb enough to step on its own toes). It
has been years since that was necessary. browserify or more recent tools like
parcel or webpack should be used instead.

~~~
parhamn
That’s not really relevant, right? No mater the loading/transpiration step,
you lose a lot of language guarantees when you can load multiple of the same
packages, including:

\- function/object identity checking I.e “package.func !== package.func” in
many cases places.

\- module level singletons and registry (e.g createContext useContext)

\- single initialization

\- others I can’t think of

That’s not typical and I was surprised by it.

~~~
jessaustin
Loading multiple versions of the same package is just something that you can
do in javascript. As complained about many other times ITT, any big node tool
does this as a matter of course. Over on the browser, react isn't my favorite,
but I'll admit that all that complexity serves some purpose. Maybe it would be
nice if it were just a bit more complex so that global contexts could be
backwards-compatible, but I guess it's not. Still, if you hire developers,
they should be able to handle this sort of thing.

------
johannes1234321
My concern with npm packages isn't the number and small size of the packages
itself, but that most of them are maintained and owned by individuals who do
work with little review by others and who could disappear any time. If an
individual goes away or takes harmful action it takes time till this is
noticed and till a surviving fork emerges.

If more libraries were maintained by groups in a shared/collaborative way,
more of those risks would go away.

------
overgard
Sadly the alternative isn't much better: the old school model of not having
packages often leads to libraries never getting updated at all. In C++ often
adding a library is such a chore once you get through all the linking issues,
as you battle the 4000 conflicting compiler switches, and compiler versions,
that once it works you never want to change it again.

I think maybe the best solution we have so far are robust "batteries included"
standard libraries like python, where even if they're huge theyre at least
curated and maintained by a central group.

~~~
fxtentacle
I feel like Bazel and Cmake have solved this sufficiently well. I regularly
compile larger projects from scratch and usually the process is to call vcvars
(I'm on Windows) and then Bazel and then an hour later I have a folder full of
cached lib and dll files so that linking my actual executable only takes a
second.

I would be so incredibly grateful to have something comparable in JavaScript
that quickly checks static typing for my entire project. But the closest I
have found is Rust/Elm, so not using JavaScript anymore.

------
Dunedan
That page itself is an example what's wrong with using JavaScript on the web:
It's used in places where it's not necessary. In this case the site doesn't
apply a stylesheet with JavaScript disabled.

JavaScript is great when it's used for enabling interactive websites, but
nowadays more and more websites use it to break features, which don't need
JavaScript at all.

------
awinter-py
express (common web framework) seems to have only 51 deps in lockfile

jekyll has 13

I don't disagree that lots of deps = supply chain risk

But there have always been variations between projects & kinds of projects re
how many deps they pull in. Try following someone's ipython data science
tutorial, it's requirements.txt for days

I think lack of a standard lib early on plus hipster functional culture made
the '10s JS leaders like small monofunctional libs.

Focus on transpilation plus lack of tree-shaking in early versions of webpack
may have also created an incentive for small things.

~~~
dgellow
A python equivalent to "express" would be Flask
[https://palletsprojects.com/p/flask/](https://palletsprojects.com/p/flask/)

I see 4 dependencies in the setup.py, and they are all from the same team of
maintainers (i.e the Pallet team):

\- Werkzeug

\- Jinja2

\- itsdangerous

\- click

[https://github.com/pallets/flask/blob/master/setup.py](https://github.com/pallets/flask/blob/master/setup.py)

[https://github.com/pallets/flask/network/dependencies](https://github.com/pallets/flask/network/dependencies)

~~~
wffurr
The Python standard library is a lot more capable than the JS equivalents
(Node or browser API). That eliminates the need for a lot of dependencies.

There's been a TC39 proposal for a JS stdlib for about 2 years now:
[https://github.com/tc39/proposal-javascript-standard-
library](https://github.com/tc39/proposal-javascript-standard-library)

------
pizzapim
So what does NPM do differently than plugin managers from other languages? I
can't name one where the dependency hell is this big.

~~~
leetrout
My personal take having “grown up” with all of this - JS is one of the worst
ecosystems for relying on external packages for everything. It’s more the
culture than the tooling.

One of the things I love about Go is generally people are a little more
forgiving to copying something around a couple times instead of making a lib
for everything.

~~~
erik_seaberg
Copying code is pinning a dependency, committing to never update it, and
hiding the evidence.

~~~
lioeters
That's not always true.

Many a time I've forked a dependency for one reason or another, and kept it up
to date with upstream. Sure, it's a few more steps to update, but allows for
deep customizations without waiting for original authors to support it (if
ever).

Another reason to fork/copy code, is that its development has ceased long ago.
In that case, it totally makes sense to carry on the work, in-house at first.

Regarding the article, sometimes copying code keeps the entire codebase easier
to understand and manage without having an external dependency.

~~~
erik_seaberg
I guess that's not impossible, but it seems to reinvent git submodules without
the tool support I'd want, and I think explicitly depending on our fork of
their library would be clearer and safer.

~~~
lioeters
> explicitly depending on our fork of their library would be clearer and
> safer.

I complete agree. To temper my comment, I've seen plenty of bad examples of
copying dependencies, bundling old un-updated versions, with undocumented
changes. In such case, it's true that copying leads to a worse kind of
dependency.

------
jessaustin
_200 megabytes is not that bad. I’ve seen it rise above 700 MB easily._

A reasonable complaint! Instead of npm, try _p_ npm [0]. It uses file links to
avoid duplication.

[0] [https://pnpm.js.org/](https://pnpm.js.org/)

------
seumars
This wouldn't be much of a problem if developers weren't as reliant on
frameworks as JS developers are — who for some reason love to blame others for
"reinventing the wheel". What do people even use lodash for? Do you really
need a whole library because you can't write a function to sort an array? This
all reminds me of the npm left-pad scandal. To me the problem was not that a
single package broke thousands of projects, but that thousands of developers
depended on a package for such a simple feature.

------
appleflaxen
If you want the 3D view of Gatsby dependencies without waiting for the initial
2D view to populate, you can go here:

[https://npm.anvaka.com/#/view/3d/gatsby](https://npm.anvaka.com/#/view/3d/gatsby)

------
calibas
If you think this is just npm, try running "cargo update" on a older Rust
project.

------
zacksinclair
I can't help but find it ironic that in middle of this oft-repeated JS bitch
fest the author's own company is promoting their NPM package.

~~~
jessaustin
You have to market differently to different people. TFA is intended for devs
who are forced to use javascript even though they prefer python 2. They would
write a different article for skilled js developers.

------
tomaszs
I treat Npm packages like a JavaScript and PHP. There was a need and it was
done. In the short run it is good enough. But in the long run it may become
complicated. For example what about essencial packages that wont be maintained
anymore? What about depending on millions of lines of code from unknown
sources? What about single dependency that is npm? It is a risk too.

------
takinola
So what exactly is the problem? I see a lot of talk about dependency hell and
node_modules bloat but I don't really hear much about what the actual downside
is.

Is it performance (I haven't experienced it)? Is it taking up disk space (I
have terabytes to spare)? Does it add complexity (I would argue it reduces
complexity)?

What exactly is the problem?

------
8bitsrule
People who sin shouldn't complain about winding up in hell.

------
dmitriid
I'm not buying the narrative of "js dependency hell what are they thinking"
anymore.

Try installing a package for Rust. Or Go. Or ... any language, really.

~~~
shakna
Gatsby has 19k dependencies. I don't think I can find something like that for
Rust.

The largest, and not exactly admirable, I can find is reqwest that drags in
97.

Whilst 100 is a huge number... It's an enormous gap from the many thousands.

~~~
mrighele
And that it is not even the worst I have seen. I think the worst offenders are
development tools like webpack.

I have a small project that I work on from time to time which uses 5 libraries
(react, a map library and a chart library), typescript, and react-scripts
(which I guess pulls in webpack and all the rest).

This is what happens when I run a npm audit (I havent' touched it for a couple
of months)

> found 38934 vulnerabilities (38916 low, 18 moderate) in 906346 scanned
> packages

> run `npm audit fix` to fix 38620 of them.

> 314 vulnerabilities require manual review. See the full report for details.

While the real numbers are probably lower because there is a lot of
duplication inside the node_modules folder, I find this ... astounding.

------
bob1029
Sometimes I feel like I am becoming increasingly out-of-touch with reality
with how far this NPM/JS ecosystem nonsense has been taken. Is anyone still
using this crap for de novo business applications that have real-world
consequences? I have no problem if you want to build fantastical, sprawling
node implementations for fun and potentially personal profit, but when your
decisions start to involve other professionals who are simply trying to get
the features implemented so they can go home...

I feel like we need to really think about the consequences of making these
kinds of technical choices, especially in 2020 given all the horror stories
circulating. Someone is eventually going to have to come in after the fact and
scrape all the gore off the walls and redo your webapp in .NET/Rust/Go/etc.
Why not start there and be done with it? You could save so much frustration.
Is it about subjective/artistic preferences? At what point should the business
owner start getting involved in these technical decisions? If the business
owner found out their webapp was trapped in hell because you had a personal
aesthetic attraction to one language over another much more pragmatic
language, how do you think they would respond?

~~~
mythrwy
Full page POST reloads on each data submission aren't going to fly with most
users, even business users. Neither is reloading a page to get new data.

My personal feeling about CRUD is this, you don't have to get fully on the
crazy train and spend all your time with packages and build tools to achieve a
nice product. Yarn is better for me. Very sparing package use. And a little
light Vue with server side templating. No I don't need you managing routes
there JS framework, that never was a good idea in my estimation. (Or maybe it
is and I don't get it but I like incremental changes and simplicity).

Once the dust settles perhaps I'll move on to a more complex setup but I don't
have time for it now. I'm trying to get stuff done. However this doesn't have
to mean eliminating modern behavior or appearance. Which means some
JavaScript. It's going to be required in general by end users.

~~~
stepbeek
Fwiw I've found turbolinks gives me enough of an SPA feel that POST reloads
aren't a huge problem. If we need a very dynamic experience then Vue would-be
my go to tool,but for the other 90% this is fine.

