Hacker News new | past | comments | ask | show | jobs | submit login
Yarn – A new package manager for JavaScript (facebook.com)
1714 points by cpojer on Oct 11, 2016 | hide | past | favorite | 469 comments

This is a huge leap forward for the JavaScript community—probably more than many people will realize right away.

I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once. npm solved this problem elegantly, but still struggles with non-determinism. I had resigned myself to thinking that maybe these were just fundamental tradeoffs in package management that could never be resolved.

Then I had the opportunity to use Cargo, the package manager for Rust. It synthesized what, in my mind, are the best features of npm and Bundler into a single package manager, with terrific speed to boot.

Yarn looks like it will be Cargo for JavaScript. After Cargo improved on many of the great ideas of npm, those ideas are returning to help improve the JavaScript ecosystem. Cross-pollination at its best.

This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.

I'm also happy to see that a significant amount of work has already gone into the governance model. Despite the largest contributors being Facebook employees, it looks like they've really outdone themselves making sure this is a community-run project. A BSD license, no PATENTS file, and an Ember/Rust RFC process[1]; this has all the hallmarks of a community open source project. That's critical for open infrastructure projects like this, and they've nailed it.

[1]: https://github.com/yarnpkg/rfcs

I'm very much looking forward to using Yarn in my own projects because it looks like it solves a lot of real problems I encounter every day. Thanks for all the hard work!

> This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.

And a shrewd move by FB: to not announce their new registry on the same day.

> Yarn pulls packages from registry.yarnpkg.com, which allows them to run experiments with the Yarn client. This is a proxy that pulls packages from the official npm registry, much like npmjs.cf.[0]

Time will tell whether they only want to be proxying NPM or will allow direct pushing to their own registry. If they do, JS ecosystem might see another big shift.

[0] http://blog.npmjs.org/post/151660845210/hello-yarn

That big shift will have to happen first. I don't see them ever making their own registry unless 99.99% of people are using yarn and are having a lot of problems with the current npm registry. While I see a lot of people using yarn, I'm not sure about 99.99% and I think npm's registry itself is pretty good.

So I don't think interests will ever align to create a new registry. Nobody wants to do that. That would have serious consquences for the JS community and would take years to recover, in my opinion.

Why would it be a bad thing to support additional repositories? Personally, I don't like how centralized the JS ecosystem is.

For example, if I refer to 'left-pad', it would default to 'npmjs.org/left-pad'. If the author goes rougue, I think it would be great to enable people to publish and consume 'thirdparty.com/left-pad'

Disclosure: I'm a FB employee with no knowledge of our plans in this regard

1. any left-pad issue has been "eliminated" with the new rules npm (the company) has enforced.

2. you can already have your own version of thirdparty/left-pad by maintaining your own npm repository. i think what your parent post is referring to is facebook saying we're going to make our own public node/javascript package registry and you should publish to our registry.

doing this at the moment does nothing for the community other causing a lot of pain points. ex) now npm authors will have to publish to both registries so developers don't have to dig to find where it was published to, then they also have to hope that someone else didn't register the module name in one of the registries..

there is just too much splintering if facebook decided to become a competing registry rather than just using npm's registry and building on top of it.

> any left-pad issue has been "eliminated" with the new rules npm (the company) has enforced.

Only to resurface again: http://status.npmjs.org/incidents/dw8cr1lwxkcr

And it will, no doubt, resurface again and again and again

Somewhat funny to have an FB employee complaining about overly centralized systems.

I used to find such comments funny until I began working for Big Name Corps myself and I realized how much one's personal philosophy could be inconsistent or sometimes even contradictory to the employer's philosophy. I make money by selling my skills to an employer despite inconsistent philosophies. I think it's like a chef that can cook meat for his/her guests although the chef has decided to refrain from consuming meat himself/herself.

It's sort of related to the principal-agent problem ..

Yup. And such a chef probably wouldn't publicly berate people for eating meat, or expect being called out if they did.

Yup. And I don't see anyone berating anybody in this thread. Do you?

Have a read here: https://github.com/nodejs/NG/issues/29#issuecomment-17431452... - it's a writeup that I made a while ago about the requirements for a stable (decentralized) package registry, and it addresses your question as well.

> So I don't think interests will ever align to create a new registry.

Possibly not soon, at least from a technical perspective, but I could definitely see a PR fiasco (security, bias, etc.) causing a loss of confidence in its stewardship.

And I don't think it'd be that big of a disruption if it were to happen; for the hypothetical case of Facebook+Yarn, they're already proxying to NPM, so they could easily continue to do so while also accepting packages that are Yarn-only.

I more or less agree with this.

Even if they don't get 99% of users (or any major percentage) I think it would still benefit the community to have an alternative to npm. Also Facebook has the advantage of not needing to make a business out of it (they already got a pretty good one) so in theory it could be entirely open source and free.

Just to expand on the stricter versioning rules that I mentioned, some things that in my opinion could improve the reliability of a package repository:

- strictly defined version update time intervals, e.g. you can't update your package more than once a week (or have to take some special actions for critical updates, e.g. contact support)

- "delayed" publishing, e.g. when you submit your package it will only be published in 24 hours, until then you can re-submit updates, etc.

- similar to above, but your package wont be published until it was tagged on github (or elsewhere) for a certain amount of time

- published packages can not be removed, but you can disassociate them with your account by marking them as "not maintained" and possibly assign new maintainers for it

- maybe introduce some way for developers to mark new versions as "backwards incompatible" if they do break backwards compatibility

I think there is definitely a "market" for some stricter node package repo.

I find it strange that the time isn't invested in already existing projects.

But at least it's a move away from NPM. I think the most problems I had with JavaScript develompent in the last 2 years came from NPM.

Yarn isn't a replacement for npm itself. It's a client that can read/write to npm, and other registries such as Bower.

Well, it seems like it could be according to the article. What would you still need to use NPM (the cli) for, other than package hosting?

According to http://blog.npmjs.org/post/151660845210/hello-yarn, it seems it doesn't work with private packages yet, which may or may not be an issue for your project. But it seems this is a complete CLI replacement for NPM.

Yes, it replaces the client, but it uses the same package repository and package format.

And thus could have been implemented as part of the existing client.

"Could have been implemented as part of the existing client" isn't the same as "Should have been implemented as part of the existing client".

I personally don't know much about either tool (don't do a ton of JS), but it's possible that fixing the existing client without either breaking backward compatibility or making it too complicated (multiple modes of operation) was too difficult or not worth it.

Also, I'm having a really hard time understanding the complaint about a new client. The value is in repository of reusable code, not the client. That you can use different clients with the same repository is a feature, not a bug.

Yeah - the npm client codebase would not be fun to do a big refactor on. And you would need to do a big refactor to make the v3 flat install process deterministic and/or performant.

These issues have been raised a few times, along with the shrinkwrap/reproducability stuff, and it didnt seem like it was a big priority for the core team. Understandably I guess they seem more focussed on the enterprise/private repo side of things and just keeping things running on the client side.

npm = node package manager = CLI tool which Yarn replaces.

npmjs.com = package repo which Yarn can use.

at least as far as I can tell

I'm almost sure complain was about npm as a tool, not about repo :)

Would you mind going a bit into detail? I work with npm on a daily basis and I am pretty happy so far.

The whole version range stuff got me many times. I went to use fixed versions on my own package.json files, but the deps of my deps could still be dynamic, which is even worse, since they sit deeper in my dependency graph AND there are more indirect deps than direct deps. (~50 direct, >200 indirect)

Also, npm isn't deterministic and it got even worse with v3. Sometimes you get a flat list of libs, if a lib is used with multiple versions, the first usage will get installed flat, the rest in the directory of the parent lib, etc.

The npm-cli is basically a mess :\

The fun of kicking off a CI build after the weekend with no commits and see stuff randomly break because some dependency of a dependency got updated and broke things in a minor version is something I've only experienced in JS - beautiful.

In fairness to the language and tools, this seems to be more of a cultural problem than anything.

You can do the same kind of version range tricks in typical Java builds, for example (Maven), but most people hardcode the values to keep builds as deterministic as possible.

For some reason, the JS community seems to prefer just trusting that new versions won't break anything. Its either very brave of them really (or maybe just foolish).

> the JS community seems to prefer just trusting that new versions won't break anything. Its either very brave of them really (or maybe just foolish).

Let's not pretend that we aren't all blindly tossing in random libs of dubious quality and origin we find on github into our package.json and hoping for the best anyway. My company talks a mean talk about "best practices", but, my god, the dependencies we use are a true horror show.

I hate non-reproducible builds and semver-relaxed dep-of-the-dep issues, but, while a broken dep fails the build for lots of people (downside), the upside of this is that very quickly (within hours of a new dep being published) there will be lots angry people complaining about it on GitHub, and a faulty dep will be typically quickly rolled back / superseded with a patch. Otherwise, some bugs might be sitting hidden for a long time.

But can yarn fix this?

Say that I use yarn to depend upon some module X, specifying a fixed version number like the good boy scout that I am. Module X resides on npmjs and in turn depends upon the bleeding edge version of module Y. And then one day module Y goes all Galaxy Note and bursts into flames.

Can yarn shield my app from that?

Yes it claims to provide a lockfile while locks all your deps all the way down the dependency tree.

You can do (and are supposed to do) the same with npm's own shrinkwrap, but people claim that it doesn't work as intended.

It's not a cultural problem. People make mistakes. People don't know what a non-breaking change is, especially those not well versed in refactor work.

I don't think Yarn solves any of these problems, tbh. It seems like what we really need is a package manager that tests the api of each package update to make sure nothing in it has broken expectation in accordance with Semver.

It is a cultural problem in that people on other platforms (eg, Maven) don't choose floating dependency versions, generally.

You shouldn't be kicking off CI builds based on unpinned deps (unless you're deliberately doing 'canary testing' etc), because of course that will break. The npm solution for this is to use 'npm shrinkwrap' and you should always have been using this at your site/project level otherwise there was no hope it could work.

It's not that npm devs were naive enough to believe that unpinned deps would be safe for reproducible builds.

However I've heard several people allude over the years that 'npm shrinkwrap' is buggy, and isn't fully reproducible (though never experienced any problems personally). This is the aspect yarn claims to address, along with a supposedly more succinct lockfile format.

With or without npm-shrinkwrap.json? Not chastising, I'm sincerely asking.

Obviously without until we broke and looked in to it :D

JS dev has been a minefield like that, the entire ecosystem reminds me of C++ except lower barrier to entry means a lot of noise from newbies on all channels (eg. even popular packages can be utter crap so you can't use popularity as a meaningful metric)

I agree 100%, but the default upgrade strategy for npm --install does not help matters: it's much saner to wildcard patch versions only and lock major.minor to specific versions.

this obviously doesn't fix anything and I think the points in this discussion stand, but I've never understood why the defaults are not more conservative in this regard.

Well, now you can do that in Cargo as well :-)

What we do currently is we lock everything to an explicit version - even libraries.

At least it's possible to get deterministic builds if you are willing to do a bit of work carefully / manually updating all of your dependencies at once.

You shouldn't have that happen with Cargo, given that we have a lockfile. Even when you specify version ranges, you're locked to a single, specific version.

If you follow the best practices for using Cargo you don't have a Cargo.lock file for libraries.

This means your library tests will not be deterministic.

Using the Cargo.lock file for libraries does solve this, but then every binary package you build that references your library will have to be specifically tied to the versions in the library.

We do this internally because it's the only way to provide deterministic builds over time.

Over time I suspect it will get harder and harder to keep this rigid web of dependencies working.

One thing we might try is to enforce the rule 'no library will contain any tests'. At first glance this kinda makes my skin crawl, but maybe if we could find a way where every project that used a library could somehow share the tests for that library it could actually work.

git submodules might actually be able to provide these tests for each binary package. If only git submodules would stop corrupting our git repos... :-(

Hmmm, I feel like there's some kind of disconnect here; could you open an issue on users or the Cargo repo? Thanks!

Ideally for libraries you would try to test with different combos of transitive deps, but it's a large space.

Anyways you can put a lockfile with your library, and it shouldn't affect downstream.

npm shrinkwrap

NPM is deterministic when there using the same package.json and there is no existing node_modules folder.

And if you want to lock versions for your entire dependency tree, npm shrinkwrap is what you're looking for (It's essentially the same as lockfiles in other development package managers). Though for security reasons I prefer to keep things locked to a major version only (e.g. "^2.0.0"). Shrinkwrapping is useful in this instance too if you need to have predictable builds (and installs as it'll use the shrinkwrap when installing dependencies too if it's a library rather than the main app) but want to ensure your dependencies stay up to date.

It's not perfect by any measure, but there are ways to make it work the way you want.

Mhm that sounds reasonable. I haven't gotten into problems with versions yet but I don't really trust npm update, because I am not always sure how it behaves.

From a build tool perspective (we use npm scripts for basically everything [and some webpack]) I am also not missing something particular.

Looking at other comparable solutions (from other languages) I'd say npm does a pretty good job.

sure, it's better than pip or something.

but it still far from perfect.

the main problem with npm is just when its used internally. It is painful really. Using it when you have internet connection is just seamless

Multiple versions may sound like it's useful, but it's almost always a bad idea. Cargo doesn't allow it either.

The problem isn't really fundamental. Bundler makes almost all the right choices already. Its major disadvantage is that it only works for Ruby.

As a practical matter, the npm ecosystem today relies on duplication, and no new client that made the "highlander rule" (there can be only one) mandatory could succeed.

Yarn does offer a `--flat` option that enforces the highlander rule, and I'm hopeful that the existence of this option will nudge the ecosystem towards an appreciation for fewer semver-major bumps and more ecosystem-wide effort to enable whole apps to work with `--flat` mode.

Plz send halp!

Explain why duplication is mandatory?

Just imagine two packages you depend on (a and b) that both have a shared dependency (x). Both start off depending on x version 1.0 but then later a is updated to 2.0 while b isn't. Now you have two packages depending on different versions of the same package and hence the need for duplication. You have a that needs x@2.0 and b that needs x@1.0, so both copies are kept.

Don't upgrade a when it wants a half-baked x. Choose versions of a and b that agree on a known-good version of x. If there aren't any, it's not sane to use a and b together unless x is written very carefully to accommodate data from past and future versions of itself.

It's not as simple as that. Lodash is a great example of why the highlander rule doesn't work within the npm ecosystem: older versions are depended on by many widely-used packages which are now "complete" or abandoned. Refusing to use any packages which depend on the latest version of Lodash is just not practical.

That's not how it works. There will be two copies of x in the require cache. They don't know of each other's existence.

I'm arguing for choosing dependency versions that don't require you to break the highlander rule. "a is updated to 2.0" doesn't mean you should start using that version of a right now.

Right but what I'm saying is that two versions of the same library will live quite happily together, because node.js absolutely abhors global scope. The two versions have their own module-level scope to store functions and data.

So go crazy and install every version of lodash! Nothing will break.

That will work for lodash, because it's just a bag of functions. But anything that creates objects that are passed outside of a single module could have problems.

There was actually a high profile bug when someone ended up with two versions of react included in the same page: https://github.com/facebook/react/issues/1939.

That seems to be a slightly different situation? As the issue poster described, a single page had invoked two copies of React, both of which assumed ownership of a particular "global" variable. (Not really global, but a particular member of a global.)

I'm not a React expert, but I don't see why that situation would only affect multiple React copies with different versions?

I'm not a large JS developer but with my other experiences managing dependencies, it doesn't always allow this as a possibility. A bug in your system is tracked down to the library b using x@1.0 but the fix is to upgrade to b with x@2.0 however a using x@1.0 doesn't have an upgrade path. Waiting for another company or organization to release with an update is not an option. Our projects have several cases of requiring different versions of the same library -- we try to avoid this using the same logic you suggest but it's not a possibility in all cases so we have to work with it.

It's placing your own release cycle at the whims of your dependencies' release cycles. In the corporate world that would not be a viable solution.

It's almost certain that a's version of x and b's version of x have distinct types whose names collide, and I don't know of any Typescript or Flow notation to prevent passing incompatible instances between them (e.g., a wants to call a method that didn't exist in b's version of x), so if anything works it's only by luck.

Edit: I haven't dug into this, but it might be possible to use Typescript namespaces to distinguish a and b's versions of x. https://www.typescriptlang.org/docs/handbook/declaration-fil...

Node's module system doesn't import dependencies into a global namespace. So the `x` required by `a` and the `x` required by `b` will have separate namespaces and won't conflict, or even know of each others' existence. There are pros and cons to this approach, but it definitely does work. Flow, at least, (and presumably Typescript) understands the module resolution logic and handles this very common situation without issue.

It's not possible to import two different versions of a module in the same module.

If a depends on x v1 and exports this dependency then your application also imports x v1. If b depends on x v2 and exports this dependency too that means your application is transitively importing both x v1 and x v2 which is not possible.

If a or b only use instances of x internally and do not accept instances of x as parameter or return an instance of x they don't have to export their dependency on x.

If either a or b do not export their dependency on x then there is no problem. Your application will only depend only on x v1 or x v2 directly.

Would it be possible to create hardlinks or symlinks to a particular package/version pair shared as a dependency between other packages? I know this only works on unix-like OSes but otherwise it could revert to the old behaviour of duplicating the dependency.

I think they're just saying that any new client that tried to not support duplication at all would likely quickly run into a large amount of npm packages/package combinations that just don't work. So within the context of using the npm registry duplication is mandatory.

Cargo definitely allows two dependencies to rely on different versions of a transitive dependency. If those deps expose a type from that dependency in their API, you can very occasionally get weird type errors because the "same" type comes from two different libraries. But otherwise it Just Works(tm).

Cargo does allow multiple versions of transitive dependencies. It tries to unify them as much as possible, but if it can't, it will bring in two different versions.

What it _won't_ let you do is pass a v1.0 type to something that requires a v2.0 type. This will result in a compilation error.

There's been some discussion around increasing Cargo's capacity in this regard (being able to say that a dependency is purely internal vs not), we'll see.

With tiny modular utilities this is very much necessity - and not a bad idea if the different versions are being used in different contexts.

For instance, when using gemified javascript libraries with bundler it is painful to upgrade to a different version of a javascript library for the user facing site while letting the backend admin interface continue using an older one for compatibility with other dependencies.

You've got to take the ecosystem into account. There are a lot of very depended-upon modules that are 1 KB of code and bump their major version as a morning habit. Forcing people to reconcile all 8 ways they indirectly depend on the module would drive them nuts, but including 4 copies of the module doesn't hurt much.

Yarn supports `yarn [install/add] --flat` for resolving dependencies to a single version

> Multiple versions [...] almost always a bad idea

If so, different major versions of the same dep should be considered different libraries, for the sake of flattening. Consider lodash for example.

That's exactly what Cargo does, but it also takes a further step of making `^` the default operator and strongly discouraging version ranges other than "semver compatible" post-1.0.

Looks like Yarn does something similar: https://yarnpkg.com/en/docs/cli/add#toc-yarn-add-exact

Personally I'm not really sure I like it. If I specify an exact revision of something, chances are I really do mean to install that exact revision. I don't see why I need an extra flag for that.

That can cause some serious problems in at least some portion of times. I've dealt with the subtle errors that have been caused by this problem in c++, and don't really know javascript libraries that well so I can't give a more concrete example. But imagine that there are the following libraries:

* LA: handles linear algebra and defines a matrix object.

* A: reads in a csv file and generates a matrix object using LA

* B: takes in a matrix object from LA, and does some operations on it

In this case, if B depends on version 5 of LA and the new version of A depends on version 6 of LA, then there's going to be a problem passing an object that A generated from version 6 and passing it to B which depends on version 5.

The problem does happen in JavaScript. But since its unityped, there is a strategy to deal with it

* Figure out early on (before 1.0) what your base interface will be.

For example, for a promise library, that would be `then` as specified by Promises/A+

* Check if the argument is an instance of the exact same version.

This works well enough if you use `instanceof`, since classes defined in a different copy of the module will have their own class value - a different unique object.

  * If instanceof returns true, use the fast path code (no conversion)
  * Otherwise, perform a conversion (e.g. "thenable" assimilation) that
    only relies on the base interface
Its not easy, but its not always necessary either. Most JS libraries don't need to interoperate with objects from previous versions of themselves.

Would this even work in what I describe? For instance, if mat.normalize() was added in LA-6, and B provides an LA-5 mat, and then A (which has been updated to use the new method) calls mat.normalize() on the LA-5 mat expecting an LA-6 mat but because of duck-typing that method doesn't exist.

It would not.

However, since A exposes outside methods that take a matrix as an argument, it should not assume anything beyond the core interface and should use LA-6's cast() to convert the matrix.

The problem is partially alleviated when using TypeScript. In that case the inferred type for A demands a structure containing the `normalize` method, which TypeScript will report as incompatible with the passed LA-5 matrix (at compile time). That makes it clearer that `cast` would need to be used.

Define the matrix class in a seperate library for compatibility purposes if it's so widely used and doesn't change. Maybe some people don't need both the linear algebra and instead only the definition of the matrix object?

Another solution is to provide version overrides and make B depend on version 6.

However if there are differences in the matrix class between different versions of the library then you're forced to write a compatibility layer in any case.

If an API expects the outside world to hand it an instance of a specific library, all bets are off. Maybe it gets `null` or the `window` object, who knows? But a library can at least declare what dependencies it wants. If you take that away, it ratchets up the uncertainty factor that much more.

This is especially true when you're going to be serving your code over the web. It's very easy when using npm to accidentally end up shipping a lot of duplicate code.

That alone has me super excited about Yarn, before even getting into the performance, security, and 'no invisible dependency' wins.

I feel like this is especially problematic when using NPM for your frontend. Now you have to run all your code through deduplication and slow down your build times or end up with huge assets. I wonder if it's really worth the trouble.

> I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once

This is due to the way that Ruby includes code, where it's available globally, vs Node where code is scoped to the including module. I'm not sure how Ruby could support multiple versions with changes to the language and/or RubyGems

Yes, that's why I said it was a Ruby limitation, not a Bundler limitation.

Right, my comment wasn't neccesarily directed at you, but to others who might not be familiar with both

Why do you think that everybody is inspired from javascript tech? It is probably the other way around. For example there were very good build tools long before npm which are still used today and unmeasurably better (like Maven or Gradle).

funny (and great!) how cargo became a gold standard for package management without any advertising. well done, rustaceans.

This may come off as a troll, but it's an honest question. I'm not a javascript guy. It's not a language I deal with at all. Why on God's green earth does it need as much tooling as it seems to have? Are people really making projects with dozens (hundreds? more?) of dependent libraries? Are there aspects of the language or runtime that reward multiple layers of configuration management? In short, what the hell is up with this ecosystem?

Basically I think it comes down to:

1. In the browser we have less control over the environment the code runs in, requiring tooling to achieve a stable, normalised environment which we can improve at our own pace, rather than the rate at which browsers are updated.

2. JS has become very widely used very quickly, which means that there have been lots of hastily built things, which have been replaced by better things (and for every 'better thing' there are many alternatives which fell by the wayside), and a larger pool of people to build them.

3. It's easier than ever to build new tools (or any kind of app, for that matter), because the npm ecosystem makes it trivial to combine existing smaller pieces of code into new combinations.

Your number 3 is the one that really drives it home for me. It's the unix philosophy taken to the extreme, and (at least in my experience) it's working very well.

It does take some getting used to, but when you stop looking for one big tool that can solve 5 semi-related problems, and start looking for 5 little tools that each solve one of the problems, things get much more clear.

And yes, I know that this kind of reliance on dependencies can cause issues, but from my experience, those issues don't seem to bite that often, and many of them can be resolved with even more tooling.

browser incompatibility is the biggest issue (if not the only issue). this is almost a non-issue if you're just writing node related programs. but if you need to support in the browser, now you have edge, chrome, firefox, and safari at minimum. then some have to continue supporting ie10/ie11, or even worse, ie7+.

don't forget about mobile devices, with ios having such a difficult-to-write-anything-new version of mobile safari.

i'm writing a new app for myself, so i don't have to worry about any backwards compatibility issues, except it needs to work on my iphone, which uses mobile safari webviews, which means i have to either use shims or restrict myself back to es5, non-html5 features (or work around their html5 limitations).

example: you cannot cache audio files to play using <audio> tags in mobile safari. safari just refetches it from your server (also has to recreate the audio object, which causes a lag in the audio being played). technically you _can_ cache if you convert all your audio files to base64 and use the webaudio api. i'm probably going to have to go this route because i'd like the app to be offline usable.

so rather than spend time on my app, i now have to write out a build step for converting audio to base64. then test that it works fine in ios and desktop browsers (at least chrome, firefox). it all just keeps adding up.

I came to the Javascript world late, starting with React. I think its tooling has gotten out of hand, but there are a couple of reasons it's not a _total_ disaster in my mind:

* Partly, there are a larger number of Javascript developers that haven't been exposed to other ecosystems, and end up reinventing some wheels

* More importantly though, I think the pace of innovation in the javascript ecosystem is actually good in a way, because other more mature languages (I'm not talking about age of the language, just maturity of the tooling and ecosystem) have gone through the same growing pains, it just took a lot longer. It's easy to forget, coming from the Python world for instance, the pains of easy_install and distutils vs seuptools, etc etc. I still find Java dependency management a bit of a pain, because I don't know it very well.

We're just watching a sped up version of a language maturing, and it's painful as individual developers trying to keep up, but I don't think it's as negative as HN often makes it.

> Are people really making projects with dozens (hundreds? more?) of dependent libraries?


> Are there aspects of the language or runtime that reward multiple layers of configuration management?

A significant fraction of the node community likes to split their project into really, really small packages and treat them as if they're independent even if they come from the same source repo and tend to get used together. As an example, the popular collection processing library Lodash publishes a package for each method and people actually do depend on individual methods.

There are two major rationales for this. The first is that JS is a dynamic language and in dynamic languages, smaller chunks of code are easier to reason about than larger ones. The second is that the JS community as a whole cares more about artifact size than pretty much any other community. Having smaller packages allows you to produce a smaller payload without having to rely on techniques like tree shaking.

I find micro packages maddening but people give me their code for free and it mostly works so I don't complain about it that much.

What do you find maddening about micro packages? I hate depending on a giant stack of frameworks (slowing installs, builds, adding behavior surface area) when I just need a function or two, so micro packages are a joy for me.

Reading the article it seems to me that this micro packages approach is what slows down everything. I never, ever have seen all these problems in even not-so-state-of-the-art dependency managers like maven or nuget. Seriously up to now it was impossible to have a build server isolated from the internet if you didn't want to check-in all the dependencies??? Simply crazy. I really can't understand how people can even think to use a dependency manager system that doesn't satisfy the essential requirement of having your CI server sandboxed.

> this micro packages approach is what slows down everything

Not really. The npm client is just flaky, very slow, buggy, has some major design flaws in its v2 incarnation, and has a completely different set of major design flaws in its v3 incarnation.

The core architecture works fine for small packages.

> Seriously up to now it was impossible to have a build server isolated from the internet if you didn't want to check-in all the dependencies

Of course not, although I admit the linked article implied it was if you didn't read closely. There's a wide number of solutions, including running a private registry, or a copy of the public registry, or a local cacheing proxy of the public repository, or a custom npm client that hits a local cache, etc. Some of the solutions work quite well, and in fact yarn itself is just a re-implementation of some existing solutions.

> I really can't understand how people can even think to use a dependency manager system that doesn't satisfy the essential requirement of having your CI server sandboxed.

As the article noted, Facebook was sandboxing their CI server and using NPM; they just didn't like the existing solutions. (Nor should they; they were a bit naff.) But that doesn't mean (nor is anyone claiming) that there were no existing solutions. Yarn looks great, but it's an extremely incremental change.

Javascript is very easy to work with if you just stick to micro packages. What slows it down is that the language has been forked to an incompatible spec (ES6) so there is essentially a hard fork in pool of libraries. To deal with this—and calm anxieties around lack of inheritance, trauma over having to type function over and over, etc—layers of pre-processing have been added: JSX, SASS, asset compilation, compatibility layers, framework booting. This pipeline distances developers from the DOM and other low-level protocols, increases the surface area of the build process, and frequently disables debugging and introspection tools. In addition megaframeworks like Angular, Ember, and React add mandatory layers of indirection and runtime dependencies between each snippet of code, introducing behavioral complexity and confusion from non-deterministic builds.

All of this is why people are excited about Yarn, but to me it's a band-aid on several architectural mistakes.

Just say no to frameworks. Just say no to ES6 and transcompilation. Just say no to asset pipelines. Viva la JavaScript. Viva la Netscape. Viva la Node.

Where would a CI server retrieve dependencies from, if not either over the network, or from within the repository? Do you keep a module / library cache on the CI server itself? In other words, what do supply as input to your CI process, besides a Git URL?

In a local nexus/nuget repository in the intranet, for sure I'm not allowing internet access directly from a CI server. And apart from the obvious and scary security concerns it is also much faster.

Got it. You can also set up an NPM repository like that, but I don't think that that is commonly done by small dev teams. This does lead to hilarity like the left-pad clusterfork.

> Reading the article it seems to me that this micro packages approach is what slows down everything.

It really isn't. The NPM client is just really, really poorly written, with no parallelization to speak of.

But a huge list of micro packages slows installs, too. It shouldn't, but NPM doesn't seem very good at parallelising operations.

I have no idea if you're familiar with the java ecosystem, but I am, so I'll use that as an example:

Node libraries are typically several orders of magnitude smaller and more single purpose than java libraries. Java libraries like Guava and Spring are typically quite large. A practical example is from unit testing: In Java, you might use JUnit+Mockito for your test dependencies. In Node, to get similar functionality, I have the following deps: Mocha, Sinon, Sinon-As-Promised, Chai, Chai-Moment, Chai, Chai-Things, Chai-Shallow-Deep-Equal, Chai-Datetime, Sinon-Express-Mock, Supertest.

Many Javascript applications both client and server functionality, and there is different tooling for both. Webpack is currently popular for building stuff that will run in the browser, whereas server side stuff is different.

It's a much newer technology than java and there is quite a lot of churn around tooling in general as the nature of the way applications are built change.

> It's a much newer technology than java

Strangely enough both Java and JavaScript were first released in May 1995. Server side JavaScript was released the same year just after the browser version.

Though I get that you are referring to the js eco systems (commonjs, npm) compared to the much more mature Java ecosystem.

Why is that? In the Java ecosystem there is Spock framework that gives you everything required for testing, you don't even need junit+mockito. In which scenario would be better to use 11 different libraries for doing the same thing as a well written framework?

In this case, I would argue it is not better. Many JS devs have moved onto simpler, greener pastures. https://github.com/substack/tape is what I prefer. Small little library. Its API has everything I need and nothing I don't. Those other test libraries listed are pretty much overkill.

Similar with http://mithril.js.org/ over React, Angular and Ember. A single drop-in library with 13 functions in it's API. The library itself is a single, understandable source code file. Mithril is pretty much the exact opposite of Angular and Ember.

A lot of front end devs are eschewing frameworks completely. With modern DOM and web APIs, you can do a lot with "just" vanillajs.

But, this is exactly what I'm saying. you start with a small, focused library that contains everything you need, which is usually fairly minimal, so that library is small. but then, along the way, you find yourself wanting, so you add a library that adds support for that specific thing.

I didn't add sinon-as-promised for fun, I added it for in-memory sequelize integration tests. same for all the other libraries I mentioned.

My point is that by keeping libraries small, you will end up with either more dependencies, or a lot more hand-rolled code. Neither of these things are inherently bad or good, and I'm not defending them. What I am trying to do is explain to the commenter why the node ecosystem seems to have such an emphasis on package management and the tooling around that.

From the readme I can't really see how to write extensive unit tests with that thing.. If I need to test something I want only one tool that works perfectly and does all the Stubbing, Mocking, verification, data driven test and behaviour driven test. I still haven't found in any language/ecosystem anything comparable to Spock in completeness, readability and ease of use. Honestly I can't really see how to do BDD with tape.

its not that hard. That being said, the current all in one turnkey testing framework for javascript is Jest, and that will pretty much do it all in one tool.

The benefit of what's happening in JS is that a lot of libraries are reasonably compartmentalized for a specific purpose.

As an example: you _could_ have one SpockJS framework that gives you React + Redux + ReactRedux, but instead each of these libraries are separate so you can use them without each other. This allows you to, say, move from React to Angular without changing the state of your application.

With everything moving so quickly in JS, this separation turns out to be valuable enough to tolerate a lot of packages and dependencies. It also makes it easier for communities to exist for solutions to specific problems, which at least in theory results in better solutions.

It's not necessarily better, just a different approach.

For testing it doesn't make any sense. I want a coherent and expressive framework that does everything. If I'm doing BDD I know that I will need everything from the start. And from the little that I have seen about jest..well it seems really a jest. I can't really think to do any serious BDD with that.

Even for testing I think it make sense. If you want to switch from Mocha to Jest and still use Chai for your assertions, you can.

Like I said, it's not better just different.

I guess you could call it "diversification" of dependencies. If one dependency stops being maintained or is not the best approach you can jump to another one.

A very limited standard library with no native (well, just approved but not ratified) support for importing/libraries. Coupled with inconsistent implementations and you end up need more layers of abstraction.

Let's consider Microsoft Office. It is a massive software and such as massive software requires a lot of (internal) dependencies and toolings.

Now imagine Office 365, the same software with the same functionalities, except running in the browsers. The task is to essentially write Microsoft Office (front-end logic) entirely in JavaScript (or TypeScript), with the additional problems of dealing with browser compatibilities issues, which are just as hard as OS compatibilities issues.

Then, imagine all the software going to the "cloud". JavaScript now has to have the capability of re-writing all these software originally written in C++, C#, Java... all sorts of languages. It is quickly becoming a superset of all ecosystems where everyone is trying to migrate their software to. So it is not surprising that people are inventing more and more tooling to make it more powerful to suit their needs.

You say tooling, but then talk about libraries. I think both are just as numerous for nearly all popular programming languages. It’s just that JavaScript is currently “hot shit”, so all this stuff is a lot more visible.

There are tons of libraries for C, many of which aim to do the same, only faster/smaller/scalable/whatever. There are many compilers, both open and closed source. And there are documentation tools, syntax checkers, unit testing frameworks and whatnot.

Of course there are also languages like C# where there’s a de-facto standard for almost everything: IDE, compiler, base library, GUI framework(s).

I think choice is great!

You say tooling, but then talk about libraries.

Because, from this outsider's perspective, it seems like most of the tooling is library management (rather than, say, profiling or linting -- does javascript even have a linter? -- or debugging).

JSLint, JSHint, ESLint. Each of which built on the good aspects of its predecessor and evolved as new requirements came up. There are a _ton_ of tooling libraries for JavaScript. Just google "X JavaScript" for any tooling you'd expect, and you'll usually have several options. Of course, on the flip side, people will complain that there are too many options. Blah blah blah. No matter what the JS community does, people will criticize.

Then you haven’t seen anything yet. x)

There are tons of tools, of course also including jslint, debuggers (e.g. IntelliJ, dunno about others). There are editors and even IDEs. Then there’s the problem of bringing (NPM) modules to the browser using tools like webpack or SystemJS or whatever. There are general purpose build/automation systems like gulp. There are transpilers that “compile” ES6 to ES5. For any given task, you can probably find at least five programs that do it.

And if you look at all the topics, you’ll find that library management makes up only a small fraction of everything.

> does javascript even have a linter?

Yes, thousands of them, npm alone has 1200 results for linter:


> does javascript even have a linter? -- or debugging

Yes, as libraries. There are multiple linter options that can be installed with a project. Debugging is available, usually through an IDE or browser.

JS has several linters, although ESLint is the one to pick if you're looking for one.

This is a valid question coming from an outsider looking in.

There are various packages that are extremely small (see leftpad and associated controversy). They often wind up in a slightly larger package, which itself winds up in a slightly larger package, which recurs until you finally wind up with one actual package of consequence that you're targeting. For example, Express (a very common Node.js abstraction) has 26 direct dependencies, yet 41 total dependencies.

A lot of this results from early Node.js' mantra of having small, focused packages. This could potentially be a good thing because instead of having three major dependencies that have their own way of leftpadding they can all rely on one library and thus code is efficiently reused. This can be bad, however, when they each need their own version of the dependency - or worse - the dependency is removed from the registry (see Leftpad and associated controversy).

One of the legitimate problems that I've seen is that there are various libraries that are just thin wrappers over functionality that is native to Node.js - but written to be "easier to grok". Thus, these thin abstractions become practical code duplication out of some mixture of Node's developers lack of effective verbosity/documentation and application developer laziness. But then they creep higher up the dependency chain because someone at a lower level used it.

On one hand it can be quite simple (and rewarding) to write code while minimizing dependencies. On the other hand, abstractions such as Express make it very easy to write code and it feels like you only have "one dependency" until you look under the hood.

41 dependencies isn't even that much.

Babel, the most popular transpiler, is 335 total dependencies, and it doesn't even do anything out-of-the-box. You need to add babel-preset-latest for 385 total dependencies if you want to actually transpile anything.

Want a linter? ESLint is 128 total dependencies.

A lot of these answers are pointing out the x-browser complexity which is true but that isn't what drives JS package mgmt. It's mostly Node stuff. The mature common JS libs for browser work are pretty robust when it comes to x-browser stuff now days.

For browser work, you can do almost everything you need with some jQuery, handlebars and moment if there are dates. A few other small libs here and there and you've got almost everything covered. Some edge cases may drive you to use some library that is heavily dependent on 32 different sub-libs but it's really not that often.

Server-side Node.js is the issue, not browser compatibility.

That's interesting because I have less JS headaches when using Node. I avoid front end JS these days

>Are people really making projects with dozens (hundreds? more?) of dependent libraries?

Yes, because the standard library provides practically nothing useful in itself.

Handling dependencies is hard. Handling dependencies for a platform you don't control is harder. Handling dependencies for a platform you don't control, where every single kilobytes count, is insane.

Add those together, and you get the need for real complex tooling.

With that said, tooling is nowhere as complex as it is for languages like C++ or Java. People are just used to it on those platforms, and the tooling is more mature/understood because it already went through the countless iterations, 15-20 years ago or more.

The tooling for building C++ consists of make or something that generates make files. There's also no "1 function" libraries. Adding a dependency is done by (compiling it if needed) and referencing it from the make file. It's super easy to understand, but involves manual work because C++ doesn't have a package manager.

Java isn't too complicated either. I've used Gradle and Maven but doing things manually like in C++ can also work.

Crucially, they don't reinvent the wheel every couple of months so a C++ or Java dev can focus on building things instead of relearning how to build things.

Javascript in my opinion is still very young. You can't compile it to web assembly (yet) and the fact that there are different browsers that interpret it in so many different ways make for libraries to exist to mitigate this. Also none of these tools are 100% necessary, they just make your life easier and tools always have room for improvement. I'm sure others can add to this. I don't think any of this is a bad thing, it's just overwhelming for newcomers that haven't been following this whole thing.

Others have touched on the subject but one of the biggest issues is browser compatibility. Building a responsive website that works on all major browsers, across multiple versions, as well as all the mobile quirks is not an easy task in the slightest ... and that's just to make a site look pretty. Then we have to start talking about building massive web apps that are front-end heavy which requires even more tooling to scale, maintain properly.

Because JS is open source and it's not controlled by an Oracle / Microsoft / Apple / XXX foundation. Everybody is free to do stuff - and does. In the end there aren't actually as many tools as people think, a lot of hobbyists projects who die quite quickly

Compared to C/C++ tooling and package managers, npm is like taking a bazooka to a knife fight.

Is npm the bazooka or C++? As someone who's worked with C/C++ (and its hilarious explosion of tooling -- half-a-dozen at Google alone), maven, bundler and Cargo, I think it's just not accurate to say that the npm ecosystem has a particularly high level of complexity.

npm is the bazooka. Covers a lot more area with a single tool, but sometimes dangerous if you use it wrong.

C/C++ tooling is a plastic butterknife you get with airplane food.

Some languages actually have standard libraries so don't need bazookas..

What're you smoking? Can I have some?

Browsers are the new OSes. Everything on them runs on javascript. You do the math.

JavaScript is the new BASIC and every millennial wants to populate his/her Github portfolio for Silicon Valley interviews.

You should check out the mayhem caused by a single contributor removing his left-pad library.


I wrote a post explaining why I'm psyched to be working on it:


- open, community governance that will support long-term evolution

- the technical details get a lot right out of the gate (decent performance, predictability, and security)

Yikes, thanks!

Interested to see the emphasis on security, it's definitely something that could use more focus in this area. On that note, are there any plans to implement TUF (https://theupdateframework.github.io/) or similar into Yarn?

Have you guys approached the ridiculous folder nesting situation? E.g. breaking out of the current/broken node_modules structure?

There has been no problems with folder nesting in `npm` in general since the version@3 came out over a year ago.

This isn't true though.

npm chooses the first version of a package it encounters to install at the top level. Every other version is installed nested. If you have N uses of version A and one use of version B, but version B is installed first, then you get N copies of the package at version A.

It's improved but it's not fixed. if a depends on c 1.0 and b depend on c 2.0, both versions of c need to be installed and one of them will be nested.

...which is actually good. With Java and maven, you cannot use multiple versions of a library.

What's the solution for that case though (one which requires no nesting)? I don't think a solution with no nesting exists given the current module resolution algorithm of Node.js, which allows for only a single version of a particular package to exist at a given level.

nesting can't be prevented in that case (at least with the current node require() design). It gets really bad if you have 10 packages that depend on c@1.0.0 and 10 packages that depend on c@2.0.0 -- one of them will install in the root directory and the other 10 will be duplicated. ied stores a single copy of each package@version and uses symlinks which is an improvement. Apparently yarn tried something similar but it breaks some packages (that check where they're located for example).


I could be prevented. Another project does this by using a hash of the project to all all dependencies at one level then symlinks the dependencies from hash to name so it references the correct version.

> no problems

Absolute and utter hogwash.

I don't even need to get into versioning to make this argument but I'll exclude it for clarity's sake. Simple scenario:

    root -> A
    root -> B
    root -> C
    A -> B
    B -> C
What folder structure is created?

Symlink that all you want, how is it in any way whatsoever thoughtful and elegant? It's not a real solution by any stretch of the imagination. Myriads of pre-existing systems that correctly deal with this scenario existed at the time that NPM was concoted. Ignored. Considering that node is touted as a cross-platform solution, how well does version@3 deal with filesystems that don't support symlinks? I've run into this: not at all.

NPM version@3 could not be further from the ethos of "engineering." It's an outright hack, and it quickly betrays its limitations. It could have been far simpler, far more considerate of pre-existing wisdom.

what you described is npm@2's behaviour. npm@3 will flatten dependencies as much as possible

How does it flatten dependencies? Symlinks?

no, it just installs everything it can into the top level node_modules. npm@2 would do this to a degree as well, it was just less sophisticated about it.

> into the top level node_modules.

Just tried using npm3 explicitly and I stand corrected.

(other than its non-determinism)

And Yarn it deterministic! :party_parrot:

fwiw npm really needs an `exec` command like bundler has (ie `npm exec bin-cmd`).

"$(npm bin)/bin-cmd" is getting really old

You can use `npm run env bin-cmd` for that. It's not as convenient as `npm exec bin-cmd` would be, but it might be easier to type than `$(npm bin)/bin-cmd`.

What is the problem with nested folders?

When do you expect ember to transition to yarn?

Are there any branches of ember-cli implementing yarn?

You say "decent performance" and "predictability". What is the basis of this claim? I've heard these all before, but unless you've actually shipped a product using this tool I don't know how you can back this up.

As for performance, Yarn is about 3-5x faster installing all of the dependencies and devDependencies of react-native. The benchmarks are here: https://yarnpkg.com/en/compare. It's much faster in most scenarios, especially the ones that used to take minutes.

Only if those dependencies are in the cache, otherwise it takes the same time.

The benchmarks test both the hot cache and cold cache scenarios for npm and Yarn. Except in the scenario when node_modules is already populated and there is no yarn.lock file, which is uncommon if you're using Yarn, Yarn is faster than npm in all of the benchmarked scenarios where there is a cold cache.

  > unless you've actually shipped a product using this tool 
Given that wycats _worked on_ this tool, I would assume it's his experience helping build it that's the basis of the claim.

(I don't work at Tilde so I can't tell you if it's been shipping with yarn, but it would shock me if it wasn't.)

We're not using Yarn at Tilde yet, but I've been using it on Yarn itself and when working on Ember packages, and it's worked well. As far as performance, we have good benchmarks tracking yarn's performance in a whole bunch of different scenarios (https://yarnpkg.com/en/compare)

Ok, didn't realize he worked on it. Bundler is great so that gives me some hope for Yarn. But in general I think its more stable in the long-term for projects to use open standards instead of vendor solutions for browser package management.

I can understand this sentiment in general, but don't think it applies here. npm is also a private company, and this does work with their registry. There currently isn't a vendor-neutral standard to follow.

There's a lot to like here - deterministic builds are great. However, yarn doesn't currently seem to support installing straight from github or private packages [0], [1].

I'm sure this will be added in the future, but it is currently a dealbreaker for me - I'm using a single package that's just straight hosted in a private git repo, as the package itself isn't ready to be published yet. I'm sure other people are using private packages for more normal reasons (closed source, pre-production, etc).

If yarn actually made it simpler to refer to a local package during development, I'd be on board with that. (I am developing the dependency, but want to also work on the thing that depends on it, so I'd rather just save locally and refresh npm on the outer project, but that's hard to get right - file urls don't always update, the easiest workflow seems to be delete the package from node_modules, and npm install, with the package having a git url rather than a version in package.json).

0: https://github.com/yarnpkg/yarn/issues/573

1: https://github.com/yarnpkg/yarn/issues/521

You are right, we aren't 100% compatible right now. Please track the issues on the Yarn GitHub repo that you linked or better yet: help us fix it. This is a community project and we are excited for the community to jump in and help us out and complete the few missing pieces.

I agree local development with many packages can be hard. For Jest we adopted `lerna`

See https://github.com/facebook/jest and https://github.com/lerna/lerna which makes cross-package development in a mono-repo a lot of fun. Maybe that solution will work for you?

This. I came here to say this also. In ruby if you want to patch a repo for yourself you just fork it and make changes and then in bundler point to the git url (often on github). Then run bundle install and you're set.

With npm I've been frustrated a few times whereby you can fork the repo and point package.json to the git repo but after running npm I usually get a bunch of compilation errors.

It would be wonderful to make this as simple as with bundler within the ruby ecosystem, point to forked url and running yarn handles the rest.

I'm very excited about this. Private package support is a must-have for it to get any traction in my company. We are more than ready to leave shrinkwrap behind.

Yeah, this is like the main reason I love npm: ability to use a Github repo as a package repo.

It is quite a congenial response and no disrespect to npm inc, but yarn will be technically far superior (if it's not already), and I can't help but interpret this as the beginning of the end for npm....It was only a matter of time though since there are a number of systemic problems with npm and (in my opinion) it is at the center of the dysfunctional world of modern JavaScript tooling. It did move the ecosystem forward significantly for a number of years, but now something better is required. Sorry to be blunt, just my opinion.

NPM can evolve; Gradle and SBT also didn't turn out to be the end of Maven, which evolved to being more of a repository and dependency management tool with multiple clients.

This would make a lot of sense. I would /never/ make a build using maven again, but having maven central around is a huge plus for the jvm ecosystem and I'm happy it survives. (plus lots of people continue to make new builds with maven and legacy maven builds will be around basically forever)

I'm glad they're okay with it but then again what else would they post? npm has some major, major issues with it and it has barely moved in years. This Yarn looks like it solves quite a few issues with npm and it took another company to build it.

That's insane. It wouldn't surprise me if yarn becomes popular and its proxy to npm slowly turns off and boom, everyone would be migrated and npm would be left with little to no users. Yeah that's probably unlikely for the immediate future but Yarn is positioned perfectly to siphon off npm's users very painlessly.

Does anyone know why npm moves so slowly?

they have millions of installs and can't break them?

Well, but remember that time they shipped an SSL cert that broke all updated clients? I'm not sure they move slowly because they don't want to break things, because that would have been a pretty easy bug to catch...


That's not really a good counter example. It's more of a mistake on a release...

It's more of a failure to double check that npm runs when you change the SSL cert you bundled with it.

I think it's a great example of moving a little too quickly, honestly. Especially since it bricked plenty of npm installs instead of being "just another bug".

Do they necessarily have to break them to move faster? It's a package installer. Yarn is handling locking / shrinkwrapping different but I would argue shrinkwrap needs to be abandoned and re-done anyway so I'm not seeing the risk. It took them, what, two years just to flatten dependencies and move to npm 3? That was actively breaking Windows usage of node!

I love the service they offer. Not a fan of the tools or pace. Npm comes in the node installer so why do they even have to care about improving?

Just my thoughts.

What do you mean by moving slowly? IMO there's not much to add. I'd like to see some speed improvements, but otherwise I'm not sure npm should evolve.

technical debt and not wanting to break people's workflows

Because they focused on hiring SJW profiles that write codes of conducts and readme's all day long instead of actually software engineers that could improve the tool.

Please resist the temptation to glom ideological snark onto unrelated threads. That's not what this site is for and it ruins threads for everyone by being simultaneously distracting and boring.

Looks like they are trying to write off yarn as just another alternative install tool. I think this is short cited and hope they are taking this threat more seriously internally.

That's pretty well written, though you can see slight hints of bitterness in the tone (though its subtle). Or maybe I'm a little biased, because I've seen some of the non-public reactions, and they were more than a little bitter...

Interesting how they point out how Yarn still depends on them since it pulls packages from NPM's package repo. Kind of like a reminder of who the dominant player still is.

I'm really liking yarn. It solves some of my main issues with npm, it's fast and it provides some great output in the CLI. It's also architected absolutely ingeniously. Seriously they have positioned themselves to seamlessly take over npm's entire business just like that. What do I mean?

So today by default yarn goes through its servers which then proxy to npm's. As time goes Facebook can add more features to its own repo (custom submissions instead of to npm or maybe it does a federated publish to both at first, nice looking pages with community engagement tools built in, better metrics (npm provides me with almost worthless metrics), etc). Then when enough people are using Yarn Facebook could simply...flip the switch. Then boom, zero npm usage.

I would be terrified if I were NPM right now. They've sat on their hands for years not improving npm. I mean npm3 had a silly CLI animation that made all downloads take orders of magnitude longer simply because of the animation!

P.S. if you're trying to use yarn with a custom npm repository just drop a .yarnrc file into your project's directory then add the following in that file:

registry "http://<my register>"

npm should not be terrified, we've been working with them and they've been very encouraging.

From their perspective it's very difficult to make breaking changes to the client because of the sheer number of people depending on them. (We've faced similar problems on Babel)

Yarn still makes use of the npm registry and all the high quality infrastructure they've built and support they provide for the ecosystem. They are a critical piece of this.

I mean I understand the intent but it turns npm into basically a dumb pipe / just infrastructure. Every business that gets reduced to that struggles to get out of that space and expand.

Granted it's far too early to write npm off. But with how slow they've moved over the years I'm unconvinced that yarn won't take over the space unless it runs into some bad problems. Npm 3 and its launch was an absolute mess and, ultimately, it's almost just a package managing tool. I am unconvinced that breaking changes is much of an issue if at all for them. They could abandon shrinkwrap into a better system and I think everyone would be happy for it; no need to keep propping up that awful system.

As Yehuda pointed out, such a thing won't happen because Yarn was launched as a community project from the beginning, and a lot of members contribute to both, so even if Facebook decided something sketchy, the community would either stop it from happening or continue with a forked version.

> the community would either stop it from happening or continue with a forked version

Would they, though? If Yarn turns out to be the better tool that doesn't change the way they work (beyond solving some pain points) then would they even care? Facebook could even offload the hosting to some open source group like Apache.

It's purely anecdotal but a lot of people I have worked with would love to see an alternative to npm. It's just slow and the features it gives to authors is pretty limited (I want better metrics, please!). I'm not even sure I would call it sketchy, it would just simply be phasing out something deprecated (I'm not talking about a swift take over more like a slowly-boiling-the-frog-in-the-water takeover).

It's only my opinion and if Facebook really doesn't want that to happen then I guess it won't but it's pretty easy to imagine it, IMO.

NPM makes money from the registry and if Yarn keeps using it though, this benefits them a lot. Let's see what happens.

on .yarnrc - anybody figured out how to get it to pull from a private git repo?

This is not a problem with the package manager. This is a problem with complexity.

When did it start becoming reasonable for a front-end only part of the MVCC pattern to have 68 dependencies?

Or for a transpiler like Babel to add 100k+ files? I'm sorry I just find it ridiculous that instead of taking a look at the disease (unbounded complexity), we are looking to engineer our way out of the problem by creating a package/dependency manager that "scales". Are the frontend problems at Facebook of showing HTML forms and buttons on a page really that complicated to warrant such a behemoth of a system?

This harkons back to the days at LinkedIn where I spent my time in a misguided effort to create a distributed build system because our codebase had grown so massive that it literally took a distributed system to build it and still release within the day. I feel bad for people working on these systems because while it is fun and "technically interesting" to solve "problems at scale", you're really just digging the ditch deeper for your company. You are putting lipstick on the 500 lbs pig; making room in the dump to pour in more technical debt.

I don't think anyone ever imagined we'd be in a situation where a javascript framework is approaching the complexity of an operating system in terms of individual source files and SLOC.

Just boggles my mind.

The Javascript ecosystem has to deal with unrelenting, multi-decade backwards compatibility requirements; supporting numerous, inconsistent implementations; and an incredibly wide set of stakeholders and participants with diverse motivations when advancing the spec.

Do any other software platforms out there face the same magnitude of complications?

To speak to one of your primary examples: Babel is freaking huge and somewhat absurd, but is there any other way to advance the Javascript language? Fortunately, this is only a development dependency, and the client doesn't have to see it.

On a happier note, these are good problems to have. This is another indicator of how successful the web platform has become. You can build some absolutely incredible projects these days, and they are instantly available to billions of people who are operating thousands of different hardware devices (with different specs and dimensions) running dozens of operating systems. No gatekeeper, bureaucrat, or corporate curator must approve your work. Just simply deploy it, and anyone with the URL can access it. It is a mind-blowing human accomplishment. I look forward to seeing how the web continues to progress over the coming decades.

Have you seen some of the things we are building for the web these days? Full applications, with multiple "pages", server-side rendering above the fold content, minified builds that load not only content--but code as well--on demand from the server, etc.

Absolutely: this is overkill for your blog or portfolio website, but part of the rising complexity with the tooling and dependency system is due to the rising complexity of the applications themselves.

What is your proposed cure for the disease?

I use JS+Node+NPM for my day job and many side projects.

Initial thoughts:

- Why didn't Facebook contribute the updates to NPM directly?

- They are coupling a package manager and registry proxy; the latter has many existing implementations already

- The differenced between Yarn and NPM+Shrinkwrap do not seem substantive; NPM made the design decision to use a sometimes non-deterministic install algo in NPM3 to speed up install times - when network is involved, there is a serious trade off between idempotency and speed

In general, FB seems to love building their own versions of existing tools:

- Flow, introduced 3 months after TypeScript

- Nuclide instead of Atom/Sublime/VSCode

- Jest instead of Jasmine/Mocha

- DraftJS instead of (insert of of the existing 100 text editors here)

- ...

I get that these are useful internally at FB, but I don't think they help the community much. It would be better to work with existing tools and contribute back to them than to reinvent the wheel every time something doesn't work perfectly for their use case.

I get that FB uses these as recruiting tools, it's useful for them to have rights for these projects, and it gives their engineers something to do and be excited about, but I do not want a whole dev tool ecosystem controlled by big FB.

Also, I find IED's approach to speeding up builds far more novel and interesting - https://github.com/alexanderGugel/ied

> Flow, introduced 3 months after TypeScript

Do you think Flow was built in 3 months as a reaction to Typescript? Not that it was a big enough problem that two different groups of people independently decided to try to solve it? EDIT: disregard this, Flow was introduced much longer than 3 months after TypeScript. The blog post introducing Flow mentions TypeScript, and is probably worth reading for background [1]

> Nuclide instead of Atom/Sublime/VSCode

Nuclide is an Atom plugin.

> Jest instead of Jasmine/Mocha

Jest provides a superset of Jasmine functionality, and uses Jasmine as its test runner.

[1] https://code.facebook.com/posts/1505962329687926/flow-a-new-...

TypeScript was released October 1, 2012. Flow was released November 18, 2014

Did Flow take 2 years to get implemented? I didn't think the initial release looked like a project that had 2 years of dev time on it.

Whoops, my mistake. I don't know enough about the history of Flow to comment further.

> Do you think Flow was built in 3 months as a reaction to Typescript?

They probably knew that TS was in progress when they started, but I don't have a window into that. I did ask some Flow devs a few weeks ago if they are interested in merging with TS, and they answered with a categorical no.

I can't speak for either project, but personally I would say that I don't think static typing in JS is a solved problem, and at this stage to me it seems a monoculture would be much less fruitful than a competition of ideas.

I would agree if they were at all different, but they seem to be syntactically and functionally identical.

As a caveat, I am a daily TS user, and have only played around with Flow.

I am not a computer scientist, but I have heard people repeatedly argue that Flow sets out to create a "sound type system" which TS intentionally does not.

What this means in practice I don't fully understand but it seems to result in enough architectural differences that it apparently makes sense to have both despite the superficial similarities in syntax and behaviour.

I think the issue is that TS is non-modular and dictates javascripts reach. It behaves like no other tool and blocks others from working. Take something like React/Redux for instance, which are built around modern day javascript, but if you transpile any example on the official Redux page right now you will see TS bail out because it can't understand some of the syntax.

Flow on the other hand solves one problem and solves it well. It behaves as any other tool would.

Not sure I understand. Can you give a concrete example?

You can only do as much as TS allows you to, unless you route through babel in a second pass. TS is a tool that controls to which extend you can use language features that are drafted. They do pick drafts here and there, but others they omitt, object spread for instance which is very common for Redux.

As i said, try to transpile this in TS: http://redux.js.org/docs/basics/ExampleTodoList.html

This is why they made Flow.

Still not sure I understand. All the code in that link is valid TS - TS is just a superset of JS.

Object spread is not part of the ES7 spec. It is a proposal which has not made it to stage 3 yet, and might still be dropped. That said, there is a TS issue tracking adding support for it: https://github.com/Microsoft/TypeScript/issues/2103.

Hey, I work on Yarn and Jest. Let me respond to why we built Jest: It solves a ton of problems no other test runner solves and it works well at Facebook's scale. We open sourced it more than two years ago but only recently started to promote the project in open source more actively because it is working so well at FB.

If you are curious to find out more about why we built it, please read our blog posts, beginning with one about testing performance and our unique constraints at FB: http://facebook.github.io/jest/blog/2016/03/11/javascript-un...

Finally, test runners tend to get stale. Any large organization will run into trouble with any kind of tooling as companies scale up engineers or codebases. Testing is no different and we aimed to solve those problems for Facebook; getting to open source our solutions is an added bonus and we are excited to work with the JS community to build the best tooling we can possibly build.

I for one am thankful for Jest and Yarn. I recently switched over to Jest and I love most everything about it and the built in watch with caching reduces friction immensely (now if we can just allow custom preprocessors to return promises that would make life easier for integration of things like rollup). And yesterday I flipped over to using yarn and it definitely solves most of my pain points with npm.

So yeah, I'm definitely on the "keep reinventing" side of this debate.

IMO it's the "javascript way" to invent new tools instead of trying to improve others (when it makes sense to do so).

Let me explain that a little bit... The javascript "ecosystem" takes the unix philosophy to the extreme in many ways. And one of the points of the unix philosophy is to try to avoid "bloating" tools with tons of options, and instead trying to create new tools where appropriate.

Many of those are perfect examples of where "improving" a current tool with the changes they wanted would end up with a more complicated and difficult to use tool.

- NPM makes tradeoffs that the majority of developers in the JS world want right now. Like it or not, deterministic installs isn't something that most people want or need (at least, they don't know that they might want or need it). Making a new tool that anyone can use when that is needed is a great solution, and it avoids bloating NPM with a hundred little flags and tons of code to cover edge cases. It's the perfect example because anyone can switch to using yarn when it's needed without any extra work.

- Nuclide is actually just a layer on top of Atom. They wanted a bunch of things to all work one way, so rather than try to get Atom to change to include what they wanted, or require developers install a ton of extensions, they packaged them up on top of Atom as one system.

- Jest had different goals at the start. Testing react components was ugly and difficult, and Jest wanted to solve that first (IIRC, i'm not very familiar with jest unfortunately). Again, it's something that can be used if needed, and can even be combined with other testing tools if you want (we almost did that at my work, used Jest to test the front-end components, and our current mocha+friends setup to test the "business logic").

> The javascript "ecosystem" takes the unix philosophy to the extreme in many ways.

Bullshit, nix systems rely on stable global dependencies, not 50 versions of the same library installed locally. I don't buy the "unix philosophy" excuse. The nodejs community just doesn't care about API stability which is why there is that inflated number of modules with 90% never maintained more than a year.

Come on now, I said that it's like the unix philosophy in many ways, not that it's unix...

- Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".

Which is what this thread is about.

- Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.

JS has wonderful stream support, and it's used everywhere in many tools. This is probably where JS is the weakest out of these.

- Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.

I mean, if JS isn't this taken to the extreme, i don't know what is...

- Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them.

This also describes JS to a T. Tons and tons of tools, each often only usable by a small number of people in niche situations.

Now i'm not saying that JS is as well designed as unix, i'm not saying that it's exactly the same as unix, i'm not saying that Unix doesn't do things better, or that JS doesn't have it's warts. I'm just saying that it's the "unix philosophy" taken to the extreme. It might not be good, it might not last, but it's working out pretty well for many people right now.

> 50 versions of the same library installed locally

Are you seriously telling me you've never had a linker (static or dynamic) choose the wrong library?

+1 for contribution to npm. Especially given npm isn't completely deterministic there is no reason for not pushing for these changes into npm itself.

Given npms years of crap I'm ready to give it a shot anyway. This might be one of FB's projects that actually gets some traction, especially given its developed in collaboration with other big names.

When you're game changer, it's hard to make current projects maintainers nod their head for you!!! I think io.js was a good example.

> Yarn, a collaboration with Exponent, Google, and Tilde.

They should mention this at the very beginning. Multiple big players investing in this package manager means that we should maybe inspect a little bit more before chanting xkcd.com/927.

927 was also my first thought. But what made me to reconsider was not more than one big name behind it (but it helped), but the fact that they rely on npm backend and did not reinvent everything from scratch.

Basically, yarn is npm client done right reusing the same npm package repo.

npm on it's own is that bad - this is nothing less than water in the desert.

That deprives me of the fun of downvoting people mindlessly misusing xkcd.com/927.

Exponent has been testing out Yarn for about a month, and it's been an incredibly pleasant experience, both in working with the Yarn team and using the tool.

Exponent uses a monolithic code repository (rather than many small code repos) to manage our many interdependent projects. Our repo is actually setup very similarly to Facebook's mono-repo, though obviously considerably smaller in size. We were encountering _many_ of the same problems as Facebook with our NPM setup -- long CI/CD build times, indeterminate node_modules directories that caused our projects and tools to work for some people on our team but not for others, and the inability to do "offline-only" npm installs in CI.

We actually talked with our friends at Facebook about these problems, and tried many of the same approaches they did -- shrinkwrap everything, checking in node_modules, uploading the node_modules folder to separate place (we used another completely separate Git repo), etc. All these approaches either didn't work well or were difficult to maintain, especially on a small team.

Yarn has fixed all these issues for us.

One not-yet-super-publicized feature of Yarn (though I'm told there is a blog post coming) that has been super useful at Exponent is the ability to build a distributable offline cache of dependencies. We have a "node_modules-tarballs" folder in our mono-repo that contains tarballs of all the dependencies for every single project that we maintain at Exponent. Yarn will pull from this cache first before fetching from the NPM registry. This offers HYPER-predictability...it's a level of confidence over and above the "yarn.lock" file, and let's me sleep well at night knowing that my team can get up and running quickly no matter where they're working, how reliable their internet connection is, etc. No more worrying about "did I forget to update the shrinkwrap" -- things just work.

As JS developers, we all, beginners and experts alike, can experience this "JavaScript fatigue" that we always hear about. Fighting with tools is never fun. We just want to write code!

As we've started to use Yarn at Exponent, both on our engineer's machines as well as in our continuous integration and deployment environments, we've at least had one less tool to fight with. Yarn works, works fast, uses the great parts of the NPM ecosystem that we know and love, and most importantly, is _predictable_.

I've been using the Shrinkpack tool ( https://github.com/JamieMason/shrinkpack ) to do that with NPM, and am trying to figure out how to do the same thing with Yarn (per my request at https://twitter.com/acemarke/status/785886788493553664 ).

Definitely looking forward to that blog post from the Yarn team, but I'd definitely be interested in your own comments on how to use that "checked-in tarballs" approach.

> [...] the ability to build a distributable offline cache of dependencies. We have a "node_modules-tarballs" folder in our mono-repo that contains tarballs of all the dependencies for every single project that we maintain at Exponent.

That's exactly what we need! How do you tell Yarn to build and use this offline cache?

As someone who just started with Spark lately, that was my first thought after being confused for a minute. It's not like people at facebook and google don't know about Apache projects. I wonder what is their motive to not avoid such name collision? I see these clashes a lot with small projects on github -ironically most of the are javascript- but rarely on a big scale like this.

generate name compare affinity in concept space if affinity > .5 repeat

yarn is framework for doing X in web programming, and this a framework for doing Y in web programming. not sure if dependency management and cluster system management are far apart enough.

They are different problem domains, I think confusing them is unlikely

Tell that to my google searches

I'm sure that will fix itself very soon.

I'm sure it will.

Search "go" on google, and the first link is the programming language. Search on any other search engine, and as one might expect, the definition or the game of go is ahead of the language.

Wait till I release my new programming language, 'the'. It's the go killer.

the haskell eliminator

wow already one of the features i'm loving in yarn is that it tells you which package is firing warnings about package incompatability.

warning electron-prebuilt-compile > electron-compilers > jade@1.11.0: Jade has been renamed to pug, please install the latest version of pug instead of jade

in npm, that would have just said the part after "jade@1.11.0" which was really vague and didn't really make you want to "fix" it because which npm module do you have to go into? who knows because npm (the package manager) didn't tell you.

i seriously want npm to stop warning me of outdated deps 3 levels down my hierarchy. so what if some shitty old unmaintained lib we started relying on a few years back use lodash v3.x? i'm not going to worry about it as long as it works.

Just experienced the same. Loved it! npm was giving vague errors about dependency's dependency. Now it's so much clearer that which package needs to be updated. Also the speed of installation has reduced a lot !

It looks like this addresses the biggest issues people have with npm's CLI, and it's coming from such huge names: Facebook, Google, and Tilde. Reproducible builds are a _huge_ issue, and this gives you that. Looks great!

One interesting little tidbit I found from diving into the source:


It's not mentioned in the post, but looks like they're running their own registry as well...

Oh! And open governance: https://github.com/yarnpkg/rfcs !

Oh hi Steve! https://registry.yarnpkg.com is just a proxy to the normal npm registry provided by Cloudflare so we can add additional caching and work on network performance (lots of stuff we could layer on top). It should hopefully help those in China as well which would be awesome. But everything you do still lives on the normal npm registry.

Makes total sense, thanks for elaborating! I don't do very much node dev, but when I do, it'll be yarn from now on.

hi! they are running a mirror of the npm registry which contains all the package data but none of the permissions/user mgmt. this means that they are still dependent on npm infrastructure.

npm, Inc strongly encourages people to build and use mirrors. for example, cloudflare (where the mirror lives, and where seb used to work) has been running https://npmjs.cf/ for a long time.

At least speaking for myself, the npm package ecosystem is critical and I wouldn't want to do anything to undermine the integrity of the npm registry.

It's not the first one addressing these problems.

Nix and ied (which borrowed from Nix) got these problems pretty much solved.

I don't understand what spoke against these approaches?

I mean okay, nix has its own language, which is probably a turnoff for JS devs, but ied?

ied didn't work on windows because of the linking strategy. many people who need to use npm use windows.

That's not 100 % accurate. Arguably symlinks are kinda weird under Windows, but that doesn't mean symlinking node_modules doesn't work (you just need admin privileges).

(Disclaimer: I'm the author of ied.)

Huh, interesting! I thought this was going to just be a client. I wonder if it resolved to npm's servers or something.

A curl -I shows CloudFlare, at least. npm's does not. So it's at _least_ doing some sort of cache...

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact