
Yarn – A new package manager for JavaScript - cpojer
https://code.facebook.com/posts/1840075619545360
======
tomdale
This is a huge leap forward for the JavaScript community—probably more than
many people will realize right away.

I loved Bundler's deterministic builds but chafed against the Ruby limitation
of only having a single version of a dependency at once. npm solved this
problem elegantly, but still struggles with non-determinism. I had resigned
myself to thinking that maybe these were just fundamental tradeoffs in package
management that could never be resolved.

Then I had the opportunity to use Cargo, the package manager for Rust. It
synthesized what, in my mind, are the best features of npm and Bundler into a
single package manager, with terrific speed to boot.

Yarn looks like it will be Cargo for JavaScript. After Cargo improved on many
of the great ideas of npm, those ideas are returning to help improve the
JavaScript ecosystem. Cross-pollination at its best.

This also highlights a shrewd move on the part of npm: a clear decoupling
between the npm registry and client, with a well-defined protocol between the
two. The strength of Node is in the staggering size of its ecosystem; how
those bits end up on disk is an implementation detail. This smart separation
allows for this kind of experimentation on the client side, without causing
the ecosystem fragmentation that happens when a new package manager requires a
new registry.

I'm also happy to see that a significant amount of work has already gone into
the governance model. Despite the largest contributors being Facebook
employees, it looks like they've really outdone themselves making sure this is
a community-run project. A BSD license, no PATENTS file, and an Ember/Rust RFC
process[1]; this has all the hallmarks of a community open source project.
That's critical for open infrastructure projects like this, and they've nailed
it.

[1]: [https://github.com/yarnpkg/rfcs](https://github.com/yarnpkg/rfcs)

I'm very much looking forward to using Yarn in my own projects because it
looks like it solves a lot of real problems I encounter every day. Thanks for
all the hard work!

~~~
lucian1900
Multiple versions may sound like it's useful, but it's almost always a bad
idea. Cargo doesn't allow it either.

The problem isn't really fundamental. Bundler makes almost all the right
choices already. Its major disadvantage is that it only works for Ruby.

~~~
_greim_
> Multiple versions [...] almost always a bad idea

If so, different major versions of the same dep should be considered different
libraries, for the sake of flattening. Consider lodash for example.

~~~
fryguy
That can cause some serious problems in at least some portion of times. I've
dealt with the subtle errors that have been caused by this problem in c++, and
don't really know javascript libraries that well so I can't give a more
concrete example. But imagine that there are the following libraries:

* LA: handles linear algebra and defines a matrix object.

* A: reads in a csv file and generates a matrix object using LA

* B: takes in a matrix object from LA, and does some operations on it

In this case, if B depends on version 5 of LA and the new version of A depends
on version 6 of LA, then there's going to be a problem passing an object that
A generated from version 6 and passing it to B which depends on version 5.

~~~
spion
The problem does happen in JavaScript. But since its unityped, there is a
strategy to deal with it

* Figure out early on (before 1.0) what your base interface will be.

For example, for a promise library, that would be `then` as specified by
Promises/A+

* Check if the argument is an instance of the exact same version.

This works well enough if you use `instanceof`, since classes defined in a
different copy of the module will have their own class value - a different
unique object.

    
    
      * If instanceof returns true, use the fast path code (no conversion)
      * Otherwise, perform a conversion (e.g. "thenable" assimilation) that
        only relies on the base interface
    

Its not easy, but its not always necessary either. Most JS libraries don't
need to interoperate with objects from previous versions of themselves.

~~~
fryguy
Would this even work in what I describe? For instance, if mat.normalize() was
added in LA-6, and B provides an LA-5 mat, and then A (which has been updated
to use the new method) calls mat.normalize() on the LA-5 mat expecting an LA-6
mat but because of duck-typing that method doesn't exist.

~~~
spion
It would not.

However, since A exposes outside methods that take a matrix as an argument, it
should not assume anything beyond the core interface and should use LA-6's
cast() to convert the matrix.

The problem is partially alleviated when using TypeScript. In that case the
inferred type for A demands a structure containing the `normalize` method,
which TypeScript will report as incompatible with the passed LA-5 matrix (at
compile time). That makes it clearer that `cast` would need to be used.

------
bandrami
This may come off as a troll, but it's an honest question. I'm not a
javascript guy. It's not a language I deal with at all. Why on God's green
earth does it need as much tooling as it seems to have? Are people really
making projects with dozens (hundreds? more?) of dependent libraries? Are
there aspects of the language or runtime that reward multiple layers of
configuration management? In short, what the hell is up with this ecosystem?

~~~
mambodog
Basically I think it comes down to:

1\. In the browser we have less control over the environment the code runs in,
requiring tooling to achieve a stable, normalised environment which we can
improve at our own pace, rather than the rate at which browsers are updated.

2\. JS has become very widely used very quickly, which means that there have
been lots of hastily built things, which have been replaced by better things
(and for every 'better thing' there are many alternatives which fell by the
wayside), and a larger pool of people to build them.

3\. It's easier than ever to build new tools (or any kind of app, for that
matter), because the npm ecosystem makes it trivial to combine existing
smaller pieces of code into new combinations.

~~~
Klathmon
Your number 3 is the one that really drives it home for me. It's the unix
philosophy taken to the extreme, and (at least in my experience) it's working
very well.

It does take some getting used to, but when you stop looking for one big tool
that can solve 5 semi-related problems, and start looking for 5 little tools
that each solve one of the problems, things get much more clear.

And yes, I know that this kind of reliance on dependencies can cause issues,
but from my experience, those issues don't seem to bite that often, and many
of them can be resolved with even more tooling.

------
wycats
I wrote a post explaining why I'm psyched to be working on it:

TLDR:

\- open, community governance that will support long-term evolution

\- the technical details get a lot right out of the gate (decent performance,
predictability, and security)

~~~
zamalek
Have you guys approached the ridiculous folder nesting situation? E.g.
breaking out of the current/broken node_modules structure?

~~~
petetnt
There has been no problems with folder nesting in `npm` in general since the
version@3 came out over a year ago.

~~~
ksherlock
It's improved but it's not fixed. if a depends on c 1.0 and b depend on c 2.0,
both versions of c need to be installed and one of them will be nested.

~~~
Kenan
What's the solution for that case though (one which requires no nesting)? I
don't think a solution with no nesting exists given the current module
resolution algorithm of Node.js, which allows for only a single version of a
particular package to exist at a given level.

~~~
ksherlock
nesting can't be prevented in that case (at least with the current node
require() design). It gets really bad if you have 10 packages that depend on
c@1.0.0 and 10 packages that depend on c@2.0.0 -- one of them will install in
the root directory and the other 10 will be duplicated. ied stores a single
copy of each package@version and uses symlinks which is an improvement.
Apparently yarn tried something similar but it breaks some packages (that
check where they're located for example).

[https://github.com/alexanderGugel/ied](https://github.com/alexanderGugel/ied)

~~~
Osiris
I could be prevented. Another project does this by using a hash of the project
to all all dependencies at one level then symlinks the dependencies from hash
to name so it references the correct version.

------
pfooti
There's a lot to like here - deterministic builds are great. However, yarn
doesn't currently seem to support installing straight from github or private
packages [0], [1].

I'm sure this will be added in the future, but it is currently a dealbreaker
for me - I'm using a single package that's just straight hosted in a private
git repo, as the package itself isn't ready to be published yet. I'm sure
other people are using private packages for more normal reasons (closed
source, pre-production, etc).

If yarn actually made it simpler to refer to a local package during
development, I'd be on board with that. (I am developing the dependency, but
want to also work on the thing that depends on it, so I'd rather just save
locally and refresh npm on the outer project, but that's hard to get right -
file urls don't always update, the easiest workflow seems to be delete the
package from node_modules, and npm install, with the package having a git url
rather than a version in package.json).

0:
[https://github.com/yarnpkg/yarn/issues/573](https://github.com/yarnpkg/yarn/issues/573)

1:
[https://github.com/yarnpkg/yarn/issues/521](https://github.com/yarnpkg/yarn/issues/521)

~~~
cpojer
You are right, we aren't 100% compatible right now. Please track the issues on
the Yarn GitHub repo that you linked or better yet: help us fix it. This is a
community project and we are excited for the community to jump in and help us
out and complete the few missing pieces.

I agree local development with many packages can be hard. For Jest we adopted
`lerna`

See [https://github.com/facebook/jest](https://github.com/facebook/jest) and
[https://github.com/lerna/lerna](https://github.com/lerna/lerna) which makes
cross-package development in a mono-repo a lot of fun. Maybe that solution
will work for you?

------
jrs235
Here is npm's response: [http://blog.npmjs.org/post/151660845210/hello-
yarn](http://blog.npmjs.org/post/151660845210/hello-yarn)

~~~
BinaryIdiot
I'm glad they're okay with it but then again what else would they post? npm
has some major, major issues with it and it has barely moved in _years_. This
Yarn looks like it solves _quite a few_ issues with npm and it took _another
company_ to build it.

That's insane. It wouldn't surprise me if yarn becomes popular and its proxy
to npm slowly turns off and boom, everyone would be migrated and npm would be
left with little to no users. Yeah that's probably unlikely for the immediate
future but Yarn is positioned _perfectly_ to siphon off npm's users very
painlessly.

~~~
sotojuan
Does anyone know why npm moves so slowly?

~~~
softawre
they have millions of installs and can't break them?

~~~
striking
Well, but remember that time they shipped an SSL cert that broke all updated
clients? I'm not sure they move slowly because they don't want to break
things, because that would have been a pretty easy bug to catch...

[http://blog.npmjs.org/post/78165272245/more-help-with-
selfsi...](http://blog.npmjs.org/post/78165272245/more-help-with-
selfsignedcertinchain-and-npm)

~~~
ssunstruck
That's not really a good counter example. It's more of a mistake on a
release...

~~~
striking
It's more of a failure to double check that npm runs when you change the SSL
cert you bundled with it.

I think it's a great example of moving a little too quickly, honestly.
Especially since it bricked plenty of npm installs instead of being "just
another bug".

------
BinaryIdiot
I'm really liking yarn. It solves some of my main issues with npm, it's fast
and it provides some great output in the CLI. It's also architected absolutely
_ingeniously_. Seriously they have positioned themselves to _seamlessly_ take
over npm's entire business just like that. What do I mean?

So today by default yarn goes through its servers which then proxy to npm's.
As time goes Facebook can add more features to its own repo (custom
submissions instead of to npm or maybe it does a federated publish to both at
first, nice looking pages with community engagement tools built in, better
metrics (npm provides me with almost worthless metrics), etc). Then when
enough people are using Yarn Facebook could simply...flip the switch. Then
boom, zero npm usage.

I would be terrified if I were NPM right now. They've sat on their hands _for
years_ not improving npm. I mean npm3 had a silly CLI animation that made all
downloads take _orders of magnitude longer_ simply because of the animation!

P.S. if you're trying to use yarn with a custom npm repository just drop a
.yarnrc file into your project's directory then add the following in that
file:

registry "[http://<my](http://<my) register>"

~~~
thejameskyle
npm should not be terrified, we've been working with them and they've been
very encouraging.

From their perspective it's very difficult to make breaking changes to the
client because of the sheer number of people depending on them. (We've faced
similar problems on Babel)

Yarn still makes use of the npm registry and all the high quality
infrastructure they've built and support they provide for the ecosystem. They
are a critical piece of this.

~~~
BinaryIdiot
I mean I understand the intent but it turns npm into basically a dumb pipe /
just infrastructure. Every business that gets reduced to that struggles to get
_out_ of that space and expand.

Granted it's far too early to write npm off. But with how slow they've moved
over the years I'm unconvinced that yarn won't take over the space unless it
runs into some bad problems. Npm 3 and its launch was an absolute mess and,
ultimately, it's almost just a package managing tool. I am unconvinced that
breaking changes is much of an issue if at all for them. They could abandon
shrinkwrap into a better system and I think everyone would be happy for it; no
need to keep propping up that awful system.

------
iamleppert
This is not a problem with the package manager. This is a problem with
complexity.

When did it start becoming reasonable for a front-end only part of the MVCC
pattern to have 68 dependencies?

Or for a transpiler like Babel to add 100k+ files? I'm sorry I just find it
ridiculous that instead of taking a look at the disease (unbounded
complexity), we are looking to engineer our way out of the problem by creating
a package/dependency manager that "scales". Are the frontend problems at
Facebook of showing HTML forms and buttons on a page really that complicated
to warrant such a behemoth of a system?

This harkons back to the days at LinkedIn where I spent my time in a misguided
effort to create a distributed build system because our codebase had grown so
massive that it literally took a distributed system to build it and still
release within the day. I feel bad for people working on these systems because
while it is fun and "technically interesting" to solve "problems at scale",
you're really just digging the ditch deeper for your company. You are putting
lipstick on the 500 lbs pig; making room in the dump to pour in more technical
debt.

I don't think anyone ever imagined we'd be in a situation where a javascript
framework is approaching the complexity of an operating system in terms of
individual source files and SLOC.

Just boggles my mind.

~~~
dwaltrip
The Javascript ecosystem has to deal with unrelenting, multi-decade backwards
compatibility requirements; supporting numerous, inconsistent implementations;
and an incredibly wide set of stakeholders and participants with diverse
motivations when advancing the spec.

Do any other software platforms out there face the same magnitude of
complications?

To speak to one of your primary examples: Babel is freaking huge and somewhat
absurd, but is there any other way to advance the Javascript language?
Fortunately, this is only a development dependency, and the client doesn't
have to see it.

On a happier note, these are good problems to have. This is another indicator
of how successful the web platform has become. You can build some absolutely
incredible projects these days, and they are instantly available to billions
of people who are operating thousands of different hardware devices (with
different specs and dimensions) running dozens of operating systems. No
gatekeeper, bureaucrat, or corporate curator must approve your work. Just
simply deploy it, and anyone with the URL can access it. It is a mind-blowing
human accomplishment. I look forward to seeing how the web continues to
progress over the coming decades.

------
bcherny
I use JS+Node+NPM for my day job and many side projects.

Initial thoughts:

\- Why didn't Facebook contribute the updates to NPM directly?

\- They are coupling a package manager and registry proxy; the latter has many
existing implementations already

\- The differenced between Yarn and NPM+Shrinkwrap do not seem substantive;
NPM made the design decision to use a sometimes non-deterministic install algo
in NPM3 to speed up install times - when network is involved, there is a
serious trade off between idempotency and speed

In general, FB seems to love building their own versions of existing tools:

\- Flow, introduced 3 months after TypeScript

\- Nuclide instead of Atom/Sublime/VSCode

\- Jest instead of Jasmine/Mocha

\- DraftJS instead of (insert of of the existing 100 text editors here)

\- ...

I get that these are useful internally at FB, but I don't think they help the
community much. It would be better to work with existing tools and contribute
back to them than to reinvent the wheel every time something doesn't work
perfectly for their use case.

I get that FB uses these as recruiting tools, it's useful for them to have
rights for these projects, and it gives their engineers something to do and be
excited about, but I do not want a whole dev tool ecosystem controlled by big
FB.

Also, I find IED's approach to speeding up builds far more novel and
interesting -
[https://github.com/alexanderGugel/ied](https://github.com/alexanderGugel/ied)

~~~
mambodog
> Flow, introduced 3 months after TypeScript

Do you think Flow was built in 3 months as a reaction to Typescript? Not that
it was a big enough problem that two different groups of people independently
decided to try to solve it? _EDIT: disregard this, Flow was introduced much
longer than 3 months after TypeScript. The blog post introducing Flow mentions
TypeScript, and is probably worth reading for background [1]_

> Nuclide instead of Atom/Sublime/VSCode

Nuclide is an Atom plugin.

> Jest instead of Jasmine/Mocha

Jest provides a superset of Jasmine functionality, and uses Jasmine as its
test runner.

[1] [https://code.facebook.com/posts/1505962329687926/flow-a-
new-...](https://code.facebook.com/posts/1505962329687926/flow-a-new-static-
type-checker-for-javascript/)

~~~
RyanCavanaugh
TypeScript was released October 1, 2012. Flow was released November 18, 2014

Did Flow take 2 years to get implemented? I didn't think the initial release
looked like a project that had 2 years of dev time on it.

~~~
mambodog
Whoops, my mistake. I don't know enough about the history of Flow to comment
further.

------
egeozcan
> Yarn, a collaboration with Exponent, Google, and Tilde.

They should mention this at the very beginning. Multiple big players investing
in this package manager means that we should maybe inspect a little bit more
before chanting xkcd.com/927.

~~~
selectnull
927 was also my first thought. But what made me to reconsider was not more
than one big name behind it (but it helped), but the fact that they rely on
npm backend and did not reinvent everything from scratch.

Basically, yarn is npm client done right reusing the same npm package repo.

------
skevy
Exponent has been testing out Yarn for about a month, and it's been an
incredibly pleasant experience, both in working with the Yarn team and using
the tool.

Exponent uses a monolithic code repository (rather than many small code repos)
to manage our many interdependent projects. Our repo is actually setup very
similarly to Facebook's mono-repo, though obviously considerably smaller in
size. We were encountering _many_ of the same problems as Facebook with our
NPM setup -- long CI/CD build times, indeterminate node_modules directories
that caused our projects and tools to work for some people on our team but not
for others, and the inability to do "offline-only" npm installs in CI.

We actually talked with our friends at Facebook about these problems, and
tried many of the same approaches they did -- shrinkwrap everything, checking
in node_modules, uploading the node_modules folder to separate place (we used
another completely separate Git repo), etc. All these approaches either didn't
work well or were difficult to maintain, especially on a small team.

Yarn has fixed all these issues for us.

One not-yet-super-publicized feature of Yarn (though I'm told there is a blog
post coming) that has been super useful at Exponent is the ability to build a
distributable offline cache of dependencies. We have a "node_modules-tarballs"
folder in our mono-repo that contains tarballs of all the dependencies for
every single project that we maintain at Exponent. Yarn will pull from this
cache first before fetching from the NPM registry. This offers HYPER-
predictability...it's a level of confidence over and above the "yarn.lock"
file, and let's me sleep well at night knowing that my team can get up and
running quickly no matter where they're working, how reliable their internet
connection is, etc. No more worrying about "did I forget to update the
shrinkwrap" \-- things just work.

As JS developers, we all, beginners and experts alike, can experience this
"JavaScript fatigue" that we always hear about. Fighting with tools is never
fun. We just want to write code!

As we've started to use Yarn at Exponent, both on our engineer's machines as
well as in our continuous integration and deployment environments, we've at
least had one less tool to fight with. Yarn works, works fast, uses the great
parts of the NPM ecosystem that we know and love, and most importantly, is
_predictable_.

~~~
acemarke
I've been using the Shrinkpack tool (
[https://github.com/JamieMason/shrinkpack](https://github.com/JamieMason/shrinkpack)
) to do that with NPM, and am trying to figure out how to do the same thing
with Yarn (per my request at
[https://twitter.com/acemarke/status/785886788493553664](https://twitter.com/acemarke/status/785886788493553664)
).

Definitely looking forward to that blog post from the Yarn team, but I'd
definitely be interested in your own comments on how to use that "checked-in
tarballs" approach.

------
manojlds
Clashes with Apache YARN - [https://hadoop.apache.org/docs/r2.7.2/hadoop-
yarn/hadoop-yar...](https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-
yarn-site/YARN.html)

~~~
vlunkr
They are different problem domains, I think confusing them is unlikely

~~~
loukrazy
Tell that to my google searches

~~~
__david__
I'm sure that will fix itself very soon.

~~~
andrewprock
I'm sure it will.

Search "go" on google, and the first link is the programming language. Search
on any other search engine, and as one might expect, the definition or the
game of go is ahead of the language.

~~~
smrtinsert
Wait till I release my new programming language, 'the'. It's the go killer.

~~~
imtringued
the haskell eliminator

------
swang
wow already one of the features i'm loving in yarn is that it tells you which
package is firing warnings about package incompatability.

warning electron-prebuilt-compile > electron-compilers > jade@1.11.0: Jade has
been renamed to pug, please install the latest version of pug instead of jade

in npm, that would have just said the part after "jade@1.11.0" which was
really vague and didn't really make you want to "fix" it because which npm
module do you have to go into? who knows because npm (the package manager)
didn't tell you.

~~~
algesten
i seriously want npm to stop warning me of outdated deps 3 levels down my
hierarchy. so what if some shitty old unmaintained lib we started relying on a
few years back use lodash v3.x? i'm not going to worry about it as long as it
works.

------
steveklabnik
It looks like this addresses the biggest issues people have with npm's CLI,
and it's coming from such huge names: Facebook, Google, and Tilde.
Reproducible builds are a _huge_ issue, and this gives you that. Looks great!

One interesting little tidbit I found from diving into the source:

    
    
      https://github.com/yarnpkg/yarn/blob/master/src/constants.js#L15
    
      https://github.com/yarnpkg/yarn/blob/master/src/registries/yarn-registry.js
    

It's not mentioned in the post, but looks like they're running their own
registry as well...

Oh! And open governance:
[https://github.com/yarnpkg/rfcs](https://github.com/yarnpkg/rfcs) !

~~~
k__
It's not the first one addressing these problems.

Nix and ied (which borrowed from Nix) got these problems pretty much solved.

I don't understand what spoke against these approaches?

I mean okay, nix has its own language, which is probably a turnoff for JS
devs, but ied?

~~~
ag_dubs
ied didn't work on windows because of the linking strategy. many people who
need to use npm use windows.

~~~
blubbi2
That's not 100 % accurate. Arguably symlinks are kinda weird under Windows,
but that doesn't mean symlinking node_modules doesn't work (you just need
admin privileges).

(Disclaimer: I'm the author of ied.)

------
STRML
This is a really big deal. We have a very large/complex web application & API
with a large number of dependencies, and not a single npm alternative (pnpm,
ied, npm-install) could figure out the tree properly.

It has been working well with our npm proxy and is basically a faster, better-
deduped drop-in replacement that removes our need for npm-cache, npm-check-
updates, and all of our shrinkwrap hacks.

This is finally the npm rewrite we've been waiting for.

In looking at the yarn repository, it's nice to see a clean, modern build, ES6
+ async/await, Flow typing, and so on. This is what a well-maintained JS
project looks like in 2016 that's built for production use.

npm, for all its flaws, was a revolution when it was first written. But since
then, it has been large, unruly, difficult to maintain, and has largely
unaddressed many important production use cases. It is nice to see a group of
companies using Node in production take the lead on addressing problems that
really matter to production JS apps.

------
chrismorgan
See also:

· [http://yehudakatz.com/2016/10/11/im-excited-to-work-on-
yarn-...](http://yehudakatz.com/2016/10/11/im-excited-to-work-on-yarn-the-new-
js-package-manager-2/)

· [https://yarnpkg.com/](https://yarnpkg.com/)

------
ef4
Kudos to the Yarn contributors for launching with a public RFC governance
model right from the start:
[https://github.com/yarnpkg/rfcs](https://github.com/yarnpkg/rfcs)

Practical, usable, deterministic lockfiles are wonderful and the lack of them
has been the single biggest pain point in the npm ecosystem.

------
tolmasky
My initial tests with this are very positive (I tried this use case here:
[https://github.com/npm/npm/issues/10999](https://github.com/npm/npm/issues/10999)
). It handled it quite well and we'll definitely be considering this for our
package.

I'm curious about the "flattening" approach though -- and why something like
ied (
[https://github.com/alexanderGugel/ied](https://github.com/alexanderGugel/ied)
) wasn't done instead. ied only ever installs one copy of a module, but allows
each module to think they have their own copy of it through clever symlinkery
(essentially making it FEEL like npm 2 while getting the benefits of
npm3/yarn). I'm not a fan of (explicit) flattening for a number of reasons:

1\. Mimicking recursive dependencies while still only having on physical copy
on disk has the benefit (or downside I suppose) of isolating and protecting
different module's dependencies from each other. AKA, if you
require("x").mutate = true, this won't be seen by other modules. Again,
depends whether you like this feature or not, but with node's treat-symlinks-
as-different-files configurable switch, you can actually control it yourself
in ied I believe.

2\. You don't run into tricky situations where require("x") can accidentally
pick something up you didn't mean to (since it is now in the parent path of
ALL your dependencies). So, if A depends on x, and you install B, it will be
able to require("x") (maliciously?) and modify it, affecting A's behavior.
Perhaps this is considered a strange possibility, but it gets particularly
weird with peer dependencies, where you expect the user to choose their own
top level dependency, but this can now be accidentally "chosen for you" by a
subdependency of another package.

3\. I just don't like clicking on my node modules folder and seeing 100 top
level directories when conceptually I typed "pkg-manager install one-thing".
This can lead to you yourself making the above mistake, by require("x") and
having it "just" work, then having it "just break" when you remove the
dependency that included x.

I'm willing to be convinced though (on RunKit we had module-fs simulate npm2
install behavior, super simple recursive installation, conceptually similar to
what ied does).

~~~
sebastianmck
Hey! I'm Sebastian McKenzie (@kittens on GitHub) and I'm the lead develop on
Yarn at Facebook. We initially used a method similar to ied when we first
started experimenting with internal usage at Facebook.

We ran into a lot of issues with OS compatibility around symlinks and existing
tools not supporting it. Windows lacks support for symlinks on non-Admin
accounts, NTFS junctions work but they're slightly quirky in their behaviour
and don't behave exactly the same way.

A lot of tooling relies on the existing node_modules structure. ESLint for
example will load tools relative to itself which means that they need to be
reachable in the tree, you can solve this with hardlinks but there's caveats
with that too. Existing tools aren't very well suited for handling cycles
either which in a system like npm are extremely common as a transitive module
could depend on one that's higher in the graph.

We care a lot about existing ecosystem compatibility and because of these
limitations we went with the current status quo. We'll continue to explore
others options though and if there's a chance we can do it without any of
these caveats then we're more than happy to explore it.

Let me know if you have any further questions!

~~~
cpojer
As Sebastian said we are more than excited to explore what we can improve in
the future. If you are interested, please start a discussion on the GitHub
issue tracker:
[https://github.com/yarnpkg/yarn](https://github.com/yarnpkg/yarn) :)

------
kzahel
"The React Native package.json currently lists just 68 dependencies, but after
running npm install the node_modules directory contains 121,358 files."

That, to me, is what is wrong with npm. The problem stems from node.js not
coming with "batteries included" so there is a proliferation of tiny libraries
that do the most trivial things.

~~~
mschuster91
> That, to me, is what is wrong with npm. The problem stems from node.js not
> coming with "batteries included" so there is a proliferation of tiny
> libraries that do the most trivial things.

This is in no way a fault of node.js but of the whole JS standard. People
these days use npm modules to run in a browser and ship stuff bundled together
with webpack (or other bundlers), so even if nodejs had a proper stdlib, you'd
still need to depend on a polyfill so that your stuff works in a browser
environment, too.

And due to the fact that even if a sane stdlib would ever be standardized, it
would take YEARS of time until it reaches significant market share (looking at
you, Android, Safari and IE), so you'd always have to ship a polyfill.

~~~
jay_kyburz
Yeah, but we could have a big, standard library polyfill, and it would quickly
find its way into everybody cache, just like jQuery.

~~~
toefraz
We do. It's called Lodash.

------
spankalee
Yarn is particularly great for front-end web apps because of its flat
installation mode.

ES6 module imports and HTML Imports both require that dependencies are
imported by URL. This means that the only reliable way to import another
module is by relative URL, like:

    
    
        import * as $ from '../jquery/jquery.js';
    

This requires that packages are installed flat, as siblings. Yarn is going to
enable native JS modules and projects like Polymer to use npm instead of
Bower. :)

~~~
dismantlethesun
You can use webpack or JSPM to resolve directories at the top level. That way
you can just import "web/X" even if the web/ directory isn't a sibling to your
current file.

It's not very helpful if you are distributing an NPM package, but for people
who aren't authoring libraries, its a godsend.

~~~
spankalee
Do you mean import '/web/X'? Import URLs have to start with a `/`, `./`, or
`//`.

Even so, that requires a build tool. I want to be able to load working sources
directly out of my packages directory.

~~~
dismantlethesun
> Do you mean import '/web/X'?

Not if you use a build tool for your frontend.

> Even so, that requires a build tool.

That was basically what I was suggesting. I'm not sure what the other option
really is. If you're working within the browser (implied by this being a
frontend), then you have to use a build tool since javascript modules/imports
haven't landed in any browser yet.

~~~
spankalee
> If you're working within the browser (implied by this being a frontend),
> then you have to use a build tool since javascript modules/imports haven't
> landed in any browser yet.

For now, but native JS module loading is right around the corner, and at that
point I don't want to have a required tool to load my modules.

Currently I use HTML Imports, which don't require tools because they use
relative URLs. It works really well, so I hope to use that pattern with JS
modules.

------
pilif
Yarn is currently being powered by a service running at
[https://registry.yarnpkg.com](https://registry.yarnpkg.com) \- at least
that's what's referenced in the yarn.lock file that's created.

I see no documentation nor code for the service powering this, nor is there a
way to tell the command line tool to use a registry at a different address.

Aren't we just replacing the dependency on npm, Inc.'s registry with a new
dependency on Facebook Inc's registry with this?

Yes. The client has a few advantages, but the registry is still closed and
while with npm there's an option of running a private registry if you pay
them, here, there's no option at all.

~~~
ag_dubs
that registry is a mirror of npm's registry. the mirror contains all package
data but does not include any of the permissions or user management, so this
tool still depends on npm infra for both that, and as a replication source.

~~~
thejameskyle
registry.yarnpkg.com is a proxy, it doesn't contain any package data except
for what gets cached by Cloudflare. Just a Cloudflare'd "CNAME
registry.yarnpkg.com. registry.npmjs.org."

------
beefsack
Is it just me, or does Facebook seem to create new competing tools and
technologies more often than the other majors?

My inner-cynic makes me feel like everyone at Facebook is really young, as
I've noticed a trait of inexperienced developers is to try to create new
things over finding and improving existing tools.

In reality it could be something as simple as Facebook releasing more FOSS
software than others, or maybe their releases having more visibility.

Either way, feels like Facebook like to pave their own way, for better or
worse.

~~~
cpojer
We don't think this way. We needed to solve this problem for Facebook,
explored alternatives and made the case to build Yarn. We work at Facebook to
solve Facebook's engineering challenges; when we can collaborate with other
companies and the open source community, that's an added bonus.

~~~
dacjames
That mindset would seem to explain why Facebook creates more competing
technologies than other players.

------
fold_left
Brilliant. I've been trying to tackle this problem for the last year or so
with shrinkpack
([https://github.com/JamieMason/shrinkpack](https://github.com/JamieMason/shrinkpack))
with _some_ degree of success and a nagging feeling that much more could be
done.

It's great to see attention is being spent on what is a really important area,
one which has been largely under-appreciated in Node.js until now (aside from
for a short period after the left-pad incident).

Kudos to Facebook, I look forward to giving it a try.

------
Benjamin_Dobell
Hmm, NIH Syndrome?

The justification for Yarn starts by ignoring the existence NPM Shrinkwrap. It
then proceeds to mention Shrinkwrap and refer to its supposed short-comings.
Whilst there absolutely were issues with Shrinkwrap in the past, they've
already been addressed by NPM.

NPM is slow, perhaps Yarn is faster and has some extra features. But why these
couldn't be contributions to NPM seems like a serious case of NIH...

~~~
brandonbloom
> NIH Syndrome

No. NPM is awful in ways that could not be fixed without braking changes. They
have a need, so they satisfied it. A need which, by the way, every one of the
dozen-or-so Node-using projects I've worked on have shared.

> Whilst there absolutely were issues with Shrinkwrap in the past, they've
> already been addressed by NPM.

This isn't even remotely true. Shrinkwrap is still a usability nightmare and
doesn't begin to address the non-deterministic install behavior that crops up
when you add new dependencies.

> why these couldn't be contributions to NPM seems like a serious case of NIH

There's no way that they could have got these changes in to NPM in anywhere
near the time it will take for Yarn to become popular; if at all! Politics (or
the avoidance thereof) are practically _the_ perfectly valid reason to justify
NIH.

I'll wait a while for Yarn to bake, but personally, I hope NPM dies a swift
and total death.

~~~
Benjamin_Dobell
> No. NPM is awful in ways that could not be fixed without braking changes.
> They have a need, so they satisfied it. A need which, by the way, every one
> of the dozen-or-so Node-using projects I've worked on have shared.

Examples, please.

> This isn't even remotely true. Shrinkwrap is still a usability nightmare and
> doesn't begin to address the non-deterministic install behavior that crops
> up when you add new dependencies.

Again, examples, please.

> There's no way that they could have got these changes in to NPM in anywhere
> near the time it will take for Yarn to become popular; if at all! Politics
> (or the avoidance thereof) are practically _the_ perfectly valid reason to
> justify NIH.

Did the Yarn developers make any attempt to contribute to NPM at all? If they
did and were turned back, then sure, I understand. However, everything I've
read about Yarn thus-far is just vague finger-pointing at NPM.

I'm by no means suggesting NPM is perfect, for one I find it incredibly slow.
However, surely that can be optimised.

However, regarding dependency version locking, which is the problem Yarn is
trumpeting, what are the actual issues with Shrinkwrap? I'd like something
specific. Since Shrinkwrap was updated to keep a flat sorted structure (a
while ago) I've been using it no fuss at all. What are the specific issues
people are running into now?

I've seen comparisons with Bundler, and how Yarn will be more like Bundler's
Gemfile.lock files. However, the thing is the current version of Shrinkwrap
works almost identically to Bundler's lockfile e.g. When I update a dependency
the shrinkwrap file _automatically_ updates (at least that's the default
behaviour).

~~~
brandonbloom
> > ways that could not be fixed without braking changes

> Examples, please.

The first and foremost: Version pinning should be the only way it works.
Obviously _requiring_ shrink-wrap would be a breaking change.

> > Shrinkwrap is still a usability nightmare

> Again, examples, please.

Having to explicitly call shrink-wrap is just silly. Devs on every time I've
used shrink-wrap with frequently forget to do it and it causes all sorts of
"works for me" problems later. My team has also experienced all kinds of
brokenness related to private NPM repositories and shrink-wrap non-
determinism.

> Did the Yarn developers make any attempt to contribute to NPM at all

I don't care about this at all. Just because you got there first doesn't mean
you should have some kind of special claim to solving the problem.

~~~
Benjamin_Dobell
> The first and foremost: Version pinning should be the only way it works.
> Obviously _requiring_ shrink-wrap would be a breaking change.

> Having to explicitly call shrink-wrap is just silly. Devs on every time I've
> used shrink-wrap with frequently forget to do it and it causes all sorts of
> "works for me" problems later.

Really? An entirely new tool because you didn't like typing "npm shrinkwrap"
once, just once per project? I... I don't even know how to respond to that...

> My team has also experienced all kinds of brokenness related to private NPM
> repositories and shrink-wrap non-determinism.

I'm unclear on what you mean? Do you mean in the past? Cause if so, sure...
but hasn't this already been addressed?

> I don't care about this at all. Just because you got there first doesn't
> mean you should have some kind of special claim to solving the problem.

This is an all too common attitude in the JavaScript community. Hence the
plethora of poorly maintained, duplicated NPM packages.

I'd rather humanity not waste the man-hours duplicating effort. Do something
productive with the time you saved.

As mentioned, if for some reason the NPM guys weren't co-operative, then I
understand, however that doesn't at all seem to be the case.

~~~
brandonbloom
> An entirely new tool because you didn't like typing "npm shrinkwrap" once,
> just once per project?

I don't know what kind of projects you work on, but new dependencies get added
with some regularity on any large node project, requiring a shrink-wrap every
time.

If you've never experienced the forgetting-to-shrinkwrap problem and its
interplay with team dynamics, then I envy you. But I also must give up trying
to explain to you, since this drives to the heart of the problem and I must
conclude that we simply do not have enough common experiences for you to ever
understand. The developers of Yarn et al clearly have lived this nightmare.

> Do you mean in the past

Where the past was < 4 months ago. If it has been fixed since that, I don't
care: too little, too late.

> This is an all too common attitude in the JavaScript community.

An equally big problem in the JavaScript community is that core pieces of
infrastructure have been "claimed" by people who have absolutely no idea what
they are doing, but are happy to community-coordinate their way to some kind
of idealistic open source victory.

> weren't co-operative

Again, I don't care if they were willing to co-operate. I have zero confidence
in them.

> I'd rather humanity not waste the man-hours duplicating effort.

Give up on this dream. People are going to re-invent everything from now until
the end-of-time, and rightly so.

~~~
Benjamin_Dobell
Clearly you haven't been using the default settings in the latest version of
NPM ;)

I can only hope the Yarn developers knew about the changes, and had a real
justification for their work.

Again, I'm not saying the Yarn developers _didn 't_ have justification, I just
haven't seen any yet. Love to be shown otherwise.

------
sdegutis
For what it's worth, the name is really cool. I just looked up the etymology
of Node and it comes from Latin `nodus` meaning knot. So if a Node program is
the knot, then the stuff that makes it up and that it comes from must be the
yarn! Language is fun.

------
wildpeaks
Note that it doesn't work with private modules yet, but it's coming soon:
[https://github.com/yarnpkg/yarn/issues/521](https://github.com/yarnpkg/yarn/issues/521)

------
andreygrehov
I just tried to install yarn and ran it to against a relatively large React
application. What's interesting is that it found an incompatible module and
could not proceed, whilst npm works fine.

The article states that running `yarn` is the same as `npm install`, which
doesn't look like the case. At that point, I'm curious what are the
differences.

~~~
thejameskyle
Sorry about that, there are very likely still some bugs out there we weren't
able to catch in all our testing which will be ironed out by the community in
the next few days. Can you open an issue with details about which package
failed to install?
[https://github.com/yarnpkg/yarn/issues/new](https://github.com/yarnpkg/yarn/issues/new)

~~~
andreygrehov
Thank you. Done. The issue is reproducible with an entirely new project.

~~~
steveklabnik
A fun side effect of reproducible builds: recreating bugs for bug reports is
easier.

------
nimish
Excellent. The NPM cli is a dumpster fire-- cf
[https://github.com/npm/npm/issues/11283](https://github.com/npm/npm/issues/11283)

A better std lib for node and module LTO would be a good step forward to
removing the proliferation of tiny packages

------
throwaway6497
Why did we have to pick the name Yarn which clashes with Apache YARN on which
Hortonworks and Cloudera depend for their survival?

------
szimek
Recently I got annoyed how hard it is to use shrinkwrap in npm and started
working on a npm wrapper that would make npm as easy to work with as Ruby
Bundler by copying its workflow as closely as possible
([https://github.com/szimek/bundlerjs](https://github.com/szimek/bundlerjs)).
Thankfully, I don't have to develop it anymore ;) Big thanks to all Yarn
developers!

------
foxwoods
yarn.lock[0] is similar to composer.lock[1]

[0]
[https://github.com/yarnpkg/yarn/blob/99dd4504469273332f01ce5...](https://github.com/yarnpkg/yarn/blob/99dd4504469273332f01ce5ad846c995378cc048/yarn.lock)
[1]
[https://github.com/composer/composer/blob/20ee689bb464edfe7e...](https://github.com/composer/composer/blob/20ee689bb464edfe7e57af6be8f03fb664f5db1d/composer.lock)

~~~
dham
Pretty close to
[https://github.com/discourse/discourse/blob/master/Gemfile.l...](https://github.com/discourse/discourse/blob/master/Gemfile.lock)

------
wale
While Yarn sounds like a nice wrapper with extra stuff. I want to ask if this
collaboration couldn't happen with the NPM team?

~~~
thejameskyle
npm's been involved since the early days, they wrote up a blog post here:
[http://blog.npmjs.org/post/151660845210/hello-
yarn](http://blog.npmjs.org/post/151660845210/hello-yarn)

~~~
wale
Oh Okay. Good one. Thanks for the blog ref #openess

------
danielrbradley
Overall looks good, but it's a shame that flat mode is opt-in instead of the
default - this is the sane way of doing package management. Would have been
nice to flip the options over so you have to explicitly allow multiple
versions of the same package - which you could automatically enable if you're
converting from npm.

~~~
steveklabnik
Elsewhere in this thread, they said that its because it doesn't work well with
enough of the ecosystem yet.

------
ensiferum
I don't know JS and I've never used these js package managers, but are these
issues such that they couldn't have been fixed in the original package
manager? Smells like NIH that is just going to create head ache and misery for
_everyone_ in the JS community in the long term.

------
colemannerd
I hope this becomes the default way to build npm projects... and then npm
install becomes yarn :-) Exactly reproducible builds is a godsend to the js
community.

------
kabes
NPM must have been the only thing the JS community has standardised on. Now
that's gone too

------
epmatsw
This looks awesome. But I have to wonder why create a whole new project rather
than fork or upstream these changes to NPM? It doesn't seem like it's doing
anything fundamentally different or outside of NPM's scope of responsibility.

~~~
pluma
Because the npm client is a hot mess?

The official npm cli is quite old and evolved along with all the different
coding styles and architecture choices of the node and JS ecosystems.

It's also full of dependencies on originally purpose-built modules that suffer
from the same problems. This compounds various issues, resulting in long-
standing bugs like `npm publish` sometimes not actually including all files in
the tar bundle it generates[0].

NPM 3 was a partial rewrite but it's still built on the same mountain of code
as NPM 2 and 1. npm Inc developers have in the past complained about not being
able to address issues for a lack of resources despite having a ton of
developers on the team[1].

The official npm client also doubles as an account management CLI to the npm
registry. Which apparently npm Inc now considers a design flaw as they
recently started creating a standalone tool for managing other aspects of
their commercial services[2].

And finally it's important to understand that having a new package manager
that is not tied into npm Inc means it's easier to replace the npm registry as
the official node module registry in the future. Which would allow the
community to get rid of the privileged position npm Inc currently holds in the
node distribution (which is otherwise governed by the Node Foundation).

Rewriting the client from scratch actually turned out to be the sanest
approach in this situation, I think.

[0]:
[https://github.com/npm/npm/issues/5082](https://github.com/npm/npm/issues/5082)

[1]: [https://github.com/npm/humans](https://github.com/npm/humans)

[2]: [https://npmjs.org/package/wombat](https://npmjs.org/package/wombat)

~~~
ag_dubs
a few points!

fwiw, only 2 of those humans work on the CLI, (3 if you count their manager,
who spends a lot of time managing). other people work in sales, marketing,
engineering on the website, engineering the registry, community + docs, etc.

wombat is not a tool for managing our other commercial services! it's a way to
manage our webhooks product, one of many products we have, though very
certainly the smallest.

yarn depends on the npm registry, which depends on npm, Inc making money to
keep it running.

~~~
pluma
> only 2 of those humans work on the CLI

Thanks for clarifying. But that does show that having a company fund two full-
time developers to work on the CLI wouldn't have been sufficient. Yarn was
mostly built by Seb (403 commits) starting in late January and Konstantin Raev
(121 commits) with contributions from several others (<30 commits each). The
commit graph[0] actually shows development mostly paused in April and May,
putting the total development time closer to half a year.

This most likely includes collaboration with the other people supporting the
project (both Seb and Konstantin work for FB) and a lot of architectural
planning -- aspects, which likely would have required an even greater
investment of time if building on the legacy codebase of the official npm
client (not to mention having to negotiate all changes with npm Inc directly).

> wombat is not a tool for managing our other commercial services!

Sorry about the misunderstanding. When wombat came out a lot of people
wondered why wombat was a standalone tool when practically every other aspect
is part of the npm cli and from what I gathered from npm devs at the time the
decision was more of a technical one (different people working on different
tools for different things) than a managerial one (different products
deserving different tools).

> yarn depends on the npm registry, which depends on npm, Inc making money to
> keep it running.

Yarn currently uses a downstream mirror of npm as the direct source of truth,
with support for private registries being actively worked on to support the
private parts of the npm registry and products like npm enterprise or free
alternatives like sinopia.

However that is obviously an implementation detail. Decoupling yarn from the
npm registry by going through an intermediary allows experimentation (as the
npm blog post points out) and ultimately allows avoiding a hard dependency on
npm Inc as a company.

If npm Inc were to shut down right now, yarn would still have a read-only
mirror. I'm sure Facebook and Google have enough experience solving problems
at scale that they could build a public registry alternative if necessary
(such as if npm Inc went out of business tomorrow).

I'm not saying yarn will kill npm Inc. But yarn can kill the npm client and
replace it with a community-supported alternative (under an open governance
model similar to Ember), which in turn makes it a lot easier to fade out the
reliance on the public npm registry maintained by npm Inc -- benefiting both
the community (by eliminating a single point of commercial failure in the node
ecosystem -- exactly like joyent was to the node project) as well as npm Inc
(by eliminating a major drag on development resources and cost center).

The only drawback from such a move to npm Inc would be the loss of its
privileged position in the ecosystem, the "first mover" advantage of being the
place to go if you want private packages (because it's more frictionless than
any of the alternatives -- exactly like GitHub in the git hosting space).

[0]:
[https://github.com/yarnpkg/yarn/graphs/contributors](https://github.com/yarnpkg/yarn/graphs/contributors)

------
mvdanj
Its disheartening to see so many duplicate efforts to solve the same problems
everyone faces. Instead of supporting an existing open-source project that
attempts to solve the problem in pretty much the same way (jspm), a
conglomerate once again builds their own from scratch. I do not think the
underlying motivations for doing so are questioned enough. Sure there is
control, but it is of course anyway-you-slice-it an aggrandizement of their
brand. We should be ashamed as a community to support the idea of yet ANOTHER
client side dependency management system. How oh how did we ever solve this
problem before Facebook came along and made Yarn in late 2016?

~~~
tomdale
I don't think it's fair to compare jspm and Yarn. jspm is an interesting
project, but is tightly coupled with SystemJS. Telling everyone to rewrite
their apps to use SystemJS and jspm is a huge lift.

Yarn, on the other hand, has focused on npm compatibility from the get-go. The
entire point is to be a drop in replacement, with no changes (or maybe very
very minimal changes) required to gain a lot of benefit.

In this slice of the industry where everyone is told they have to rewrite to
the latest thing every year or two, I find it very refreshing to see Yarn's
focus on backwards compatibility, improving existing infrastructure, and
having a clear, painless adoption path.

~~~
mvdanj
I don't agree with any of your statements, and I think they are inaccurate.

jspm aims to be type-agnostic with respect to modules (supports all major
types, hence why it is called a universal module loader). jspm certainly
doesn't advocate rewriting your modules (you can use exports, AMD, ES6,
whatever). In fact, when the browsers natively support module loading,
systemJS will go away (which is really just a polyfill for this
functionality).

jspm also has npm compatibility.

------
smrtinsert

      yarn global add typescript
      yarn link typescript
      error No registered module found called "typescript".
      info Visit http://yarnpkg.com/en/docs/cli/link for documentation about this command.
    

what?

edit: also,

    
    
      $ yarn add typescript
      yarn add v0.15.1
      [1/4] Resolving packages...
      [2/4] Fetching packages...
      [3/4] Linking dependencies...
      error ENOENT: no such file or directory, open  'C:\Users\******REDACTED******\@types\react\index.d.ts' at Error (native)
      info Visit http://yarnpkg.com/en/docs/cli/add for documentation about this command.

~~~
vjeux
Can you open an issue in the repository please :)

------
geofree
Nice name too: [https://www.getyarn.io/yarn-
clip/6f28eef4-98fe-4ae5-96c0-da3...](https://www.getyarn.io/yarn-
clip/6f28eef4-98fe-4ae5-96c0-da3aa65ce319)

------
zuck9
The node_modules folder is my least favorite part of the JS ecosystem. I
realize now that's mostly because of npm-cli.

npm-cli's job was simple. Take in a `package.json` as input and output the
same `node_modules` folder as fast as possible. But somehow it couldn't do
that even after getting 6 years old. On top of that, every now and then with
when you ran `npm update` or `npm dedupe`, it messed up the `node_modules`
folder with extraneous and invalid packages.

------
nodesocket
Does yarn support custom `run` commands like npm?

For example:

    
    
        "scripts": { "dev": "node api.js --config config-dev.json" }
        $ npm run dev

~~~
steveklabnik
Yes. See the diff here for an example of moving a project from one to another:
[https://github.com/rust-lang/crates.io/pull/457](https://github.com/rust-
lang/crates.io/pull/457)

lots of "npm run" -> "yarn run" there.

~~~
nodesocket
Sweet. Would be cool to have yarn support passing dynamic arguments into
scripts/run commands.

------
edibleEnergy
Just installed it and ran it on a largish project (
[https://www.bugreplay.com](https://www.bugreplay.com) ), the parallel
downloads really sped things up and outputted some great diagnostic info about
problematic dependencies. BugReplay is a webapp for recording webapp bugs, so
having some confidence that our dependency chain isn't introducing our own
bugs would be awesome :)

------
joepie91_
Well, this is a problematic case of "throwing out the baby with the
bathwater". Semantic versioning isn't the problem here (the lack of
deterministic install trees is!), yet it's thrown out pretty much entirely,
without even so much as considering why it exists in the first place.

I wonder whether Facebook realizes just how much damage they are going to be
doing to the JS ecosystem with this approach.

~~~
steveklabnik
How is this throwing away semver?

~~~
joepie91_
As I understand it, it automatically locks in the (exact) version of
everything you install, rather than having this as a separate, optional
feature (like shrinkwrap does, despite it not working well - but that's an
implementation problem).

The big deal about semver and how it's implemented in NPM is that there is a
_choice_ of how to deal with it - either you automatically allow compatible
updates and risk an occasional maintainer fuckup, or you explicitly pin
everything and take on the burden of constantly updating your versions
manually and tracking security releases.

The latter might work for Facebook and other large organizations, but for many
smaller developers, the consequences of an accidental break are much, _much_
less serious than those of constantly having to keep pinned dependencies up to
date. End result: people don't bother _at all_ , and insecure versions never
get upgraded for the majority of the userbase.

This, in turn, removes the incentive for package authors to do anything with
semver _at all_ , once Yarn's approach becomes commonplace. Why bother with
semantic versioning if everybody is expected to manually check and pin their
dependencies anyway?

Facebook also needs to take into account their influence over the ecosystem.
The reality is that whatever project Facebook puts out is immediately seen as
gospel and "naturally, a well-engineered thing and a good choice to use!" \-
this is literally a _daily_ argument in #Node.js when somebody is trying to
argue why a poorly-fitting tool is somehow appropriate for their usecase. _"
Well, Facebook uses it!"_

In reality, the Yarn workflow is designed to _work for Facebook_ and
organizations of similar size, and in all likelihood it will break down in
other situations. People will not realize this until they've already bought
into it, and by then it's too late.

~~~
steveklabnik

      > Why bother with semantic versioning if everybody is expected to manually
      > check and pin their dependencies anyway?
    

I think this is the disconnect here: the lockfile does protect me from
updates, but ranges are still useful: they allow a tool to help me upgrade my
dependencies. With the lockfile-based approach, you usually _don't_ pin to
specific versions: with Cargo, we even defaulted to the ^ operator, so 'foo =
"1.2.3"' is the same as 'foo = "^1.2.3"'. So this is what happens:

    
    
      1. foo v1.2.3 comes out and I put that in my Cargo.toml.
      2. foo v1.3.0 comes out. My build doesn't change.
      3. I decide "hey, there's some new versions of my dependencies, let's upgrade."
      4. I type "cargo update". Now, my lockfile is updated to have 1.3.0 instead.
    

That is, ranges allow you to use semver as a tool. The big issue here is
transitive dependencies; yes, for my own deps, I could have specified an exact
version and done it every time. But what about the dependencies of my
dependencies? Or the dependencies of the dependencies of my dependencies?

I totally understand your concerns about cargo-culting tooling, and that
Facebook-sized organizations have different needs than non-Facebook sized
ones. But repeatable builds are a big deal, and managing dependency ranges so
you can still use semver while getting repeatable builds is a big deal. Even
for smaller organizations.

(And that's beyond the speed and security aspects of yarn.)

~~~
joepie91_
> That is, ranges allow you to use semver as a tool.

I understand that, but you're looking at it from a _technical_ perspective,
while it's partially a _psychological_ issue - namely, the question of how you
drive certain desirable behaviour.

When you make upgrades explicit, rather than automatically installing newer
versions, you _reduce_ the deployment of (security) bugfixes, because many
people simply will not run `upgrade`.

The reason is that you've created an _expectation_ of manually checking
versions before updating them - after all, why would it be a manual action
otherwise? - and people do not want to commit to doing that. _That_ is why
this will harm the ecosystem - it undermines the mutual trust that a
maintainer will be using semver, and that you can rely on it to automatically
get updates.

The solution is simple: make lockfiles opt-in. That doesn't mean that you
can't aggressively require a developer to lock their versions once they _have_
opted in - it just shouldn't be a feature that is enabled by default.

> But repeatable builds are a big deal, and managing dependency ranges so you
> can still use semver while getting repeatable builds is a big deal. Even for
> smaller organizations.

For many people, it really isn't. Comparing the tradeoffs of manual upgrade
management vs. the chance of occasional breakage due to non-determinism, many
people will pick the latter.

And I'm not just talking about "organizations" here either - there's plenty of
development going on outside of any organization, on personal projects, and
_that_ is precisely the kind of development you'd want to encourage.

~~~
steveklabnik
Thanks for elaborating. I think we fundamentally disagree, but I understand
your perspective a lot better now.

------
hswolff
This is super exciting! Really excited to try this out and glad to see
innovation in this area.

I'm also bummed cuz I've been working on a static site generator for over a
year written in node called Yarn as well[1].

I guess it's time for a rename! Any suggestions? :D

[1] [https://github.com/yarnjs/yarn](https://github.com/yarnjs/yarn)

~~~
wayfarer2s
Oh that sucks..How about "yarnish"? It's available on npm, sticks to your
roots, and you can play on varnish if you like. Or you can always go the
thesaurus route and use fleece, wool, cotton fiber, flaxen thread...

------
akhatri_aus
So how do you install packages like `npm install` without adding it to the
package.json file?

~~~
skrebbel
You don't. If you want to be able to do stuff like that (and inevitably run
into "hmm, works on my machine"-type bugs once you start sharing your code
with others), you need to keep using npm.

This kind of liberty is the sort of stuff npm was designed for. It is also
_why_ npm has been causing trouble in many cases.

~~~
akhatri_aus
I was under the impression that this was about speed.

There are many use cases for this functionality, the design of a software
package shouldn't instill dogma into its users. Users use software for their
needs, not to sign up to a religion.

~~~
skrebbel
No, speed is bonus. Yarn is about reproducible dev/test/build setups. Same
yarn.lock - same node_modules content. That's the primary goal.

If you think consistency across computers is a religion then I don't think
we're going to find a lot of common ground here though.

Finally, yarn is not intended to replace the npm client. It is intended to
replace it _for a certain use case_ , which happens to be one many companies
have. Unlike Yarn, Npm is highly flexible and obviously you like it like that.
I strongly doubt it's going to stop being developed all of a sudden because
some other people made Yarn. So what, exactly, is the problem?

~~~
akhatri_aus
> So what, exactly, is the problem?

Follow users needs, not your idea of what they are or the problems with npm
are.

------
sontek
Has anyone successfully got this to work? I was excited to try it so I did:

    
    
        $ rm -rf ./node_modules
        $ npm install yarnpkg
        $ ./node_modules/yarnpkg/bin/yarn
    

and it just freezes up and never finishes:

    
    
        [4/4]   Building fresh packages...
        [1/3] ⠈ node-sass
        [2/3] ⠈ radium
        [3/3] ⠈ phantomjs-prebuilt
        [-/3] ⠈ waiting...
        [-/3] ⠈ waiting...
    
    

This is not a particularly complex app, the dependencies:

[http://paste.openstack.org/show/585495/](http://paste.openstack.org/show/585495/)

------
substack
One of the features for yarn mentioned is better offline support, but you can
also do this with vanilla npm:

    
    
      npm install --cache-min Infinity
    

I have this aliased to `npmi` on my system and use it all the time.

~~~
theanswerer
For unknown reasons it will still try to contact the npm servers once in a
hundred runs even though none of the package versions changed.

------
thunfisch
This is a very good step in the right direction.

It still doesn't solve the biggest problem the node.js and javascript
ecosystem has IMHO: dependency hell, thousands of trivial packages, just utter
chaos.

------
bouk
I'm so excited about this. Having wasted lots of time migrating Shopify to use
npm for JS package management, it seems that yarn fixes everything that had
been a nuisance with it. Woo!

------
petetnt
Rather impressed with this with the few tests I've run. Kudos to the creators
and can't wait to see this project progress (and to contribute into!) in the
near future.

------
gell_mann
This is a somewhat unrelated question. I'm a systems programmer (I spend most
of my days in the land of C and Go). I do understand the basic concepts of web
apps and how you develop one. I have even built a few websites using Rails.

Can somebody please explain to me: 1) How does npm fit in in the process of
developing a web app. How does it manage the js files / libraries and where
does it place them? 2) How is yarn going to make your life easier

Please go easy on me.

~~~
steveklabnik
1\. npm == bundler. One difference: it puts the source of your dependencies
into a node_modules folder in the project itself, rather than in some global
spot

2\. npm doesn't have bundler's lockfile, but this does. It's also faster.

~~~
gell_mann
I never guessed building a package manager / dependency tracker could be such
a complex problem.

~~~
steveklabnik
It's really tough! I like this post, if you want to learn more:
[https://medium.com/@sdboyer/so-you-want-to-write-a-
package-m...](https://medium.com/@sdboyer/so-you-want-to-write-a-package-
manager-4ae9c17d9527)

------
hiendv
Yarn shouldn't be installed from the package repository.The inconsistency
between the version of nodejs which yarn is running on and the version of
nodejs which npm is running on brings significant effects.
[https://github.com/yarnpkg/rfcs/issues/9](https://github.com/yarnpkg/rfcs/issues/9)
Any idea how to reach out to one of the authors?

------
losvedir
Very interesting! How does the lockfile work with optional dependencies? We've
run into this issue[0] on npm with a bad interaction between optional
dependencies and the shrinkwrap. We develop on OS X and our CI server is
linux, and fsevents works on one but not the other.

[0]
[https://github.com/npm/npm/issues/2679](https://github.com/npm/npm/issues/2679)

------
mrmrben
What does this mean for NPM Inc the company?

Is a replacement / mirror / rethink of the NPM infrastructure coming next?

If I was NPM Inc I would think about adopting yarn if possible.

------
theanswerer
A big +1 for yarn from me. I had no end of trouble with npm needing internet
connectivity at all times and its non-determinism. Thanks to all involved!

------
nawitus
Should you commit the yarn lockfile to version control?

~~~
quaunaut
Yes. This makes it part of code reviews and is what keeps everyone who might
ever use the project(whether human or CI) using the same libraries.

------
currywurst
I wish some kind of "auto-bundle" feature prevented the creation of multiple
hundred MB's worth of tests, readmes and docs on disk when something like
single "babel.bundle.js" file would do.

Is it really important that one knows that a tool dependency uses left-pad ?
What could be the drawbacks of such bundling?

~~~
thejameskyle
You can run `yarn clean` and it will clear out all the extra files you don't
need:
[https://yarnpkg.com/en/docs/cli/clean](https://yarnpkg.com/en/docs/cli/clean)

~~~
currywurst
Great !! That sounds like it should help atleast .. hope the heuristics
improve to capture most npm patterns

------
atomaka
Why another project? If npm has been suitable and only falls short in certain
cases, why not contribute back to it?

~~~
bronson
It's unlikely npm would accept patches moving it in the direction Yarn is
taking. Different priorities.

------
LoSboccacc
so, where are the packages stored? how is this more secure than npm? how does
this solve the leftpad problem?

~~~
nailer
leftpad was a social issue. It's been solved by policy on the registry side
stopping packages depended on by many others being unpublished.

~~~
ag_dubs
actualyl at this point, unless there is a legal request or a serious security
issue, we don't allow any unpublishes. we encourage transferring the package
to the npm user and deprecating it. yay immutable registry!

~~~
steveklabnik
It is unclear why this is downvoted: this is an npm employee who made the
requisite policy changes after left pad happened.

[http://blog.npmjs.org/post/141905368000/changes-to-npms-
unpu...](http://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-
policy)

------
jbverschoor
"Linking: Finally, Yarn links everything together by copying all the files
needed from the global cache into the local node_modules directory."

Stop copying stuff. Just make it global and link.

Next step is to make packages immutable and signed.

I'm happy with this step and the fact that facebook will be able to push this.

~~~
wycats
This is something I really want to explore more through less-compatible modes.
It was the original way yarn worked, but it wasn't compatible enough to be the
default mode:
[https://github.com/yarnpkg/yarn/issues/57](https://github.com/yarnpkg/yarn/issues/57)

~~~
jbverschoor
Hmm.. is it still available as a configuration option somewhere? I couldn't
find it on the site.

I'm a huge fan of bundler - it's dependency heaven. I'm also a big fan of the
rubygems repo. It does not allow changes in released versions.

Even without symlinks it's a much needed improvement in the javascript
ecosystem

~~~
wycats
This is something I hope to work on over the next few months. It's a priority
for Ember :)

------
slobbermaster
It appears that yarn.lock resolves exactly one hash version of a package for
one semvar version of it. What happens when your project depends on packages A
and B, which both depend on the same semvar version, but due to manipulation
by the author, rely on different hash versions?

------
ateevchopra
My results on the speed of yarn vs npm. Same repo, side by side comparison.

[https://twitter.com/ateevchopra/status/785914522364026880](https://twitter.com/ateevchopra/status/785914522364026880)

------
cdnsteve
"Excited to to recommend the @yarnpkg JS package manager

npm install – 2m5s

yarn (no cache) – 9s

yarn (cache) – 2s"

[https://twitter.com/yeoman/status/785888558544334848](https://twitter.com/yeoman/status/785888558544334848)

------
AnsemWise
JavaScript has enough issues that I prefer to not touch it, focusing more on
backend Python work. Yarn is one of the first steps I've seen that makes me
interested in JavaScript again. Excited to see where this leads the community.

------
po1nter
They should update the getting started section. I get a warning when trying to
install it:

    
    
        C:\Users\******>npm install -g yarnpkg  
        npm WARN deprecated yarnpkg@0.15.1: Please use the `yarn` package instead of `yarnpkg`

------
michaelsbradley
I think the write-up should specify `npm install -g yarnpkg` not `npm install
-g yarn`.

~~~
nailer
Could be worth finding the author of 'yarn' on npm and asking them for the
name - it's been unmaintained for 4 years:

[https://github.com/samholmes/yarn](https://github.com/samholmes/yarn)

~~~
FLGMwt
He's since donated it : )

------
ranyefet
As a JavaScript developer I can't say how much excited I am about this new
tool.

Most of my problems with developing modern JavaScript apps are around npm
client, long installs and huge dependencies folder.

I hope it will improve everybody workflows.

------
thecity2
So this is not a Hadoop thing?

~~~
andmarios
Just an easy way for javascript devs to add big data keywords into their CV.
:p

------
nikolay
Coming soon to Homebrew: [https://github.com/Homebrew/homebrew-
core/pull/5832](https://github.com/Homebrew/homebrew-core/pull/5832)

------
electic
Doesn't seem to be very bug free atm:

yarn yarn install v0.15.1 info No lockfile found. [1/4] Resolving packages...
error [https://registry.yarnpkg.com/babel-code-
frame](https://registry.yarnpkg.com/babel-code-frame): socket hang up at
createHangUpError (_http_client.js:252:15) at TLSSocket.socketCloseListener
(_http_client.js:284:23) at emitOne (events.js:101:20) at TLSSocket.emit
(events.js:188:7) at TCP._handle.close [as _onclose] (net.js:493:12) info
Visit
[http://yarnpkg.com/en/docs/cli/install](http://yarnpkg.com/en/docs/cli/install)
for documentation about this command.

------
eknkc
Oh god it's fast.. It's freaking fast.

Thank you, Whoever touched this. Thank you.

------
jiyinyiyong
In China, we have using part of yarn for long, installation speed can be
boosted with [https://github.com/cnpm](https://github.com/cnpm) .

------
z3t4
I think it's naive to not review dependency updates. Also if a minor or patch
update touches tens of thousands of files, you are dealing with a monster. And
monsters need to be in cages.

------
spasquali
Or just use shrinkpack:
[https://github.com/JamieMason/shrinkpack](https://github.com/JamieMason/shrinkpack)

But that would be too easy...

~~~
remi
Did you read the article paragraph about shrinkwrap?

~~~
spasquali
Not `shrinkwrap` -> `shrinkpack`.

Not sure why Facebook spent so much time and money to create a more
complicated "solution", or why so many people don't realize that.

------
ephimetheus
Hm I like the idea, but in my testing, yarn is about 2 times slower than
regular npm3...

EDIT: Even when it's hitting the cache, it's only ~5% faster than clean NPM
with networking for me...

------
philwhln
Just glad to see that there is no "yarn isntall" command.

------
incredulous
It's incredible that a tool that we've tolerated for years, npm, is instantly
and unceremoniously dumped for a superior offering, yarn. Technology changes
on a dime.

------
itsbits
Surely reliability will be improved with lockfile concept but why do we need
another package manager with this. Can't we suggest same to npm team?

------
dorianm
Github link:
[https://github.com/yarnpkg/yarn](https://github.com/yarnpkg/yarn)

------
fooyc
Why not contributing these improvements to non instead ?

Why is the JavaScript community continuously creating new tools instead of
contributing to existing ones ?

------
niix
Awesome! So far this is working great with a Dockerized Node application I
have. Great job Facebook and the rest who contribute!

~~~
mfornasa
It could finally solve the major pain in using Node over Docker for CI and
local development.

[https://medium.com/@mfornasa/using-yarn-with-
docker-c116ad28...](https://medium.com/@mfornasa/using-yarn-with-
docker-c116ad289d56)

------
stincity
I get NPM for backend development but I don't understand why frontend
developers require NodeJS just to utilize it's package management. It really
does not make any sense to me considering that NodeJS is server side
javascript. If my backend is Python or Go or .NET or Java or whatever, now
I'll need NodeJS for the frontend and only for NPM (and now this). It's a
reason why I don't enjoy dealing with "modern" frontend technologies.

~~~
gkoberger
Is requiring Node really that bad? I don't write Bash or Ruby or Python or Go,
but I have them all installed because various tools need them.

~~~
stincity
It doesn't make sense to require a backend tool for frontend technologies.
That's just me though. So that's why I'm asking. Couldn't they just create a
package system that's not based on some backend tech?

~~~
qudat
We have that and it's called `bower`. We moved away from it because having two
package managers to manage javascript (one for front-end and one for back-end)
is more insane than the issue you described.

~~~
stincity
Love that. Let's include a backend tool for just so our frontend stuff can
work. Makes sense.

------
nodesocket
Does registry.yarnpkg.com always just proxy to the npm registry, or is it
caching results and serving directly as well.

~~~
steveklabnik
Elsewhere in the thread, the devs have said its just a CNAME.

------
skybrian
The algorithm sounds very much like how Dart's "pub" command works. Anyone
know how they differ?

------
ostater
yarn's immediate acceptance by the community over npm appears to be a
wholesale rejection of semver and an embrace of dependency determinism. Every
dependency is locked down to the patch level with yarn. Instead of
major.minor.patch, it might just as well be a single incrementing number now.

------
ilaksh
Maybe instead of reinventing npm use a more general staging or deploment
tool/strategy like Docker.

------
tehchromic
It's a great read and feels good to know FB is solving problems that we all
run into!

------
tehchromic
It's a great read and feels good to know FB is solving problems that we all
run into

------
Kiro
So, as a casual user of node... Should I switch? I don't see any immediate
benefits.

~~~
ohitsdom
You don't? Consistent and reliable dependency versioning across all machines,
and speed:

[https://yarnpkg.com/en/compare](https://yarnpkg.com/en/compare)

~~~
Kiro
Sure, not saying there aren't any benefits but as a casual user they don't
feel big enough to justify a switch. I may be wrong though (hence why I'm
asking).

~~~
ohitsdom
It took less than 20 minutes for me to change over a decently large project.
The builds on the CI server increased by about 25 seconds by switching from
npm install to yarn install. It was definitely worth it for me.

~~~
seandavidfisher
Wait, your build times increased on the CI server? What makes that worth it?

------
johncoltrane
If only emojis were optional…

------
simplify
Am I correct in understanding Yarn works similar to Ruby's bundler?

~~~
thejameskyle
Similar, but even more similar to Rust's Cargo
[https://crates.io/](https://crates.io/)

------
pylinux
Anybody know what yarn does when it encounters a "*" version?

------
niahmiah
Many of these problems could have been avoided by using containers

------
burgerdev
The acronym means "Yes: another repo's name!".

------
jshawl
>We decided to zip the entire node_modules folder and upload it to an internal
CDN so that both engineers and our continuous integration systems could
download and extract the files consistently.

why not mirror npm modules there then?!

------
rbanffy
Yet Another... What does the R and the N stand for?

~~~
__raccoons__
Yet Another Redundant NPM

~~~
Gigablah
Redundancy for package registries? Sounds like a good thing to me ;)

~~~
rbanffy
Redundancy needs consistency. Not sure how that will be done.

------
jfmercer
Dear Facebook,

Thank you, thank you, thank you, thank you, thank you.

------
romanovcode
Doesn't work for me. My gulp builds are failing because they are missing some
dependencies.

I'll continue using npm thank you very much.

------
b123400
So many package managers, we need a package manager for that!

------
z3t4

      console.log("u got powned")

------
ChoHag
You're going to need these: "nother"

------
vacri
How is checksumming 'mega-secure'? (looking at
[https://yarnpkg.com/](https://yarnpkg.com/))

------
andrewvijay
The 'attempts at scaling npm client' section is incredibly cringy even to
read. Those are real complex problems that every one who works on a massive
scale will face at some point in time.

~~~
cpojer
Can you make yarn out of hair?

------
klosnet
awesome

------
karliky
[https://twitter.com/k4rliky/status/785875228710858753](https://twitter.com/k4rliky/status/785875228710858753)
please no

------
koolba
> Finally, updating a single dependency with npm also updates many unrelated
> ones based on semantic versioning rules. This makes every change much larger
> than anticipated, and having to do things like committing node_modules or
> uploading it to a CDN made the process less than ideal for engineers.

You shouldn't check in node_modules. You should use a shrinkwrap file to
indicating the specific sub-dependency versions you're using.

~~~
epmatsw
They address pretty throroughly why that does not work for them in the linked
article.

~~~
koolba
> They address pretty throroughly why that does not work for them in the
> linked article.

Not really. They just mention it in passing:

From the article

>> Shrinkwrap files aren't generated by default and will fall out of sync if
engineers forget to generate them, so we wrote a tool to verify that the
contents of the shrinkwrap file matches what's in node_modules. These files
are huge JSON blobs with unsorted keys, though, so changes to them would
generate massive, difficult-to-review commits. To mitigate this, we needed to
add an additional script to sort all the entries.

Comparing node_modules to shrinkwrap isn't necessary. Building the app will
re-generate node_modules and tests should catch whether or not the app works
as intended.

The sorting issue for shrinkwrap is legit, but the answer to that is sort it!
Sure you'll get some deep diffs if the dependencies change, but that reflects
the real changes to the app.

The simpler answer seems to be to add a build test to mandate the existence of
a shrinkwrap file, verify it's sorted, and (optionally) verify it's up to
date. The latter doesn't even have to be done every build.

