
Yarn's Future – v2 and beyond - arcatek
https://github.com/yarnpkg/yarn/issues/6953
======
scrollaway
Very happy to see yarn.lock will finally be a proper format that won't need
its own parser. YAML subset is a pretty good choice, though I think
canonicalized, indented JSON would be a better choice for this use case.
Incidentally that's what npm uses as lockfile, I wonder if there's room to
have the two package managers share the format (or even share the file
itself).

Very excited to see shell compatibility guarantee in scripts as well. Using
environment variables in scripts is a pain right now.

Finally one of the biggest news is the switch from Flow to Typescript. I think
it's now clear that Facebook is admitting defeat with Flow; it brought a lot
of good in the scene but Typescript is a lot more popular and gets overall
much better support. Uniting the JS ecosystem around Typescript will be such a
big deal.

~~~
wopian
npm's lockfile is a pain to diff in PRs because of the JSON format where what
was maybe 20 changed lines in yarn is upwards of 80 from the brackets.

With YAML and whatever format yarn.lock was in, the only changed lines are
changes to the version resolutions, hash and dependencies.

~~~
donatj
I'd say safely merging YAML diffs however could be trouble.

I don't know how restricted their YAML subset is, but in my experience it's so
loose a format the only way to be sure YAML says what you think it says is to
run it through a parser.

~~~
DuskStar
I think if you're merging lockfile diffs, you're doing something wrong! Merge
the package.json diffs and regenerate the lockfile.

~~~
tlrobinson
Yarn automatically resolves conflicts in yarn.lock if you run "yarn":
[https://github.com/yarnpkg/yarn/pull/3544](https://github.com/yarnpkg/yarn/pull/3544)

------
misiti3780
> The codebase will be ported from Flow to TypeScript. To understand the
> rational please continue reading, but a quick summary is that we hope this
> will help our community ramp up on Yarn, and will help you build awesome new
> features on top of it.

Another major project moving from flow to typescript

~~~
tonyhb
This sucks because, in the first few years, flow had much better soundness and
typescript had some serious issues. I'm a little disappointed and feel as
though, similar with Kube vs Swarmkit, the worse technology is winning.

~~~
4di
I worked with Typescript for 2 years, then worked with Flow for 2 years and
now I'm back on Typescript working for a big company.

I agree that Flow had better capabilities around soundness. But the tooling
around Typescript really made me jealous, specifically in VSCode.

Near the end of working with Flow, Typescript was getting some cool
capabilities like refactoring JSX for React apps.

Nowadays I can easily say that Typescript is a far better experience than
Flow. There are updates every two months, adding some neat features that you
might find in other languages.

Then you add in the power of surrounding tools and _their_ ecosystems like
TSLint and it really feels like a next-level coding experience where the tools
start writing the mundane code for you, driven the core TS static analysis.

~~~
wafflesraccoon
I'd agree with everything you've said. Flow isn't bad but after working with
TypeScript I could never go back. You're spot on with the tooling and
ecosystem.

~~~
ggregoire
Also, TypeScript has a roadmap and when you report a bug, the devs actually
reply. Flow doesn't have any of that.

------
CGamesPlay
> The log system will be overhauled - one thing in particular we'll take from
> Typescript are diagnostic error codes. Each error, warning, and sometimes
> notice will be given a unique code that will be documented - with
> explanations to help you understand how to unblock yourself.

Why do programmers love error codes? As an end user, they are useless
indirection to me and the only way this would even be tolerable is if the
explanation was printed directly next to the error code, so why bother? Is it
code quality, since you don't have to write a long error message when you emit
a similar error? But doesn't that have the drawback of encouraging error code
reuse when it might not be appropriate?

[append]

Thanks for the replies! I guess I could understand error codes for search
ability that also provide information about the problem specifics.

Counterexample:

    
    
      $ yarn add foo
      Error YARN1001: Incompatible peerDependencies.
      $ yarn explain YARN1001
      # Some longer text about how two of my modules have
      # incompatible peerDependencies
    

Better example:

    
    
      $ yarn add foo
      Error Yarn1001: Incompatible peerDependencies.
      * my-package@1.0.0
      |-* foo@1.0.0
      |-* bar@1.0.0
      |-* left-pad@1.0.1 (peerDependency of bar@1.0.0)
      |-* left-pad@0.9.0 (peerDependency of foo@1.0.0)
      $ yarn explain YARN1001
      # Some longer text about how two of my modules have
      # incompatible peerDependencies

~~~
lost_my_pwd
Humans are not always the consumer of error output. Distinct error codes
simplify parsing and eliminate ambiguity.

It also makes it easier for developers look up a specific error in the
documentation, assuming it's been documented.

~~~
talltimtom
How does an error code make it easier to look up an error? The workflow is
either “get error, google it”, or “get errorcode google it”. In both cases the
docs will be the top hit.

If we where talking 20 years ago I might agree, but I really can’t see the
argument with todays tooling.

~~~
simion314
Updates can improve on the error messages, making them more clear, the error
code does not change but the message changes.

Also many times the error message is interpolated with your specific details
like

"Syntax error at line 123 in file /home/user/myfile , symbol X is not allowed
here" this is a silly example but often enough when i google this errors I
have to first strip out my data from them.

------
Aeolun
I’m glad that Yarn will continue. Npm has improved, but it’s still less
pleasant to work with than yarn (which basically always does what I expect,
not so for npm).

~~~
h1d
Which part of npm acts unexpectedly?

~~~
dhritzkiv
Not that this is a showstopper, but I have issues with the package-lock.json
file where the `resolved` field (the package's registry URL) constantly flip
flops between http and https protocols, depending on which machine I'm on
(home, work, or docker container), whenever I run `npm install`. Sounds not so
bad, but it becomes a mess in git, and causes any docker build caches to
become invalidated.

~~~
scrollaway
That sounds pretty bad and I'm not so sure it's a npm bug. Do you have a diff
of the change in question? Is it on the npm registry or a custom one?

~~~
dhritzkiv
It's on the npm registry, affecting the npm client:
[https://npm.community/t/some-packages-have-dist-tarball-
as-h...](https://npm.community/t/some-packages-have-dist-tarball-as-http-and-
not-https/285/51)

------
yaseer
The JS ecosystem has its flaws, but one has to appreciate the speed at which
momentum shifts, making clear winners obvious.

The move towards TypeScript 'winning' has been fast, and to everyone's
benefit.

~~~
wanted2
> , and to everyone's benefit.

Why? I've worked on large codebases in Coffeescript, ES6 and Typescript.
Whatever this whole community sings and believes, but Coffeescript still wins
for me. ES6 is still trying to catch up but will probably never reach the
beauty and ease of Coffeescript. Both are transpilers, only ES6 with Babel is
a total horror to manage (just upgraded a large codebase to Babel 7..).

Typescript takes about 2x the time to write if you want to create all your
typings properly. I hear you say; only in the beginning, later it will speed
up the development process. I've never seen that in reality! I've actually
never seen a proper codebase in Typescript. Show me a Typescript codebase not
using the type 'any'! In a decent system language you can't get away with
that, it's just a fake sense of security.

A good codebase should not be dependant at all by Typescript or whatever hype
comes next. Writing a good codebase is IMHO a craft and should not depend on
the language or a bunch of tooling. If Typescript is way to go, what about
Python, Ruby, abandon it, deprecated? Are those inferior languages compared to
Typescript? Typescript is just another hype, very smart play by Microsoft btw.

~~~
kjaer
> Show me a Typescript codebase not using the type 'any'! In a decent system
> language you can't get away with that, it's just a fake sense of security.

I don't know the comparison to system languages is fair though, because the
use-case for JS is quite different from system languages.

Javascript (and by extension, Typescript) is commonly used to interface
between the user and the network, both of which often are outside the bounds
of the type system. Add to that any code that interfaces with plain JS, such
as external libraries or legacy code. When dealing with those, it's natural to
use statically untyped values and type ascriptions based on reasonable
assumptions.

Taking that into account, I actually think Typescript's type system is fairly
well-designed for the use-case. The problem isn't really with Typescript, it's
just intrinsic to the use-case of JS.

~~~
s_tec
Exactly. Even strongly-typed languages have this problem. In C, it looks like
`void *`, while in Java, it's `Object`. The `any` keyword is just the latest
in a long line of escape hatches. Pretty much every language has one.

~~~
dragonwriter
> Even strongly-typed languages have this problem. In C

C is clearly statically typed, but among statically typed language it is quite
weakly typed.

------
symlinkk
I've got to say I'm not a big fan of some of these changes and I think they're
biting off more than they can chew.

> Writing posix command lines inside your scripts field will work regardless
> of the underlying operating system. This is because Berry will ship with a
> portable posix-like light shell that'll be used by default.

> Scripts will be able to put their arguments anywhere in the command-line
> (and repeat them if needed) using $@. Similarly, scripts will have access to
> $1, $2, etc.

If you use either of these features your package.json will no longer work with
NPM. Maybe they should call it yarn-package.json?

> Starting from Berry, we made it an explicit goal that each component of our
> pipeline can be switched to adapt to different install targets. In a way,
> Yarn will now be a package manager platform as much as a package manager. If
> you're interested into implementing PHP, Python, Ruby package installers
> without ever leaving Yarn, please open an issue and we'll help you get
> started!

Noooo, god no. Package management is a gargantuan, complicated task, and these
languages all have their own solutions already.

That being said, it's cool that they're rewriting it in TypeScript.

~~~
maw
I don't see the need for PHP, because composer is one of the few things I like
about it. And I don't know enough about Ruby to have an opinion there.

But something for Python that actually works? Yes, please!

~~~
nhanb
For python we've been using poetry[1] at work. Dep resolution is a bit slow
but otherwise it works well enough, definitely a saner choice than pulling in
a completely different stack just for one tool.

[1]: [https://poetry.eustace.io/](https://poetry.eustace.io/)

------
sebazzz
There appears to be a real movement to move from Flow to typescript. Is Flow
dying?

~~~
mrspeaker
Does anyone know of any "third choice" around typing JavaScript? I would love
to add types to my code, but I want to write "real" JavaScript: so the code I
input is the code that is executed by the browser. I just want the compile
step to strip away the type annotations.

There was initially talk of Flow using comments to actually work without
touching the source code at all, but I don't think anything came of that... is
there anything else on the horizon?

[Edit: Nevermind: I went looking for the github issue about adding types as
comments, and it turns out it's already supported by flow:
[https://flow.org/en/docs/types/comments/](https://flow.org/en/docs/types/comments/)
\- is there anything like this for TypeScript?]

~~~
_hardwaregeek
TypeScript works with JSDoc annotations:
[https://github.com/Microsoft/TypeScript/wiki/JsDoc-
support-i...](https://github.com/Microsoft/TypeScript/wiki/JsDoc-support-in-
JavaScript)

~~~
sephoric
Can confirm, have sometimes added /* * @type Foo */ above a type and VS Code
assumed that type was Foo everywhere else in my project, as if I typed `foo:
Foo` directly. Didn't even have TypeScript in the project itself, purely an
IDE feature. Extremely well thought out and useful ecosystem.

------
Vinnl
> your previous yarn.lock will be silently migrated

I hope that's not _that_ silent, because at that point, everybody who works on
that project will have to upgrade as well.

That said, shipping the light-weight POSIX-like shell will make it a lot
easier for scripts to be multiplatform. That's the improvement I'm looking
forward to most.

~~~
arcatek
We'll make sure to add a notice at runtime to make it clear (plus, a
consequent diff at review).

Also note that we recommend using `yarn policies set-version` to enforce the
version of Yarn used by everyone in your team with very little friction:

[https://yarnpkg.com/en/docs/cli/policies#toc-policies-set-
ve...](https://yarnpkg.com/en/docs/cli/policies#toc-policies-set-version)

~~~
Vinnl
There's a feature I didn't know about - that's enormously useful, thanks! Now
to remember applying that to all my different repo's...

------
mydpy
For the uninitiated / confused, this refers to yarnpkg, the JavaScript
dependency manager, not (Hadoop) YARN, the cluster manager.

------
talkingtab
I'm curious as to why yarn instead of contributing to NPM? I am aware that
yarn was the inspiration for many improvements for NPM by providing an
alternative, but going forward do we need two systems? Is the plan for yarn to
be compatible with NPM and package.json?

~~~
spricket
From what I've heard, the Yarn codebase is much cleaner than NPM. If anything,
I think it would make more sense to migrate NPM over to yarn

------
jcolella
The addition of vulnerability scanning was the only reason our company
switched back to npm from yarn. Other than that, yarn offers a great
experience

~~~
symlinkk
Yarn has this too (although it uses the NPM audit database): `yarn audit`.

~~~
coolreader18
Oh, I didn't know that! Here's some resources about it if you haven't heard of
it either:

documentation;
[https://yarnpkg.com/lang/en/docs/cli/audit/](https://yarnpkg.com/lang/en/docs/cli/audit/)

original feature issue:
[https://github.com/yarnpkg/yarn/issues/5808](https://github.com/yarnpkg/yarn/issues/5808)

release comment in that issue:
[https://github.com/yarnpkg/yarn/issues/5808#issuecomment-441...](https://github.com/yarnpkg/yarn/issues/5808#issuecomment-441989817)

------
mderazon
> Writing posix command lines inside your scripts field will work regardless
> of the underlying operating system

That is very nice. No need to install other dependencies just to do 'rm -rf'

~~~
arcatek
Note that this is mostly about the command line syntax, not so much the
commands themselves which will be executed just like now.

That being said, maybe we'll offer some builtin as well (possibly in a similar
way to what CMake offers?[1]). That would be worth an RFC later on :)

[1]
[https://cmake.org/cmake/help/v3.2/manual/cmake.1.html#comman...](https://cmake.org/cmake/help/v3.2/manual/cmake.1.html#command-
line-tool-mode)

------
jclay
Since it seems the devs are here answering questions:

Which lightweight shell will be used on Windows? Does it also bundle standard
unix tools (if a script pipes to grep or less for example)?

How will paths be translated on Windows? I’ve attempted something similar
recently and had to do a fair amount of regex magic + using cygwins built in
path translation utility to preprocess commands. Curious to see if there’s a
better way to solve that.

~~~
arcatek
> Which lightweight shell will be used on Windows? Does it also bundle
> standard unix tools (if a script pipes to grep or less for example)?

It will be in-house, and very basic. We don't intend to rewrite bash, just to
provide the basic experience that is usually needed when adding script into
the `scripts` field. For more complex needs we'll simply offer a way to opt-
out and use the native shell, or to call Node scripts.

> How will paths be translated on Windows?

The current Yarn tries to do this by using the `path` native module. It's
quite error-prone since backslashes tend to appear in the worst possible
places. For the v2 I plan to work with all paths in a posix style, and convert
them into Windows paths right before they reach the filesystem (which is
similar to what Cygwin does, as you mentioned). It would be a bit slower on
Windows, but massively simpler in the codebase.

~~~
exogen
I'm assuming that lifecycle scripts (and scripts called by lifecycle scripts)
in particular will still need to use common Windows-supported syntax? Even if
the devs of the package are guaranteed to be using Yarn, people _installing_
the package might still be using npm. So I assume some caveats apply to some
scripts, right?

p.s. I love Yarn :)

~~~
arcatek
The `postinstall` scripts would likely be better off without using those
features, indeed. But in the end, your packages would be better off without
`postinstall` scripts anyway ;)

~~~
exogen
Yeah I don't recommend `postinstall`, but `prepare` (and all the potential
build-type scripts it could run) is actually useful (esp. for allowing people
to install from unpublished versions via git), and would need special
consideration. :(

------
spullara
If you use the yarn cli and have tried the npm cli recently, why do you still
use yarn? Are there big gaps that you find that NPM has failed to close?

~~~
exogen
Unfortunately npm is probably the biggest source of my daily development
frustrations, even on the latest version.

I still come across bugs (that are definitely in npm itself) that have been
around absolutely forever, like:

> npm ERR! cb() never called!

It sometimes gets its primary purpose, dependency resolution, wrong. I'll give
it a perfectly reasonable package.json to install, which it will do, and then
`npm ls` will still error with "missing dependency!" in some package. This
should not be possible.

Related, it will put packages from the flattened tree in the wrong place. I
can have a dependency (that other dependencies need, specified in their
peerDependencies) specified in my top-level package.json and bafflingly, npm
will still move it from top-level node_modules into the node_modules of
something else that happens to use it, breaking the peerDependencies I was
trying to satisfy.

On the install process: even if the total time to install is roughly on par
with Yarn, I find that Yarn is much smoother. Whatever they're doing, they're
yielding the CPU a lot more, and the result is I can actually work while it
installs. npm meanwhile doesn't yield much during install and slows the whole
system to a crawl.

Some npm commands are extremely neglected. The "success" message printed by
one of the user/permissions related commands is simply: {}

Lastly, I will leave this terrifying comment here:
[https://github.com/npm/npm/issues/16528#issuecomment-3075400...](https://github.com/npm/npm/issues/16528#issuecomment-307540010)
– note this comment was left 2–3 years after receiving $10M in funding.

I don't blame them for being one-upped by Yarn at every turn, but they still
fail to get the basics right, let alone innovate on things.

------
juancampa
If package.json is already in JSON format, why not use the same for yarn.lock?
Honest question, there must be a good reason.

~~~
arcatek
We want the lockfile to be easy to review by humans. In our experience, JSON
doesn't quite fit the bill once you reach a critical mass of data.

Our lockfile format worked fine for the past three years (bar the unfortunate
YAML incompatibilities that we're about to fix). Don't fix what isn't broken
:)

------
misiti3780
serious question, if you are starting a new project today why would you choose
to use npm over yarn?

~~~
azangru
Because npm is a standard (default) package management tool that comes with
node.

After getting `package-lock.json` and `npm ci`, I would rather wonder why
choose yarn instead of npm.

~~~
MehdiHK
"npm ci" deletes node_modules directory every time. Try using that when you
depend on some native modules. :(

~~~
azangru
I thought `npm ci` is more suited for ci machines, where it's crucial that
packages are pinned to specific versions. For local development, I just npm
install. Perhaps with no-save option in order to avoid updating package-lock.
Works fine.

------
towaway1138
Yarn has over 1500 open bugs. Rather than working on changes, it'd be nice to
stop and address these.

~~~
arcatek
Most of those have been fixed a long time ago, but we simply don't have the
resources to triage them.

This effort we start is in no small part to decrease the number of issues that
will be created by empowering the users to unblock themselves* and solidifying
Yarn's codebase.

* You wouldn't believe the number of issues that are simply about things working as they should - we can't really blame their authors because it can be quite hard to find the right paragraph in the documentation, but it's extremely taxing on a small team. Similarly, we often have issues created against older releases, or without reproducible test case.

~~~
towaway1138
I noticed the issue count while using yarn for the first time yesterday. It
encountered a fatal error (Disk Quota Exceeded) and then proceeded blithely
on. It's only one data point, but doesn't inspire confidence.

