
Node Modules at War: Why CommonJS and ES Modules Can’t Get Along - dfabulich
https://redfin.engineering/node-modules-at-war-why-commonjs-and-es-modules-cant-get-along-9617135eeca1
======
rjeli
<quote>

I’ll conclude with three guidelines for library authors to follow:

Provide a CJS version of your library

Provide a thin ESM wrapper for your CJS

Add an exports map to your package.json

</quote>

This seems like the same fallacy of python’s 2to3, and why they should have
written 3to2 instead. If ESM is the new way, library authors should instead
have a CJS shim to prevent fossilization of cruft.

~~~
kqr
> This seems like the same fallacy of python’s 2to3, and why they should have
> written 3to2 instead.

This is incredibly insightful. I'm frankly a bit stunned. I would never have
thought of this (and clearly the Python people also didn't) but it is
absolutely true. It's also a very transferable strategy for any time
compatibility is to be broken.

Is this an original thought of yours or do you have references to further
reading?

~~~
rjeli
It's a common refrain when the py2/3 discussions get going.
[https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...](https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=3to2&sort=byPopularity&type=comment)

------
indeyets
Sorry about nitpicking, but I feel that we shouldn't say that CJS has named
exports. It does not.

CJS has a single export which might be an object. And objects have named
properties/fields.

On the other hand, ESM CAN have unnamed "default" export and CAN have
arbitrary number of named exports.

~~~
DaiPlusPlus
> On the other hand, ESM CAN have unnamed "default" export and CAN have
> arbitrary number of named exports.

And this is a constant source of confusion and frustration for me when working
with so many npm modules lately...

------
tlarkworthy
Wow that was informative. So ES Modules is really about the Red functions [1]
getting module support.

I find it fascinating watching our understanding of async computation mature
over the years.

[1] [http://journal.stuffwithstuff.com/2015/02/01/what-color-
is-y...](http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-
function//)

~~~
wruza
The last time I asked here why async-await is even a thing, when you can just
introduce light/green threads, coroutines, you name it, everyone jumped that
it is _even better_ when you mark awaiting points explicitly. Now that it goes
from a blog post of some popular language designer, seems no one gonna
disagree.

>async computation mature over the years

A good thing, really, but would be much better if the experience from few
decades ago wasn't ignored by cool kids who mature.

~~~
gridlockd
The reason why async/await is a good fit for Javascript is that it neatly
wraps up asynchronous callback hell and Promises, which were already
ubiquitous in Javascript code, because the web runtime enforces non-blocking
IO.

~~~
wruza
It may be the reason, but not the only option.

    
    
      function cpsfoo(cb) {...}
      function foo() {
        var thread = this_thread()
        cpsfoo(x => thread.resume(x))
        return thread.yield()
      }
    

Error and immediate cb invokation handling omitted for clarity. Similar
wrapper or wrapper-generator (uncps(f), unpromise(f)) may be done for other
primitives.

~~~
cookiengineer
> Error and immediate cb invokation handling omitted for clarity.

... but this is exactly the reason why callback hell existed. Error handling
down the callback branches were a total mess, especially with networked state
transfers.

~~~
wruza
Error handling can be easily added to the example wrapper above, completely
hiding all the burden from a user of foo(), who then could do their usual try-
catch. It's just my phone keyboard that prevents me from implementing it right
here.

------
spankalee
In two-and-a-half months the last 5 stable releases on Node, the last 2 LTS
releases of Node, and all current major browsers will support standard JS
modules.

It's time we all published standard JS modules and only standard JS modules to
npm.

~~~
dfabulich
> the last 2 LTS releases of Node

(Author here.)

Node 14 supports ESM, but Node 12 only supports ESM behind the --experimental-
modules flag, and its error handling for incorrectly importing CJS is not
good.

In 2020, I think it's a bit early to go ESM only, especially since it's
straightforward for CJS libraries to support both CJS and ESM clients. (I
document how to do this in the article.)

~~~
colinr
The experimental flag was recently removed in v12.17.0 (though the release
notes still say ESM is an experimental feature)

[https://github.com/nodejs/node/releases/tag/v12.17.0](https://github.com/nodejs/node/releases/tag/v12.17.0)

------
e12e
Meanwhile, I found this on how Deno handles modules (it's all esm):

[https://www.sitepoint.com/deno-module-system-a-beginners-
gui...](https://www.sitepoint.com/deno-module-system-a-beginners-guide/)

Note that - if you can get an esm out of your cjs - and it doesn't use any
nodejs apis unsupported by deno, then you might be able to use it:

> (...) Several CDNs can convert npm/CommonJS packages to ES2015 module URLs,
> including: Skypack.dev, jspm.org, unpkg.com (add a ?module querystring to a
> URL)

Ed: submitted as:
[https://news.ycombinator.com/item?id=24069037](https://news.ycombinator.com/item?id=24069037)
since I couldn't find an earlier submission, and maybe it'll generate some
interesting discussion on deno / modern js vs node.

In particular I wonder if it'll in practice be easier to share (more) code
between backend and front-end.

~~~
e12e
There's also a crutch - I'm not sure how attractive it would be in a project
using deno, though. Maybe to get up and running quickly on deno when moving a
legacy project from nodejs?

[https://deno.land/std@0.63.0/node#commonjs-module-
loading](https://deno.land/std@0.63.0/node#commonjs-module-loading)

------
cel1ne
JavaScript just gives to much options. I keep having problems with libraries
because ES6 allows

    
    
      import x from "library";
      import {x} from "library";
      import * as x from "library";
      import {x as y} from "library";
    

Just to compare Java uses this:

    
    
      import java.lang.Double;
      import java.lang.*;
    

No renaming, not a gazillion options during in- and export, but one statement.
If you want to rename something, assign it to a variable.

I don't understand why ES-module authors made it that complicated, when they
had the chance to introduce a proper module system.

Why did they not just copy Java's system and be done with it.

~~~
greggman3
Java's packaging system sucks as does all the similar ones like C#, C++ etc
compared to JavaScript's IMO.

All of those the space of namespaces is global. You can't have 2 modules named
math.quaterinions in any of those languages. You can in JavaScript. To put it
another way, in order to prevent name clashes in C++/C#/Java you have to know
the name of every other project on the planet. If someone uses the same
namespace then if you ever need to use those 2 projects together one of them
will have to be re-factored.

That problem doesn't exist in modern JavaScript. There are no namespaces.
There is just floating references to code.

~~~
smt88
I honestly have no idea where you got the idea that namespace conflicts are a
problem in C# or Java. Both have solved this problem (C# with aliasing and
Java with packages).

~~~
greggman3
aliases help you use bar from foo.bar and moo.bar. They don't help you with 2
files both of which declare foo.bar

~~~
manigandham
Why would you declare the same type in 2 different files in the same project?

~~~
greggman3
you don't have control of all 3rd party libraries.

~~~
manigandham
Then it's not the same project and the fully qualified type name will be
different.

------
onion2k
Rollup makes building both CJS and ES versions of a library fairly
straightforward.
[https://rollupjs.org/guide/en/#outputformat](https://rollupjs.org/guide/en/#outputformat)

~~~
desmap
Just tinkering with Rollup: How can I let Rollup work with a pnpm monorepo, so
that it's resolving local packages declared via pnpm's pnpm-workspace.yaml?

------
no_wizard
I use the ESM dependency to load ESM modules as a CJS shim:

[https://www.npmjs.com/package/esm](https://www.npmjs.com/package/esm)

I highly recommend it. Really let’s you unify on ESM modules without much
thought

~~~
jcheng
For those who, like me, wondered how this is possible without hitting the top-
level await obstacle mentioned in the article, the options doc includes this
line (emphasis mine):

> "await":false - A boolean for top-level await in modules _without ESM
> exports_. (Node 10+)

So, yeah, seems to rely on "most ESM modules don't use top-level await". Seems
like a useful option to have, thanks!

------
Humphrey
Wow, I've been coding JS/Typescript frontends for years and somehow completely
missed out knowing anything about CJS & ESM or their differences.

I guess it's all irrelevant when you don't use Node, aren't publishing
libraries, and are compiling all JavaScript into bundles for deployment and
never read the bundled code.

~~~
beart
Unless you are developing libraries which will be published to the nodejs
ecosystem, you really don't need to worry about this.

You pick one of these when you first create your app which is typically based
on whatever framework or template you are starting from, and you just always
stick with that.

~~~
recursive
Also, typescript can compile `import`s into `require()`s.

------
maple3142
I think CJS can export 1 more thing than ESM is a problem too.

    
    
        // cjs.js
        module.exports = (a, b) => a + b
        exports.default = 'default'
        exports.nice = 69
        // esm.js
        import * as mod from './cjs'
        // what is `mod` now?
    

In babel, `mod` will be `{ default: [Function], nice: 69 }`, and the string
'default' will be discarded because ESM always exports a `{ [key: string]: any
}` while CJS can export whatevey they want (`any`).

------
phpnode
I really feel like default exports were a mistake, everything is simpler
without them, they don't even work with things like `export * from './foo'`

------
fireattack
Another issue about module import I encountered before and wasted me 1 hour to
find the culprit:

When converting TypeScript that using `import default xxx` to JS using `tsc`,
it would convert it to something like `exports.default = xxx`.

To import it, you have to use `const XXX = require('./mo.js').default` instead
of just `const XXX = require('./mo.js')` in plain ES6 JS.

This is fine but Babel _used to_ [1] be able to convert `const XXX =
require('./mo.js')` to valid code, which made lots of people wrote it
(incorrectly) that way (I was very baffled why people wrote like that since it
does not work in plain ES6).

[1]
[https://github.com/babel/babel/issues/2212](https://github.com/babel/babel/issues/2212)

~~~
desmap
But how do you solve it in a tsc context?

Using the default prop doesn't work in tsc and breaks all type information
because tsc just doesn't know of any default prop.

~~~
fireattack
Not exactly sure what you mean but you can check this:
[https://github.com/microsoft/TypeScript/issues/2719](https://github.com/microsoft/TypeScript/issues/2719)

~~~
desmap
Thanks for the link. I meant following case:

A TS file which compiles to an es6 ("esnext") module cannot import cjs. Either
node breaks or TS' type-checking fails. Former if you omit default when
importing and latter if you introduce .default in the TS codebase.

It's a bit complicated, what I just wrote about is reflected in the chart in
step 4-6 here
[https://github.com/microsoft/TypeScript/issues/18442#issueco...](https://github.com/microsoft/TypeScript/issues/18442#issuecomment-666190219)

------
kohtatsu
I spent half an hour yesterday banging my head against the wall on exactly
this.

Thank you.

------
EdwardDiego
Holy crap, this redefines dependency hell.

~~~
quickthrower2
Gimme COM+ back! I’m sorry I hated you...

------
jokethrowaway
While I understand why it was done, I feel ESM vs CJS Will be the breaking
point for node.js. All my clients are using Babel to avoid this single issue
but it's slow enough to be a DX problem.

I wonder if deno (maybe with a tsc reimplementation written in rust) will take
its place for new projects. Personally, I've already suffered enough with
python 2/3 and decided, after 9 years writing node, to use uniquely rust for
new backend development.

Frontend development is quite a slow process as well, but all the faster
alternatives imply knowing a different language - which impacts developers
availability (given historically frontend developers are mostly JS).

------
jorams
It's always surprising to read about the trouble this causes for Node. I've
been writing mostly Webpack-compiled JS for the past few years, and it seems
to allow any random mixture of require and import. Have I just been avoiding
trouble areas by chance?

~~~
jakub_g
Webpack + babel are basically rewriting `import` and `require` calls into
their own `__webpack_require__` helper and converting ESM to CJS. Whereas
`require` and `import` are node built-ins and `import` is browser built-in.
Making stuff interoperate requires some trickery and lax spec compliance.

There are high chances you are relying on some non-spec compliant behaviors
but you don't know about it. For example, ESM exports have immutable bindings,
which means, mocking exports for unit testing is impossible. If it works for
you, it's because you transpile the code from ESM to CJS (either via `babel`,
`rewire` or similar tool).

------
foreigner
The description of how require() works is not quite correct. Node does not
always fully initialize the dependency module before returning from require().
It can't work that way, because Node allows circular require() dependencies.
Try it!

~~~
dsun179
The reason is not correct. Nodejs cache the require lot and so circular
dependencies can work without returning half finished loads.

~~~
foreigner
Nodejs does cache the results of require(), but that doesn't help with this
problem. Example:
[https://repl.it/repls/SwiftOlivedrabBrace](https://repl.it/repls/SwiftOlivedrabBrace)

------
jariel
This is an important issue to consider because as we are witnesses a Cambrian
Explosion in 'new tech' (and by the way, in 'new' we can include evolutions of
'current') ... we need to grasp that 'a new piece' is is just going to be 'one
of many pieces'.

My first instinct these days is how that funky new bit of Lego fits with all
the other bits of Lego.

When you start there, it's becomes easy to understand just how incumbent C++
and javascript are, even with all their necessary spells and arcane
witchcraft.

------
wildpeaks
Honestly, all I need is a standardized way to publish and consume packages
directly as Typescript (right now you can point "main" & "types to it, but I'd
rather something like export maps) because the transpilation should be done by
the end user as you can't assume which build target they might prefer.

~~~
Androider
Typescript transpilation is already pretty slow when just used for your own
application, if all the node_modules directory entries were in TS I'd probably
be looking at hours for the transpilation.

~~~
wildpeaks
I removed Babel from my toolchains a while ago (and instead use TS to
transpile even dependencies), so the compilation times are fine.

------
desmap
fwiw, TypeScript and ES Modules are just not compatible and never will be:
[https://github.com/microsoft/TypeScript/issues/18442#issueco...](https://github.com/microsoft/TypeScript/issues/18442#issuecomment-666190219)

~~~
WorldMaker
That chart shows that Typescript is incompatible with CJS modules, because it
inherits ES Modules incompatibilities with CJS, not that it it is incompatible
with ES Modules.

------
aPoCoMiLogin
You can use ESM (including dynamic import) in node 6+ (without babel) by using
[https://github.com/standard-things/esm](https://github.com/standard-
things/esm) and cut the `.mjs` and "it can't work like that" crap.

~~~
jkrems
You can definitely continue to use transpilers like esm or babel to continue
using CommonJS behind the scenes while writing module syntax. But it's not
quite the same as using native modules, as implemented by JavaScript engines.

------
djohnston
it seems so insane that there are competing module patterns in a single
language, but perhaps i'm ignorant. are there other languages with such
incompatibilities on a fundamental component?

~~~
WorldMaker
Lisp is the easiest classic example that took ages to settle on a module
standard and has had different competing standards over the years.

PHP and Perl are probably more recent examples comparable to JS here that
provided mostly only low level plumbing (include/require) support and nearly
wound up with multiple incompatible module loading patterns.

Even C/C++ the "module" system started as a text processing hack in a macro
pre-processor that used to be a separate tool (though I don't think any modern
C/C++ toolchain uses a separate macro pre-processor, as there are tons of
optimization layers now and some of the macro pre-processor knowledge makes
its way all the way through to the linker these days).

Historically, modules were always just "smash these two files together in this
order" and it was only later that languages started asking hard questions such
as "which symbols from this other file are available".

The modern concept of modules as a fixed "shape" of self-describing public
symbols is as much a later abstraction out of Object-Oriented Programming as
anything else. Our idea of what a module should be in 2020 is much more
informed by "recent" notions such as Component systems like COM and languages
such as Java and C#.

~~~
djohnston
very cool thanks for the context!

------
ravenstine
Is this really such a big deal? Using ESM with imported CJS packages has been
pretty straight forward for me. It's hardly the worst thing about Node.js
development.

------
rk06
If library authors make esm as a thin wrapper over cjs, won't it hinder tree
shaking.

And for tools that want end to end esm, like vite. Won't it cause degradation
in behaviour?

~~~
yabberdabbafoo
> If library authors make esm as a thin wrapper over cjs, won't it hinder tree
> shaking.

Correct. I wouldn't follow the article's advice wrapping ESM with CJS as it
leads to larger bundles.

> Won't it cause degradation in behaviour?

For web apps - certainly. The forced sync behavior of CJS wouldn't be very
noticeable for local applications however.

------
the_gipsy
And the article is not even talking about default imports on transpiling with
tsc or babel (or both!). Too many combinations.

~~~
desmap
Oh yes, especially with tsc (it's just not working), see my other post here
with a Github issue link

------
lf-non
Not taking seamless CJS interop into account was IMHO the biggest failure of
TC39 till date.

Given the widespread popularity of CJS they should have prioritized CJS
backward compatibility over things like better support for circular
dependencies and mutable exports. Many languages are doing quite fine without
them.

------
praestigiare
npm install esm

Then everything just works. Top level await and all.

~~~
wonderlg
Wait till you throw a package.json with "type": "module" in there.

Everything used to work until Node 12.15, then 12.16 started throwing when
trying to require type:module packages with `require` (which is what `esm`
does)

[https://github.com/standard-
things/esm/issues/868](https://github.com/standard-things/esm/issues/868)

------
draw_down
I haven’t seen CommonJS modules for years now, only ES modules. Didn’t realize
this was such an issue still.

> You can’t require() ESM scripts

That’s one way to put it! Another would be to say that you don’t have to.
require() came about because we didn’t have the import keyword.

~~~
dfabulich
(Author here.)

You were not using native ESM in Node years ago. Node 14 shipped with
unflagged ESM support just this year. (If I had to take a guess, you were
probably transpiling "import" statements into "require" statements and/or
bundles, which honestly works better than native ESM, because there's no
interop issue.)

Almost all of the most depended on NPM packages are CJS, and provide no ESM
exports.
[https://www.npmjs.com/browse/depended](https://www.npmjs.com/browse/depended)

lodash, react, chalk, commander, moment, express, axios, and vue are all CJS.

------
austincheney
A major benefit of node modules using ESM, at the moment, is that they aren’t
swimming in a swamp of NPM packages.

------
iso8859-1
Enough with the hornet hate. They are great animals. If you know German, you
can learn how great they are in the knowledge podcast of the Bavarian
Broadcast: [https://www.br.de/mediathek/podcast/radiowissen/die-wespe-
un...](https://www.br.de/mediathek/podcast/radiowissen/die-wespe-ungeliebtes-
insekt-mit-grossem-nutzfaktor/1799970)

Or if you prefer this Hessian article:

[https://www.faz.net/podcasts/wie-erklaere-ich-s-meinem-
kind/...](https://www.faz.net/podcasts/wie-erklaere-ich-s-meinem-
kind/faszination-wespen-was-das-tolle-an-den-tieren-ist-16882996.html)

~~~
pjmlp
They might be great animals, but as you know here trying to have a picnick
with them arriving as an uninvited gang doesn't turn out a great outdoors
experience.

~~~
ThePadawan
I can't say I have ever been bothered by hornets as opposed to wasps.

(Central Europe)

~~~
throwanem
US here, but I've never been particularly bothered by wasps, at a picnic or
otherwise. I suppose it helps that I don't mind them, or mind sharing food
with them; they're peaceful until given cause not to be, cleanly in their
personal habits, and quite beautiful.

One of my earliest memories is of sitting on the open tailgate of my mom's
truck, carefully holding a slice of pizza so as not to disturb the black-and-
yellow mud dauber who had landed on it to nibble daintily at the edge of a bit
of pepperoni. In the general run of human theory, I gather, such a moment
should require all sorts of histrionics, but neither she nor I saw any need
for them, and no one else was around to disturb her appetite or my
fascination. On reflection, I suppose that moment must have done something to
set a tone for my life in terms of my relationship with aculeates.

And it's not as if they eat enough for a human appetite to notice, anyway.
They're hell on caterpillars - I've seen a few P. metricus foragers entirely
depopulate a thriving fall webworm nest in the course of a couple of days -
but, having never developed a taste of my own for such things, that really
doesn't bother me. And I do very much enjoy watching them hunt!

Squirrels, on the other hand, can all go straight to hell.

