Hacker News new | past | comments | ask | show | jobs | submit login
Node Modules at War: Why CommonJS and ES Modules Can’t Get Along (redfin.engineering)
193 points by dfabulich on Aug 6, 2020 | hide | past | favorite | 147 comments



<quote>

I’ll conclude with three guidelines for library authors to follow:

Provide a CJS version of your library

Provide a thin ESM wrapper for your CJS

Add an exports map to your package.json

</quote>

This seems like the same fallacy of python’s 2to3, and why they should have written 3to2 instead. If ESM is the new way, library authors should instead have a CJS shim to prevent fossilization of cruft.


> This seems like the same fallacy of python’s 2to3, and why they should have written 3to2 instead.

This is incredibly insightful. I'm frankly a bit stunned. I would never have thought of this (and clearly the Python people also didn't) but it is absolutely true. It's also a very transferable strategy for any time compatibility is to be broken.

Is this an original thought of yours or do you have references to further reading?


It's a common refrain when the py2/3 discussions get going. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...


This is something like what the future package and tools do, though I can say from experience it's not without its issues.


> should instead have a CJS shim

OP wrote, note that it’s easy to write an ESM wrapper for CJS libraries, but it’s not possible to write a CJS wrapper for ESM libraries.


It's not possible to write a nice automated transparent one for arbitrary libraries because of top level await, but it's totally possible for the 95%.


You cannot write a CJS shim for any library because the CJS shim cannot expose any ESM file with a synchronous API because importing ESM always returns a Promise.

You can - and people widely do - compile a CJS version of your library from ESM sources. You'll accept that there may be multiple copies of your library in the program but it does work. But in any case there's no reason to actually author in CJS, everything including the ESM shim can be generated from ESM.


It is possible, but you have to hack the Node require() loader to do it, teach it a lot about ESM loading, and then hope the downstream CJS don't have timing bugs around "very long" synchronous require() loads.

https://github.com/standard-things/esm


I'm not sure that's really relevant here because esm is just one of the transpilers for module syntax, it's not running native modules at all. The same can be achieved with the babel CLI or TSC etc. and the right configuration. But it's not really "ESM loading". It's compiling ESM to CJS and then running CJS. It's true that transpiling to CJS will continue to work, no matter what.


To my understanding this ESM module uses native ESM loading on Node versions that support it, which is why others point out its behavior with top-level await is deficient in the same ways that Node's own behavior is, though it tries.


Re your point OP also commented/quoted someone with “I don’t think designing a system with the blanket assumption that some feature just won’t get used is a viable path.”


Yes, someone cannot design a system to handle arbitrary libraries with the assumption.

A library author can provide cjs bindings to their library for as long as cjs remains relevant with the assumption that they won't use top level await.


Yea they are looking at the ecosystem as a whole. But if you are a library author you have the power to control these things.


I'm pretty sure services like pika.dev/skypack.dev/jspm.io are trying to solve the CJS to ESM problem so you can use it directly in the browser (or deno).

Last time I checked they seemed to lean into using rollup to take care of the translation under the covers.


That's not the point here. It's about that a thin CJS wrapper to an ESM module isn't possible but the way around is.


Sorry I misread your comment. But is it not easier to write an CJS wrapper around ESM given that its format is a lot looser?

From what I have seen seems pretty typical to simply export the named properties using a fresh module.exports = {...} and then use default as the main export?


2to3 was still useful because all code was in python 2 when python 3 first came out.


Sure, but maybe upgrading the existing Python 2 code wasn't the problem: the vast majority of the Python 2 code alive when Python 3 was released was probably scrapped by the time Python 2 was EOL'd. Over a decade is a very long time for a codebase to live. (Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so.)

The (comparatively) few Python 2 projects that were alive by the release of Python 3 and still alive by the EOL of Python 2 could probably have been upgraded manually, rather than with a tool.

On the other hand, people not wanting to write new code in Python 3 because it would be incompatible with the Python 2 ecosystem – that was a real problem. That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem.

If only people would have written new code in Python 3 right away, we would have been in a far better position by the time Python 2 would have been EOL'd, purely due to natural turnaround of code bases.

Porting Python 2 code to Python 3 was never the problem. Writing new code in Python 3 was.


> Porting Python 2 code to Python 3 was never the problem.

You're, frankly, high as a kite. As the author of one such port on a large codebase, porting Python 2 code to Python 3 was absolutely a problem.

The issue with 2to3 is that it assumed you'd do the conversion as a one-shot and straight flip over with entirely separate codebases (or dropping Python 2 entirely) from there on, which is completely insane.

What the vast majority of transitioning projects needed was a way to migrate progressively through cross-version codebases:

* libraries weren't going to drop older Python versions, the very few which attempted that (I'm aware of dateutil at least) were extremely painful to users and IIRC ended up rolling back to a more progressive codebase

* programs being distributed were not going to drop Python 2 until EOF at least

* large codebases simply could not afford to perform a completely untested and uncheckable one-shot transition

That's why the community created cross-version support tools like six and friends, and lobbied hard for the reintroduction of cross-version features into Python 3 e.g. reintroduction of `callable` (3.2), reintroduction of the "u" prefix (3.3), reintroduction of % on bytes (3.5) and I'm sure others.

> Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so. […] Python 3 because it would be incompatible with the Python 2 ecosystem

What do you think "the python 2 ecosystem is" except for a ton of projects which needed to be ported from Python 2 to Python 3 exactly? "the vast majority of websites or data processing scripts" is not an ecosystem, tooling and libraries are.

Do you somehow think numpy and lxml and django just scrapped their entire Python 2 codebase and rewrote the entire thing from scratch? Of course not.

> That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem.

3to2 would not have worked any better better than 2to3. The syntactic incompatibilities between the version were not the hard part, the semantic ones were, and 3to2 would not have succeeded any better than 2to3 there because it simply couldn't.

All 3to2 would have given you is a broken Python 2 codebase from a Python 3 codebase you could not even run.


> You're, frankly, high as a kite.

https://news.ycombinator.com/newsguidelines.html

"Be kind. Don't be snarky. "


Isn't "migrating progressively through cross-version codebases" exactly what a 3to2 compiler would allow you to do? You'd write new code (and incrementally port old code) to Python 3, and then when you build the project you'd compile the Python 3 code down to Python 2 before building the entire project as Python 2, including all dependencies.

So for over a decade, you'd be writing Python 3 code but building a Python 2 project. After a decade, one would hopefully have managed to port everything over to Python 3, and can then drop the "compile to Python 2" step.

A 3to2 tool would solve exactly the problems you're bringing up, with no intoxication needed!

----

I don't see why 3to2 would not have worked. It's a compiler. We have written compilers before. We can write compilers. Compilers work. Both languages are Turing complete and about the same in expressiveness, so clearly compiling one to the other is something we can do.

What we cannot (reasonably) do is produce nice, human-looking target code from a compiler. This is what 2to3 attempted to do, and what a 3to2 would never need to do.

Who cares if the Python 2 code it produced looked a little wonky, as long as it was correct and roughly recogniseable as being built from the original Python 3 code? The Python 3 code is the one humans write and edit. The Python 2 code is just compiler output to make it compatible with all of the Python 2 ecosystem until one is ready to go fully Python 3.

----

This is also not something completely foreign and unheard of. This is exactly how I've been porting legacy JavaScript applications to a more modern ECMAScript approach: write new code in the modern way, compile it down to legacy code compatible with the existing legacy code, and then run that.

Slowly but surely, more and more things are being written in the modern fashion, and eventually, when everything has caught up, the compatibility conversion to the legacy code can be dropped.


> Isn't "migrating progressively through cross-version codebases" exactly what a 3to2 compiler would allow you to do?

No. You'd have to migrate everything to Python 3 at once, and then "maintain" two different and incompatible codebases, one generated from the other.

> A 3to2 tool would solve exactly the problems you're bringing up, with no intoxication needed!

It would half solve (at best) one of them.

> I don't see why 3to2 would not have worked. It's a compiler.

Unless your 3to2 would literally reimplement Python 3 in Python 2, it would be a translator / bridge. Which can't work due to the semantics difference between Python 2 and Python 3: not every change translates mechanically and reliably. If they did, 2to3 would have worked.

> Who cares if the Python 2 code it produced looked a little wonky,

The maintainer who has to debug, probably.

> as long as it was correct

Which it wouldn't be.

> This is also not something completely foreign and unheard of. This is exactly how I've been porting legacy JavaScript applications to a more modern ECMAScript approach: write new code in the modern way, compile it down to legacy code compatible with the existing legacy code, and then run that.

The "modern way" of javascript is additive, it's about adding new features to the language, some of which can be implemented in terms of the old one (and which you can thus "backport" through a compiler or extending the existing APIs).

You don't run into pieces of code which are syntactically identical but semantically divergent. Which you absolutely do in P2 v P3.


> You'd have to migrate everything to Python 3 at once, and then "maintain" two different and incompatible codebases, one generated from the other.

Absolutely not. Well, you could, but the better approach would be to write individual modules in Python 3, and then compile those down to Python 2 at which point they can talk to the existing Python 2 code freely.

> Unless your 3to2 would literally reimplement Python 3 in Python 2

That's exactly what I'm suggesting. Take the existing Python 3 compiler and swap out the backend to generate Python 2 code instead of bytecode.

This is stuff people do every day for other languages. It is very far from rocket science.


Amen. Babel gets a lot of things right (and arguably has a tougher job than a 3to2 compiler would, given the speed of change and the target breadth / problem surface area).


> arguably has a tougher job than a 3to2 compiler would

It doesn't. Not that its job is easy, but its job is to take constructs which are brand new and translate them to an older version of the language.

Of the same language.

It doesn't have to deal with the language or the APIs literally working differently when used the same way.

The Python version of Babel would be 3to3 compiler.


Babel is not limited to JavaScript frontends. You can take any language and write a Babel front-end for it. This is, for example, how Fable compiles F# code to JavaScript. Those two languages are very different from each other -- far more differences than Python 2 and Python 3 -- and it works brilliantly.


I made all my Py2 code forward compatible with 2to3. It was mostly a smooth process when you're using the future imports and know what landmines to avoid. It still required manual intervention at times with Unicode support and libraries that have breaking API changes between versions.

The end result is something that can be Py3 native whenever I want. 2to3 is the right tool for the job. There is no way to map all Py3 features back into Py2 so 3to2 would be a broken mess that only supported a subset of the language.


> There is no way to map all Py3 features back into Py2 so 3to2 would be a broken mess that only supported a subset of the language.

Oh but there is. If you can map them to bytecode, you can map them to Python 2, which is a fully capable language. It won't be a one-to-one map, but that's also not required.


Not going to work for async.


Why not? Provide your own future implementation, transform the async code into a state machine. Babel does exactly this for JavaScript async.

Sure the generated code is a horrific mess to look at but even that's a solved problem with sourcemaps if you need to debug it.


good theory, doesn't align with my (limited) experience. writing new code to be both 2 and 3 compatible was quite possible and not too tricky. six helped, `from __future__` imports, etc. however, everywhere i've worked still had some python 2 only code running - even to this day unfortunately.

the real issue is 2to3 didn't work so well, or exposed the exact issues why python 3's breaking changes were needed (80% sloppy bytes/str/unicode handling).


PSF wanted people to migrate to python3 for reasons including not wanting to support multiple languages. Why would they write some tool to convert new python3 code back to python2 that would ultimately work against their agenda? It doesn't make sense to me.


Sorry about nitpicking, but I feel that we shouldn't say that CJS has named exports. It does not.

CJS has a single export which might be an object. And objects have named properties/fields.

On the other hand, ESM CAN have unnamed "default" export and CAN have arbitrary number of named exports.


> On the other hand, ESM CAN have unnamed "default" export and CAN have arbitrary number of named exports.

And this is a constant source of confusion and frustration for me when working with so many npm modules lately...


> On the other hand, ESM CAN have unnamed "default" export and CAN have arbitrary number of named exports.

To nitpick on the nitpick: ESM only has named exports. It just treats the export named "default" differently in some scenarios and has special syntax for it.

So, roughly:

* CJS: Exports a single value which may or may not be an object. When assigning the exported value to local variables, destructuring can be used to access individual exports properties.

* ESM: Exports a namespace of named bindings. When importing the bindings to local aliases, the name of the alias can be changed using `as`. There's special syntax for importing the binding with the name "default".

It's somewhat unfortunate that the syntax _looks_ so close without being all that close.


Wow that was informative. So ES Modules is really about the Red functions [1] getting module support.

I find it fascinating watching our understanding of async computation mature over the years.

[1] http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...


No, this is not correct. The split long predates async in JS The fundamental reason behind the difference between ES modules and CJS is that require() does synchronous I/O in a function, but that's not acceptable in the browser context. So you either need to make require an asynchronous function (meaning with a callback, back in 2010 when this happened) or move the I/O out of the execution of module itself, as import does.


Sync xhr is being depricated. Meanwhile Google and Facebook analytics/tracking freeze the main thread for a whooping two seconds. Why not push the module dependencies with http2 push? The problem is not technical. Its big money wanting to break backwards compatibility.


Even HTTP/2 Push is an order of magnitude slower than local File I/O on a good day with perfect (internet) weather. ESM isn't even the first approach to try to deal with this, for instance AMD had nothing (at first) to do with "big money" and everything to do with average web developers trying their best to practically solve the module problem for the web and CommonJS was never going to be an option in a web browser (even if AMD or ESM spec authors had magic foresight to HTTP/2 or HTTP/3 they'd still have the same concerns with CommonJS).

CommonJS was a mistake that the JS world is going to live with for years more.


The Node.js module system is only loosely based on CommonJS. The Node.js module system is probably the biggest reason of the success of the Node.js ecosystem. Ifyou want to attack Node.js you would start with the module system.


The last time I asked here why async-await is even a thing, when you can just introduce light/green threads, coroutines, you name it, everyone jumped that it is even better when you mark awaiting points explicitly. Now that it goes from a blog post of some popular language designer, seems no one gonna disagree.

>async computation mature over the years

A good thing, really, but would be much better if the experience from few decades ago wasn't ignored by cool kids who mature.


The reason why async/await is a good fit for Javascript is that it neatly wraps up asynchronous callback hell and Promises, which were already ubiquitous in Javascript code, because the web runtime enforces non-blocking IO.


It may be the reason, but not the only option.

  function cpsfoo(cb) {...}
  function foo() {
    var thread = this_thread()
    cpsfoo(x => thread.resume(x))
    return thread.yield()
  }
Error and immediate cb invokation handling omitted for clarity. Similar wrapper or wrapper-generator (uncps(f), unpromise(f)) may be done for other primitives.


> Error and immediate cb invokation handling omitted for clarity.

... but this is exactly the reason why callback hell existed. Error handling down the callback branches were a total mess, especially with networked state transfers.


Error handling can be easily added to the example wrapper above, completely hiding all the burden from a user of foo(), who then could do their usual try-catch. It's just my phone keyboard that prevents me from implementing it right here.


Sure, it's not the only option, just the better option for Javascript. Your example isn't representative of Javascript.

This is an example for something that might come up in Javascript, using async/await:

    const a = await fetchA();
    const b = await fetchB(a);
    const stuff = await fetchStuff(a, b);
    const transformedStuff = await Promise.all(stuff.map(someAsyncTransform));
Error handling is just try/catch.

Note that you generally have no control over most of these function being asynchronous. Threads or coroutines do nothing to help you write straightforward code when all your libraries are based on fine-grained callbacks/promises. Writing out that chain of dependencies in CPS would be a nightmare.


They really do though. In the modal case you would just write the same code as in js but without all the extra keywords. If you need fanout you'd call some special function to do that, like you do now.


well coroutines are a way better option and working way better than promises, unfortunatly promises were introduced since it's easier to convert a callback function to a promise based one


Never should have had to worry about that stuff to begin with, unless you were actively using it. I’ve read and written way more code that had to wrestle async back to sync than code that’s actually been able to let async happen at the local scope. Of course if you’re working in a framework (Express, say) one might expect that it would run your entire mostly-sync pile of code async, so it can hop over and process another request. And that’s fine. That’s how it should have worked. Most things should execute in an async context so other things can happen while they’re waiting on IO or whatever, but within them async behavior should have been what required a keyword or a bunch of hideous nesting or whatever.


Do Promises and async-await work together better than, say, Promises and Lua-like stackful coroutines?


This description is probably far more abstract than most would like (and it is probably wrong by not being exact enough, I am not a mathematician I am a software developer), but might be useful to some: coroutines are a join calculus and Promises are a monad. async/await is syntactic sugar for a monad (less general than Haskell's do-notation, more general than only Promises, though rarely in practice used much beyond Promises). You could plumb Promises on top of coroutines easily (and while JS engines are more traditionally event loops; other languages that support async/await such as Python use much more coroutine-like underbellies), and while you don't get the full "join calculus" power/flexibility by putting the monad abstraction on top of the join calculus, you get the benefit of monad transformers like async/await.

Those benefits being that the async/await syntax and semantics for working with Promises more closely resembles the imperative code it replaces, whereas the join calculus is its own [sometimes easy, admittedly] learning curve with its own syntax and semantics. Most of the complaints about Promises are the usual complaints about monadic wrapping types that the types become "viral" and "color" all associated code. But that just means that the failure cases are more easily picked up by a static type checker than semantic failures in learning the join calculus.


I dont know Lua, but async/await in JavaScript is syntax sugar for a coroutine.


In regards to JS, any reasons for a sync await seems to be rationalization.

Node is built on V8, which is built for browsers. Browsers have a single threaded execution model that also hides an event loop that runs the entire tab.


Yes, async await is a bug, introduced to workaround the lack of proper concurrency in the runtime.


The whole point of that article is that it hasn't really matured at all


I’m not sure if that’s true. I remember that at some point there were more than 2 systems, none were standards, and they were widely incompatible, I remember AMD for example: https://github.com/amdjs/amdjs-api/blob/master/AMD.md.

I cannot find the others though.


In two-and-a-half months the last 5 stable releases on Node, the last 2 LTS releases of Node, and all current major browsers will support standard JS modules.

It's time we all published standard JS modules and only standard JS modules to npm.


> the last 2 LTS releases of Node

(Author here.)

Node 14 supports ESM, but Node 12 only supports ESM behind the --experimental-modules flag, and its error handling for incorrectly importing CJS is not good.

In 2020, I think it's a bit early to go ESM only, especially since it's straightforward for CJS libraries to support both CJS and ESM clients. (I document how to do this in the article.)


The experimental flag was recently removed in v12.17.0 (though the release notes still say ESM is an experimental feature)

https://github.com/nodejs/node/releases/tag/v12.17.0


I feel the opposite. ESM is a nightmare and I hope I can avoid using it in node as long as possible.


that's the attitude that brought us sites only working in Internet Explorer (and now Chrome)

IE had many non-standard APIs that were much more convenient to use than what the standard was providing (innerHTML, document.all)

And rather than moving to the standard, more and more code was written targeting IE which also had a conveniently large user-base.

ESM is the standard. CJS is the solution proprietary to node.js.


Meanwhile, I found this on how Deno handles modules (it's all esm):

https://www.sitepoint.com/deno-module-system-a-beginners-gui...

Note that - if you can get an esm out of your cjs - and it doesn't use any nodejs apis unsupported by deno, then you might be able to use it:

> (...) Several CDNs can convert npm/CommonJS packages to ES2015 module URLs, including: Skypack.dev, jspm.org, unpkg.com (add a ?module querystring to a URL)

Ed: submitted as: https://news.ycombinator.com/item?id=24069037 since I couldn't find an earlier submission, and maybe it'll generate some interesting discussion on deno / modern js vs node.

In particular I wonder if it'll in practice be easier to share (more) code between backend and front-end.


There's also a crutch - I'm not sure how attractive it would be in a project using deno, though. Maybe to get up and running quickly on deno when moving a legacy project from nodejs?

https://deno.land/std@0.63.0/node#commonjs-module-loading


> (it's all esm)

Always has been.


JavaScript just gives to much options. I keep having problems with libraries because ES6 allows

  import x from "library";
  import {x} from "library";
  import * as x from "library";
  import {x as y} from "library";
Just to compare Java uses this:

  import java.lang.Double;
  import java.lang.*;
No renaming, not a gazillion options during in- and export, but one statement. If you want to rename something, assign it to a variable.

I don't understand why ES-module authors made it that complicated, when they had the chance to introduce a proper module system.

Why did they not just copy Java's system and be done with it.


1. Get the default export as x

2. Get the exported value x

3. Get all exported values and call the object x

4. Get the exported value x and call it y

They're all different because they allow you to do different things, and fairly readable.

Edit: Just realised dimgl said the same 45min ago, my bad!


I think people (like the author of this article) should explain the syntax of destructuring assignment {x} briefly to make it make more sense for the unfamiliar.

[1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...


This looks similar to detructuring assignment, but it isn't:

- no recursive destructuring

- `import {y} from 'x'` is not the same as `import x from 'x'; const {y} = x`


Thanks for pointing it out, didn't know that.

But I think your second point is more to do with the fact `import` by default imports `default` instead of the whole thing.

If you use import * as x from 'x'; then `const {y} = x;` works.

I personally will consider it is `import from` being different from a typical object instead of {x} here being different from typical destructuring assignment (by that I mean they still follow the same pattern; not necessarily say they're the same thing.)


Java's package system has its own problems, and all of those examples you gave are very straightforward.

1. Import the default export as the name `x`.

2. Import named export `x` as the name `x`.

3. Import all of the named exports under an object named `x`.

4. Import named export `x` as the name `y`.


The order of operations was different: the packages ecosystem came before the imports language construct. The location and the name of libraries are not correlated (PHP is closer to Java here, in the default case). Also, the idea of tree-shaking means you don’t want to import the whole thing (options 1 and 3) when you use something like webpack. Renaming let’s you use competing names. There are decades worth of history and libraries to accommodate.

Your last sentence is like a C dev asking a Java dev why they don’t just compile to x64 and be done with it. It comes across as flippant and will likely attract downvotes.


So what happens when you need to deal with java.time and Joda Time in the same class (e.g. because Joda was part of your public API but you're migrating to Java time). Fully qualified names everywhere, when it would be nicer to have a construct like:

import org.joda.time.Instant as JodaInstant;

The rest of the weirdness is to make it look like JS destructuring


Version the API and switch to Java-time everywhere. Or forward everything to another class.


Unless you're going to duplicate the logic too, you're still going to have the facade class providing the legacy Joda API that also needs to use the new types to forward onto the new API.

That class doing that transformation will need to refer to both the Joda and java.time class.


C# has exactly that FWIW: using foo.lib = libfoo


Java's packaging system sucks as does all the similar ones like C#, C++ etc compared to JavaScript's IMO.

All of those the space of namespaces is global. You can't have 2 modules named math.quaterinions in any of those languages. You can in JavaScript. To put it another way, in order to prevent name clashes in C++/C#/Java you have to know the name of every other project on the planet. If someone uses the same namespace then if you ever need to use those 2 projects together one of them will have to be re-factored.

That problem doesn't exist in modern JavaScript. There are no namespaces. There is just floating references to code.


I honestly have no idea where you got the idea that namespace conflicts are a problem in C# or Java. Both have solved this problem (C# with aliasing and Java with packages).


aliases help you use bar from foo.bar and moo.bar. They don't help you with 2 files both of which declare foo.bar


Why would you declare the same type in 2 different files in the same project?


you don't have control of all 3rd party libraries.


Then it's not the same project and the fully qualified type name will be different.


That’s not true. You can use aliases just fine. .NET is great with this.


in one file declare

    namespace foo {
      class Bar ...
    }
In another file declare

    namespace foo {
      class Bar ...
    }

Now use both foo.Bar from file 1 and foo.Bar from file 2 in the same project. Aliases don't help with this


Different assemblies with the same assembly name, namespace and class? Use an extern alias (which is how different versions of the same assembly can be used if necessary): https://docs.microsoft.com/en-us/dotnet/csharp/language-refe...

Different assemblies with different assembly names but same namespace and class? Just use the fully qualified type name (and alias it for a shorter name).

Single assembly/project and namespace with the same class in multiple files? If you're just spreading out the code then use partial classes, otherwise why would you define the same type twice?

C# (and other languages) that use virtual namespaces can do everything that file-based namespaces can do, but also support many more scenarios.


> Java's packaging system sucks as does all the similar ones like C#, C++ etc compared to JavaScript's IMO.

There is one gazillion things that are much more a problem than "namespace conflicts" in packaging and package management.

Including security & code authenticity, parallel multi-version handling, sensibility to side effects, speed, auditing and reproducibility.

And in these domain the JavaScript packaging system generally range from being 'catastrophic' to 'absolutely terrible'.

If I have to take an example package management system for the world I would certainly quote cargo, Nix, Guix or Spack but certainly not npm which is quite close to what you should really not do for a package manager.


As a Java programmer dropped into JS land, I totally agree. Keep it simple, one way to achieve something wherever you can. Pythonic, in other words. What I don’t understand is why we didn’t learn from other major languages that had gotten this right already.


You can also do `import x, {y} from ...`. Very confusing to learn.

I think they did it like that for compatibility with CommonJS.


Rollup makes building both CJS and ES versions of a library fairly straightforward. https://rollupjs.org/guide/en/#outputformat


Just tinkering with Rollup: How can I let Rollup work with a pnpm monorepo, so that it's resolving local packages declared via pnpm's pnpm-workspace.yaml?


I use the ESM dependency to load ESM modules as a CJS shim:

https://www.npmjs.com/package/esm

I highly recommend it. Really let’s you unify on ESM modules without much thought


For those who, like me, wondered how this is possible without hitting the top-level await obstacle mentioned in the article, the options doc includes this line (emphasis mine):

> "await":false - A boolean for top-level await in modules without ESM exports. (Node 10+)

So, yeah, seems to rely on "most ESM modules don't use top-level await". Seems like a useful option to have, thanks!


Another similar option is `@babel/cli` which also allows using module syntax with require-like semantics. It's a bit more widely supported in other tools which can be an advantage.


How did this escape my attention till now? Thanks for the link! :)


Wow, I've been coding JS/Typescript frontends for years and somehow completely missed out knowing anything about CJS & ESM or their differences.

I guess it's all irrelevant when you don't use Node, aren't publishing libraries, and are compiling all JavaScript into bundles for deployment and never read the bundled code.


Typescript and your bundler both have to know a lot about it. Typescript went all in on ESM syntax since 1.x so you at least know ESM syntax by now if you are in Typescript and can pretend that CJS is a bad dream, but Typescript will happily spit out CJS for you with a compiler flag if you need it for node.

So much of the complexity of webpack, parcel, rollup, pika, snowpack, et al is dealing with the weird mix of CJS and ESM spread across npm at this point.

One of the main differences between the bundlers at this point is their opinion of CJS and ESM handling. You may not notice it as a bundler user, but you probably see a lot of indirect results in things like bundle size and ease of use. So it is not entirely irrelevant to frontend work, but you just may not realize that why you prefer one bundler over another today is heavily influenced by their approach to the CJS / ESM split. (A good bundler you shouldn't have to realize that behavior is what contributes so much to the bundler's "feel" and your gut-reaction/instinct to working with it.) Learning the differences can be useful if you need to better pick your next bundler based on technical underpinnings, though admittedly that sort of "comparison shopping" is probably a rare need.


I know how you feel. It really feels like tribal knowledge. You learn this stuff bit by bit from struggles and HN posts. There’s just too much JS stuff out there!


Unless you are developing libraries which will be published to the nodejs ecosystem, you really don't need to worry about this.

You pick one of these when you first create your app which is typically based on whatever framework or template you are starting from, and you just always stick with that.


Also, typescript can compile `import`s into `require()`s.


I think CJS can export 1 more thing than ESM is a problem too.

    // cjs.js
    module.exports = (a, b) => a + b
    exports.default = 'default'
    exports.nice = 69
    // esm.js
    import * as mod from './cjs'
    // what is `mod` now?
In babel, `mod` will be `{ default: [Function], nice: 69 }`, and the string 'default' will be discarded because ESM always exports a `{ [key: string]: any }` while CJS can export whatevey they want (`any`).


I really feel like default exports were a mistake, everything is simpler without them, they don't even work with things like `export * from './foo'`


Another issue about module import I encountered before and wasted me 1 hour to find the culprit:

When converting TypeScript that using `import default xxx` to JS using `tsc`, it would convert it to something like `exports.default = xxx`.

To import it, you have to use `const XXX = require('./mo.js').default` instead of just `const XXX = require('./mo.js')` in plain ES6 JS.

This is fine but Babel used to [1] be able to convert `const XXX = require('./mo.js')` to valid code, which made lots of people wrote it (incorrectly) that way (I was very baffled why people wrote like that since it does not work in plain ES6).

[1] https://github.com/babel/babel/issues/2212


But how do you solve it in a tsc context?

Using the default prop doesn't work in tsc and breaks all type information because tsc just doesn't know of any default prop.


Not exactly sure what you mean but you can check this: https://github.com/microsoft/TypeScript/issues/2719


Thanks for the link. I meant following case:

A TS file which compiles to an es6 ("esnext") module cannot import cjs. Either node breaks or TS' type-checking fails. Former if you omit default when importing and latter if you introduce .default in the TS codebase.

It's a bit complicated, what I just wrote about is reflected in the chart in step 4-6 here https://github.com/microsoft/TypeScript/issues/18442#issueco...


Typo: I meant to say "... using `export default xxx`...".


Holy crap, this redefines dependency hell.


Gimme COM+ back! I’m sorry I hated you...


I spent half an hour yesterday banging my head against the wall on exactly this.

Thank you.


It's always surprising to read about the trouble this causes for Node. I've been writing mostly Webpack-compiled JS for the past few years, and it seems to allow any random mixture of require and import. Have I just been avoiding trouble areas by chance?


Webpack + babel are basically rewriting `import` and `require` calls into their own `__webpack_require__` helper and converting ESM to CJS. Whereas `require` and `import` are node built-ins and `import` is browser built-in. Making stuff interoperate requires some trickery and lax spec compliance.

There are high chances you are relying on some non-spec compliant behaviors but you don't know about it. For example, ESM exports have immutable bindings, which means, mocking exports for unit testing is impossible. If it works for you, it's because you transpile the code from ESM to CJS (either via `babel`, `rewire` or similar tool).


I do the same. Babel makes life a lot simpler, indeed, but there are still some edge-cases. Especially around tree-shaking, optimization.

And dynamic imports are still complicated sometimes


The description of how require() works is not quite correct. Node does not always fully initialize the dependency module before returning from require(). It can't work that way, because Node allows circular require() dependencies. Try it!


The reason is not correct. Nodejs cache the require lot and so circular dependencies can work without returning half finished loads.


Nodejs does cache the results of require(), but that doesn't help with this problem. Example: https://repl.it/repls/SwiftOlivedrabBrace


> Node does not always fully initialize the dependency module before returning from require().

Unsure what this means. The only reason this happens is because an unfinished copy is provided to the dependency that is causing a circular reference.


It means you can get a half-initialized module back from a require() call.


Does that mean the required modules are not run?


They are run, but if there are circular dependencies they are run _after_ require() returns.


While I understand why it was done, I feel ESM vs CJS Will be the breaking point for node.js. All my clients are using Babel to avoid this single issue but it's slow enough to be a DX problem.

I wonder if deno (maybe with a tsc reimplementation written in rust) will take its place for new projects. Personally, I've already suffered enough with python 2/3 and decided, after 9 years writing node, to use uniquely rust for new backend development.

Frontend development is quite a slow process as well, but all the faster alternatives imply knowing a different language - which impacts developers availability (given historically frontend developers are mostly JS).


This is an important issue to consider because as we are witnesses a Cambrian Explosion in 'new tech' (and by the way, in 'new' we can include evolutions of 'current') ... we need to grasp that 'a new piece' is is just going to be 'one of many pieces'.

My first instinct these days is how that funky new bit of Lego fits with all the other bits of Lego.

When you start there, it's becomes easy to understand just how incumbent C++ and javascript are, even with all their necessary spells and arcane witchcraft.


Honestly, all I need is a standardized way to publish and consume packages directly as Typescript (right now you can point "main" & "types to it, but I'd rather something like export maps) because the transpilation should be done by the end user as you can't assume which build target they might prefer.


Typescript transpilation is already pretty slow when just used for your own application, if all the node_modules directory entries were in TS I'd probably be looking at hours for the transpilation.


I removed Babel from my toolchains a while ago (and instead use TS to transpile even dependencies), so the compilation times are fine.


fwiw, TypeScript and ES Modules are just not compatible and never will be: https://github.com/microsoft/TypeScript/issues/18442#issueco...


That chart shows that Typescript is incompatible with CJS modules, because it inherits ES Modules incompatibilities with CJS, not that it it is incompatible with ES Modules.


You can use ESM (including dynamic import) in node 6+ (without babel) by using https://github.com/standard-things/esm and cut the `.mjs` and "it can't work like that" crap.


You can definitely continue to use transpilers like esm or babel to continue using CommonJS behind the scenes while writing module syntax. But it's not quite the same as using native modules, as implemented by JavaScript engines.


it seems so insane that there are competing module patterns in a single language, but perhaps i'm ignorant. are there other languages with such incompatibilities on a fundamental component?


Lisp is the easiest classic example that took ages to settle on a module standard and has had different competing standards over the years.

PHP and Perl are probably more recent examples comparable to JS here that provided mostly only low level plumbing (include/require) support and nearly wound up with multiple incompatible module loading patterns.

Even C/C++ the "module" system started as a text processing hack in a macro pre-processor that used to be a separate tool (though I don't think any modern C/C++ toolchain uses a separate macro pre-processor, as there are tons of optimization layers now and some of the macro pre-processor knowledge makes its way all the way through to the linker these days).

Historically, modules were always just "smash these two files together in this order" and it was only later that languages started asking hard questions such as "which symbols from this other file are available".

The modern concept of modules as a fixed "shape" of self-describing public symbols is as much a later abstraction out of Object-Oriented Programming as anything else. Our idea of what a module should be in 2020 is much more informed by "recent" notions such as Component systems like COM and languages such as Java and C#.


very cool thanks for the context!


`require()` isn't really a part of the language. It was a library added by nodejs, since javascript had no way of expressing modularity at all at the time.


Is this really such a big deal? Using ESM with imported CJS packages has been pretty straight forward for me. It's hardly the worst thing about Node.js development.


If library authors make esm as a thin wrapper over cjs, won't it hinder tree shaking.

And for tools that want end to end esm, like vite. Won't it cause degradation in behaviour?


> If library authors make esm as a thin wrapper over cjs, won't it hinder tree shaking.

Correct. I wouldn't follow the article's advice wrapping ESM with CJS as it leads to larger bundles.

> Won't it cause degradation in behaviour?

For web apps - certainly. The forced sync behavior of CJS wouldn't be very noticeable for local applications however.


And the article is not even talking about default imports on transpiling with tsc or babel (or both!). Too many combinations.


Oh yes, especially with tsc (it's just not working), see my other post here with a Github issue link


Not taking seamless CJS interop into account was IMHO the biggest failure of TC39 till date.

Given the widespread popularity of CJS they should have prioritized CJS backward compatibility over things like better support for circular dependencies and mutable exports. Many languages are doing quite fine without them.


npm install esm

Then everything just works. Top level await and all.


Wait till you throw a package.json with "type": "module" in there.

Everything used to work until Node 12.15, then 12.16 started throwing when trying to require type:module packages with `require` (which is what `esm` does)

https://github.com/standard-things/esm/issues/868


I haven’t seen CommonJS modules for years now, only ES modules. Didn’t realize this was such an issue still.

> You can’t require() ESM scripts

That’s one way to put it! Another would be to say that you don’t have to. require() came about because we didn’t have the import keyword.


(Author here.)

You were not using native ESM in Node years ago. Node 14 shipped with unflagged ESM support just this year. (If I had to take a guess, you were probably transpiling "import" statements into "require" statements and/or bundles, which honestly works better than native ESM, because there's no interop issue.)

Almost all of the most depended on NPM packages are CJS, and provide no ESM exports. https://www.npmjs.com/browse/depended

lodash, react, chalk, commander, moment, express, axios, and vue are all CJS.


I guess you don't work on Node.js backends.


A major benefit of node modules using ESM, at the moment, is that they aren’t swimming in a swamp of NPM packages.


Enough with the hornet hate. They are great animals. If you know German, you can learn how great they are in the knowledge podcast of the Bavarian Broadcast: https://www.br.de/mediathek/podcast/radiowissen/die-wespe-un...

Or if you prefer this Hessian article:

https://www.faz.net/podcasts/wie-erklaere-ich-s-meinem-kind/...


They might be great animals, but as you know here trying to have a picnick with them arriving as an uninvited gang doesn't turn out a great outdoors experience.


I can't say I have ever been bothered by hornets as opposed to wasps.

(Central Europe)


US here, but I've never been particularly bothered by wasps, at a picnic or otherwise. I suppose it helps that I don't mind them, or mind sharing food with them; they're peaceful until given cause not to be, cleanly in their personal habits, and quite beautiful.

One of my earliest memories is of sitting on the open tailgate of my mom's truck, carefully holding a slice of pizza so as not to disturb the black-and-yellow mud dauber who had landed on it to nibble daintily at the edge of a bit of pepperoni. In the general run of human theory, I gather, such a moment should require all sorts of histrionics, but neither she nor I saw any need for them, and no one else was around to disturb her appetite or my fascination. On reflection, I suppose that moment must have done something to set a tone for my life in terms of my relationship with aculeates.

And it's not as if they eat enough for a human appetite to notice, anyway. They're hell on caterpillars - I've seen a few P. metricus foragers entirely depopulate a thriving fall webworm nest in the course of a couple of days - but, having never developed a taste of my own for such things, that really doesn't bother me. And I do very much enjoy watching them hunt!

Squirrels, on the other hand, can all go straight to hell.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: