This is an interesting case of critical mass. A few years ago Rust was a pretty suboptimal language to start JavaScript tooling in (or Python for that matter). Some crazy folks however did not care, and just went for it. As time went on, that base level of investment into tooling all the sudden meant that there was so much to leverage, that it went from an obscure language choice to a pretty reasonable one.
Cargo and crates.io make it a cinch to include libraries which implement discrete chunks of the solution. The effect of opinionated defaults and an easy path to publishing coupled with the Rust community's passion for correctness and efficiency really play off each other to result in rapid expansion of library capabilities. Crates seem to be well modularized, there seems to be widespread agreement about which crates provide good solutions, and that results in widespread compatibility in the Rust ecosystem between complex codebases doing similar things.
Working in such an environment is a dream. Need to add a feature? Chances are that someone out there has already built every piece of it and all you need to do is track them down and stitch them together.
1) esbuild isn’t good at splitting
2) rollup is too slow
Why wasn’t it an option to improve esbuild’s splitting functionality or to improve rollup performance? Why is it the best option to introduce yet another tool?
I can't speak for the authors of Rolldown, but esbuild is essentially a one-man show and everything is bespoke and written for esbuild. Not saying this as a bad thing, Evan single-handedly improved bundling performance for Node and started us down this path of tools in Rust. The libraries and ecosystem for writing tooling for other languages is really taking off with projects like oxc, SWC, LightningCSS, Biome, rslint, Deno, rspack, napi-rs/Neon for JS/TS/Node etc plus ruff, uv, rattler, pixi for Python and so on, so you get a proliferation of libraries and authors that can share and help each other.
Plus, on a personal level, using a language with pattern matching and algebraic data types makes writing tooling for parsers and such much more ergonomic than in languages without.
My understanding (could be totally wrong) is that esbuild has historically been less open to widespread collaboration than some other open source projects, so adding the features they needed may have been untenable.
I have no authority and will not stand behind these claims if challenged.
Also, it was originally written in both Go and Rust, and the author decided to continue in Go. Both are fine languages, but it makes an interesting case study:
Wow, thank you for sharing this. The author echoes a lot of thoughts I've had working with rust.
However, doesn't the explanation given for why the Go version was 10% faster mean that esbuild was built to take advantage of a fixed number of cores rather than all of them? Sorry if this is a dumb question, I'm not really that experienced with parallel computation
I'm not sure. From the description, it sounds like whatever rust does on one thread (compute + destructors) is split across multiple threads in Go (compute + gc), but it's not clear if the base computation workload is spread across multiple threads.
Thinking about the problem, I think that at the very least parsing could be parallelized. Assembling everything into one output might not be parallelizable. But I haven't looked at what happens in esbuild.
I do know that it's fast enough that when I switched from webpack I had to check that it actually did something, because it returned immediately.
> Why wasn’t it an option to improve esbuild’s splitting functionality or to improve rollup performance?
Rollup is written in javascript; not only is the runtime slow, the multicore story is terrible.
Improving esbuilds splitting functionality is hard because splitting is full of hard tradeoffs; the spec allows module imports to have side-effects and requires that they be evaluated in order. It's therefore impossible to do optimal code-splitting without violating the esmodule spec (potentially introducing subtle errors), and the `esbuild` project values correctness even more highly than performance.
Personally, I'd be very happy with "If you enable the 'make it fast' option and have modules with top level side-effects, you will get weird bugs", but I respect where they're coming from in not wanting to do that.
It'll be equally hard to write it from scratch in rust as it would be to add this to esbuild.
Maybe there are some considerations I don't know about but it does seem like hard forking esbuild would be much faster way to get there (hard fork so that they are not blocked or slowed down by whatever esbuild owner wants).
It really depends on who (which team) is doing it.
I would almost surely give up before understanding what esbuild does and how, and what needs to be modified. (Because Go basically gives me the creeps.) So someone with a lot of Rust experience probably simply opts for the rewrite.
That said, it seems folks on the Vite team decided to give this a try because of oxc_parser (a JS parser written in Rust), so probably it's not because of Rust-Go sentimentalism.
// Though ... there's also otto[1] (a JS parser in Go) ... so who knows! :))
// And of course even more context here[2] and here[3] (this one is from Evan You)
Speaking of things “written in javascript” for the longest time all I kept hearing is how incredibly fast things written js are…
Is it fair to say that it was a bit of a sales pitch?
When key components of the ecosystem that were written in the language of the ecosystem itself are being rewritten in other languages - I think it’s hard to find a more compelling evidence that all is not as great as it was advertised.
> for the longest time all I kept hearing is how incredibly fast things written js are…
I am sure no-one ever claimed that things (properly) written in JS are as fast as things (properly) written in C, Go, or Rust, and running in a native environment.
JS on the server may have been faster than php or python, due to the non-blocking nature of the event loop; but even that may not be the case anymore.
Generally, the pitch has been that js is "fast enough" for most tasks.
I don't know what OP is referring to here, but from discussions like this:
https://stackoverflow.com/a/4417485
It seems like circa 2010 folks were referring to JavaScript as "fast" because it was more performant in some ways than other interpreted languages like Ruby and Python and those were the only major popular options at the time that people would consider for some use cases.
The event loop model of JavaScript led to some creative use cases for having a single-threaded application handle many highly parallel requests that require very little CPU usage or computation but a large amount of blocking I/O, which I think was also pitched as "fast".
But I think it's always been uncontroversial that interpreted languages like JS, Ruby, and Python are all "slow" compared to compiled ones for CPU heavy use cases.
I find VSCode is fast enough for me. I work on a project with ~800k files. VSCode launches fast, finds files fast, greps fast enough, etc... So to me, JS is working as advertised. Choose the right solutions and algorithms and it works fast.
I get that VSCode might not be performant for others. I'm on a M1 Mac with 32gig of ram. I never notice any slowdown compared to any other editor I've used. I'm not saying other editors aren't measurably faster. I'm saying I don't notice any issues personally.
Node/JS (really any scripting language) is fast enough in scenarios where the productivity gains are worth the performance trade off (example: many web backends). Tooling is not one of those use cases.
Productivity gains of using Node.js are a myth. You will be better served by C# + ASP.NET Core with massive performance gains while writing about the same amount of code. Or sometimes a little more but cutting down 10 times on ensuring it works correctly.
Yes, perhaps if the CLI tool is being invoked many many times in succession.
But otherwise JITing isn't an issue: either (a) your program runs quickly in which case the Ignition interpreter is used and the JIT is not used, or (b) your program runs slowly in which case JIT (Sparkplug/Maglev/Turbofan) has time to warm up.
> Yes, perhaps if the CLI tool is being invoked many many times in succession.
Each execution will be a fresh VM though. A CLI tool isn't a long-running daemon. There is also the whole lack of parallelism without forking in this ecosystem.
For esbuild: We know other teams that have attempted to improve code splitting based on esbuild and found it very difficult. A big part of it is that in order to be fast, esbuild applies multiple features (bundle, treeshaking, transforms) in as few AST visits possible, but that comes at the cost of the logic of different features not being layered / decoupled nicely. It is difficult for anyone other than Evan Wallace himself to add non-trivial new mechanisms to esbuild, and although we didn't directly talk to Evan about this, we felt it would be too much to ask for a substantial refactor of esbuild in order to get what we need. In addition, the people interested in making Rolldown happen has much more experience in Rust than in Go - and there is a lot more to leverage (e.g. napi-rs & Oxc) in the Rust-for-JS ecosystem.
For Rollup: the Rollup team itself has been trying to incrementally improve Rollup's performance, e.g. by swapping acorn with a Rust-based parser. But there's only so much you can gain starting from a pure JavaScript base, especially considering multicore utilization. Another aspect of the performance is in the back-and-forth between Rollup (on the JS side) and native transforms (swc, esbuild) - there is a lot of overhead repeatedly parsing / serializing ASTs and then passing strings across JS/native. By building on top of Oxc (which will ship transforms in the future) we hope to be able to stay on the native-side as much as possible to avoid such overhead.
Thanks for the detailed explanation. It’s really interesting how much of an impact language choice has on outcomes. Given your description of esbuild, it almost feels like there is space for a bundler compiler that can generate an optimized bundler from a pipeline.
esbuild is and has always been governed strongly by Evan Wallace. I can't vouch for whether or not the Evans (both Wallace and You) talked to each other about it, but I'm almost 100% certain Wallaces' vision for the esbuild project is different than what rolldown is ultimately trying to achieve.
In some respects, rolldown is also the next generation of Rollup[0] and is a direct answer to rspack[1], which is a webpack compatible rust based toolchain from ByteDance
[0]: Which switched to SWC as its underlying parser from acorn, which makes me wonder if that influenced the decision to use OXC, perhaps they found SWC to be troublesome. I know the Rollup and Vite teams are extremely close, and wouldn't be shocked if any troubles they had with SWC weren't shared with the Vite team when considering underlying technologies
I've found esbuild is pretty good at splitting when source formats and outputs are ESM. (Given the name esbuild a focus on/priority for ESM input/output makes sense to me.)
Is this maybe Vite team saying that they still have too many brownfield CJS dependencies and still don't believe in ESM output for production bundles in 2024?
esbuild handles CJS dependencies just fine and had been what Vite was using in development for a long time anyway. It doesn't split CJS dependencies or tree-shake them as automatically/smartly as it could, but it still supports splitting them and there are manual workarounds.
I think the bigger problem, from my experience here, is still the question of not moving to ESM as the default production output. That seems out of date.
1) I know Rust, not Go (esbuild is written in go-lang)
2) I don't have to worry as much about breaking Rollup, and I'm not nearly as confident in my ability to write performant JS as I am in writing performant Rust.
My experience is that the main reason why Rust is faster than JS in some applications is that the latter tends to do a lot of copying and other memory-intensive tasks - especially those related to running the JS virtual machine.
If you can reduce the problem to a series of operations on pre-allocated typed arrays, JS is fast enough.
Unfortunately in the case of Rollup that would be hard to do.
Personally I feel like moving a JS tool to rust means 99% of JS/TS programmers are unlikely to be able to supply a patch to fix a bug or participate in most ways. I suspect some people will consider that a good thing though.
Both of those options imply not using Rust, hence why.
I tend to think Rust isn't suitable to rewrite JavaScript tooling, any compiled language would do, or even native addons, but apparently dealing with borrow checker for userspace software that doesn't require 1us performance within 512 KB is appealing.
Copy and Clone traits have different semantics. Copy is only for bitwise (and cheap) copies. Clone is for explicit, potentially expensive copies. Also what entails cloning differs from type to type. String, for example, needs to duplicate the pointed to string buffer in the heap, while Arc only increments a reference count.
Yes, I know. But in the context of what’s being described here someone getting started with Rust and wanting to avoid issues with lifetimes could very easily be using copy in the way the OP described clone.
I cannot think of any mainstream language that is AOT-compiled and create standalone executable for the three major platforms other than Rust and Go* (Zig is not stable yet), which are precisely the two languages that are used for most newer JS toolings. For Python you only have Rust, as you need C FFI. Maybe one day we will see more software in Nim, Crystal, and Janet, but they are far, far from mainstream.
What specific languages do you have in mind?
*and C and C++ but writing tools in them is like self-lobotomy.
>"...and C and C++ but writing tools in them is like self-lobotomy."
C is a lower level language than Rust so sure it is harder to write complex tools. Modern C++ - writing tools is at worst as easy than Rust and way better in practice because of a huge amount of libraries to suit every taste and C++ being more expressive / supporting more programming paradigms.
Statement like yours are not doing any service to Rust. Rust advocacy can do very much without such "help". Hopefully it does not represent culture of Rust community in general.
I am not advocating for Rust and don't care at all if these tools are written in Assembly or C++; as long as they exist and work well, I am happy.
When I wrote that statement, I didn't really mean C++ the language, but C++ the developer experience. I have tried to write a few command-line utilities in C++, and while the language proper is quite capable, dealing with CMake, package management, header files, and testing is just more headache-inducing than in Rust for me. Most of it probably stems from my lack of experience with C++, but it doesn't change the fact that despite C++ being 20 years older than Rust and having an order of magnitude more users, most of new JS toolings are not written in it.
Feel free to prove me wrong by pointing me to good tooling projects in C++. I actually want to learn more C++ best practices for working with another language.
I mainly use CLion by JetBrains for development. It does use CMake.
I can't point you for particular tooling C++ project. The only 3rd party source code I use are some C++ libraries. All it takes in my particular case is adding couple of lines in CMakeLists.txt file and I am very far from being CMake expert.
CMake is actually a perfect demonstration of how annoying C++ development tooling can be.
The documentation is massively verbose but doesn’t really say anything helpful. There’s basically no “Modern CMake” examples that anyone can agree upon. (I do see that there’s some GitHub repos these days that maybe do show some of it). There’s very little information on “The CMake way of doing things”. You really need to learn by examination, and your assumptions in the end will very likely be wrong.
Even just adding a package fetch in CMake is infuriating. What is the currently accepted way to do this? (I admit that I haven’t used CMake in a while, but as of a year ago, the internet and CMake docs did not agree on the correct way, even internally)
The Professional Cmake book is an indispensable, fantastic resource I've used on a couple of work projects, I highly recommend it to anyone working with Cmake.
That being said, Cmake is a far cry from the ease of use of cargo (albeit with potentially much more flexibility).
We must be living in a different world. It literally takes me seconds to create new executable project.
>"Even just adding a package fetch in CMake is infuriating."
I am not sure how to do it because I do not let build process / environment fetch things from the Internet. I always download / deploy / update manually upon real need. I would agree that having IDE do it for you is convenient but since I only use very few libs there is no need for me to do it at all.
If I need to create new project I just copy a template, fire up CLion IDE and it all takes few seconds from the - I am going to do it, to start actual coding.
Using external libraries is important because it lets you focus on what's important for your project. C++'s lack of a proper package manager means we're all reinventing the wheel because adding dependencies is non-trivial.
Cargo/Rust and Go show how compiled languages can have drastically superior development experiences. CMake is primitive in comparison. Its only upside is its flexibility, which I reckon isn't worth it for most cases.
A while back I needed to write a small library in C++.
It took me about the same amount of time to write few dozen lines of CMake (to get build working with all the external libraries), as it took me to write few thousands lines of C++.
No, but it was a rewrite of C# code. Using MSVC compiler, targeting Windows and Linux. C++20 with modules. External libraries: libsodium, lmdb, roaring & rocksdb. First time using CMake.
That’s fine, but most modern tooling “automates” this for you and gives you easy tooling to manage versions as well.
Additionally, you get a nice list in a simple spot of exactly what you need that isn’t just documentation. Anyone can easily pull your project and contribute.
Being able to easily manage deps has issues (such as a tendency to trend toward left-pad), but there’s also significant benefits as well that become readily apparent when you compare cargo with CMake (I understand they’re not completely comparable)
The Rust ecosystem is much more coherent than the C++ ecosystem, broadly speaking. Starting from cargo, which isn't perfect but is a fantastic tool.
As the main author of cargo-nextest, I'd challenge you to write a tool with a similar complexity and quality level in C++. I got so much leverage from being able to use the Rust ecosystem, including being able to port to new platforms with literally zero code changes. (Be sure to get the signal handling exactly right.)
> Statement like yours are not doing any service to Rust. Rust advocacy can do very much without such "help". Hopefully it does not represent culture of Rust community in general.
Ugh. You’re disagreeing about which languages are best for some task. Are either one of you being-an-advocate? That’s up to interpretation.
It’s not fair to say that someone who vouches for Rust is being-an-advocate while someone who vouches for C++ is just arguing normally. What basis is there for that?
I could say that you are not doing any service to Rust-naysayers. That I hope that the culture of Rust-naysayers are not represented by people like you. Would that be fair?
You specifically seemed to be saying "Why use rust when you don't want top tier performance", to which my response was "I do want top tier performance."
That aside, with regards other languages I'd consider candidates for high performance tooling, I'm either not proficient with them, or I don't enjoy working with them.
Andrew Kelley talks about this and the effects he’s trying to achieve with Zig tooling.
Undoubtedly, language tooling being fast improves your ability to iterate. 1 second vs 60 seconds doesn’t “seem” like much, but when you have that speed, it becomes very difficult to give it up, and when you know it can be faster, it is incredibly frustrating when it’s slow.
Iteration speed is crucial for shipping code fast. It can mean the difference between spending a day understanding and fixing a bug to spending 15 minutes doing the same thing.
Although I suspect that as your domain knowledge of the software improves, the effect flattens out.
Because as mentioned, it isn't a game engine fighting for fitting everything into 16ms into a games console.
Similar performance levels can be attained with any AOT compiled managed language, with much better developer productivity.
Either one needs to deal with the borrow checker for performance that matters, or if the performance doesn't matter to put .clone() all over the place, then what is the point beyond CV building?!?
Once you know Rust, the borrow checker is not impacting your productivity.
This leaves you with a fast language, with robust and relatively easy multithreading, and a rich ecosystem of high quality libraries. I don’t know any other AOT language that can come close to such productivity. The closest may be golang, but it’s not ideal for parsing.
AFAIK no other language has this kind of thread safety in the type system, enforced at compile time. The next best thing is having a particular share-nothing or message-passing paradigm built into the language.
Golang for example does not enforce use of synchronization when mutating shared objects from multiple threads, even though it can lead to crashes and vulnerabilities, just like in C and C++.
It may be hard to appreciate Rust's fearless concurrency until you see for yourself how it can save you from heisenbugs. You can invoke any function, which could be even a 3d party dependency using more dependencies and touching a million lines of code, and Rust will tell you right away if any of that code has some thread-unsafe behavior (e.g. some method may have internal cache without proper locking). With the safety net of the borrow checker you can use constructs that would have been considered irresponsibly dangerous in other languages, like spawning threads referencing on-stack values of other threads.
>"It may be hard to appreciate Rust's fearless concurrency until you see for yourself how it can save you from heisenbugs."
Writing multithreaded native applications is my bread and butter. Including enterprise grade application servers with multiple clients and various IT systems accessing it. I can't recall when was the last time I had any problems caused by improperly mutating data.
Mozilla, a project full of C++ experts, tried to write a parallel CSS engine in C++ twice. They couldn't get it working either time. But their first attempt at using Rust worked out -- even though many of the devs working on it were completely new to Rust.
>"Mozilla, a project full of C++ experts, tried to write a parallel CSS engine in C++ twice. They couldn't get it working either time."
The fact that Mozilla can not do a particular thing in particular way has zero value for others not in the same boat. I am not Mozilla. Do not know their problems and do not care. Maybe parallel CSS is too complex by nature. I can only relate to my own situation. My commercially deployed systems work and serve bazillion of clients with no problems.
What weight do you assign Mozilla's data point? It isn't rational to say zero, is it? Also consider that Chromium still doesn't have a parallel CSS engine -- my understanding is that they tried and failed as well.
I've also written systems in Rust, some embarrassingly parallel and some that are more complex than that. The embarrassingly parallel ones never took more than 10 minutes of effort to do. The more complex ones are harder but still possible.
I agree that your data point should also be assigned non-zero weight.
It is well above zero. But this is for very specific situation - parallel CSS.
I on the other hand had never had problem managing / mutating state in my multithreaded programs. I did have one situation with deadlock when interacting with Directshow signal processing. That was around 2002 I think. Took me one day to analyze and fix the code after the bug was reported. But Directshow is an incredibly complex piece with some weird behaviors. Other then that I do not recall any real problems. Btw that particular software was desktop application and written in Delphi, not even C++.
So yes to me the opinion of Mozilla's team is totally irrelevant. It is of course totally mutual but I suspect that my situation - C++ application servers is way more common than parallel CSS.
It is much easier to write complex multithreaded code (not just data-parallel) in Rust than in any other language. And data-parallel code is also easier.
If you don't care about the best possible performance you can (and so most people should) just do whatever is easiest to reason about and not waste your time trying to figure out how to satisfy the borrow checker.
But also yeah, Rust is just a really nice programming language, I prefer to write Rust so obviously I'm going to write software in Rust and that's likely true for others too.
Doesn't Rollup already use quite a bit of rust[0]? It's actually why I had to abandon it for a project, where they didn't offer binaries for our build platform and I needed to bundle, like 2 ES6 javascript libraries so I just grabbed esbuild instead.
An illumos distribution which is supported by rust, and the rust stuff would successfully build but `npm install` wouldn't because they didn't provide binaries for that platform and it was much faster to just switch to esbuild than figure out all that was needed to get it supported.
I only tried rollup from the beginning because the ES6 project I was trying to bundle suggested it.
Mostly curious, why do you care about the readability of the generated JS? Surely if you need to debug something for a production bundle you can use source maps.
Rollup produces an AST (with acorn) then it manipulates the original source code as a string with MagicString which is a less than ideal hack for code transformations. Will Rolldown eschew the MagicString approach altogether and transform the AST directly? The advantage would be that the emitted code would not have to be reparsed again for downstream use.
JS tooling is mostly single threaded. It is expensive to stringify a large AST and then parse it again over IPC vs just transferring an object (which is not allowed by runtime). This cost incentives more things happening in the same process, so there’s a limit to how fast you can be with JS
There was a moment a few years ago when steelbrain's pundle was the only tool I knew of with proper instant hot module replacement.
It went the way that many one-man projects with no community backing so, which made me sad. I moved to Rollup, and then recently moved to Vite to get back on the instant HMR train.
Thanks for your work in this space, steelbrain! pundle was awesome!
Their about page (https://rolldown.rs/about) describes why they want to do this, but after reading it I'm still unsure why they can't accomplish their goals by adding features to esbuild? Maybe the project goals are too different?
esbuild is and has always been governed strongly by Evan Wallace. I can't vouch for whether or not the Evans (both Wallace and You) talked to each other about it, but I'm almost 100% certain Wallaces' vision for the esbuild project is different than what rolldown is ultimately trying to achieve.
In some respects, rolldown is also the next generation of Rollup (potentially) and is a direct answer to rspack[0], which is a webpack compatible rust based toolchain from ByteDance
One of the interesting things about rollup is how it compiles code. Instead of building each module as a closure, separating the scopes, it compiles everything as one big closure. This is actually faster to parse and run, and allows you to do much better tree shaking and other optimizations.
If you're doing all that, you could probably support rollup plugins as well, which it sounds like they want to do.
In many ways its not an alternative, in particular, SWC does not handle resolving and other bundler specific concerns, and dropped the concept of JS interop with plugins (its all rust built WASM now).
I also blame Vercel, they have put zero investment in documenting SWC. The person behind the project used to be more prolific in documenting things, and that all stopped once Vercel scooped them and the project up for use in turbopack (which has not materialized as a standalone solution like they promised, funnily enough)
I'm not sure how the projects ended up where they are, but Oxc which this relates to has a very specific vision of how to improve the experience for the JavaScript tooling ecosystem which I'm not sure SWC shares. https://oxc-project.github.io/docs/guide/introduction.html
I have used SWC internals recently for a toy project of mine and was generally let down a bit by some of the rather unwieldy internals, insufficient documentation and pretty hefty footprint.
Example: you use a large plotting library, but it's only needed on some pages. Chunking allows you to selectively load that code only where it's needed.
Great! Hopefully this makes using dynamic import easier when you say want to chunk your language dependencies and only import them as needed during runtime. As I just tried this with Vite it was quite a hassle and I decided it was easier to just import all async after initial render.
I’d love to get to a point where “dev mode” is just “build mode plus”, as in the build is identical except for additive things like automatically launching a local dev server, some notion of live updates/HMR that doesn’t require different dependency sources, dev specific affordances as environment flags rather than separate modules…
For my own (edit: as in personal) projects I tend to just “dev in build mode” with some subset of that bolted on, because I’ve been burned so many times by “works in dev” issues.
I actually can't understand javascript bundling, am I just dumb? I try to build a monorepo and every single time I get errors building my nodejs apis because of esm / cjs module bundling.
Why do I even need to bundle a nodejs api? My current project's backend is completely in Rust because I actually found that easier to deal with. Kubernetes was easier to figure out for me.
Bundling (including dead code elimination) is more used for frontend code to reduce the amount of JavaScript sent over the wire to browser.
You don't need to bundle server-side code unless you're using a cloud provider that limits how much source code you can upload or some other limitation like that.
More importantly, how is this being funded? I find it hard to believe that core team is running off sponsor dollars. Is this project going to be around next year? Maybe it's coming from Vite sponsorship backers.
Its a Vite project and very much the same core team. This is another Evan You lead project, which gives me alot more hope than I normally do for these kinds of things, he has a knack for generating long term sponsorship of projects
