God I hate multi-threading on the web/nodejs. Rather than implementing syncronization primatives like mutexes or rwlocks that capture the contained values and make them "transferrable" between JavaScript contexts (v8 isolates) - they introduced SharedArrayBuffer that is almost entirely unusable for anything meaningful.
Syncronizing between threads involves thunking and copying data through RPC layers.
Sucks for me because our production app has grown faster than we are able to rewrite it and uses 70-100gb of ram (written before my time). To try to get around this, we are investigating exotic solutions like using native code to manually manage pages of shared memory that contains custom data structures and as little serialization/deserializing logic as possible - but of course v8 uses utf16 for string encoding which means working with JavaScript values in the native layer is expensive.
It's actually an internal build tool (forked from an open source tool) used to build our public applications and runs in Nodejs. We are rewriting it in Rust but it's a complex tool and that takes a lot of time.
In the mean time, a "build" using this Nodejs based tool takes about an hour and uses 100gb of RAM.
By contrast, preliminary unoptimized results of the Rust tool brings build times down from 1 hour to 1 minute and maxes out at 6gb of ram - but that'll be finished in 2 years.
It literally costs millions per year to run and is a nightmare to maintain.
To be fair, the JavaScript itself is actually surprisingly performant - it's the multi-threading that's the real problem.
It would be a non issue if there were memory sync primitives that could be sent between threads but because of how v8 isolates work, this isn't possible without changes to v8. So we are forced to copy gigantic graphs of data between threads and manually synchronize them with postMessages.
Running a build on the main JS thread (no workers) uses 10gb of ram and takes 5 hours. So shared memory would solve our issue and avoid the need to rewrite it.
Attempts to centralize the data (in a DB or similar) perform worse due to the de/serialization of the data. Other attempts of using a shared in-memory database shares the same issue. We also need to modify every shared structure in the codebase to use getters/setters with a copy on read approach (so memory blows up any way).
It's not that JavaScript itself is the wrong language, but that the isolated nature of the runtimes fail in multi-threaded use cases.
I find the gaps in the Rust standard library annoying. The batteries-not-included design just doesn't do it for me, especially when it's for commonly used things such as the guid type and common interfaces.
That's fair. The std lib is the biggest pain point for me also but it's easy enough to work around. That and the poor ergonomics of async/await/futures
Bundler engineer here. ESM is great when it comes to build-time optimisations. When a bundler runs into CJS code it literally deopts the output to accommodate - so from that side it's amazing.
But also, there's a special place in hell for the people that decided to add default exports, "export * from" and top level await.
Commonjs is also very weird as a "module" instance can be reassigned to a reference of another module
module.exports = require('./foo')
and there's no way to do this in ESM (for good reason, but also no one warned us). In fact major projects like React use CJS exports and the entire project cannot be optimized by bundlers. So, rather than porting to ESM, they created a compiler LOL
If I may, the evil is not in the top level await but in the use of a top level await in a module that exports. That is evil. But a top level await in a programs main script seems ok to me.
As a kiwi living in Australia and working for mini FANG, the thought of moving to Southeast Asia and living off savings for a couple of years in the hope that I might wait out this jobs/rental crisis has crossed my mind.
Not necessarily a bad plan per se, but don’t forget to take into account the fact that this jobs/rental crisis might be due to deep systemic+geopolitical issues that are only going to get worse and that we might not be going back to the abundance of the 2010s anytime soon.
Finding yourself in an even worse job market 5 years from now is a very real possibility.
I think it is alleviating slowly. In Melbourne at least, rent seems to have peaked already. The state government has done great work in overruling councils to get more housing built faster. And it seems to be working.
To be honest, the perfect solution and you have a high chance of finding something there that makes you happy.
Housing there is more affordable also but not for long, people are buying insane houses there coz cost "nothing".
If this is something that you already considered and have enough saving to do so, personally, I would start looking for what can I do over there job wise to avoid using all my savings and screw it, I am off.
No matter how small is the income coz you will need to start from the bottom, I am getting tired of IT, too much drama, all the time, if living in Asia doing something completely different but that pays the bills and that I am happy with, I wouldn't come back.
Australia within the last 11 years went to shit big time, and it is getting worse by the day.
I have considered the same, I think it might be a little to late for me now, who knows.
> Australia within the last 11 years went to shit big time, and it is getting worse by the day.
Most major wester economies did the same in past 11 years: stagnating wages, exploding CoL. Life was good 11 years ago when salaries were great and housing still abundant and affordable to buy but this isn't the case anymore.
If you expect an improvement, moving anywhere there right now would be like jumping from a lake to a pond.
Housing in Australia was not affordable at all 11 years ago! Perhaps 20 years ago yeah… but I mean, compared to almost anywhere else, salaries for the average folk are great even when you compensate for that. Only the top tier is underpaid if you compare with the USA, much like Europe.
It's so funny to me how you Australians have no idea how fortunate you are. You've been born into one of the luckiest places on earth and never stop moaning about it. When I lived there all anyone ever did was bitch and moan while pulling in huge salaries, soaking in some of the best weather on earth and knowing their country's immense mineral wealth and isolated status would always be a security blanket.
Truly some of the most fortunate, least deserving people on the planet. I am thankful I can get my special category visa and when my parents eventually pass away I will set my roots down in Australia. Despite all of the Australians.
I'm interested where in SE Asia you think you can can just barge in, buy a property and live. It might pay to actually look into what it takes to do that legally, it might not be as simple as you think.
WSL1 got my hopes up that we were on the path to Windows supporting the Linux user space API but then it was cancelled and replaced with a virtual machine based solution that I didn't need WSL2 to implement myself (with more flexability and capabilities).
I'd much prefer a proper compatibility layer that converts Linux system calls to their equivilent Windows calls and those calls be exposed from the Windows kernel itself.
That way I could just run Linux applications, bash, zsh and development tools directly on top of Windows without needing any "remote development tools" in my IDE or whatever.
Something closer to MSYS2/git bash/busybox for win - but where tools can ignore their Windows specific stuff like the filepath seperator.
I'd love to see the EU aggressively expand investment in the Linux desktop. I'd absolutely drop my job and jump on FOSS projects if there was financial incentive to do so - I'd even take a hit on compensation to participate.
Windows is making people annoyed more and more (saying this as a guy who grew up with it and is still not anti windows) and with the recent trump 2 / use-european-tech .. it might be a clear enough narrative for people to try linux distros. Fedora 41 on a 2015 laptop is lean and cute.. enough for most normies IMO. It could be a boost for everybody, no need to buy a new laptop, simpler times without Microsoft pulling shenanigans every year, more open source market share.
Yes, the are making consumers more and more annoyed. The are not making enterprises annoyed. All the bloat and ads that we hear about again and again is not present in their enterprise offering.
Same, I'd love if the EU sponsored the development of open source alternatives. I'm tired of the American tech industry and how it's towing the (anti-European) party line - just give me a way to make a living while building competitors!
What they already contribute is amazing, honestly. My concern is that, in order to build a genuine competitor to Windows, investment needs to increase substantially.
As it stands, getting into kernel development or working on something like a desktop environment (or adjacent software like Firefox or Chromium) is quite difficult in a way that supports you financially.
I mean, as individuals, we still invest personal time into public domain projects because we love to tinker and build stuff - but sadly we can't afford to do that full time.
The way Linux kernel stuff works, is you do a Linux internship with GSoC or Outreachy or LKMP , and then core Linux contributing companies like Intel, AMD etc gobble you up. Or you join FOSS consultancies Like Igalia, Bootlin, Collabora etc. Or you go the rockstar Patreon way like Asahi folks, trading off money for independence.
This is where governments need to step in. We need grants to open source contributors as a national security priority. If you're helping build alternatives to American Big Tech, the state should sponsor that.
Yeah, nah. My M1 is basically dead to me for gaming. There are a _few_ games that run _okay_ but I now just carry a second computer for gaming. Salty because it's such a great computer nerfed by walled garden business practices.
but also WSL2 doesn't add anything of value to the development experience on Windows that wasn't already possible with a VM.
For my use cases, I just want to target unix-like APIs and use bash-isms so I can reuse my build scripts and tooling. I don't really care if the binary format I consume or compile to is different to Linux - as long as my development experience is identical across MacOS, Windows and Linux.
A thin layer on top of Windows that mimics Linux so I can run bash _properly_ is all I really need.
The closest I've come is using Msys2 with zsh and uutils and it is so close but there are still cases where it simply doesn't work. WSL1 was pretty close but it falls short by needing remote development tools and having isolated storage/poor performance interacting with host storage.
WSL2 is DOA for me, I just hand roll my own Linux VM and call it a day.
The thinness is actually part of the problem. POSIX and Windows APIs don't work like each other.
For example, if you were a Unix archiver extracting a file, you'd call stat() open() write() close() chmod() utimes(). On Linux/BSD that's 1 file open/close, but on Windows, that's 4 file opens/closes because the Windows APIs need an open filehandle, while the POSIX APIs take a filepath. Linux/BSD have LRU caches of filepath->inode because they're designed around these APIs, Windows doesn't. Cygwin has to open/close a Windows filehandle each time because it can't make the calling code pass a filehandle instead.
So it may be more comfortable and familiar, but also Windows APIs are unsympathetic to code designed for Unix originally.
I do think that talk is way too kind to the windows design. they're trying to make the argument that windows filesystem isn't slow, but the talk is how fixing the problem took 3 months of optimization and direct help from Microsoft. all of this could be avoided if Microsoft made their filesystem not ridiculously slow
File system filters are pluggable kernel drivers. For example, ProcMon (Sysinternals tool) monitors file systems via a file system filter -- so fun fact, if you outright disable file system filters on a ReFS volume, you won't get any ProcMon results! This was a 'duh' moment for me.
I find the filesystem and network integration to be a lot nicer than I what I get from a Virtualbox VM. Having the WSL system able to see the host as /mnt/c and also have the WSL filesystems appear in Windows Explorer is pretty darn convenient.
I know conventional VMs do this kind of thing too, but I've always found it more awkward to set it up, you have to install special tools or kernel modules, etc. With WSL it just works out of the box.
pulling up terminal running WSL instead of running a VM is a superior experience to me when all I need is terminal coding and python/bash scripting without having to block off a chunk of RAM for a virtual machine
I'd love to see more competition in the OS space. I feel that a big reason there isn't more competition is the gatekeeping of drivers by hardware vendors and the locking of bootloaders.
Why can't I install Linux on a Snapdragon laptop? Why can't I install OpenWRT on my router? Why can't I install Linux on my Pixel? It's certainly not because they are not capable - it's just that the vendors don't want you to (for one reason or another) or decide not to support a platform without giving users the ability to support it themselves.
With the liability that is an over-reliance on American core software (OS, firmware, etc) - I believe the EU should mandate unlocked bootloaders on electronic devices and force vendors to either distribute driver sources or provide enough documentation for bootleggers to write drivers.
This is simply "the right to repair" extended to the software component of hardware.
Although for Europeans this is probably more like "the right to not have (potentially hostile) foreign countries backdoor all your devices". Whether it's Microsoft spying on you or Samsung preinstalling unremovable spamware, I'm all about making the world a better place.
Syncronizing between threads involves thunking and copying data through RPC layers.
Sucks for me because our production app has grown faster than we are able to rewrite it and uses 70-100gb of ram (written before my time). To try to get around this, we are investigating exotic solutions like using native code to manually manage pages of shared memory that contains custom data structures and as little serialization/deserializing logic as possible - but of course v8 uses utf16 for string encoding which means working with JavaScript values in the native layer is expensive.
reply