TLDR: The attacker injected an attack code as coa's `preinstall` script, which executes an obscurely-named file ("compile.bat"). This file is fully obfuscated, but what it does is basically to pull exploit DLLs from the attacker's server and install 'em.
I think the fortunate part of this accident is that the attacker failed to deploy the malware in his/her first attempt; v2.0.3 only contained the half of the changeset that the exploit needs to work (which accidentally broke tons of CI builds); So some developers could notice that something is wrong a bit early.
ahhh nice, it's been years since i've seen a obfuscated .bat!
very nice use of substring, but a bit too linear... with some input redirects and nested spaced variables it would have become more robust and unpredictable, but i suppose nowadays batch chiselers are rare.
edit: by the way in the article is missing a -useless- decoded line (n.4)
The thing that should be causing concern is not so much these very loud obvious attacks, but how many better attacks that are harder to detect, are currently happening.
With 1.7M packages and an ecosystem that favours lots of 3rd party package usage, NPM is a large target. Whilst NPM isn't the only repository to have this kind of issue, it's definitely the largest attack surface.
Reminder that people should seriously consider disabling the install-scripts.
Personal system-wide config:
npm/yarn config set ignore-scripts true -g
and add & commit a .npmrc/.yarnrc file with
ignore-scripts true
Yes, this will cause headaches in some (increasingly rare) cases where some package actually needs those scripts. You can fix this with custom install scripts that take care of running install for those specific packages.
And yes yes, as people love to point out, this isn't exactly a bulletproof solution either. The attacker could just put the malicious code inside the package's code and wait for it to be actually executed. But again and again, they don't, they choose to use the package's install scripts as the place to do their dirty work.
So in practice this policy would've alrady protected you from who knows how many of these attacks, and my guess is that it'll continue to do so.
And to clarify, you should run this once by hand to collect the list, don't add the grepping to the actuall installation script or you are back to square one :D
Here's what I use with yarn, install.sh:
#!/usr/bin/env bash
yarn
function run_install() {
local dependency="node_modules/$1"
if [[ -d "$dependency" ]]; then
yarn --cwd "$dependency" run postinstall
fi
}
run_install 'esbuild'
run_install 'other_package'
...
NB: Setting “ignore-scripts” globally means you can’t even “npm run” scripts manually, for packages you’re authoring (or at least that appears to be happening for me).
Edit: Actually, on NPM 8.1.0, setting "ignore-scripts" globally still allows you to run scripts manually with "npm run".
The behavior I described (where it just blanket ignores ALL scripts, even ones you try to run) existed on NPM 6 (which is what I was using until a few minutes ago). I couldn't find the exact version where this behavior was fixed.
A separate advisory says the npm package "rc" is also compromised. That's also a highly popular one, according to the npmjs stats (1,323 dependents; 14.2 million weekly downloads).
This makes me appreciate Deno's focus on security. Having things like file and network access 'opt in' seems like a no brainer when we see how easy it is to simply install an npm package and find yourself vulnerable to malware.
Any protection offered by Deno isn't robust. Only OS-level protection is secure. Any tooling will require access to file system - and file system access is enough to compromise developer system.
Let's assume that you have Deno compiler for other language. You run it through seemingly innocent deno run "https://..." --allow-write=. src/ (you use optional parameter to --allow-write, right?).
Unfortunately, webpage hosting script was compromised. Now our compiler can write to .git/hooks, .npmrc (npm can do arbitrary script execution in version 6 or lower, even on npm --version), .idea/ etc.
Are other languages/runtimes also that risky as Node with npm?
npm packages seem like a cardhouse.
I know that the node_modules folder is often times criticized for its sheer amount of 3rd party libraries. Is it because of JavaScripts "missing" standard library?
I will be downvoted to hell for saying this. But javascript ecosystem is where most newbies come. (Low barrier to entry and it also seems hip). With no regards to security, maintainability or reliability, fashion chasing blog-happy hipsters.
The amount of churn in JS ecosystem, security incidents like this and general crappiness of websites can be generally explained by how immature these hipsters are.
Not disagreeing with organization around frameworks in PHP, but apparently PHP suffered a lot from low barrier, in terms of security especially. That was the time package management wasn't that widespread yet, which IMO limited this kind of stuff. But there were, for sure, many applications where PHP could be blamed for security incidents and general unreliability. PHP improved quite well though.
Now PHP isn't hip anymore, node js is super popular hip thing, and every tom dick and harry from art school in US or 3rd tier engineering college in India will slap together three todo list applications on Resume and wants to call himself full stack developer. Internet is fast, hardware is fast, no one cares about pile of dependencies sitting beneath them.
Add to that resume driven development where every JS wants to write libraries and become github-famous. 30-line libraries will be considered a joke and it will be shameful to brag about such things in any other ecosystem.
In 2019, 11% of all vulnerabilities listed by the National Vulnerability Database were linked to PHP; historically, about 30% of all vulnerabilities listed since 1996 in this database are linked to PHP. [1]
Also early this year there were news that git.php.net was compromised and a backdoor was introduced into PHP, but lucky enough the backdoor was catch before a production release.[2][3]
Npm is pretty unique in the low bar it sets for security. What is really frightening is how these cases are discovered, more or less by accident, rather than by some kind of verification process that ensures this simply can not happen before QA catches it on the way to a release.
What happened there was that he got the broken update, 2.0.3 which just referenced and used compile.js, but didn't include the file.
Then 2.0.4 came out which included compile.js and compile.bat.
Had he updated a couple of minutes later, this error would not have appeared. Not sure if /Users/ is a MacOS thing, but it is a Windows path structure, which might indicate that he was running this on Windows. And in that case he would have been compromised.
What a worthless advisory, how about sharing who could possibly be affected at the very top, or at least anywhere?
Going to the issue, it seems the `preinstall` field was changed to `start /B node compile.js & node compile.js",` which means this would only run on Windows machines, everyone else seems to be unaffected.
Here is how you can find out if you have the affected package on your machine/instance:
That post doesn’t say much about `coa`, besides “new versions started appearing and builds started failing”. The bug report linked from GitHub advisory does a good job of describing the issue, though: https://github.com/veged/coa/issues/99
So if I’m reading this correctly, only Windows hosts are directly affected by the malware. On macOS and Linux one only needs to rollback to a healthy version of the package?
It seems that all of these should be cryptographically signed by a developer's private key before publication and then verified by others before use. Is that not the case?
I agree that all software package repositories -- NPM, Rubygems, PyPI, Maven, NuGet, Crates, etc -- should have two minimum baseline security policies:
1. All accounts are MFA, no exceptions.
Only CI/CD usecases justify some laxity. Only push tokens should be non-MFA-able for CI/CD purposes, and they should only be usable for push. Tokens should only be obtainable with MFA.
Ideally these too would be part of a multi-step OAuth2 or OIDC refresh token to access token flow, so that any given access token is only used once and CI/CD jobs never get to see the refresh token.
And while we're on tokens: they should all expire, after a short period of time, without exception.
2. Signing packages is opt-out, not opt-in.
The two reasons package signing is relatively rare are that (1) in most ecosystems it is unwieldy, (2) not actually that effective at providing guarantees (self-signed certs are no better than a pinky promise).
But once these are solved, and I believe they can be, making signing opt-out means that we'd see signing rate percentages in the 90s, instead of languishing in single digits as they do today.
If you work on NPM and would like to swap notes, I would love to talk to you with my professional hat on -- email in profile.
There are just too many micropackages to properly look at. A basic react app created using official method (CRA) was 200MB something last time I tried.
(Of course this malware was in a preinstall script, which should also be disabled... but any module you import in a node app can do bad things when you run your app, preinstall script or no.)
If I'm reading this correctly, the malicious code was new (higher) versions of the releases.
Would this mean any project using a package.lock/yarn.lock was 'safe' going through deploys? So only new installs and builds without lock files could have grabbed the higher version?
If so, I wonder if it's hard or impossible to swap a release version on NPM. Seems like that would hit a much wider audience before being detected.
My read of the headline was that this was malware in embedded hardware electronics systems, or describing some exploit / attack surface for same.
May I suggest that a clearer phrasing would be ‘Malware embedded in Coa’? Or is ‘embedded malware’ a somewhat confusing term-of-art in the cybersec community?
I know npm feels like wild west, but you can audit. Its quite a challenge to review imany of the C, C++, libraries out there that are just a .zip file stored in a website.
My point is that: Npm is auditable, trackable. I'm not challenging the bug itself, neither the security issue..
The receiving server would trigger it. The first review should be done by the owner, who should be contactable via email and authenticate via 2FA and acknowledge the modification.
Then I'd guess that Microsoft has enough information with NPM's history to train an AI. Specially the modifications made in these versions could easily trigger suspicious activity.
Did you look at the diffs?
Also, three years of inactivity and then a sudden upload should easily trigger a manual review, even if it is by automatically opening an issue with a review request on that project's GitHub page.
I don't see how this is an NPM issue at all. If the attacker made a Linux version and stuck it inside a Makefile, plenty of devs and build servers would end up executing the malicious code.
Yes, but good luck getting kernel.org to distribute that. They're not perfect (nothing is, and kernel.org has been compromised in the past, though quite long ago, and even then it did not lead to compromise of the Linux kernel sources), but to put the Linux kernel maintainers on the same level as the people maintaining the NPM distributions is a bit much.
On npm, the average application uses hundreds (or thousands?) of dependencies, none of which are audited by the application author. On Linux, ffmpeg uses dozens or hundreds of dependencies, but they're each trusted and audited by the Linux distro packaging the dependency packages, who would hopefully catch malware shipped in a Makefile. And whatever vendored libraries a C/C++ app bundles (so you can build it on distributions that don't ship the library or a new-enough version) are usually self-contained rather than having transitive dependencies. As a result, a C++ app's author count grows linearly with the number of dependencies you bring on (and you generally evaluate each author's trustworthiness, and hopefully skim over the library and build scripts as well), rather than polynomially in the dependency tree's branching factor and exponentially in its depth (you generally don't evaluate the trustworthiness of your transitive dependencies' authors, since there are so many and they're not shown to you when you add a dependency).
Additionally, an app's vendored dependencies don't automatically update until you choose to update them. And when you update a vendored dependency, you have a chance of finding malware when you review the diff before committing the update. The downside is that vendored dependencies introduce friction in updating your dependencies (I've experimented with extracting tarballs, git subtree, and git subrepo, and generally avoid submodules, and they're all somewhat awkward), and don't automatically receive patch/security updates like distribution libraries do.
I'm more familiar with crates.io and PyPI than npm, but I think they all work similarly (unvetted package repository allowing libraries to depend on other libraries, and transitive dependencies get pulled in when you install the library, and can be updated without the library author's knowledge). The primary difference IMO is the degree that packages use transitive dependencies vs. building/copying the subset of code they need (the average Python app uses dozens of dependencies, Rust apps use hundreds, Node apps use thousands).
My wish is that for each language, a team of Linux distribution maintainers or community members would "sign off on" a set of trusted packages, each of which depends on only trusted packages. And when a trusted package gets updated, the new version isn't marked as trusted until the maintainer verifies the diff isn't malicious. Then the language package manager would add a mode where apps or libraries can choose to only depend on trusted packages (which only depend on trusted packages, etc.).
(In the Rust world, where the compiler and package manager are developed by the same non-commercial organization, I hope the Rust organization will create the team handling "trusted packages"; sadly they seem opposed to it because they consider it favoritism, but IMO it's a good thing for Arch to package wget and curl, but not "joe's 1337 haxxor tool". In the Node world, V8 and node.js and npm are different groups/companies I think, so I don't know which organization is community-led and neutral enough to make a trustworthy host for this effort.)
Crev takes the "web of trust" approach, where everyone can choose who to trust. However, it doesn't ship a "deny untrusted dependencies" mode as part of Cargo. And as with PGP webs of trust, I haven't met any library developers in person, don't know which developers sign off on packages, and would prefer having semi/centralized institutions (like Linux distributions) who stake their reputation on the non-maliciousness of packages they ship, and allow different people to agree on which packages to trust. Having institutions reduces the responsibility placed on the individual app developer. Right now, you need to trust an distro to take responsibility for an OS, and a language team to take responsibility for a compiler. With crev, you'd have to also search for trustworthy people who've reviewed enough package that you can feasibly build apps using only those dependencies, instead of having the distro or language team provide that service out of the box.
Maybe I'm just too lazy to find trustworthy people. But currently there are not enough Crev reviews at all. Searching through https://web.crev.dev/rust-reviews/crates/, there's no mention of gtk and pipewire, and rusqlite and sqlite are reviewed but not libsqlite3-sys. And if I were to start writing reviews, would people trust me?
My hope is that Linux distributions will enforce a policy of only packaging applications where all dependencies are trusted packages (on top of reviewing the app's own codebase as usual). As difficult as this may sound, this is already the norm for C/C++ applications, where it's easy to find either distribution-packaged or self-contained libraries, and there are organized teams of maintainers selecting trustworthy packages, auditing and packaging libraries based on user demand, and telling audio libraries that require 70 packages to bind to PipeWire to clean up their act. Unfortunately, in languages where package managers and communities have embraced unvetted transitive dependencies, it's difficult to even find libraries with small or trusted dependency trees. This normalization of deviance has seriously harmed our ability to understand and/or trust the code making up our apps, and it will take changes in our package managers and community norms to restore trust.
I know npm feels like wild west, but you can audit. Its quite a challenge to review many of the C, C++, libraries out there that are just a .zip file stored in a website.
My point is that: Npm is auditable, trackable.
I'm not challenging the bug itself, neither the security issue..
I would say that it's not that different from others I've seen, just more visible because of the size and activity of the repository.
One thing NPM does (and I believe Python too) is to allow install scripts -- this has been a reliable vector for attackers to steal credentials. Not every package repository system has that.
authors cannot revoke their compromised keys to immediately halt all distribution, and you don't have any process to verify package<->author ownership beyond the upload secrets.
What exactly is the malicious code? I assume it's in `compile.js` and only can be found in published (now removed) npm package instead of source code repo?
It gives you a good glance how likely it is that something is a threat - if an overwhelming majority of AV products detect it, it's quite likely that it's bad, and if only a few products find it, it's either very new or really just a bunch of false positives. Like many things, there's no simple "yes" or "no" answer, thus interpreting Virustotal output still requires some literacy on that topic. The site can also help software vendors to find which AV products reject their software, so that they can file false-positive requests to those AV vendors.
The most common ways these things seem to happen is either password reuse with no 2fa or that the npm token (in ~/.npmrc) was harvested by another compromised package/program. IIRC there were a few that were due to phishing too.
Almost certainly one of these. It's not a typosquatting attack, since it's an existing package. And it's not a repository compromise, since they had to create new versions instead of silently altering an existing version.
https://github.com/veged/coa/issues/99#issuecomment-96153687...
TLDR: The attacker injected an attack code as coa's `preinstall` script, which executes an obscurely-named file ("compile.bat"). This file is fully obfuscated, but what it does is basically to pull exploit DLLs from the attacker's server and install 'em.
I think the fortunate part of this accident is that the attacker failed to deploy the malware in his/her first attempt; v2.0.3 only contained the half of the changeset that the exploit needs to work (which accidentally broke tons of CI builds); So some developers could notice that something is wrong a bit early.