Hacker News new | past | comments | ask | show | jobs | submit login
The xz attack shell script (swtch.com)
605 points by todsacerdoti 10 days ago | hide | past | favorite | 313 comments

As a longtime autotools-hater I would say this justifies my position, but any build system complex enough to be multiplatform is going to be inscrutable enough to let somebody slip something in like this. But it really is a problem that so much software is built with what's essentially become a giant cargo cult-style shell script whose pieces no one person understands all of.

I think the common usage of bash and other languages with a dense and complicated syntax is the root problem here. If build scripts were written in python, you would have a hard time obfuscating backdoors, because everyone would see that it is weird code. Bash code is just assumed to be normal when you can’t read it.

I think the issue is that build systems are often build on a completely different language - python with some weird build framework is likely as inscrutable as bash and autotools to someone who doesn't use python.

You can write pretty straightforward readable bash, just as I'm sure you can write pretty gnarly python. Especially if you're intentionally trying to obfuscate.

Man, this sounds right, but I dunno ... I feel like even "simple" shell scripts tend toward more inscrutability than all but the most questionable python code I've ever written. Just comparing my dotfiles - every bit of which I understood when I wrote them - to the gnarliest sections of a big python app I work on ... I just really feel like at least some of this inscrutability issue can truly be laid at the feet of shell / bash as a language.

Shell is evil mainly because it's so old. It's largely unchanged since 1978 and had to run on incredibly resource limited machines. What it could achieve on those machines was amazing. Tradeoffs would be different today, e.g. PowerShell.

A big pet peeve of mine is that shell is written off as evil but the only reason I ever hear is basically a variation of “I don’t understand it, therefore it scares me.” The reality is, unlike _really_ old languages like COBOL or RPG, bash is still literally everywhere, it’s installed on practically every linux machine by default which makes deploying a shell script completely trivial. It’s certainly under appreciated and because it’s ubiquitous, widely used in build processes, there’s a responsibility to learn it. It’s not hard, it’s not a wildly complex language.

I don’t think these issues would necessarily be solved at all by waving a hand and replacing it with similarly complex build tools. Bazel, for example, can be a daunting tool to fully grasp. Any tool used should be well understood. Easier said than done of course.

If a language has more footguns than guns-pointed-at-not-my-feet, I consider that a problem with the language.

And shell has a lot of footguns.


As long as you apply the same standards to what seems to be everyones darling: Javascript.

Javascript has the same amount of footguns as PHP and Bash but has gotten away with it by being cute (and having a whole menagerie if support tools around it to make it possible for ordinary people to write workable code in it).

(Yes, I am qualified to rant about Javascript BTW. I wrote a working map rendering system with pan and zoom and automatic panning based on GPS location using ECMAScript and SVG back in the spring of 2005. I think roughly half a year before Google Maps became public. Back before all the modern JS tooling existed. All I had was JEdit with syntax highlighting. Perl at least let me put breakpoints in my code even back then.

And yes, I have written more JS since then.)

In the context of build systems and the vulnerabilities that exist in xs, one of server-side JavaScript’s biggest footguns that cannot be ignored is its dependency management. Very few people I know ever really dig into the dependency tree and audit all packages 10 levels deep. The attack surface there is huge and objectively much wider than PHP/Bash. It’s also a built-in automatic entryway into a corporate network.

I specifically avoided web development because of Javascript and its plethora and ever-changing set of frameworks. So IMO, it absolutely counts.

FWIW it has actually become better the last few years:

Now you can at least just stick to React and TypeScript and bundle it using Webpack and have months of relative sanity between each time you have to throw something out and replace it.

Shell has a lot of stuff going for it too, though:

-> REPL essentially for free (the language IS a REPL)

-> enormous installed base

-> No compilation (well, unless you have something--like autotools--using shell as essentially a transpilation target)

-> No need for "libraries" in most cases: the ordinary CLI that $vendor already ships can be used right away, with no need for a custom SDK for whatever "real" language you would otherwise be using. For example, if you are already familiar with the "aws" CLI program, it's trivial to treat it as an "API" for a quick shellscript instead of needing to dig into the boto3 docs to do something equivalent the "right" way.

-> Pretty good integration with standard *nix facilities (redirecting STD{ERR,OUT}, checking existence of files/pipes, checking/setting exit codes, etc.)

I’d argue that every language is loaded full of foot guns. If you’re encountering those foot guns on a regular basis, it’s an issue with the author.

That said, what can help drastically here are well-defined best practices and conventions built into the language which, admittedly, bash really doesn’t have.

Can you point to 3 python foot guns, for instance?

I’m generally in agreement with the anti-bash camp, but I can name about that many :)

- Mutating default arguments to functions, so subsequent calls have different behavior

- Somewhat particular rules around creating references vs copies

- Things that look like lambda captures but aren’t quite

Yep, every language has footguns and other kinds of quirks, but I contend that the "footguns per character" ratio in shell is unusually high. (It is not unique in having a high ratio though; other popular languages, like c++ for instance, also have this issue.)

The worst (level of nastyness * usage) offenders all probably have a reason for being popular despite their flaws:

- Bash: installed everywhere you want to work (yes, who actually wants to work on Windows ;-)

- C/C++: when speed/size matters there was no alternative except Assembly until recently

- Javascript: until recently this was the most sane option for client side code on the web (Active X and Java applets existed yes but managed to be even worse.)

- PHP: Low cost hosting, Function-As-A-Service way before that became popular, shared nothing architecture, instant reload for local development bliss

Couldn't agree more.

- string vs. list "in" ('a' in 'a' is True, but 'a' in ['a'] is also True)

- cannot know which object attributes are private or public (and some classes use settable properties so you can't say "just don't set any attributes on non-dataclass objects")


I think a good way to evaluate languages from this perspective is through the lens of how easy it is to maintain understanding over time. I have learned bash well three or four times now, but I know now that I'm never going to remember enough of its quirks through the interim periods where I'm not focused on it, to be able to grok arbitrary scripts without refreshing my memory. This is very different for languages like java, go, python, and some others, which have their quirks, sure, but a much lower quirks per character ratio.

I might agree with "it's not hard to learn it", but I don't agree with "it's not hard to remember it".

I don't think PowerShell is a big improvement, though. Still allows no signature checking of functions. Shells are optimized for fast entry, not for writing comprehensible (or even secure) programs.

> Shell is evil mainly because it's so old.

I really don't understand this point, its a script language, how old is it doesn't make any difference. I've come accross some Powershell scripts that were unreadable down to its verbosity with certain things, and if you don't already know all the flags and options for it, it's hopeless to try and understand.

Both serve a purpose, neither are 'evil'.

There is Xonsh which is a Python shell. I don't know why everyone hasn't switched to it already as default.

> I feel like even "simple" shell scripts tend toward more inscrutability than all but the most questionable python code I've ever written.

When you use long options in bash (scripts), it becomes very readable, but it's not a widespread practice. I always use long options while writing scripts.

Consider these two examples, which is very straightforward:

- curl -fsSL $URL | bash

- curl --fail --silent --show-error --location $URL | bash

The second one almost "talks you through".

IMO, that's not where the inscrutability comes from. Rather, it is things like many useful (and thus used) constructs being implemented with line noise like `"$(foo)"`, `2>&1 &`, `${foo:-bar}`, etc., finicky control flow syntax, surprising defaults that can be changed within unclear scopes, etc.

You're right, but you can also expand these. If not syntactically, by adding temporary variables and comments around these complex parts.

It's true that Bash and Perl has one of the most contractible syntax around, but it's not impossible to make it more understandable.

However, these parts of codebases are considered "supportive" and treated as second class citizens, and never receives the same love core parts of the codebases enjoy. That's a big mistake IMO.

When you make something more readable all around, hiding things becomes harder exponentially.

I have seen people write bash scripts with a lot of love and attention trying really hard to pay attention to modern best practices and avoiding known pitfalls and using shellcheck. It still looked like shit imo.

I challenge you to find a single easily readable >100 line bash script and link it here (I do think small scripts can be fine).

Yep, my experience of this was at google, where I heard "gbash is actually pretty good!", so I learned that system well (because shell scripts are incredibly useful!), but still found the results unsatisfyingly inscrutable.

While not completely up to my own standards, this is pretty neat and readable: https://rclone.org/install.sh

I may add more if I have time today.

Must say I'm pleasantly surprised.

The irony of talking about readability of an example of curl to sh. You really don't need to understand the flags here to understand that this is a voluntary RCE and while I'm generally for long options in scripts, in this case it might make it harder to see the real problem because it increases the distance between the download command and the shell invocation.

> The irony of talking about readability of an example of curl to sh.

Who says I blindly curl2sh these scripts? I download them, read them and understand what they do before running them actually.

Long options make obscure command lines readable and understandable, actually, and if you vet them, you understand what they are doing.

Short options make things harder because I need to open the man of the command to understand what it is doing.

You don't have to do everything blindly as the quick start documents outline.

This is totally true and the bash apologists are delusional.

So are some Python advocates, too. The thing that's worse than a bash script made of Perlish line noise, is a piece of "clean code" dead simple Python that's 80% __language__.boilerplate, 20% logic, smeared over 10x the lines because small functions calling small functions are cool. No one has enough working memory to keep track of what's going on there. Instead, your eyes glaze over it and you convince yourself you understand what's going on.

Also, Python build scripts can be living hell too, full of dancing devils that could be introducing backdoors left and right - just look at your average Conan recipe, particularly for larger/more sensitive libraries, like OpenSSL or libcurl.

FWIW, my comment wasn't meant to single out python as particularly good. I think the comparison I drew between its inscrutability and that of shell / bash would apply to nearly all other languages as well.

That's a lot of emotive language, can you actually link to some actual examples.

I can understand the frustration when those small functions are not well-named, so you have to inspect what they do.

You already have to inspect everything if you want to review/audit a build script. Small functions - and I specifically mean functions being written small because of misguided ideas of "clean code", as opposed to e.g. useful abstraction or reusability - become especially painful there, as you have that much more code to read, and things that go together logically (or execution-wise) are now smeared around the file.

And you can't really name such small functions well anyway, not when they're broken down for the sake of being small. Case in point, some build script I saw this week had function like `rename_foo_dll_unit_tests` calling `rename_foo_dll_in_folder` calling `rename_foo_dll` calling `rename_dlls`, a distinct call chain of four non-reused functions that should've been at most two functions.

Are all Python build scripts like that? Not really. It's just a style I've seen repeatedly. The same is the case with inscrutable Bash scripts. I think it speaks more about common practices than the language itself (notwithstanding Bash not really being meant for writing longer programs).

Sounds like DRY run amok indeed. Maybe a compiler or linter could detect these cases and complain "this function is only called from one place" :)

I'm glad I'm not the only one with this particular frustration.

I mentioned Conan recipes, didn't I? :). Those are my most recent sources of frustration.

I've never heard of a conan language, and a couple of URLs to some bad recipes would not go amiss.

Conan is a package manager for C/C++, written in Python. See: https://conan.io/.

The way it works is that you can provide "recipes", which are Python scripts, that automate the process of collecting source code (usually from a remote Git repository, or a remote source tarball), patching it, making its dependencies and transitive dependencies available, building for specific platform and architecture (via any number of build systems), then packaging up and serving binaries. There's a lot of complexity involved.

Here are the two recipes I mentioned:

libcurl: https://github.com/conan-io/conan-center-index/blob/master/r...

OpenSSL v3: https://github.com/conan-io/conan-center-index/blob/master/r...

Now, for the sake of this thread I want to highlight three things here:

- Conan recipes are usually made by people unaffiliated with the libraries they're packaging;

- The recipes are fully Turing-complete, do a lot of work, have their own bugs - therefore they should really be treated as software comonents themselves, for the purpose of OSS clearing/supply chain verification, except as far as I know, nobody does it;

- The recipes can, and do, patch source code and build scripts. There's supporting infrastruture for this built into Conan, and of course one can also do it by brute-force search and replace. See e.g. ZLib recipe that does it both at the same time:

https://github.com/conan-io/conan-center-index/blob/7b0ac710... -- `_patch_sources` does both direct search-and-replace in source files, and applies the patches from https://github.com/conan-io/conan-center-index/tree/master/r....

Good luck keeping track of what exact code goes into your program, when using Turing-complete "recipe" programs fetched from the Internet, which fetch your libraries from somewhere else on the Internet.

That was a really, really good answer, thanks.

It depends on the use case. Bash code can be elegant, Python code can be ugly. I'm not saying those are the average cases but complex code regardless of the language often is ugly even with effort to make it more readable.

Oh I get it. Sometimes bash is the right tool for the job. I think that’s just mostly an unfortunate historical artifact though. It’s hard to argue it’s intuitive or “clean” or ${POSITIVE_ADJECTIVE} in the average case.

I'm a bash apologist and I totally stand by your words. It's delusional. Bash totally has to go.

The problem is that obfuscated bash is considered normal. If unreadable bash was not allowed to be committed, it would be much harder to hide stuff like this. But unreadable bash code is not suspicious, because it is kind of expected. That’s the main problem in my opinion.

Lots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly.

But all this relies on one specific thing about the autotools ecosystem - that shipping the generated code is considered normal.

I know of no other build system that does this? It feels weird, like shipping cmake-generated makefiles instead of just generating them yourself, or something like scons or meson being packaged with the tarball instead of requiring an eternal installation.

That's a lot of extra code to review, before you even get to any kind of language differences.

> Lots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly.

That's why you don't commit auto-generated code. You commit the generating code, and review that.

Same reason we don't stick compiled binaries in our repositories. Binary executables are just auto-generated machine code.

But then you have the problem that enabled this backdoor. It's normal to have uncommitted autogenerated unreadable shell code in the tarball. Nobody is going to review test, it was just generated by automake, right? That makes it so easy to sneak in a line that something slightly different. At least with cmake, you have none of this nonsense, people need cmake to build the project, it doesn't try to save users from it by generating a ton of unreadable shell code.

> Nobody is going to review test, it was just generated by automake, right?

Well, there's your problem. If you have unreviewed code, anything can be snuck in. Doesn't really matter too much where in your system the unreviewed code is.

> It's normal to have uncommitted autogenerated unreadable shell code in the tarball.

You need to review everything that goes into the tarball. Either directly, or indirectly by reviewing the sources it gets built from. (And then making sure that your build process is deterministic, and repeated by a few independent actors to confirm they get the same results bit for bit.)

This probably varies widely, because unreadable bash is absolutely not considered normal, nor would pass code review in any of my projects.

On a slightly different note, unless the application is written in python, it grosses me out to think of writing scripts in python. IMHO, if the script is more complex that what bash is good at (my general rule of thumb is do you need a data structure like an array or hash? then don't use bash), then use the same language that the application is written in. It really grosses me out to think of a rails application with scripts written in python. Same with most languages/platforms.

What if your application is written in Rust or C? Would you write your build scripts in these languages, too? I would much prefer a simpler scripting language for this. If you’re already using a scripting language as the main language, you don’t necessarily need to pull in another language just for scripts, of course.

Writing a build script in Rust is fine-ish.

Writing anything in C is a bad idea these days, and requires active justification that only applies in some situations. Essentially, almost no new projects should be done in C.

Re-doing your build system, or writing a build system for a new project, counts as something new, so should probably not be done in C.

In general, I don't think your build (or build system) should necessarily be specified in the same language as most of the rest of your system.

However I can see that if most of your system is written in language X, then you are pretty much guaranteed to have people who are good at X amongst your developers, so there's some natural incentive to use X for the tooling, too.

In any case, I would mostly just advice against coding anything complicated in shell scripts, and to stay away from Make and autotools, too.

There are lots of modern build systems like Shake, Ninja, Bazel, etc that you can pick from. They are all have their pros and cons, just like the different distributed version control systems have their pros and cons; but they are better than autotools and bash and Make, just like almost any distributed version control is better than CVS and SVN etc.

C is probably the best example where I would be fine with scripts in Python (for utility scripts, not build scripts). Though, if it were me I'd use Ruby instead as I like that language a lot better, and it has Rake (a ruby-ish version of Make), but that's a "flavor of ice cream" kind of choice.

Also zig is a good example of not a scripting language which does this job.

build.rs is a thing FYI

or make.go, for some project it makes sense to not add another language for scripting and building tasks. It way easier for every one to have to master multiple language.

I think the main issue is auto tools tries to support so many different shells/versions all with their own foibles, so the resulting cross-compatible code looks obfuscated to a modern user.

Something built on python won't cover quite as wide a range of (obsolete?) hardware.

Python actually covers quite a lot of hardware. Of course, it does that via an autotools nightmare generated configure script.

Of course, you could do the detection logic with some autotools-like shenanigans, but then crunch the data (ie run the logic) on a different computer that can run reasonable software.

The detection should all be very small self-contained short pieces of script, that might be gnarly, but only produce something like a boolean or other small amount of data each and don't interact (and that would be enforced by some means, like containers or whatever).

The logic to tie everything together can be more complicated and can have interactions, but should be written in a sane language in a sane style.

...The main problem is some asshat trying to install a backdoor.

I use bash habitually, and every time I have an inscrutable or non-intuitive command, I pair it with a comment explaining what it does. No exceptions.

I also don't clean up after scripts for debuggability. I will offer an invocation to do the cleanup though after you've ascertained everything worked. Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing.

There was a lot of intentionally obfuscatory measures at play here and tons of weaponization of most conscientious developer's adherence to the principle of least astonishment, violations of homoglyphy (using easy to mistake filenames and mixed conventions), degenerative tool invocations (using sed as cat), excessive use/nesting of tools (awk script for the RC4 decryptor), the tr, and, to crown it all, malicious use of test data!!!

As a tester, nothing makes me angrier!

A pox upon them, and may their treachery be returned upon them 7-fold!

> Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing.

If your hammer is repurposed shoe, it's fair to blame the tools.

> pair it with a comment

A good practice, but not really a defense against malice, because if the expression is inscrutable enough to really need a comment, then it's also inscrutable enough that many people won't notice that the comment is a lie.

Nothing short of reading the damn code is a defense against malice. I have yet to have any luck in getting people to actually do that.

Let's not get into a all-or-nothing fallacy here, my point is that there is an important security difference between:

1. Languages where obfuscation techniques look a lot like business as usual.

2. Languages where obfuscation techniques look weird compared to business as usual.

The presence or lack of comments in #1 situations won't really help to bridge the gap.

You assert a dichotomy where no split really exists. Qt the end of the day, things like tr or xz or sed invocations are not part of bash as a language. They are seperate tool invocations. Python or any other programming language could have things hidden in them just as easily.

And the other major issue here, is that xz is a basic system unit, often part of a bare bones, ultra basic, no fluff, embedded linux deployment where other higher level languages likely wouldn't be. It makes sense for the tools constituting the build infra to be low dependency.

And yes. In a low dependency state, you have to be familiar with your working units, because by definition, you have fewer of them.

Unironically, if more people weren't cripplingly dependent on luxuries like modern package managers have gotten them accustomed to, this all would have stuck out like a sore thumb, which it still did once people actually looked at the damn thing*.

“Why can’t humans just be better computers!” is a naive exercise in futility. This is not the answer, and completely ignores the fact that you yourself certainly make just as many mistakes as everyone else.

I'm mot asking people to be better computers. Nor do I believe myself to blissfully free of making mistakes. I'm saying, from an information propagation standpoint, it is 100% impossible for you to make a knowing based statement without having first laid eyes on the thing which you are supposed to make knowledge based statements about.

This fundamental limitation on info prop will never disappear. There is nothing harder to do than to legit get somebody to actually read code.

This feels a bit like saying you can run as fast as Usain Bolt. Theoretically many things are possible, but I don't think I've ever seen a readable bash script beyond a trivial oneliner and I've seen a lot of bash in my life. Or to maybe explain from a different perspective, ask a room full of developers to write a bash if-else without looking at the docs and you'll probably come back with more different options than developers in the room. Ask the same for a language such as Python and you'll mostly get one thing.

We'll write build scripts in highly obfuscated Perl. There, now no one is happy.

Python makes it relatively hard to write inscrutable code, and more importantly, it's very non-idiomatic, and there would be pushback.

WTFing at shell scripts is normal.

Yep. Python is my day job. If anyone on my team put something up that was as unreadable as a typical bash script, without really good justification, I’d immediately hit “request changes”.

This isn’t just because I’m more familiar with Python. I don’t even think it’s the main reason. It’s just that Python is more likely to be able to be read in a ‘natural language’ sort of way. It’s better at doing what it says on the tin. It’s more able to be read via pure intuition by a programmer that’s not familiar with Python specifically.

In bash land? “What the hell is [[?”

And yes, I could come up with 20 ways off the top of my head that Python is a far-from-perfect language.

And I’m not even saying that Python is the right tool for the job here. Maybe we’re better off with one of the many more modern attempts at a shell language. The main thing is that we should acknowledge that bash has overstayed its welcome in many of the areas in which it’s still used.

> WTFing at shell scripts is normal

"WTFing". This is brilliant.

I always think of this now-old comic: https://www.osnews.com/story/19266/wtfsm/

Yeah, nah. I’m all for “every tool has its place”, “religious wars over languages are silly”, etc. Don’t get me wrong. But anyone that claims that all but the 1% most simple bash scripts are “straightforward” and “readable” is, to be blunt, showing their age.

The reality is that we as a society have made meaningful progress as far as designing readable languages goes. And we LITERALLY have many orders of magnitude more resources to make that happen. That’s something to feel good about. It’s unreadable to continue to mischaracterise some sysadmin greybeard’s’ familiarity with bash as an indication that it is in any way intuitive or readable.

Like, sheesh, now we all sound like C developers trying to justify an absurdly footgun-laden standard library just because we happen to know the right secret incantations, or think that we know them, anyway. But now this is definitely becoming a religious war…

It's rather easy to monkeypatch Python into doing spooky things. For something like this you really want a language that can't be monkeypatched, like I think Starlark.

Where are you going to hide your monkey patching though? As long as your code is public, stuff like this is always going to stand out, because no one writes weird magic one liners in python.

The way python's run time literally executes code when you import a module makes it seem pretty easy to taint things from afar. You only need to control a single import anywhere in the dependency hierarchy and you can reach over and override any code somewhere else.

There are lints that will warn you, if your imported module does anything apart from define functions and classes.

(Though not sure how fool-proof these are.)

Oh yeah, that's a fantastic point.

This code was essentially monkey patched from a test script. Python automatically runs any code in a imported module, so not hard to see a chain of module imports that progressively modifies and deploys a similar structure.

I'm not a security-oriented professional, but to me a place I could hide this logic is by secretly evaling the contents of some file (like the "corrupt archive" used in xz) somewhere in the build process, hiding it behind a decorator or similar.

I’m not a security professional either, but that doesn’t sound very plausible to me. If you assume a maintainer who checks every commit added to the codebase, he’s hopefully blocking you the second he sees an eval call in your build script. And even a code audit should find weird stuff like that, if the code is pythonic and simple to read. And if it’s not, it should not be trusted and should be treated as malicious.

Well, the threat model here is that a maintainer themselves is the saboteur.

That was true for this project, which was almost orphaned to begin with. We'll run out of nearly-unmaintained critical infrastructure projects sometime. Larger projects with healthier maintenance situations are also at risk, and it's worth reasoning about how a group of honest developers could discover the actions of one malicious developer (with perhaps a malicious reviewer involved too).

Would stick out like a sore thumb

eval() is a big security hole

> In the beginning of Unix, m4 was created. This has made many people very angry and has been widely regarded as a bad move.

autoconf creates a shell script by preprocessing with m4. So you need to know not just the intricacies of shell scripting, but also of m4, with its arcane rules for escaping: https://mbreen.com/m4.html#quotes

If autoconf used m4 to generate python scripts, they would also look like https://pyobfusc.com/#winners

The things that Bash is good at can wind up obfuscated in Python code due to the verbosity and complexity that it translates to in Python.

Bash is great at dealing with files, text, running other programs, job handling, parallelism and IPC.

Those things in combination can end up being more complex in Python, which creates more opportunities for obfuscation.

Just wait for AI to start pumping spaghetti code everywhere.

That's AI phase one

Hey, I phase 2 is even better disguised exploit code hiding behind acres of seemingly plausible AI generated code

Any language can become a turing tarpit if you try hard enough.

Some languages make you try harder, some less so.

And not all languages are Turing complete in the first place. Not even all useful languages.

Yet it is python code which has the highest amount of security vulnerabilities found in public repos. And the most often times that pre-compiled code is commited as well.

Cliffy is also pretty good: https://cliffy.io

And type safe.

I see what everyone is saying about autotools, and I never envied those that maintained the config scripts, but as an end-user I miss the days when installing almost any software was as simple as ./configure && make && make install.

Running that command is simple, sure. The development workflows when that doesn't work is full stop horrible.

Also the basic workflows for the alternative build systems have maybe ten more characters to type. It's not bad.

Autotools need to go away, but most of the underlying use cases that cause build system complexity comes down to dependency management and detection. The utter lack of best practices for those workflows is the root cause of complexity. There is no way to have a mybuild.toml build config on the face of those challenges.

Agreed, and I think this leads to the question: how much risk do we face because we want to support such a wide range of platforms that a complex system is required?

And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation?

> And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation?

The project in question contained a compression library, but was not limited to it; it also contained a set of command line tools (the "xz" command and several others).

And a modern compression library needs more than just "math and memory allocation"; it also needs threads (to make use of all the available cores), which is historically not portable. You need to detect whether threads are available, and which threading library should be used (pthreads is not always the available option). And not only that, a modern compression library often needs hand-optimized assembly code, with several variants depending on the exact CPU type, the correct one possibly being known only at runtime (and it was exactly in the code to select the correct variant for the current CPU that this backdoor was hidden).

And that's before considering that this is a library. Building a dynamic library is something which has a lot of variation between operating systems. You have Windows with its DLLs, MacOS with its frameworks, modern Linux with its ELF stuff, and historically it was even worse (like old a.out-based Linux with its manually pre-allocated base address for every dynamic library in the whole system).

So yeah, if you restrict yourself to modern Linux and perhaps a couple of the BSDs, and require the correct CPU type to be selected at compilation time, you could get away with just a couple of pages of simple Makefile declarations. But once you start porting to a more diverse set of systems, you'll see it get more and more complicated. Add cross-compilation to the mix (a non-trivial amount of autotools complexity is there to make cross-compilation work well) and it gets even more complicated.

Right there with you. It's really tempting to blame this entire thing existing on m4 but that's the trauma talking.

Let me reword that for you:

>No one has any business saying they know what something does until they've actually read it.

Beneath the placid surface of abstraction is the den of the devil.

I'm glad to see I'm not a minority of one here. Looking at autoconf and friends I'm reminded of the dreck that used to fill OpenSSL's code simply because they were trying to account for every eventuality. Autotools feels like the same thing. You end up with a ton of hard to read code (autogenerated bash, not exactly poetry) and that feels very inimical to safety.

I do not agree with your generalisation, the Meson build system is well thought, it has a clear declarative syntax that let you just express what you want in a direct way.

The designer of Meson explicitly avoided making the language turing complete so for example you cannot define functions. In my experience this was an excellent decision to limit people tendency to write complex stuff and put the pressure on the Meson developer to implement themselves all the useful functionalities.

In my experience the Meson configuration are as simple as they can be and accommodate only a modicum of complexity to describe OS specific options or advanced compiler option one may need.

Please note that some projects' Meson file have been made complex because of the goal to match whatever the configure script was doing. I had in mind the crazy habits of autotools to check if the system has any possibly used function because some system may not have it.

Meson is way better than autotools, but if you're too level to depend on python, you're probably needing to customize your build scripts in the ways you mention. I don't see meson being a silver bullet there.

Also, meson's build dependencies (muon, python) are a lot for some of these projects.

If a projects build system can't depend on python, that let's leave them in the dust, ffs..

What if the project is python or a dependency of python?

This just results in 500x worse build.sh on top of meson/ninja/whatever.

The devil is in the detail and nothing obscures details like complexity.

Same reason why I don't like TypeScript in its current form. It's not worth the extra complexity it brings.

Every time someone explains to me autotools, the individual pieces sort of make sense, yet the result is always this inscrutable unreadable mess.

I don't know why.


I haven’t used it for some time but autoconf always seemed like a horrible hack that was impossible to debug if it didn’t work properly.

That was bad enough back in the days where one was mostly concerned with accidents, but in more modern times things that are impossible to debug are such tempting targets for mischief.

Just the fact that you have multiple platforms suggests few people will fully understand the entire complex.

The issue was that built artifacts weren’t immutable during the test phase and/or the test phase wasn’t sandboxed from the built artifacts.

The last build system I worked on separated build and test as separate stages. That meant you got a lot of useless artifacts pushed to a development namespace on the distribution server, but it also meant later stages only needed read access to that server.

The malicious .o is extracted from data in binary test files, but the backdoor is inserted entirely in the build phase. Running any test phase is not necessary.

So if you ran build without test files it would fail. I get that this is hindsight thinking - but maybe removing all non essential files when packaging/building libraries reduces the surface area.

As a developer, this amazes me, and it just shows what — to me — feels like a top-tier attack method, is probably only entry-to-mid level complexity for the folks working at that stage. Some of the things I see posted here on HN are well above this level, so I'd assume for the right kind of money (or other incentives), this is only the beginning of what's possible. And, if you think of ALL the packages and ALL the many millions of libraries on GitHub, this vector is SO EFFECTIVE, there will be hundreds of cases like it uncovered in the next few months, I am certain of it.

I worry about all the con/pro-sumer hardware makers, from Philips Hue to Alexas, from the SumUps to the camera makers, from Netgear to TP-Link. All their products are packed with open-source libraries. And I am 100% certain that most of their dev teams do not spend time scanning these for obscure injection vectors.

> And I am 100% certain that most of their dev teams do not spend time scanning these for obscure injection vectors.

This rationale baffles me, it feels that the dependency-hell circlejerk crowd is working on making OSS maintainers look even more bad with this scenario.

Any given commercial operation that claims any credibility for itself does supply chain analysis before adopting a dependency. This is, among other things why ordinarily you'd pay RedHat to maintain a stable Linux Release for you and why projects such as FreeBSD severely limit the software they ship in the default install.

If you are affected by this mess, I'm sorry to say, but it's your fault. If you are worried about developers of software you use for free, as in free beer, going rogue, either put in incentives for them to not do that (i.e. pay them) or fork the project and implement your own security measures on top of what's already there.

If you're worried that you could encounter exploits from dependencies in commercial software you use, you should negotiate a contract that includes compensation from damages from supply chain attacks.

If you're unwilling to do that, sorry mate, you're just unprofessional.

Inb4: Yes, I am really trying to say that you should check the supply chain of even your most basic dependencies such as SSH.

Unfortunately that's "industry standard" nowadays. I lost count how often I had that discussion over the last two decades.

Just look at stuff like pip, npm or pretty much any "modern" package manager in use by developers - they're all pretty much designed to pull in a shitload of arbitrary unaudited and in some causes unauditable dependencies.

And nobody wants to listen. That's why I prefer to work in heavily regulated areas nowadays - that way I can shorten that discussion with "yeah, but regulatory requirements don't let us do that, sorry"

The absolute basic should be having a local archive of dependencies which at least received a basic sanity check, and updates or additions to that should review changes being added. CI gets access to that cache, but by itself does not have network access to make sure no random crap gets pulled into the build. You'd be surprised how many popular build systems can't do that at all, or only with a lot of workarounds.

Seems like a lot of this could be solved with a whitelist of trusted dependencies.

There are already lots of groups maintaining internal lists and analysis of dependencies they trust. If there was a platform for reporting safety, rather than reporting vulnerability, one could say "Only allow packages that someone from a fortune 500 company publish an analysis of".

Package managers that use git are less prone to this kinda of attack (goland, rust).

Cargo does not “use git.”

Cargo absolutely uses git, or else we wouldn't've had that thing where setting it to use the cli git instead of some reimplementation lead to massive speedups

What I mean is that it doesn’t use git as the source of truth for packages (unless you point at a git repo instead of crates.io). It does use a git repo for the index.

What they meant was probably, that you have the option to rely entirely on using git repositories for your dependencies or even just paths to other projects on your disk.

You can also setup your own dependency registry and only work with that.

That is possible but it’s not the default and isn’t as nice as the default is.

> they're all pretty much designed to pull in a shitload of arbitrary unaudited and in some causes unauditable dependencies.

No they're not. The dependency circle jerk went so far to prompt NPM to display all subsequent dependencies on each libraries page.

The issue lies with the industry as a whole exploiting the work of OSS developers for their own gain and having the audacity to complain when these volunteers won't additionally do a security audit for free.

> Any given commercial operation that claims any credibility for itself does supply chain analysis before adopting a dependency. This is, among other things why ordinarily you'd pay RedHat to maintain a stable Linux Release for you and why projects such as FreeBSD severely limit the software they ship in the default install.

That sounds like you assume RedHat would've caught the vulnerability in xz-utils, before shipping it in the next release of RHEL. I'm not so sure about that, as there is only so much you can do in terms of supply chain analysis and such a sophisticated vulnerability can be pretty hard to spot. Also mind that it only got discovered by accident after all.

I don't know if RedHat would have caught it. But the benefit of Red Hat is, they would be the one to fall on the sword. Your product is built on RHEL. This happens. You get to shift blame to RHEL and RedHat would eat it. The positive is, after the dust has settled Red Hat could choose to sort of adopt the compromised piece (invest engineering effort and take it over) or take some stewardship (keeping an eye on it and maybe give a hand to whoever is maintaining it after).

Sorry, but what? What does “shifting the blame” mean here? Ever heard of “it’s not my fault but it’s my problem?”.

It really sounds like you’re speaking from the perspective of a hypothetical employee looking to not get PIPd or whatever.

GP is talking about something quite different, and you’ve run off taking some sort of great personal offence to someone dare implying that there are downsides to open source, not even that it’s worse overall, but that there are downsides.


They're right though, the benefit of paying absurd amounts to your linux vendor is raking in certs that you can use with your insurance provider to cover your a** in case something like this happens. That's the sole reason of certs after all. Though I'd like to figure out if RedHat really is going to eat it if push comes to shove.

In theory they should and hopefully in practice they would. How rigorously is that tested, I am not sure. But if they weren't willing to eat it when they sell you support for a distro they package has a vulnerability, then people would need to ask, what is the point in paying Red Hat? Seems like their entire business would go out the window, especially because they are required or one of a few small number of options certain business domains require. What is the advantage of RHEL over the free Debian ISO and support there that I currently deploy to production environments? I also don't work in as heavily of regulated domain.

> I also don't work in as heavily of regulated domain.

I feel this is the crux of it for the thread. Most places where I've worked have been regulated and this has been interested to read/follow.

This 'fall on the sword' thing is real. The 'engineer on a PIP' thing is too, in a twisted sense. This has multitudes/depth.

Consider business terms/liability. Your certification/ability to do business depends on implementing certain things, sometimes by buying things (ie: RHEL) from those who also carry certifications. The alternative is to do it yourself at great expense.

If 'it' hits the fan, you can [hopefully] point at due diligence. It's not an engineer doing this to cover themselves... but businesses.

I don't know how approachable the distribution providers are as a smaller business. We, at fairly large enterprises, were able to work closely with them to get fixes regularly - but that says very little.

Anyway: I say all this to neither defend or deride the situation. It's sort of like a cartel, insurance, and buying merch for a band on tour, all in one.

I've benefited from this situation but also lost years of my life to it

Then please enlighten me as how the hell Red Hat's business model is supposed to work if that isn't true. You pay for Red Hat for quality guarantees and certifications, which in some industries is required. The main business model of Red Hat is, pay for a curated distro with support and we will take care of some things for you. We will ensure a secure and managed repo of third party tools and yada yada. Otherwise, why would anyone pay Red Hat and not just deploy fleets of Debian servers? For sure, some people do just deploy Debian, this is what I do at work. But some businesses do pay for Red Hat.

I'm not saying its not a companies problem if this exploit got into their RHEL environments. But from a company perspective when it comes down to law suits, they will get to shift the blame to RHEL. And for a business, that is what matters. Do you really think companies care about having secure systems? I would be willing to bet money, if companies could be protected from lawsuits from data breeches, they wouldn't give two shits about security. For them, data breeches are just potential multi-million or multi-billion dollar legal liabilities. And this is part of RHEL's business model. You get to shift some of that legal liability to RHEL.

I said "ordinarily". I meant "this is what you'd expect from them by paying them". Obviously this is a big fauxpas on their end and I'd reconsider using their services after this scenario. After all, security hardening upstream packages is among the reasons you're supposed to use them.

I think it's more than an individual or an organisation. The industry as a whole has favoured not caring about dependencies or where they come from in order to increase velocity. Concerns about supply chain (before we even had the term) were dismissed as "unlikely and we won't be blamed because everyone's doing it").

The organisations that did have some measures were complained about loudly, and they diluted their requirements over time in order to avoid stagnation. Example: Debian used to have a "key must be signed by three other Debian developers" requirement. They had to relax the requirement in part because, from the perspective of the wider ecosystem, nobody else had these onerous requirements and so they seemed unreasonable (although Covid was the final straw). If we'd had an ecosystem-wide culture of "know your upstream maintainer", then this kind of expectation as a condition of maintainership would be normal, we'd have much better tooling to do it, and such requirements would not have seemed onerous to anyone. It's like there's an Overton Window of what is acceptable, that has perhaps shifted too far in favour of velocity and at the cost of security, and this kind of incident is needed to get people to sit up and take notice.

This incident provides the ecosystem as a whole the opportunity to consider slowing down in order to improve supply chain security. There's no silver bullet, but there are a variety of measures available to mitigate, such as trying to know the real world identity of maintainers, more cautious code review, banning practices such as binary blobs in source trees, better tooling to roll back, etc. All of these require slowing down velocity in some way. Change can only realistically happen by shifting the Overton Window across the ecosystem as a whole, with everyone accepting the hit to velocity. I think that an individual or organisation within the ecosystem isn't really in a position to stray too far from this Overton Window without becoming ineffective, because of the way that ecosystem elements all depend on each other.

> If you're unwilling to do that, sorry mate, you're just unprofessional.

There are no professionals doing what you suggest today, because if they did, they'd be out-competed on price immediately. It's too expensive and customers do not care.

>, is probably only entry-to-mid level complexity for the folks working at that stage.

On the contrary: the developers and maintainers who are more informed than us described it as highly sophisticated attack. I also read early InfoSec (information security) articles which were able to only describe a part of the code, not the whole strategy behind the attack because, again, the attack and code are sophisticated. You can also read early InfoSec articles which describe the attack in different ways simply because it was not that simple to understand. Then I read articles saying something like this: "Finally it seems it's an RCE attack".

Of course, now that even a scanner is developed to detect that vulnerability on your server, we can all claim: "Oh that was a so simple and stupid attack, how come no one detected it much earlier ?!"

That's why I don't see e.g. TP-Link basing their router firmware on OpenWRT as a win, and why I want the "vanilla" upstream project (or something that tracks upstream by design) running on my devices.

Applies to all of my devices btw. I don't like Android having to use an old kernel, I didn't like MacOS running some ancient Darwin/BSD thing, etc. The required effort for backporting worries me.

Don't get me wrong, I'm not saying OSS has no vulns.

More orgs directly contributing to upstream is best in my eyes too. I'm not against forking, but there are usually real benefits to running the latest version of the most popular one.

One opposite of this I've seen is Mikrotik's RouterOS. I'm under the understanding that they usually reimplement software and protocols rather than depending on an upstream.

I'd imagine that is what leads to issues such as missing UDP support in OpenVPN for 10 years, and I'm not sure it gives me the warmest fuzzy feeling about security. Pros and cons, I suppose. More secure because it's not the same target as everybody else. Less secure because there are fewer users and eyes looking at this thing.

> there are usually real benefits to running the latest version of the most popular one.

Using the absolute latest version is acting as a beta tester for everyone else and this is not the first case where it means you get absolutely hosed.

Any moderately well run shop will have a mechanism to get updates when a dependency of theirs has a security issues, depending on the line of business it may actually be required by a regulator or certification body (eg PCI etc)

We should probably be more afraid of the backdoors you can’t see in proprietary that would almost never be found.

> there will be hundreds of cases like it uncovered in the next few months, I am certain of it.

"Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the "fixes" mentioned above." (https://www.openwall.com/lists/oss-security/2024/03/29/4)

So, it's like story of those security researchers injecting bugs into the kernel https://thehackernews.com/2021/04/minnesota-university-apolo...

I'm saying this isn't that easy to pull off, and it's unlikely we'll see hundreds of similar cases.

> this isn't that easy to pull off,

How much does it cost in money? 400k? 600k? 1M? That's peanuts for any three letter agency from the US, UK, China, etc.

I haven't seen Thomas Roccia's infographic mentioned here yet: https://twitter.com/fr0gger_/status/1774342248437813525

That seems to be largely grasped at straws and connecting dots without reason.

For example oss-fuzz was building xz by cloning the github repo directly. There was never a chance for oss-fuzz to discover the backdoor, because only the tarball had it, not the repo itself. So that oss-fuzz PR might genuinely just be a genuine thing unrelated to the backdoor.

The ifunc part looked more legitimate, it was used to switch between optimized implementations of CRC. So that part was in the git repo. https://github.com/google/oss-fuzz/pull/10667

Jia Tan asked distros to update quickly just before it became public. How possible is it, there is another account / person who learned earlyer from people around Andreas Freund, the backdoor would become public. How possible is it, there is still another insider around?

That's probably because of this (as mentioned in the timeline):

>2024-02-29: On GitHub, @teknoraver https://github.com/systemd/systemd/pull/31550 to stop linking liblzma into libsystemd. It appears that this would have defeated the attack. https://doublepulsar.com/inside-the-failed-attempt-to-backdo... that knowing this was on the way may have accelerated the attacker’s schedule. It is unclear whether any earlier discussions exist that would have tipped them off.

rwmj did mention about an unintentional embargo break, so I wonder whether this is GitHub issue is actually it.


I'm the author of such PR. My purpose was to trim down the size of the initram files by removing unneeded dependencies.

I couldn't imagine that liblzma had a backdoor.

Hi! Was there any discussion on any mailing lists ahead of time, or was your PR the first public mention of that idea? Thanks.

Yes, there was an effort to turn all the dependencies into on demand loads, where possible. It started in 2020 with libpcre2: https://github.com/systemd/systemd/pull/16260

But many others followed, like libselinux: https://github.com/systemd/systemd/pull/19997

libqrencode: https://github.com/systemd/systemd/pull/16145

p11kit: https://github.com/systemd/systemd/pull/25771

tpm2-util: https://github.com/systemd/systemd/pull/28333

libiptc: https://github.com/systemd/systemd/pull/29836

libkmod: https://github.com/systemd/systemd/pull/31131

Exactly during the development of the libkmod PR, someone noted that libxz could be lazily loaded too: https://github.com/systemd/systemd/pull/31131#issuecomment-1...

And so I proposed myself to to the job, nothing less, nothing more.

If you look at the code of the other PRs, you see that they are very very similar, there are also macros to easy this task, like DLSYM_FUNCTION()

Hi @rsc, I just saw your update on the timeline.

To be more precise, the first public comment asking to dlopenify lzma was dated 30 Jan by Daan: https://github.com/systemd/systemd/pull/31131#issuecomment-1...

The day after, it was reiterated by Lennart: https://github.com/systemd/systemd/pull/31131#issuecomment-1...

But if you look in the systemd repo there is a TODO file with a section of libraries which needs to be lazy loaded. liblzma was added in this list in June 2020 (https://github.com/systemd/systemd/commit/cdfd853744ee934869...) by Lennart, and removed by me just after that my PR was merged.

Updated. Thanks!

There were also changes to systemd happening around that time which would have prevented the backdoor from working. See the timeline article by the same author linked in this one.

Right. Way too much coincidence. Jia Tan found out that it was about to become public and threw a Hail Mary. How did he find out?

I think the RedHat Valgrind report on 2024-03-04 made the Jia Tan team panic, since the one public rwmj stack trace pointed the finger directly at the backdoor. All it would take is someone looking closely at that failure to expose the whole operation. They fixed it on 2024-03-09, but then two weeks later distros still had not updated to the new version, and every day is another day that someone might hit the Valgrind failure and dig. I think that's why the sockpuppets came back on 2024-03-25 begging Debian to update. And then on the Debian thread there was pushback because they weren't the maintainer (except probably they were), so once Debian was updated, Jia Tan had to be the account that asked Ubuntu to update.

That seems like a breach that they went forward with the update based on some random persons request. Oh you're getting pushy? I guess we better listen to this guy.

The update was pulling from trusted upstream archives. I'm sure Debian verified that.

If the stakes weren't so high, this would be a damn fun game of murder mystery.

This just gives a partial high-level look at how the exploit gets planted in liblzma, it doesn't cover how the exploit works or its contents at all.

Thx. Timeline shows attack begun by adding ignore entry in .gitignore file. That is hard to detect nowadays.

As a naive bystander, the thing that stands out most to me:

> Many of the files have been created by hand with a hex editor, thus there is no better "source code" than the files themselves.” This is a fact of life for parsing libraries like liblzma. The attacker looked like they were just adding a few new test files.

Yes, these files are scary, but I can see the reason. But at least can we keep them away from the build?

> Usually, the configure script and its support libraries are only added to the tarball distributions, not the source repository. The xz distribution works this way too.

Obligatory auto tools wtf aside, why on earth should the tarballs contain the test files at all? I mean, a malicious test could infect a developer machine, but if the tars are for building final artifacts for everyone else, then shouldn’t the policy be to only include what’s necessary? Especially if the test files are unauditable blobs.

It's pretty common to run tests on CI after building to verify your particular setup doesn't break stuff.

Last time we were doing that we were preferring git upstream, though, and generated autocrap as needed - I never liked the idea of release tarballs containing stuff not in git.

This strengthens the argument I’m making, no? You bring in the source repo when doing development and debugging. In either case - tarball or not - it doesn’t seem that difficult to nuke the test dir before building a release for distribution. Again, only really necessary if you have opaque blobs where fishy things can hide.

The distributions often run the same tests after it’s built to make sure it’s working correctly as built in the distribution environment. This can and does find real problems.

We'd usually put the tests in a subpackage so it can be installed and run on devices later on.

Because, despite containing some amount of generated autoconf code, they are still source tarballs. You want to be able to run the tests after compiling the code on the destination machine.

Besides for verifying that the compiled program works on the target, tests are also required to compile with PGO because you need to have a runtime example to optimize for.

> The first difference is that the script makes sure (very sure!) to exit if not being run on Linux.

The repeated check is indeed mysterious. My only hypothesis is that the attacker may have thought that it should look plausible as a test input to a compression library, hence repetition.

It can be to make space for script changes: you may overwrite the first bytes of the script.

Or just add some laziness.

I also thought it was odd. There's also different random bytes (not random text, actual random bytes) prefixed to the start the scripts, but the bytes are prefixed by a hash symbol, which comments them out, so they don't affect the script. It seems intentional, but I can't think of why they would be there. I thought maybe xz would skip compression if the input was short/not complex enough or something, so they were added to pad the size, but removing them and re-compressing with xz seems to properly compress it, none of the original plaintext is in the compressed archive bytes.

One thing I noticed while trying to reproduce the exact bytes included in the .xz file committed to git is that the script's xz stream doesn't seem to be compressed by any of the default xz presets, I was only able to reproduce it by using `xz --lzma2=dict=65536 -c stream_2`. All the default numbered presents chose a different dictionary size. Another odd seemingly intentional choice, but again I don't understand the reasoning.

Ah, I think I understand the random bytes at the start of the script now. They're prepended to make the partial output of the "corrupted" (pre-tr'd) test stream look like random data. Without those random bytes, you will see part of the start of the script if you observe the partially decompressed output before xz throws an error. They really thought quite carefully about hiding this well.

Still not sure about the repeated lines, though now I'm convinced there must be some reason for it.

Can it be to enlarge/or obfuscate parts of the compressed test file? Perhaps without the repetitions the compressed file has some strange binary triggering some security or antivirus software?

Is it though? The attacker probably has Linux x86 target(s) in mind and IFUNC support isn't guaranteed to work with other platforms.

Checking for Linux makes sense. Doing the exact same check for Linux five times in a row is mysterious.

It would be an interesting plot twist if the whole thing was an AI hallucination.

How about an AI trying to make itself some spare CPU cycles available ?

...or a stalking horse by somebody in Microsoft's marketing division.

Perhaps expected a very fast machine that might blast straight through the first few checks. Like how you needed two STOP statements in a Cray Fortran program in case it blew through the first one at 80MIPS.

why doesn't IFUNC work on Linux ARM64?

IFUNC is supported on several architectures, including ARM64.

The malicious code that the xz backdoor inserts into the library is a compiled x86_64 object file so it only is targeting one platform.

yea i know the backdoor is AMD64 only. the parent comment said IFUNC isn't supported on ARM64 which is incorrect.

Maybe if run on non Linux it could be found out either by crashing or leaving some trace because of OS differences


Malware has no reason to edit what uname returns.

Why not? Most malware utilizes uname to either whitelist or blacklist infection on a system, so an APT may benefit from lying.

It would break a bunch of other legitimate stuff that also checks uname and expects stuff to be there.

Maybe but only once in a while, I wonder how much legitimate software relies on the kernel name at runtime (vs compile time)?

Also consider it may only trigger if five syscalls to uname come in sequence from the same process. I suspect without evidence this pattern is less common.

anyone know any examples of malware which have affected uname results?

ChatGPT could not provide any, fwiw, but it's an enticing idea.

Maybe a machine designed specifically to produce "enticing" and "very plausible"-seeming ideas with no basis in reality is a bad idea. [1]

The mention to other "malware" is spurious. This was a highly engineered/targeted attack. Mentioning "malware" just sounds plausible because your brain files the word in the same region as "RCE".

Running `strace` shows `/usr/bin/uname` just calls `uname()`, which just reads out some string literals in the kernel. It is unlikely to be randomized accidentally.

And deliberately signalling the presence of other hacks by making the `uname` syscall non-deterministic would be highly inefficient (Return 8 strings just for one bit of data?), highly unreliable (Check how many times to be sure?), highly visible (Human-readable, and human-targeted.), and unnecessarily difficult (Kernel-level hack?).

…And now you see how much effort it takes, to even start to examine/debunk an implausible and unfounded idea that cost you seconds to prompt and OpenAI fractions of pennies to generate, which took root and seemed "plausible" and "enticing" specifically because GPT was trained in an adversarial way to exploit how us humans can fall for bullshit if it's packaged in the right way…


1: The word for that is "Lying". ChatGPT is a Lying Machine. Sometimes the lies line up with parts of reality, but that's more luck than intent. And it's usually still lying even in those cases, just in smaller, subtler, or less blatant ways.

You can call it what you like but I’ve had plenty of hypothetical conversations with people and we never accused each other of lying. I think you are assuming that I treat GPT like an oracle rather than as a sci-fi author tripping on acid.

I’m not going to submit a paper to a scientific journal treating the information as a fact but this is a public discussion forum on the Internet, so I’m happy to speculate in the open using the hallucinations of an algorithm if they seem interesting.

People have the neurological, sensory, and social hardware to be able to conceive of such concepts as external reality, internal biases, internal intentions, and sufficient theory of mind to be able to predict and understand your mental state and try to shape their words to honestly convey those external and internal realities— Including when the mutual internal intention itself is hypothetical or fantastical.

I'm not calling LLMs lying machines just because they sometimes hallucinate and get things wrong. I'm calling them Lying Machines because they literally don't have either the hardware or progeny to be able to care about any concept of the truth, whether regarding explicit simple physical facts or about the intentions of a reciprocal conversation. They are, literally, designed, trained, and selected to fool humans so Sam Altman (and many others) can get his bag.

I find LLMs to be useful tools like any other piece of software. Prompting to get creative ideas is just like using any other software and coding it to respond with various options by say, pruning a large set of options based on a heuristic. In this case it’s a black box and the software is a bunch of words strung together. Also, I don’t pay for ChatGPT and often do this on local LLMs because it’s fun to use new technology.

You seem to have an axe to grind, good for you, I think we need lots of variety of opinion and passion as a society.

I think it's poor form to bring the unvetted outputs of a pseudorandom algorithm to a discussion with real people who are investing their own mental effort and emotional energy. If you couldn't be bothered to have the thought, why would you expect other people to listen to you?

That's the opposite of having "lots of variety of opinion and passion as a society". It's outsourcing cognition to a machine designed to emulate the most banal, superficial, and mediocre aspects of human expression.

That’s your opinion, but I was clear about the origin of the information which is accessible to anyone, it’s up to others to vet and validate information they find online, and furthermore, what you think is poor form is strictly your personal opinion and has little to do with the implied social etiquette of this forum, and nothing to do with its actual rules.

You continue to grind your axe without considering or responding to my opinion about LLMs utility as a tool. Like any tool, it can be used for good and bad things.

No matter what you feel about LLMs and other generative “AI,” they’re here to stay. If you choose to be ignorant of how to use such tools to your benefit, nobody is going to stop you.

> That’s your opinion, but I was clear about the origin of the information which is accessible to anyone, it’s up to others to vet and validate information they find online, and furthermore, what you think is poor form is strictly your personal opinion and has little to do with the implied social etiquette of this forum, and nothing to do with its actual rules.

Evidently not.

> codezero 1 day ago

> [flagged]

Do you always get this upset, argumentative, and perceive personal attack when people disagree with you?

I've been very civil, being flagged doesn't mean I broke the rules, flags are crowdsourced and it's more likely related to your axe grinding and the four downvotes which I'm fine with. If you are implying I broke the rules, please cite the rule I broke.

The comment I responded to literally said:

> My only hypothesis is that the attacker may have thought that it should look plausible as a test

The entire premise of this thread is speculation, which I did. Furthermore, the GPT response prompted me to come up with my own speculation (I won't call it a hypothesis, as this is not science):


I never said anything about a personal attack. You didn't disagree with me (There wasn't anything to agree to, I was speculating), you went on a tirade about how much you dislike people using ChatGPT. I don't have a problem with you, or your opinions about ChatGPT, in fact, I respect them.

You also haven't considered one thing: Posting trash responses by various LLMs on public forms will help weaken future LLMs by poisoning their inputs. If you hate LLMs so much, you should consider doing your part to poison their inputs so they become irrelevant, case in point, this thread is the third result for "uname used in malware" on Kagi, and first on Google. Stop being a passive participant and fight LLMs at the front if you care so much.

It's kind of tragically amusing how heinously complex and unnecessarily inscrutable modern technology is, and it's only getting worse. I think developers sadistically enjoy it.

This is probably the worst take ever.

It's pretty amazing how the tools keep up with the increasing complexity of the products we make.

And to be honest, in most cases just make it simpler, I think people just don't like to learn new stuff.

Whoever hired these people to infiltrate this project spent a shit ton of hours building it in such a way to avoid detection this long. Fortunately, it was so complicated that they couldn't account for all of the factors.

This is why open source will always outperform closed source in terms of security. Sure it pointed a massive flaw in the supply chain. Sure it highlights how under appreciated the foundational elements of FOSS, leaving maintainers subject to manipulation.

But the same attack within a private company? Shit, probably wouldn't even need advanced obfuscation. With a large enough PR and looming deadlines, could easily sneak something like this with a minimal amount of effort into production systems. By the time company even realizes what happens, you are already flying off to a non-extradition country and selling the exfiltrated data on Tor (or dark web).

I’m gonna cry survivorship bias here. How do we know how many similar attempts succeeded? How many of the ones discovered have been written off as honest mistakes? How can we know for sure that e.g. Heartbleed wasn’t put there by someone on purpose (and that that someone isn’t filthy rich now)?

When you get hired to a private company, they know who you are. That’s an immediate deterrent against trying anything funny. On Github, no one knows who you are. It might be harder to backdoor a project without getting noticed, but there is no risk to getting noticed. You can try as many times as you like. Jia Tan is still at large, and didn’t even have to plan their whole life around living in a non-extraditing country (if they aren’t in one already).

https://en.wikipedia.org/wiki/Industrial_espionage (aka corporate espionage)

Happens all the time. Maybe it's a state actor. Maybe it's a disgruntled employee. It's just not in the same lens as you expect (software supply chain attack).

Apple has trouble keeping the lid on top secret projects. Leaks about designs happen all the time prior to scheduled debut at WWDC.

MS has had trouble in the past as well when it came to developing the Xbox (One?).

"Owners of China-Based Company Charged With Conspiracy to Send Trade Secrets Belonging to Leading U.S.-Based Electric Vehicle Company" - https://www.justice.gov/usao-edny/pr/owners-china-based-comp...

"Ex-Google engineer charged with stealing AI trade secrets while working with Chinese companies" - https://www.latimes.com/world-nation/story/2024-03-07/ex-goo...

"The US Hits Huawei With New Charges of Trade Secret Theft" - https://www.wired.com/story/us-hits-huawei-new-charges-trade...

"U.S. charges China-controlled company in trade secrets theft" - https://www.pbs.org/newshour/economy/u-s-charges-china-contr...

"Ex-Google and Uber engineer Anthony Levandowski charged with trade secret theft" - https://www.theverge.com/2019/8/27/20835368/google-uber-engi...

In the case of Levandowski, the dude didn't even bother with covering his tracks. Straight up just downloads trade secrets from source control and transfers them to personal computer - https://www.justice.gov/usao-ndca/press-release/file/1197991...

In this small sample of cases, were the exfiltration attempts as elaborate as the "xz attack"? Probably not, but all of these people were vetted by internal procedures and that did nothing to stop them from acting maliciously.

Forget back dooring the project when getting through the front door is so much easier! People are very relaxed in their walled off garden and cubicle :)

Stealing trade secrets (read: copying source code) is a whole different ballgame from trying to inject backdoors into ubiquitous pieces of software.

This kind of demonstrates my point. Every single one of these headlines indicates the bad actor has been "charged" (with serious consequences in the case of Huawei).

Has Jia Tan been "charged" with anything?

I’m gonna cry survivorship bias here. How do we know how many similar attempts succeeded?

Right, we don’t know that, but we have yet to hear of one case where a supply chain attack on a FOSS project resulted in someone getting arrested.

I don't buy it. Most private companies would require a face-to-face meeting when you start. Even if you're fully remote, the expectation would be that at some point you would meet your coworkers at meetspace, most likely before you get to commit anything substantial. The ones that are worth penetrating will almost certainly require a background check.

And then, once you're in, you cannot just commit to your target project willy-nilly, as your manager and your manager's manager will have other priorities. A for-profit company's frequently dysfunctional management would actually work as deterrent here: you don't just need to justify your code, you will have to justify why you were working on it in the first place.

face to face, background checks. they are all superficial.

A smooth talker can get you to relax your guard.

Identity can be faked, especially if you have a nation state backing you.

How much would you bet against the NSA having a team full of leetcode and interview experts, who's job is to apply at tech companies and perform excellently through the hiring process, so that the offensive team at NSA can infiltrate and work remotely without ever needing to meet there new "coworkers"?

I suspect a "professional job seeker" with the resources of the NSA behind them and who lands 1st and subsequent interviews dozens of times a year - would be _way_ better at landing interviews and jumping through stupid recruiting hoops that even the best senior or "10x engineers", who probably only interview a dozen or two times in their entire career.

Today no one knows what Jia Tan looks like. That means Jia Tan come back as Gary Smith and work on the next exploit. If we knew what he looked like, we could have at least prevented him from coming back.

Jia Tan might be more than one person.

Requiring him/her to show their face might have prevented that.

Why? Just have one person show up and everyone else funnel the work through their account.

That's still one named person the police can go after, a good starting point for investigation into the criminal network.

Also, that's still one named person that no one would like to end up being, so that alone acts as a deterrent.

On open source projects you can vet people based on whether you can personally identify them as real people. Moving forward be suspicious of anonymity for core contributions.

Debian does things that way, their developers have to get their key signed by other developers, and a DD who signs someone's key is supposed to check ID. But there's no similar protection for their upstream.

And using a freaking VPN for everything. Definitely more vetting is needed.

off topic, but how many actual non-extradition countries are there these days (for the US)?

Even countries we have strained relationships with will extradite as part of a negotiation when it's convenient for them politically.

Russia probably wouldn't have even kept Snowden if it wasn't state secrets he revealed. If it was just some random data breach they would have prisoner-swapped him for an oligarch caught money laundering elsewhere.

Our adversaries aren't going to extradite their citizens to the west. And obviously if it's a state-level actor they aren't going to extradite their own actors.

As someone old enough to remember the tail end of the early hacker eras (e.g. Mitnick), I don't think anyone SHOULD be extradited over this, in particular if they're not being charged with actually using the exploit. Prosecute them where they live. Should they be prosecuted 193 times over in every state on Earth? What's the nexus? Github? Every server that installed the compromised xz utils?

But you are right they will deport (not extradite) foreigners who are inconvenient to them or when it is politically expedient to do so, if the foreigners are a nuisance, or as part of a political negotiation or prisoner exchange.

The whole "extradition treaties" meme is a misconception. You will only get extradited if you flee to a country where you are a citizen (even dual citizen), or the ability to assert citizenship/nationality there. A fugitive fleeing to a country without an extradition treaty is subject to deportation. Every country on earth reserves the right to deny entry to or deport foreign fugitives. They might choose not to if someone is found to be a refugee, subject to the death penalty in a non-death-penalty state, etc.

Russia and China would never extradite for this, they would hire the people involved if they aren't already in their employ, and I wouldn't blame them. I'm not even sure if they could be charged with more than misdemeanor charges anyway.

> off topic, but how many actual non-extradition countries are there these days (for the US)?

Several countries do not extradite their own citizens. For citizens of these countries, going back to their own home country would be enough.

Could be interesting as a class project to investigate other small relatively innocuous but near ubiquitous projects that could be treated the same way and investigate whether something similar could be done or has been done already. Just making a list of them would be useful if nothing else.

The lesson I take away from this incident is that we probably shouldn't be allowing anonymity for core contributers in critical open source projects. This attack worked and the attacker will likely get away with it free of consequence, because they were anonymous.

No thanks.

That's not going to help, and will be fairly easy to circumvent for nation state actors or similar advanced persistent threats who will not have a problem adding an extra step of identity theft to their attack chain, or simply use an agent who can be protected if the backdoor is ever discovered.

On the other hand, the technical hoops required for something like that will likely cause a lot of damage to the whole open source community.

The solution here is learn from this attack and change practices to make a similar one more difficult to pull off:

1. Never allow files in release tar-balls which are not present in the repo.

2. As a consequence, all generated code should be checked in. Build scripts should re-generate all derived code and fail if the checked in code deviates from the generated.

3. No inscrutable data should be accessible by the release build process. This means that tests relying on binary data should be built completely separately from the release binaries.

It's easy to steal or craft an identity. Having a person adopt that identity and use it over multiple in-person meetings around the world over an extended period of time is not.

Part of the appeal of cyber operations for intelligence agencies is that there's basically no tradecraft involved. You park some hacker in front of a laptop within your territory (which also happens to have a constitution forbidding the extradition of citizens) and the hacker strikes at targets through obfuscated digital vectors of attack. They never go in public, they never get a photo taken of them, they never get trailed by counterintelligence.

If you start telling people who want to be FLOSS repo maintainers that they'll need to be at a few in-person meetings over a span of two or three years if they want the keys to the project, that hacker has a much harder job, because in-person social engineering is hard. It has to be the same person showing up, time after time, and that person has to be able to talk the language of someone intimately familiar with the technology while being someone they're not.

It's not a cure-all but for supply chain attacks, it makes the operation a lot riskier, resource-intense, and time-consuming.

Many OSS contributors likely don't have "fly to distant country for mandatory meeting" money.

You are excluding a ton of contributors based on geography and income.

It's not common that I find this line actually decent but check your privilege with this kind of comment.

This is really a small step away from segregation.

There are multiple ways to vet identity. Even just knowing location and such. This person used a VPN. A big red flag.

Stop trying to support such a variety of images too? Maybe?

Two problems with this:

1. Many important contributors, especially in security, prefer to be pseudonymous for good reasons. Insisting on identity drives them away.

2. If a spy agency was behind this, as many people have speculated, those can all manufacture "real" identities anyway.

So you'd be excluding helpful people and not excluding the attackers.

> The lesson I take away from this incident is that we probably shouldn't be allowing anonymity for core contributers in critical open source projects. This attack worked and the attacker will likely get away with it free of consequence, because they were anonymous.

This would be impossible to enforce, and might not be a good idea because it enables other ranges of attacks: if you know the identities of the maintainers of critical open source projects, it’s easier to put pressure on them.

If this was a state-actor (which it definitely looks like it) then what validation are you going to do? They can probably manufacture legitimate papers for anything.

Driver’s license, SSN, national ID, passport, etc. If the government is in on it then there’s no limits.

The only way would be to require physical presence in a trusted location. (Hopefully in a jurisdiction that doesn’t belong to the attacker…)

Many more chances to mess up.

Who designates it as critical?

If someone makes a library and other people start using it, are they forced to reveal their identity?

Do the maintainers get paid?

It might prevent attacks under different aliases, but a determined organization will be able to create a verified account, if only because nobody, certainly noy github, has the will and means to verify each account themselves.

The attack almost worked because of too few eyes

if anyone has worked in any development. (closed or open) you know half the time developers are lazy and just approve PRs. Linus Torvalds is like the glimming exception where he will call out shit all day long.

Second this.

And in the event someone is pedantic enough to actually care: that person will be considered a pariah that all development stifles due to.

Tensions with the team for nitpicking etc;

FD: I have a situation like this now, I am not the one being picky- one of the developers I hired is. I had to move him out of the team because unfortunately his nitpicky behaviour was not well regarded. (he also comes from eastern europe and has a very matter-of-fact way of giving feedback too which does not aid things).

This cracks me up. I had the privilege of working on a team where about 5 or 7 of us out of 10 were extremely picky about code and documentation, all different nationalities. We would sometimes argue about a word in a doc for hours and didn't get a whole lot done. Feelings were hurt a lot, but what we did get accomplished was very high quality. I've also worked on teams where we were all cowboys and would sling code just about as fast as we could type it.

Both can be fun, but as you probably already know there is a balance.

The set of unix utilities have been tested a long time. I just wish the kernel and key utilities keeps fixed and not changed. Unless absolutely necessary. Don’t fix it if it ain’t broken. The software empire seems out of control.

It is all broken. The quick hack to make things kind of work today, compounded across fifty years and a million developers.

There are occasional shining lights of stuff done right but I think it's fair to say they're comprehensively outcompeted.

What's the advantage of IFUNCS, over putting a function in the library that selects the implementation, either via function pointers or a switch/if? In particular given that they seem to be quite fragile and exploitable too.

I don't have much experience in low-level optimization, but would a modern CPU not be able to predict the path taken by a brach that tests the CPU features.

The advantage is that it avoids an extra function pointer indirection when the function you are trying to swap out is already an exported library function. For purely internal functions like in this case the only advantage is that it allows this exploit.

> either via function pointers or a switch/if?

> but would a modern CPU not be able to predict the path taken by a brach that tests the CPU features.

That's true, but the CPU has finite branch predictor state, and now you've wasted some of it. Indirect calls hurt too, especially in you need retpolines.

This is a great read: https://www.agner.org/optimize/microarchitecture.pdf

The Linux kernel has interfaces for doing the same thing, more explicitly than ifunc:



Since I'm a bit late to the party and feeling somewhat overwhelmed by the multitude of articles floating around, I wonder: Has there been any detailed analysis of the actual injected object file? Thus far, I haven't come across any, which strikes me as rather peculiar given that it's been a few days.

Your best bet may be in the chat (from https://www.openwall.com/lists/oss-security/2024/03/30/26 ):

Matrix: #xz-backdoor-reversing:nil.im

IRC: #xz-backdoor-reversing on irc.oftc.net

Discord: https://discord.gg/XqTshWbR5F

I agree, I haven't seen anything about decompiling the object file.

If I had a project to develop a backdoor to keep persistent access to whatever machine I wanted, it would make sense that I would have a plug-in executable that I would use for multiple backdoors. That's just decent engineering.

Few notes:

- "auto" tools are installer tools and, after all, do good job. Maybe we can download any project that use it, run it once, parse results and put it in /etc ? - becouse it's already there. Then next step: build just-detection tools. Side note: just straight 'make all' will detect it too :) And it show additional next problem that auto's already solved - setting up all that inc/lib paths...

- m4 should be replaced and not by monstrosity like cmake

- GNU's came after MIT's and BSD's, evolved from them. But corporations adapted and assimilated. We need next evolution step: something that forces corporations to spend money on what they use, fund devs or participate in development. OSlicensecertification.orgs should help with that instead of helping corps to be predators becouse some outdated bullshit talks

- yea, auto* tools are way overcomplicated

- finally do that TTY 2.0 !

So people are talking about the review process but what if the maintainer and the reviewers are all complicit?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact