Hacker News new | past | comments | ask | show | jobs | submit login
Timeline of the xz open source attack (swtch.com)
1069 points by todsacerdoti 13 days ago | hide | past | favorite | 461 comments





Excellent summary of the events, with all the links in one place. This is the perfect resource for anyone who want to catch up, and also to learn about how such things (especially social engineering) unfold in the wild, out in the open.

One thing that could be added, for the sake of completeness: in the part "Attack begins", toward the end, when they are pushing for updating xz in the major distros, Ubuntu and Debian are mentioned but not Fedora.

Looks like the social engineering/pressuring for Fedora started at least weeks before 2024 March 04, according to a comment by @rwmj on HN [1]. I also found this thread on Fedora's devel list [2], but didn't dig too much.

[1] https://news.ycombinator.com/item?id=39866275

[2] https://lists.fedoraproject.org/archives/list/devel@lists.fe...


It would be intetesting if Lasse Collin published his off-list interactions with 'Jia Tan' and any of the other pseudonyms, to get an even better angle on the social engineering parts. Apparently a large part of the campaign was via private channels to Lasse.

Collin has been writing more extensively on IRC. A screenshot of one of his posts can be seen in [this YouTube video](https://youtu.be/0pT-dWpmwhA?t=1158).

He notes that while he was unsatisfied with some of the changes Tan introduced, Tan was nonetheless extremely helpful.


> [...] unsatisfied with some of the changes Tan introduced [...]

That's one way to put it


I was wondering if that would be released also. I hope so but wouldn't blame them if they decide not to.

I wouldn't blame then if they've thrown all their computers into the sea, changed their name, and taken upon a job building timber furniture...

I, too, have had days like that

as an alaskan, those aren't mutually exclusive

Missing the whole Fedora timeline. I was emailed by "Jia Tan" between Feb 27 and Mar 27, in a partially successful attempt to get the new xz into Fedora 40 & 41. Edit: I emailed Russ with the details.

I wondered about this. I saw the note at the bottom "RedHat announces that the backdoored xz shipped in Fedora Rawhide and Fedora Linux 40 beta" but saw nothing in the timeline explaining when/how it made it into Fedora.

These are the Fedora packages for xz that are vulnerable. If you click through the links you can see when they were added:

https://lists.fedoraproject.org/archives/list/devel@lists.fe... (https://archive.ph/diGNB)

This is the rough sequence of events in Fedora:

https://lists.fedoraproject.org/archives/list/devel@lists.fe... (https://archive.ph/e0SdX)


One big take away for me is that we should stop tolerating inscrutable code in our systems. M4 has got to go! Inscrutable shell script have got to go!

Its time to stop accepting that the way we've done this in the past is the way we will continue doing it ad infinitum.


That's a great first step, but ready your pitchforks for this next take, because the next step is to completely eliminate Turing-complete languages and arbitrary I/O access from standard build systems. 99.9% of all projects have the capability to be built with trivial declarative rulesets.

Java's Maven is an interesting case study, as it tried to be this:. A standard project layout, a standard dependency mechanism, pom.xml as standard metadata file, and a standard workflow with standard target(clean/compile/test/deploy). Plugins for what's left.

There might have been a time where it worked, but people started to use plugins for all kinds of reasons, quite good ones in most cases. Findbugs, code coverage, source code generation,...

Today, a maven project without plugins is rare. Maven brought us 95%, but there is a long tail left to cover.


Most of these could still be covered with IO limited to the project files though.

There is a sizable movement in the Rust ecosystem to move all build-time scripts and procedural macros to (be combined to) WASM. This allows you to write turing-complete performant code to cover all use-cases people can reasonably think of, while also allowing trivially easy sandboxing.

It's not perfect, for example some build scripts download content from the internet, which can be abused for information extraction. And code generation scripts could generate different code depending on where it thinks it's running. But it's a lot better than the unsandboxed code execution that's present in most current build systems, without introducing the constraints of a pure config file.


But the xz backdoor didn’t involve a build script that tried to compromise the machine the build was running on. It involved a build script that compromised the code being built. Sandboxing the build script wouldn’t have helped much if at all. Depending on the implementation, it might have prevented it from overwriting .o files that were already compiled, maybe. But there would still be all sorts of shenanigans it could have played to sneakily inject code.

I’d argue that Rust’s biggest advantage here is that build scripts and procedural macros are written in Rust themselves, making them easier to read and thus harder to hide things in than autotools’ m4-to-shell gunk.

But that means it’s important to build those things from source! Part of the movement you cite consists of watt, which ships proc macros as precompiled WebAssembly binaries mainly to save build time. But if watt were more widely adopted, it would actually make an attacker’s life much easier since they could backdoor the precompiled binary. Historically I’ve been sympathetic to dtolnay’s various attempts to use precompiled binaries this way (not just watt but also the serde thing that caused a hullabaloo); I hate slow builds. But after the xz backdoor I think this is untenable.


Ultimately you're going to have to be adept at stuff like the Underhanded C Contest to spot this kind of thing in any Turing-complete language, so the idea of auditing the source is unreliable at worst. So I'd take another page from the Java/Maven ecosystem and require hashed+signed binaries, with the possible addition of requiring builds to be performed on a trusted remote host so that at least we can verify the binary is produced from the same source and results in the same output.

But determined actors with access are always going to try to thwart these schemes, so verification and testing is going to need to step up.


> to spot this kind of thing in any Turing-complete language

I’m not sure I follow. The bar for passing review and audits is “people can understand it”, at least to the point of preventing smuggling in a custom public keys and other shenanigans – as opposed to “all valid programs in the language”. While I agree Turing-completeness opens up undeniable complexity, it’s the best we got. And there are huge variations in obscurity across languages and tooling.


Underhanded C is meant to create code that is transparent enough to convince the reader that it's obviously and clearly doing one thing, while subtly doing something else.

reproducible builds never made sense to me. if you trust the person giving you the hash, just get the binary from them. you don't need to reproduce the build at all.

if you trust that they're giving you the correct hash, but not the correct binary, then you're not thinking clearly.


One thing a culture of reproducible builds (and thus stable hashes) however does provide is a lack of excuse as to why the build isn't reproducible. Almost nobody will build from source - but when there's bugs to be squashed and weird behavior, some people, sometimes, will. If hashes are the norm, then it's a little harder for attackers to pull of this kind of thing not because you trust their hash rather than their blob, but rather because they need to publish both at once - thus broadening the discovery window for shenanigans.

To put it another way: if you don't have reproducible builds and you're trying to replicate an upstream artifact then it's very hard to tell what the cause is. It might just be some ephemeral state, a race, or some machine aspect that caused a few fairly meaningless differences. But as soon as you have a reproducible build, then failure to reproduce instantly marks upstream as being suspect.

It's also useful when you're trying to do things like tweak a build - you can ensure you're importing upstream correctly by checking the hash with what you're making, and then even if your downstream version isn't a binary dependent on upstream (e.g. optimized differently, or with some extra plugins somewhere), you can be sure that changes between what you're building and upstream are intentional and not spurious.

It's clearly not a silver bullet, sure. But it's not entirely useless either, and it probably could help as part of a larger ecosystem shift; especially if conventional tooling created and published these by default such that bad actors trying to hide build processes actually stick out more glaringly.


Actually, this xz-utils was a sort of reproducible build issue. But the twist it is wasn't the binary not built reproducibly. It was the tar ball. The natural assumption is it just reflected the public git repository. It didn't.

Debian's response is looking to be mandating the source must come from the git repository, not a tar ball. And it will be done using a script. And the script must produce the same output every time, ie be reproducible. Currently Debian's idea of "reproducible" means it reflects the source Debian distributes. This change means it will reproducibly reflect the upstream sources. That doesn't mean it will be the same - just that it's derived in a reproducible way.

As for trusting the person who gave you the hash: that's not what it hinges on. It hinges on the majority installing a binary from a distro. That binary can be produced reproducibly, by an automated build system. Even now, Debian's binaries aren't created just once by Debian. There are Debian builders all around the planet creating the binary, and verifying it matches what's being distributed.

Thus the odds of the binary you are running not being derived from the source are vanishingly low.


So if something is hidden in the source, a reproducible build will give you the confidence to believe that the source is fully vetted and clean.

I see what you’re saying, but I don’t buy that reproducible builds actually solve anything, especially long term. As this whole xz thing has shown us, lots of things fly under the radar if the circumstances are right. This kind of thing will absolutely happen again. In the future it may not even require someone usurping an existing library, it could be a useful library created entirely by the hacker for the express purpose of infiltration a decade later.

Reproducible builds are a placebo. You must still assume there are no bad actors anywhere in the supply chain, or that none are capable of hiding anything that can be reproducibly built, and we can no longer afford to make that assumption.

A reproducible vulnerability is still a vulnerability.


You're focusing on the wrong issue. Just because reproducible builds don't solve all avenues of attack doesn't mean they're worthless. No, a reproducible build does not give you any confidence about the quality of the source. It gives you confidence that you're actually looking at the correct source when your build hash matches the published hash, nothing more.

A window that can be smashed in is still a vulnerability, so there is no value in people locking their front doors.


> I see what you’re saying, but I don’t buy that reproducible builds actually solve anything,

You're right. They don't solve anything, if you restrict "anything" to mean stops exploits entering the system. What reproducible builds do is increase visibility.

They do that by ensuring everybody gets the same build, and by ensuring everybody has the source used to do the build, and by providing a reliable link in the audit trail that leads to when the vulnerability was created and who did it. They make it possible to automatically verify all that happened without anybody having to lift a finger, and it's done in a way that can't be subverted as Jia Tan did to the xz utils source.

Ensuring everyone has the same build means you can't just target a few people, everyone is going to get your code. That in turn means there will be many eyes like Andres looking at your work. Once one of them notice something, one of them is likely to trace it back to the identity that added it. Thus all these things combine to have one overall effect - they ensure the problem and it's cause are visible to many eyes. Those eyes can cooperate and combine their efforts to track down the issue, because they are all are absolutely guaranteed they are looking at the same thing.

If you don't think increasing visibility is a powerful technique for addressing problems in software then you have a lot to learn about software engineering. There are software engineers out their who have devoted their entire professional lives to increasing visibility. Without them creating things that merely report and preserve information like telemetry and logging systems software, would be nowhere as reliable as it is now. Nonetheless, if the point you are making is when we shine a little sunlight into a dark corner, the sunlight itself does nothing to whatever we find there, you're right. Fixing whatever the sunlight / log / telemetry revealed is a separate problem.

You aren't correct in thinking visibility doesn't reduce these sorts of attacks. By making the attack visible to more people, it increases the chance it will be notice on any given day, thus reducing it's lifetime and making it less useful. Worse, if the vulnerability was maliciously added the best outcome for the identity is what happened here - the identity was burned, all its contributions where also burned and so a few years of work went down the drain. At worst someone is going to end up in jail, or dead if they live in authoritarian state. The effect is not too different from a adding surveillance cameras in self-checkouts. Their mere presence makes honesty look like a much better policy.


But you can build the code yourself, and verify that your hash is the same as theirs, meaning the binary is the same.

Just get their binary, then. Why do you need to build it yourself?

If you trust the people giving you the hash of the binary, just get the binary from them, too.


Reproducible builds let you combine trust from multiple verifyers. If verifyers A, B and C verify that the build produces the stated hash then you can trust the binary if any of A, B or C is trustworthy.

Or in other words, the point is not for everyone to verify the build produces the expected binary since that would indeed make the published binaries pointless. Instead, most people trust the published hash because it can be independently verified and anyone can call out the publisher of the binary if it doesn't match.


It's easier to hide nastiness in a binary than it is in source. And indeed this XZ hack too relied on multiple obfuscated binary blobs. The hash helps because it makes it a little harder to hide things like this. It's not a silver bullet, but it would have in this specific instance made it harder to hide that malicious m4/build-to-host.m4 in the tarball - after all, had the attacker done that despite publishing a hash they would have needed to use the hash including the modified build script, but then anybody building from the git repo would have had a different hash, and that's a risk for detection.

Reproducible builds and hashes thereof aid in transparency and detecting when that transparency breaks down. Of course, it doesn't mean hackers can't hide malicious code in plain sight, but at least it makes it slightly harder to hide between the cracks as happened here.


"slightly harder" isn't enough. That's what I'm saying that people are not accepting.

The days of 3rd party libraries simply being trusted because they're open source are slowly coming to an end. the problem is not unreproducible builds, the problem is that we implicitly trust source code found online.

Projects relying on 100s of 3rd party libraries are a major problem, and no one seems to care. They like it when their Rust build lists all the libraries that they didn't have to write as it compiles them. They like it when they can include an _extremely_ small Node library which itself relies on 4 other _extremely_ small Node libraries, which each rely on another 4 _extremely_ small Node libraries, until you have a node_modules directory with 30,000 packages in it.

We don't know how to write software any more. We know how to sew bits of things together into a loincloth, and we get mad when the loincloth makes us itch. Of course it does, you didn't LOOK at anything you are relying on. You didn't stop to think, even for a moment, that maybe writing the thing instead of trusting someone else with it was even an option.

As we continue doing this, relying on 3rd parties to write the heavy lifting code, we lose the skill to write such code ourselves. We are transferring the skills that we need into the hands of people that we can't trust, when viewed from a high level.

We need to get back to small applications, which we know because we wrote them in their entirety, and that someone else can trust because the code for the thing is 2500 lines long and has extremely few, if any, dependencies.

We need to get away from software which imports 30,000 (or even 100) third party libraries with implicit trust because it's open source.

"All bugs are shallow with enough eyes" requires that people use their eyes.


Sometimes the perfect is the enemy of the good. As the XZ saga shows, even clearly exceptionally well organized attackers don't have an easy time injecting hacks like this; and things that increase the attackers costs or risks, or reduce their reward can be useful even if they don't solve the problem entirely. Reproducible builds are useful; they don't need to be silver bullet.

compression. encryption. handshakes. datetime. rng or prng. i'm right there with ya but nontrivial tasks require nontrivial knowledge. i don't have an answer for obfuscated backdoors like here, or bad code that happens, but i do know if i tried to audit that shit, i'd walk away telling everyone i understood nothing

Maybe not untenable.

If everything is done carefully enough with reproducible builds, I think using a binary whose hash can be checked shouldn't be a great extension of trust.

You could have multiple independent autobuilders verifying that particular source does indeed generate a binary with the claimed hash.


The xz backdoor did hide the payload in test data though. Proper sandboxing could have meant that eg. building the binary wouldn't have access to test data.

How does the sandboxing help if the compiler and/or build scripts or whatever modifies its own output?

When your programmatic build steps are isolated in plugins, then you can treat them like independent projects and apply your standard development practices like code review and unit tests to those plugins. Whereas when you stuff programmatic build steps into scripts that are bundled into existing projects, it's harder to make sure that your normal processes for assuring code quality get applied to those pieces of accessory code.

My standard development practices like code review and unit tests do not scale to review and test every dependency of every dependency of my projects. Even at company-wide scale.

I'm not saying that. I'm just saying that improved ability to apply such development practices is one benefit of using a plugin-style architecture for isolating the programmatic steps of your build pipeline. It's not perfect but in many ways it's still a significant improvement upon just allowing arbitrary code right in the pipeline definition.

Maybe now that we have things like GitHub Actions, Bitbucket Pipelines, etc., which can run steps in separate containers, maybe most of those things could be moved from the Maven build step to a different pipeline step?

I'm not sure how well isolated the containers are (probably not very – I think GitHub gives access to the Docker socket) and you'd have to make sure they don't share secret tokens etc., but at least it might make things simpler to audit, and isolation could be improved in the future.


> Findbugs, code coverage, source code generation,...

For the purpose of this conversation we mostly just care about the use case of someone grabbing the code and wanting to use it in their own project. For this use case, dev tools like findbugs and code coverage can be ignored, so it would suffice to have a version of the build system with plugins completely disabled.

Code generation is the thornier one, and we can at least be more principled about it than "run some arbitrary code", and at least it should be trivial to say "this codegen process gets absolutely no I/O access whatsoever; you're a dumb text pipeline". But at the end of the day, we have to Just Say No to things like this. Even if it makes the codebase grodier to check in generated code, if I can't inspect and audit the source code, that's a problem, and arbitrary build-time codegen prevents that. Some trade-offs are worth making.


The xz debacle happened partiallybecause the generated autoconf code was provided. Checking in generated code is not that much better. It's a bit more visible, but not much people will spend their limited time to validate it, as it's not worth it for generated code. xz also had checked in inscrutable test files, and nobody could know it was encrypted malware.

I'm not a fan of generated code. It tends to cause misery, being in a no mans land between code and not-code. But it is usefull sometimes, e.g rust generating an API from the opengl XML specs.

Sandboxing seems the least worst option, but it will still be uninspected half code that one day ends up in production.


> The xz debacle happened partiallybecause the generated autoconf code was provided.

The code was only provided in a roundabout way that was deliberately done to evade manual inspection, so that's not a failure of checking in generated code, that's a failure of actually building a binary from the artifacts that we expect it to be built from. Suffice to say, cutting out the Turing-complete crap from our build systems is only one of many things that we need to fix.


In this case, I think the GP is absolutely right. If you look at the infamous patch with a "hidden" dot, you may think "any C linter should catch that syntax error and immediately draw suspicion." But the thing is, no linter at the moment exists for analyzing strings in a CMakeLists.txt file or M4 macro. Moreover, this isn't something one can reliably do runtime detection for, because there are plenty of legitimate reasons that program could fail to compile, but our tooling does not have a way to clearly communicate the syntax error being the reason for a compilation failure.

> there are plenty of legitimate reasons that program could fail to compile

It's worse than that, these small C files are required to fail unless the target build environment meets some specific requirements.


It could be required to declare which error should happen, and at what line it should trigger. And the C program should still be syntactically valid and pass linting. One could create a tool which would have a folder of .c files to check with such declarations. And output similar things to as the configure script.

Configure scrips tend to be a very slow part of projects, so a new tool that would support parallel compile/run of these checks would also have a considerable speedup. In addition to being more reviewable, verifiable and less error prone.


All good things if you want to use the autotools. And other tools too.

I've been porting and compiling Unix programs since 5 years before Linux. There was a lot of work needed to port between Unix variants, or even worse, non-Unix variants. So I can see the problem that the autotools solve.

But I never spent much time learning the details on how the autotools work. When './configure' has some problem, it is a very steep uphill battle to get any traction on the issue.

So I'd be more in the camp of looking for a different solution than autotools.


Yeah a lot of the complexity that configure scripts are supposed to solve are massively reduced now. Both because there is better harmonization between platforms, and many checks in a N year old are probably no longer relevant.

So a first pass should probably be to eliminate as much checks as possible. But there will almost always be a need for a couple of build time checks/switches - should be supported by some kind of tooling.


I'm not very familiar with CMake, would it be viable and better to change over to that?

Cmake has a set of standardized macros to perform checks on the platform being built for. Including many specialized for typical usecases, like checking if a C header is present. And the generic CheckCompiles allows specifying a regex to match for check to be considered "failed".

https://cmake.org/cmake/help/latest/manual/cmake-modules.7.h...

So it seems considerably better than configure scripts. Probably also better than my idea :p I don't know how easy it would be for an adversary to do something "underhanded" though, or how easy that would be to spot such.


What would that accomplish? It certainly wouldn't have stopped this attack.

> 99.9% of all projects have the capability to be built with trivial declarative rulesets.

Only if you forbid bootstrapping, which all projects ultimately rely on at some point in their supply chain.


> What would that accomplish? It certainly wouldn't have stopped this attack.

We could write an entire PhD thesis on the number of dire technical failings that would need to be addressed to stop this attack, so while this alone wouldn't have stopped it, it would have required the actor to come up with another vector of code injection which would have been easier to find.

> Only if you forbid bootstrapping

Codebases that bootstrap are the 0.1%. Those can be built via `bash build.sh` rather than deceptively hiding a Turing-complete environment behind a declarative one. Even if you need to have these in your trusted computing base somewhere, we can focus auditing resources there, especially once we've reduced the amount of auditing that we need to do on the other 99.9% of codebases now that we've systematically limited the build-time shenanigans that they can get up to.


Concretely, what security issues are solved by forcing the build specification language to be Turing incomplete? My guess is the answer is "none."

At worst, you're actually creating more holes. The reason autoconf/automake exist and M4 scripts are innocuous in the first place is because the build system uses an underpowered language and developers have to turn to code generation to get around it.

If you kneecap the build system's language you're not solving problems. You're creating them.

> it would have required the actor to come up with another vector of code injection which would have been easier to find.

If make was standardized and could programmatically determine the environment its run under and write full programs then the attack vector wouldn't exist in the first place.

> Codebases that bootstrap are the 0.1%.

We have different experiences, because ime it's close to 100% especially when you include transitive dependencies. When you care about supply chain security you care about being able to bootstrap from sources for your code and all your dependencies, and it's almost guaranteed that one of your dependencies needs to be bootstrapped.


I think that what we need is good sandboxing. All a sandboxed Turing-complete language can do is perform I/O on some restricted area, burn CPU and attempt to escape the sandbox.

I would like to see this on the language level, not just on the OS level.


I have been thinking the exact same thing, and specifically I would like to try implementing something that works with the Rye python manager.

Say I have a directory with a virtualenv and some code that needs some packages from PyPI. I would very much like to sandbox anything that runs in this virtualenv to just disk access inside that directory, and network access only to specifically whitelisted URLs. As a user I should only need to add "sandbox = True" to the pyproject.toml file, and optionally "network_whitelist = [...]".

From my cursory looking around, I believe Cloudflare sandbox utils, which are convenience wrappers around systemd Seccomp, might be the best starting point.

Edit: or just use Firejail, interesting...

You mention sandboxing on the language level, but I don't think it is the way. Apparently sandboxing within Python itself is a particularly nasty rabbit hole that is ultimately unfruitful because of Python's introspection capabilities. You will find many dire warnings on that path.


Can you explain more in-depth what you mean? I'm also unaware of how you could have declarative rulesets in a non turing-complete language.

Sounds like it would be impossible but maybe my thinking is just enclosed and not free.


Haven’t used it much but Dhall is a non Turing complete configuration language: https://dhall-lang.org/

What does modern C project management look like? I'm only familiar with Autotools and CMake.

Redis. Simple flat directory structure and a Makefile. If you need more, treat it as a sign from God you're doing something wrong.

This is a better take. Though I'm sure people can obfuscate in source code just as much as in build configuration code.

until you have to integrate with the rest of the world sure

Looking at the state of software security in the rest of the world, this may not be much of a disincentive. At some point we need to knuckle down and admit that times have changed, the context of tools built for the tech of the 80s is no longer applicable, and that we can do better. If that means rewriting the world from scratch, then I guess we better get started sooner rather than later.

This doesn't read as a technical failure to me. This was 99% social engineering. I understand that the build system was used a vector, but eliminating that vector doesn't mean all doors are closed. The attacker took advantage of someone having trouble.

While I do agree that M4 is not great I don't believe any alternative would have prevented this attack: you could try translating that build-to-host file to say python with all the evals and shell oneliners and it would still not be immediately obvious for a distro package maintainer doing a casual review after work: for them it would look just like your average weekend project hacky code. Even if you also translated the oneliners it wouldn't be immediately obvious if you were not suspicious. My point is, you could write similarly obfuscated code in any language.

What's inscrutable code? Was it m4 or sh or the combination of the two?

Who will pay for all the rewriting you want done? Or even just for the new frameworks that are "scrutable"? How do we guarantee that the result is not inscrutable to you or others?

There is so much knee-jerking in this xz debacle.

(And I say this / ask these questions with no love for autoconf/m4/sh.)


I think this is pretty straight forward. Don't accept PRs for code that cannot be reasoned about easily.

We are all paying right now for it not being done.

I do think we have enough eyeballs at this point that we should stop entertaining the Dancing Bear in low level libraries and start insisting on crisp, self-explaining code. There are a lot of optimizations pushed into compilers these days, and there are a lot of architectural changes that can make things fast without making them inscrutable.

We should be moving from No Obvious Bugs to Obviously No Bugs (Tony Hoare).


>start insisting on crisp, self-explaining code

That’s the beauty of FOSS: if you don’t like what’s written, you can write your own replacement.


And now we have 14 standards.

That would be an improvement for sure... but this is not fundamentally a technical problem.

Reading the timeline, the root cause is pure social engineering. The technical details could be swapped out with anything. Sure there are aspects unique to xz that were exploited such as the test directory full of binaries, but that's just because they happened to be the easiest target to implement their changes in an obfuscated way, that misses the point - once an attacker has gained maintainer-ship you are basically screwed - because they will find a way to insert something, somehow, eventually, no matter how many easy targets for obfuscation are removed.

Is the real problem here in handing over substantial trust to an anonymous contributor? If a person has more to lose than maintainership then would this have happened?

That someone can weave their way into a project under a pseudonym and eventually gain maintainership without ever risking their real reputation seems to set up quite a risk free target for attackers.


> Is the real problem here handing over substantial trust to an anonymous contributor?

Unless there's some practical way of comprehensively solving The Real Problem Here, it makes a lot of sense to consider all reasonable mitigations, technical, social or otherwise.

> If a person has more to lose than maintainership then would this have happened?

I guess that's one possible mitigation, but what exactly would that look like in practice? Good luck getting open source contributors to accept any kind of liability. Except for the JiaTans of the world, who will be the first to accept it since they will have already planned to be untouchable.


Create an organization of paid professionals who are responsible for maintaining these libraries (and providing support to library users).

It's heartblead all over again (though those were honest mistakes, not an intentional attack


> I guess that's one possible mitigation, but what exactly would that look like in practice? Good luck getting open source contributors to accept any kind of liability. Except for the JiaTans of the world, who will be the first to accept it since they will have already planned to be untouchable.

It's not necessary to accept liability, that's waived in all FOSS licenses. What I'm suggesting would only risk reputation of non-malicious contributors, and from what I've seen, most of the major FOSS contributors and maintainers freely use their real identity or associate it with their pseudonym anyway, since that attribution comes with real life perks.

Disallowing anonymous pseudonyms would raise the bar quite a bit and require more effort from attackers to construct or steal plausible looking online identities for each attack, especially when they need to hold up for a long time as with this attack.


Agreed it's mainly a social engineering problem, BUT also can be viewed as a technical problem, if a sufficiently advanced fuzzer could catch issues like this.

It could also be called an industry problem, where we rely on other's code without running proper checks. This seems to be an emerging realization, with services like socket.dev starting to emerge.


Autotools-based build infra is always crufty, fragile, slow, and fugly. In a lot of cases, it sucks because it's what we have right now without doing the work to replace it with something equally flexible.

Build infrastructure should be minimal, standardized, and not subject to endless special, undocumented fragility.

cmake, just, conan, meson, and bazel (and forks) exist.

But I've still yet to see a proper build system that does feature detection in parallel and concurrently, or supports live, incremental, continuous, cached rebuilding.


Functionality like IFUNC is completely inexcusable outside of dev/debug builds imo. It's rot.

That and we need to pay open source maintainers and find new ways to support them.

And all code that gets linked into security critical applications/libraries needs to be covered by under some sort of security focused code review.

So no patching the compression code that openSSL links to with random junk distribution maintainers.


That's an improvement, but ultimately not enough, actually. The article hits at this, and you should definitely read it:

Being able to send xz patches to the Linux kernel would have been a nice point of leverage for Jia Tan's future work. We're not at trusting trust [1] levels yet, but it would be one step closer.

[1]: https://research.swtch.com/nih


With all the CI tooling and containerization, it seems to be going in the opposite direction.

[flagged]


Nonsensical argument. JavaScript and TypeScript projects are developed with patches of the original source not some compressed artefact. Take your trolling elsewhere

[flagged]


Javascript is almost always executed in sandboxed, unprivileged environments. The issue here is that this type of obfuscation is easy to add in core os libraries. The JavaScript ecosystem, for all the hate that it gets, makes it super easy to sandbox any running code.

It doesn't matter if it's minified or obfuscated because you basically have to run unknown, untrusted code everywhere while browsing the web with JavaScript turned on. So the ecosystem and tooling is extremely resilient to most forms of malicious attacks no matter how minified or obfuscated the js you're running is. The complete opposite is true for bash and shell scripting in general


Javascript is pretty much guaranteed to be permanent. It is the language of the web.

(There's webassembly too, but that doesn't remove js)


I don't think anything is "pretty much guaranteed": things evolve

Hence the "pretty much".

Great, let the cmake dummies wander off into their own little dreamworld, and keep the professionals dealing with the core stuff.

I think one of the good things to come out of this may be an increased sense of conservatism around upgrading. Far too many people, including developers, seem to just accept upgrades as always-good instead of carefully considering the risks and benefits. Raising the bar for accepting changes can also reduce the churn that makes so much software unstable.

> Far too many people, including developers, seem to just accept upgrades as always-good instead of carefully considering the risks and benefits.

Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability, since it was introduced early in the 2.x series. The old 1.x log4j versions had other known vulnerabilities, but only if you were using less common appenders or an uncommon server mode or a built-in GUI log viewer (!); the most common use of log4j (logging into a local file) was not exposed to any of these, and in fact, you could remove the vulnerable classes and still have a functional log4j setup (see for instance https://www.petefreitag.com/blog/log4j-1x-mitigation/ which I just found on a quick web search).

Did log4shell (and a later vulnerability which could only be exploited if you were using Java 9 or later, because it depended on a new method which was introduced on Java 9) lead people to question whether always being on the "latest and greatest" was a good thing? No, AFAIK the opposite happened: people started to push even harder to keep everything on the latest release, "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".


> Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability

Lol, this exact thing happened at my last gig. When I first learned of the vulnerability I panicked, until I found out we were so outdated it didn't affect us. We had a sad laugh about it.

> "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".

I think there is some truth to this motivation though - if you are on an ancient 1.X version and have to jump a major version of two, that almost always causes pain depending on how critical the service or library is. I don't pretend to know the right answer but I always tend to wait several versions before upgrading so any vulnerabilities or fixes can come by the time I get to the upgrade.


A lot of people were in that exact same situation. So many, that the original author of log4j 1.x released a fork to allow these people to keep using the old code while technically being "up to date" and free of known vulnerabilities: https://reload4j.qos.ch/

> "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".

Isn't this still true, generally speaking? How many servers are daily compromised due to out of date, exploitable versions of software versus XZ-like sophisticated attack?


About a decade ago, the industry shifted from slow, carefully evaluated infrequent updates (except for security) to frequent, almost daily updates that are mandatory. I'd say this was pioneered by the Chromium team and proved to be beneficial. The rest of the industry followed.

Now we're in a position where most projects update so quickly, that you don't really have a choice. If you need to update one component, there's a good chance it will require that many more as dependencies in your project will require an update to be compatible.

The industry as a whole sacrificed stability and some aspects of security for faster advancements and features. Overall I'd say the net benefit is positive, but its times like these that remind us that perhaps we need to slow things down just a little and do a bit of a course correction to bring things into a better balance.


All things considered I'm not sure that'd be such a good thing

How many security issues spring from outdated packages vs packages updated too hastily?


On top of that:

A newly-introduced security issue tends to have very limited exploitability, because it's valuable, not-yet well understood, and public exploits are yet to be developed.

Compare to that a similar vulnerability in an older package: chances are that everything about it has been learned and is publicly known. Exploits have become a commodity and are now part of every offensive security distro on the planet. If you run that vulnerable version, there's a real risk that a non-targeted campaign will randomly bite you.


> valuable, not-yet well understood, and public exploits

Except in the scenario that is this exact case: Supply chain attacks that are developed with the exploit in mind.


I agree in principle. But even if the backdoor is deliberate (as is the case here), there’s limited risk for the average person. Nobody in their right mind is going to attack Jane Doe and risk burning their multi-million dollar exploit chain.

For an old vulnerability, however, any unpatched system is a target. So the individual risk for the average unpatched system is still orders of magnitude higher than in the former scenario.


and when you do get a security issue and you're using a 10 year old version the upgrade is going to be really really difficult vs incremental upgrades when they are available. or are you going to fork and assume responsibility for that library code too?

This drove me crazy at previous companies where our build system dependencies were specified with wildcards so they would just auto pull in the latest version whenever you built the project. Not only are there security issues with that, as seen here, but it destroys your ability to have deterministic builds; every time you build the project it might be building against a different set of dependencies. You lose the ability to even compare builds against each other, because even though the builds are for the same exact project code, the dependency binaries might be different, and so the two builds could run differently.

The 2017 WannaCry ransomware attack would be a good counter example, the virus spread even though it was already fixed in an update from MS a bit more than a month before.

I'm kinda the opposite. Way too many times I've seen "upgrades" actively remove things I liked and add things I hate. I hold off on letting mobile apps update because they almost always get worse, not better.

I think it was Ted Neward who argued that libraries should come with a core and ancillary packages. Only the stuff really necessary and broadly popular goes into the base library, and all of the low use features should go into one or more ancillary libraries.

I still think this is the way to go, but it does move some of the problem around in that you have to pick good cleave lines such that people aren’t forced to pull 5 libraries in for 5 features they need. You want one software domain to use two or three and another to use an overlapping set of three.


Wouldn't that just result in exploits written for old versions? A successful exploit for something that everyone is running might be worse, than a backdoor on blending edge systems.

Everyone being on different versions results in something like a moving target


well, it's the bazaar vs the cathedral, isn't it? bazaar moves a lot faster. Everyone likes that part, except when it breaks things, and when they have to chase an upstream that's constantly churning, etc. but most people don't consider that a cathedral itself might have some engineering merit too. cathedrals are beautiful and polished and stable.

I highly encourage people to try freeBSD sometime. Give ports a try (although the modern sense is that poudrie is better even if you want custom-built packages). See how nicely everything works. All the system options you need go into rc.conf (almost uniformly). Everything is documented and you can basically operate the system out of the FreeBSD Handbook documentation (it's not at all comparable to the "how to use a window or a menu" level intro stuff the linux provides). You can't do that when everything is furiously churning every release. everything just works, everything is just documented, it's an experience when you're coming from linux.

and that forum post from 2007 on how to tweak a service script is probably still valid, because BSD hasn't had 3 different init systems over that timespan etc.

just like "engineering is knowing how to build a bridge that barely doesn't fall over", engineering here is knowing what not to churn, and fitting your own work and functionality extensions into the existing patterns etc. like it doesn't have to be even "don't make a bigger change than you have to", you just have to present a stable userland and stable kernel interface and stable init/services interface. the fact that linux doesn't present a stable kernel interface is actually fairly sketchy/poor engineering, it doesn't have to be that way, a large subset of kernel interfaces probably should be stable.


I don't believe this will result in conservatism around upgrading nor that it would be good if it did. The vast, vast majority of security vulnerabilities are due to regular bugs and once the details are known, they can be exploited. Not patching leaves you open to these exploits.

Supply chain attacks are a real threat too, but being conservative about upgrading to improve your security posture is like saying "I'm worried someone with the key to my house might rob me, so I'm going to remove the lock from my door."


That's a double-edged sword. What happens when you need to upgrade in order to get vulnerability fixes?

Maybe one of the outcomes of this could be a culture change in FOSS towards systematically banning rude consumers in Github issues, or, just in general, a heightened community awareness making us coming down on them way harder when we see it happen.

The attackers will leverage any culture that helps them accomplish their goals.

If being rude and pushy doesn’t work, the next round will be kind and helpful. Don’t read too much into the cultural techniques used, because the cultural techniques will mirror the culture at the time.


Even if the security outcome is the same, I would still count people being kind and helpful online instead of rude as an improvement.

As always, there's an xkcd for that https://xkcd.com/810/

That's amazing.

Spot on. The counter should be sound regardless of any social or cultural context, a process where being polite or rude, pushy or not is irrelevant.

Agree. I think a more core issue here is that only 1 person needed to be convinced in order to push malware into xz

The Jia Tan character was never rude. If you make rudeness the thing that throws a red flag, then ‘nice’ fake accounts will bubble up to do the pressuring.

The assumption is that the group behind this attack had sock puppets that were rude to Lasse Collin, to wear him down, and then Jia Tan swept in as the savior.

Jia Tan wasn't rude, but the original maintainer Laser Collin probably wouldn't have been as burned out and willing to give responsibility to them if the community wasn't as rude and demanding of someone doing free work for them.

I think we need to start paying more of these open source maintainers and have some staff/volunteers that can help them manage their git hub issue volume.


The article covers that those rude accounts may have been sybils of the attacker to create pressure. It's effectively good cop/bad cop for open source.

There was definitely good cop/bad cop going on. That’s a really powerful psychological tool.

Pressuring the maintainer is already rude in itself and being polite about it won't help them

If they want things done quickly they can do it themselves


> If they want things done quickly they can do it themselves

I mean they kind of did. And that was the problem.


I want to caution against taking a good thing too far.

There's a certain kind of talented person who is all too conscious of their abilities and is arrogant, irascible, and demanding as a result. Linus Torvalds, Steve Jobs, Casey Muratori come to mind. Much as we might want these characters to be kinder, their irascibility is inseparable from their more admirable qualities.

Sometimes good things, even the best things, are made by difficult people, and we would lose a lot by making a community that alienates them.


That's a tough one - It's hard to fully disagree but in my experience you can have all the benefits without the poison. Accepting the poison just because of the benefits is kind of just giving up. I don't feel like the your hypothesis that the two are irrevocably linked holds up under examination.

Linus Torvalds is apparently trying to do better (although I haven't followed up with the progress), but more importantly, while he might be (have been) unnecessarily rude and aggressive, he's not entitled (as far as I know). I don't think he would jump into an issue tracker of some project he doesn't maintain and demand that certain changes be made.

After watching a recent talk with Linus Torvalds[0], it seems that he's become more soft spoken.

[0] clip about the Nvidia "incident" https://youtu.be/wvQ0N56pW74


There are plenty of hyper-competent technical people in the field who are also kind and patient. Being smart doesn't turn someone into a jerk.

People have been bullied out of 'nice' communities. See the 'Actix' debacle in Rust.

While I don't condone some of the treatment he received, that situation was extremely different.

A user reported a safety issue, the maintainer said it was safe. Then it was proven that it was in fact unsafe, and the maintainer justified it with performance. Then a PR was filed which was safe and did not regress performance, and the maintainer rejected it with "this patch is boring"

The behavior of both sides was deeply unacceptable. If someone identifies a legitimate issue and files a PR to fix it, don't insult them by calling the patch "boring" and don't reject it solely on that basis.


That was mostly redditors though. Reddit is not a nice community.

Most "nice" communities aren't all that nice if they consider you to be part of the out group.

Redditors on r/rust are ostensible the same as saying Rust programmers.

I don't want to excuse rudeness or a sense of entitlement. But I think we can still understand where it comes from. A lot of these people probably work on crappy codebases where "let's just add a random dependency without any vetting" was the norm, they might have to deal with production issues etc. There's probably a systemic issue behind it, that our industry relies too much on unpaid labour and is usually not willing to contribute back.[0]

[0] Funnily enough, just a week or two ago, I fixed an issue in an OS project that we introduced at work. It was an easy frontend fix even for someone like me who doesn't do frontend and barely knows how to spell Vue. And more importantly, in the issue description somebody already wrote exactly what causes the bug and what would need to change - the only thing left was finding the place where to make the (one-line) change. Somehow that issue had been open for 2 years but nobody of the several people who complained (nor the maintainer) had bothered to fix it. After I made a PR, it was merged within a day.


Just don't do anything crazy. There are legitimately crazy people asking for crazy things, not necessarily backdoors.

Being rude is... unimportant. A lot of people think being passive aggressive is being polite when it's actually being rude + deceitful. There's nothing wrong with being direct, which some mistake for rude. I find it refreshing.

Never allow yourself to be bullied or pressured into action. As a maintainer, the more a contributor or user nags, the less likely I am to oblige.

The issue here is the attackers will quickly move away from an individual attacking you to the group attacking you. The person writing the infected code will never be a jerk to you at all. You'll just suddenly see a huge portion of your mailing list move against you ever so slightly.

We've complained about bots in social media for a long time, but how many people in open source discussions are shady manipulative entities?


This is so true, distrusting the guys never being jerk to you while others do is the hardest thing ever. Intentions will never be public :/

These days you can even have these emails automatically taken in by an LLM and have the LLM argue with the maintainer for you, no humans needed!

Maintainers will need LLM sockpuppets of their own to automatically answer these automatic emails.

But in this case he was getting hit by both someone willing to help and then multiple people complaining that things were taking too long. And when you yourself feel like things are taking too long then you’re probably more susceptible to all this.

That sounds nice.

I did an engineering/program manager role for 8 years and people pretty much always did what I asked if I showed up at their desk or bothered their boss.

"Squeaky wheel gets the grease?"

But I too like to think that I prioritize my children on merit rather than fuss level. For some reason they continue to cry despite me claiming I don't react to it.


True, but a determined adversary like JiaTan/Jugar has an ace up their sleeve: they are good enough, and patient enough, to be able to fork the base project, spend a year or two making it better than the original (releasing the head of steam built up from legitimate requests that the old, overworked maintainer never got too, building goodwill in the process) and then convincing the distros to pick up their fork instead of the older original. At which point it really is game over.

> and then convincing the distros to pick up their fork instead of the older original.

Given the current situation, I'm slightly worried about Fedora's planned move to zlib-ng instead of zlib in the next release (https://fedoraproject.org/wiki/Changes/ZlibNGTransition).


The social side of this is really haunting me over the last days. It's surprisingly easy to pressure people to giving up control. I've been there myself. I can't even imagine how devastating this must be to the original author of XZ, especially if he is dealing with other personal issues as well. I hope at least this will serve a strong example to other open source people, to never allow others to pressure them into something they are not comfortable with.

It makes Rich Hickey’s „Open Source Is Not About You” [0] particularly poignant.

As a hobbyist developer/maintainer of open source projects, I strive to remember that this is my gift to the world, and it comes with no strings attached. If people have any expectations about the software, it’s for them to manage; if they depend on it somehow, it’s their responsibility to ensure timely resolution of issues. None of this translates to obligations on my part, unless I explicitly make promises.

I empathize with Lasse having been slowed down by mental issues. I have, too. And we need to take good care of ourselves, and proactively prevent the burden of maintainership from exacerbating those issues.

[0]: https://gist.github.com/g1eny0ung/9e7d4d0f72547a8d156452e76f...


>having been slowed down by mental issues

Anyone and everyone in the OSS world should be concerned about this too. You have nation state level actors out there with massive amounts of information on you. How much information have you leaked to data brokers? These groups will know how much debt you're in. The status of your relationships. Your health conditions and medications? It would not take much on their part to make your life worse and increase your stress levels. Just imagine things like fake calls from your bank saying that debt of yours has been put in collections.


Not just nation state actors. All that data is available to anyone with a credit card from legit data dealers.

This is why I find some disclaimers in some open source projects quite superfiscial, that the software is provided as is without any warranty. Of course it is, this should be the obvious default.

If there is a law that would entitle a user to more, it is a bug in legislation that needs urgent fixing.


I see this as sort of the pivot on how people choose an open source license. When you feel like you are building the thing for others use a gplish license, it has all sorts of clauses around getting everyone to play nice. Building the thing for yourself however, I think the bsd style license makes more sense. you don't really care what anyone else is doing with it, you don't want to form a community. however, because it is trivial to share source code, you do so.

Look how brilliantly they selected their target project:

(1) xz and the lib are widely used in the wild including linux kernel, systemd, openSSH; (2) single maintainer, low rate of maintenance; (3) the original maintainer has other problems in his life distracting them from paying closer attention to the project.

I am wondering how many other OSS projects look similar and can be targeted in similar ways?


I'm thinking 95% of home automation which is full of obscure devices and half baked solutions which get patched up by enthusiasts and promptly forgotten about.

Controlling someone's lights is probably less important than Debian's build fleet but it's a scary proposition for the impacted individual who happens to use one of those long tail home assistant integrations or whatever.


A lot of home automation controls EV charging these days too. Imagine an attack that syncs a country’s EV fleet to charge in a minute where demand is at a peak. You could cause some damage at the switchgear I bet if not worse

A takeaway for me is to be extremely tight with personal information on the internet. People will use this to craft a situation to fool you.

Are you married? Have a house? Pets? Children? Sick parent? Gay? Trans? Mental health issues? Disabled? All of this can be used against you. Be careful where and how you share stuff like this. I know it's not "cool" to be mysterious online anymore, but it creates a much larger attack surface. People can still engage with groups around these things, but better to do it with various personas than to have one trackable identity with everything attached to it.


Many.

We're in a tech slowdown right now. There are people who got used to a certain lifestyle who now have "seeking work" on their LinkedIn profiles, and who have property taxes in arrears that are listed in county newspapers-of-record. If you're an intelligence operative in the Silicon Valley area, these guys should be easy pickings. An envelope full of cash to make some financial problems go away in exchange for a few commits on the FOSS projects they contribute to or maintain.


Yes it seems a lot like a case of a predator picking off a weak and sick individual.

The Jigar Kumar nudges are so incredibly rude. I would have banned the account, but perhaps they contributed something positive as well that isn't mentioned.

I wonder if it would be possible to crowdsource FOSS mailing list moderation.


There is a good chance that everyone in that thread except the original maintainer is in on the act. It's likely that all those accounts are managed by a single person or group. Targeting just one account for rudeness isn't going to help, if that's true.

The mechanism employed here seems like the good cop, bad cop interrogation/negotiation technique. There is the one person who has taken care to show cultural and mission alignment. Then there are several misaligned actors applying pressure which the first person can relieve.

How to identify and defuse: https://www.pon.harvard.edu/daily/batna/the-good-cop-bad-cop...


Reminds me of the "no soap radio" joke. Joke being euphemism for collective gas lighting, but typically a "joke" played by kids on each other.

Play is just preparing for the same game but when stakes are higher?

https://en.wikipedia.org/wiki/No_soap_radio


It does help on the social/psychological side. If you, as an open source project maintainer, have a policy that such rudeness is not acceptable, you are much less likely to become a successful victim of a social attack like this.

It's entirely possible for an evildoer to make someone feel bad while remaining completely polite.

First send a message to the mailing list as "Alice" providing a detailed bug report for a real bug, and a flawless patch to fix it.

Then you reply to the mailing list as "Bob" agreeing that it's a bug, thanking "Alice" for the patch and the time she spent producing such a detailed bug report, then explaining that unfortunately it won't be merged any time soon, then apologising and saying you know how frustrating it must be for Alice.

Your two characters have been model citizens: Alice has contributed a good quality bug report, and code. Bob has helped the project by confirming bug reports, and has never been rude or critical - merely straightforward, realistic and a bit overly polite.


As someone else said in this thread, scammers are often rude, because it makes people act fast, polite responses give them time to think. Of course, people are very easily manipulated. But by completely rejecting rudeness and having the mindset to not let others put pressure on me, you will improve the odds by a lot.

That would be true if you could ban the person from using new emails, but I don't think that's true when the thread if rife with sock puppet accounts. You ban the first rude email account, then there will be 2 new accounts complaining about both the lack of commits and the "heavy-handed mailing-list moderation" stifling differing views.

Yep, as the attacker you bias the entire playing field to your side. If a mailing list has 20 or so users on it, you create 50 accounts over time that are nice, helpful, and set a good tone. Then later you come in with your attack and the pushy assholes. Suddenly those 50 puppets just slightly side with the asshole. Most people break under that kind of social pressure and cave to the jerks request.

Absolutely right. Considering there is a whole cottage industry about asshole replies from Linus Torvalds on linux mailing lists.

For lesser/individual maintainers there is no way to survive this kind of mob attack. Corporate maintainers may be able to manage as it could be considered just as paid job and there are worse ways to make money.


The act relies on there being an extreme reluctance to ban. Once the banhammer has been used, the act kind of falls apart. Of course, difference pressure campaigns can then be brought to bear.

We live in an adversarial environment, time to stop playing naively nice. Ideally it isn't the maintainer that has to do all this work.


The xz list traffic was remarkably low. More than a few times over the years, I thought it broke or I was unsubscribed.

Messages like Jigar’s are kind of par for the course.


I think that is intentional and that the goal would have been achieved even if Jigar (who probably is the same guy as Jia) had been banned.

It seems from the reading of this article that jigar is in on the scam. That said, I agree.

> I would have banned the account

Yeah, same. We should be much more willing to kick jerks out of our work spaces. The work is hard enough as it is without also being shit on while you do it.


Yea people are too accepting of allowing asshats like the Jigar messages.

Simple ban and get the fuck out. Too often I've dealt with people trying to rationalize it as much as "o its just cultural, they don't understand". No, get the fuck out.

But hey I'm a NYer and telling people to fuck off is a past time.


Jigar was the same person/group as Jia. They were the bad cop and Jia was the good cop. Banning wouldn't have changed anything. Even if Jigar had been banned, the maintainer would still have appreciated the good cop's helpful contributions in contrast to the unhelpful bad cop. Jia would have become a maintainer anyway.

Not surprising, unfortunately. You'd think malicious actors would be nice to people they're trying to deceive. But after watching a few Kitboga videos, I learned that they more often yell, abuse, and swear at their victims instead.

Being nice gives people time to think.

Being mean is stressful and stops your brain from working properly. If someone doesn't allow you to be abusive, then they are not a mark. Predators look for prey that falls into certain patterns.


>I wonder if it would be possible to crowdsource FOSS mailing list moderation.

I think this could be a genuine use of an AI: to go through all of the shit, and have it summarized in a fashion that the user wants: distant and objective, friendly, etc. It could provide an assessment on the general tone, aggregate the differently phrased requests, many things like that.

Crowdsourcing would works best with the reddit / hacker news model I feel, where discussion happens in tree styled threads, and users can react to messages in ways that are not text, but something meta, like a vote or a reaction indicating tone.

Both of these have significant downsides, but significant upsides too. People pick the mailing list in a similar way.


A big problem is that people allow this sort of thing as part of the culture. I've followed the Fedora and PHP development mailing lists a few different times over the years ans this sort of thing was tolerated across the board. It doesn't matter if you crowdsource the moderation if nobody thinks the behavior is bad in the first place.

Trying to do something about it was called censorship.


I'm sorry I don't understand your point clearly. Why is it a big problem, and whose problem it is?

The premise of the post I replied to is that the mailing list moderation is currently not great and that it allows people to be abusive. It suggest that we should crowdsource this moderation. I assume they think this will lower the burden.

I myself do not think that this is the actual problem. I think the actual problem is that many FOSS communities have fostered an idea that cracking down on certain types of behavior is censorship. Of course they might all agree to ban somebody who is saying certain well known explicit words or being really really aggressive, but there's a lot of behavior that I would consider bannable but they do not.

I'm trying to avoid mentioning specific cases because I don't wanna either reopen old wounds or cause another similar dustup. I can say that it reminds me a lot of the really old post about the 5 geek social fallacies https://plausiblydeniable.com/five-geek-social-fallacies/ In particular social fallacy #1

Obviously we're not talking about geek culture here (in general anyways). After rereading it, I saw something else that we actually do a see a lot even today.

This in particular has happened on many a mailing list, including the ones I'm not specifically mentioning

>> <SNIP> when nothing smacking of social selectiveness can be discussed in public, people inevitably begin to organize activities in secret. </SNIP>

Well-known annoyances get tolerated for years, and the people who do the work start moving their discussions to bug trackers, irc, or in personal emails to avoid interacting with them.


I agree with this. In group settings I also look out for group-destructive behavior, and address it as best as I can. I'm the kind of person who thinks that certain rules, while restrictive on the surface, enable a sort of sustainable freedom down the line, similar to how the GPL license works.

I feel for Lasse.

It's time for more of the big vendors who use these projects in their offerings to step up and give people running these small projects more resources and structure. $20k to have maintainers for each project actually meet twice a year at a conference is chump change for the biggest vendors, especially when compared against the cost of the audits they'll now be doing on everything Jia Tan and Co. touched.


As an OSS maintainer, $20k wouldn't help me enough unless I was retired. The issue is not money (or not just money), but time. If a maintainer has a full-time job, they may not have time, and developers/maintainers tend to have full-time jobs, so...

Now maybe one could build a career out of OSS maintainerships, with work/time funded by lots of donations much smaller than a salary but amounting to a salary.


I was thinking more of a fix to the issue of "who the hell's maintaining this package our distro/service/whatever is based on" than a way to make money. The bigger projects (like the kernel) and vendors (MS, IBM/Red Hat, Canonical, Google, etc.) all have a vested interest in knowing the actual identity and basic personalities of people who maintain the important packages. If maintainers avail themselves for a weekend at a conference twice a year (or maybe even a lighter commitment like a few short meetings with a manager) they get some resources for their efforts. The flip side of this, of course, is that these organizations will prefer to include packages from maintainers who agree to this arrangement over those who don't.

Furthermore, these organizations are in a place to put experienced, trustworthy contributors on projects that need maintainers if need be. If Lasse had been able to go to, idk, the Linux Foundation and say, "Listen, I'm getting burnt out, got anyone?" and they said "Sure, we've got this contributor with an established record who would love to help maintain your project", none of this is happening right now.


I’ve given semi-popular projects that I no longer had the bandwidth to maintain to random people who bothered to email, no pressuring needed. While those projects are probably four to five magnitudes less important than xz, still thousands of people would be affected if the random dude who emailed was malicious. What should I have done? Let the projects languish? Guess I’ll still take the chance in the future.

If it's open source they can just fork it, and if you're no longer maintain yours you can put a link over to their fork. (Or any other active forks). It's still on the user to vet new forms.

> What should I have done? Let the projects languish?

Yes, if you can't find a successor you trust then let someone fork the project and build trust from 0 rather than transferring trust others' trust in you by handing over the project. This doesn't just apply to security concerns btw. - plenty of other ways in which a new maintainer might end up making the project worse (intentionally or through incompetence) compared to it not receiving any updates.


I guess all you can do is not give the brand away.

Put a link saying "Hey this guy forked my project, I won't maintain it anymore, he may add malware, review and use at your own risk"


I'm reminded of the short story "The Strange Case of Mr Pelham", in which a man is stalked and eventually replaced by a doppelganger.

https://en.wikipedia.org/wiki/The_Strange_Case_of_Mr_Pelham


In "Ghost in the Wires" Kevin Mitnik details one of the ways he obtained information was via a law enforcement receptionist* who he managed to trick into believing he was law enforcement over the phone. He obtained information this way multiple times over multiple years, and fostered a phone based friendship with this woman. He seemed to have no qualms in doing this.

He was also turned on by multiple people who he considered close friends. In the book it did not seem that he had considered that it might not be a "them" problem.

*my details may be off here, I read it some time ago


It's bizzare enough as it is to start asking questions to confirm that "mental issue" had natural cause.

Your experiences may differ, but I'd say pretty much anyone who lived through the past few years has reason enough to pay careful attention to their mental health.

Thank the lord Lasse wasn’t maintaining the nuclear codes.

In another thought, I hope the nukes aren’t on a rolling Debian distro.


> merges hidden backdoor binary code well hidden inside some binary test input files. [...] Many of the files have been created by hand with a hex editor, thus there is no better "source code" than the files themselves.

So much for the folks advocating for binary (driver) blobs in OSS t support otherwise unsupported hardware.

It's either in source form and reproducable or it's not there.


Not just for hardware support: https://github.com/serde-rs/serde/issues/2538

!!

>It's either in source form and reproducable or it's not there.

Wanna know how I know you haven't read into the discussion much?

There are a whole lot of binary test cases in software. Especially when you're dealing with things like file formats and test cases that should specifically fail on bad data of particular types.


GP is talking about executable blobs (drivers) more than anything. Yes, binary protocols will lead to binary test blobs, so what.

The attack was embedded in a binary test blob, or did you just not happen to read anything about the xy attack?

You can't avoid having to have binary blobs of data. And again, GP was talking about closed-source drivers, not specifically the xz attack.

> There are a whole lot of binary test cases in software.

That's not how I read GP's point. If even binary blobs in test cases are a place where backdoors are, now as a matter of fact, hidden then, certainly, among the folks advocating for binary drivers in FOSS, there are some who are already --or planning to-- add backdoors there.

Binary blobs are all terrible, terrible, terrible ideas.

Builds should be 100% reproducible from source, bit for bit. At this point it's not open up for discussion anymore.


Then you figure out how to build a 'source' test case of a bad zip, or bad jpg, or word document or whatever else exists out there. Also figure out how to test that your bit4bit perfect binary isn't doing the wrong damned thing in your environment with actual real data.

In cryptography, there's the concept of a nothing-up-my-sleeve number. [1]

Instead of obscure constants, you use known constants, or at least simple methods to derive your constants.

You can do the same thing to come up with your test cases. Bad zip? Construct a good zip of 10 files, each containing the first 10,000 prime numbers. Then corrupt the zip by seeking to position (100/pi) and write a thousand zeroes there.

Bad JPEG? Use Imagemagick to render the first 1000 prime numbers as text into a JPEG file, then apply a simple nothing-up-my-sleeve corruption operation.

There are still cases where this approach isn't going to work: that new icon, helpfully proposed by a contributor, meant to be used in production, might contain malicious code, steganographically embedded. I think there's little you can do to prevent that.

[1]: https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number


> It's either in source form and reproducable or it's not there.

Forbidding binary data in source repositories is neither practical nor solving the issue.

Binary code can be hidden steganograhically in e. g. image files or text files.


Are you running linux-libre?

> It's also good to keep in mind that this is an unpaid hobby project.

That's the root cause. At some point a corp/gov consortium needs to adopt key projects and hire out the maintainers and give them real power and flexibility, perhaps similar to the way a nation might nationalize key infrastructure.

But this is against the core ethos at some level. Freedom and safety can be antagonistic.


You are implying that governments are more technically competent and more trustworthy than open source communities..

Many open source projects often already do receive US government funding, mostly through an onerous grant-application process. Nationalizing American open source projects could make them operate more like European open source where their EU funding is open and clear. The detrimental trade-off, however, is that the American agencies most capable to support and contribute directly to infrastructure security have burned away all trust from the rest of the world. Direct contributions directly from those USG agencies would reduce global trust in those projects even worse.

I said nothing of the sort.

I'm implying they are richer.


Avoid compromise with one simple trick: surrender to the attackers

I don't understand how "Accept public money to work full time on key software" is surrender, but I think everyone misunderstood what I said, which usually means I didn't say what I meant.

> gov consortium

Personally I wouldnt trust a govt to not backdoor everything.


A consortium is a great way to get money and power into those maintainers. Never said they should take the power from them or provide code. I think people are hearing their own mind here, not mine.

> Evan Boehs observes that Jigar Kumar and Dennis Ens both had nameNNN@mailhost email addresses

This is the second time I've read this "observation", but this observation is just wrong? Jigar's email is "${name}${number}@${host}", yes, but Dennis's is just "${name}@${host}" — there's not a suffixed number. (There's a 3, but it's just a "leetcode" substitution for the E, i.e., it's semantically a letter.)

(They could still both be sockpuppets, of course. This doesn't prove or disprove anything. But … let's check our facts?)


Where are the email addresses visible? I've also seen this a few times, but never the actual addresses.

eg "Hans Jansen" is <hansjansen162@outlook.com>

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708


I'm maybe naive in assuming that there is some kind of investigation going on that isn't playing out in public, but I would assume that years of emails going through Microsoft servers might make the identities of the cohorts in this attack difficult to hide?

When making sockpuppet email accounts use a plausible birth year as your numeric extension.

In the OP, many of the links (e.g., "first, innocuous patch") lead to mail-archive.com. From there, click the button with the name that appears after "Reply via email to", and it will open your email client, with the To: filled in with the email of the person.

I couldn't spot email addresses directly in plaintext for those who weren't submitting patches (e.g. Jigar), however if you look at one of the links to his (?) responses then there's a mailto link with the text 'Reply via email'

Joe Cooper's take on pressuring project maintainers: https://forum.virtualmin.com/t/dont-panic-ssh-exploit-in-ble...

somewhat ironic, but I'd say effective.


I don't even think it's necessarily moderation so much as maintaining these high-traffic projects is now akin to a full-time job minus the pay.

At a certain point, companies need to step up and provide funding or engineering work or they should just keep expecting to get owned.


If there is a larger shift to companies paying for the software they rely on that is currently free, it's not going to be companies paying open-source maintainers, it's going to be companies paying other companies that they can sign contracts with and pass legal responsibility to.

Why would they when they can just... not?

This seems very difficult to defend against. What is a project with a single burnt-out committer to do?

lcamtuf's two posts argue that this may simply not be an open-source maintainer's job to defend against. ("The maintainers of libcolorpicker.so can’t be the only thing that stands between your critical infrastructure and Russian or Chinese intelligence services. Spies are stopped by spies.")

That doesn't mean we shouldn't try to help burnt out committers, but the problem seems very hard. As lcamtuf also says, many things don't need maintenance, and just paying people doesn't address what happens when they just don't want to do it anymore. In an alternate universe with different leadership, an organization like the FSF might use donated funds to pay a maintenance staff and an open-source maintainer might be able to lean on them. Of course, that still doesn't address the problem of Jia Tan getting a job with this organization.

https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-ba... https://lcamtuf.substack.com/p/oss-backdoors-the-allure-of-t...


Why the assumption this is a Russian or Chinese intelligence service? Western governments aren't above this sort of conduct: https://www.mail-archive.com/cryptography@metzdowd.com/msg12...

Why are people assuming it's any intelligence service/state actor? With cryptocurrency valuations, it would seem like remote rooting gajillions of machines would be highly incentivized for a private person/collective. Not to mention other financial incentives. Our digital infrastructure secures enormous value much of which can be pilfered anonymously.

I admit, the op has a "professional/polished vibe" to me as well, but we seem to know very little except for what work time/zones were preferred by the possibly collective/possibly singular human(s) behind the Jia Tan identity. Does anyone have slick linguistic tools to assess if the writer is a single author? Maybe an opportunity to show off.. It's sort of how they caught Ted Kaczynski.

It also absolutely makes sense to think of all the state actors (I agree including as you say the US/UK) as part of the ongoing threat model. If the KGB/Ministry of State Security/NSA/MI6 were not doing this before then they surely might in the future. Maybe with more gusto/funding now! They all seem to have an "information dominance at all costs" mentality, at least as agency collectives, whatever individuals inside think.


people often assume state actors are the pinnacle of sophistication, and especially long games. (Also notably Chinese culture is very attuned / trained for long games, relative to American impatience). This was a sophisticated attack, therefore presumption.

Fair enough - I agree that sophistication inspires the presumption, but it's still just that (not that you said otherwise - just emphasizing).

Anyway, I've yet to hear of anything in the xz work beyond the ability of 1-3 skilled black hats with a lot of post-COVID time on their hands. The NSA ensuring the Intel ME/AMT could be turned off seems another level of sophistication entirely, for example { a "higher pinnacle" in your nice phrasing :-) }.

In terms of sheer numbers, my impression is that the vast majority of attacks blocked at almost every sophistication level are more "criminals" than "states". Admittedly, that may partly be states just acquiring ability to act surgically rather than launching big compromise initiatives like botnets (or otherwise states going undetected). I'm sure it's hard to know.

Maybe we're just in a sad era of increasing attack sophistication measured in "Kilo-Sophistic-meters"? The AntiVirus industry has been having a boom lately.

It's probably already been mentioned many times, but besides economic & geopolitical incentives, maybe the attacker(s) was a hard core `systemd` or IBM/RedHat hater or one of the people who supposedly issued death threats to Lennart Poettering now at Microsoft or even an open source hater burnt out or wanting to burn the world down like The Joker in Batman. In light of that, Russ' Setting The Stage Prelude could perhaps profitably add the introduction of that lib dependency into `systemd` and also various distros defaulting to `systemd`.

Anyway, premature conclusions are to be cautioned against over & over. That's all I was trying to do. { And I'm not claiming you were concluding anything - you seem pretty open-minded about it all. I was always just amplifying parent comments. I absolutely agree long games "feel" more State Actor - part of what I meant by "vibe", but to quote the detective in V For Vendetta - "it's just a feeling". To Old People, 2 years famously doesn't seem as long as to a 25- or 15-year old. ;-) It's actually short on the 5+ year Linux distro maintenance time scales. }


I don't think there's necessarily an assumption that it was Russia or China this time (the Chinese-sounding name is almost a dead giveaway that it's someone other than China), but the security world does believe quite strongly that it has been Russia (SolarWinds) and China (Juniper) before, so they are the usual examples.


There are efforts from industry to try to secure open source, e.g., https://cloud.google.com/security/products/assured-open-sour...

I suspect some variant of this will grow so that some companies, MS/GitHub for example, audit large body of code and vet it for everyone else.


As a burnt out creator of open source projects with thousands of GitHub stars who's received numerous questionable maintainership requests: either very carefully vet people or let it stagnate. I chose the latter and just waited until others forked it, in large part because I didn't want to be responsible for someone hijacking the project to spread malware.

If I had ever received a request from a well-known figure with a longstanding reputation, who's appeared in-person at talks and has verifiable employment, I might've been more receptive. But all the requests I got were from random internet identities that easily could've been fabricated, and in any case had no previous reputation. "Jia Tan" and their other sockpuppets very likely are not their real identities.


This is not Lasse Collin’s responsibility. What is a burnt out committer supposed to do? Absolutely nothing would be fine. Doing exactly what Lasse Collin did and turn over partial control of the project to an apparently helpful contributor with apparent community support is also perfectly reasonable.

It's not a perfect solution, but I think Big Companies could have a role to play here. Some kind of tenure/patronage system, where for every $1B a Big Company makes in profit, they employ one critical open source maintainer with a decent salary (like $200k or something). The job would only have two requirements: 1) don't make everyone so mad that everyone on the Internet asks for you to be fired, and 2) name a suitable replacement when you're ready to move on. The replacement would become an employee at Big Company, which means Big Company would need to do whatever vetting they normally do (background checks, real address to send paychecks and taxes, etc).

In this scenario, Jia Tan would not be a suitable replacement, since they don't friggin' exist.

Yes, there's problems with this approach. When money gets involved, incentives can get distorted. It limits the pool of acceptable maintainers to those employable by a Big Company. But I think these are solvable problems, especially if there's a strong culture of maintainer independence. It provides a real improvement over the current situation of putting so much pressure on individuals doing good work for no benefit.


> "It's not a perfect solution"

Is it a solution at all? Say Oracle offer Lasse Collin $200k for maintaining xz but he doesn't want to work for Oracle so he refuses, then what? Amazon offer Lasse $200k but they require fulltime work on other open source packages which he isn't experienced with or interested in, so he refuses, then what? Google employ someone else for $200k but they can't force Lasse Collin to hand over commit rights to xz to Google, or force him to work with a fulltime Google employee pestering with many constant changes trying to justify their job, and they can't force Debian to accept a new Google fork of xz, so then what? And NetFlix, Microsoft, Facebook, Uber, they can't all employ an xz maintainer, xz doesn't need that many people, but if they just employ 'open source maintainers' scattering their attention over all kinds of random projects they have no prior experience with, how would they catch this kind of subtle multi-pronged long-term attack on some low-attention, slow moving, project?

Google already employ a very capable security team who find issues in all kinds of projects and publicise them, they didn't find this one. Is it likely this attack could have made its way into ChromeOS and Android if it wasn't noticed now, or would Google have noticed it?

> "1) don't make everyone so mad that everyone on the Internet asks for you to be fired"

So it's a sinecure position, doing nothing is the safest thing?

> "and 2) name a suitable replacement when you're ready to move on"

How could Lasse Collin have named a more suitable replacement than someone who seemed technically capable, interested in the project, and motivated to work on some of the boring details like the build system and tests and didn't seem to be doing it for the hype of saying "I improved compression by 10%" for their resume? Are they needing to be skilled in hiring and recruitment now?


I think you've misunderstood my suggestion. I said the job has two responsibilities. No more. You added a bunch of other responsibilities, I'm saying those wouldn't be allowed. It would be in the employment agreement that is purely payment for doing the maintainership tasks they were already doing. It would be a culture expectation that the company not apply pressure on maintainers.

> but if they just employ 'open source maintainers' scattering their attention over all kinds of random projects they have no prior experience with

They would pay the people who are doing the work now. Under this hypothetical, one of the Big Companies would have hired Lasse as the xz maintainer, for example. His job responsibilities are to maintain xz as he had been doing, and identify a successor when he's ready to move on. Nothing else.

> So it's a sinecure position, doing nothing is the safest thing?

No. Not doing the maintenance tasks would make everyone mad, violating one of the two job responsibilities.

> How could Lasse Collin have named a more suitable replacement than someone who seemed technically capable, interested in the project, and motivated to work on [it]

Lasse would suggest Tan as a suitable replacement. Big Company's hiring pipeline would approach Tan and start the hiring process (in person interviews, tax docs, etc etc). At some point they would realize Tan isn't a real person and not hire him. Or, the adversary would have to put up "a body" behind the profile to keep up the act, which is a much higher bar to clear than what actually happened.


Leaving aside issues of how it could work, Lasse Collin wasn't the one who saw this attack and stopped it so how would paying him have helped against this attack?

> "Lasse would suggest Tan as a suitable replacement. Big Company's hiring pipeline would approach Tan and start the hiring process (in person interviews, tax docs, etc etc). At some point they would realize Tan isn't a real person and not hire him"

What if they find that Tan is a real person but he doesn't want to work for Amazon or legally can't (This is before knowing of his commits being malicious, we're assuming he's a fake profile but he could be a real person being blackmailed)? Collin can't leave? Collin has to pick someone else out of a candidate pool of people he's never heard of? Same question if they find Tan isn't a real person - what then; is there an obligation to review all of Tan's historical commits? Just committing under a pseudonym or pen name isn't a crime, is it? Would the new maintainer be obliged to review all historic commits or audit the codebase or anything? Would Amazon want their money back from Lasse once it became clear that he had let a bad actor commit changes which opened a serious security hole during his tenure as maintainer?

> "No. Not doing the maintenance tasks would make everyone mad, violating one of the two job responsibilities."

What he was doing already was apparently ignoring issues for months and making people like Jigar Kumar annoyed. Which is fine for a volunteer thing. If "Jigar Kumar" is a sock-puppet, nobody knew that at the time of their posts; Lasse Collins' hypothetical employer wouldn't have known and would surely be on his case about paying him lots of money for maintenance while complaints are flowing on the project's public mailing list, right? Either they're paying him to do what he was doing before (which involved apparently ignoring work and making some people mad) or they're paying him to do more than he was doing before (which is not what you said).

It doesn't seem like it would work - but if it did work it doesn't seem like it would have helped against this attack?


> Lasse Collin wasn't the one who saw this attack and stopped it so how would paying him have helped against this attack?

Well, there's a few issues I'm trying to target. I'm trying to work backwards from "how do we stop bad actor Tan from getting maintainer access to the project?" Creating an identify-verified relationship (employment) is a good fit for that, I think. And it nicely solves some other related issues with the current volunteer maintainership model. Lasse may not have felt the strong pressure/health issues if he was being paid to do the work. Or, if he was feeling burnt out, he may have felt more comfortable passing the torch earlier if there was a clear framework to do so, backed by an entity that can do some of the heavy lifting of naming/validating a successor.

> What if they find that Tan is a real person but he doesn't want to work for Amazon or legally can't

I think this would be a fairly rare occurrence, but it's one I called out as a potential problem in my original post, yeah ("smaller pool of possible maintainers"). If there isn't a clear successor, I think the maintainer could inform the Big Company that they'd like to move on in the next year or two, and Big Company could maybe find an internal engineer who wants to take over the role. Or maybe this more formal sponsored-maintainership arrangement would create incentives for outside contributors to aim for those positions, so there's more often someone waiting to take over (and then be verified by Big Company's hiring process).

> is there an obligation [for the maintainer] to review all of Tan's historical commits? Would the new maintainer be obliged to review all historic commits or audit the codebase or anything? Would Amazon [fire] Lasse once it became clear that he had let a bad actor commit changes which opened a serious security hole during his tenure as maintainer?

(I tweaked your questions a tiny bit to rephrase them as I interpreted them. I think the spirit of your questions was kept, I apologize if not.) If these tasks fall under the "don't make everyone mad" job responsibility, then yes. If not, then no, to all of these. There are no obligations other than the two I mentioned: don't piss off the community and help name a successor. It's up to the project's community to decide if the maintainer is not meeting their obligations, not the sponsoring Big Company.

> What he was doing already was apparently ignoring issues for months and making people like Jigar Kumar annoyed.

I'm not sure. It seems like Kumar was a bad actor. Was there actually a real maintenance issue? If so, maybe it could have been avoided in the first place by the sponsorship arrangement, like I mentioned at the top of this reply. Or, the community could raise the problem to Big Company, who can do the work of verifying that there is a problem and working with the maintainer to resolve it. Instead what happened here, which was for one burned out guy deciding to hand the keys over to some email address.


> "I'm trying to work backwards from "how do we stop bad actor Tan from getting maintainer access to the project?" Creating an identify-verified relationship (employment) is a good fit for that, I think."

It would stop a sock puppet, but Jai Tan might be a real person, a real developer paid or blackmailed by a hostile group; Amazon might just have hired him and handed over maintainer access to him thinking it was above board, if a problem hadn't been found yet. I don't know where Jai Tan claimed to be from, but it's quite possible they would say "I don't have a passport", "I can't leave my family to travel to America for an in-person interview", "I'm not in good health to travel", "I don't speak English well enough for an in-person interview", "I live in a poor country without a functioning government and have no tax documents", or etc. etc. excuses which are quite plausible

> "Or, if he was feeling burnt out, he may have felt more comfortable passing the torch earlier if there was a clear framework to do so, backed by an entity that can do some of the heavy lifting of naming/validating a successor."

Your suggested $200k is equivalent to £160k GBP in the UK; look at this UK average salary list: https://uk.jobted.com/ no job comes close; not Managing Director, IT director, Finance Director, Aerospace engineer, DevOps engineer, neurosurgeon, nothing on the list is above £110k. Sure there are many people earning that much as a senior devops AI cloud security specialist in a fast paced London based fintech trading house, but the idea that someone would comfortably pass up a salary around the 98th percentile of incomes in the country for like 2 days a month of work because they're "feeling burnt out" is unthinkable. Anyone sensible would hold onto that until they pried it out of one's cold dead hands; American tech salaries are almost literally unbelievable. Even moreso if we consider a maintainer in a poorer country.

> "I tweaked your questions a tiny bit to rephrase them as I interpreted them. I think the spirit of your questions was kept, I apologize if not"

I started writing Tan, but then changed it. A lot of your reply is assuming that we know there were malicious patches and suspect Jigar Kumar was a bad actor and that the big company would be somewhat US friendly. But we can't plan to know all that for all situations like this. Some people will be speculating that the previous paid maintainer was complicit and all their work and merges are now suspect. The billion dollar company who hired Collins in this hypothetical maintainer could be Baidu or Yandex or Saudi Aramco, and then people would be suspicious. It's one thing to have your task be "don't make people mad" but doesn't that change if people getting mad can give you unbounded retrospective work and responsibility?

> "If these tasks fall under the "don't make everyone mad" job responsibility, then yes. [...] Was there actually a real maintenance issue? [...] Or, the community could raise the problem to Big Company, who can do the work of verifying that there is a problem and working with the maintainer to resolve it."

As soon as the internet becomes aware that they can get anything merged ASAP by threatening to get mad, everyone will be mad about everything all the time. Whom at the BigCo will do the work of verifying whether there is a problem? I mean, let's put Lasse Collins on a specific team along with other employees who are expected to work 40-80 hour weeks while he isn't. The pressure on the team manager to ditch the maintainer and distribute his salary among the other team members would be constant. If those other team members see him doing less work for similar or more money it would be a morale killer and they would want to leave. If they also have to know his project well enough to follow all the drama and things people are complaining about and tease out what is and isn't a real problem and coerce him to do his job, sorry 'work with him', well, they won't be very motivated to do that.


I think the best solution would be governments forcing companies to secure the entire pipeline and setting up a non profit that does this for open source packages. Have security researchers work for a non profit and force companies that use software from some guy in Nebraska to pay into the non profit (could be in the form of labor) to get the code checked and certified.

The guy in Nebraska is still not getting anything but will also not have the stress of becoming one of the main characters/victims in a huge attack.


This is the problem - we have record numbers of computer programmers, having increased exponentially from the 90s, however very few of them want to work just for fun on some library buried deep in the system, most are just out for a quick buck.

Look at the number of people who make meaningful commits to ffmpeg vs the number of people that wrap it in some UI and somehow persuade thousands of people that their solution is amazing, and ffmpeg is a sexy project.


Until we can patch humans, social engineering will always work. Burnt-out comitter or not. Just be vigilant.

The same as what the ant colonies do when one of their own gets infected by the Ophiocordyceps unilateralis fungus.

That's a great question and instinctively I'd say better to halt development than cave to pressure.

Check the GitHub profile of anybody that commits. Is there a photo of the person? Can you see a commit history and repos that help validate who they seem to be.

In this instance, noticing the people emailing to pressure you have fake looking names that start with adjacent letters and the same domain name.

Be more paranoid.


> Is there a photo of the person?

Does that even matter these days?

Especially if we're talking nation state level stuff convincing histories are not hard to create to deflect casual observers.

>Be more paranoid.

Most people in OSS just want to write some code to do something, not defend the world against evil.


“Does it even matter?”

Yes, it would have prevented this attack. It isn’t totally sufficient but it’s quick and easy and would have prevented this attack.

“Most people don’t want …”

I get it. I think the issue is that pushing junk code from malicious contributors into your project causes more hassle in the long run. If you just want to code and make stuff work, you should probably be careful who you pull from. It’s not just for the benefit of others, it’s first and foremost to protect the code base and the time and sanity of other contributors.


"Sorry, we had to kill open source software because bad people exist" -Microsoft laughing all the way to the bank.

The more paranoid walls you put up the more actual contributors getting into the movement say "eh, screw this, who wants to code anyway".

This isn't just a problems with OSS, this is a fundamental issue the internet as a whole is experiencing and no one has good answers that don't have terrible trade offs of their own.


Only non-paranoid people have a photo of themselves online.

get the project taken over by a foundation eg the Apache Foundation.

tell people to fork your project and fuck off

My god, Hans Jansen? You need to be 60 years old to have that name in The Netherlands, it's so stereotypical that it would definitely ring some alarm bells if I saw it.

It's like seeing a "Bill Johnson, real American patriot" account on Twitter: would not believe it at face value.


I wonder, once the attacker gained commit permissions, were they able to rewrite and force push existing commits? In that case rolling back to older commits may not be a solution.

If my speculation is correct then the the exact date on which access was granted must then first be known, after that a trusted backup of the repo from before that date is needed. Ideally Lasse Collin would have a daily backup of the repo.

Although perhaps the entire repo may have to be completely audited at this point.


Force pushes tend to be noticed easily. All it takes is for one external developer to try to pull to see the failure. And it's actually hard to do because you need to comb through the tree to update all the tags that point to the old commits. On top of that it obviously breaks any external references to the commit IDs (e.g. in distro or build configurations), all the way up to cryptographic signatures that might have been made on releases.

I think it's a pretty reasonable assumption that this didn't happen, though it would be nice to see a testimony to that effect from someone trustworthy (e.g. "I restored a xz checkout from a backup taken before 5.6.0 and all the commit IDs match").


Of course, that's only since the attack was caught early. Since it affected the machines that build packages, a next step could have been, say, to add code to Git to make it ignore certain commit IDs or mask them in some way. Maybe even modify Firefox, Chromium etc to not show those changes on github. With a backdoor like this and enough time undetected, you could control basically everything.

> And it's actually hard to do because you need to comb through the tree to update all the tags that point to the old commits.

Isn't this part just a few pages of code, if that?

I agree that it will be blindingly obvious for the reasons you list.


The way the hack works is incredibly sophisticated and has specifically sought out how to get past all normal checks. If messing with some commits would be possible this entire rube goldberg hack would not have been set up.

There are trusted copies of historic releases from third-party sources (at least Linux distributions, but there's probably other sources as well), it's pretty easy to check whether the tags in the git repository match those. (This can be done as the tarballs are a superset of the files in the git repository, the other way around doesn't work).

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: