The malicious changes were submitted by JiaT75, one of the two main xz Utils developers with years of contributions to the project.
“Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system,” an official with distributor OpenWall wrote in an advisory. “Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the ‘fixes’” provided in recent updates. Those updates and fixes can be found here, here, here, and here.
On Thursday, someone using the developer’s name took to a developer site for Ubuntu to ask that the backdoored version 5.6.1 be incorporated into production versions because it fixed bugs that caused a tool known as Valgrind to malfunction.
“This could break build scripts and test pipelines that expect specific output from Valgrind in order to pass,” the person warned, from an account that was created the same day.
One of maintainers for Fedora said Friday that the same developer approached them in recent weeks to ask that Fedora 40, a beta release, incorporate one of the backdoored utility versions.
“We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added),” the Ubuntu maintainer said.
He has been part of the xz project for two years, adding all sorts of binary test files, and with this level of sophistication, we would be suspicious of even older versions of xz until proven otherwise.
This is the best summary I could come up with:
Researchers have found a malicious backdoor in a compression tool that made its way into widely used Linux distributions, including those from Red Hat and Debian.
An update the following day included a malicious install script that injected itself into functions used by sshd, the binary file that makes SSH work.
So-called GIT code available in repositories aren’t affected, although they do contain second-stage artifacts allowing the injection during the build time.
In the event the obfuscated code introduced on February 23 is present, the artifacts in the GIT version allow the backdoor to operate.
“This could break build scripts and test pipelines that expect specific output from Valgrind in order to pass,” the person warned, from an account that was created the same day.
The malicious versions, researchers said, intentionally interfere with authentication performed by SSH, a commonly used protocol for connecting remotely to systems.
The original article contains 810 words, the summary contains 146 words. Saved 82%. I’m a bot and I’m open source!
So-called GIT code available in repositories aren’t affected
I wonder what convinced the model to treat git as an acronym
Dude seems like a foreign asset
Jia Tan, University of Hong Kong in China. He’s been the sole maintainer of the package for almost two years.
Looks like he’d done a lot for various US companies on his LinkedIn.
I would not be surprised if he was previously legit but pressured into doing this by the CCP.
Maybe he wasn’t sloppy by accident if he was indeed coerced by someone. I don’t think we’ll ever find out the backstory of this though.
It would make more sense to compromise developers in trusted positions, or steal their credentials, than going through the time and effort of building trusted users and projects only to burn them with easily spotted vulnerabilities.
This wasn’t easily spotted. They use words like sloppy, but it all started with someone digging in because starting ssh season was about a half second slower that it used to be. I could easily imagine 99.99% of people shrugging and deciding just something in the chain of session startup took a bit longer for a reason not worth digging into.
Also, this was a maintainer that just started two years ago. xz is much older than that, just he took over.
And that’s why you cannot trust open source software blindly.
And yet with closed-source software you have no choice but to trust it blindly. At least open source software has people looking at the code.
Bet you anything there were more pairs of eyes on SolarWinds code than this. Sick of this open source bystander effect.
Code scanners check for vulnerabilities not malicious code. Ain’t no one running full coverage dynamic scanners to trigger all branches of code on this thing, otherwise this would’ve been caught immediately
Vulnerabilities are caught and fixed faster in open source projects than closed ones.
The researchers found that open-source programmers fixed Linux issues in an average of only 25 days. In addition, Linux’s developers have been improving their speed in patching security holes from 32 days in 2019 to just 15 in 2021.
Its competition didn’t do nearly as well. For instance, Apple, 69 days; Google, 44 days; and Mozilla, 46 days.
Coming in at the bottom was Microsoft, 83 days, and Oracle with 109 days.
By Project Zero’s count, others, which included primarily open-source organizations and companies such as Apache, Canonical, Github, and Kubernetes, came in with a respectable 44 days.
There is, it’s Google Fuzz which the maintainer of XZ handily disabled the codeshare for.
You are an idiot. It’s not blind. That’s how it was found.
Not having world accessible SSH is the real fix here.
You are an idiot. It’s not blind. That’s how it was found.
From the article…
Will Dormann, a senior vulnerability analyst at security firm Analygence, said in an online interview. “BUT that’s only because it was discovered early due to bad actor sloppiness. Had it not been discovered, it would have been catastrophic to the world.”
The fact that it was discovered early due to bad actor sloppiness does not imply that it could not have also been caught prior to wide spread usage via security audits that take place for many enterprise grade Linux distributions.
Opensource = fast detection
Opensource + sloppiness = faster detection
Closedsource = never detected
Closedsource + sloppiness = maybe detected
As opposed to what? If you said “thats why you cannot trust any software blindly” it would have been not that wrong.
Single point of failure on the lone maintainer of a popular package, vs having to hack an entire company like SolarWinds and make a backdoor that bypasses their entire SDLC. Which is harder?
There’s plenty of closed source packages or components with a single actor ultimately accountable for it.
Imagine a tester even bothering to open a bug that starting a session takes 500ms longer to start than it used to. Imagine what the development manager is going to do with that defect. Imagine a customer complaining about that and the answer the company will give. At best they might identify the problematic component then ask the sole maintainer to give the “working as designed” explanation, and that explanation won’t be held to scrutiny, because at that point it’s just a super minor performance complaint.
No, closed source is every bit as susceptible, of not more so because management is constantly trying to make all those tech people stop wasting time on little stuff that doesn’t matter, and no one outside is allowed to volunteer their interest in investigating.
Imagine trying to make a helpdesk of a proprietary company take your “it’s taking 0.5 seconds longer to login” complaint seriously…
So many vulnerabilities were found due to time to login that one of the security features was to take longer to respond to a bad login so they couldn’t tell what part failed. Here’s an article I found about one such vulnerability.
And precisely that’s why the exploit was found. If it was a closed source programme, a lone threat actor modifying the code and passing it in a release can happen, and no one will find it out. In that case everyone trusts the internal security team of a closed source company blindly. I really don’t see this as an open source issue. These are malicious actors.
The backdoor appears to specifically target RSA public key authentication, so they must have had a target in mind that they know uses RSA keys.
Would another less complex answer simply be that many (most?) people and organizations use RSA because it was first and elliptic signing is not yet as prevalent?
Going with Occam’s Razor here…
I genuinely don’t understand if I missed something here, and would love more explanation.
I would highly recommend Curve25519, etc., just because such keys are faster and less common than RSA public-private keys in today’s world. RSA 2048-bit keys are considered weak today, while the Curve25519 256-bit keys remain stronger. Also, the ChaCha20-Poly1305 cipher has an interesting backstory and doesn’t necessarily need hardware acceleration (which, in theory, could be borked by the HW-vendor) to obtain good performance.
Unfortunately, some SSH front-ends don’t play nice with Curve25519 public-private keys yet… (I’m pointing at the putty SSH client, but that may have improved from the last time I had to use it)
The backdoor has existed for a month at least. Yikes.
