A near-miss on Linux hackers exposed the fragility of the internet

One of the most fascinating and terrifying events in the history of computer security began in 2022, when several aggressive emails were received on the mailing list of a small, one-person open source project.

A user submitted a complex piece of code that is now awaiting review by maintainers. But another user named Jigar Kumar thinks it can’t happen fast enough. “The patch spent years on this mailing list,” he complained. “The 5.2.0 release was 7 years ago. There’s no reason to think anything will happen anytime soon.”

A month later, he followed up: “It’s been more than a month and it’s still far from the merger. Not surprising.” [sic]

A month later: “Is there any progress on this?” Kumar stayed here for about four months, complaining about the speed of updates, and then was never heard from again.

A few weeks ago, the world took a shocking turn. “Jijal Kumar” doesn’t seem to exist at all. Aside from the aggressive emails, there is no record of anyone by that name. He and a number of other accounts were apparently part of a campaign that compromised nearly every computer in the world running Linux. (Linux is an open source operating system, as opposed to closed systems from companies like Apple, and runs on tens of millions of devices.)

Experts believe the campaign was likely the work of a well-resourced nation-state actor who nearly succeeded in launching an attack that allowed attackers to remotely access millions of computers and effectively assume the identity of anyone they wanted. Log in. The security impact will be huge.

How to hack (almost) everything

Here’s how it happened: In 2005, software engineer Lasse Collin wrote a series of tools to better compress files (similar to the process behind .zip files). He made the tools available for free online, and Collin’s work was incorporated into many large projects that eventually became known as XZ Utils.

Colin’s tools became part of the vast open source ecosystem that powers the modern Internet.We might think of something as vital to modern life as the internet as having a professionally maintained structure, but as XKCD comics were published long before the hack Showing that “all modern digital infrastructure” relies on “a project that someone in Nebraska has been painstakingly maintaining since 2003” is closer to the truth. XZ Utils is one such project – and yes, you should find it a bit alarming, since there are a lot of them.

Starting in 2021, a user named “Jia Tan” (who also doesn’t appear to exist anywhere else) began contributing to the XZ project. At first, they were small, harmless fixes. Tan then started submitting more supplementary content.

The way open source projects like this work is that the maintainer (in this case Collin) must read and approve every such commit. In fact, Tan gave Colin too much homework.

Just then, “Kumar” showed up, complaining that Colin was taking too long. Another account that doesn’t appear to exist joined the chorus. They decided that Collin was clearly not capable of maintaining the project on his own and urged him to add “Jia Tan” as another maintainer.

“They appeared to be fakes created to force Lasse to give Jia more control,” engineer Russ Cox wrote in a detailed timeline of events. “Effective. Over the next few During the month, Jia started replying authoritatively to posts about the upcoming 5.4.0 version on xz-devel.” He has become a trusted “maintainer” who can personally add code to XZ Utils.

Why is this important? Because one of the many open source tools that happens to be incorporated into XZ Utils is OpenSSH, which is used to access computers remotely and is used by millions of servers around the world.

“Tan” carefully added some carefully disguised code to XZ Utils that compromised OpenSSH, effectively allowing the creator to remotely log into any computer running OpenSSH. Files containing (heavily disguised) code are accepted as part of a larger project.

Fortunately, almost all of the millions of potential target computers were not affected, as new updates like this are usually first released in “unstable” form (meaning some bugs are expected), and most administrators Will wait for the subsequent “stable” version.

Prior to this, the work of “Jatan” was caught. Andres Freund, a software engineer at Microsoft, was off work and running some tests on a computer where an “unstable” new version had been released. In most cases, the hacker worked seamlessly, but in the cases he tested, it slowed down SSH performance. He dug deeper and soon unraveled the entire conspiracy.

This means that, thanks to some work done by a Microsoft engineer after hours, your computer is still safe – at least as far as I know.

Could we do better than try our luck?

It was not inevitable that the hack was discovered this time. Many others are running the unstable new version and haven’t noticed any issues. What first aroused Freund’s suspicion was not the suspicious code, but an error “Jatan” had inadvertently introduced.

If the Armor team had avoided this mistake, they would likely have succeeded. Freund later said of Mastadon that it “really took a lot of coincidence” to catch the suspicious code.

No one wants to believe that modern computer security essentially relies on “a lot of coincidence.” We would rather have a reliable process. But I hope this narrative makes clear how difficult it is to reliably protect our improvised internet from such attacks.

The people behind “Jia Tan” spent more than two years building the access required for this attack. Some of the details have to do with the dynamics of open source software, where projects from decades ago often sit in a quiet maintenance phase from which, as we’ve seen, radical actors can wrest control. But with the same resources and dedication behind “Jia Tan”, you can also be hired by a software company to achieve the same goals on closed source software.

The bottom line is that it’s difficult to guess whether this attempted attack was unprecedented or unusual simply because it was caught. This means we don’t know if there are other landmines lurking within the Internet.

Personally, as someone who doesn’t work in computer security, my main takeaway from this is less about specific policy prescriptions and more about a sense of awe and appreciation. Our world runs on the unsung contributions of engineers like Colin and Freund, who spend their spare time building stuff, testing stuff, and sharing what they build for everyone’s benefit. This is inconvenient for security, but also cool.

I could not reach Colin for comment. (His website says: “To the media and reporters: I will not respond for now because first I need to understand the situation thoroughly enough. It will be sufficient to reload this page every 48 hours to check if this message has changed.”) But. I hope he will eventually come to regard this rather extraordinary effort as making his work on the XZ utils feel inadequate and become a personal goal, in fact a remarkable testament to its importance.

A version of this story originally appeared in future perfect communication. Sign up here!