Supply chain attacks have been a trendy topic in the past years. Rather than directly attacking their primary target, attackers infiltrate less secure assets, such as software depenencies, firmware, or service providers, to introduce malicious code. In turn, these components also have their own layers of dependencies, and we can start to understand why this becomes a very complex problem.

Most of the coverage of such attacks focusses typosquatting issues, where attackers register in hope of developers using these dependencies by mistake. Software registries are flooded with such malicious packages, but the risk is minimal.

More recently, in March 2024, a “Jia Tan” carried out a supply chain attack on several Linux distributions by compromising an important upstream dependency called XZ Utils. They contributed to the project for three years to later became an official maintainer and start pushing malicious code to liblzma on February 2024. While liblzma is not a direct dependency of OpenSSH, Linux distributions often patched the server to add libsystemd, which in turn brought liblzma. The analysis revealed that the backdoor allowed the attacker to bypass the SSH administration protocol’s authentication process under certain conditions.

The resources needed to perform such an attack are substantial: it required real and credible development work over three years to become a maintainer of a software component of the target, with a high risk of detection. Only states or criminal groups can afford this and make it happen at scale.

This was undoubtedly out of reach for us. At the same time, it precisely targeted a specific library in the dependency tree of OpenSSH. But what would it take to compromise an entire Linux distribution directly through their public infrastructure? Is it possible to perform such a compromise as simple security researchers with no available resources but time?

Attacks against the infrastructure of Linux distributions

In the wild, we are only aware of a few detected compromises of the infrastructure of open-source projects.

In 2003, a Gentoo mirror server was targeted by a remote exploit in rsync, as well that Debian and ftp.gnu.org. In 2011, kernel.org, the organization that handles the development of the Linux kernel, was also targeted. An attacker gained root access to the organization’s main server from what appear to be a credential stuffing attack after the compromise of the machine of a developper. In 2016, attackers targeted the Linux Mint forums and blogs to change download links and point to backdoored ISO images.

Public researchers have also done their part. Research has been conducted on the security of package management systems like CocoaPods, PyPI, Composer, PEAR, etc.

The development model of Linux distributions

The development process of distributions is handled by maintainers. Linux distributions are made up of an ecosystem of independent open source projects built around a Linux kernel. These projects are developed, tested, and maintained by their own communities and developers. The maintainers of Linux distributions are responsible for retrieving the source code of each independent project (upstream), modifying the source code for integration or to address bug issues, building, and providing the packages to the distribution’s users. In parallel, downstream distributions (e.g Red Hat is downstream of Fedora) can also reuse these packages and add their own touch to it.

This cycle is summed up in the schema below:

Linux distribution maintainers need a software infrastructure to manage source code, build packages, sign them to ensure authenticity for users, and then distribute them to their communities.

If we consider compromising an entire Linux distribution, we have to compromise its software infrastructure to backdoor packages before the signing process. Otherwise, the community won’t consider the backdoored package as a legitimate package. Therefore, we need to compromise one of the following three steps:

• Directly in the dependent projects (UPSTREAM).
• The source code management system (PACKAGING).
• The package build toolchain (BUILD).

The XZ compromise has shown that the resources needed to compromise an upstream are substantial and the risks of detection are high (from the attacker’s perspective). We also don’t want to engage in any kind of social engineering and take the risk of harming maintainers or users.

Our research

With these things in mind, we could start by looking at the weakest links of these infrastructures. These distributions are built in the open and their documentation helped us identify the various services they deploy. For instance, in the case of Fedora, their Apps Directory is a good start.

The good thing about all these services is that they are all likely open-source, thus easy to deploy in our testing lab, and even sometimes developed by contributors of these distributions because of the unique constraints of their development model. These are also often self-service applications, open to all contributors, which increases their overall attack surface.

Another important factor of our approach is that we are only interested in quick wins—no need for exhaustivity here and a single good bug will do. We always had a sweet tooth when it came to argument injection bugs, so we fired up strace and started working.

Results

We successfully identified vulnerabilities in the Pagure, the Git forge used by Fedora to store their package definitions. We also compromised Open Build Service, the all-in-one toolchain used and developed by the openSUSE project for compilation and packaging.

Their exploitation by malicious actors would have led to the compromise of all the packages of the distributions Fedora and openSUSE, as well as their downstream distributions, impacting millions of Linux servers and desktops.

Technical details of our findings are described in the articles below:

• Open-Build-Service
• Pagure

We also presented them at the conference Insomni’hack on March 14th, we will update the article with the recording and slides as soon as they are published.

Compared to XZ, these attacks are within the reach of most developers and security professionals. Both rely on the same common bug class (argument injections), and only required a few days of work (disclosure included).

What now?

Even though we only have limited expertise on this topic, we don’t think there’s a silver bullet to address the risks caused by the compromise of such central pieces of infrastructure. We looked into artifact integrity frameworks like SLSA but for now the threat Threat: Exploit a vulnerability in the implementation of the source code management system to bypass controls is out of the scope of SLSA and that’s understandable!

For Pagure, we think the focus would need to be on producing third-party attestations for all package specifications, patches, and configuration files created by distribution maintainers. This system needs to be decorelated from Pagure so pushing to a Git repository on a compromised instance wouldn’t produce a valid attestation. It would also require all data fetched from upstream sources to be authenticated in some fashion too, which is still unrealistic to expect because of the wide range of upstream development processes and support level. Note that per Fedora Packaging Guidelines, the validation of upstream GPG signatures, if any, happens in the spec file and could also be removed if not signed.

For Open Build Service, the situation is more complex because compromised workers can alter both input and output files, breaking previous attestations in subtle ways. Reproducible build also become a necessity and are a notoriously hard problem to tackle. At the same time, pulling pre-compiled packages is optional and companies could absolutely deploy their own internal instance on their infrastructure.

We also think that SCA tooling has a role to play here. It can help identify which supply chains you depend on, and understand their security level to reduce exposure or contribute back to them for the benefit of everyone.

These are often maintained and operated by a handful of individuals in their free time, and they could greatly benefit from security audits and sponsored feature development. A great example of that is the recent addition of package attestations to PyPy, designed / funded by the Sovereign Tech Agency and the Google Open Source Security Team, and implemented by Trails of Bits. We hope to see this happen everywhere else as well.

Finally, the fact we found these vulnerabilities doesn’t mean these distributions are less secure that others. On the contrary, the reactivity and efficiency of the maintainers we worked with is beyond what we see in the industry—kudos again to them.

The slides of the talk at insomni'hack 2025 are available below.