![]() |
Supply Chain Attacks on Linux distributions - Fedora Pagure |
As discussed in the meta-article, we picked Pagure from the Fedora Apps Directory and already had a technical approach in mind. A software forge is likely to be a good target for an argument injection: we can expect the backend to shell out even when libgit2
bindings are used.
In addition, this is a self-service application, in the sense that anyone can create a Fedora contributor account and gain authenticated access to various services. For instance, this allows users to report packaging issues and contribute back directly on Pagure.
Fedora packages are made of several text files, for instance in the case of OpenSSH:
openssh.spec
.sources
.sshd.service
.While the sources
file allows validating the integrity of upstream code, attackers modifying any other file could easily sneak in malicious code.
Our effort resulted in CVE-2024-47516, an argument injection in PagureRepo.log()
. It allows writing to arbitrary files, allowing the execution of arbitrary code on any Pagure instance. We could confirm the vulnerability both on our (painfully broken) local instance and the Fedora staging server with their permission after our initial report.
We also uncovered 3 other vulnerabilities that we won’t be detailing in this article—feel free to try to PoC them based on their CVEs. The first one is a classic path traversal, while the two others are related to the way Pagure deals with repository files, and all lead to RCE as well:
view_issue_raw_file()
_update_file_in_git()
follows symbolic links in temporary clonesgenerate_archive()
follows symbolic links in temporary clonesThese bugs would have allowed us to modify any of the repositories stored on Pagure and thus the specification of any Fedora package to change its upstream sources, scripts or distribution patches. Quite a nice impact for such simple bugs eh?
As we expected, strace
and a quick manual bottom-up code review reveal many calls to the git
binary, despite Python bindings around libgit2
being available in the project.
In the snippet below, note the docstring explaining why Python bindings are not used, and the presence of --
in the command line. This is not the POSIX end-of-options
switch but a separator specific to Git to differentiate references and paths, and while Git’s --end-of-options
should have been used, only log_options
and fromref will be injectable here.
lib/repo.py
@staticmethod
def log(path, log_options=None, target=None, fromref=None):
"""Run git log with the specified options at the specified target.
This method runs the system's `git log` command since pygit2 doesn't
offer us the possibility to do this via them. [...]
"""
cmd = ["git", "log"]
if log_options:
cmd.extend(log_options)
if fromref:
cmd.append(fromref)
if target:
cmd.extend(["--", target])
return run_command(cmd, cwd=path)
There’s only one cross-reference for this function, in which it is clear that fromref is tainted by the query parameter identifier
:
lib/repo.py
@UI_NS.route("/<repo>/history/<path:filename>")
@UI_NS.route("/<namespace>/<repo>/history/<path:filename>")
# [...]
def view_history_file(repo, filename, username=None, namespace=None):
# [...]
branchname = flask.request.args.get("identifier")
if repo_obj.is_empty:
flask.abort(404, description="Empty repo cannot have a file")
# [...]
try:
log = pagure.lib.repo.PagureRepo.log(
flask.g.reponame,
log_options=["--pretty=oneline", "--abbrev-commit"],
target=filename,
fromref=branchname,
)
# [...]
Through identifier
, we can inject the option-argument --output
to git log to create a new file, or replace an existing one if permissions allow. It will contain a copy of the Git history of the file pointed by filename
.
$ man git-log
NAME
git-log - Show commit logs
...skipping...
--output=<file>
Output to a specific file instead of stdout.
For instance, by sending a request to http://pagure.local/test/history/README.md?identifier=--output=/tmp/foo.bar
, it creates the file as follows with the history of the file README.md
:
This is a powerful primitive, and the mandatory prefix (the short Git commit identifier) will likely not be a big hurdle if we target scripts rather than configuration files. For instance, with Python script, we can start the commit message with a colon and then add arbitrary code, as long as the commit identifier doesn’t start with a digit.
We also noticed that asking for the history of a non-existent file works, and will only result in an empty file. If the destination file exists, it will be truncated.
Because triggering the injection doesn’t require an account on the Pagure instance, we started thinking about what could be truncated or replaced with the history of a repository that was not under our control, like Git hooks or configuration files.
But again, this is a self-service application on which we can create an account freely and have our own repository (and commit history), so why bother!
For instance, we already know that Git repositories can be interesting when we have the ability to truncate arbitrary files, but this wouldn’t work here because Pagure stores them as bare repositories. If we tried to race the edition of a repository via the web UI, we would still need to guess / find the name of the temporary folder.
We could also be overriding Python files of the application. It worked locally on a Pagure deployed from source, but when validating the finding on the staging instance, we noticed that this idea and a few other ones simply didn’t work! The instance must have been deployed from the Fedora RPM packages where application files are owned by root
.
We need to find other target files to overwrite, where the mandatory prefix will hopefully not break anything. We can’t touch Pagure code and will need to fall back to a configuration file or data it stores: we picked the custom authentication system behind OpenSSH
.
Ever wondered how GitHub, GitLab, and others let all users authenticate as git over SSH? In our case, Pagure configures sshd to call keyhelper.py
at every connection, with any SSH key via AuthorizedKeysCommand
. OpenSSH automatically passes the information about the current system user (git and its home, /srv/git/
), the key type, and its fingerprint for use by keyhelper.py
.
/etc/ssh/sshd_config
Match User git
AuthorizedKeysCommand /usr/libexec/pagure/keyhelper.py "%u" "%h" "%t" "%f"
AuthorizedKeysCommandUser git
At this point, the user is still not authenticated by SSH, and keyhelper.py
needs to identify if a Pagure account has this key in their account settings.
/usr/libexec/pagure/keyhelper.py
# [...]
username, userhome, keytype, fingerprint = sys.argv[1:5]
# [...]
pagure_url = pagure_config["APP_URL"].rstrip("/")
url = "%s/pv/ssh/lookupkey/" % pagure_url
data = {"search_key": fingerprint}
# [...]
headers = {}
if pagure_config.get("SSH_ADMIN_TOKEN"):
headers["Authorization"] = "token %s" % pagure_config["SSH_ADMIN_TOKEN"]
resp = requests.post(url, data=data, headers=headers, verify=False)
if not resp.status_code == 200:
print(
"Error during lookup request: status: %s" % resp.status_code,
file=sys.stderr,
)
print(resp.text)
sys.exit(1)
result = resp.json()
if not result["found"]:
# Everything OK, key just didn't exist.
sys.exit(0)
print(
"%s %s"
% (pagure_config["SSH_KEYS_OPTIONS"] % result, result["public_key"])
)
It returns a line in the AuthorizedKeys
format, with SSH_KEYS_OPTIONS
set to restrict,command="/usr/libexec/pagure/aclchecker.py %(username)s"
if the public key is known. restrict
forbids features like port forwarding and the execution of files like ~/.ssh/rc
(it would have been a good target for us!), while command forces the execution of aclchecker.py
.
This script finally takes the SSH command the user intends to execute, makes sure it is a Git fetch one (git-receive-pack, git-upload-pack
), and executes it:
/usr/libexec/pagure/aclchecker.py
if "SSH_ORIGINAL_COMMAND" not in os.environ:
print("Welcome %s. This server does not offer shell access." % sys.argv[1])
sys.exit(0)
# [...]
args = os.environ["SSH_ORIGINAL_COMMAND"].split(" ")
# Expects: <git-(receive|upload)-pack> <repopath>
if len(args) != 2:
print("Invalid call, too few inner arguments", file=sys.stderr)
sys.exit(1)
cmd = args[0]
gitdir = args[1]
if cmd not in ("git-receive-pack", "git-upload-pack"):
print("Invalid call, invalid operation", file=sys.stderr)
sys.exit(1)
# [...]
runargs = [arg % result for arg in runner]
if env:
for key in env:
os.environ[key] = env[key] % result
os.execvp(runargs[0], runargs)
Indeed, trying to connect to the server through SSH and requesting a shell will correctly authenticate us, but show us an error because we’re not asking to perform any of these Git operations:
thomas@foobar ~ % ssh git@pagure.local
PTY allocation request failed on channel 0
Welcome thomas. This server does not offer shell access.
Connection to pagure.local closed.
Let’s run a strace of this whole process during a git clone—notice
anything interesting?
[pid 3817] execve("/usr/libexec/pagure/keyhelper.py", ["/usr/libexec/pagure/keyhelper.py", "git", "/srv/git", "ssh-ed25519", "SHA256:GgKi0ddkGVKnfUzd8kwjxIM9e"..
.], ["PATH=/usr/local/bin:/usr/bin:/us"..., "USER=git", "LOGNAME=git", "HOME=/srv/git", "LANG=en_US.UTF-8"]) = 0
[...]
[pid 3834] execve("/bin/bash", ["bash", "-c", "/usr/libexec/pagure/aclchecker.p"...], ["USER=git", "LOGNAME=git", "HOME=/srv/git", "PATH=/usr/local/bin:/usr/bin:
/us"..., "SHELL=/bin/bash", "MOTD_SHOWN=pam", "XDG_SESSION_ID=71", "XDG_RUNTIME_DIR=/run/user/1001", "DBUS_SESSION_BUS_ADDRESS=unix:pa"..., "XDG_SESSION_TYPE=tty"
, "XDG_SESSION_CLASS=user", "SSH_CLIENT=192.168.77.1 56903 22", "SSH_CONNECTION=192.168.77.1 5690"..., "SSH_ORIGINAL_COMMAND=git-upload-"...]) = 0
# [...]
[pid 3834] openat(AT_FDCWD</srv/git>, "/srv/git/.bashrc", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 3834] execve("/usr/libexec/pagure/aclchecker.py", ["/usr/libexec/pagure/aclchecker.p"..., "thomas"], ["SHELL=/bin/bash", "PWD=/srv/git", "LOGNAME=git", "XDG
_SESSION_TYPE=tty", "MOTD_SHOWN=pam", "HOME=/srv/git", "SSH_ORIGINAL_COMMAND=git-upload-"..., "SSH_CONNECTION=192.168.77.1 5690"..., "XDG_SESSION_CLASS=user", "US
ER=git", "SHLVL=0", "XDG_SESSION_ID=71", "XDG_RUNTIME_DIR=/run/user/1001", "SSH_CLIENT=192.168.77.1 56903 22", "PATH=/usr/local/bin:/usr/bin:/us"..., "DBUS_SESSIO
N_BUS_ADDRESS=unix:pa"..., "=/usr/libexec/pagure/aclchecker"...]) = 0
# [...]
The command directive is executed with the user’s shell bash, so it tried loading their .bashrc
!
This may sound strange for this user to have a shell, but this is actually required for this system to work: they need to execute the Python scripts with several arguments to authentify the Pagure user, and because all users connect as git
, these scripts are responsible for authorizations checks too. It sounds very reasonable for SSH to use the user’s shell to execute this forced command.
We can’t change git’s shell to /sbin/nologin
or /bin/false
, or users wouldn’t be able to connect over SSH.
(The platform sourcehut uses a similar model, we strongly suggest reading What happens when you push to git.sr.ht, and why was it so slow? to learn more about this implementation).
Back to our case: we only have to override git
’s .bashrc
and we will obtain a proper shell before the execution of the script aclchecker.py
. As shown earlier, we will first need to create our repository and send a request to http://pagure.local/test/history/README.md?identifier=--output=/srv/git/.bashrc
.
Bash is quite permissive so the mandatory prefix will not be an issue, we can use the operator ||
to execute another command as the first one will not be found:
This is what it looks like after exploiting the argument injection to override /srv/git/.bashrc
: any Pagure user will obtain a shell.
thomas@foobar ~ % ssh git@pagure.local
PTY allocation request failed on channel 0
/srv/git/.bashrc: line 1: cc75d10: command not found
uname -a
Linux pagure.local 6.8.9-100.fc38.aarch64 #1 SMP PREEMPT_DYNAMIC Thu May 2 19:13:01 UTC 2024 aarch64 GNU/Linux
id
uid=1001(git) gid=1001(git) groups=1001(git)
/srv/git/.bashrc: line 2: 573f846: command not found
Welcome thomas. This server does not offer shell access.
Connection to pagure.local closed.
We disclosed this vulnerability to Pagure maintainers via Red Hat’s Bugzilla on April 2024, and it was promptly patched on production systems a few hours later (!!). We then stayed in the loop and reviewed the patches before the official release of Pagure 5.14.1 in May along with our other reports.
We should note these are all one-off fixes that don’t really address the deeper root cause. There are still a bunch of external git invocations rather than calls to libgit2
, but at least we are satisfied with the patches of the specific vulnerabilities we reported.
In a decision unrelated to our work, Fedora decided to migrate from Pagure to Forgejo, a fork of Gitea. We can only welcome this change from a security standpoint, as Forgejo benefits from an active community. It went a long way from Gogs—where Thomas basically found basically the same vulnerabilities—and avoids a GitLab monoculture. We were only surprised by Forjero’s security track record and their 3 CVEs in total. Instead, they publish short advisories on their bug tracker when a new security release is out.
Overall, this is a very simple bug that was only made technically interesting because of the little exploitation twist and its huge impact. The migration to Forgejo will hopefully make Fedora’s package hosting platform less of an easy target and reduce the likelihood of such supply chain attacks.
Thomas Chauchefoin (@swapgs@infosec.exchange) is a Principal Application Security Engineer at Bentley Systems. With a strong background in offensive security, he helps uncover and responsibly disclose 0-days in major open-source software. He also participated in competitions like Pwn2Own or Hack-a-Sat and was nominated for two Pwnies Awards for his research on PHP supply chain security.