Trellix automates patching for 62,000 open-source projects linked to a 15-year-old Python bug

The Trellix research team said they have patched nearly 62,000 open-source projects that were susceptible to a 15-year-old path traversal vulnerability in the Python programming ecosystem.

The team identified the bug, tracked under CVE-2007-4559, in Python’s tarfile module late last year. It was first reported to the Python project in 2007 but left unchecked. Since then, it’s presence has greatly expanded as it has been used in approximately 350,000 open-source projects and countless other closed-source or proprietary software projects.

To minimize the vulnerability surface area the team drew inspiration from security researcher Jonathan Leitschuh’s DEFCON 2022 talk on fixing vulnerabilities at scale, spending months conducting automated patching to close the vulnerability in 61,895 open-source projects, according to a Jan. 23 Trellix blog post.

As many open-source projects lack dedicated staff and resources, mass patching and automated patching can be an effective tool for lessening the attack surface. While still a relatively new concept, with the first major real-world application introduced by Leitschuh last year, Trellix researchers told SC Media that their successful patch this time paves the way for the open-source community to leverage similar tactics in the future to better defend their projects from potential exploitation.

“Though it takes time and requires collaboration, our mass and automated patching effort tells the industry that this can be done,” said Kasimir Schulz, vulnerability researcher at Trellix Advanced Research Center.

Schulz said one of the biggest challenges around using an automated approach has been balancing the need to patch as many projects as possible without lowering the quality of the patch. To avoid spamming projects that weren’t actually vulnerable, the team focused the code scanning tool on the most commonly occurring formats of the vulnerability.

Specifically, the team’s patching process can be broken into two steps: the patching phase and the pull request phase, both of which are automated.

During the patching phase, with the help of quick delivery of actionable data from GitHub, the team compiled a unique list of repositories to scan after by tracking a list of repositories and files containing the keyword “import tarfile.” When the list was delivered, the team used Creosote — a free tool they built for developers to check if their applications are vulnerable — to determine which repositories needed patching. Once a vulnerable repository was identified, the team patched the file and created a local patch diff so users could compare the two files.

The team then moved to the pull request phase, first reviewing a list of local patch diffs, then creating a fork of the repository on GitHub. After cloning the fork, they replaced the original file with the patched file if the original file had not changed.

“We then committed the changes to the repository and created a pull request from our forked repository back to the original repository along with a message detailing who we were and why we were doing the pull request. At this point, it was now up to the owner of the repository to accept or reject our changes,” the blog post explained.

Automated patching is a broadsword, not a scalpel

While Trellix’s success can serve as a model to the broader open source community, security researchers say massive, automated patching comes with its own set of challenges and security risks.

“I think the automated patching approach taken by Trellix researchers is extremely helpful, yet I’m afraid it can only be practiced in specific situations, such as the one described in the blog post in which the conditions for the vulnerability to be applicable are very well defined,” said Yotam Perkal, director of vulnerability research at Rezilion.

One risk is that automated patching at scale can function more as a broadsword than a scalpel, one that may be less effective depending on the specific IT environment.

“It would be ideal if a researcher could define a CodeQL query that pinpoints a specific vulnerable code pattern without generating too much noise [or] false detections,” said Perkal. However, he noted that creating such a query isn’t straightforward, as often, multiple logical conditions have to be in place for a vulnerable code pattern to be exploitable.

Testing is another challenge with mass patching. Often, instead of just patching something and hoping it doesn’t cause harm, researchers test the software following a patch to ensure nothing is broken and the update actually mitigates the real problem, said Tim Mackey, head of software supply chain risk strategy at Synopsys.

 “If appropriate testing safeguards are in place, mass patching offers significant promise to alleviate legacy vulnerabilities,” Mackey said. “If anyone thinks this can be accomplished by a simple string replacement of ‘bad version’ with ‘good version’, without investing in testing the results of the effort, they’re likely to create more harm than good.”