The LiteLLM Dependency of OpenClaw Has Been Compromised

By Bing Jun, Wang Chen and Cheng Tang

On March 24 at 19:00 Beijing time, the LiteLLM version 1.82.7 appeared in the PyPI repository. Just 13 minutes later, version 1.82.8 followed closely. These two versions did not go through the GitHub PR process, and there are no corresponding commits found in the official repository.

The malicious code can steal sensitive data from the victim's system (including system environment variables, SSH keys, cloud service credentials, K8s/Git/Docker/database configurations, SSL certificate private keys, encrypted wallet configurations and keys, etc.), with the data ultimately being RSA encrypted and sent to the attacker’s server; moreover, the attacker would also create script backdoors in the target system and use system services to maintain backdoor persistence.

The total historical download volume of the LiteLLM SDK in the PyPI repository has exceeded 480 million times, and the well-known open-source project OpenClaw also depends on this open-source project.

Karpathy warned on the X platform: "Supply chain attacks like this are basically the most terrifying thing in modern software." Musk retweeted and commented, "Caveat emptor" (let the buyer beware).

Some Background Information

PyPI is the official package management repository for Python, where all maintainers of Python projects publish new versions.

When enterprise projects depend on a certain Python open-source project, developers typically run pip install -r requirements.txt, and pip automatically downloads all dependencies from PyPI, which facilitates automated version management.

The PyPI official conducts basic technical checks on all released packages, such as package format validation, metadata integrity, identity verification like PyPI Token or password, automated reviews like malware scans, etc., but does not provide services for code audits, source consistency verification, or sandbox testing that require human intervention or paid resources.

Therefore, this trust is based on: believing that the packages on PyPI are safe, trusting that package maintainers will not upload malicious code, and trusting that PyPI Tokens will not be leaked.

It is precisely because PyPI is so convenient and widely used that it has become an ideal target for attackers. Attackers only need to:

Steal a PyPI Token.
Push a malicious version.
Wait for developers around the world to download it automatically.

And developers are almost impossible to discover because:

pip install is automated, and no one checks every package.
The dependency tree is too deep to audit comprehensively.
People habitually trust PyPI.

The incident of LiteLLM being poisoned is speculated to have originated from the theft of the developer's PyPI Token.

Coincidentally.

Previously, DockerHub also experienced the OneAPI image poisoning incident. DockerHub is the world's largest container image hosting service, with over 100,000 container images from software vendors and open-source projects. Many open-source software projects choose to publish their container images here, making it very convenient for users to obtain, install, and use this software.

How to Prevent - Image Poisoning

The author of this article is a maintainer of another open-source AI gateway project, Higress. While paying attention to LiteLLM, I noticed this issue, so I want to share Higress's relevant experience in preventing such risks.

Higress is an open-source gateway software developed by Alibaba Cloud, built on API gateway capabilities and maintained by the research and development team behind Alibaba Cloud API Gateway, rather than an individual project. Higress has continuously used Alibaba Cloud's container image service for image storage and has its own official Helm repository (installation package management for K8s environments).

Using Alibaba Cloud's container image service has at least two benefits:

Not affected by DockerHub network restrictions, friendlier to domestic users, and faster image pull speeds.
Can perform image security scans, automatically intercept risk-laden image submissions.

The second point is also the core of preventing open-source image poisoning.

The cloud-native delivery chain function based on Alibaba Cloud's container image service can conduct malicious script scans immediately after the image is pushed, and if risks are found, the image can be deleted immediately.

Moreover, every time a new version is released, it is automatically done by the program rather than relying on people, which is also important. After each version release of Higress, automatic container images and installation packages are created through GitHub Actions, and the image repository key is stored based on GitHub Secrets. Permissions for published versions can be given to other collaborators in the community without needing to provide the password for the image repository to the collaborators.

How to Prevent - PyPI Package Poisoning

Unlike image poisoning, we can avoid it through Alibaba Cloud's container image service. However, for PyPI package poisoning, we must respond from the perspective of a security prevention system.

Alibaba Cloud API Gateway (with the open-source project Higress as its kernel) prioritizes security in its architectural design. Against typical threats such as credential leakage, malicious attacks, and plugin poisoning, Alibaba Cloud API Gateway has built a triple-layer defense line.

First Line of Defense: KMS Key Management, Credentials Never in Plaintext

As a traffic entry point, the API Gateway needs to manage various sensitive credentials. In the LiteLLM incident, the attackers primarily targeted the plaintext keys stored in environment variables and configuration files.

Alibaba Cloud API Gateway deeply integrates with Alibaba Cloud Key Management Service (KMS), where credentials are encrypted and managed through KMS, rather than stored in plaintext in the gateway configuration. For consumer authentication, Alibaba Cloud API Gateway supports creating independent consumer identities for API callers, authenticating them through API Keys. The consumer's API Key is stored encrypted by KMS, and only the encrypted reference is retained in the gateway configuration, with the plaintext credentials unified managed by KMS.

This mechanism brings multiple layers of security: first, credentials do not appear in plaintext in configuration files, environment variables, or logs, reducing the risk of leakage from the source; second, behind KMS is the complete permissions system of Alibaba Cloud RAM (Resource Access Management), which has strict identity authentication and permission verification for credential access; in addition, different consumers have independent API Keys and access permissions; even if a consumer's credentials are leaked, the impact is strictly limited to that consumer's permission range and will not affect other credentials.

Second Line of Defense: WAF Firewall Interaction, Building a Security Barrier at the Traffic Entry Point

The Alibaba Cloud API Gateway can easily integrate with Alibaba Cloud Web Application Firewall (WAF), adding a powerful layer of security protection at the API gateway's entry point.

As the entry point for all requests, the API Gateway naturally faces threats from various malicious traffic. After integrating WAF, every request entering the gateway will undergo deep inspection by WAF:

Malicious Request Interception: WAF maintains a real-time updated threat intelligence database that can automatically identify and intercept common web attacks such as SQL injection, XSS, and command injection.
Abnormal Traffic Detection: Conducts deep analysis of request content to identify abnormal parameter constructs and attack characteristics.
CC Attack Protection: Prevents attackers from launching large-scale malicious calls to the API, ensuring the stability of backend services.
Bot Protection: Identifies and stops automated attack tools from probing and scanning.

This is akin to installing an intelligent "security checkpoint" at the entrance of the API gateway, where malicious traffic is identified and intercepted before reaching the backend services.

Third Line of Defense: Wasm Sandbox Plugins, Extending Capabilities but Isolating Risks

The extensibility of an API gateway is a necessity, but this often comes with security risks—if the plugin code shares the same process space as the core of the gateway, a problematic plugin could impact the entire gateway.

Alibaba Cloud API Gateway is based on the Higress kernel and adopts aWasm (WebAssembly) sandbox plugin mechanism to solve this problem. Each Wasm plugin runs in an independent sandbox environment, supporting development in Go, Rust, JavaScript, and more:

Memory Isolation: Plugins cannot access the memory space of the gateway core process, nor can they read sensitive information from other plugins or gateway configurations.
Limited System Calls: The Wasm sandbox strictly restricts the system calls that plugins can initiate, preventing plugins from scanning the file system, reading environment variables, or performing arbitrary I/O operations.
Hot Updates Without Loss: The installation, update, and uninstallation of plugins do not require restarting the gateway process, supporting independent version upgrades and having zero impact on traffic.
Multi-language Support: Supports development in multiple languages such as Go, Rust, JavaScript, with community-contributed plugin code fully open source and auditable.

Even if a certain Wasm plugin contains security vulnerabilities, the impact is strictly confined within the sandbox and will not affect the credentials and other sensitive resources managed by the gateway. While maintaining flexible extensibility, security isolation is achieved at the architectural level.

It is worth mentioning that the Alibaba Cloud API Gateway also provides a dedicated WebIDE plugin development environment, supporting online coding, AI-assisted generation, and one-click Wasm plugin building. The WebIDE is integrated with Alibaba Cloud's enterprise build pipeline, and the plugin building process is completed in a VPC build cluster hosted by Alibaba Cloud—build tasks are executed within a proprietary enterprise network, and the code pulling, dependency downloading, and artifact production are all completed within the VPC, ensuring that sensitive information is not exposed to the public network. Reflecting on the LiteLLM incident, it was precisely the introduction of the poisoned third-party tool (Trivy) in the CI/CD pipeline that led to the theft of release credentials. In contrast, the plugin building pipeline of the Alibaba Cloud API Gateway is managed by Alibaba Cloud, with the building environment isolated from the public network, thus reducing the risk of external poisoning penetration from the building stage itself.

More Security Capabilities: Comprehensive Protection of API Security

In addition to the three core lines of defense mentioned above, Alibaba Cloud API Gateway also provides a series of security capabilities:

mTLS Two-way Authentication: Communication between the gateway and backend services is authenticated through mutual TLS certificates, preventing man-in-the-middle attacks.
JWT/OIDC Authentication: Built-in support for multiple standard authentication protocols, allowing integration with existing enterprise identity systems.
Refined Access Control: Management of calling permissions and quotas based on consumer dimensions, using independent credentials and quotas for different teams and projects.
Observability: Comprehensive monitoring panel for real-time viewing of call volumes and delays for each route and each consumer, enabling quick detection of any abnormal calling patterns.

Currently, LiteLLM versions 1.82.7 and 1.82.8 have been removed from PyPI.

Alibaba Cloud API Gateway: https://www.alibabacloud.com/en/product/api-gateway

Higress Open Source Community: https://higress.io/en

Community

The LiteLLM Dependency of OpenClaw Has Been Compromised

Some Background Information

How to Prevent - Image Poisoning

How to Prevent - PyPI Package Poisoning

First Line of Defense: KMS Key Management, Credentials Never in Plaintext

Second Line of Defense: WAF Firewall Interaction, Building a Security Barrier at the Traffic Entry Point

Third Line of Defense: Wasm Sandbox Plugins, Extending Capabilities but Isolating Risks

More Security Capabilities: Comprehensive Protection of API Security

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Container Compute Service (ACS)

Container Service for Kubernetes

DevOps Solution

Alibaba Cloud Flow