Lessons from the 1.107.0 Incident: Higress and Its Wasm Sandbox and Immutable Releases
At 19:39 Beijing time on October 21, LiteLLM 1.107.0 appeared on PyPI. Seven minutes later, tags on Docker Hub and GHCR, including latest, main-stable, and others, were also pointed to poisoned images.
LiteLLM is an open-source LLM proxy with approximately 23 million downloads per week. This was already its second supply-chain poisoning incident in 2026. The previous incident involved 1.82.7 / 1.82.8 on March 24, just seven months earlier.
In both incidents, the attackers were after almost the same things: cloud credentials, SSH private keys, K8s Tokens, and the LiteLLM proxy's centrally stored API Key credentials for every LLM provider used across the organization. Security firm Wiz observed that, less than five hours after the malicious release was published, the attackers began using stolen AWS credentials to call bedrock:ListFoundationModels.
In March, we published The LiteLLM Dependency of OpenClaw Has Been Compromised, explaining how Higress guards against image poisoning and PyPI package poisoning. This time, the attack technique had evolved: the source code was clean, and only the build artifacts were poisoned. Release approval was not bypassed either; it was ‘properly’ granted using stolen accounts.
So this article seeks to answer a more fundamental question: poisoning will eventually happen; how can we ensure that malicious code gets nothing even after it gets in?

Figure 1: LiteLLM 1.107.0 Attack Timeline (Beijing Time)
According to LiteLLM's official incident report and analyses by security teams at Wiz, Chainguard, Endor Labs, Datadog, and others, this attack can be broken down into four steps.
Step 1: Obtain the keys. The attackers obtained GitHub CLI OAuth Tokens belonging to three developers. LiteLLM's official assessment is that these Tokens most likely leaked through logs from an internal LiteLLM deployment exposed to the public internet.
Step 2: Rewrite the release process. At 18:08, an attacker attempted to modify the release workflow but was blocked by branch protection. Five seconds later, the attacker used a Token with organization Owner privileges to override the protection and force the changes through. The modified workflow no longer fetched source code from the official repository BerriAI/litellm; instead, it built from the impersonating repository Berri-AI/litellm, whose name differed by just one hyphen. Next, credentials for a second account ‘approved’ the production release; after the release, credentials for a third account were used to delete the workflow run records.
Step 3: Poison only the artifacts, not the source code. The malicious change was hidden in the build configuration: pyproject.toml's maturin configuration gained a single line, data = "cook". That one line caused three files to be placed in the root of site-packages during installation:
cook.pth: contains just one line, import cook. Each time the Python interpreter starts, it executes lines in .pth files that begin with import. You do not need to import litellm; starting pip, an IDE background process, or any Python script will trigger it;cook.py: decompresses and runs an approximately 61 KB infostealer;sitecustomize.py: uses fork to spawn a daemon detached from the terminal. Every hour, it reads a server address from a ‘dead drop’ on GitHub, then downloads and executes unsigned Python code.The source code in the official Git tag was clean; the source used for the malicious build differed from it in 309 files. Chainguard pointed out that anyone reviewing only the tagged source would miss this attack entirely. These files were also outside the litellm package directory, so simply uninstalling litellm would not remove them.
Step 4: Take everything that can be taken. The infostealer had 16 modules covering:
GetCallerIdentity to verify whether the credentials were usable;There was also a module written specifically for LiteLLM: whenever DATABASE_URL was present in the environment, it connected directly to the proxy database and queried the LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable tables. These stored the upstream model providers' API Key credentials used by the proxy to forward requests for all downstream teams.
The stolen data was encrypted with AES-256-CBC, with the encryption key wrapped using RSA-4096. It was then packaged, disguised as a health check, and sent via POST to the impersonating domain www.litellm[.]dev/healthcheck.
Floating tags amplified the impact. On Docker Hub, latest, main-latest, and main-stable were all pointed to poisoned images. This meant that even without deliberately upgrading to 1.107.0, any deployment that pulled these tags during that 2.5-hour window could be affected—for example, if a Pod using latest happened to restart. Endor Labs summed it up this way: floating tags and installations without version pinning allowed a bad release to affect deployments that had never chosen it.
PyPI quarantined 1.107.0 after approximately 1.7 hours, and all the Docker Hub tags were taken down after approximately 2.5 hours. But while a package can be removed, credentials already exfiltrated cannot be taken back. LiteLLM's official recommendation was straightforward: if 1.107.0 ran on a host with external network access, treat all credentials as ‘already exfiltrated’ and rotate them all.

Figure 2: Two Poisoning Incidents: Different Entry Points, the Same Outcome
In March, the entry point was Trivy, the security scanner used in CI. TeamPCP first poisoned Trivy, then used it to steal the LiteLLM CI system's PyPI publishing Token and push malicious releases directly to PyPI. In October, the entry point shifted to developers' OAuth Tokens and a tampered release workflow.
The doors were different, but what happened after entry was almost identical:
.pth startup hooks, with no import required.We cannot guarantee that any open-source project will never be poisoned, including Higress itself. Maintainer accounts can be stolen, CI dependencies can be compromised, and approvals can be 'granted' using stolen accounts. So a more important question than 'Will it be poisoned?' is: once poisoning happens, how large is the blast radius?
Chainguard put it plainly in its analysis: a single compromised proxy host can expose keys for many upstream AI providers at once.
A self-hosted LLM proxy such as LiteLLM combines three characteristics:
The attackers' subsequent behavior observed by Wiz reinforces this point: after obtaining AWS credentials, their first batch of calls included bedrock:ListFoundationModels. What the attackers want is your permission to invoke models.
Centralized key management is a good thing in itself: unified access, auditing, and rotation all depend on it. The problem lies in the third characteristic: all extension code has the process's full privileges, so one poisoned dependency exposes the entire keychain. What truly determines the blast radius is how much privilege extension code has.

Figure 3: The Same Malicious Code: Where It Runs Determines What It Can Steal
Higress's data plane is based on Envoy (C++). Core gateway capabilities such as routing, load balancing, and TLS are implemented natively by Envoy. Extension capabilities such as AI proxying, authentication, rate limiting, content security, and MCP run as Wasm plugins in the Wasm virtual machine embedded in Envoy. The dozens of official plugins provided by Higress, including ai-proxy, ai-token-ratelimit, and ai-security-guard, all run this way.
Wasm plugins differ fundamentally from Python dependencies: Python dependencies have the process’s full privileges by default, whereas Wasm plugins have only the limited capabilities explicitly exposed by the host. Everything a plugin does must go through the Proxy-Wasm interfaces provided by the host (Envoy): reading and writing requests and responses, reading its own configuration, reading and writing key-value data shared between plugins, sending requests to registered services, and recording logs and metrics. Beyond that, it can do nothing.
This can be verified directly in the source code of proxy-wasm-cpp-host, which Envoy uses:
.pth or sitecustomize. A plugin is loaded only when explicitly referenced in a WasmPlugin resource, and its code runs only in callbacks for request processing and configuration delivery.path_open always returns ‘not found’ (NOENT). The comment for fd_read states that ‘reading any file is not supported,’ and the comment for fd_readdir states that ‘there is no real filesystem in proxy-wasm.’ The plugin cannot read a single byte from ~/.aws/credentials, ~/.ssh/id_rsa, or /var/run/secrets/kubernetes.io/serviceaccount/token.environ_get returns only variables explicitly configured for this VM. In Higress, these are the variables individually declared in WasmPlugin vmConfig.env. To the plugin, the environment variables of the host process and other processes simply do not exist.sock_send and sock_recv both return ‘not supported’ (ENOTSUP). To send an external request, a plugin must ask Envoy to send it on its behalf. Envoy first checks whether the target cluster exists and rejects the request outright if it does not. Whether the plugin wants to POST data to www.litellm[.]dev or access 169.254.169.254 to obtain cloud instance credentials, it has no path to do so unless those addresses have been registered as services in the gateway.fork/exec: it cannot spawn daemons, write systemd services, or, much less, deploy privileged Pods in the cluster.FAIL_CLOSE or FAIL_OPEN policy determines whether requests are rejected or allowed through.Let us walk through each malicious action in 1.107.0 and see what would happen inside a Wasm sandbox:

Figure 4: Examining Each Malicious Action in 1.107.0 Inside a Wasm Sandbox
One point needs to be clear: a sandbox prevents plugins from ‘crossing boundaries’; it cannot stop them from acting maliciously within their own scope of responsibility. A plugin can still read and write the requests and responses it handles, giving it an opportunity to exfiltrate data through responses. It can read its own configuration and access key-value data shared between plugins through Proxy-Wasm. For example, ai-proxy must have the upstream API Key to forward requests.
So, in addition to the sandbox, the following measures must work together:
WasmPlugin supports a sha256 field and refuses to load a downloaded plugin if its checksum does not match. In production, we recommend referencing plugins by ‘version number + sha256’ rather than latest.The sandbox determines ‘what malicious code can do after it gets in’; the release pipeline determines ‘whether it can get in.’
Starting with v2.2.4 (released in August 2026), official Higress Go/Rust Wasm plugins use an immutable release process: ‘build once, review the snapshot, and promote by digest.’ The process design and workflows are publicly available in the repository: docs/developers/immutable-plugin-releases.md.

Figure 5: LiteLLM's Breached Release Links and Higress's Corresponding Designs
Let us compare each stage of this LiteLLM incident:
org.opencontainers.image.revision, io.higress.plugin.input-hash). A formal release only copies previously reviewed candidate artifacts by digest; it never rebuilds them.plugin-release-production environment. Official release tags require manual approval by the release manager and can be created only by a dedicated release App.latest and main-stable were repointed to poisoned images. Higress's public version tags are immutable: a release fails immediately if an existing tag with the same name has a different digest. latest advances only after the entire batch of plugins passes validation and only when the version number increases rather than decreases. Released default configurations for the console, gateway, and Standalone all pin specific versions and do not depend on latest.plugins/release/snapshots/2.2.4.json locks 43 official plugins, each with its version, image digest, input hash, and source commit recorded for anyone to verify:{
"logicalId": "ai-proxy",
"version": "2.0.1",
"ociRef": "higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/ai-proxy:2.0.1",
"digest": "sha256:754c4d6c9e8aed2cabc4aa37c21f1e6dbf3b4973f01c18765ab83c17ad4d40db",
"inputHash": "sha256:fc42f8a92ee198eacb59689be844cbeb00dab79b359b583cb404f4106fef5cee",
"sourceCommit": "7774663e4353652f5c3cecd7e98a1a43b05ee15c"
}
In addition, Higress images are hosted in Alibaba Cloud Container Registry (ACR) and undergo security scanning after being pushed, as described in detail in our March article.
To be honest, no process can prevent every account compromise. The point of immutable tags and digest pinning is that, even if an incident occurs, the version you are already running cannot be silently swapped out, and a simple check reveals whether it has been replaced.
If you use LiteLLM, first follow the official incident-response guidance:
find and sha256sum to inspect site-packages for cook.pth, cook.py, and sitecustomize.py, as well as /tmp/.pc_once_* and /tmp/remote_code_loader.lock, and terminate the daemon holding that lock file;litellm==1.106.2 and reference images by digest.If you are evaluating an AI gateway, Higress covers the core scenarios of the LiteLLM proxy: access to leading model providers in China and abroad, as well as self-hosted models such as vLLM / Ollama, through a unified OpenAI-compatible protocol (ai-proxy); consumer authentication (key-auth, jwt-auth, and others); Token-level rate limiting (ai-token-ratelimit); multi-model load balancing and fallback; invocation observability and Token statistics (ai-statistics); content security (ai-security-guard); and MCP Server hosting. If you do not want to deploy it yourself, you can also directly activate Alibaba Cloud AI Gateway's Serverless Standard Edition, starting at CNY 0 (see Section VII).
If you already use Higress, here are a few configuration recommendations:
sha256; do not use latest;vmConfig.env should inject only the variables a plugin actually needs. Use the HOST method for passing through host environment variables with caution;FAIL_CLOSE. The comments in the WasmPlugin CRD also explicitly advise against using fail open for authentication plugins;Open-source Higress gives you a secure architecture, but the security of a self-hosted gateway ultimately depends on operational discipline: where images are pulled from, whether versions are pinned, how often credentials are rotated, and how quickly vulnerabilities are addressed.
LiteLLM's official advisory contains an easily overlooked statement: customers using its private enterprise image registry were not affected by this incident. The value of managed services and enterprise distribution channels lies in entrusting professional teams with enforcing this discipline.
Alibaba Cloud AI Gateway is the Enterprise Edition of Higress, sharing the same codebase as open-source Higress. It provides a unified entry point for LLMs, Agents, and MCP services, along with a migration path compatible with Higress.
The barrier to getting started is low: the Serverless Standard Edition starts at CNY 0, has no instance fee, and charges based on actual usage. It is suitable for trials, development and testing, or low-traffic workloads. Once validated and ready for production, you can switch to the Serverless Enterprise Edition.

Figure 6: Higress Enterprise Edition: Three Additional Layers Beyond the Sandbox
Custom plugins still run in the Wasm sandbox, with the platform providing lightweight isolation, strict permission controls, and code validation on top of it.
Isolation comes at a cost. When open-source Higress's Wasm plugins read and parse a complete request body, the data must be copied into the plugin's linear memory. When multiple plugins separately process the same large payload, copying and allocation overhead grow with both request size and concurrency. Wasm linear memory has another characteristic: it can grow but cannot directly shrink, so memory usage may remain high under sustained high concurrency.
The Enterprise Edition draws a clear division between security and performance: user-defined and third-party extensions continue to run in the Wasm sandbox. Core AI processing paths developed and maintained in-house by the platform, such as model protocol conversion and large-payload parsing, instead use Envoy's native plugin mechanism. They are specifically optimized for AI request processing and memory usage to reduce repeated data copying and unnecessary memory consumption. This code is maintained and delivered by the platform within a controlled development and release system.
The Alibaba Cloud AI Gateway team used the same load-test source code and synthetic JSON requests against the same Mock backend with no waiting, comparing the Serverless Enterprise Edition with a self-hosted open-source Higress deployment (the two sides used different deployment resources and access endpoints, so the results reflect performance under their respective deployments):

Figure 7: Gateway Processing Latency for 1 MB Requests (Source: Alibaba Cloud AI Gateway Team)

Figure 8: Gateway Processing Latency for 10 MB Requests (Source: Alibaba Cloud AI Gateway Team)
Consumer scale is another scenario that widens the gap. In the Higress Serverless Enterprise Edition tests published in July, when the consumer count reached 10,000, the Enterprise Edition's authentication p50 latency was 1.47 ms, compared with 50.32 ms for the open-source implementation—a difference of approximately 34 times. At 20,000 consumers, the Enterprise Edition maintained a 100% success rate, while the open-source implementation dropped to 77.14%. The reason is that the Enterprise Edition separates consumer data from authentication configuration, so the cost of a single authentication does not grow proportionally with the total number of consumers.
‘Starting at CNY 0’ means that the Standard Edition has no instance fee. The gateway itself is not billed by elapsed time, but only by actual usage (request charges and public internet traffic). When traffic is low, costs are low too; you do not have to pay upfront for an idle gateway. Once your workload grows and you need a dedicated entry point and an SLA, you can upgrade to the Enterprise Edition.
The cost of a self-hosted open-source gateway includes not only compute, networking, and redundant replicas, but also capacity reserved for traffic peaks. Image maintenance, plugin compatibility checks, upgrade regression testing, and troubleshooting also require ongoing staffing investment. According to estimates published by Higress in July, the Serverless model can reduce costs by 70%–90% compared with self-hosting in small- and medium-scale scenarios.

Figure 9: Self-Hosted vs. Managed (Source: Alibaba Cloud AI Gateway Team)
Learn about the product: Alibaba Cloud AI Gateway
Supply-chain attacks will not stop. PyPI, npm, Docker Hub, GitHub Actions—any link could be the next entry point.
In an 'execution upon installation' ecosystem, security depends on luck: you have to hope that every dependency package, every maintainer account, and every CI pipeline remains uncompromised. In a ‘no privileges by default’ sandbox, security depends on architecture: even if malicious code gets in, it can only stay in its cage.
Open-source Higress confines extension code to a Wasm sandbox and locks its release pipeline into an immutable process. Higress Enterprise Edition then entrusts credentials, the supply chain, and protection to the platform's managed services. Which you choose depends on where your team wants to focus its effort. If you want to try the managed model first, the Serverless Standard Edition starts at CNY 0, without an upfront instance fee.
From Long-Horizon Agents to Super-agents: Unlocking More Scenarios
820 posts | 61 followers
FollowAlibaba Cloud Native Community - March 30, 2026
Alibaba Cloud Native Community - April 13, 2026
Alibaba Cloud Indonesia - July 3, 2026
ApsaraDB - September 2, 2026
CloudSecurity - April 2, 2026
Xi Ning Wang(王夕宁) - April 8, 2023
820 posts | 61 followers
Follow
API Gateway
API Gateway provides you with high-performance and high-availability API hosting services to deploy and release your APIs on Alibaba Cloud products.
Learn More
Managed Service for Prometheus
Multi-source metrics are aggregated to monitor the status of your business and services in real time.
Learn More
AgentCore
AgentCore unifies agent construction, governance, and execution capabilities so enterprises can deploy diverse agents with confidence.
Learn More
Qwen
Full-range, open-source, multimodal, and multi-functional
Learn MoreMore Posts by Alibaba Cloud Native Community