All posts
Supply Chain Compromise

LiteLLM PyPI Attack: Every Hop Was a Machine Identity

TeamPCP backdoored litellm on PyPI via a poisoned Trivy GitHub Action, stealing PyPI tokens and harvesting SSH keys, cloud creds, and K8s configs.

Securityv0 Intelligence Team OWASP: ASI06 sv0 finding: nhi_compromise
supply-chain nhi pypi ci-cd litellm teampcp

The Incident

On March 24, 2026, threat actor TeamPCP published two backdoored versions of litellm to PyPI — a package downloaded roughly 3.4 million times per day and embedded in virtually every serious LLM application stack. Versions 1.82.7 and 1.82.8 were live for approximately three hours before PyPI quarantined them. The compromise was assigned CVE-2026-33634 (CVSS 9.4) and added to the CISA Known Exploited Vulnerabilities catalog with a federal remediation deadline of April 9, 2026.

The payload in 1.82.7 was twelve lines of base64-encoded Python injected into litellm/proxy/proxy_server.py, firing on any import litellm.proxy. Version 1.82.8 added a more dangerous delivery mechanism: litellm_init.pth, a 34,628-byte file placed directly in site-packages/. Python executes .pth files at interpreter startup, before any user code runs — running pip list, opening a file in an IDE, or starting a cron job is sufficient to trigger the payload with no import statement needed. Both variants read ~/.ssh/, ~/.aws/, ~/.kube/config, ~/.docker/config.json, .env files, and shell history, and queried the AWS Instance Metadata Service and the Kubernetes API directly for live, short-lived credentials. Everything was bundled as tpcp.tar.gz, encrypted with AES-256-CBC under an RSA-4096-wrapped session key, and exfiltrated to models.litellm.cloud — a domain registered one day before the attack to impersonate the legitimate LiteLLM infrastructure. A sysmon.service systemd unit persisted and polled checkmarx.zone/raw every five minutes for follow-up commands. On hosts that held Kubernetes service account tokens, the payload deployed privileged Alpine pods named node-setup-{node_name} across the kube-system namespace, requesting hostPID, hostNetwork, and full filesystem mounts — full cluster compromise from a single developer laptop. Wiz later estimated that roughly 36% of cloud environments had litellm present, most via transitive dependencies such as DSPy, MLflow, OpenHands, CrewAI, and LangChain integrations.

The attack was discovered by accident. A FutureSearch engineer was testing a Cursor MCP plugin that pulled litellm in as a transitive dependency. The .pth payload had a reentrancy bug — it spawned child Python processes that triggered the .pth file again, creating an exponential fork bomb that crashed the test machine. Without that bug, the payload would have run silently across thousands of CI/CD pipelines for days. FutureSearch filed with security@pypi.org; PyPI quarantined both versions within about 25 minutes. MITRE ATT&CK coverage: T1195.002 (Supply Chain Compromise: Compromise Software Supply Chain), T1546.018 (Event Triggered Execution: Python Startup Hooks — the .pth mechanism), T1552 (Unsecured Credentials — sub-techniques .001, .004, and .005 for credentials in files, private keys, and cloud instance metadata), and T1610 (Deploy Container).

The Authority Path That Failed

Not a single human credential was stolen to execute this attack. Every hop in the kill chain consumed one non-human identity and produced access to the next. In late February 2026, TeamPCP exploited a pull_request_target misconfiguration in Trivy’s CI pipeline and stole a privileged GitHub PAT. On March 19, that PAT was used to force-push the release tags of aquasecurity/trivy-action — 76 of 77 tags were rewritten to point at malicious code, including the widely pinned v0.69.4. On March 23, the same campaign compromised Checkmarx KICS via a cx-plugins-releases service account. LiteLLM’s CI pipeline installed Trivy unpinned from apt and ran the poisoned action on a runner that held the PYPI_PUBLISH token. Trivy read the token. TeamPCP owned PyPI. On March 24, they used that token to publish 1.82.7 and 1.82.8 under the legitimate maintainer account.

Every signature check passed. Every hash matched. Dependabot would have reported no known vulnerabilities. The credentials used to publish were legitimately obtained — just not by the maintainer. The authority carried by each machine identity in the chain — the Trivy CI runner’s PAT, the PyPI publish token, the end-user’s cloud and cluster credentials — was long-lived, broadly scoped, and exercised with no human in the loop. The trust anchor that failed first was a mutable GitHub Action tag: aquasecurity/trivy-action@v0.69.4 is a pointer, not a hash, and anyone with repository write access can silently redirect it. From there, every subsequent hop was a machine identity whose scope and lifetime nobody had audited.

SecurityV0 Perspective

An organization running SecurityV0 would see nhi_compromise surface for any machine identity in the attack chain whose scope, rotation cadence, or issuer-controlled state deviated from its justified authority baseline — starting well before March 24. The finding type applies because each identity in the chain — the Trivy CI PAT, the PYPI_PUBLISH token, the Docker Hub and PyPI registry tokens, the Kubernetes service accounts harvested at the endpoint — held authority that was neither time-bound to the action it performed nor anchored to an immutable build artifact. SecurityV0 enumerates every machine identity a CI/CD pipeline, package registry, or deployed agent can assume, maps each one to the artifact or action it is justified to authorize, and flags the delta.

The evidence pack for this finding would show: the full trust chain from CI runner to package registry to end-user install (every third-party GitHub Action reference with its resolved SHA versus its mutable tag), the scope and last-rotation timestamp of every publish token, the set of secrets reachable from each CI job’s environment, and — at the endpoint — the set of .pth files present in every active Python environment with file hashes and modification timestamps. That pack gives a security team what they need before exfiltration begins: a specific identity, a specific unpinned reference, a specific broadly-scoped token, and a specific remediation path. After the fact, the same pack answers the only questions that matter: which machines ran the malicious package, which tokens were reachable from those processes, and which clusters were exposed to node-setup-* pod deployment.

What To Do

  • Pin every third-party GitHub Action to a commit SHA, not a version tag. uses: aquasecurity/trivy-action@v0.69.4 is a mutable pointer that anyone with repository write access can silently redirect. uses: aquasecurity/trivy-action@abc123def456... is not. Treat every Action in your CI pipeline as an NHI trust anchor and hash-pin it accordingly. Dependabot can keep SHAs current with diff visibility; tags cannot be diffed.
  • Adopt PyPI Trusted Publishers (OIDC) for every package you publish. Trusted Publishers replaces long-lived API tokens with short-lived OIDC credentials minted per workflow run. Had LiteLLM been using Trusted Publishers, the stolen PYPI_PUBLISH token would not have existed in GitHub Secrets to steal. This is the single highest-leverage control against this attack pattern, and the same model exists for npm, crates.io, and RubyGems.
  • Audit your NHI trust chain, not just your SBOM. For every CI/CD pipeline, enumerate every non-human credential it touches: what token, what scopes, who issues and revokes it, when it was last rotated, and what artifact it is authorized to produce. The question defenders should ask is not “is this package version safe?” but “what machine identities have publish authority over this package, and have any of them behaved anomalously?”
  • Monitor site-packages/*.pth as a detection surface. The .pth mechanism is a legitimate Python feature that is trivially weaponized — and because it fires on interpreter startup rather than on import, it evades most SCA tooling. Inventory .pth files across every Python environment and alert on creation, modification, or unexpected content. The affected file signature is litellm_init.pth, 34,628 bytes.
  • Subscribe to registry-level behavioral signals, not just CVE feeds. Sonatype’s automated tooling blocked 1.82.7 and 1.82.8 within seconds of publication based on anomalous publish behavior — before any researcher had named the attack. The signal is not “this version is known bad” but “this version was published from an unusual IP, at an unusual time, without a corresponding git tag in the source repository.” Registry firewalls surface that delta before CVE databases do.

Sources

All posts