Blog
The litellm Supply Chain Attack: Credentials Harvested at Python Startup
Malicious litellm 1.82.8 harvested SSH keys, cloud tokens, and K8s secrets on every Python startup. Supply chain attack analysis, IOCs, and remediation guide.
Root team
The Root team
Published :
Mar 25, 2026
It's Monday evening. A compromised PyPI credential quietly pushes a new litellm release. By Tuesday morning, your CI/CD pipelines have pulled it, your containers have rebuilt with it, and a .pth file is executing on every Python process startup, harvesting SSH keys, cloud tokens, and Kubernetes configs before exfiltrating them to an attacker-controlled server.
By the time PyPI yanks the package, the credentials are already gone.
This is supply chain compromise at scale. The manual response — audit every environment, purge caches, rotate all credentials, hunt for persistence — takes days across security, platform, and development teams. The attack takes hours.
Attack Type: Malicious PyPI release (supply chain compromise) | Affected: litellm 1.82.8 | Published: March 24, 2026 | Status: Yanked by PyPI
Who's at risk: Any organization that installed or upgraded litellm on March 24, 2026 — including automated dependency updates in CI/CD pipelines, containerized Python environments, and development machines. With 5M+ monthly downloads and deep embedding in the AI toolchain, blast radius was massive.
Check now:
The Attack: Credential Harvesting on Python Startup
litellm 1.82.8 was not a normal release. No GitHub tag. No release PR. Someone compromised the maintainer's PyPI credentials and pushed directly.
The payload was a .pth file, a Python path configuration file that executes automatically on every Python process startup. No import required. No user interaction. If litellm is installed, it runs.
What This Means: Every Python process that started after installation, including background workers, CI jobs, and containerized services, ran the attacker's code. This isn't a vulnerability you trigger by doing something wrong. It's a booby-trapped package that fires on contact.
The attack had three stages:
Collection swept for SSH keys, cloud credentials from all three major providers, Kubernetes configs, database passwords, git credentials, shell history, and environment variables. It also queried the cloud metadata endpoint at 169.254.169.254 to harvest instance-level tokens.
Exfiltration encrypted everything with a hardcoded 4096-bit RSA key (AES-256-CBC) and POSTed it to models.litellm.cloud — a domain with no affiliation to the legitimate litellm project.
K8s spread activated if a Kubernetes service account token was present. The malware enumerated secrets across all namespaces and attempted to deploy privileged node-setup-* pods on every node, installing a persistent backdoor at /root/.config/sysmon/sysmon.py.
Business Impact: Any credentials present in affected environments — cloud provider tokens, database passwords, API keys, SSH keys, Kubernetes service account tokens — should be considered compromised. If your cluster's service account had cluster-admin scope, the attacker had full cluster access.
One accidental upside: the .pth mechanism triggered on child processes too, creating a fork bomb that crashed infected machines. A bug that inadvertently limited exfiltration time.
How Root Handles Supply Chain Compromise
Root monitors your open-source dependency graph continuously. When litellm 1.82.8 appeared on PyPI without a corresponding GitHub release or signing attestation, Root flags it as an anomalous release before your environments pull it.
Phase | Traditional (days) | Root (hours) |
|---|---|---|
Detection | Day 0: Malicious release published | Hour 0: Anomalous release flagged — no tag, no signed attestation |
Notification | Day 1–2: PyPI yanks it — but pipelines already pulled it | Hour 1: Alert surfaced with affected environments mapped |
Triage | Day 2–4: Security team audits environments, maps exposure | Hour 2–4: Automated triage — which containers pulled 1.82.8, which CI jobs ran |
Remediation | Day 4–7: Credential rotation across AWS, GCP, Azure, SSH, databases | Hour 4–8: Pinned to known-good version across all environments, containers rebuilt |
Verification | Day 7–10: Container rebuilds, K8s pod audits, persistence hunting | Hour 8–24: Credential rotation initiated, persistence IOCs checked |
Post-mortem | Day 10–17: Verification, incident documentation, post-mortem | — |
What makes Root different:
No guesswork on blast radius. Root knows exactly which containers, pipelines, and environments have the compromised package — you're not hunting manually.
Version pinning without dependency hell. Root pins affected packages to known-good versions and rebuilds without breaking the rest of your dependency tree.
Application-level remediation. Root doesn't wait for you to coordinate upgrades across teams. It handles the fix and surfaces the credential rotation workflow automatically.
Continuous attestation monitoring. Releases without matching GitHub tags, signed attestations, or release PRs are flagged before they reach production.
Root-secured litellm
Root maintains a verified, uncompromised build of litellm 1.82.6 in its package registry. If you're a Root customer, you can pull the clean version directly — no manual version pinning, no exposure to compromised PyPI releases.
Book a demo to see how Root handles supply chain incidents end-to-end, detection through remediation.
Continue Reading
