And how to lock down its supply chain before the next poisoned release reaches your pip install.
Update (October 27, 2026). On October 23, LiteLLM published its final root cause analysis of the 1.107.0 incident, and it corrects their initial hypothesis: the first link in the chain was not a gateway's logs, but a master key leaked to a public repository in May and never revoked. We have updated section 2.3 and the affected controls. As for the vulnerability LiteLLM had scheduled for disclosure on October 27: as of 20:00 UTC that day, no security advisory identifying it had been published. The same day, LiteLLM shipped patch releases 1.103.10, 1.104.8, 1.105.6 and 1.106.5 with a change labeled as a security fix to how rejected requests are logged (PR #48599). LiteLLM has not publicly tied it to the embargo, so we are not assuming it is the same issue.On October 21, 2026, someone ran pip install litellm without pinning a version, or started a container tagged latest. They never imported the library or started the proxy. All it took was for Python to start in that environment: a credential harvester began reading model provider keys, cloud credentials, GitHub tokens, AI tool and MCP server configuration, encrypted all of it and sent it to a domain that imitated the official one.
This is not a hypothetical. It is what LiteLLM's own maintainers and Microsoft Security Research describe for version 1.107.0, published with malware to PyPI and to the official container images. Eight days earlier, Check Point Research had disclosed two serious vulnerabilities in the same gateway. And in March, the same project had already suffered a different compromise of its publishing pipeline.
This is the second post in our series on best practices for building AI products. The first covered agent security against prompt injection; today we go one layer down, to the infrastructure almost nobody reviews because "it's just a proxy". We are not writing this to point fingers at LiteLLM. It is a useful project, it has responded transparently, and what happened to it can happen to any dependency in your stack. We use it because it is the clearest and most recent example of a pattern that affects every team taking AI to production.
1. What an LLM gateway is, and why it concentrates so much risk
An LLM gateway sits between your applications and your model providers. Instead of every service calling OpenAI, Anthropic or Google directly, all requests go through a layer that centralizes credentials, routing between models, fallbacks, rate limits, request logging and spend control. LiteLLM, Portkey, the cloud providers' own gateways or an in-house implementation all play this role.
It is a good architectural decision. It keeps API keys out of ten different services and gives you a single point for observability and cost control. But that single point has a consequence worth saying out loud: the gateway is, by design, where all the keys live. Microsoft puts it this way in its incident analysis: in that position, a gateway may have access to provider API keys, database connection strings, workload identities, routing configuration and per-tenant policy data.
Two more properties make it more sensitive than a typical microservice:
- It sees all the traffic. Prompts, responses, documents retrieved by RAG, personal data a user pastes into a chat. If it logs full bodies, its logs are a copy of all of that.
- The same dependency lives in many places. The package does not only run on the proxy server. It also runs on developer laptops, CI agents, test environments and notebooks. Microsoft specifically points out that this widened the set of exposed environments.
With that in mind, let's look at what happened over the last few weeks.
2. Three incidents in one month, three different lessons
2.1 October 13: broken access control in the gateway itself
Check Point Research disclosed two vulnerabilities in LiteLLM (analyzed on version 1.77.7 and fixed from 1.83.9 onward):
- CVE-2026-35029: the admin endpoint
POST /config/updatedid not check the user's role. Any authenticated user, including a read-only one, could change the configuration at runtime. From there, the researchers demonstrated arbitrary file reads, admin account takeover and remote code execution by loading a Python module from a bucket. - CVE-2026-59823: an SSRF in the client-side credentials feature let an attacker redirect the gateway's calls to their own server and capture the provider API keys.
The most instructive technical detail of the first one is how the request got through. The list of routes allowed for web UI sessions included /config (probably meant for reading the configuration), and the check used startswith. /config/update starts with /config, so it passed. Check Point's conclusion sums it up: these are not new attack classes, but well-known access control and input validation failures.
Lesson: the gateway is backend software with an admin panel, and everything you already know about web security applies to it. Don't expose it to the internet "because it's internal", and don't give people accounts on it unless they need one.
2.2 October 20: the inference server is attack surface too
The same month, vLLM published advisory GHSA-6757-hggm-hp2w: in deployments with the shared-memory multimodal cache enabled (--mm-processor-cache-type shm), an authenticated user of the inference API could forge a reference that ended up in a pickle.loads() on the worker. In other words, arbitrary code execution. It affects versions up to 0.25.1 and is fixed in 0.32.0.
Lesson: if you self-host models, "authenticated" does not mean "trusted". An inference API key you hand to a customer or another team can be enough to run code on your GPUs if you are behind on patches.
2.3 October 21: the poisoned release
This is the incident behind the title of this post. According to LiteLLM's incident page:
litellm==1.107.0was live on PyPI with malware for about two hours (11:39 to 13:23 UTC).- The Docker tags
v1.107.0,latest,main-latestandmain-stablepointed to a compromised image between 11:46 and 14:20 UTC. - The official source tag did not contain the malicious files: the artifacts were built from an attacker-controlled repository with an almost identical name (
Berri-AI/litellminstead ofBerriAI/litellm).
The uncomfortable part is how it got there. After the March incident, the team had hardened its pipeline: releases from a private repository, publishing through Trusted Publishers, immutable Docker tags and signed images. Their root cause analysis, published on October 23 and presented as the canonical source of truth, reconstructs the full chain:
- May 27. A configuration file containing the master key of a LiteLLM Agent Platform deployment, an in-house tool for orchestrating agents, was accidentally committed to a public repository. The commit was force-pushed over, but the master key was never revoked.
- October 10. The attacker used that key against the platform's API. A request listing the configured agents returned the GitHub CLI OAuth token of one of the organization's two owners. It was the token the agents used to open PRs: personal, not scoped down and not tied to a service account.
- October 10 and 12. With that token, the attacker pushed throwaway branches whose CI workflows posted the repositories' secrets to an external endpoint. That is how they got the tokens of two more maintainers, stored as CI secrets. Then they deleted those workflow runs.
- October 21. Using the owner's token, the attacker bypassed branch protection and pointed the release workflow at the lookalike repository (
Berri-AI/litellm). Earlier that same week, the production approval had been temporarily relaxed to allow more frequent backports: any two engineers could trigger and approve a release. The attacker approved the release with the second token and deleted the run records with the third. The legitimate pipeline signed and published the malicious artifacts.
The first hypothesis, published while the incident was still unfolding, pointed to tokens that a coding assistant had embedded in responses, which then ended up in the logs of an internal, internet-facing LiteLLM deployment. The final analysis rules that out for this case: the request logs show the first token came from a different service, the agent platform's API.
Put simply: a key leaked five months earlier and never revoked, on an agent platform that stored and returned a GitHub token with owner permissions, was the first link in compromising the gateway's own release pipeline. None of the pieces is exotic on its own. The problem was the combination.
Microsoft's analysis adds the technical detail that explains why the blast radius was so large. The package dropped cook.pth, cook.py and sitecustomize.py outside the library directory. Python processes .pth files and sitecustomize every time the interpreter starts, so importing LiteLLM was not required: in one case observed on Windows, a routine Python invocation to install an unrelated dependency triggered the harvester. The harvester looked for AI assistant, IDE and MCP integration configuration; AWS, Azure and Google Cloud credentials (including metadata endpoints); Kubernetes tokens; SSH keys; browser data; environment variables of other processes; and credential records from the proxy's database. On Unix systems, it also left behind a loader capable of downloading and running additional code from an attacker-controlled server.
The March incident followed a similar pattern through a different door. Versions 1.82.7 and 1.82.8 were published with a credential stealer, and the investigation points to Trivy, the security scanner used in CI, as the entry point. According to JFrog, the pipeline script installed Trivy with apt-get install trivy without pinning a version or verifying a checksum, so it automatically pulled a malicious release of the scanner, which took the publishing tokens with it. One relevant detail: LiteLLM said users of its official Docker image were not affected in March because that path pinned dependencies in requirements.txt.
3. What these incidents teach (beyond LiteLLM)
Four ideas we apply in every architecture review of an AI product:
- Signing is not verifying the source. In October, the artifacts were signed by the legitimate pipeline. A signature proves who built the artifact, not from which code. You also need to verify provenance (source repository and commit) and, above all, stop accepting new versions blindly.
- Mutable tags are an open door.
latestandmain-stablepointed to a malicious image for hours. Anyone deploying by digest, or pinned to an earlier version, received nothing. Hash-pinned versions don't stop a project from being compromised, but they do stop that compromise from flowing into your production on its own. - Security and AI tools are prime targets. Endor Labs notes that the March campaign deliberately went after security-adjacent tools (Trivy, KICS) and then an LLM proxy, because they run in environments full of credentials. An AI gateway, a scanner in CI or an MCP server on a developer's laptop are exactly that kind of environment.
- Agent credentials are production credentials. The first token stolen in October was not on a production server. It sat in the configuration of an internal agent platform, and it was a personal token with owner permissions. If an agent needs to open PRs, give it a service account or a GitHub App with minimal permissions and short-lived credentials, and don't let any API return secrets in plain text. It is the same least-privilege principle we covered in the first post of this series, applied to infrastructure.
OWASP covers this as LLM03:2025 Supply Chain in its Top 10 for LLM applications: on top of the classic risks of third-party packages, the AI supply chain includes pre-trained models, LoRA adapters, deployment platforms and the providers' terms of use. Among its mitigations, it recommends keeping a signed, up-to-date inventory of components (SBOM) and evaluating AI BOMs or ML-BOMs, starting with CycloneDX.
4. Seven controls we would put in place this week
Ordered from cheapest to most expensive. The first four take an afternoon.
4.1 Hash-pinned versions, in CI too
A requirements.txt with litellm>=1.80, or a pip install litellm in a Dockerfile, means "install whatever the person who controls that account published last". Pin the version and the hash of every artifact. With pip, hash-checking mode is all or nothing: if a single line has --hash, it is required for every line.
# Generate requirements with hashes (uv or pip-tools) uv pip compile requirements.in --generate-hashes -o requirements.txt # Install requiring hashes and without building packages from source pip install --require-hashes --only-binary :all: -r requirements.txt
If you use uv with projects, uv.lock already records hashes. What matters is that CI and your Dockerfiles install from the lockfile and fail if it is out of sync:
uv sync --locked # fails if uv.lock doesn't match pyproject.toml
The same goes for pipeline tooling: the March compromise came in through a scanner installed without a pinned version. Pin GitHub Actions to the full commit SHA, as GitHub's own Actions security guidance recommends:
# Bad: whoever controls the repository can move the v4 tag - uses: actions/checkout@v4 # Good: full commit SHA, with the version as a comment - uses: actions/checkout@<full-40-character-sha> # v4.x.y
4.2 Images by digest, never by floating tag
# Bad FROM ghcr.io/berriai/litellm:main-stable # Good: the digest identifies immutable content FROM ghcr.io/berriai/litellm@sha256:<verified-digest>
The same applies to Kubernetes manifests and Helm charts. Microsoft lists this explicitly in its recommendations: digests are preferable to mutable tags because they identify immutable content.
4.3 A cooldown for new versions
Both LiteLLM compromises were available for a few hours (about 40 minutes in March; about two hours on PyPI in October). A delay of a few days before accepting a new version would have kept you from installing either one. Some tools support this natively:
# pyproject.toml — uv only considers artifacts uploaded more than 7 days ago # (relative durations require a recent version of uv) [tool.uv] exclude-newer = "7 days"
# .github/dependabot.yml — don't propose versions less than 7 days old
version: 2
updates:
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
cooldown:
default-days: 7
semver-major-days: 14Dependabot applies a 3-day cooldown to version updates by default, and that wait does not apply to security updates, which is exactly what you want: critical patches (like LiteLLM's 1.83.9) land fast, and everything else waits until the community has had time to spot anything odd.
4.4 Watch Python's startup hooks
The October malware (and version 1.82.8 in March) used .pth files with import lines, which Python executes at startup. Python 3.15 has deprecated those lines, but they still run, and they will stay in today's environments for years. A cheap control is to keep a baseline of which executable .pth files and sitecustomize.py files exist in your images, and fail the build if a new one shows up:
#!/usr/bin/env sh
# check-startup-hooks.sh — deliberately does not invoke Python: if the
# environment were compromised, starting the interpreter would run the payload.
SITE_DIRS="${SITE_DIRS:-/usr/local/lib /opt/venv/lib}"
found=$(
find $SITE_DIRS \( -name '*.pth' -o -name 'sitecustomize.py' -o -name 'usercustomize.py' \) 2>/dev/null |
while read -r f; do
case "$f" in
*.pth) grep -qE '^import[[:space:]]' "$f" && echo "$f" ;;
*) echo "$f" ;;
esac
done | sort
)
# hand-reviewed allowlist (some legitimate packages use executable .pth files)
if ! echo "$found" | diff -u startup-hooks.allowlist - ; then
echo "New Python startup hook detected. Review before deploying." >&2
exit 1
fiIt is not a replacement for a malware scanner, but it turns the specific technique used twice this year into a CI alert.
4.5 Isolate the gateway and close its egress
The gateway needs to talk to a handful of known hosts (your model providers' endpoints, the database, your observability system). It doesn't need to talk to anything else. If the October harvester had not been able to make its POST to the attacker's domain, the stolen credentials would never have left the host. Microsoft recommends that workstations, build agents, test environments and AI gateway hosts should not have unrestricted outbound HTTP access, but go through allowlist-based egress controls.
In practice:
- The admin panel and management endpoints (
/config,/key,/user...) are not published to the internet. Access over VPN or a private network, and only for administrators. - Outbound internet access only through an egress proxy with a domain allowlist, enforced at the network level (Kubernetes
NetworkPolicyworks with IPs and selectors, not domains, so filtering by hostname requires an egress proxy or a CNI that supports it). - Block access to the cloud metadata endpoint (
169.254.169.254) from the gateway container if it doesn't need it, or require IMDSv2 on AWS. The October harvester tried to pull credentials from there.
4.6 Gateway logs without secrets, and with short retention
LiteLLM's first hypothesis about October pointed to tokens that had ended up in its gateway's logs. The final analysis ruled that out for this case, but the risk is documented: in March, the project itself published an incident in which responses from custom guardrails could carry plain-text Authorization headers into spend logs and OpenTelemetry traces. And on October 27, it shipped a security fix so that logs for rejected requests are built from a fixed list of fields: until then they were built straight from the request body, and the caller could influence where that log was sent. Logging full request and response bodies is tempting for debugging, but it turns your logs into a store of prompts, personal data and, sometimes, credentials.
- Turn off full-body logging by default; enable it only temporarily and with a narrow scope.
- Redact secret patterns before persisting (
ghp_/gho_tokens,sk-,AKIA, JWTs,Authorizationheaders). - Short retention and restricted access to logs. If your LLM observability tool stores full traces, apply the same policy to it (we covered observability in this post).
Design bonus: if you write your own access control (in the gateway or any API), match routes and permissions exactly or with explicit patterns, never by prefix:
# Bad: "/config/update".startswith("/config") is True
READ_ONLY_UI_ROUTES = ["/config", "/spend", "/key/info"]
def allowed(route: str) -> bool:
return any(route.startswith(r) for r in READ_ONLY_UI_ROUTES)
# Better: exact routes and explicit method; anything admin requires the role
READ_ONLY_UI_ROUTES = {("GET", "/config"), ("GET", "/spend"), ("GET", "/key/info")}
def allowed(method: str, route: str, role: str) -> bool:
if (method, route) in READ_ONLY_UI_ROUTES:
return True
return role == "admin"4.7 An inventory (AIBOM) and a rehearsed rotation plan
When an advisory like October's comes out, the first question is "are we affected?". If answering it means asking every team on Slack, you are already late. Microsoft asks you to look for the compromised version in installed environments, uv and pip caches, internal mirrors, lockfiles, build artifacts and images. That is only fast if you have an inventory.
A minimal inventory for an AI product should include, per service and environment:
| Component | What to record |
|---|---|
| LLM gateway | Exact version, image digest, where it runs, which secrets are mounted |
| SDKs and frameworks (LLM, agents, RAG) | Version and hash from the lockfile |
| Self-hosted inference servers | Version, relevant flags (e.g. shared caches), network exposure |
| Models | Provider and version identifier; for your own or third-party weights, origin and hash |
| MCP servers and tools | Origin, pinned version, permissions and credentials they use |
| CI tooling | Pinned scanner versions, Actions by SHA |
CycloneDX has a dedicated format for this (ML-BOM), and OWASP suggests it as a starting point. But don't wait for the perfect tool: a table generated from your lockfiles and manifests already answers almost everything you need to know when someone asks "where are we running this?".
The other half is rotation, and it starts before the incident: October's chain began with a key exposed in May whose commit was overwritten but which was never revoked. Removing a secret from the history does not invalidate it; you have to revoke it and check that the revocation actually took effect. For incident response, LiteLLM's guidance for affected users is to rotate the instance IAM role, every provider key stored in the proxy database, the master and virtual keys, the database password and any secret in the environment of any process on the host. Microsoft adds revoking active sessions, because rotating a password does not automatically invalidate every token already issued. If you have never rotated your model provider keys, rehearse it now, calmly, and not at three in the afternoon on a Tuesday with an open incident. And whenever the provider supports it, prefer workload identities with short-lived credentials over static keys.
5. Two mistakes that undo everything above
Mistake 1: "we use the official image, so we're safe". In October, the official image tagged main-stable was the compromised one. Official and signed does not mean reviewed by you. What protects you is pinning by digest, delaying adoption of new versions and limiting what the container can reach if something goes wrong.
Mistake 2 (the counterweight): freezing dependencies and never updating. The Check Point vulnerabilities were fixed in 1.83.9; anyone who stayed on an earlier version "for stability" had a gateway that any authenticated user could take over. Pinning versions is not the same as not updating. It means updating deliberately, with a cooldown for anything new and a fast lane for security patches.
Checklist: LLM gateway and supply chain before going to production
Dependencies and artifacts
- [ ] All Python dependencies are installed from a lockfile with hashes (
--require-hashesoruv sync --locked), in CI and Dockerfiles too. - [ ] No image is deployed by floating tag (
latest,main-stable); all by digest. - [ ] GitHub Actions are pinned to a full SHA and CI tools (scanners included) have pinned versions.
- [ ] There is a cooldown for new versions (
exclude-newer, Dependabotcooldownor equivalent) and a documented fast lane for security patches. - [ ] The build fails if an executable
.pthfile or asitecustomize.pynot on the allowlist appears.
Gateway and inference
- [ ] The gateway's admin panel and endpoints are not reachable from the internet.
- [ ] Gateway accounts follow least privilege and are reviewed regularly (no forgotten "test" users).
- [ ] The gateway and inference servers run a version with no known vulnerabilities (checked against the project's advisories).
- [ ] The gateway's outbound traffic goes through allowlist-based egress, and the cloud metadata endpoint is blocked or restricted.
- [ ] Each application uses its own virtual key with spend limits; provider keys never leave the gateway.
Logs and secrets
- [ ] The gateway does not log full bodies by default and redacts secrets before persisting.
- [ ] Logs and LLM traces have short retention and restricted access.
- [ ] Credentials are short-lived (workload identities) wherever possible.
- [ ] Any exposed secret (in a repository, a log or a ticket) is revoked immediately, not just deleted, and the revocation is verified.
- [ ] Agents and CI workflows use service accounts or GitHub Apps with minimal permissions, never personal admin tokens.
Response
- [ ] There is an inventory (AIBOM) that answers "where is version X of this component running?" in minutes.
- [ ] There is a runbook for rotating model provider, cloud, database and CI keys, and it has been rehearsed at least once.
- [ ] The runbook says to investigate suspicious environments from the outside, without running
pythonorpipinside them. - [ ] Someone on the team follows the security advisories of your critical AI dependencies (gateway, inference server, SDKs, MCP servers).
Conclusion
An LLM gateway is one of the best architectural decisions an AI product can make: it centralizes keys, costs and observability. That is exactly why it is the component that most deserves to be treated as critical infrastructure, not as "a proxy we set up in an afternoon".
What the last few weeks have shown is not that one particular tool is insecure. It is that the chain carrying third-party code into your production has many links, and one is enough: a leaked key nobody revoked, an owner token in an agent's configuration, a scanner with no pinned version, a tag that suddenly points somewhere else. None of the controls in this post is exotic. They are long-standing software engineering practices applied to the part of the stack that is being built the fastest and reviewed the least.
Next week we continue with another piece that usually gets left out of the demos. If you run a gateway or an inference server in production and want to review its exposure and its supply chain, we'd be glad to have that conversation. No pitch: a 30-minute call with one of our engineers.
Sources
- LiteLLM. Contained: Oct 21, 2026 LiteLLM release v1.107.0 Compromised (Oct 21, 2026, updated Oct 23). https://docs.litellm.ai/blog/security-incident-oct-21-2026
- Microsoft Security Research. Inside the LiteLLM 1.107.0 supply chain compromise: Credential theft through Python startup hooks (Oct 23, 2026). https://www.microsoft.com/en-us/security/blog/2026/10/23/inside-the-litellm-1-107-0-supply-chain-compromise-credential-theft-through-python-startup-hooks/
- Check Point Research. When LLM Gateways Go Wrong: Findings from LiteLLM (Oct 13, 2026). CVE-2026-35029, CVE-2026-59823. https://research.checkpoint.com/2026/when-llm-gateways-go-wrong-findings-from-litellm/
- Check Point. Check Point Research Discovers High Risk Vulnerabilities in LiteLLM, One of the Most Widely Used LLM Gateways (Oct 13, 2026). https://blog.checkpoint.com/security/check-point-research-discovers-high-risk-vulnerabilities-in-litellm-one-of-the-most-widely-used-llm-gateways/
- vLLM. GHSA-6757-hggm-hp2w: Public multimodal SHM handle forgery reaches worker-side pickle.loads() (Oct 20, 2026). https://github.com/vllm-project/vllm/security/advisories/GHSA-6757-hggm-hp2w
- LiteLLM. Security Update: Suspected Supply Chain Incident (March 2026). https://docs.litellm.ai/blog/security-update-march-2026
- JFrog Security Research. Popular litellm Python package is the latest victim of TeamPCP's ongoing supply chain attack (March 2026). https://research.jfrog.com/post/litellm-compromised-teampcp/
- Datadog Security Labs. LiteLLM and Telnyx compromised on PyPI: Tracing the TeamPCP supply chain campaign (2026). https://securitylabs.datadoghq.com/articles/litellm-compromised-pypi-teampcp-supply-chain-campaign/
- Endor Labs. TeamPCP Isn't Done: Threat Actor Behind Trivy and KICS Compromises Now Hits LiteLLM (Mar 24, 2026). https://www.endorlabs.com/learn/teampcp-isnt-done
- OWASP Gen AI Security Project. LLM03:2025 Supply Chain. https://genai.owasp.org/llmrisk/llm032025-supply-chain/
- CycloneDX. Machine Learning Bill of Materials (AI/ML-BOM). https://cyclonedx.org/capabilities/mlbom/
- Python documentation. site — Site-specific configuration hook (
.pthfiles,sitecustomize, deprecation ofimportlines in 3.15). https://docs.python.org/3/library/site.html - pip documentation. Secure installs (hash-checking mode,
--require-hashes,--only-binary). https://pip.pypa.io/en/stable/topics/secure-installs/ - uv documentation. Settings: exclude-newer. https://docs.astral.sh/uv/reference/settings/#exclude-newer · Locking and syncing. https://docs.astral.sh/uv/concepts/projects/sync/
- GitHub Docs. Dependabot options reference: cooldown. https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference
- GitHub Docs. Secure use reference (pinning Actions to a full SHA). https://docs.github.com/en/actions/reference/security/secure-use
- PyPI Docs. Trusted Publishers. https://docs.pypi.org/trusted-publishers/ · Attestations. https://docs.pypi.org/attestations/
- LiteLLM. The attack chain behind the v1.107.0 LiteLLM release (root cause analysis, Oct 23, 2026). https://docs.litellm.ai/blog/security-incident-oct-21-2026-attack-chain
- LiteLLM. Incident Report: Guardrail logging exposed secret headers in spend logs and traces (Mar 18, 2026). https://docs.litellm.ai/blog/guardrail-logging-secret-exposure-incident
- LiteLLM. PR #48599, fix(security)!: build proxy-only failure logs from an allowlist and drop caller guardrail info (Oct 27, 2026), shipped in v1.103.10, v1.104.8, v1.105.6 and v1.106.5. https://github.com/BerriAI/litellm/pull/48599 · https://github.com/BerriAI/litellm/releases/tag/v1.106.5