AlamedaDev

Harden Your Inference Server

Marina CamachoNovember 9, 2026

What the latest vLLM, llama.cpp and CPython advisories teach teams that serve their own models, and how to set things up so the next flaw isn't an incident.

More and more teams serve open models on their own infrastructure: for cost, for latency, or because the data can't leave the building. The component that almost always shows up in that architecture is an inference server such as vLLM or llama.cpp (llama-server), and it's almost always deployed the same way: a GPU container, --api-key set, done.

The last few weeks made it clear that this isn't enough:

  • vLLM published a batch of security advisories on October 20, fixed in 0.32.0, another on October 27 and three more on November 2, fixed in 0.33.0 [1][2][11]. The most serious one (GHSA-6757-hggm-hp2w) lets a client with ordinary API access run Python code in the worker, even with insecure serialization turned off [3].
  • llama.cpp got CVE-2026-116816 on October 26: a crafted GGUF file writes outside a heap buffer just by being loaded, before any prompt is processed [4][5].
  • RunSybil showed on November 6 code execution inside the vLLM process from a modified chat_template.jinja shipped with a model, escaping the Jinja sandbox through a CPython memory bug [6][7].
  • And that same week, LiteLLM (the gateway we covered in the second post of this series) published CVE-2026-114612, where an internal user becomes admin and ends up running commands on the host [8].

None of these flaws is exotic. They all follow the same pattern: the inference server trusts something that comes from outside, whether a request, a model file or a template. This post is about designing the deployment so that trust isn't needed.

This is the fourth post in our series on best practices for AI products. The previous one covered marking synthetic output under Article 50; today we go down to the inference server layer.

1. Three ideas worth getting straight

"Authenticated" doesn't mean "trusted". The vLLM shared memory flaw isn't exploited by an anonymous attacker: anyone with a valid API key can exploit it through three ordinary HTTP routes [3]. If several customers, teams or tenants can reach your server, each of them is a potential attacker of the others.

vLLM's API key doesn't protect the whole server. vLLM's own security documentation is blunt about it: --api-key only protects endpoints under /v1, /v2, /inference and /cohere. Routes such as /invocations (which serves the same inference as /v1), /pooling, /classify, /score, /rerank, /pause, /abort_requests, /update_weights or /metrics stay open even with the key configured. The official advice is not to rely on --api-key alone and to put a reverse proxy in front that only allows the endpoints you need [9].

A model is code, not data. A model repository ships weights, but also config.json, a tokenizer, a chat template and sometimes Python code that runs with --trust-remote-code. vLLM's November 2 advisory (GHSA-qrj2-j723-m6w8) shows how a crafted config.json can redirect which weights and tokenizer get loaded to another directory on the host [10]; the llama.cpp one, how a GGUF corrupts memory on load [4]; and RunSybil's, how a template runs code [6]. Pulling a model from a public registry is the same as running a stranger's code.

2. What happened, in a bit more detail

vLLM: the multimodal cache and `pickle`

With --mm-processor-cache-type shm, vLLM shares image and audio preprocessing between workers through shared memory. Each cached item is identified by an (address, counter) pair, and the worker treated that pair as permission to read memory. A public request could supply its own pair pointing at bytes the attacker had staged in the buffer earlier, and the worker would end up calling pickle.loads() on them. The advisory documents the chain with three routes, /v1/chat/completions/render, /v1/chat/completions and /inference/v1/generate, and it works with VLLM_ALLOW_INSECURE_SERIALIZATION=0 [3].

The operational takeaway is this: the setting every deployment guide tells you to turn off didn't help here, because the flaw sat in a deserializer that's always on [3][11]. A lot of hardening protects against last year's bug, not the next one.

The rest of the batch is less spectacular but more common: an EPS file labelled as an image that reaches Ghostscript on the host (GHSA-h3rc-j82p-2c3h), out-of-vocabulary token ids that take the engine down with a single request (GHSA-q97c-p368-75c4, and GHSA-q9v9-qpjp-fc24 in --skip-tokenizer-init mode), or requests with many remote images that exhaust memory and connections before the prompt length is even checked (GHSA-5r3c-5x4v-8gjh) [1][12][13]. On a shared server, one client's denial of service is everyone's outage.

llama.cpp: the model attacks on load

CVE-2026-116816 affects llama.cpp builds from b5899 up to (not including) b10261. A BYTE-type token with text like <0x-1> in the PLaMo-2 tokenizer turns into a negative index and writes outside the buffer. VulnCheck scores it 8.5 on CVSS 4.0. The vector is local and requires the victim to load the file, which is exactly what anyone pulling a GGUF from a public registry does [4][5].

One detail says a lot about the ecosystem: the fix shipped in b10261 on August 4, about twelve weeks before the CVE [5]. If you only react to CVEs, you're late; and if you use an app or runtime that bundles its own llama.cpp, the version that matters is the one inside it, not the one in the repository.

CPython and Jinja: the chat template as an attack vector

Hugging Face chat templates are rendered with Jinja's sandbox. RunSybil found a memory bug in CPython's handling of generic types that allows escaping that sandbox, and demonstrated it with a Qwen2.5-0.5B-Instruct model with a malicious chat_template.jinja added: when served with vLLM, the template ran code in the process [6][7].

The Python Security Response Team doesn't consider it a CPython vulnerability, because running untrusted code inside a Python sandbox is outside its threat model. The patch went into the 3.13, 3.14 and 3.15 branches as a regular bug fix, not into 3.12 [7]. The researchers' conclusion is the one that matters for us: treat template sandboxes as a best-effort defense, not a security boundary [7]. This hits hardest if you let customers upload their own models, but it applies to anyone serving a model without reviewing its template.

3. How we'd set it up: defense in layers

The underlying idea is to assume the inference server will have flaws and design so that a flaw doesn't give access to anything else. In practice, five layers.

3.1 Network: the inference server doesn't talk to the world

vLLM documents that inter-node communication (PyTorch Distributed, KV cache transfer, parallelism) is insecure by default, and that torch.distributed can open a TCPStore listening on all interfaces even on a single host [9]. So:

  • The inference server lives on a private subnet. Nothing reaches it except the proxy or gateway.
  • Internal ports (KV cache, torch.distributed, data parallelism) are only reachable between cluster nodes.
  • Egress is closed. The server doesn't need the internet in production if models are already local. If it serves multimodal models with remote URLs, restrict domains with --allowed-media-domains and disable redirects with VLLM_MEDIA_URL_ALLOW_REDIRECTS=0, which is the SSRF protection vLLM recommends [9].

In Kubernetes, something like this:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vllm-isolation
  namespace: inference
spec:
  podSelector:
    matchLabels:
      app: vllm
  policyTypes: [Ingress, Egress]
  ingress:
    # Only the proxy/gateway can call the API
    - from:
        - namespaceSelector:
            matchLabels: { name: llm-gateway }
          podSelector:
            matchLabels: { app: inference-proxy }
      ports:
        - port: 8000
  egress:
    # Internal DNS and nothing else. No internet egress.
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
      ports:
        - { port: 53, protocol: UDP }

If the server is multi-node, add an ingress and egress rule between the vLLM pods themselves for the internal ports, and nothing more.

3.2 Proxy: an endpoint allowlist, not a blocklist

This is the measure with the best effort-to-benefit ratio, and it's what vLLM's own documentation recommends [9]. The proxy only lets through what your product uses and blocks everything else, including control endpoints and the ones vLLM doesn't authenticate.

# We only expose chat completions and embeddings. Everything else, 404.
limit_req_zone $http_authorization zone=per_key:10m rate=10r/s;

server {
    listen 443 ssl;
    client_max_body_size 2m;          # no 100 MB bodies

    location = /v1/chat/completions {
        limit_req zone=per_key burst=20 nodelay;
        proxy_pass http://vllm.inference.svc:8000;
        proxy_read_timeout 120s;
    }

    location = /v1/embeddings {
        limit_req zone=per_key burst=20 nodelay;
        proxy_pass http://vllm.inference.svc:8000;
    }

    # /invocations, /pooling, /score, /pause, /update_weights,
    # /metrics, /render, /inference/v1/generate... don't exist from outside.
    location / { return 404; }
}

Two caveats:

  • /metrics is useful, but let Prometheus scrape it over the internal network, not through the public proxy.
  • The render routes and /inference/v1/generate only exist with --enable-scale-out or --tokens-only [9]. If you don't use them, don't turn those modes on; if you do, don't expose them outside the cluster.

If you already have an LLM gateway in front (LiteLLM, Envoy AI Gateway, your own), this is its natural place: per-client authentication, per-key limits and logging. But remember what we saw with LiteLLM this week: the gateway is software with bugs too, so the inference server has to be isolated even from the gateway beyond the API port [8].

3.3 Models: treat them as third-party code

What changed most in these weeks is this: the model supply chain is as critical as the package supply chain. What we'd do:

  • An internal model registry. Production only loads models from your own storage, never straight from Hugging Face. Setting HF_HUB_OFFLINE=1 in the container prevents surprise downloads.
  • Pinned revision and verified hashes. Every approved model has a specific revision (commit) and the SHA-256 of every file.
  • No --trust-remote-code unless the repository's code has been reviewed and its revision pinned.
  • Safetensors by default. Avoid pickle-based formats (.bin, .pt, .pth) for third-party models.
  • Review the chat template and config.json. They're small, readable files. If a template has odd calls or dunder attributes, or config.json carries keys that point to paths (like speculators_config in the vLLM advisory [10]), it doesn't get in.

A simple admission check that runs in CI before the model is copied into the internal registry:

"""Third-party model admission: runs in CI before the internal registry."""
import hashlib, json, pathlib, re, sys

PICKLE_EXT = {".bin", ".pt", ".pth", ".pkl", ".ckpt"}
SUSPICIOUS_TEMPLATE = re.compile(r"__\w+__|\bimport\b|\bos\.|\bsubprocess\b|\beval\b")
PATH_KEYS = {"speculators_config", "auto_map"}   # keys that load something else

def sha256(p: pathlib.Path) -> str:
    h = hashlib.sha256()
    with p.open("rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()

def admit(model_dir: str, lockfile: str) -> list[str]:
    root = pathlib.Path(model_dir)
    expected = json.loads(pathlib.Path(lockfile).read_text())  # {"file": "sha256"}
    problems = []

    files = {str(p.relative_to(root)): p for p in root.rglob("*") if p.is_file()}
    for name, path in files.items():
        if path.suffix in PICKLE_EXT:
            problems.append(f"pickle-based format: {name}")
        if path.suffix == ".py":
            problems.append(f"Python code in repo (needs trust_remote_code): {name}")
        if name not in expected:
            problems.append(f"file not approved: {name}")
        elif sha256(path) != expected[name]:
            problems.append(f"hash differs from approved: {name}")

    cfg = json.loads((root / "config.json").read_text())
    for key in PATH_KEYS & cfg.keys():
        problems.append(f"config.json contains '{key}': review by hand")

    for tpl in list(root.glob("chat_template*.jinja")) + [root / "tokenizer_config.json"]:
        if tpl.exists() and SUSPICIOUS_TEMPLATE.search(tpl.read_text()):
            problems.append(f"template with suspicious constructs: {tpl.name}")
    return problems

if __name__ == "__main__":
    issues = admit(sys.argv[1], sys.argv[2])
    for i in issues:
        print("BLOCKED:", i)
    sys.exit(1 if issues else 0)

It's not a model antivirus, and a regex won't catch a well-obfuscated template. What it does is force every model that reaches production to have an explicit revision, a lockfile and a person who approved it. It's the same principle we applied to dependencies with pinned versions and hashes in the gateway post.

3.4 Process: fewer privileges, fewer features

The goal is that if someone gets code execution in the worker, they find very little:

docker run --rm \
  --gpus all \
  --read-only --tmpfs /tmp \
  --cap-drop ALL --security-opt no-new-privileges \
  --user 10001:10001 \
  --network inference-internal \
  -e HF_HUB_OFFLINE=1 \
  -e VLLM_API_KEY \
  -v /models/approved/qwen2.5-7b@<revision>:/model:ro \
  vllm/vllm-openai:<patched-version> \
  --model /model \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 16384

Adapt the example to your image and version (some images need to write to a cache, which you can mount as tmpfs), but keep the ideas:

  • Read-only filesystem and the model mounted ro. Nothing to write, nothing to persist.
  • No capabilities, no root, no privilege escalation.
  • No credentials in the container beyond what it needs. No cloud keys, no Hugging Face tokens with write access, no secrets from the rest of the platform. In vLLM's case, the process hit by the shared memory flaw can read the weights, the tokenizer, mounted credentials and the inference traffic going through it [11].
  • Turn off what you don't use. Never VLLM_SERVER_DEV_MODE=1 in production (it exposes /collective_rpc, which runs arbitrary RPCs on the engine), no profiler, and no --enable-tokenizer-info-endpoint unless you need it [9]. If you don't need the shared memory multimodal cache, don't turn on --mm-processor-cache-type shm; if you do, treat the upgrade to 0.32.0 or later as a security patch [3][11]. The same goes for hot LoRA loading (VLLM_ALLOW_RUNTIME_LORA_UPDATING).

In multi-tenant setups, the next step is separation per instance: one server (or at least one process) per customer or trust level, with a more isolated container runtime (gVisor, Kata) for models uploaded by customers. Sharing an engine between tenants means sharing its memory, its cache and its availability.

3.5 Limits: one request can't take everyone down

Half a dozen of the recent vLLM advisories are single-request denial of service [1][12][13]. Upgrading fixes part of it; limits fix the rest:

  • At the proxy: maximum body size, per-key rate limits, timeouts.
  • In vLLM: --max-model-len that matches your use case, --limit-mm-per-prompt for the number of images or audio clips per request, and VLLM_MAX_N_SEQUENCES far below the default of 16384 (the docs suggest 64 or 128 for public deployments). Also review the media download and decode limits (VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB, VLLM_MAX_MEDIA_REQUEST_DECODE_BYTES) and don't set them to 0 [9].
  • On the platform: health checks that test real generation, not just /health. One of the October 6 advisories describes an oversized min_tokens that hangs the engine while /health stays green [12].

4. Patching: the problem isn't the patch, it's knowing what you run

The pace of security advisories for inference servers now matches any exposed software: vLLM has published at least twenty advisories since September alone [12]. Without a process, every batch is a surprise. What works for us:

  1. A real inventory. Which version of vLLM, llama.cpp, Python and CUDA runs in each service, taken from the deployed image and not from the manifest. If you use a platform or distribution that bundles vLLM, the version you see is theirs, not vLLM's: ask for the upstream commit [11].
  2. Advisory subscriptions. GitHub security advisories for vllm-project/vllm and ggml-org/llama.cpp, plus your dependency scanner over the images (Trivy, Grype or whatever you use). In the llama.cpp example, the fix was available twelve weeks before the CVE [5].
  3. Triage by configuration, not just version. Many flaws depend on a flag: shm for the multimodal cache, --skip-tokenizer-init, --enable-scale-out. Keep each deployment's flags in the same inventory so you know in minutes whether you're exposed.
  4. A fast path to production. Rebuilt image, smoke tests (including a minimal eval suite to catch behavior changes), gradual rollout and rollback. vLLM releases often carry breaking changes to request handling, so test the client before rolling out [11].

Checklist: inference server in production

Network

  • [ ] The inference server isn't reachable from the internet; only the proxy or gateway reaches its API port.
  • [ ] Internal ports (KV cache, torch.distributed, parallelism) are only reachable between cluster nodes.
  • [ ] Egress is closed; if there's remote media, --allowed-media-domains is set and redirects are disabled.

Exposure

  • [ ] The proxy enforces an endpoint allowlist and returns 404 for everything else.
  • [ ] /invocations, /pooling, /score, /rerank, /pause, /update_weights and the render routes aren't reachable from outside.
  • [ ] VLLM_SERVER_DEV_MODE, the profiler and --enable-tokenizer-info-endpoint are off in production.
  • [ ] We don't rely on --api-key alone: per-client authentication and limits live in the proxy or gateway.

Models

  • [ ] Production only loads models from an internal registry, with HF_HUB_OFFLINE=1.
  • [ ] Every model has a pinned revision, a hash lockfile and a recorded approval.
  • [ ] We don't use --trust-remote-code without reviewing the code; we prefer safetensors.
  • [ ] We review each model's chat_template and config.json before admitting it.

Process and isolation

  • [ ] The container runs without root, without capabilities, with a read-only filesystem and the model mounted ro.
  • [ ] The container holds no credentials it doesn't need.
  • [ ] Different tenants or trust levels don't share an engine; customer-uploaded models run in an isolated runtime.

Limits

  • [ ] The proxy has body size limits, per-key rate limits and timeouts.
  • [ ] --max-model-len, --limit-mm-per-prompt and VLLM_MAX_N_SEQUENCES are tuned to the use case.
  • [ ] Health checks test real generation.

Patching

  • [ ] We know which version of vLLM, llama.cpp and Python runs in each service, taken from the deployed image.
  • [ ] We're subscribed to the security advisories of the projects we use and we scan the images.
  • [ ] No vLLM build is older than 0.33.0, and no llama.cpp build older than b10261.
  • [ ] There's a documented path to get a patch into production in days, not weeks.

Conclusion

Serving your own models isn't a lab curiosity anymore: it's production infrastructure, and it gets security advisories at the pace of any web server. The difference is that many teams still deploy it like an experiment, with an API key and little else.

None of the measures in this post is new. An isolated network, an endpoint allowlist, least privilege, pinned dependencies and a patching process are what we'd do for any exposed service. What's new is accepting that the model is a dependency too, and that an authenticated client's request is still untrusted input.

If you're serving your own models and want a second pair of eyes on the deployment (what's exposed, what would happen if the worker fell into the wrong hands and how long patching would take you), we'd be happy to have that conversation. No pitch: a 30-minute call with one of our engineers.

Keep reading.