What the Cursor, NEAR AI and Phala incidents teach us about observability for LLM products, and how to design it so your traces aren't the prize.
Every team that ships an LLM product ends up instrumenting it: without traces you can't tell why the agent made a decision, what each user costs or what changed when you bumped the prompt version. Tools like Langfuse, LangSmith, Phoenix or any OpenTelemetry backend make this very easy. So easy that the usual move is to turn on "capture everything" on day one and never think about it again.
The problem is what's inside that "everything". A prompt contains what the user typed (including the API key they pasted so the agent could "set it up"), what the tools returned (including the .env file the agent read to understand the project) and what the model answered. The last few weeks have given us several examples of what happens when all of that is stored as is:
- Cursor explained on November 13 that an attacker used a key it had issued to Fireworks AI, which Fireworks exposed, to copy a bucket of coding-agent traces, mostly from 2025. Some traces held unredacted credentials; with them the attacker reached internal SaaS tools and, from there, a credential used to publish a malicious copy of the Claude Code extension on Cursor's marketplace [1][2].
- NEAR AI reported on November 9 that an attacker exploited a remote code execution flaw in SGLang, its inference engine, and took the chat history, settings and logs of 18 agents. Two customers' API keys were in those logs [3].
- Phala asked users on November 10 whose requests to one model went through its deployment between October 21 and November 8 to rotate any secrets included in those requests, because they were visible in the clear during processing [4].
- And in the wave of attacks on AI developer platforms that Wiz dubbed "Black October", Warp reported that an attack on one of its vendors reached agent traces and telemetry for about 1% of its users [5][6].
None of these incidents started with the traces. But in several of them, the traces were what turned partial access into a serious incident. This post is about designing observability for an LLM product so that doesn't happen.
This is the fifth post in our series on best practices for AI products. The previous one covered hardening the inference server; today we look at traces and logs.
1. Three things worth being clear about
Your traces are a production database. Probably the most sensitive one you have: in one place, nicely indexed, they hold what your users write, what your tools read and what your model replies. If your main database has encryption, access control, retention and auditing, your tracing backend needs the same.
Secrets arrive through the prompt, not just through config. Security teams usually watch where the application's API keys are stored. But in an LLM product, secrets come in through the front door: a user pastes a token, the agent runs cat .env, a tool returns an Authorization header, a provider error quotes the key it received. The researcher who uncovered the Phala incident did it exactly this way: putting canary credentials in requests and watching whether anyone used them [4].
Every copy of a trace is a new attack surface. The traces in the Cursor incident weren't stolen from the main system but from a bucket shared with a vendor for training [1]. Exporting to an observability SaaS, copying into an eval dataset, sharing with a fine-tuning provider: every copy inherits the risk and almost never inherits the controls.
2. What happened, in a bit more detail
Cursor: from a trace bucket to a marketplace
According to Cursor's report, the bucket shared with Fireworks held traces from sessions without Privacy Mode, which the company was permitted to use for training. Fewer than 0.01% of customers had API credentials exposed in them. With those credentials the attacker reached Cursor SaaS tools and took, among other things, support data, summaries of agent conversations and credentials users had sent to the support team. With a credential obtained there, they published malicious versions 2.1.312 and 2.1.313 of the Claude Code extension, available for about 11 hours and potentially received by up to 5,000 users [1][2].
The lesson for any team isn't "don't use vendors". It's that a trace with a secret in it is a stored secret, with all that entails, even if nobody decided to store it.
NEAR AI: agent logs as loot
At NEAR AI the entry point was the inference server (the same kind of component we covered last week). But what the attacker took from the 18 affected agents was chat history, settings and logs, and those logs contained customer API keys, which the company revoked [3]. Had the logs recorded metadata instead of content, or redacted the keys, the impact would have been smaller.
Phala: what travels in the prompt is yours too
Phala offers confidential inference, and its advice to affected users was still to treat their request contents as exposed and rotate any secrets in them [4]. NEAR AI reached a similar conclusion: confidential computing protects against the operator and the host, not against a flaw in the software running inside [3]. For product builders the consequence is simple: if a secret or a piece of personal data goes into the prompt, you're sending it to your model provider, to your tracing backend and to anyone who compromises either.
A recurring pattern: errors are logs too
It doesn't take a sophisticated attack. On November 10 a flaw was published in pgEdge RAG Server, fixed in 2.0.0, where the API returned the LLM provider's error text to the client, and that text could partially quote the operator's API key [7]. The same thing happens when you log a full SDK exception with the request and its headers inside.
3. How we'd set it up
The core idea is the one OpenTelemetry sums up in its sensitive data guide: the best way to avoid leaking sensitive data is not to collect it [8]. Everything else (redaction, retention, access control) is a layer in case the first one fails. OWASP lists credentials and personal data among the examples for its LLM02 risk (sensitive information disclosure) and recommends restricting access and redacting before processing [16].
3.1 Decide what you record: metadata always, content by exception
Most questions about running an LLM product can be answered without the text: which model, which prompt version, how many tokens, how long it took, what it cost, why it stopped, which tools were called and whether they failed. That's the level we'd always record.
OpenTelemetry's semantic conventions for generative AI point the same way: attributes like gen_ai.input.messages are opt-in, and the spec warns they are likely to contain sensitive information and personal data [9][10]. The official Python instrumentation doesn't capture content unless you explicitly turn it on [11]:
# Production: metadata only (model, tokens, latency, finish_reason...) OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENT # Isolated debugging environment, with synthetic or consented data: # OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY
When you do need content (to debug a case, build evals or review quality), treat it as a named exception:
- Sampling and scope. A small percentage, or only tenants who opted in, or only during a specific investigation with an end date.
- A separate store. Metadata in your usual tracing backend; content in a separate store with its own access control and short retention. The OpenTelemetry Python instrumentation can upload prompts and completions to your own storage and keep only a reference in the trace [11].
- Sensitive tools, no output. If an agent tool reads files, environment variables or internal API responses, record that it was called, the outcome and how many bytes it returned, not the content.
An example of what we'd store by default for each call:
{
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"tenant_id_hash": "t_9f2c...",
"gen_ai.request.model": "model-x-2026-10",
"prompt.version": "support-v14@3a7e1c2",
"gen_ai.usage.input_tokens": 1834,
"gen_ai.usage.output_tokens": 212,
"latency_ms": 2140,
"gen_ai.response.finish_reasons": ["stop"],
"tools": [{"name": "lookup_order", "status": "ok", "output_bytes": 812}],
"redaction": {"secrets": 1, "pii": 2}
}Note the last field: counting how many secrets and personal data items were redacted is a valuable metric. If it spikes one day, someone is pasting credentials into your product or a tool is returning more than it should.
3.2 Redact before it leaves the process
If you capture content, redact it in the application, before export. Redacting in the observability backend means the data has already travelled and been stored, even if only briefly.
A minimal redactor for secrets and some common personal data formats (the ID patterns here are Spanish DNI/NIE; swap in your own markets):
import re
# Known credential formats. Add your own providers' formats.
SECRET_PATTERNS = [
("openai_key", re.compile(r"\bsk-(?:proj-)?[A-Za-z0-9_-]{20,}")),
("anthropic_key", re.compile(r"\bsk-ant-[A-Za-z0-9_-]{20,}")),
("aws_access_key", re.compile(r"\b(?:AKIA|ASIA)[0-9A-Z]{16}\b")),
("github_token", re.compile(r"\bgh[pousr]_[A-Za-z0-9]{36,}\b")),
("jwt", re.compile(r"\beyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+")),
("bearer", re.compile(r"(?i)\bbearer\s+[A-Za-z0-9._~+/=-]{16,}")),
("private_key", re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]+?-----END [A-Z ]*PRIVATE KEY-----")),
("assignment", re.compile(r"(?i)\b(api[_-]?key|secret|token|password|passwd)\b\s*[:=]\s*['\"]?[^\s'\"]{8,}")),
]
PII_PATTERNS = [
("email", re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")),
("dni", re.compile(r"\b\d{8}[A-HJ-NP-TV-Z]\b")),
("nie", re.compile(r"\b[XYZ]\d{7}[A-HJ-NP-TV-Z]\b")),
("iban", re.compile(r"\b[A-Z]{2}\d{2}(?:[ ]?[A-Z0-9]{4}){4,7}\b")),
("phone", re.compile(r"(?<!\d)(?:\+34[ ]?)?[6789]\d{2}[ ]?\d{3}[ ]?\d{3}(?!\d)")),
]
def redact(text: str) -> tuple[str, dict[str, int]]:
counts: dict[str, int] = {}
for kind, pattern in SECRET_PATTERNS + PII_PATTERNS:
text, n = pattern.subn(f"[REDACTED:{kind}]", text)
if n:
counts[kind] = counts.get(kind, 0) + n
return text, countsAnd how to hook it into the exporter. With Langfuse, for instance, the Python SDK lets you modify each span's attributes right before export with mask_otel_spans [12]:
from langfuse import Langfuse
from langfuse.types import MaskOtelSpansParams, MaskOtelSpansResult, OtelSpanPatch
def mask_otel_spans(*, params: MaskOtelSpansParams) -> MaskOtelSpansResult | None:
patches = {}
for span_id, span in params.spans.items():
changes, total = {}, 0
for key, value in span.attributes.items():
if isinstance(value, str):
clean, counts = redact(value)
if clean != value:
changes[key] = clean
total += sum(counts.values())
if changes:
changes["redaction.count"] = total
patches[span_id] = OtelSpanPatch(set_attributes=changes)
return MaskOtelSpansResult(span_patches=patches) if patches else None
langfuse = Langfuse(mask_otel_spans=mask_otel_spans)Two details from the Langfuse docs worth noting: if the function raises an exception, the whole batch is dropped (it fails closed, which is what you want), and the hook only affects what that client exports; if you send the same spans to another backend, it gets its own unredacted copy [12].
As a second, centralised layer for every service, the OpenTelemetry Collector's redaction processor works as an allowlist: attributes not in allowed_keys are removed before any value checks [13]:
processors:
redaction/genai:
allow_all_keys: false # fail closed: anything not listed is dropped
allowed_keys:
- gen_ai.operation.name
- gen_ai.provider.name
- gen_ai.request.model
- gen_ai.response.model
- gen_ai.response.finish_reasons
- gen_ai.usage.input_tokens
- gen_ai.usage.output_tokens
- prompt.version
- tenant_id_hash
- redaction.count
blocked_key_patterns:
- ".*(token|secret|password|api_key|authorization).*"
blocked_values:
- "sk-[A-Za-z0-9_-]{20,}"
- "(AKIA|ASIA)[0-9A-Z]{16}"
- "eyJ[A-Za-z0-9_-]+\\.eyJ[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+"
summary: info # don't list which keys were removedThree honest caveats about redaction:
- Regular expressions don't catch everything. A secret with no known format, a postal address or a medical diagnosis in free text sails through. For personal data in free text there are tools like Microsoft Presidio, which combine patterns and NLP models [14], but they aren't perfect either. Redaction is a layer, not the defence.
- Hashing isn't anonymisation. OpenTelemetry warns about this: a hashed identifier can be reversed when the input space is small and predictable [8]. For user IDs, use an HMAC with a key that doesn't live in the tracing backend.
- Test the redactor like any other code. A CI test with a corpus of fake secrets and personal data that must come out clean. If someone changes the instrumentation and starts exporting a new attribute, the test should break.
3.3 Errors, without content
The pgEdge case is a useful reminder [7]. Three simple rules:
- Give the client a generic error and an ID. Never the LLM provider's error text: it can quote the key, the prompt or configuration details. The detail goes to the internal log, redacted.
- No headers or full bodies in exception logs. Many SDKs include the request in the exception; capture the error type, status code and the provider's request ID, and drop the rest.
- Watch out for "debug modes".
DEBUG-level logs in SDKs and gateways often dump entire requests. Whether they're off in production should be checked at deploy time, not left to anyone's memory.
3.4 Keep it briefly, show it to few
For whatever you decide to keep:
- Tiered retention. Our starting point: metadata for as long as you need it to operate and bill; content for days or a few weeks, except cases promoted to a curated dataset. Tools like Langfuse let you configure retention so deletion is automatic and doesn't depend on someone remembering [12]. The exact period depends on your case and your legal analysis, but "forever, just in case" isn't a period.
- Minimal, audited access. Viewing trace content should be a separate permission from viewing metrics, with a record of who looked at what. Most of the team needs dashboards, not conversations.
- Eval and training datasets through a separate path. If you want to use real traces for evaluation or training, run them through an explicit pipeline: selection, redaction, review and a store with its own access control. Never a raw copy of the tracing backend.
- Anything shared with vendors gets an expiry date. The Cursor incident started with a long-lived key handed to a third party to access a bucket [1]. Temporary credentials, access to the minimum data needed, and revocation when the work is done.
3.5 Know who receives your prompts
Take inventory: model provider, gateway, observability backend, eval tool, fine-tuning provider. For each one: what it sees (metadata or content), how long it keeps it, whether it trains on it and which region it's in. For data protection purposes, many of them will be processors and should be documented as such.
And towards users, the same message Phala gave after its incident: what you send in a request can end up exposed [4]. If your product encourages people to paste credentials or sensitive data into the chat, give them better options (a built-in secrets manager, a form that doesn't go through the model) and say so in the UI.
3.6 Be ready for the awkward question
When someone tells you one of those vendors had an incident, the question will be: which secrets and what personal data were in what they held? If you can't answer, you'll have to rotate everything and notify blindly. To be able to answer:
- An inventory of what's exported and where, versioned, just like your dependencies.
- The redaction counter from section 3.1, per service and per destination, so you at least know whether there were secrets and when.
- Canary tokens. This is the technique that uncovered the Phala incident [4]: include fake but unique credentials in periodic test requests and watch for anyone trying to use them. If one shows up, you know which path leaked and since when.
- A tested rotation procedure. The Cloud Security Alliance's Black October note stresses something often forgotten: deleting a key from the breached platform doesn't revoke it; you have to revoke it at the provider that issued it [6].
4. And what does the GDPR say about all this?
If your traces contain personal data, and if they store conversations they almost certainly do, they're a processing of personal data like any other. Several GDPR principles translate almost literally into engineering decisions [15]:
- Data minimisation (Art. 5(1)(c)): record only what's needed for the purpose. "Capture everything just in case" is hard to justify.
- Storage limitation (Art. 5(1)(e)): the tiered retention from section 3.4.
- Data protection by design and by default (Art. 25): the instrumentation default should be not to capture content.
- Security of processing (Art. 32): encryption, access control and the ability to restore and verify.
- Breach notification (Art. 33): in the event of a personal data breach, the controller must notify the supervisory authority without undue delay and, where feasible, within 72 hours. Hard to do if you don't know what was in your traces.
This isn't a substitute for a legal analysis of your case, but it does make clear that good observability hygiene and compliance point in the same direction.
Checklist: LLM traces and logs in production
What we record
- [ ] By default we only record metadata: model, prompt version, tokens, latency, cost,
finish_reasonand tool status. - [ ] Content capture is off in production (
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENTor equivalent). - [ ] When we capture content, it's sampled, time-boxed and goes to a separate store.
- [ ] Tools that read files, environment variables or internal APIs don't log their output.
Redaction
- [ ] We redact secrets and personal data in the application, before export.
- [ ] The Collector applies a second layer with an attribute allowlist.
- [ ] Every exporter goes through redaction, not just the main one.
- [ ] A CI test checks that a corpus of fake secrets and personal data comes out clean.
- [ ] We measure how many redactions happen and alert on spikes.
- [ ] User IDs are pseudonymised with an HMAC, not a plain hash.
Errors
- [ ] Clients get generic errors with an ID, never the provider's text.
- [ ] Exception logs don't include request headers or bodies.
- [ ]
DEBUGlevels in SDKs and gateways are off in production, and that's checked at deploy time.
Retention and access
- [ ] Retention is automatic, shorter for content than for metadata.
- [ ] Viewing trace content is a separate, audited permission.
- [ ] Eval and training datasets come from a curated, redacted pipeline, not raw copies.
- [ ] Credentials we share with vendors are temporary and minimally scoped.
Third parties and incidents
- [ ] We have an inventory of who receives prompts and traces, what they keep, for how long and where.
- [ ] There are canary credentials in test requests to detect leaks.
- [ ] We can revoke at the source and rotate any key within hours, and we've rehearsed it.
- [ ] We know what we'd report within 72 hours if an observability or inference vendor is breached.
Conclusion
Observability is essential for running an LLM product properly: without traces there are no evals, no cost control and no way to understand an agent. But this past month's incidents show that, done carelessly, it's also where your users' credentials and conversations pile up, ready for whoever gets there first.
None of the measures in this post are exotic. Recording metadata by default, redacting before export, keeping data briefly and knowing who sees what are practices any team would apply to a customer database. What changes with LLMs is that this database builds itself, on every request, and nobody consciously decided to store an API key in it.
If you'd like a second pair of eyes on your product's observability (what's being captured, where it goes, who can see it and what would happen if one of your vendors shows up on the next incident list), we'd be happy to have that conversation. No pitch: a 30-minute call with one of our engineers.