How Prompt Injection Detection Works (and Where It Fails)

Every prompt injection detection product is sold with a percentage attached, and on its own that percentage means very little. The same detector can be described as catching 65 percent of attacks or 94.8 percent, truthfully, in the same paper, on the same data. What changes is a setting nobody puts in the headline.

This post is about that setting, what published benchmarks measure, and where detection stops being the thing you rely on. If you want the basics first, how prompt injection works and how to reduce it is the place to start.

Prompt injection detection rates move with the false alarm threshold

Detectors do not return yes or no. They return a score, and somebody picks a line. Set the line low and you catch more attacks and interrupt more innocent users. Set it high and the opposite.

The PromptShield benchmark makes the size of that trade concrete. Evaluated on its own benchmark, at a false positive rate of 0.1 percent, the best prior detector caught 9 percent of attacks and the paper’s own detector caught 65 percent. Loosen the threshold to a 1 percent false positive rate and the same two detectors reach 20.4 percent and 94.8 percent.

Read those together. One detector, two honest descriptions, an enormous gap between them. Any vendor figure without its false positive rate is unfalsifiable.

How the reported prompt injection detection rate changes with the false positive threshold, using published benchmark figures

The same detector, two thresholds. The headline number is chosen, not measured.

Base rates make one percent expensive

A one percent false positive rate sounds tolerable until you think about the traffic it runs against. The PromptShield authors put it plainly: “because of the base rate problem, most inputs will be benign, and attacks are very rare, so it is more important to keep the false positive rate low.”

Work it through. On ten thousand ordinary messages a day, a one percent false positive rate means a hundred interruptions of people doing nothing wrong, while real attacks might number in the single digits. The alarms drown the signal, and the predictable human response is to raise the threshold until the noise stops, which is the same as turning the detector down.

That is why the paper sets the bar where it does, arguing a practical detector needs “an extremely low false positive rate across a wide range of data distributions.” At that bar, most of what was available when they wrote fell short. Their own summary is blunt: many existing detectors are “not yet suitable for practical deployment.”

Benchmarks flatter detectors more than production does

There is a second problem underneath the first, and it is easy to miss because it looks like good news on a chart.

A 2026 study titled When Benchmarks Lie trained classifiers on 18 datasets and then tested them on datasets held out entirely from training, rather than on a random split of familiar data. Its finding is that “current evaluation practices in this literature overestimate generalization.” Standard cross-validation reported pooled AUC 8.0 to 16.5 points higher than the stricter test, with per-dataset accuracy gaps spanning 1 to 25 points.

The diagnosis is the interesting part. Between 28 and 44 percent of the features these classifiers leaned on were dataset-specific shortcuts rather than signals of an attack, and a classifier trained only to identify which dataset a sample came from reached 96.6 percent accuracy. The models were partly learning what the test looked like.

Two caveats keep this honest. Those figures come from activation-based probes rather than the commercial guardrails you might buy, and AUC is not accuracy. The transferable point is narrower and still uncomfortable: a detector’s benchmark score is an upper bound, not a forecast.

Each technique is blind somewhere different

Detection is not one thing. Four approaches show up repeatedly, and they fail in different places, which is the argument for combining them rather than choosing one.

Pattern and classifier matching on the input

Scan incoming text for instruction-shaped language. Cheap, fast, and the easiest to evade, because an attacker can rephrase indefinitely while a filter list cannot.

Canary tokens in the system prompt

Put a unique high-entropy string where the model can see it but the user cannot, then watch for it appearing in output. This does not detect attempts, it proves leaks, which makes it unusually trustworthy evidence and useless as prevention.

Behavioural detection on what the agent does next

Ignore the text and watch for actions with no cause in the conversation: a tool call, a file read, an outbound request that nothing asked for. This survives rephrasing because it targets the effect rather than the wording. It also fires after the fact, so its value depends on whether the action was reversible.

Session-level scoring across the whole conversation

Score the arc rather than each message. This is the answer to attacks assembled gradually, where every individual turn is innocuous and only the sequence is hostile. Per-message filters miss those by construction, and the same blind spot applies to a person skimming a transcript, which is why spotting an injection by behaviour beats reading the text.

Pair detection with something that does not depend on being right

Given all of the above, the sensible position is that detection is a layer with a known failure rate, not a control you can lean on. What pairs well with it are boundaries that hold even when detection misses.

Keep untrusted content out of the instruction channel, so retrieved text is data the model reads rather than commands it follows. Put the boundary in code rather than in prompt wording, since a rule written into your system survives a refactor and a rule written into prose does not. Gate the actions that matter: the MCP specification asks that there “SHOULD always be a human in the loop with the ability to deny tool invocations.” That word is a recommendation and not a requirement, a gap MCP tool poisoning takes apart in detail. Where the action cannot be undone, implement it anyway. Limiting AI agent permissions covers how to decide which actions those are.

None of that requires the detector to be right. That is the point of it.

One thing I should say plainly. Researching this site has put agents of mine through a lot of untrusted pages, vendor documentation, forum threads, security write-ups, PDFs I did not vet first, and I have never once seen something I could point at and call an injection attempt. That is not reassurance and I am not offering it as any. Most of what a detector would be catching, I would have no way of noticing, because I read the answer rather than the retrieval, and an attempt that failed leaves nothing behind. My honest position is that I have no personal evidence either way, which is a different thing from evidence that it is rare. It is also the reason I would rather spend effort on which actions need approval than on getting better at spotting the text.

FAQ

Do prompt injection detection tools work in practice?

They work in the sense that they catch a measurable share of attacks, and not in the sense that they can be relied on alone. On the PromptShield benchmark the best prior detector caught 9 percent of attacks at a 0.1 percent false positive rate, and the paper’s own detector caught 65 percent at the same threshold. Both numbers rise sharply if you accept more false alarms. Treat any product figure without its false positive rate as unverifiable.

How do I test my own setup for prompt injection?

Build a small set of adversarial inputs that match how your system is really used, then run it whenever the prompt or the tool list changes, the same way you would run tests. Include multi-turn attempts, not just single messages, and include the untrusted sources your agent really reads, such as web pages, files or tickets. The useful signal is not whether the model says something odd, it is whether it takes an action nothing in the conversation asked for.

Can a detector catch an attack spread over many messages?

Only if it scores the conversation rather than each message. Attacks built gradually across many turns can be composed entirely of individually innocuous messages, which is exactly what a per-message filter is designed to pass. Session-level scoring addresses this in principle, and it is the least mature of the common techniques, so treat a vendor’s claim here as something to test rather than assume.

Is prompt injection solved yet?

No, and the reason is structural rather than temporary. Models read instructions and data as one stream, so the defence has to come from what surrounds the model rather than from the model reliably telling them apart. Detection reduces exposure and published work keeps improving it, but no current approach removes the underlying problem, which is why the durable advice is still to limit what an assistant can reach and what it can do without a human.

Sources

Benchmark figures are measured on the cited papers’ own evaluation data and are not production measurements; PromptShield’s comparison includes the authors’ own detector. Detection rates are meaningless without the false positive rate they were measured at, so both are given throughout. Descriptions of detection techniques in practice reflect ongoing practitioner discussion rather than a formal survey.

Written by Grace

I test AI tools and agents in my own workflow, and write down what I find, including the settings that surprised me. About Grace and how posts are verified