AI agentsCoding agentsAgent guardrailsClaude CodeInstruction followingAgent evaluationAgent security
Every team that runs coding agents writes rules for them. Don't touch that folder. Commit through this tool. Never paste a script into the shell. We wanted to know which of those rules hold, so we measured it three ways.
In the recall test, the rules loaded at start held; rules the agent had to find depended on effort. In live work, a loaded instruction still needed a hook to catch prohibited calls. Reading a rule, following it and enforcing it are different things. A deny rule also has limits.
Real guards versus prose: written instructions, retrieved rules and enforced controls.
A rule the agent has to find: retrieval and reasoning effort
Many setups keep a short instruction file loaded at start and route the rest through an index: one line that says when to open which note.
We built a test of 20 questions. Each is a situation that never names a file. Eight need a memory note, four need a rule reached only through an index line, three a skill, three a script, and two test rules already loaded. We ran it on three models at three effort levels, one run each.
Strict recall-test scores out of 20; one run per model and effort
Effort — strict score out of 20
Opus 5.5
Fable 5.1
Sonnet 5
medium
13
11
6
high
16
16
13
xhigh
17
16
12
The routed rules are the sharp part. Pooled across the three models, a rule reached only through an index line fired 4 of 12 times at medium effort and 9 of 12 at high. The two loaded-rule checks passed in every recall-test run. Sonnet at medium answered 12 of the 20 without looking anything up.
In these runs, moving from medium to high improved every model's strict score. Moving beyond high brought little improvement overall; this small test does not establish a universal best effort level.
Routed-rule results and full-test scores use different denominators. Each model-effort combination was run once.
“Did it open the note?” is the wrong primary score
Then we looked at real work: 31 sessions of one leader seat. For every note whose trigger names a file or a tool, we asked whether the seat opened the note before acting.
It did so about 3 to 5 percent of the time: 19 of 675 moments by a loose rule, 18 of 398 by a stricter one.
That reads like a disaster. It is not. When we read the misses, most were not misses. Either the trigger was narrower than the file it named, or the lesson already lived somewhere else. One note says to commit through a particular tool. The seat “missed” that note 53 times, and all 53 were commits made through that tool. The lesson had moved into the procedure.
A team's lessons reach an agent through scripts, guards and the instruction file as often as through notes. Score the result, and keep note-opening as a second column.
Note-opening is an activity proxy, not a failure or recall rate. The commit example is one counted case.
A sentence is a request. A hook is a guard.
We are not alone here. The abstract of a public audit of 481 CLAUDE.md files (arXiv 2608.23550) reports that only about 4 to 16 percent of the security rules it retrieved had a matching built-in control, depending on the matching standard. This is a control-mapping estimate, not a measured rule-violation rate.
We put one sentence behind a hook and counted what the hook caught. The rule: no inline scripts in the shell. In one 24-hour session the guard refused 41 of 968 shell calls, 4.2 percent. Across ten large sessions the rate sat between 3.6 and 5.1 percent in nine of them.
It looked like habit, not intent. 31 of the 41 were the agent clipping output with head or tail to save context. Two clips need two pipes, and the second pipe is what this guard refuses.
The rule was loaded the whole time. The seat's own note says a pipeline “feels like a lookup, not a script”.
A large context did not appear to drive it. The rate was highest in the first calls after a start or a compaction, when the agent looks around.
The human's push mostly did not appear to drive it either. 25 of the 41 came half an hour or more after the human's last message.
A refusal did not produce a lasting change in the observed pattern. The rate in the five calls after a refusal was the same as everywhere else.
The price was small: 41 extra round trips, reported as 2.0 percent of the session's calls. That is a different denominator from the 968 shell calls. These were blocked attempts, not successful breaches.
What a hook or permission deny does not guarantee
A tool-specific deny pattern is not a universal file-access boundary. With a shell, the same effect has many spellings. In a test the human asked for, a short script reached a file that a deny rule named. A second attempt was stopped by the harness's own classifier.
One more edge is documented: a command hook that times out on PreToolUse does not itself block the call; the normal permission flow still applies. Agent SDK callback hooks have different timeout behaviour (hooks reference). A guard has to decide fast, and its failure behaviour matters.
Protect secrets with a real boundary: operating system permissions, a separate restricted account, or vault access the agent cannot obtain. A deny pattern and the agent's conduct are not substitutes for that boundary.
Blocked attempts are not successful breaches. A hook can enforce covered rules, but coverage, failure behaviour and underlying access controls still matter.
What to do with it: practical AI agent guardrails
Keep the rules that must always apply few, and load them at start.
If a rule must hold, back it with an enforcement control. For tool-call rules, a tested hook can help. Look at what it refuses: it shows you a habit. Test its coverage and failure behaviour too.
Test routed rules at high effort or above. That was the more reliable setting in this recall test; validate it on your own tasks.
Protect secrets with a boundary, not a pattern.
How we know: sources and limitations
The recall test: late September 2026, headless Claude Code, one run per model and effort. Model names and effort labels are retained as recorded. A gap of one or two questions between single runs may be noise.
The observed sessions are one seat's. The 3 to 5 percent is a floor on note-opening, not a recall rate, and “most misses were not misses” is our reading, with one case counted.
The guard counts are one seat and one hook, over six days.
The audit is someone else's work. We read its abstract, not the paper.
