KN-20 · Reviewing generated scripts
An agent built a migration script in 16 minutes. Nineteen tests passed. The builder assessed the plan as safe, and a fresh reviewer reported three high-severity findings two minutes later. The operator used a plan file, independent reviews and a rehearsal to decide whether to apply it.
A drive rename meant updating rules for a security hook, configuration files and an instruction file. Two related projects had their own relocators. The operator needed to review a proposed set of changes before applying it. A one-off script made that review especially important because its first production run would touch the real files.
The default mode produced a plan file and a readable diff without modifying targets. Apply required an explicit plan and refused to continue if a target had changed since planning. It backed up files, checked the result and rolled back on failure. A protected list identified files the tool could never touch. Delegates produced previews by default and needed a separate option to run for real. The builder developed on copies, leaving the real apply to the operator.
The builder reported 19 tests at completion. When asked to inspect the generated plan 22 minutes later, it compared the plan against the live files, called it safe and still recommended waiting for the cold review already under way.
Each reviewer started fresh on a different model and received the same brief. It could read the named files and had no permission to run the script or its tests. The brief asked for one readiness verdict and a concrete fix for each finding. It also asked what the operator and builder could change about their own process.
| Round | Verdict | Findings | Blocking highs on the tool | Tests after fixes |
|---|---|---|---|---|
| 1 | Not ready | 5 | 3 | 30 |
| 2 | Not ready | 6 | 1 | 34 |
| 3 | Not ready | 8 | 0 | 41 |
Round two’s six included three high-severity findings on a separate design paper. The three rounds saw successive versions and used reviewer-assigned severity. From the first review’s launch to the last fix was 54 minutes. The reviews took about 14 minutes, and fixes about 16. Much of the remaining time went to verification interrupted by the security hook.
| Round | Finding | Coverage gap identified in the field note |
|---|---|---|
| 1 | Partial write could escape rollback | Test writes failed cleanly |
| 1 | Append mode could skip a new rule | Fixtures lacked a similar existing rule |
| 1 | A link could bypass the protected list | Fixtures lacked a link |
| 1 | Missing checker let apply report success | Fixtures had the checker |
| 1 | Delegates could act beyond the reviewed plan | Tests ran delegates as previews |
| 2 | Plan output could overwrite an input | Output did not point at a target in tests |
| 2 | Delegate approval checked code, not proposed changes | Behavior could differ between preview and apply |
| 3 | A setting could redirect validation to another file | The environment variable was unset in tests |
| 3 | Validation proved syntax, not protection | Tests did not ask if the new rule denied a path |
After the third review, apply asked the hook about one path under every newly written rule and rolled back if any answer was anything other than blocked. The builder disputed one medium finding because an existing test covered the path, then hardened it anyway. The coverage explanations are the field note author’s reading of the review and test descriptions.
Figure 01Evidence comparison
What the review foundA partial write could escape rollback.
What the fixture missedThe fixture’s write failed cleanly.
What the review foundA link could bypass the protected list.
What the fixture missedThe fixture contained no link.
What the review foundA rule could parse without denying the target.
What the fixture missedThe test did not ask the hook to deny a path under the new rule.
Examples from successive reviews of one migration script. The source authors read the review and test descriptions. They did not run the script or its tests.
The old drive name still worked through a mapped share during the move. The builder treated the alias as a lasting feature of the design. A first reviewer questioned it. The builder initially set the concern aside because the reviewer lacked local context. The second reviewer made the failure specific: after removal of the bridge, the old root could stop resolving, and a rule rooted there could stop protecting the destination. The builder corrected the design. The bridge was removed five hours later.
All three reviewers advised agreeing on the requirement and acceptance conditions, then fixing within that scope. The plan format changed twice between reviews. Reviewers in rounds two and three received plans the corresponding script versions would refuse. The operator fixed the mode, deferred delegates from the first apply and authorized fixes against the agreed scope.
After the final fixes, the builder reported end-to-end runs in both modes against copies of the real files, 134 passing hook checks and byte-for-byte matches to expected output. The operator applied the tool and ran the delegates in a separate apply from a fresh plan about 40 minutes later. The review record contained no affirmative verdict on the full tool. The operator made the run decision after the additional checks.
Figure 02Process trace
Set the apply mode and acceptance checks.
The operator deferred delegates from the first apply and authorized changes within the agreed scope.
Pair the proposed writes with the script version.
The plan format changed twice. Two later rounds received plans the corresponding script would refuse.
Check effects as well as syntax.
The builder reported byte-for-byte matches and 134 passing hook checks after the final fixes.
The operator owns the run decision.
The operator applied the reviewed scope first and used a separate fresh plan for delegates about 40 minutes later.
One reported migration. The operator decided to run after additional checks. The review record contained no affirmative verdict on the full tool.
The container engine lost its registration after the drive letter changed. Its paths may have needed updating first, but the cause was not established. A separate editor replacement touched 397 files, including historical records. The team had to restore 312 record files from Git, and one byte in a memory file was corrupted. These two manual steps had no plan file or independent review.
The repair journal initially attributed the large rewrite to the script using a file timestamp and corrected that attribution within the hour. After the correction, the journal attributed no repair to the gated tool. The record cannot establish that the reviews caused that outcome.
Treat the reviewed plan as part of the release. Regenerate it if the format, target state or requested changes move. Set the end-to-end checks before reviewers see the plan, and require one that exercises the actual goal. Document the operator’s decision on what remains open, including manual steps outside the script.
This November 2026 field case draws on the builder’s recorded replies, three returned reviews, the plan and manifest, and a repair journal. The source authors read the reviews and counted findings, tests and elapsed times from the record. The finding table paraphrases nine of the nineteen. They did not inspect the tool code or run its tests. This publication did not rerun the case.
The rounds used different reviewers and different code versions, so their findings cannot rank the models. There is no comparison with an ungated run of the same script. The gated run and two manual incidents show what the record attributed to each step on one night.