GuildBuild Inc.
← All Insights

KN-06 · Memory summary fidelity

What a cheaper model loses when it summarizes your agent's memory

6 min readGuildBuild Team
AI agentsAgent memoryMemory summarizationSummary faithfulnessContext engineeringAI evaluationLLM cost optimization

Agent memory piles up. Sooner or later someone has a cheaper model read it and write a digest. In our test, that digest cost well under half as much. We measured what it lost, and learned something about measuring it on the way.

KN-06: what did your memory summary leave out? Distinguish coverage—what survived—from faithfulness—is it true? A two-file study with different summary lengths, not a universal model ranking.
KN-06 article cover and social preview.

A digest is a map to its source

Conceptual source-to-digest map. A retained fact can point to source lines; an omitted fact leaves no cue in the digest alone. This is an illustration, not a three-fact test result.
Illustration of the retrieval problem, not a three-fact experiment: a retained fact can point back to its source; an omitted fact leaves no cue in the digest alone.

The test: two agent-memory files

We took two big files from a real agent's memory.

  1. A dense ledger, 55 KB: checkpoints, identifiers, sizes.
  2. A narrative file, 67 KB: what happened and why.

Sonnet and Opus each wrote a digest of both. A digest cost $0.36 to $0.47 on Sonnet and $1.05 to $1.12 on Opus, and Opus's digest was about twice as long.

This was not an equal-length comparison, so it does not isolate model quality from summary length.

Then we checked what survived. We looked for three kinds of fact a later reader needs: a size or percentage, a failure event, and a mechanism.

What survived the memory summaries

What survived the memory summaries
Kept Sonnet Opus
Ledger, of 56 facts 30 to 32 48 and 53
Narrative, of 81 rows 54 and 63 75

On the dense ledger Sonnet kept about 55 in 100 and Opus about 90.

Two things stood out.

  1. In these runs, Sonnet lost most in long entries. In entries of 5,000 bytes or more it kept fewer facts than in shorter ones, in both files. Both models cut a long entry short. Sonnet cut it to about 3 KB.
  2. Its losses are whole facts. That matters more than it sounds. Each digest entry points back to its source lines. A figure missing from a kept fact can be fetched. A fact dropped whole gives the reader no cue to fetch it from the digest alone. It may still be found through a separate search of the source.
Reported retention: ledger, 56 sampled facts—Sonnet 30 to 32, Opus 48 and 53. Narrative, 81 sampled rows—Sonnet 54 and 63, Opus 75. Opus digests were about twice as long; this was not an equal-length comparison.
Reported counts, not confidence intervals. Ledger facts and narrative rows have different denominators. Opus’s digests were about twice as long, so the comparison does not isolate model quality at equal length.

The sample that flattered the digest

Our first measurement said Sonnet was fine on the narrative file. It was wrong, and the reason is the useful part.

The first sample was drawn by a reader's judgment: the facts that looked important. The digest was written the same way. A sample drawn by judgment and a digest written by judgment can agree with each other. That agreement is not coverage of the source.

We withdrew it. The second sample was drawn by a written rule, a fixed walk through the file. On the ledger it was hashed before its reader saw any digest. On the narrative it was scored with the digests' makers masked. On those samples both files told the same story.

The rule we keep: fix the sample from the source before the output exists.

Is the digest at least true?

Dropping is one failure. Saying something false is another. We had a fresh auditor read Sonnet's digests of big files entry by entry against the source.

  1. The auditor's first grade: 9 of 15 entries stated something false or lost a failure.
  2. A second reader then read every serious line by hand, and the disputed ones were ruled on. 5 of 15 stood.
  3. The auditor's quotes matched the source 31 of 31 times. Its false alarms were of grade, not of fact.

What the failing entries did: marked a fix as made when the source only ordered it, put a statement in the wrong mouth, gave a mechanism the source never gives, and left a failure in no field.

Most of the other flagged lines were lost detail. We read a sample of those and they were right. That is a digest compressing, which is its job.

The first judgment-selected coverage sample was withdrawn. Its replacement followed a fixed source walk, with ledger sample hashing before evaluator exposure and masked narrative scoring. In a separate audit, 9 of 15 entries were initially flagged and 5 of 15 findings stood after review; 31 of 31 quotations matched their sources. Quotation count and entry count are different denominators.
Two separate checks: source coverage and summary faithfulness. The audit counted entries; the quotation check counted passages. Nine initial flags became five upheld findings after review.

What we do now

  1. In our current workflow, files of 50,000 bytes or more go to the larger model. This is a routing heuristic, not a validated boundary.
  2. A digest of a big file is a map to its source lines, not a substitute for them.
  3. When something turns on a digest's claim, we open the lines it points to.

Summary faithfulness is a wide public field. This is its agent-memory case: the reader is another agent, and what it needs is failures, sizes and causes. Anthropic likewise recommends preserving relevant information before tightening a compaction summary in its context-engineering guidance.

Three practical steps: route large files carefully, preserve source-line pointers, and verify consequential claims against the original text. The team’s 50,000-byte routing threshold is a heuristic, not a validated model boundary. The next suggested evaluation matches summary budgets and tests downstream tasks.
The team’s current workflow, not a universal size threshold. Matching output budgets and testing the next task are suggested follow-up evaluations, not additional results.

How we know: sources and limitations

  1. Two files, one reader per file, late September and early October 2026.
  2. Two Sonnet runs and one or two Opus runs per file. The two Sonnet runs on the narrative differ by 9 rows of 81, so small gaps are inside the spread.
  3. The audit is 15 entries from 9 digests by Sonnet 5.5 at medium effort. Five findings stood after review. This small sample is not a general failure-rate estimate.
  4. No small file has been measured on a rule-drawn sample. The cheaper model below 50,000 bytes is untested.
  5. Why long entries lose more is not shown. Identifier density and entry length are tangled in one of the two files.
  6. Costs are the study's reported per-digest figures from September–October 2026, not a current price quote. The note does not specify the currency or pricing basis, or every model version and effort used in the retention comparison.
  7. Opus's digests were about twice as long. The counts do not establish an equal-length quality ranking or downstream task success.

Related field note: Where the tokens go in a real agent team.