An AI summary of a ninety minute design call read cleanly, named every topic, and would have passed any review. It was also missing the three things that changed what we build next. Summarizing is lossy in a specific direction: it drops reversals, negative findings, and things that never came up, which is exactly the class of fact that changes a decision. Any pipeline where an agent reads a compiled artifact instead of the original inherits that loss silently, and every extra hop compounds it. This post covers why the loss is directional, where it enters a real pipeline, and the mechanisms we use to keep it away from decisions.
Why the loss runs in one direction
A summarizer optimizes for something close to “what was this about”. Under that objective, facts are not equally cheap to keep.
A decision that was made has text behind it. A decision that was reversed has text twice, and the summarizer has to notice that the second block cancels the first rather than reinforcing it. A finding that something does not exist is a sentence about an absence, competing for space with a dozen sentences about things that do. And an item that sat on the agenda and was never raised has no text at all. It exists only as the difference between two documents, and the summarizer only ever sees one of them.
Factory.ai landed the general form of this while benchmarking context compression for coding agents: “generic summarization treats all content as equally compressible. A file path might be ‘low entropy’ from an information-theoretic perspective, but it is exactly what the agent needs to continue working.” (Factory.ai, “Evaluating Context Compression for AI Agents”). Compressibility and importance are different axes, and nothing in the objective ties them together.
Negation takes the worst of it. MIT researchers testing vision-language models on queries containing “no” and “not” found retrieval performance dropped by nearly 25% on negated captions, and the best models managed about 39% accuracy on multiple choice questions about negation, several of them at or below random chance. They named the shortcut affirmation bias: the model ignores the negation word and latches onto the objects (MIT News, May 2025). That study is on vision-language models, so read it as an illustration of the shape rather than a measurement of your text pipeline. The shape is the point: a system trained to notice what is there is not thereby trained to notice what is absent.
Accuracy is not completeness, and reviewers check the easy one
The cleanest demonstration comes from healthcare. In a PLOS Digital Health study of 100 randomly selected emergency department encounters, GPT-4 drafted the encounter summaries and clinicians graded them. The result: “Summaries generated by GPT-4 were mostly accurate, with inaccuracies found in only 10% of cases, however, 42% of the summaries exhibited hallucinations and 47% omitted clinically relevant information” (PLOS Digital Health, 2025). Only 33% were error free across every domain assessed. The models are a generation old now and the numbers will have moved, but the split will not: the accuracy score and the omission score measure different things and they disagreed by a factor of four.
Accuracy asks whether what the summary says is true. Completeness asks whether what it left out mattered. Only the first can be checked by reading the summary. Checking the second means going back to the source, which is the work the summary was supposed to save you. So the property people verify is the property that is cheap to verify, and the summary passes.
The same split shows up in meeting notetakers, where most teams meet this problem without noticing. Evaluating six advisor-focused notetakers, the Oasis Group found transcription essentially solved, with one tool hitting 100% on a scripted meeting, while summarization captured key data points at 85% to 96% and action item accuracy fell to 62% to 87% (Michael Kitces, “The Risks Of AI Meeting Notetakers”). Note the gradient. The layer that merely transcribes is near perfect. The layer people actually act on is the weakest one in the stack.
The trap: the summary is accurate, so the record looks settled. It reads as settled precisely because the unsettled parts are the ones that did not survive.
Every hop is another compilation
Draw the chain most teams are running and the problem stops being about notetakers.
Four compilations before anyone decides anything, and each one applies the same directional filter to the output of the last. Nothing in the chain ever reintroduces what an earlier step dropped, because no step after the first has access to the original.
This is measurable inside agent harnesses too, where the same operation is called compaction. Factory.ai compared three production approaches (their own, Anthropic’s SDK compression, and OpenAI’s /responses/compact endpoint) across 36,611 messages on six dimensions. On artifact trail, the ability to keep track of the files a session had touched, all three scored badly: 2.45, 2.33 and 2.19 out of 5.0 respectively. Not one vendor’s summarizer reliably kept the concrete references the work depended on.
Anthropic says the same thing about its own mechanism, plainly, in its engineering guidance: “Overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later” (Anthropic, “Effective context engineering for AI agents”). The phrase worth sitting with is only becomes apparent later. At compaction time the dropped fact looked like noise. That is not a tuning failure, it is the structure of the problem: the summarizer is asked to rank importance before the thing that makes a fact important has happened.
Retrieval has the same hole. If what you indexed is a digest rather than the original, retrieval cannot return what the digest dropped, and a confident answer built on the recall is indistinguishable from a correct one.
What it cost us, three times
We hit this three times in a single week, in three different shapes. The specifics are anonymized, the mechanism is not.
A client design call, ninety minutes, on a system we are building. The notetaker summary was accurate and read as though the design had been settled. Read against the transcript, three things were missing. A ticketed piece of build work had lost its scope entirely, because the client moved a check upstream mid-call and the summary carried both statements as topics rather than as a reversal. A static input file named in the spec turned out not to exist, traced live during the call to an unverified months-old generated artefact, which is a negative finding and therefore a sentence about an absence. And four items the pre-call register listed as open were never raised at all, which is a non-event and had no text to summarize. The most consequential output of that call was the list of things that did not get decided, and it was the one thing no summary could contain.
An agent writing its own completion receipt. Our engineering loop requires a session to record, for each place its work touched, either the file it updated or an explicit reason it did not. Written from memory of what the session did, that receipt is the one check in the system that can pass while being entirely false, and it fails in the direction that hides the gap. A script can fail midway while the shell sails on, so a receipt line appended before the edit landed records a write that never happened.
A status claim verified against our own prose. Before asserting a document was outstanding, someone confirmed it against the record that summarized the thread, rather than against the sent mail. A record’s prose is a compiled layer. A working summary written earlier in the same session is a second compilation. Status flags are exactly what degrades across compilations, and the confident wrong answer was one hop away from the mailbox that would have settled it.
Five mechanisms that keep the loss away from decisions
None of these are clever. They are all a version of the same rule: put the raw artifact between the compiled one and the decision.
Read raw at the decision point, and use the summary as an index. A summary is genuinely good at telling you where in ninety minutes something happened. It is not a substitute for what was said there. The shape we use:
def decision_context(meeting_id: str, topics: set[str]) -> str:
"""The summary locates. The transcript grounds."""
summary = notetaker.summary(meeting_id)
spans = [s.timestamp for s in summary.sections
if s.topic in topics]
return transcript.slice(meeting_id, spans) # the raw readPin the immutable source in the record. Every compiled record we write carries the original it was compiled from, so any later agent can reopen it rather than trusting the prose. Re-reading a note and rewriting it is not re-verification; reopening the source is.
---
type: delivery-note
slug: design-call-scope-change
# `source` points at the immutable original, never at a summary
source: fathom://recording/<id>#t=41m20s
last_confirmed: 2026-08-03
---Verify receipts against the file, never against recall. The question is not “did I write it down”, it is “does the file contain it”. That is one command, and it is the difference between a receipt and a claim:
# Fails loudly instead of confirming a write that never happened.
grep -q "compilation loss" .learnings/inbox/2026-08-03.md \
|| { echo "receipt would be false: not written"; exit 1; }Diff the call against the register, and give the non-events their own heading. Nothing generates a record of what was never discussed, so it has to be computed as a set difference against whatever listed the open items beforehand. This is the mechanism that recovers the class of fact no summarizer can reach:
open_items: set[str] = load_register("decision-agenda.md")
decided = {d.item for d in call_record.decisions}
never_raised = sorted(open_items - decided) # write these downThen write them into the record under a heading that says they are open, not merely omit them. An item left out of a record reads as closed to the next person, and the failure this prevents is quiet: a call that feels conclusive lets a blocking dependency sit unowned until the sprint that needs it.
Confirm status claims in the channel of record, in both directions. Before asserting a document is outstanding or that it was already delivered, check the sent folder or the thread, not a record summarizing it and not your own earlier summary. And treat a negative asymmetrically. Re-running the same query is repetition, not corroboration: two identical reads of one source are one piece of evidence. If you are about to assert that something did not happen, reach for a structurally different read path, and if none exists, say you cannot see it rather than stating the absence as fact.
What this costs, honestly
Raw is expensive. A ninety minute transcript is tens of thousands of tokens against a summary’s few hundred, it is slower to pull, and pushing every original into an agent’s context is exactly the failure mode that makes long runs degrade in the first place. The rule is not “never summarize”. Compilation is what makes a corpus usable, and an agent that reads everything raw runs out of room before it runs out of task.
The rule is narrower and cheaper than that: never let a summary be the last artifact before a decision. Summarize freely for navigation, for triage, for knowing where to look. Then pay for the raw read on the small number of moments where being wrong costs something, which in practice is a handful per week, not per hour. That makes this a retrieval discipline rather than a context-size problem, which is the same conclusion we reached about the four levers of context engineering: the question is never how much context you can hold, it is what you go and fetch at the moment it matters.
The takeaway
A compiled artifact is a claim about a source, not the source. It drops reversals, negatives, and non-events by construction, it passes review because accuracy is easy to check and completeness is not, and the loss compounds silently at every hop between the original and the decision. Read raw where the decision is made, pin the source in the record, and verify what you wrote against the file rather than your memory of writing it.
This is the same failure we wrote about in stop asking the LLM how confident it is, one level up. There the point was that a model’s report of how sure it is is not a measurement. Here it is that a model’s report of what happened is not a record. In both cases the fix is the same shape: stop accepting the system’s account of itself, and go and measure the thing.
Sources
- Anthropic, “Effective context engineering for AI agents”: compaction is lossy, and the loss surfaces later
- Factory.ai, “Evaluating Context Compression for AI Agents”: three production compressors benchmarked across 36,611 messages
- PLOS Digital Health, “Evaluating large language models for drafting emergency department encounter summaries”: 10% inaccurate, 47% omitting clinically relevant information
- Michael Kitces, “The Risks Of AI Meeting Notetakers: Evaluating Accuracy”: near-perfect transcription, 62% to 87% on action items
- MIT News, “Study shows vision-language models can’t handle queries with negation words”: affirmation bias, and a 25% retrieval drop on negated queries