← Back to the Shed
Part 1 of 3 · Part 2: The Intervention · Part 3: The Weaponisation

AI Summarisation Mechanics: What Gets Lost and Why

Research Note |

The Core Misunderstanding

Most people assume AI summarisation works like human summarisation: read the text, identify key points, condense.

This is wrong.

AI summarisation is generating new text, not pulling out key sentences. The model:

  1. Reads the input and compresses it into an internal representation
  2. Generates new text that pattern-matches to "summary-like" outputs based on its training
  3. Produces something coherent that captures general themes

The summary is written fresh, not copied out. This distinction matters because generated text can:

It Looks Like Copying, But It Isn't

When you read an AI summary, you'll often see phrases that sound exactly like the original document. It's natural to assume these were pulled out directly — like highlighting key sentences.

They weren't. The AI regenerated them.

Sometimes the regenerated text matches the original word-for-word. Sometimes it's subtly different — a word changed, a qualifier dropped, a meaning shifted slightly. The problem is you can't tell which is which just by reading.

A sentence that looks like a direct quote might have been altered. A sentence that sounds freshly written might happen to match the source exactly. There's no way to know without checking against the original — which defeats the purpose of having a summary.

This makes the trust problem worse, not better. You can't treat familiar-sounding phrases as reliable quotes. Everything in an AI summary is generated text — even the bits that look copied.

The Mechanism: Tokens and Vectors

From Words to Numbers

Words don't enter the model as text. Each token (word or word-fragment) is immediately converted into a vector — a list of thousands of numbers representing position in high-dimensional space.

"Cat" isn't stored as letters. It's stored as something like [0.234, -0.871, 0.125, 0.456, ...] extending across 4,096 or more dimensions.

These numbers encode relationships learned from training data. Words that appear in similar contexts cluster together in vector space. "Cat" and "dog" are close; "cat" and "democracy" are far apart.

Attention: What Gets Weight

The attention mechanism determines how tokens influence each other. For each token, the model:

  1. Creates a Query vector (what am I looking for?)
  2. Computes similarity against all other tokens' Key vectors
  3. Uses those similarity scores to create weighted averages of Value vectors
  4. Updates the token's representation based on this weighted combination

When "bank" appears near "river," the attention mechanism assigns high weight to "river," pulling "bank" toward the riverbank meaning in vector space.

Key insight: This isn't information being passed in packets. It's vectors being pulled through high-dimensional space by gravitational influence from contextually relevant tokens. Subtle signals can be overwhelmed by stronger pulls.

See it visually → Interactive token and vector diagram

What Gets Lost: Systematic Biases

The Lost in the Middle Problem

Research by Liu et al. (2024) demonstrated that LLM performance degrades significantly based on where information appears in input text.

Key finding: Performance is highest when relevant information appears at the beginning or end of context, and significantly degrades when information is in the middle — even for models explicitly designed for long contexts.

Implication: If your key qualification appears on page 4 of a 10-page document, it's more likely to be lost than content on pages 1 or 10.

The Overgeneralisation Bias

Research by Peters & Chin-Yee (2025) tested 10 prominent LLMs on 4,900 scientific text summarisations.

Key finding: LLM summaries were nearly five times more likely to contain broad generalisations than human-authored summaries. Even when explicitly prompted for accuracy, models overgeneralised in 26-73% of cases.

Paradox: Prompting for accuracy made it worse. Summaries retrieved with accuracy prompts were about twice as likely to contain generalised conclusions.

The Drama Bias

Research published in PNAS (2023) found that LLMs show biases for content that is gender-stereotype-consistent, social, negative, threat-related, and biologically counterintuitive.

The soap opera effect: The mathematics is rediscovering what TV commissioners have known for decades — drama holds attention. A visit to the library doesn't make the cut.

Mechanism: These biases exist in training data because humans write more about dramatic things. The attention economy becomes encoded in vector space.

The Thermostat Analogy

Default AI interaction is like a thermostat set to 21°C. The model produces serviceable, predictable outputs. Nothing wrong with it, but nothing emerges either.

Temperature (a technical parameter) controls how the model samples from probability distributions:

Engagement changes temperature metaphorically: Pushing back, asking follow-ups, taking ideas in unexpected directions — these reshape the probability landscape the model samples from. The "profound" outputs that sometimes emerge aren't the model being clever; they're interaction effects between human input and probability sampling.

Why This Happens: Potential Landscapes

LLMs don't learn rules; they learn to navigate an implicit "energy landscape" where generation follows gradients toward states that feel closer to completion.

For summarisation:

Token pressure compounds this: Unlike humans, LLMs cannot pause, reflect, or choose not to generate. They must produce tokens. If the "honest uncertainty" well isn't deep enough, the "plausible completion" well wins.

This explains why qualifiers get dropped (they add complexity without improving coherence-feel), confidence increases (confident tone feels more "complete"), and drama gets emphasised (dramatic content has stronger training signal).

The Monoculture Problem

There's a closed feedback loop forming:

  1. LLMs trained on human output
  2. Humans increasingly rely on LLM output
  3. LLMs retrained on the mixture
  4. System converges toward shrinking basin of possibilities

Summaries amplify this convergence. If millions of people read AI summaries rather than original documents, and those summaries systematically emphasise drama over nuance, drop qualifiers and limitations, generalise beyond source material, and favour beginning/end over middle — then the "level of consciousness" processing information narrows. The same biases get reinforced. Alternative framings become less probable.

Practical Implications

For Users of AI Summaries

  1. Never trust a summary for high-stakes decisions. The mechanism actively works against preserving caveats and limitations.
  2. Check the middle. If something seems missing, it might be — positional bias loses content from the middle of documents.
  3. Treat summaries as orientation, not replacement. They're useful for knowing roughly what something is about. They're not substitutes for reading source material.
  4. Notice the confidence. AI summaries tend to be more confident than sources warrant. If it sounds definitive, check whether the source was actually that certain.

For Organisations

  1. Don't use AI summaries in decision chains without human verification. This is an open-loop failure waiting to happen.
  2. Structure documents with position bias in mind. Key qualifications and limitations should appear early and late, not buried in the middle.
  3. Establish verification protocols. Someone needs to check whether summaries accurately represent source nuance.

For Researchers

  1. The "lost in the middle" problem may not be fixable with prompting. It's architectural, not behavioural.
  2. Accuracy prompts can backfire. They may create pressure toward authoritative-sounding output.
  3. Human-authored summaries remain the benchmark. They're five times less likely to overgeneralise.

The Boardroom Question

If presenting this to a board, start here:

Do you use AI summaries? Do you trust them? Do you know how they actually work?

Then walk through:

  1. It's not reading and condensing — it's generating new text from compressed representations
  2. Position matters — content in the middle of documents is systematically lost
  3. Drama wins — the maths discovers what TV commissioners know; the nuanced qualifier doesn't make the cut
  4. Qualifiers vanish — summaries are five times more likely to overgeneralise than human-written versions
  5. Confidence inflates — the mechanism favours completion over uncertainty

The implication: When you read an AI summary of a report, the nuanced caveats are probably gone and the confident claims are probably emphasised. That's not a bug in this particular AI — it's how the mechanism works.

Key Sources

Questions or corrections?

keiron@curiosityshed.co.uk

Revision Note (13 January 2026)

The original version of this article explained the mechanism of AI summarisation but didn't address the intervention point — what users can do about it.

This gap was identified through peer review. The feedback: "You've explained the problem, but you haven't told people what they can do about it."

That was right. Explaining mechanism without intervention creates learned helplessness. Understanding how something works should lead to understanding how to work with it.

The companion article "Influencing Token Attention Through Prompting" now addresses this gap, explaining the three mechanisms of influence: explicit weighting instructions, context priming, and relationship override.

Keiron Northmore & Claude | January 2026