Who is speaking?

We recently compared what a long-running public commentator writes with what he says on air. The written side was easy to pin down: about 800 signed columns over four decades, each with a date and a byline. The spoken side was several hundred recorded talks: interviews, panels, lectures and podcast episodes, most of them available only as video-platform captions, a few dozen as podcast audio. Before we could ask whether a spoken claim contradicted a written one, we had to answer a duller question for every one of more than twelve thousand spoken claims: did he say it?

This note describes how we answered it, what we measured, and where the method still fails. It says nothing about the commentator’s views.

The problem

A quote in a transcript is not a quote by a person. Three things get in the way.

Captions have no speakers. Automatic captions are short timed lines with no speaker labels, and one line can hold the end of a question and the start of an answer. A host’s summary (“so you are saying that…”) reads exactly like the guest’s own words.

Interviews hand over mid-sentence. The guest starts a sentence, the host completes it, the guest says “exactly” and moves on. On panels, other guests address him by name and then make their own point.

Speech recognition mishears exactly what matters. Names and figures are the tokens a fact-check depends on and the tokens recognisers get wrong most often. In our captions we saw a surname turned into a different surname, an acronym turned into an ordinary word, and a duration in years heard as hours. A claim that rests on one number can be an artefact of transcription.

Each error produces a false “inconsistency”: someone else’s sentence, or a word nobody said. So attribution came first, as a separate stage with its own checks.

How the work is organised

The study ran on our agent pipeline. Deterministic code does every mechanical step (fetching, hashing, transcription, timing, matching), and a swarm of language-model agents does the reading. Each agent has one narrow role and written instructions:

• screener: a quick triage of each transcript (is he speaking, or is this a re-upload, a translation, a talk about him?);
• reader: assigns every line to a speaker and extracts claims from his turns only;
• finder: given spoken and written claims on one group of subjects, proposes candidate inconsistencies;
• skeptic: tries to refute each candidate, re-reading the transcript around the quote;
• re-sourcer, prediction checker and final reviewer: check the survivors against captured public sources, against events, and once more before anything is shown.

Work is cut into batches sized for one agent (a reading batch is about 170,000 characters of transcript), so agents run in parallel. The pilot took 27 agent tasks. After it, the full study took nearly 300: 116 reading batches, 67 finder and 67 skeptic bundles, and 47 batches of re-sourcing, prediction checking and final review.

One design rule matters more than the rest: no agent is trusted on its own word. Agents propose and code verifies. Quotes are matched word for word, turn boundaries are enforced by code, hashes are recomputed, and every finding is challenged by a different agent with the opposite brief. Speaker attribution is the clearest case. The reading agent’s judgement is one of two independent signals, and the other involves no language model at all.

The method

We used two checks that do not share information: one reads the conversation, the other listens to the voice. A spoken claim is kept only when both agree that it is his. Where only one check is possible, we say so on the claim.

Check 1: context

A reading agent goes through each talk as numbered, timed lines and assigns every line to a turn: subject, host, other (another guest, a caller, a clip, named as introduced) or unclear. Every line must be covered. For podcasts, lines are cut at sentence ends or at pauses of 1.5 seconds or more, so that a change of speaker tends to fall between lines rather than inside one; for videos, one line is one caption.

The rule is strict. A turn is his only with a cue in the transcript: a question addressed to him, his name, an introduction, or first-person facts only he could state. Content that merely sounds like his views is not enough. Someone addressing him by name opens someone else’s turn; on panels, any stretch that could be another panellist’s is unclear; a line holding two speakers stays unclear, even if that loses a quote. Garbled stretches yield no claims.

Claims come only from his turns, and each quote is checked mechanically: found word for word in the lines, every line inside one of his turns. Otherwise it is set aside automatically.

Check 2: voice

Where we hold the audio (the podcasts), we also ask who is speaking by voice alone. Everything runs locally on CPU, with no external service:

• The episode’s audio is decoded to 16 kHz mono.
• The recogniser’s word timestamps are grouped into chunks of about three seconds, with a new chunk at any pause over 0.6 seconds; chunks shorter than one second are dropped.
• Each chunk gets a speaker embedding from an open-source model (SpeechBrain’s ECAPA-TDNN trained on VoxCeleb).
• Chunks are clustered (agglomerative clustering, cosine distance, average linkage, distance threshold 0.75). Each cluster is a candidate speaker.
• A reference voiceprint is the mean embedding of a 25-second stretch that is his beyond doubt by context: he answers in the first person about his own past work while the host is silent.
• The cluster whose centroid is closest to the reference is labelled as him if the cosine similarity is at least 0.45; all others are labelled “other”.
• A quote’s voice verdict is the share of its seconds that fall in his cluster: 0.6 or more is “him”, 0.4 or less is “not him”, anything between is “unsure”.

The two never overlapped, though the margin is narrow at the bottom. Across the roughly thirty episodes with a voice file, spanning many years of recordings and different studios, his cluster scored between 0.47 and 0.94 against the reference. The closest any other substantial speaker came (at least 20 seconds of speech) was 0.40, and the next closest 0.30.

The rule

• Podcasts: kept only if context says him and voice says him. Any disagreement, and any “unsure”, sets the claim aside.
• Videos: we did not download video audio, so the voice check is not available. These claims are kept on context alone and carry the label “context only” wherever they appear.

Setting aside costs little: a misattribution, once published, is hard to undo.

What we measured

We ran a pilot of 22 talks (about 15 hours; 16 videos and 6 podcasts), audited it, adjusted the reading instructions, and then ran the method on every talk we had found: several hundred, over four hundred of them videos and about thirty podcast episodes, close to 190 hours in all.

Three results matter most.

On the pilot, the two checks never contradicted each other. On 290 podcast claims the voice check either confirmed the reader’s attribution or was unsure (5 cases, set aside). It never pointed to another speaker.

On the full run, they did, rarely. Of 979 podcast claims that passed the context check, the voice check said “not him” for 14 (about 1.4%) and was unsure for 11. We set all 25 aside without trying to adjudicate them. Several rest on very little audio: six of the 25 had one second or less of voiced speech under the quote. They are short replies, the kind of turn that context reading handles worst and voice embedding also handles badly.

Context alone held up on video, with one caveat. An independent reader re-judged 60 randomly drawn video claims from the pilot: 57 his, none someone else’s, 3 impossible to decide. All three undecidable claims sat at a hand-over between speakers. Outside the sample, a skeptic agent re-reading a finding caught one misattribution: a panellist’s sentence, said to him by name, had been read as his. Later review set aside a few more video claims under the strict rule, mostly stretches given to him by vocabulary or theme without a cue.

Where it fails

• Hand-overs. Every undecidable case in the audit, and the one confirmed misattribution, sat where the floor changed. A short interjection merged into a caption line is invisible in text. The rule “a line with two speakers stays unclear” helps only if the reader notices that there are two speakers.
• Overlap and short answers. One-second embeddings are noisy, and overlapping speech fits neither cluster cleanly. A one-word answer gives the voice check almost nothing and is easy for a reader to place on the wrong side. The voice check cannot rescue the context check’s weakest cases; it can only flag them.
• Caption errors. Any quote containing a number or a name is flagged for a listening check; about 39% of video claims are. For podcasts, a second recogniser (Whisper large-v3) transcribes every part independently of the first (Qwen3-ASR-1.7B), and each quote gets the share of its words both heard. The median is 1.0; about 4% of scored claims fall below 0.8.
• Silent recogniser failures. A 32-minute segment once came back with under 18 minutes transcribed. We now cut audio into 10-minute parts, detect output that stops short or repeats one word twelve or more times, and re-send such parts in halves.

For any quote a published finding rests on, we go back to the audio. The quote’s span plus three seconds either side goes to both recognisers again. In one case both models, hearing only the clip, agreed on a different figure from the one in the full transcript; the finding was corrected before publication. For videos there is no stored audio, so a finding resting on a caption carries a link to the exact second and the label “listen at link”. Video claims are checked less well than podcast claims, and every one says so.

Provenance

Every kept claim carries enough to be re-checked by someone who does not trust us:

• the talk, the first and last transcript line it spans, and its start and end time to the second (word-level for podcasts);
• a link to that second for videos, and minute and second within the episode for podcasts;
• the SHA-256 of the raw transcription response it came from and, for podcasts, of the audio file itself;
• which speaker checks it passed (“both” or “context only”), its voice share, and whether it needs a listening check.

Raw service responses are stored under their SHA-256 with URL and retrieval date, and claim identifiers stay stable across re-runs. Before any finding is shown, a script re-verifies it mechanically: the quote is found word for word in the stated lines, the link’s time matches the start of those lines, and the stored transcript and audio files still hash to the recorded values.

What we would change next

• Voice on video, where licences allow. Most claims here rest on context alone; a voice check on video audio would show how often context is wrong at scale.
• Repeat the random audit on the full run. Our only audit covers 60 pilot claims. A stratified sample, weighted towards panels, would give an error rate with an interval.
• More than one reference voiceprint. Ours is one 25-second stretch from one recent recording. Separation held across years, but voices age; references from several periods would remove a single point of failure.
• Treat hand-overs as a class. Every hard case sat within seconds of a speaker change. Sending only quotes near a turn boundary to a second reader would put human effort where the errors are.
• Decide the 25 instead of dropping them. Listening to each would tell us whether voice or context was wrong, the one number this note cannot yet give.

Little in the attribution stage is specific to one person: mainly the reference voiceprint and the cues that identify him in a conversation. Any comparison of what someone says with what they write, or with what they said before, depends on getting the speaker right, and the errors that come from getting it wrong look exactly like findings.