We Improved 74 of 86 Transcripts and Still Refused to Ship
Our proposed transcript-recovery change improved raw coverage in 74 of 86 comparable recordings. We still refused to ship it.
That result sounds irrational if the release question is, “Did the average get better?” It becomes straightforward when the product may turn recovered words into findings, memories and actions.
The study exposed what I call the Coverage Trap: a system can recover more content overall while becoming less trustworthy record by record.
This is a build log about the decision, not a public benchmark. The evaluation used a private, controlled set of 100 workplace recordings. Fourteen records did not support a clean comparison, leaving 86 comparable pairs. I am publishing aggregate results and the release method, not customer identity, audio, transcripts, providers or private infrastructure.
The aggregate result looked good
The candidate recovery method produced these results across the 86 comparable records:
| Result | Records | Share of comparable set |
|---|---|---|
| Raw coverage improved | 74 | 86.0% |
| Improvement was material | 57 | 66.3% |
| Coverage regressed | 12 | 14.0% |
| Complete recovery proven | 0 | 0% |
One record lost roughly nine percentage points of coverage and 47 words. That is not a rounding error to the person whose conversation contains those words.
The candidate was useful. It was not safe to promote automatically.
Coverage measures presence, not truth
We were comparing how much time-aligned speech each method recovered from the same recording. Coverage is useful because silence, truncation and missing spans are mechanically detectable. It is not a semantic quality score.
A longer transcript may include:
- speech the original method missed;
- duplicated phrases;
- invented connective language;
- words assigned to the wrong speaker;
- a plausible reconstruction of something nobody said.
This is why “more words” and “better transcript” cannot be treated as synonyms. The candidate first had to show that it recovered a missing span. Then a reviewer had to decide whether the recovered language was faithful enough to use.
The Coverage Trap appears whenever a measurable proxy is easier to improve than the thing a user actually needs. Retrieval systems optimize similarity while the user needs grounded claims. Coding agents optimize completed tasks while the maintainer needs correct changes. Speech systems optimize word error rate while the operator needs the right commitment attached to the right person.
The proxy is not useless. It simply cannot authorize the product decision.
Averages hide exactly the cases that matter
Suppose a change adds 20 useful words to nine recordings and removes 20 decisive words from one. The average movement is strongly positive. The system is also capable of deleting the sentence that changes a business outcome.
That asymmetry changes the release standard.
We evaluated every record rather than declaring victory from the average. That exposed the twelve regressions and the absence of any record for which the method alone proved completeness.
The most important number was zero.
Comparison eligibility was part of the result
Fourteen records did not enter the 86-record comparison. Excluding them was not data cleaning performed to improve the headline. A pair needed enough matching source identity and measurement context for the difference to mean anything.
This matters because evaluation sets acquire false authority quickly. A row in a spreadsheet looks comparable even when one side used a different audio duration, source object or processing boundary. We kept the full 100-record cohort visible, reported the comparable subset and refused to convert missing comparison evidence into a neutral result.
The denominator is part of the claim. “74 improved” without “out of 86 comparable records from a 100-record controlled set” tells a different and much stronger story than the evidence supports.
The system needed two fields, not one verdict
The unsafe design would have stored a recovered transcript and marked it better. The safer design preserved two independent statements:
| Field | Question |
|---|---|
provesCompleteness |
Does the evidence establish that the recovered record is complete? |
authorizesAction |
May downstream systems use it for memory, findings, claims or action? |
A candidate can improve coverage while both fields remain false.
That state is not failure. It is a useful review object. The system can show the original and candidate spans, highlight additions and losses, and ask a person to resolve the disputed portion. What it cannot do is convert “probably better” into “approved truth.”
The distinction follows the Evidence Layer: observed material, model inference, proposed correction and approved record remain separate until responsibility changes hands.
We closed the automatic path
The tempting response was to ship the recovery method for the majority and create an exception path for regressions. That would have been reasonable if the system could reliably identify every dangerous record before promotion.
It could identify coverage movement. It could not prove semantic completeness. The automatic merge path was therefore closed.
This cost us momentum. It preserved the integrity of everything downstream.
Once a recovered transcript becomes canonical, derived systems may extract decisions, assign owners, generate reports, update memory or present a customer-facing claim. Correction becomes more expensive at every step. A small uncertainty near the source turns into confident product behaviour further away.
The right place to stop the chain was the first boundary we could not prove.
The next experiment became narrower
Rejecting automatic fusion did not mean discarding the candidate method. It changed the job we gave it.
Instead of “produce the new transcript,” its role became:
- Identify spans where the canonical transcript appears sparse.
- Produce an alternative only for those spans.
- Show additions and regressions beside the source audio.
- Ask a reviewer to accept, correct or reject the candidate.
- Preserve the review result as evidence.
That is less magical than automatic repair. It is much more useful.
The product question also changed. We stopped asking whether one provider could replace another and started asking where independent methods disagree in ways a human can resolve. Agreement may increase confidence. Disagreement tells us where to spend attention.
What I got wrong
I expected independent transcription to behave like redundant storage: if one copy had a gap, the other would often fill it. Speech does not fail so neatly. Background noise, overlapping voices, short fragments and domain language can cause different systems to omit, distort or invent different material.
Two incomplete witnesses do not automatically produce one complete witness.
The experiment was still successful. It prevented a misleading release, exposed the records that need review and gave us a more honest product boundary. This is what the Release Proof Ladder is supposed to do: make a positive technical result earn its way into a user claim.
The open question is economic. Human review preserves trust, but it cannot expand without limit. The next system has to become good at allocating review attention without pretending that allocation is the same as proving truth.