AI Systems Need an Evidence Layer

The records, review screen, permission checks, and action receipts that turn AI output into something an operator can inspect and safely use.

An Evidence Layer is not a dashboard, a confidence score, or a collection of model logs.

It is the part of an AI product that stores the source behind a claim, shows that source to the person responsible, records the exact action they approved, and blocks every action they did not approve.

In practical terms, a product builder has to create four things:

1. Evidence records that keep source material, observations, inferences, gaps, and proposed actions separate.
2. A review screen where an operator can inspect the source, correct the interpretation, and see the consequence before approving it.
3. Permission checks that prevent an unapproved or changed proposal from executing.
4. Action receipts that show what happened, where it happened, and whether it can be reversed.

If those four things do not exist, the product may display evidence. It does not yet have an Evidence Layer.

This is what the complete path looks like:

The arrows are not decoration. Each one represents a product rule about what may move forward.

Start with one ordinary operator decision

Consider a voice AI product that listens to a sales call and detects a follow-up.

The model returns:

Asha will send the revised commercial proposal on Friday.

That sentence looks usable. A conventional product may immediately create a task in the customer account. An Evidence Layer first creates a candidate.

The candidate contains:

• the retained recording;
• the exact audio span from 18:42 to 18:57;
• the transcript text in that span;
• the system’s interpretation that this is a commitment;
• the proposed owner and due date;
• the fact that speaker identity is not yet verified; and
• the proposed consequence: create one follow-up task in account 482.

Nothing has been written to the customer account yet.

The operator opens the candidate, listens to the fifteen-second span, confirms who spoke, corrects the due date if needed, and approves one named action. The product then creates the task and stores the returned task ID. If the task creation fails, the candidate stays approved but visibly unexecuted. If the operator changes the proposal after approval, the approval becomes invalid and the action locks again.

That sequence is the Evidence Layer in use.

What the evidence record contains

The underlying record does not need to be elaborate. It needs to preserve the distinctions the product depends on.

Here is a deliberately small example. The values are illustrative, but the shape is usable:

{
  "evidence_item_id": "ev_1042",
  "source": {
    "source_id": "recording_781",
    "kind": "recording",
    "retained_at": "recordings/recording_781",
    "checksum": "sha256:...",
    "captured_at": "2026-09-04T11:30:00Z",
    "availability": "retained"
  },
  "observation": {
    "start_ms": 1122000,
    "end_ms": 1137000,
    "text": "I will send the revised commercial proposal on Friday.",
    "speaker_label": "speaker_2"
  },
  "claim": {
    "state": "inferred",
    "type": "commitment",
    "text": "Asha will send the revised commercial proposal on Friday.",
    "derived_from": ["recording_781:1122000-1137000"],
    "derivation_version": "commitment-extractor:12"
  },
  "gaps": [
    {
      "code": "speaker_identity_unverified",
      "blocks": ["create_follow_up"]
    }
  ],
  "proposal": {
    "action": "create_follow_up",
    "target": "crm/account_482",
    "payload": {
      "owner": "asha",
      "due_date": "2026-09-11",
      "summary": "Send revised commercial proposal"
    }
  },
  "approval": null,
  "execution": {
    "state": "locked",
    "reason": "speaker_identity_unverified"
  }
}

The important part is not the field names. It is what cannot be collapsed:

• The observation is the retained source span.
• The claim is the system’s interpretation of that span.
• The gap states what is unresolved and which action it blocks.
• The proposal names the exact external change the product wants to make.
• The approval is empty until a person accepts that exact proposal.
• The execution remains locked while a blocking gap exists.

A single confidence: 0.87 field cannot replace any of these.

What the operator sees

The review screen should mirror the evidence record. It should not make the operator reconstruct the chain from an activity log.

A useful screen has three visible regions.

1. Source

The left side shows the retained recording or document at the cited span. The operator can play the audio, read the surrounding transcript, or open the original page. The product states whether the source is complete, partial, stale, or unavailable.

2. Interpretation

The middle shows the candidate claim. Every inferred field is editable or explicitly read-only. Gaps and contradictions appear beside the affected field, not in a separate diagnostics page.

For the example above, the owner field would say speaker identity needs confirmation. The operator can choose Asha, select another person, or leave the field unresolved.

3. Consequence

The right side says exactly what approval will do:

Create one follow-up in CRM account 482. Assign it to Asha. Set the due date to 11 September. Do not send a message.

The available actions are equally literal:

• Reject candidate: retain the evidence and record the rejection. Make no external change.
• Save correction: update the candidate. Invalidate any earlier approval. Make no external change.
• Approve and create follow-up: authorize only the displayed CRM write.

“Approve” by itself is too vague. The button must name the consequence.

What happens after approval

Approval is a versioned permission, not a boolean.

At minimum, store:

Approval field	Why it exists
Reviewer	Identifies who accepted responsibility
Reviewed evidence version	Proves which source and interpretation were visible
Proposal hash	Binds approval to the exact target and payload
Permitted action	Prevents approval from expanding to another operation
Approved at	Establishes the decision time
Expiry, if needed	Stops old approval being used after the context changes

The execution check can remain simple:

run the action only when:

  approval exists
  AND approval permits this action
  AND approval proposal hash matches the current proposal
  AND no blocking gap remains
  AND the target still belongs to the correct customer or workspace

If any condition fails, return a visible reason and keep the action locked.

This makes several dangerous behaviors structurally impossible. Editing a due date after approval cannot silently reuse the old permission. Approving a CRM task cannot also authorize an email. Moving the candidate to another customer account changes the target and invalidates the approval.

The action produces another piece of evidence

The chain does not end when the API call starts.

After execution, append a receipt:

{
  "action_receipt_id": "receipt_902",
  "evidence_item_id": "ev_1042",
  "action": "create_follow_up",
  "target": "crm/account_482",
  "requested_at": "2026-09-05T09:12:10Z",
  "completed_at": "2026-09-05T09:12:11Z",
  "state": "completed",
  "provider_result_id": "task_5519",
  "reversible": true,
  "reversal_action": "delete_follow_up"
}

The operator sees Follow-up created with a link to task 5519. If the provider times out, the product shows Result unknown and reconciles before retrying. It does not create a second task because the first response was missing.

The source, candidate, approval, and receipt remain connected. Deleting or expiring the raw source should change the evidence item’s availability state rather than pretending the source is still inspectable.

Six product components are enough to start

You do not need a new platform before shipping this behavior. Add the minimum components around one consequential AI action.

1. Source reference. Store the canonical ID, retained location, source time, checksum, and availability. Refuse to present a claim as inspectable when its source cannot be found.

2. Observation. Store the exact page, quote, timestamp, image region, or event. Do not copy the model’s summary into this field.

3. Claim. Store the interpretation, its supporting observations, and the derivation version. Keep its state inferred until the product has a separate reason to promote it.

4. Gap. Store each missing, contradictory, stale, or unverified condition and the operations it blocks. Do not bury known weaknesses inside one score.

5. Approval. Store the reviewer, permitted action, proposal version, and time. Invalidate it when the proposal changes.

6. Action receipt. Store the attempt, result, external object ID, failure state, and reversal path. Do not treat an API request as a completed customer outcome.

These may be six tables, six document types, or one append-only event stream. That is an engineering choice. Their meanings and transitions are product choices.

Build it in three passes

Pass 1: Make the result inspectable

Retain the source. Store exact observations separately from claims. Add a read-only review page that lets an operator move from a claim back to its supporting span.

Do not add external actions yet. The visible result is a candidate with evidence, gaps, and a review state.

Pass 2: Add one scoped approval

Choose one action whose consequence is easy to describe and reverse, such as creating a draft or internal follow-up. Show the exact target and payload. Bind approval to that version of the proposal.

Any edit after approval must lock the action again.

Pass 3: Add execution and recovery

Store an idempotency key before calling the external system. Record the returned external ID. Reconcile unknown results before retrying. Give the operator a link to the changed object and a reversal path where one exists.

Only then add the next action type.

Tests that prove the boundary

The Evidence Layer is working when these tests pass:

1. Remove the retained source. The claim becomes unavailable for approval.
2. Change an inferred field after approval. The action locks again.
3. Approve a CRM follow-up. Attempt to send an email with that approval. The system refuses.
4. Leave speaker identity unresolved. The follow-up remains a candidate.
5. Make the provider return an unknown result. The product reconciles before retrying.
6. Complete the action. The operator can open the receipt and reach the external object.
7. Reject the candidate. No external state changes, but the source and rejection remain available for review.

These are product acceptance tests. Model quality tests and agent traces still matter, but they answer different questions. A trace tells the builder how the machine produced an output. The Evidence Layer tells the operator why this claim is present and what they are being asked to permit.

What this costs

An Evidence Layer creates more visible unfinished work. Candidates remain unresolved. Missing sources stay missing. Approval takes time. Source retention costs money. External actions need reconciliation and reversal paths.

That is the tradeoff.

Without the layer, the product looks faster because it hides the review work and stores inference as truth. The cost appears later as duplicate tasks, incorrect customer memory, untraceable decisions, and operators who stop trusting the system.

In a controlled transcript-recovery study, 74 of 86 comparable records improved in raw coverage, 12 became worse, and none proved complete recovery. The practical Evidence Layer decision was not “the new method scores better.” It was: retain the candidates, show the differences, and block their promotion into customer truth. I describe that release decision in We Refused a Better Transcription Result.

That is the standard I now use: an AI result is not ready because the model produced it or because an evaluation score improved. It is ready for use when the responsible person can inspect its source, understand its gaps, approve a specific consequence, and see what happened next.