Agents Need a Source Supply Chain, Not Just Search
An agent that found a good source once does not have a dependable research system.
It has a successful search result.
The distinction became unavoidable while building SourceFoundry, a service that keeps external sources fresh for agents and applications. Search quality mattered. The harder product problems appeared after the first useful result: when to fetch again, how to recognize the same story from several places, which URL is canonical, whether the source is becoming unreliable, how much provider work is allowed and what the consuming agent should do when supply fails.
I call the complete operating path the Source Supply Chain. It turns external information into recurring, inspectable input rather than a pile of links that happened to rank well today.
Each stage removes a different reason for an agent to be confidently stale.
Search answers a query. Supply maintains a promise.
A search API is optimized for the request in front of it. An application usually has a continuing obligation: keep track of this topic, monitor these sources, surface material changes and say when the stream becomes unreliable.
| Search request | Source supply |
|---|---|
| Begins with a query | Begins with a standing information need |
| Returns ranked results | Maintains a versioned stream |
| Success is a response | Success includes freshness over time |
| Duplicate links are inconvenient | Duplicate events corrupt downstream judgment |
| Provider errors affect one request | Silent failures make the product stale |
| The caller manages retry and spend | Policy bounds retry, frequency and cost |
This is why a generic /search endpoint was the wrong primary shape for SourceFoundry. The consuming agent should be able to describe feeds, searches and source policy. The service should own schedules, provider lanes, retries, normalization, deduplication, provenance and health.
The agent’s job begins after that. It interprets, ranks, writes or proposes an action.
Provider output is not a product contract
Different discovery providers return different fields, scores and notions of relevance. Passing those responses directly to every consuming application couples product behaviour to the current vendor.
We introduced a neutral signal contract containing the things a consumer actually needs: canonical URL, source identity, provider, query when available, content hash, fetched and published time, age, completeness, source reliability and consecutive failure state.
The contract does two jobs. It makes providers replaceable, and it stops downstream agents from reconstructing missing evidence differently.
If publication time is absent, it stays absent. If content is partial, the item says so. If a source is failing repeatedly, the consumer sees degradation rather than receiving an old result with a fresh wrapper.
This is vendor neutrality at the product boundary. The promise belongs to SourceFoundry, not to the provider it used for one fetch.
Deduplication is an epistemic problem
Three articles can report one event. Ten posts can repeat the same claim. Counting each URL as independent support inflates apparent confidence.
SourceFoundry groups candidates under a stable signal identity and retains a bounded set of observations. One source contributes at most one observation to a bundle. Repetition does not increase the representative score by itself.
That rule is conservative by design.
An agent needs to distinguish corroboration from syndication. Independent reporting with separate evidence may strengthen a claim. Copies of the same press release do not. URL-level deduplication helps, but source identity, content hashes, canonicalization and provenance are what make the bundle interpretable.
This is where the Source Supply Chain feeds the Evidence Layer. The agent receives not only content, but the information needed to judge what kind of support it represents.
Freshness is a state, not a timestamp
A recent fetched_at value can coexist with an old article, a partial response or a source that returned nothing useful. Freshness needs several clocks:
- when the source says the item was published;
- when the system first observed it;
- when it last fetched the source successfully;
- how old the returned content is;
- how many consecutive attempts failed.
These fields let the consumer behave differently. A breaking-news product may reject an item after hours. A research archive may accept years-old material if the canonical source remains available. A monitoring agent may alert on source failure before it alerts on content.
The policy belongs to the consumer. The facts about freshness belong to the supply layer.
Safe collection needs operating limits
An autonomous source service can create security and cost problems before an LLM becomes involved.
A supplied URL may resolve to a private network address. Redirects can leave the approved origin. A response can be unexpectedly large. A recurring schedule can become a spend loop. Several workers can claim the same job. A failed job can remain active forever.
We added bounded fetch behaviour, tenant-scoped credentials, response limits, durable jobs, duplicate-active-job prevention and dry-run reconciliation. Provider-backed collection is capped by source count, frequency and run budget. Paid enrichment is rejected before its first request when the maximum batch cost exceeds policy.
These controls are not the differentiated product on their own. They are what make the differentiated promise credible.
Cloudflare’s durable workflow model expresses a related runtime principle: each step can be retried independently and execution resumes from a successful checkpoint after interruption. Their guide shows the mechanism clearly. A source product still has to define idempotency, canonical identity and what a retry is allowed to spend.
Failure should travel with the data
The easiest interface returns an empty list when nothing new was found. That hides several meanings:
| Empty result | Required response |
|---|---|
| Source was checked; nothing matched | Healthy, zero new items |
| Provider rejected the request | Explicit provider failure |
| Source violated safety policy | Rejected with reason |
| Fetch timed out | Partial or failed attempt |
| Worker never ran | Stale schedule state |
An agent cannot make a responsible decision if all five look identical.
Visible failure is part of the product output. It lets a consuming system lower confidence, use another source, ask for help or decline to make a claim. Silence pushes the failure downstream, where it appears as poor model judgment instead of missing supply.
What I got wrong
I initially described SourceFoundry as an evidence request API. That emphasized the most legible demo: ask for information and receive results. The service already contained recurring feeds, durable jobs and source health. I had positioned the product around one request instead of the continuing job it performed.
The better promise was already in the system: keep a bounded set of sources useful over time and expose when that promise breaks.
This also changes how I think about RAG. Validation still matters, but validation begins before retrieval. If the supply layer has lost provenance, confused repetition with corroboration or gone stale silently, no reranker can reconstruct the missing truth.
The remaining question is ownership. When an agent bases a decision on a degraded source stream, should the source service block delivery, should the consuming product lower confidence, or should the user choose? A neutral supply layer can expose the facts. Someone still has to own the consequence.