TL;DR
- An agent that runs out of evidence doesn't stop. It keeps writing in the same confident register. Prompting it to "cite sources" or "flag uncertainty" asks the least reliable part of the loop to police itself.
- REA, an open-source MCP server for reverse engineering, moves that job into the tool contract. Every Evidence record carries a confidence (
observed,derived,inferred) and an authority (what kind of source the fact speaks for), and both are part of the record's identity.- Coverage is three-state:
complete,partialwith mandatory limitations, orunknownwith a mandatory reason. An empty caller list and an unread caller list stop looking the same.- Unknowns are durable records with IDs derived from the question itself. A "verified" resolution cannot exist without at least one Evidence ID behind it.
- Its agent evaluation grades a claim on value, confidence, authority, and whether the cited Evidence ID was actually delivered by a tool in that session. A correct answer the model recalled from memory fails.
- The cost is real: heavier payloads, more schema, and a taxonomy only as honest as the code that assigns it. Worth it where plausible fiction is cheap to produce and expensive to catch. Ceremony for a tool that books meetings.
- The rule to take away: if a missing fact and a negative fact serialise to the same JSON, your tool is lying by omission, and the agent will repeat the lie fluently.
- REA
Reverse engineering is close to the worst possible domain for a language model. Decompiler output is partial by nature. Indirect calls don't resolve, jump tables get misread, and pseudocode is a reconstruction of what a compiler probably did. Hand that to a model, ask how search works in some application, and you get a fluent, well-structured explanation. Part of it was read out of the binary. Part of it is how search usually works. Both parts read the same.
That is why I spent time in REA ("Reverse Engineer Anything"). It's an MCP server and CLI that puts Hopper, Ghidra, IDA, headless JADX, a browser, and a long list of format parsers behind one tool surface. The tool count isn't the interesting part. The interesting part is where the project decided honesty has to be enforced. Not in the prompt. In the types.
The prompt is the wrong layer
The usual answer to an over-confident agent is more instructions. REA has those too. Its packaged agent skill says every conclusion "must distinguish observations, inferences, and unknowns" and must "never imply that static analysis observed execution."
But an instruction is a request made to the component least able to tell what it knows from what it pattern-matched. If a tool returns a bare list of callers, the model has no way to separate "these are all the callers" from "these are the callers the decompiler resolved before it gave up." That information was destroyed before the model ever saw it, and no prompt recovers it.
So the design question moves. What would the tool's output need to look like for the honest answer to be the easy one?
Confidence and authority are fields, and they are part of identity
Every observation REA returns is an Evidence record, validated by a strict Zod schema in src/domain/evidence.ts. Two fields carry most of the weight:
confidence: z.enum(["observed", "derived", "inferred"]),
authority: z.enum([
"shipped-artifact",
"controlled-replay",
"historical-reference",
"external-service",
"analyst-inference",
]),Confidence is how the fact was arrived at. Authority is what kind of source it can speak for: bytes in the shipped artefact, behaviour seen in a controlled replay, a historical reference, a remote service, or somebody's reasoning. The two are deliberately independent. A call edge read statically from a binary is observed, with shipped-artefact authority, and it still says nothing about whether that path ever runs. The skill's rule about static analysis and execution depends on exactly that distinction. Here it's a value in a field rather than a sentence in a prompt.
The detail I like most is how a record gets its ID. evidence_id is a SHA-256 over a canonical "semantic projection" of the record: the subject's digest and format, the provider, operation, parameters, raw and normalised results, confidence, authority, limitations, coverage, locations and links. The file's local path and display name are left out. When a record is parsed, the digest is recomputed and a mismatch is rejected as tampered.
Two consequences follow. The same binary analysed from a different folder yields the same Evidence ID, because identity is the content, not where you happened to find it. And you cannot quietly promote an inference. Change inferred to observed, or drop a limitation, and you have a different record with a different ID. Any claim citing the original no longer points at it.
If that sounds familiar, it's the rule from the previous article, The Deduplication Key That Was Set to Random, applied to knowledge instead of messages. Derive the identifier from what the thing is, and nothing else can be substituted under the same name.
Unknown is not absent
The second decision is about missing information. REA's tool-design guide puts it in five words: "Missing coverage is unknown, not absence."
It's most concrete for caller and callee relationships, which in a decompiler are exactly where the gaps hide. Coverage is a discriminated union, not a boolean, in src/domain/relationshipCoverage.ts:
const completeCoverage = z.strictObject({ status: z.literal("complete") });
const partialCoverage = z.strictObject({
status: z.literal("partial"),
limitations: z.array(z.string().min(1)).min(1),
});
const unknownCoverage = z.strictObject({
status: z.literal("unknown"),
reason: z.string().min(1),
});Look at the .min(1) calls. You cannot claim partial coverage without saying what's missing, and you cannot claim unknown without saying why. Even the strong state is scoped carefully: the guide defines complete as the provider's observed direct relationships returned without a known read limit, and says outright that it "does not claim every possible runtime edge."
An empty array in the naive shape means three different things. No callers exist, none were found, or nobody looked. An agent will choose whichever reading makes the cleanest paragraph. Making the tool say which one it means takes that choice away.
The same thinking extends to questions that outlive a single call. src/domain/residualUnknown.ts models them as durable records with a lifecycle (open, investigating, blocked, contradicted, resolved), a severity, the authority and confidence needed to close them, and recommended probes. The ID is a hash of the normalised question, its domain and its scope, so asking the same question twice lands on the same unknown instead of creating a second one. Revisions are hash-chained to their predecessor. And the schema refuses some convenient lies outright:
- a
contradictedunknown must cite contradicting evidence - only a
resolvedunknown may carry a resolution, and every resolved one must - a
verifiedresolution requires at least one Evidence ID
That's a lot of machinery for "I don't know." It's also the only version of "I don't know" that survives a context window rolling over, or a second agent picking up the work tomorrow.
The obligation ledger REA uses when you rebuild a feature from what it found takes the same stance. A static fact creates a candidate obligation; it never closes one. Closure needs a unique owner, the required test fixtures, and a passing verifier whose authority is comparable to the original observation. The empty example request in its own documentation returns an unknown ledger, because "absence of source Evidence never claims closure."
Grading the agent on provenance
What convinced me this is a design and not just a vocabulary is the evaluation code. src/evaluation/KnownAnswerEvaluation.ts scores an agent's final answer against fixtures with known answers, and every claim must take this shape:
z.strictObject({
id: z.string().min(1),
value: jsonValueSchema,
evidence_id: z.string().regex(/^ev_[a-f0-9]{64}$/u),
confidence: confidenceSchema,
authority: authoritySchema,
})The assessor checks the value, then the claimed confidence and authority, then whether the cited evidence_id is one a tool actually delivered during that session, collected from the session transcript. If it isn't, the failure is unknown_evidence. If the same ID arrived twice with different content, it's conflicting_evidence.
Consider what that rules out. An agent that already knows the answer, because the target is popular and the answer is in its training data, and writes it down without fetching it: fails. Right value, wrong epistemics. An agent that invents a well-formed ev_ hash: fails. An agent that read a static fact and reported it with the wrong authority: fails, even with the correct value.
Most agent evaluations I've seen grade the answer. This one grades whether the agent was entitled to the answer. For a tool whose whole purpose is explaining how somebody else's software works, that's the thing worth measuring.
What it costs
None of this is free, and I'd be wary of anyone presenting it as a pattern to copy wholesale.
Payload size. Inline evidence with raw and normalised results, limitations and locations is heavy. REA has a dedicated encoder, src/server/toolResultEncoding.ts, that measures a result against the wire budget before allocating it, and the 6.4.0 release notes lead with summaries and selected views so agents can find retained evidence without loading every payload. Provenance competes with the context window, and the project is visibly still negotiating that trade.
The taxonomy is only as honest as its producers. The schema guarantees a confidence is present and that changing it changes the ID. It cannot guarantee that a provider adapter labelled the fact correctly in the first place. Trust has moved from the model to the adapter code. That's a real improvement, because adapter code is reviewable and testable and the repository's AGENTS.md is unusually strict about which layer owns each interpretation. But the trust hasn't disappeared, it has moved.
Load on agents and on contributors. More schema means more ways for an agent to call a tool badly, and the design guide says so plainly: "Schema validity and keyword heuristics do not establish successful tool selection." For people, the bar in AGENTS.md (preserve unknowns across every boundary, normalise once at the owning boundary, never treat unresolved identity as absent) is high. That's throughput paid for deliberately.
Fit with the domain. Plenty of agent tools need none of this. A tool that creates a calendar event or reads a ticket has a crisp success state, and wrapping it in confidence and authority is ceremony. The machinery earns its keep when two conditions hold together: plausible wrong answers are cheap to produce, and expensive to detect. Reverse engineering is the extreme case. Architecture review of an unfamiliar estate, incident forensics and security triage are closer to it than most people building MCP servers assume.
The part worth taking
You don't need REA's full model to use the idea. If you're writing tools for an agent, ask three questions of every return type:
- Can the caller tell "none" from "didn't look"? If an empty list means both, add a coverage field with a reason, and make the reason mandatory.
- Does the result say what kind of fact it is? Read from the artefact, computed from other facts, or guessed. If the tool knows, put it in a field. The model cannot reconstruct it later.
- Can a claim be traced to a specific result, and would changing the claim break the trace? Content-derived IDs give you that almost for free.
Then evaluate your agent the way REA does: not on whether it reached the answer, but on whether it can point to the result that earned it.
An agent's honesty doesn't live in its system prompt. It lives at the narrowest point the information passes through, which is the tool contract. If uncertainty is erased there, nothing downstream gets it back.