Observable evidence in AI answers

Keep three evidence layers separate

A usable observation record distinguishes the submitted input, the returned public output, and the observer’s interpretation. Mixing the layers makes later review almost impossible: an analyst cannot tell whether a label came from the interface, the provider documentation or the researcher.

The first two layers can be recorded directly when terms and rights allow. The third layer is legitimate analysis, but it needs its own field, method and confidence. It should never overwrite the raw observation.

• Observed: text and controls visible in the public interaction.
• Supported: a mechanism described in named provider documentation.
• Inferred: an interpretation derived from observed patterns.
• Unknown: information neither observed nor documented for the case.

Minimum capture for a reviewable observation

Record a stable observation ID, timestamp with timezone, surface and route, locale, stable question ID, protocol version, visible source URLs, and a missing-data reason when an expected field is unavailable. If raw answer retention is restricted, record the permitted representation and the rule that produced it.

A screenshot alone is weak provenance. It may show appearance, but often omits route, locale, timing, configuration and machine-readable source normalization. Pair visual evidence with a structured record whenever possible.

Do not upgrade an observation into a system claim

A source link appearing in an answer proves that the link was visible in that captured output. It does not prove how the system found, ranked, read or weighted the page. Similar answers across two runs show similarity across those runs, not permanent model stability.

This is the most common reporting error in AI-search studies: the observation is real, but the sentence travels beyond it. A reviewer should be able to trace every claim back to the exact field or documentation source that supports it.

A compact claim test

Before publication, ask four questions: What exactly was captured? Which part of the sentence comes from documentation? Which part is analysis? What plausible alternative explanation remains? If any answer is missing, reduce the claim or collect better evidence.

The result may sound less dramatic, but it becomes more useful. Readers can repeat the observation, challenge the interpretation and understand what would change the conclusion.