ai-fanout.comOpen the lab
Menu

L-13 · Field guide

How to compare AI answers over time

Primary intent
Run a transparent time comparison of public AI answers with explicit comparability and trend gates.
Evidence state
Source-grounded reference
Review owner
Matthias Ramahi · independent review not claimed
Last reviewed
2026-08-22
Direct answer

Compare answers over time

To compare AI answers over time, repeat a frozen control set under the same declared conditions, preserve raw and normalized observations, measure predefined answer and source features, audit missingness and comparability breaks, and report dated differences before making any trend claim. Two windows show a change between two samples, not a durable trend.

Use this guide when moving from a one-off capture to a repeatable longitudinal observation series.
01#

1. Set the comparison contract

Freeze the control question set, surface, route, locale, session rule, repetition schedule, capture schema and measures. Define the minimum number of comparable windows and the conditions that break the series.

Choose whether the study describes within-window variability, between-window change or both. The design and number of repetitions differ.

02#

2. Collect each window as a versioned batch

Give every batch an identifier, protocol version, start and end time, environment note, rights state and cost record. Run the same validation and missing-data checks before calculating results.

Do not backfill a missed run with a later observation and label it with the original date. Keep the gap visible.

03#

3. Compare components, not impressions

Calculate the preregistered components: availability, claim continuity, structural features, source overlap, citation persistence, diversity and missingness as applicable. Preserve question-level results so an aggregate can be traced back to individual controls.

Review changes in the question mix, source UI and missingness before interpreting the main metric. A stable average can hide offsetting changes; a large shift can be a capture-method effect.

04#

4. State the evidence level

Report the exact windows, sample, eligible denominator, observed difference and uncertainty or variability available from the design. Identify any segmented series and show the protocol version beside the data.

Use “difference” for two comparable points. Reserve “trend” for the preregistered number of comparable windows across the required duration. For ai-fanout.com, the accepted public launch gate is at least three comparable batches across at least 60 days, plus the remaining evidence and ownership gates.

  • Difference: a measured contrast between named windows.
  • Pattern: repeated behavior within the observed sample.
  • Trend: a preregistered longitudinal claim that passed the series gate.
  • Hypothesis: an explanation that still needs targeted evidence.
S

Source notes

These sources support the definitions, standards or project boundaries named in this reference. They do not prove that a public observation dataset exists.

  1. portfolio-dossier
    Canonical ai-fanout.com domain dossier

    Confirmed ownership, accepted public Evidence Lab purpose, named Research Owner, indexable website launch and separately gated provider research.

    Owner record
  2. nist-ai-rmf-genai
    NIST AI RMF Generative AI Profile

    Supports explicit measurement, documentation, monitoring and limitations for generative-AI evaluations.

    Open
  3. w3c-prov-o
    PROV-O: The PROV Ontology

    Provides provenance concepts for entities, activities, agents, derivations, sources and versions.

    Open
  4. rfc-3339
    RFC 3339: Date and Time on the Internet

    Supports an interoperable timestamp representation tied to UTC.

    Open
Next
  1. L-04Answer stabilityMeasurement
  2. L-05Citation persistenceMeasurement
  3. L-08Comparability breaksMethod