How to measure AI answer stability

Stability has several dimensions

Exact text matching is too brittle for most generated answers, while a single semantic score can hide important changes. Treat stability as a feature set. Record whether an answer appeared, which defined claims were present, how the response was structured, which sources were visible and which citations persisted.

Choose the features before collection. Retrofitting the measure after seeing the answers invites selective interpretation and makes repeated batches harder to compare.

• Availability: answer returned, unavailable or interrupted.
• Claim continuity: preregistered propositions present or absent.
• Structural continuity: sections, ordering or answer form.
• Source continuity: URL or domain overlap.
• Citation persistence: recurrence for the same control question.

Control the observation conditions

Repeat the same stable question through the same declared route under the same locale and configuration. Record timestamps and any visible model or surface label. If account state, personalization or session context cannot be controlled, record that limitation rather than pretending the runs are identical.

Multiple repetitions within a window help separate within-window variability from change between windows. The number and timing of repetitions belong in the protocol.

Prefer inspectable components to a magic score

A composite score can be useful only when its components, weights and missing-data behavior are published. Otherwise it gives precision without interpretability. Report the component measures first and keep any composite secondary.

For claim continuity, publish the coding guide and reviewer process. For source overlap, publish normalization rules. For text or embedding similarity, publish the model, version, threshold and sensitivity analysis where rights allow.

Two observations are a difference, not a trend

A change between two points is descriptive. A trend claim needs repeated comparable windows, enough observations to assess normal variation, and a protocol that survived the interval. A provider or interface change may require a new series.

The practical reporting language is simple: name the dates, sample and observed difference; state whether the series met the preregistered trend gate.