How to sample public AI answers

Name the population you want to describe

“AI answers” is not a useful population. Define the provider surface, access route, locale, time period and question space. A study of English informational questions in one public search interface cannot automatically describe other languages, commercial tasks, logged-in assistants or API outputs.

Write the scope as a sentence another researcher could use to decide whether a new observation belongs in the sample. Ambiguous membership produces ambiguous denominators.

Build a question frame before choosing examples

Group the question space by dimensions that matter to the research question: task type, answer form, topic volatility, source requirement, locale or another defensible class. Select within those strata using a documented rule.

Convenient or hand-picked prompts may still support an exploratory pilot, but label the result accordingly. Do not present a convenience sample as a market-wide benchmark.

• Define inclusion and exclusion rules.
• Assign stable question IDs before collection.
• Separate fixed controls from rotating exploratory questions.
• Record the source of the question frame without exposing personal data.

Separate breadth from repetition

More unique questions improve coverage of the declared question space. More repetitions per question help estimate response variability. They are not interchangeable. A design needs enough of each for the intended measure.

Fix the timing rule: sequential runs, spaced runs, fixed daily windows or another method. Avoid silently changing the schedule between batches.

Run a protocol pilot, not a headline pilot

The first batch should test whether questions are unambiguous, capture fields work, missing cases are classified, costs are sustainable and provider terms permit retention. It is a method test, not a public trend result.

Freeze the protocol only after documenting pilot changes. If those changes affect the observation unit or measure, start the comparable series after the freeze.