AI Answer Sample-Size Calculator

Find how many AI answers you need to measure a mention rate within a margin of error, or the margin your answers give.

How to use

What it checks and how it works

The calculator uses the normal approximation for a proportion: n = z² × p × (1 - p) / E², where z comes from the confidence level, p is the expected rate and E is the margin as a proportion. It is the standard sample-size formula for estimating a proportion, described in the statistics references below. At p = 50% it gives 9,604 answers for a margin of ±1 point at 95% confidence, and 385 for ±5 points. It assumes every answer is an independent draw. Repeated runs of the same question are correlated, so many questions with a few runs each give more information than one question run many times. The figure is a planning number: the real uncertainty is wider when the answers are correlated, and it does not cover differences between assistants or markets unless you size each one separately.

Why does 50% need the most answers?

A rate near 50% has the most variation, so measuring it takes the largest sample. If you expect a rate near 10% or 90%, you need fewer answers for the same margin.

Is a run of the same question a new answer?

It is a new answer, but not an independent one, because the question and the market are the same. For a mention rate that stands for your category, use many different questions and a few runs of each.

Does the margin apply to each assistant separately?

Only if you collect that many answers from each one. The margin applies to the group of answers the rate is computed from. To compare two assistants, size each group on its own.

Sources

Related tools

Privacy

Runs in your browser; nothing is uploaded. The interactive tool needs JavaScript.