Survey sample size and margin of error: how many answers are needed and what can be read by segment
From the confidence level, the tolerable margin, the expected share, the size of the base, the effect of quotas and the response rate counts how many answers are needed, how many invitations to send and what it costs. For an already collected sample shows the margin of the overall estimate and of each segment: a segment several times smaller gives a wide interval.
At a glance
- What question does it answer?
- How many answers are needed for the chosen margin, how many invitations to send, what it costs and what margin each segment of an already collected sample has.
- What data is needed?
- The tolerable margin and the confidence level: decided by the team before collection.
- How to read the result?
- Example at the default values (95%, ±5 pp, base 12,000, quota effect 1.2, 12% responding): 444 answers are needed, 3,700 invitations, RUB 53,280; for three segments with the same margin 1,332; with 400 collected the margin of the overall estimate is 5.3 pp Example on the default values: Answers needed: 444; Without base and quota corrections: 384; Invitations needed: 3,700; Cost of collection: RUB 53,280. The figures are illustrative: enter your own to see your result.
Why and how to reason
A survey gives an estimate of a share, for example “how many buyers would come back for the same item”, and every estimate has an interval. Its width depends on the number of answers: the margin equals the quantile multiplied by the root of p(1 − p) divided by n. The quantile is set by the confidence level (1.96 for 95%), the share p when unknown is taken as 50% because then the sample is the largest. Quotas and weighting enlarge the needed number of answers several times, and for a small base there is a finite-population correction. In the example for ±5 pp with a base of 12,000 and an effect of 1.2, 444 answers are needed; at a response rate of 12% that is 3,700 invitations and RUB 53,280. If three segments are read with the same margin, 1,332 answers are needed, because each segment is a separate survey.
When to use it
Apply before a survey to understand how many answers and how much money are needed, and after collection to read the result and the segments honestly. It does not replace interviews: to find the reasons and the customers’ words 8–15 conversations are enough, and a sample is needed only to estimate shares. Not suitable if the respondents are selected by their own wish: the bias is larger than the margin. For a small base the finite-population correction works.
How to read the result
Look at the number of answers, the invitations and the margin of the segments. If 400 answers are collected, the overall estimate reads with a margin of about 5.3 pp: a difference smaller than that between two shares cannot be noticed. A segment of a quarter of the sample (100 answers) reads with a margin of about 11 pp The decision: either enlarge the sample, or read the segments only by a large difference, or merge them. Do not forget bias: no number of answers fixes that only the most loyal responded. Compare the composition of respondents with the composition of the base by recency and purchase amount.
Common mistakes
A segment is read as if it had the margin of the whole sample. The response rate is taken from someone else’s survey. Invitations that did not reach the end are not included in the cost. The shares of two surveys with different wording of the question are compared. The margin is taken for accuracy: it does not account for selection bias.
What data is needed?
- The tolerable margin and the confidence level: decided by the team before collection.
- The size of the base sampled from and the expected share of the answer (when in doubt 50%).
- The response rate and the price of an answer from past surveys; the segments that you will read separately.
Inputs
| Field | Unit | Value in the example | Meaning |
|---|---|---|---|
| Confidence level | % | 95 | With what probability the true share falls into the interval: 95% is most often taken. The Excel export supports 90, 95 and 99. The example value is illustrative. |
| Tolerable margin of error | pp | 5 | The half-width of the interval in percentage points: “±5 pp” means that a 40% share reads as 35 to 45%. The example value is illustrative. |
| Expected share of the answer | % | 50 | If the share is unknown, 50% is taken: the sample then is the largest and gives a reserve. For rare events enter the expected share. The example value is illustrative. |
| Size of the base sampled from | people | 12,000 | How many people are in the base of buyers or subscribers; 0 means a very large base without the finite-population correction. The example value is illustrative. |
| Effect of quotas and weights | × | 1.2 | By how many times quotas and weighting enlarge the needed number of answers; 1 for a simple random sample. The example value is illustrative. |
| Response rate of those invited | % | 12 | How many of the invited get to the end of the questionnaire: for a mailing to your own base the rate is usually lower than for a panel; take it from past surveys. The example value is illustrative. |
| Cost of one answer | ₽ | 120 | The incentive, the panel, the time for mailing and processing per one completed answer, excl. VAT. The example value is illustrative. |
| Segments read separately | segments | 3 | How many groups you want to compare with each other with the same margin: each needs its own sample. The example value is illustrative. |
| Answers already collected | pcs | 400 | The actual number of completed questionnaires: the margin of the overall estimate and of the segments is counted from it. The example value is illustrative. |
How it is calculated
| Step | Formula |
|---|---|
| Number of answers | n₀ = effect × z² × p × (1 − p) / e² |
| Base correction | n = n₀ / (1 + (n₀ − 1) / N) |
| Invitations | n ÷ response rate |
| Segment margin | z × √(effect × p × (1 − p) / n of the segment) |
Worked numbers
Default values from the app; in the interactive version you enter your own.
- Answers needed
- 444Completed questionnaires for the set margin with the finite-population correction and the quota effect.
- Without base and quota corrections
- 384The simple formula for a large base: z² p (1 − p) / e².
- Invitations needed
- 3,700
- Cost of collection
- 53,280 ₽
- Answers for all segments if each has the same margin
- 1,332Each segment is read as a separate survey: the sample is multiplied by the number of segments.
- Margin of the overall estimate at the collected answers
- 5.3 % p.p.
Data as a table
| Measure | Margin |
|---|---|
| 100 | 10.7 |
| 200 | 7.5 |
| 400 | 5.3 |
| 800 | 3.7 |
| 1600 | 2.5 |
| 3200 | 1.6 |
Margin by segment at the collected answers
| Segment | Share | Answers | Margin, p.p. |
|---|---|---|---|
| Loyal customers | 30 % | 120 | 9.6 |
| New customers | 45 % | 180 | 7.9 |
| Lapsed customers | 25 % | 100 | 10.6 |
The smallest segment reads with a margin of about 10.6 p.p. against the tolerable 5: segments cannot be compared by a small difference.
The sample works only if the respondents resemble the whole base: the answers of those who gladly respond are biased, and no margin fixes that. 8–15 in-depth interviews are enough to find reasons and wording; to estimate a share a sample is needed.
Nuances and limits
- Reading segments as if they had the same margin as the whole sample: it is wider for a segment.
- Thinking that a large sample fixes bias: the loyal respond most readily.
- Not setting the response rate: the number of invitations turns out several times smaller than needed.