Customer behavior and research methods: from a hypothesis to a decision (interviews, surveys, A/B, JTBD, sample size)
The document helps to choose a method of researching the customer for the question: interviews to find reasons, a survey to estimate a share, an A/B test to check cause and effect, behavioral data to see what people actually do. The decision it supports: which method to launch, how many answers to collect, what it will cost and what difference in results to consider noticeable.
Why this document
Research most often breaks not on the analysis but on the choice of method: people survey when they should listen, listen when they should measure, and draw conclusions from a handful of answers. The document is read by the head of CRM and marketing, a product manager, a category manager and an analyst before research. It turns a question into a hypothesis, chooses a method, counts the sample size and the margin of error and fixes the decision rule in advance so that the result is not fitted to what is wanted.
At a glance
- What is this document and why is it needed?
- The document helps to choose a method of researching the customer for the question: interviews to find reasons, a survey to estimate a share, an A/B test to check cause and effect, behavioral data to see what people actually do. The decision it supports: which method to launch, how many answers to collect, what it will cost and what difference in results to consider noticeable.
- Who owns the document and how often is it rebuilt?
- Brand: a product manager or a customer-base analyst, before launching a line, changing a price or a channel; the commercial director decides. Retail: category analytics and CRM, before changing the assortment, the format or the loyalty program.
- What mistakes are the most common?
- Surveying when one should listen → answers to pre-invented options, the reason is not found → start with interviews and only then a survey. Reading a segment with the margin of the whole sample → a difference between groups is seen where there is none → count the sample for each segment. Not writing the decision rule before collecting → the result is fitted to what is wanted → fix the threshold and the action in advance.
For whom
- Brand
- A brand communicates with the end customer less often than a retailer: the main sources are its own sales, subscribers, reviews and partners’ data. A brand needs the reasons for choice and refusal (interviews and JTBD), estimates of shares in its own base (a survey) and a check of prices and pages (A/B).
- Retail
- A retailer sees customer behavior in receipts, traffic and the loyalty card and can survey at the till and in the app. Differences between stores and clusters matter more to it, switching between brands and reasons for leaving: so segments and their sample play the main role.
What it consists of
Hypothesis and choice of method
- A hypothesis is written as a statement that can be refuted: “customers return dresses because of the length and not the fabric”; the question “what do customers think” is not research.
- An interview answers the question “why”: 8–15 conversations per segment until saturation, when new conversations stop giving new reasons; it does not measure shares.
- A survey answers the question “how many”: shares and comparison of groups; a sample counted in advance is needed, and questions without a hint.
- An A/B test answers the question “what causes”: one version against another on randomly split traffic; the number of observations is counted before launch and the winner is chosen by a rule set in advance.
- Brand: for a new line start with interviews and data of platforms and partners, and only then a survey of your base; do not start with a survey until you know which answer options to ask.
- Retail: for a change of assortment start with receipt and loyalty-card data (what is bought together, what is substituted), then a short survey at the till and interviews with lapsed customers.
JTBD: the job for which people buy
- JTBD (jobs to be done) describes a situation and a desired outcome: “I want to look put-together at a meeting and not think about ironing”, and not age and income.
- In an interview they ask about the last purchase: what happened, what was compared, what stopped, what decided; the answers are used to build a map of forces: what pushes towards the purchase and what pulls back.
- Brand: the job defines the assortment and communication: one style may be hired for different jobs (office, a date, a trip), and then the description and photos speak about different things.
- Retail: the job defines the display and related goods: a set “for a week-long business trip” is assembled by job and not by categories.
- The result of an interview is hypotheses for a survey and A/B, and not conclusions about the market: do not multiply the number of conversations to get “statistics”.
Survey and sample size
- The sample size is set by the margin and the confidence level: for ±5 pp at 95% about 400 answers are needed, for ±3 pp about 1,000, for ±10 pp about 100 (a simple random sample from a large base).
- A segment is read as a separate survey: if you need three groups with a margin of ±5 pp each, the sample is multiplied by three; otherwise a segment of a hundred answers gives a margin of about ±10 pp
- Brand: invitations to your own base bias towards the loyal; supplement with a panel survey or competitors’ buyers if the question is about going beyond the base.
- Retail: a survey at the till and in the app biases towards those who are satisfied and in a hurry; record the time, the store and the channel of the survey and compare only within one channel.
- The response rate turns answers into invitations: at 12% for 444 answers 3,700 invitations are needed; set it from past surveys.
The decision rule
- Before collecting, write down what difference will make you act: for example “if the share of those who recommend is below 40% among new customers, we change the packaging”; without a rule any result is interpreted in one’s favor.
- A difference within the margin means “we do not know” and not “the same”: for a decision either the sample is enlarged or the decision is made on other grounds.
- Brand: the result is recorded in a hypotheses register with the date, method, sample and decision; after a season it is checked whether the conclusions were confirmed by sales.
- Retail: the result is passed to category managers and suppliers by brands; assortment decisions are tied to previously fixed hypotheses.
What differs for a brand and a retailer in research
- Brand: compare the composition of respondents with the composition of the base by recency and purchase amount; if there are more loyal ones than in the base, recalculate the weights or add an external panel.
- Brand: start an A/B test of a card or page after counting the number of visitors per arm: at low traffic take a larger step of change or a longer period.
- Brand: interviews with wholesale buyers about the reasons for declining a style give hypotheses faster than a survey of end customers and cost less.
- Retail: limit a survey at the till to three questions and one channel; compare stores only within this channel and at the same time.
- Retail: to compare stores and clusters count the sample for each in advance: a small store rarely gives enough answers, so similar ones are merged.
- Retail: loyalty card data answers the question “what do they do”, a survey the question “why”; match them for the same customers with their consent.
Where the data comes from
| Data source | Fields |
|---|---|
| Own data | receipts and orders, traffic, the loyalty card, returns and reasons, reviews and requests |
| Surveys and interviews | questionnaires, transcripts of conversations, participants’ consents, the channel and time of collection |
| Experiments | versions A and B, traffic by groups, conversion, buyout and return, duration |
How to build it step by step
- Step 1: Formulate the business question and turn it into a hypothesis that can be refuted.
- Step 2: Determine what is needed: a reason (interview), a share (survey), an effect (A/B) or behavior (data).
- Step 3: Brand: check what data already exists in sales, reviews and at partners before surveying customers.
- Step 4: Retail: check receipts, traffic and the loyalty card: many questions are solved without a survey.
- Step 5: Count the sample size and the invitations for the margin and the segments; assess the cost.
- Step 6: Write down the decision rule before collecting: which difference and which result lead to which action.
- Step 7: Brand: conduct interviews about the last purchase, build a map of forces and formulate hypotheses for a survey.
- Step 8: Retail: launch a short survey in one channel and compare stores or clusters within it.
- Step 9: Read the result with the margin, apply the rule, record the decision and the date of checking the effect.
Formulas
| Measure | Formulas | Note |
|---|---|---|
| Sample size | n₀ = z² × p × (1 − p) ÷ e² | z: the quantile (1.96 for 95%), p: the expected share (50% when unknown), e: the tolerable margin. |
| Base and quota correction | n = effect × n₀ ÷ (1 + (effect × n₀ − 1) ÷ N) | N: the size of the base; the effect is 1 for a simple random sample. |
| Margin of a segment | e = z × √(effect × p × (1 − p) ÷ n of the segment) | A smaller segment gives a wider interval. |
| Invitations and cost | Invitations = n ÷ response rate; cost = n × the price of an answer | The response rate is taken from past surveys of the same channel. |
Formulas in notation
- Finite population correction
n = n₀ / (1 + (n₀ − 1)/N)
N: the size of the base; for a large base the correction is small.
Worked example
The sample for a survey of a base of 12,000 customers: 95%, quota effect 1.2, 12% responding, an answer costs RUB 120; all values illustrative
| Margin, ± pp | Answers needed | Invitations | Cost, RUB | Answers for three segments |
|---|---|---|---|---|
| 10 | 115 | 959 | 13,800 | 345 |
| 7 | 231 | 1,925 | 27,720 | 693 |
| 5 | 444 | 3,700 | 53,280 | 1,332 |
| 4 | 680 | 5,667 | 81,600 | 2,040 |
| 3 | 1,158 | 9,650 | 138,960 | 3,474 |
For ±5 pp 444 answers, 3,700 invitations and RUB 53,280 are needed; a margin 1.7 times smaller (±3 instead of ±5) enlarges the sample about 2.6 times. Three segments with the same margin require 1,332 answers. Calculation: “Survey sample size and margin of error”.
Figures
Data as a table
| Answers | Margin |
|---|---|
| 100 | 10.7 |
| 200 | 7.5 |
| 400 | 5.3 |
| 800 | 3.7 |
| 1600 | 2.5 |
| 3200 | 1.6 |
Margin of error at different numbers of answers. To halve the margin fourfold more answers are needed: the first hundreds of answers give the most.
From the question to the decision. The method is chosen by the question, the sample is counted before collection, the decision rule is written down in advance.
- Business question → Hypothesis
- Hypothesis → Interview: why; Survey: how many; A/B: what causes
- Interview: why → Decision rule
- Survey: how many → Decision rule
- A/B: what causes → Decision rule
- Decision rule → Action and effect check
- Action and effect check
Specifics by category
- Apparel
- Apparel: the main themes are fit, size and fabric; an interview about the last return quickly shows the reasons, a survey estimates their shares; A/B checks the size chart and photos.
- Footwear
- Footwear: the last and the size decide; it is important to ask about fitting (did they take several sizes?), the sample by sizes needs to be wider because size groups have different reasons.
- Accessories
- Accessories: often bought as a gift, so in interviews they ask about the occasion and the recipient; a survey distinguishes buying for oneself and as a gift, and their shares must be counted separately.
- Kids
- Children’s: the parent buys, the child wears; the parent is surveyed about the child and separately about the child’s age and height; the reasons for returns are tied to growth and the season and not only to quality.
- Men
- Men’s: respondents less often answer long questionnaires, more invitations are needed for the answers; a short survey and behavioral data work better than a long questionnaire.
- Women
- Women’s: more occasions and scenarios, so JTBD interviews matter more and there are more segments in a survey; make sure the sample for each segment is sufficient.
Common mistakes
- Surveying when one should listen → answers to pre-invented options, the reason is not found → start with interviews and only then a survey.
- Reading a segment with the margin of the whole sample → a difference between groups is seen where there is none → count the sample for each segment.
- Not writing the decision rule before collecting → the result is fitted to what is wanted → fix the threshold and the action in advance.
- Asking leading questions → answers confirm the author’s hypothesis → neutral wording and a trial survey on 10 people.
- Taking the margin for accuracy → selection bias stays invisible → compare the composition of the sample with the composition of the base.
Russian specifics
- Surveys and interviews tied to a customer are the processing of personal data: consent, purpose and retention period are needed; for aggregated results keep depersonalised data. Rule: Law 152-FZ of 27.07.2006: Federal Law of 27.07.2006 No. 152-FZ on personal data.
- Sending survey invitations with advertising content (a discount for taking part) may be considered advertising: check the requirements of the advertising law for mailings and the recipient’s consent. Rule: Law 38-FZ of 13.03.2006: Federal Law of 13.03.2006 No. 38-FZ on advertising (rule check not completed).
- An A/B test of the price on a site must follow the sale rules: the price at the time of the order matches the one shown, and different groups do not receive hidden terms; check the terms of the platform and the offer.