Ask ChatGPT for a score
“Rate this product from 1 to 5.”
ChatGPTReal people
ChatGPT’s ratings cluster at 3. Human ratings are spread more widely.
Every score comes from one published scoring method, run on a fixed protocol. This page sets out where simulated shoppers go wrong, how each test guards against it, the evidence that it works, and where it stops working.
When a language model is asked to rate a product from 1 to 5, its ratings cluster around 3 and vary far less than human ratings. If both versions score close to 3, a real difference between them is hard to detect.
Ask ChatGPT for a score
“Rate this product from 1 to 5.”
ChatGPTReal people
ChatGPT’s ratings cluster at 3. Human ratings are spread more widely.
Ask TestFeed
“What do you think of this?”
TestFeed’s simulated shoppersReal people
Hundreds of simulated shoppers answer in free text, and the rating is derived from that text. The resulting distribution closely resembles the human one.
Illustration. Levels run from 1, would not buy, to 5, would definitely buy. The pattern is measured in Maier et al., 2025.
Every test follows the same six steps on one versioned protocol.
Hundreds of shoppers from census profiles, shaped by your store’s totals.
Step 1 in detail ↓ Census microdata + store aggregatesQuestions in the out-of-scope list below, such as taste, smell, brand recall or sales forecasts, are stopped here. Blocked questions do not use up a test.
Out-of-scope filterThe same shoppers see every version. Each test varies one element, so any difference, apart from sampling noise, is attributable to that change.
Within-audience comparisonEach shopper writes a reaction. Nobody sees a scale or a yes/no.
Open-text elicitationEach answer becomes odds of buying. The odds combine into a likely buyer share with a range (note 2).
Steps 4 and 5 in detail ↓ Semantic Similarity RatingA fixed decision rule assigns the verdict. No one at TestFeed chooses the winner.
Step 1 in detail
Built from real population data, then shaped by your store.
US profiles from government census microdata, plus a smaller Australian set. Age, location, income and job are drawn from real census records.
Your store contributes aggregate order totals, never individual records. These totals weight the census-based audience without replacing it. Stores with more orders produce a closer match.
The same shoppers see every version, so the tested change is the only variable apart from sampling noise.
Steps 4 and 5 in detail
We use Semantic Similarity Rating (SSR), published by Maier et al. in 2025. Each written answer is compared with five reference statements, one per level of intent, and the closeness of the match becomes the odds for each level. No shopper is ever asked for a number.
This simplified keyword version runs in your browser. The production method compares meaning using text embeddings.
There are two kinds of evidence for TestFeed scores. Each is labelled with its source and its limits.
The same two versions went to hundreds of simulated shoppers and to a 200-person human panel.
| Compared | TestFeed vs panel |
|---|---|
| People asked | Hundreds simulated, 200 real |
| Verdict | ✓ Agreed |
| Top reasons given | ✓ The same four |
What it does not show. This is a single test in one category. It shows agreement in this case and does not establish accuracy across categories.
Bae Juice, 2026 · note 1of human test-retest reliability, across 57 personal care product surveys and 9,300 human responses. The spread of simulated answers also closely resembled real answers, with a distribution similarity above 0.85.
What it does not show. The benchmark covers personal care only. Other categories rest on one human panel check so far.
Maier et al., 2025 · independent published researchTest-retest reliability is how closely the same people agree with themselves when asked the same question twice. It is treated as the practical ceiling for any method. Recovering 90% of it means the method ranked products about 90% as well as asking the same people twice. The paper measured average purchase intent. TestFeed’s likely buyer share is built from the same scores, but the paper did not test that measure itself.
The two checks answer different questions. The panel check tests a full TestFeed run against real people. The benchmark tests the scoring method on its own.
Everything else on this page cites published research, linked below.
The published evidence covers personal care, and language models perform best on everyday consumer products. The further a category sits from that, the less weight a result should carry.
Common questions about the reliability of simulated shopper results.
Every score carries a range. A clear winner needs both conditions at once, a gap of at least ten points and ranges that stay apart. A ten-point gap alone does not count. The ranges guard against chance, and showing both versions to the same shoppers keeps them tighter than two separate groups would. Treat a Leaning result as a reason to retest or to test a larger change.
The published benchmark covers personal care only. Outside it we rely on one human panel check so far, the juice test above. Treat results in categories far from everyday shopping with more caution.
The score is not a sales forecast. It is the likely buyer share among hundreds of simulated shoppers, and it ranks versions against each other. It does not predict your conversion rate or revenue, which is why sales forecasts are out of scope.
Possibly, if it is a large brand. Language models learn about well-known brands in training, and simulated shoppers can carry some of that. That is why brand awareness, recall and loyalty are out of scope.
Yes. Every result groups the reasons, counts them and shows the shoppers’ own words.
The scoring method we use, and the main published critiques of synthetic respondents.
The scoring method behind every TestFeed score, tested on 57 personal care surveys and 9,300 human responses.
2023 Known limitation Whose Opinions Do Language Models Reflect? Santurkar et al. · ICMLMeasures how a model’s views differ from real survey groups.
2024 Critique Synthetic Replacements for Human Survey Data? The Perils of Large Language Models Bisbee et al. · Political AnalysisShows synthetic answers lose variety and drift over time.
2024 Known limitation Towards Understanding Sycophancy in Language Models Sharma et al. · ICLRDocuments how trained models agree with the person asking.
2024 Known limitation Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design Sclar et al. · ICLRShows small formatting changes in a prompt can change accuracy by up to 76 points.
2023 Critique Can AI language models replace human participants? Dillion et al. · Trends in Cognitive SciencesWhich research tasks synthetic respondents can and cannot replace.