The science behind every score.

Every score comes from one published scoring method, run on a fixed protocol. This page sets out where simulated shoppers go wrong, how each test guards against it, the evidence that it works, and where it stops working.

A 1950s research scientist in a white lab coat holding a red clipboard, mid-explanation
Published benchmark ~90% of human test-retest reliability
Maier et al., 2025 · 57 surveys, 9,300 responses

Why not just ask ChatGPT?

When a language model is asked to rate a product from 1 to 5, its ratings cluster around 3 and vary far less than human ratings. If both versions score close to 3, a real difference between them is hard to detect.

  • Real peopleHuman shoppers asked the same question
  • ChatGPTOne AI model asked directly for a rating
  • TestFeedHundreds of simulated shoppers, scored from their words

Ask ChatGPT for a score

“Rate this product from 1 to 5.”

ChatGPTReal people

ChatGPT’s ratings cluster at 3. Human ratings are spread more widely.

Current box 3.3New box 3.3No clear difference

Ask TestFeed

“What do you think of this?”

TestFeed’s simulated shoppersReal people

Hundreds of simulated shoppers answer in free text, and the rating is derived from that text. The resulting distribution closely resembles the human one.

Current box 52New box 67Clear winner
  • ✓No rating scale. Shoppers never see a scale, which reduces the pull towards the midpoint.
  • ✓Hundreds of distinct profiles. Each shopper starts from a census profile weighted by your store’s order totals, which spreads the audience across real census profiles.
  • ✓Stated reasons. Each result includes the reasons shoppers gave, in their own words.

Illustration. Levels run from 1, would not buy, to 5, would definitely buy. The pattern is measured in Maier et al., 2025.

Six problems with AI and how we fix them.

Sycophancy

They agree too easily

✕A leading question gets the answer you hoped for.
✓Shoppers never see a scale or a yes/no question. Questions are neutral and open-ended, which gives the model no stated view to agree with.
Addressed in step 4 ↓ Sharma et al., 2024
Central tendency bias

They pile on the middle

✕Asked for a rating from 1 to 5, a model over-selects 3 and under-selects the end points.
✓No numeric rating is requested. The score is derived from the text afterwards.
Addressed in steps 4 and 5 ↓ Maier et al., 2025
Variance collapse

They all sound the same

✕Even with demographic conditioning, model outputs vary less than human answers.
✓Audiences are built from census data, and every result shows its range, so low variation appears as a narrow range.
Addressed in step 1 ↓ Bisbee et al., 2024
Demographic misalignment

They don’t speak for every group

✕A model’s default opinions under-represent many demographic groups.
✓Results are reported for the whole audience. Sub-group breakdowns are not offered.
Addressed in step 1 ↓ Santurkar et al., 2023
Persona brittleness

Small wording changes move the answer

✕Small formatting changes to a prompt change the result.
✓There are no hand-written personas. Every test runs on one fixed, versioned protocol.
Held constant by the protocol ↓ Sclar et al., 2024
No sensory grounding

They can’t taste or remember

✕A model has no sensory experience and no memory of past campaigns.
✓Those questions are blocked before they run.
Addressed in step 2 ↓ Dillion et al., 2023

How a test runs.

Every test follows the same six steps on one versioned protocol.

  1. Build the audience

    Hundreds of shoppers from census profiles, shaped by your store’s totals.

    Step 1 in detail ↓ Census microdata + store aggregates
  2. Check the question

    Questions in the out-of-scope list below, such as taste, smell, brand recall or sales forecasts, are stopped here. Blocked questions do not use up a test.

    Out-of-scope filter
  3. Show each version

    The same shoppers see every version. Each test varies one element, so any difference, apart from sampling noise, is attributable to that change.

    Within-audience comparison
  4. Answer in words

    Each shopper writes a reaction. Nobody sees a scale or a yes/no.

    Open-text elicitation
  5. Score the words

    Each answer becomes odds of buying. The odds combine into a likely buyer share with a range (note 2).

    Steps 4 and 5 in detail ↓ Semantic Similarity Rating
  6. Call the verdict

    A fixed decision rule assigns the verdict. No one at TestFeed chooses the winner.

    Tie0 to 4 pts Leaning5 to 9 Clear winner10+, ranges apart
    Published verdict rule

Step 1 in detail

Where the shoppers come from.

Built from real population data, then shaped by your store.

01 · Census Rows of filing cabinets with a line of people walking out of one drawer

Census-based profiles

US profiles from government census microdata, plus a smaller Australian set. Age, location, income and job are drawn from real census records.

02 · Your store

Shaped by your sales patterns

Your store contributes aggregate order totals, never individual records. These totals weight the census-based audience without replacing it. Stores with more orders produce a closer match.

How often people rebuySpend bandsWhat sells togetherWhere orders go
03 · Your audience

Hundreds of simulated shoppers

The same shoppers see every version, so the tested change is the only variable apart from sampling noise.

Stripped in code before anything is stored: Names (removed) Emails (removed) Phone numbers (removed) Addresses (removed) Totals from groups too small to stay anonymous are dropped (note 3).

Read the privacy policy

Steps 4 and 5 in detail

How words become a score.

We use Semantic Similarity Rating (SSR), published by Maier et al. in 2025. Each written answer is compared with five reference statements, one per level of intent, and the closeness of the match becomes the odds for each level. No shopper is ever asked for a number.

01 · The answer

A shopper writes what they think

“Looks less sugary than the big brands. I’d try a box if the price was close to what I pay now.”
Open-text response, no scale shown
02 · Compared

Matched to five reference answers

Would not buy.16
Unlikely.33
Might.77
Likely.74
Definitely.30
Text embeddings, cosine similarity to Likert anchor statements
03 · Spread

Turned into odds for each level

1
2
3
4
5
Similarities normalised into a probability mass function over 1 to 5
04 · Result

Combined across hundreds of shoppers

67 likely buyer share, against 52 for the current box Clear winner Illustration. Aggregated over the full audience, with a range and the verdict rule

Write a shopper’s reaction and score it.

This simplified keyword version runs in your browser. The production method compares meaning using text embeddings.

Closest reference answers
Odds for each level
0% of this shopper’s intent sits at levels 4 and 5
likely or definitely would buy

Checked against real people.

There are two kinds of evidence for TestFeed scores. Each is labelled with its source and its limits.

Human panel check

A juice test, run twice

The same two versions went to hundreds of simulated shoppers and to a 200-person human panel.

ComparedTestFeed vs panel
People askedHundreds simulated, 200 real
Verdict✓ Agreed
Top reasons given✓ The same four

What it does not show. This is a single test in one category. It shows agreement in this case and does not establish accuracy across categories.

Bae Juice, 2026 · note 1
Published benchmark
~90%

of human test-retest reliability, across 57 personal care product surveys and 9,300 human responses. The spread of simulated answers also closely resembled real answers, with a distribution similarity above 0.85.

People asked twice100
SSR scoring~90

What it does not show. The benchmark covers personal care only. Other categories rest on one human panel check so far.

Maier et al., 2025 · independent published research

How to read these numbers

Test-retest reliability is how closely the same people agree with themselves when asked the same question twice. It is treated as the practical ceiling for any method. Recovering 90% of it means the method ranked products about 90% as well as asking the same people twice. The paper measured average purchase intent. TestFeed’s likely buyer share is built from the same scores, but the paper did not test that measure itself.

The two checks answer different questions. The panel check tests a full TestFeed run against real people. The benchmark tests the scoring method on its own.

Method notes

  1. Juice test. A single paired study run with Bae Juice in 2026, comparing a full TestFeed run with a 200-person human panel on the same two versions. The full write-up is not yet published.
  2. Score ranges. Each range reflects how much a likely buyer share can move by chance. The interval method behind it is not yet published. For scale, the standard error of a single score is at most about 3.5 points, a 95% margin of about ±7 points.
  3. Small groups. Store totals leave out any group too small to keep customers anonymous. The minimum group size is not stated on this page.

Everything else on this page cites published research, linked below.

Where it works, and where it doesn’t.

The published evidence covers personal care, and language models perform best on everyday consumer products. The further a category sits from that, the less weight a result should carry.

A scientist giving a thumbs up beside a bar chart with a red tick

✓ Strong

  • Everyday consumer products, closest to the personal care benchmark
  • Purchase intent, the most reliable measure
  • Packs, claims, ad ideas and descriptions
  • A price checked in context
  • Head-to-head comparisons, including ties
  • Finding the version more of your visitors would buy

✕ Out of scope

  • B2B and specialist, expert-led categories
  • Taste, smell and texture
  • Brand awareness, recall and loyalty
  • Predicting your exact sales or conversion rate

The hard questions.

Common questions about the reliability of simulated shopper results.

How do I know a result isn’t a fluke?

Every score carries a range. A clear winner needs both conditions at once, a gap of at least ten points and ranges that stay apart. A ten-point gap alone does not count. The ranges guard against chance, and showing both versions to the same shoppers keeps them tighter than two separate groups would. Treat a Leaning result as a reason to retest or to test a larger change.

Has it been checked outside personal care?

The published benchmark covers personal care only. Outside it we rely on one human panel check so far, the juice test above. Treat results in categories far from everyday shopping with more caution.

Does a higher score mean more sales?

The score is not a sales forecast. It is the likely buyer share among hundreds of simulated shoppers, and it ranks versions against each other. It does not predict your conversion rate or revenue, which is why sales forecasts are out of scope.

Does the model already know my brand?

Possibly, if it is a large brand. Language models learn about well-known brands in training, and simulated shoppers can carry some of that. That is why brand awareness, recall and loyalty are out of scope.

Can I read what the shoppers said?

Yes. Every result groups the reasons, counts them and shows the shoppers’ own words.

Try it on your own store.

Book a demo and we'll run your first test with you.