Synthetic Research in 2026 — What the Studies Actually Say

The number shows up in every other product pitch deck of the past eighteen months: 85 percent. 85 percent agreement, supposedly, between AI-generated answers and real people. Depending on the source, it becomes “85 to 92 percent” or “near-human.” What rarely gets mentioned: what was measured, under which conditions, against which benchmark.

This article reads the key studies in detail. It is deliberately technical, because synthetic research becomes dangerous without methodological understanding — not because it is wrong, but because it creates an air of exactness where differentiation is needed.

What Park et al. (2024) actually did

The Stanford HAI paper by Joon Sung Park and colleagues (Generative Agent Simulations of 1,000 People, arXiv 2024) is currently the most-cited piece of evidence for synthetic research. It deserves a close reading.

The study recruited 1,052 participants from the US, screened through the Prolific platform. Each participant gave a two-hour, semi-structured interview, which was then fed into a generative agent as a context document. The agents ran on GPT-4o and used a specialized persona framework (reflective memory).

Two weeks later, the same participants came back — both to answer their own questions again (test-retest reliability) and to answer new questions that were also put to the agents in parallel. The primary metric was how closely the agents’ answers matched the later human answers, normalized to the test-retest reliability of the humans themselves.

The central finding:

The agents reached 85% of the participants’ own test-retest reliability (General Social Survey) — meaning they were about as consistent with the later human answers as the humans were with their own earlier answers.

That is a different statement from “85 percent of the answers were identical.” Test-retest reliability measures how dependably a person repeats themselves. People typically do not repeat themselves perfectly on Big Five items: the usual two-week correlation sits at r = 0.75–0.85 (not a hit rate, but a correlation coefficient). As a rough illustration: if you multiply the Park et al. figure of 85% by this human self-reliability, you land somewhere around two-thirds real consistency with the later answer — that is not clean math, just a rule of thumb to gauge the order of magnitude.

This is a genuinely remarkable finding — and it is less spectacular than the headlines suggest. It means AI agents can approximate individual people, but they are not identical to them.

What test-retest reliability really means

If you are not familiar with survey methodology, this takes a moment of orientation. Test-retest reliability indicates how closely a measurement matches itself when taken at two points in time. For psychological constructs like the Big Five, values typically sit at r = 0.75–0.85 over two weeks. In other words: even without any external change, people do not answer 100% identically.

This fact matters because it sets the ceiling for every synthetic method. An agent can never be more reliable than the person themselves. When Park et al. reach 85% of human reliability, they are hitting a theoretical ceiling, not the limits of their method.

The reverse also holds: if someone promises 95% or 99% agreement, they have a methodological problem. Values like that are not plausible below self-reliability — and not informative above it.

Where synthetic research is reliable

From Park et al. (2024) and from Argyle et al. (2023) (Out of One, Many: Using Language Models to Simulate Human Samples, Cambridge University Press), three conditions emerge under which synthetic research becomes methodologically robust:

Condition 1: Personality items and attitude questions

Synthetic agents reach their highest agreement on personality-adjacent and values-adjacent items: Big Five scales, political attitudes, basic ethical stances, trust questions. That is consistent with these constructs being stable over time and well operationalized in language.

Condition 2: Broad behavioral tendencies

Would this person rather take a risk or seek safety? — agents answer tendency questions like this with a hit rate of roughly 70–85%, depending on context depth. That is enough for many design decisions.

Condition 3: Copy and framing sparring

One of the undervalued applications: synthetic personas are excellent at catching off notes in language. A text run past fifteen differently characterized agents surfaces inconsistencies a single reader would miss. That is not research in the strict sense, but it is a valid testing method — comparable to code review in software development.

Where synthetic research fails

Just as important: the limits. In a 2025 article by Kate Moran, the Nielsen Norman Group systematically listed where synthetic user research falls short:

Failure 1: Extreme reactions are underestimated. Synthetic agents trend toward average answers. A real test participant who quits the interview out of frustration, or who delivers an unexpected, idiosyncratic insight — agents produce such moments less often.

Failure 2: Context-sensitive behavior. How does the person react when the page takes three seconds to load, the form has a typo, and a package is being delivered outside? — agents are too smooth for multifactorial situations like this. They construct plausible, but not realistic, reaction sequences.

Failure 3: New categories with no analogy. When a product is conceptually new — not the tenth SaaS variant, but a push into a new category — agents lack the basis for valid projection. They fall back on near analogies, which leads to systematic underestimation of behavioral variance.

The Nielsen Norman Group (Moran, 2025) describes qualitatively what teams observe in practice: synthetic personas come across as “one-dimensional,” produce “a flat average of many experiences,” and lean toward sycophancy — they rate even weak ideas approvingly. Quantitative measurements of variance reduction in synthetic personas are still sparsely published in 2026; teams that want to test category innovation should expect systematically over-optimistic signals and budget for real user samples.

What bias audits actually need to check

Anyone setting up synthetic research on a scientifically sound footing needs a bias audit loop. Concretely, that means:

1. Control for demographic drift. LLMs carry training biases. An uncontrolled synthetic sample is often too Western-urban in age, gender, and education level. Weight or stratify deliberately.

2. Check for answer homogeneity. If all thirty agents answer in the same direction, something is off. Real user samples produce wider spread. Homogeneity is a warning sign of prompt leakage or persona uniformity.

3. Compare against real samples periodically. At least once a quarter, a question that was answered synthetically should also be tested with real users. The gap between the two is your method-quality metric.

4. Document prompt versions. An agent that ran on GPT-4 six months ago answers differently on GPT-4o today. Without version control, longitudinal research is impossible.

How to set up synthetic studies cleanly

Taken together, the evidence points to a practical approach for teams that want to use synthetic research validly:

1. Define the purpose. Pre-testing hypotheses? Copy sparring? Broad attitude measurement? Each class comes with different reliability requirements.
2. Choose the persona granularity. Archetypes (generic) are enough for framing tests. Individual personas with psychologically dense profiles (Big Five, bias patterns, cultural context) are needed for behavioral-tendency studies.
3. Build in redundancy. At least five, ideally fifteen agents per test run. A single agent is not a study.
4. Combine qualitative and quantitative analysis. Scores alone miss the patterns that surface in the free-text answers.
5. Validate selectively. Expensive studies with real users are not replaced — they are deployed more precisely. Synthetic research shortens the hypothesis funnel; real research confirms the final picks.

What to expect by 2030

Three methodological developments are taking shape:

First: validation corpora are getting bigger. Park et al. (1,052) was the state of the art in 2024; in 2026, studies with five to ten thousand participants are emerging. The consequence: tighter confidence intervals and more precise estimates of where the method works.

Second: reliability on behavioral predictions will improve — but more slowly than reliability on attitude items. Behavioral context is multifactorial by nature, and LLMs are structurally poor at modeling interactions they have not seen in training data.

Third: the replacement claim will disappear. The industry is moving from “synthetic personas replace real research” to “synthetic personas extend real research.” That is a correction, not a defeat.

What follows from this

Synthetic research in 2026 is a valid but precisely bounded method. Selling it as a replacement for user interviews means selling a promise the evidence does not cover. Using it as a first sieve — for copy sparring, hypothesis screening, persona stress tests — yields a tool that significantly raises research productivity.

The honest summary: the 85 percent figure is real, but it is a statement about approximating reliability, not about identical answers. Between those two readings lies the difference between a serious tool and a marketing claim.

For the methodological foundations of our own implementation — Big Five validation, bias composition rules, and bias audit loops — see the Science page.