What AI Changes About User Research, and What It Does Not

A recurring claim in product circles holds that language models have made user research slower and more expensive than it needs to be. If a model can be asked what a nurse, a warehouse supervisor or a first-time buyer would think, why recruit one, schedule a session and pay for the hour?

The question deserves a serious answer rather than a defensive one. Three distinct things are happening at once, and they pull in different directions.

Models as substitutes for participants

The first is the use of “synthetic users”, generated profiles asked to answer as though they were members of a population. The empirical record here is now large enough to draw on rather than speculate about.

MeasuringU’s review of twelve peer-reviewed studies published between 2023 and 2026 counted nine encouraging results against fourteen discouraging ones. The pattern across them is consistent and worth stating precisely: models reproduce surface statistics reasonably well, and fail on everything underneath. Aggregate means sometimes land close to human ones, while subgroup means, variance and regression coefficients do not. In one replication attempt, only about a fifth of fourteen classic studies came out usable. In a survey comparison, roughly seventy percent of policy question means differed significantly from the human responses they were meant to stand in for.

Behavioural studies reach the same conclusion by a different route. Kuric, Demcak and Krajcovic, in What Would GPT Click, compared model and human interaction on real interfaces and found systematic divergence in what each chose to click. The models were not noisy versions of people. They were differently behaved.

Two failure modes recur often enough to name. Generated participants tend toward agreement, praising concepts that human participants reject, which makes them worse than useless for evaluation because they fail in a flattering direction. And they are confined to their training distribution, so they remix what has already been recorded rather than surfacing what is currently true of a specific population in a specific context.

The Nielsen Norman Group’s position on synthetic users draws the line where the evidence puts it. Generated profiles are defensible for desk research, for drafting hypotheses and for piloting an interview guide. They are not defensible for concept validation, prioritisation or any finding presented as though it came from real people. Treat their output as a hypothesis awaiting a test, never as the test.

Models as the object of research

The second shift is more consequential and receives less attention. When the product itself incorporates a model, the assumptions underpinning conventional usability testing quietly stop holding.

Classical usability method assumes a system that responds identically to identical input. Much of the discipline depends on it: a task either succeeds or fails, a defect reproduces, two participants meeting the same obstacle constitute a pattern. A generative feature satisfies none of this. It may answer well and then poorly, be confidently wrong, or behave differently for the same person on consecutive attempts.

This changes the unit of observation. The question is no longer only whether someone completed the task, but whether they could tell a good answer from a bad one, and what they did when the system erred. Nielsen Norman Group’s research agenda for generative AI in UX frames this as an open problem: how to evaluate systems that change over time, and which interface patterns genuinely support transparency rather than merely gesturing at it.

Design guidance for this class of system predates the current wave. Amershi and colleagues published eighteen guidelines for human-AI interaction at CHI in 2019, validated against twenty deployed products. Most concern behaviour under uncertainty: making clear what the system can do, how well it does it, and what recourse exists when it is wrong. These remain a reasonable starting checklist.

The reliance problem

The third shift is the one that most firmly settles the question, and it is counterintuitive enough to be worth stating carefully.

The relevant measure for a system that is usually right is not trust but what the literature calls appropriate reliance: accepting the system’s output when it is correct, and catching it when it is not. Raees and Papangelis, reviewing measurement constructs in human-AI decision-making, note that these two things are routinely conflated, and that a product can score well on trust while performing badly on reliance.

The uncomfortable finding is that the obvious remedies do not reliably work. Adding explanations tends to raise acceptance of answers, including wrong ones, because users read the presence of an explanation as a signal of competence rather than evaluating its content. Interventions meant to calibrate trust, including uncertainty indicators, have been observed to attract the same unexamined confidence they were introduced to correct. Automation bias, the tendency to defer to a machine against one’s own better judgement, persists in expert users and is not reliably trained away.

The practical consequence is direct. Whether an interface produces appropriate reliance cannot be deduced from the design. It has to be observed, in the population that will use it, on tasks where the system sometimes fails. There is no analytical shortcut, and a generated participant cannot supply the answer because the behaviour in question is precisely the human one being modelled away.

For readers outside the discipline

Three conclusions, stated without the vocabulary.

Asking a model what your users think is desk research, not research. It is a reasonable way to arrive at a question and an unreasonable way to answer one. The error is not that the output is useless; it is that it is plausible, which makes it harder to discard than an obvious mistake would be.

A feature that does not behave the same way twice needs a different kind of testing. Watching ten people succeed proves less than it used to. What matters is what happens on the occasions the system gets it wrong, which means those occasions have to be part of the study rather than excluded from it.

Explanations and confidence scores are not evidence that users understand the system. They are frequently read as reassurance. If a product’s correctness matters, the only way to know whether people catch its mistakes is to watch them try.

None of this makes research a brake on shipping. The cost of a poor design decision has not fallen, and where a system is convincing regardless of whether it is correct, that cost has risen. Studying real people remains the only method available for finding out what a product actually does to the people using it.