Can AI Agents Simulate A/B Test Outcomes?

An evidence-ready record begins with provenance. Give the experiment a stable protocol ID and preserve the exact business question, decision owner, preparation date, source cutoff, and intended use. Attach the control and treatment artifacts exactly as agents received them, including surrounding context, timing, channel, and any interaction affordances. List every source used to construct the population and distinguish observed attributes from modeling assumptions. If an analyst cannot reconstruct which claim entered which agent, the result may still inspire discussion, but it is not ready to influence a consequential shortlist. Provenance also prevents a subtle form of hindsight: quietly replacing a weak source, refining a treatment, or changing eligibility after seeing outputs while continuing to describe the work as one experiment.

The population record should describe coverage, not merely persona count. Name the eligible population, segment definitions, evidence behind segment proportions, attributes intentionally held constant, and groups for which evidence is sparse. Record how individual agents were instantiated, whether histories were reused across arms, and how social relationships or network positions were assigned. A hundred generated profiles do not represent a market simply because they are different. Diversity in names and prose can coexist with identical incentives or stereotypes. Reviewers need to see which real dimensions of heterogeneity could affect the treatment and whether those dimensions were grounded, sampled, stress-tested, or omitted. When population proportions are unknown, run explicit alternative mixes and report whether the shortlist changes rather than choosing one convenient synthetic census.

The assignment record should make the contrast auditable. For a between-subject design, retain group allocation, stratification or matching variables, repeated assignment seeds where supported, and baseline balance summaries. For a within-subject design, retain exposure order, context-reset procedure, counterbalancing, and any washout or independent-state method. When agents interact, document the randomization level and possible spillover between control and treatment worlds. Individual assignment inside one shared network can contaminate the contrast because treated information reaches controls; world-level assignment estimates a different effect. Neither choice is automatically wrong, but the estimand must match it. If assignment is described only as 'we ran both prompts,' the experiment lacks the structure required to interpret a difference.

The measurement record translates generated behavior into evidence. Define one primary outcome tied to the decision, material guardrails, and a small set of exploratory mechanisms. Store the coding rubric, examples, counterexamples, missing-output rule, coder identity or model version, and agreement check. Free-form summaries should link back to coded runs so a reviewer can inspect minority and tail cases. Avoid scoring generic enthusiasm when the decision concerns action: an agent can praise a treatment while refusing to buy, share, comply, or continue. Report the full arm distributions, number of agents, number of repeated worlds, ambiguous cases, and exclusions. A percentage without its denominator and assignment structure invites readers to mistake synthetic frequency for estimated population prevalence.

The technical run manifest separates treatment signal from system drift. Preserve model provider and identifier, dated model version when available, system and user prompts, temperature or sampling settings, tool definitions and versions, retrieval configuration, timestamps, concurrency, retry rules, context limits, and random seeds where the stack supports them. Record failures rather than rerunning silently until a complete response appears. If a provider updates a model during a study, mark the boundary and treat it as a technical variant. Exact transcript reproduction may be impossible in stochastic hosted systems, but MIASE's minimum-information principle still applies: another team should be able to reconstruct the procedure, identify configuration differences, and compare outcome distributions under a meaningfully equivalent setup.

The analysis record should preserve prespecified and exploratory work separately. State how arm contrasts, paired differences, rank ordering, uncertainty, and sensitivity variants were calculated. If the team selected the best of many outcomes or treatments, record the multiplicity rather than reporting the final comparison as though it were the only question asked. Keep null and contrary findings in the same ledger as supporting findings. Generated explanations can suggest mechanisms, but reviewers should trace them to behavior and source evidence instead of rewarding fluency. When sample sizes are small or calibration is absent, use terms such as recurring, uncommon, or design-sensitive instead of attaching a probability that resembles a real-world rate.

The calibration record links simulation to observations without erasing misses. Before revealing a completed experiment's outcome, freeze the synthetic prediction and its confidence label. Then compare control baseline, direction, magnitude, rank, segments, and mechanisms. Record whether the case belonged to the build set, adjustment set, or untouched evaluation set. If a change is made after a miss, version it and evaluate the new procedure on different held-out cases. Do not let one aggregate score hide where the model fails: a workflow may rank message variants usefully but fail on prices, new markets, or treatments that depend on operational friction. Calibration should end in a narrow usage statement and an expiration condition tied to material model, prompt, evidence, or population changes.

Finally, the decision record connects the experiment to accountability. Name which candidates advance, which are revised or dropped, the reason under the prespecified rule, the unresolved risks, and the next source of evidence. For an advancing treatment, link the live protocol: eligible units, randomization, primary metric, guardrails, power basis, analysis window, stop conditions, owner, and rollback path. State explicitly that the simulation did not establish a live causal effect. After production results arrive, append them to the same record and compare the frozen expectations with observations. This final step turns agentic experimentation into a learning system. Without it, teams accumulate attractive simulations; with it, they accumulate local evidence about where simulation helps, where it fails, and how much weight future pretests deserve.

A mature program separates three kinds of uncertainty that are often collapsed into one confidence statement. World uncertainty concerns facts the team does not know about the market, affected groups, incentives, or future context. Model uncertainty concerns whether agents, prompts, interaction rules, and language-model behavior represent the intended mechanism. Sampling uncertainty concerns variation across assignments and stochastic runs. Each demands a different response: gather evidence for the world, redesign or calibrate the model, and repeat or rebalance the experiment for sampling. Adding more synthetic runs addresses only the third category. It cannot repair a missing stakeholder or an invalid treatment contrast. The protocol should label each material uncertainty, assign an owner, and state the observation or test that could reduce it.

Teams should also distinguish prediction from preparation. A simulated outcome may be operationally valuable even when it is not calibrated as a forecast. If several grounded worlds expose the same objection, the team can prepare an answer, instrument that objection in research, or add a guardrail to the live test without claiming the objection will occur at a particular rate. This is often the safest route to value: use generated behavior to expand the mechanisms considered, then let interviews, telemetry, experiments, and domain review determine which operate in reality. The decision memo should say whether an action is justified by expected impact, low preparation cost, reversibility, or calibrated evidence. These rationales are not interchangeable.

Governance should be proportionate but visible. A low-risk creative screen may need one analyst, a versioned protocol, and a live experiment owner. A pricing, eligibility, or public-policy scenario may require domain review, affected-group representation, legal or compliance approval, stronger held-out evidence, and an explicit prohibition on autonomous action. NIST AI RMF's govern-map-measure-manage cycle provides a useful structure: establish accountability, map context, measure performance and harm, and manage deployment and monitoring. The ledger can hold these controls without pretending that one checklist makes every use safe. Rigor, independence, and escalation should increase with the consequence of being wrong.

Schedule a dated review of the protocol and its sources; a pretest built for yesterday's population or product state should not silently govern tomorrow's experiment.