Simulation-to-Live A/B Test Checklist

A handoff cannot repair an ambiguous simulation. Confirm that the control represents the current experience and the treatment differs only as declared. Archive source, prompt, model, population, assignment, and coding versions. Retain all material runs, including nulls and failures.

Summarize distributions and mechanisms rather than selecting a persuasive conversation. Record how results changed across treatment order, population variants, model configuration, and contested evidence. A fragile recommendation may still enter a live test, but its fragility should shape monitoring and interpretation.

Give each candidate a status: advance, revise, drop, or unresolved. The simulation's job is to reduce and improve the live experiment set, not to certify a production effect.

Review affected users, access, price, safety, privacy, fairness, and legal constraints. Simulation can surface failure stories but cannot authorize exposing real people to an unsafe condition. Domain and compliance owners must approve the live protocol where their expertise is required.

Check operational readiness. A promising variant is not testable if assignment leaks, logging differs by arm, support cannot handle failures, or rollback is unavailable. Add guardrail metrics and real-time alerts for known risks without turning every available metric into a decision endpoint.

NIST AI RMF recommends mapping context and managing risk throughout the lifecycle. Apply more scrutiny as impact and irreversibility increase. For high-stakes decisions, simulated support should remain secondary to independent evidence and accountable review.

Define the estimand in operational terms. Specify whether the analysis is intent-to-treat, how repeated visitors are handled, what counts as exposure, and how missing data is treated. Confirm that instrumentation has equivalent semantics in both arms.

Base sample-size planning on the live metric's historical variance and a meaningful minimum detectable effect, not an unvalidated simulated lift. CUPED or other variance-reduction methods may improve sensitivity when their assumptions hold, but they belong in the live analysis plan.

Register primary and guardrail metrics separately from exploratory diagnostics. Set the experiment duration and stopping process. Repeated peeking followed by an unplanned stop changes error behavior and weakens the conclusion.

Begin with an instrumentation or low-percentage ramp when feasible. Check sample-ratio balance, eligibility, exposure, event completeness, latency, and treatment fidelity before interpreting outcome movement. A technical failure can look like a behavioral effect.

Monitor guardrails at a cadence appropriate to harm. The team should know who can pause the experiment, how rollback works, and which incident threshold overrides the statistical plan. Document any intervention and treat the resulting analysis accordingly.

Do not use the simulation narrative to explain away an unfavorable early signal. The live data may reveal a mechanism that agents omitted. Preserve surprise as evidence rather than forcing it into the earlier story.

Report assignment counts, exposure, duration, estimate, interval, primary outcome, and material guardrails. Investigate sample-ratio mismatch and instrumentation defects before drawing conclusions. Segment analyses that were not prespecified should generate future hypotheses rather than definitive subgroup claims.

A null result is information. It may mean the treatment effect is smaller than the detectable threshold, implementation diluted exposure, or the simulated mechanism does not operate in the real population. Do not reframe a null solely as a simulation success because one qualitative mechanism appeared.

State where the result applies: population, product version, channel, geography, and period. Causal validity within the experiment does not guarantee transport to every future context.

Append the live result to the calibration ledger even when it contradicts the simulation. Compare control baselines first, then treatment contrast, ranking, subgroup patterns, and observed mechanisms. A correct winner for the wrong reason should not receive full credit.

Update population evidence, treatment context, and coding rules where the mismatch identifies a real gap. Do not overwrite the original configuration; preserve it so reviewers can distinguish learning from retrospective editing. Evaluate the revised process on other held-out cases before raising its confidence label.

Over time, the ledger shows which domains and decisions benefit from synthetic screening. It may justify deeper use in one narrow workflow and less use elsewhere. That local evidence is more useful than a generic claim that AI agents can or cannot predict behavior.