How to Calibrate AI Agent Experiments

A model can rank variants correctly while exaggerating every effect, or match average magnitude while reversing the most important cases. Calibration therefore needs several views: baseline fit, sign agreement, rank ordering, absolute error, segment error, and mechanism consistency. Each metric answers a different decision question.

Calibration is not prompting until past predictions look right. The experiment specification and adjustment procedure must be frozen before outcomes in the evaluation set are revealed. Otherwise the team is fitting a story to known answers and has no evidence that it will generalize.

The output is a usage boundary such as: suitable for removing clearly weak message variants in this market, not suitable for estimating revenue lift. That statement is more operational than a single accuracy score.

Historical volume alone does not create relevance. A set of email subject-line tests may not calibrate agents for pricing pages; consumer acquisition experiments may not transfer to enterprise renewals. Record eligibility, geography, channel, assignment unit, dates, variants, sample size, metric definition, and known implementation defects.

Exclude experiments whose outcome cannot be reconstructed reliably, but retain hard and null cases. Filtering only for clean wins inflates apparent directional agreement. Note interference, novelty, seasonality, sample-ratio mismatch, and post-hoc metric changes because observed experiments also contain error.

Split the reference set before building. Use one portion to identify systematic bias and a separate portion to evaluate the locked process. When data is scarce, use rolling or leave-one-out evaluation and report uncertainty rather than claiming a stable calibration curve.

Compare the simulated control baseline with the observed historical baseline by outcome and segment. Persistent gaps may reflect missing population evidence, unrealistic constraints, or a measurement mismatch. Correct the underlying representation when possible before applying a numerical adjustment.

[Experimental] The 2026 agentic-experimentation preprint reports a two-phase calibration design that uses pre-period information and substantially reduces prediction error in its dataset. This result is promising but preliminary; the procedure must be reproduced in the target context before it informs prospective decisions.

CUPED also uses pre-experiment data, but for a different purpose: reducing variance in randomized online experiments using correlated covariates. It does not turn a synthetic comparison into a randomized causal estimate, and its name should not be applied to generic baseline correction.

Sign agreement is intuitive but treats a tiny noisy positive effect like a decisive win. Add rank correlation when selecting among several variants and absolute or squared error when magnitude changes resource allocation. Plot predicted against observed contrasts instead of hiding variation behind one average.

Segment error matters when the overall effect masks harm or reversal for an important group. Mechanism error matters when the model predicts the right direction for the wrong reason; that mistake may fail under a new treatment. Review generated explanations against qualitative or behavioral evidence without treating eloquence as proof.

Always include a simple baseline: random sign, no-effect prediction, historical average, or a conventional model. An elaborate agentic pipeline that cannot beat the relevant baseline has not earned operational complexity.

Language models may already contain public descriptions of famous experiments. Avoid well-known cases when possible, date-bound the scenario, and inspect outputs for details unavailable in the supplied evidence. Private or recent completed experiments can provide a stronger leakage test when governance permits their use.

Do not use the held-out outcome to rewrite agent personas and then count the same case as successful validation. Preserve every adjustment in a changelog and start a new evaluation round. Model upgrades and prompt changes can alter both average behavior and variance, so an old calibration label should not silently carry forward.

MIASE's minimum-information principle is useful here: another reviewer should know what was simulated, how it was configured, what outputs were measured, and how to repeat the procedure.

Set the threshold before evaluation. Screening may require reliable rank ordering of obviously different candidates, while budget forecasting requires much stronger magnitude calibration. A model can pass the first and fail the second. Document both successful and prohibited uses.

Use confidence labels tied to evidence: exploratory when no relevant backtest exists, directional when held-out sign or rank performance is useful, and calibrated only when magnitude behavior is stable enough for the stated decision. Never remove the sample size or date range from the label.

Continue sending material decisions through live validation. After every production experiment, append the frozen synthetic prediction and observed outcome to the calibration ledger. The record should update trust in both directions.