Sensitivity Analysis for AI Scenario Simulations

Traditional sensitivity methods attribute output variance to model inputs. LLM-agent systems add new sources of variation: natural-language interpretations, generated plans, interaction order, tool output, model version, and sampling. The same principle still applies—change inputs deliberately and observe which conclusions move.

Begin with the decision metric, not the most visible text. A launch simulation might track the prevalence of a trust objection, time to escalation, or effectiveness of a mitigation. Without a defined outcome, teams often compare wording differences while missing a decision reversal.

List the assumptions behind the conclusion and score each for uncertainty and leverage. A well-established product price may be low uncertainty but high leverage; a speculative audience weight may be high on both. Start with the assumptions in the high-high quadrant.

Include omissions. Adding an overlooked regulator, distributor, minority audience, or internal veto holder can matter more than changing an existing persona. The purpose is to test the structure of the scenario, not merely tune numeric settings.

One-at-a-time variants are easy to explain and useful for screening. They can miss interactions, so follow with combined variants for influential inputs. For example, a skeptical audience may become decisive only when a trusted intermediary also delays its response.

Keep everything else fixed and versioned. Use the same output coding scheme across variants. If the model or prompt changes during the test, treat that as a separate technical variant rather than silently mixing runs.

A result can be directionally stable while the path varies. If most variants produce the same objection through different actors, preparation may still be justified. Conversely, a similar average score can hide distinct failure modes that demand different mitigations.

Use tables or plots that show distributions by variant. Report how many runs support each conclusion and mark configurations that reverse the action. Avoid precise probability language unless the design and calibration support it; generated frequencies are properties of the simulation setup, not direct estimates of population prevalence.

Fragility is useful information. It can identify a fact to research, a stakeholder to interview, a mitigation to test, or a decision that should remain reversible. Do not hide a fragile result behind an average across incompatible scenarios.

Distinguish model fragility from world fragility. If changing only the language model reverses the result, the technical layer needs investigation. If competing but credible source assumptions reverse it across models, the real-world uncertainty may be the dominant issue.

Tie each influential variable to an action. Monitor it if it will become observable, research it if evidence can reduce uncertainty, or build a contingency if neither is possible. A sensitivity report that does not change the decision process is only technical documentation.

Preserve the matrix with the decision record. When real signals arrive, compare them with the tested variants and update the scenario. This makes simulation a learning loop instead of a one-time forecast artifact.