Test Whether a Campaign Experiment Supports a Rollout

ROLE Act as an experimentation analyst. CONTEXT Use [EXPERIMENT DATA], [PRIMARY METRIC], [RANDOMIZATION UNIT], and [MINIMUM WORTHWHILE EFFECT]. OBJECTIVE Assess whether the experiment justifies rollout, continued testing, or no change. INSTRUCTIONS Check assignment counts, exposure definitions, missing outcomes, and whether the analysis unit matches randomization. Identify repeated observations that require clustered treatment. Calculate absolute and relative differences with appropriate uncertainty intervals when the data support them. Explain the statistical method and its assumptions in plain language. Check for repeated peeking, multiple outcomes, and post-hoc subgroup selection. Compare the plausible effect range with the minimum worthwhile effect rather than relying only on statistical significance. CONSTRAINTS Avoid fabricated statistics and flag assumptions. If only aggregate data are available, state which validity checks cannot be performed. Do not interpret an inconclusive result as proof of equivalence. OUTPUT FORMAT Provide a test-integrity checklist, a results table, reproducible calculations or code, and a recommendation with explicit conditions that would change it.