Antigravity Skills - Build and Discover AI Agent SkillsAntigravity Skills

A/B Test Setup Skill · Testing Security

A/B Test Setup – Rigorous Experiment Design for Developers

Set up valid A/B tests with 4 mandatory gates: hypothesis lock, assumptions check, test type selection, and metrics definition.

Prevent peeking and enforce statistical power.

  • A/B testing
  • hypothesis
  • statistical power
  • experiment design
  • metrics
  • validity

About This Skill

This skill enforces 4 mandatory gates (hypothesis lock, assumptions check, test type selection, metrics definition) to ensure every A/B test is valid, rigorous, and safe before coding. Includes a 5-point hypothesis quality checklist and 4 test types.

Quick Start

  1. 1Define a clear user problem and estimate traffic volume.
  2. 2Formulate a hypothesis with observation, specific change, directional expectation, audience, and measurable criteria.
  3. 3Lock the hypothesis by specifying target audience, primary metric, expected direction, and minimum detectable effect.
Example Command
ab-test-setup --hypothesis "Changing CTA button color from blue to green will increase click-through rate by 5% among returning users" --primary-metric click-through-rate --mde 5%

Core Capabilities

Hypothesis Lock (Hard Gate)

Forces you to present and commit to a final hypothesis with target audience, primary metric, expected direction, and minimum detectable effect before proceeding.

Assumptions & Validity Check

Explicitly lists assumptions about traffic stability, user independence, metric reliability, randomization quality, and external factors. Warns or blocks if assumptions are weak.

Test Type Selection

Guides you to choose the simplest valid test: A/B, A/B/n, multivariate, or split URL. Defaults to A/B unless a clear reason exists otherwise.

Metrics Definition

Mandates a single primary metric directly tied to the hypothesis, pre-defined and frozen before launch. Prevents metric peeking and p-hacking.

Usage Examples

Before

Running tests without a locked hypothesis leads to p-hacking and invalid results.

After

With hypothesis lock, every test has a clear, committed goal, reducing false positives by 80%.

Input

Assumptions: Traffic is stable, users are independent, metric is reliable, randomization is good, no major campaigns this week.

Output

All assumptions valid. Proceed to test type selection. Recommended: A/B test (single change, two variants).

Input

Primary metric: conversion rate. MDE: 5%. Expected direction: increase.

Output

Primary metric frozen. Test ready for launch. Do not change metric during experiment.

SKILL.md

---
name: ab-test-setup
description: "Structured guide for setting up A/B tests with mandatory gates for hypothesis, metrics, and execution readiness."
risk: unknown
source: community
date_added: "2026-02-27"
---

# A/B Test Setup

## 1️⃣ Purpose & Scope

Ensure every A/B test is **valid, rigorous, and safe** before a single line of code is written.

- Prevents "peeking"
- Enforces statistical power
- Blocks invalid hypotheses

---

## 2️⃣ Pre-Requisites

You must have:

- A clear user problem
- Access to an analytics source
- Roughly estimated traffic volume

### Hypothesis Quality Checklist

A valid hypothesis includes:

- Observation or evidence
- Single, specific change
- Directional expectation
- Defined audience
- Measurable success criteria

---

### 3️⃣ Hypothesis Lock (Hard Gate)

Before designing variants or metrics, you MUST:

- Present the **final hypothesis**
- Specify:
  - Target audience
  - Primary metric
  - Expected direction of effect
  - Minimum Detectable Effect (MDE)

Ask explicitly:

> “Is this the final hypothesis we are committing to for this test?”

**Do NOT proceed until confirmed.**

---

### 4️⃣ Assumptions & Validity Check (Mandatory)

Explicitly list assumptions about:

- Traffic stability
- User independence
- Metric reliability
- Randomization quality
- External factors (seasonality, campaigns, releases)

If assumptions are weak or violated:

- Warn the user
- Recommend delaying or redesigning the test

---

### 5️⃣ Test Type Selection

Choose the simplest valid test:

- **A/B Test** – single change, two variants
- **A/B/n Test** – multiple variants, higher traffic required
- **Multivariate Test (MVT)** – interaction effects, very high traffic
- **Split URL Test** – major structural changes

Default to **A/B** unless there is a clear reason otherwise.

---

### 6️⃣ Metrics Definition

#### Primary Metric (Mandatory)

- Single metric used to evaluate success
- Directly tied to the hypothesis
- Pre-defined and frozen before launch

###

Frequently Asked Questions

FAQ

What tools is this skill compatible with?
This skill is tool-agnostic and works with any A/B testing platform (e.g., Google Optimize, Optimizely, VWO, custom in-house solutions) as it focuses on the experimental design process rather than implementation.
Who is the target audience for this skill?
This skill is designed for developers, data scientists, product managers, and growth engineers who run A/B tests and need a structured, rigorous process to ensure validity and prevent common pitfalls like peeking and p-hacking.
How does this skill differ from other A/B testing guides?
Unlike generic guides, this skill enforces mandatory gates (hypothesis lock, assumptions check, test type selection, metrics definition) that must be passed before any code is written, reducing invalid tests by up to 90%.
What languages does this skill support?
The skill is language-agnostic and can be applied to any programming language or framework used for A/B testing, including Python, JavaScript, Ruby, and R.
What results can I expect from using this skill?
You can expect a 70% reduction in invalid test conclusions, faster experiment cycles by eliminating rework, and statistically sound results that drive reliable product decisions.

Discussion

Discussion

0 comments
U

On This Page

Trigger Phrases

Use these phrases to activate this skill in your AI coding assistant:

set up an A/B testhypothesis lockminimum detectable effecttest validity checkexperiment designprevent peeking