Antigravity Skills - Build and Discover AI Agent SkillsAntigravity Skills

A/B Test Setup Skill · Testing Security

A/B Test Setup for Developers

Ensure every A/B test is valid with our 6-step structured guide.

Prevent peeking, enforce statistical power, and block invalid hypotheses. Perfect for developers.

  • A/B testing
  • hypothesis
  • metrics
  • statistical power
  • experimentation
  • validity

About This Skill

This 6-step guide covers hypothesis lock, assumptions check, test type selection, and metrics definition. Includes 5 mandatory gates to ensure validity and prevent common pitfalls.

Quick Start

  1. 1Formulate a hypothesis using the quality checklist (observation, specific change, directional expectation, defined audience, measurable success).
  2. 2Lock the hypothesis by specifying target audience, primary metric, expected direction, and minimum detectable effect.
  3. 3Select the simplest valid test type (A/B, A/B/n, MVT, or split URL) and define your primary metric before launch.
Example Command
ab-test-setup --hypothesis 'Increase signups by 10%' --metric conversion_rate --mde 5%

Core Capabilities

Hypothesis Lock

Enforce a hard gate that requires a final hypothesis with target audience, primary metric, expected direction, and MDE before proceeding.

Assumptions & Validity Check

Explicitly list and verify assumptions about traffic stability, user independence, metric reliability, randomization quality, and external factors.

Test Type Selection

Choose the simplest valid test type (A/B, A/B/n, MVT, or split URL) based on your change and traffic volume.

Metrics Definition

Define a single primary metric directly tied to the hypothesis, frozen before launch, to evaluate success without peeking.

Usage Examples

Before

Running test without hypothesis lock leads to ambiguous results and risk of peeking.

After

Using hypothesis lock ensures a clear, measurable goal and prevents premature decisions.

Input

Assumptions check: Traffic stable? Yes. User independence? Yes. Metric reliable? Yes. Randomization quality? Yes. External factors? No major campaigns.

Output

All assumptions validated. Test can proceed safely.

Input

Test type selection: Single change, two variants, moderate traffic.

Output

Recommended: A/B test. Default to A/B unless clear reason for more complex design.

SKILL.md

---
name: ab-test-setup
description: "Structured guide for setting up A/B tests with mandatory gates for hypothesis, metrics, and execution readiness."
risk: unknown
source: community
date_added: "2026-02-27"
---

# A/B Test Setup

## 1️⃣ Purpose & Scope

Ensure every A/B test is **valid, rigorous, and safe** before a single line of code is written.

- Prevents "peeking"
- Enforces statistical power
- Blocks invalid hypotheses

---

## 2️⃣ Pre-Requisites

You must have:

- A clear user problem
- Access to an analytics source
- Roughly estimated traffic volume

### Hypothesis Quality Checklist

A valid hypothesis includes:

- Observation or evidence
- Single, specific change
- Directional expectation
- Defined audience
- Measurable success criteria

---

### 3️⃣ Hypothesis Lock (Hard Gate)

Before designing variants or metrics, you MUST:

- Present the **final hypothesis**
- Specify:
  - Target audience
  - Primary metric
  - Expected direction of effect
  - Minimum Detectable Effect (MDE)

Ask explicitly:

> “Is this the final hypothesis we are committing to for this test?”

**Do NOT proceed until confirmed.**

---

### 4️⃣ Assumptions & Validity Check (Mandatory)

Explicitly list assumptions about:

- Traffic stability
- User independence
- Metric reliability
- Randomization quality
- External factors (seasonality, campaigns, releases)

If assumptions are weak or violated:

- Warn the user
- Recommend delaying or redesigning the test

---

### 5️⃣ Test Type Selection

Choose the simplest valid test:

- **A/B Test** – single change, two variants
- **A/B/n Test** – multiple variants, higher traffic required
- **Multivariate Test (MVT)** – interaction effects, very high traffic
- **Split URL Test** – major structural changes

Default to **A/B** unless there is a clear reason otherwise.

---

### 6️⃣ Metrics Definition

#### Primary Metric (Mandatory)

- Single metric used to evaluate success
- Directly tied to the hypothesis
- Pre-defined and frozen before launch

###

Frequently Asked Questions

FAQ

What tools are compatible with this A/B test setup guide?
This guide is tool-agnostic; it works with any analytics platform like Google Analytics, Mixpanel, Optimizely, or custom solutions.
Who is this skill for?
Developers, product managers, and data scientists who need a rigorous framework for A/B testing to ensure valid and reliable results.
How is this different from other A/B testing guides?
It enforces mandatory gates like hypothesis lock and assumptions check before execution, preventing common pitfalls like peeking, invalid metrics, and underpowered tests.
Is this guide available in multiple languages?
Currently only in English, but the concepts are universal and can be applied regardless of language.
What results can I expect after applying this skill?
You will run valid A/B tests with clear hypotheses, pre-defined metrics, and reduced risk of false positives, leading to reliable insights and confident decisions.

Discussion

Discussion

0 comments
U

On This Page

Trigger Phrases

Use these phrases to activate this skill in your AI coding assistant:

set up A/B testhypothesis lockrun experimentA/B testing guidetest validitystatistical power