A/B TEST DESIGN SCENARIO

Product Context: You’re at NetworkPro, a B2B SaaS platform for sales team coordination. The platform has strong network effects: more users from a company = more value.

Current Metrics:

• Daily active users: ~5,000
• Average company size: 12 users
• Free → paid conversion: 8%
• Revenue: $29/user/month

Proposed Change: Team Invitation Incentive - 20% discount for inviter and invitee if they upgrade within 30 days.

Your Task: Design the A/B test addressing randomization strategy, sample size with clustering, and network effect complications.

EVALUATION AND FEEDBACK

Key Insight: Cluster Randomization Required

With network effects, individual user randomization creates contamination. Company-level randomization is essential.

Design Effect Calculation:

# For cluster randomized trial
avg_company_size = 12
icc = 0.15  # Intra-cluster correlation
design_effect = 1 + (avg_company_size - 1) * icc  # = 2.65

# Base sample size (individual)
from scipy.stats import norm
import math

baseline = 0.08
target = 0.092  # 15% relative lift
z_alpha, z_beta = 1.96, 0.84

pooled_p = (baseline + target) / 2
n_base = ((z_alpha + z_beta)**2 * 2 * pooled_p * (1-pooled_p)) / (target - baseline)**2
# n_base ≈ 10,900

# Adjusted for clustering
n_adjusted = n_base * design_effect  # ≈ 28,885 per arm
companies_needed = n_adjusted / avg_company_size  # ≈ 2,407 per arm

Duration: ~12 days enrollment + 30 days observation = 42 days total

OVERALL SCORE: 87/100

Excellent On: Cluster randomization choice, mixed-effects model specification, risk mitigation
Practice More: Sample size arithmetic, articulating ICC assumptions