A/B Testing Implementation

Meta-Description: A/B Testing für Web-Anwendungen. Feature Flags, statistische Signifikanz und Testing-Frameworks mit Next.js.

Keywords: A/B Testing, Split Testing, Feature Flags, Experimentation, Statistical Significance, Conversion Optimization

Einführung

A/B Testing ist essentiell für datengetriebene Entscheidungen. Mit Feature Flags, statistischer Auswertung und modernen Testing-Tools können Hypothesen validiert und Conversions optimiert werden. Dieser Guide zeigt Implementation mit Next.js.

A/B Testing Overview

┌─────────────────────────────────────────────────────────────┐
│              A/B TESTING ARCHITECTURE                        │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Experiment Flow:                                           │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  User Visit                                         │   │
│  │      ↓                                              │   │
│  │  Random Assignment (Hash-based)                     │   │
│  │      ↓                                              │   │
│  │  ┌──────────────┬──────────────┐                   │   │
│  │  │  Control (A) │ Variant (B)  │                   │   │
│  │  │    50%       │    50%       │                   │   │
│  │  └──────┬───────┴──────┬───────┘                   │   │
│  │         ↓              ↓                            │   │
│  │    Track Events   Track Events                      │   │
│  │         ↓              ↓                            │   │
│  │    ┌──────────────────────────┐                    │   │
│  │    │   Statistical Analysis   │                    │   │
│  │    │   (Significance Test)    │                    │   │
│  │    └──────────────────────────┘                    │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│  Key Metrics:                                               │
│  ├── Conversion Rate                                       │
│  ├── Click-Through Rate (CTR)                              │
│  ├── Revenue per Visitor                                   │
│  ├── Time on Page                                          │
│  └── Bounce Rate                                           │
│                                                             │
│  Statistical Requirements:                                  │
│  ├── Sample Size: Min 1000 per variant                     │
│  ├── Significance Level: 95% (p < 0.05)                    │
│  ├── Power: 80% minimum                                    │
│  └── Test Duration: Min 1-2 weeks                          │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Custom A/B Testing System

// lib/ab-testing/types.ts
export interface Experiment {
  id: string;
  name: string;
  description: string;
  variants: Variant[];
  targetingRules?: TargetingRule[];
  status: 'draft' | 'running' | 'paused' | 'completed';
  startDate?: Date;
  endDate?: Date;
  metrics: string[];
}

export interface Variant {
  id: string;
  name: string;
  weight: number; // 0-100
  config: Record<string, unknown>;
}

export interface TargetingRule {
  type: 'percentage' | 'userProperty' | 'url' | 'device';
  operator: 'equals' | 'contains' | 'greaterThan' | 'lessThan';
  value: string | number;
}

export interface ExperimentAssignment {
  experimentId: string;
  variantId: string;
  userId: string;
  assignedAt: Date;
}

export interface ExperimentEvent {
  experimentId: string;
  variantId: string;
  userId: string;
  eventName: string;
  eventValue?: number;
  timestamp: Date;
  metadata?: Record<string, unknown>;
}

// lib/ab-testing/assignment.ts
import { createHash } from 'crypto';

export function assignVariant(
  userId: string,
  experiment: Experiment
): Variant {
  // Deterministic hash-based assignment
  const hash = createHash('md5')
    .update(`${experiment.id}:${userId}`)
    .digest('hex');

  // Convert first 8 chars to number (0-100)
  const hashValue = parseInt(hash.substring(0, 8), 16) % 100;

  // Assign based on weights
  let cumulativeWeight = 0;
  for (const variant of experiment.variants) {
    cumulativeWeight += variant.weight;
    if (hashValue < cumulativeWeight) {
      return variant;
    }
  }

  // Fallback to first variant
  return experiment.variants[0];
}

// Check if user matches targeting rules
export function matchesTargeting(
  user: { id: string; properties?: Record<string, unknown> },
  rules: TargetingRule[]
): boolean {
  if (!rules || rules.length === 0) return true;

  return rules.every(rule => {
    switch (rule.type) {
      case 'percentage':
        const hash = createHash('md5').update(user.id).digest('hex');
        const value = parseInt(hash.substring(0, 8), 16) % 100;
        return value < (rule.value as number);

      case 'userProperty':
        const propValue = user.properties?.[rule.operator];
        return propValue === rule.value;

      default:
        return true;
    }
  });
}

// lib/ab-testing/client.ts
import { Experiment, Variant, ExperimentEvent } from './types';
import { assignVariant, matchesTargeting } from './assignment';

class ABTestingClient {
  private experiments: Map<string, Experiment> = new Map();
  private assignments: Map<string, Variant> = new Map();
  private userId: string;
  private eventQueue: ExperimentEvent[] = [];

  constructor(userId: string) {
    this.userId = userId;
  }

  async loadExperiments(): Promise<void> {
    const response = await fetch('/api/experiments');
    const experiments: Experiment[] = await response.json();

    experiments
      .filter(exp => exp.status === 'running')
      .forEach(exp => this.experiments.set(exp.id, exp));
  }

  getVariant(experimentId: string): Variant | null {
    // Check cache first
    const cached = this.assignments.get(experimentId);
    if (cached) return cached;

    const experiment = this.experiments.get(experimentId);
    if (!experiment) return null;

    // Check targeting
    if (!matchesTargeting({ id: this.userId }, experiment.targetingRules || [])) {
      return null;
    }

    // Assign variant
    const variant = assignVariant(this.userId, experiment);
    this.assignments.set(experimentId, variant);

    // Track assignment
    this.trackEvent(experimentId, 'experiment_viewed');

    return variant;
  }

  trackEvent(
    experimentId: string,
    eventName: string,
    eventValue?: number,
    metadata?: Record<string, unknown>
  ): void {
    const variant = this.assignments.get(experimentId);
    if (!variant) return;

    const event: ExperimentEvent = {
      experimentId,
      variantId: variant.id,
      userId: this.userId,
      eventName,
      eventValue,
      timestamp: new Date(),
      metadata
    };

    this.eventQueue.push(event);
    this.flushEvents();
  }

  private flushTimeout: NodeJS.Timeout | null = null;

  private flushEvents(): void {
    if (this.flushTimeout) return;

    this.flushTimeout = setTimeout(async () => {
      if (this.eventQueue.length === 0) return;

      const events = [...this.eventQueue];
      this.eventQueue = [];

      try {
        await fetch('/api/experiments/events', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify(events)
        });
      } catch (error) {
        // Re-queue on failure
        this.eventQueue.unshift(...events);
      }

      this.flushTimeout = null;
    }, 1000);
  }
}

// Singleton
let client: ABTestingClient | null = null;

export function getABClient(userId: string): ABTestingClient {
  if (!client || client['userId'] !== userId) {
    client = new ABTestingClient(userId);
  }
  return client;
}

React Integration

// contexts/ABTestingContext.tsx
'use client';

import { createContext, useContext, useEffect, useState, ReactNode } from 'react';
import { getABClient } from '@/lib/ab-testing/client';
import { Variant } from '@/lib/ab-testing/types';

interface ABTestingContextValue {
  getVariant: (experimentId: string) => Variant | null;
  trackConversion: (experimentId: string, eventName: string, value?: number) => void;
  isReady: boolean;
}

const ABTestingContext = createContext<ABTestingContextValue | null>(null);

export function ABTestingProvider({
  children,
  userId
}: {
  children: ReactNode;
  userId: string;
}) {
  const [isReady, setIsReady] = useState(false);
  const [client, setClient] = useState(() => getABClient(userId));

  useEffect(() => {
    client.loadExperiments().then(() => setIsReady(true));
  }, [client]);

  const getVariant = (experimentId: string) => {
    return client.getVariant(experimentId);
  };

  const trackConversion = (experimentId: string, eventName: string, value?: number) => {
    client.trackEvent(experimentId, eventName, value);
  };

  return (
    <ABTestingContext.Provider value={{ getVariant, trackConversion, isReady }}>
      {children}
    </ABTestingContext.Provider>
  );
}

export function useABTesting() {
  const context = useContext(ABTestingContext);
  if (!context) {
    throw new Error('useABTesting must be used within ABTestingProvider');
  }
  return context;
}

// hooks/useExperiment.ts
export function useExperiment<T = unknown>(
  experimentId: string,
  defaultValue: T
): { variant: T; isLoading: boolean; trackConversion: (event: string, value?: number) => void } {
  const { getVariant, trackConversion, isReady } = useABTesting();

  const variant = getVariant(experimentId);
  const value = variant?.config as T ?? defaultValue;

  return {
    variant: value,
    isLoading: !isReady,
    trackConversion: (event: string, value?: number) =>
      trackConversion(experimentId, event, value)
  };
}

// components/ABTest.tsx
'use client';

import { useExperiment } from '@/hooks/useExperiment';
import { ReactNode } from 'react';

interface ABTestProps {
  experimentId: string;
  control: ReactNode;
  variants: Record<string, ReactNode>;
  fallback?: ReactNode;
}

export function ABTest({
  experimentId,
  control,
  variants,
  fallback = null
}: ABTestProps) {
  const { variant, isLoading } = useExperiment<{ variantId: string }>(
    experimentId,
    { variantId: 'control' }
  );

  if (isLoading) return fallback;

  if (variant.variantId === 'control') {
    return <>{control}</>;
  }

  return <>{variants[variant.variantId] || control}</>;
}

// Usage Example
export function CTASection() {
  return (
    <ABTest
      experimentId="cta-button-color"
      control={<CTAButton color="blue">Get Started</CTAButton>}
      variants={{
        green: <CTAButton color="green">Get Started</CTAButton>,
        orange: <CTAButton color="orange">Start Free Trial</CTAButton>
      }}
    />
  );
}

// Tracked CTA Button
function CTAButton({
  color,
  children
}: {
  color: string;
  children: ReactNode;
}) {
  const { trackConversion } = useExperiment('cta-button-color', {});

  const handleClick = () => {
    trackConversion('cta_clicked');
    // Navigate or perform action
  };

  return (
    <button
      onClick={handleClick}
      className={`btn bg-${color}-600 text-white`}
    >
      {children}
    </button>
  );
}

Statistical Analysis

// lib/ab-testing/statistics.ts

interface ExperimentResults {
  control: {
    visitors: number;
    conversions: number;
    conversionRate: number;
  };
  variant: {
    visitors: number;
    conversions: number;
    conversionRate: number;
  };
  lift: number;
  significance: number;
  isSignificant: boolean;
  confidenceInterval: [number, number];
  requiredSampleSize: number;
}

// Z-Score for confidence levels
const Z_SCORES: Record<number, number> = {
  0.90: 1.645,
  0.95: 1.96,
  0.99: 2.576
};

export function calculateResults(
  controlVisitors: number,
  controlConversions: number,
  variantVisitors: number,
  variantConversions: number,
  confidenceLevel: number = 0.95
): ExperimentResults {
  const controlRate = controlConversions / controlVisitors;
  const variantRate = variantConversions / variantVisitors;

  // Lift
  const lift = ((variantRate - controlRate) / controlRate) * 100;

  // Standard Error
  const se = Math.sqrt(
    (controlRate * (1 - controlRate)) / controlVisitors +
    (variantRate * (1 - variantRate)) / variantVisitors
  );

  // Z-Score
  const zScore = (variantRate - controlRate) / se;

  // P-Value (two-tailed)
  const pValue = 2 * (1 - normalCDF(Math.abs(zScore)));

  // Significance
  const significance = (1 - pValue) * 100;
  const isSignificant = pValue < (1 - confidenceLevel);

  // Confidence Interval
  const z = Z_SCORES[confidenceLevel] || 1.96;
  const marginOfError = z * se;
  const confidenceInterval: [number, number] = [
    (variantRate - controlRate - marginOfError) * 100,
    (variantRate - controlRate + marginOfError) * 100
  ];

  // Required Sample Size (for 80% power)
  const requiredSampleSize = calculateSampleSize(
    controlRate,
    controlRate * 1.1, // 10% MDE
    0.05,
    0.8
  );

  return {
    control: {
      visitors: controlVisitors,
      conversions: controlConversions,
      conversionRate: controlRate * 100
    },
    variant: {
      visitors: variantVisitors,
      conversions: variantConversions,
      conversionRate: variantRate * 100
    },
    lift,
    significance,
    isSignificant,
    confidenceInterval,
    requiredSampleSize
  };
}

// Normal CDF approximation
function normalCDF(x: number): number {
  const a1 = 0.254829592;
  const a2 = -0.284496736;
  const a3 = 1.421413741;
  const a4 = -1.453152027;
  const a5 = 1.061405429;
  const p = 0.3275911;

  const sign = x < 0 ? -1 : 1;
  x = Math.abs(x) / Math.sqrt(2);

  const t = 1.0 / (1.0 + p * x);
  const y = 1.0 - ((((a5 * t + a4) * t + a3) * t + a2) * t + a1) * t * Math.exp(-x * x);

  return 0.5 * (1.0 + sign * y);
}

// Sample Size Calculator
export function calculateSampleSize(
  baselineRate: number,
  expectedRate: number,
  alpha: number = 0.05,
  power: number = 0.8
): number {
  const zAlpha = Z_SCORES[1 - alpha] || 1.96;
  const zBeta = 0.84; // 80% power

  const p1 = baselineRate;
  const p2 = expectedRate;
  const pBar = (p1 + p2) / 2;

  const n = Math.ceil(
    2 * Math.pow(zAlpha * Math.sqrt(2 * pBar * (1 - pBar)) +
    zBeta * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2)), 2) /
    Math.pow(p1 - p2, 2)
  );

  return n;
}

Results Dashboard

// app/admin/experiments/[id]/page.tsx
'use client';

import { useState, useEffect } from 'react';
import { calculateResults, ExperimentResults } from '@/lib/ab-testing/statistics';

interface ExperimentData {
  experiment: Experiment;
  results: {
    control: { visitors: number; conversions: number };
    variants: Record<string, { visitors: number; conversions: number }>;
  };
}

export default function ExperimentResultsPage({
  params
}: {
  params: { id: string }
}) {
  const [data, setData] = useState<ExperimentData | null>(null);
  const [analysis, setAnalysis] = useState<ExperimentResults | null>(null);

  useEffect(() => {
    fetch(`/api/experiments/${params.id}/results`)
      .then(res => res.json())
      .then(setData);
  }, [params.id]);

  useEffect(() => {
    if (!data) return;

    const variant = Object.values(data.results.variants)[0];
    if (!variant) return;

    const results = calculateResults(
      data.results.control.visitors,
      data.results.control.conversions,
      variant.visitors,
      variant.conversions
    );
    setAnalysis(results);
  }, [data]);

  if (!data || !analysis) return <div>Loading...</div>;

  return (
    <div className="p-6 space-y-6">
      <h1 className="text-2xl font-bold">{data.experiment.name}</h1>

      {/* Results Cards */}
      <div className="grid grid-cols-3 gap-4">
        <ResultCard
          title="Control"
          visitors={analysis.control.visitors}
          conversions={analysis.control.conversions}
          rate={analysis.control.conversionRate}
        />
        <ResultCard
          title="Variant"
          visitors={analysis.variant.visitors}
          conversions={analysis.variant.conversions}
          rate={analysis.variant.conversionRate}
          isWinner={analysis.isSignificant && analysis.lift > 0}
        />
        <div className="bg-white p-4 rounded-lg shadow">
          <h3 className="text-sm text-gray-500">Lift</h3>
          <p className={`text-3xl font-bold ${analysis.lift > 0 ? 'text-green-600' : 'text-red-600'}`}>
            {analysis.lift > 0 ? '+' : ''}{analysis.lift.toFixed(2)}%
          </p>
        </div>
      </div>

      {/* Statistical Significance */}
      <div className="bg-white p-6 rounded-lg shadow">
        <h2 className="text-lg font-semibold mb-4">Statistical Analysis</h2>

        <div className="grid grid-cols-2 gap-4">
          <div>
            <p className="text-sm text-gray-500">Confidence Level</p>
            <p className="text-xl font-semibold">
              {analysis.significance.toFixed(1)}%
            </p>
          </div>
          <div>
            <p className="text-sm text-gray-500">Status</p>
            <span className={`px-3 py-1 rounded-full text-sm ${
              analysis.isSignificant
                ? 'bg-green-100 text-green-800'
                : 'bg-yellow-100 text-yellow-800'
            }`}>
              {analysis.isSignificant ? 'Significant' : 'Not Significant'}
            </span>
          </div>
          <div>
            <p className="text-sm text-gray-500">Confidence Interval</p>
            <p className="text-xl font-semibold">
              [{analysis.confidenceInterval[0].toFixed(2)}%, {analysis.confidenceInterval[1].toFixed(2)}%]
            </p>
          </div>
          <div>
            <p className="text-sm text-gray-500">Required Sample Size</p>
            <p className="text-xl font-semibold">
              {analysis.requiredSampleSize.toLocaleString()} per variant
            </p>
          </div>
        </div>
      </div>

      {/* Recommendation */}
      {analysis.isSignificant && (
        <div className={`p-4 rounded-lg ${
          analysis.lift > 0 ? 'bg-green-50 border border-green-200' : 'bg-red-50 border border-red-200'
        }`}>
          <h3 className="font-semibold">Recommendation</h3>
          <p>
            {analysis.lift > 0
              ? 'The variant significantly outperforms the control. Consider rolling out to 100% of users.'
              : 'The control outperforms the variant. Consider keeping the original version.'}
          </p>
        </div>
      )}
    </div>
  );
}

Best Practices

Fazit

A/B Testing erfordert:

1. Deterministic Assignment: Hash-basierte Zuweisung
2. Statistical Rigor: Signifikanz-Tests
3. Proper Tracking: Events und Conversions
4. Patience: Ausreichende Sample Size abwarten

Datengetriebene Entscheidungen führen zu besseren Produkten.

Bildprompts

1. "A/B testing dashboard showing variant comparison, charts and metrics"
2. "Statistical significance visualization, confidence intervals"
3. "Experiment funnel diagram, control vs variant flow"

Quellen

• Google Optimize Best Practices
• Evan Miller A/B Testing Calculator
• VWO A/B Testing Guide
• Statistical Methods in A/B Testing