Experiment Preferences jobs-apply

Event-driven A/B testing with auto-promotion will produce measurable conversion improvements vs the stalled static-JSON system

System fully operational. 18 events verified in Neon via browser test. 8 experiments running. Consent gate unified. Threshold counters incrementing. A

saasab-testingconversioncareer
Hypothesis

Event-driven A/B testing with auto-promotion will produce measurable conversion improvements vs the stalled static-JSON system

Result: confirmed
Key Findings

System fully operational. 18 events verified in Neon via browser test. 8 experiments running. Consent gate unified. Threshold counters incrementing. Auto-evaluation and promotion wired end-to-end.

Changelog

DateSummary
2026-04-27Event-driven rebuild shipped. 15-task plan executed. All 8 experiments running. Result: confirmed.
2026-04-239-issue audit revealed system 100% non-operational. Consent gate split-brain, 7/8 paused.
2026-04-06Audited: added Changelog, domain tags, stamped last_audited
2026-03-28Initial creation

Hypothesis

A/B testing marketing page copy and layout will improve conversion rate. The original system (shipped 2026-03-28) used a static JSON config with cron-based evaluation, but a 9-issue audit on 2026-04-23 revealed it was completely non-operational: consent gate split-brain dropped 100% of events, 7/8 experiments were paused, the evaluator had never run, conversion goals targeted a dead waitlist_submit metric.

The hypothesis evolved: can an event-driven architecture with threshold-triggered evaluation and auto-promotion make A/B testing truly autonomous?

Method

Architecture (event-driven, shipped 2026-04-27):

  • Experiments live in the experiments DB table (not static JSON)
  • Middleware reads running experiments via 60s TTL cache (edge-compatible)
  • Event route increments event_count per experiment on every analytics event
  • When event_count crosses eval_threshold, fires background evaluation
  • Evaluation: Beta-Binomial posterior (20k MCMC), 6-tier stopping rules
  • Auto-promotion: conclude → determine new control → dequeue challenger → create successor
  • Daily cron as safety net

8 experiments running across all marketing pages (ToFu/MoFu/BoFu).

Results

System confirmed operational via browser interaction test (2026-04-27):

  • 18 events landed in Neon from a single browsing session
  • 8 experiment assignments created
  • event_count incrementing on all experiments
  • Evaluation fires automatically on threshold crossing

Findings

  1. The original system was 100% broken for 30 days. Zero events collected. Zero evaluations run.
  2. Event-driven is correct for low-traffic A/B. Threshold triggers evaluate when data exists.
  3. Edge runtime constraints matter. Next.js middleware can’t import node
    .
  4. Auto-promotion eliminates the human bottleneck.