Hypothesis Test
A formal bet: state what you expect, run the experiment, and let the data confirm or reject it.
A hypothesis test is a formal structure for deciding whether observed data provides enough evidence to support a claim. State two opposing positions: the null hypothesis (H0: “nothing interesting is happening”) and the alternative (H1: “something is happening”). Collect data and compute how unlikely your result would be under H0 : that’s the p-value. If p < 0.05, you reject H0 and accept H1. Critical caveat: you never prove a hypothesis true; you only accumulate enough evidence to reject the alternative. No hypothesis, no experiment : just tinkering.
How It Works
Define H0 and H1 → choose significance threshold (typically 0.05) → run experiment → compute p-value → if p < threshold, reject H0. Common traps: peeking early, testing multiple metrics without correction, and treating “not significant” as “definitely no effect.”
Example
The oil model tests “expanding the MC time window to 360 days will improve forward accuracy.” The LinkedIn engine tests “adding a hydration wait will increase form submission rate.” Every experiment file starts with the hypothesis. Without it, you can’t evaluate the result. Framework for A/B tests and statistical significance.