Quantitative Audit
Multi-round validation: check internal consistency, then forward accuracy, then external benchmarks.
A quantitative audit validates numerical outputs in three escalating rounds that each catch what the previous missed. Round 1 (internal consistency) checks whether outputs are self-consistent: do totals add up, are distributions normalized, are there impossible jumps? Round 2 (forward accuracy) compares predictions against ground truth: MAE, MAPE, R², confusion matrices. Round 3 (external benchmarking) compares against an independent standard : a prediction market, published benchmark, or human expert. A system that passes all three rounds has earned production trust at every level of scrutiny.
How It Works
Each round is a filter: Round 1 catches math bugs, Round 2 catches model drift, Round 3 catches systematic bias that only an external reference reveals. Failure at Round 1 may inflate Round 2 scores (bugs attributed to model error). Always run in order.
Example
Oil v16’s 3-round audit: Round 1 found two normalization bugs (MC probabilities didn’t sum to 1.0). Round 2 confirmed MAE $1.12, MAPE 2.26%, R² 0.913 over 34 days. Round 3 benchmarked against Polymarket: +2.16pp Brier edge. Three rounds, three different classes of problem found. Details in Oil v16.