Heaviest data day: 64MB, four projects shipping in parallel
2026-04-05
Signal
The heaviest single-day data volume yet (64MB across main sessions alone) driven by four projects shipping in parallel: rusty-bloomnet completed its SQLite migration, the oil model leapt from v21 to v22 with Monte Carlo forward calibration, jobs-apply deployed a full E2E test suite, and the internal agentic crawl crossed the two-thirds mark.
Evidence
- rusty-bloomnet: SQLite migration complete via new
rusty-datacrate:bloomnet.db(7.9MB) anddakka.dbcreated, 218 tests passing across 75 sessions - oil model v21: full universe scanner covering 1,640 instruments, 1,209 identified with edge
- oil model v22: Monte Carlo forward calibration with throughput trend and fatigue-adjusted jumps: Brier delta improved from +5.25pp to +7.19pp vs Kalshi
- oil: live Robinhood/Kalshi and USO options data populated
- jobs-apply: E2E dashboard test suite shipped: 13 specs, 31 scenarios, all 11 pages covered
- jobs-apply: Cloudflare WAF bypass: resume upload switched to JSON+base64, path renamed
/api/profiles/importto/parse-resume, JSON fields renamed to evade content inspection - jobs-apply:
pdf-parsereplaced with native PDF vision for resume import - internal audit pipeline: agentic crawl A/B test reached 610/914 venues crawled with DQI comparison complete
So What (Why Should You Care)
The SQLite migration in rusty-bloomnet is the persistence layer that all downstream analytics depend on. Moving from flat files to structured databases (bloomnet.db + dakka.db) means the developer analytics dashboard can now query historical data efficiently, and the 218-test harness locks in correctness before any further schema changes land. This is infrastructure that compounds.
The oil model jumping two versions in a single day signals that the feedback loop between instrument scanning and probabilistic calibration is tightening. A +7.19pp Brier improvement over Kalshi is not noise; it means the model’s forward curve is materially more accurate than the prediction market’s implied pricing, which is the entire thesis for the project.
On the jobs-apply side, the E2E suite and WAF bypass fixes are defensive investments. The test suite locks in 31 scenarios that would otherwise regress silently, and the WAF workarounds (base64 encoding, path/field renaming) solve a class of Cloudflare false-positive blocks that were silently dropping resume uploads. Both reduce the surface area for undetected failures.
What’s Next
- Wire rusty-bloomnet’s
bloomnet.dbinto the live dashboard data pipeline, replacing the JSON flat-file ingest - Oil model v23: integrate the auto-refresh launchd cron (12h cadence) so forward calibration updates without manual runs
- Jobs-apply: expand E2E coverage to channel-specific flows (LinkedIn, Greenhouse, Workday) beyond the dashboard pages
- internal agentic crawl: push past 610/914 toward full coverage and begin the Sonnet-vs-Node DQI comparison analysis
- Validate WAF bypass durability: monitor for Cloudflare rule updates that could re-block the renamed paths
Log
Session 1: rusty-bloomnet SQLite Migration (75 sessions, ~2.9MB)
- rusty-data crate: complete SQLite migration
- bloomnet.db (7.9MB), dakka.db created
- 218 tests passing across the crate
Session 2: Oil Model (10 sessions)
- v21: Full universe scanner: 1,640 instruments, 1,209 with edge
- v22: MC forward calibration: throughput trend + fatigue-adjusted jumps
- v22 ratchet: Brier delta +5.25pp -> +7.19pp vs Kalshi
- Populate live Robinhood/Kalshi + USO options data
Session 3: Jobs Apply (48 sessions, ~1.5MB)
- E2E dashboard test suite: 13 specs, 31 scenarios, all 11 pages
- Cloudflare WAF bypass: resume upload switched to JSON+base64
- Path rename: /api/profiles/import -> /parse-resume (WAF blocks “import”)
- Field rename: JSON fields renamed to bypass WAF content inspection
- Resume import: replaced pdf-parse with native PDF vision
Session 4: internal audit pipeline (5 sessions)
- Agentic crawl A/B test: 610/914 venues crawled, DQI comparison complete
Session 5: Main Sessions (42 sessions, ~64MB)
- Vault maintenance and cross-project coordination