Pitfall Preferences jobs-apply

Lever Networkidle Timeout Cascade

pitfallplaywrightlevernetworking

What Happened

The Lever channel had zero job discoveries across every session since the project’s DB wipe on Apr 11. 35 boards configured, each with dozens to hundreds of postings. Every session reported 0 discovered, 0 matched, 0 submitted. One session ran 44 cycles over 10 hours and still found nothing. The title pre-filter fix (bb7a544) was committed on Apr 13 to solve this, but Lever remained dead.

When we finally got terminal logs (the Electron app swallows console output), the error cascade was clear:

[Lever] Scanning board: Palantir: net::ERR_TIMED_OUT
[Lever] Scanning board: ShieldAI: Navigation interrupted by another navigation to "chrome-error://chromewebdata/"
[Lever] Scanning board: Zoox: Navigation interrupted by another navigation to "https://jobs.lever.co/shieldai"
[Lever] Scanning board: Hive: Navigation interrupted by another navigation to "https://jobs.lever.co/zoox"

The first board timed out. Every subsequent board crashed because the previous timed-out navigation was still pending on the shared page, causing a navigation collision.

Root Cause

navigateTo() used waitUntil: 'networkidle', which waits for the network to have zero requests for 500ms. Lever board pages maintain persistent WebSocket and polling connections that never go idle. Playwright waited for the full timeout (30s default), then threw. The shared this.page was left in a broken navigation state. Each subsequent page.goto() collided with the pending navigation from the previous board, producing “Navigation interrupted” errors. The try/catch in searchBoard() swallowed every error silently, so each board failed independently and the adapter returned 0 postings total.

How to Avoid

Use waitUntil: 'domcontentloaded' for server-rendered pages. Lever renders all posting HTML on the server; the DOM is complete before any async WebSocket connections fire. There is no reason to wait for network idle on a page where the data you need is in the initial HTML.

// Before (hangs forever on Lever):
await this.page.goto(boardUrl, { waitUntil: 'networkidle' });

// After (completes in <1s):
await this.page.goto(boardUrl, { waitUntil: 'domcontentloaded', timeout: 30_000 });

After the fix: 908 postings found from just 8 boards. Palantir alone had 248.

The broader lesson: networkidle is a heuristic for SPAs that hydrate client-side. For server-rendered pages, it is an anti-pattern that turns fast loads into timeouts. Always match the wait strategy to the page’s rendering model.