Journal

2026-04-02

2026-04-02

Signal

Three of six scheduled vault cron jobs were silently broken : each for a different reason : revealing that cron jobs need their own validation layer because they fail silently in environments that differ from interactive shells (locked keychains, GNU-only flags, exit code conventions).

Evidence

  • Project: projects/dakka/_index : Full audit of all 6 cron/scheduled jobs; 3 were broken:
    • usage-probe: Exit 1 always (keychain locked in cron context); fix: graceful degradation to local data
    • obsidian-cron: Crashes at head -z (GNU-only flag on macOS) + grep without || true; fix: removed GNU head -z, guarded grep
    • vault-stale: Never fired, would log false failure; fix: exit code normalization (finding stale notes = success)
  • Additional hardening: Watchdog thresholds raised 1500→2000MB warn, 2500→3000MB kill; bloomnet-usage-probe.jsonl now has actual data (was empty since creation); all 6 jobs verified exit 0 via manual wrapper invocation
  • Project: projects/peon-notify/_index : Session backfill pipeline: _backfill_sessions() in peon-obsidian-cron.sh; first run indexed 286 sessions; vault_path fixed; _obsidian_dir() helper with dir_map config keys
  • Vault: Temporal enrichment spec : renamed 114 vault files with date prefixes; 93 living documents got ## Changelog sections; week ordering reversed in monthly reviews; 8 templates and 9 audit skills updated
  • New dimension: False Positives at topics/false-positives/ : 10 source systems, audit skill with 9 checks, integrated into lifecycle chain
  • ADRs linked: projects/dakka/adrs/2026-04-02-003-temporal-enrichment-spec, projects/peon-notify/adrs/2026-04-02-004-session-backfill-pipeline

So What (Why Should You Care)

Cron jobs fail silently in ways that interactive scripts don’t, and the three failure modes today illustrate exactly why. The cron environment is not your development environment : it has no keychain access, it runs as a different user, and it uses the system’s default tool versions rather than the ones installed by Homebrew.

The usage-probe failure (Exit 1 always in cron because the keychain was locked) shows the user-session dependency problem. The keychain is only unlocked for interactive sessions. A cron job that calls any macOS API requiring keychain access will fail silently every time it runs, logging nothing useful, until someone checks the exit code manually. The fix : graceful degradation to local data when the API is unreachable : makes the job succeed regardless of keychain state.

The obsidian-cron failure (crashes on head -z) shows the GNU/BSD tool divergence problem. macOS ships with BSD tools. Homebrew installs GNU tools. Your interactive shell has both, with GNU tools usually shadowing BSD tools. The cron environment typically uses system defaults, which means BSD tools. head -z (null delimiter) is a GNU extension that BSD head doesn’t support. Any script that relies on GNU-specific flags will fail silently in cron on macOS.

The vault-stale failure (exit code convention mismatch) shows the semantic gap between “job purpose” and “monitoring expectation.” Finding stale notes means the job ran successfully and found something : that’s a success by any reasonable definition. But the job was exiting with non-zero status when it found stale notes, which monitoring systems interpret as failure. The job was working correctly and being reported as broken.

The validation protocol that works: run each cron job manually via its wrapper script, check exit code, check output, verify behavior matches intent. Do this before you consider the job operational. The 286-session backfill after the session indexing fix is a concrete demonstration of what fixing pipeline plumbing reveals: two months of session data that had been silently dropping the entire time.

What’s Next

  • Monitor all 6 cron jobs through their next scheduled run to confirm fixes hold
  • Continue vault dimension health audits using the now-functional cron pipeline

Log

  • projects/dakka/_index: full audit of all 6 cron/scheduled jobs; 3 were broken
  • usage-probe: Exit 1 always in cron (keychain locked); graceful degradation fix : writes local data when API unreachable
  • obsidian-cron: Crashes at head -z (GNU-only) + grep without || true; fix deployed
  • vault-stale: never fired, would log false failure; exit code normalization deployed
  • Watchdog memory thresholds raised 1500→2000MB warn, 2500→3000MB kill
  • bloomnet-usage-probe.jsonl now has actual data : was empty since creation
  • All 6 jobs verified exit 0 via manual wrapper invocation
  • projects/peon-notify/_index: _backfill_sessions() implemented in peon-obsidian-cron.sh (Step 2b)
  • Scans ~/.claude/projects/*.jsonl, extracts per-session metadata, writes to ~/vault/meta/sessions.jsonl
  • Cursor-based incremental, capped at 200/run; first run indexed 286 sessions
  • Fixed peon.json vault_path from ~/Documents/Obsidian to ~/vault
  • Added _obsidian_dir() helper with dir_map_* config keys for directory mapping
  • Vault temporal enrichment spec: renamed 114 files with date prefixes
  • 93 living documents got ## Changelog sections
  • Week ordering reversed in monthly reviews (newest first)
  • 8 templates and 9 audit skills updated
  • New dimension: False Positives at topics/false-positives/; 10 source systems; audit skill (9 checks); integrated into lifecycle chain
  • Bugs fixed: topics/bugs/2026-04-02-obsidian-cron-head-z-gnu-only, topics/bugs/2026-04-02-usage-probe-keychain-locked-in-cron, topics/bugs/2026-04-02-obsidian-cron-grep-status-missing-field