2026-03-23

Signal

When enrichment files were missing expected data, comparing the enrichment prompt directly against actual output: rather than comparing output against expected output: exposed a systematic field naming mismatch: the bug wasn’t in the logic, it was in the contract between what the prompt instructed and what the parser expected.

Evidence

• Project: internal audit: Two sessions investigating missing enrichment data; compared enrichment prompt against actual LLM output field by field
• Root cause identified: LLM output field names didn’t match what the downstream parser expected: a contract mismatch rather than a logic failure
• Debug methodology: Went upstream to the prompt (the API contract) rather than downstream to the output (the symptom), which found the real cause faster
• New events investigated: Checked new events added to the pipeline for the same mismatch pattern
• Project: projects/jobs-apply/_index: Iteration 2 fixes deployed and run; separate familiarization session to read documentation and Claude file before starting work
• Volume: 76 automated code-review sessions (autosearch: 31, internal audit pipeline: 24, jobs-apply: 21)

So What (Why Should You Care)

LLM-based enrichment pipelines have a failure mode that traditional software doesn’t: the prompt is the API contract, and prompts can drift. When a prompt changes slightly: or when the model’s output format shifts across versions: the field names it produces may no longer match what the downstream parser expects. The data doesn’t error out; it silently disappears. The enrichment pipeline runs, the LLM produces output, the parser reads its expected fields and finds nothing.

The debug approach that works here is different from debugging a logic error. For a logic error, you check the code. For a prompt-parser contract mismatch, you check the prompt against the parser side-by-side: field by field, name by name: to find where the contract broke. This is the LLM equivalent of checking API schema compatibility before integration.

The broader lesson: in any system that uses LLMs as data transformers, the prompt is infrastructure, not configuration. Changes to prompts need the same versioning, review, and compatibility testing that API schema changes need. The field naming mismatch today happened because the prompt was treated as easy-to-edit natural language rather than as a contract with a typed interface.

The fact that the enrichment pipeline continued to run after the field naming mismatch: producing empty enrichment files rather than errors: is what made the bug hard to find. A traditional API returning a 400 error on a malformed request is immediately visible. An LLM producing output that uses different field names than the parser expects produces no error at all; it produces empty output, which looks the same as “nothing to enrich.” Silent failures are always harder to detect than loud failures. Building explicit field validation between prompt output and parser expectations is the change that makes this class of failure loud.

The separate familiarization session before starting work on projects/jobs-apply/_index: reading documentation and Claude file before touching any code: is a practice worth noting. It takes 15-20 minutes but significantly reduces the probability of making a change that conflicts with a decision already made, or repeating work that’s already been done. On a project that’s actively changing across multiple sessions, this context-loading step is the difference between informed iteration and uninformed exploration.

76 automated code-review sessions across three projects (autosearch: 31, internal audit pipeline: 24, jobs-apply: 21) reflects a day of maintenance and debugging rather than major feature development: smaller session counts per project suggest targeted fixes rather than broad changes.

What’s Next

• Fix the field naming mismatch in the enrichment pipeline
• Add field name validation between prompt output and parser expectations
• Validate enrichment pipeline output against full event set after fix

Log

• internal audit: two sessions investigating missing enrichment data
• Compared enrichment prompt against actual LLM output field by field
• Identified systematic field naming mismatch between prompt output and parser expectations
• Debug methodology: went to the prompt (the contract) rather than the output (the symptom)
• Investigated new events added to the pipeline for the same mismatch pattern
• projects/jobs-apply/_index: iteration 2 fixes deployed and run
• Separate familiarization session: read documentation and Claude file before starting work
• 76 automated code-review sessions (autosearch: 31, internal audit pipeline: 24, jobs-apply: 21)