audit-lifecycle-chain

Changelog

260428: provenance chain check

• Added check 9: provenance chain integrity for lifecycle nodes (WARN, ERROR after 2026-05-28). Flags llm-synthesized nodes without validated_by field.

260420: multiple edits

• v_migrate: Changelog migrated from table to YYMMDD H3 format per versioning-standard rule 2 (V1.6 of skills upgrade plan)
• v6: Added license, sources per V6.1/V6.2 of skills upgrade plan.
• v1.5: Added ## Quality Checks section per V1.5 of ~/vault/plans/2026-04-20-vault-skills-upgrade-plan.md

260406: v1.2: Added check 8 (PII and secrets scan per _config/pii-rules.yaml)

260403: multiple edits

• Added Agent to mounted_allowed_tools + post-audit visual enrichment trigger
• Added Visual Enrichment section + self-improving-agent-patterns cross-reference

260402: v1.1: Breakthrough endpoint is now directory-based (breakthroughs/). Chain ends at type: breakthrough note, not just #breakthrough tag.

260401: multiple edits

• Added false-positive → idea lifecycle edge
• Updated experiment path patterns for date-prefix naming (temporal enrichment spec)
• Initial creation

---

Description

Cross-cutting audit skill that validates the vault’s lifecycle chain end-to-end. The lifecycle chain is the vault’s core value proposition: it connects raw lessons to codified knowledge through a defined progression:

pitfall → idea (addresses_pitfalls) → experiment (idea_source) → skill (experiments) → breakthroughs/ note
                                                                                         ↑
false-positive → idea (FP triggers detection system redesign) ──────────────────────────┘

False positive integration (2026-04-01): FPs feed the lifecycle when a recurring false alarm motivates an idea to redesign the detection system. The edge is: false-positive.source_system → idea.addresses_pitfalls (where the pitfall is the noisy detector). If the idea is promoted to an experiment that succeeds, it becomes a skill or breakthrough in detection calibration.

Each dimension has its own audit skill that checks internal consistency. This skill checks the connections between dimensions: the edges that form the chain. Without these edges, the vault is a collection of isolated notes rather than a connected knowledge graph.

Why this skill exists: The deep remediation audit (2026-04-01) discovered that the lifecycle chain had ZERO edges despite all dimensions being populated. All 17 experiments had idea_source: null, no idea referenced a pitfall, and no skill referenced source experiments. The chain existed in spec but not in data. This skill prevents that regression.

Interface

Trigger: Run after any bulk note creation, after remediation audits, or on a monthly cadence alongside the dimension-scorecard review.

Inputs:

• vault_path: root of the Obsidian vault (default ~/vault)

Outputs:

• chain_report: edge counts per link type, orphan node lists, complete chain traces
• gap_candidates: nodes that should have lifecycle connections based on project/topic overlap

Provenance

Created to measure the health of the vault’s canonical lifecycle pattern: pitfall → idea → experiment → skill → breakthrough. This chain is the spine of the knowledge-compile-wiki pattern (see skills/self-improving-agent-patterns Pattern 4): pitfalls seed ideas, ideas become experiments, successful experiments crystallize into skills, and skills that shift the baseline become breakthroughs. A broken chain means the vault isn’t compounding knowledge.

Inspired by Karpathy’s LLM-compiled knowledge base pattern (see research/2026-04-02-karpathy-llm-knowledge-base-pattern). Measurement methodology builds on graph theory edge-density metrics; orphan detection from classic data-lineage tooling.

Usage Notes

• Run monthly or after bulk-editing any dimension. Chain integrity can silently degrade when an experiment is deleted without its idea_source frontmatter being cleaned up elsewhere.
• Forward edges (pitfall → idea) and backward edges (breakthrough → source_note) are audited separately: one-way edges are a common drift.
• A chain-completion baseline ≥ 50% is the target; below that suggests the vault is accumulating isolated notes rather than compounding knowledge.
• Orphan nodes (e.g., a skill not linked from any experiment) get WARN severity; they’re often in-progress patterns that will get connected later.
• The canonical chain order is strict: pitfalls do NOT link directly to skills (they go through idea → experiment first). Skipping a stage is a chain violation.

What to Check

1. Forward Edges: Pitfalls → Ideas (WARN)

Count how many pitfalls are referenced by at least one idea’s addresses_pitfalls array.

echo "=== PITFALL → IDEA EDGES ==="
total_pitfalls=$(ls ~/vault/topics/pitfalls/*.md 2>/dev/null | wc -l | tr -d ' ')
addressed=0

for p in ~/vault/topics/pitfalls/*.md; do
  slug=$(basename "$p" .md)
  # Check if any idea references this pitfall
  if grep -rl "pitfalls/$slug" ~/vault/ideas/*.md >/dev/null 2>&1; then
    addressed=$((addressed + 1))
  fi
done

echo "Pitfalls addressed by ideas: $addressed / $total_pitfalls"
pct=$((addressed * 100 / total_pitfalls))
echo "Coverage: ${pct}%"
[ "$pct" -lt 30 ] && echo "WARN: <30% pitfall coverage: lifecycle chain is starved at the source"

Target: >=50% of pitfalls addressed by at least one idea.

2. Forward Edges: Ideas → Experiments (WARN)

Count how many ideas have been promoted to experiments (status=promoted, promoted_to set).

echo "=== IDEA → EXPERIMENT EDGES ==="
total_ideas=$(ls ~/vault/ideas/*.md 2>/dev/null | wc -l | tr -d ' ')
promoted=0

for f in ~/vault/ideas/*.md; do
  status=$(grep '^status:' "$f" | head -1 | sed 's/status: *//')
  promoted_to=$(grep '^promoted_to:' "$f" | head -1 | sed 's/promoted_to: *//')
  if [ "$status" = "promoted" ] && [ "$promoted_to" != "null" ] && [ -n "$promoted_to" ]; then
    promoted=$((promoted + 1))
  fi
done

echo "Ideas promoted to experiments: $promoted / $total_ideas"

Target: >=1 promoted idea (pipeline is flowing). WARN if all ideas are captured with none promoted.

3. Backward Edges: Experiments → Ideas (ERROR after 2026-04-01)

Every experiment must have idea_source set. For experiments created before 2026-04-01, "predates ideas dimension" is valid.

echo "=== EXPERIMENT → IDEA BACKWARD EDGES ==="
total=0; linked=0; missing=0

for f in ~/vault/experiments/**/*.md; do
  [ ! -f "$f" ] && continue
  total=$((total + 1))
  idea_src=$(grep '^idea_source:' "$f" | head -1 | sed 's/idea_source: *//')
  created=$(grep '^created:' "$f" | head -1 | sed 's/created: *//')

  if [ -z "$idea_src" ] || [ "$idea_src" = "null" ]; then
    echo "MISSING: $(basename "$(dirname "$f")")/$(basename "$f" .md) (created: $created)"
    missing=$((missing + 1))
  else
    linked=$((linked + 1))
  fi
done

echo "Experiments with idea_source: $linked / $total"
echo "Missing: $missing"

Target: 100% of experiments have idea_source set (wikilink or “predates ideas dimension”).

4. Forward Edges: Experiments → Skills (INFO)

Count experiments referenced by skills in their experiments array.

echo "=== EXPERIMENT → SKILL EDGES ==="
# Collect all experiment paths referenced by skills
skill_exp_refs=""
for f in ~/vault/skills/*.md; do
  grep -oP '\[\[experiments/[^\]]+\]\]' "$f" 2>/dev/null | sed 's/\[\[//;s/\]\]//' >> /tmp/skill_exp_refs.txt
done

total_experiments=$(find ~/vault/experiments -name "*.md" | wc -l | tr -d ' ')
referenced=$(sort -u /tmp/skill_exp_refs.txt 2>/dev/null | wc -l | tr -d ' ')

echo "Experiments referenced by skills: $referenced / $total_experiments"
rm -f /tmp/skill_exp_refs.txt

Target: Experiments that produced reusable patterns should be referenced by skill notes.

5. Breakthrough Directory + Tag Consistency (WARN)

Breakthroughs are now a directory-based dimension at breakthroughs/ with type: breakthrough notes. Source experiments/skills retain the #breakthrough tag for cross-referencing. Both must exist (dual-write model).

echo "=== BREAKTHROUGH DIRECTORY ==="
dir_count=$(ls ~/vault/breakthroughs/*.md 2>/dev/null | wc -l | tr -d ' ')
echo "Breakthrough notes in breakthroughs/: $dir_count"
[ "$dir_count" -lt 8 ] && echo "WARN: <8 breakthrough notes: high-impact wins are under-documented"

echo "=== BREAKTHROUGH TAG CROSS-CHECK ==="
tag_count=$(grep -rl 'breakthrough' ~/vault/experiments/ ~/vault/skills/ 2>/dev/null | while read f; do
  grep -A 20 '^tags:' "$f" | grep -q 'breakthrough' && echo "$f"
done | wc -l | tr -d ' ')
echo "Source notes tagged #breakthrough: $tag_count"

echo "=== ORPHAN CHECK ==="
# Every breakthroughs/ note should have a source_note that exists and is tagged
for f in ~/vault/breakthroughs/*.md; do
  source=$(grep 'source_note:' "$f" | head -1)
  echo "  $(basename $f): $source"
done

Target: >=8 breakthrough notes in breakthroughs/. Every breakthrough note has a valid source_note. Every tagged source note has a corresponding breakthroughs/ note.

6. Orphan Detection: Nodes with Zero Lifecycle Connections (WARN)

A lifecycle orphan is a note in a lifecycle dimension (pitfall, idea, experiment, skill) that has zero connections to any other lifecycle dimension.

echo "=== LIFECYCLE ORPHANS ==="

# Pitfalls not referenced by any idea
for p in ~/vault/topics/pitfalls/*.md; do
  slug=$(basename "$p" .md)
  if ! grep -rl "pitfalls/$slug" ~/vault/ideas/ >/dev/null 2>&1; then
    echo "ORPHAN pitfall: $slug (no idea addresses it)"
  fi
done

# Ideas not referencing any pitfall AND not promoted
for f in ~/vault/ideas/*.md; do
  slug=$(basename "$f" .md)
  has_pitfall=$(grep 'addresses_pitfalls' "$f" | grep -v '\[\]' | grep -c '\[\[')
  status=$(grep '^status:' "$f" | head -1 | sed 's/status: *//')
  if [ "$has_pitfall" -eq 0 ] && [ "$status" = "captured" ]; then
    echo "ORPHAN idea: $slug (no pitfall link, not promoted)"
  fi
done

# Experiments with no idea_source AND no skill references them
for f in ~/vault/experiments/**/*.md; do
  [ ! -f "$f" ] && continue
  slug=$(basename "$f" .md)
  proj=$(basename "$(dirname "$f")")
  idea_src=$(grep '^idea_source:' "$f" | head -1 | sed 's/idea_source: *//')
  
  # Check if any skill references this experiment
  has_skill_ref=0
  if grep -rl "experiments/$proj/$slug" ~/vault/skills/ >/dev/null 2>&1; then
    has_skill_ref=1
  fi
  
  if { [ -z "$idea_src" ] || [ "$idea_src" = "null" ]; } && [ "$has_skill_ref" -eq 0 ]; then
    echo "ORPHAN experiment: $proj/$slug (no idea_source, not referenced by any skill)"
  fi
done

Target: Zero orphan experiments. Pitfall orphans acceptable if <50% of total.

7. Complete Chain Count (INFO)

A complete chain traces from pitfall through idea through experiment to skill. Count how many such chains exist.

echo "=== COMPLETE CHAINS ==="
chains=0

for skill_file in ~/vault/skills/*.md; do
  # Skip audit skills
  [ "$(basename "$skill_file")" == audit-* ](/ "$(basename "$skill_file")" == audit-* ) && continue
  
  skill_name=$(basename "$skill_file" .md)
  
  # Get experiments this skill references
  grep -oP '\[\[experiments/[^\]]+\]\]' "$skill_file" 2>/dev/null | sed 's/\[\[//;s/\]\]//' | while read exp_path; do
    exp_file=~/vault/"${exp_path}.md"
    [ ! -f "$exp_file" ] && continue
    
    # Get idea_source from the experiment
    idea_src=$(grep '^idea_source:' "$exp_file" | head -1 | sed 's/idea_source: *//' | sed 's/.*\[\[//;s/\]\].*//')
    [ -z "$idea_src" ] || [ "$idea_src" = "null" ] || [ "$idea_src" = "predates ideas dimension" ] && continue
    
    idea_file=~/vault/"${idea_src}.md"
    [ ! -f "$idea_file" ] && continue
    
    # Get pitfalls this idea addresses
    grep -oP '\[\[topics/pitfalls/[^\]]+\]\]' "$idea_file" 2>/dev/null | while read pitfall_link; do
      pitfall_path=$(echo "$pitfall_link" | sed 's/\[\[//;s/\]\]//')
      if [ -f ~/vault/"${pitfall_path}.md" ]; then
        echo "CHAIN: $(basename "$pitfall_path") → $(basename "$idea_src") → $(basename "$exp_path") → $skill_name"
      fi
    done
  done
done

echo "Complete chains found (see above)"

Target: >=1 complete chain demonstrates the lifecycle is functioning. As the vault matures, target >=5.

8. PII and Secrets Scan (ERROR/WARN)

Every note in the lifecycle chain must be free of real API keys, database credentials, and personal contact information. Env var names in documentation are fine; env var values are not. See _config/pii-rules.yaml for the canonical blocked and redact pattern lists.

Blocked patterns (ERROR): API keys (sk-, AIza, ghp_, xoxb-, AKIA, whsec_, sk_live_, sk_test_, re_), database connection strings with embedded passwords, env var assignments with real values.

Redact patterns (WARN): Personal email addresses, GCP project numbers, phone numbers, SSNs, credit card numbers, street addresses. Replace with placeholders (e.g., <personal-email>, <gcp-project-id>).

VAULT=~/vault
PII_ISSUES=0
for dir in "$VAULT"/topics/pitfalls "$VAULT"/ideas "$VAULT"/experiments/*/ "$VAULT"/skills "$VAULT"/breakthroughs; do
  for f in "$dir"/*.md; do
    [ -f "$f" ] || continue
    slug=$(basename "$f")
    if grep -qE 'sk-[a-zA-Z0-9]{20,}|AIza[a-zA-Z0-9_-]{35}|ghp_[a-zA-Z0-9]{36}|xoxb-|AKIA[A-Z0-9]{16}|whsec_|sk_live_|sk_test_|re_[a-zA-Z0-9]{20,}' "$f"; then
      echo "ERROR (PII): $slug contains a blocked secret pattern"
      PII_ISSUES=$((PII_ISSUES + 1))
    fi
    if grep -qE 'postgres(ql)?://[^:]+:[^@]+@|mongodb(\+srv)?://[^:]+:[^@]+@' "$f"; then
      echo "ERROR (PII): $slug contains a database connection string with credentials"
      PII_ISSUES=$((PII_ISSUES + 1))
    fi
    if grep -qE 'alexdgutierreza@gmail\.com|616560719313' "$f"; then
      echo "WARN (PII): $slug contains personal email or GCP project number: replace with placeholder"
      PII_ISSUES=$((PII_ISSUES + 1))
    fi
  done
done
echo "PII scan (lifecycle chain): $PII_ISSUES issues found"

Severity: ERROR for blocked secrets (must fix before commit). WARN for redact patterns (fix before any content flows to public destinations).

Reference: _config/pii-rules.yaml: canonical pattern list and zone-based escalation rules.

9. Provenance Chain: LLM-Synthesized Source Validation (WARN)

If any lifecycle node (pitfall, idea, experiment, skill, breakthrough) has provenance_type: llm-synthesized and is referenced as a source by another lifecycle node, it must have a validated_by field. This prevents feedback corruption where LLM-generated content feeds unverified into future LLM operations.

echo "=== LIFECYCLE PROVENANCE CHECK ==="
ISSUES=0

for dir in "$VAULT"/topics/pitfalls "$VAULT"/ideas "$VAULT"/experiments/*/ "$VAULT"/skills "$VAULT"/breakthroughs; do
  for f in "$dir"/*.md; do
    [ -f "$f" ] || continue
    slug=$(basename "$f" .md)
    provenance=$(grep '^provenance_type:' "$f" | head -1 | sed 's/provenance_type: *//')

    if [ "$provenance" = "llm-synthesized" ]; then
      validated=$(grep '^validated_by:' "$f" | head -1 | sed 's/validated_by: *//')
      if [ -z "$validated" ] || [ "$validated" = "null" ]; then
        echo "WARN [provenance]: $slug is llm-synthesized with no validated_by field"
        ISSUES=$((ISSUES + 1))
      fi
    fi
  done
done

echo "Lifecycle provenance issues: $ISSUES"

Severity: WARN (lint warning, not blocker). Promoted to ERROR after 30-day false-positive stabilization period.

---

What Complete Looks Like

A healthy lifecycle chain meets these criteria:

Metric	Target	Status
Pitfall→Idea coverage	>=50% of pitfalls addressed	Check 1
Idea→Experiment flow	>=1 promoted idea	Check 2
Experiment→Idea backlinks	100% have idea_source	Check 3
Experiment→Skill references	confirmed experiments referenced by skills	Check 4
Breakthrough notes	>=8 in breakthroughs/ + matching tags	Check 5
Lifecycle orphans	0 orphan experiments	Check 6
Complete chains	>=1 end-to-end	Check 7

Score Summary Template

Lifecycle Chain Audit: {date}
──────────────────────────────────────
Pitfall→Idea coverage:    {n}/{total} ({pct}%)
Idea→Experiment promoted: {n}/{total}
Experiment→Idea linked:   {n}/{total} ({pct}%)
Experiment→Skill refs:    {n}/{total}
Breakthrough notes:       {n} in breakthroughs/ ({n} tagged sources)
Lifecycle orphans:        {n} (target: 0 experiments)
Complete chains:          {n} (target: >=1)
──────────────────────────────────────
Edge density:             {total_edges} edges across {total_nodes} nodes
Overall:                  {pass|fail}

---

How to Fill Gaps

Stamp last_audited

Every note you audit or create must have last_audited: YYYY-MM-DD in its frontmatter (today’s date).

Disconnected pitfalls (no idea addresses them)

For each unaddressed pitfall:

1. Read the pitfall to understand the problem pattern
2. Check if an existing idea could address it (add the pitfall to addresses_pitfalls)
3. If no existing idea fits, create a new idea note with addresses_pitfalls linking to the pitfall

Missing idea_source on experiments

For experiments created before 2026-04-01:

idea_source: "predates ideas dimension"

For experiments created on or after 2026-04-01:

1. Find or create the idea that prompted the experiment
2. Set idea_source: "[ideas/{slug}](/ideas/{slug})"

No promoted ideas (pipeline stalled)

Review captured ideas by priority_score. For the highest-priority idea:

1. Change status: exploring and begin investigation
2. When ready, change status: promoted and set promoted_to: "[experiments/{project}/{YYYY-MM-DD}-{slug}](/experiments/{project}/{YYYY-MM-DD}-{slug})"
3. Create the experiment note with idea_source: "[ideas/{idea-slug}](/ideas/{idea-slug})"

Missing breakthrough notes (dual-write)

Review confirmed experiments with >=10% metric improvement. For each:

1. Add breakthrough to the experiment’s tags array.
2. Create a note in breakthroughs/YYYY-MM-DD-{project}-{slug}.md with type: breakthrough frontmatter (title, project, source_type, source_note, metric_before, metric_after).
3. Also tag the related topic and skill if they exist.

---

Quality Checks

Automated

1. Edge count script (checks 1-4): Run the bash scripts above from vault root
2. Breakthrough census: ls ~/vault/breakthroughs/*.md | wc -l (directory) + grep -rl 'breakthrough' ~/vault/experiments/ ~/vault/skills/ | wc -l (tags)
3. Orphan scan (check 6): Run the orphan detection script
4. Chain trace (check 7): Run the complete chain script

Manual

5. Walk one chain: Pick a skill with experiments links. Follow the chain backward: skill → experiment → idea_source → idea → addresses_pitfalls → pitfall. Every link should resolve to a real note.
6. Dashboard verification: Open dashboards/dimension-scorecard.md in Obsidian. All lifecycle dimensions should show non-zero counts.
7. Graph view: Filter Obsidian graph to show only lifecycle dimensions. Verify visible edges between clusters (not isolated islands).

Regression Prevention

• After creating a new experiment, always set idea_source (create the idea first if needed)
• After resolving a pitfall, create or update the idea that addressed it
• Run this audit monthly alongside the dimension scorecard review

Visual Enrichment

When this audit produces output that benefits from visualization:

Finding Type	Tool	Specification
Knowledge graph density, edge counts	R viz (skills/r-visualization-pipeline)	Family: NET, Template: Journal
Full lifecycle (pitfall->idea->experiment->skill->breakthrough)	Figma MCP (generate_diagram)	Type: State diagram

See topics/visual-output-routing for the full routing decision framework.

Self-improvement context: This audit skill implements the lint cycle of Pattern 4 (Compiler Wiki) from skills/self-improving-agent-patterns. The vault’s audit skills collectively form the lint+heal loop described in research/2026-04-02-karpathy-llm-knowledge-base-pattern.

Post-Audit Visual Enrichment Trigger

After completing all quality checks, dispatch a Sonnet subagent to generate visuals for this dimension:

[!tip] Auto-generate visuals after audit Use the Agent tool to dispatch a subagent that runs skills/wiki-visual-enrichment for the lifecycle-chain dimension. This generates R charts and Figma diagrams based on the Visual Enrichment specifications above. Content-hash dedup ensures only changed articles get new visuals.