Pitfall Persistence dakka

Log Buffer Requeue On Flush Failure

pitfallloggingdata-lossSTATE

What Happened

The Dakka log buffer flushed to disk, the write failed (disk full, permissions error, file locked), and the log entries were silently dropped. The buffer reported success because the entries were dequeued : but they never reached persistent storage. Under sustained write failures, minutes of log data vanished with no indication anything had gone wrong.

Root Cause

See definitions/root-cause-analysis for the analytical framework. Specific cause: the typical flush pattern : (1) pop entries from the buffer, (2) serialize, (3) write to disk : removes entries in step 1 before confirming step 3 succeeded. If the write fails, the entries exist only in a local variable scope and are garbage-collected when the function returns. No retry, no error propagation: just silent data loss.

How to Avoid

On flush failure, prepend the failed batch back to the front of the buffer and retry on the next flush cycle. Use unshift so failed entries maintain their original order and are the first to be written on the next attempt. Add a retry counter to prevent infinite loops : after N failures, drop the batch and write a warning to stderr.

try {
  fs.appendFileSync(logPath, serialized);
} catch (err) {
  buffer.unshift(...failedBatch);
  console.error(`Flush failed, ${failedBatch.length} entries re-queued: ${err.message}`);
}