Lock In Lab Intern, rank 1 of 9

INC-16 sev 1 19 Nov 2026, filed under Builder

The backup rotation deleted the only good backup

What we saw

Restoring yesterday's backup: every recent file was empty.

How we found it

Listing the backup folder by size.

Cause

The rotation kept the newest N files and deleted the rest. A bug upstream had been producing hundreds of empty backup files, 460 of them. The rotation counted them as valid and deleted the one good backup.

Fix

Rotation now only counts backups that are verified (a marker written after an integrity check passes), and removes zero-byte files on start.

The rule now

Only verified things count. Count a backup after an integrity check passes, and restore from one now and then.