The bad: Yesterday I found that the system backup tool (dump(8)) for FreeBSD had malfunctioned on one of my level 0 dumps (read: a complete snapshot of a disk), garbling the internal ordering of records.
The worse: The dump was for my “/pub” partition, which included the multimedia collection (mmc) that I populated over nearly 10 years. -_-
The not-so-bad: After a close inspection of what had gone wrong (which involved extensive use of hexdump(8)) and analysis of the dump(8) source code, I concluded:
All the backup data blocks were present in the dump, simply locally reordered;
Due to the way dump(8) works, each backup data block could be displaced by ±(n−1) blocks at most, where n is the number of worker threads/processes;
Because the blocks did not replace one another, but were simply reordered around, there could be at most n!−1 possibilities of reordering;
Since dump(8) used 3 worker processes by default, it meant that there were only five reordering variations to consider.
The good: Each header block (which precedes a 256KiB chunk of file data) had its block number recorded; this meant that it could be easily detected if the header block was displaced, simply by comparing the recorded block number, which is where the block should have been seen, against where the block was actually seen in the backup. I wrote a C++ program that 1) verifies my conclusion and 2) analyzes the displacement patterns to see if it is one of the five possible variations. I left it to run overnight (because the backup tape had over 150GB of data *sigh*)…
The better: … And I woke up this morning to find that my program determined that only one displacement pattern was seen over the entire corrupt region of the backup.
The best: With this displacement pattern, I will be able to write a program that reorders the backup data blocks back into the correct sequence then use it to recover the entire backup set, possibly except the region where the initial displacement had been detected.
And the bestest: The region of initial displacement was a 4MiB stretch, contained completely within a backup image of my own DVD, so I wouldn't care even if that region were gone. In other words, nothing was lost! Well, okay, maybe it lost me a couple of hours of time, but this feeling of accomplishment totally justifies that time lost then some more!