tl;dr:“The test ran hundreds if not thousands of times successfully. What a waste of time... But then, one day, we started observing test failures. Not many, maybe three over the course of a few weeks. The test actually crashed with a Segmentation Fault, so it was clear that it was a severe error. Interestingly, none of the code under test had actually changed. Well, that's definitely something we had to investigate! I spare you the details of the search for the error, but eventually, I was able to reproduce the problem while a debugger was attached, so the entire context of the problem was handed to me on a silver platter.”
tl;dr:Benjamin concludes with what he's learned from debugging: (1) No matter how battle-tested and old the code and how reputable the distributor - the code contains bugs. (2) Old bugs can manifest themselves seemingly out of nowhere, caused by subtle changes in timing or memory layout. (3) Whenever the file system is involved, there is a significant danger that bugs are caused by race conditions. And more.