Mind the Gap: What Working With Developers on Fuzz Tests Taught Us About Coverage Gaps

Can fuzzers generate partial tests that developers find useful enough to complete into functional tests (e.g., by adding assertions)? To address this question, researchers developed a prototype within the Mozilla ecosystem and open bug reports proposing partial generated tests for currently uncovered code.

They found that the majority of the reactions focus on whether the targeted coverage gap is actually worth testing. To investigate further which coverage gaps developers find relevant to close, they designed an automated filter to exclude irrelevant coverage gaps before generating tests. From conversations with developers about whether the remaining coverage gaps are worth closing when a partially generated test is available, they learned that the filtering indeed removes clearly nontest-worthy gaps. The developers propose a variety of additional strategies to address the coverage gaps and how to make fuzz tests and reports more useful for developers.