Abstract
Code coverage is usually used as a measurement of testing quality and as adequacy criterion. Unfortunately, code coverage
is very sensitive to modifications of the code structure, and, therefore, the same test suite can achieve different degrees
of coverage on the same program written in two syntactically different ways. For this reason, code coverage can provide the
tester with misleading information. In order to understand how a testing criterion is affected by code structure modifications,
we introduce a way to measure the sensitivity of coverage to code changes. We formalize the modifications of the code structure
using semantic preserving code-to-code transformations and we propose a framework to evaluate coverage robustness to these
transformations, extending actual existing coverage criteria. This allows us to define which programs and which test suites
can be considered robust with respect to a certain set of transformations. We can identify when the obtained coverage is fragile
and we extend the concept of coverage criterion by introducing an index that measures the fragility of the coverage of a given
test suite. We show how to compute the fragility index and we evidence that also well-written industrial code and realistic
test suites can be fragile. Moreover, we suggest how to deal with this kind of testing fragility.
[read the copyright and download the pdf file] [DOI] [presentation]