Abstract
Nowadays, there exists an increasing demand for reliable software systems able to fulfill their requirements in different
operational environments and to cope with uncertainty that can be introduced both at design�time and at runtime because
of the lack of control over third�party system components and complex interactions among software, hardware infrastructures
and physical phenomena. This article addresses the problem of the discrepancy between measured data at runtime and the design�time
formal specification by using an inverse uncertainty quantification approach. Namely, we introduce a methodology called METRIC
and its supporting toolchain to quantify and mitigate software system uncertainty during testing by combining (on�the�fly)
model�based testing and Bayesian inference. Our approach connects probabilistic input/output conformance theory with statistical
hypothesis testing in order to assess if the behaviour of the system under test corresponds to its probabilistic formal specification
provided in terms of a Markov decision process. An uncertainty�aware model�based test case generation strategy is used
as a means to collect evidence from software components affected by sources of uncertainty. Test results serve as input to
a Bayesian inference process that updates beliefs on model parameters encoding uncertain quality attributes of the system
under test. This article describes our approach from both theoretical and practical perspectives. An extensive empirical evaluation
activity has been conducted in order to assess the cost�effectiveness of our approach. We show that, under same effort constraints,
our uncertainty�aware testing strategy increases the accuracy of the uncertainty quantification process up to 50 times with
respect to traditional model�based testing methods.
[download the pdf file] [DOI]