Optimal test activity allocation for covariate software reliability and security models☆
Introduction
Traditional software reliability growth models (SRGM) (Farr and Smith, 1984) characterize the fault discovery process during testing as a non-homogeneous Poisson process (NHPP). These models predict future faults as a function of testing time or effort, enabling inferences such as the number of faults remaining, additional time required to achieve a specified reliability, and optimization problems such as optimal release (Zhao and Xie, 1993) and effort allocation (Fiondella and Gokhale, 2008). However, the vast majority of these NHPP SRGM do not identify the underlying software testing activities that lead to fault discovery. Thus, effort allocation based on these models can only provide general guidance on the amount of effort to invest and limited information regarding the effectiveness of specific testing activities.
More recently, bivariate NHPP SRGM (Ishii et al., 2008) and covariate models (Rinsaka et al., 2006), which are capable of characterizing faults discovered as a function of multiple software testing activities such as calendar time, number of test cases executed, and test execution time have been proposed. Covariate models are an especially attractive alternative to testing effort models (Yamada et al., 1986) because they only introduce a single additional parameter per metric and do not require sequential model fitting procedures, although covariate models still require stable and efficient numerical methods. To realize the full potential of covariate models, generalized optimization procedures such as test activity allocation are needed to guide the distribution of limited resources among specific test activities in order to maximize fault discovery, correction, and improved reliability.
Early metrics based models include the work of Khoshgoftaar and Munson (1990), who performed regression analysis to study the relationship between software complexity metrics and the number of errors found during test and validation. Alternative estimation techniques to create linear models of software quality (Khoshgoftaar et al., 1992b) were subsequently evaluated. Khoshgoftaar et al. (1992a) also applied nonlinear regression to model the relationship between software metrics and the number of faults in a program module. Hudepohl et al. (1996) implemented these methods in the Emerald (Enhanced Measurement for Early Risk Assessment of Latent Defects) tool and applied them to telecommunications software.
More recent covariate models include the work of Rinsaka et al. (2006) combined the proportional hazards model and nonhomogeneous Poisson process to provide a generalized fault detection process possessing time-dependent covariate structure. Shibata et al. (2006) subsequently extended this to a cumulative Bernoulli trial process. Okamura et al. (2010) proposed a multi-factor software reliability model based on logistic regression and an efficient algorithm to estimate the model parameters. Okamura and Dohi (2014) combined Poisson regression-based fault prediction and generalized linear models (Okamura and Dohi, 2015) with metrics-based software reliability growth models. Shibata et al. (2015) implemented these methods in the M-SRAT (Metrics-based Software Reliability Assessment Tool).
Covariate models exhibit substantially improved prediction capabilities over NHPP software reliability growth models. Therefore, to enhance the utility of covariate models and encourage their use in practice, this paper makes the following primary contributions
- •
A software reliability growth model possessing a discrete Cox proportional hazard rate to incorporate covariates. This approach is analogous to the model introduced in Shibata et al. (2006), which has been constructed based on a discrete time model developed by Kalbfleisch and Prentice (1973). However, formulation and estimation of our model takes into account the fact that the counts of software faults in disjoint intervals of the point process are independent random variables possessing a Poisson distribution with non-homogeneous rates, which is an assumption of a NHPP by design.
- •
Efficient expectation conditional maximization (ECM) (Nagaraju et al., 2017) algorithms to estimate the numerical parameters of a model that best characterize the data.
- •
A generalization of the testing effort allocation problem (Yamada et al., 1993) to covariate models referred to as the optimal testing activity allocation problem to (i) maximize fault discovery within a budget constraint or (ii) minimize the budget required discover a specified number of additional faults.
The illustrations apply the model to two data sets from the literature with the ECM algorithms and then solves the optimal testing activity allocation problem. The results indicate that periodic application of testing activity allocation could more effectively guide the type and amount of specific testing activities throughout the software testing process in order to discover more faults despite limited resources.
The remainder of the paper is organized as follows: Section 2 describes the mathematical formulation of the NHPP SRGM as well as the discrete Cox proportional hazards model. Section 3 discusses proportional hazards modeling incorporating covariates and formulates the optimal test activity allocation problem to maximize fault discovery. Section 4 describes model estimation, assessment, and selection. Section 5 illustrates the proposed approach on two data sets from the literature, while Section 6 provides conclusions and future research.
Section snippets
Point process based SRGMs
This section formulates the nonhomogenous Poisson process software reliability growth model and Discrete Cox Proportional Hazards (DCPH) model upon which the subsequent covariate model is developed.
Software faults are detected at random discrete times, which are consistent with the abrupt nature of software failures. Let be the number of faults within the software and , the time the fault is first discovered. The subscript is not used further because it is assumed that the
Proportional hazards modeling incorporating covariates
This section describes the formulation of a discrete Cox proportional hazards model incorporating covariates.
Testing occurs in discrete intervals . An NHPP software reliability model assumes that the point process possesses independent Poisson distributed increments. In otherwords, the numbers of faults detected over these disjoint time intervals are independent. Suppose that we are interested in investigating the effect of software test activities on fault detection. We denote the
Model estimation, assessment, and selection
This section describes model estimation, assessment, and selection methods. Estimation methods include maximum likelihood estimation (MLE) and the expectation conditional maximization (ECM) algorithm as well. An example derivation of the ECM update rules is provided in the context of the geometric hazard function. Initial parameter estimation, measures of goodness of fit, and model selection are also discussed.
Illustrations
This section demonstrates the model, ECM algorithm, and optimal covariate allocation. The DS1 and DS2 data sets (Shibata et al., 2006) respectively consisting of and weeks of observations are employed. Both data sets possess three covariates, including execution time (E) in hours, failure identification work (F) in person hours, and computer time failure identification (C) in hours. Here, F may indicate activities such as code reviews, where one or more programmers read but did not
Conclusions and future research
This paper presents a covariate software reliability growth model and formulates the optimal test activity allocation problem to maximize fault discovery despite budget constraints or minimize the budget required to discover a specified number of faults. Expectation conditional maximization algorithms were derived and applied to two data sets from the literature. The optimal test activity allocation problem was then solved to illustrate how it can increase the number of faults discovered by
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This material is based upon work supported by the National Science Foundation, United States under Grant Number (#1749635). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
V. Nagaraju was sponsored by the Army Research Laboratory, United States and was accomplished under Cooperative Agreement Number . The views and conclusions contained in
Vidhyashree Nagaraju is a Ph.D. candidate in the Department of Electrical and Computer Engineering at the University of Massachusetts Dartmouth (UMassD), where she received her MS (2015) in Computer Engineering. She received her BE (2011) in Electronics and Communication Engineering from Visvesvaraya Technological University in India.
References (28)
- et al.
Determining objective weights in multiple criteria problems: The critic method
Comput. Oper. Res.
(1995) - et al.
Practical software reliability engineering with the software failure and reliability assessment tool (SFRAT)
SoftwareX
(2019) - et al.
Introduction to Software Testing
(2016) - et al.
Statistical modeling and estimation of reliability functions for software (SMERFS) users guide
(1984) - et al.
Software reliability models incorporating testing effort
Opsearch
(2008) - et al.
Software reliability model with bathtub-shaped fault detection rate
- et al.
Emerald: Software metrics and models on the desktop
IEEE Softw.
(1996) - et al.
Bivariate extension of software reliability modeling with number of test cases
Int. J. Reliab. Qual. Saf. Eng.
(2008) - et al.
Marginal likelihoods based on Cox’s regression and life model
Biometrika
(1973) - et al.
Predicting software errors, during development, using nonlinear regression models: A comparative study
IEEE Trans. Reliab.
(1992)
Predicting software development errors using software complexity metrics
IEEE J. Sel. Areas Commun.
Predictive modeling techniques of software quality from software measures
IEEE Trans. Softw. Eng.
Software reliability metrics selecting method based on analytic hierarchy process
Handbook of Software Reliability Engineering
Cited by (0)
Vidhyashree Nagaraju is a Ph.D. candidate in the Department of Electrical and Computer Engineering at the University of Massachusetts Dartmouth (UMassD), where she received her MS (2015) in Computer Engineering. She received her BE (2011) in Electronics and Communication Engineering from Visvesvaraya Technological University in India.
Chathuri L. Jayasinghe received the B.Sc. (Hons.) degree in statistics from the University of Sri Jayewardenepura, Nugegoda, Sri Lanka, in 2007, the M.Sc. degree (with high distinction) in statistics and operations research in 2009 and the Ph.D. degree in statistics in 2013, both from the RMIT University, Australia. She was a Researcher in the School of Sciences, RMIT University and she is currently working with the University of Sri Jayewardenepura, Sri Lanka as a Senior Lecturer in statistics. Her research interests include software reliability growth models, nonparametric and parametric estimation methods, and survival analysis.
Lance Fiondella is an assistant professor in the Department of Electrical & Computer Engineering at UMassD. He received his Ph.D. (2012) in Computer Science & Engineering from the University of Connecticut. Dr. Fiondella’s papers have been the recipient of twelve conference paper awards, including seven with his students. His research has been funded by the Department of Homeland Security, National Aeronautics and Space Administration, Army Research Laboratory, Naval Air Warfare Center, and National Science Foundation, including a CAREER Award.