Optimal test activity allocation for covariate software reliability and security models

https://doi.org/10.1016/j.jss.2020.110643Get rights and content

Highlights

  • A NHPP software reliability growth model incorporating covariates is developed.

  • Efficient expectation conditional maximization algorithms are derived.

  • The optimal testing activity allocation problem maximizes fault exposure.

  • Results suggest periodic application of testing activity allocation can guide test.

Abstract

Traditional software reliability growth models enable quantitative assessment of the software testing process by characterizing the fault detection in terms of testing time or effort. However, the majority of these models do not identify specific testing activities underlying fault discovery and thus can only provide limited guidance on how to incrementally allocate effort. Although there are several novel studies focused on covariate software reliability growth models, they are limited to model development, application, and assessment.

This paper presents a non-homogeneous Poisson process software reliability growth model incorporating covariates based on the discrete Cox proportional hazards model. An efficient and stable expectation conditional maximization algorithm is applied to identify the model parameters. An optimal test activity allocation problem is formulated to maximize fault discovery. The proposed method is illustrated through numerical examples on two data sets.

Introduction

Traditional software reliability growth models (SRGM) (Farr and Smith, 1984) characterize the fault discovery process during testing as a non-homogeneous Poisson process (NHPP). These models predict future faults as a function of testing time or effort, enabling inferences such as the number of faults remaining, additional time required to achieve a specified reliability, and optimization problems such as optimal release (Zhao and Xie, 1993) and effort allocation (Fiondella and Gokhale, 2008). However, the vast majority of these NHPP SRGM do not identify the underlying software testing activities that lead to fault discovery. Thus, effort allocation based on these models can only provide general guidance on the amount of effort to invest and limited information regarding the effectiveness of specific testing activities.

More recently, bivariate NHPP SRGM (Ishii et al., 2008) and covariate models (Rinsaka et al., 2006), which are capable of characterizing faults discovered as a function of multiple software testing activities such as calendar time, number of test cases executed, and test execution time have been proposed. Covariate models are an especially attractive alternative to testing effort models (Yamada et al., 1986) because they only introduce a single additional parameter per metric and do not require sequential model fitting procedures, although covariate models still require stable and efficient numerical methods. To realize the full potential of covariate models, generalized optimization procedures such as test activity allocation are needed to guide the distribution of limited resources among specific test activities in order to maximize fault discovery, correction, and improved reliability.

Early metrics based models include the work of Khoshgoftaar and Munson (1990), who performed regression analysis to study the relationship between software complexity metrics and the number of errors found during test and validation. Alternative estimation techniques to create linear models of software quality (Khoshgoftaar et al., 1992b) were subsequently evaluated. Khoshgoftaar et al. (1992a) also applied nonlinear regression to model the relationship between software metrics and the number of faults in a program module. Hudepohl et al. (1996) implemented these methods in the Emerald (Enhanced Measurement for Early Risk Assessment of Latent Defects) tool and applied them to telecommunications software.

More recent covariate models include the work of Rinsaka et al. (2006) combined the proportional hazards model and nonhomogeneous Poisson process to provide a generalized fault detection process possessing time-dependent covariate structure. Shibata et al. (2006) subsequently extended this to a cumulative Bernoulli trial process. Okamura et al. (2010) proposed a multi-factor software reliability model based on logistic regression and an efficient algorithm to estimate the model parameters. Okamura and Dohi (2014) combined Poisson regression-based fault prediction and generalized linear models (Okamura and Dohi, 2015) with metrics-based software reliability growth models. Shibata et al. (2015) implemented these methods in the M-SRAT (Metrics-based Software Reliability Assessment Tool).

Covariate models exhibit substantially improved prediction capabilities over NHPP software reliability growth models. Therefore, to enhance the utility of covariate models and encourage their use in practice, this paper makes the following primary contributions

  • A software reliability growth model possessing a discrete Cox proportional hazard rate to incorporate covariates. This approach is analogous to the model introduced in Shibata et al. (2006), which has been constructed based on a discrete time model developed by Kalbfleisch and Prentice (1973). However, formulation and estimation of our model takes into account the fact that the counts of software faults in disjoint intervals of the point process are independent random variables possessing a Poisson distribution with non-homogeneous rates, which is an assumption of a NHPP by design.

  • Efficient expectation conditional maximization (ECM) (Nagaraju et al., 2017) algorithms to estimate the numerical parameters of a model that best characterize the data.

  • A generalization of the testing effort allocation problem (Yamada et al., 1993) to covariate models referred to as the optimal testing activity allocation problem to (i) maximize fault discovery within a budget constraint or (ii) minimize the budget required discover a specified number of additional faults.

The illustrations apply the model to two data sets from the literature with the ECM algorithms and then solves the optimal testing activity allocation problem. The results indicate that periodic application of testing activity allocation could more effectively guide the type and amount of specific testing activities throughout the software testing process in order to discover more faults despite limited resources.

The remainder of the paper is organized as follows: Section 2 describes the mathematical formulation of the NHPP SRGM as well as the discrete Cox proportional hazards model. Section 3 discusses proportional hazards modeling incorporating covariates and formulates the optimal test activity allocation problem to maximize fault discovery. Section 4 describes model estimation, assessment, and selection. Section 5 illustrates the proposed approach on two data sets from the literature, while Section 6 provides conclusions and future research.

Section snippets

Point process based SRGMs

This section formulates the nonhomogenous Poisson process software reliability growth model and Discrete Cox Proportional Hazards (DCPH) model upon which the subsequent covariate model is developed.

Software faults are detected at random discrete times, which are consistent with the abrupt nature of software failures. Let M be the number of faults within the software and Tj, j=1,2,,M the time the jth fault is first discovered. The subscript is not used further because it is assumed that the

Proportional hazards modeling incorporating covariates

This section describes the formulation of a discrete Cox proportional hazards model incorporating covariates.

Testing occurs in discrete intervals i=1,2,,n. An NHPP software reliability model assumes that the point process possesses independent Poisson distributed increments. In otherwords, the numbers of faults detected over these n disjoint time intervals are independent. Suppose that we are interested in investigating the effect of p software test activities on fault detection. We denote the

Model estimation, assessment, and selection

This section describes model estimation, assessment, and selection methods. Estimation methods include maximum likelihood estimation (MLE) and the expectation conditional maximization (ECM) algorithm as well. An example derivation of the ECM update rules is provided in the context of the geometric hazard function. Initial parameter estimation, measures of goodness of fit, and model selection are also discussed.

Illustrations

This section demonstrates the model, ECM algorithm, and optimal covariate allocation. The DS1 and DS2 data sets (Shibata et al., 2006) respectively consisting of n=17 and n=14 weeks of observations are employed. Both data sets possess three covariates, including execution time (E) in hours, failure identification work (F) in person hours, and computer time failure identification (C) in hours. Here, F may indicate activities such as code reviews, where one or more programmers read but did not

Conclusions and future research

This paper presents a covariate software reliability growth model and formulates the optimal test activity allocation problem to maximize fault discovery despite budget constraints or minimize the budget required to discover a specified number of faults. Expectation conditional maximization algorithms were derived and applied to two data sets from the literature. The optimal test activity allocation problem was then solved to illustrate how it can increase the number of faults discovered by

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This material is based upon work supported by the National Science Foundation, United States under Grant Number (#1749635). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

V. Nagaraju was sponsored by the Army Research Laboratory, United States and was accomplished under Cooperative Agreement Number W911NF1920224. The views and conclusions contained in

Vidhyashree Nagaraju is a Ph.D. candidate in the Department of Electrical and Computer Engineering at the University of Massachusetts Dartmouth (UMassD), where she received her MS (2015) in Computer Engineering. She received her BE (2011) in Electronics and Communication Engineering from Visvesvaraya Technological University in India.

References (28)

  • DiakoulakiD. et al.

    Determining objective weights in multiple criteria problems: The critic method

    Comput. Oper. Res.

    (1995)
  • NagarajuV. et al.

    Practical software reliability engineering with the software failure and reliability assessment tool (SFRAT)

    SoftwareX

    (2019)
  • AmmannP. et al.

    Introduction to Software Testing

    (2016)
  • FarrW. et al.

    Statistical modeling and estimation of reliability functions for software (SMERFS) users guide

    (1984)
  • FiondellaL. et al.

    Software reliability models incorporating testing effort

    Opsearch

    (2008)
  • FiondellaL. et al.

    Software reliability model with bathtub-shaped fault detection rate

  • HudepohlJ. et al.

    Emerald: Software metrics and models on the desktop

    IEEE Softw.

    (1996)
  • IshiiT. et al.

    Bivariate extension of software reliability modeling with number of test cases

    Int. J. Reliab. Qual. Saf. Eng.

    (2008)
  • KalbfleischJ. et al.

    Marginal likelihoods based on Cox’s regression and life model

    Biometrika

    (1973)
  • KhoshgoftaarT. et al.

    Predicting software errors, during development, using nonlinear regression models: A comparative study

    IEEE Trans. Reliab.

    (1992)
  • KhoshgoftaarT. et al.

    Predicting software development errors using software complexity metrics

    IEEE J. Sel. Areas Commun.

    (1990)
  • KhoshgoftaarT. et al.

    Predictive modeling techniques of software quality from software measures

    IEEE Trans. Softw. Eng.

    (1992)
  • LiH. et al.

    Software reliability metrics selecting method based on analytic hierarchy process

  • Handbook of Software Reliability Engineering

    (1996)
  • Cited by (0)

    Vidhyashree Nagaraju is a Ph.D. candidate in the Department of Electrical and Computer Engineering at the University of Massachusetts Dartmouth (UMassD), where she received her MS (2015) in Computer Engineering. She received her BE (2011) in Electronics and Communication Engineering from Visvesvaraya Technological University in India.

    Chathuri L. Jayasinghe received the B.Sc. (Hons.) degree in statistics from the University of Sri Jayewardenepura, Nugegoda, Sri Lanka, in 2007, the M.Sc. degree (with high distinction) in statistics and operations research in 2009 and the Ph.D. degree in statistics in 2013, both from the RMIT University, Australia. She was a Researcher in the School of Sciences, RMIT University and she is currently working with the University of Sri Jayewardenepura, Sri Lanka as a Senior Lecturer in statistics. Her research interests include software reliability growth models, nonparametric and parametric estimation methods, and survival analysis.

    Lance Fiondella is an assistant professor in the Department of Electrical & Computer Engineering at UMassD. He received his Ph.D. (2012) in Computer Science & Engineering from the University of Connecticut. Dr. Fiondella’s papers have been the recipient of twelve conference paper awards, including seven with his students.  His research has been funded by the Department of Homeland Security, National Aeronautics and Space Administration, Army Research Laboratory, Naval Air Warfare Center, and National Science Foundation, including a CAREER Award.

    V. Nagaraju and L. Fiondella were supported by the US National Science Foundation under Award (#1749635). V. Nagaraju was sponsored by the Army Research laboratory, USA under Cooperative Agreement Number W911NF1920224.

    View full text