Crash recovery support for variable strength t-way test generation strategy
Loading...
Date
2016-06-01
Authors
Syahrul Afzal Che Abdullah
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Often, the biggest challenge in software testing relates to the fact that it is not feasible to test for all the input parameters exhaustively owing to constraints in costs, resources and time. Considering these factors, software testers must appropriately sample the test cases in order to best utilize the resources at hand. Within the context of Combinatorial Testing, testers often resort to t-way test generation strategy (where t indicates the strength of interaction). Empirical evidence in the literature indicated that t-way test generation strategy has managed to minimize the test cases significantly whilst maintaining the fault detection capability of the testing process. Much useful progress has been achieved as far as the development of t-way test generation strategy is concerned. Nevertheless, some issues remain especially in the context of addressing ever increasing complexity and size of software (i.e. lines of code) resulting into high number of interaction among input parameters. Firstly, the test generation can be painstakingly long, interruption is expensive as the whole generation process needs to be restarted from scratch. Time and efforts will also be wasted. Secondly, existing strategies commit too early on selection of the best value of input parameters when sampling of the test cases. For this reason, these strategies were less sufficient in terms of generating optimal test suite size. Here, to enhance the t-way testing capability, there is also a need to consider variable-strength strategy. This approach is often favored because of the compromise in terms of test suite size as the strategy focuses testing where it has the most potential value which usually is associated with a risk analysis and priority. In order to address these issues,
this research develops a variable-strength (VS) interaction t-way test generation strategy, called Test Suite Generator with Crash Recovery support (TSGCR). Unlike existing strategies, TSGCR adopts Multilevel Greedy algorithm, which delays choosing the best value until it satisfies certain rules. To provide a reliable operation, TSGCR also permits crash recovery support integrated as part of the strategy itself. As the test generation can potentially be long lasting processes (i.e. due to large selection of input parameters and values), TSGCR tolerates involuntary transaction failures (e.g. such as power failure or system errors) or voluntary execution suspension (e.g. to give ways for other computations) enabling restoration of state and data to the last consistent state. To evaluate the competitiveness of TSGCR, the test generator is tested with uniform and mixed input parameters and the performance (in terms of the generated test suite size) is compared with existing variable strength t-way test generation strategies using well-known standard benchmark configurations (based on six sets of experiments). Benchmarking results showed that for VS interaction configurations for uniform input parameters, TSGCR is able to get seven Δ with 0 value, i.e. similar value to the best solution obtained by other strategies, seven Δ with +ve values, i.e. able to get the best solution; from fourteen out of forty four experimental results. While for VS interaction configurations for mixed input parameters, TSGCR is able to get twenty seven Δ with 0 value, eight Δ with +ve values; from thirty five out of forty one experimental results. Hence, the results demonstrated that TSGCR produces competitive results as far as the size of the test suite is concerned against most existing strategies.