Candidato: Alan Kelon Oliveira de Moraes
Título do trabalho: On the Influence of Test Adequacy Criteria on Test Suite Reduction for Model-Based Testing of Real-Time Systems
Orientador(es): Patrícia D. L. Machado e Wilkerson L. Andrade
Banca examinadora: Adenilso Simão (USP), Fátima Mattiello (INPE), Jorge César Abrantes de Figueiredo (UFCG), Tiago Massoni (UFCG).
Resumo: Model-based testing is a testing approach that relies on the existence of abstract models of an application to generate, execute and evaluate tests. Test case generation plays an important role in model-based testing. Since it consists of a systematic search for test cases that can be extracted from models, model-based testing usually generates large test suites which are too expensive to execute in full. Test suite reduction techniques have been proposed to address this problem. The goal of the techniques is to obtain reduced test suites that are both cheaper to execute and as effective at detecting faults as the original suite, given that the reduced test suites maintain the same coverage level of the complete test suite required by a test selection criterion. These criteria define which parts of the system are going to be tested, how often and under what circumstances. Nevertheless, little attention has being paid to the impact of the criterion choice in test suite reduction research. On the other hand, real-time systems are reactive systems whose behavior is constrained by time. Consequently, time-related faults are specific to these systems. In order to cope with this issue, models for real-time systems must deal with time and, consequently, there are specific test selection criteria for them. However, test suite reduction research is focused on non real-time systems, therefore the impact of test selection criteria for models of real-time systems on test suite reduction is unknown. In this doctoral research, we aim at investigating the influence of test selection criteria on the outcomes of test suite reduction techniques in the context of model-based testing of real-time systems. We are particularly interested in the Timed Input-Output Symbolic Transition Systems (TIOSTS) model because it is an expressive transition system in which data and time are symbolically defined, and transition systems are the basis for conformance testing of real-time systems. In order to achieve the research objective, first, we defined 18 test selection criteria for TIOSTS models. The defined criteria include transition-based criteria, data-flow-oriented criteria and real-time systems criteria. Next, we formalized a hierarchy with these criteria which is partially ordered by strict inclusion. Second, we evaluated the cost-effectiveness of nine criteria in an empirical study of test suite reduction. We evaluated the size, execution time and fault detection of reduced test suites obtained from a random test suite reduction technique and the criteria evaluated. In the experiment, we used a TIOSTS specification model of a burglar alarm system, a trace generator for that model, and faulty trace generators to simulate faults in model implementations. Results show that, despite being a criterion for non real-timed systems, the All-Transition-Pairs criterion was the most cost-effective criterion. Finally, future works of this research intend to evaluate more test suite reduction techniques in combination with other test case generation algorithms.