Runtime-Guided Change-Impact Analysis for Explainable Regression Test Selection in Evolving Software Platforms: A Corrected Benchmark and Defects4J Pilot
DOI:
https://doi.org/10.63282/3117-5481/AIJCST-V7I1P111Keywords:
Regression Testing, Regression Test Selection, Change-Impact Analysis, Runtime Coverage, Defects4j, Negative Results, Reproducibility, Explainable Software EngineeringAbstract
Runtime-guided regression test selection can connect changed code to previously observed test behavior, but evaluation granularity may determine whether competing methods can be distinguished. Objective: We examine an explainable method, TRACE-RTS, and test whether a class-level real-fault pilot can evaluate its function-level change-impact contribution. Method: TRACE-RTS combines runtime footprints, changed functions, bounded dependency propagation, history, freshness, execution cost, and per-test explanations. A corrected deterministic benchmark uses 40 generated subjects, 20 releases per subject, five cost budgets, sparse faults, and independently degraded selection evidence. A pre-specified Defects4J 3.0.1 pilot evaluates nine Lang faults using test classes, production-class coverage, strict measured-cost ceilings, and 1,000 Random orders per fault-budget pair. Results: At a 30% synthetic budget, TRACE-RTS detected faults in 77.63% of 800 releases, versus 82.13% for Direct-Rank. Direct-Rank had significantly higher fault-revealing-test recall, while TRACE-RTS attained 99.74% of the impact weight reachable by the full suite and reached a first fault after 6.69% of full-suite cost. In Defects4J, TRACE-RTS, Direct-Rank, and Direct-Greedy produced identical results at every budget. Class-impact coverage was zero below 30% and saturated at 100% thereafter. Random was descriptively higher at four of five budgets, but nine-fault bootstrap intervals included zero. Conclusion: The evidence supports a tradeoff, not TRACE-RTS superiority. More importantly, the pilot demonstrates that relevant-tests-only, class-level evidence is too coarse to evaluate a function-level RTS contribution. Method-level, all-tests, multi-project evaluation is required.
References
[1] G. Rothermel and M. J. Harrold, "Analyzing regression test selection techniques," IEEE Transactions on Software Engineering 22(8) (1996) 529-551. https://doi.org/10.1109/32.536955.
[2] G. Rothermel and M. J. Harrold, "A safe, efficient regression test selection technique," ACM Transactions on Software Engineering and Methodology 6(2) (1997) 173-210. https://doi.org/10.1145/248233.248262.
[3] G. Rothermel and M. J. Harrold, "Empirical studies of a safe regression test selection technique," IEEE Transactions on Software Engineering 24(6) (1998) 401-419. https://doi.org/10.1109/32.689399.
[4] R. Kazmi, D.N.A. Jawawi, R. Mohamad, I. Ghani, "Effective regression test case selection: A systematic literature review," ACM Computing Surveys 50(2) (2017), Article 29. https://doi.org/10.1145/3057269.
[5] H. Zhong, L. Zhang, S. Khurshid, "TestSage: Regression test selection for large-scale web service testing," Proceedings of ICST (2019) 430-440. https://doi.org/10.1109/ICST.2019.00052.
[6] M. Machalica, A. Samylkin, M. Porth, S. Chandra, "Predictive test selection," Proceedings of ICSE-SEIP (2019) 91-100. https://doi.org/10.1109/ICSE-SEIP.2019.00019.
[7] G. Zhang, L. Liu, Z. Chen, J. Wang, "Hybrid regression test selection by integrating file and method dependences," Proceedings of ASE (2024) 1557-1569. https://doi.org/10.1145/3691620.3695525.
[8] R. Just, D. Jalali, M.D. Ernst, "Defects4J: A database of existing faults to enable controlled testing studies for Java programs," Proceedings of ISSTA (2014) 437-440. https://doi.org/10.1145/2610384.2628055.
[9] S. Yoo, M. Harman, "Regression testing minimization, selection and prioritization: A survey," Software Testing, Verification and Reliability 22(2) (2012) 67-120. https://doi.org/10.1002/stvr.430.
