Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit

CSCI426/CSCI926 Software Testing and Analysis

Lab – Week 3

1. Read this article https://test.io/resources/blog/what-is-regression-testing about regression testing, and watch an example of how automated regression testing was conducted in practice https://youtu.be/wDpBDqOXRHc.

2. Answer the following questions:

a. What is regression testing?

(Answer is given in the post; also refer to Chapter 22 of the textbook)

What is regression?

When a new version of software no longer correctly provides functionality that should be preserved, we say that the new version regresses with respect to former versions.

Why regression occur?

When building a new version of a system (e.g., by removing faults, changing or adding functionality, porting the system to a new platform, or extending interoperability), we may also change existing functionality in unintended ways. Sometimes even small changes can produce unforeseen effects that lead to new failures. The non-regression of new versions (i.e., preservation of functionality), is a basic quality requirement. Disciplined design and development techniques, including precise specification and modularity that encapsulates independent design decisions, improves the likelihood of achieving non-regression.

What is regression testing?

Testing activities that focus on regression problems are called (non) regression testing. Usually “non” is omitted and we commonly say regression testing.

b. Why is selecting a subset of regression test cases to be executed sometimes needed?

Even when we can identify and eliminate obsolete test cases, the number of tests to be re-executed may be large, especially for legacy software. Executing all test cases for large software products may require many hours or days of execution and may depend on scarce resources such as an expensive hardware test harness. For example, some mass market software systems must be tested for compatibility with hundreds of different hardware configurations and thousands of drivers. (In some software companies, developers could run ~100k regression testing overnight every time a change is made in the code.) Many test cases may have been designed to exercise parts of the software that cannot be affected by the changes in the version under test. Test cases designed to check the behaviour of the file management system of an operating system is unlikely to provide useful information when re-executed after changes of the window manager. The cost of re-executing a test suite can be reduced by selecting a subset of test cases to be re-executed, omitting irrelevant test cases or prioritizing execution of subsets of the test suite by their relation to changes.

c. Subsets of test cases can be selected according to any of several different criteria. An interesting property of some regression test selection criteria is that they do not exclude any test case that could possibly reveal a fault. How would you classify such a property according to the concepts of safe, sound and complete discussed in the lecture (c.f., lecture 2, slide 12)?

This test selection criteria property is a safe property. A safe analysis accepts only correct programs.

An analysis of a program P with respect to a formula F is sound if the analysis returns True only when the program actually does satisfy the formula.  If satisfaction of the formula F is taken as an indication of “passing all tests that could possibly reveal a fault”, then the test selection criteria property also algins with soundness (i.e., no optimistic inaccuracy).