Abstract
Verification & Validation: any activities that seek to assure that a software system meets the users’ needs.
The principle objectives are
the discovery of defects in a system, and
the assessment of whether or not the system is usable in an operational situation.
The most familiar form of V&V is testing.
Through the rest of this module, we will indeed be taking a close look at testing, unit testing in particular. Before doing that, however, it’s worth noting that testing is not the only way to do V&V.
We will look at various forms of
as alternatives or, more often, supplements to testing.
Verification & Validation
Verification:
Validation:
Testing
Testing is the act of executing a program with selected data to uncover bugs.
Testing is the most common, but not the only form of V&V
What Can We Find?
Fault: A defect in the source code.
Failure: An incorrect behavior or result.
Error: A mistake by the programmer, designer, etc., that led to the the fault.
Industry figures of 1-3 faults per 100 statements are quite common.
Static Verification
Verifying the conformance of a software system and its specification without executing the code
Involves analyses of source text by humans or software
Can be carried out on ANY documents produced as part of the software process
Discovers errors early in the software process
Usually more cost-effective than testing for defect detection at the unit and module level
Allows defect detection to be combined with other quality checks
Static verification effectiveness
It has been claimed that
More than 60% of program errors can be detected by informal program inspections
More than 90% of program errors may be detectable using more rigorous mathematical program verification
The error detection process is not confused by the %existence of previous errors
Inspecting the code in an effort to detect errors
Desk Checking
Inspection
An exercise conducted by the individual programmer.
“Playing computer” with the aid of a listing.
Can be done with pseudocode, diagrams, etc. even before code has been written
A good way to find fundamental flaws in algorithms, especially before actually writing the code.
Useful in checking the results of an intended change during debugging.
Formalized approach to document reviews
Intended explicitly for defect detection (not correction)
Defects may be logical errors, anomalies in the code that might indicate an erroneous condition (e.g. an uninitialized variable) or non-compliance with standards
Inspection pre-conditions
A precise specification must be available.
Team members must be familiar with the organization standards.
Syntactically correct code must be available.
An error checklist should be prepared.
Management must accept that inspection will increase costs early in the software process.
Management must not use inspections for staff appraisal.
Inspection procedure
Code and associated documents are distributed to inspection team in advance
Inspection takes place and discovered errors are noted
After inspection meeting,
Inspection teams
Inspection rate
Inspection checklists
Checklist of common errors should be used to drive the inspection
Error checklist is programming language dependent
The “weaker” the type checking, the larger the checklist
Examples: Initialization, Constant naming, loop termination, array bounds, etc.
Inspection checks
What kinds of faults would appear in a checklist?
Data Faults
Control Faults
I/O Faults
Interface faults
Storage Mgmt Faults
Exception Mgmt Faults
Stylistic/standards Faults
Verification is based on mathematical arguments which demonstrate that a program is consistent with its specification
Programming language semantics must be formally defined
The program must be formally specified
Program proving
Rigorous mathematical proofs that a program meets its specification are long and difficult to produce
Some programs cannot be proved because they use constructs such as interrupts.
The cost of developing a program proof is so high that it is not practical to use this technique in the vast majority of software projects
Program verification arguments
Less formal, mathematical arguments can increase confidence in a program’s conformance to its specification
Must demonstrate that a program conforms to its specification
Must demonstrate that a program will terminate
Model Checking
Simplified models on which properties can be proved
Focus on properties short of correctness
Machine-Assisted
Software tools for source text processing
Try to discover potentially erroneous conditions in a program and bring these to the attention of the V & V team
Very effective as an aid to inspections. A supplement to but not a replacement for inspections
Static analysis checks
What kinds of faults can be detected by static analysis?
Data faults
Control faults
Interface faults
Storage mgmt faults
Stages of static analysis
Control flow analysis.
Data use analysis.
Interface analysis.
Information flow analysis.
Path analysis.
These stages generate vast amounts of information.
The name is derived from the ‘Cleanroom’ process in semiconductor fabrication. The philosophy is defect avoidance rather than defect removal.
Cleanroom process teams
Responsible for developing and maintaining the system specification
Responsible for developing and verifying the software. The software is NOT executed during this process
Responsible for developing a set of statistical tests to exercise the software after development.
Reliability growth models used to determine when reliability is acceptable
Results in IBM have been very impressive with few discovered faults in delivered systems
Some independent assessment shows that the process is no more expensive than other approaches
Fewer errors than in a ‘traditional’ development process
Not clear how this approach can be transferred to an environment with less skilled or less highly motivated engineers