Last modified: Mar 11, 2014
Verification & Validation: assuring that a software system meets the users’ needs
Principal objectives:
The discovery of defects in a system
The assessment of whether or not the system is usable in an operational situation.
Verification & Validation
Testing
Testing is the most common, but not the only form of V&V
What Can We Find?
Fault: A defect in the source code.
Failure: An incorrect behavior or result.
Error: A mistake by the programmer, designer, etc., that led to the the fault.
Industry figures of 1–3 faults per 100 statements are quite common.
Static Verification
Verifying the conformance of a software system and its specification without executing the code
Involves analyses of source text by humans or software
Can be carried out on ANY documents produced as part of the software process
Discovers errors early in the software process
Usually more cost-effective than testing for defect detection at the unit and module level
Allows defect detection to be combined with other quality checks
Static verification effectiveness
It has been claimed that
More than 60% of program errors can be detected by informal program inspections
More than 90% of program errors may be detectable using more rigorous mathematical program verification
The error detection process is not confused by the %existence of previous errors
Inspecting the code in an effort to detect errors
Desk Checking
Inspection
An exercise conducted by the individual programmer.
Can be done with pseudocode, diagrams, etc. even before code has been written
A good way to find fundamental flaws in algorithms, especially before actually writing the code.
Useful in checking the results of an intended change during debugging.
Formalized approach to document reviews
Intended explicitly for defect {\EM detection} (not correction)
Defects may be logical errors, anomalies in the code that might indicate an erroneous condition (e.g. an uninitialized variable) or non-compliance with standards
Inspection pre-conditions
A precise specification must be available
Team members must be familiar with the organization standards
Syntactically correct code must be available
An error checklist should be prepared
Management must accept that inspection will increase costs early in the software process
Management must not use inspections for staff appraisal
Inspection procedure
System overview presented to inspection team
Code and associated documents are distributed to inspection team in advance
Inspection takes place and discovered errors are noted
Inspection teams
Inspection rate
Inspection checklists
Checklist of common errors should be used to drive the inspection
Error checklist is programming language dependent
The “weaker” the type checking, the larger the checklist
Examples: Initialization, Constant naming, loop termination, array bounds, etc.
Inspection checks
What kinds of faults would appear in a checklist?
Data Faults
Control Faults
I/O Faults
Interface faults
Storage Mgmt Faults
Stylistic/standards Faults
Inspection checks
Inspection checks
Inspection checks
Inspection checks
Inspection checks
Inspection checks
Inspection checks
Verification is based on mathematical arguments which demonstrate that a program is consistent with its specification
Programming language semantics must be formally defined
The program must be formally specified
Program proving
Rigorous mathematical proofs that a program meets its specification are long and difficult to produce
The cost of developing a program proof is so high that it is not practical to use this technique in the vast majority of software projects
Program verification arguments
Less formal, mathematical arguments can increase confidence in a program’s conformance to its specification
Must demonstrate that a program conforms to its specification
Must demonstrate that a program will terminate
Model Checking
Machine-Assisted
Software tools for source text processing
Try to discover potentially erroneous conditions in a program and bring these to the attention of the V & V team
Very effective as an aid to inspections. A supplement to but not a replacement for inspections
Static analysis checks
What kinds of faults can be detected by static analysis?
Data faults
Control faults
Interface faults
Storage mgmt faults
Static analysis checks
Static analysis checks
Static analysis checks
Static analysis checks
Stages of static analysis
The name is derived from the ‘Cleanroom’ process in semiconductor fabrication. The philosophy is defect avoidance rather than defect removal.
Cleanroom process teams
Reliability growth models used to determine when reliability is acceptable
Results in IBM have been very impressive with few discovered faults in delivered systems
Fewer errors than in a ‘traditional’ development process
Not clear how this approach can be transferred to an environment with less skilled or less highly motivated engineers
Testing stages
Regression Test: Unit/Integration/System tests that are repeated after a change has been made to the code.
Acceptance Test: A test conducted by the customers or their representatives to decide whether to purchase/accept a developed system.
Testing goals
Unit Test: does it work?
Integration Test: does it work?
System Test: does it work?
Regression Test: has it changed?
Acceptance Test: should we pay for it?
By testing modules in isolation from the rest of the system
Easier to design and run extensive tests
Much easier to debug any failures
Errors caught much earlier
Main challenge is how to test in isolation
Scaffolding
To do Unit tests, we have to provide replacements for parts of the program that we will omit from the test.
Scaffolding is any code that we write, not as part of the application, but simply to support the process of Unit and Integration testing.
Scaffolding comes in two forms
Drivers
Stubs
Drivers
A driver is test scaffolding that calls the module being tested.
Stubs
Replacements for code begin called from the unit under test
Integration testing is testing that combines several modules, but still falls short of exercising the entire program all at once.
Integration testing usually combines a small number of modules that call upon one another.
bottom-up
(start by unit-testing the modules that dont’call anything else, then add the modules that call those starting modules and thest the combination, then add the modules that call those, and so on until you are ready to test main().)
or top-down
(start by unit-testing main() with stubs for everything it calls, then replace those stubs by the real code, but leaving in stubs for anything called from the replacement code, then replacng those stubs, and so on, until you have assembled and tested the entire program).
Types of testing
Testing Process Components
Oracle: The process, person, and/or program that determines if test output is correct
Failure: An execution on which incorrect behavior occurs
Fault: A defect in the code that (may) cause a failure
Error: A human mistake that results in a fault