IMB - USER ACCEPTANCE TEST STANDARDS

USER ACCEPTANCE TEST STANDARDS

Version 1.0 February 16, 2000

This document contains the User Acceptance Testing standards for Application Development used within the Ministry. It includes a template and instructions for the creation of a User Acceptance Test Plan.

Table of Contents

VERSION CONTROL

1. INTRODUCTION

1.1    Audience
1.2    Purpose
1.3    Assumptions
1.4    Other Standards

2. USER ACCEPTANCE TEST STANDARDS

2.1    Overview
2.2    New System Test
2.3    Regression Test
2.4    Limited Testing

2.4.1    Form-Based Testing
2.4.2    Business Process Testing
2.4.3   Report Testing

3. ASSESSING THE TEST TYPES FOR A TEST PLAN

3.1 Overview
3.2 Assessing the Test Type Required

4. CREATING A USER ACCEPTANCE TEST PLAN

4.1    Test Instructions
4.2    Form-Based Test Scripts
4.3    User Security Matrix
4.4    Business Process Test Scripts
4.5    Report Test Scripts
4.6    Defect Tracking Form
4.7    Defect Tracking Log

5. CONCLUSION

APPENDICES

Appendix A: USER ACCEPTANCE TEST PLAN TEMPLATE

VERSION CONTROL

This section of the document records the various versions or releases of this document.

Version	Description	Distribution	Date	Author	Organization
0.1	First Draft	Whole Document	January 17, 2000	Diane Barrand	Pangaea
1.0	QA/Edits	Whole Document	February 3, 2000	Diana Von Ratenberg	MSRM IMB

1. Introduction

This section outlines the Audience and Purpose of this standards document.

1.1 Audience

This document is to be used by the Application Managers or anyone tasked with preparing a User Acceptance Test Plan for the Ministry.

1.2 Purpose

The purpose of this document is to define the standards and deliverables for User Acceptance Testing for the Ministry.

This document also provides a template and instructions for developing a User Acceptance Test Plan for a specific application.

1.3 Assumptions

In order to make effective use of this standard; the audience must be familiar with the Ministry 's System Development Life Cycle Overview, of which this standard is part.

1.4 Other Standards

Refer to System Development Life Cycle Overview for more information.

User Acceptance Testing is part of the Implementation Phase in the MSRM Systems Development Life Cycle (the 6^th Phase in the Systems Development Life Cycle). This is the phase where the application is delivered to the Ministry for user acceptance.

2. User Acceptance Test Standards

The purpose of this section is to define the levels of testing required for User Acceptance Testing, depending on the extent of change being applied to the application.

2.1 Overview

This section is a descriptive overview, which will assist in developing a User Acceptance Test Plan as described in Section 3 of this document.

The possible levels of testing required in a User Acceptance Test Plan are:

‹ New System Test: where the application to be tested is entirely new (is not an enhancement or system upgrade).

‹ Regression Test: where the amount of change in an existing application requires a full system retest.

‹ Limited Test: where the amount of change in an existing application requires only change specific testing.

2.2 New System Test

Purpose:

‹ To ensure that the system meets all specified objectives.

‹ To ensure that all requirements are included in the new application.

When to begin planning: Plan the User Acceptance Test as early in the life cycle as possible. It is important that the original project scope is verified in the User Acceptance Test Plan by including the key points of the original project documentation in the test scripts.

The Following Material can Contribute to the Test Plan (if available):

‹ Systems Analysis documentation

‹ System Design Document

‹ Project Statement

‹ Technical documentation

‹ Online Help/Documentation

‹ User Training material

‹ User Manual

‹ Application Change Requests

‹ Contractor Test plans

‹ Contractor Test Results

‹ Similar User Acceptance Test Plans (from other applications)

Include Test Script Types :

‹ Form-Based test scripts

‹ Business Process test scripts

‹ Report test scripts

2.3 Regression Test

Purpose:

‹ To ensure the "entire" system is working correctly.

‹ To ensure that application changes have not changed existing functionality.

‹ To ensure that the new development work meets all requirements.

Material to Review:

‹ Requirement documentation

‹ System Design documentation

‹ Project Statement

‹ Technical documentation specific to the changes

‹ Online Help/Documentation

‹ User Training material

‹ User Manual

‹ Contractor Test plans

‹ Contractor Test Results

‹ Previous User Acceptance Test Plans

Include Test Script Types :

‹ Form-Based test scripts

‹ Business Process test scripts

‹ Report test scripts

2.4 Limited Testing

The section below outlines the different types of testing in cases where a full system test is not required.

There are three types of limited testing: :

‹ Form-Based test scripts

‹ Business Process test scripts

‹ Report test scripts

2.4.1 Form-Based Testing

Purpose:

‹ To verify that individual application forms are performing correctly.

‹ To ensure that all required fields and buttons exist on the application forms.

‹ To ensure that the flow of information and data entry is logical and correct based on the application 's business requirements.

‹ To ensure that a new form added to an existing application is functioning according to specifications.

‹ To verify that new functionality added to an existing form has not adversely affected the existing functionality on that form.

‹ To ensure that the flow of fields on a new form is sensible.

‹ To check common relationships of fields between forms.

‹ To verify navigation between forms.

Material to Review:

‹ Requirement documentation

‹ Technical documentation

‹ Analysis documentation

‹ Contractor Test plans

‹ Contractor Test Results

2.4.2 Business Process Testing

Purpose:

‹ To ensure that all business related functions and processes are supported by the application.

‹ To ensure that the flow of data and screens is logical for each business process.

‹ To ensure that security and access requirements are met.

‹ To test the application with "real" scenarios.

‹ To test all batch processes.

‹ To test for performance related problems.

‹ To test the application 's interfaces.

Material to Review:

‹ Requirement documentation

‹ Technical documentation

‹ Analysis documentation

‹ Contractor Test plans

‹ Contractor Test Results

‹ Previous User Acceptance Test Plans

‹ Security Information

‹ Business Process Documentation

2.4.3 Report Testing

Purpose:

‹ To ensure that the new report meets its requirements.

‹ To ensure that the data extracted for the new report is correct.

‹ To ensure that the report format is correct and logical.

‹ To ensure that the print process is working correctly.

Material to Review:

‹ Requirement documentation

‹ System Design Documentation

‹ Technical Specifications

‹ Contractor Test Plans

‹ Contractor Test Results

3. Assessing the Test Types for a Test Plan

The purpose of this section is to describe the process of planning and assessing the level of testing required for a specific application.

3.1 Overview

To complete a User Acceptance Test Plan for a specific application, the Application Manager must first plan the tests based on the initial application project documentation. By researching the purpose of the development work, and the degree to which this development work will affect the rest of the application, the test planner can then begin to create the scripts for testing. (A Ôtest script ' includes the individual test steps to be executed in order to verify an application is working as expected).

The development of a User Acceptance Test Plan involves a number of iterative steps:

1. Assess the type of testing required

2. Develop the procedures and instructions for testing

3. Develop the necessary test scripts

4. Execute the test scripts

5. Report any defects

6. Retest any fixes

This section covers the first of these steps. The remaining steps are covered in Section 4 of this document.

3.2 Assessing the Test Type Required

The criteria for determining the degree and type of testing that may be required is listed in the chart below. Use this chart as a guide for determining what test scripting will be required for your particular User Acceptance Test Plan. If the application changes to be tested fall in more than one criterion Ð multiple test script types may be required. Once you have determined the appropriate test types to follow, complete the User Acceptance Test Template as described in the section below.

Application Criteria	Test Criteria
New Application – (not replacing an existing application).	The Application Manager developing the User Acceptance Test Plan should be as involved as possible in the design and reviews of the new application with the contractor. The User Acceptance Test Plan should be developed with communication from the contractor and with as much information gathered through the system documentation as possible.
New Application – (replacing an existing application).	The Test Plan should be developed using the required aspects of the system that is being replaced. Test scripts for any enhancements to the new (replacement system) should be developed as if the application is a new application – basing information of requirement and design documents. Running a parallel test with the old system, and comparing critical report results would be the optimal test scenario for any functionality that is to be duplicated in the new system. .
Database Change – (no other change to the application e.g. upgrade from Oracle 7.3 to Oracle 8.05)	In this case running a parallel test with the system using the old database and the system using the new database is advised. Comparing critical report results would be the optimal test scenario to ensure that the new application database and drivers are producing identical results to the old system. Performance testing should be included in these test criteria. Create a User Acceptance Test Plan to manage the parallel test.
Application Enhancements – (the existing application is enhanced with new functionality).	Form-based test scripts should be developed for any forms that are being added to the existing application to ensure that each of the new forms is behaving correctly and is meeting the requirements. Business process test scripts should be developed and tested to ensure that the new functionality is integrated with the existing application correctly, and to ensure that no existing functionality has been lost in the enhancement process. Test scripts should be developed based on the specific requirements for any new reports. In this case, unchanged forms may not require testing. E.g. if they are not involved in any business processes which have been changed.
Application Enhancement – (the existing application is being enhanced to change the existing functionality).	New form-based test scripts should be developed based on the requirements documents to ensure that the application meets its enhanced requirements. Existing test scripts can be amended in this case. Business process test scripts should be developed and tested to ensure that the enhanced functionality is integrated with the existing application correctly, and to ensure that no existing functionality has been lost in the enhancement process. Test scripts should be developed based on the specific requirements for any enhanced reports. In this scenario, any form that has been changed, or is affected by a change, new data or new functionality should be tested to assure that existing functionality is not lost. This will ensure that the new requirements are being met.
Infrastructure Change – (the application is not changing but a test is required to port it to a new environment, new server, or the application be being utilized for a new division or purpose).	In this case, a full regression test of the form-based test scripts and all business processes should be repeated as new environments can create unexpected problems for an existing application.

4. Creating a User Acceptance Test Plan

The User Acceptance Test Plan Template is an independent document, which is intended to serve as a tool for the creation of an application specific User Acceptance Test Plan. This template can be downloaded and edited in MSWord.

4.1 Test Instructions

How to Customize the Instruction Set:

‹ Include instructions for components assessed as necessary.

‹ Delete the unnecessary instructions from the template.

‹ Modify the appropriate application specific information and resources.

4.2 Form-Based Test Scripts

How to Customize the Form-Based Test Module List and Scripts:

‹ List all modules to be included in the User Acceptance Test

‹ Add any application specific steps to the test script

‹ Remove any steps from the sample test script that are not relevant to your application

Examples:

Module ID	Module name
CRS0000	Main Menu
CRS1020	Enter a License Scenario
CRS1080	Enter an Authorization Scenario
CRS1050	Enter an Exam Mark Scenario
CRS1030	View all Licenses Scenario
CRS1020	Update a License Scenario

Figure 1: List of Modules Example

Form Based Testing Component	Date	Initials
1. Are all fonts, colors, shading and toolbars consistent with standards and project guidelines?
2. Is the online help module available?
3. Are all date formats correct (DD-MON-YYYY) Are the date edits being correctly applied? Are dates greater than 2000 accepted?
4. Does all text wrap when displayed in the text editor?
5. Is there a scroll bar on every applicable block?

Figure 2: Form-Based Test Script Example

4.3 User Security Matrix

How to Customize the User Security Matrix:

Fill in the blanks for the table for each appropriate User Security Level to be tested. If your application does not have multiple security levels or multiple modules, this matrix will not be necessary for your User Acceptance Test Plan.

‹ User Role : List the Security Level to be tested

‹ Module Label: List the technical id for the Module (eg. CRS1010)

‹ Module description: describe the Module name and purpose (eg.Update License information)

‹ Module Type: Select from types: UPDATE, VIEW, CREATE NEW or REPORT

Examples:

	User Security/Access Level Matrix
	USER ROLE	Module Label	Module Description	Details	Module Type
1	SYS_ADMIN	CRS0000	Main Menu	Access to all buttons	Menu
2	SYS_ADMIN	CRS1020	Enter License	Access to all functionality	Update
3	CLERK_1	CRS0000	Main Menu	Access to View Buttons only Ð other buttons "grayed" out	Menu

Figure 3: User Security Matrix Example

4.4 Business Process Test Scripts

How to Customize the Business Process Test Scripts:

‹ Delete the sample business process test scripts

‹ List all business functions per user Security level as test steps

Examples:

Acceptance Testing Action	Date	Initials
1. System Administrator logs on
2. Is the correct module name/date/version # displayed at the top of the window?
3. Are the appropriate buttons available (system admin has access to all)?
4. Are the appropriate menu options available (system admin has access to all)

Figure 4: Business Process Test Script Example (Main Menu)

4.5 Report Test Scripts

How to Customize the Report Test Scripts:

‹ Delete the sample business process test scripts

‹ List all report functions as test steps

Examples:

Acceptance Testing Action	Date	Initials
1. Can you access the Report module from the main menu?
2. Are all appropriate reports listed based on the Security of the user being tested?
3. Can you access the Detailed Information Report from the list of available reports?
4. Is the correct report name/date/version # displayed at the top of the window?
5. Are the appropriate buttons available (based on the security matrix)?

Figure 5: Report Test Script Example

4.6 Defect Tracking Form

How to Customize the Defect Tracking Form:

‹ Modify the User Security levels listed in the form to match your application

‹ Add any key information you wish to have reported with defects

4.7 Defect Tracking Log

How to Customize the Defect Tracking Log:

‹ Add new columns for application specific information added to the Defect Tracking Form

‹ Remove any columns containing information not required in your summary information

Examples:

Defect #	Log #	Form #	Tester	Description	Date Reported	User Level Tested	Severity (1,2,3,4,5)	Repeated (E,S,O,1x)	Date Reported to Contractor	Status
1		CRS001	DB	View License button not available for Clerk 1	99/10/5	CLRK1	1	E	99/10/06	Fixed
2		CRS020	LM	Error on printing	99/10/08	All	1	S	99/10/06	Open
3		CRS030	DB	Field width cuts off description	99/10/09	All	2	E	99/10/10	Fixed
4		CRS030	KA	Sort order incorrect	99/10/10	All	2	1x	99/10/10	Open

Figure 6: Example Defect Tracking Log

5. Conclusion

The test scripts created for a new system, for regression testing or for limited testing should be recycled throughout the application 's life. By editing existing test scripts the User Acceptance Test Planning process can save time and money, as well as maintain the test quality, as key requirement information will be re-used.