Module 9 - T201

T201: How to Write a Test Plan

HTML of the PowerPoint Presentation

(Note: This document has been converted from a PowerPoint presentation to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)

Slide 1:

(Extended Text Description: Graphic image of introduction information. A large dark blue rectangle with a wide, light grid pattern at the top half and bands of dark and lighter blue bands below. There is a white square logo box with words “Standards ITS Training” in green and blue on the middle left side. The word “Welcome” in white is to the right of the logo. Under the logo box are the words “RITA Intelligent Transportation Systems Joint Program Office.” )

Slide 2:

Welcome

Shelley Row, P.E., PTOE

Director

ITS Joint Program Office

Shelley.Row@dot.gov

(Extended Text Description: Slide 2: Screen capture snapshot of RITA website - for illustration only. Below this image is a link to the current website: https://www.its.dot.gov/pcb - this screen capture snapshot shows an example from the RITA website from June 3, 2011. At the top of the page it shows the RITA logo with the text Research and Innovative Technology Administration - Intelligent Transportation Systems. Below the main site banner, it shows the main navigation menu with the following items: About RITA, Communities of Interest, Contact Us, Press Room, RITA Offices, Site Map, and a Search button. Below the main navigation menu, it shows a sub-navigation menu with the following items: About Us, T3 Webinars, ITS Peer-to-Peer, Resources, Local ITS PCB and Testimonials. Beneath the sub-navigation menu, the page is sub-titled "ITS Professional Capacity Building Program" and is divided into sub-sections such as "Welcome to ITS Professional Building", "News", "ITS Technical Assistance" and "Scheduled T3 Webinars". Again, this image serves for illustration only. The current website link is: https://www.its.dot.gov/pcb)

ITS PCB Home

(Note: There is additional text attached to this slide that includes the following introductory information from Shelley Row):

"ITS Standards can make your life easier. Your procurements will go more smoothly and you’ll encourage competition, but only if you know how to write them into your specifications and test them. This module is one in a series that covers practical applications for acquiring and testing standards-based ITS systems.

I am Shelley Row the director of the ITS Joint Program Office for USDOT and I want to welcome you to our newly redesigned ITS standards training program of which this module is a part. We are pleased to be working with our partner, the Institute of Transportation Engineers, to deliver this new approach to training that combines web based modules with instructor interaction to bring the latest in ITS learning to busy professionals like you.

This combined approach allows interested professionals to schedule training at your convenience, without the need to travel. After you complete this training, we hope that you will tell colleagues and customers about the latest ITS standards and encourage them to take advantage of the archived version of the webinars.

ITS Standards training is one of the first offerings of our updated Professional Capacity Training Program. Through the PCB program we prepare professionals to adopt proven and emerging  ITS technologies that will make surface transportation safer, smarter and greener which improves livability for us all. You can find information on additional modules and training programs on our web site ITS PCB Home.

Please help us make even more improvements to our training modules through the evaluation process. We look forward to hearing your comments. Thank you for participating and we hope you find this module helpful."

Slide 3:

T201

How to Write a Test Plan

Slide 4:

Target Audience

• Engineering staff
• Operational staff
• Maintenance staff
• Testing staff (testing personnel and systems integrators)

Slide 5:

Instructor

Ken Vaughn, P.E.

President

Trevilon Corporation Herndon, VA, USA

Slide 6:

Curriculum Path (Testing)

(Extended Text Description. This flow chart with a series of six boxes is entitled "Curriculum Path (Testing)." There are three horizontal blue boxes with rounded, beveled edges. The titles for all boxes appear in white inside the boxes. The first box on the left is entitled "T101 Introduction to ITS Standards" and it has a black arrow from its right side pointing to the right at the next box. The second (middle) box is highlighted with words "T201 How to Write a Test Plan" and a black arrow from its right side pointing to the right to the third box. The third box is entitled "T202 Overview of Test Design Specifications, Test Cases, and Test Procedures." This final box has three arrows leading from its upper right, middle right and lower right sides to a series of three vertical blue boxes with rounded, beveled edges. The titles for these boxes also appear in white in the center of each box. The top box (to which the upper right arrow is pointing) is entitled "T311 Applying Your Test Plan to NTCIP 1203 v03 DMS Standard." The second (middle) box underneath the first box (to which the middle right arrow is pointing) is entitled "T313 Applying Your Test Plan to the NTCIP 1204 v03 ESS Standard." The third box at the bottom (to which the bottom right arrow is pointing) is entitled "T3XX Applying Your Test Plan to NTCIP / TMDD / ATC Standards.")

Slide 7:

Recommended Prerequisites

• T101: Introduction to ITS Standards Testing
• Helpful to have knowledge of
  • Intelligent Transportation Systems (ITS)
  • Systems engineering process (SEP)
  • Acquisition process for standards-based ITS procurements

Slide 8:

Learning Objectives

1. Discuss the role of a test plan within the testing lifecycle and SEP
2. Summarize the characteristics of a good test plan
3. Present the outline of a test plan
4. Describe relationship among test plans and test design specifications

Slide 9:

Slide 10:

Online Poll

• When should you test?
  1. When there is a prototype
  2. Prior to delivery (e.g., in manufacturer's factory)
  3. Upon installation at the site
  4. All of the above
  5. It depends on the system being acquired

(Additional note from the original PowerPoint presentation: In the original PowerPoint presentation, Slide 10 appeared as above, then afterwards, a red arrow appeared pointing to "5. It depends on the system being acquired")

Slide 11:

Learning Objective #1

Testing and the Project Lifecycle

(Extended Text Description: Systems Engineering Process (SEP). A graphic of the systems engineering process (SEP). The main graphic of the SEP is a V-shaped diagram in gradated yellow with some additional horizontal extensions on the left and right side of the top of the V shape. Each section is separated with dark blue lines. There is a key at the lower right showing the blue separator lines, and designating them as “Decision Gate.” This modified version of the V Diagram begins on the left horizontal extension with a blue line and "Project Planning" leading into sections (separated by a white space) "Systems Engineering Management Planning." At this point the sections begin to descend the left side of the V with (blue line) “Concept of Operations,” (blue line) “System Requirements,” (blue line) “High-Level Design,” “Subsystem Requirements,” (blue line) “Detailed Design,” (blue line) and “Software Coding / Hardware Fabrication” at the bottom juncture of the V shape. Underneath the bottom point of the V shape are the words "Life Cycle Time Line” and a long thin horizontal arrow pointing to the right. There is a long thin diagonal arrow pointing down along the left side of the V labeled “Decomposition and Definition.” From the bottom point of the V, the sections begin to ascend up the right side of the V with (blue line) “Unit Testing,” (blue line) “Subsystem Integration,”“Subsystem Verification,” (blue line) “System Integration,”“System Verification,” (blue line) “Initial Deployment,” “System Validation,” and (blue line) “Operations and Maintenance.” There is a long thin arrow pointing up along the right side of the V shaped labeled “Integration and Recomposition.” At this point the sections on the right “wing” of the V are labeled with a white curving space to “Changes and Upgrades.” Between the V shape there are a series of black arrows pointing from left to right, connecting the related sections on each left/right side of the V shape. The first arrow (top) connects “Concept of Operations” on the left and “System Validation” on the right. The second arrow connects “System Requirements” on the left and “System Verification” on the right. The third arrow connects “Subsystem Requirements” on the left and “Subsystem Verification” on the right. The last arrow (at the bottom) "Detailed Design” on the left and “Unit Testing” on the right. There is a large red oval shape placed over the right diagonal side of the V diagram, seeming to include all sections from "Unit Testing" at the bottom to "System Validation" at the top.)

Slide 12:

Learning Objective #1

When to Test

• It depends on the system being acquired
• Test as needed, which may include:
  • When there is a prototype
  • Once design is complete
  • In manufacturer's factory, prior to delivery
  • Upon delivery
  • Upon installation at the site
  • After all components are integrated together

Slide 13:

Learning Objective #1

Types of Testing: Verification

• Ensuring the system is built "right" (according to specifications) through:
  • Inspection
  • Demonstration
  • Analysis
  • Testing

Slide 14:

Slide 15:

Online Poll

• Who should perform verification testing on behalf of the client?
  • Development staff
  • Engineering staff
  • Operational staff
  • Testing staff
  • Other (please send chat to explain)

(Additional note from the original PowerPoint presentation: In the original PowerPoint presentation, Slide 15 appeared as above, then afterwards, red arrows appeared pointing to "Development Staff" and "Testing Staff")

Slide 16:

Learning Objective #1

Types of Testing: Validation

• Ensuring the "right" system has been built - A system that meets the real user needs

Slide 17:

Slide 18:

Online Poll

• Who should perform validation testing on behalf of the client?
  • Development staff
  • Engineering staff
  • Operational staff
  • Testing staff
  • Other (please send chat to explain)

(Additional note from the original PowerPoint presentation: In the original PowerPoint presentation, Slide 18 appeared as above, then afterwards, a red arrow appeared pointing to "Operational Staff")

Slide 19:

Learning Objective #1

Strategy for Testing

• "V" diagram specifies several testing steps
• Each project must define:
  • When each requirement is tested
  • Where requirement is tested
  • How each requirement is tested
  • Who tests each requirement
• All requirements need to be tested
  • Functional, interface, environmental, etc.

Slide 20:

Slide 21:

Learning Objective #1

Sample Strategy: New Standard

• VDOT Testing for NTCIP 1203v1 (Version 1 Message Sign)
  • Prototypes were required to pass 85% of NTCIP tests to be pre-qualified to be used on bids
  • Factory acceptance required passing 100% of NTCIP tests (as well as most other tests)
  • Site acceptance required integration with system

Slide 22:

Learning Objective #1

Sample Strategy: Stable Standard

• Typical DMS testing today
  • Factory acceptance for hardware requirements
  • Site acceptance testing of initial sign for all NTCIP tests
  • Site acceptance required integration with system

Slide 23:

Learning Objective #1

Sample Strategy: Management System

• Testing for an ATMS (i.e., central system)
  • Inspection of test reports from developer
  • Testing of system in agency test lab with sample devices
  • Load testing of system in agency test lab with simulated users and devices
  • Testing of partial deployment
  • Testing of full deployment

Slide 24:

Slide 25:

Online Poll

• How many test plans should be developed for a project?
  • One
  • Two
  • One for each test phase
  • Multiple for each test phase
  • It depends

(Additional note from the original PowerPoint presentation: In the original PowerPoint presentation, Slide 25 appeared as above, then afterwards, a red arrow appeared pointing to "It depends")

Slide 26:

Learning Objective #2

Introduction to IEEE 829

• At least one test plan per testing phase
• May have distinct plans for different categories of testing
  • Functional
  • Interface
  • Environmental
• All test plans are developed after requirements
• Each test plan is developed prior to starting tests

Slide 27:

Slide 28:

Learning Objective #2

Sample Test Plan

• Please find the sample test plan in your participant supplement

Go to page 6 in the supplement

Slide 29:

Learning Objective #3

Introduction

• Test plan identifier
• Objectives
  • Types of requirements
  • Testing phase
• Project Background
• References

Page 6 in the supplement

Slide 30:

Learning Objective #3

Identifying the Test Items

• Item to be tested
  • Version of the product
  • Specific version of the requirements

Page 7 in the supplement

Slide 31:

Learning Objective #3

Features to be Tested

• Identifies requirements that will be tested
  • Compare to Protocol Requirements List (PRL) contents
  • Listing is as long as necessary
• Identifies requirements that are not tested

Pages 8 and 13-19 in the supplement

Slide 32:

Slide 33:

Online Poll

• Where do you find the requirements list when the standard does not include SEP content?
  • Define them in the test plan
  • Refer to project requirements
  • Refer to design specification
  • Refer to user guide

(Additional note from the original PowerPoint presentation: In the original PowerPoint presentation, Slide 33 appeared as above, then afterwards, a red arrow appeared pointing to "Refer to project requirements")

Slide 34:

Learning Objective #3

Approach: Standards with Test Cases

• Refer to Requirements to Test Case Traceability Table in standard
  • See Participant Supplement for sample
• Identify activities to be performed
• Identify tools that are needed
• Enough detail to estimate amount of work

Page 8 and 20-21 in the supplement

Slide 35:

Learning Objective #3

Approach: Standards without Test Cases

• High-level overview of how item will be tested
  • Identify activities to be performed
  • Identify tools that are needed
  • Enough detail to estimate amount of work

Page 8 in the supplement

Slide 36:

Learning Objective #3

Pass/Fail Criteria

• Must clearly specify what constitutes passing
  • Prevents debates later
  • Usually requires 100% success

Page 8 in the supplement

Slide 37:

Learning Objective #3

Suspending the Test

• Testing takes time
• Where can testing be paused
• What steps must be undertaken to restart testing

Page 8 in the supplement

Slide 38:

Learning Objective #4

Test Deliverables: Preparation

• Requirements
• Test design specification
• Test case specification
• Test procedure specifications
• Test item transmittal report

Page 8 in the supplement

Slide 39:

Learning Objective #4

Test Deliverables: Results

• Test summary
• Test incident reports
• Test logs

Page 9 in the supplement

Slide 40:

Learning Objective #3

Testing Tasks

• Task description
• Predecessors
• Responsible party
• Skills required
• Effort required

Page 9 in the supplement

Slide 41:

Slide 42:

Tasks Involving Testing

• What are some of the tasks involved in testing?
  • Developing the test plan, test cases, and procedures
  • Receiving equipment
  • Setting up the test environment
  • Performing the tests
  • Recording test results
  • Summarizing test results

Slide 43:

Learning Objective #3

Environmental Needs

• Major components and connections
• Testing software used
• Configuration of each piece of equipment
• Practical and logistical needs to perform the test
  • Electrical outlets
  • Tables, chairs, lighting, protection from elements
  • Safety considerations such as safety vests

Page 9 in the supplement

Slide 44:

Learning Objective #3

Roles and Responsibilities

• Define each major stakeholder in the test
• Identify responsibilities of each stakeholder

Page 11 in the supplement

Slide 45:

Learning Objective #3

Schedule

• Define the expected start and end for each testing task
• Identify dependencies on other project tasks
  • Dependencies within the project
  • Dependencies with other projects
  • Dependencies on resources
• Schedule often shown as weeks from start of testing
• Schedule defines the length of the relevant step in the "V" diagram

Page 12 in the supplement

Slide 46:

Slide 47:

Discussion

• During testing, what are some of the problems that may arise on a project? Can anyone give concrete examples that may have happened to you or your colleagues?

Slide 48:

Learning Objective #3

Risks and Contingencies

• What are the risks?
  • Delay in development
  • Delay in other projects
  • Resources unavailable
  • Defects found during testing
• What happens if delays occur
  • Many times the delay just delays the testing
  • May be constrained by other events
    • System being installed for a special event

Page 12 in the supplement

Slide 49:

Learning Objective #3

Approvals

• Approves the plan before testing starts
  • Agency
  • Developer
  • Tester

Page 12 in the supplement

Slide 50:

What We Learned

• Testing occurs throughout theRight side of the "V" diagram.
• Testing should follow an overall Strategy .
• Test plans should follow the IEEE829 outline.
• Test plan is one of several testing Documents .
  • Detailed steps are defined using:
    • Test Design Specifications_
    • Test Case Specifications
    • Test Procedure Specifications_
  • Results are reported in:
    • Test Summary
    • Incident Reports
    • Test Log

Slide 51:

Curriculum Path (Testing)

(Extended Text Description. This flow chart with a series of six boxes is entitled "Curriculum Path (Testing)." There are three horizontal blue boxes with rounded, beveled edges. The titles for all boxes appear in white inside the boxes. The first box on the left is entitled "T101 Introduction to ITS Standards" and it has a black arrow from its right side pointing to the right at the next box. The second (middle) box is entitled "T201 How to Write a Test Plan" and a black arrow from its right side pointing to the right to the third box. The third box is highlighted and entitled "T202 Overview of Test Design Specifications, Test Cases, and Test Procedures." This final box has three arrows leading from its upper right, middle right and lower right sides to a series of three vertical blue boxes with rounded, beveled edges. The titles for these boxes also appear in white in the center of each box. The top box (to which the upper right arrow is pointing) is entitled "T311 Applying Your Test Plan to NTCIP 1203 v03 DMS Standard." The second (middle) box underneath the first box (to which the middle right arrow is pointing) is entitled "T313 Applying Your Test Plan to the NTCIP 1204 v03 ESS Standard." The third box at the bottom (to which the bottom right arrow is pointing) is entitled "T3XX Applying Your Test Plan to NTCIP / TMDD / ATC Standards.")

Slide 52:

Where to Learn More

• Module Supplement

Slide 53: