Module 48 - T204 Part 2 of 2
T204 Part 2 of 2: How to Develop Test Procedures for an ITS Standards-Based Test Plan
HTML of the Student Supplement
(Note: This document has been converted from the Student Supplement to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)
T204 Part 2 of 2: How to Develop Test Procedures for an ITS Standards-Based Test Plan, Part 2 of 2
Table of Contents
Introduction/Purpose - 2
Samples/Examples - 2
Reference to Other Standards - 8
Glossary - 8
References - 9
Study Questions - 9
Module Description
This module is part of the non-systems engineering path, thus students must learn "How to Write a Test Plan" (T201), followed by an "Overview of Test Design Specifications, Test Cases, and Test Procedures" (T202), and then "How to Develop Test Cases for an ITS Standards-Based Test Plan" (T203 Part 1 and 2). Part 1 of this module (T204, Part 1) extended this discussion by providing participants with detailed information on how to prepare their own test procedure specifications, including using the test cases and creating test logs, test summaries, and anomaly reports. Part 1 ended with an overview of the Test Procedure Generator, which is an automated tool available at no cost from USDOT that students were allowed to download and try before beginning Part 2 of this module. Part 2, covered here, provides examples of how to perform tests using the TPG.
1. Introduction/Purpose
The purpose of this module is to teach the student how to develop a Test Procedure Specification (TPS) and how that document fits into the testing process and relates to the Test Plan, Test Design Specification, and Test Case Specification. In addition, the student will be taught how to develop the TPS to meet the specific project requirements for the interface (as found in the Protocol Requirements List (PRL) or Needs to Requirements Traceability Matrix (NRTM).
2. Samples/Examples
2.1. Review of T203 and T204.1
(Extended Text Description: The figure depicts the testing flowchart consisting of Test Plan box at the top with a downward arrow to Test Design Specification box. That box has three arrows directed towards three boxes labeled Test Case 1, Test Case 2 and Test Case 3 that are arranged in a horizontal row below. Test Case 1 box has one arrow directed to Test Procedure 1 box below. Test Case 2 box has one arrow directed to Test Procedure 1 box below. Test Case 3 box has two arrows. The first arrow is directed to Test Procedure 2 box, while the second arrow is directed to Test Procedure 3 box. The horizontal row of Test Case boxes is identified by the text Module 203. The horizontal row of Test Procedure boxes is identified by the text Module 204.)
2.2. Using the Test Procedure Generator to generate Test Procedures for a variety of equipment.
- NTCIP 8002 Annex B1
- Role of TPG
- TPG successful installation
- Example Test Procedure for NTCIP 1203v03 Dynamic Message Sign (DMS)
2.3. Installation and operation of TPG
(Extended Text Description: Author's relevant description: The Graphical User Interface (GUI) screenshot is shown, consisting of a Microsoft Windows 7 screen with TPG pulldown menu items on the top banner. These pulldown menu items from left to right are "File", "Test Procedure", "Reports", "Tools", "Session Panel" and "Help" button selections. The left side of the GUI screenshot consists of the vertical "Session Panel" with a list of each item pertaining to the selected standard. Below the TPG menu items are document tabs consisting of "Standard", "Set of Test Procedures", "Current Test Procedure" and "Reports", followed by Embedded Microsoft Word 2010 Document Menu Items consisting of "File", "Home", "Insert Page Layout", "References", "Mailings", "Review" and "View" button selections. Below and to the left of the Session Panel is a main screen area with a sample NTCIP 1203 version v03 title page. Below the title page is a horizontal TPG session status text area. At the bottom is the TPG Command status text area. This GUI layout will be referenced in the later slides. Note that the course presentation does not address each item shown in each area of each screenshot; rather, the relevant pull-down buttons are described along with their effect so that the student is aware of the major GUI functions. Likewise, this description also describes the same pulldown button operation without describing each item shown on the screen. The TPG Graphical User Interface screenshot is shown taken with permission from Center-to-Field Test Procedure Generator User Manual, v2.0, FHWA, July 15, 2015.)
(Extended Text Description: Author's relevant description: Screenshot of the TPG GUI is shown. The embedded MS Word document depicts a two-column table with "Test Procedure 01.00" in the upper left table entry. "Test Procedure" menu is pulled down and "Define Test Procedure Header" is selected in blue. "Test Procedure Header" screenshot is shown taken with permission from Center-to-Field Test Procedure Generator User Manual, v2.0, FHWA, July 15, 2015.)
2.4.Output of TPG in XML format
(Extended Text Description: Author's relevant description: Screenshot of the TPG GUI with the embedded MS Word document depicting the set of Test Procedures in XML format. Again, we do not describe the XML shown, only that the box displays XML outputs that occur during the execution of that command "Displaying XML Test Procedure" screenshot is shown taken with permission from Center-to-Field Test Procedure Generator User Manual, v2.0, FHWA, July 15, 2015.)
2.5.Adapt the generated Test Procedure to procurement contract Terms and Conditions for successful project conclusion
- Lessons learned from unsuccessful ITS projects and how to avoid repetition.
(Extended Text Description: Odds of Success: Pie chart graphic reproduced from FHWA Systems Engineering Handbook, 2007. The pie chart indicates 34% Succeeded, 15% Failed and 51% Challenged.)
- Find and correct problems early in the project
(Extended Text Description: Graphic depicts a grid of five horizontal lines and six vertical lines as a three dimensional graph. The vertical axis is labeled "Phase the Defect is Created", "ConOps", "Requirements", "Design" and "Construction" from top to bottom and located to the left of the first vertical line. The horizontal axis is labeled "ConOps", "Requirements", "Design", "Construction" and "O&M" from left to right and located below the lower horizontal line. Four perpendicular blue triangular planes are located within each horizontal space between the five horizontal grid lines. Each plane extends in decreasing height from the upper plane to the lower plane. A blue oval encircles the left-most tip of the four planes that is labeled "Fix It Here". A black arrow directed vertically is located to the right of the grid labeled "Cost to Correct". "Phase the Defect is Corrected" graphic is shown taken with permission from Systems Engineering for Intelligent Transportation Systems, USDOT, January 2007.)
- Test Procedure defines the steps to perform the test
2.6. Synchronize the Test Procedure specification to the contract Terms and Conditions (T&Cs) for successful contract execution
- The odds are against success. The typical highway project model does not work well for ITS projects
- Structure the contract Terms and Conditions from the viewpoint of the project's end, including test specification, test plan, test scripts, and common equipment, if possible.
(Extended Text Description: Graphic depicts the USDOT Systems Engineering Process (SEP) VEE model. The left-hand side of the VEE depicts six blue boxes, descending towards the lower right with a downward arrow labeled "Decomposition and Definition". The box and the bottom valley of the VEE is labeled "Implementation" below. The boxes are labeled "Regional Architecture", "Feasibility Study / Concept Exploration", "Concept of Operations", "System Requirements", "High Level Design" and "Detailed Design", arranged from upper right to lower left to the bottom of the VEE. The right-hand side of the VEE depicts seven boxes ascending towards the upper right labeled "Integration and Recomposition with an arrow directed upwards. The boxes are labeled "Unit / Device Testing", "Subsystem Verification", "System Verification and Deployment" "System Validation", "Operations and Maintenance", "Changes and Upgrades" and "Retirement / Replacement". Connecting the left and right sides of the VEE and at the bottom of the VEE is one box labeled "Software / Hardware Development Field Installation". "VEE workflow" graphic is shown taken with permission from Systems Engineering for Intelligent Transportation Systems, USDOT, January 2007.)
2.7.Write the reports produced at the end of testing and understand their relationship to successful Procurement Contracts
- Logs, including the data, information, files, and fulfilled requirements that are captured during the test
- Anomaly Report, including a failure description and the investigation process to resolve
- Level Test Report, providing a measure of success compared to the stated goals and scope of the Test Plan
- Master Test Report covering Level Test Reports
(Extended Text Description: Graphic depicts the testing flowchart consisting of Test Plan box at the top with a downward arrow to Test Design Specification box. That box has three arrows directed towards three boxes labeled Test Case 1, Test Case 2 and Test Case 3 that are arranged in a horizontal row below. Test Case 1 box has one arrow directed to Test Procedure 1 box below. Test Case 2 box has one arrow directed to Test Procedure 1 box below. Test Case 3 box has two arrows. The first arrow is directed to Test Procedure 2 box, while the second arrow is directed to Test Procedure 3 box. The Test Procedure 1 box has two arrows, one directed towards Output Records 1 box below and a second arrow directed towards Output Records 2 box below. Test Procedure 2 box has one arrow directed towards Output Records 3 box below. Test Procedure 3 has one arrow directed towards Output Records 4 box below.)
2.8. Use tools to develop the Test Procedures for a sample TPS structure
- Test Procedure Generator (TPG) is an automated tool
- TPG guides the development of uniform Test Procedures
- How to obtain the TPG from USDOT at no cost
- Installation of TPG
- Step-by-step TPG use and the results
- Demonstrating a typical error and how it is handled
- Use of uniform Key Words with a common understanding of the ITS standards
(Extended Text Description: Author's relevant description: Screenshot of the TPG GUI with the embedded MS Word document table filled out in entirety except the "Variable(s) entry remains grayed out. "Populated Test Procedure Document" screenshot is shown taken with permission from Center-to-Field Test Procedure Generator User Manual, v2.0, FHWA, July 15, 2015.)
2.9. Use XML script outputs from TPG as inputs to automated test tools
(Extended Text Description: Author's relevant description: Screenshot of the TPG GUI with the embedded MS Word document depicting the set of Test Procedures in XML format. Again, we do not describe the XML shown, only that the box displays XML outputs that occur during the execution of that command "Displaying XML Test Procedure" screenshot is shown taken with permission from Center-to-Field Test Procedure Generator User Manual, v2.0, FHWA, July 15, 2015.)
3. Reference to Other Standards
- IEEE 829-2008, IEEE Standard for Software and System Test Documentation. IEEE Computer Society, July 18, 2009
- Institute of Transportation Engineers, ATC 5201 Advanced Transportation Controller (ATC) Standard Version 06. ATC Joint Committee, 30 July 2012.
- Institute of Transportation Engineers, ATC 5202 Model 2070 Controller Standard Version 03. ATC Joint Committee, 28 December 2012.
- Institute of Transportation Engineers, Intelligent Transportation System (ITS) Standard Specification for Roadside Cabinets v01.02.17b. ATC Joint Committee, 16 November 2006.
- National Electrical Manufacturers Association, NEMA Standards Publication TS 2-2003 v02.06 Traffic Controller Assemblies with NTCIP Requirements. NEMA, 2003
4. Glossary
| Term | Definition |
|---|---|
| Agency Specification | A document that has been prepared by an agency to define requirements for a subject item or process when procured by the agency. |
| Compliance | A condition that exists when an item meets all of the requirements of an agency specification. |
| Concept of Operations | A document that describes the purpose of a system project, including a description of the current and proposed system, as well as key user needs that the new system is required to address. |
| Conformance | A condition that exists when an item meets all of the mandatory requirements as defined by a standard. It can be measured on the standard as a whole, which means that it meets all mandatory (and applicable conditional) requirements of the standard or on a feature level (i.e., it conforms to feature X as defined in section X.X.X), which means that it meets all mandatory (and applicable conditional) requirements of the feature. |
| IT | Information Technology |
| T&C | Terms and Conditions |
| TCS | Test Case Specification |
| TDS | Test Design Specification |
| TPG | Test Procedure Generator |
| TPS | Test Procedure Specification |
| XML | Extensible Markup Language |
5. References
- IEEE 829: IEEE Standard for Software and System Test Documentation http://standards.ieee.org/findstds/standard/829-2008.html
- NTCIP 1204 v03 ESS, www.ntcip.org
- PCB Website: Module T201, T202, stds training.aspx
- US Department of Transportation, Systems Engineering for Intelligent Transportation Systems. USDOT, January 2007
- Center to Field Test Procedure Generator User Manual, v1.7, Federal Highway Administration. October 18, 2012
6. Study Questions
1. How does the TPG relate to NTCIP 8002?
- Draft standards are verified to NTCIP 8002 compliance
- Unbroken traceability from requirements through testing
- Uniformity of test procedure content and numbering
- All of the above
2. Which is the order of TPG example workflow?
- XML Requirement Step Object Variable
- Object Variable Requirement Step XML
- Requirement Variable Step Object XML
- Object Requirement Variable Step XML
3. ITS project cost and schedule should be developed in which of the following orders?
- From the test procedures back through the project workflow
- From the requirements only
- To enforce the contract terms at the end of the project
- To minimize the planning costs up front