Module 58 - T307

T307: Applying Your Test Plan to the Advanced Transportation Controller Based on ATC 5201 Standard v06

HTML of the Student Supplement

(Note: This document has been converted from the Student Supplement to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)

T307: Applying Your Test Plan to the Advanced Transportation Controller Based on ATC 5201 Standard v06

Table of Contents

Module Description - 2

Introduction/ Purpose - 2

Case Study - 3

Glossary - 5

References - 7

Study Questions - 7

Icon Guide - 8

1. Module Description

This module is based on the IEEE 829-2008 formats for testing documentation. Note that the ATC 5201 v06 Standard includes an Environmental and Test Procedures section, but does not offer testing documentation preparation information. The IEEE 829-2008 approach has already been amply covered by testing series modules. This module will help users leverage their ATC 5201-based specifications to produce and apply a sample test plan, including test design specifications, test case specifications, and test procedure specifications for ATC. Thus, the development of clear and unambiguous ATC 5201 v06-based testing documentation can be used by system developers and integrators during procurement specifications preparation, system acceptance, and ongoing maintenance efforts. The module will guide agencies in verifying that delivered products conform to ATC 5201 standards and comply with the agency's specifications.

The logical step for the participant is to consider modules in the testing lifecycle that lead up to the T307 module: The Advanced Transportation Controller Based on ATC 5201 Standard v06. These modules are T101, T201, and T202; T203 Parts 1 and 2; and T204.

2. Introduction to Advanced Transportation Controller (ATC)

Advanced Transportation Controller (ATC) is an environmentally hardened field computer used to support roadside software applications such as Signal Control, Ramp Meter, and others. The ATC is normally installed in the field and connected over a wide area network to a central Traffic Management System (TMS) via the NTCIP standard communications protocols. The current ATC 5201 standard is a non-systems engineering (SE) process that was developed by merging elements of shelf-mount NEMA TS 2 Standard and elements of rack-mount agency procurements specifications into an open source Linux platform including standardized hardware, operating system, and Board Support Package (BSP) tools for software application development. ATC 5201 Standard v5.2b was successfully balloted by ITE, NEMA, and ITS, and then published on September 25, 2006. After five years, ATC 5201 can be either reaffirmed or updated from comments received by the Working Group. Rather than reaffirmation, ATC 5201 was updated to become v06 covered by this module. The current standard provides information on environmental testing procedures, self-test software, and BSP. ATC 5201 Standard v06-based user needs and requirements were covered under modules A307a and A307b, respectively, (see reference below). This module will direct participants to consult these two modules for gaining necessary knowledge and understanding of ATC user needs and specifying ATC requirements prior to preparation of ATC testing documentation covered in this module.

You will learn what to test, when to test, and why to test an ATC so that users get what they have specified (a computer platform with interfaces to appropriate cabinet wiring) and learn to prepare testing documentation needed for the agency's ATC procurement specification. Keeping this overall need in mind, the module aims to create ATC test documentation based on standardized formats (IEEE 829-2008) and relating the formats to key elements of ATC 5201 Standard v06. This will ensure central

TMS connectivity and data communication interface with the field ATCs, as well as appropriate configuration of optional elements to interface with cabinet wiring.

3. Case Study: An Outline of an ATC Test Plan

Testing is a process that uses a documented test plan designed to check conformance to the ATC standard. Testing of ATC is done to verify that requirements are fulfilled, reduce risks of misinterpretation between agency and manufacturers, and ensure interoperability. In addition, ATC testing also verifies that open source Linux obligations are covered by testing, as well as that software development tools are supplied by the ATC manufacturer along with sample self-test software source code. A testing process is guided by a test plan, which prescribes the Scope, Approach, Resources, and Schedule for the testing. Some of the testing aspects covered by a test plan include:

• Item(s) to be tested
• Features to be tested
• Features not to be tested
• Testing tasks to be performed
• Personnel responsible for each task
• Risks associated with the plan

The following outline is typically used for ATC testing based on IEEE 829-2008 formats and guidance provided. An agency may be able to inject local project needs and requirements using this template.

1.0 Introduction

1.1 Testing Documentation Identifier

ATC_TP v01.01
ATC Test Plan v01.01
11 June 2016, City of Midsize

1.2 Scope

1.3 References

1.4 Level Test Plan Testing to be Covered

2.0 Details of LTP: Unit/Bench Testing

2.1 Test items and their identifiers

2.2 RCTM (Test Design/Test Procedures)

2.3 List of ATC Features to be tested

• Core features that are required by all ATC configurations
• Optional features needed for interface to cabinet wiring and user interface

2.4 Objects to be tested (RTM)

2.5 Approach

2.6 Item Pass/Fail criteria

2.7 Suspension Criteria and Resumption Requirements

2.8 Test Deliverables

ATC Test Plan
ATC Test Designs
ATC Test Cases
ATC Test Procedures

Reporting results

ATC Test Logs
ATC Test Incident Reports
ATC Interim Test Status Reports
ATC Test Reports (one for each test design)

3.0 Test management

3.1 Planned activities and tasks; test progression

3.2 Environment/infrastructure

3.3 Responsibilities and authority

3.4 Resources and their allocation

3.5 Training

3.6 Schedule: As per project schedule (see main contract)

4.0 General

4.1 Quality assurance procedures

The testing quality will fall under the Testing QA Procedures established by the city and Company Name.

4.2 Metrics

The percentage of test cases passed per test design will be recorded.

4.3 Test coverage

All data elements specified by the SSM PRL and RTM shall be included in at least one test using nominal values. RTCTM guides the linkages.

4.4 Glossary

4.5 Document change procedures and history

4. Acronyms, and ATC Concepts and Terminology

API - Application Program Interface

ATC - Advanced Transportation System

BSP - Board Support Package

DUT - Device Under Test

EEPROM - Electrically-Erasable Programmable Read-Only Memory

FIO - Field Input/ Output

FIT - Fitness Self-Test Software

HW - Hardware

ID - Identification

IP - Internet Protocol

ITS - Intelligent Transportation System

LITSR - Level Interim Test Status Report

LTL - Level Test Log

LTP - Level Test Plan

LTR - Level Test Report

MTP - Master Test Plan

NTCIP - National Transportation Communications for ITS Protocol

OS - Operating System

SEP - Systems Engineering Process

RTCTM - Requirements to Test Case Traceability Matrix

RTM - Requirements Traceability Matrix

SW - Software

TMC - Traffic Management Center

TMS - Traffic Management System

TTM - Test Traceability Matrix

USB - Universal Serial Bus

v - Version

Testing Process: Consider Testing as an activity that is carried out with a series of steps in a lifecycle of an ITS project. To ensure that the system interface delivers what the users have specified, a testing process is necessary to ensure outcomes. Such requirements are "communicated" in the testing documentation, beginning with a Test Plan (Test Case Specification [TCS] is a component of the Test Plan). The purpose of software and software-based systems testing is:

• To help build quality into the software and system during the lifecycle processes and to validate that the quality was achieved
• To determine whether the products of a given lifecycle activity conform to the requirements of that activity, and whether the product satisfies its intended use and user needs
• To Include inspection, demonstration, analysis, and testing of software and software-based system products
• To perform test activities in parallel with development efforts, not just at the conclusion of the development effort

Test Plan: A test plan provides a description of the overall approach to testing all of the requirements to be verified. The Test Plan outlines the scope, approach, resources, and schedule of testing activities. Breakdown of Key Point: This first key point describes a test plan - the master document that will include the test cases. Explaining this key point shows the hierarchical structure (test plan - test design specification - test case) that is required to develop a fully system-engineered test plan.

Test Design Specification (TDS): A test design breaks apart testing into smaller efforts and describes a test design specification - the specification that outlines the requirements to be tested and which test cases cover which requirements. Explaining this key point shows the hierarchical structure (test plan - test design specification - test case) that is required to develop a fully system-engineered test plan.

Test Case Specification (TCS): A test case identifies and specifies the inputs, outcomes, and conditions for execution of a test and is included in a document called Test Case Specification (TCS) as part of an ITS project overall Test Plan. It identifies a specific input and/or output that needs to be tested, and records the purpose of the test, a description of the test, the input and output test specifications, the environmental needs, and references the test procedure and describes the results of the test.

The suggested outline for a TCS is shown below:

• Test Case Identifier
• Objective
• Inputs
• Outcomes
• Environmental Needs
• Special Procedural Requirements
• Intercase Dependencies

What Does a Test Case Verify?

• To verify the requirements related to information exchanged between two systems by:
  • Verifying the sequence of information exchanged is correct:
    • Standards use dialogs to define information exchange sequence
  • Verifying the structure of information exchanged is correct
    • Standards define the order of Messages-Data Frames-Data Elements
  • Verifying the content of information exchanged is correct
    • Standards define the valid value rules (e.g., value ranges) for data exchanged

Test Procedure Specification (TPS): defines the steps to execute a test. Multiple Test Cases may reference a single Test Procedure.

Requirements to Test Case Traceability Matrix (RTCTM) for SSM

• RTCTM table provides traceability from requirements to test cases to test procedures
• Each ATC Test Design has an RTCTM for the requirements and test cases applicable to the Test Design

Term	Definition
Agency Specification	A document that has been prepared by an agency to define requirements for a subject item or process when procured by the agency.
Compliance	A condition that exists when an item meets all of the requirements of an agency specification.
Concept of Operations	A document that describes the purpose of a system project, including a description of the current and proposed system as well as key user needs that the new system is required to address.
Conformance	A condition that exists when an item meets all of the mandatory requirements as defined by a standard. It can be measured on the standard as a whole, which means that it meets all mandatory (and applicable conditional) requirements of the standard or on a feature level (i.e., it conforms to feature X as defined in section X.X.X), which means that it meets all mandatory (and applicable conditional) requirements of the feature.

5. References

• ATC5201 v06: Advanced Transportation Controller (ATC) Standard Version 06
  https://www.ite.org/tripgenappendices/
• IEEE 829-2008, IEEE Standard for Software and System Test Documentation. IEEE, 2008.
  http://standards.ieee.org/findstds/standard/829-2008.html
• PCB Modules 24-25: A 307a and A307b, including Student Supplement :
  stds training.aspx

6. Study Questions

1. What is NOT an ATC procurement deliverable?

1. Open Source Declaration
2. Board Support Package (BSP)
3. Open Source Distribution License
4. Application Program Interface (API)

2. Which is NOT a reason to use the IEEE 829 Standard?

1. IEEE 829 is part of the ATC 5201 Standard v06
2. Provides familiar documents
3. Standard definition of terms
4. Reuse in later projects

3. What is the primary purpose of RTCTM?

1. Sets the ATC testing workflow sequences
2. Correlates ATC User Needs to Requirements
3. Contains only ATC test cases
4. Traces ATC Requirement to ATC Test Case to Test Procedure

4. Manufacturers will have a single ATC that conforms to all versions of the ATC 5201 Standard

1. True
2. False

7. Icon Guide

The following icons are used throughout the module to visually indicate the corresponding learning concept listed below, and/or to highlight a specific point in the training material.

1) Background information: General knowledge that is available elsewhere and is outside the module being presented. This will be used primarily in the beginning of the slide set when reviewing information readers are expected to already know.

2) Tools/Applications: An industry-specific item a person would use to accomplish a specific task, and application of that tool to fit the need.

3) Remember: Used when referencing something already discussed in the module that is necessary to recount.

4) Refer to Student Supplement: Items or information that are further explained/detailed in the Student Supplement.

5) Example: Can be real-world (case study), hypothetical, a sample of a table, etc.

6) Checklist: Used to indicate a process that is being laid out sequentially.