Module 35 - T315

T315: Applying Your Test Plan to the NTCIP 1202 ASC Standard

HTML of the Student Supplement

(Note: This document has been converted from the Student Supplement to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)

Cover image for T315: Applying Your Test Plan to the NTCIP 1202 ASC Standard. Please see the Extended Text Description below.

(Extended Text Description: Large graphic cover page with dark blue background with the title in white letters "T315: Applying Your Test Plan to the NTCIP 1202 ASC Standard." At the middle left is the "Standards ITS Training" logo with a white box and letters in blue and green. The words "Student Supplement" and "RITA Intelligent Transportation Systems Joint Program Office" in white lettering are directly underneath the logo. Three light blue lines move diagonally across the middle of the blue background.)

T315: Applying Your Test Plan to the NTCIP 1202 ASC Standard

Table of Contents

Introduction/Purpose - 2

Example Needs and Requirements - 2

Other Issues for ASC - 11

Sample Test Plan - 13

Sample Test Case/Procedure - 22

Sample Test Summary Report - 24

Sample Test Incident Report - 27

Sample Test Log - 28

Glossary - 30

References - 30

Study Questions - 31

1. Introduction/Purpose

This module assists user agencies in their efforts to create test plans specific to their ASC needs based on the NTCIP 1202 v02 and related standards. Prior to developing such a test plan, the user is expected to be knowledgeable of the NTCIP 1202 standard and testing methodologies. The agency is also expected to develop their own user needs and requirements to the NTCIP 1202 standard.

2. Example Needs and Requirements

The following text provides the User Needs and Requirements that were developed as part of the A315b Module (Specifying Requirements for Actuated Traffic Signal Controllers Based on NTCIP 1202 Standard Part 1) and are referenced by this module. For a complete discussion of these requirements, please see A315b Part 1.

2.1. User Needs

The user needs are presented as informational background to give better insight into the intent of the requirements.

2.1.1. Architectural User Needs

2.1.1.1. Live Data Exchange

The typical operational environment allows the management system to monitor and control the DMS ASC by issuing requests (e.g., requests to access information, alter information, or control the signcontroller). In this environment, the DMS ASC responds to requests from the management station immediately (e.g., through the provision of live data, success/failure notice of information alteration, or success/failure of the command).

NTCIP 1203 v03 Clause 2.4.2.1 with revisions shown

2.1.1.2. Logged Data Exchange

Some operational environments do not have always-on connections (e.g., dial-up links). In such environments, a transportation system operator may wish to define conditions under which data is placed into a log, which can then be uploaded at a later time. For example, the operator may wish to maintain a log of all error conditions. maintain a log of all messages posted on the sign, regardless of which management station or algorithm posted the message.

NTCIP 1203 v03 Clause 2.4.2.2 with revisions shown

2.1.1.3. Support Legacy Communications Networks

Associated with the previous need is that aAgencies may need to use SSMASC’s within existing communications networks. Therefore, the SSM ASC needs to provide for the efficiencies needed by old low- speed communications, and to support those same efficiencies used by SSLs.

NTCIP 1210 v01 Clause 2.4.4 with revisions shown

2.1.2. Configure User Needs

2.1.2.1. Safely Configure Signal for Intersection Layout

The agency needs a signal controller that can be safely configured to control any intersection within its jurisdiction, including those with atypical layouts.

This will result in lower overall maintenance costs for the Maintenance Division. However, the operators must still be able to control every intersection, including 6-legged intersections. In addition, due to the safety critical nature of this information coupled with the static nature of the information, this process should require the presence of a field technician to verify any change.

A315a Slide 67

2.1.3. Control User Needs

2.1.3.1. Manually Control Selection of Timing Pattern

The Agency needs to be able to adjust intersection timing to accommodate the dynamic demands on the signal while also providing “green waves” for smooth and efficient traffic flow.

The pattern will typically be selected from a predefined list by the central system based on network conditions, but the local controller needs to be able to default to a specified schedule if any problems occur with receiving these commands and field personnel should be able to override these commands. The controller should allow for storage of sufficient patterns for all of these purposes.

A315b Slide 19

2.1.3.2. Support a Timing Pattern Schedule

This feature enables the operator to configure the ASC DMS to activate messages timing patterns at a future time (“scheduling”). The operator can indicate a series of times at which an associated message timing pattern will be activated. The associated message timing pattern can be any message timing pattern stored in the sign controller ASC, including a blank message.

NTCIP 1203 v03 Clause 2.5.2.3.5 with revisions shown

2.1.4. Monitor User Needs

2.1.4.1. Monitor Signal Configuration

<<TBD>>

2.1.4.2. Monitor Signal Timing

The agency needs a signal controller that will allow the management system to monitor the status of each signal indication with a one second resolution so that the agency has a way to remotely verify that the traffic signal is operating as expected.

A315a Slide 68

2.1.4.3. Monitor Timing Pattern Selection

<<TBD>>

2.1.4.4. Monitor Signal Diagnostics

<<TBD>>

2.1.4.5. Determine Device Identity

This feature allows the operator to determine basic information about the ASC, DMS, such as the type, technology, manufacturer, model, and version number of the DMSASC. It includes the ability to access information about both hardware and software elements of the ASC. DMS.

NTCIP 1203 v03 Clause 2.5.1.1 with revisions shown

2.1.5. Backwards Compatibility User Needs

2.1.5.1. Manufacturer X Model 1

Operators need the central system to work with Manufacturer X Model 1 controllers that are currently deployed, as these are too costly to update.

A315a Slide 82

2.2. Requirements

A component claiming compliance to this specification shall fulfill all of the requirements defined in this specification according to the designs defined in the Requirements Traceability Matrix.

2.2.1. Configure Requirements

2.2.1.1. Configure a Timing Pattern

When requested by the Operator, the central system shall configure the ASC with timing pattern information subject to ASC-imposed validation rules.

A315b Slide 33 with revisions as suggested on subsequent slides

2.2.1.2. Configure Timing Pattern Selection Logic

<<TBD>>

2.2.1.3. Configure Logging Service

The central system shall configure the ASC with event logging service information, including configuration of the event classes and event types to log.

Derived from NTCIP 1203 v03 Clause 3.4.2.2

2.2.1.4. Configure Timing Pattern Transition Mechanism

<<TBD>>

2.2.1.5. Configure Access

When requested by the Administrator, the central system shall configure the ASC with access settings for all access levels. The ASC shall support at least ______ access level(s) in addition to the administrator access level.

Derived from NTCIP 1203 v03 Clause 3.4.4.2

2.2.1.6. Set Time

The central system shall configure the ASC with the current coordinated universal time to the nearest second.

Derived from NTCIP 1203 v03 Clause H.2.2.1.

2.2.1.7. Set Time Zone

The central system shall configure the ASC with time zone information.

Derived from NTCIP 1203 v03 Clause H.2.2.2.

2.2.1.8. Set Daylight Savings Operations

The central system shall configure the ASC so that it knows whether or not to apply day light savings time adjustments when determining local time.

Derived from NTCIP 1203 v03 Clause H.2.2.3.

2.2.1.9. Define a Schedule

The central system shall configure the ASC with daily schedules of actions with a time resolution of one minute; the rules for selecting a daily schedule to run shall allow schedule configuration up to a year in advance.

Derived from NTCIP 1203 v03 Clause 3.5.2.3.4.2

2.2.2. Control Requirements

2.2.2.1. Activate Timing Pattern Remotely

<<TBD>>

2.2.2.2. Activate a Timing Pattern per a Schedule

<<TBD>>

2.2.2.3. Override Timing Pattern

<<TBD>>

2.2.2.4. Clear Log

The central system shall clear the ASC of log entries of a given event class that are less than or equal to a given time.

Derived from NTCIP 1203 v03 Clause 3.4.2.4

2.2.3. Monitor Requirements

2.2.3.1. Verify Current Time

The central system shall monitor the ASC to determine the current time settings.

Derived from NTCIP 1203 v03 Clause H.2.2.4.

2.2.3.2. Determine Current Configuration of Logging Service

The central system shall monitor the ASC to determine the current configuration of the event logging service, including the classes and types of events that are currently configured.

Derived from NTCIP 1203 v03 Clause 3.4.2.1

2.2.3.3. Retrieve Logged Data

The central system shall monitor the ASC to discover any logged events.

Derived from NTCIP 1203 v03 Clause 3.4.2.3

2.2.3.4. Determine Capabilities of Logging Service

The central system shall monitor the ASC to determine the capabilities of the event logging service, including the number of classes, number of event types, and number of events that can be supported.

Derived from NTCIP 1203 v03 Clause 3.4.2.5

2.2.3.5. Determine Total Number of Events

The central system shall monitor the ASC to determine the total number of logged events since power up.

Derived from NTCIP 1203 v03 Clause 3.4.2.6

2.2.3.6. Retrieve a Schedule

The central system shall monitor the ASC to determine the current configuration of the schedule.

Derived from NTCIP 1203 v03 Clause 3.5.2.3.4.1

2.2.3.7. Determine Maximum Number of Schedules

The central system shall monitor the ASC to determine the maximum number of schedules and day plans supported.

Derived from NTCIP 1203 v03 Clause H.2.3.1

2.2.3.8. Monitor Current Schedule

The central system shall monitor the ASC to determine which schedule entry has been selected.

Derived from NTCIP 1203 v03 Clause H.2.3.2.

2.2.3.9. Monitor Timing Pattern Configuration

<<TBD>>

2.2.3.10. Monitor Current Timing Pattern

The central system shall monitor the ASC to determine which timing pattern is currently active.

A315b Slide 34

2.2.3.11. Monitor Last Timing Pattern Requested

<<TBD>>

2.2.3.12. Monitor Source of Last Timing Pattern

<<TBD>>

2.2.3.13. Monitor Transition Mechanism Selected

<<TBD>>

2.2.3.14. Monitor Timing Pattern Schedule

<<TBD>>

2.2.3.15. Determine Current Access Settings

When requested by the administrator, the central system shall monitor the ASC to determine the current access settings.

Derived from NTCIP 1203 v03 Clause 3.4.4.1

2.2.3.16. Determine Device Component Information

The central system shall monitor the ASC to determine identification information for each module contained in the device including:

  1. An indication of the type of device
  2. The manufacturer of the module
  3. The model number or firmware reference of the module
  4. The version of the module
  5. An indication of whether it is a software or hardware module

Derived from NTCIP 1203 v03 Clause H.2.1

2.2.3.17. Determine Supported Standards

The central system shall monitor the ASC to determine the supported NTCIP standards.

Derived from NTCIP 1203 v03 Clause H.2.4

2.2.4. Supplemental Requirements

To claim compliance to this specification, a component shall support the full range of all objects referenced in the Requirements Traceability Matrix (Clause 2.5), except as ranges are explicitly defined in the following clauses.

2.2.4.1. Support at Least 32 Timing Patterns

The component shall support at least 32 timing patterns and 32 splits.

2.2.4.2. Support a Number of Day Selection Patterns

The device shall support at least _____ day selection pattern(s).

Derived from NTCIP 1203 v03 Clause H.2.5.1

2.2.4.3. Support a Number of Day Plan Events

The device shall support at least _______ day plan events per day plan.

Derived from NTCIP 1203 v03 Clause H.2.5.2

2.2.4.4. Support a Number of Day Plans

The device shall support at least _____ day plan(s).

Derived from NTCIP 1203 v03 Clause H.2.5.3

2.2.4.5. Support a Number of Actions

The ASC shall support at least _____ actions.

Derived from NTCIP 1203 v03 Clause 3.6.10.1

2.2.4.6. Support the Activate Timing Pattern Action for the Scheduler

The ASC shall allow the scheduler to be configured to activate any timing pattern supported by the ASC and currently valid.

Derived from NTCIP 1203 v03 Clause 3.6.10.2

2.2.4.7. Perform Actions at Scheduled Times

The ASC shall perform the actions configured in the scheduler at the times identified.

Derived from NTCIP 1203 v03 Clause 3.6.10.3

2.2.4.8. Maximum Response Time for Requests

The ASC shall process all requests in accordance with all of the rules of the relevant base standards (i.e., NTCIP 1103 v01 and NTCIP 2303:2001), including updating the value in the database and initiating the transmission of the appropriate response (assuming that the ASC has permission to transmit) within the Maximum Response Time. The Maximum Response Time shall be 100 milliseconds. The Maximum Response Time is measured as the time between the receipt of the last byte of the request and the transmission of the first byte of the response.

Derived from NTCIP 1204 v03 Clause 3.6.21

2.2.4.9. Record and Timestamp Events

The device shall support the capability to record configured event types with timestamps, in a local log (log contained in the device controller), upon request by the user and/or the management station.

NTCIP 1203 v03 Clause H.2.6.1

2.2.4.10. Support a Number of Event Classes

The device shall support at least _________ event class(es).

Derived from NTCIP 1203 v03 Clause H.2.6.2

2.2.4.11. Support a Number of Event Types to Monitor

The device shall support at least ________ event type(s).

Derived from NTCIP 1203 v03 Clause H.2.6.3

2.2.4.12. Support Monitoring of Event Types

Supplemental requirements for monitoring types of events are provided in the following subsections.

NTCIP 1203 v03 Clause H.2.6.4

2.2.4.12.1. Support On-Change Events

The device shall allow any event type configuration to monitor data for changes in value.

NTCIP 1203 v03 Clause H.2.6.4.1

2.2.4.12.2. Support Greater Than Events

The device shall allow any event type configuration to monitor data for values exceeding a defined threshold for a period of time.

NTCIP 1203 v03 Clause H.2.6.4.2

2.2.4.12.3. Support Less Than Events

The device shall allow any event type configuration to monitor data for values falling below a defined threshold for a period of time.

NTCIP 1203 v03 Clause H.2.6.4.3

2.2.4.12.4. Support Hysteresis Events

The device shall allow any event type configuration to monitor data for values exceeding an upper limit or dropping below a lower limit.

NTCIP 1203 v03 Clause H.2.6.4.4

2.2.4.12.5. Support Periodic Events

The device shall allow any event type configuration to monitor data on a periodic basis.

NTCIP 1203 v03 Clause H.2.6.4.5

2.2.4.12.6. Support Bit-flag Events

The device shall allow any event type configuration to monitor one or more bits of a value becoming true (e.g., obtaining a value of one).

NTCIP 1203 v03 Clause H.2.6.4.6

2.2.4.12.7. Support Event Monitoring on Any Data

The device shall allow a management station to configure any event type to monitor any piece of data supported by the device within the logical rules of the type of event (e.g., ASCII strings should not be monitored with greater than or less than conditions).

NTCIP 1203 v03 Clause H.2.6.4.7

2.2.4.13. Support a Number of Events to Store in Log

The device shall support at least _________ event(s) in the log.

Derived from NTCIP 1203 v03 Clause H.2.7

2.2.5. Architectural Requirements

2.2.5.1. Retrieve Data

The central system shall retrieve data from the ASC.

Derived from NTCIP 1203 v03 Clause 3.4.1.1

2.2.5.2. Deliver Data

The central system shall deliver data (e.g., configuration data, commands, etc.) to the ASC.

Derived from NTCIP 1203 v03 Clause 3.4.1.2

2.2.5.3. Explore Data

The central system shall dynamically discover what data and data instances are supported by the ASC.

Derived from NTCIP 1203 v03 Clause 3.4.1.3

2.2.5.4. Configure Block Objects

The central system shall configure the ASC by utilizing standard block objects.

Derived from NTCIP 1210 v01 Clause 3.3.1.9.1

2.2.5.5. Retrieve Block Objects

The central system shall retrieve ASC configuration data by utilizing standard block objects.

Derived from NTCIP 1210 v01 Clause 3.3.1.9.2

2.2.5.6. Retrieve Block Status

The central system shall retrieve the status of block transactions.

Derived from NTCIP 1210 v01 Clause 3.3.1.9.3

2.2.5.7. Support STMP

The central system shall support the STMP for communications.

Derived from NTCIP 1210 v01 Clause 3.3.1.9.4

2.2.5.1. ASC Conformance

The ASC shall properly respond to all requests that conform to the requirements defined in this document.

3. Other Issues for ASC

The presentation discussed several issues that should be considered when dealing with ASCs, including:

The following issues should also be considered.

3.1. Response Times

Many ASCs are deployed in multi-drop environments on low-speed communication channels. In this configuration it is critical that devices respond in a timely manner so that the central system can proceed to poll the next device in order.

Module A315b suggests that the user to specify a maximum response time for any request. This can easily be verified by using a data analyzer to monitor the entire test plan. At the end of the test, the log file can be sorted based on the time differential (delta) between each message exchanged and the largest value for a response packet identified. If this value exceeds the Maximum Response Time specification, then the device fails the test.

3.2. Updating Global Set ID

Another challenge exists in managing ASCs is the ability to determine if the database has been reconfigured since the last check. The size of the database makes it prohibitive to upload and check every value, so the designers of the standard ensured that there was a separate object that is required to change values any time the database changes. This is another feature that should be tested.

3.3. Setting Time Over a Network

Another challenge facing ASC deployments is coordinating clocks. ASCs are called upon to coordinate their operation with other ASCs in order to provide smooth traffic flow – this requires coordination to the second. Unfortunately, depending on the network design, delays in transmitting requests could exceed this one-second limit.

If your network design does not provide a timely delivery of messages, you should consider using an alternative mechanism to synchronize your clocks such as GPS or WWV.

3.4. Security

An often over-looked problem is security of signal systems. Traditionally, signal systems have used proprietary protocols over dedicated infrastructure, which made it difficult, but not impossible for these systems to be hacked. However, with the introduction of a public protocol and a move to connect ASCs via wireless networks, security becomes a pressing issue. Even on dedicated circuits, there is a potential threat for hackers to gain access.

Anyone deploying ASCs is strongly encouraged to investigate ways to provide security to their system as normal SNMP provides virtually no security.

3.5. Data Link Control Issues

Another area of testing that is often over-looked is testing the lower layers. For example, if the signal implements PMPP over a serial connection, does it handle long addresses and broadcast addresses as required by the standard? Does it reject messages that contain CRC errors? All aspects of all requirements should be tested prior to accepting a device.

3.6. Protocol Specific Objects

In addition to the lower layer protocols, most of these protocols have their own management objects that are required by NTCIP. These are objects that can be monitored just like normal SNMP objects, but rather than monitoring and controlling things like signal phase status, they monitor things like how many SNMP messages have been received, how many have had bad community names (which may indicate a hacker), etc. Once again, all of these requirements should be tested prior to accepting a device.

3.7. Compound Testing

Finally, ASCs should be subjected to a variety of tests simultaneously prior to acceptance. This is similar but even more stressful than load testing. Whereas load testing is making sure the signal operates when under a heavy communications load, compound testing adds to that set-up by adding other conditions such as a reboot due to power failure and restoration, high temperatures, etc. during railroad preemption. The goal is to subject the ASC to a realistic worst-case scenario and ensure that it still operates as required. It is better to have it fail in the laboratory than in the field.

4. Sample Test Plan

4.1. Test Plan Identifier

The test plan identifier for this document is:

ASC-CI-20130927-FAT-TP-v1

4.2. Introduction

4.2.1. Objectives

This test plan (TP) has been developed to document the plan for factory acceptance testing (FAT) of the communications interface (CI) for Actuated Traffic Signal Controllers (ASC) that claim compliance to the Agency's NTCIP 1202 v02 specifications dated September 27, 2013 (20130927). This is the first version (v1) of this test plan.

4.2.2. Background

This test plan covers an extensive test of the NTCIP 1202-related requirements for the ASC. This includes requirements related to:

Other test plans may supplement this test plan by performing:

4.2.3. References

4.2.4. Definitions

The following terms shall apply within the scope of this test plan.

Agency Project Manager - Person designated by the organization purchasing the equipment to be responsible for overseeing the successful deployment of the equipment.

ASC – Actuated Traffic Signal Controller - A controller unit that can be configured to operate a traffic signal according to NTCIP 1202.

ASC Test Procedures – The composite of all test procedure documents listed under references.

Developer - The organization providing the equipment to be tested.

System - The combination of all ASCs and the management station with associated communications infrastructure.

Test Analyst - The person who performs the testing according to the test procedures and interprets and records the results.

Test Manager - The designated point-of-contact for the entire test team. Frequently, this is the same person as the Test Analyst.

4.3. Test Item

This test plan is designed to test the NTCIP-related operation of an ASC. The following documents will provide the basis for defining correct operation with the document of highest precedent listed last:

NOTE: The above documents reference specific versions of NTCIP standards that are the originating source of many requirements to be tested. Correct operations are defined by the specific versions of the standards as referenced in the above documentation (unless explicitly overridden by other documents); the specific version numbers are not listed in this test plan in order to avoid the introduction of any inconsistency.

The test item should be delivered with its own user's manual.

4.4. Features to be Tested

The features to be tested are identified in Annex A as items that are mapped to specific test cases.

4.5. Features Not to be Tested

Features that are not defined in NTCIP 1202 v02 are not directly covered by this test plan. These features typically include, but are not limited to:

While some aspects of these features may be tested (e.g., all NTCIP communications rely upon the basic operation of lower layer protocols), this test plan does not generally focus on these types of requirements because they are not the focus of NTCIP 1202.

In addition, features listed in Annex A that reference other test plans are not covered by this test plan.

Other test plans that relate to this device include:

4.6. Approach

The Test Analyst will perform each selected test case from the ASC Test Procedures identified for this test plan in Annex A.

4.7. Item Pass/Fail Criteria

In order to pass the test, the ASC shall pass all test procedures included in this test plan without demonstrating any characteristic that fails to meet project requirements.

4.8. Suspension Criteria and Resumption Requirements

The test may be suspended at the convenience of test personnel between the performances of any two test procedures. The test shall always resume at the start of a selected test procedure.

If any modifications are made to the ASC, a complete regression test may be required in order to pass this test plan.

4.9. Test Deliverables

The Test Manager will ensure that the following documents are developed and entered into the configuration management system upon their completion:

All test documentation will be made available to both the Agency and the Developer.

All test documentation will be made available in a widely recognized computer file format such as Microsoft Word or Adobe Acrobat. In addition, the files from the test software shall be provided in their native file format as defined by the test software.

4.10. Testing Tasks

Task Task Name Predecessor Responsibility NTCIP Knowledge
1 Complete the Test Item Transmittal Form and transmit the component to the Test Group Implement Interface Developer Limited
2 Perform Tests and produce Test Log and Test Incident Reports 1 Test Analyst Expert
3 Resolve Test Incident Reports 2 Developer, Test Manager Extensive
4 Repeat Steps 1-3 until all test procedures have succeeded or project otherwise concludes 3 N/A N/A
5 Prepare the Test Summary report 4 Test Analyst Moderate
6 Transmit all test documentation to the Agency Project Manager 5 Test Manager Limited

4.11. Environmental Needs

All Test Cases covered by this test plan require the device under test to be connected to a test application as depicted in Figure 3-1. A data analyzer may also be used to capture the data exchanged between the two components. The test environment should be designed to minimize any complicating factors that may result in anomalies unrelated to the specific test case. Failure to isolate such variables in the test environment may result in false results to the test. For example, the device may be conformant with the standard, but communication delays could result in timeouts and be misinterpreted as failures.

A sample test environment as incorporated from NTCIP 8007, Page 13. Please see the Extended Text Description below.

(Extended Text Description: A sample test environment as incorporated from NTCIP 8007, Page 13. A "device under test (DUT)" is connected to a communications cloud which is connected to a "test application" running on a PC. Also connected to the communications cloud is a data analyzer, also running on a PC, with the word "Optional" underneath it.)

Figure 3-1: Field Device Test Environment

The specific test software and data analyzer to be used are identified in the Tools clause of the Approach section of this test plan.

The tests will be performed at a location to be determined by the Agency. This location will provide the following:

4.12. Tools

The following tools (latest version of each), or their equivalent shall be used:

All tests shall be performed using the following communications environment, unless otherwise defined in the specific test procedure.

4.13. Responsibilities

The following roles are defined in this test plan:

4.14. Staffing and Training

The following staffing is expected for this test plan:

If the Agency Project Manager is not familiar with NTCIP Testing, s/he should become familiar with NTCIP 9012 and FHWA guidance on the procurement of ITS systems. The Test Manager and Test Analyst must be familiar with how to use the test software. Many software systems come with extensive on-line help, but the test personnel may also need detailed knowledge of the NTCIP standards to fully perform their duties.

4.15. Schedule

Each round of testing will commence within 4 weeks of the receipt of the hardware from the manufacturer. The first round of testing is expected to take three months of dedicated testing time with an additional two weeks to prepare all associated documentation. Subsequent rounds of testing are expected to require less time due to the resolution of anomalies. The final round of testing (with no failures) is expected to take three weeks of dedicated testing time with an additional week to prepare all associated documentation.

4.16. Risks and Contingencies

The number of rounds of testing will be dependent upon when the agency receives a device that passes all tests. For planning purposes, the agency assumes that five rounds of testing will be required totaling 10 months of actual testing time with an additional 10 weeks for documentation purposes.

If the ASC experiences significant failures during any round of testing, the agency may stop the testing and return the device to the manufacturer for corrections in order to minimize the impact to the project budget.

The Developer shall correct any problems identified with the ASC. Upon completion of the modifications, the Developer shall re-submit the component for another complete test consisting of all test cases.

If the device fails to pass all defined tests within the allotted budget defined above, the agency may reject the device or assess penalties on the Developer per the contract terms.

4.17. Approvals

_________________________________ __________________
Agency Project Manager Date
_________________________________ __________________
Developer Date
_________________________________ __________________
Test Manager Date

4.18. Annex A – Requirements to Test Case Traceability Matrix

Requirement Test Case
ID Name ID Name
2.2.1.1 Configure a Timing Pattern 2.3.1.1 Configure a Valid Timing Pattern
2.3.1.2 Incorrectly Configure a Timing Pattern
2.2.1.2 Configure Timing Pattern Selection Logic See Manufacturer Factory Acceptance Test Plan
2.2.1.3 Configure Logging Service 2.3.2.1 Configure Logging Service
2.2.1.4 Configure Timing Pattern Transition Mechanism 2.3.1.4 Configure Timing Pattern Transition Mechanism
2.2.1.5 Support Database Transactions 2.3.6.1 Download a Valid Database Configuration
2.3.6.2 Download a Database Configuration with Syntax Errors
2.3.6.3 Download a Database Configuration with Consistency Errors
2.3.6.4 Download a Valid Database Configuration with Other Valid Requests
2.3.6.5 Download a Valid Database Configuration with Other Invalid Requests
2.3.6.6 Download a Database Configuration with Commands Mixed with Database Objects
2.3.6.7 Cancel a Database Download
2.2.2.1 Activate Timing Pattern Remotely 2.3.1.5 Activate Timing Pattern Remotely
2.2.2.2 Activate a Timing Pattern per a Schedule 2.3.4.2 Activate a Timing Pattern per a Schedule
2.2.2.3 Override a Timing Pattern 2.3.1.6 Override a Timing Pattern
2.2.2.4 Set Time 2.3.3.1 Set Time
2.2.2.5 Set Time Zone 2.3.3.2 Set Time Zone
2.2.2.6 Set Daylight Savings Mode 2.3.3.3 Set Daylight Savings Mode
2.2.2.7 Clear Log 2.3.2.2 Clear Log
2.2.2.8 Define a Schedule 2.3.4.1 Define a Schedule
2.2.3.1 Verify Current Time 2.3.3.4 Verify Current Time
2.2.3.2 Determine Current Configuration of Logging Service 2.3.2.3 Determine Current Configuration of Logging Service
2.2.3.3 Retrieve Logged Data 2.3.2.4 Retrieve Logged Data
2.2.3.4 Determine Capabilities of Logging Service 2.3.2.5 Determine Capabilities of Logging Service
2.2.3.5 Determine Total Number of Events 2.3.2.4 Retrieve Logged Data
2.2.3.6 Retrieve a Schedule 2.3.4.3 Retrieve a Schedule
2.2.3.7 Determine Maximum Number of Schedules 2.3.4.3 Retrieve a Schedule
2.2.3.8 Monitor Current Schedule 2.3.4.3 Retrieve a Schedule
2.2.3.9 Monitor Timing Pattern Configuration 2.3.1.7 Monitor Timing Pattern Configuration
2.2.3.10 Monitor Current Timing Pattern 2.3.1.8 Monitor Current Timing Pattern
2.2.3.11 Monitor Last Timing Pattern Requested 2.3.1.8 Monitor Current Timing Pattern
2.2.3.12 Monitor Source of Last Timing Pattern 2.3.1.8 Monitor Current Timing Pattern
2.2.3.13 Monitor Transition Mechanism Selected 2.3.1.8 Monitor Current Timing Pattern
2.2.3.14 Monitor Timing pattern Schedule 2.3.4.4 Monitor Timing pattern Schedule
2.2.3.15 Monitor Database Configuration 2.3.9.1 Verify Updates to globalSetID
2.2.3.16 Monitor SNMP Status 2.3.10.1 Verify Support for SNMP MIB
2.2.4.1 Support at Least 32 Timing Patterns 2.3.1.1 Configure a Valid Timing Pattern
2.3.1.3 Configure Timing Pattern 32
2.2.4.2 Support a Number of Day Selection Patterns 2.3.4.5 Fill Scheduling Table
2.2.4.3 Support a Number of Day Plan Events 2.3.4.5 Fill Scheduling Table
2.2.4.4 Support a Number of Day Plans 2.3.4.5 Fill Scheduling Table
2.2.4.5 Support a Yellow Change Interval 2.3.5.1 Verify Yellow Change Interval
2.2.4.6 Respond within 100 ms 2.3.8.1 Verify Response Times
2.2.5.1 Support STMP 2.3.7.1 Configure a Dynamic Object
2.3.7.2 Get a Dynamic Object
2.3.7.3 Set a Dynamic Object
2.3.7.4 Configure a Dynamic Object with Incorrect Data
2.3.7.5 Attempt to Configure a Dynamic Object Using the Wrong Process
2.3.7.6 Use a Dynamic Object While It Is Being Modified
2.3.7.7 Load Test the Device
2.2.5.2 Support MIB Views 2.3.11.1 Get with Administrator Rights
2.3.11.2 Set with Administrator Rights
2.3.11.3 Get with Full Access User Rights
2.3.11.4 Set with Full Access User Rights
2.3.11.5 Get with Read-Only Rights
2.3.11.6 Set with Read-Only Rights
2.3.11.7 Change Access Rights
2.2.5.3 Support PMPP 2.3.12.1 Verify Default Address
2.3.12.2 Reject Incorrect Address
2.3.12.3 Verify Alternate Address
2.3.12.4 Reject Large Incorrect Address
2.3.12.5 Support Broadcast Message
2.3.12.6 Support Group Address
2.3.12.7 Accept Unset Poll Bit
2.3.12.8 Reject Invalid Control Byte
2.3.12.9 Reject Invalid IPI Byte
2.3.12.10 Reject Invalid CRC
2.3.12.11 Support for RS-232 Objects
2.3.12.12 Support for LapB Objects

5. Sample Test Case/Procedure

A sample test case/procedure is provided below (as modified slightly from NTCIP 1203):

Test Case 2.3.3.1 Title Set Time
Description This test case verifies that the ASC properly tracks time. It advances the clock by a user-defined amount, waits a few seconds, retrieves the time, and verifies it indicates an appropriate value.
This test will advance the ASC clock by Time_Offset seconds.
Variables Time_Offset as defined in the test plan.
Pass/Fail Criteria The DUT shall pass every verification step included within the Test Case to pass the Test Case.
Step Test Procedure Results
1 CONFIGURE: Determine the number of seconds to advance the clock in the ASC. RECORD this information as: Time_Offset
2 Determine the time the test started according to the test computer. RECORD this information as: »Test_Time
3 GET the following object(s):
»globalTime.0
»controllerStandardTimeZone.0
»globalDaylightSaving.0
»controllerLocalTime.0 		Pass/Fail
(Clause H.2.2.4)
4 RECORD the RESPONSE VALUE for globalTime.0 and controllerLocalTime.0 as:
»UTC_Time
»Local_Time 		 
5 Calculate the time difference between Local_Time and UTC_Time. RECORD this information as:
»Time_Diff 		 
6 Calculate the value of UTC_Time plus Time_Offset. RECORD this information as:
»New_UTC_Time 		 
7 SET the following object(s) to the value(s) shown:
»globalTime.0 = New_UTC_Time NOTE--This advances the clock by Time_Offset 		Pass/Fail
(Clause H.2.2.1)
8 Calculate UTC_Time plus Time_Offset plus the amount of time that has elapsed since Step 1. RECORD this information as:
»Expected_Time 		 
9 GET the following object(s):
»globalTime.0
»controllerStandardTimeZone.0
»globalDaylightSaving.0
»controllerLocalTime.0 		Pass/Fail
(Clause H.2.2.4)
10 VERIFY that the RESPONSE VALUE for globalTime.0 is roughly equal to Expected_Time. Pass/Fail
(Clause H.2.2.4)
11 VERIFY that the RESPONSE VALUE for controllerLocalTime.0 is roughly equal to Expected_Time plus Time_Diff.  
12 DELAY for 15 seconds.  
13 GET the following object(s):
»globalTime.0
»controllerStandardTimeZone.0
»globalDaylightSaving.0
»controllerLocalTime.0 		Pass/Fail
(Clause H.2.2.4)
14 VERIFY that the RESPONSE VALUE for globalTime.0 is roughly equal to Expected_Time plus 15 seconds. Pass/Fail
(Clause H.2.2.4)
15 VERIFY that the RESPONSE VALUE for controllerLocalTime.0 is roughly equal to Expected_Time plus Time_Diff plus 15 seconds. Pass/Fail
(Clause H.2.2.4)
16 Calculate the time to set in the agent to restore the original value. RECORD this information as:
»Restore_UTC_Time 		 
17 SET the following object(s) to the value(s) shown:
»globalTime.0 = Restore_UTC_Time 		Pass/Fail
(Clause H.2.2.1)

6. Sample Test Summary Report

6.1. Identifier

The identifier for this document is:

ASC-CI-20130927-FAT-TS-2013-37-1

6.2. Summary

This test summary report (TS) provides an overview of the testing results of the first round of factory acceptance testing (FAT) of the communications interface (CI) for the Actuated Traffic Signal Controllers (ASC) procured under project 2013-37 that claim compliance to the Agency's NTCIP 1202 v02 specifications dated September 27, 2013 (20130927).

The tested ASC was a Manufacturer X Model Y version Z actuated traffic signal controller.

The test was performed at the Agency Workshop on December 2, 2013 by ___________. This test used the _____ test application and the _______ data analyzer.

The test followed the guidelines set forth in ASC-CI-20130927-FAT-TP-v1. The full test log is provided in ASC-CI-20130927-FAT-TL-2013-37-1 and the testing resulted in 14 Incident Reports identified as ASC-CI-20130927-FAT-TIR-2013-37-1-x, where the trailing "x" is a number, 1 through 14.

6.3. Variances

The test items appeared to be as specified except as explained in the documented incident reports.

The testing was performed according to the plan.

6.4. Comprehensive Assessment

Testing was performed as defined in the test plan.

6.5. Summary of Results

ID Test Case Result
2.3.1.1 Configure a Valid Timing Pattern PASS
2.3.1.2 Incorrectly Configure a Timing Pattern PASS
2.3.1.3 Configure Timing Pattern 32 PASS
2.3.1.4 Configure Timing Pattern Transition Mechanism PASS
2.3.1.5 Activate Timing Pattern Remotely PASS
2.3.1.6 Override a Timing Pattern PASS
2.3.1.7 Monitor Timing Pattern Configuration PASS
2.3.1.8 Monitor Current Timing Pattern PASS
2.3.2.1 Configure Logging Service PASS
2.3.2.2 Clear Log PASS
2.3.2.3 Determine Current Configuration of Logging Service PASS
2.3.2.4 Retrieve Logged Data PASS
2.3.2.5 Determine Capabilities of Logging Service PASS
2.3.3.1 Set Time PASS
2.3.3.2 Set Time Zone PASS
2.3.3.3 Set Daylight Savings Mode PASS
2.3.3.4 Verify Current Time PASS
2.3.4.1 Define a Schedule PASS
2.3.4.2 Activate a Timing Pattern per a Schedule PASS
2.3.4.3 Retrieve a Schedule PASS
2.3.4.4 Monitor Timing pattern Schedule PASS
2.3.4.5 Fill Scheduling Table PASS
2.3.5.1 Verify Yellow Change Interval PASS
2.3.6.1 Download a Valid Database Configuration FAIL
2.3.6.2 Download a Database Configuration with Syntax Errors FAIL
2.3.6.3 Download a Database Configuration with Consistency Errors FAIL
2.3.6.4 Download a Valid Database Configuration with Other Valid Requests FAIL
2.3.6.5 Download a Valid Database Configuration with Other Invalid Requests FAIL
2.3.6.6 Download a Database Configuration with Commands Mixed with Database Objects FAIL
2.3.6.7 Cancel a Database Download FAIL
2.3.7.1 Configure a Dynamic Object FAIL
2.3.7.2 Get a Dynamic Object FAIL
2.3.7.3 Set a Dynamic Object FAIL
2.3.7.4 Configure a Dynamic Object with Incorrect Data FAIL
2.3.7.5 Attempt to Configure a Dynamic Object Using the Wrong Process FAIL
2.3.7.6 Use a Dynamic Object While It Is Being Modified FAIL
2.3.7.7 Load Test the Device FAIL
2.3.8.1 Verify Response Times PASS
2.3.9.1 Verify Updates to globalSetID PASS
2.3.10.1 Verify Support for SNMP MIB PASS
2.3.11.1 Get with Administrator Rights PASS
2.3.11.2 Set with Administrator Rights PASS
2.3.11.3 Get with Full Access User Rights PASS
2.3.11.4 Set with Full Access User Rights PASS
2.3.11.5 Get with Read-Only Rights PASS
2.3.11.6 Set with Read-Only Rights PASS
2.3.11.7 Change Access Rights PASS
2.3.12.1 Verify Default Address PASS
2.3.12.2 Reject Incorrect Address PASS
2.3.12.3 Verify Alternate Address PASS
2.3.12.4 Reject Large Incorrect Address PASS
2.3.12.5 Support Broadcast Message PASS
2.3.12.6 Support Group Address PASS
2.3.12.7 Accept Unset Poll Bit PASS
2.3.12.8 Reject Invalid Control Byte PASS
2.3.12.9 Reject Invalid IPI Byte PASS
2.3.12.10 Reject Invalid CRC PASS
2.3.12.11 Support for RS-232 Objects PASS
2.3.12.12 Support for LapB Objects PASS

This was the first round of testing; at this time all identified failures are still unresolved.

6.6. Evaluation

The device appears to comply with the specifications with two major exceptions:

These limitations prevent the deployment of this equipment at this time. The agency should not deploy signal controllers with a known tendency to lock up and without the support of STMP, the device should not be deployed on any of the agency's low-speed links.

6.7. Summary of Activities

The first round of testing required 10 weeks of time, which is less than the three months estimated.

6.8. Approvals

_________________________________ __________________
Agency Project Manager Date
_________________________________ __________________
Developer Date
_________________________________ __________________
Test Manager Date

7. Sample Test Incident Report

7.1. Identifier

The identifier for this document is:

ASC-CI-20130927-FAT-TIR-2013-37-1-5

7.2. Summary

This test incident report (TIR) documents the fifth anomaly discovered during the first round of factory acceptance testing (FAT) of the communications interface (CI) for the Actuated Traffic Signal Controllers (ASC) procured under project 2013-37 that claim compliance to the Agency's NTCIP 1202 v02 specifications dated September 27, 2013 (20130927).

The tested ASC was a Manufacturer X Model Y version Z actuated traffic signal controller.

Test Case: 2.3.6.1 - Download a valid database configuration

7.3. Incident Description

We attempted to set dbCreateTransaction.0 to a value of 'transaction' in Step 202. Per the defined dialog, the device should have accepted this value and changed its state to 'transaction'. However, the device failed to respond and stopped responding to any subsequent request for any object. We also observed that signal timing stopped. Immediately prior to this, we had verified that the value of dbCreateTransaction.0 was normal as required by the dialog definition. We had to reboot the device in order to resume testing. We then repeated the test and obtained the same result. This appears to be a flaw in the implementation.

First Attempt: 10:05am December 2, 2013
Second Attempt: 10:18 am December 2,2013

The incident is recorded in the following log files at the indicated timestamps.

7.4. Impact

This anomaly is an apparent violation of Clause 2.3.1 of NTCIP 1201 and can be traced to the following:

Until this issue is fixed, the central system will not be able to download "P2" database parameters. In addition, if the central system attempts to use the transaction mode, the signal timing will stop.

8. Sample Test Log

8.1. Identifier

The identifier for this document is:

ASC-CI-20130927-FAT-TL-2013-37-1

8.2. Description

This test log (TL) provides a textual log of all activity associated with the first round of factory acceptance testing (FAT) of the communications interface (CI) for the Actuated Traffic Signal Controllers (ASC) procured under project 2013-37 that claim compliance to the Agency's NTCIP 1202 v02 specifications dated September 27, 2013 (20130927).

The tested ASC was a Manufacturer X Model Y version Z actuated traffic signal controller.

The test was performed at the Agency Workshop on December 2, 2013 by ___________. This test used the _____ test application and the _______ data analyzer.

8.3. Tester Log

<<Insert any notes taken by the tester; for example>>

The first testing session lasted from 9:00 am until 12:00 noon on Monday, December 2, 2013. One anomaly was identified as recorded in Incident Report ASC-CI-20130927-FAT-TIR-2013-37-1-5.

8.4. Test Application Log

The full testing results can be found in a set of files located at <Project Directory>\ASC\CI\20130927\FAT\Round_1\File_x.ntd where the "x" in "File_x" represents a sequential file number. A textual log of this file is attached below.

<<Insert the textual output of the test application; for example, for each test case performed there may be an entry such as >>

8.4.1. Set Time

Test Procedure Identifier: SetTime

Test Name: Set Time

Description: This test case verifies that the ASC properly tracks time. It advances the clock by a user-defined amount, waits a few seconds, retrieves the time, and verifies it indicates an appropriate value. This test will advance the ASC clock by Time_Offset seconds.
Time: 18:25:08 on Thursday, December 2, 2013

Result: PASS
Steps:

Description Notes Time Result
Step 1: Time_Offset = 33   18:25:08 N/A
Step 2: GET globalTime.0   18:25:08 Passed
Step 3: RECORD globalTime.0 = 123456789   18:25:08 N/A
Step 4: SET globalTime.0 = 123456822   18:25:08 Passed
Step 5: Delay for 15 seconds   18:25:08 N/A
Step 6: Get globalTime.0   18:25:23 Passed
Step 7: Verify globalTime.0 (123456836) ~= 123456837   18:25:24 Passed

8.5. Data Analyzer Log

The full data analyzer results can be found in a set of files located at <Project Directory>\ASC\CI\20130927\FAT\Round_1\File_x.cfa where the "x" in "File_x" represents a sequential file number. A textual log of this file is attached below.

<<Insert a textual output of the data analyzer; for example, for each test case performed there may be an entry such as:>>

Frame ID Type Community Error Object 1 Timestamp
1 1 Get public No Error globalTime.0 18:25:08.731
2 1 Response public No Error globalTime.0 18:25:08.785
3 2 Set public No Error globalTime.0 18:25:08.801
4 2 Response public No Error globalTime.0 18:25:08.894
5 3 Get public No Error globalTime.0 18:25:08.902
6 3 Response public No Error globalTime.0 18:25:08.999

9. Glossary

Term Definition
Agency Specification A document that has been prepared by an agency to define requirements for a subject item or process when procured by the agency.
Compliance A condition that exists when an item meets all of the requirements of an agency specification.
Concept of Operations A document that describes the purpose for a system project, including a description of the current and proposed system, as well as key user needs that the new system is required to address.
Conformance A condition that exists when an item meets all of the mandatory requirements as defined by a standard. It can be measured on the standard as a whole, which means that it meets all mandatory (and applicable conditional) requirements of the standard or on a feature level (i.e., it conforms to feature X as defined in section X.X.X), which means that it meets all mandatory (and applicable conditional) requirements of the feature.

10. References

Testing

Actuated Traffic Signal Controllers

NTCIP

Systems Engineering

11. Study Questions

Quiz/poll questions and answer choices as presented in the PowerPoint slide

Question 1: Why is it important to test Actuated Traffic Signal Controllers?

  1. Implementation errors might result in conflicting green displays
  2. Implementation errors might decrease traveler safety
  3. Testing is not important for ASCs
  4. It gives peace of mind for a trivial cost

Question 2: Which of the following statements is not true?

  1. Every requirement should be tested
  2. Some testing may be performed by the manufacturer
  3. You should only need to perform your test plan once
  4. Traceability tables can help you to assess the impact of a test failure

Question 3: Where can you find definitions for terms that can be used in NTCIP test steps?

  1. IEEE 829
  2. ISO 9001
  3. NTCIP 8007
  4. Student Supplement

Question 4: Which of the following statements is true?

  1. The transaction mode must be used for all data downloads
  2. A manufacturer may not impose its own consistency checks
  3. STMP testing only needs to worry about "your" dynamic objects
  4. There is no guarantee of off-the-shelf interoperability

Question 5: Which document(s) discuss potential impacts of testing failures?

  1. Test Summary
  2. Test Incident Report
  3. Test Log
  4. Test Summary and Test Incident Report