Module 17 - T311

T311: Applying Your Test Plan to the NTCIP 1203 v03 DMS Standard

HTML of the Student Supplement

(Note: This document has been converted from the Student Supplement to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)

T311: Applying Your Test Plan to the Dynamic Message Signs based on NTCIP 1203 DMS Standard v03

Table of Contents

1. Module Description - 2

2. Introduction/Purpose - 2

3. Samples/Examples - 2

4. Reference to Other Standards - 10

5. Case Studies - 11

6. Glossary - 11

7. References - 12

8. Study Questions - 13

9. Icon Guide - 14

1. Module Description

This updated module provides participants with the information needed on how to create a test plan specific to a procuring agency's dynamic message sign (DMS) needs based on the NTCIP 1203 v03 standard. This module helps the participant understand the elements of the DMS standard that are required to apply test plans to verify that the agency's DMS system meets the design specifications and is conformant to the NTCIP 1203 v03 standard while following standard testing methodologies.

2. Introduction/Purpose

This module assists user agencies in creating a test plan specific to their dynamic message system needs based on the NTCIP 1203 v03 standard. NTCIP 1202 v03 builds on the NTCIP 1202 v02 standard by adding test procedures for testing conformance to the standard. Prior to developing such a test plan, the user is expected to be knowledgeable of the NTCIP 1203 v03 standard and testing methodologies. This module will cover material related to elements of the 1203 v03 standard required to apply test plans to verify that an agency's product or system meets design specifications and other requirements of the NTCIP 1203 DMS standard, while following standard testing methodologies.

The module includes a brief description of the DMS standard with examples in order to perform the verification mentioned above. This module will cover the role of other modes of testing including compliance, manufacturing and acceptance tests, and verification and validation as part of the testing life cycle.

This module shall use a sample test plan for NTCIP 1203 v03 DMS to demonstrate the proper way to create a test plan specific to the user needs and requirements based on the DMS standard, including test procedures. It will walk participants through the process of correctly creating at test plan. This module also explains the types of testing applied within the acquisition process.

This module will be placed in the context of the systems engineering process as well in the acquisition curriculum path. The complete series of ITS Standards Training Modules for acquisition of a DMS is as follows: I101, A101, A102, A201, A311a, A311b, T101, T201, T202, T203, T204, and T311. This module is the final module in the DMS acquisition series.

3. Samples/Examples

The presentation slides uses the functional requirement, Activate Pixel Testing, as an example on how to use the standard to test that a functional requirement has been fulfilled. The full details of this requirement, including the description of the requirement, the design, and the test case and test procedure for this requirement is provided below.

The functional requirement is:

3.5.3.1.1.2 Activate Pixel Testing

The DMS shall allow a management station to initiate a pixel test.

From Annex A, in the Requirements Traceability Matrix (RTM), this functional requirement traces to the following design:

Requirements Traceability Matrix (RTM)
FR ID Functional Requirement Dialog ID Object ID Object Name Additional Specifications
3.5.3 Monitor the Status of the DMS        
3.5.3.1 Perforin Diagnostics        
3.5.3.1.1 Test Operational Status of DMS Components        
3.5.3.1.1.1 Execute Lamp Testing 4.2.4.1    
    5.11.2.5.3 lampTestActivation  
3.5.3.1.1.2 Activate Pixel Testing 4.2.4.2    
    5.11.2.4.3 pixelTestActivation  

The dialog ID requirement 3.5.3.1.1.2 traces to is: 4.2.4.2 Activating Pixel Testing

The standardized dialog for a management station to command the DMS to activate pixel testing shall be as follows:

  1. The management station shall SET pixelTestActivation.0 to 'test'.
  2. The management station shall repeatedly GET pixelTestActivation.0 until it either returns the value of 'noTest' or a maximum time-out is reached. If the time-out is reached, the DMS is apparently locked and the management station shall exit the process.
  3. (PostCondition) The following objects will have been updated during the pixel test to reflect current conditions. The management station may GET any of these objects as appropriate.
    1. pixelFailureTableNumRows
    2. any object within the pixelFailureTable

The object ID requirement 3.5.3.1.1.2 traces to is:

5.11.2.4.3 Pixel Test Activation Parameter

pixelTestActivation OBJECT-TYPE SYNTAX INTEGER {

--other (1), -retired
noTest (2),
test (3),
clearTable (4) }

ACCESS read-write
STATUS mandatory
DESCRIPTION

"<Definition> Indicates the state of the pixel testing. The actual test routine can vary among different manufacturers. The results of the pixel failure test shall be stored in the pixel failure table. The pixel failure table, pixelFailureTableNumRows objects are cleared (both messageDisplay and pixelTest types), when a pixel test is started (test) or a table is cleared (clearTable). Setting the value to test starts the test, meaning this test is executed once. Pixel failures identified by setting this object to test are entered into the pixelTest type of the pixelFailureDetectionType. The sign controller automatically sets the value of this object back to noTest after completion.

<Object Identifier> 1.3.6.1.4.1.1206.4.2.3.9.7.4"

DEFVAL {noTest} ::= { statError 4 }

-- In v02, the enumerated value of 'other' is RETIRED to improve

-- interoperability.

From Annex C, in the Requirements to Test Case Traceability Matrix (RTCTM), this requirement traces to the following test cases:

Requirement Test Case
ID Title ID Title
3.5.3 Monitor the Status of the DMS
3.5.3.1 Perform Diagnostics
3.5.3.1.1 Test Operational Status of DMS Components
3.5.3.1.1.1 Execute Lamp Testing
    C.3.5.21 Verify Lamp Test with No Errors
    C.3.5.22 Verify Lamp Test with Errors
3.5.3.1.1.2 Activate Pixel Testinc  
    C.3.5.1 Pixel Test - No Errors
    C.3.5.2 Pixel Test - Errors
3.5.3.1.1.3 Execute Climate-Control Equipment Testing
    C.3.5.3 Climate-Control Equipment Test - No Errors
    C.3.5.4 Climate-Control Equipment Test - Errors

The test cases requirement 3.5.3.1.1.2 traces to are C.3.5.1 and C.3.5.2, which are reproduced below:

C.3.5.1 Pixel Test - No Errors

Test Case: 5.1 Title: Pixel Test - No Errors
  Description: This test case verifies that the DMS executes a pixel test and verifies that there are no failed pixels.
Variables: Pixel_Test_Time From Manufacturer's Documentation
Message_Display_Test_Time From Manufacturer's Documentation
  Pass/Fail Criteria: The DUT shall pass every verification step included within the Test Case to pass the Test Case.
Step Test Procedure Results Additional References
1 CONFIGURE: Determine the maximum period of time that the pixel test should require (based on manufacturer documentation). RECORD this information as:
»Pixel_Test_Time 		   
2 CONFIGURE: Determine the maximum period of time that the message display pixel test should require (based on manufacturer documentation). RECORD this information as:
»Message_Display_Test_Time 		   
3 SET-UP: Ensure that all pixels are functioning prior to this test.    
4 SET the following object(s) to the value(s) shown:
»pixelTestActivation.0 = 'test' (3)
NOTE--Valid enumerated values are defined in Section 5.11.2.4.3 (Pixel Test Activation Parameter). 		Pass / Fail (Section 3.5.3.1.1.2) Section 4.2.4.2 Step a
5 GET the following object(s):
»pixelTestActivation.0 		Pass / Fail (RFC 1157) Section 4.2.4.2 Step b
6 IF the RESPONSE VALUE for pixelTestActivation.0 equals 'test' (3), then GOTO Step 5; otherwise, GOTO Step 7.
NOTE--If the RESPONSE VALUE remains at 'test' (3) for more than Pixel_Test_Time seconds, this test fails. 		   
7 GET the following object(s):
»dmsPixelStatus.1
»dmsPixelFailureTestRows.0
»dmsPixelFailureMessageRows.0 		Pass / Fail (Section 3.5.3.1.3.3) Section 4.2.4.2 Step c
8 VERIFY that the RESPONSE VALUE for dmsPixelFailureTestRows.0 is equal to 0. Pass / Fail (Section 3.5.3.1.3.3)  
9 GET the following object(s):
»shortErrorStatus.0 		Pass / Fail (Section 3.5.3.1.2)  
10 VERIFY that the RESPONSE VALUE for shortErrorStatus.0 has bit 5 (pixel error) cleared. Pass / Fail (Section 3.5.3.1.2)  
11 PERFORM the test case labeled 'Activate a Message' (C.3.7.6). Pass / Fail (Section 3.5.2.3.1)  
12 DELAY for Message_Display_Test_Time seconds.    
13 GET the following object(s):
»dmsPixelFailureTestRows.0
»dmsPixelFailureMessageRows.0
»pixelFailureTableNumRows.0 		Pass / Fail (Section 3.5.3.1.3.3) Section 4.2.4.2 Step c
14 VERIFY that the RESPONSE VALUE for dmsPixelFailureMessageRows.0 is equal to 0. Pass / Fail (Section 3.5.3.1.3.3)  
15 VERIFY that the RESPONSE VALUE for pixelFailureTableNumRows.0 is equal to 0. Pass / Fail (Section 3.5.3.1.3.3)  
16 GET the following object(s):
»shortErrorStatus.0 		Pass / Fail (Section 3.5.3.1.2)  
17 VERIFY that the RESPONSE VALUE for shortErrorStatus.0 has bit 5 (pixel error) cleared. Pass / Fail (Section 3.5.3.1.2)  
18 PERFORM the test case labeled 'Blank the Sign' (C.3.7.15). Pass / Fail (Section 3.5.2.3.1)  
Test Case Results
Tested By: Date Tested: Pass / Fail
Test Case Notes:

C.3.5.2 Pixel Test - Errors

Test Case: 5.2 Title: Pixel Test - Errors
Description: This test case verifies that the DMS executes a pixel test and verifies that there are failed pixels to be detected by unplugging the power or signal to the pixel boards.
Variables: Pixel_Test_Time From Manufacturer's Documentation
Message_Display_Test_Time From Manufacturer's Documentation
Pass/Fail Criteria: The DUT shall pass every verification step included within the Test Case to pass the Test Case.
Step Test Procedure Results Additional References
1 CONFIGURE: Determine the maximum period of time that the pixel test should require (based on manufacturer documentation). RECORD this information as:
»Pixel_Test_Time 		   
2 CONFIGURE: Determine the maximum period of time that the message display pixel test should require (based on manufacturer documentation). RECORD this information as:
»Message_Display_Test_Time 		   
3 SET-UP: Unplug the power or signal to several pixels to simulate failed pixels to detect within this test procedure.    
4 SET the following object(s) to the value(s) shown:
»pixelTestActivation.0 = 'test' (3)
NOTE--Valid enumerated values are defined in Section 5.11.2.4.3 (Pixel Test Activation Parameter). 		Pass / Fail (Section 3.5.3.1.1.2) Section 4.2.4.2 Step a
5 GET the following object(s): »pixelTestActivation.0 Pass / Fail (Section 3.5.3.1.1.2) Section 4.2.4.2 Step b
6 IF the RESPONSE VALUE for pixelTestActivation.0 equals 'test' (3), then GOTO Step 5; otherwise, GOTO Step 7.
NOTE--If the RESPONSE VALUE remains at 'test' (3) for more than Pixel_Test_Time seconds, this test fails. 		   
7 GET the following object(s):
»shortErrorStatus.0 		Pass / Fail (Section 3.5.3.1.2)  
8 VERIFY that the RESPONSE VALUE for shortErrorStatus.0 has bit 5 (pixel error) set. Pass / Fail (Section 3.5.3.1.2)  
9 GET the following object(s):
»vmsSignHeightPixels.0 »vmsSignWidthPixels.0 		Pass / Fail (Section 3.5.1.2.2.1)  
10 RECORD the RESPONSE VALUE for vmsSignHeightPixels.0 and vmsSignWidthPixels.0 as:
»Actual_Height_Pixels
»Actual_Width_Pixels 		   
11 Calculate the number of pixels in the sign. RECORD this information as:
»Total_Pixels
NOTE--In general, the number of pixels in the sign can be determined by multiplying the sign height in pixels by the sign width in pixels. This algorithm is not valid if the pixels on the sign do not form a perfectly rectangular matrix. 		   
12 Calculate the number of pixel status objects required to be retrieved using the formula: Total_Pixels / 3200, rounded up to the next integer. RECORD this information as:
»Num_Pixel_Blocks 		   
13 GET the following object(s):
»dmsPixelFailureTestRows.0
»dmsPixelFailureMessageRows.0 		   
14 FOR EACH value, N, from 1 to Num_Pixel_Blocks, perform Steps 14.1 through 14.2.
NOTE--For example, if Total_Pixels equals 3201, N shall be assigned a value from 1 to 2. 		   
14.1 GET the following object(s):
»dmsPixelStatus.N 		Pass / Fail (Section 3.5.3.1.3.3)  
14.2 RECORD the RESPONSE VALUE for dmsPixelStatus.N, dmsPixelFailureTestRows.0, dmsPixelFailureMessageRows.0 as:
»Pixel_Status[N]
»Pixel_Failure_Test_Rows
»Pixel_Failure_Message_Rows 		   
15 VERIFY that the number of bits set in all of the Pixel_Status[N] parameters equals Pixel_Failure_Test_Rows. Pass / Fail (Section 3.5.3.1.3.3)  
16 FOR EACH value, N, from 1 to Pixel_Failure_Test_Rows, perform Steps 16.1 through 16.7.    
16.1 GET the following object(s):
»pixelFailureXLocation.2.N
»pixelFailureYLocation.2.N
»pixelFailureStatus.2.N 		Pass / Fail (Section 3.5.3.1.4.3) Section 4.2.4.6 Step c
16.2 Determine the X and Y location of the pixel. RECORD this information as:
»X
»Y 		   
16.3 Calculate the text string describing the location of the failed pixel. RECORD this information as:
»Failed_Pixel_Location 		   
16.4 VERIFY that the RESPONSE VALUE for pixelFailureXLocation.2.N and pixelFailureYLocation.2.N identify one of the failed pixels that has not been previously identified. Pass / Fail (Section 3.5.3.1.4.3)  
16.5 VERIFY that the RESPONSE VALUE for pixelFailureStatus.2.N is not equal to 0. Pass / Fail (Section 3.5.3.1.4.3)  
16.6 Calculate the unique pixel number of the failed pixel. RECORD this information as:
»Subject_Pixel
NOTE--The unique pixel number is defined by its position on the sign, where the top and left-most pixel is pixel 0, the next one to the right is pixel 1, etc. Assuming a perfectly rectangular matrix sign, the pixel number can be calculated by multiplying the (Y position minus 1) by the sign width in pixels and adding the X position minus 1. 		   
16.7 VERIFY that the bit corresponding to the Subject_Pixel in Pixel_Status[N] is set to one. Pass / Fail (Section 3.5.3.1.4.3)  
17 PERFORM the test case labeled 'Activate a Message' (C.3.7.6).
NOTE--This step is allowed to fail if the sign has internal logic to prevent the display of the message due to an excessive number of pixel failures. 		Pass / Fail (Section 3.5.2.3.1)  
18 VERIFY that the disconnected pixels are blank. Pass / Fail (Section 3.5.3.1.3.3)  
19 DELAY for Message_Display_Test_Time seconds.    
20 GET the following object(s):
»shortErrorStatus.0 		Pass / Fail (Section 3.5.3.1.2)  
21 VERIFY that the RESPONSE VALUE for shortErrorStatus.0 has bit 5 (pixel error) set. Pass / Fail (Section 3.5.3.1.2)  
22 GET the following object(s):
»dmsPixelFailureTestRows.0
»dmsPixelFailureMessageRows.0 		Pass / Fail (Section 3.5.3.1.3.3)  
23 Determine the RESPONSE VALUE. RECORD this information as:
»Pixel_Failure_Test_Rows
»Pixel_Failure_Message_Rows 		   
24 FOR EACH value, N, from 1 to Pixel Failure Message Rows, perform Steps 24.1 through 24.7.    
24.1 GET the following object(s):
»pixelFailureXLocation.3.N
»pixelFailureYLocation.3.N
»pixelFailureStatus.3.N 		Pass / Fail (Section 3.5.3.1.4.3) Section 4.2.4.6 Step c
24.2 Determine the X and Y location of the reported pixel failure. RECORD this information as:
»X
»Y 		   
24.3 Calculate the text string describing the location of the failed pixel. RECORD this information as:
»Failed_Pixel_Location 		   
24.4 VERIFY that the RESPONSE VALUE for pixelFailureXLocation.3.N and pixelFailureYLocation.3.N identify one of the failed pixels that has not been previously identified in the message. Pass / Fail (Section 3.5.3.1.4.3)  
24.5 VERIFY that the RESPONSE VALUE for pixelFailureStatus.3.N is not equal to 0. Pass / Fail (Section 3.5.3.1.4.3)  
24.6 Calculate the unique pixel number of the failed pixel. RECORD this information as:
»Subject_Pixel
NOTE--The unique pixel number is defined by its position on the sign, where the top and left-most pixel is pixel 1, the next one to the right is pixel 2, etc. Assuming a perfectly rectangular matrix sign, the pixel number can be calculated by multiplying the (Y position minus 1) by the sign width in pixels and adding the X position minus 1. 		   
24.7 VERIFY that the bit corresponding to the Subject_Pixel in Pixel_Status[N] is set to one. Pass / Fail (Section 3.5.3.1.4.3)  
25 SET-UP: Reconnect the power or signal to the pixels from which it was removed.    
26 GET the following object(s):
»shortErrorStatus.0 		Pass / Fail (Section 3.5.3.1.2)  
27 VERIFY that the RESPONSE VALUE for shortErrorStatus.0 has bit 5 (pixel error) cleared. Pass / Fail (Section 3.5.3.1.2)  
28 PERFORM the test case labeled 'Blank the Sign' (C.3.7.15). Pass / Fail (Section 3.5.2.3.1)  
Test Case Results
Tested By: Date Tested: Pass / Fail
Test Case Notes:

4. Reference to Other Standards

Dynamic Message Signs

Systems Engineering

Testing

5. Case Studies

6. Glossary

Term Definition
Agency Specification A document that has been prepared by an agency to define requirements for a subject item or process when procured by the agency.
Compliance A condition that exists when an item meets all of the requirements of an agency specification.
Conformance A condition that exists when an item meets all of the mandatory requirements as defined by a standard. It can be measured on the standard as a whole, which means that it meets all mandatory (and applicable conditional) requirements of the standard or on a feature level (i.e., it conforms to feature X as defined in section X.X.X), which means that it meets all mandatory (and applicable conditional) requirements of the feature.
Device Under Test (DUT) A device that is the object of testing.
Dialogs A sequence of information or message exchanges.
DMS Dynamic Message Sign.
DUT Device Under Test.
Dynamic Message Sign Any sign system that can change the message presented to the viewer, such as VMS, CMS, and BOS. It includes the following major components: sign face, sign housing, controller, and, if present, the controller cabinet.
Interoperability The ability of two or more systems or components to exchange information and use the information that has been exchanged.
NTCIP National Transportation Communications for ITS Protocol.
PRL Protocol Requirements List.
Protocol Requirements List A table mapping user needs with their associated requirements.
(PRL) This table allows procurement personnel to specify the desired features of a DMS or can be used by a manufacturer to document the features supported by their implementation.
Requirements to Test Case Traceability Matrix (RTCTM) A table that defines the traceability from a requirement to the associated test case
RTCTM Requirements to Test Cases Traceability Matrix.
RTM Requirements Traceability Matrix.
TDS Test Design Specification.
Test Case Specification A document that specifies the test inputs, execution conditions, and predicted results for an item to be tested.
Test Design Specification Documentation specifying the details of the test approach for a software feature or combination of software features and identifying the associated tests.
Test Documentation Documentation describing plans for, or results of, the testing of a system or component. Types include test case specification, test incident report, test log, test plan, test procedure, and test report.
Test Incident Report A document reporting on any event that occurs during the testing process that requires investigation.
Test Log A chronological record of relevant details about the execution of tests.
Test Plan A document describing the scope, approach, resources, and schedule of intended test activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning.
Test Procedure Specification See Test Procedure.
Test Summary Report A document summarizing testing activities and results. It also contains an evaluation of the corresponding test items.
TPG Test Procedure Generator.
VMS Variable Message Sign.

7. References

8. Study Questions

To include the quiz/poll questions and answer choices as presented in the PowerPoint slide to allow students to either follow along with the recording or refer to the quiz at a later date in the supplement.

1. What does a "test case specification" do?

  1. Specifies the inputs, predicted results, and the conditions for one or more functions in the test item.
  2. Specifies the details of the test approach for a feature or combination of features.
  3. Describes the scope, approach, and resources for the testing activities.
  4. Specifies the sequence of actions for the execution of a test.

2. What is the purpose of the Requirements to Test Case Matrix?

  1. Identify the requirements that are part of project specification.
  2. Identify all the test cases that must be passed to verify the requirement is fulfilled.
  3. Identify the design content to fulfill a requirement.
  4. Identify one of the possible test cases that must be passed to verify a requirement is fulfilled.

3. Which of the following information is not provided in a test plan?

  1. What item is being tested?
  2. Who is to test the item?
  3. What are the inputs and outputs for the test case specification?
  4. What are the test deliverables?

4. What is the Requirements to Test Case Traceability Matrix (RTCTM) in a Test Design Specification based upon?

  1. Includes all the requirements supported by the standard.
  2. Includes only the requirements selected in the PRL that the Test Design Specification is based upon.
  3. Includes only those requirements that are mandatory to conform to the standard.
  4. Includes all the requirements that are contained in the project specifications.

9. Icon Guide

The following icons are used throughout the module to visually indicate the corresponding learning concept listed below, and/or to highlight a specific point in the training material.

1) Background information: General knowledge that is available elsewhere and is outside the module being presented. This will be used primarily in the beginning of the slide set when reviewing information readers are expected to already know.

Background information icon indicates general knowledge that is available elsewhere and is outside the module being presented.

2) Tools/Applications: An industry-specific item a person would use to accomplish a specific task, and application of that tool to fit the need.

Tools/Applications icon. An industry-specific item a person would use to accomplish a specific task, and applying that tool to fit your need.

3) Remember: Used when referencing something already discussed in the module that is necessary to recount.

Remember icon. Used when referencing something already discussed in the module that is necessary to recount.

4) Refer to Student Supplement: Items or information that are further explained/detailed in the Student Supplement.

Supplement icon indicating items or information that are further explained/detailed in the Student Supplement.

5) Example: Can be real-world (case study), hypothetical, a sample of a table, etc.

Example icon. Can be real-world (case study), hypothetical, a sample of a table, etc.

6) Checklist: Used to indicate a process that is being laid out sequentially.

Checklist icon used to indicate a process that is being laid out sequentially.