Module 35 - T315

T315: Applying Your Test Plan to the NTCIP 1202 ASC Standard

HTML of the PowerPoint Presentation

(Note: This document has been converted from a PowerPoint presentation to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)

Slide 1:

(Extended Text Description: Slide 1: Welcome - Graphic image of introductory slide. A large dark blue rectangle with a wide, light grid pattern at the top half and bands of dark and lighter blue bands below. There is a white square ITS logo box with words "Standards ITS Training" in green and blue on the middle left side. The word "Welcome" in white is to the right of the logo. Under the logo box are the words "RITA Intelligent Transportation Systems Joint Program Office.")

Slide 2:

Welcome

Ken Leonard, Director

ITS Joint Program Office

Ken.Leonard@dot.gov

(Extended Text Description: Intro Slide: Screen capture snapshot of RITA website - for illustration only. Below this image is a link to the current website: https://www.its.dot.gov/pcb - this screen capture snapshot shows an example from the RITA website from April 2013. At the top of the page it shows the RITA logo with the text U.S. Department of Transportation Research and Innovative Technology Administration - Intelligent Transportation Systems Joint Program Office - ITS Professional Capacity Building Program/Advanced ITS Education. Below the main site banner, it shows the main navigation menu with the following items: About, ITS Training, Knowledge Exchange, Technology Transfer, ITS in Academics, and Media Library. Below the main navigation menu, the page shows various content of the website, including a graphic image of professionals seated in a room during a training program. A text overlay has the text Welcome to ITS Professional Capacity Building. Additional content on the page includes a box entitled What's New and sections entitled Available E-Training (free), Free ITS Training and T3 Webinars. Again, this image serves for illustration only. The current website link is: https://www.its.dot.gov/pcb)

ITS PCB Home

(Note: There is additional text attached to this slide that includes the following introductory information from Ken Leonard):

"ITS Standards can make your life easier. Your procurements will go more smoothly and you'll encourage competition, but only if you know how to write them into your specifications and test them. This module is one in a series that covers practical applications for acquiring and testing standards-based ITS systems.

I am Ken Leonard, director of the ITS Joint Program Office for USDOT and I want to welcome you to our newly redesigned ITS standards training program of which this module is a part. We are pleased to be working with our partner, the Institute of Transportation Engineers, to deliver this new approach to training that combines web based modules with instructor interaction to bring the latest in ITS learning to busy professionals like yourself.

This combined approach allows interested professionals to schedule training at your convenience, without the need to travel. After you complete this training, we hope that you will tell colleagues and customers about the latest ITS standards and encourage them to take advantage of the archived version of the webinars.

ITS Standards training is one of the first offerings of our updated Professional Capacity Training Program. Through the PCB program we prepare professionals to adopt proven and emerging ITS technologies that will make surface transportation safer, smarter and greener which improves livability for us all. You can find information on additional modules and training programs on our web site ITS PCB Home

Please help us make even more improvements to our training modules through the evaluation process. We look forward to hearing your comments. Thank you again for participating and we hope you find this module helpful."

Slide 3:

Slide 4:

T315:

Applying Your Test Plan to the NTCIP 1202 ASC Standard

Slide 5:

Instructor

Kenneth L. Vaughn, P.E.
President
Trevilon LLC
Magnolia, TX, USA

Slide 6:

Target Audience

• Engineering staff
• Operations and maintenance staff
• System integrators
• Device manufacturers
• Testing contractors

Slide 7:

Recommended Prerequisite(s)

• T101: Introduction to ITS Standards Testing
• T201: How to Write a Test Plan
• T202: Overview of Test Design Specifications, Test Cases, and Test Procedures
• C101: Introduction to the Communication Protocols and Their Uses in ITS Applications
• A315a: Understanding User Needs for Actuated Traffic Signal Controllers (ASC) Based on the NTCIP 1202 Standard
• A315b: Understanding Requirements for Actuated Traffic Signal Controllers (ASC) Based on the NTCIP 1202 Standard

Slide 8:

Curriculum Path

(Extended Text Description: A graphical illustration indicating the sequence of training modules that lead up to and follow this course. Each module is represented by a box with the name of the module in it and an arrow showing the logical flow of the modules and the current module highlighted. This slide focuses on the modules that lead up to the current course. The first box is labeled "T101 Introduction to ITS Standards Testing." An arrow from this box connects it to a box labeled "T201 How to Write a Test Plan." An arrow from this box connects it to a box labeled "T202 Overview of Test Design Specifications, Test Cases, and Test Procedures." An arrow from this box connects it to a box located at the start of the next line and labeled "C101 Introduction to the Comm Protocols and Their Uses in ITS Applications." An arrow from this box connects it to a box labeled "A315a Understanding User Needs for ASC Based on NTCIP 1202 Standard." An arrow from this box connects it to a box labeled "A315b Understanding Requirements for ASC Based on NTCIP 1202 Standard." Finally, an arrow from this box connects it to a highlighted box designating this module, which is located at the start of the next line and labeled "T315 Applying Your Test Plan (ASC) to NTCIP 1202 ASC Standard.")

Slide 9:

Learning Objectives

1. Recognize the importance of testing ASCs
2. Apply the rules for developing a sample ASC test plan
3. List the rules for developing test case specifications and procedures for NTCIP 1202
4. Develop sample test case specifications and procedures for NTCIP 1202
5. Understand testing results for NTCIP 1202

Slide 10:

Learning Objective #1 — Recognize the Importance of Testing ASCs

• Safety implications
• Complexity issues
• Functional vs. communications testing
• Repeatability
• Budgeting

Slide 11:

Learning Objective #1

Recognize the Importance of Testing

Safety Implications

• ASCs directly control traffic
  • Relatively unique for NTCIP
  • Conflict monitor only provides a base level of safety
  • Improper timing can still result in safety issues
  • Poor timing can increase crash rates
• Improper implementation of NTCIP can result in undesired timing

Slide 12:

Learning Objective #1

Recognize the Importance of Testing

Complexity Issues

• ASCs have perhaps the most complex NTCIP interface
  • Complex logic increases potential for errors
  • ASCs have several complex features
  • ASCs have many features that continually interact
  • Time sensitivity complicates interactions
• Testing needs to be performed in steps
  • Isolate features and test
  • Combine proven features and test again

Slide 13:

Learning Objective #1

Recognize the Importance of Testing

Functional vs. Communication Testing

What is NTCIP testing?

• Communications testing (i.e., syntax)
  • Ensuring that the yellow change interval is set to the desired value
• Functional testing (i.e., semantics)
  • Ensuring that the ASC properly displays the yellow for the defined time

Both are defined by NTCIP, but test plan may separate for convenience

Slide 14:

Learning Objective #1

Recognize the Importance of Testing

Repeatability

• Proper test documentation ensures repeatable tests
• Especially important for complex devices
• Unexpected operation may occur due to:
  • Implementation errors
  • Tester errors
  • Errors in test procedure
  • Problems in specification
  • Problems in standard

Slide 15:

Learning Objective #1

Recognize the Importance of Testing

Budgeting

• Software testing often reveals errors
  • Even mainstream computer applications experience bugs
• Errors should be corrected before deployment
• Devices should be retested after any software update
• Full testing is likely require multiple rounds of testing
  • Budget impacts
  • Schedule impacts

Slide 16:

Slide 17:

Learning Objective #1

Why is it important to test ASCs?

1. Implementation errors might result in conflicting green displays
2. Implementation errors might decrease traveler safety
3. Testing is not important for ASCs
4. It gives peace of mind for a trivial cost

Slide 18:

Learning Objective #1

Review of Answers

a) Implementation errors might result in conflicting green displays
Incorrect; a properly configured conflict monitor will ensure that conflicting displays are never shown.

b) Implementation errors might decrease traveler safety
Correct! An error can result in an undesired configuration and decrease safety.

c) Testing is not important for ASCs
Incorrect; testing is especially important for ASCs due to their ability to directly control traffic operations.

d) It gives peace of mind for a trivial cost
Incorrect; proper testing requires significant effort.

Slide 19:

Summary of Learning Objective #1

Recognize the Importance of Testing ASCs

• Safety implications
• Complexity issues
• Functional vs. communications testing
• Repeatability
• Budgeting

Slide 20:

Learning Objective #2 — Develop a Sample Test Plan

• Identify requirements to test for each testing phase
• Identify test methodology
• Describe requirements to test case traceability matrix
• Plan logistics of testing
• Estimate level of effort for testing
• Evaluate risks
• Plan project close-out

Slide 21:

Learning Objective #2

Develop a Sample Test Plan

Test Documentation

• IEEE 829 Defines Test Documentation
  • Provides sample outline of Test Plan
  • Discussed in detail in Module T201

Slide 22:

Learning Objective #2

Develop a Sample Test Plan

Identify Requirements to Test

• Module A315b identified how to define ASC requirements
  • See Participant Student Supplement for list of sample requirements
• Every requirement should be tested:
  • During at least one test phase
  • Using at least one method
  • By at least one party
• Extent of agency testing is a risk management issue

Slide 23:

Learning Objective #2

Develop a Sample Test Plan

Identify Test Phase

• Each test phase will have its own test plan
  • Prototype
  • Design Approval
  • Factory Acceptance
  • Incoming Device
  • Site Acceptance
  • Burn-in
• Often further divided
  • NTCIP testing
  • Hardware testing
  • Etc.

Slide 24:

Learning Objective #2

Develop a Sample Test Plan

Identify Test Methodology

• Inspection
• Analysis
• Demonstration
• Formal testing

• Consider testing scenarios
  • Positive test(s)
  • Negative test(s)

Slide 25:

Learning Objective #2

Develop a Sample Test Plan

Requirements to Test Case Traceability Matrix

Req't ID	Requirement	TC ID	Test Case
2.2.1.1	Configure a Timing Pattern	2.3.1.1	Configure a Valid Timing Pattern
		2.3.1.2	Incorrectly Configure a Timing Pattern
2.2.4.1	Support at least 32 Timing Patterns	2.3.1.1	Configure a Valid Timing Pattern
		2.3.1.3	Configure Timing Pattern 32
2.2.1.2	Configure Timing Pattern Selection Logic	See Manufacturer Factory Acceptance Test Plan
2.2.2.1	Activate Timing Pattern Remotely	2.3.1.5	Activate Timing Pattern Remotely
2.2.2.2	Activate a Timing Pattern per a Schedule	2.3.1.6	Activate a Timing Pattern per a Schedule

* Req't and TC ID reference clauses in the Participant Student Supplement

Slide 26:

Learning Objective #2

Develop a Sample Test Plan

Identify Test Environment

(Extended Text Description: A sample test environment as incorporated from NTCIP 8007, Page 13. A "device under test (DUT)" is connected to a communications cloud which is connected to a "test application" running on a PC. Also connected to the communications cloud is a data analyzer, also running on a PC.)

Source: NTCIP 8007, page 13

Slide 27:

Learning Objective #2

Develop a Sample Test Plan

Plan Logistics of Testing

• Where will tests be performed?
  • Safety during on-site testing
• Who is responsible for what?
  • Power
  • Tools
  • Tables
  • Protection from elements
  • Local assistance for remote testing
• What happens if testing is suspended?

Slide 28:

Learning Objective #2

Develop a Sample Test Plan

Estimate Level of Effort

Estimate effort, schedule, and budget for:

• Preparing test plan
• Preparing test cases
• Preparing test procedures
• Performing multiple rounds of testing
  • Performing tests
  • Investigating problems
  • Preparing test documentation

Slide 29:

Learning Objective #2

Develop a Sample Test Plan

Evaluate Risks

• What happens if the device does not pass?
  • How many rounds of testing will be conducted?
  • Who pays for additional rounds of testing?
  • Do external factors control the schedule?
  • What happens if the delivered equipment is unsatisfactory?
• How are disputes resolved?
  • What happens if there is an problem with the specification or standard?

Slide 30:

Learning Objective #2

Develop a Sample Test Plan

Understanding the Impact of a Failure

Req' t ID	Requirement	TC ID	Test Case
2.2.1.1	Configure a Timing Pattern	2.3.1.1	Configure a Valid Timing Pattern
		2.3.1.2	Incorrectly Configure a Timing Pattern
2.2.4.1	Support at least 32 Timing Patterns	2.3.1.1	Configure a Valid Timing Pattern
		2.3.1.3	Configure Timing Pattern 32
2.2.1.2	Configure Timing Pattern Selection Logic	See Manufacturer Factory Acceptance Test Plan
2.2.2.1	Activate Timing Pattern Remotely	2.3.1.5	Activate Timing Pattern Remotely
2.2.2.2	Activate a Timing Pattern per a Schedule	2.3.1.6	Activate a Timing Pattern per a Schedule
H.2.2.1	Set Time	2.3.3.1	Set Time

* Req't and TC ID reference clauses in the Participant Student Supplement

Slide 31:

Learning Objective #2

Develop a Sample Test Plan

Understanding the Impact of a Failure

UN ID	User Need	RID	Requirement
2.1.3.1	Control Selection of Timing Pattern
		2.2.1.1	Configure a Timing Pattern
		2.2.4.1	Support at Least 32 Timing Patterns
		2.2.1.2	Configure Timing Pattern Selection Logic
		2.2.2.1	Activate Timing Pattern Remotely
		2.2.2.2	Activate a Timing Pattern per a Schedule
		2.2.2.3	Override Timing Pattern
		2.2.3.6	Retrieve a Schedule
		2.2.2.8	Define a Schedule
		NTCIP 1203v03 H.2.2.1	Set Time
		NTCIP 1203v03 H.2.2.2	Set Time Zone
		NTCIP 1203v03 H.2.2.3	Set Daylight Savings Mode
		NTCIP 1203v03 H.2.2.4	Verify Current Time
		...	...

* UN and R ID reference clauses in the Participant Student Supplement

Slide 32:

Learning Objective #2

Develop a Sample Test Plan

Plan Project Close-out

• Have a plan
• Understand the impacts of accepting a failure

Slide 33:

Slide 34:

Learning Objective #2

Which of the following statements is not true?

1. Every requirement should be tested
2. Some testing may be performed by the manufacturer
3. You should only need to perform your test plan once
4. Traceability tables can help you to assess the impact of a test failure

Slide 35:

Learning Objective #2

Review of Answers

a) Every requirement should be tested
True; every requirement should be tested.

b) Some testing may be performed by manufacturer
True; testing may be performed by the agency, the manufacturer, or a third party.

c) You should only need to perform your test plan once
False (correct). Testing will often reveal problems; these should be fixed and the device retested.

d) Traceability tables can help you to assess the impact of a test failure
True; traceability tables allow you to identify the user needs that will not be completely fulfilled.

Slide 36:

Summary of Learning Objective #2

Develop a Sample Test Plan

• Discussed assigning requirements to test phases
• Identified test methodology
• Requirements to test case traceability matrix
• Discussed level of effort
• Considered risks

Slide 37:

Learning Objective #3 — List the Rules for Developing Test Case Specifications and Procedures for NTCIP 1202

• Review guidance from IEEE 829 and NTCIP 8007
• Apply guidance to sample dialog

Slide 38:

Learning Objective #3

Test Case Specifications and Procedures

Review Guidance

• IEEE 829 defines other test documentation
  • Test design specification
  • Test case specification
  • Test procedure specification
• Can be grouped together
  • "Each organization using the standard will need to specify the specific documents required for [a] particular test phase."

Slide 39:

Learning Objective #3

Test Case Specifications and Procedures

Components

NTCIP 8007 describes how these can be combined for NTCIP testing

• Test case identifier
• Purpose
• Inputs
• Pass/Fail criteria
• Procedure steps
  • Steps
    • Can reference other often used procedures
  • Expected outputs
  • Features tested

Slide 40:

Test Case Specifications and Procedures

Sample Basic Dialog

• Set Time Dialog

(Extended Text Description: A slide showing a sample basic dialog. It is a UML sequence diagram with a "TMS" sending a "Set(globalTime.0)" message to an "ASC" followed by the "ASC" sending an unnamed response message to the "TMS".)

Slide 41:

Learning Objective #3

Test Case Specifications and Procedures

Specify Each Test Case

• Define test case before test procedure

Req't ID	Requirement	TC ID	Test Case
H.2.2.1	Set Time	2.3.2.1	Set Time

Test Case 2.3.3.1	Title	Set Time
	Description	This test case verifies that the ASC properly tracks time. It advances the clock by a user-defined amount, waits a few seconds, retrieves the time, and verifies it indicates an appropriate value. This test will advance the ASC clock by Time_Offset seconds.
	Variables	Time_Offset as defined in the test plan.
	Pass/Fail Criteria	The DUT shall pass every verification step included within the Test Case to pass the Test Case.

NOTE: If requirement is from an SEP standard, you may be able to reference the standard test case as well

Slide 42:

Learning Objective #3

Test Case Specifications and Procedures

Specify Procedures

• Data exchanges should follow defined dialogs
• Return the device to its original state (generally)
• Verification steps should cite the relevant requirement
  • A test case typically tests multiple requirements
• NTCIP 8007 precisely defines standardized step types
  • A "SET" operation includes 9 specific verification checks related to the Simple Network Management Protocol (SNMP) response packet.

Slide 43:

Learning Objective #3

Test Case Specification and Procedures

Sample Procedure

Step	Test Procedure	Results
1	CONFIGURE: Determine the number of seconds to advance the clock in the ASC. RECORD this information as: Time_Offset
2	GET the following object(s): globalTime.0	Pass/Fail (Section H.2.2.4)
3	RECORD the RESPONSE VALUE for globalTime.0 as Start_Time
4	SET the following object(s): globalTime.0 = Start_Time + Time_Offset	Pass/Fail (Section H.2.2.1)
5	DELAY for 15 seconds
6	GET the following object(s): globalTime.0	Pass/Fail (Section H.2.2.4)
7	VERIFY that the RESPONSE VALUE for globalTime.0 is roughly equal to Start_Time + Time_Offset + 15	Pass/Fail (Section H.2.2.4)

Slide 44:

Slide 45:

Learning Objective #3

Where can you find definitions for terms that can be used in NTCIP test steps?

1. IEEE 829
2. ISO 9001
3. NTCIP 8007
4. Student supplement

Slide 46:

Learning Objective #3

Review of Answers

a) IEEE 829
Incorrect; IEEE 829 defines sample outlines for test documentation, but does not define steps for NTCIP

b) ISO 9001
Incorrect; ISO 9001 deals with quality management, but does not deal directly with NTCIP testing

c) NTCIP 8007
Correct! NTCIP 8007 defines a number of terms that can be used in test steps for NTCIP testing

d) Student Supplement
Incorrect; the student supplement provides samples of test procedures, but it does not define the test terms.

Slide 47:

Summary of Learning Objective #3

List the Rules for Developing Test Case Specifications and Procedures for NTCIP 1202

• Reviewed IEEE 829 and NTCIP 8007
• Applied logic to a simple dialog

Slide 48:

Learning Objective #4 — Develop Sample Test Case Specifications and Procedures for NTCIP 1202

• Database transactions
• Consistency checks
• Simple Transportation Management Protocol (STMP)
• Load testing
• Other key requirements to consider

Slide 49:

Develop Sample Test Cases and Procedures Database Transactions

• ASCs are required to support transaction mode
  • Allows complex data transfers
  • Allows complex consistency checks on data
  • Rejects entire transaction if there is a consistency error
• Need to test that the mode works correctly for all object categories

Slide 50:

Learning Objective #4

Develop Sample Test Cases and Procedures Database Transactions

• NTCIP 1202v02 Annex A categorizes every object
  • C (Control) objects can be set at any time and without delay
  • P (Parameter) objects can be set in normal mode, but are buffered when in transaction mode
  • P2 (Parameter 2) objects can only be set when in transaction mode
  • S (Status) objects can never be set
  • Any object can be retrieved at any time

Slide 51:

Learning Objective #4

Database Transactions

Sample User Need

Agencies need to be able to fully configure an ASC remotely. Fully configuring an ASC requires setting a large number of inter-related parameters that may not fit into a single request and may require considerable time to validate; however, the entire configuration must be implemented simultaneously due to their interrelationships.

Slide 52:

Learning Objective #4

Database Transactions

Sample Requirement

Until all inter-related database parameters have been downloaded, the ASC shall not implement new values for any database parameter but will process operations on non-database parameters normally.

Slide 53:

Learning Objective #4

Database Transactions

Sample Dialog Part 1

(Extended Text Description: A slide showing part 1 of the sample dialog for database transactions. It is depicted by a UML sequence diagram between a "TMS" and an "ASC". The dialog starts with the "TMS" sending a "Get(dbCreateTrtansaction.0)" message to the "ASC". This is followed by the "ASC" sending an unlabeled response message to the "TMS". If the response is normal, the sequence continues and the "TMS" sends a "Set(dbCreateTransaction = transaction)" message to the "ASC", which responds with an unnamed response. The "TMS" then enters a loop, designated with a box labeled "repeat as needed in any order" which includes: 1. The "TMS" sends a "Set(any combination of writable objects to a valid values") and the "ASC" responds with a response message. 2. The "TMS" sends a "Get(any object)" message and the "ASC" responds with a response message.)

Slide 54:

Learning Objective #4

Database Transactions

Sample Dialog Part 2

(Extended Text Description: A slide showing part 2 of the sample dialog for database transactions. It is depicted by a UML sequence diagram between a "TMS" and an "ASC". The dialog continues with the "TMS" sending a "Set(dbCreateTrtansaction.0 = verify)" message and the "ASC" responds with a response message with a note that the ASC will "respond and then perform consistency checks". The "TMS" then enters a loop, designated with a box labeled "repeat until dbCreateTransaction.0 = done"; the box contains: 1. The "TMS" sending a "Get(dbCreateTransaction.0)" message and the "ASC" responding with a response message. The dialog then continues with the "TMS" sending a "Get(dbVerifyStatus.0, dbVerifyError.0)" message and the "ASC" responding with a response message. The "TMS" then sends a "Set(dbCreateTransaction.0 = normal)" message and the "ASC" responds with a response message.)

Slide 55:

Learning Objective #4

Database Transactions

Sample Test Cases

• Download a valid database configuration
• Download a database configuration with syntax errors
• Download an database configuration with consistency errors
• Download a valid database configuration with other valid requests
• Download a valid database configuration with other invalid requests
• Download a database configuration with commands mixed with database objects
• Cancel a database download

Slide 56:

Learning Objective #4

Database Transactions

Sample Test Case

Test Case 6.1	Title	Download a valid database configuration
	Description	This test case verifies that the ASC properly buffers database parameters until all inter-related parameters are downloaded. It initializes, starts the database transaction, downloads a number of parameters, retrieves the parameters, verifies that they are unchanged, completes the transaction, retrieves the parameters, and verifies that they have been updated. This test modifies the database configuration for the ASC.
	Variables	Phase_Startup_1, ... Phase_Options_1, ... Phase_Ring_1, ... ...
	Pass/Fail Criteria	The DUT shall pass every verification step included within the Test Case to pass the Test Case.

Slide 57:

Learning Objective #4

Database Transactions

Sample Procedure - Initialize and Start

Step	Test Procedure	Results
1	CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Phase_Startup_1
...	...
100	GET the following object(s): phaseStartup.1 phaseStartup.2 ...	Pass/Fail (NTCIP 1202 - 2.2.2.20)
101	RECORD phaseStartup.1 as Orig_Phase_Startup_1
...	...	...
200	GET the following object(s): dbCreateTransaction.0	Pass/Fail (NTCIP 1201 -2.3.1)
201	SET-UP: VERIFY that the value of dbCreateTransaction.0 is equal to 'normal' (1).
202	SET the following object(s) to the values shown: dbCreateTransaction.0 = 'transaction' (2)	Pass/Fail (NTCIP 1201 -2.3.1)

Slide 58:

Learning Objective #4

Database Transactions

Sample Procedure - Download

Step	Test Procedure	Results
203	SET the following object(s) to the values shown: phaseStartup.1 = Phase_Startup_1 phaseStartup.2 = Phase_Startup_2 ...	Pass/Fail (NTCIP 1201 -2.3.1; NTCIP 1202 -2.2.2.20)
...	...	...
300	GET the following object(s) : phaseStartup.1 phaseStartup.2 ...	Pass/Fail (NTCIP 1202 - 2.2.2.20)
301	VERIFY that phaseStartup.1 equals Orig_Phase_Startup_1	Pass/Fail (NTCIP 1201 -2.3.1)
...	...	...
400	SET the following object(s) to the values shown: dbCreateTransaction.0 = 'verify' (3)	Pass/Fail (NTCIP 1201 -2.3.1)

Slide 59:

Learning Objective #4

Database Transactions

Sample Procedure - Verify

Step	Test Procedure	Results
401	GET the following object(s) : dbCreateTransaction.0	Pass/Fail (NTCIP 1201 -2.3.1)
402	IF dbCreateTransaction.0 equals 'done', GOTO Step 403; otherwise GOTO Step 401
403	GET the following object(s) : dbVerifyStatus.0 dbVerifyError.0	Pass/Fail (NTCIP 1201 -2.3.6, 2.3.7)
404	VERIFY that dbVerifyStatus.0 equals 'doneWithNoError'	Pass/Fail (NTCIP 1201 -2.3.1)
405	SET the following object(s) to the following values: dbCreateTransaction.0 = 'normal'	Pass/Fail (NTCIP 1201 -2.3.1)
406	GET the following object(s) : phaseStartup.1 phaseStartup.2 ...	Pass/Fail (NTCIP 1202 -2.2.2.20)
407	VERIFY that phaseStartup.1 equals Phase_Startup_1	Pass/Fail (NTCIP 1201 -2.3.1)
...	...	...

Slide 60:

Learning Objective #4

Database Transactions

Sample Procedure

• Procedures can be lengthy
• Manual performance is
  • Time consuming
  • Prone to error
• In practice, procedures need to be automated with scripts

Slide 61:

Learning Objective #4

Develop Sample Test Cases and Procedures Consistency Checks

• NTCIP 1202v02 Annex B defines 40 consistency checks
  • Ensures phase order is serviceable without conflict
  • Ensures phase references are valid in each context
  • All 40 are required for a conforming device
• Integral part of the design of the transaction feature
• Manufacturers are allowed to define additional checks
  • Agency needs to ensure that these do not affect interoperability with their management system
• Example on next slides

Slide 62:

Learning Objective #4

Consistency Checks

Sample Test Case

Test Case 6.3	Title	Download a database configuration with consistency errors
	Description	This test case verifies that the ASC properly rejects a database download that attempts to concurrently time two phases from the same ring. After initialization, it downloads several database parameters using the transaction mode and then ensures that the transaction verification results in the correct error. The DUT must start in the standard 8-phase, dual-ring configuration.
	Variables	<none>
	Pass/Fail Criteria	The DUT shall pass every verification step included within the Test Case to pass the Test Case.

Slide 63:

Learning Objective #4

Consistency Checks

Sample Procedure - Initialize and Start

Step	Test Procedure	Results
1	PRE-CONDITION: The ASC must be configured for standard 8-phase dual-ring operation. NOTE: This means that phaseConcurrency.1 and phaseConcurrency.2 will equal 0x0506, among other things.
2	GET the following object(s): dbCreateTransaction.0	Pass/Fail (NTCIP 1201 -2.3.1)
3	SET-UP: VERIFY that the value of dbCreateTransaction.0 is equal to 'normal' (1).
4	SET the following object(s) to the values shown: dbCreateTransaction.0 = 'transaction' (2)	Pass/Fail (NTCIP 1201 -2.3.1)
5	SET the following object(s) to the values shown: phaseConcurrency.1 = 0x020506 phaseConcurrency.2 = 0x010506	Pass/Fail (NTCIP 1201 -2.3.1; NTCIP 1202 -2.2.2.23)
6	GET the following object(s) : phaseConcurrency.1 phaseConcurrency.2	Pass/Fail (NTCIP 1202 -2.2.2.23)
7	VERIFY that phaseConcurrency.1 equals 0x0506	Pass/Fail (NTCIP 1201 -2.3.1)

Slide 64:

Learning Objective #4

Consistency Checks

Sample Procedure - Download

Step	Test Procedure	Results
8	VERIFY that phaseConcurrency.2 equals 0x0506	Pass/Fail (NTCIP 1201 -2.3.1)
9	SET the following object(s) to the values shown: dbCreateTransaction.0 = 'verify' (3)	Pass/Fail (NTCIP 1201 -2.3.1)
10	GET the following object(s) : dbCreateTransaction.0	Pass/Fail (NTCIP 1201 -2.3.1)
11	IF dbCreateTransaction.0 equals 'done', GOTO Step 12; otherwise GOTO Step 10
12	GET the following object(s) : dbVerifyStatus.0 dbVerifyError.0	Pass/Fail (NTCIP 1201 -2.3.6, 2.3.7)
13	VERIFY that dbVerifyStatus.0 equals 'doneWithError'	Pass/Fail (NTCIP 1202 -B.1)
14	VERIFY that dbVerifyError.0 equals "PHASE 01 CONCURRENCY FAULT" or "PHASE 02 CONCURRENCY FAULT"	Pass/Fail (NTCIP 1202 - B.1)

Slide 65:

Learning Objective #4

Consistency Checks

Sample Procedure - Verify

Step	Test Procedure	Results
15	SET the following object(s) to the following values: dbCreateTransaction.0 = 'normal'	Pass/Fail (NTCIP 1201 -2.3.1)
16	GET the following object(s) : phaseConcurrency.1 phaseConcurrency.2	Pass/Fail (NTCIP 1202 -2.2.2.23)
17	VERIFY that phaseConcurrency.1 equals 0x0506	Pass/Fail (NTCIP 1201 -2.3.1)
18	VERIFY that phaseConcurrency.2 equals 0x0506	Pass/Fail (NTCIP 1201 -2.3.1)

Slide 66:

Learning Objective #4

Develop Sample Test Cases and Procedures STMP

• Designed for low-bandwidth environment
• Can reduce overhead by 80-95%
  • SNMP has ~25 bytes of overhead for each message
  • SNMP has ~20 bytes of overhead for each object
  • STMP eliminates virtually all of this overhead
  • STMP eliminates the echo of values
  • A request for a single one-byte object
• Requires a complex set-up procedure
  • Adds one-time overhead
• Requires an additional level of processing

Slide 67:

Learning Objective #4

STMP

Sample User Need

• Agencies need to be able to monitor signal operations in real-time over low-speed communication networks.

Slide 68:

Learning Objective #4

STMP

Sample Requirement

• The ASC shall allow a TMS to configure up to 13* user-defined, bandwidth efficient messages for monitoring and control operations.

* The limitation of 13 messages is derived from NTCIP 1103 Clause 5.1.1.3.

Slide 69:

Learning Objective #4

STMP

Sample Dialog

(Extended Text Description: A slide showing a sample STMP configuration dialog. It is a UML sequence diagram between a "TMS" and an "ASC". It starts with the "TMS" sending a "Set(dynObjConfigStatus.x = invalid)" message and the "ASC" responding with a response message. The "TMS" then sends a "Set(dynObjConfigStatus.x = underCreation)" message and the "ASC" responds with a response message. The "TMS" then sends a "Set(dynObjConfigOwner.x, dynObjVariable.x.1, …, dynObjVariable.x.n)" message and the "ASC" responds with a response message. Finally, the "TMS" sends a "Set(dynObjConfigStatus.x = valid)" message and the "ASC" responds with a response message.)

From NTCIP 1103v02 Page 37

Slide 70:

Learning Objective #4

STMP

Sample Test Cases

• Configure a dynamic object
• Get a dynamic object
• Set a dynamic object
• Configure a dynamic object with incorrect data
• Attempt to configure a dynamic object using the wrong process
• Use a dynamic object as it is being modified

Testing should include dynamic objects that your system plans to use plus additional dynamic objects

Slide 71:

Learning Objective #4

STMP

Sample Test Case

Test Case 7.1	Title	Configure a dynamic object
	Description	This test case verifies that the ASC allows a user to configure a dynamic object. After initialization, it configures the dynamic object and verifies that the settings were saved. This test changes the definition of a dynamic object.
	Variables	Dynamic_Object_Number, Dynamic_Object_Owner, Object_1, Object_2, Object_3
	Pass/Fail Criteria	The DUT shall pass every verification step included within the Test Case to pass the Test Case.

Slide 72:

Learning Objective #4

STMP

Sample Procedure - Initialize and Start

Step	Test Procedure	Results
1	CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Dynamic_Object_Number
2	CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Dynamic_Object_Owner
3	CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Object_1
4	CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Object_2
5	CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Object_3
6	SET the following object(s): dynObjConfigStatus.Dynamic_Object_Number = 'invalid'	Pass/Fail (NTCIP 1103 -5.2.4)
7	SET the following object(s): dynObjConfigStatus.Dynamic_Object_Number = 'underCreation'	Pass/Fail (NTCIP 1103 -5.2.4)

Slide 73:

Learning Objective #4

STMP

Sample Procedure - Download

Step	Test Procedure	Results
8	SET the following object(s): Object_1.Dynamic_Object_Number.1 Object_2.Dynamic_Object_Number.2 Object_3.Dynamic_Object_Number.3	Pass/Fail (NTCIP 1103 -5.2.4)
9	SET the following object(s): dynObjConfigStatus.Dynamic_Object_Number = 'valid'	Pass/Fail (NTCIP 1103 -5.2.4)
10	GET the following object(s): dynObjConfigStatus.Dynamic_Object_Number	Pass/Fail (NTCIP 1103 -5.2.4)
11	VERIFY that dynObjConfigStatus.Dynamic_Object_Number equals 'valid'	Pass/Fail (NTCIP 1201 -2.3.1)

Slide 74:

Learning Objective #4

Develop Sample Test Cases and Procedures Load Testing

• Communications loading deals with the amount of data exchanged across a communications channel
• ASCs are often:
  • On multi-dropped channels
  • Require frequent communications
• Communications must not interfere with signal timing

Slide 75:

Learning Objective #4

Load Testing

Sample Test Case / Procedure

• This test case verifies that the ASC will continue to time phases correctly while processing a large number of complex messages.
• Procedure Set-up
  • Define a dynamic object containing status information
  • Define a dynamic object that can read time
  • Define a dynamic object that can be used to set time

Slide 76:

Learning Objective #4

Load Testing

Sample Test Procedure

• Procedure Details:
  • Requests first dynamic object from the device
  • Verify data is valid
  • Request second dynamic object from the device
  • Verify data is valid
  • Sends a SetNoReply request for time to another device
  • Repeat
  • During above steps, verify that the timing for each signal phase is appropriate for the actuations provided

Slide 77:

Learning Objective #4

Develop Sample Test Cases and Procedures Other Key Requirements to Consider

• Response Times
• Updating globalSetID
• Setting time over a network
• Security
• Data link control issues
• Protocol-specific objects (for status)
• Compound testing

Slide 78:

Learning Objective #4

Develop Sample Test Cases and Procedures Implications for Interoperability

• The standard provides a limited definition
  • No standardized user needs
  • No standardized requirements
  • No standardized dialogs
  • Various non-standard features used in industry
  • Manufacturer-specific consistency checks
  • No standardized test procedures
• These limitations require custom test procedures
  • No guarantee of off-the-shelf interoperability
  • Standard still offers benefits in reducing customization

Slide 79:

Slide 80:

Learning Objective #4

Which of the following statements is true?

1. The transaction mode must be used for all data downloads
2. A manufacturer may not impose its own consistency checks
3. STMP testing only needs to worry about "your" dynamic objects
4. There is no guarantee of off-the-shelf interoperability

Slide 81:

Learning Objective #4

Review of Answers

a) Transaction mode must be used for all downloads
Incorrect; transaction mode is only required for objects designated as "P2 "parameters.

b) Device may not impose additional consistency checks
Incorrect; a manufacturer may define additional checks and agencies need to be aware of these.

c) Only test your dynamic objects
Incorrect; testing should ensure that dynamic objects will work with any future system upgrade.

d) No guarantee of off-the-shelf interoperability
Correct! Your specifications can impact whether off-the-shelf devices are interoperable.

Slide 82:

Summary of Learning Objective #4

Develop Sample Test Case Specifications and Procedures for NTCIP 1202

• Reviewed how to test:
  • Database transactions
  • Consistency checks
  • STMP
  • Load testing
• Discussed where to find testing information on other key requirements

Slide 83:

Learning Objective #5 — Understand Testing Results for NTCIP 1202

• Explain a sample test summary
• Investigate failures with test incident reports
• Interpret what a failure means
• Inspect test logs to evaluate test results
• Appreciate need to repeat tests

Slide 84:

Learning Objective #5

Understand Testing Results

Sample Test Summary

• Identifier
• Summary
• Variances
• Comprehensiveness
• Summary of results
• Evaluation
• Summary of activities
• Approvals

See Participant Student Supplement for example

Slide 85:

Learning Objective #5

Understand Testing Results

Investigate Failures with Test Incident Reports

• Test summary only indicates pass or fail for a test
• Test incident report contains details
  • Identifier
  • Summary
  • Incident description
  • Impact
• Test summary references every test incident report

See example in Participant Student Supplement

Slide 86:

Learning Objective #5

Understand Testing Results

Investigate Failures with Test Incident Reports

Incident Description

• Inputs
• Expected results
• Actual results
• Other anomalies
• Date and time
• Procedure step
• Attempts to repeat
• Any other observations

Slide 87:

Learning Objective #5

Understand Test Results

Investigate Failures with Test Incident Reports

Incident Description

• Failure may be
  • Failure of a defined verification step
  • Any non-compliant anomaly noticed

Slide 88:

Learning Objective #5

Understand Testing Results

Investigate Failures with Test Incident Reports

Incident Description

• Should provide an assessment of cause
  • Implementation failure
  • User error in conducting test
  • Test procedure problem
  • Specification problem
  • Standard problem

Slide 89:

Learning Objective #5

Understand Testing Results

Interpret What a Failure Means

Incident Impact

• Traceability should help identify
  • Step that failed should indicate requirement
  • Requirement traces to user need(s)
• May be secondary impacts due to inter-relationships
• Impacts may include unrelated device operation

Slide 90:

Learning Objective #5

Understand Testing Results

Interpret What a Failure Means

Incident Description and Test Log

• Description should reference details
  • Developer can investigate exact details
  • Should reveal how to reproduce conditions
  • Assists in diagnosing the problem

Slide 91:

Learning Objective #5

Understand Testing Results

Inspect Test Logs

Description	Notes	Time	Result
Step 1: Time_Offset = 33		18:25:08	N/A
Step 2: GET globalTime.0		18:25:08	Passed
Step 3: RECORD globalTime.0 = 123456789		18:25:08	N/A
Step 4: SET globalTime.0 = 123456822		18:25:08	Passed
Step 5: Delay for 15 seconds		18:25:08	N/A
Step 6: Get globalTime.0		18:25:23	Passed
Step 7: Verify globalTime.0 (123456836) ~= 123456837		18:25:24	Passed

Actual test file will often provide more details about each step.

Slide 92:

Learning Objective #5

Understand Testing Results

Inspect Test Logs

Frame	ID	Type	Community	Error	Object 1	Timestamp
1	1	Get	public	No Error	globalTime.0	18:25:08.731
2	1	Response	public	No Error	globalTime.0	18:25:08.785
3	2	Set	public	No Error	globalTime.0	18:25:08.801
4	2	Response	public	No Error	globalTime.0	18:25:08.894
5	3	Get	public	No Error	globalTime.0	18:25:08.902
6	3	Response	public	No Error	globalTime.0	18:25:08.999

Actual test file will often provide more details about each step.

Slide 93:

Learning Objective #5

Understand Testing Results

Appreciate need to repeat tests

• Testing is intended to identify failures
  • Even mainstream software has frequent updates
  • Proper testing goes beyond "does it work"
• Identified failures need to be corrected
• Final product needs pass all tests
  • Often requires multiple rounds of testing

Slide 94:

Learning Objective #5

Understand Testing Results

Appreciate need to repeat tests

• Testing requires
  • Budget
  • Schedule time
• Implications of multiple rounds of testing
  • Who pays?
  • Penalties for failures
  • Penalties for schedule delays
  • Schedule constraints
  • Is there a maximum number of rounds?
• Proper planning will minimize close-out problems

Slide 95:

Slide 96:

Learning Objective #5

Which document(s) discuss potential impacts of testing failures?

1. Test summary
2. Test incident report
3. Test log
4. Test summary and test incident report

Slide 97:

Learning Objective #5

Review of Answers

a) Test Summary
Incorrect; incident reports contain an impact clause.

b) Test Incident Report
Incorrect; the test summary contains an evaluation.

c) Test Log
Incorrect; the log does not contain this information.

d) Test Summary and Test Incident Report
Correct! The Test Summary includes an Evaluation clause providing an overview and each Incident Report provides a detailed Impact clause.

Slide 98:

Summary of Learning Objective #5

Understand Testing Results for NTCIP 1202

• Test summary
• Test incident reports
• Test log
• Understand what a failure means
• Importance of planning for multiple rounds of testing

Slide 99:

What We Have Learned

1. Testing ASCs is important due to their safety- related issues.
2. Every ASC requirement should be tested at least one time in at least one test phase using at least one method and by at least one party.
3. In NTCIP, the test design specification, test case specification, and test procedure are typically defined in a combined test procedure table.
4. Due to various complications there is no guarantee of off-the-shelf interoperability for ASCs.
5. The Test Summary and Test Incident Reports reveal the impact of failures on a project.

Slide 100:

Resources

• NTCIP 1202:2005 v02.19
  • Object Definitions for Actuated Traffic Signal Controller (ASC) Units - version 02
  • www.ntcip.org
• NTCIP 9001:2009 v04
  • The NTCIP Guide
  • www.ntcip.org
• IEEE 829-2008
  • IEEE Standard for Software Test Documentation
  • www.ieee.org

Slide 101: