Module 5 - T101
T101: Introduction to ITS Standards Testing
HTML of the PowerPoint Presentation
(Note: This document has been converted from a PowerPoint presentation to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)
Slide 1:
(Extended Text Description: Slide 1: Welcome - Graphic image of introductory slide. A large dark blue rectangle with a wide, light grid pattern at the top half and bands of dark and lighter blue bands below. There is a white square ITS logo box with words “Standards ITS Training” in green and blue on the middle left side. The word “Welcome” in white is to the right of the logo. Under the logo box are the words “RITA Intelligent Transportation Systems Joint Program Office.”)
Slide 2:
Welcome
Shelley Row, P.E., PTOE
Director
ITS Joint Program Office
(Extended Text Description: Slide 2: Screen capture snapshot of RITA website - for illustration only. Below this image is a link to the current website: https://www.its.dot.gov/pcb - this screen capture snapshot shows an example from the RITA website from June 3, 2011. At the top of the page it shows the RITA logo with the text Research and Innovative Technology Administration - Intelligent Transportation Systems. Below the main site banner, it shows the main navigation menu with the following items: About RITA, Communities of Interest, Contact Us, Press Room, RITA Offices, Site Map, and a Search button. Below the main navigation menu, it shows a sub-navigation menu with the following items: About Us, T3 Webinars, ITS Peer-to-Peer, Resources, Local ITS PCB and Testimonials. Beneath the sub-navigation menu, the page is sub-titled "ITS Professional Capacity Building Program" and is divided into sub-sections such as "Welcome to ITS Professional Building", "News", "ITS Technical Assistance" and "Scheduled T3 Webinars". Again, this image serves for illustration only. The current website link is: https://www.its.dot.gov/pcb)
(Note: There is additional text attached to this slide that includes the following introductory information from Shelley Row):
"ITS Standards can make your life easier. Your procurements will go more smoothly and you’ll encourage competition, but only if you know how to write them into your specifications and test them. This module is one in a series that covers practical applications for acquiring and testing standards-based ITS systems.
I am Shelley Row the director of the ITS Joint Program Office for USDOT and I want to welcome you to our newly redesigned ITS standards training program of which this module is a part. We are pleased to be working with our partner, the Institute of Transportation Engineers, to deliver this new approach to training that combines web based modules with instructor interaction to bring the latest in ITS learning to busy professionals like you.
This combined approach allows interested professionals to schedule training at your convenience, without the need to travel. After you complete this training, we hope that you will tell colleagues and customers about the latest ITS standards and encourage them to take advantage of the archived version of the webinars.
ITS Standards training is one of the first offerings of our updated Professional Capacity Training Program. Through the PCB program we prepare professionals to adopt proven and emerging ITS technologies that will make surface transportation safer, smarter and greener which improves livability for us all. You can find information on additional modules and training programs on our web site ITS PCB Home.
Please help us make even more improvements to our training modules through the evaluation process. We look forward to hearing your comments. Thank you for participating and we hope you find this module helpful."
Slide 3:
T101: Introduction to ITS Standards Testing
Slide 4:
Target Audience
- Engineering staff
- Operational staff
- Maintenance staff
- Decision makers (in feedback loop) in a procurement capacity (acquisitions department)
Slide 5:
Instructor
Gary B. Thomas, P.E., Ph.D.
Research Engineer
Texas Transportation Institute
College Station, TX, USA
Slide 6:
Recommended Prerequisites
- A100
- A200
- A300 level courses as appropriate
Slide 7:
Curriculum Path (Testing)
(Extended text description. Curriculum Path (Testing). A chart showing the curriculum path for the testing topics. A linear box chart (boxes are in blue or highlighted with rounded corners and embossed effect on side edges) starting with "T101 – Introduction to ITS Standards Testing" (highlighted box) with an arrow leading to "T201 – How to Write a Test Plan" (blue box) with an arrow leading to "T202 – Overview of Test Design Specifications, Test Cases and Test Procedures" (blue box) with a divided arrow leading up to "T311 – Applying Your Test Plan to the NTCIP 1203 V03 DMS Standard" (higher blue box) and lower arrow leading down to "T313 – Applying Your Test Plan to the NTCIP 1204 V03 ESS Standard" (lower blue box).)
Slide 8:
Learning Objectives
- Explain the need for and benefits of testing
- Describe how ITS standards testing fits into the overall scope of a systems test and a systems life cycle
- Discuss how to test an implementation for conformance to standards
- Distinguish the difference between standard conformance and project compliance
Slide 9:
What is Testing?
Two types:
1. Standards testing
- Provides information to users on the reliability, interoperability, functionality, and performance of systems using the standard
2. System's conformance to documented test plan based upon an ITS standard
- The topic of this Webinar
Slide 10:
Slide 11:
Learning Objective #1
Benefits of Testing
- What do you see as possible benefits of testing?
- Use the chat pod to answer
Slide 12:
Learning Objective #1
Benefits of Testing
- Verify that requirements are fulfilled
- Reduce the risk of misinterpretation between agency and manufacturers
- Reduce the risk of financial mismanagement
- Reduce the risk of perceived lack of oversight
- Ensure interoperability to allow system expansion
Slide 13:
Learning Objective #1
Who Does the Testing?
- Agency should identify party
-
Typical testing parties:
- Agency (includes agency's contractor)
- Manufacturer
- Independent laboratory
-
All parties should have a clear understanding of their testing responsibilities
- Clear description of the consequences of test failure(s)
Slide 14:
Terminology
- Conformance: A condition that exists when an item meets all of the mandatory and selected optional requirements as defined by a standard.
- Compliance: A condition that exists when an item meets all of the requirements of an agency specification.
Slide 15:
Terminology
-
Validation: Making sure a system, when placed in operation, will support agency needs
- In other words: Have we built the right thing?
-
Verification: Making sure a design complies with requirements and that the systems (as proposed and delivered) comply with both design and requirements
- In other words: Have we built the thing right?
- Traceability: A tool to help determine if the agency's requirements are fulfilled by the design and that implementation was done correctly
Slide 16:
Learning Objective #2
Testing and the Systems Engineering Process
(Extended Text Description: Testing and the Systems Engineering Process. A graphic of the systems engineering process (SEP). The main graphic of the SEP is a V-shaped diagram in gradated blue with some additional horizontal extensions on the left and right side of the top of the V shape. Each section is separated with dark blue lines. There is a key at the lower right showing the blue separator lines, and designating them as “Decision Gate.” The left horizontal extension is labeled as “Lifecycle Processes” and include the sections “Regional Architecture” (separated by a white space) to the second section labeled “Needs Assessment,” then “Concept Selection,” (blue line) “Project Planning,” and “Systems Engineering Management Planning” (blue line). At this point the sections begin to descend the left side of the V with “Concept of Operations,” (blue line) “System Requirements,” (blue line) “High-level Design,” “Subsystem Requirements,” (blue line) “Detailed Design,” (blue line) and “Software Coding / Hardware Fabrication” (blue line) at the bottom juncture of the V shape. Underneath the bottom point of the V shape is a long thin horizontal arrow pointing to the right labeled “Time Line.” There is a long thin diagonal arrow pointing down along the left side of the V labeled “Decomposition and Definition.” From the bottom point of the V, the sections begin to ascend up the right side of the V with “Unit Testing,” (blue line) “Subsystem Integration,” (blue line) “Subsystem Verification,” (blue line) “System Integration,” (blue line) “System Verification,” (blue line) “Initial Deployment,” (blue line) “System Validation,” and (blue line) “Operations and Maintenance.” There is a long thin arrow pointing up along the right side of the V shaped labeled “Integration and Recomposition.” At this point the sections on the right “wing” of the V are labeled with “Changes and Upgrades” and (white space) “Retirement/Replacement.” Between the V shape there are a series of gray dashed arrows connecting the related sections on each left/right side of the V shape. The first arrow (top) is labeled “System Validation Plan” and connects “Concept of Operations” on the left and “System Validation” on the right. The second arrow is labeled “System Verification Plan (System Acceptance)” and connects “System Requirements” on the left and “System Verification and Deployment” on the right. The third arrow is labeled “Subsystem Verification Plan (Subsystem Acceptance)” and connects “High-Level Design” on the left and “Subsystem Verification” on the right. The last arrow (at the bottom) is labeled “Unit/Device Test Plan” and connects “Detailed Design” on the left and “Unit/Device Testing” on the right. The empty space in the middle of the V is highlighted in light blue. The right side V diagonal from "Unit Testing," through "Subsystem Verification Subsystem Integration," "System Verification System Integration," to "System Validation Initial Deployment." There is a large word "Traceability" at the top of the graphic with a curved arrow pointing downward to the highlighted space between the V shape. There is another large word "Testing" on the right side of the diagram with a curved arrow pointing into the sections of the right side of the V diagram.)
Slide 17:
Learning Objective #2
Testing and the Systems Engineering Process
Unit/Device Testing
- Focuses on comparing an implementation against the standards and specified options
- May be performed by inspecting the code to use "proven" software to send test messages to the device.
(Extended Text Description. Testing and Systems Engineering Process (SEP) V Diagram, middle portion. The image fades in in the left wing section beginning with “Project Planning,” (blue line representing a "Decision Gate" according to the key to the right of the V Diagram) into “Systems Engineering Management Planning” (blue line). At this point the sections begin to descend the left side of the V with “Concept of Operations,” (blue line) “System Requirements,” (blue line) “High-level Design,” “Subsystem Requirements,” (blue line) “Detailed Design,” (blue line) and “Software Coding / Hardware Fabrication” (blue line) at the bottom juncture of the V shape. There are horizontal arrows pointing from each of these left side sections to the corresponding sections on the right side. Underneath the bottom point of the V shape is a long thin horizontal arrow pointing to the right labeled “Life Cycle Time Line.” There is a long thin diagonal arrow pointing down along the left side of the V labeled “Decomposition and Definition.” From the bottom point of the V, the sections begin to ascend up the right side of the V with “Unit Testing,” (blue line) “Subsystem Integration,” (blue line) “Subsystem Verification,” (blue line) “System Integration,” (blue line) “System Verification,” (blue line) “Initial Deployment,” (blue line) “System Validation,” and (blue line) “Operations and Maintenance.” There is a long thin arrow pointing up along the right side of the V shaped labeled “Integration and Recomposition.” The sections on the right “wing” of the V begin to fade with “Changes and Upgrades.” The section "Unit Testing" on the right side and the corresponding middle space of the V Diagram is outlined in a thick red line and highlighted in light blue.)
Slide 18:
Learning Objective #2
Testing and the Systems Engineering Process
Subsystem Testing
- Consists of connecting two or more devices together and exchanging data
- Assumes devices and components have passed a designed unit test plan
- Assumes devices or subsystem components support same ITS operational and/or functional features.
(Extended Text Description. Testing and Systems Engineering Process (SEP) V Diagram, middle portion. The image fades in in the left wing section beginning with “Systems Engineering Management Planning" (blue line representing a "Decision Gate" according to the key to the right of the V Diagram). At this point the sections begin to descend the left side of the V with “Concept of Operations,” (blue line) “System Requirements,” (blue line) “High-level Design,” “Subsystem Requirements,” (blue line) “Detailed Design,” (blue line) and “Software Coding / Hardware Fabrication” (blue line) at the bottom juncture of the V shape. There are horizontal arrows pointing from each of these left side sections to the corresponding sections on the right side. Underneath the bottom point of the V shape is a long thin horizontal arrow pointing to the right labeled “Life Cycle Time Line.” There is a long thin diagonal arrow pointing down along the left side of the V labeled “Decomposition and Definition.” From the bottom point of the V, the sections begin to ascend up the right side of the V with “Unit Testing,” (blue line) “Subsystem Integration,” (blue line) “Subsystem Verification,” (blue line) “System Integration,” (blue line) “System Verification,” (blue line) “Initial Deployment,” (blue line) “System Validation,” and (blue line) “Operations and Maintenance.” There is a long thin diagonal arrow pointing up along the right side of the V labeled "Integration and Recomposition." The sections on the right “wing” of the V begin to fade with “Changes and Upgrades.” The section "Subsystem Verification Subsystem Integration" on the right side and the corresponding middle space of the V Diagram is outlined in a thick red line and highlighted in light blue.)
Slide 19:
Learning Objective #2
Testing and the Systems Engineering Process
System Testing
- Highest level of testing
- Performed after all lower level testing is successfully completed
- Performed in an operational environment
- Includes acceptance testing
(Extended Text Description. Testing and Systems Engineering Process (SEP) V Diagram, middle portion. The image fades in in the left wing section beginning with “Systems Engineering Management Planning" (blue line representing a "Decision Gate" according to the key to the right of the V Diagram). At this point the sections begin to descend the left side of the V with “Concept of Operations,” (blue line) “System Requirements,” (blue line) “High-level Design,” “Subsystem Requirements,” (blue line) “Detailed Design,” (blue line) and “Software Coding / Hardware Fabrication” (blue line) at the bottom juncture of the V shape. There are horizontal arrows pointing from each of these left side sections to the corresponding sections on the right side. Underneath the bottom point of the V shape is a long thin horizontal arrow pointing to the right labeled “Life Cycle Time Line.” There is a long thin diagonal arrow pointing down along the left side of the V labeled “Decomposition and Definition.” From the bottom point of the V, the sections begin to ascend up the right side of the V with “Unit Testing,” (blue line) “Subsystem Integration,” (blue line) “Subsystem Verification,” (blue line) “System Integration,” (blue line) “System Verification,” (blue line) “Initial Deployment,” (blue line) “System Validation,” and (blue line) “Operations and Maintenance.” There is a long thin diagonal arrow pointing up along the right side of the V labeled "Integration and Recomposition." The sections on the right “wing” of the V begin to fade with “Changes and Upgrades.” The section "System Verification System Integration" on the right side and the corresponding middle space of the V Diagram is outlined in a thick red line and highlighted in light blue.)
Slide 20:
Learning Objective #2
IEEE 829
- Standard that specifies the form of a set of documents for use in eight defined stages of software testing
-
Each stage may have its own document
- Specifies the format of these documents
- Does not stipulate whether they all must be produced
- Does not include any criteria regarding adequate content for these documents
- Software testing standard applicable to other types of testing
- A framework for testing within the life cycle
Slide 21:
Slide 22:
Learning Objective #3
What Should a Test Plan Cover?
- What do you think a test plan should cover?
- Use the chat pod to answer
Slide 23:
Learning Objective #3
Testing Considerations
- Maturity of device, software, interface, or system
- Number of units (if a device)
- Agency's ability to test
- Available expertise
- Significance of specification requirement
Slide 24:
Learning Objective #3
Test Plans
- Management planning document that covers the "who, what, when, and how"
- Defined in IEEE 829-1998
- Convey the scope, approach, resources, and schedule of testing activities
-
Identifies:
- Items and features to be tested
- Tasks to be performed
- Personnel responsible
- Associated risks
- A well-designed test plan feeds into the more detailed test procedures
Slide 25:
Learning Objective #3
Test Plan Benefits
- Provides framework and process to verify that the system meets user needs
- Improve stakeholder participation
- More adaptable, resilient systems
- Verify functionality and fewer defects
- Higher level of reuse from one project to the next
- Better documentation
Slide 26:
Learning Objective #3
Test Procedures
- Details regarding how to run each test
- Includes set-up preconditions and steps to be followed
- Allows for flexibility
-
Format for test procedures
- Included in NTCIP 8007 v.01
- Should reference agency specification requirements
Slide 27:
Learning Objective #3
Systems Engineering Testing Phases
- Prototype test and inspection
- Design approval test and inspection
- Factory acceptance test
- Incoming device test
- Site acceptance test
- Burn-in and observation test
Slide 28:
Learning Objective #3
Systems Engineering Testing Phases
Prototype Test and Inspection
- Purpose: Verify the electrical and mechanical design
- Number of units: One prototype
- Test Location: Test laboratory
Design Approval Test and Inspection
- Purpose: Verify the final design
- Number of units: Pre-production or small percentage of units
- Test Location: Laboratory
Slide 29:
Learning Objective #3
Systems Engineering Testing Phases
Factory Acceptance Test
- Purpose: Verify production units are identical to final design and production quality
- Number of units: Percentage of production units
- Test Location: Production factory
Incoming Device Test
- Purpose: Inspect for damage due to shipping/handling
- Number of units: All delivered units (including spares)
- Test Location: Agency
- Conformance testing begins
Slide 30:
Learning Objective #3
Systems Engineering Testing Phases
Site Acceptance Test
- Purpose: Full functionality of the entire system
- Number of units: All installed units
- Test Location: Final location for operation
- Conformance testing continues
Burn-in and Observation Test
- Purpose: Monitor proper operation of the installed unit
- Number of units: All installed units
- Test Location: Final location for operation
Slide 31:
Learning Objective #3
Methods of Testing
(Extended Text Description. Methods of Testing. A graph of five blue boxes with rounded corners. The center box is labeled "Testing Methods." A light blue bi-directional arrow points to a blue box directly above labeled "Inspection." Another bi-directional arrow points to a box on the left labeled "Formal Test." Another bi-directional arrow points to a box below marked "Demonstration." Another bi-directional arrow points to a box labeled "Analysis" on the right.)
Slide 32:
Learning Objective #3
Methods of Testing
Inspection
- Verification by physical and visual examination
- Reviewing descriptive documentation
- Comparing appropriate characteristics
-
Examples:
- Measuring cabinets sizes
- Matching paint color samples
- Observing printed circuit boards
Slide 33:
Learning Objective #3
Methods of Testing
Analysis
- Verification by evaluation or simulation using mathematical means
- Can by used to extrapolate past performance for scaled up deployment
-
Examples:
- Internal temperature gradients for dynamic message sign
- Review of power supply designs to verify temperature and voltage limitations
Slide 34:
Learning Objective #3
Methods of Testing
Demonstration
- Functional verification by observing an operation or exercise performed under specific condition
-
Examples:
- Accuracy of displays
- Comparison of system outputs with test cases
- System recovery from induced failures
Slide 35:
Learning Objective #3
Methods of Testing
Formal testing
- Verification that requirement is met by testing under controlled exercises using real or simulated stimulus
-
Examples:
- System performance
- System functionality
- Data distribution
Slide 36:
Learning Objective #4
Conformance
- Exists when an item fulfills all mandatory requirements defined by a standard and any selected optional portions of the standard
- Conformance testing is designed to verify that the device fulfills the mandatory and optional requirements of the applicable standard(s)
- Prior to testing, ensure that a set of well-defined requirements exists and can be verified
- Testing addresses all parts of the standards that are mandatory for the ITS device
- Conformance is tested against all mandatory and selected optional requirements of the standard
Slide 37:
Learning Objective #4
Compliance
- Exists when an item meets all requirements of an agency specification
- Compliance testing is designed to verify that the device fulfills the requirements of an agency's specification
- Creating a specification is necessary to ensure that all optional elements in the standard are required for your deployment
- Specifications also allow for the explicit removal of optional elements that do not apply
- Compliance is tested against the requirements developed as part of a project
Slide 38:
Slide 39:
Learning Objective #4
Why is Conformance Necessary?
- Why do you think conformance is necessary?
- Use the chat pod to answer.
Slide 40:
Learning Objective #4
The Need for Conformance
- Helps to build the right system correctly
- Able to validate using standardized testing
- Interoperability is a chief benefit
Slide 41:
Learning Objective #4
Return on Investment
- Testing reduces risk
-
Risk imposes a cost
- risk = cost of remediation x probability of worst case
-
Cost of rejecting devices after installation include:
- Replacement cost
- Legal fees
- Lost benefit during delay of implementing devices
Slide 42:
Review of Learning Objectives
Explain to decision makers the need for and benefits of standardized testing.
Describe how ITS standards testing fits into the overall scope of a systems test and a systems life cycle.
Discuss how to test an implementation for conformance to standards.
Distinguish the difference between standard conformance and project compliance.
Slide 43:
Student Supplement
- Excerpt from NTCIP 9001 v4.06: NTCIP Testing
- Excerpt from NTCIP 9012 v1.27: Test Documentation and Test Execution
Slide 44:
For More Information
RITA/ITS web site
ITE web site
ITS Architecture Implementation Program
NTCIP web site
Systems Engineering Guide for ITS
Slide 45:
Curriculum Path (Testing)
(Extended Text Description. Curriculum Path (Testing). A chart showing the curriculum path for the testing topics. A linear box chart with highlighted and blue boxes (rounded corners, embossed side edges) starting on the left with "T101 – Introduction to ITS Standards Testing" (blue box) with an arrow leading toward the right to "T101 – Introduction to ITS Standards Testing" (highlighted box) leading to the right to "T202 – Overview of Test Design Specifications, Test Cases, and Test Procedures" (blue box) with a divided arrow leading up to "T311 – Applying Your Test Plan to the NTCIP 1203 V03 DMS Standard" (blue box) and the other arrow leading down to "T313 – Applying Your Test Plan to the NTCIP 1204 V03 ESS Standard" (blue box).)
Slide 46:
