Lessons Learned

Lessons Learned during Interoperability Testing amongst the three Connected Vehicle Pilots

Through a collaborative effort, the CV Pilot sites (NYCDOT, THEA, and WYDOT) worked with the USDOT and USDOT’s support contractor to plan and conduct a limited Interoperability Test at FHWA’s Turner-Fairbank Highway Research Center (TFHRC) in McLean, VA in June 2018. In addition to testing interoperability among connected vehicle devices from the three sites across six device vendors, the testing resulted in valuable CV-relevant experience for the participants. Planning for the testing event was jointly led by the CV Pilot sites in coordination with TFHRC and USDOT staff. TFHRC and its Saxton Transportation Operations Laboratory (STOL) contractor provided support to the CV Pilot sites as well as the facility and supporting equipment for the testing. This brief highlights some lessons learned from the testing experience for the benefit of other CV developers and deployers.

Planning/Coordination Lessons

• Since the testing involved multiple sites, their vendors and support from USDOT and its contractors, coordination and planning were essential. In the months leading up to the actual test date, bi-weekly technical roundtable meetings were held to track progress toward critical elements needed for the test, with supplemental communications to address more detailed items. For example, coordination was needed with the STOL testbed team to ensure that devices were properly installed on the test vehicles and software/firmware on devices were updated to the correct versions well in advance (two weeks) of the test dates. Installation of equipment went relatively smoothly as a result of this planning effort conducted in advance.

• Both a formal test plan and test readiness review were developed to ensure that protocols and schedules were documented in advance. The sites found the test plan document to be generally thorough, clear and concise. Since the sites had only conducted limited cross-site testing, a full day was scheduled for setup and understanding the testbed layout, configuration updates and checkout and execution of 20-30 minute “dry runs” to allow the test drivers and other participants to understand the procedures for each scenario. A pre-meeting was held to make sure everyone understood their specific role in the procedure, whether as driver, flagger, or recorder, and included both an overview and Q&A discussion in case clarifications were needed. Real-time communication and coordination were enabled by walkie-talkies assigned to the observers in each vehicle and the test leads and recorders on the ground. Time allotted to each run was allocated conservatively at 10 minutes to minimize the possibility of falling behind schedule, but in actual practice the runs took less time. The test documentation and pre-testing review helped sites to be well prepared for the tests.

• All parties found the testing to be a beneficial experience, both from a technical perspective as well as interacting with the other roles and teams. The testing allowed the participants to test applications using devices and equipment that they generally lack access to and take advantage of other supporting resources at the STOL testbed such as a DSRC sniffer. It should be noted that the USDOT arranged for a support team to handle the bulk of the test plan development, thereby allowing the sites to review and suggest extensions, alternatives, and identify some details and issues based on a draft test plan. Initial efforts to have the sites develop the test plans proved to be challenging due to the other activities also going on within the projects. Thus having the USDOT’s support team work closely with the sites allowed all the sites to address the issues without concern for the impact on the documentation for the test procedure.

CV Technical Lessons

• From the sites’ perspective, the interoperability testing enabled first-time testing of how their devices worked with other sites including using credentials from the supporting Security Credential Management System (SCMS). All sites and the STOL team worked to enroll devices in the commercial “test” SCMS in advance of testing, and as a result the devices were able to use the credentials to sign messages being communicated with other devices including the Basic Safety Message (BSM). This was a significant achievement and also helped to increase the familiarity with the SCMS for both the sites and the STOL testbed.

• Data captured on vehicle devices (OBUs), including sent and received BSMs, were downloaded to external storage after each test run to ensure that test data was not lost due to storage capacity or other issues. Some sites brought vendor support to facilitate the offloading of data after each run at the test garage (starting/ending point) while the next run was taking place, so as not to delay the testing. At the end of each test day, teams were able to discuss any issues the drivers encountered and identify any retests needed.

• Another benefit of the testing was the ability to run informal impromptu testing between the sites outside of the formal test runs. Site teams, supported by the STOL testbed staff, were able to explore variations to better understand the device behavior and interactions across sites beyond what was defined in the test plan document.

• The CV Pilots Independent Evaluation team was also present during testing to observe first-hand the nature of the devices and the application functionality and interact with the site teams. On the last day, time was allotted for a focus group with all the participants to allow the Independent Evaluation team to gather subjective feedback while testing was fresh in participants’ minds.

• While the devices’ positioning capabilities were not the focus of this testing, V2V applications like Forward Collision Warning (FCW) were used to assess the reception and processing of BSMs. As a result, the devices’ GPS accuracy was sometimes a factor in whether an alert was reported, since the host vehicle needed to classify the remote vehicle in the same lane for FCW alerts. GPS coverage was good throughout the testing, with 8-11 satellites usually in view. Initial testing for some devices encountered issues with triggering alerts, but after updating lane width configurations in the devices, consistency was greatly improved. In addition, the NYCDOT team took the opportunity to conduct follow-up testing based on the observed behavior in the tests. This follow-up testing was able to demonstrate significantly improved positioning accuracy resulting from the chip vendor updating the positioning firmware. The findings from this testing helped to inform not only interoperability-related issues but also considerations for operational performance.

• Another factor in triggering of application alerts was the variability of test driver behavior. Tests that relied upon applications such as IMA and EEBL were more challenging to execute since they required hard braking or more precise timing. The teams felt that repeatability could be improved by adjusting application configuration parameters or using more specific cues, cones or guides for the test drivers. The issue of repeatability and variability in driver behavior is likely to remain a challenge in both application testing as well as performing “tuning” adjustments aimed at different driving conditions.

For more information and findings from the CV Pilots Interoperability Testing, please visit the USDOT CV Pilots website where the test report will be available.