ITS Evaluation

Research Progress & Insights

ITS Evaluation is designed to determine the effectiveness and benefits of deployed ITS and the value of ITS investments. The vision for the ITS Evaluation program is to ensure progress toward the vision of integrated ITS, achieving ITS deployment goals, as well as understanding the value, effectiveness, and impact of National ITS Program activities and allow for the program’s continual refinement.

Program Management Plan

The objective of the ITS evaluation program is to determine the effectiveness and benefits of deployed ITSand the value of ITS program investments. Evaluations are critical to ensuring progress toward the vision of integrated intelligent transportation systems and achieving ITS deployment goals. Evaluations are also critical to an understanding of the value, effectiveness, and impacts of the ITS Program activities and to allow for the continual refinement of the ITS Program’s strategy. To carry out these efforts, the ITS Evaluation Program has activities in six tracks:

• Track 1: ITS Research Evaluation: Efforts in this track seek to establish a common point of reference for evaluation and comparing different research activities. Oversight is provided to ensure that methodologies are aligned with Federal government guidelines.
• Track 2: ITS Deployment Tracking Surveys: This is a continuing effort to track and analyze results of surveys provided every three years to deploying agencies. These efforts have established a one-of-a-kind database that supports important longitudinal analysis to gather insights into direct and indirect effects of decisions, incentives, and other types of support.
• Track 3: ITS Deployment Evaluation: This effort uses the databases developed with the ITS survey results to perform analysis for the ITS Program and modal partners. The results assist the ITS Program in making targeted program investments that more effectively support deployment.
• Track 4: ITS Program Evaluation: Activities in this track are focused on providing evaluations for major ITSresearch initiatives such as the Integrated Corridor Management or the Urban Partnership Agreements. Current efforts including developing a plan for evaluation of the Connected Vehicle initiatives, including coordination with the Safety Pilot Model Deployment test plans.
• Track 5: Knowledge Management: The wealth of insights and results from surveys and evaluations forms a broad and deep knowledge base to support current and future ITS deployments. Efforts in this track turn these resources into databases and other, easily accessible materials that are used in decision making, training, and providing assistance.
• Track 6: Knowledge Transfer: Efforts in this track seek to link the knowledge management resources to training and decision making.

Track 1: ITS Research Evaluation

• In 2010, the ITS Evaluation program made a significant update to the existing ITS Evaluation Guidelines, a legislative requirement. The revisions incorporate the Net Present Value (NPV) calculation of research, applying principles of the Office of Management and Budget (OMB) Circular A-94, bringing the ITS Program into conformity with government-wide practices for evaluating research.

• The guidelines were also revised to ensure that contractors involved in evaluating deployment successes or lessons learned are not also engaged in deployment teams.

• The shift to incorporate the OMB principles results in a common point of reference for comparing different research activities by their NPV. As a result, ITS research initiatives, and the potential for high-risk/ high-reward results, can be viewed on equal footing with other research initiatives across the Department.
• The change in contractor qualifications eliminates a potential conflict of interest.

• Implement a framework that describes the value derived from research while in progress.

Track 2: ITS Deployment Tracking Surveys

• The ITS Evaluation program continued its work with deployment tracking surveys with analysis of the results from the 2010 survey being presented throughout the 2011 year. Both the website and summary reports were released in 2011. Analysis and other materials include:
  • Deployment Tracking Survey Results:The full 2010 survey database is available for download for other types of analysis.
  • Deployment of ITS: A Summary of the 2010 National Survey Results, August 2011.
  • Presentations from the 2011 ITS World Congress.

• The ITS Evaluation Program has collected over 1years of deployment data. It is the only database of its kind that is able to view, longitudinally, the results from different policies and programs designed to support effective ITS deployment.
• Rigorous analysis in 2011 offered the first opportunity to use historical data to understand how ITS implementation occurs. The results will be used to inform and support investment decisions for connected vehicle implementation.

• Design the 2013 Deployment Tracking Surveys.
• Provide the 2010 survey database on https://www.data.gov.

Track 3: ITS Deployment Evaluation

• The US DOT published ITS Technology Adoption and Observed Market Trends from ITS Deployment Tracking, which compiled statistical evidence on what influences ITS adoption and deployment.
• A second report, An Analysis of the Factors Influencing ITS Technology Adoption and Deployment, looked at how decisions are made through historical adoption patterns, and examined the maturity of different ITS markets.
  • Available at: https://www.itskrs.its.dot.gov/deployment.
  • ITS Strategic Research Plan, 2010–2014

• With the 2011 ITS survey results, the program was able to perform analysis on the patterns of ITS adoption to understand how external factors, and in particular, those that are policy related, may affect ITS diffusion.
• Results from this analysis provide insight into intervention levers that could be used to positively affect adoption and deployment. This information can be used to inform future ITS strategic planning and decision making. Important insights include:
  • Since the late 1990s, there has been a relatively steady increase in the adoption and deployment of ITS technologies. The data reveal that most adopters are “imitators” — those who make the decisions to deploy ITS after the technologies have been proven successful at other sites.
  • Data and analysis proves that earmarks are not effective at encouraging ITS technology adoption. Instead, the results observe that budget, regional architecture involvement, and the effect of peer behavior are among the key factors affecting ITS adoption and implementation.
  • Analysis also revealed that developing or updating a regional architecture is a cost-effective way to increase adoption of ITS technologies.

• The report represented the final step in this track. A new project in 2012 will attempt to gather more detailed empirical data on technology diffusion.

Track 4: ITS Program Evaluation

• The SafeTrip-21 evaluation was completed in 2011.
• A number of upcoming evaluations are in different stages of moving forward:
  • The Urban Partnership Agreement (UPA) Evaluation Plans and Test Plans were submitted for Seattle, Minneapolis-Saint Paul, Atlanta, and Los Angeles.
  • A UPA evaluation was completed for Miami.
  • ICM Evaluation Plans were submitted for Dallas and San Diego and an overall ICM evaluation framework is being developed.

• To gain a more accurate analysis of the impact of deployment and to help isolate the various factors affecting deployment outcomes, the program stresses analyses of factors beyond transportation, including employment and economic indicators, to accurately state the impact of ITS research deployments with precision.
• Test plans and evaluation plans are looking at the effectiveness of strategies consisting of combinations of tolling, transit, telecommuting/transportation demand management, and technology.
• The Miami UPA results reveal that conversion from High Occupancy Vehicle (HOV) to High Occupancy Tolling (HOT) lanes improved travel times and on-time performance for transit express bus service. The HOT lanes also encouraged the shift from driving in cars to taking transit.
• Total person throughput on all modes increased 42 percent for AM/PM peak periods.

• Complete UPA and ICM evaluations.
• Advise on the structure for evaluation of the Safety Pilot.
• Progress Update 2012

Track 5: Knowledge Management

• The ITS Evaluation program continued to update and maintain three knowledge management resources that are critical to decision making about deploying ITS. Over 10,000 monthly users refer to these resources for information. Significant new resources have been added since 2008. The databases include:
  • ITS Benefits Database
  • ITS Costs Database
  • ITS Lessons Learned Database
• The ITS Evaluation program revised the web presence of the databases by creating a new website with more graphics, ensuring the look is more consistent with the ITS website, and providing a new geographic component (a beta version of a mapping application helps to locate sites with their benefits information).
• The program has added a new online feature, “Rate this Benefit,” that allows end users to express the value of the information received. End users now have a simplified mechanism to supply self-developed evaluation reports.

• The knowledge management resources are easier to use. Users have been pleased, particularly with the mechanism to weigh in on the ITS evaluation results.
• Users respond to the website with their own analysis of the evaluation results or they request more information.

• Keep databases populated, seek out reportsevaluations include those benefits costs, and lessons learned data in our database.

Track 6: Knowledge Transfer

• Collaborated on linking knowledge resources into PCB training materials.

• Stakeholder engagement has improved website usage.
• Collaborating with ITS PCB program on opportunities from deployment tracking analysis and evaluations.
• Launched Longitudinal Study to analyze the factors that motivate deployment community.

• Developing evaluation methodologies that describe value of research in progress and that also considers impact of stakeholder development on deployment stability.
• The databases can be accessed at:
  • ITS Benefits Database at: https://www.itskrs.its.dot.gov/benefits.
  • ITS Costs Database at: https://www.itskrs.its.dot.gov/costs.
  • ITS Lessons Learned Database at: https://www.itskrs.its.dot.gov/lessons.