CSA scores would be set using an approach that pulls in a wider stream of data such as driver compensation, vehicle miles traveled by state, and other factors under improvements recommended by a congressionally mandated panel charged with reviewing the system.

The Safety Measurement System is used to identify commercial motor vehicle carriers at high risk for future crashes. It's the heart of the Federal Motor Carrier Safety Administration's Compliance, Safety, Accountability enforcement regime, known as CSA. After numerous criticisms of the methodology from the industry, Congress called for the review of SMS as part of the Fixing America’s Surface Transportation (FAST) Act of 2015.

The resulting study by the National Academies of Sciences, Engineering, and Medicine recommended replacing the current system over the next two years with a method using "item response theory," also known as latent trait theory.

FMCSA should develop a “more statistically principled approach” for the task, it said, based on an item response theory (IRT) model. Also known as latent trait theory, it’s an approach that has been used for policy decisions in other areas such as hospital rankings, according to the panel.

SMS Data Shortcomings

FMCSA uses information collected primarily during roadside inspections to identify motor carriers that are operating unsafely and therefore theoretically at higher risk for future crashes. These carriers are subject to interventions from FMCSA, such as warning letters and investigation.

Since it rolled out in 2010, the report noted that the system has been criticized for, among other things:

• Using highly variable assessments

• Not accounting for crashes where the motor carrier is not at fault

• Including carriers that have very different tasks in the same peer groups

• Using measures that are sensitive to effects from one or more individual states

• Using measures that are not predictive of a carrier’s future crash frequency

• Using measures that are not reflective of a carrier’s efforts to improve its safety performance over time.

The National Academies panel reported, “We have found, for the most part, that the current SMS implementation is defensible as being fair and not overtly biased against various types of carriers, to the extent that data on MCMIS can be used for this purpose.

“However, we believe some features of SMS implementation can be improved upon, and some of the details of the implementation are ad hoc and not fully supported by empirical studies. Many of these details of implementation would be easily addressed if the algorithm currently used were replaced by a statistical model that is natural to this sort of discrimination problem. “

The report identified a number of data quality issues.

It recommended that FMCSA should continue to collaborate with states and other agencies to improve the collection of data on vehicle miles traveled and on crashes, data which are often missing and of unsatisfactory quality. Including vehicle miles traveled data by state and month will enable SMS to account for varied environments where carriers travel – for example, in icy winter weather in the North.

In addition, there is information available in police narratives not represented in the data used that could be helpful in understanding the contributing factors in a crash.

The report also says FMCSA should research ways of collecting data on carrier characteristics – including driver turnover rates, type of cargo, and method and level of compensation. For example, compensation levels are relevant, says the panel, contenting that "it is known that drivers who are better-compensated, and those not compensated based on miles traveled, have fewer crashes."

This additional data collection would require greater collaboration between FMCSA and the states to standardize the effort and to protect carrier-specific information, it noted.

The committee said it was unable to recommend whether SMS percentile ranks should be made public, because it would require a formal evaluation to understand the consequences of public knowledge of the information. SMS data was pulled from public view in December 2015 after the FAST Act prohibited the public display of property carriers’ relative percentiles. At the same time, the agency removed the raw CSA data from public view to allow time to revise its SMS website, returning only that raw data to the website in March 2016.

Industry: What We've Been Saying All Along

Trucking groups say the study confirmed much of what they’ve been saying all along about the shortcomings of the CSA program.

“This report has confirmed much of what we have said about the program for some time," said American Trucking Associations President and CEO Chris Spear. “The program, while a valuable enforcement tool, has significant shortcomings that must be addressed, and we look forward to working with FMCSA to strengthen the program.”

Specifically, ATA noted the NAS study validated the trucking industry’s concerns about the inclusion of certain types of violations in the CSA system, that geographic enforcement disparities can have a significant impact on carriers’ scores, and that the collection and use of clean inspections is critical to the accuracy of the program.

“We strongly agree with the National Academies’ conclusion that CSA should be based less on the ‘subject-matter expertise’ of enforcement and more on the empirically validated data,” said ATA Director of Safety Policy Sean Garney. “We also see great potential in the Academies’ recommendation that FMCSA overhaul the current CSA methodology in favor of a new, more adaptive, data-centric model with the potential to address serious flaws in the system.

“To maximize CSA’s potential, there is an urgent need to address issues regarding data sufficiency and accuracy – specifically when looking at crash and carrier exposure data."

Similarly, the Truckload Carriers Association supports the notion of CSA — in theory. The study, however, “reaffirms what our industry has been saying all along, that there are inconsistencies with the Safety Measurement System,” said TCA Vice President of Government Affairs David Heller. “Safety-minded, forward-thinking carriers like the members of TCA should be able to stand behind their safety accomplishments, and being judged on those achievements, in an equitable manner, should be the basis of what CSA is all about.”

The Owner-Operator Independent Drivers Association said it is still reviewing the report, "but in general we hope the agency will take the suggestions seriously, as it appears the panel agrees with many of our stated concerns about the flawed data used to evaluate the safety of carriers."

What Will Happen to Safety Fitness Determination?

In March of this year, FMCSA announced it was scrapping a proposed rule that would have used the SMS data for issuing safety-fitness determinations for motor carriers. A large coalition of trucking industry groups wrote to Secretary of Transportation Elaine Chao to argue that the safety-fitness determination proposal should take a back seat to FMCSA fixing its CSA program.

In announcing its withdrawal of that proposal, FMCSA said that if it “determines changes to the safety-fitness determination process are still necessary and advisable in the future, a new rulemaking would be initiated that will incorporate any appropriate recommendations from the National Academies of Science and the comments received through this rulemaking.”

Updated 3:50 p.m. EDT 6/27/17 to add ATA and TCA comments

Updated 3 p.m. EDT 6/28/2017 to add OOIDA comments