Commentary: CSA Gets Ready for Another Take

Trucking’s got a new TLA (three-letter acronym) to add to the multitudes we already have: IRT. It marks a new approach to the Federal Motor Carrier Safety Administration’s controversial CSA program, and so far it looks like it will address some big complaints about it – but will fall short in one crucial area.

FMCSA had a good reason for implementing CSA, which stands for Compliance, Safety, Accountability: It wanted a better way to identify carriers that warranted a compliance review.

CSA aimed to use inspection results and other data to help identify unsafe companies for “interventions,” which could be a warning letter, compliance review, or other action. Eventually, the system was supposed to be used for issuing safety-fitness determinations for motor carriers.

But from the beginning, CSA was fraught with problems, including the fact that brokers, insurers, attorneys, and others were using the publicly available scores in ways DOT never intended. Eventually, trucking convinced Congress, in 2015, to order FMCSA to shut down public access to that data while it revamped the program. And Congress had a very specific way it wanted the agency to address it – by getting actual scientists involved from the National Academies of Science, Engineering and Medicine.

What the NAS panel set up to study CSA came back with was a recommendation that FMCSA develop a “more statistically principled approach” for the task, based on an item response theory (IRT) model. IRT is widely used in the education field (think SAT tests), and has been used for policy decisions in other areas, such as hospital rankings.

At the heart of CSA is the Safety Measurement System, which is supposed to identify carriers at high risk for future crashes. But NAS found that some of the details of how those numbers are calculated weren’t really scientifically based, or as the report said, “ad hoc and not fully supported by empirical studies.”

The NAS report also identified a number of data quality issues. For instance, it said FMCSA needs to improve the collection of data on vehicle miles traveled by state and month, to help account for factors such as icy winter weather in the North.

In July, FMCSA delivered a report to Congress on its “corrective action plan” for CSA outlining how it’s implementing those recommendations. It is indeed moving forward to develop and test an IRT model.

While FMCSA hasn’t released many details about what that IRT model will look like, one of my favorite data geeks, Steve Bryan with Vigillo (now part of Samba Safety), was in every public meeting of the NAS panel as it developed its recommendations. He describes IRT as “very complex, but time-tested science.” His company went out and licensed sophisticated IRT modeling software to develop its own version, which he says customers can use to get a feel for how they’re going to perform under the new CSA system.

Bryan outlines three things to remember:

• This new methodology is happening, and the industry needs to start thinking about it.
• The new methodology is not about trying to predict crash risk, but focusing on the “safety culture” of a motor carrier.
• Variety kills. A variety of violations across different areas will tank your scores much worse than a record showing just one area where you’re falling short.

Bryan said he’s a fan of the proposed methodology, but the agency “largely ignored some of the data quality questions that still linger.” When it comes to the data recommendations put forth by the NAS panel, FMCSA’s report basically said, “We’ll have to research that more.”

FMCSA in July said its target is to roll out the new system by September of 2019. While I wouldn’t bet on that (the agency doesn’t have a great track record of meeting timelines), I would bet that IRT is something you’ll be hearing more about in the coming year.