Although not much in the news of late, the ongoing Congressionally mandated process to improve the federal Compliance, Safety, Accountability safety-compliance regime for commercial vehicles was recently put through an internal audit and found to be solid on the whole, but lacking in data – which could make it harder to roll out changes aimed at improving the assessment of carrier safety performance.

To back up a bit, it was in the summer of 2018 that the Federal Motor Carrier Safety Administration submitted to Congress a required “corrective action plan” that outlined how the agency planned to address recommendations made in the National Academy of Sciences’ “Improving Motor Carrier Safety Measurement” report.

The NAS report, commissioned by FMCSA to comply with a provision of the FAST Act highway bill of 2015, focused on the effectiveness of using the percentile ranks produced by the agency’s Safety Measurement System for identifying high-risk carriers, and if not, what alternatives might be preferred.

NAS recommended that FMSCA consider adopting a more-scientific statistical-modeling approach known as “Item Response Theory” to make the SMS function more accurately.

The expectation was that an IRT-based method would help develop “an estimate of the measure of ‘safety culture’ for each carrier and be used to monitor and identify carriers in need of intervention,” rather than just zero in on violations, said FMCSA Director of Compliance and Enforcement Joe DeLorenzo on Oct. 22, 2017, during an educational session at that year’s American Trucking Associations’ Management Conference & Exhibition.

The FAST Act also directed the DOT’s Office of the Inspector General to audit FMCSA’s action plan in light of OIG’s own prior recommendations and those made by NAS and the Government Accountability Office.

Specifically, according to OIG, the audit’s objectives were to “assess the extent to which FMCSA’s corrective action plan addresses the NAS recommendations and relevant OIG and GAO recommendations and identify challenges FMCSA may face when implementing the corrective action plan.”

The 28-page audit report, released on Sept. 25, states that “FMCSA’s [action] plan addresses many of the NAS recommendations. However, it lacks details about the benchmarks, costs, and program reforms required to fully address the act’s requirements as they address the NAS recommendations. As a result, FMCSA faces the risk that it will be unable to improve its system for prioritizing motor carriers in an accurate, efficient, and timely manner.”

More specifically, OIG said that the FMCSA plan “lacks implementation details for improving transparency and its assessment of carrier safety rankings. For example, in response to recommendations from NAS and GAO, the agency is testing an Item Response Theory statistical model to gauge how it prioritizes motor carrier safety interventions.

"Regarding the NAS recommendation on collecting more accurate and diverse types of data, FMCSA determined that much of the data do not exist. As a result, FMCSA no longer plans to collect additional data.”

OIG also commented that the plan “describes putting datasets on a publicly available website but does not discuss making them user-friendly, or outline costs and implementation steps— hindering FMCSA’s efforts to make its data, safety measures, and rankings more transparent. Finally, the complexity of the IRT model may make implementation and public outreach difficult.”

The OIG report concludes with two recommendations for FMCSA, both of which are tied to specific recommendations made in the NAS study report:

• Regarding NAS Recommendation 5 (“FMCSA should undertake a study to better understand the statistical operating characteristics of the percentile ranks to support decisions regarding the usability of public scores”), OIG now wants FMCSA “to provide (a) cost estimates that account for staffing, enforcement, and data collection; and (b) benchmarks for completion.”

• As for NAS Recommendations 4 and 6 (which concern structuring “a user-friendly version of the MCMIS data file without personally identifiable information” and advice on computing safety scores conditionally within groups of similar carriers, as well as unconditionally for all carriers), OIG says FMCSA should “provide (a) cost estimates that account for staffing, enforcement, and data collection; (b) benchmarks for completion; and (c) potential programmatic reforms, revisions to regulations, or proposals for legislation.

However, under the heading “Action Required” in the audit report, OIG stated that it “considers [its] Recommendations 1 and 2 resolved but open pending completion of planned action.”

HDT reached out to several key trucking stakeholder groups that keep close tabs on FMCSA’s regulatory activities for comment on the audit report. At the time of this posting, only David Heller, vice president of government affairs for the Truckload Carriers Association, had responded.

“The Truckload Carriers Association supports the efforts of FMCSA to accurately determine the safety fitness of motor carriers across the country,” Heller told HDT. “However, that determination must be transparent and based on sound data so that the safety performance of our motor carriers operating on our highways will be as correct.  

“In other words, we, as an industry, shouldn’t mind being judged based upon our safety principles and performance,” he added. "We should insist that we are judged properly.”