The Federal Motor Carrier Safety Administration announced on July 16 that the agency has delivered a required “Correlation Study Corrective Action Plan” for its Compliance, Safety, Accountability program to Congress.

FMCSA’s new corrective action plan outlines how the agency will address recommendations made in the National Academy of Sciences’“Improving Motor Carrier Safety Measurement”report, which examines the effectiveness of the use of the percentile ranks produced by SMS for identifying high-risk carriers, and if not, what alternatives might be preferred.

In addition, that report evaluates the accuracy and sufficiency of the data used by SMS, to assess whether other approaches to identifying unsafe carriers would identify high-risk carriers more effectively, and to reflect on how members of the public use the SMS and what effect making the SMS information public has had on reducing crashes.

FMCSA noted that it has pulled from public view a preview website that displayed changes that the agency had proposed to its Safety Measurement System. Those changes had been “released prior to the NAS report,” so they are now outmoded.

The NAS study and subsequent report was commissioned by FMCSA to comply with a provision of the FAST Act highway bill of 2015 that mandated a study be conducted of the agency’s Compliance, Safety, Accountability program and its SMS component.

According to the agency, the report to Congress presents FMCSA's corrective action plan for addressing the six recommendations made by the National Academy of Sciences’ study.

A key element of the NASA report is the recommendation that FMSCA study a more scientific modeling approach known as “Item Response Theory” and then consider implementing it to make the SMS function more accurately.

The hope is that an IRT-based method would help develop “an estimate of the measure of ‘safety culture’for each carrier and be used to monitor and identify carriers in need of intervention,” rather than just zero in on violations, said FMCSA Director of Compliance and Enforcement Joe DeLorenzo on Oct. 22, 2017, during an educational session at the American Trucking Associations’ annual Management Conference & Exhibition In Orlando, Florida.

Actions that FMCSA stated it will take in its 10-page corrective action plan include:

• Moving forward to develop and test an IRT model. That model will be used to inform the agency’s work, with opportunities for public input. As recommended by NAS, if the new IRT model performs well, it will replace the existing SMS. FMCSA will not substantively modify SMS while we are testing the IRT model “to ensure we have a stable comparison to evaluate effectiveness.”

• Agreeing that more frequent and more detailed Vehicle Miles Traveled data from motor carriers would reduce the need for FMCSA to use substitute values and would improve the quality of the data in SMS. FMCSA currently only collects carrier VMT data every two years. Access to this data, by State, on a monthly basis is not currently feasible, but the agency will continue to identify possible sources for this important data.

• Agreeing that additional information about carrier operations might improve the agency’s analysis and identification of non-compliant motor carriers. However, the collection of this data would come at a cost, and the benefits are unknown.

• Agreeing that there could be benefits from making MCMIS data available to researchers and carriers. As a result, the agency’s first effort will be to improve data availability. FMCSA will develop a web page where researchers, carriers, safety consultants, and the public can obtain simplified MCMIS data snapshots.

• Gathering public input from motor carriers, insurance companies, and shippers regarding the ways in which the public uses SMS data. Using the data collected in these public listening sessions, FMCSA will scope and complete a study specific to the issue of percentile ranks and the usability of public scores.

• Once IRT modeling is complete, FMCSA will evaluate the use of absolute measures for set intervention thresholds. Absolute measures and percentiles are products of the SMS system. At this time, it is not known how these would be affected once the Agency completes the modeling recommended by NAS. Therefore, FMCSA defers action on this recommendation until it is confirmed this would be relevant.

As for “next steps,” FMCSA said those will include working with the input of the NAS Standing Committee to develop and run a small scale IRT model by September of 2018 and after evaluating the results, running a full scale IRT model by April of 2019. Additionally, the agency said it will work with the NAS Standing Committee on planning and scheduling public meetings throughout IRT development to further discuss the recommendations.  

Related: Proposed CSA Scoring-Improvement Method Gets Thumbs Up