ORLANDO – Boiled down to its essence, the congressionally mandated review of the Compliance, Safety, Accountability program by the National Academies of Sciences found that the Federal Motor Carrier Safety Administration should move to a more scientific method to drive the Safety Measurement System that signals which motor carriers are at highest risk for future crashes, according to an FMSCA official. 

“The National Academies said to FMCSA, ‘The model you have is pretty good, but it’s not based on a principled scientific approach. We would like to see a better statistical model behind SMS,’” FMCSA Director of Compliance and Enforcement Joe DeLorenzo, said on Oct. 22 during an  educational session here at the American Trucking Associations’ annual Management Conference & Exhibition.

He said NAS recommended that FMSCA take about two years to study a more scientific modeling approach known as “Item Response Theory” and then consider implementing it to make the SMS function more accurately. “Our focus,” he told the audience, “remains focused on the prevention of crashes, not the prediction of crashes.”

DeLorenzo said the hope is that an IRT-based method would help develop “an estimate of the measure of ‘safety culture’ for each carrier and be used to monitor and identify carriers in need of intervention,” rather than just zero in on violations.

To accomplish all that may take digging into a variety of data points, such as driver pay, driver turnover, and miles travelled. He likened such an approach to the IRT-type scoring used in such standardized tests as the SAT (Scholastic Aptitude Test), familiar to generations of students applying for college admission, that weights answers not just on whether they are “right” or “wrong” but also on how they reflect a student’s depth of understanding.

“This is a long-term development,” DeLorenzo conceded. “The next step will be developing an action plan [in response to the NAS survey] per the FAST Act [which mandated the survey]. FMCSA will continue working with the National Academies on this over the next several years. All this [work] is very data-intensive; it will take a long time to run [all the numbers].”

Steve Bryan, president of president of data analysis firm Vigillo, Inc., which provides safety analytics, told the audience that after attending all of the public meetings on the SMS study held by NAS and digesting the 134-pg. report produced, he is in favor of seeing an IRT-based model adopted.

“IRT is a very sophisticated statistical model,” said Bryan. “But the good news is it has been battle-tested in other industries. IRT won’t predict the next crash, but it will shine a light on the cultural weaknesses of carriers that lead to higher crash rates.

He allowed that incorporating the NAS recommendations into the CSA system will be “a heavy lift,” but said the result will be worth it as trucking will end up with “a much better safety scoring system.”