CSA, IRT, and a Culture of Safety [Commentary]

How do you measure safety? It’s a question the Federal Motor Carrier Safety Administration has struggled with for more than a decade. 

When CSA 2010 was rolled out, the acronym stood for Comprehensive Safety Analysis. Somewhere along the line, it became Compliance, Safety, Accountability. The agency didn’t exactly put out a press release about why it changed. Perhaps it had something to do with the fact that safety and compliance are not necessarily the same thing.  

The core of CSA 2010 was a new Safety Measurement System (SMS), which FMCSA described as “a tool to identify high-risk motor carriers requiring interventions in order to improve safety on the nation's roads.” 

From the beginning, while applauding the concept in general, the trucking industry had major concerns about how the SMS measured motor carrier safety. Some of those issues led to lawsuits and lobbying, and in 2017, Congress demanded the agency commission an independent study of SMS. The resulting study by the National Academy of Sciences (NAS) recommended using a better statistical model: Item Response Theory, or IRT. 

Self-described data geek Steve Bryan, founder of Vigillo (now part of Samba Safety), had attended every public meeting of the NAS panel as it developed its recommendations. In a 2018 presentation, he explained that one of the things the NAS panel was tasked with was looking at whether CSA scores actually predict future crash risk.  

“The existing CSA program is about predicting crash risk,” Bryan said. “None of us believe that ever worked. It does a terrible job.” 

Instead, he said the IRT model meant, “We’re moving away from pretending we can predict future crashes and toward evaluating safety culture.” The new IRT method, he explained, would move beyond the seven BASICs and create a single safety culture score. 

OK, that seems like a good idea. But how do you measure safety culture?  

A Virginia Tech Transportation Institute study found that fleets that reported building a strong safety culture, with management and driver buy-in, increased safety outcomes. But the actual strategies varied widely.  

As VTTI’s Matt Camden explained in a 2019 guest article, “safety culture has been a popular topic in the safety literature since the Chernobyl disaster in 1986; however, the exact definition of safety culture has been widely debated. 

“Perhaps safety culture can best be thought of as a ‘lens’ through which employees see how an organization feels about safety. In other words, a fleet’s safety culture can be assessed by the shared beliefs, values, and attitudes all their employees have toward safety.” 

If the safety experts can’t come up with a good definition of safety culture, perhaps it’s not surprising that the FMCSA decided to reject the IRT recommendation in its February announcement of proposed changes to the SMS. 

“It's easy to think of the term safety culture and consider that to be kind of meaningless business-speak,” said Brandon Wiseman, owner and president of Trucksafe Consulting, in a 2021 HDT webinar. 

FMCSA regulations don’t use the term, but instead refer to “safety management controls.” However, he said, a culture of safety “is the why behind the steps that a motor carrier takes to bolster their safety program. And it’s the foundation on which all of those policies and procedures sit, and it’s that foundation that the FMCSA expects to see when they come in and look at all your files.” 

One of the most passionate advocates I know for building a culture of safety is Chris Woody, safety manager at M&W Logistics in Nashville, Tennessee, a 2020 HDT Truck Fleet Innovator.  

In that same webinar, he said, “I think everybody's got a safety culture. The question is, is it a good safety culture or a bad safety culture?” 

Which are you?