City of Tulsa Fleet Maintenance Manager Mike Wallace didn’t expect to be thrown back to the Stone Age when he headed to work in early May 2021.

“I came in thinking it would be a normal Monday morning for fleet operations,” he said in a GovCast interview titled “What the Tulsa Fleet Learned From its Ransomware Incident.” “I soon discovered we had no FMS [fleet management system], no phones, no printers, no Internet, no Wi-Fi and no email,” he says.

He learned city-wide outages had wreaked havoc on every city system, from public safety to utility billing to the city website. Then he learned a ransomware attack was to blame.

A ransomware attack is a type of malicious software or malware that prevents users from accessing their computer files, systems or networks until they pay a ransom for their return. The City of Tulsa had received a message on its server that said, “We have compromised your server, contact us on the dark web at this address so we can negotiate terms for you to pay a ransom.” The message linked to images that showed hackers had access to their data.

Effects on Fleet Operations

The City of Tulsa prides itself on operating an efficient and well-maintained fleet and has earned recognition for its efforts.

The Equipment Management Division in the Asset Management Department won the 2021 No. 1 Fleet Award from The 100 Best Fleets in the Americas—including North and South America. The 100 Best Fleets Organization recognized the City of Tulsa Equipment Management Division as the No. 8 rated public sector fleet in North America. The fleet also has received recognition from the National Association of Fleet Administrators as a top-50 green fleet.

But even with these accolades commending them for a job well done, fleet managers were hard pressed to deal with difficulties presented by the attack.

“We broke out three carbon copy work orders from ages ago and the senior guys got to show old technology to the new guys,” Wallace says. “They were used to having everything on computers.”

It took three weeks for FBI officials, Microsoft teams and internal IT professionals to identify the hackers and months to fix the problems they caused. Wallace’s team shifted to a paper-based system in the meantime.

“We developed a workflow process where everybody knew the steps,” he says. “We also were fortunate to have storekeepers who knew their parts rooms because they no longer could look up inventories. They needed to know what they had on the shelf.”

The operation also relied on vendors for help. But calling them on the phone meant using personal cell phones. Desk phones no longer worked.

“It’s good knowing people’s phone numbers and who to reach out to when you lack vehicle histories or come in on a Monday hoping to pick up where you left off on a repair,” he says. “These were challenges we had to work through. But the guys adapted quickly.”

He adds the outage greatly affected vehicle diagnostics because everything is Internet driven. “It’s all based on Wi-Fi scanners and computers that do diagnostics,” he says.

John Reel, administrative supervisor for the City of Tulsa, echoed Wallace’s challenges and added a few others. He notes the city not only lost access to its FMS, but also lost access to its fuel tracking system.

“We stuck a clipboard with notepads on the fuel islands, then turned on the pumps and relied on everyone to write down their unit number and the type of fuel and quantity they got,” he says.

Then they appointed a person to collect this paperwork daily and to track fuel use by vehicle manually.

The service department had an advantage because technicians used Chrome devices that did not connect into the city network, Reel adds. They brought in whole-home Wi-Fi devices and Reel’s personal firewall so technicians could access needed data.

“We had people using their own phones as hotspots so they could pull up the information they needed and gain access to the sites they needed,” he says.

Fleet operations worked this way until the city resolved the issue, he adds.

Months to Resolution

The ransomware attack demolished the city’s entire network, requiring IT officials to rebuild it from the ground up. With all city systems suffering damage, fleet management took a backseat to more pressing priorities, like restoring police and fire systems.

“We were down about two months, which correlated with the end of our fiscal year,” Wallace says. “So not only did we need to get our systems back online, but we also needed to close out our fiscal year. That meant we had to enter nearly 3,000 work orders and 15,000 fuel transactions into our system within two weeks.”

The city fleet department first considered hiring temps to input data, but quickly realized it would take too much time to train them.

“We had to utilize our own resources, which meant working a lot of overtime and shifting resources around,” he says.

Be Prepared

Never say never, stress Wallace and Reel. Ransomware and other cyber attacks happen every day and can happen to anyone.

“You cannot take for granted that when you leave on a Friday, everything will be there on Monday,” Wallace says. Experience shows most ransomware attacks happen over the weekend.

Tulsa fleet operations now has its FMS system set to run reports on all information they wished they’d had after the ransomware attack. For instance, they run monthly reports that list all maintenance work due on vehicles, a complete inventory report on all parts, and a work order export for all shops.

“If this ever happens again, we’d refer to these reports, which automatically go to our email,” Wallace says. “Our email is web-based. As long as we can get online, we can access these reports. If it happens again, we have a list of all vehicles and what’s due on them, inventory reports for all locations, and work order statuses for every vehicle.”

Most fleets have a disaster recovery plan for floods, tornadoes, other natural disasters, and now even for pandemics. But most still lack data recovery plans, he adds.

“I wouldn't want any other fleet to go through this. It was an interesting couple of months,” Wallace concludes. “Put things in place to be ready for it.”