The Data Miner's Song

I work in the trucking data mine.

Mining is woven into the American story through songs, movies, television shows, art, books and magazines. I expect even those who have never set foot in a mine or talked with a miner probably have a mental image of the work.

Mining, at any scale of operation, generally involves moving a lot of material, termed ore, to yield a small amount of mineral. It seems the more valuable the mineral, the more raw ore has to be moved. A 2013 Business Insider report shows that depending on the mine, a miner has to move from 7.5 to 91 tons of soil to yield one single ounce of gold. Some mine shafts go down more than two miles.

Freight data mining is just like that. Data mining involves going through mountains of raw material to find something valuable.

Digital data is ubiquitous. Trucks are continuously generating mountains of data. Devices such as electronic logging devices are listening to all that data and providing a way to store and analyze it. Global positioning systems can continuously keep track of a vehicle’s movement down to inches. These systems are built into smartphones and truck and trailer tracking systems. Shippers have the ability to track and report on the transit of their products from the warehouse to the delivery location. Manufacturers assemble their vehicles from digital sales orders and build instructions. Servicing and warranty are digitized. Almost anything that involves a financial transaction likely is digitized. Used truck sellers have digital profiles of their offerings. Registrations and taxes are all digitized.

That’s only a sample of today’s data.

The future promises (or threatens) even more data. Battery-electric, fuel-cell electric and autonomous trucks likely will be the most instrumented vehicles ever to hit our roads, with more sensors and software generating data. New instrumentation on roads and facilities themselves likely will increase the amount of data generated.

That’s great job security for computer chip makers and data analysts. It also should be good news for the freight industry. We should be able to know everything, nearly all the time, about anything associated with moving freight. We should be able to spot trends and patterns and properly plan for them.

There are a lot of smart people in the trucking industry — at the OEMs, at the fleets, at the suppliers, at the shippers. Today they have access to these massive amounts of digital data on nearly everything. There are also countless third-party data aggregators and analysists with business models providing specialized critical skills and capabilities. Many of these third parties have no prior background in trucking but know the ins and outs of processing big data.

Which begs the question: How is it that the industry can’t seem to predict something as significant as a multi-year supply chain disruption? Can’t seem to predict a shortage of key components? Can’t seem to manage inventories properly? Can’t get their assets into the right locations? Can’t predict asset shortages like equipment, drivers, and technicians? And can’t take proactive steps to plan for these issues?

It’s a bit embarrassing, don’t you think? It’s not like this industry has never had major disruptions to deal with. History is littered with examples of unplanned and planned factors disturbing the North American freight market.

It’s 2022. Why does the freight industry seem so chaotic at a time where we have a world of data at our fingertips?

Why the Trucking Data Miner's Job is Hard

While all that data is out there, it is not well managed, and much of it is secured in unconnected silos. Much of it is carefully shrouded as “proprietary” information. Much of the data is not even valued — it’s generated and collected, but then its owners have no idea they are sitting on a potential motherlode. Still others over-hype the value of some information, dissuading knowledgeable data users from further digging for gold. Still others collect the data but don’t provide access to it — their business models consider any data they have collected as “theirs” and charge the data generators for access to limited reporting.

The value of data is hard to predict. Much of it is waste, the data mine tailings of the data mining process. But what is waste to one user may be an income stream to another. The foundation of research starts from humbly admitting that we don’t know everything. Sure, we may have opinions about everything, but we really don’t know everything. Fact based decision making requires data. Risk reduction planning requires data.

Access to a wide range of data also can spark innovation. Freight efficiency is all about measuring a baseline and then trying out something different to see if it improves. The better the access to the diverse silos of data, the better the ability of the data miners to find the gold. When working with fleets in the past, one of the first steps in addressing a fleet’s truck performance concern was to install a data logger and collect real-world data on actual operations of the truck. A site visit also was required to collect photographs and conduct interviews. Several times I found that what the driver and fleet manager thought were different — and both differed from what the data showed.

Industry groups have the ability to greatly influence the accessibility, collection and use of data. They have the ability to safely anonymize data and make it available. They have the opportunity to standardize information formats, nomenclature and taxonomy. Those industry groups that lead in doing this will enable researchers — the data miners — to find freight gold. These groups should not be viewed by innovators as constricting innovation, but rather as empowering it.

Data mining is the future of freight efficiency improvement. The mountains of data coming in will continue to grow as new technologies reach the market. Fleets will need to develop or hire skilled data miners. The need for finding freight efficiency gold will always be there.