An anonymous, exasperated executive from a car manufacturer recently squirmed during an off-the-record interview when asked about his company’s intentions to address ongoing quality issues. He referenced the internal desire to measure the “Cost of Poor Development Quality” and bemoaned, “You don’t understand. There’s no way to measure that. Each engineering group and set of suppliers has its own tools and methods to track defects. Having an ongoing, real-time, transparent view of that is essentially impossible.”

Not only does his conundrum have a nightmarish web of complexity (including politics and negotiations), but it already assumes the correct Key Performance Indicator (KPI) is the Cost of Poor Development Quality and untold assumptions about the associated calculations (e.g., Labor rate? Time to fix each bug?). And as the oft-incorrectly-attributed-to-Peter-Drucker quote says, “It you cannot measure it, you cannot manage it,” so the typical corporation defines KPIs around things they can easily measure regardless of how key their value truly is. The silos in the automotive OEM space – and their suppliers – means that real-time data, to insight, to decision – at the speed of Tesla – is very challenging without next-gen observability tools.

Few would argue that having well-informed strategies for the struggles they’ll encounter will define the winners and losers as automotive companies pivot during the upcoming era of the step-function changes (e.g., Software-Defined Vehicle or connected car). But our lamenting executive is correct: finding the better metric and its supporting data is not easy. For example: should that executive’s Cost of Poor Development Quality include how long into development it took for a defect to be identified at each tier of the supply chain (e.g., Before it was coded? Before it was launched)? And what if the defect created a cybersecurity vulnerability for vehicles in production? That cost of clean-up is significantly higher.

“Many times you don’t know the questions you are going to ask of the data upfront,” states Tom Wilkie, Chief Technology Officer of Grafana Labs, which provides observability solutions built around data visualization and dashboarding. “The common technologies out there either store all of the data – which is typically quite expensive in terms of resources – or uses sampling, which often doesn’t answer questions which weren’t conceived ahead of time because all of the raw data is no longer available.”

Therein, creating management dashboards driven by the key-est of key indicators is no easy task. Such a desired outcome is predicated on two upfront ways of approaching the issue: prioritizing the goals and designing in flexibility.

Prioritizing The Goals

Gathering too much data and attempting to oversee or manage too many metrics creates Analysis Paralysis. Per a global Oracle survey conducted in April, “97% of respondents … said they wanted data to help them make better and faster decisions, reduce risk and increase profits [yet]

72% admitted that the volume of data—and their lack of trust in it—has prevented them from making any decisions at all.” Analysis Paralysis.

“Start with the organizational goals,” says Bob Rapp, formerly the first ever Principal Partner at Microsoft (and eventually technical leader for Cloud, AI and Data) and now the Director of Partnerships at Envorso. “And not a few, high-level ‘pillars of the organization.’ Those don’t help inform operational issues. Create a set of specific goals that can inform which questions are paramount. Pick outcome-based insight from broad goals – and make sure you are gaining data, insight, decision and advantage in real-time – not simply taking the KPI’s that exist – but creating new value from KPIs you need.”

Designing The Flexibility

Interestingly, there are multiple aspects to flexibility which are … you guessed it … key: flexibility of inputs, flexibility of inquiries and flexibility of switching.

Flexibility Of Inputs

As automotive designs get more complex, the sources of input data have a greater likelihood of disparate systems. Take, for example, our frustrated executive: the defect information he’s seeking is probably within multiple supplier’s tools (e.g., Jira, Userback, Asana), which might not even be coordinated between each company’s I.T. and engineering departments.

So a flexible design must assume upfront a malleable interface for accepting, storing and eventually consuming raw data from whatever source.

Flexibility of Inquiries

As previously described, knowing the exact overarching or temporal questions is extremely difficult, so it’s paramount to select a tool that provides a low-cost storage of all the data, and flexibly assists with finding the next interesting question. “The underpinning of agile – both in development and metrics – is we can’t know where we want to go,” pronounces Wilkie. “There’s no amount of effort we can do upfront that will actually give us confidence in our original answer because it’s unknowable. And, therefore, the only way to find out is to just do it and iterate. Do a little bit and learn.”

Flexibility of Switching

As detailed in New Relic’s “Top Five Observability Pricing Traps,” the tools that make your metrics transparent are ironically not always transparent with theirs. So sometimes it won’t become obvious that one solution is high-cost until overage fees or penalties come due, and then switching becomes arduous and expensive.

So before cobbling together an expensive interface or customized solution, look for the ease of switching tool providers.

“There’s something strange – good strange – about the observability tooling market,” muses Wilkie. “It’s a very competitive space with thirty or so billion-dollar players. This is hyper-rare, especially without consolidation. As such, the value that customers get has a huge recency-weighting to it. And that’s kind of cool. If [customers] have a very low switching cost, and there’s a new kid on the block who offers a better service, [they] can switch to him very easily. If any of us ever take our foot off the gas on innovating, it’s going to work against [us]. The people who win in this space are the users and consumers because this market dynamic fuels being responsive to what the market needs.”

Author’s Note

I have also seen the opposite of Analysis Paralysis: jumping quickly to a dashboard without considering the Goal-Question-Metric (GQM) Approach, as recommended by my long-term mentor. Over-haste typically ends miserably with information that’s either uninformative or, even worse, misleading. Usually such debacles happen because diverse roles with associated perspectives and goals are not included for the sake of speed and, voilà, the result is the favorite cliché of my 3^rd grade teacher, Mrs. Jackalow: “haste makes waste.”

So, yes, be agile. But running through a door quickly in the opposite direction doesn’t result in great velocity. Take a moment to at least start with a strategy.

That’s key.

Follow me on Twitter or LinkedIn.