Volume 1, Issue 6

July 6, 2010

The Devil Wore Data

"If you can’t measure it, you can’t improve it," might be just irony.  A quote attributed to Lord Kelvin, it has long been the adage of quality gurus, who are constantly looking for ways to measure the soft aspects that matter.  Concrete numbers aren’t all they are trumpeted to be, when math anxiety is real for about half of adults, and "innumeracy seems to be a part of our normal human condition", according to Stanislas Dehaene.  Maybe for people like Lord Kelvin, the inventor of the absolute temperature scale, the humour is in his iconoclasm.  After all, we have been measuring the BP disaster in time, damage, costs and barrels, and are still struggling to improve it.  Perhaps it’s time to change the mandate.

How Numbers Fail to Communicate

• 45% of Canadians polled didn’t know how many millions were in a billion. (1,000 - the difference of three zeros, and the perhaps the easiest of the four numbers to grasp.)
• Ask people to plot 50 dots randomly on a page, and an even distribution emerges, showcasing the failure to understand the size, nature and impact of randomness.
• When 68% chance of dying is the same as 32% chance of living, why do patients select the surgery only 18% in the first framing and 44% in the second?  Words matter more.
• Like Mother Teresa who has been attributed as saying "If I look that the mass, I will never act.  If I look at one, I will", big numbers are impressive, not motivating.
• Israeli flight instructors concluded that criticism improves performance based on personal experience.  Not just skewed memory, or a lack of concrete measurement, but a failure to understand regression to the mean, notes Nobel winner Daniel Kahneman. (In fact, it is neither criticism nor praise, but predicable variation.)
• World Series fans might not know that the chances of the inferior team winning are quite high, with 269 games required for a statistical valid result for a 5% edge.  Or maybe that’s why their team didn’t win last year?

Forget Data Analysis...Sometimes

When a baseball is struck by a bat, the future landing spot for that ball can be determined by measuring certain variables, knowing the appropriate equations, and crunching the data.  The precise position that the outfielder should stand in order to catch that ball would be known, with the accuracy and precision determined from the quality of the data.
However, if a person were to collect this required data, such as wind speed, humidity and temperature, the imparted force, trajectory and surface area of contact, the ball would have already landed and the player already circled the bases, increasing their score, gaining the competitive edge.  Much time, money and energy spent on determining the exact solution would be wasted, yielding no benefit whatsoever. 
In fact, outfielders catch balls all the time without complicated data analysis.  Well, not all the time, but pretty often.  Even considering salaries, the costs are much lower than the data crunching method.
Instead of crunching data, the outfielder relies on a rule of thumb.  He makes a prediction, keeps his eye on the ball, and makes corrections.  His final position is unknown to him, until the ball is in his glove and he surveys his surroundings.  With experience, predictions improve.  Practice makes perfect.
Data crunching, as with any activity, is appropriate if it adds appropriate value.  Do you need advance notification of a future event?  Is a high degree of accuracy and precision required?  Can you let an expert run with it, focusing instead on measuring effectiveness, such as the number of times the ball is dropped?  Ensure to right-size the analysis, the resources and the requirements of the solution to the problems that you are trying to solve.

The Bottom Line

When compelling numbers such as Dr. Atul Gawande’s success in reducing surgical morbity rates from 1.5% to 0.8% with the use of checklists translate into an adoption rate of only 20%, what is wrong with the numbers?  Data collection and crunching will always have an important place in improvement, and perhaps we need different ways to take advantage of their story, and know when to swap to story instead.