The Tyranny of Metrics
This is the title of a very easy to read book on the obsession of metrics by Jerry Z. Muller. It should be mandatory reading for anyone in the safety industry.
‘The challenges of ‘tick and flick’ in the culture of safety is endemic. We know from the research by Herbert Simon (https://www.researchgate.net/publication/262630113_Herbert_A_Simon_and_the_concept_of_rationality_Boundaries_and_procedures) and others that when humans a ‘flooded’ with excessive paperwork that they seek ways of coping with those excesses. This creates a new risk that gives the appearance of having undertaken a risk assessment without actually having done so. This shifts the emphasis from the process of risk assessment to the production of a paper outcome which should be evidence of a risk assessment.
Similarly, the focus on low level measurable goals such as injury data, tell us nothing about the culture of a site or an industry. High quality goals like: trust, listening, learning, care and collaboration disappear into the background as people attribute some value of safety to a measurement. The non-measurable goals that are critical for success in business and in safety remain devalued simply because they cannot be measured quantitatively. Imagine if we scored our relationships based on the same logic!
When I do training on the psychology of motivation and goals we explore this fixation on quantitative goals and attribution of value to low order goals. Safety seems the only industry that seeks to change culture by measuring what doesn’t matter. The following semiotics is taken from that workshop, illustrating the challenges of setting goals in the three goal states (both conscious and unconscious).
What we observe from the Goal State Model ™ is that the things most important to us are immeasurable. We certainly know intuitively when they are not present.
We learned recently about the corruption of banks in Australia, a corruption that has been brooding for many years since the ideology of KPI measurement became endemic. This interesting piece by Ross Gittens shows just how the tyranny of metrics has encouraged corruption in the banks (https://www.smh.com.au/business/the-economy/banks-commission-ross-gittins-metrics-kpi-20180424-p4zbbj.html). We now have bank executives paid in tens of millions of dollars presiding over a corrupt system rewarded by KPIs measured against everything that doesn’t matter (http://www.abc.net.au/news/2018-04-28/banking-royal-commission-financial-industry-culture-exposed/9705504 ). However, we now know that the qualities that matter most, that cannot be measured, have been missing for some time: honesty, integrity, respect, listening, compassion and trust.
We need to confess that we tend to value what we can measure not measure what we value. What can be measured is not always worth measuring. What gets measured may have no relationship to what we really want to know, and all the energy and cost put into measuring things that are not significant for safety may give little value for all that work.
The things we measure tend to draw value away from the things we should most care about, higher order goals and motivations. As one wise person said: ‘not everything that can be counted counts and, not everything that counts can be counted’.
Dr Rob Long
Latest posts by Dr Rob Long (see all)
- Welcome to the Nightmare, Safety Creates its Own Minefield (as usual) - January 23, 2023
- Safety is the Wrong Anchor - January 23, 2023
- Foresight Blindness, Hindsight Bias and Risk - January 23, 2023
- Getting the Balance Right in Tackling Risk - January 23, 2023
- What is SPoR? - January 23, 2023
Bernard Corden says
I often find it most interesting to listen to any safety crusaders preaching about statistics and performance metrics.
Many are unaware of the difference between qualitative and quantitative data or counting and measuring and descriptive or inferential statistics.
Moreover, many still believe the risk matrix is a quantitative device although it is no better than a Hasbro Ouija board or phrenology.
If you ask any safety proselytes the difference between the mean, mode and median the dark fog clouds soon come rolling in and most are completely baffled by terms such as categorical, nominal or ordinal data and interval or ratio scales of measurement.
This is before moving onto hypothesis testing and terms such as statistical significance or confidence intervals.
Then throw in the statistician’s wonderful caveat that correlation does not imply causation and it leaves most of the evangelists floundering.
I recently asked a general practitioner what was the repeatability and reproducibility of the Covid-1984 diagnostic test.
The response was an incredulous…….”I don’t know what you are talking about.”
Indeed statistics often conceal more than they reveal.
Rob Long says
None of this is studied in WHS curriculum or in the AIHS BoK. The compulsory mis-education of people in safety is astounding. It is as if some engineers got in a room, were shown all that was important in understanding human persons and risk and then deliberately decided to ignore it.