The Madness Of Measuring Nothing

First published Phil La Duke on his blog See the whole article here

Phil says: In a business climate where every company seems to live and die by its metrics one important business element is often over looked, or worse yet is misleading business leaders. Safety has been poorly measured for almost 100 years, and in this week’s post I offer better ways to measure worker safety. I hope you will give it a read and tell me what you think.

These days’ organizations live and die by measurements. It seems that no matter where we work we are confronted with the dreaded balanced score card and so we are tasked with measuring “safety”. I’ve said for a long time, “the absence of injuries does not denote the presence of safety” and zero injuries doesn’t tell us a heck of a lot about the risk of injuries within a given population.

The traditional measures of safety, i.e. injuries or days away or restricted time, don’t help us predict the likelihood of future performance, and yet that is the best we seem able to come up with. We play at leading indicators like near miss reporting (as somehow indicating the level of participation of workers in safety) but even these measurements are fraught with statistical noise that can lead us to conclusions not in evidence; so many of our indicators mislead us that one has to wonder if there is any value in them at all.

Recently I was asked to address a meeting of the senior leaders of a multi-national manufacturer and I was asked what some good predictive measures would be for safety. I was pressed for time and I’m afraid I didn’t have the luxury of a prolonged discussion on metrics (my topic was Creating A Culture Of Safety Excellence). So given the fervour set in motion from my last three posts I thought I would add a bit of metaphorical fuel to the fire and lay out for professional debate, what I see as some good ways for correlating business measures to future performance in safety.

Risk Factor #1: Worker Stress and Distraction

Worker stress has a profound impact not only on human error, but on risk taking, and worker’s health as well. Highly stressed workers are distracted and distraction leads to mistakes which lead to injuries. Some measures that I think directly correlate to worker stress are:

Worker absenteeism. Absenteeism rates are indicators of both worker stress and worker competence. Research has shown that stressed workers tend to miss more work, and when a worker misses work, his or her job is done by someone less skilled, less practiced at the job, and therefore more likely to deviate from the standard. In other words, the worker stuck doing the job is at greater risk of injury than the worker whose muscle memory is completing many of the tasks by rote. Of course this isn’t universally the case, but it is true often enough to correlate, and when it comes to prediction, correlation is the best we’ve got.

Number of calls to employee assistance programs. When we talk about worker distraction, we tend to think in terms of distractions borne in the workplace. Workers who are worried about financial problems, divorce, or other “off-hours” problems while working face the same dangers as those distracted by work issues. The number of calls to EAP lines can provide a good idea of how much distraction is in the workplace which correlates to human error, behavioral drift, lapses in judgement and ultimately workplace injuries.

Worker turnover. Employee turn over creates risk in much the same way absenteeism does: it introduces greater variation into our work processes which in increases the risk of injuries. The greater the worker turnover rates the higher the risk of injuries as newer, less competent and skilled workers replace higher performing, more experienced workers.

Engagement survey scores. Engaged workers tend to do things because these things are the right thing to do. The lower the level of employee engagement the higher the risk of worker injuries.

In all these cases we have to remember that we seldom have a perfect correlation (a case where every time factor A is true factor B is also true) and even in those rare cases where there is a perfect correlation such a condition does not mean that there is a cause and effect relationship between two factors. But since we are looking at the measurement’s predictive value there is always a margin for error, statistical anomalies and statistical outliers. If we had a perfect way of predicting exactly where and when an injury would occur we would be using it.

Risk Factor #2: Worker Incompetence

When we talk about worker incompetence, we’re not talking about the nincompoop who doesn’t seem able to do even the most rudimentary task without screwing things up, rather, we are talking about the skill level at which a worker is able to perform his or her job.

There is a strong correlation between level of mastery at which a worker performs the tasks associated with his or her job and the risk of injuries. To that end these measurements are appropriate and predictive:

Required training % complete. Assuming that we require training because it is necessary to do one’s job, the lack of this training would indicate process variability. Tracking the percentage of training provides us with a glimpse of how much risk a worker faces of being injured because he or she performed a task improperly. The greater the percentage of people who have completed training the lower the risk of injury because of a gap in essential skills.

% of licenses and certificate expired. Just as the percentage of required training that is complete provides us with an understanding of approximately how many people are likely working out of process (it’s tough to do the job right simply by guessing) so too does the percentage of workers who are working despite having expired licenses and certificates.

Time to complete required training. The longer it takes to complete required training the longer a worker is exposed to workplace risk associated with a skills gape.

Worker performance appraisal scores. This particular measure is tricky—it assumes that the worker appraisals are fair assessments of the worker’s ability to accurately complete tasks and do the job. Assuming that there is a robust worker performance appraisal assessment the lower scored individuals should be at greater risk that those who are performing at higher levels.

Risk Factor #3: Leader Incompetence.

Workers generally perform in ways for which they are rewarded and eschew behaviors for which they are punished. Low-performing leaders often exacerbate safety issues by behaving inappropriately in their interactions with workers. Some measures that I think directly correlate to leadership competency are:

360 Reviews. 360 Reviews, that is, reviews where a leader’s team members, boss, and peers all contribute to the review, are often excellent indicators of how well a leader interacts with his or her team. The weaker the leader the higher the risk of process variation and hence a rise in the risk of injuries.

Leader performance review. Leaders who perform poorly are generally allowing more variation into the work area the higher the performance of the leader the less likely workers will be harmed on his or her watch. It’s important to note that the leader’s performance review will most likely include things like the productivity of his or her team, general performance in things like cost, quality, and efficiency, in other words, things that will either directly or indirectly impact the risk of injuries.

Worker morale. Of course worker morale can be effected by a host of things unrelated to the leader, but worker morale is heavily influenced by the performance of the leader. Workers suffering from poor morale generally perform at lower levels which fall outside the processes control limits. The worse the morale the higher the risk of variation and ultimately injuries.

% of safety reviews completed on time. I am not a fan of “behavioral observations”; I’ve always felt the time watching someone work could be better spent taking a more holistic view of worker safety by reviewing the risk conditions (procedural, physical, or behavioral). That having been said, it is important that leaders conduct routine and repeated inspections of the workplace to identify hazards. The percent of safety reviews/tours/inspections/observations completed on time is a, at least ostensibly, an indicator of the time to which workers are exposed to hazards.

% or performance reviews completed on time. Completing performance reviews on time isn’t just about making employees feel good, it is also about assessing competency. The more reviews that are completed on time, the more skills and performance gaps are identified in a timely manner.

% Attendance at safety meetings. The percentage of safety meetings that a leader attends provides a good insight into the level of priority on which the leader places on safety.

Risk Factor #5: Process Capability

Process variability creates risk; to the product, to the equipment, and to the workers. The frequency and duration of non-standard or out of process work is a good predictive indicator of risk of injury. Good measures of process capability (relative to safety) are:

% of nonstandard work. Statistically speaking nonstandard work tends to be more dangerous and the injuries associated with nonstandard work tend to be more lethal than its standard counterpart. The percentage of work that is nonstandard can indicate a substantial bump in risk associated with any operation.

% of jobs with completed JSAs. A complete and current Job Safety Analysis (JSA) is crucial for the safe execution of work, yet I don’t know any company that has 100% of it’s jobs with JSAs, and many companies don’t have a good track record of keeping the JSA’s current with the standard operating procedure. Understanding the percentage of your tasks have good and current JSAs is a good predictor of future risk (the higher the percentage the lower the risk).

% of jobs with Standard Work Instructions. Personally, I prefer Standard Work Instructions (SWI) to JSAs (a good SWI should address all the safety concerns of a job), but SWIs suffer from the same problems that I discussed regarding JSAs above.

% behind in production. I still have nightmares about my days working an assembly line and falling “in the hole” screams of “man in the hole” booming above the cacophony of hand tools, presses, and industrial vehicles still give me chills. Whenever ever workers are struggling to catch up because they are behind in production the risk of injuries rises.

% parts shortages. When there are part shortages (or tools shortages, or materials shortages, or labor shortages for those of you who work outside manufacturing) workers are forced to work outside the standard process. This is incredibly dangerous because the standard process is designed with protections against injuries embedded in the tasks. When a worker is working outside the process the organization is relying on luck to protect them.

Risk Factor #6: Worker Engagement In Safety

We’ve discussed worker engagement in a broad sense, but I think it is important enough to look at worker engagement specific to safety. Engaged workers will work safely for no more reward than because working safely is the right thing to do. Worker engagement in safety can be measured by:

Number of reported near misses. Some will argue, correctly, that near misses are lagging indicators, but whether or not a worker choses to report a near miss correlates to the level of worker engagement in the safety process. This measurement, admittedly, is difficult to get accurately. Since we don’t know the total actual number of near misses we can’t say with certainty whether the current level of reporting is a high or low percentage. Even so, the number of workers who report, even more so than the raw numbers of near misses, can provide a good glimpse into the level of importance workers place on safety.

Number improvement suggestions. Workers who take an interest in improving the organization are generally interested in finding and eliminating failure modes, which will include those failure modes that will ultimately place workers at risk of injury. The greater the number of suggestions the lower the risk.

Participation in continuous improvement workshops. Eliminating variation, risk, and hazards are part and parcel of the continuous improvement process so it should surprise no one that the level of participation in these activities correlate to the level of risk.

Number of worker grievances. Worker grievances shed valuable light in to many of the other risk factors identified here and generally the greater the number of grievances the higher the level of risk of injuries.

Number of disciplinary actions for safety violations. The number of disciplinary actions for safety violations are indicative of two things: the number of unsafe acts being committed and the extent to which these incidents are taken seriously.

Of course one has to be careful in designing and managing these measurements to avoid unintended consequences (for example, one could easily reduce the number of disciplinary actions by not applying appropriate discipline, or one could raise worker performance evaluation simply through “score inflation” but the risk of these unintended consequences can be reduced by solid management practices and random sampling audits.

The Imperfection Of Predictive Measures

To some extent we can never have a perfect set of measures. In many ways it’s like predicting the weather, since we are talking about probability there is always a chance that the organization will beat the odds. In fact, there isn’t one of these measures that I couldn’t construct a convincing argument against. What’s important is to use those of these measures that make sense and use them in conjunction with each other. One correlation does not a pattern make, but when we look at multiple areas of risk and analyze them in a holistic context we can find a more useful way to measure safety than counting bodies and broken bones.