Since 1998, the aviation industry and the U.S. Federal Aviation Administration (FAA) have reduced the fatality risk of air travel by a full 83 percent. The FAA’s goal is to cut that risk in half by 2025, one reason why the agency is requiring most commercial airlines to implement a Safety Management System (SMS) by 2018.
Achieving such ambitious safety goals calls for more than just a “check the box” approach, requiring companies take a hard look at whether or not their SMS is truly effective.
With that in mind, this post offers a variety of metrics you can use to measure SMS effectiveness.
Employee training is the bedrock of your safety program. Human behavior may seem impossible to measure, however there are some key measures that reveal a lot about the effectiveness of your employee training program:
- Test results: Using the SMS to analyze the results of post-training assessments or quizzes can help you identify weaknesses in your training program.
- Number of training course updates: If you aren’t updating employee training course requirements regularly, you’re missing opportunities to share lessons learned.
- Number of changes that trigger new training requirements: It’s essential that employees are trained up on any process changes. Use your SMS to compare the number of change management initiatives in the past year against how many of them involved new training requirements.
Many safety professionals treat incident management as a rear-facing process. A truly effective SMS takes a proactive approach to incidents, using data and lessons learned to minimize the chance of recurrence.
Metrics that indicate whether your incident management program is working include:
- Repeat incidents: Repeat occurrences are a sure sign you’re missing something, and likely not identifying the true root cause of incidents.
- Proportion of incidents assessed for risk: Without identifying high-risk incidents, you have no way of knowing which ones need action first.
- Time required for corrective action: Taking too long to resolve problems just increases the likelihood of recurrence. An automated SMS routes corrective action requests for you, providing alerts when tasks are overdue.
It’s not enough to have safety plans, procedures and training. You also need an effective audit program to see where things are (and are not) working so you can make adjustments. This is the all-important “check” step in the plan-do-check-act process.
The following audit metrics are particularly revealing:
- Number of repeat noncompliances: Recurring noncompliances indicate a problem with your audit follow-up system. Your SMS should allow you to launch corrective actions directly from the audit record to prevent this issue.
- Proportion of high-risk findings: If you consistently identify lots of high-risk noncompliances, it’s a sign that something is missing in your policies or training process.
- Consistency between internal and external audits: If external audits regularly uncover more noncompliances than internal audits, you probably need to reevaluate your audit procedures and checklists.
Many companies mistakenly equate risk assessment with risk management, when the truth is risk assessment is just one part of a continuous cycle.
Metrics to help you analyze your risk management efforts include:
- Hazards or observations reported in the SMS: This tells you whether people are engaged in the process, or if you need to offer incentives or quotas to boost SMS participation.
- Number of new controls implemented: Looking at the number of new controls instituted over the past quarter or year says a lot about your investment in risk management.
- Proportion of changes that triggered a risk assessment: Change introduces significant risk for an organization, which is why it’s critical each change management initiative involve risk assessments throughout the process.
At the end of the day, an SMS is just a tool—what really matters is how you use it. To make real progress, aviation companies must dig deeper into their metrics and ask if their safety programs are achieving their intended goals.