5 Steps to Make Better Audit Decisions
Making audit decisions is always a challenge. When is the right time to schedule an audit? How many audits are necessary? What are the weak points of the company to be audited? Which auditor(s) should you send? How is their performance? And where should you set measures to improve efficiency, without sacrificing quality?
If you struggle to answer these questions and questions alike, this article will be of great help. Follow me along as I provide you with a 5-step framework to using your data to make better audit decisions and lay the foundation for a risk-based audit approach that will make a difference.
1. Define Your Problem and What You Would Like to Know
Whether you work at a certification body or standard setter, or are responsible for the internal audit management of your company, you certainly have valuable data available. Which of your data is valuable depends largely on what you want to know, and it tends to be buried in loads of other data irrelevant to your questions.
So, before your start analyzing your data, it is utterly important to define what you would like to know from your data, for it being very hard to draw insights if you are not clear about what you are looking for. Most of the attempts to setting up a sound business intelligence (BI) already fail at this point because the goals were not defined clearly enough or this step was skipped at all.
… before your start analyzing your data, it is utterly important to define what you would like to know from your data …
Best start by writing down all the challenges you face and defining the question you would like to have answered. The questions I raised at the beginning of this article can be a great starting point. However, you should rely on your experience and industry knowledge to find the questions most relevant to your concrete needs. After you have defined your questions, determine which data could help you to answer them clearly.
For example, if you want to make risk-based decisions in your audit management, you need to look at the data of previous audits. This data needs to be cleansed and prepared for analysis based on the questions you have defined; then, the interesting part begins.
If you have a data specialist team available at your company, you should get in touch with them and ask for their support. If not, consider to consult an external data specialist such as Intact. In any way, the data maturity model as outlined in figure 1 will help you to understand the way ahead of you.
2. Descriptive Analysis
Descriptive analysis allows us to describe our problem(s) by using the data we have available in a straight forward way, with the result being a number of pie, bar, and line charts, or descriptive numbers such as the mean and the standard deviation. They allow us to see trends, clusters, and extreme values. Most people stop here, because they think that now they understand everything. This, however, is only the starting point for us to get a good general understanding of our data and a feeling for the solution ahead.
3. Diagnostic Analysis
In the book Thinking Fast and Slow, Nobel laureate Daniel Kahneman brought up the interesting concept that our brain works in two different systems. System 1 is a fast one; it is activated by doing things fast, automatically, frequently, emotionally, stereotypically, and unconsciously. This would be the case if you , e.g., look at a data set and instantly draw conclusions from it. You see that the audit performance of auditor A is way worse than from the rest and you instantly think that you need to talk to him. And while this might be true, it most probably is not.
You should brace yourself and use System 2, which is slow, effortful, infrequent, logical, calculating, and conscious. Getting back to our example, you might now think that the probability is high, that you did not normalize your data. That happens quite often when clicking data together, regardless of the tools used. Probably this auditor had more audits then the others? Or always audited a special kind of company or scope in which a higher number of non-conformties is normal? With this in mind, you should start to think about what really causes your challenges.
If you really want to base your audit management on risks, you should ask for the biases of your companies, auditors, and certification bodies. Can you cluster them to see those biases? Can you educate a cluster of low performing auditors differently to bring them back up to the average? Can you educate the average cluster to get as good as your top auditors? Can you … well, there are more questions than we could possibly cover here. However, the point is this: you can find the answers to your questions with diagnostic analysis. In this step you will also define what your answers are going to look like, and you can create dashboards to control your now defined Key Performance Indicators (KPIs).
4. Predictive Analysis
Once you have your diagnostic analysis in place, it is time to create scenarios. Why would you want to do this? Because KPIs only allow you to look back at your historic data. Predictive analysis on the other hand can give you an outlook on what is likely to happen based on the data you have.
Imagine having a statistical model that could predict audit outcomes based on patterns within your data; patterns including all the biases of company, auditor, and certification body characteristics. You could even go further and include outside events such as corruption, weather, or findings of official controls to make your predictions more accurate or comprehensive.
Until recently, we were very bad in recognizing patterns and predicting events. This changed with machine learning, which today allows us to better explore how different factors play together and thus become better and better in recognizing patterns and predicting future events. And it really pays off. In a pilot project, Intact was able to predict audit non-conformities with a starting accuracy of over 82%, with the results getting more accurate over time. This not only allowed their client to proof that its risk-based audit approach worked, but also to make data-driven decision when planning future audits. This is, what the fifth and final step is about.
5. Prescriptive Analysis
After having conducted the steps 1 to 4, you have a lot of information available to base your audit decisions on. Now, it is time to sit down with your colleagues and make risk-based decision about your next steps. Below, you can find some exemplary findings and the resulting decisions:
- We see that companies in a certain commodity and area are performing so well over time that we can and will skip an audit.
- We see that the audit performance of certain auditors is significantly below average. We will arrange extra education to improve their performance.
- In order to improve the audit performance in regard to finding hygiene non-conformities, we will select auditors with biases to hygiene.
Start, Complete, Repeat
In order to achieve the best results and keep your risk-based audit approach up-to-date, you should set these five steps up as a continuous improvement cycle; even if you see good results right away. By implementing your data and statistical analysis dynamically, you can constantly update your dashboards and predictions models and, thus, safely steer your organization based on hard data and well-founded predictions.