Determinants of internal audit efficiency

This paper describes a case study in which Data Envelopment Analysis (DEA) methodology was combined with regression analysis to evaluate the efficiency of an Internal Audit (IA) department over twelve consecutive months. Efficiency of audit projects was first estimated using DEA. These results were then used as one of the outputs to perform a multi-period DEA study with a choice of other inputs and outputs specific to the Internal Audit department under review. The efficiency of audit projects is viewed as one of the key outputs of an IA department and an explanation of these efficiencies would therefore be useful (necessary) to enhance insights gained from the DEA model applied to the twelve months. To assist in this explanation a multiple regression model was employed in which the efficiency score obtained from the DEA computations for the audit projects was used as the dependent variable. Following a description of the models and data, the results are discussed and notes are made of certain aspects pertaining to the department reviewed.


Introduction
One of the essential elements of the managerial control function is performance evaluation. In the evaluation of profit-seeking organisations, management often utilises traditional aggregate performance indicators such as return on investment (ROI), residual income (RI), profitability, etc. The organisation's performance is then usually compared with itself over time or by comparison with other organisations in the same industry. Non-profit organisations however, are faced with special challenges during performance evaluations. This is because non-profit organisations often have less clear-cut objectives. This is true not only for non-profit organisations but also for nonprofit or service departments within organisations. For example, an Internal Audit (IA) department may be established within an organisation with the specific objective to reduce costs by providing a risk identification,analysis and -solving service. There is an element of conflict in this objective as it does not provide revenue to the organisation but it focuses on cost savings. In performing their duties, IA departments, just like other departments or organisations, use resources to provide their services. These services are a measure of output for the IA department and comparative efficiency assessment of how well inputs are used to produce these services, then, becomes an important measure of performance.
There are also other reasons why efficiency assessment for IA departments is important. Firstly, IA departments are service departments and the issue of whether such departments are reasonably efficient is an ongoing concern for management. IA departments sometimes account for substantial employment and investment and contribute to the growth and economic strength of an organisation. Understanding the determinants of their efficiency then becomes relevant to management to determine whether IA departments deliver sustained contributions to the well being of an organisation. Secondly, the efficiency of an IA department will influence decisions made by policy makers within the organisation in terms of the control of excessive expenses, search for better resource utilisation etc. Efficiency evaluations may, in the third place, also impact the IA department's planning function and performing of audit projects. By examining annual changes in efficiencies, IA management may get additional insights that may lead to improvement (change) of audit plans, resources allocated to specific audit areas, types of audits performed etc.
To address the above-mentioned aspects, this study investigates the combination of Data Envelopment Analysis (DEA) methodology and regression analysis to evaluate the efficiency of an IA department over twelve consecutive months.
The remainder of this paper is organised as follows. The IA department selected for the case study is briefly introduced in the next section. A description of the methodology used and the problem specification is presented next followed by modelling results and a discussion of these results. The paper is concluded with some general comments and a brief overview of DEA is given in Appendix A.

Background
The study described here concerns the IA department of an international gold mining company with operations in all the gold producing provinces of South Africa as well as considerable operations in North Africa, North and South America, and Australia. The IA function is centrally managed from the company's head office in Johannesburg with regional offices in the Free State and North West provinces. For the purpose of this study one of the regional offices in the North West province was selected for evaluation of efficiency. This choice was based on a management request as well as the fact that the environment (staff, infrastructure, projects, etc.) was reasonably stable. Management of the office in terms of administration, planning, execution and documenting projects was well established and the required data could be obtained fairly easily from a management information system. The staff complement of the office consists of a regional audit manager and twelve audit staff members ranging from junior auditors to middle management level. Two audit staff members work half day and the secretary also works half day. The office performs on average 65 audit projects a year, which originates mainly from the annual audit plan, management requests and special investigations e.g. fraud.

Methodology
Data Envelopment Analysis (DEA) is a non-parametric methodology used to calculate a measure of efficiency for an individual 'Decision Making Unit' (DMU) with respect to the other units within the same group. DMU's are regarded as the entities responsible for converting inputs into outputs and the measure is then based on the ratios of linear combinations of the inputs and outputs of the units. To optimise the efficiency of a DMU with respect to the others, mathematical programming techniques are used. The DEA approach was developed by Charnes, Cooper and Rhodes (1978) and has since been applied in a number of areas. Some of these areas are referenced by Wong and Beasley (1990) and include education, health care, banking, local governments, electric utilities, tax collection etc. In the field of IA, Anderson, Cooper and Lockhart (1994) and Metzger (1994), among others, applied the technique to evaluate the efficiency of operational audits. Mathematical details of DEA do not form part of this paper and is limited to a brief overview in Appendix A. Good descriptions of the technical details of efficiency estimation using DEA and an exposition of details of the various models within the DEA framework can be found in Charnes, Cooper, Lewin andSeiford (1994), Fried, Knox Lovell andSchmidt (1993) and Seiford and Thrall (1990).
DEA requires only input and output quantities for computational purposes. These computations yield a firstorder measure of relative efficiencies with no real explanation of the variations in such efficiency patterns. Furthermore, the efficiency estimates are made with the assumption that inputs, outputs and operating characteristics are homogenous. Different DMU's, or in this case, different time periods or different audit projects may be affected by different characteristics (which may include the inputs and outputs). It seems therefore appropriate, and necessary, to perform a second-order analysis. In the case of this study, the 'follow-up' analysis can be done through the use of multiple regression analysis. Due to the constraint of degrees of freedom, it is not appropriate to perform a multiple regression analysis on the twelve time periods. However, the importance of the DEA results for the audit projects warrants further investigation. A multiple regression analysis can be performed where the efficiency scores for each audit project is used as the dependent variable in a regression model with a number of independent variables representing factors likely to impact on efficiency performance of the chosen audit project. This second-order analysis will then ultimately assist in explaining efficiencies for the twelve time periods.

DEA problem specification
The first objective of the case study was to try to determine relative efficiency measures for twelve consecutive time periods using DEA. The period January 2000 to December 2000 was chosen and each month was treated as a different DMU.
There is a concern amongst some researchers that technical changes can take place when working with different time periods. For example, staff experience might be at a higher level in the last period than in the first period. Thanassoulis and Emrouznejad (1996) have proposed a method to deal with such cases. The method is based on input/output 'paths' mapped out by operating units over time. This concern was not dealt with in this paper as the total time period used for modelling purposes was relatively short (one year) and would have little influence on possible technical changes.
Input and output specification relied on discussions with IA management and requests received from them. The IA department makes use of a computerised management information system and, as would be expected, priority has been given to data relevant to IA operations. The specification of input and output was therefore a compromise between desirable model formulation and available data.

Input variables
In most operations where staff is employed, labour is one of the major inputs in terms of its effect on services (output) and cost. The contribution of labour for each month was measured as hours available for each month, i.e. number of staff x number of working hours per day x working days for the month. The second input was specified as a monetary value (financial resources/input) related to the actual cost per month. To be able to express the cost on a monthly basis, the actual hours for that specific month was expressed as a percentage of the total actual hours and this percentage was then used to determine the cost for the month. Cost in this case means the 'cost to the company' and would therefore include all cost allocated to a staff member.
In the literature, see for example Anderson et al (1994), other input and output variables pertaining to IA departments are also used with success. The number of auditors and the average audit team experience level are popular choices as input variables. It was decided not to use these inputs as the case study is concerned with the evaluation of only one office over time and it was felt that the number of auditors and the experience level would remain more or less constant over this time period.

Output variables
The most important, output variable acknowledged by senior management was related to audit reports produced. The first result of any audit project is an audit report. This report contains information on the objective and scope of the audit, the methodology used to perform the audit, a list of strengths and weaknesses of the area audited and applicable findings and recommendations. In many cases, audit reports are the only visible evidence of work performed that is presented to management and audit committees and therefore is often used by them to evaluate the service of the IA department. It seems therefore reasonable to say that the primary objective of an internal audit office is to perform efficient audit projects -this means that to measure the office's economy, efficiency and effectiveness, the efficiency of projects should be taken into account. To do this an additional DEA model (for which input and output variables will be defined subsequent to the main model specification) was formulated to estimate the relative efficiency of specific projects performed during the period January 2000 to December 2000. The obtained efficiency scores per project were then averaged for each month and used as an output variable for the main DEA model. The second output variable used was the total number of actual hours spent on projects per month. The last output variable, called total number of findings with root cause analysis (RCA) per month, was included at the request of IA management. Auditors can produce a number of findings but efficient findings are regarded as those with explanations and analyses of the cause and effect of the situations that lead to the finding. This is a variable that should be treated with care, as an audit project can be highly efficient even if there were no findings. The inclusion of the variable will be further explained and motivated when defining the additional DEA project model.
As in the case of input variables, the literature provides also other choices for output variables. One of the outputs frequently used is the estimated monetary value saving by the audit team. It was agreed with IA management that this output variable is inappropriate, simply because the objective of an audit project is not necessarily to save money but rather to identify and address control risks. Saving of money is often a result of an audit but is not always easily quantifiable or even recognisable.

Additional DEA model to estimate project efficiency
A total of 45 projects were selected to determine the efficiency that was used as an output measure in the main DEA model. Inputs for the project evaluations were identified as the number of audit staff working on the project, the actual hours taken to complete the project and the actual cost of the project. The actual cost was calculated as the number of hours each employee booked against the project times the 'cost to company' rate for that employee.
Four outputs were used. Each one of the four outputs was requested by IA management to be included in the model and was already used in practice by them to evaluate the audit function.
The first two outputs identified were the number of significant findings for a project (a finding is regarded as significant in this study, and in the company reviewed, if the finding causes a control objective not to be achieved or to be partially achieved and if the auditee agrees with the finding), and the number of findings with RCA. As mentioned earlier, these type of variables (number of findings) should be treated with care and may be criticized by some researchers as an efficient audit does not necessarily have to produce any findings -other than maybe a finding that the area audited was well-controlled with no weaknesses. Why were these variables then included in the model and what contribution can they make to the evaluation of an audit project? Anderson et al. (1994) used the following four (out of five) output variables to evaluate performance audit projects with the DEA approach: They have based their selection of variables on the fact that 'These variables are mentioned frequently in the audit literature as important indicators of performance'. It is clear that none of these variables would have significant values in a well-controlled area with little or no weaknesses. They have also pointed out that an alternative formulation of a DEA model is possible should some of these variables be zero -this was not necessary in the case study described in this paper as all the variables were non-zero.
More evidence on the use of these type of variables as performance indicators can be found at the website In Part 1, the number of recommendations presented during the 2000 fiscal year is reported on as an output amongst other inputs and outputs. Part 2 gave as performance indicators the following, amongst others: the number of recommendations reported to management, number of recommendations accepted by management and the number of recommendations implemented by management. All these outputs and performance indicators are directly linked to the number of findings and will be influenced by efficient audits in well-controlled areas with no or only a few findings.
Another reason for accepting the variables as valid lies in the nature of DEA models. A DEA model captures how multiple inputs affects simultaneously the multiple outputs of the transformation process carried out by the unit (audit project) being assessed. This means that an audit project can still be evaluated as being efficient with a low or zero number of findings whenever the other output and/or input variables are measured high for the outputs or low for the inputs. Investigations into the audit projects used in this case study have shown that such audits in well-controlled areas with no or only a few weaknesses usually comply with this assumption. A more detailed explanation on this and other aspects regarding the nature of DEA models can be found in Thanassoulis (2001).
The final and maybe most important reason for including these two variables in the model, was the fact that IA management, in the organisation reviewed, used them in practice to evaluate the performance of the audit function. An interesting observation was that IA management uses them to assist with the planning function. IA management investigates audit projects with no significant findings and should it be an audit performed in an area with no problems, management would question the inclusion (or frequency) of the same audit or type of audit in future audit plans. The objective is to focus the audit effort on those areas where control weaknesses are and not to spend too much time in areas with no (major) problems, thereby enhancing the efficiency and effectiveness of the audit function.
Despite the above arguments in favour of the inclusion of variables linked to the number of findings, it remains an issue that should be treated with care. The next section where the modelling results are presented also introduces a sensitivity analysis exercise that was performed to ensure that the outcomes of the models are stable and not negatively influenced by any such variables. The process to perform the sensitivity analysis is described in the next section.
The other two output variables used in this model were a project rating based on the type of audit and a satisfaction rating. There were no actual satisfaction ratings for the audit projects available and for the purpose of the study a satisfaction rating for each project was calculated based on the time and financial resources budgeted against the actual time and cost to complete the project.
To summarise, the selection of input and output variables used in both the main DEA model and the additional model to estimate project efficiency was based on the following: • IA management and staff agreed that these variables were important and applicable to their audit projects and the success of the IA department.
• The variables are mentioned in audit literature and other studies as important indicators of performance.
• Measures (for each of the 45 audit projects) for these variables were available from the office reviewed. Table 1 summarises the input and output variables used in the two DEA models. The study was performed using Excel for data preparation and data input. Mathematical programming was done using OSL (Optimisation Subroutine Library) from IBM.

Project efficiency
It was decided to split the 45 audit projects into two groups to ensure that the same type of audit is compared. Group 1 represents those audits that are generally not planned for, like management requests and fraud investigations, while Group 2 consists of the audits that form part of the annual audit plan. DEA was then applied to each of the 45 audit projects. Concise results are presented in Table 2. Each row of Table 2 represents the solution to a linear program, which maximises the efficiency rating of the corresponding audit project under the constraints dictated by the output/input relationships operating in the complete data set. Of the 45 audit projects, ten have a maximum efficiency rating of one. In the third column, next to the relatively inefficient audit projects, appears the corresponding efficiency reference set. This is the subset of relatively efficient audit projects to which the audit project in question has been most directly compared in deriving its efficiency rating. The last column indicates the number of times each efficient audit project appears in the reference set for an inefficient audit project.
To ensure that the results in Table 2 are stable and not negatively influenced by variables such as the number of findings, a sensitivity analysis was performed. Schlamp and Fecker (2002) described a sensitivity analysis process that they applied in a study to examine the overall efficiency of Swiss commercial banks. The same process was followed and briefly entails the following: Starting with the audit project (DMU) having the highest reference set frequency, efficient audit projects are successively deleted from the set of audit projects. After each step, a DEA analysis is performed on the remaining audit projects, which are then ranked based on their overall efficiencies. The initial result in Table 2 will then be considered stable if the ranking does not change dramatically by removing individual audit projects from the dataset. To test this, the rank correlation coefficient between the original ranking and the current ranking is calculated. If the rankings agree fully, the rank correlation coefficient is unity. If there is no correlation, it is zero.
Below are the results of the sensitivity analysis. The rank correlation coefficient remains high for both groups of audit projects, even after deleting all 5 efficient audit projects in each group. The results are hence considered to be stable.

Results of sensitivity analysis
Audit project (

Regression analysis
In conjunction with the IA management team, eight possible explanatory variables capturing the characteristics of an audit project were identified. Six of them, together with two input variables from the DEA model, were used in the regression analysis. The two input variables from the DEA model (see Table 1) were the number of full time equivalent audit staff working on the project (PERS) and actual hours to complete the project (HOURS).
The chosen six explanatory variables were all qualitative variables and indicate whether the specific factor would have an impact or not. They were defined as follows.
EXP -the level of experience of an individual auditor working on the project. In certain areas, audits can be conducted fairly easily without any specific experience of that area while others require a certain level of experience.
CULT -the working culture in specific areas may impact the efficiency of audit projects and requires from the auditor a certain level of knowledge of how business is conducted, management and employees' attitudes etc. It refers to the way that people in the area audited perform their tasks.
GEO -the location of auditors plays a role in cases where the area audited is geographically away from the auditor's home office.
TYPE -the type of audit to be performed. Various types of audits, e.g. management request, follow-up, fraud, executive request etc., are performed. Depending on the type of audit, certain aspects such as planning, priority, resource allocation etc., might be influenced.
MAT -auditee maturity. In some instances the auditee is more receptive towards auditors and their work. They have a higher level of understanding of the audit function and the need thereof. This often results in better co-operation and faster results. This variable refers to the auditee's experience, perception and attitude towards auditors and the audit function and should not be confused with the variable CULT. The variable CULT refers to the auditee's way of performing their duties regardless of what they think of the auditors or audit function.
REQ -audience/requestor. The requestor of the audit also influences audit projects. Requestors are closely related to the audience who will receive the final audit results.
The efficiency scores from the Project DEA model were used as the dependent variable in a multiple regression model fitted to the eight explanatory variables. Ordinary least squares (OLS) regression was used and the results are given in Table 3. The DEA results (Table 2) and regression analysis results (Table 3) present a lot of information that was used to analyse issues surrounding specific audit projects. The DEA results were also used as one of the output measures to determine the IA department's efficiency. These results will be further discussed in section 6.

Office efficiency
Using the results from Table 2 and the inputs and outputs described earlier on, DEA was then applied to twelve consecutive months, treating each month as a different DMU. Two additional constraints were introduced to provide for value judgements from IA management. The constraints were i) The weight of actual hours worked should be at least 25% greater than the weights of the efficiency ratings for the audit projects but not exceeding 40%.
ii) The weight of the RCA should be at least 30% of the weights of the actual hours worked.
Consensus on the above weights was reached amongst those IA managers and staff most familiar with the situation being modelled. Questions, such as those suggested by Wong and Beasley (1990), were used in the process. The questions were of the form 'Do you think that the importance of input/output measure i could be as low/high as x%' and 'Should the importance of input/output measure i be allowed to be as low/high as x%'.
The result of this exercise is summarised in Table 4. Five of the 12 months have a maximum efficiency rating of one with November and December rated as the most inefficient months.

Project efficiency
Apart from identifying inefficient DMU's, DEA can also provide additional insight about the degree of inefficiency for each DMU. Consider audit project 16 from Group 2 in Table 2. The efficiency rating is 0,8990 and the reference set consists of audit projects 2 and 26. The solution to the linear program (not presented in this paper) for project 16 gave dual prices of 0,64286 and 0,42857 for the reference audit projects 2 and 26 respectively. Suppose we create a composite audit project by combining 0,64286 of audit project 2 with 0,42857 of audit project 26, i.e. 0,64286 x 1 + 0,42857 x 1 x i -input i .
. . . x n x n Doing this, the following result was obtained for audit project 16: The results show that the actual inputs used for audit project 16 were greater than the derived composite efficiency reference set. Specifically, the output achieved for audit project 16 could have been achieved using 0,86 less staff members, 20,05 less hours and R1040,18 less cost. Each variation can now be investigated.
Similar analyses were done for all the inefficient audit projects, which enabled IA management to investigate individual audit projects. It was also possible to make general observations from the DEA results that pertain to the audit function in general. The planned projects (Group 2) were generally more inefficient than those not planned for (Group 1). This could be explained by findings such as the satisfaction ratings that were on average lower for the planned projects than for the unplanned ones (20 against 98). The number of staff working on planned projects was on average higher than for the unplanned projects (6 against 4) and the average number of hours spent on planned projects was higher (194 against 81).
The above general findings from the DEA model highlighted certain aspects that were used as a starting point by IA management to re-consider and re-evaluate specific functions within the IA department. Some of these internal functions include: • the budgeting and planning process for audit projects (use data from completed projects -both efficient and inefficient -as a guideline. Efficient projects would provide guidelines on how to budget and plan for future projects, whereas inefficient projects would provide experiences to be avoided for similar projects); • the control of planned hours (who may book hours against a project, stricter control on the scope of the project); • the control of project cost (level of staff booking hours against a project); and • keep projects as small as possible in terms of number of hours (sub-projects that form one large project can be controlled more easily and completed more efficiently, however, care should be taken not to lose sight of the overall project objectives).

Regression analysis
From the regression results in Table 3 it can be observed that the magnitudes and signs of the coefficients for the explanatory variables are consistent with expectations. The negative coefficients for PERS and HOURS support the view that a higher number of staff and a higher number of hours will reduce the efficiency.
The remaining six variables were qualitative with values of 0 or 1. Their coefficients indicate the additive differential effects on the height of the regression line for any given levels of PERS and HOURS. For example, the positive coefficient of EXP indicates how much higher the efficiency would be if an experienced auditor is used, while GEO's negative coefficient indicates how much lower the efficiency would be if an auditor's home office is not located at the site where the audit is performed.
Simply by looking at the coefficients of the regression model it would be worthwhile to IA management to ensure that auditors have the necessary experience and a reasonable knowledge of the working culture in the area that is being audited. This is especially true for those audits that require a higher number of hours to complete. To achieve this, management may want to consider more training for less experienced auditors or having junior auditors performing their audit work with the help of a more experienced senior auditor.
The type of audit to be performed should also be carefully considered when planning or budgeting for an audit project, while the remaining variables (GEO, MAT, and REQ) would have a lesser impact.

Office efficiency
The ultimate goal of the study was to express an opinion on the efficiency of the IA department over twelve consecutive months. The results of the DEA model summarised in Table  4 provided a framework for this.
As in the case of the audit projects, each relatively inefficient month was analysed by constructing a composite time period based on the appropriate reference sets. An output-oriented approach was used and below is a summary of the averaged figures of the analysis. The analysis shows that on average, during the inefficient months, 18,76% more hours should have been spent on project work, project efficiency should have been 22,27% higher and 137,98% more findings with RCA should have been produced.
Further explanatory comments on these results are as follows.

•
The high percentage deviation of actual hours for November and December were the result of study leave taken during November by staff who had to write official examinations during that month while December is traditionally the month where most of the staff are on annual leave.
• As discussed earlier, the project efficiency was obtained from the project DEA model. The same comments and recommendations for this model are applicable here.
• The figures for RCA should be treated with care. To put the high average into perspective it should be mentioned that the number of RCA findings was generally small and the 137,98% represent on average only about 3 RCA findings. It is advisable to analyse each month separately because the number of RCA findings per month is the total number for all those projects that were completed in that specific month. This means that whenever there is a month with a low completion rate of projects, the number of RCA findings will also be low. In addition, one should also be careful where efficient projects were completed with no or a low number of findings (an audit project can be highly efficient even if there were no findings). Taking all this into account it still indicates that there is possible room for improvement.
What does all this mean to the IA department?
As far as the hours are concerned, it appears as if there are no serious concerns. Acceptable explanations exist for the high differences during November and December. Deviations for the other inefficient months would be reduced when changes to the planning and budgeting process (discussed under project DEA results) are made. The project efficiency played a major role in the determination of monthly efficiency -to improve monthly efficiency, project efficiency will have to be improved. The comments made earlier on regarding the project DEA model are applicable here.
With regards to the RCA variable, IA management may want to consider a more rigorous quality assurance process to ensure that, where appropriate, no findings without RCA are included in final reports. This also forms part of the project efficiency and should be addressed with project issues raised in the project DEA model.

Conclusion
In this paper a case study was described where DEA methodology was combined with regression analysis to evaluate the efficiency of an Internal Audit department over twelve consecutive months.
The results obtained indicated that measuring performance in a non-profit department should be treated cautiously. One cannot simply look at the efficiency of performance itself, but the identification and evaluation of relevant characteristics should also form part of the exercise, as all of them are important factors associated with efficient performance.
The results of a specific DEA model should also be treated with care. DEA should be used (as in this case study) as a catalyst for change. It offers an opportunity to communicate possible changes and enhancements to the different business processes. Although there is no hard and fast rule to determine the accuracy, validity and success of a DEA model, the discussion and communication of results to management will be indicative of the value of the modelling process. In many cases, management may be able to supply a straightforward explanation to a low efficiency rating, e.g. the low efficiency rating for December that was due to annual leave taken by staff. A more valuable contribution, however, will occur when the modelling results contradict management's accepted knowledge about a certain process, e.g. the budgeting and planning process to allocate resources to audit projects was not as efficient as management believed. In these cases, in-depth investigations may be prompted and this may lead to information about processes that until that time had been ignored. DEA should therefore be seen as part of a process of information generation and understanding.
On a high level, this case study -as part of an information generation and understanding process, has suggested that IA management (for the audit department under review) consider adjustments to the following: • budget and planning process for audit projects; • control of planned hours and project cost once an audit project has commenced; • taking into account the experience of auditors and cultural diversity for certain types of audit projects; and • quality assurance process for final audit reports.