Risk factors affecting software projects in South Africa

This paper reports on an analysis of risk factors relevant to South African software projects. Seven of the most widely cited studies in the research literature regarding software project risk were evaluated along with a detailed examination of the 53 risk factors developed by Schmidt, Lyytinen, Keil & Cule (2001). Forty completed questionnaires, submitted by software project managers, were analysed.


Introduction
The development of information technology (IT) related projects have become a critical aspect of most businesses because of organisational dependence on computer-based systems to remain competitive (Jiang, Klein & Ellis, 2002).As business reliance on software grows, so does the business-related consequences of software failure.Billions of dollars are lost on cancelled projects, late delivery, overbudget delivery and limited functionality.Data from the Standish Groups' 2002 survey showed that 51% of information systems (IS) projects overrun their schedule and budgets, 15% are cancelled and only 34% are completed on time and within budget (Standish Group International, 2002).
Among advocated methods for improving software project success, the concept of software project risk management has gained prominence (Schmidt, Lyytinen, Keil & Cule, 2001).Risk management is aimed at taking counter measures to either prevent risks from affecting the project or to reduce their impact (Heemstra & Kusters, 1996), and should be viewed as a fundamental component of the project management process.
Software project risk has been defined as the product of uncertainty associated with project risk factors and the magnitude of potential loss due to project failure (Barki, Rivard & Talbot, 1993).Thus, the key elements to be controlled are the project risk factors (Schmidt et al., 2001).Furthermore, the relative importance of these risk factors needs to be established.As DeMarco and Lister (2003) stated, 'in this period of turmoil, a willingness to run risks is utterly essential'.
According to Chapman and Ward (2003) a key motivation is the identification of opportunities to change the base plans and develop contingency plans in the context of a search for risk efficiency, taking a level of risk that is appropriate, with the view to long-term corporate performance maximisation.
Risk management is not a one-time activity, but an ongoing process of identification, assessment and action, which needs to be well integrated into every part of IS management.Effective risk management involves taking a holistic approach to risk, developing a risk management policy, establishing clear accountabilities and responsibilities, balancing risk exposure against controls, being open about risks to reduce conflict and information hiding, enforcing risk management practices and learning what works and does not from past experience (Smith, McKeen & Staples, 2001;DeMarco & Lister, 2003).

Core risks of software projects
A project is usually deemed as successful if it meets requirements (of measures such as functionality, reliability, maintainability, portability, efficiency, integration and operability) and is delivered on time and within budget (Addison & Vallabh, 2002).Variation in success of software projects is explained by a wide variety of technical, economic and behavioural factors (Jiang & Klein, 2001).One consistent factor influencing project success is the various risks associated with developing projects (Jiang & Klein, 1999).Consequently, risk identification has been the topic of many research endeavours (Jiang, Cheng & Klein, 2002;Jiang & Klein, 2001;Boehm, 1989;Addison & Vallabh, 2002;Kloppenborg & Tesch, 2004;Schmidt, et al., 2001;Keil, Cule, Lytinen & Schmidt, 1998).

Boehm's 'top ten' risks
In terms of previous efforts to identify risk factors, Boehm's work (1989) has probably had more influence on the practitioner community than any other (Keil et al., 1998).Boehm's 'top ten list of software risk items' was built upon his experiences in the defence industry in the 1980s.Figure 1 lists the 'top ten' risk factors as identified by Boehm (1989).A study conducted by Keil et al., (1998) employed a variation on the traditional Delphi survey approach.The study was designed to elicit opinions from a panel of experts through iterative, controlled feedback.Recruiting from among experienced project managers in the USA, Hong Kong and Finland, three expert panels were formed.Figure 2 lists the risk factors identified by all three panels ordered by relative importance.Rivard and Talbot (1993).The results of this study produced a list of nine ranked software project risks, shown in Figure 3.

Oz and Sosik's 'top five' risks
In a survey of IS executives, Oz and Sosik (2000) collected quantitative and qualitative data about reasons why IS projects are abandoned in an extensive literature search (Kloppenborg & Tesch, 2004).Thirty items were identified as reasons for IS project abandonment from trade and academic journals.Five factors emerged from the factor analysis of surveys; lack of corporate leadership, poorly communicated goals/deliverables, inadequate skills and means, poor project management, and deviation from timetable/budget.
Addison and Vallabh's 'top ten' risks Addison and Vallabh (2002) conducted a study in South Africa and identified the ten most important risk listed in Figure 4.In a subsequent study similar to Keil et al. (1998), Schmidt, et al. (2001) developed an 'authoritative' list of 53 risk factors (including 27 factors derived from the literature) using input from a multicultural set of 41 practicing project managers (Kloppenborg & Tesch, 2004).Three panels were formed on the basis of their cultural backgrounds.The study was conducted using a three phase Delphi survey process with nine panellists in Hong Kong, 13 panellists in Finland and 19 panellists in the USA.
Comparison of this list of risk factors with those generated in previous studies suggested that this list is fairly comprehensive.From this list of risk factors, a set of risk factor groups was derived.The groups created were found to be consistent with the five risk groups previously established in Barki et al. (1993).These risk groups included corporate environment, sponsorship/ownership, relationship management, project management, scope, requirements, funding, scheduling, development process, personnel, staffing, technology, external dependencies and planning.In addition to developing this comprehensive list of risk factors, the authors also determined which of those risk factors were considered most important (Schmidt et al., 2001).Listed in Figure 5 are the eleven factors common to all three panels in order of their average relative ranks.A comparison between the lists of ranked risk factors from all 7 studies discussed in this paper does not provide any overall agreement as to the most important risks.Not one single risk factor was present in all 7 lists.The significant overlaps that can be identified between the different lists are shown below: • 'Lack of team expertise/ Lack of required knowledge/skills' was present in 5 of the 7 lists (Jiang & Klein, 2001;Schmidt et al., 2001;Keil et al., 1998;Addison & Vallabh, 2002;Oz & Sosik, 2000).
Due to the lack of consensus between the ranked lists of 'top' risk factors from the seven studies considered, alternative means of merging the risk factors was considered.By increasing the coverage of possible risks using Schmidt et al.'s (2001) 'authoritative' list of 53 risk factors, a certain level of amalgamation was possible.The authors compared Schmidt et al.'s (2001) list of 53 risks to a merger of other risk item lists.The merger consisted of lists produced by Boehm (1989), DeMarco and Lister (2003), Schmidt et al. (2001), Keil et al. (1998), Addison and Vallabh (2002), Oz and Sosik (2000) and Jiang and Klein (2001).
A union of these seven lists was used for this comparison.A comparison between these various lists was completed and the analysis suggested that Schmidt et al.'s (2001) list was more encompassing, and, due to its elicitation procedure, also more reliable.

Differences in risk factor lists
The authors expected some differences in Schmidt et al.'s (2001) list and the union of previous lists.Given the radical changes that have occurred in the IT and business environments, it was assumed that some risk items may have remained relatively stable whilst others have changed in importance.
The first subset of risk factors considered were those that were expected to remain stable over time.Although there were 31 risk factors identified by Schmidt et al.'s (2001)  Another major topic dealt with inadequate project management methodologies and project management skills.This identified an awareness for disciplined management practices and recognised their absence as an important source of risk (Schmidt et al., 2001).Although such factors have not been recognised in previous risk management research, they had been previously mentioned in process improvement research (Humphrey, 1989).
A further new risk topic concerning systems development dealt with the turbulence of the business environment including changes in the business ownership or senior management.
Although organisational politics and organisational culture were not reflected in the risk lists, these had been long standing issues in the IS literature (Kwon & Zmud, 1987;Chengalur-Smith & Duchessi, 2000) and thus did not represent new findings.
Overall, the list of 53 risk factors identified in Schmidt et al.'s (2001) study provided a useful starting point for further research as the list of risk factors produced was deemed 'authoritative' and 'fairly comprehensive' in the literature (Kloppenborg & Tesch, 2004).Furthermore, a comparison of the list of risk factors generated by Schmidt et al. (2001) with those reported in previous studies (Jiang & Klein, 2001;Schmidt et al., 2001;Addison & Vallabh, 2002;Oz & Sosik, 2000;Boehm, 1989;DeMarco & Lister, 2003) suggested that the Schmidt, et al.'s (2001) list was a fairly comprehensive one, including, with very few exceptions, practically all of the major risk elements that were identified in the other studies considered in this paper.

Research definition
As shown in the literature review, numerous authors including Schmidt et al. (2001), DeMarco and Lister (2003) and Addison and Vallabh (2002), have published works on software risk factors.One of the more recent contributions to this field that has received much critical acclaim is the book Waltzing with Bears in which DeMarco and Lister (2003) identify five core risks of software projects.While the literature does cover software risk extensively, the question remains which of these abundant risk factors are relevant to South African software project managers.

Hard versus soft risks
Each identified risk factor was classified as either a 'hard' risk or 'soft' risk.The framework used for this classification scheme is shown in Figure 6.Risks items that related to quality, cost, time, requirements or methodology were classified as hard or mechanical risks, whilst factors related to people, relationships or change were classified as soft or dynamic risks.Of the 53 risk factors produced in the questionnaire, 28 were classified as soft risks and 25 were classified are hard risks.This framework (see Figure 7) identifies three main dimensions of competency -PM Knowledge, PM Performance and Personal Competency -and provides guidelines for assessing generic project managers in each of the three areas.
Project manager experience was evaluated according to how well these 3 dimensions were satisfied.

Research objectives and hypotheses
The following objectives and hypotheses were developed to satisfy the aforementioned research problem.

•
Objective 1: Determine whether experienced project managers perceive different risks to be important than inexperienced project managers.
This objective can be tested by the following hypotheses: Hypothesis 1 H 0 : Experienced PMs perceive the same risks to be important as inexperienced PMs

•
Objective 2: Identify whether soft risks are perceived to be as important as hard risks This leads to the following two hypotheses: Hypothesis 2 H 0 : All PMs perceive soft risks to be as important as hard risks

Questionnaire design
Section A of the questionnaire was designed to distinguish between experienced and inexperienced project managers.The questions were derived from the Project Management Competency Development Framework (2001).
Section B of the questionnaire listed 53 risks that were identified by Schmidt et al. (2001) and these were grouped numerically under the following headings: These headings had been used in the paper by Schmidt et al. (2001) but the questionnaire used in this study did not explicitly state these headings in the interest of fitting the entire Section B onto one page for ease of answering.The numbering did, however, suggest that the questions belonged under logical categories.This was done to increase usability and make it easier for the project managers to answer this section.
In the column labelled 1, respondents were asked to choose the risks they felt were important.Those risks identified in column 1 then had to be ranked from one to ten in column 2 -one being the most important.In the appendix of the questionnaire a glossary with a short description of each risk as well as an example of how to answer section B, was provided.

Sampling approach
The sampling method chosen was 'convenient' sampling, which involved distributing the survey to potential respondents within a target population who can be contacted.The questionnaire was first emailed to 2 targeted project management organisations, both involved in the software project management business, who then distributed the questionnaire to software project managers.A total of 37 usable responses were initially received.These project managers either worked fulltime for or consulted in large corporations.
After the data was collected and some provisional analysis done, it was found that 20 of the respondents were experienced project managers and 17 inexperienced.Due to the fact that a significant aspect of this research involved comparing the perceptions of inexperienced and experienced project managers, an equal number of respondents from each group would 'give the most power to detect a difference between the groups' (Hopkins, 2001).Therefore a further request was sent to the 2 organisations which resulted in further returns.The first three returns from inexperienced project managers were accepted.Further returns were not considered.The total number of valid returns was thus 40, consisting of 20 experienced and 20 inexperienced respondents.

Perceived importance of software risk factors
Section B in the questionnaire asked project managers to first mark the risks that were important in Column 1 before ranking the important risks in Column 2. The results of Column 2 were used to develop the list of top ten risk factors.The column 1's data was analysed first.
As stated previously, the risks in Section B of the questionnaire were grouped numerically to indicate that they could be classified under certain risk areas.Figure 8 shows the percentage of risks which were identified as important under each risk area.From the figure, some interesting trends can be observed.Even though the sample consisted of an equal number of experienced and inexperienced PMs, it is observed that in general experienced PMs perceived more risks to be important than inexperienced PMs.The risks grouped under the Personnel risk area were thought to be significantly more important by experienced project managers.It is also interesting to note that all the risks under the aforementioned area were classified as soft risks.However, all risks grouped under the area Relationship Management were also classified as soft risks and in this case, inexperienced project managers perceived this area to be slightly more important.
From Figure 8 the main differences in perception between experienced and inexperienced PMs lay in the fact that experienced PMs perceived a greater number of risks to be important than inexperienced PMs.Two notable exceptions were risks falling under Planning and Relationship Management which were areas found more important by inexperienced PMs.Whether the differences in perceptions were statistically significant or not, will be analysed in the next section.

Perceive importance of hard and soft risks
To ascertain whether hard and soft risks were perceived differently by project managers with varying degrees of experience, the number of times a risk was marked as important was aggregated and categorised as either hard or soft.Of the risks marked as important by experienced project managers, 54% were hard risks and 46% soft.For inexperienced project managers 52% of the risks marked as important were hard risks with the remaining percentage soft risks.From the above figures, there does not seem to be a significant difference in the perceptions of hard and soft risks by experienced and inexperienced project managers.This was tested for statistical significance and is reported later.

Findings Objective 1 -Level of experience
The purpose of objective 1 was to determine whether there was a difference in how project managers of different levels of experience perceive software risk.Hypothesis 1 tests this statement.
Hypothesis 1 H 0 : Experienced project managers perceive the same risks to be important as inexperienced project managers H 1 : Experienced and inexperienced project managers perceive different risks to be important.
To test this hypothesis, the Chi-Squared Goodness of Fit test was used.This method compares a column of observed frequencies to a column of expected frequencies and evaluates whether the differences are significant.For the purposes of this research, the observed column consisted of the number of times each risk was marked as important by an experienced project manager, while the expected column contained the same type of data but for inexperienced project managers.To satisfy the required condition for this test, the Rule of 5, which states that the frequencies in each column must not be less than 5, the frequencies associated with each risk were pooled into their respective risk areas.For example, the number of times the first four risks were marked as important was pooled under the risk area Corporate Environment.This was done for both the experienced and inexperience project managers.Conclusion: There is sufficient evidence at the 5% significance level to reject H 0 and infer that experienced project managers perceived different risks to be important compared with inexperienced project managers.
The Chi-Squared test confirmed what was speculated after the general analysis of the data; namely that experienced and inexperienced project managers have different perceptions of the risk factors.
Objective 2 -Hard versus soft risks The purpose of objective 2 was to determine whether hard risks were perceived to be as important as soft risks.
Hypothesis two and three were used to satisfy this objective.
Hypothesis 2 H 0 : All PMs perceive Soft Risks to be as important as Hard Risks.
H 1 : Experienced and inexperienced project managers have different perceptions of the importance of Hard and Soft Risks.
A two-way Analysis of Variance (ANOVA) test was used to test hypothesis 2. This method made use of the number of times a risk was marked as important and grouped this data into two columns for inexperienced and experienced project managers and two rows for hard and soft risks.A limitation of this test was that an equal number of hard and soft risks were required.The list of 53 risks consisted of 28 soft and 25 hard risks.For the purpose of this test, the three soft risks with the least responses were removed.These were Growing sophistication of users lead to higher expectations, Managing multiple relationships with stakeholders and Poor productivity.The first risk was only marked as important by four respondents and only ranked once, the second was only perceived to be important twice and not ranked at all, while the third risk was also only marked as important four times and ranked twice.Leaving these risks out would, therefore, not affect the test significantly but nonetheless remains a limitation of this method.To test hypothesis two, the p value for Factor B -the classification of a risk as either hard or soft -was used.
H 0 : There is no difference between the means of the two levels of Factor B H 1 : The means of Factor B differ.

P -Value: 0,118
Conclusion: There was insufficient evidence at the 5% significance level to reject H 0 .
Hypothesis 3 H 0 : The degree of PM experience and a risk's classification as either hard or soft do not interact to affect its perceived importance H 1 : The degree of PM experience and a risk's classification interact to affect its perceived importance The same ANOVA test discussed above was used.To test this hypothesis, the p value for Interaction was used.
H 0 : Factors A and B do not interact to affect the perceived level of importance H 1 : Factors A and B do not interact to affect the perceived level of importance P -Value: 0,644 Conclusion: There is insufficient evidence to reject H 0 .
Both hypotheses one and two could not be rejected which confirms what was observed from the data -namely that both inexperienced and experienced project managers do not regard either hard or soft risks as more important.The data had shown that for experienced project managers, 54% of the risks marked as important were hard risks while for inexperienced project managers, 52% of the risks marked were hard risks.The ANOVA test revealed this difference not to be significant and had further shown that there is no interaction between the level of PM experience and the type of risk.

Objective 3 -Top ten risks
The purpose of this objective was to develop a list of the top ten most important risk factors.However, since the conclusion from hypothesis one showed marked differences in the perceptions of experienced and inexperienced project managers, two separate top ten lists for experienced and inexperienced project managers were also developed.
Several methods to develop a list of top ten risks were considered.One aspect that had to be kept in mind was the relatively small sample size and the large number of risks in the questionnaire which resulted in a large spread of answers.This meant that a ranking based purely on the mean and standard deviation might not give an accurate reflection of the top risks.To overcome this problem, it was decided to only compare the means of risks that were ranked ten or more times.Risks were ranked an average of seven times and it was felt that by using ten as the cut off, a fair reflection of the top risks would be achieved.This approach was a potential weakness in the analysis.The risks ranked more than ten times were then evaluated on their mean, the risk with the highest mean receiving the highest rank.The mean was chosen to compare the risks since the data would not be affected by outliers as the choices were limited to between one and ten.
In addition, two further lists were developed for experienced and inexperienced project managers.Since the sample consisted of half experienced and half inexperienced project managers, these respective lists of top risks were developed by considering only those risks ranked five or more times.This method of ranking risks was viewed as acceptable statistically.
From Table 1 it can be seen that Lack of top management commitment to the project was undoubtedly perceived to be the most important risk.It received the highest mean (bearing in mind that the data was recorded inversely, with a rank of one being stored as ten) and was also ranked the most times.The second most important risk, Unclear/ misunderstood scope/ objectives, received the second highest mean but was only ranked ten times.When compared to the 5 th most important risk, No planning or inadequate planning, which had a lower mean but was ranked 20 times, a flaw of the method used to rank the risks becomes apparent.
However, overall the method does seem to hold true with the mean decreasing steadily as the ranks become lower and the frequency spread fairly evenly.
As noted previously, two further lists of risk factors were developed due to the difference in perceptions of experienced and inexperienced project managers regarding software risk.These two lists show marked differences in ranked risk items, as was expected.As the conclusion to hypothesis one stated, there is a significant difference in the perceptions of experienced and inexperienced project managers as to the top ten risk factors.
Failure to identify all stakeholders was ranked as the second most important risk by inexperienced PMs but was not in the top ten list of experienced PMs.Similarly, Lack of adequate user involvement, the fifth most important risk for inexperienced PMs was not perceived to be one of the ten most important risks by experienced PMs.Unclear/misunderstood scope/objectives was ranked as the most important risk by inexperienced PMs and was only the 9 th most important risk for experienced PMs.This suggests that as project managers become more experienced, the minimising of these risks becomes more internalised.Suggesting further reasons accounting for the difference of perceptions of experienced and inexperienced PMs is however beyond the scope of this paper.

Conclusions
The top ten software risk factors determined from this research was compared to the findings from the research literature.Table 4 indicates those risk items identified in this study that were present in prior studies and those risks that were not mentioned in previous ranked risk factor lists.
It is interesting to note that lack of top management commitment, which was ranked as the most important risk factor in this study, also received highest rank ratings in both Keil et al.'s (1998) and Schmidt et al.'s (2001) risk lists.Additionally, unclear/misunderstood scope/objectives was ranked first in Addison and Vallabh's (2002) risk list and schedule flaw was present in DeMarco and Lister's (2003) listing of core risks.However, the risks ranked 5, 6, 7, 8 in this study were not cited in previous studies of ranked risk factors.Within the constraints of the small sample, this could indicate that these risk factors are especially important to South African software projects.Possible reasons for different risks being important to South African software projects could be based on the unique organisational culture of South African companies or on the fact that the software industry is less mature in this country than for example in America or Europe.
Further research with a larger sample could increase the statistical validity of these findings.Extending the research to compare risk factors in other industry sectors like the mining and construction industries in South Africa would also be insightful.Poor risk management Oz & Sosik (2000)

Figure 2 :
Figure 2: Keil et al.'s 'top eleven' list of software risk items.(Keil et al., 1998) Jiang and Klein's 'top nine' risks In this study, Jiang and Klein made use of software risk measurement instrument pertaining to various characteristics of a software development project developed by Barki et al.Rivard and Talbot (1993).The results of this study produced a list of nine ranked software project risks, shown in Figure3.

Figure 5 :
Figure 5: Schmidt et al's (2001) 'top eleven' software risk items A comparison of risk item lists An interesting observation was made between Keil et al's (1998) first study and Schmidt, Lyytinen, Keil and Cule's (2001) subsequent study.Although there is not perfect consensus between the rankings of each individual risk, the 'top eleven' risks identified in the earlier study (1998) are all present in the subsequent 'top eleven' risks identified in the later study (2001).

Figure
Figure 6: Hard (mechanical) and soft (dynamic) risk classification The degree of PM experience and a risk's classification as either hard or soft do not interact to affects its perceived importance•Objective 3: Develop a list of the top ten most important software risk factors relevant to South African software projects.

Figure 10 :
Figure 8: Perceived % importance of software risk areas