Performance measurement frameworks in a state controlled research organization: Can the Balanced Scorecard (BSC) be modified?

The development of multiple stakeholder performance measurement frameworks (PMF) is a complex process in the public research sector. The objective of the paper was to investigate whether the performance measurement framework (PMF) of a state owned research organization could be adapted to the Balanced Scorecard (BSC). A conceptual framework developed a hybrid form of BSC that could be applied in a research setting before describing a case study setting. A case study, as well as a survey was employed to investigate the research questions. The results suggest that the research sector PMF incorporated the properties the organization’s value chain, as well as reflected its strategies and key actions. Furthermore, suitable performance measures had been created for all the organization’s stakeholders. Further inductive analysis indicated that the PMF resembled a hybrid form of BSC based on the Moullin (2002) public sector framework. Finally, the results indicated that the PMF motivated its researchers at operations level, however, a survey of opinions suggests there was a significant difference in opinion between different categories of researchers.


Introduction
The complex nature of performance measurement is underlined by the need to measure human effort, the functioning of a system and the efficiency of organizational processes within these systems (Kerssens-van Drongelen, Nixon & Pearson, 2000). Performance measurement in the R&D domain is further complicated by the difficulty of isolating research outcomes, long time periods, multifunctional team members, the subjective nature of assessment and high levels of uncertainty (Pappas & Remer, 1985;Brusoni, Prencipe & Salter, 1998;Kerssens-van Drongelen, 1999;Groen et al., 2002;Jamsen, Suomala & Paranko, 2002;Loch & Tapper, 2002;Baglieri, Chiesa, Grando, & Manzini, 2001;Kerssens-van Drongelen et al., 2000). These problems are exacerbated when an R&D organization is state controlled and its agenda includes outcomes that promote national development agendas, a better quality of life and preserving the environment. Furthermore, performance measurement is complicated because state owned research organizations must address the expectations of a wider range of stakeholders that include the government, society and alliance partners (Jordon & Malone, 2006;Kerssens-van Drongelen et al., 2000;Ojanen & Vuola, 2006;Hofstede, 1981;Merchant, 1998;Anthony & Govindarajan, 2001;Negash, 2005).
State owned research organizations are under increasing pressure to improve their performance (Jamsen et al., 2002;Jordon & Malone, 2006). The CSIR Materials Science and Manufacturing (MSM) is an operating unit of the Council for Scientific and Industrial Research (CSIR) in South Africa. The CSIR operates under the Scientific Research Council Act 46 of 1988, and is mandated to foster industrial and scientific development in the national interest through multidisciplinary research and technological innovation, either by itself or in collaboration with a range of local and international partners. The CSIR undertakes approximately 10% of all research and development on the continent of Africa and recorded a total revenue of R1.1 billion in 2006/7 with approximately 40% being received by way of a parliamentary grant from the South African government through its Department of Science and Technology (DST). In recent times the CSIR has introduced a performance measurement framework (PMF) which it has labelled as a balanced scorecard (BSC). The objective of the paper is to investigate whether the PMF of a state owned research organization can be adapted to the Kaplan and Norton balanced scorecard (BSC). In particular, the following four questions are investigated: Are the research organizations' value chain, strategy and key actions required to implement strategy reflected in the PMF, are performance measures developed for all the organizations' stakeholders and is the research organizations' PMF a form of BSC. Finally, a fourth research question investigates whether the PMF motivated operations at a business unit level The importance of this study is underlined by the strong linkages between national research agendas and the growth rate of national economies (Jamsen et al., 2002). In South Africa, for instance, the need for higher levels of research and development has been identified as a critical success factor to achieve a growth rate of above 5% (Wray, 2004). The article makes a contribution by evaluating the deployment and efficacy of the BSC in a complex multiple stakeholder environment (Johnson, 2001;McAdam, Hazlett & Casey, 2005;Neely, 2005;Chang, 2007). The paper also contributes to the public sector performance measurement debate (Collier, 2006).
The remainder of the paper is sequenced as follows: Section 2 develops a conceptual framework to test whether a PMF is a form of BSC. Section 3 outlines the data and method, Section 4 discusses the case study setting and Section 5 develops and discusses the results. Finally, Section 6 concludes the study and makes certain recommendations with respect to future research.

A conceptual framework
A review of the performance measurement framework literature indicates a number of seminal models over the last hundred years. These include, amongst others, the Du Pont Framework of Financial Ratios (Anderson & McAdam, 2004), the Balanced PMS Matrix of Keegan, Eiler and Jones (1989), the Performance Prism of Kennerley and Neely (2000), the balanced scorecard of Norton (1992, 1996b) and the Performance Pyramid of Lynch and Cross (1991). According to Neely (2005), the Kaplan-Norton Balanced Scorecard continues to be the most referenced framework in the literature and a majority of other frameworks since 1992 have been critiques of the shortfalls of the BSC. Although PMF prior to the BSC had incorporated non financial indicators (NFI), the BSC was he first to configure these in a causally linked framework that showed the relationship between a spectrum of causal factors and their related financial outcomes (Norreklit, 2000;Voelpel et al., 2005). In this regard, the BSC consists of four principal dimensions that are based on pre-defined strategy. The initial dimension, namely, learning and innovation influences the efficiency of business processes which, in turn, promotes customer satisfaction. Finally, customer satisfaction translates into the financial dimension that traditionally focuses on profit maximization. The BSC emphasizes strong linkages to strategy (Kaplan & Norton, 1996a;1996b; and has evolved from a performance measurement tool to a strategy implementation framework that is appropriate for the management of intangible assets in the "new innovation economy". Its authors also claim it can be customized for public and non profit organizations (Kaplan & Norton, 2001;2007).
Performance measurement in the public sector is complex and there has been much debate as to whether private sector PMF can be successfully implemented (McAdam, Hazlett & Casey, 2005). In this regard, the New Public Management (NPM) literature is premised on private sector practises that can be incorporated in the public sector (Flak & Dertz, 2005). The development of a generic PMF for the public sector remains an illusive target. A well referenced PMF for the public sector, namely, the Moullin (2002) framework incorporates five performance dimensions that include strategy or key performance outcomes, service quality, operational excellence, financial management and innovation and learning. This framework incorporates many of the characteristics of the Fitzgerald et al. (1991) model that was developed for a service sector application. , appears to be a form of BSC that has been modified to accommodate a public service sector setting. Neely (2005) regards both these models as hybrid forms of the BSC. The Moullin model, however, has been adapted to include a broader range of stakeholders. With regard to a stakeholder based PMF, like the Moullin model, it has been proposed that the Performance Prism can be incorporated to identify and categorize stakeholders (Neely et al., 2002) thus transforming it into a (fairly) flexible generic PMF for the public sector.
A critique of BSC application in the public sector reveals a number of problems. One criticism of the BSC in a public sector application is that it has a strong shareholder focus and does not meet the expectations of other stakeholders Another limitation is that it fails to link stakeholder objectives to local operations and it ignores the influence of power with respect to stakeholder influence (Johnsen, 2001;McAdam et al., 2005;Chang, 2007). A stakeholder based BSC is, therefore, difficult to apply in the public sector because of the need to align central government objectives with local service provision. Furthermore, the assumed linear relationship between the performance dimensions has been described as overly simplistic, especially in a public sector framework that incorporates a stakeholder focus. In this regard, the traditional BSC ignores time lags between processes-determinants and outcomes-results, as well as reverse causality and complex interdependent relationships. Finally, the conflicting expectations of stakeholders tend to result in the development of overly complex performance measurement frameworks that include an overload of performance measures (Brignall, 2002;Wisniewski & Steward, 2004;McAdam et al., 2005;Chang, 2007).
The following (hybrid) form of BSC is suggested for a public research organization. The first step is to develop the organization strategy in conjunction with stakeholder analysis (theory) as a failure to attend to stakeholder interests is imperative (Wiesniewski & Stewart, 2004;Bryson, 2004;Flak & Dertz, 2005). The next step, which has been recommended by various researchers, is to relax the definitions of the four BSC performance dimensions (Kaplan & Norton, 2004;2007;Sloper, Linard & Paterson, 1999). The financial dimension, therefore, is changed from a profit maximization focus to a financial sustainability objective and the customer dimension broadened to include a wider range of stakeholders. The BSC can also relaxed by increasing the number of dimensions, as well as renaming them (Kaplan & Norton, 2007). In this regard, the Moullin (2002) PMF, illustrated in Figure 1, can be regarded as a reformatted BSC that has renamed and rescheduled some of the dimensions. The learning and innovation dimension is the same in both scorecards, as well as the dimension of operational excellence. In a research context, however, operational excellence has a two way relationship with financial sustainability, in that it influences and is influenced by this dimension. Conversely, in the BSC, the financial dimension is the final outcome and it focuses on the profitability of the organization. In this regard, the final dimension of a public entity is the key strategic outcomes that satisfy stakeholder objectives. The following causal chain, illustrated in Figure 1, links the five dimensions in the research sector as follows: Innovation and learning, in conjunction with financial sustainability, provides the necessary skills, facilities and technologies to ensure operational excellence. In turn, operational excellence ensures the efficient management of funds, as well as higher levels of contract work in order to ensure financial sustainability. Operational excellence leads to research outputs like publications and patents, as well as the development of local and international collaborative alliances. Finally, research outputs translate into strategic outcomes in the public research sector that include stakeholder satisfaction that is based on the transfer of knowledge generation and technology for national priorities and the public good.  The relevant stakeholders can be configured into the reformatted BSC using a combination of stakeholder analysis and the Performance Prism Wiesniewski & Stewart, 2004;McAdam et al.. 2005). It is also suggested that the reformatted BSC captures the key properties of the organizations value chain (Wilson, Hagarty & Gauthier, 2003). In this regard, the value chain of a research organization can be likened to an input-output function (Brown & Svenson, 1998) that is activated by a range of inputs like funding, human capital, information and investment in equipment. These inputs are translated into outputs as a result of a series of research activities like planning, collaborating and managing resources. In this regard, outputs like publications and patents ensure long standing alliances, a healthy proportion of paying customers and help ensure financial sustainability (Jordon & Malone, 2006). Finally, in a public research agenda, outputs are translated into outcomes like public goods that meet the expectations of various stakeholders like the state, society, alliance partners and industrial clients (Jordon & Malone, 2006;Kerssens-van Drongelen et al., 2000;Ojanen & Vuola, 2006;Hofstede, 1981;Merchant, 1998;Anthony & Govindarajan, 2001;Negash, 2005).

Data and method
The objective of the paper was to investigate whether the PMF of a state owned research organization could be adapted to the Kaplan and Norton balanced scorecard (BSC). In order to achieve this objective, as well as because of the qualitative nature of the data, a case study method was adopted to resolve the first and second research questions. The first research question was whether the organizations' value chain, strategy and key actions were captured in the PMF. The second research question was whether performance measures had been developed in the PMF for all the organizations' stakeholders. A case study method was chosen not only because of the qualitative nature of the data, but also because of the inductive nature of the study and the fact that it allowed the researchers to collect data from multiple sources. (Leedy, 1993;Yin, 1994;Leedy & Omrod, 2001). In this regard, the study obtained data from interviews, company documents, press releases and websites. The data from the case study setting were primarily collected from a number of semi-structured interviews, illustrated in Table 1. Other data, consisting of company documents, was made available to the researchers in electronic or hard copy format or by way of the provision of appropriate websites. With respect to the interviews, a senior CSIR Group manager for Research and Development was interviewed twice before he referred the researchers to suitable managers at unit level. Because of the wide range of responses, as well sources of data, a content analysis was used to assemble the data in themes and patterns (Breakwell, Hammond & Fife-Schaw, 2000). The data were also constantly compared to theory that was reviewed in the literature. In order to ensure that the data had been reliably recorded, the researchers wrote up their account of the interviews, as well as their understanding of other data and verified this with the respective interviewee.
Finally, in order to test the third research question, namely, whether the divisions' PMF was a form of BSC, the results of the two previous research questions were assessed in conjunction with additional data from the interviews and company documents. Further evidence and theory from the literature was then revisited to check whether key characteristics of the organizations' PMF (dimensions, causality) were similar to that of the BSC.
In order to test the fourth research question, namely, whether the PMF of a state owned research organization had motivated operations at a business unit level, a survey of 39 researchers at three different levels of seniority, was conducted. The reliability of the small sample size was strengthened by the fact that all of the respondents could be regarded as experts in the research domain (Lenth, 2001). The survey involved five questions. The respondents were required to indicate their answer on a five point Likert scale before basic descriptive statistics was employed to describe the median scores and confidence intervals of the three different levels of researcher. A series of Kruskal-Wallis tests and a Box Plot diagram were employed to see if any significant difference in opinion existed across the three groups of respondents with respect to each question asked, as well as the aggregated score of the five questions.

The case study setting
The Materials Science and Manufacturing (MSM) division, a business unit of the CSIR, reports to the executive committee of the CSIR. The strategic intent of MSM is to improve industry effectiveness, address national human resource development by conducting research and innovation in the fields of materials and manufacturing. The unit conducts a range of applied research, experimental development and technology transfer activities in six main areas. These are: Metals and Metals Processes whose main focus is on primary processes, alloying and engineering design of light metals. The Polymer and Bioceramics Area concentrates on new drug delivery systems and biomaterials. The Fibres and Textiles Area conducts R&D in nonwovens, fibre modification and composites. The Manufacturing Science and Technology Area focus is on the development of advanced manufacturing technologies that include digital and micro manufacturing, robotics and micro fluidics. The Energy and Processes Area investigates clean coal technologies and renewable energies and Sensor Science and Technology focuses on smart structures, electro-optic sensing and imaging and sonar technologies. In addition, MSM hosts a national Nanotechnology Centre of Innovation concerned with the synthesis and characterisation of nanostructured materials.
In order to provide a backdrop to the development of the division's PMF, the case study setting first explains the characteristics and dynamics of the MSM value chain before introducing the relevant stakeholders.
The annual operational plan is then discussed in order to explain exactly the structure and management of the PMF.

The MSM value chain
The MSM value chain, illustrated in Figure 2, demonstrates how inputs are translated into outcomes. The funding the division receives ensures that it is able to finance inputs like human capital, equipment, information technology and management control systems that underpin the activities undertaken by the division. R&D activities involve the coordination of the research process of knowledge creation and application in order to solve business problems. In this regard, researchers in the MSM Division are continuously involved in planning, collaborating and the implementation of project related activities that translate into outputs like patents, research publications and technology transfers. Simultaneously, the division generates its cash flow from these outputs in order to maintain its financial sustainability. Finally, the long term implications of quality outputs translate into solid strategic alliances and good relationships with the division's stakeholders. Other longer term outcomes are the improvement of manufacturing efficiency in South Africa in general, as well as the promotion of a cleaner environment.

The stakeholders
The MSM Division receives a government grant to pursue its research agenda. The South African Government, therefore is a key stakeholder as both a financier, as well as a client because MSM also undertakes research on behalf of a number of government departments. Furthermore, the division collaborates with many local and foreign research institutions like universities, technical institutes and national research foundations. Its client base, moreover, includes numerous local and international corporations in the private sector. The outcomes of many of these activities are developed for a far broader audience, however, than just the private sector and extend to other national agendas in Africa (Nepad), as well as the improvement of the environment and South African business efficiency in general. Society per se, is therefore, a stakeholder from two perspectives. Firstly, society is a provider of tax revenue that funds government activities (including its funding of MSM) and secondly as a beneficiary (albeit indirect). Finally, the MSM division has a mandate to develop its staff into world class researchers not only to ensure the success of its own operations but also to translate into improving the competitive edge of South Africa's manufacturing sector. The wide range of stakeholders, as well as the complex interrelationships between the stakeholders complicates the development of a performance measurement system.

The operational plan
The operational plan of MSM is largely based on an environment in which the objectives of the state and society are promoted. This suggests that many of the unit's programs are directed in the national interest. Because of this, the expectations of a large number of stakeholders need to be met including the state, society, the environment, employees, research partners and customers. In support of achieving unit objectives, a three year strategic and operational (rolling) plan is developed on an annual basis. The business plan to implement this strategy is developed by MSM using guidelines provided by the CSIR Executive team. Planning begins in October each year and a draft business plan is submitted in December for feedback and approval. The planning process begins with an analysis of the external and internal environments, including a review of the recent performance of the unit. Strategic objectives are then derived from these analyses and configured with CSIR level strategic objectives before detailed plans are developed in support of these.
The format of business plans is standardized across all CSIR operating units. The detailed one year operational plan outlines the key initiatives and actions to support the strategic objectives, the key performance measures (PM) and targets The strategic and operational plan is then agreed with the CSIR Executive team who may request changes to the plan or PM targets. The unit would then make the necessary changes before a final plan is agreed on with the CSIR Executive team. The CSIR Executive then prepares an organisation level strategy and operational plan that is partially an aggregation of the various unit plans, and submits it to the CSIR Board for approval or rectification until a final plan is agreed on. Only at this stage are the unit level performance measures and targets finalized for the next one and three-year planning periods MSM's operational plan for 2006/7 included four primary strategies as illustrated in Table 2. The first strategy, namely, to strengthen the science and technology base by way of improving the quantity and quality of its research outputs, was supported by a number of key actions. These actions included the promotion of R & D alliances with local and international partners, the improvement of the quality and mix of research undertaken and the need to invest heavily in equipment and infrastructure. The second strategy was focused on building and transforming human capital by improving the qualification profile of its staff, as well as correcting demographic imbalances. The third strategy, namely, to perform relevant research and development and to transfer technologies is primarily directed at improving the outcomes of the research outputs identified in the first strategy. In this regard, the quality of outcomes is based on key actions to promote more contract research with the private and public sectors, as well as to improve R&D outcomes like the commercialization of intellectual property and the development of technologies for the public and social good. The fourth strategy, namely, to secure financial sustainability through operational excellence, focused on improving the unit's contract revenue and net margin and adhering to good governance standards. These standards included compliance with the national black economic empowerment charter, compliance with international quality and environment management standards and a reduction in energy consumption.

The performance measurement system
The PMF, illustrated in Table 2, has been "labelled" by the CSIR as a (type) of BSC. The construction of the PMF begins with a detailed assessment of the research environment in conjunction with the divisions' performance in the previous year. This assessment matches progress against expectations with respect to each of the four strategic focus areas and highlights specific challenges that need to be addressed in the annual development of the PMF. The scorecard operates on a quarterly basis and actual progress is measured against phased targets and a brief report summarizes achievements and challenges.
In addition, a series of actions are listed with respect to overcoming any challenges, as well as meeting the attainment of the future phased targets for the remainder of the year. A revised forecast for the year is also prepared. Individual performance measurement reports are produced for staff members every 6 months in a similar format, however, the choice of PM 's within the main strategic dimensions, is determined by the specific contributions of individual staff members. The performance measures (PM 's) are grouped in the four main strategic focus areas. A range of five targets is developed for each PM ranging from "outstanding" (rating of 1) to "does not meet expectations" (rating of 5). For simplicity sake, the targets cited in table 2 are those which represent "meets all requirements" corresponding to a rating of 3. The review of MSM's performance involves comparing the actual performance achieved with the target set for each KPI, and assigning a rating of 1-5.

Results and discussion
This section develops the results of the four research questions. In this regard, the questions investigated include whether the organizations' value chain, strategy and key actions were captured in the PMF, whether the PMF developed performance measures for all of its stakeholders and whether the organizations PMF was a form of BSC. The final research question investigated whether the PMF had motivated operations at a business unit level.
Are the value chain, strategy and key actions required to implement strategy captured in the PMF?
In order to link the division's value chain to its strategies, the strategies have been reformatted to better reflect their causal relationships. The reformatted PMF, illustrated in Table 3, commences with Strategy 2, namely, the need to develop human capital. Strategy 2 can be linked to the inputs segment of the value chain that includes human capital development and the provision of facilities. Strategy 4, namely, to secure financial stability through operational excellence can be tracked to the activities section of the value chain that includes all the activities that need to be undertaken in order to produce quality research outputs. A causal link exists between Strategies 2 and 4 (see arrows showing direction of causality), namely, that the investment in equipment and human capital translates into operational excellence. Operational excellence, however, promotes Strategy 1 that has the objective of strengthening the science and technology base by way of producing quality research outputs and collaborative alliances. In the value chain this can be tracked to the outputs section. Finally, the quantity and quality of research outputs (Strategy 1) translate into the quantity and quality of research outcomes, reflected in the outcomes section of the value chain, as well as by Strategy 3, namely, to improve research outcomes in the form of public goods. The properties of the division's value chain (input-output function), therefore, appear to be reflected in the division's strategy The division's reformatted PMF has four principal dimensions. The PMF dimensions start with the division's learning and innovation dimension (called building and transforming human capital) that is based on Strategy 2. The key actions listed to implement this strategy involve the development of human capital and the transformation of staff demographics. A total of 21 performance measures are based on these key actions. The next PMF dimension (Financial sustainability through operation excellence) is based on strategy 4 and can be likened to the BSC dimension of internal efficiencies. A further 9 performance measures are based on the key actions of improving financial sustainability, operational excellence and good governance. The next PMF dimension is a type of customeroutputs dimension that is based on Strategy 1 that develops the key actions of improving the quantity and quality of research output and alliances. A total of 9 performance measures have been developed to implement strategy 1. Finally, the PMF dimension of stakeholder outcomes, based on Strategy 3, has developed 9 performance measures that are based on the key actions of improving the quantity and quality of research outcomes. The four dimensions of the division's PMF, incorporating 48 performance measures, are thus based on the division's key strategies and actions.

Are performance measures developed for all the stakeholders?
Although the division does not conduct formal stakeholder analyses, numerous performance measures, have been specifically developed for all of the division's stakeholders. Stakeholder driven performance measures, illustrated in Table 4, have been developed for society at large, the state (Department of Science and Technology), local and international customers, local and international research partners and employees. The PM 's clearly illustrate the broader objectives of the MSM Division to promote national objectives and strategies like NEPAD. In this regard, close to 50% of the PM 's deliberately target stakeholders that are not normally included in the PMF of private sector companies. Furthermore, these performance measures have been monitored over long periods of time to ensure stakeholder objectives are achieved.

Is the division's PMF a form of BSC?
In order to determine whether the PMF was a form of BSC, the four dimensions of the division's PMF and their relevant strategies were reformatted. The reformatted PMF was then compared to the generic research based BSC illustrated in Table 3, as well as the Moullin (2002) framework and the traditional BSC of Kaplan and Norton (1992;1996a) outlined in Section 2. The division's first PMF dimension, based on Strategy 2, is similar to the Innovation and Learning dimension in all versions of the BSC. This strategy results in a range of performance measures to ensure the development and training of personnel. In this regard, the division's first PMF dimension appears to ignore the provision of other inputs. The second PMF dimension, based on Strategy 4 develops performance measures to achieve the objective of operational excellence and financial sustainability. This dimension appears to closely reflect that of the both the Moullin and generic public research sector scorecards. In a state research organization, however, an output of operational efficiency is financial sustainability. Operational efficiency and financial sustainability have a causal link with the division's next PMF dimension that develops a series of performance measures to promote the quantity and quality of research outputs articulated in Strategy 1. In this regard, the division's third performance dimension is the same as the generic R&D model but has a broader stakeholder focus than the Moullin PMF that focuses service quality. It should be noted, however, that research outputs like patents and publications also have a reverse influence on the creation of financial sustainability. Finally, the division's fourth PMF dimension, based on Strategy 3, develops performance measures in order to secure quality research outcomes. This stakeholder based dimension in the research sector is not found in the traditional BSC but partially reflected in the Moillon framework. The four performance dimensions of the division's PMF, therefore, concur with the dimensions of the generic research sector BSC developed in Section 2, as well as closely resemble those of the Moullin (2002) public sector PMF. The causal linkages between the four dimensions of the traditional BSC are a central aspect that differentiated the BSC from all other frameworks in 1992. In this regard, it can be shown that a reformatted version of the division's four performance dimensions demonstrate a casual chain (see direction of arrows in Table 3). The PMF dimension of learning and innovation, that is based on the strategy to develop human capital, has a logical linkage with operational efficiency. In this regard, highly trained scientists, funding, facilities and systems logically translate into operational efficiency. Operational excellence also influences the management and creation of funding that contribute towards financial sustainability, a key objective of the division. Operational efficiency and financial sustainability, in turn, influence the research outputs dimension. The quantity and quality of research outputs is a logical function of operational efficiency and financial sustainability as demonstrated in the value chain of the division. This cause and effect relationship is also partially explained by the Moullin (2002) framework that links service quality outputs to operational excellence and financial sustainability. In a research organization, however, outputs influence (and are influenced by) financial sustainability. Finally, the outcomes based PMF dimension that incorporates measures of stakeholder satisfaction is directly linked to the quantity and quality of outputs. This causal relationship has been assumed by many studies in the PMF literature covering the research sector (Pappas & Remer, 1985;Brusoni, Prencipe & Salter, 1998;Kerssensvan Drongelen, 1999;Groen et al., 2002;Jamsen, Suomala & Paranko, 2002;Loch & Tapper, 2002;Baglieri, Chiesa, Grando, & Manzini, 2001;Kerssens-van Drongelen et al., 2000). The four dimensions of the division's PMF, therefore, demonstrate a causal chain that complies with the properties of a modified BSC.
The results of the first and second research questions appear to confirm the fact that the division's PMF was based on its strategies, that the performance measures were based on key actions and that performance measures had been developed for all the stakeholders. In particular, the linkage of the PMF to strategy is cited as a critical compliance factor in the public sector PMF (Niven, 2008). Furthermore, the four dimensions of the divisions' PMF are directly reflected in the generic BSC created in Section 2 for the research sector. The dimensions, moreover, demonstrate clear causal linkages albeit that they are more complex than the traditional BSC. In this regard, the division's PMF refutes the argument that causal relationships cannot be demonstrated in a stakeholder based context. It cannot, however, claim to capture all the complex interdependent relationships thereby supporting the critique of certain studies (Brignall, 2002;McAdam et al., 2005;Chang, 2007). The divisions PMF is also not overly complex and includes a comprehensive, yet limited, set of performance measures for each of the dimensions (Wisniewski & Steward, 2004). Although the performance measures are specific to the research domain it is clear they include both financial and non financial measures. The division's PMF, therefore resembles a hybrid form of the BSC, based on the Moullin (2002) scorecard that has been modified to incorporate the objectives of a state owned research organization. The PMF supports the claims of Kaplan and Norton (2007), namely, that the BSC can be utilized in a public sector context. Finally, it is clear that many organizations believe any dashboard of performance measures can be labelled as a BSC confirming the pervasive marketing of this framework.

Has the PMF motivated operations at business unit level?
The results, illustrated in Table 5, indicate a largely positive response with respect to the role of the motivational role of the PMF. In this regard, a median score of 1 suggested the PMF has not motivated the researcher whilst a score of 5 suggests the PMF was very successful with respect to meeting its objectives. The respondents indicated that the PMF had helped to communicate strategy to them as individuals. However, a series of Kruskal-Wallis tests indicated that there was a significant difference in opinion (5%), illustrated in Table 6, between Level 1 researchers (more senior) and Level 3 researchers (less senior). Furthermore, the PMS was strongly instrumental in motivating individuals to achieve strategic objectives, as well as fostering learning and facilitating decision making. In this respect, there was a significant difference in opinion (5%) between Levels 1 and 3 with respect to the motivational role of the PMF to achieve strategic differences, as well as its role as an aid to decision making. The disaggregated results, illustrated in Figure 3, further illustrate that significant differences in opinion existed across the three levels of researchers with respect to the role of the PMF and its motivational role. In this regard, the more senior managers (level 1) were most positive about the role of the PMS scoring a median value of 4 whilst non management employees (level 3) were the least positive scoring a median value of 3.44 that is approaching an ambivalent stance.
The results suggest that, from a divisional perspective, the stakeholder based BSC was successful motivating good performance and employee participation at operational level. In this regard, a series of five reliable proxies for motivation were measured. This would refute one of the criticisms of stakeholder based PMF, namely, that they are often not capable of reconciling stakeholders objectives with those of employees at operational level (Johnsen, 2001;McAdam et al., 2005;Chang, 2007) . What is clear, however, is that the deployment of the measurement system was not as successful with respect to more junior research staff. This supports a contention that performance measurement systems like the BSC are often only operationalized at more senior levels in the organization (Merchant, 1998;Kaplan & Atkinson, 1998;Anthony & Govindarajan, 2001).

Conclusion and recommendations
The problematic nature of developing a BSC type performance measurement system for a state owned research organization was briefly discussed before developing a conceptual framework for a research sector BSC. On the basis of the results, it would appear as if a stakeholder type PMF can be developed in a public research organization that displays the properties of a BSC. The performance dimensions of a BSC, however, need to be modified to capture the extra dimension of research outcomes in order to satisfy a wider range of stakeholders than those generally contained in a private sector organization. The causal linkages, although linear are different in some respects to the traditional BSC because a combination of both research outputs and financial sustainability (usually the final dimension in the BSC) influence longer term research outcomes. The usefulness of the study is that it illustrates how research organizations can set about creating a PMF along BSC type principles. In this regard, the results suggest the case study PMF should be re-arranged in a causal format and certain performance measures should be relocated to other performance dimensions. For example, performance measures like the investment in equipment, currently in the research outputs dimension, should be relocated to the human capital development dimension of the division's PMF which should be broadened into a more traditional innovation and learning dimension. The sequence of strategies, their key actions and related performance measures are currently arranged in a somewhat illogical fashion that is at odds with the properties of the value chain.
A further important recommendation is that the organization engages in a formal analysis of its stakeholders with a view to the development of its strategic plans. The results also demonstrate that many organizations are under the impression that all PMF that involve a dashboard of measures can be labelled as balanced scorecards. It is also debateable whether many PMF' s are a hybrid form of BSC or a performance prism or a combination of these and other frameworks. Finally, the results do not refute the contentions of Kaplan and Norton (1996;2007) that the BSC can be applied in a public company context.  The results suggest that the division's PMF was capable of motivating local operations despite the fact that it had a stakeholder focus. A useful aspect of the study highlights the important role of a PMF, namely, that it should foster learning, facilitate decisions, as well as be used for a wide range of other objectives. The study, however, demonstrated that different levels of employees may have had very different thoughts about the role of the PMF and its impact on the achievement of objectives. In this regard, lower level employees were significantly less enthusiastic about the role of the PMF system than more senior level staff. This issue highlights a common complaint with respect to the deployment of performance measures to lower levels of an organization and the need to ensure ownership and participation at all levels.
We fully accept the limitations of this inductive type of study and the concomitant problems of generalizing case study conclusions. The study is, therefore, intended to open up new areas of investigation as much as answer certain questions. In conclusion, the increasing importance of national research agendas suggests that future research should examine the appropriateness of a range of PMF in state controlled research organizations. Further research is also suggested with respect to the development and deployment of different PMF frameworks across a range of subject levels in the public research sector.