The development of a measurement instrument to measure the quality of internet investor relations

The purpose of this study was to develop a measurement instrument to measure the quality of Internet investor relations (IIR). This study will aid future research to examine IIR and provide guidance to companies in the development of an IIR strategy. The development of the instrument was based on best practice guidelines issued by the Investor Relations Society, an extensive literature review and a pilot study. The result was a measurement instrument that consists of 346 attributes.


Introduction
King III (2009) states that transparent and effective communication with stakeholders is essential for building and maintaining their trust and confidence.Stakeholders include, but are not limited to, shareholders, suppliers, employees and creditors.Marston (1996: 477) defined investor relations as the link between a company and the financial community in terms of which information is provided to the financial community for evaluating the company.The activity of investor relations involves all information types, for example mandatory and voluntary, financial and non-financial, as well as shareholder services to facilitate relationship management and/or strategic marketing.
Based on the investor recognition hypothesis (Merton, 1987), a well-developed investor relations strategy will increase company visibility, which in turn leads to increased share liquidity (Brown & Hillegeist, 2007).Economic theory links increased liquidity to the cost of capital through information asymmetry (Botosan, 2000).
While South Africa has recently been ranked number one by the World Economic Forum for the strength of its auditing and reporting standards for the sixth consecutive year (IRBA, 2016), the quality of investor relations, as a wider concept, is largely unresearched in South Africa.
Investor relations communication channels available to companies include, but are not limited to, annual reports, presentations, media releases, corporate websites, and social 1 With a few exceptions, such as the Alternative Investment Market (AIM) in the UK and the European transparency directive media (e.g.Twitter).The rapid increase of information channels has driven users closer to sources directly controlled by the company, such as corporate websites (Jones, 2009).
The corporate website as a communication medium has specific advantages for both companies and investors.For companies, it is cost-effective and flexible in format; while, for investors, it may potentially be an easy, quick, cheap, complete, reliable and up-to-date source of information.
The following are, on the other hand, often discussed in the literature as hindrances in the use of the corporate website as communication channel: outdated and incomplete information (FASB, 2000), disorientation (Debreceny, Gray & Mock, 2001: 10), lack of clear boundaries (DeStefano & LeFevre, 2007: 1616), information overload (Lybaert, 2002), the lack of assurance regarding the credibility of information (Healy & Palepu, 2001: 425), and the absence of rules to standardise content (Khadaroo, 2005: 61).
The decision to use the corporate website as investor relations communication channel by Johannesburg Stock Exchange (JSE) listed companies is voluntary. 1Notwithstanding the voluntary nature thereof, empirical findings have shown that nearly all the largest listed companies in South Africa have corporate websites with dedicated investor relations sections (Barac, 2004;Nel & Baard, 2007;Esterhyse & Wingard, 2016).
The majority of related studies to date have either used an indirect disclosure proxy (a proxy that is not based on an (2004/109/CE), corporate website investor communication is also not regulated in the international environment.examination of the original disclosure vehiclee.g.analyst ratings issued by the Association for Investment Management and Research) or involved an examination of the annual report.Studies that entail an examination of the original disclosure vehicle (e.g.content analysis using a measurement instrument) may be defined as direct disclosure proxy studies (Hassan & Marston, 2010).
Direct disclosure proxy studies that have measured either investor relations or the use of the corporate website are scarce.Measurement instruments used in the literature to measure the use of corporate websites as communication channel can be criticised for the following:  Ignoring important attributes (e.g. a number of instruments measured merely the level of financial reporting on corporate websites and made no attempt to measure the wider concept of investor relations and presentation-related attributes);  No attempt either to distinguish between quantity and quality or to measure quality;  Use of weights without a clear rationale; and  Failure to perform applicable reliability and validity tests.
The purpose of the current study was to develop a measurement instrument that could be used to measure the quality of Internet 2 investor relations (IIR 3 ).This study will contribute to the body of knowledge as follows.Firstly, through a methodical discussion of existing measurement instruments and the issues that should be considered in the development of an instrument.Secondly, it will aid future research to examine IIR.Finally, it will provide guidance to companies and investor relations practitioners in the development and implementation of an IIR strategy.
The study achieved its purpose by means of a four-step methodology: review of best practices as published by the Investor Relations Society (IRS),4 an extensive literature review of existing instruments, a pilot study, and, finally, a reliability and validity assessment based on the IIR measurement results of a sample of 85 JSE-listed companies.All measurements were done during 2015.The result was a measurement instrument that consists of 346 attributes, organised into 11 categories.

Literature review
Although indirect and direct disclosure5 proxies each has specific advantages, both are criticised in the literature for various reasons.Indirect disclosure proxies are criticised for 2 It should be noted that although companies' use of the Internet is not limited to corporate websites only (e.g. companies also use Twitter, Facebook and YouTube to communicate with investors), Internet is defined to include only corporate websites for the purpose of this study. 3The remainder of this article will use the abbreviation, IIR, to refer to Internet investor relations.Although not all prior studies that have examined the use of the corporate website as communication channel have explicitly measured investor relations, such studies three reasons: sample bias (Welker, 1995), analyst bias (Healy, Hutton & Palepu, 1999: 489), and the relative small investor relations component (Agarwal, Taffler, Bellotti & Nash, 2016: 32).Given the labour-intensity6 of performing a manual content analysis, an important advantage of using an indirect disclosure proxy, on the other hand, is larger sample sizes.Froidevaux (2004: 75) points to the risk of missing information in conducting a corporate website content analysis, given the large and complex nature of websites.Direct disclosure proxies are also criticised for judgement error and subjectivity in deciding on what attributes to measure (Froidevaux, 2004: 53).Trabelsi, Labelle and Dumontier (2008) identified the following three weaknesses of measurement instruments that have examined the corporate website as disclosure medium:  The use of dichotomous variables which could reduce cross-sectional variability;  Too much emphasis on website design (presentation) over the actual content; and  The ignoring of important variables.
Contrary to the second weakness listed above by Trabelsi et al. (2008), Hamid (2005: 12) emphasises the omission of presentation attributes as a weakness of measurement instruments.Francis, Nanda and Olsson (2008: 62-63), on the other hand, listed the following advantages of using a selfconstructed measurement instrument:  Increased confidence that the metric captures what it is intended to capture; and  The ability to include any company in the sample.Esterhuyse and Wingard (2016) used a measurement instrument consisting of 201 items to assess the websites of 205 JSE-listed companies during the course of 2012.A separate discussion of the Esterhuyse and Wingard (2016) study is justified, but to ease comparison with the current study, is discussed near the end of this article.
Table 1 summarises the literature that has examined corporate websites as disclosure vehicle according to: (1) the number of attributes measured; (2) the use of weights; (3) whether a distinction is made between quantity as opposed to the quality of IIR; and (4) whether the results of reliability and validity tests were reported.Each of these will now be discussed in turn.
will be categorised as IIR in this article, given the wide definition of investor relations as discussed.

Attributes measured
Earlier studies merely measured whether companies had a web presence and whether annual reports or summaries were available on the website 7 .Xiao, Yang and Chow (2004: 197) criticise earlier studies for treating corporate website disclosure as if it were just an electronic version of the hardcopy annual report, i.e. ignoring the advantages available through the use of technology.Hedlin (1999) and Lymer, Debreceny, Gray and Rahman (1999) both used a three-stage model to classify the IIR practices of companies.
Attributes can be categorised into two broad categories: content and presentation, with content referring to all financial and non-financial information and presentation referring to the use of presentation technologies to enhance the information (FASB, 2000).Marston and Polei (2004: 297), Loehnis (2007) and Chang, D'Anna, Watson and Wee (2008: 376) all argued that although investors are mainly 7 Examples are Debreceny, Gray and Rahman (2002), Craven andMarston (1999), andAshbaugh, Johnstone andWarfield (1999).
interested in the extent to which information has been provided (i.e.content), they also need to find this information as quickly and easily as possible (via clear presentation).
Other authors have, moreover, argued that the presentation of information determines the ease of navigation, users' ability to access relevant information (Khadaroo, 2005: 64), as well as the timeliness, verifiability, quality and usefulness of information (Debreceny, Gray & Rahman, 2002).In the opinion of Wade and Forbes (2000), the usability of IIR is often seen by institutional investors as a prerequisite hurdle to clear before they would engage in using corporate websites as an information source.

The use of weights
As is evident from Table 1, the majority of studies did not assign weights.Studies that did assign weights could further be categorised into studies that had assigned weights to individual attributes as opposed to assigning weights to categories.

Quantity versus quality of IIR
The majority of studies examined in the literature review made no attempt to measure the quality rather than quantity of IIR.Core (2001: 452) emphasised two problems that could hinder metrics to measure the quality of IIR: labour-intensity and judgement error.Three approaches are suggested in the literature as proxies for IIR quality: the number and type of attributes measured, the use of weights and through an assessment of the actual content.
Studies that have assigned weights to capture the importance of attributes were discussed above.Froidevaux (2004: 78) suggested an assessment of the actual content of the disclosed items as one alternative to measure the quality of information.Core (2001: 452) suggested the use of techniques in natural language processing (e.g.artificial intelligence) to lower the cost of calculating such disclosure metrics.

Reliability and validity
Reliability and validity assessments are scarce in direct disclosure proxy studies (Hassan & Marston, 2010) (also see Table 1).Reliability can be assessed through one or a combination of the following tests: test-retest (i.e.stability of results over time); inter-coder (i.e.stability of results when the same analysis is conducted by more than one coder); and internal consistency (i.e. an assessment of how well different attributes measure the same subject matter).
Validity, on the other hand, can be assessed with one or a combination of three tests: criterion; content or face; and construct.Criterion validity is a measure of how well the measurement instrument results compare with another instrument (the criterion variablee.g.external awards or analyst ratings).Content or face validity, involves an assessment through third-party (expert) judgement.Finally, construct validity measures the extent to which a measure performs in accordance with theoretical expectations.

Attributes measured
As a first step, attributes were identified from the IRS best practice corporate website guidelines (IRS, 2013).From these guidelines, 105 attributes were identified.
Next, an extensive literature review was done for three reasons: firstly, to measure as widely as possible so as to mitigate the risk of important attributes being omitted, as also argued by Froidevaux (2004) and Loehnis (2007); secondly, to improve the comparability with previous studies; and, finally, for guidance on how to measure specific attributes.
As the next step, a pilot study was conducted to evaluate the practicality of measuring the attributes as identified, to identify omitted attributes that may only be applicable to JSElisted companies and to perform preliminary reliability and validity assessments.For the pilot study, eight JSE-listed companies were randomly selected.

Use of weights
Given the subjectivity involved in the assignment of weights (as discussed above), the fact that corporate websites are used by various types of stakeholders and investors (with different needs in terms of both content and presentation), and previous research that documents similar results for studies using weighted and unweighted instruments, the current study opted not to apply weights to either individual attributes or to categories.

Quality of Internet investor relations
Various approaches are suggested in the literature to measure the quality of IIR (as discussed above).This study attempted to measure quality by first of all measuring content as widely as possible and by measuring the presentation of information (accessibility, timeliness and navigation), and, furthermore, by allowing for the measurement of attributes as only partially available (scored as 0.5), as discussed below.
Although the majority of attributes were measured as either available (1) or absent (0), 50 attributes were measured as partially available (0.5), based on the breadth and depth of content available. 9Further, based on timeliness and usability, some attributes were measured as only partially available (0.5).Outdated information was assessed as either partially available (0.5) or absent (0). 10 Where information was available, but as a result of factors such as poor layout, inconsistencies and incompleteness was not fully useful, the attributes were assessed as partially available (0.5).Nonfunctional 11 and non-useful 12 links were assessed as absent (0).

Reliability and validity
To assess reliability, internal consistency was tested in two ways: by applying Cronbach's alpha and, in accordance with Froidevaux (2004), Cheng, Courtenay and Krishnamurti (2006), and Kelton and Yang (2008), by examining the correlation coefficients between disclosure categories. 13 Following Hail (2002), Trabelsi et al. (2008), Chang et al.
9 For example, ownership that was discussed only in qualitative terms (e.g.director A is a substantial shareholder) was assessed as only partially available (0.5), as opposed to ownership discussed in quantitative terms (e.g.director A has a 20% interest), which was assessed as being available (1). 10 Information older than five years was assessed as absent if updated information was a reasonable expectation (e.g.shareholders' information).Information older than a year was assessed as partially available, unless updated information was not a reasonable expectation or unless it was separately measured as archived information (e.g.archived annual reports).
11 For example: "This page cannot be displayed".
12 For example: "Click here for share price history" with links to www.jse.co.za homepage. 13Test-retest is not applicable as only one IIR measurement per company was done in the current study.As all measurements were (2008), Froidevaux (2004) and Orens et al. (2010), construct validity was applied to validate the use of the measurement instrument. 14  To conduct these tests, measurement results were required.For this purpose, a sample of 25% of JSE-listed companies 15 was selected using stratified (JSE industry) random sampling with proportional allocation.A total of 85 companies were included in the sample.As only companies with working corporate websites were included in the population, technically all sample elements 'replied' and there were therefore no non-responses.

Results
This section will begin with a discussion on the measurement instrument that was developed, followed by the results of reliability and validity tests.To assist future research, some of the decisions that were made in conducting the content analysis is discussed.Finally, Esterhuyse and Wingard's (2016) contribution is discussed.

The measurement instrument
Based on the IRS best practice guidelines, a literature review and a pilot study, a measurement instrument that consists of 346 attributes, organised into 11 categories, was developed.Table 2 lists these categories 16 , as well as the number of attributes per category 17 .Each of these categories will now be discussed in turn.  1The population was defined as all companies currently listed on the JSE that had not been suspended, had traded since inception date, had published 2013 annual reports, and had a dedicated, working website. 16Based on the categories used by the IRS. 17For practical reasons the full measurement instrument is not published here, but is available on request.

Accessibility
According to the IRS (2013), a best practice website should be available to the widest possible audience and should be fully accessible to allow equal access to all users, including those with visual, hearing, cognitive and motor impairments.
Companies can improve accessibility by, for example, providing alternative formats (e.g. transcripts of meetings) for users with specific impairments (e.g.hearing impairment) and by making greater use of graphics with a text-only option to accommodate users with visual impairments.Presentation technologies such as an option to adjust the font size can further improve accessibility for users with visual impairments.

Navigation
Corporate websites should be easy to navigate and their use should be intuitive (IRS, 2013).The search function, help function and sitemap are useful navigation tools that can assist users to find specific information they are looking for.
Although hyperlinks can potentially assist navigation, they can also result in disorientation18 and cognitive overload if not properly used.Drill-down links can be used as an effective tool to prevent information overload.To help prevent disorientation, users should always know where they are on the website and a 'back to the homepage' link should be clearly visible on all pages.Breadcrumb trails and/or navigation panes can be used by companies to help users orient themselves on webpages.

Timeliness
An important potential advantage of corporate websites is the provision of timely information.For investors to judge the timeliness of information, it should be dated.In some cases, undated information (e.g. about market capitalisation) may be misleading or even useless.Instead of clearly dating information, companies can emphasise timeliness by clearly labelling information as being 'the latest' or by having a dedicated area or areas on the homepage for the latest information.Timeliness can further be improved by providing the option for users to subscribe to an e-mail alert service or RSS19 feed, and the option to synchronise their own calendars with the company's financial calendar.

Company information
Telling the company story is one of the main functions of the corporate website (IRS, 2013).Attributes measured in this category are categorised into 10 sub-categories: dedicated 'about us' link, history, contact details, organisational chart and group structure, vision and mission, customer information, product and services, suppliers, properties, and critical success factors.

Financial information
One of the main reasons that investors visit corporate websites is to access reliable and up-to-date financial information (IRS, 2013).Attributes measured in this category are categorised into four sub-categories: financial reports, presentations, financial analysis, and archives.
Besides PDF20 integrated annual and interim reports, alternative formats should be provided (e.g.HTML 21 and spreadsheets).Research found that different types of users prefer different types of formats (Beattie & Pratt, 2003;Lybaert, 2002), suggesting that alternative formats are not mutually exclusive.
Presentations refer to presentations held by the company to present annual and interim results. 22As with financial reports, the accessibility of presentations can be improved with alternative formats (e.g.webcasts, podcasts and transcripts).
Three different types of attributes were measured under the financial analysis category: financial highlights or summaries, financial ratios, and key performance indicators.Four different types of attributes were measured under the archive category: annual reports, interim reports, results presentations, and other investor presentations.

Relevant news
A key element of a good corporate website is keeping investors up to date with news about the company, its strategy and operating environment (IRS, 2013).Attributes measured in this category are categorised into three sub-categories: SENS, 23 media (press) releases made by the company, and news published by the financial media.

Investment case
The corporate website should provide users with a clear statement of strategy and vision (IRS, 2013).Attributes measured in this category are categorised into four subcategories: investment pack, forecasts, industry information, and corporate profile.

Shareholder information
According to the IRS best practice guidelines (IRS, 2013), the corporate website should provide an up-to-date source of shareholder information.Attributes measured in this category are categorised into 13 sub-categories: dedicated investor relations section, investor contact details, shareholder 21 Hypertext Markup Language. 22Investor presentations (all other presentations, excluding annual general meeting presentations) were measured under the investment case category.Annual general meeting presentations were measured under the shareholder information category. 23Stock Exchange News Service.
communications, PAIA 24 , company advisors, analysts, share price information, dividend information, shareholder information, shareholder meetings, financial calendar, listing information, and other shareholder services/information.

Bondholder information
Companies should provide clear information about their corporate debt situation, as applicable (IRS, 2013), by providing information such as their credit ratings and information on listed debt instruments.

Corporate governance
According to the IRS (2013), it is no longer sufficient for companies simply to refer website users to relevant parts of the annual report for corporate governance information.
Attributes measured in this category are categorised into 11 sub-categories: dedicated corporate governance link, corporate governance report, King III, directors, executives and management, board committees, management committees, code of conduct, memorandum of incorporation, insider trading policy, and whistle-blowing policy.

Corporate responsibility
A best practice corporate website should contain comprehensive information about the company's sustainability policies and data, including a detailed review of how policies are linked to environmental and social management (IRS, 2013).Attributes measured in this category are categorised into seven sub-categories: dedicated corporate responsibility link, reports, policies, BBBEE, 25 employees, corporate citizenship, and stakeholders.

Reliability and validity
As discussed, reliability and validity tests were based on the measurement results of 85 JSE-listed companies.An IIR score for each company was calculated by adding the scores of all individual attributes together.

Reliability
Cronbach's alpha was calculated for each of the 11 categories that were used in this study to categorise attributes and is shown in Table 3.
Although Cronbach's alpha for three categories (accessibility, timeliness and company information) is below the often-suggested acceptable level of 0.7 (e.g.Serrano-Cinca, Fuertes-Callén & Gutiérrez-Nieto, 2007;Nunnaly, 1978;Kelton & Yang, 2008), it can still be viewed as acceptable given the alpha of 0.51 as reported by Gul and Leung (2004) and 0.64 reported by Botosan (1997).Gul and Leung (2004) did however admit that the low alpha reported in their results suggested that random measurement error could reduce the power of the empirical tests in their study.
Part of the process in calculating Cronbach alpha is to determine the level of redundancy in the measurement instrument.No attributes in this study indicated significant redundancy.This implies that all attributes were used in the analysis.
Further to Cronbach's alpha, Pearson correlation coefficients were used to assess internal consistency.This study examined the relationship between the total IIR score and its disclosure components (i.e. the 11 categories) and the results are reported in Table 4.
Table 4 shows that the categories are all positively related to each other and that all relationships are statistically significant at the 1% level, except for three at the 5% level, one at the 10% level and one that is not statistically significant. 26

Validity
Statistically significant (all at the 5% or better level) positive correlations between the IIR score and size 27 , leverage 28 and listing status 29 , and negative correlation between the IIR score and ownership structure 30 were found.
These correlations are all, as expected, and are consistent with the results of former studies that have used these company characteristics to confirm the validity of their measurement instruments (as used to measure website contents - Chang et al., 2008;Orens et al., 2010;Trabelsi et al., 2008;Froidevaux, 2004).

The content analysis
Contrary to prior studies, the current study did not limit the content analysis only to the investor relations section 31 or to the homepage and a specified maximum number of drill downs32 .All internal hyperlinks available on the corporate website were therefore examined.Hyperlinks to third party websites were ignored, unless there was a clear link to the website with a specific and clear indication of the type of information (e.g.share price information) that would be accessed. 33n accordance with previous studies34 and given the importance to distinguish between the availability as opposed to the accessibility of information, the content of PDF documents (e.g. annual reports) was not examined, unless a dedicated and descriptive hyperlink was provided to guide users to the information (e.g.'For more information about the directors, click here').
27 Average market capitalisation of all trading days from December 2014 to November 2015 28 Ratio between debt and assets 29 Dummy variable representing one if the company is dually listed 30 Dummy variable representing one if one shareholder has more than 20% of issued shares 31 Froidevaux (2004), Bollen et al. (2006), and Kelton and Yang (2008), for example, examined only the investor relations section of corporate websites.

Esterhuyse and Wingard (2016)
Esterhuyse and Wingard (2016) developed their measurement instrument using the guidelines published by the Nielsen Norman Group35 and the IRS.The current study did consider the guidelines published by the Nielsen Norman Group, but as they were not as freely available as the IRS guidelines, the Nielsen Norman guidelines were not used.
From an analysis of the categories used by Esterhuyse and Wingard (2016), it is clear that their study focused on usability, financial information and shareholder information.
The usability focus was probably the result of the use of the Nielsen Norman usability guidelines.Granting that usability is extremely important, the current study argues that a valid measurement of investor relations should include attributes such as relevant news, investment case, bondholder information, corporate governance and corporate responsibility.Esterhuyse and Wingard (2016) further assessed all attributes as either available or absent, has used weights to emphasise the importance of presentation attributes and failed to report on any reliability or validity assessments.

Summary and conclusion
The purpose of this study was to develop a measurement instrument to measure the quality of IIR.Based on IRS published best practice guidelines, an extensive literature review and a pilot study, 346 attributes were identifiedorganised into 11 categories.
This study carefully considered the assigning of weights, but opted not to assign weights to either individual attributes or categories.The measurement instrument intends to measure the quality of IIR by measuring content as widely as possible, by including presentation attributes (i.e.accessibility, timeliness and navigation) and by allowing for the measurement of attributes as partially available (0.5) as opposed to being merely absent (0) or available (1).Breadth (comprensiveness), usability and timeliness criteria were mainly used to assess attributes as partially available.
The reliability and validity of the measurement instrument was confirmed based on the measurement results of a sample of 85 JSE-listed companies.This study will aid future research to examine IIR, and provide guidance to companies in the development of an IIR strategy.This study, however, made no attempt to distinguish between the information needs of the various users of information (corporate website users, for instance, vary from the naïve decision maker, to the institutional investor and analyst).Further research (e.g. through surveys and web server logs) is needed to ascertain to what extent the information available on corporate websites fulfils the information needs of specific investors.

Table 3 : Cronbach's alpha per measurement instrument category
: Ac refers to the accessibility category, N to navigation, T to timeliness, Ci to company information, Fi to financial information, Rn to relevant news, Ic to investment case, Si to shareholder information, Bi to bondholder information, Cg to corporate governance and Cr to corporate responsibility.
Notes24Public Access to Information Act.25Broad-based black economic empowerment.26Allfive of these associations relate to the bondholder information category.The bondholder information category measures only five of the total 346 attributes.

Table 4 : Correlation matrix: measurement instrument categories
: Ac refers to the accessibility category, N to navigation, T to timeliness, Ci to company information, Fi to financial information, Rn to relevant news, Ic to investment case, Si to shareholder information, Bi to bondholder information, Cg to corporate governance and Cr to corporate responsibility.*** = significant at the 1% level; ** = significant at the 5% level and * = significant at the 10% level Notes