The performance of Information Technology and the role of human and organisational factorsReport to the Economic and Social Research Council, UKby
Version 1.0, January 1996 Table of Contents
1.Summary1.1 The principal objective of this study was to gather and collate information from the major researchers and consultants in the UK regarding the performance of Information Technology (IT) and the role of human and organizational factors. The aims for the next stage of this work involve abstraction of general lessons arising from these results, development of plans to deliver improvements in this area, and wider dissemination of these findings. 1.2. The study was undertaken by members of OASIG, a DTI-supported Special Interest Group (SIG) concerned with the Organizational Aspects of Information Technology. 1.3. Substantial interviews were conducted with 45 leading researchers and consultants in the UK. Collectively these 'experts' drew on over 900 years professional work in this field and on findings from a total sample of approximately 14,000 user organisations. Between them these experts have researched and consulted in all the major sectors of UK economic activity, working with a wide array of users and suppliers, and with all forms of information technology. As such the authors of this report have confidence in the validity and generalizability of these findings for the UK. 1.4. The main findings from the first stage of this study are: The performance of IT (see section 8.1.)
The context of technical change (see section 8.2.)
Managing the development and implementation of IT (see section 8.3.)
The role of human and organizational factors (see section 8.4.)
The role of end users (see section 8.5.)
The role of managers (see section 8.6.)
Variations in performance (see section 8.7.)
'Best practice' in companies (see section 8.8.)
Wider improvements (see section 8.9.)
2.Objectives2.1.The objectives of this study are:
2.2.This report covers the first of these objectives. Plans for completion of the other objectives are outlined in section 9. 3.Background3.1.It has become clear that many new investments in Information Technology fail to meet their objectives. This is evident both in well publicised instances (such as the London Stock Exchange Taurus System and the London Ambulance Service), and also in everyday experiences across many organisations. Thus, whilst it is widely recognised that the new information technologies increasingly underpin most forms of commercial, industrial and governmental activity, there is also an abundance of evidence that many of these innovations are ineffective and under-utilised. 3.2.This study has been undertaken under the auspices of OASIG, a Special Interest Group concerned with the Organizational Aspects of Information Technology. OASIG was formed at the request of the DTI in 1989. It is managed by a small volunteer group of organisers, chaired by Professor Chris Clegg (University of Sheffield). A statement of the objectives of OASIG and a list of members of its organising committee are presented in Appendix 1. 3.3.OASIG is supported by the DTI but it is self-managing and funds its own activities. OASIG organises around 4 public meetings each year in the form of seminars and workshops. Attendees are drawn from a range of backgrounds, principally comprising practitioners from commerce and industry, consultants and academics. 4.Study group4.1.A number of people have been involved in planning and undertaking this study, in developing the research instrument, in collecting, collating and interpreting the data, and in preparing this report. They are all active members of OASIG and their names and that of a contact person are given in Appendix 2. 5.Research methods5.1.A group of OASIG members met at the Ergonomics Training Centre (located in the Business Design Centre), Islington in July, 1994 to plan this study. The overall goal was to examine, from as wide a range of sources as possible, the levels of performance of IT investments and the role of human and organizational factors. 5.2.It was decided at the outset to gather and collate information from the leading experts working in this field, each of whom has widespread experience of working with large numbers of user organisations. The aim was to provide findings on a scope and scale from which valid, generalizability, national conclusions could be drawn. This strategy was selected to complement findings from the existing body of detailed case studies whose principal weakness concerns their generalizability. 5.3. The study group met again in November, 1994 at the Ergonomics Training Centre. At this meeting the research approach was specified, the principal sources of information identified, and the first draft of the data collection method was developed. 5.4.It was planned to collect data from around 30 key sources 'representing' the major research groups and programmes, and the leading consultancies working in this field. 5.5.Data were collected by interviewing researchers and consultants using a semi-structured interview schedule. A copy of this schedule is provided in Appendix 3. Interviewees were briefed in advance by telephone and by letter. Some preparation was required by them. Each interview lasted, on average, around 1.5 to 2 hours, though some lasted considerably longer. Where practicable, interviews were tape-recorded and subsequently transcribed. After each visit a synopsis of the findings from their interview was sent to each interviewee for confirmation and elaboration. Interviewees were encouraged to add supplementary information to their interview reports. Several provided further documentation describing their work. 5.6.During the interviews data were collected on:
5.7.Further meetings were held at the Ergonomics Training Centre in February, May, September and December 1995 to interpret and collate the findings and to plan the next stages of the study. This work culminates in the presentation of this report. Future plans are described in section 9. 6.Sources of information6.1.Information has been collected from 45 leading researchers and consultants drawing on results from around 14,000 user organisations. 6.2.The full list of people interviewed is given below:
7.The experts and their experiences7.1.Personal experience7.1.1.Summary. Information was collected from a sample of experts with a wide range of experience and expertise. 45 of the leading experts in this field in the UK were interviewed. On average they each have over 20 years personal experience in this field, a cumulative knowledge base of over 900 years. To the best of the authors' knowledge such a collation of collective wisdom has not been attempted before in the UK. 7.1.2.As can be seen in section 6.2., the 45 interviewees are employed primarily by Universities or Consultancies. 31 of these (69%) include consultancy work as a major component of their work, and 27 (60%) include research; many do both. 7.1.3.Their work involves use of a wide range of methods and techniques, for example including interview and survey techniques, workshops and discussion groups. The work commonly involves gathering information, analysing results, giving feedback, advising clients, and evaluating outcomes. 7.1.4.The professional areas of expertise of the interviewees varies considerably, but most reported expertise in the domains of management, business, and social science. A small number of those interviewed have a background in engineering. 7.1.5.Interviewees were asked to give their personal views, rather than speak 'on behalf of' their employers. Nevertheless, as well as being leading figures in their own right, most of those interviewed work for some of the most prestigious organizations in the UK, many of them with world-wide reputations for excellent work. 7.2.Sample on which the findings are based7.2.1.Summary. Interviewees drew on a very broad and comprehensive sample of experiences. Between them they have researched and consulted in all the major sectors of UK economic activity, working with a wide array of users and suppliers, with all forms of information technologies, and using a full range of approaches and methods. The total number of user organizations represented in this sample was around 14,000. 7.2.2.The interviewees were explicitly asked to identify the sample of organizations and technologies from which they were describing their particular findings and experiences. Their answers varied enormously. For example, some limited their answers to a small number of detailed projects, whilst others drew on a vast range of professional work. Some chose to draw on a mix of detailed projects at the same time as surveys across a wide sample of companies. The average number of sampled companies was in excess of 300 for each interviewee, and the total number of organizations represented in this sample, including those surveyed, was around 14,000. 7.2.3.The sizes of the companies in which the interviewees have worked also varied. Given the nature of their work (ie. research and consultancy) it is perhaps unsurprising that most of these companies are medium sized to large. Nevertheless some of the interviewees explicitly included reference to small and medium enterprises, particularly in the manufacturing sector. Several interviewees stressed that their work tended to focus on those working at the forefront of developments in this field, both technically and in performance terms. 7.2.4.Interviewees were asked to identify the sample of sectors and technologies from which they were drawing their findings. All categories of sector were included in the experiences of those interviewed, ranging across the public and private sectors, manufacturing and services, basic industries and utilities, and local and central government. The most heavily represented sectors were: banking and financial services; manufacturing; distribution and transport; communications and telecommunications; central government; health services; and defence. The sectors represented are summarised in table 1 below. Table 1: List of sectors represented in study Agriculture, Forestry and Fishing Energy and Water supply Extraction of Minerals and Ores Food manufacture (incl. Brewing) Manufacture of Electronic and Electrical goods Manufacture of Chemicals, Metals, Mineral products (incl. manufacture of Oil, Steel, Glass, Cement) Metal goods engineering and Vehicles Textiles Tobacco Other manufacturing Banking and Financial services Communications and Telecommunications (incl. Postal services) Construction Distribution (including Retailing and Wholesaling) Education Entertainment and Leisure Health services Hotels and Catering Media Research and Development Security services Social services and Welfare Transport Central Government Local Government Defence Other services 7.2.5.Similarly the coverage of technologies was comprehensive. All the categories of technology were covered by the interviewees, including manufacturing and office based technologies, communications systems, and software engineering tools. The most commonly represented technologies were: information systems; communications systems; office systems of various kinds; CAD; EDI; manufacturing systems of various kinds; and decision support systems. The technologies covered are summarised in table 2 below. Table 2: Types of technology represented in study communications systems computer aided design (CAD) computer aided design computer aided manufacturing (CADCAM) computer aided engineering (CAE) computer aided production management systems (CAPM) computer aided software engineering (CASE) tools computerised numerically controlled (CNC) machine tools databases decision support systems (DSS) electronic data interchange (EDI) electronic mail expert systems intelligent knowledge based systems (IKBS) management information systems (MIS) manufacturing systems office systems order processing systems robots software engineering tools and methods teleconferencing/ videoconferencing telematics transaction systems word processing systems 7.2.6.The interviewees were primarily engaged in some mix of research and consultancy roles. Their work incorporated a wide range of methods and approaches. Methods included: interviews; questionnaires; observations; group discussions; use of company records; videos; task analysis; simulation; electronic data collection techniques; protocol analysis; and others. Approaches were similarly varied, and were variously described as interpretive, ethnographic, positivistic, socio-technical, action research, user centred design, organizational development, experimental or phenomenological. 8.FindingsThe major findings are described below, organized in 9 sections:
8.1.The performance of IT8.1.1.Summary. Interviewees were critical of performance levels achieved with new investments in IT. In the majority of cases new systems do not meet their objectives and are counted at best as partial successes, at worst as failures. Most systems are delivered late and over budget. Failures in this area are rarely purely technical in origin. Most organizations are poor at evaluating the effectiveness and impact of their investments in this area. 8.1.2.Interviewees were asked the extent to which new investments in IT meet their objectives. Some were highly pessimistic in this regard, claiming that no systems in their experience had met their objectives. Others were more sanguine, but very few reported success rates in excess of 50%. Of those who had examined this systematically, the most optimistic response was around 25 - 35%. On average interviewees estimated around 80-90% of investments in new technology fail to meet their objectives. 8.1.3.Many of those interviewed reported that performance evaluation is a difficult area to assess. They drew attention to the fact that careful and systematic evaluation of the operational performance of investments against their objectives very rarely takes place. There are a number of reasons for this, some of them political. For example, it is not necessarily in managers' interests to undertake such evaluations. Problems in this area are compounded by the fact that pre-specified objectives are often political statements to persuade senior managers to make an investment, rather than a commitment to expected levels of performance. In many cases too, objectives are not clearly specified, making evaluation even harder. This whole area of objective setting, performance review and learning represents a major set of factors contributing to the disappointing delivery of IT investments. 8.1.4.Interviewees were asked the extent to which new systems are delivered on time and within budget. Again detailed hard evidence proved to be in short supply. Several reported that systems are never delivered either on time or within budget as originally specified. The most favourable judgements were that 20-30% were successful in this regard. Most however reported fewer than 20% meeting time and cost targets. It was also noted that managers often alter plans and milestones in the light of events, adjusting time and cost targets as they go along. In part this reflects the fact that the organizations and environments within which IT investments take place, are themselves subject to change. In addition, people are rewarded for getting a new system in 'on time'—they can 'get it right' later (for example, in later versions). 8.1.5.Interviewees were asked to estimate the overall success and failure rates for new technology investments. When asked about projects that were abandoned, experiences varied markedly, for example ranging from as low as 5% to as high as 60%. Highest rates for abandoned projects were reported in the defence sector. Those that were implemented but failed ranged from 5% to 40%. Putting these two categories together and counting them as system failures, the average response was over 40%. Turning to systems that were implemented and were partial successes (ie. met some of their objectives), responses ranged from 10% to 70%. The average score here was 40%. Regarding systems that were implemented and met all their success criteria, responses varied between 0% and 30%, the most common response being around 10-20%. 8.1.6.Interviewees were asked the extent to which new systems meet expectations and give value for money. Opinions here varied. Some felt that developers' expectations are usually frustrated and that users have expectations which are unrealistically high. For some the whole issue of 'managing expectations' is a major issue for management and the other parties involved. 8.1.7.Given the other findings as described above, most did not feel that IT investments delivered value for money. Nevertheless some felt that organizations can get value for money even when the level of performance is disappointing, in part because their markets are such that some organizations cannot function and compete without investment in new technology. One further problem concerns the difficulty of assessing and obtaining value for money from infrastructure investments, ie. major IT projects that enable other investments to perform well. For some this represented an issue of growing importance given the pressures and opportunities to develop more integrated and larger scale systems. 8.1.8.People were asked why systems fail to meet their objectives. Many reasons were given and most respondents pointed to the organizational complexities in this area. Lack of attention to the human and organizational aspects of IT is a major explanatory factor and is manifest in poor management generally, poor project management, poor articulation of user requirements, inadequate attention to business needs and goals, and a failure to involve users appropriately. These issues are discussed further in the rest of section 8. For all the variations here, one trend is clear—failures of IT are rarely purely technical in origin. So far as this group was concerned the technology per se is not the problem. In the words of one of our respondents "80-90% of the important issues today are organizational". 8.2.The context of technical change8.2.1.Summary. Many interviewees stressed the huge significance of investments in IT, the technical and organizational complexities inherent in this area, and the immense difficulties these create for managers in particular. 8.2.2.Most of our respondents stressed the significance of IT, the fact that it has become an essential aspect of most businesses. This was expressed in a variety of ways, for example that the new technologies are now at the core of many operations, that they represent the major source of new opportunities, and that they can function as the principal catalyst for improved competitiveness. For all the problems that may exist with implementing and using new technologies, many organizations cannot function successfully without them. It was also recognised that investments in IT represent, for many businesses, their major sources of opportunities, costs and risks. 8.2.3.The field of new technology was seen as an area of increasing uptake and complexity. The technologies are becoming more pervasive in organizations but at the same time more complex. Complexity arises in part from the increased capability and functionality of new systems, and from their increased potential for integration. The scope of technical change and the speed at which it is taking place are increasing. New markets and new functionality are driven by the suppliers and this contributes substantially to a focus on technology and technical change. Unsurprisingly perhaps this is a world of fad and fashion (see section 8.6.7.). 8.2.4.More specific trends within the field of IT were also described. In particular IT budgets are becoming more decentralized and more spending is falling within the control of users than previously (in comparison with a centralized IT function). Outsourcing of development has become more common as has the purchase and tailoring of standard packages. These trends suggest a move away from the centralised power of the IT function within organizations. At the same time there is an increasing trend for IT to be linked with telecommunication systems in novel ways that provide major emerging opportunities for organizations that many are only just beginning to recognise. A further trend concerns the adoption and application of ideas and techniques from the emerging field of business process re-engineering (see section 8.4.4.). 8.2.5.Many of those interviewed pointed to a further area of complexity. The organizational impacts of IT systems are themselves very complex, difficult to predict and plan, and outcomes are the result of many interacting forces. Whilst certain principles can be identified many pointed to the uniqueness of organizational outcomes. 8.2.6.In addition, large systems evolve over considerable periods of time. The technologies themselves are shaped by complex organizational processes and decisions. This adds to organizational complexity. Often too, new systems are not developed in greenfield sites, and must fit with, or at least operate alongside, existing 'legacy' systems. Again the technical and organizational outcomes are difficult to predict. 8.2.7.Taken together, these factors mean that it is difficult, if not impossible, for any single individual or indeed professional group, to understand all aspects of change in these areas. Those who understand the technology tend not to appreciate the wider organizational issues, and those who have knowledge of these are often technically naive. This places a very high premium on finding ways of integrating different forms of knowledge and expertise (see section 8.6.6.). 8.2.8.These difficulties are compounded by four further factors. First, the business climate in the UK over the last few years had been dominated by recessionary forces, by pressures to downsize, delayer, etc.. IT is often used as an instrument to further these goals, and this makes it exceptionally difficult to focus on important human and organizational aspects in a satisfactory way. For example, it makes it difficult to ask the recipients of new technology to participate in what they may regard as their own demise. 8.2.9. Second, the organizations within which these multi-faceted changes are taking place are themselves often highly fragmented and differentiated, working with different goals, assumptions and languages. This makes co-ordinated effort (in this and indeed other areas) very difficult—this issue was stressed again and again by our interviewees. 8.2.10. Third, some trends within the IT industry further the processes of fragmentation and specialisation. An example here is the increasing use of outsourcing as a mechanism for developing software. 8.2.11. And fourth, for the reasons above, the whole domain of organizational and technical change must, at least in part, be analysed as a political issue within organizations. Many of our interviewees pointed to the importance of organizational politics. 8.3.Managing the development and implementation of IT8.3.1.Summary. Changes are seen as technology-led, addressing too narrow an agenda, and reflecting too technical an emphasis. Management of this area is heavily criticised for a lack of understanding of the non-technical aspects of change, for under-estimating the wider human and organizational issues, and for failing to act upon them. Cost reduction is the major motive for investing in IT. A number of methods and tools are in widespread use to support development and change in this area, but these are widely criticised for their strong technical orientation. 8.3.2. A wide range of motives were recorded for implementing new technology. The most common included cost reduction, increased output, improved quality and reliability, and innovations in products and processes. Most are concerned in some way with improving competitiveness. The most common single reason is focused on cost reduction. In many cases it was also recognised that the new technologies open up and offer new opportunities, previously not available to organizations. 8.3.4.Specific questions were also asked about the process of change and the ways in which it is managed. Five main conclusions can be drawn from the answers here. First, most of the respondents argued that change is predominantly technology-led (ie. the technology is considered first and commands most of the resources). A substantial group however, believed that organizations are becoming more business-led, motivated by a desire to meet and develop the needs of the business. Nevertheless some of the interviewees held somewhat cynical views, for example that: some businesses who believe they are business-led are, in fact, often led by their IT people who have to justify their existence; organizations may set out with business goals but the changes get hijacked by the technologists; and, whilst companies may espouse a strong business orientation, some do not necessarily behave in that way. 8.3.5.Second, a minority of respondents argued that organizations should be technology-led in the sense that advanced technologies provide the major source of new opportunities for organizational improvement, for example in the field linking IT and telecommunications. A key point here, however, is the recognition that their human and organizational aspects remain critical to their success. 8.3.6.Third, project management methods and tools are in widespread use; in part this reflects the common concern that projects run over time and over budget. Unfortunately these same techniques are widely criticised. Particular criticisms are that: they can create large bureaucratic infrastructures, adding more problems than they solve; they provide an excuse not to think (fostering a 'tick-box' mentality); these techniques neglect critical factors, most especially the human and organizational issues; and they help promote a technocentric orientation. 8.3.7.Fourth, structured methods and tools are often and increasingly used to undertake analysis and design work. Some strong criticisms are made of these methods; indeed this is an area that can evoke strong reactions. For example, some reported that these methods simply do not work, that they omit too much, and that they are too technically oriented. On a more positive note, some argued that one way forward is to incorporate explicitly human and organizational factors in these methods, in part as a means of ensuring that attention is paid to them. 8.3.8.Finally here, it is clear that IT development and implementation work is hugely constrained by a number of factors, including for example, the expertise of those undertaking the work, their orientation towards the task in hand, the expectations of their clients, managers and others, and the time available. 8.4.The role of human and organizational factors8.4.1.Summary. In general organizations are not successful at attending to the non-technical aspects of changing technology. It was widely felt that there is little consideration of these issues, that they are not well understood, that their importance is under-estimated and that action in this area is under-resourced. In particular not enough attention is paid to the impact of new technology on organizational structures and processes, or on job designs. 8.4.2.Questions here were concerned with a more detailed examination of the major non-technical issues that appear so problematic in the findings above. One question was concerned with the extent to which new technology systems are successfully integrated with business goals and business needs. Most respondents thought organizations are still not good at this, though they recognised there have been improvements over the last few years. The most optimistic view was that 25 - 50% of new projects achieve this integration of technology and business goals. Most estimates however were considerably lower than this. The 'blame' for this lack of integration was laid at the door of senior management and this issue is discussed in section 8.6. 8.4.3. Regarding attention to the impact of new technologies on organizational structures and processes, the answers were even less positive. The most common responses were that little or no attention is paid to these issues. In the words of one interviewee this area is "massively under-estimated". This represents a major gap in current practice. 8.4.4.Several interviewees drew attention to the emergence of a business process perspective in organizations, for example under the guise of Business Process Re-engineering (BPR) or Business Process Management. Many organizations have attempted to make changes to their working practices through the adoption of such a perspective, especially when introducing new technology. However, some interviewees expressed some criticisms of work in this area. For example, some described how BPR in particular is too focused on traditional tools and techniques and fails to address important organizational issues, particularly concerning organizational change and politics. Some were also critical of leading exponents of BPR who appear to espouse beliefs in the 'obliteration' of existing knowledge and practices, values which run counter to a commitment to organizational and individual learning and development. There was a perceived danger that organizations get carried away by the hype of BPR and that it becomes another managerial fad (see also section 8.6.7.). Some of our interviewees reported that the majority of BPR exercises are failing to meet their goals and expectations. 8.4.5.Regarding the impact of new technology on the way work is organized and upon individual job designs, the majority view was that this is hugely important but largely ignored in practice. Again this was seen as a topic that is significantly under-estimated. Where it is addressed this is because the job design implications of technical change are discovered, usually relatively late in the development process. 8.4.6.The above findings (sections 8.4.2. to 8.4.5.) demonstrate that IT remains technology-led. IT is not seen in an integrated way as raising sets of related business and organizational issues. 8.4.7.Questions were also asked regarding the attention paid to usability issues (and 'user-friendliness'). Respondents saw this as one area of significant improvement over the last few years. The field of usability is at least recognised as of importance for the performance and effectiveness of IT systems. Interviewees reported that usability issues are successfully addressed in as many as 60-70% of projects. However, there were criticisms that the area is still not sufficiently well understood, that technologists adopt a rather mechanistic view of usability, that users are still not involved enough in improving usability, and that the area is often defined in technological terms (eg. the problem of usability is translated into one of moving from character-based to graphical user interfaces). Nevertheless, the usability community has made successful efforts over the last few years in getting these issues accepted as legitimate and significant concerns during new system development and implementation. This has been assisted by the development and uptake of practical methods and tools to improve human-computer interface design. 8.4.8.Responses on the attention paid to health and safety issues were less encouraging. Most felt that legislation had helped this area but that the issues themselves are not fashionable. Lip service may be paid but in most cases health and safety are not given proper attention. Similar findings were reported regarding ergonomic issues. These are sometimes successfully taken into account but not in the majority of cases. Many new systems are badly designed from an ergonomic perspective. 8.4.9.More positively, our interviewees reported that issues associated with training and skills are often considered, with particular focus on how to use and operate the new systems. Critics however argued that practice remains technically driven; thus users are trained in how to operate the technology rather than in how to do the job better. There were also criticisms that work in this area is often under-resourced, that training is often provided after the system is in operation, that training plans fail to be linked to consideration of new job designs and working practices, that standards are often too low, and that corners can get cut in the race to get systems implemented. Thus, whilst there may be large spending on training, it is not necessarily effective. Overall our respondents estimated that training and skills are successfully addressed in around 30-40% of cases. 8.5.The role of end users8.5.1. Summary. All interviewees argued that users have important roles, but were critical of practice. Behind all the rhetoric in this field, respondents reported disappointing practice. In most cases users do not have a substantial influence on system development. Whilst there is a trend towards more user participation, this is from a low baseline of activity. Lack of substantial progress in this area is clearly related to the dominant technical orientation described in previous sections and the widespread failure to address human and organizational issues. There are substantial barriers to undertaking successful user participation. 8.5.2.All interviewees argued that users have a crucial role in system development, most especially in identifying their needs and requirements and in actively helping design and implement new systems. Unfortunately users are rarely successfully involved in these ways and actual levels of user involvement are typically low. The whole area of user participation is seen as poorly understood and managed. 8.5.3.When users are involved this is most often in analysis work (where an analyst may observe and interview them to 'capture their needs') and subsequently in acceptance testing (ie. once the system is developed). Many described that user involvement is somewhat sporadic and inconsistent. In particular, users are rarely influential in the actual design of new systems. Some reported that this has been made worse by the use of structured methods and tools for system development, which can act to constrain user roles. Professional users of new technologies are more powerful and have more success in making their views heard. 8.5.4.Several of our interviewees reported that the trend is towards more user involvement, but this is from a low baseline of activity. 8.5.5.In the majority of cases developers are still seen as the 'owners' of most new technologies, and the most powerful party in their development. Even the user managers who ultimately will be responsible for managing and running a new system are seen as less influential and lacking in ownership. 8.5.6.A number of interviewees described the costs or penalties of failing to involve users. These include: threats to system viability; an increased likelihood that the new system will not be integrated into the business; and increased chances of system failure. The benefits of involving users are that it helps create a better system, at the same time as increasing user ownership. 8.5.7. Many interviewees identified substantial barriers to successful user participation. These included: lack of understanding and skills on the part of all parties, including the users themselves, their managers and the developers; sets of expectations regarding the limited role of users; the pressure to deliver systems to tight deadlines, and more generally a lack of money, time and other resources; the use of structured methods which militate against user involvement; the lack of established methods and tools that promote and foster user participation; the antagonistic, adversarial relationships that can develop between developers and users; the unwillingness of managers to expose what they may feel are their own weaknesses; and the problems managers face in managing complex change projects with all the uncertainties and risks involved. A substantial investment of time, effort and money is required to overcome these barriers, in particular in the fields of education and training. This sample of respondents believed that involving users effectively costs money; but not involving them, costs more. 8.6.The role of managers8.6.1.Summary. Management issues pervade this area. In so far as managers bear a major responsibility for action and performance within organizations, their roles in the context of new technology bear scrutiny. Senior managers in particular are criticised for their lack of understanding of the links between technical and organizational issues, and for their emphasis on cost reduction. Managerial and organizational behaviour more generally are criticized for the high levels of organizational fragmentation and for the high incidence of fads and fashion. 8.6.2.Several of our interviewees drew attention to the general lack of understanding on the part of senior managers in the area of new technology and its impact. At one level this may seem unsurprising in that the new technologies have become influential relatively recently and certainly after most current senior managers have been educated and trained. Nevertheless, senior managers were criticised for not developing a much better understanding of the business opportunities and difficulties arising from new technology. Some interviewees believed that senior managers have abdicated their responsibilities in these areas, and left the technology to the technologists. Where this is true, organizations pay a heavy price. 8.6.3.A more specific criticism of senior managers was that they lack an integrated view of new technology, failing to pay adequate attention to the human and organizational issues described in section 8.4. above. Clearly these issues are interconnected. 8.6.4.Managers were also criticised for their motives in investing in new technology. Most are perceived as pursuing cost reduction, for using technology to reduce staff and skill levels, and for seeking rapid payback and short term gain (see section 8.3.2.). Related to this they are criticised for the failure to adopt a longer term perspective, for not taking a strategic view, and for failing to ensure their investments in technology reflect and further the needs and goals of the business. 8.6.5.Some interviewees also stressed the issue of managerial values. Their view was that too many managers perceive their staff as costs rather than investments, as units of production, and as sources of error and unpredictability. In this perspective, managerial lack of attention to the human and organizational issues as described above is no accident; rather it reflects a set of commonly held values. 8.6.6.One of the strongest and clearest messages that emerges from these interviews concerns the levels of fragmentation within organizations. Company functions and specialisms were described as highly differentiated and separate, often with their own set of professional interests, agendas and specialist languages. This view of fragmented organizational life and its associated territorial and political battles helps explain why co-ordinated effort proves very difficult to manage. This fragmentation is a pervasive characteristic of organizational practice and cannot be stressed too much (see section 8.2.7.). It is especially germane to the domain of new technology; for example, interviewees repeatedly described the dislocations that exist between strategy and development, development and use, and use and evaluation. But this approach to the development and implementation of IT can be regarded as a special case of a more general phenomenon, concerned with the fragmented and political nature of organizational life and managerial practice. 8.6.7.A further recurring criticism of managerial behaviour concerned fads and fashions. Several interviewees reported that organizational life is prone to waves of fashion; furthermore some argued that the life cycle of these fads is getting shorter. For example, the fashion that was Total Quality Management was superseded by Business Process Re-engineering. Just In Time manufacturing went Lean and then became Agile. Management gurus, whether senior business figures, academics or consultants, help engender and feed the hype and these various searches for excellence. The fashions are also fed by technologists and others working in the supply industry. Interestingly in this context, several interviewees were keen that the non-technical aspects of new technology, the human and organizational factors, do not become the next fashion (see section 8.9.7.). 8.6.8.Project managers were also criticised for their lack of attention to the human and organizational aspects of the systems they are responsible for developing and introducing. In their defence however, it was recognised that people working in these roles are usually not expected to address these issues, are not rewarded for doing so, are not educated or trained in them, nor supported adequately in any endeavours they may make in these areas. 8.7.Variations in performance8.7.1.Summary. Interviewees were asked to identify variations in their samples, in particular whether any particular sectors or technologies performed better (or worse) than others. Overwhelmingly, respondents reported that the variations that occurred in their experience were not a function of a particular market or sector or technology. Excellent performance is open to all and is more related to good management than to any other single factor. Nevertheless, some variations in performance were described and three minor trends can be discerned. 8.7.2.First, several interviewees reported that technical systems that operate within departments, as opposed to across functions, tend to operate more effectively. This is because they are not prone to the problems of integrating effort across different parts of the business. Such systems also tend to be on a smaller scale, which also makes their management easier. 8.7.3.Second, some differences were perceived comparing different sectors of the economic activity. Thus manufacturing industry for example was viewed, by some respondents, as further advanced and more effective than most others. The health sector was perceived as performing rather poorly in the field of IT. Similarly the defence sector was seen as prone to problems, in large part perhaps because of the size and scope of its investments and the long lead times involved. 8.7.4.Third, although not a major focus of this study, some respondents made international comparisons. For example, some reported that people in the UK have a relatively negative attitude to new technology and that British managers are too conservative, lacking understanding in this area. Several reported that UK companies perform worse than competitors in the USA and Japan. 8.8.'Best practice' in companies8.8.1.Summary. Many interviewees described experiences of working with very successful organizations and were able to summarise what they regarded as 'best practice' in this area. They reported that the research and consultancy communities now understand a great deal about these issues. There were remarkably high levels of consensus in these findings. Several reported that the problems in this area are mainly concerned with disseminating this understanding and messages concerning 'best practice''. These findings will be elaborated in the next stages of this project (see section 9.). 8.8.2.As described in section 8.7.1., a critical initial point is that high levels of performance are not restricted to particular markets, business sectors or indeed types of technology. 8.8.3.A second general point is that organizations that perform well in these areas tend to undertake many of the activities described below; these are not alternative routes to success. Many of these factors are interconnected and potentially reinforcing. They should be seen as a set of necessary but not sufficient ingredients for success. 8.8.4.The following features were widely described as characterising 'best practice' in this area. Their order is not intended to bestow any significance.
8.9.Wider improvements8.9.1. Summary. Interviewees were asked for their judgements on what needs to be done on a national scale to improve performance and practice in this area. Improvements were not seen as the sole responsibility of managers working within individual companies. A key goal was for the understanding of these issues to become embedded in practice as part of the normal culture of managing organizational and technical change. A number of far-reaching suggestions were made to promote improvements, for example in the areas of education, the professions and their societies, national policy, and research and development. These were seen as supporting 'best practice' initiatives within companies (as described in section 8.8.). These findings will be elaborated in the next stages of this project (see section 9.). 8.9.2. Most respondents argued that improvements in this area are not solely the responsibility of individual organizations and the managers within them. Whilst senior managers in particular, can achieve a great deal in the pursuit of 'best practice' within companies (section 8.8.), there are some wider issues that demand attention. In that sense most of the interviewees were unashamedly interventionist in their views—market forces alone would not improve practice sufficiently. 8.9.3. Some also believed that no single interest group or profession can 'unlock' all the forces that currently 'conspire' to produce poor performance. In this view the poor performance of new IT systems is the result of a complex and interacting set of forces that will not be easy to change. A great deal of integrated effort is required. Indeed if it were easy, it would have been done already. 8.9.4. On a more positive note, several people argued that 'big ideas' have their time and that there is a critical mass of thinking that now recognises the essential role of human and organizational factors. This is reflected in the consensus in perspective and viewpoint that exists across all the experts we interviewed. 8.9.5. Many respondents reported that the research and consultancy communities know and understand a great deal about this area; it is a source of frustration that there seems to be a need to keep re-learning these lessons, 'endlessly reinventing the wheel'. A major priority becomes the diffusion of existing knowledge and expertise. 8.9.6. At the same time, several interviewees felt concerned that efforts to focus on human and organizational issues do not become another management fad, only to be followed 2 or 3 years later with damning reports that 70-80% of interventions in this area fail to meet their objectives. 8.9.7. A key goal then is that these issues become 'institutionalized' and embedded in practice. They become part of the natural way of doing things, part of the culture of managing organizational and technical change. This requires a multi-faceted and long term strategy. 8.9.8. Some of the professional judgements offered by the interviewees are listed below. These should be seen as preliminary ideas; more work on these issues will be undertaken in the next stages of this study (see section 9.).
9.What next?9.1.A number of further activities are planned to complete the objectives of this study. To abstract the general lessons from the data, to plan actions for improvement and to disseminate the findings (objectives 2, 3, and 4, see section 2.1.) a series of Focus Group meetings will be held during 1996. The objectives of each meeting will be to interpret the findings so far, to plan a set of actions for improving practice (both within companies and on a wider national scale), and to discuss the optimum strategy for their wider discussion and dissemination. 9.2.At this stage meetings with three different groups of people are envisaged. Focus groups meetings will be held in a private company which is a major developer and user of IT, with a panel of users with extensive experience in this field, and with the OASIG study group itself. Preliminary discussions have been held with British Telecom and the HUSAT Research Centre in Loughborough who have given provisional support to this venture. 9.3.Thereafter the general lessons, key findings and actions for improvement will be summarised in a short report for widespread circulation. Other dissemination activities are also envisaged. 9.4.An application for funding will be made to ESRC to support this work. 10.Acknowledgments10.1.The authors wish to thank all the interviewees who so willingly spent time describing their views and experiences. The findings described in this report represent a compilation and interpretation undertaken by the authors; they do not necessarily reflect the views of any single individual. 10.2.The active support given by the ESRC in part-funding this work is also acknowledged. 10.3.The authors thank Tom Stewart of System Concepts Ltd for use of the Ergonomics Training Centre in London, and also Claire Calcutt for making the arrangements. Particular thanks are also due to Glynis Eblet of the University of Sheffield for helping organize and administer this work. 11.Appendices11.1. Appendix 1. The objectives of OASIG and members of its Organizing CommitteeThe objectives of OASIG are:
The current members of the Organizing Committee are:
For further information regarding OASIG please contact: Mrs Glynis Eblet
11.2. Appendix 2: Members of the study groupThe members of the study group responsible for this work are:
For further information regarding this study please contact: Professor Chris Clegg
11.3. Appendix 3: Interview ScheduleThe performance of Information Technology and the role of human and organizational factors A National Study undertaken by OASIG Background briefing We are undertaking a study gathering, collating and disseminating information based on the experiences of those engaged in research and consultancy on these issues. The principal aim of this study is to gather and collate information from the major researchers and consultants in the UK regarding the performance of IT and the importance of the human and organisational aspects. NOTE 1.all views expressed are confidential and non-attributable 2.all informants will receive a draft synopsis of the interview for their comment/ confirmation 3.all informants will receive a copy of the report 4.in our report we plan to list the research groups and consultancies from whom data has been gathered Section A.The informant's PERSONAL EXPERIENCE and expertise in this area Talking about you personally: *For how many years have you personally worked in this field? *How many different companies/ organizations have you worked in/ consulted in/ or researched? (NB this question is about organization in which you have consulted or researched; it is not asking about the number of employers/ jobs you have had) *What are your areas of relevant expertise/ special knowledge? *What is your normal role? (eg consultant/ researcher) *What does this normally involve? Section B.The RESEARCH GROUP'S or consultancy's experience and expertise in this area Talking now of the organization/ company in which you work: *For how long has your company/ group been working in this field? How many person years effort has your company/ group invested in this field? *How many companies/ organizations has your organization worked in/ consulted in/ researched? *What are your organization's areas of relevant expertise/ special knowledge? *What are their normal roles? (eg consultancy/ research) *What does this normally involve? Section C.The SAMPLE of research/ consultancy on which findings are given We need to be clear about the sample of work on which the findings you describe later are based *How many companies/ organizations are you drawing on? *Of what sizes? (eg by turnover? by employee numbers?) *In which sectors? see list 1 *What types of new technology have you covered? see list 2 *What has been the nature of your relationship with them? *In general, what has been your remit? *What methods of study/ data collection have been used? *(For research groups, has there been a particular theoretical approach/ epistemology eg positivistic, interpretive, ethnomethodological?) Section D.The FINDINGS from the group's research/ consultancy Either: Can you please describe to me your major findings from this collected body of work? (as described under C above) (looking for a set of 8 - 12 major findings in the interviewees own words; any questions or prompts are to check our understanding) Can you give me some specific examples please? Or: Can you please describe the major findings from say two to three projects of particular interest? And, how do these relate to your more general experiences? Can you give me some specific examples please? VARIATIONS *Given the pattern of findings reported above, are there any systematic differences across companies/ organizations? Eg by size, by sector Section E.SPECIFIC ISSUES & THEMES Here we prompt the interviewee according to our specific interests NB In each case questions are in relation to the sample described in Section C above PERFORMANCE *To what extent have the systems/ projects met their objectives? (To what extent have such systems/ projects been evaluated against their initial objectives? if not, why not?) *To what extent have they met the expectations and aspirations of their investors? (ie people who pay for them) their developers? their users and their managers? (to what extent have user requirements been met?) the customer/ consumer/ citizen? *To what extent have the systems/ projects given value for money? and/ or made an acceptable return on investment? *To what extent have they been delivered on time? and within budget? *What percentage of systems/ projects: have been abandoned? have been implemented but failed? have been implemented and been partial successes? have been implemented and have met their success criteria? would not have been funded if the actual levels of performance had been forseen? *Where systems have failed to meet their objectives, why has this been the case? technical failings? technical complexity? human and organizational reasons? financial failings? other? please specify ...... THE PROCESS OF CHANGE *Why have organizations/ companies undertaken changes in this area? (eg implemented IT) what motives? *To what extent has the process of change been technology-led? (ie techy considered first and commands most resources) been 'project managed'? used methods and tools? (eg project mgt tools, such as Prince; structured development methods, such as SSADM) been business-led? (ie to meet and develop the needs of the business) been customer driven? (ie to meet and develop the needs of the market) *What has been the role of users? when have they been involved? and how? what was their brief? what training/ preparation did they receive to support their involvement? how powerful were the users? (eg was there a powerful champion for users?) if conflict occurred between project managers and users, whose views would usually prevail? HUMAN & ORGANIZATIONAL FACTORS *To what extent have the systems/ projects SUCCESSFULLY included consideration of the following issues during development and implementation? (NB success here means not just that the issue was considered, but also that the outcome was successful) their contribution to business goals/ business needs? their impact on organization structures and processes? their impact on work organization and job design? their usability (and user friendliness)? their impact on health and safety? their ergonomic aspects (eg physical layout)? their impact on training and skills? other? please specify ...... *where they have been considered, can you summarise how? VARIATIONS *Given the pattern of findings reported above, are there any differences across companies/ organizations? Eg by size, by sector Section F.The informant's PROFESSIONAL JUDGEMENTS in this area *Please summarise your professional judgements of this area? *What are your key messages? And who needs to hear them? *Who needs to do what to make improvements in this area? *What do you think are the underlying values prevalent in this area? Are these problematic in any way? *Are there any other topics you would like to raise? THANK YOU Want to comment on our website? Fill in our 2-minute questionnaire. Copyright ©2000 System Concepts Ltd All rights reserved. |