eGovernment for Development

Evaluation

Causes of eHealth Success and Failure: Design-Reality Gap Model

• The Model
• Examples
• Archetypes
• Action

Why eHealth Projects Succeed or Fail: The Design-Reality Gap Model

Central to e-health success and failure is the amount of change between 'where we are now' and 'where the e-health project wants to get us'.

'Where we are now' means the current realities of the situation.  'Where the e-health project wants to get us' means the model or conceptions and assumptions built into the project's design.  eHealth success and failure therefore depends on the size of gap that exists between 'current realities' and 'design of the e-health project'.

The larger this design-reality gap, the greater the risk of e-health failure.  Equally, the smaller the gap, the greater the chance of success.

Analysis of e-health projects indicates that seven dimensions - summarised by the ITPOSMO acronym - are necessary and sufficient to provide an understanding of design-reality gaps:

• I nformation
• T echnology
• P rocesses
• O bjectives and values
• S taffing and skills
• M anagement systems and structures
• O ther resources: time and money

Putting these dimensions together with the notion of gaps produces the model for understanding success and failure of e-health that is shown in Figure 1.

Figure 1. The ITPOSMO dimensions of e-health project design-reality gaps

Design-Reality Gap Examples of eHealth Success and Failure

Full Case Study Example:

1. Computerising a Central Asian Epidemiology Service

eHealth Success Thumbnail Sketch:

A 'ComputerLink' scheme for 26 home-based AIDS patients was set up that put them in touch with each other individually and in groups, and with an encyclopaedia.  In this case, system design and reality for the AIDS patients were often well matched, along the ITPOSMO dimensions:

• The information dimension : ComputerLink was designed in a way that offered great flexibility of information access either via the encyclopaedia or via queries in public/private email messages.  Because of this flexibility, users were easily able to meet their real information needs through the system.
• The technology dimension : ComputerLink was designed around simple and straightforward technology using basic PCs, simple software and existing networking links, creating little gap between desiged and actual technology.
• The process dimension . ComputerLink was designed so that it supported the real pre-existing information-seeking and communication processes that AIDS patients undertook prior to computerisation, creating virtually no gap on this dimension.
• The objectives and values dimension .  ComputerLink's design met patients' real needs and provided something they said was valuable to them: the ability to interact with other AIDS patients and to have more information about their condition.  It was also designed to meet the very different individual objectives found in reality, from those who wanted to interact with a group on a daily basis to those who wanted occasional individual communication or anonymous 'support' (from the encyclopaedia).
• The staffing and skills dimension . ComputerLink was designed to require the input of only one project nurse on an irregular basis and to require only a limited number of new skills for system use.  There was thus little gap between designed and real human capability requirements.
• The management and structures dimension . ComputerLink was designed to fit within the existing health structures and, as noted, required only the addition of a fairly simple management framework of one nurse to monitor the system.
• The other resources dimension . ComputerLink was designed to have very low implementation and operation costs.  Its financial design was therefore well matched to the real finances available.  It was also designed to meet the time resource realities of patients by providing 24-hour access.  Participants were therefore able to match expenditure of time to their availability periods.  For example, there were as many logins between 10pm and 3am as between 10am and 3pm.

All of this meant only limited gaps between ComputerLink design and patient/health system reality.  The result was success.  The scheme was a qualitative success, as judged by participant ratings, and a quantitative success, as judged by the fact that:

• ComputerLink was used, on average, 300 times during the six-month evaluation period by each patient;
• its private email and public email forum components were used more than 10,000 times; and
• its electronic encyclopaedia of AIDS-related information was accessed nearly 800 times.

eHealth Failure Thumbnail Sketch:

A hospital attempted to introduce an expert system for computerised coloscopy.  There were significant gaps between system design and hospital reality, along dimensions including:

• The information dimension : the expert system was designed to produce a set of statistical information on coloscopy, but it emerged that there was no significant demand in reality for this information.
• The technology dimension : the expert system required a relatively powerful technological infrastructure, which differed markedly from the hospital's existing technological realities.
• The process dimension : the expert system was designed to automate many of the currently human decision-making processes around coloscopy, creating a significant gap between process design and current process reality.
• The objectives and values dimension : because of the process automation, the system's design did not match well with the real objectives and values of medical staff who feared automation and who believed that human inputs remained critical.  Nor did the expert system's objectives match well with the real priorities of senior hospital managers (leading to their providing little, if any, support for the project).
• The staffing and skills dimension : the expert system was relatively difficult to use and there was thus a significant gap between its design requirements and the reality of availability and expertise of hospital staff.
• The other resources dimension : the expert system was both time-consuming and costly to operate. This created a serious gap between the system's design requirements and the realities of resource availability within the hospital.

Overall, there was too great a gap between the expert system design and the realities of the hospital context into which it was being introduced.  The result was failure: the project was abandoned.

Design-Reality Gap Archetypes of eHealth Failure

eHealth failures come in more varieties than Heinz.  However, archetypes of failure do exist: situations when a large design-reality gap - and, hence, failure - is more likely to emerge.

Hard-Soft Gaps .  Health information systems can often be designed according to hard, rational models.  These can come from various sources.  Where technical staff dominate design, they can impose a very mechanistic design.  Where managerial staff dominate design, they can impose a very financially-rational design.  Where doctors dominate design, they can impose a very medically-rationally design.  The trouble is, many healthcare organisations don't adhere to these hard models.  In reality, they can be dominated by 'soft' factors: people, politics, emotions and culture.  When a hard e-health design meets a soft reality, there's a large gap, and a strong likelihood of failure.

Private-Public Gaps .  Private healthcare systems and public healthcare systems are different.  Private systems have different objectives, different levels of resourcing, different values, different case mixes, and different information priorities.  No problem.  Except that too many IT vendors, consultants, systems managers et al forget this.  They pick up a health information system designed for the private sector.  Then they try to shoehorn it into a very different public sector reality.  It's a classic case of square pegs and round holes.  The large design-reality gap generates lots of heat and noise, not much light and, ultimately, plenty of failure.

Country Context Gaps .  It sometimes seems only the first half of 'Think Global, Act Local' gets remembered.  Health professionals seeking quick fixes try to pull solutions off-the-shelf from other countries.  But New Delhi isn't New York, and Johannesburg isn't Jakarta.  So there's often a large design-reality gap when you try to introduce in country X an e-health system designed for country Y. The frequent result: tears before bedtime.

Taking Action on Design-Reality Gaps

Follow this link for further details about actions to take to reduce the risk of e-health failure.

Basis for analysis: a) Brennan, P.F. and Ripich, S. (1994) 'Use of a home-care computer network by persons with AIDS', International Journal of Technology Assessment in Health Care , 10(2), 258-272, for success thumbnail; b) Guah, M.W. (1998) Evaluation and Analysis of Multimedia Information System Design and Implementation at the Coloscopy Unit of St. James University Hospital, Leeds, UK , MSc dissertation.  School of Management, UMIST, Manchester, for the failure thumbnail; c) the cited full e-health case study; and d) Heeks, R., Mundy, D. & Salazar, A. (2000) 'Understanding success and failure of healthcare information systems', in Healthcare Information Systems , A. Armoni (ed.), Idea Group Publishing, Hershey, PA, 95-128