Health IT – TjilpiDoc

Photo by Toa Heftiba u015einca on Pexels.com

Software Design is an arcane subject, a long way from the day to day practice of Health Practitioners. Yet we all use computer systems – indeed they are central to our day to day practice. Good Health IT design is important to efficiency, work satisfaction and Clinical Safety, yet it appears not to be considered as an important factor in the commissioning of Health IT systems .

In this article I explore some relevant issues.

“Technical Debt” (Ref 1)

The design of a new system is a significant investment in money and resources. There is pressure to deliver on time and on budget. During the process compromises will be made on design and system functionality. Sometimes problems that should be solved now are deferred in favour of a “bandaid” solution. This incurs a “Technical Debt” which may have to be repaid later with further design work.

Data Modelling is an example of “technical debt”. This can be important when different systems need to communicate but is not so critical within a single system. Data sent across the interface between the systems must be carefully structured or serious problems can arise when the format of the data is not compatible. For example, a numerical quantity can be specified in various ways – signed integer, unsigned integer, floating point decimal to name a few. In programming, these quantities have different lengths and are represented and manipulated in different ways. If this is not recognized and allowed for in design, an error will result.

A special program or script may have to be devised to parse and convert data when it crosses from one system to another. If the work of data modelling and interface standard specification has been deferred at the design phase, “Technical debt” has been incurred which must be “repaid” at a later date. (ref Sundberg). There does not seem to be much interest from Vendors or Buyers in a formal Data Modelling system.

Data Modelling – why?

Any significant software system must store data to be useful. Health systems require large quantities of often complex data which to be stored, recalled and manipulated. A particular data entity can be represented in various ways. For example a pulse rate is likely to lie within a certain range. It will not be negative and is not measured in fractions Thus it could be represented by an unsigned integer quantity – this takes up much less memory space than a floating point number for example. On the other hand a vaccine temperature measurement will require recording in decimal fractions of a degree and might be negative. Thus a floating point number is required to represent this measurement. Additional “metadata” might be required for interpretation of the figure such as the means of measurement. Suitable ranges could be included in the data definition. This would allow alerts to be generated in decision support systems when the measurement falls outside the normal range.

Data modelling is the process of looking at data entities to be stored in a system database and specifying such constraints and data types. It makes the data computable. It then becomes more useful for search and decision support systems. It also allows the specification of data standards at interfaces between systems and may remove the need for “middleware” to connect systems. The internet is a good example where standards for data and interfaces have been agreed – this allows many disparate systems to communicate.

OpenEHR (ref 2) is an example of a data modelling system which is gaining increasing acceptance throughout the world.

Standards and Interoperability

– “the good thing about Standards is that there are so many to choose from”

There are strong commercial reasons why standardization is so hard to achieve. In everything from razors to printers there appears a plethora of shapes, sizes and standards. Of course when there is a need to link components together a standard interface matters. A razor blade will not link to a handle unless the relevant components are the correct shape and size. Reducing the shape and size of all razor blades to a common standard allows competition and invariably results in a reduction in price. Thus there is a strong commercial incentive for manufacturers to resist standardization, particularly if they hold market dominance. Windows and Microsoft made Bill Gates one of the richest men in the world. Apple is one the largest corporations in the world on the back of its proprietary operating systems and products. This intellectual property is jealously guarded. There are many examples where corporations have used their market power and closed intellectual property to maintain their market position and increase profits. Microsoft has been fined repeatedly by the EU for anticompetitive behaviour. One of the largest IT corporations in US health IT (Epic Systems) had to be forced by the Obama administration to open its systems to interface with outside systems. (Ref 3)

This commercial incentive to resist standardization and interoperability appears not to be acknowledged as an issue when governments are procuring Health IT systems. Just as the razor manufacturer can charge a premium for blades that fit their handle, health IT vendors appear able to charge eyewatering figures for their programs which are proprietary and do not interface with other systems. The user must pay an ongoing licence fee to continue to use the software. Moreover, because their source code is proprietary, no one else can work on it for upgrades and bugfixes – thus they can charge a premium for ongoing support. Changing to another system is expensive. Data will have to be migrated – the vendor will charge to access and modify data. Because it is not standardized and modelled, software will have to be written to migrate it to another system. Users will have to be retrained. All these costs mean that the customer is effectively “locked-in” to an expensive system, often of mediocre quality.

The NHS upgrade of 2002-11 was probably one the most spectacular and expensive Health IT failures of all time (see “Why is Health IT so hard?”). At the time there was much talk of Open Source systems and Interoperability. Yet more than 10 years later the NHS IT scene is still dominated by large commercial corporations, and Government is still paying large amounts for disparate systems. The Nirvana of Interoperability seems as far away as ever.

Software quality and why it matters.

Large software systems may contain millions of lines of source code. They are constantly modified and extended often over a long period. Many programmers may work on the system over time.

A high quality system has less bugs, is less likely to fail catastrophically, but perhaps most importantly can be extended and modified.

This is important because there will inevitably be software errors (“bugs”) in a large system. The probability density of bugs increases with software complexity and can even be quantified. Many are “edge cases” or of nuisance value. But some are critical and must be fixed.

The user will almost certainly want to make changes to the software as their business needs evolve or issues never addressed at the initial deployment become apparent. Some of these issues arise as a result of “Technical Debt” (see above). Some arise because of the “Complex” nature of the system and the fact they could not have been easily predicted as a result. (see “Complexity and Software Design”)

Some software is regarded as “legacy” (ref 4) – this means essentially that it cannot be modified without unpredictable failures – it has become “frozen”. While the program may function well enough, the quality of the underlying source code is such that making changes and fixing bugs is difficult if not impossible. This happens when the initial design is poor and changes are not carefully curated and managed over time. The code is not well abstracted into components, it is not well documented, interfaces are not discrete and well described, variables are “hard-coded” in different locations, and the codebase is not testable. A well designed program is generally separated into database, business logic, and user interface “layers”. The interfaces between these layers are well described and “discrete”. It should be possible to substitute different programs in each layer and have the whole system continue to function.

A modern software development suite will automatically test code after changes are made and will employ a versioning system to manage change. One approach to Software Design is to write a test for a section of code and then write the code to meet the test.

Another characteristic of “legacy” code is that it is usually proprietary. It may even use it’s own proprietary language and coding framework. It takes time and resources to develop software components that we are all used to such as typical window behaviour, “drag and drop” and “widgets” that form part of Graphical User Interfaces. If the code is proprietary it may not have had the benefit of the work of many programmers over time that Open languages and frameworks have had. This may explain the “primitive” interface look and function that many large enterprise systems have.

It is not standard practice, even when acquiring large enterprise level systems, to require an independent analysis of underlying software source code and coding systems quality. Customers generally look only at the function of the program. This is surprising given the sums of money that are routinely spent on these projects. The vendor will usually cite commercial confidentiality to prevent such scrutiny. This may also be a reason why the “waterfall” (see above) process of software design is preferred by Vendors and Buyers alike. Software Quality will have a big impact on how easy it is to extend and develop a system.

Software Security and Open Source

Many systems are connected to the internet nowadays. This means they are open to penetration by bots and hackers which are increasingly sophisticated. There may be hundreds of hacking attempts in an hour in a typical system.

Indeed this has recently become topical with the penetration and theft of data in large Health related IT systems. “Ransomware” attacks are now commonplace.

Open Source software by definition is public and open to scrutiny. Some argue that this makes the code vulnerable to hackers as they can easily analyse the source for potential vulnerabilities. The counter argument is that “many eyes” can look at the codebase and identify these vulnerabilities, allowing them to be corrected.

Proprietary software also seems to have it’s share of hacks. Perhaps the source code is available to the hacker, perhaps it has been “reverse engineered” from object code, perhaps the hacker has simply tried a suite of hacking tools.

In the case of a recent large attack “development” code was inadvertently deployed to a production server.

“Trojans” or backdoor access may have been built into the code by the original coder. There was a famous case of an undocumented “backdoor” built into a large commercial database widely used in enterprise systems. This hack was said to be exploited by US intelligence. Of course if the code is secret these Trojans will never be discovered. High quality code is less likely to have vulnerabilities and if they are present it is possible to correct them.

The User Interface

This is a focus for commercial mobile phone and computer app providers where usability is critical to uptake in a competitive environment. But in large Health IT systems, usability and the quality of the User Interface does not attract the same attention. Is this yet another adverse effect of the commercial contracting process and “Vendor Lockin”?

The User Interface has also been well studied in the context of Safety Critical systems. Poor User Interface design and poor software quality were factors in the Therac-25 incident. Here a malfunction in a Radiotherapy machine caused harm to several patients and some deaths. (ref 5)

In my view it is also a factor in many poor clinical outcomes in Health when critical previous history data is obscured in the record. The treating Clinician does not access this data and makes a clinical error as a result.

In my experience this factor is not considered in most “after the fact” reviews and “Root Cause Analyses”. Usually the Clinician is held responsible for the outcome and a common response is to require “further training”.

Some principles of User Interface design: (ref 6)

Simplicity Principle – This means that the design should make the user interface simple, communication clear, common tasks easy and in the user’s own language. The design should also provide simple shortcuts that are closely related to long procedures.

Visibility Principle – All the necessary materials and options required to perform a certain task must be visible to the user without creating a distraction by giving redundant or extraneous information. A great design should not confuse or overwhelm the user with unnecessary information.

Feedback Principle – The user must be fully informed of the actions, change of state, errors, condition or interpretations in a clear and concise manner without using unambiguous language.

Tolerance Principle – This simply means that the design must tolerant and flexible. The user interface should be able to reduce the cost of misuse and mistakes by providing options such as redoing and undoing to help prevent errors where possible.

Reuse Principle – The user interface should be able to reuse both internal and external components while maintaining consistency in a purposeful way to avoid the user from rethinking or remembering. In practice this means data used in several locations should only be entered once.

In all the various Health IT systems that I have used, the User interface does not appear to conform with many of these principles. Typically, the program is complex with multiple layers of dialogs.

There is a lot of redundancy of headings, poor use of screen “real estate”, poor formatting of dialogs, and displays of multiple “null” values

Often a lot of irrelevant “administrative” information is displayed. The end result is poor usability and poor “data visibility”. Important clinical data is hidden in layers of dialogs or poorly labelled documents.

These failures reduce efficiency and user satisfaction and increase risk in an already difficult and at times dangerous Clinical Environment.

Conclusion

Software Quality is important to cost, extensibility, Interoperability and Clinical Safety. It should receive more attention in the commissioning and upgrading of Health IT systems. The design of the User Interface is a Clinical Safety Issue and should be considered as a factor when adverse clinical outcomes occur.

References

1. Scalability and Semantic Sustainability in Electronic Health Record Systems

Erik Sundvall

Linköping University, Department of Biomedical Engineering, Medical Informatics. Linköping University, The Institute of Technology.

http://liu.diva-portal.org/smash/record.jsf?pid=diva2%3A599752&dswid=3087

2. Data Modelling – OpenEHR

https://www.openehr.org/

3. Obama legislation

https://www.modernhealthcare.com/article/20160430/MAGAZINE/304309990/how-medicare-s-new-payment-overhaul-tries-to-change-how-docs-use-tech

4. Legacy software

https://www.manning.com/books/re-engineering-legacy-software#toc

5. The Therac-25 Incident

https://thedailywtf.com/articles/the-therac-25-incident

6. User Interface design

http://ux.walkme.com/graphical-user-interface-examples/

(Why Large IT Projects Fail)

From the literature and the cases above it is apparent that large IT system deployments fail more often than not, and are almost never completely delivered. Is there a fundamental reason for this?

Large IT projects are certainly complex in the conventional sense. Bar-Yam (ref) argues that they are also “complex” in the mathematical sense. There may be several independent large systems interacting in unpredictable ways. This complexity property makes them difficult to analyse and therefore difficult if not impossible to predict. It is therefore practically impossible to specify the outcome of a complex system design at the outset. Thus the traditional “waterfall” (see below) specification process is likely or even bound to fail.

Conventional large system engineering has been based on historic designs such as the Manhattan Project, or the Apollo project to land a man on the moon. While these appear “complex” the outcome sought was relatively discrete and simple in an engineering sense. Health IT systems have a complex output in many records which are used in various ways for individual, management and population purposes. The Health IT environment is inherently complex with large amounts of data of different types.

From Bar Yam – “The complexity of a task can be quantified as the number of possible wrong ways to perform it for every right way. The more likely a wrong choice, the more complex the task. In order for a system to perform a task it must be able to perform the right action. As a rule, this also means that the number of possible actions that the system can perform (and select between) must be at least this number. This is the Law of requisite variety that relates the complexity of a task to the complexity of a system that can perform the task effectively. “

Clearly in the case of Health records there are many “wrong ways” and probably many “right ways” to record a health encounter!

The governance of large projects in government is itself complex – there are many stakeholders with various agendas interacting in various ways, not necessarily under the control of the design team.

Bar Yam argues that a fundamentally different approach should be taken to the design of complex systems. Rather than attempting to specify a single solution at the outset, the design process should focus on the process of evolution towards a solution.

Even agile software design processes (see below) are not “evolutionary” in this sense. They do not have many possible solutions competing with only the “fittest” surviving – they are a single solution evolving by iterative small changes towards a solution. The Open Source software development process is perhaps closer – here there may be several projects developing a similar solution to a particular problem.

There are design processes which are a combination of these techniques. Toyota are regarded as world leaders in design and their vehicles are market leaders. They use a traditional systems engineering approach, but have several teams working in parallel on the same problem. Each team is allowed to progress their work to an advanced stage before a single solution is chosen. This redundancy would appear to be wasteful but in fact produces superior designs.

Where change is being made to a large complex system, Bar Yam advocates small changes at various places in the system. These gradually replace the current components over time, with the old system being maintained for a period for redundancy and safety.

The internet can be regarded as a large system that was developed using this “complex systems engineering” approach, being created and revised incrementally over decades. The internet has brought change to the world equivalent to the industrial revolution. It is a Complex Systems Engineering success story.

Software design approaches – “Waterfall” vs “Agile”

The traditional approach to commissioning or designing a software system is termed the “Waterfall” process.

Software designers or vendors attempt to capture a detailed set of specifications from users which completely describe the functions of the new system. The customer assesses the new system against the specifications. When the two parties agree that the specifications have been met to an acceptable degree, the system is delivered. Further evolution of the system is a separate issue – often not considered at the initial design phase. So the design effort is essentially a one-off “waterfall” process using a traditional systems engineering approach. In practice, there are a number of problems with this approach in large Health IT systems.

Firstly the vast majority of new Health IT systems are not designed from scratch – they are usually created by modifying a system already held by the software vendor. There are claims by the vendor that their system can be “customized” to fit the customer’s requirements. In practice, the customer usually accepts significant compromises as to which specifications are not addressed. Contracts generally limit the writing of new software to address specifications, because this is expensive and uncertain. Indeed if the product being considered is not well designed and maintained, it may be impossible to modify it to a significant extent (see “Legacy Software” in a further article)

Secondly the process of capturing accurate specifications is difficult – misunderstandings may remain between customer and software engineer. To make the process more reliable, formal processes and even software languages have been written for this purpose.

Thirdly, the customer’s needs will always change and evolve. There is often “Specification Creep”, particularly if there are many “stakeholders” and if the process is not tightly managed. In large Government projects there may be hundreds of stakeholders with various agendas. Invariably the delivery of specifications is incomplete. If the process of updating and improving the software is not explicitly planned and funded changes after the startup date will be difficult. Bureaucrats generally want to deliver a specific outcome on a certain date – funding is always contested and limited. A system which is developed incrementally over time is less politically attractive than a showpiece startup announcement. They do not generally expect to fund an ongoing process of evolution and improvement.

Finally, the project seeks to replace a current system with a “big bang” approach. The old system is not necessarily retained for safety and redundancy

Thus, if we accept that that these systems are “complex” as discussed above, then this approach is almost certain to fail.

An alternative design process has emerged in recent years termed “agile” or “extreme” programming. Here the designer starts with a partial but quickly produced solution to the customer’s requirements. The customer assesses the solution and proposes changes and extensions. The designer produces these and the customer assesses the system again and proposes improvements. Through an iterative series of such cycles, the system evolves towards a solution. This approach would overcome several of the problems above – in particular some of the issues of complexity and difficulty in specification would be better addressed. However this approach is not generally adopted in large Government funded systems.

Next – Open Source and Software Design

Reference

When Systems Engineering Fails — Toward Complex Systems Engineering

SMC’03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme – System Security and Assurance (Cat. No.03CH37483), 2003

Yaneer Bar-Yam