DevOps in medical software
Together with Henrik Lynge and Volkan Kocak
Over the past 3 years I have been working the creation of different IT solutions that are used in the Pharmaceutical drug process that must follow FDA (USA) and EMA (Europe) regulations regarding Good Clinical Practices (GCP).
A critical challenge that the pharmaceutical industry is facing today is that quality assurance (QA) units have a set of tools, processes and practices that are incompatible and conflicting with modern IT development and DevOps. This results in a major clash between QA and software development teams that has a significant impact on both the speed of development and potentially also loss to both opportunity and quality.
In this blog, we will make a root cause analysis of the problem and propose a solution at the end.
Today, quality assurance (QA) for IT solutions used in the pharmaceutical industry is a meticulous process comprised of a significant number of hand-held activities that aim at establishing intended use, risk mitigation, fitness for intended use, and the preservation of a validated state.
Modern software is developed and released under CI/CD, which is a largely automated process of verifying, installing and deploying the code as functional software. Through Agile, it mitigates risks derived from project planning, management. Through behavior driven development, it mitigates risks derived from misalignment between user requirements and functional specification.
Almost paradoxically, although these processes seem incompatible in form, their objectives are the same: improve quality and reduce risk using a risk-based approach.
The primary consequence of having these two cultures is that organizations usually have 2 teams working in QA — “software QA”, comprising of roles such as DevOps engineers and software developers, and “process QA”, comprising of roles such as validation leads and QA leads. These teams usually produce two parallel sets of deliverables:
- Software and documentation for developers
- Documentation for QA, audits and inspections
The former comprises of test automation at the unit, integration and system level, pull request templates, .yaml files describing the CI/CD, README, wiki pages, etc.
The latter comprises a set of “approved“ documents including SOPs, risk assessment, trace matrices, that the QA unit deems necessary based on its interpretation of the requirements set out by regulatory agencies. These are usually word documents, excel sheets and process diagrams. These documents contain an interpretation, by QA, of the software and its development process.
As the software evolves, the “documentation for inspections” has a high risk of drifting from the software and the process itself. Consequently, when presenting the process in an inspection, the “process QA” risks presenting not the process, but a shadow of the process. Overall, there is a risk of making audits and inspections assess a shadow process, resulting, at best, in a missed opportunity by all parties (development team, QA team, organization, regulatory agency and the industry) to improve overall patient safety, product quality and data integrity.
This risk of drift results in a significant impact to software quality. To understand this, we need to describe how this risk is mitigated: QA and developers spend a lot of time and resources keeping the documentation for inspections aligned with the CI/CD process and developed software at all times. Software developers consider this a waste of time as these documents are not used by them, and therefore not adding value to them. This results in developer dis-satisfaction and ultimately that the industry is losing experienced developers to other less regulated industries as they get overwhelmed by these documentation activities. Ultimately, this results in a reduced software quality and thus potentially decreases patient safety, product quality and data integrity.
This in turn puts a burden to innovation and ultimately impacts the cost of drugs — the barrier to enter the Pharmaceutical Industry with service offerings is high as both startups and large enterprise alike are challenged to attract exceptional talent to produce innovative solutions.
In my opinion, the solution to this problem is to take the best of the two worlds, and for this we need to understand how each world adds value. This allow us to find the potential sweet spot where the value of one world adds to the value in the other world in a complementary way where hopefully 1 + 1 = 3.
The core value of modern DevOps is that it increases quality through automation — it makes humans’ limited mental space focused on the non-trivial aspects by automatizing the trivial ones in systematic and reproducible ways. I think that we can agree that how we operate the current Quality process is largely a manual, point and click process, and that our bread and butter is an effort of maintaining a consistent state of the documentation. This results in a reduced mental space to apply critical thinking to the process and its resulting quality.
The core value of QA is that it increases quality by applying controls to the process. I think that we software developers can agree that how we behave when producing and operating software has risks with critical impact to quality. One example is the god complex behaviour in assessing our own capabilities.
Under this perspective, we must accept that each side should compromise by adopting aspects that the other side excels at. Each side will need to accept that part of what they are doing today will be different, but that such changes lead both to a more productive collaboration and ultimately better patient outcomes.
So, what is the proposal?
1. Use software development practices for documentation and configuration management
This includes
- use git version control system to produce and manage documentation
- use a formal review and approval processes of documentation (no more emails to gather acknowledgements, use pull requests)
- use automated documentation checks (e.g. ids uniqueness is checked automatically)
- focus the writing on the content, not on the form (e.g. use markdown)
This is “documentation as code”, in line with other “.. as code” initiatives.
Software developer, you may wonder what is the benefit of having the documentation in the git repository. The two core arguments are:
- in the pharmaceutical industry, software without up-to-date, high quality documentation is not releasable. Consequently, documentation, like tests that follow the implementation, should be in the repository.
- documentation is largely a data transformation or aggregation where the raw data is the source code, and the aggregated result is natural language. Because such transformation cannot be done automatically (yet), we humans need to maintain it ourselves.
Note that this is no different process than the documentation of an open source software library when APIs and guides are well documented and maintained.
QA lead, you may wonder how do you benefit from this. I do not have the time to present to you the full argument, but I do not need to — your development team will passionately explain you the benefits (pssssst, developer, show the git commit history, git blame, and describe how we control over what we deploy).
2. Use QA’s risk controls to ensure quality of our process and deliverables
This includes
- specify requirements in a way that users can read and accept. This will define “intended use” and “acceptance criteria” of your software and make it much easier for you to declare “fitness for use” when you have finalized all your validation activities. Remember that requirements can also be risk-based, some are more critical than others.
- review and demo your software design. Will the design meet the user requirements/intended use? Have requirements in specifications been fully considered in the intended design? Does the design take any needed interfaces into consideration?
- specify how the software is tested (yes, including that critical undocumented manual step that the releaser does in production). Leverage your testing activities made as part of your test driven development. Document why your development environment is similar to the production environment in terms of configurations, resources and data. Describe how you ensure that all your requirements are covered.
- document the risks that we accepted and why (e.g. what are the accepted risks of not rotating secrets?)
- describe all items that the application is composed of (e.g. include the secret that someone created 8 months ago and that will expire in 4)
- describe how the software is released and how no unauthorized and unintended changes can happen in production (right??!)
- describe how to conduct post-mortem analysis of incidents in production
QA lead, you may wonder what is the benefit of asking developers to write these themselves (and risk not being good for an inspection). The three core arguments here are:
- risk controls that are described in a document but not applied to a process do not improve quality, and the process happens on over what the developers behave
- a good audit will interview developers and will ask them to show then the process. The developer will refer to another document, not the one QA wrote, as that is the de-facto reference.
- “Quality” includes the developer’s willingness to apply risk controls. Keeping experienced software developers and having then unconsciously apply risk controls is far more important than a pristine PQP.
Software developer, you may wonder how do you benefit from this. I do not have the time to present to you the full argument, but I do not need to — your QA lead will passionately explain you the benefits (pssssst, QA lead, explain them from first principles — avoid presenting how you do it, but rather what are the risks that you are trying to control for — developers will understand, as they are the same risks that developers mitigate in DevOps).
To facilitate this process, I created an open-source project where you can find
- a CI/CD tool to help achieving step 1.
- a generic PQP compatible with modern DevOps (that should pass an inspection) to achieve step 2
The aim is to have these fulfill:
- Regulatory agencies’ requirements regarding software supporting GxP regulated processes, including good documentation practices
- QA quality controls such as requirements gathering and approval, design establishment and review, proper testing of your software, support and operations as configuration management and control, access management and control, audit trail etc.
- Modern DevOps and software development practices such as CI, git, markdown, and CD
- Risk-based approach on all quality activities and topics.
I have divided these in two repositories:
- an “out-of-the-box” documentation specification and CI tool to check it, https://github.com/medical-software-quality/documentation-as-code, and
- a documentation-only repository with a PQP for modern DevOps, https://github.com/medical-software-quality/project-quality-plan
I welcome everyone to contribute to it. Have fun and I look forward to seeing you there!
