CSA—To Formally Test or Informally Test, That Is the Question

The FDA’s Case for Quality initiative seeks to shift the focus from validation as a check-the-box regulatory activity to one that centers around quality. The shift will ensure that patients and end users have access to safe, effective care. Recognizing that innovation drives continuous improvement, which, in turn, leads to higher quality and efficiencies in products and healthcare; the FDA issued the Computer Software Assurance (CSA) draft guidance in September 2022 to make validation of regulated systems less burdensome and more strategic.

It is important to note that as of this writing, the guidance is still in draft form and is not final. That said, companies can start preparing for CSA now. The proposed shift advocates for applying critical thinking and a risk-based approach to determine the testing rigor for requirements. The risk-based approach will lead to resourcing efficiencies in any validation project. By design, CSols has used a risk-based approach for validation for more than 20 years. A main difference between CSA and computer system validation (CSV) is the emphasis on informal testing versus formal testing. This blog will look at what we know about CSA—the good news is that our clients shouldn’t notice many changes in our validation approach.

Computer Software Assurance (CSA) in a Nutshell

CSA puts the focus of importance in software validation on demonstrating confidence in a regulated system’s ability to perform its intended use. In order for companies to implement innovative technologies in an ever-changing technology landscape, while still remaining compliant with laws and regulations, CSA encourages a quality-centric perspective based on the documented identified use and risk-based analyses of the documented requirements (user and/or functional requirements) instead of applying a blanket test-it-all approach to validation of regulated systems. As discussed in our recent CSA vs CSV webinar, the FDA encourages companies to define their validation process so that it results in a right-sized approach based on intended use and risk.

Right sizing includes leveraging applicable past validation work by the company as well as qualified vendor testing documentation. CSA doesn’t stop once the laboratory, production, or quality system is validated; the process should include the necessary steps to maintain the system in a validated state throughout its lifecycle, which will require periodic review and potentially, revalidation.

Once the validation process for the system or software is documented you must be confident in its ability to support the technology, whether in its current form or when updates are released. Also, you should be prepared to tell your story to an auditor in a way that justifies your process and conveys your confidence that the intended use of said system or software within your organization has been validated for its intended use to your satisfaction.

Supplier Qualification—It’s Important!

To leverage testing documentation from a supplier, you must first demonstrate your due diligence in ensuring that the software has quality built into it. The supplier qualification evaluates all aspects of the supplier’s processes in software development and includes a questionnaire and/or in-person audit, both of which are conducted every two years at minimum and whenever an event impacts the supplier’s qualification. Examples of suggested qualification topics include the following, as applicable:

• Product Development
  • Is there a software development life cycle (SDLC) process in place and followed?
  • Describe what testing (functional, regression, integration, etc.) is conducted
  • Defect reporting and management
    • In the development phase
    • For a released version
• Performance Testing
  • Describe what aspects of the system are tested and how
  • Are the results of performance testing available for the customer’s review?
• Configuration Management
  • What is the change control process?
  • How are customers notified of required patches or updates?
  • What is the warranty information?
• Security
  • How is physical access restricted?
  • Are there safeguards in place to restrict software/IT access, as needed?
• Technical Documentation
  • Are the following available to the customer?
    • Specifications—functional, design, etc.
    • The user manual
    • Diagnostic or troubleshooting information
• Training for the customer—is it provided?
• Customer reviews, including responses by the supplier to reported defects and feature requests

A Risk-based Approach for Validation Testing

Once the intended use is identified and the requirements are documented, it’s time to apply the quality-centric risk-based evaluation of the software through the lens of efficacy. It’s important to ask if the software’s failure to perform as intended will have a negative impact on patient or product safety. Another factor linked to the risk assessment is the implementation method. Software features can generally be classified into three implementation categories:

• Custom: the feature requires changes to the source code
• Configured: the feature is enabled through a software setting; no change to the source code
• Out of the Box (OOTB): the feature works once installed and master data is added, as applicable

The risk-based approach can be summarized as shown in the risk matrix (Figure 1), where the numbers (and colors) represent the risk rating to the patient or end user.

The risk rating scale, as it applies to the suggested testing effort required, can be defined as in Table 1 for a given company’s validation procedure:

These categories of testing, as discussed in our Test Smarter, Not Harder white paper, allow more freedom for a company to right size the testing effort per each requirement, based on the assigned risk category. Of course, your organization can define what type of testing and other value-added assurance activities are assigned to each risk rating.

Unscripted CSA Testing—How Is It Documented?

In the CSA draft guidance, the FDA encourages the use of unscripted testing whereby the tester is not constrained by prescriptive test case steps and obtaining large amounts of objective evidence. That said, when using unscripted testing methods, there needs to still be enough evidence to show that the function, feature, or operation was evaluated. In the CSA draft guidance, the FDA describes the minimum testing record components as:

• The intended use of the function, feature, or operation
• The determined risk level
• A brief description of what testing was performed, which may contain high-level test objectives
• A description of any issues found and their resolution
• Concluding statement indicating whether the testing results were acceptable or not
• Date and who performed the testing

Regarding objective evidence for testing efforts, the FDA recommends leveraging the electronic records such as system logs, audit trails, and other software-generated data to document the assurance activities.

Don’t Be Afraid to Adopt CSA

CSA with its risk-based approach to the testing effort required is not a new concept. It reinforces software quality assurance best practices and encourages companies to document their validation processes with what is relevant to their needs. The draft CSA guidance emphasizes that companies must think critically about the intended use of the software and come to a robust understanding of the foreseeable risks posed to patient or end user safety. CSA helps organizations establish the level of assurance activities needed to demonstrate that the software functions as intended.

The draft CSA guidance from September 2022 can be applied when validating any computer software or automated system that supports your company’s quality efforts. However, as of the posting of this blog, this remains draft guidance. We have not seen the final industry-commented and approved/issued guidance. We will be looking forward to that in the future as there may be some changes in the final version. This blog will be revised if necessary when the final text is released.

---

What questions remain in your mind about Computer Software Assurance? Let us know in the comments.