Validating AI in a Regulated Environment: Validation Begins Before Model Selection
Artificial intelligence has made its way into the life sciences industry. Companies are exploring applications for quality management, production, visual inspection, documentation, master data, and many other processes.
However, as this technical capability becomes available, a key question arises:
“How can an AI-based system be validated in a regulated environment?”
The answer is not: with a completely new validation method. The established fundamental principles remain relevant. However, they must be supplemented with AI-specific aspects—in particular, the model and data lifecycle, data governance, controls (e.g., guardrails, human-in-the-loop), and performance requirements for continuous monitoring.
What's Different About AI-Powered Systems
- What data requirements apply to the model?
- Is the data used suitable and representative for the purpose?
- How are training, evaluation, and test data separated from one another?
- What performance criteria apply to the model?
- How is the model’s release candidate selected?
- How are model performance and changes monitored during operation?
- How is a model change evaluated?
Estimating Validation Effort: Key Activities in the Concept Phase
The starting point for assessing the validation effort remains the intended use (defined purpose), the associated risk, and the system’s complexity.
For AI-based systems in a regulated environment, another risk perspective comes into play in addition to GxP risks (patient safety, product quality, and data integrity). The EU AI Regulation classifies AI systems across industries on a risk-based basis. Both perspectives must be considered and evaluated together—they are not mutually exclusive. [1]
One example is an AI-based visual inspection of pharmaceutical packaging. From a general AI risk perspective, the application may be classified differently than from a GxP perspective. Therefore, the specific process context is crucial for validation.
When describing the intended use, particular attention must be paid to the interaction between humans and the system, the distribution of tasks, and decision-making responsibilities.
When assessing the complexity of AI-based systems, it is not only the intended model type and the degree of autonomy that are relevant. The context of use—in particular, the type, source, and quality of the data used—as well as the operating conditions during actual operation must also be considered.
The interplay of these elements makes AI-based applications complex systems. Therefore, the conceptual design phase should be expanded to systematically analyze the overall structure. This lays the foundation for a robust understanding of the system and a precisely defined intended use. Both are prerequisites for selecting an appropriate and safe model. The results of the conceptual design phase are incorporated into the risk assessment, the requirements for the model and data, as well as the performance metrics and acceptance criteria.
The proof of concept is part of the risk assessment
In AI projects, the proof of concept plays a particularly important role during the concept phase. Even at this early stage, the details gathered during the definition of the intended use should be reviewed:
- What problem is the application designed to solve?
- Which models are generally suitable?
- What data is available?
- What performance characteristics and limitations can be expected?
- What risks arise from misclassifications, non-representative data, or model changes?
The proof of concept should therefore not be viewed as an experimental space separate from the validation project. Rather, it is a key component for assessing requirements, risks, and the fundamental suitability of a solution at an early stage.
Advanced CSV in Operation: Detecting Drift, Ensuring Suitability
A model that has been successfully tested once does not automatically remain unchanged or maintain the same level of performance over time. During operation, data, processes, user behavior, or environmental conditions may change. This can lead to data drift or model drift, among other things.
The validation strategy must therefore ensure suitability through expanded AI-specific activities during ongoing operations. In addition to existing processes such as change management and periodic review, the following are required depending on the application:
- defined monitoring KPIs,
- Technical controls (guardrails)
- Thresholds and escalation procedures,
- Rules for Model Changes and Reassessment,
- Guidelines for handling unusual results,
- Clear lines of responsibility between the business unit, IT, QA, CSV, and, if applicable, Data Science.
The classic V-model is being supplemented—not replaced
A risk-based approach to AI-supported systems is based on established CSV/CSA principles. In particular, it is supplemented by:
- Definition of the intended use and its integration into the business process,
- Requirements for the model, data quality, and data governance, and verification in the proof of concept,
- Iterative model development and selection of a release candidate,
- model-based verification,
- Integration into the end-to-end tests of the overall system,
- Continuous monitoring during operation.
This means there is no one-size-fits-all standard method for every AI application or AI-supported system. The specific approach must continue to be defined on a risk-based basis, taking into account the intended use, the model, the data, the degree of autonomy, and the system’s complexity.
What Companies Should Do Now
Companies that want to use AI in a regulated process should start with these five steps:
- Clearly describe the intended use
What is the system supposed to do, and in which process will it be used? - Assessing Risk from Multiple Perspectives
GxP criticality, the perspective of the EU AI Regulation, system complexity, and data risks must be considered together. - Define Model and Data Requirements
Requirements regarding performance, representativeness, data quality, and traceability should be established early on. - Conduct a controlled proof of concept
Early experiments should be documented and incorporated into the subsequent decision-making and validation process. - Plan Operations and Continuous Monitoring
Monitoring (trends, guardrails), periodic reviews, change management, and handling drift must be considered before go-live.
Conclusion
AI validation does not mean completely replacing established CSV/CSA principles. The goal is to adapt them for learning, probabilistic, and evolving systems.
The most important starting point here is not whether a system uses generative AI, machine learning, or another model. What matters is:
“What role does AI play in this specific process, what risks does this entail, and how is control ensured throughout the entire lifecycle?”
In this comprehensive practical guide, DHC demonstrates how intended use, GxP criticality, model and data lifecycles, and monitoring can be integrated into a coherent validation approach.
How do you validate AI-powered systems using a risk-based approach?
In this comprehensive practical guide, DHC demonstrates how intended use, GxP criticality, model and data lifecycles, and monitoring can be combined into a transparent validation approach.
Sources and Editorial Note
[1] European Commission: AI Act – Regulatory Framework for Artificial Intelligence
[2] ISPE: ISPE GAMP® Guide: Artificial Intelligence
[3] European Commission: Stakeholder Consultation on EudraLex Volume 4, Chapter 4, Annex 11, and Proposed Annex 22
Frequently Asked Questions About the Validation of AI Systems
Why isn't traditional CSV validation sufficient for AI systems?
Traditional systems are defined by software logic, configuration, and established rules, and are therefore predictable and testable. AI systems learn their behavior from data. Their results may change when presented with novel inputs and cannot always be fully verified in advance. Validation is therefore expanded to include data, models, and performance. The proven CSV and CSA principles remain valid in this context.
What is “intended use,” and why is it so important in AI?
The intended use describes the defined purpose of a system and the process in which it is embedded. It forms the basis for risk assessment, validation efforts, and requirements for the model and data. In the context of AI, this also includes human-system interaction, the distribution of tasks, and decision-making responsibilities.
What risk considerations must be taken into account when using AI in a regulated environment?
On the one hand, there are the GxP risks related to patient safety, product quality, and data integrity. On the other hand, there is the risk-based classification under the EU AI Regulation. Both considerations must be evaluated together, as they are not mutually exclusive. An AI application may be classified differently under the AI Regulation than from a GxP perspective.
Why is the proof of concept part of the validation strategy?
During the concept phase, the proof of concept assesses at an early stage which problem needs to be solved, which models are suitable, what data is available, and what performance can be expected. It also highlights risks such as misclassifications, non-representative data, or model changes. Experiments should be documented and incorporated into the subsequent decision-making and validation processes.
What are data drift and model drift, and how can they be controlled?
Data drift and model drift occur when data, processes, user behavior, or environmental conditions change. As a result, a model’s suitability in production may decline, even though it was successful during validation. Control measures include defined monitoring KPIs, technical guardrails, thresholds with escalation procedures, rules for model changes and reassessments, and clear lines of responsibility. These measures complement change management and periodic reviews.
Will AI validation replace the traditional V-model?
No. The V-model is supplemented, not replaced. Additional elements include, among others, the definition of intended use, requirements for model and data quality, iterative model development with the selection of a release candidate, model-based verification, integration into end-to-end tests, and ongoing monitoring during operation.
What steps should companies take to get started?
Companies should follow five steps: clearly describe the intended use, assess risks from multiple perspectives, define model and data requirements early on, conduct a controlled proof of concept, and plan for operations and ongoing monitoring before going live.