Published on 02/12/2025

Change Control for Models: Verification vs Re-Validation

The integration of Artificial Intelligence (AI) and Machine Learning (ML) models in Good Practice (GxP) regulated environments has introduced complexities in pharmacovigilance, clinical operations, and regulatory compliance. As pharma professionals navigate these complexities, it’s essential to establish a robust framework for model validation, focusing on the distinction between verification and re-validation, particularly in the parameters of intended use, data readiness, and compliance with regulatory entities such as the FDA and the EMA.

Understanding Model Validation in GxP Frameworks

Model validation ensures that AI/ML models perform reliably and as intended within GxP frameworks. This process involves both verification and re-validation, critical aspects of maintaining compliance and ensuring data integrity. Establishing an understanding of these concepts starts with the following definitions:

• Verification: This process confirms that the models meet the specified requirements and perform their intended functions. It is usually executed during the model development phase and involves checking for correctness, performance consistency, and suitability for the intended application.
• Re-Validation: In contrast, re-validation occurs when there are changes to the operational parameters of the models or when new data is introduced. This ensures ongoing compliance and establishes that the models maintain their intended use and data readiness throughout their lifecycle.

Both verification and re-validation are intrinsically linked to data quality, intended use, bias and fairness testing, and compliance with GxP standards such as 21 CFR Part 11 and GAMP 5. Understanding how these factors interplay helps laboratories and regulatory professionals in developing a compliant AI/ML framework in their analytics operations.

Framework for Verification of AI/ML Models

Verification serves as the foundation for establishing trust in AI/ML models by assessing their design and function. The following steps outline a straightforward framework for verification:

1. Define Model Requirements

The first step involves defining comprehensive requirements for the AI/ML model. This should encompass the following:

• Intended use and expected outcomes.
• Applicable performance metrics and benchmarks.
• Compliance needs aligned with regulatory guidelines.

2. Data Readiness Curation

It is critical to evaluate the data inputs intended for model training and testing. This entails:

• Assessing data quantity and quality.
• Ensuring that data reflects the intended population and context.
• Evaluating data sources for integrity and relevance.

3. Execute Thorough Testing

Testing plays a vital role in model verification. Conduct the following:

• Performance testing by comparing model outputs against known outcomes.
• Robustness testing to evaluate performance under varied conditions.
• Stress testing to simulate atypical scenarios.

4. Document Findings and Develop Audit Trails

Documentation is critical for accountability and traceability. Maintain clear records of:

• Testing protocols and results.
• Any identified discrepancies and resolutions.
• Decision-making processes.

5. Review and Approve Verification Results

All verification outcomes should be reviewed by a qualified individual or team. Upon approval, the model can be cleared for operational use, ensuring that future audits can reference the verification documentation.

Re-Validation Process: When and Why?

Models should be considered for re-validation under specific circumstances. Potential triggers include changes in:

• Input data characteristics.
• Operational environment (e.g., changes in software or hardware).
• Regulatory guidelines.
• Intended use or scope of the model’s application.

Establishing a structured re-validation process is essential for GxP compliance. Follow these steps for effective re-validation:

1. Trigger Identification

Clearly outline the conditions under which re-validation is necessary and ensure that personnel are trained in recognizing these triggers.

2. Impact Analysis

Conduct an impact analysis to evaluate how changes may affect the model’s performance and its alignment with initial specifications.

3. Re-Validation Testing

Perform testing similar to the original verification. Focus on:

• Re-evaluating model outputs against updated data sets.
• Assessing the models for any bias introduced by the changes.
• Performing additional fairness testing as needed.

4. Documentation and Audit Trail

Maintain thorough documentation of the re-validation process, including:

• Change requests and approvals.
• Re-validation results, discrepancies, and corrective actions.
• Executive summaries for stakeholders, ensuring transparency in decision-making.

5. Final Review and Approval

As with verification, the re-validation findings should undergo thorough review and approval, establishing a formalized reference for future audits.

Drift Monitoring & Re-Validation: A Critical Component

Drift is a significant concern for AI/ML models as their performance can degrade due to changes in underlying data distributions over time. Implementing drift monitoring as part of the re-validation strategy ensures ongoing model performance and accuracy in its predictions. The following components should be included:

1. Continuous Model Evaluation

Establish protocols for regularly assessing model performance metrics, such as accuracy, precision, and recall, against predefined acceptable ranges. This includes monitoring the input data for signs of drift that can indicate potential shifts in the underlying population or patterns.

2. Anomaly Detection Systems

Utilize anomaly detection systems to identify deviations from expected model outputs and operational behaviors. This is vital in providing early warnings that a model may require re-validation.

3. Automated Reporting Mechanisms

Build automated reporting systems that alert stakeholders whenever performance falls below acceptable thresholds, supporting timely decision-making regarding potential re-validation or re-training of the models.

Governance and Security: The Foundation of Model Trust

A robust governance structure is essential for ensuring that AI/ML models meet ethical standards while complying with regulatory requirements. This includes:

1. AI Governance Framework

Develop an AI governance framework that outlines responsibilities, performance standards, and ethical considerations. The framework should enforce compliance with international standards and best practices governed by entities like the WHO and PIC/S.

2. Security Protocols

Implement comprehensive security measures that protect data integrity and model availability. This includes:

• Data encryption standards.
• Access control measures to enforce role-based access.
• Regular security audits and penetration testing.

3. Training and Awareness Programs

Ensure that all personnel involved in GxP processes are trained in the governance framework along with compliance needs related to 21 CFR Part 11 and Annex 11. This forms the cornerstone of an organization’s broader regulatory compliance strategy.

Documentation & Audit Trails in Validation Practices

Effective documentation and audit trails are not only essential for regulatory compliance but also serve to provide transparency and traceability throughout the model lifecycle. Implement the following practices:

1. Standard Operating Procedures (SOPs)

Develop comprehensive SOPs for both verification and re-validation processes. These should include:

• Clear roles and responsibilities.
• Cascade of documentation requirements.
• Templates for recording outcomes and decisions.

2. Version Control Mechanisms

Utilize version control systems to track changes to model specifications, testing protocols, and all relevant documentation. This ensures that historical records are preserved for audits and inspections.

3. Regular Reviews and Audits

Conduct internal reviews and audits of model validation processes and documentation to identify areas for improvement and ensure adherence to compliance standards.

Conclusion: Building a Resilient AI/ML Model Validation Strategy

In conclusion, establishing a robust change control framework for AI/ML models in pharmaceutical laboratories is essential for ensuring compliance with regulatory standards such as those set forth by the FDA, EMA, and GAMP 5. By systematically following the outlined verification and re-validation processes, incorporating drift monitoring, prioritizing governance and security, and maintaining rigorous documentation and audit trails, organizations can confidently deploy AI/ML technologies in their GxP analytics. Staying ahead in understanding regulatory expectations and best practices will ultimately lead to greater reliability and trustworthiness in AI applications.