Published on 01/12/2025

Missing Data & Imputation: Guardrails for Regulated Use

In the evolving landscape of pharmaceutical analytics and artificial intelligence (AI), understanding the challenges associated with missing data and imputation is critical. As AI and machine learning (ML) gain traction in Good Automated Manufacturing Practice (GxP) environments, adherence to regulatory standards such as FDA, EMA, and MHRA becomes paramount. This article serves as a step-by-step guide for industry professionals to navigate the complexities of AI/ML model validation, focusing on intended use, data readiness, bias and fairness testing, model verification and validation (V&V), and compliance with standards such as 21 CFR Part 11, Annex 11, and GAMP 5.

1. Understanding Missing Data and Its Implications

Missing data is an inherent challenge in any analytical process, especially in regulated environments. In the context of pharmaceutical data analytics, missing data can lead to biased results and compromise decision-making. The significance of addressing missing data stems from the potential risks it poses to patient safety and the integrity of clinical trials.

The implications of missing data are multifaceted:

• Impact on Model Performance: Insufficient data can lead to overfitting or underfitting, hampering the predictive accuracy of AI/ML models.
• Regulatory Compliance: Incomplete datasets may result in non-compliance with regulatory guidelines, leading to delays in drug approval or market entry.
• Ethical Considerations: The lack of comprehensive datasets can lead to biased outcomes, raising ethical concerns about data fairness and equality.

Hence, addressing the issue of missing data is critical before progressing to validation phases. The next steps involve assessing data readiness and implementing appropriate imputation techniques.

2. Assessing Data Readiness for AI/ML Models

Data readiness is a vital aspect of successful AI/ML implementation in GxP analytics. Regulatory bodies require that data utilized for analytical purposes is robust, complete, and representative. The assessment of data readiness consists of the following steps:

2.1 Data Collection

The first step involves gathering data from various sources, including clinical trials, historical datasets, and real-world evidence. Sources should be well-documented to ensure traceability and compliance with regulations. Confirm that data types conform to expected formats and standards, as outlined in guidance from ICH and others.

2.2 Data Cleaning

Data cleaning involves identifying and rectifying inaccuracies, inconsistencies, and incompleteness in the dataset. Tools such as statistical methods can help identify outliers or anomalies that may skew results. Essential considerations during this phase include:

• Evaluating the impact of outliers on model training.
• Documentation of all cleaning activities as part of maintaining audit trails.

2.3 Data Imputation Techniques

Once data readiness is established, the next phase involves choosing appropriate imputation techniques. Various methods can be employed, depending on the nature and extent of the missing data:

• Mean/Median Imputation: Substituting missing values with mean or median values is simple but may not be suitable for all datasets.
• Regression Imputation: Using regression models to predict and fill in missing data based on other variables.
• Multiple Imputation: A more complex method that accounts for uncertainty by creating multiple complete datasets and then aggregating results.

Best Practice: The selected imputation technique must be justified and documented thoroughly to comply with regulatory expectations.

3. Addressing Bias and Fairness in Models

As AI/ML models are increasingly utilized for decision-making in clinical and operational roles, ensuring fairness in model predictions is crucial. This section delineates best practices for conducting bias and fairness testing in the model validation process.

3.1 Identifying Sources of Bias

Bias can arise from various sources, including:

• Data Collection Bias: Systematic errors occurring during data collection can skew results.
• Label Bias: Human bias in labeling training data can lead models to make biased predictions.
• Algorithmic Bias: The algorithms themselves may encode biases present in the training data.

3.2 Conducting Fairness Testing

Fairness testing assesses how different demographic groups are treated by the AI/ML model. Techniques for fairness assessment include:

• Disparate Impact Analysis: Evaluating whether a model disproportionately impacts one demographic over others.
• Equal Opportunity Analysis: Ensuring equal true positive rates across groups.

Incorporating these practices during the model validation phase enhances compliance with ethical standards and regulatory expectations.

4. Verification and Validation of AI/ML Models

Verification and validation (V&V) are critical steps in the lifecycle of AI/ML systems, particularly in regulated settings. V&V ensures that models perform as intended and meet user specifications accurately.

4.1 Verification Process

The verification process assesses whether the model complies with technical specifications and requirements. Key components include:

• Static Testing: Evaluating the model code for technical errors before deployment.
• Dynamic Testing: Assessing the model’s performance under various scenarios using test datasets.

4.2 Validation Process

Validation ensures that the AI/ML model meets the specified requirements and is fit for its intended purpose. Validation should include:

• Usability Testing: Ensuring the system meets user needs.
• Performance Evaluation: Running the model against predefined validation datasets to ensure accuracy and reliability.

Documentation of all V&V activities is essential for compliance and future auditing as per 21 CFR Part 11 requirements.

5. Model Drift Monitoring and Re-Validation

AI/ML models may experience changes in performance over time due to shifts in data patterns—referred to as “model drift.” Continuous monitoring and re-validation are critical to maintaining compliance and ensuring model efficacy.

5.1 Setting Baseline Performance Metrics

Establishing baseline performance metrics during the validation phase is essential for effective drift monitoring. Common metrics include:

• Accuracy
• Precision and Recall
• F1 Score

5.2 Implementing Drift Detection Techniques

Several techniques can be employed to monitor model drift:

• Statistical Process Control: Utilizing control charts to assess model metrics over time.
• Performance Monitoring Dashboards: Visualizing model performance against baseline metrics.

5.3 Re-Validation Protocols

In instances of detected drift, re-validation is necessary. This process should include:

• Comparative analysis against original validation results.
• Documentation of all analyses and adjustments made during re-validation.

This ensures that the model remains compliant with appropriate regulatory standards, including guidance from GAMP 5.

6. Documentation, Audit Trails and Governance

Robust documentation and audit trails are essential components of compliance and governance in AI/ML model development. Regulatory bodies mandate that all processes, analyses, and validations be thoroughly documented.

6.1 Documentation Requirements

All activities related to data handling, model training, testing, and V&V must be documented. Key documentation points include:

• Data sources and collection methods.
• Selection and justification of imputation methods.
• V&V outcomes and any identified discrepancies.

6.2 Maintaining Audit Trails

Audit trails must track all changes and developments in the model lifecycle, including:

• User modifications made to datasets or models.
• Performance metrics and outcomes from monitoring and testing.

6.3 AI Governance and Security

As AI technologies evolve, ensuring AI governance and security is paramount. Organizations must implement policies that govern the use of AI/ML systems, ensuring ethical implementation and compliance with all regulatory requirements.

Developing a governance framework helps facilitate accountability, transparency, and trust in AI/ML deployments across the pharmaceutical sector.

Conclusion

This tutorial has highlighted crucial aspects of AI/ML model validation in GxP analytics, specifically addressing the challenges posed by missing data and the importance of rigorous validation processes. A focus on data readiness, bias and fairness testing, robust verification and validation practices, drift monitoring, and impeccable documentation will help ensure that AI/ML implementations comply with regulatory expectations and uphold the highest standards of patient safety and ethical responsibility.

As the pharmaceutical industry continues to embrace AI/ML technologies, ongoing education, adherence to guidelines, and commitment to quality assurance will serve as foundational pillars of success in the regulated landscape.