Published on 02/12/2025

Model Use Restrictions: Where Not to Use AI

The integration of Artificial Intelligence (AI) and Machine Learning (ML) models into pharmaceutical environments has accelerated innovation and enhanced efficiencies. However, their application is not without concerns. This tutorial guide will elucidate the intricacies of AI/ML model validation, particularly focusing on the intended use, data readiness, bias and fairness testing, model verification and validation, and associated documentation. Understanding these aspects is essential in navigating the regulatory landscape defined by authorities such as the FDA, EMA, and MHRA.

Understanding AI/ML Model Validation

AI/ML model validation refers to the systematic evaluation of algorithms used to ensure they meet predefined requirements. This process is vital for any model intended to support decision-making in regulated environments. Regulatory standards provide clear guidelines on how AI/ML models should be validated, emphasizing the importance of intended use risk assessments. Here are the critical steps involved in the validation process.

Step 1: Define Intended Use and Scope

Before commencing validation, it is imperative to define the intended use of the AI/ML model clearly. The intended use encompasses the specific purpose for which the model is developed and the context in which it will be utilized. Follow these steps:

• Clearly articulate the use case: Document what decisions the model will inform, such as patient diagnosis or treatment recommendations.
• Identify regulatory requirements: Determine which regulations, such as 21 CFR Part 11 for electronic records and signatures, and Annex 11 relating to computerized systems apply to your model.
• Assess risk: Perform a risk assessment to evaluate the implications of incorrect outputs from the model and the potential impact on patient safety.

Step 2: Ensuring Data Readiness and Curation

Data is the cornerstone of any AI/ML model. Ensuring data readiness through appropriate curation processes is crucial to the model’s performance and predictive capabilities. Address the following:

• Data Integrity: Ensure that data sources meet GxP (Good Automated Manufacturing Practice) standards, which may include validation of input data.
• Cleaning Data: Conduct data cleaning to remove any inconsistent or erroneous entries which could skew the results, ultimately affecting bias and fairness testing.
• Documentation: Maintain meticulous documentation of data sources, cleaning methodologies, and transformations applied for audit trails.

Step 3: Bias and Fairness Testing

The potential for bias within AI/ML models poses a substantial concern, particularly in sensitive applications such as healthcare. Here’s how to conduct effective bias and fairness testing:

• Define fairness metrics: Establish metrics that suit your intended use. This might include statistical parity, equal opportunity, and predictive parity.
• Evaluate underrepresented groups: Ensure that the model performs equitably across diverse demographics.
• Tool Utilization: Employ tools and libraries designed for fairness testing to quantify and rectify bias in predictions.

Step 4: Model Verification and Validation

Verification and validation (V&V) of an AI/ML model are critical not only for compliance with regulatory standards but also for ensuring the model’s reliability. The steps to achieve robust V&V include:

• Establish a validation framework: Design a structured framework that outlines roles, responsibilities, and validation processes.
• Testing Methods: Utilize techniques such as cross-validation, holdout validation, and external validation datasets to comprehensively evaluate the model’s performance.
• Result Interpretation: Analyze the validation results, ensuring that findings align with the legitimate intended use established initially.

Explainability and XAI in Pharmaceutical AI/ML Models

Explainable AI (XAI) techniques are critical in the pharmaceutical domain, where stakeholders require transparency in AI/ML-driven decision-making processes. These methods not only enhance trust and acceptance but also aid in regulatory compliance. Consider the following:

Step 5: Implementing Explainability Techniques

• Model-Agnostic Interpretation Tools: Utilize techniques such as LIME (Local Interpretable Model-agnostic Explanations) and SHAP (Shapley Additive Explanations) to explain predictions irrespective of the underlying algorithm.
• Prototype Explanations: Develop prototype models that allow stakeholders to understand how inputs affect outputs, enabling them to evaluate model behavior better.
• Transparent Documentation: Maintain comprehensive documentation detailing the rationale behind model decisions, which is necessary for audits and compliant practices.

Step 6: Drift Monitoring and Revalidation

Models can become outdated as new data becomes available or as the underlying environment changes; thus, drift monitoring and periodic revalidation are vital for sustained performance:

• Establish Drift Monitoring Protocols: Set up systems to continuously monitor the model’s performance over time, including alert mechanisms for when performance degrades.
• Implement Revalidation Procedures: Define criteria for revalidation based on pre-established metrics and triggers of model drift.
• Update and Retrain Models: Regularly incorporate new data into the model and retrain to adapt to new trends or insights.

Documentation and Audit Trails

Maintaining comprehensive documentation is non-negotiable in ensuring compliance and facilitating audits. An effective documentation strategy should encompass:

Step 7: Create Comprehensive Documentation Practices

• Validation Plans: Document your model validation plans, outlining objectives, methodologies, and expected outcomes.
• Change Control Documentation: Create records for any changes introduced to the model post-validation, including revalidations prompted by drift or other interventions.
• Audit Trails: Ensure electronic systems provide secure and detailed audit trails as specified under regulations like 21 CFR Part 11 and Annex 11 to facilitate the review of all activities related to the model.

AI Governance and Security

In a landscape filled with data privacy concerns, establishing robust governance and security protocols is essential.

Step 8: Establish AI Governance Frameworks

• Assign Governance Roles: Designate responsibilities for governance, ensuring stakeholders from diverse departments contribute to the AI model oversight.
• Security Protocols: Implement security measures to protect data integrity and confidentiality throughout the model’s lifecycle.
• Regular Audits: Conduct audits regularly to ensure adherence to AI governance policies and regulatory requirements.

Conclusion

AI/ML models present substantial opportunities and challenges in the pharmaceutical sector. By understanding the tenets of AI/ML model validation, including intended use risk, data readiness curation, bias and fairness testing, model verification and validation, explainability (XAI), drift monitoring, and proper documentation protocols, pharmaceutical professionals can leverage these technologies while maintaining regulatory compliance. Engaging with these steps within the regulatory frameworks of the WHO, FDA, EMA, and MHRA will support the creation of safe, effective AI/ML applications that transform healthcare.