Published on 01/12/2025

Synthetic Data in Regulated Analytics: When and How

Introduction to Synthetic Data in Regulated Analytics

The use of artificial intelligence (AI) and machine learning (ML) is becoming increasingly prevalent in the pharmaceutical sector, particularly in regulated analytics. As the industry trends towards these advanced technologies, understanding the context in which synthetic data can be utilized becomes essential. This tutorial will provide a comprehensive guide on AI/ML model validation within GxP analytics, with a specific focus on synthetic data, intended use risk, data readiness curation, bias and fairness testing, model verification and validation, and explainability (XAI).

The regulatory bodies such as the US FDA, EMA, MHRA, and PIC/S establish stringent guidelines for the deployment of novel technologies. Organizations must navigate these guidelines effectively to maintain compliance while maximizing the utilization of synthetic data in modeling efforts.

Understanding Synthetic Data and Its Importance

Synthetic data refers to artificially generated data that can be utilized for training machine learning models while ensuring privacy and compliance with regulations. Its importance stems from the necessity to obtain diverse datasets that reflect a wide range of scenarios, thus enhancing model reliability and generalization.

In regulated analytics, synthetic data can help mitigate issues related to data scarcity and sensitivity. Given that patient data is often protected under regulations like 21 CFR Part 11 and GDPR, synthetic data provides a viable alternative, allowing organizations to fulfill their data needs without compromising ethical obligations.

Step 1: Defining the Intended Use of AI/ML Models

The first critical step in AI/ML model validation is to establish the intended use of the model. This involves defining not only what the model will predict or classify but also the context in which it will operate. The details of the intended use should be thoroughly documented and aligned with both the regulatory guidelines and the organization’s business objectives.

• Identify the Specific Application: Clearly define what clinical or operational decision the model will support.
• Understand Regulatory Implications: Different applications may fall under various regulatory frameworks. Understanding these relates directly to the intended use.
• Document the Intended Use Statement: This statement should be detailed and include all relevant information regarding the application of the model, user personas, and expected outcomes.

Step 2: Ensuring Data Readiness and Curation

Once the intended use is defined, the next step is ensuring that the data used for model training is ready and appropriate. Data readiness involves assessing the quality, quantity, and representativeness of the data that will be utilized.

Key components of data readiness curation include:

• Data Quality Assessment: Evaluate completeness, accuracy, and consistency of the dataset to ensure reliable model outcomes.
• Bias Detection: Identify any biases in the data that could influence model outcomes. This can be examined through statistical analysis and other methodologies.
• Data Enrichment: Where appropriate, supplement existing datasets with synthetic data to improve model robustness.

A thorough documentation process should outline how data readiness was verified. This step is fundamental to complying with regulations such as 21 CFR Part 11 (as it relates to electronic records) and GAMP 5 standards.

Step 3: Entering the Analysis: Bias and Fairness Testing

Bias and fairness testing is a crucial phase in AI/ML model validation. The integrity and reliability of AI models hinge significantly on their ability to provide equitable outcomes across diverse patient populations. Therefore, organizations must engage in systematic bias and fairness testing to identify and mitigate potential risks associated with model deployment.

This involves the following:

• Conduct Comprehensive Analyses: Use statistical methods and AI techniques to analyze potential bias across demographic factors.
• Evaluate Outcomes: Assess model predictions and the disparate impact of those predictions on different population groups.
• Enhance Explainability: Utilize explainable AI (XAI) methodologies to provide clarity and uncover the reasoning behind predictions made by models.

Fairness testing should also be documented as part of the overall validation process, demonstrating compliance with both internal governance policies and external regulatory standards.

Step 4: Model Verification and Validation (V&V)

The verification and validation of AI/ML models are paramount to ensuring that models perform as intended in their designated applications. The V&V process involves various strategies to confirm that the model meets its specifications and intended use claims.

The key activities involved in model V&V include:

• Verification: Confirm that the model is developed correctly according to pre-specified protocols and standards.
• Validation: Ensure the model performs effectively and reliably within its intended use context by comparison against real-world datasets.
• Continuous Monitoring: Implement drift monitoring and re-validation strategies to continuously validate model functions over time and against changing real-world conditions.

Comprehensive documentation and audit trails are required throughout the V&V process, fortifying the foundational elements of regulatory compliance.

Step 5: Governance, Security, and Documentation

As AI/ML technology evolves, the establishment of a robust governance framework is crucial for overseeing model deployment and maintaining compliance. This includes ensuring proper data governance and security measures to protect sensitive information derived from patients or clinical settings.

• AI Governance: Establish protocols for compliance with regulatory expectations and internal standards (including ongoing monitoring).
• Security Measures: Implement robust cybersecurity practices to safeguard data integrity and confidentiality.
• Comprehensive Documentation: Maintain a thorough repository of all model-related documentation. This includes intended use, data quality assessments, V&V activities, and results from bias testing.

Adherence to frameworks such as GAMP 5 and relevant regulatory guidelines helps demonstrate a commitment to quality and accountability in AI/ML deployments.

Step 6: Training and Continuous Improvement

As new developments in AI/ML technologies emerge, so must the training of personnel involved in model development and validation processes. Continuous improvement is vital in fostering an adaptive organizational culture adept in utilizing innovative technologies successfully.

This final step includes:

• Ongoing Training Programs: Implement structured training programs that focus on emerging trends and regulatory requirements.
• Feedback Mechanisms: Develop systems for continuous feedback on model outcomes and processes to drive iterative improvements.
• Collaboration: Enhance engagement across teams and disciplines to foster interdisciplinary collaboration in understanding and improving AI/ML applications.

Conclusion

Integrating synthetic data into AI/ML validations presents significant opportunities and challenges for the pharmaceutical industry. As highlighted in this guide, organizations must be proactive in understanding the regulatory landscape, adequately preparing data, validating models, and establishing solid governance frameworks. By adhering to the steps outlined, pharmaceutical professionals can effectively harness AI/ML technologies while ensuring compliance with standards set forth by authorities such as the EMA.

By maintaining rigorous processes for validation and evaluation, pharmaceutical companies can improve not only the efficacy and safety of their AI/ML models but also boost their competitive stance in an increasingly data-driven industry.