Published on 09/12/2025

PQ for AIS: Challenge Sets, Acceptance Criteria, and Robustness

Understanding the Importance of Performance Qualification (PQ) for Automated Inspection Systems (AIS)

Performance Qualification (PQ) is a critical phase in the validation lifecycle of Automated Inspection Systems (AIS) within the pharmaceutical industry. As regulatory scrutiny intensifies worldwide, maintaining compliance with standards set forth by the FDA in the US, EMA in the EU, and MHRA in the UK becomes a necessity. The PQ process ensures that the AIS functions within predefined parameters and meets the required specifications for visual inspection, thereby contributing to product quality and patient safety.

In the validation framework, PQ aims to establish that the system consistently performs as intended within its operational environment. This phase follows Installation Qualification (IQ) and Operational Qualification (OQ), forming part of a comprehensive validation process that often references regulatory guidelines such as 21 CFR Part 11 and the EU Annex 15.

This guide aims to provide a step-by-step approach to establishing effective PQ protocols for AIS, incorporating challenge sets, acceptance criteria, and robustness testing. We will delve into the critical aspects of visual inspection qualification, ensuring a thorough understanding of each element involved in the PQ process.

Step 1: Defining User Requirements Specifications (URS)

The first step in the PQ process involves developing a detailed User Requirements Specification (URS). The URS outlines the expected functions, capabilities, and performance criteria of the AIS, engaging all relevant stakeholders to ensure alignment with production needs and regulatory requirements.

Key components of the URS should include:

• Operational attributes: Define the types of defects the AIS must identify (e.g., particulate matter, container integrity issues).
• Performance metrics: Specify acceptable false reject rates, accuracy, and reliability standards.
• Regulatory references: Incorporate relevant regulatory guidelines and standards that the AIS must comply with.
• Integration with existing systems: Outline how the AIS will interface with current production and quality systems.

Once the URS is established, it should undergo review and approval by all stakeholders, ensuring that the requirements are clear and achievable.

Step 2: Installation Qualification (IQ)

The next step is Installation Qualification (IQ), where the physical installation of the AIS is verified against the URS. During IQ, the following aspects must be evaluated:

• Equipment setup: Confirm that the system is installed according to manufacturer’s specifications.
• Environmental conditions: Ensure that the installation site meets required temperature, humidity, and calibration standards.
• Configuration checks: Validate user access controls and alignment with 21 CFR Part 11 requirements.

Documenting each step of the IQ process is critical, as this will ensure traceability and compliance in future audits.

Step 3: Operational Qualification (OQ)

In the Operational Qualification (OQ) stage, the focus shifts to assessing the functional performance of the AIS. The objective of OQ is to ensure that the system operates within defined parameters under simulated conditions. Key activities in this phase include:

• Testing against protocols: Perform various tests that challenge the system to ensure it can handle a range of operational scenarios.
• Usability assessments: Evaluate user interface and ease of operation for personnel managing the AIS.
• Review of software features: Validate that the software meets the specifications outlined in the URS, including reporting capabilities and data integrity measures.

Like the IQ phase, thorough documentation during OQ provides a basis for future validation efforts and supports regulatory inspections.

Step 4: Developing Challenge Sets for PQ

With IQ and OQ completed, the focus now turns to Performance Qualification (PQ). One of the crucial components of PQ is establishing effective challenge sets. Challenge sets are collections of samples that simulate the production environment and test the AIS’s ability to accurately identify defects. When developing challenge sets, consider the following factors:

• Defect library: Create a comprehensive defect library that includes a range of defect types in varying sizes and shapes.
• Realistic scenarios: Simulate production conditions to ensure the challenge sets reflect actual operational challenges encountered.
• Statistical methods: Utilize attribute sampling methods to deliberately introduce defects at various rates, assessing how often the AIS correctly identifies these issues.

Incorporating diverse scenarios within the challenge sets is paramount for assessing the robustness of the AIS. The resulting data will not only inform regulatory compliance but also help form CAPAs (Corrective and Preventative Actions) when necessary.

Step 5: Defining Acceptance Criteria

Establishing acceptance criteria is integral to the PQ process. Acceptance criteria determine the threshold for system performance considered acceptable. A few key points to consider when defining acceptance criteria include:

• Threshold settings: Determine appropriate thresholds for false reject rates based on industry standards and product specifications. Typically, this involves setting a maximum acceptable false reject rate (often expressed as a percentage) during validation runs.
• Regulatory expectations: Ensure that the acceptance criteria are in line with regulatory guidance. Referencing guidelines such as EU Annex 15 provides a framework for setting these criteria.
• Consistency checks: Acceptance criteria should not only evaluate the performance of the AIS during validation but should also be integrated into routine quality checks to ensure ongoing compliance.

Documenting both the challenge sets and acceptance criteria is crucial for maintaining inspection-readiness throughout the equipment lifecycle.

Step 6: Executing Performance Qualification (PQ) Testing

After establishing the challenge sets and acceptance criteria, the next step is to execute the PQ testing. During this phase, operators will utilize the challenge sets to assess the AIS’s performance. Steps involved in the PQ execution include:

• Test execution: Testing should be conducted in batches, employing the prepared challenge sets sequentially to gather data over multiple runs.
• Data recording: Accurately record performance data, noting both successful defect identifications and any false rejects encountered during the testing process.
• Data analysis: Analyze the compiled data to assess compliance with the established acceptance criteria. Statistical tools may assist in evaluating performance against defined thresholds.

Monitoring performance throughout this phase allows for effective trending analysis and facilitates the identification of patterns in AIS performance, improving long-term reliability.

Step 7: Documentation and Reporting

Following the completion of PQ testing, comprehensive documentation is essential. The documentation should include:

• PQ report: Summarize all test parameters, challenge sets used, data collected, analysis conducted, and outcomes regarding acceptance criteria.
• Change control records: Document any changes made to the AIS or processes during the PQ phase, including corrective actions taken.
• Review and approval: Ensure the final PQ report is reviewed and approved by quality assurance and other relevant stakeholders.

Thorough documentation not only supports compliance with cGMP standards but also serves as an invaluable resource for addressing issues identified during validation or audits.

Step 8: Trending and Continuous Monitoring

Post-PQ execution, the focus shifts to the ongoing monitoring of the AIS’s performance. Establishing a trending process allows ongoing quality assurance and helps mitigate risks associated with false rejects. Consider the following strategies:

• Routine Checks: Implement a schedule for regular performance assessments of the AIS in line with established acceptance criteria.
• Data Collection: Continuously collect and analyze data on AIS performance, paying close attention to any deviations from established patterns.
• CAPA Initiatives: When monitored metrics exceed set limitations, establish CAPAs to address the root cause of the issues, ensuring corrective measures are executed.

This continuous monitoring provides crucial feedback for system improvements, ensuring that the AI systems remain aligned with ever-evolving regulatory expectations and operational needs.

Conclusion: Ensuring Robustness and Compliance in PQ for AIS

Implementing a rigorous PQ process for Automated Inspection Systems is vital to ensuring regulatory compliance and product quality. By following the outlined steps—from defining URS to developing acceptance criteria and challenge sets, executing tests, and monitoring performance—pharmaceutical professionals can establish a robust validation framework.

As technology and regulatory environments evolve, maintaining a proactive approach to validation, including the integration of continuous monitoring and trending, will be essential in mitigating risks associated with false rejects and ensuring the long-term success of AIS. A successful PQ process will not only comply with existing regulations but will also foster confidence among stakeholders and enhance product integrity throughout its lifecycle.