How Can Pharma Use GenAI Responsibly for FDA-Regulated Products?

Why "human in the loop" and expert review are essential to effective use in the pharmaceutical industry.

Earlier this year, the U.S. Food & Drug Administration (FDA) issued a first-of-its-kind warning letter to a third-party manufacturer that used generative AI (GenAI) to create drug product specifications, standard operating procedures, and production and control records. These documents resulted in violations noted during FDA's inspection of the facility. When asked why basic precautions and testing were not performed, the company stated that GenAI neglected to tell the manufacturer about a needed step in the process. This situation highlights the pitfalls that can arise from overreliance on GenAI in pharmaceutical manufacturing, and how FDA may intend to regulate its use going forward. 

What did FDA find?

FDA highlighted two major violations related to the use of GenAI. The first violation centers on the company's failure to thoroughly review the AI-generated process and manufacturing documents that ultimately affected product quality. Under FDA regulation 21 CFR 211.22(c), manufacturers must review these documents to ensure they are accurate and compliant with Current Good Manufacturing Practice (cGMP). The second GenAI-related violation (21 CFR 211.100) was associated with process validation, which the company failed to perform. 

From the FDA letter: ". . . FDA investigators found that you had not conducted process validation prior to distribution of your drug products, as required under 21 CFR 211.100, and informed you as such. You replied that you were not aware of the legal requirement, as the AI agent you used . . . never told you it was required."

The specific GenAI-associated violations noted highlight the importance of expert human oversight in the successful deployment of this technology in the pharmaceutical industry. While FDA did not discourage GenAI tools altogether in its warning letter, the agency stated that any AI-based systems used must be validated, including those deployed across manufacturing, quality control, and marketing. FDA also stated that if GenAI systems are intended for medical or diagnostic use, manufacturers must obtain the necessary FDA clearance, approval, or authorization.

How to effectively manage GenAI

Use of GenAI for document generation and review, including quality system procedures like those described in the FDA warning letter, is rapidly growing. These uses present challenges due to the bespoke, non-repeatable nature of the workflows used to generate documents, meaning established verification and validation frameworks may no longer be valid. 

Additionally, there are several critical risks associated with the use of GenAI in FDA-regulated settings:

• Loss of Data Integrity: GenAI-generated summaries may distort source material, undermining traceability between raw data, interpretation, and reported conclusions.
• Propagation of Errors Across Stakeholders: Once inaccuracies are introduced into downstream documents (e.g., internal briefings, regulatory submissions), they can be amplified across cross-functional teams, including legal and quality functions.
• Confidentiality and Data Security Risks: Use of external or uncontrolled GenAI tools may expose confidential information — including proprietary manufacturing processes, contamination findings, and regulatory strategies — to unintended third parties. In many cases, there is limited transparency regarding how data are stored, retained, or potentially used for model training, creating potential risk to intellectual property and regulatory confidentiality obligations.
• Increased Cost and Resource Burden: Errors introduced by GenAI tools can lead to significant rework, offsetting any perceived efficiency gains. In addition, inaccurate or inconsistent regulatory submissions may prompt extensive follow-up questions from regulatory authorities, requiring further investigation, clarification, and documentation. These downstream impacts can materially increase project cost, extend timelines, and divert subject matter expert (SME) resources from higher-value activities.
• False Confidence in Automation: The perceived efficiency of GenAI tools may lead to overreliance, particularly when outputs are not rigorously validated by qualified SMEs.
• Regulatory Exposure: Submission of inaccurate or incomplete information to regulatory authorities increases the risk of inspection findings, credibility loss, and potential enforcement actions.

Teams leveraging GenAI for document generation outside the scope of traditional frameworks can look to expert human quality system document review. Effective oversight of AI-generated documents depends on experts who can spot not only hallucinations or content errors but also omissions, which are more difficult to find. GenAI content can exhibit several critical failure modes:

• Hallucination: Introduction of statements, interpretations, or conclusions not supported by the underlying data.
• Omission: Exclusion of key technical findings or contextual qualifiers necessary for accurate interpretation.
• Loss of Confidential Control: Submission of sensitive investigation data into AI tools without clear governance over data retention, reuse, or access.

The value of a human in the loop

The "human in the loop" concept (in which human involvement is required in automated systems) supports accountability for decisions that impact product quality or patient safety. Transparency and explainability are also important. Regulators and users might request or require an explanation of how a GenAI system generates its results. AI models are expected to be reasonably interpretable (i.e., the inner workings are known and understood), with clear documentation on their logic, training methods, and limitations.

Some key principles that can support the responsible use of GenAI in regulated environments include:

• Restrict Inputs to Verified, Controlled Information: GenAI tools can be restricted to well-defined, quality-controlled datasets. Use of base foundation models, draft, unverified, or interpretive content increases the likelihood of generating inaccurate or misleading outputs.
• Require SME Review and Accountability: All GenAI content intended for technical or regulatory use can undergo thorough review by qualified subject-matter experts to confirm accuracy, completeness, and appropriate interpretation.
• Avoid Reliance on AI for Primary Technical Synthesis: AI tools may support administrative or low-risk tasks; however, their use for summarizing complex technical investigations, particularly those intended for regulatory review, poses risks that benefit from a cautious approach.

How can GenAI meet FDA and other requirements?

Compliance with regulatory guidance underpins responsible GenAI use. Data privacy laws like HIPAA and GDPR must be observed when personal or clinical data are used. To coordinate these practices, companies may benefit from creating an internal governance structure to oversee ethics, monitor bias, establish vendor qualification processes, and perform continuous audits of AI performance.

Responsible GenAI in the pharmaceutical industry means treating these systems as regulated, auditable components of a company's quality and compliance framework. By emphasizing rigorous validation, trustworthy data, human oversight, interpretability, and ethical governance, pharmaceutical innovators can harness GenAI's benefits while maintaining patient safety and regulatory integrity. SMEs are crucial in this and other highly specialized technical fields that require precise documentation for regulatory and research purposes. Technical experts have the experience and reasoning skills to spot omissions and hallucinations and check references. Even as GenAI tools evolve and improve, their use will continue to require expert guidance to be effective in the pharmaceutical industry.