FDA Issues Detailed Guidance for Biopharma Industry on What Information to Provide to FDA About Use of AI Models

At a Glance

• FDA lists three main challenges to related regulatory decision-making and recommends the solutions to those challenges.
• The guidance sets out FDA’s proposal of a “risk-based credibility assessment framework” for use by industry and FDA “to establish and assess the credibility of an AI model output for a specific COU [context of use] based on model risk.” This involves a seven-step process, and FDA provides recommendations for each.

On January 7, 2025, the Food and Drug Administration (FDA) made available a draft guidance for industry (GFI) titled “Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products.” After reviewing some potential benefits of AI use and listing potential use cases in the development of drugs, FDA lists three main challenges to related regulatory decision-making and recommends the solutions to those challenges: 

First, FDA is concerned that variability in training datasets (size, representativeness, etc.) “may introduce bias and raise questions about the reliability of AI-driven results.” Therefore, AI models must be proven “fit for use,” requiring use of data that is both relevant (i.e., includes key elements and is representative) and reliable (i.e., accurate, complete and traceable). 

Second, providing methodological transparency is essential but challenging given that models are based on “complex computational and statistical methodology.” Sponsor descriptions of the methods and processes used to develop AI models should be sufficiently detailed for FDA to be able to understand not only how the models were developed but also how they arrive at their conclusions. 

Third, FDA notes that it is challenging to “interpret, explain, or quantify” the “uncertainty of accuracy” element of model output. Lastly, FDA calls out the issue of data drift — e.g., resulting from new data inputs and/or use of the model in different deployment environments — and the resultant necessity for life-cycle model maintenance. 

The guidance sets out FDA’s proposal of a “risk-based credibility assessment framework” for use by industry and FDA “to establish and assess the credibility of an AI model output for a specific COU [context of use] based on model risk.” This involves a seven-step process; and FDA provides recommendations for each, broadly paraphrased below. For steps 1 through 3, FDA provides two helpful and detailed examples, one in the clinical development phase and another in the manufacturing context: 

Step 1: Define the question of interest that will be addressed by the AI model. 

• Be specific. 
• Remember that current good manufacturing practice (CGMP) regulations apply, too. For finished drug products, note quality unit responsibilities under 211.22 and 211.68, in particular. (For API, see ICH Q7.)

Step 2: Define the COU for the AI model (i.e., role and scope of model to address question of interest).

• Provide details about what will be modeled and how the model’s outputs will be used.
• Specify all evidentiary sources (e.g., in vitro testing, in vivo animal testing).
• If uncertain about evidentiary sources, engage early with FDA.

Step 3: Assess the AI model risk (i.e., model influence and decision consequence).

• Use a variety of evidentiary sources “in conjunction with evidence generated from the AI model” to address the question (p. 7; emphasis in original).
• Consider how sources of evidence are relevant when assigning a risk rating (FDA examples used low, medium, high ratings) to “model influence,” that is, “the contribution of the evidence derived from the AI model relative to other contributing evidence.”
• Characterize risk from “decision consequence” by describing the significance of adverse outcomes resulting from incorrect decisions. See footnote 22 on page 8 for additional information. 
• Provide a model risk matrix with model influence and decision consequence axes to explain the overall model risk given its COU. 

Steps 4, 5 and 6: Develop and execute a plan to assess the credibility of a model’s output given the COU, and document the results of a credibility assessment plan. 

• See Section 4, footnote 25, for more on how FDA meetings may be appropriate times for submission of proposed model credibility assessment plans. (pp. 9-10)
• Activities should be tailored to the COU and commensurate with the model’s assigned risk level.
• Explain/describe model inputs and outputs, architecture, features (e.g., clinical measurements, demographics, clinical imaging data), features selection process, and model parameters (i.e., variables that affect how the model computes outputs). In addition, provide a rationale for choosing the described modeling approach.
• Describe training data (model weight, connections, components, etc.) and tuning dataset (different architectures and hyperparameters explored, etc.). Remember to provide relevance and reliability of data to support its fitness for use. 
• Describe development datasets and model training and evaluation process in close accord to the breakdowns provided by FDA on pages 12 through 14 of the guidance. These are comprehensive and sponsors are wise to tick every box, separately and distinctly, erring on the side of inclusiveness over worry for repetitiveness. 

Generally, FDA wants industry to engage with it on steps 3 and 4. Consult Table 1, “Engagement Options Other than Formal Meetings,” to determine which meeting type to request (pp. 18-20). FDA envisions companies consulting with the Agency through submissions that cover steps 1 through 3 and some of step 4: Sponsors should propose “higher-level” credibility assessment activities, and FDA’s feedback on those will inform development of a “more detailed credibility assessment plan drafted after the iterative process” (p. 10).

• Once the fully developed plan is executed, record results in a “credibility assessment report.” Include any deviations. During FDA engagement opportunities, ask FDA to advise on the report’s format and on when and how to share it with the agency.

Step 7: Determine the adequacy of the AI model for the COU.

• If model creditability is not sufficiently established, downgrade its influence and/or otherwise adjust the model per FDA feedback.

FDA then explains how important it is to ensure any AI model remains fit for use throughout its deployment. It recommends performance metrics monitoring. Sponsors should take a risk-based approach when assessing the impacts of any changes implemented (e.g., manufacturing change), or a model’s otherwise changing outputs, when determining what activities will be sufficient to maintain credibility of the model’s use in the COU. Sponsors should document all such output credibility maintenance activities and report to FDA on any changes they make to models, as is required for other kinds of post-approval changes. When determining the appropriate mechanism to use to report model changes post-approval, consider how the changes will affect the model’s output and any impact on product quality.