Taken together, these two guidances seek to provide greater clarity for addressing the acceptable degree of clinical "uncertainty" about a medical device's safety and effectiveness in moving innovative products to market quickly to meet patient needs while also protecting public health from potentially unsafe products. Notably, although the final uncertainty guidance did not differ materially from the draft guidance issued in September 2018, the updated benefit-risk guidance features a completely new worksheet for benefit-risk assessment. The new worksheet follows a decision-tree format (rather than the prior table format) and includes new questions and a sequential method for considering benefit-risk and how to mitigate risks.
The uncertainty guidance includes illustrative examples of the impact on clinical trial size under different hypothetical scenarios of uncertainty designated "conventional," "modest," and "high," along with recommended shifts to postmarket data collection associated with each level of uncertainty. Although these examples reflect specific statistical thresholds in uncertainty analysis, the guidance stresses that the hypotheticals are not meant to convey established criteria for determinations. It remains to be seen whether this will have any meaningful impact on the threshold level of safety and effectiveness that needs to be established, or whether it will lead to more postapproval study orders because FDA has yet another incentive to ask for them.
In a rare move for FDA, the uncertainty guidance devotes a substantial portion of the document to discussing the purpose for the guidance. This background section emphasizes the "flexible" and "tailored" approach FDA takes in reviewing each device with consideration for the totality of the evidence and the context in which it was generated, such as "the applicable patient population's willingness to accept more uncertainty…particularly when there are no acceptable alternatives available." Thus, while it appears that the agency is seeking a path toward tolerating greater uncertainty in premarket clinical studies, it remains to be seen whether FDA will continue to default to the requirement for randomized, controlled studies in large populations.
FDA is holding a webinar to discuss this new guidance on 16 October 2019.
What's new: benefit-risk guidance
Although the uncertainty guidance in its final form closely tracked the draft, the companion release of the updated final benefit-risk guidance replaces the existing table for benefit-risk assessment with a tool that is completely new in both form and substance.
The format of the benefit-risk assessment is substantially changed; what had been a table of factors and mostly high-level questions with space for reviewer notes has become a highly specific checklist in the form of a decision tree. Uncertainty no longer appears as an "additional" factor; it now appears as an integral component throughout the sequencing of questions. Although it is highly prescriptive, the worksheet also leaves considerable room for the discretion of the reviewer.
At a high level, in the new worksheet, FDA reviewers are asked to walk through a checklist of evidence of clinical benefit followed by an assessment of the "degree of uncertainty" as low, medium, or high. A similar process is repeated for assessment of risk, with an assessment of the extent of uncertainty for the risks. Under each, the reviewer is to summarize the assessment of benefits and risks, considering the factors (e.g., type, magnitude of benefit, duration of effects) that appeared in the table in the prior guidance. Reviewers are then asked if the benefits outweigh the risks, taking into account a list of additional considerations (e.g., patient preferences, availability of alternative therapies). Reviewers must then consider whether risks can be mitigated via labeling and training, so that benefits outweigh the risks, and then whether the benefits outweigh the risks considering possible postmarket data collection. If the reviewer is unable to determine whether the benefits outweigh the risks, they are directed to go back to the beginning of the decision tree and review any possible modified indications for use where there is evidence of clinical benefit.
In general, questions prior to the end of the decision tree prompting the reviewer to weigh benefits against risks can be answered with either "yes" or "unable to conclude" (rather than "no"), and also instruct the reviewer to consider all options, so it appears to lean more toward clearance/approval. It also provides a more structured way to consider these options in a particular order (labeling/training first, then postmarket studies, then revised indications).
Assessment of benefits and risks
For assessment of benefit, the flow chart allows for "any evidence of clinical benefit" followed by a checklist with a number of clinical scenarios, which includes the possibility of nonclinical or modeling instead of clinical data. Interestingly, the instructions note that "benefit should be considered based on the assessment of the data, whether or not the results are statistically significant." As such, this decision tree appears to allow more variability in demonstrating evidence of benefit than is typical for FDA, which tends to require both statistical and clinical significance of results. If there is no evidence of clinical benefit, modified indications for use should be considered.
For assessment of risk, the question is framed as "Are known/probable risks more than minimal?" Regardless of whether the answer is "yes" or "no," the decision tree moves on to the next question.
For all the questions in the decision tree, the scenarios are very specific, which is a significant departure from the previous (table) version of the assessment. For example, under the assessment of benefit, options include: "A favorable change in at least 1 clinical assessment that is equal to or greater than seen in the control group"; "A favorable change in at least 1 clinical assessment that meets a predetermined performance goal"; etc. The high degree of specificity in the checklists likely reflects the most common scenarios that FDA typically sees in submissions, but could not cover all scenarios. The checklists provide the option of "Other," presumably to deal with this issue.
How does uncertainty factor into benefit-risk determinations?
Although the decision tree provides highly detailed checklists for uncertainty, it is not clear how the uncertainty analysis figures into the benefit-risk determination. The worksheet requests a determination of whether the extent of uncertainty is "low," "medium," or "high" without defining these categories. It is not clear whether these categories relate to the scenarios in the uncertainty guidance described as "conventional," "modest," or "high" uncertainty. Moreover, the flow chart provides for no divergent path depending on which level of uncertainty is specified. Whether the reviewer checks "low," "medium," or "high," the worksheet simply directs the user to the next question.
Thus, although there is now ample accounting for uncertainty in the new benefit-risk worksheet, there is still significant lack of clarity in how uncertainty factors into the analysis, despite this being the stated purpose of the update to the benefit-risk guidance.
These two guidances provide insight into FDA's approach to determining the appropriate extent of uncertainty for a device in premarket review. Yet considerable uncertainty remains as to how this analysis will be applied. The benefit-risk worksheet is new, containing both new factors and a new, more prescriptive decision-tree format. Given the criticality of the benefit-risk assessment to premarket approval (PMA) and De Novo product reviews, the new worksheet has the potential to impact reviewer decisions on these products. Companies should review the new worksheet carefully and ensure that the factors are addressed in their product submissions.
Authored by Janice Hogan, Randy J. Prebula and Kristin Zielinski Duggan