Eli Lilly and Company - Comment

July 17, 2009 Division of Dockets Management (HFA-305) Food and Drug Administration 5630 Fishers Lane, Rm. 1061 Rockville, MD 20852. Via Internet to: http://www.regulations gov Re: Document ID: FDA-2009-N-0192-0005 Dear Sir(s)/Madame(s), We are pleased to submit comments on the Sentinel Initiative contractual report entitled “Defining and Evaluating Possible Database Models to Implement the FDA Sentinel Initiative” [Document ID: FDA-2009-N-0192-0005]. We support the Food and Drug Administration (FDA) in their efforts to develop the Sentinel Initiative as required by Section 905 of the Food and Drug Administration Amendments Act (FDAAA) of 2007. We also applaud FDA in soliciting input and comments from stakeholders and making this information available to the public. We look forward to the continued development of the plans for implementing the Sentinel Initiative. We also look forward to providing additional comments and insights as more specific information regarding the implementation of the Sentinel Initiative is outlined. As the Sentinel Initiative is envisioned to strengthen FDA’s ability to monitor the performance of marketed products, there are a number of elements that will need to be addressed in order to have a successful and sustainable Sentinel system. These include topics such as governance, operations, database models, data sources, scientific methods, communications, privacy, and legal considerations. We would like to commend Brown and colleagues for their efforts to describe Sentinel system user needs and to define characteristics and capabilities for potential database models. Overall, the document provides a very nice summary of the key issues and challenges that will need to be addressed in the Sentinel system, as well as possible approaches that might be utilized. At a high-level the recommendations are reasonable; however, they appear to oversimplify expectations placed on database owners. As very limited information is provided on implementation, this leaves the reader with questions regarding how to resolve many of the challenges that will be encountered by stakeholders. The details of our comments are provided below. Specific Comments on Sentinel Initiative - Possible Database Models 1) Possible Uses of Sentinel System: Primary users provided input on needs relative to the primary goals of the system (page 3). In that list there is mention of safety as well as monitoring the adoption and diffusion of interventions. In the input provided by potential secondary users, “drug effectiveness studies and comparative effectiveness studies” are also mentioned. While we recognize that the primary purpose of the Sentinel Initiative is the evaluation of the potential relationship between medications and adverse events, we believe it is important that any communication resulting from analysis of Sentinel data is done in a way that provides appropriate context relative to the benefit as well as potential risks of the medication. 2) Duplications of Cases: Given the focus on a distributed system relying on central aggregation of results from different databases from different types of entities (payers with claims data, electronic medical record vendors with data warehouses, etc.), it will be important to provide more specifics on approaches to eliminate potential duplication of cases (page 4). 3) Anticipated Attributes to User Needs: External stakeholders noted that the System should be available for “real-time” surveillance (page 5). Due to the nature of administrative and claims data, “real-time” surveillance will be a challenge. Guidance will need to be provided on the types of activities that will be conducted using the various data sources. For the minimum data requirements listed (page 5), we suggest that laboratory data be added to the list. 4) Validation of Exposures and Outcomes: The authors reviewed the strengths and limitations of both claims data and electronic medical record (EMR) data (page 7). One recognized limitation is that diagnoses based on claims data may not always accurately reflect the actual diagnoses in patients’ medical records. The use of claims data may thus impact the appropriate identification of the particular condition that is the focus of the pharmacovigilance study and also the conditions that are potential confounders in the research analysis. The sensitivity and specificity of diagnostic codes in claims data varies depending upon the health condition or adverse event. Researchers have explored this issue in a variety of conditions. For example, researchers reported high sensitivity and specificity values in studies of claims-based algorithms for diabetic macular edema (Bearelly et al, 2008), high positive predictive values for claims for acute myocardial infarction (Kiyota et al, 2004) and chronic kidney disease (Winkelmayer et al, 2005), and high specificity but lower sensitivity (66%) for identifying cases of Parkinsonism (Noyes et al, 2007). Researchers observed relatively low sensitivity (47.8% to 66.2%) for claims-based algorithms for identifying patients with pneumonia (Aronsky et al, 2005), poor sensitivity in studies using claims data to identify cases of outpatient hyponatremia (Shea et al, 2008), poor positive predictive values for using claims data for postoperative deep vein thrombosis and pulmonary embolism (Zhan et al, 2007), and the potential for misclassifying cases when administrative data were used in a study of diagnoses of gout (Harrold et al, 2007). While the referenced studies represent important contributions to medical research efforts, further research in a wide range of disease states needs to be conducted in order to enhance confidence in the results of analyses conducted through the Sentinel Initiative. Such research, including the knowledge gained through chart validations in Sentinel analyses, could contribute to the development of a framework to guide research decisions. Such a framework might utilize a referenced, living document of the sensitivity, specificity, and positive predictive values for claims data for disease states or events of interest in proposed observational studies. Further, as EMR databases mature, the document might also include research on whether the availability of EMR data improves sensitivity and specificity values. For some conditions, claims data may yield sensitivity and specificity values that researchers consider to be acceptable, while for other conditions, the sensitivity and specificity values may be so low that researchers conclude that utilizing claims data alone leads to poor quality analyses with potentially incorrect conclusions. This framework might also depend on the prevalence and severity of specific conditions. It is important to note that the Sentinel Initiative’s emphasis on the importance of chart review for validation should contribute greatly to the quality of analyses conducted through Sentinel. While this approach may allow validation of almost all suspected cases in which a health condition or event is relatively rare, it is important to be as explicit as possible at this stage about how this validation approach might be utilized when a research study focuses on a relatively common event or condition. If only a sample of potential cases are subject to validation through chart review, Sentinel will still need to determine whether the sensitivity and specificity determined by the chart review are sufficient to justify conducting the analysis utilizing claims data primarily. It may also be worth exploring through pilot activities whether medical record review by researchers might be supplemented by an approach involving the treating physicians. In cases in which there is uncertainty about claims data, there may be a role for working with a subset of the treating physicians to provide greater clinical context to help understand potential adverse events. We also suggest that when describing the chart validation process, roles and responsibilities will need to be defined, including who will be responsible for conducting chart reviews. 5) Development of Analysis Plan: The authors state that “analyses be developed centrally and distributed to the data owners for execution against a common data model” (page 10). While we appreciate the potential advantages of using a centrally developed analysis plan, it will be important for the Sentinel Initiative to be as specific as possible about the process by which these analysis plans will be developed. At the very least, it will be important to allow the participating researchers and data holders to contribute insights into development of the analytic plan after having had an opportunity to think through the research question, understand the quality and coding of the required data elements within their respective databases, gain a sense of the distribution of relevant values, and consider potential confounding factors and sensitivity analyses. The ability of the Sentinel Initiative to design and conduct studies of the highest quality may depend on the degree to which Sentinel leverages the ability of excellent researchers in different data environments to think through research questions and then work collaboratively with other researchers to determine an optimal analysis plan. It may also be appropriate to consider a variation in this approach such that researchers in different data environments might first think through the research question and independently develop prospective analysis plans and then conduct the analyses. With the benefit of greater insights into the research question and relevant data, collaborative discussions between the groups of participating researchers might then lead to refinements of the analyses. 6) Handling of Inconsistent Study Results: The document puts forward a model in which results from different data environments are submitted to a central group and aggregated to determine the results of an analysis (page 10). It will be important for the Sentinel Initiative to detail, to the extent possible at this stage, how the results from each group of researchers working in different data environments might be considered. For example, in cases in which the individual data environments have sufficient numbers to address the research question independently, how might the FDA and the Collaborative deal with inconsistent study results from different data environments? If inconsistent findings are not considered and only aggregated results are utilized in determining study conclusions, might there be a risk that incorrect conclusions are reached in the aggregated analysis? If only data aggregated across data environments are considered relevant to study conclusions, might the findings be weighted heavily by the largest data environments – even if their findings are different from those of some of the smaller data environments? At the very least, inconsistent findings from different data environments should prompt the central group to look closely into why the findings are inconsistent. Inconsistency in study findings might be due to a wide range of factors including, for example, differences in coding practices within different data environments, the underappreciated impact of formulary status changes within participating health plans, variability in the length of time that individuals are followed in different databases, and differences in the quality of data and the amount of missing data. Further, in a centrally-driven analysis, how will the Sentinel Initiative deal with cases in which some data environments have access to additional data elements that are not present in all or even most of the participating data environments? Examples might include analyses in which body mass index, smoking status, blood pressure values, and information to better understand disease severity such as echocardiographic assessment of ejection fraction are available. Would this information only be evaluated in the data environments with these data elements or would these elements be excluded from a centrally-developed analysis plan? 7) Clarify Usage of Common Data Model ETL Procedure: The authors recommend that “initial implementation should include a common data model using an ETL procedure” (pages 10-11). While there are strengths of a common data model, it will be important to determine if data holders are allowed to circumvent the ETL approach if they are willing and capable of porting the analysis programs to run in their native database environment and generate results that can be effectively consumed. This determination may affect the participation of some data holders and could also result in analyses of data closer to real-time than some ETL processes may support. 8) Clarification of Linkages: The term linkage is used in several sections of the document. For example, on page 4 there is reference to the “ability to link individuals across data sources,” page 5 notes that the system must be able to “link across datasets” and page 16 mentions the possibility of “cross- institutional linkages.” It will be important to provide additional details on how such linkages can be accomplished in a distributed network and what types of databases might be required. We appreciate the opportunity to comment on the current report on possible database models for the Sentinel Initiative. Please let us know if you have questions or need additional clarification. We would also welcome the chance to provide additional comments and insights as more detailed information is released on the Sentinel Initiative. Sincerely, Donald G. Therasse, M.D. Vice President, Global Patient Safety Lilly Research Laboratories Eli Lilly and Company