- Search for Research Summaries, Reviews, and Reports
- DEcIDE Project
- Registries for Evaluating Patient Outcomes: A User's Guide
- Research Report May 16, 2007
Executive Summary – May 16, 2007
Registries for Evaluating Patient Outcomes: A User's Guide
Table of Contents
- Front Matter
- User's Guide to Registries Evaluating Patient Outcomes: Summary
- Evaluating Registries
The Effective Health Care Program of the Agency for Healthcare Research and Quality (AHRQ) conducts and supports research focused on the outcomes, effectiveness, comparative clinical effectiveness, and appropriateness of pharmaceuticals, devices, and health care services. More information on the Effective Health Care Program and electronic copies of this report can be found at www.effectivehealthcare.ahrq.gov.
This report is the summary of Registries for Evaluating Patient Outcomes: A User's Guide, which was produced under contract to AHRQ as a collaborative effort between two centers involved in the Effective Health Care Program: The Outcome DEcIDE Center (Developing Evidence to Inform Decisions about Effectiveness) under Contract No. HHSA29020050035I TO1, and the Duke EPC (Evidence-based Practice Center) under Contract No. 290-02-0025 TO1. The AHRQ Task Order Officer for this project was Elise Berliner, Ph.D. The findings and conclusions in this summary and the larger document are those of the authors, who are responsible for the contents; the findings and conclusions do not necessarily represent the views of AHRQ or the U.S. Department of Health and Human Services. Therefore, no statement in this report should be construed as an official position of AHRQ or the U.S. Department of Health and Human Services.
Registries for Evaluating Patient Outcomes: A User's Guide and User's Guide to Registries Evaluating Patient Outcomes: Summary are both copyrighted by the Agency for Healthcare Research and Quality (AHRQ). The products and their contents may be used and incorporated into other materials on the condition that the contents are not changed in any way (including covers and front matter) and that no fee is charged by the reproducer of the products or their contents for their use. The products may not be sold for profit or incorporated into any profit-making venture without the expressed written permission of AHRQ.
- When one of the documents is reprinted, it must be reprinted in its entirety without any changes.
- When parts of the documents are used or quoted, the following citation should be used.
*Note: Registries for Evaluating Patient Outcomes: A User's Guide contains material copyrighted by others. For material noted as copyrighted by others, the user must obtain permission from the copyright holders identified in the document. This is a summary of a longer, more comprehensive report.
Citation for full report should be used: Gliklich RE, Dreyer NA, eds. Registries for Evaluating Patient Outcomes: A User's Guide. (Prepared by Outcome DEcIDE Center [Outcome Sciences, Inc. dba Outcome] under Contract No. HHSA29020050035I TO1.)
AHRQ Publication No. 07-EHC001-1. Rockville, MD: Agency for Healthcare Research and Quality. April 2007.
Full Report: AHRQ Publication Number 07-EHC001-1
Summary Report: AHRQ Publication Number 07-EHC001-2
User's Guide to Registries Evaluating Patient Outcomes: Summary
Patient registries are organized systems that collect data for scientific, clinical, or policy purposes. Registries are a valuable complement to randomized controlled trials in determining real-world outcomes in the practice of medicine. They do not generally have restrictive inclusion or exclusion criteria, nor do they specify what therapy the health care provider must adhere to. They can be used to evaluate outcomes for diverse purposes ranging from the natural history of a disease, to the safety of drugs or devices, to the real-world effectiveness of therapies.
The Effective Health Care Program of the Agency for Healthcare Research and Quality (AHRQ) conducts and supports research focused on the outcomes, effectiveness, comparative clinical effectiveness, and appropriateness of pharmaceuticals, devices, and health care services. As part of the Effective Health Care Program, Outcome Sciences, Inc., a DEcIDE (Developing Evidence to Inform Decisions about Effectiveness) center, and the Duke EPC (Evidence-based Practice Center) collaborated in a study of registries and the many elements involved in creating a registry.
Outcome Sciences, with the assistance of the Duke EPC, coordinated a group of 39 contributors and 35 reviewers to develop a handbook to serve as a guide to the design, implementation, analysis, interpretation, and evaluation of the quality of a registry for understanding patient outcomes.
This is a summary of the full handbook. The "Overview" presents basic information on the main areas to consider in setting up a registry, from selection of data elements and protection of patient privacy to analyzing results and publishing findings. "Evaluating Registries" outlines elements of quality to be considered in setting up or evaluating a registry. Basic good registry practices are given, as well as future directions for practices that could enhance scientific rigor but may not be practical for every registry.
The full report is available online at www.effectivehealthcare.ahrq.gov. Free print copies are available by calling 800-358-9295. Ask for Registries for Evaluating Patient Outcomes: A User's Guide, AHRQ Publication Number 07-EHC001-1.
For the purpose of this document, a patient registry is an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves a predetermined scientific, clinical, or policy purpose(s). The registry database is the file (or files) derived from the registry.
Although registries can serve many purposes, the focus here is on registries that are created for one or more of the following purposes: to describe the natural history of disease, to determine clinical effectiveness or cost effectiveness of health care products and services, to measure or monitor safety and harm, and/or to measure quality of care. Registries are classified according to how the populations are defined. For example, product registries include patients who have been exposed to biopharmaceutical products or medical devices. Health services registries consist of patients who have had a common procedure, clinical encounter, or hospitalization. Disease or condition registries are defined by patients having the same diagnosis, such as cystic fibrosis or heart failure.
There are several key steps in planning a patient registry, including articulating the purpose of the registry, determining whether the registry is an appropriate means for addressing the research question, identifying stakeholders, defining the scope and target population, assessing feasibility, and securing funding. The registry team and advisors should be selected based on expertise and experience. The plan for registry governance and oversight should clearly address such issues as overall direction and operations, scientific content, ethics, safety, data access, publications, and change management. It is also helpful to plan for the entire lifespan of a registry, including how and when the registry will end and any plans for transitioning the registry at that time.
A patient registry should be designed with respect to its major purpose, with the understanding that different levels of rigor may be required for registries that are designed to address focused analytical questions to support decisionmaking, in contrast to those intended primarily for descriptive purposes. The key points to consider in designing a registry include formulating a research question; choosing a study design; translating questions of clinical interest into measurable exposures and outcomes; choosing patients for study, including deciding whether a comparison group is needed; determining where data can be found; and deciding how many patients need to be studied and for how long. Once these key design items have been determined, the registry design should be reviewed to evaluate potential sources of bias (systematic error); these should be addressed to the extent that is practical and achievable.
The specific research questions of interest will guide the registry design, identification of exposures and outcomes, and definitions of the target population (the population to which the findings are meant to apply). The registry population should be designed to approximate the characteristics of the target population as much as possible. The number of study subjects desired and length of observation (followup) should be planned in accordance with the overall purpose of the registry. The desired study size (in terms of subjects or person-years of observation) is determined by specifying the magnitude of an expected clinically meaningful effect or the desired precision of effect estimates. Study size determinants are also affected by practicality, cost, and whether or not the registry is intended to support regulatory decisionmaking. Depending on the purpose of the registry, internal, external, or historical comparison groups strengthen the understanding of whether the observed effects are indeed real and, in fact, different from what would have occurred under other circumstances.
Registry study designs often restrict eligibility for entry to individuals with certain characteristics to assure that they will assemble enough information for analysis (e.g., age restriction) or use some form of sampling-random selection, systematic sampling, or a haphazard, nonrandom approach. The potential for bias refers to opportunities for systematic errors to influence the results. The information value of a registry is enhanced by its ability to provide an assessment of the potential for bias and to quantify how this bias could affect the study results.
The selection of data elements requires balancing such factors as their importance for the integrity of the registry and for the analysis of primary outcomes, their reliability, their contribution to the overall burden for respondents, and the incremental costs associated with their collection. Selection begins with identifying relevant domains. Specific data elements then are selected with consideration for established clinical data standards, common data definitions, and the use of patient identifiers. It is important to determine which elements are absolutely necessary and which are desirable but not essential. In choosing measurement scales for assessing patient-reported outcomes, it is preferable to use scales that have been appropriately validated, when such tools exist. Once data elements have been selected, a data map should be created, and the data collection tools should be pilot tested. Testing allows assessment of respondent burden, accuracy, and completeness of questions, and potential areas for missing data. Inter-rater agreement for data collection instruments can also be assessed, especially in registries that rely on chart abstraction. Overall, choice of data elements should be guided by parsimony, validity, and a focus on achieving the registry's purpose.
A single registry may integrate data from various sources. The form, structure, availability, and timeliness of the required data are important considerations. Data sources can be classified as primary or secondary. Primary data are collected for direct purposes of the registry. Secondary data are comprised of information that has been collected for purposes other than the registry, and they may not be uniformly structured or validated with the same rigor as primary data. Sufficient identifiers are necessary to guarantee an accurate match between secondary sources and registry patients. Furthermore, a solid understanding of the original purpose of the secondary data and how they were collected is advised, because the way that those data were collected and verified or validated will help shape their use in a registry. Common secondary sources of data linked to registries include medical records systems, institutional or organizational databases, administrative health insurance claims data, death and birth records, census databases, and related existing registry databases.
Ethics, Data Ownership, and Privacy
Critical ethical and legal considerations should guide the development and use of patient registries. The Common Rule is the uniform set of regulations on the ethical conduct of human subjects research from the Federal agencies that fund such research. Institutions that conduct research agree to comply with the Common Rule for federally funded research and may opt to apply that rule to all human subjects activities conducted within their facilities or by their employees and agents, regardless of the source of funding. The Health Insurance Portability and Accountability Act of 1996 (HIPAA) and its implementing regulations (collectively, the Privacy Rule) are the legal protections for the privacy of individually identifiable health information created and maintained by health care providers, health plans, and health care clearinghouses (called "covered entities"). The research purpose of a registry, the status of its developer, and the extent to which registry data are individually identifiable largely determine applicable regulatory requirements. Other important ethical and legal concerns include transparency of activities, oversight, and data ownership.
Patient and Provider Recruitment and Management
Recruitment and retention of providers (as registry sites) and patients are essential to the success of a registry. Recruitment typically occurs at several levels, including facilities (hospital, practice, pharmacy), providers, and patients. The motivating factors for participation at each level and the factors necessary to achieve retention differ according to the registry. Factors that motivate participation include the perceived relevance, importance, or scientific credibility of the registry, as well as the risks and burdens of participation and any incentives for participation. Because provider and patient recruitment and retention can affect how well a registry accurately represents the target population, well-planned strategies for enrollment and retention are critical. Goals for recruitment, retention, and followup should be explicitly laid out in the registry planning phase, and deviations during the conduct of the registry should be continuously evaluated for their risk of introducing bias.
Data Collection and Quality Assurance
The integrated system for collecting, cleaning, storing, monitoring, reviewing, and reporting on registry data determines the utility of those data for meeting the registry's goals. A broad range of data collection procedures and systems are available. Some are more suitable than others for particular purposes. Critical factors in the ultimate quality of the data include how data elements are structured and defined, how personnel are trained, and how data problems are handled (e.g., missing, out-of-range, or logically inconsistent values). Registries may also be required to conform to guidelines or standards of specific end users of the data (e.g., 21 Code of Federal Regulations, Part 11). Quality assurance aims to affirm that the data were, in fact, collected in accordance with established procedures and that they meet the requisite standards of quality to accomplish the registry's intended purposes and the intended use of the data.
Requirements for quality assurance should be defined during the registry's inception and creation. Because certain requirements may have significant cost implications, a risk-based approach to developing a quality assurance plan is recommended. It should be based on identifying the most important or likely sources of error or potential lapses in procedures that may impact the quality of the registry in the context of its intended purpose.
Adverse Event Detection, Processing, and Reporting
The U.S. Food and Drug Administration defines an adverse event (AE) as any untoward medical occurrence in a patient administered a pharmaceutical product, whether or not related to or considered to have a causal relationship with the treatment. AEs are categorized according to the seriousness and, for drugs, the expectedness of the event. Although AE reporting for all marketed products is dependent on the principle of "becoming aware," collection of adverse event data falls into two categories: those events that are intentionally solicited (meaning data that are part of the uniform collection of information in the registry) and those that are unsolicited (meaning that the AE is volunteered or noted in an unsolicited manner).
Determining whether the registry should use a case report form to collect AEs should be based on the scientific importance of the information for evaluating the specified outcomes of interest. Regardless of whether or not AEs constitute outcomes for the registry, it is important to develop a plan for detecting, processing, and reporting AEs for any registry that has direct patient interaction. If the registry receives sponsorship, in whole or in part, from a regulated industry (for drugs or devices), the sponsor has mandated reporting requirements, and the process for detecting and reporting AEs should be established and registry personnel trained on how to identify AEs and to whom they should be reported. Sponsors of registries designed specifically to meet requirements for surveillance of drug or device safety are encouraged to hold discussions with health authorities about the most appropriate process for reporting serious AEs.
Analysis and Interpretation
Analysis and interpretation of registry data begin with answering a series of core questions. Who was studied? How were the data collected, edited, and verified, and how were missing data handled? How were the analyses performed? Four populations are of interest in describing who was studied: the target population, the accessible population, the intended population, and the population actually studied (the "actual population"). The representativeness of the actual population to the target population is referred to as generalizability.
Analysis of registry outcomes first requires an analysis of the completeness of data collection and data quality. Considerations include an evaluation of completeness for most if not all important covariates and an understanding of how missing data were handled and reported. Analysis of a registry should provide information on the characteristics of the patient population, the exposures of interest, and the endpoints. Descriptive registry studies focus on describing frequency and patterns of various elements in a patient population, whereas analytical studies concentrate on associations between patients or treatment characteristics and health outcomes of interest. A statistical analysis plan describes the analytical plans and statistical techniques that will be used to evaluate the primary and secondary objectives specified in the study plan. Interpretation of registry data should be provided so that the conclusions can be understood in the appropriate context and so that any lessons from the registry can be applied to the target population and used to improve patient care and outcomes.
Although registries can provide useful information, there are levels of rigor that enhance validity and make the information from some registries more useful for guiding decisions than the information from others. The term "quality" can be applied to registries to describe the confidence that the design, conduct, and analysis of the registry can be shown to protect against bias and errors in inference-that is, erroneous conclusions drawn from a registry. Although there are limitations in any assessment of quality, this handbook uses a quality component analysis to evaluate high-level factors that may affect results and differentiates between research quality (which pertains to the scientific process) and evidence quality (which pertains to the data/findings emanating from the research process). Quality components are classified as either "basic elements of good practice," which can be viewed as a basic checklist that should be considered for all patient registries, or as "potential enhancements to good practice" that may strengthen the information value in particular circumstances. The results of such an evaluation should be considered in the context of the disease area(s), the type of registry, and the purpose of the registry, and should also take into account feasibility and affordability.
Registries are undertaken for many purposes, ranging from descriptive studies intended to contribute to scientific understanding of patient outcomes to studies used to inform policy decisions. Some are undertaken with great urgency, whereas others proceed with more deliberation. Budgetary support ranges from spartan to adequate. Most importantly, registries often serve multiple purposes and change over time to accommodate these various purposes-in fact, these are hallmarks of registries. Although all registries can provide useful information, there are levels of rigor that enhance validity and make the information from some registries more useful for guiding decisions than others.
To date, no standards have been developed by which to guide evaluation of registries, and the research into quality aspects of registries has been sparse. This is an overview of key components of the design, execution, and analysis of a registry that promote reliability and validity of data on patient outcomes.
The aim here is to provide a simple and user-friendly system that allows registries to be described and evaluated in the context of the purpose for which they are conducted. Information is presented to help distinguish between:
- Basic good registry practices that are desirable to meet certain purposes.
- Future directions for practices that could enhance scientific rigor but may not be achievable because of practical constraints.
The items listed as "basic elements of good practice" are applicable to all patient registries. While it may not be practical or feasible to achieve all of the basic elements of good practice, it is useful to consider these characteristics in planning and evaluating registries. The information described in this handbook, and particularly in this chapter, is also designed to be used in reporting registry study results, much as CONSORT guidelines have been used to improve reporting of clinical trials.
A definition of "quality" for registers was adapted from a definition developed for randomized controlled trials; the term is used to refer to the confidence that the design, conduct, and analysis of the trial or registry can be shown to protect against bias (systematic error) and errors in inference-that is, erroneous conclusions drawn from a study. As used here, quality refers both to the data and to the conclusions drawn from analyses of these data. For more information about bias, validity, and inference, readers are encouraged to consult the handbook and epidemiologic textbooks.
There are two major difficulties with assessing quality in registries:
- It can often be difficult to differentiate between the quality of the design, the study conduct, and the information available.
- There is a lack of empirical evidence for evaluating parameters purported to indicate quality and impact on the evidence produced from registries.
In addition, registries vary widely in methodology, scope, and objectives, and therefore attributes that are important in one scenario may be less important in another. Furthermore, registries may be very useful vehicles for providing clinically relevant real-world information, even when they meet relatively few of the basic elements of good practice (typically because of budgetary limitations). In many cases, some data are better than no data, and even registries that fall short of including all the basic elements of good registry practice may still provide valuable insights about real-world medical and consumer practices and disease etiology. Evaluations of the quality of any registry must therefore be done with respect to the context-specific purpose of the registry, must take into account both the internal and external validity of the data, and should be tempered by considerations of cost and feasibility.
The most commonly used method to assess quality of studies is a quality scale; there are numerous quality scales of varying length and complexity in existence, with strong views being expressed both for and against their use. Different scales emphasize distinctive dimensions of quality and therefore can produce disparate results when applied to a given study. In most situations, a summary score is derived by adding individual item scores, with or without weighting. This method, however, ignores whether the various items may lead to a bias toward the null (suggesting the erroneous interpretation that there is no effect) or tend to exaggerate the appearance of an effect when none really exists, and the final score produced does not reflect individual components.
Rather than develop a checklist, the approach suggested here is to undertake a quality component analysis, an investigation of the components that may affect the results obtained. In the quality component analysis, a differentiation is made between two domains: research quality, which pertains to the scientific process (in this instance, the design and operational aspects of the registry), and evidence quality, which relates to the data/findings emanating from the research process. According to Lohr, "The level of confidence one might have in evidence turns on the underlying robustness of the research and the analysis done to synthesize that research."*
Several key elements identified in previous research studies were reviewed in selecting the quality components for analysis presented here.
The results of the quality component analysis must be considered in conjunction with context-specific substantive components that relate to the disease area, the type of registry, and the purpose of the registry. (See Table 1.) For example, a disease-specific registry that has been designed to look at natural history should not be deemed low quality simply because it is not large enough to detect rare treatment effects.
For research, the quality domains are planning; design; data elements and data sources; and ethics, privacy, and governance. For evidence, the quality domains are described separately for registry participants; data elements and data sources; data quality assurance; analysis; and reporting.
Table 2 shows the basic elements of good registry practice for research, and Table 3 shows additional practices that have the potential to enhance scientific rigor, and thus the validity and reliability of information resulting from registries. Similarly, Table 4 shows the basic elements of good registry practice for evidence, and Table 5 shows additional practices that may enhance the evidence quality. It is important to weigh efforts taken to promote the accuracy and completeness of evidence in balance with the public health urgency of a problem, the types of interventions that are available, and the risks to public health from coming to a wrong conclusion. These lists of components are most likely incomplete, but the level of detail provided should be useful for high-level quality distinctions.
Most importantly, the basic elements of good practice, as well as the potential enhancements to good practice, depend to a great extent on the resources and budget available to support registry-based research.
* Lohr KN. Rating the strength of evidence: relevance for quality improvement programs. Int J. Qual Health Care 2004;16(1):9-18.
Table 1: Overview of Registry Purposes
- Determining clinical effectiveness, cost effectiveness, or comparative effectiveness of a test or treatment, including evaluating the acceptability of drugs, devices, or procedures for reimbursement.
- Measuring or monitoring safety and harm of specific products and treatments, including comparative evaluation of safety and effectiveness.
- Measuring or improving quality of care, including conducting programs to measure and/or improve the practice of medicine and/or public health.
- Assessing natural history, including estimating the magnitude of a problem; determining the underlying incidence or prevalence rate; examining trends of disease over time, conducting surveillance; assessing service delivery and identifying groups at high risk; documenting the types of patients served by a health provider; and describing and estimating survival.
|Data elements and data sources||
|Ethics, privacy, and governance||
|Data elements and data sources||
|Ethics, privacy, and governance||
|Data elements and data sources||
|Data quality assurance||
|Data elements and data sources||
|Data quality assurance||