Skip Navigation
Department of Health and Human Services

Back to "About This Webcast"

New and Improved Registries for Evaluating Patient Outcomes and HIT

Webcast Transcript, September 27, 2010
Elise Berliner, Ph.D., Moderator

Title Slide

DR. BERLINER: Good morning everyone. Welcome to New and Improved Registries for Evaluating Patient Outcomes and HIT. This is the release of the Second Edition of our registries handbook.

Slide 2: Agenda

My name is Elise Berliner, and I'm the task force officer for this project. We've been working on this update for a number of years now, and we're just so proud of it. Our senior editors are Richard Gliklich and Nancy Dreyer from Outcome, which is one of the DEcIDE Centers for the Effective Health Care Program. They are going to talk about the major changes and new sections. And, then at the end, we will have an open discussion on the Third Edition. We hope the Third Edition will be even larger than the Second Edition, and we want to hear your suggestions about what it should include.

Slide 3: Background

The First Edition was published in 2007. It has been widely used as a reference for designing, operating, and evaluating patient registries, which we're so happy about. I forgot to acknowledge, Michelle Leavy, the managing editor of the Handbook for her contributions. Michelle did a literature search to see how many citations there were for the Registry Handbook and she found over 60 citations in the literature and significant government publications such as the Federal Register and the FCC Report to the President. So we are happy that this document is getting used.

Slide 4: Purpose of the Second Edition

So why did we do a Second Edition? As registries continue to evolve, many new methodological and practical issues have arisen. We wanted to update and expand the First Edition of the User's Guide with new information gathered from recent publications and reported experiences encountered by researchers and other professionals who utilize registries for research. And, we wanted to expound on selected topics in the original guide and add new topics that deserve further in-depth discussion.

Slide 5: Process of Creating the Guide

The topics were identified based on public comments received for the First Edition. For the new sections, we gathered author and reviewer teams with similar backgrounds to the First Edition; all of the chapters include authors from industry, government, physician groups, and patient representatives. We tried to get a balance of stakeholders in order to have everyone's perspective represented. We posted the new sections for public comment and revision, and then we posted the whole draft of the Second Edition for another round of public comment. For the original chapters, the original authors and reviewers were invited to participate. Additions were made for new topic areas when necessary, and you'll hear about that from Rich and Nancy. We also had an open call for case examples. One of the great things about this handbook is that throughout we have cases where people who have developed real registries tell us about problems they've had and how they've solved them. We think is useful for people as they face their own problems, they can see how other people have solved those problems.

Slide #5: Second Edition Collaborators

So our Second Edition. We had 55 contributors from industry, academia, health plans, physician societies, and government. We had 49 invited peer reviewers plus we invited anyone to comment during the public review process. We had 38 case studies illustrating challenges and solutions. There are 20 new examples. Rich and Nancy were the editors and Michelle Leavy was our managing editor.

Slide 7: Second Edition: Table of Content

Here is the table of contents and in yellow (on this slide) are the major new sections. Every chapter was updated as needed. In Chapter 3, we added a discussion of planning for the end of a patient registry. In Chapter 4, we added a whole new section on the use of registries and product safety assessment. In Chapter 7, we looked at linking registry data to other sources of data and to other technical and legal considerations. And, in Chapter 11 we looked at interfacing registries with electronic health records. All of these are very timely topics.

With that, I will turn it over to Rich and Nancy.

Slide 8: Registries for Evaluating Patient Outcomes: A User’s Guide Second Edition

DR. GLIKLICH: Thank you, Elise. Good morning. It is both an honor to be here and a great feeling to be talking about this second edition, which you can see is a bit thicker than the first one. As Dr. Berliner said, this is the work of a large number of collaborators and hopefully a very valuable contribution in the field.

Slide 9: Review of Major Changes
This morning we're going to primarily focus on how the Second Edition builds on and expands the First Edition with a particular emphasis on the four new chapters or chapter parts and case examples.

As many of you know, the User's Guide is divided into three sections: creating, operating and evaluating registries. The first two sections provide basic information on the key areas of registry development and operations, which highlight the spectrum of practices in each of those areas and their potential strengths and weaknesses.

First, I should say that every part of every chapter was reviewed and there are many updates and changes; however, this morning I'll just review the key changes to the original chapters, discuss the new case studies and present a more expanded review of two of the new chapters. I'll cover linking registry data and interfacing registries with EHRs and my colleague, co-editor Dr. Dreyer, will review the other two major additions as well as the changes to the last chapter which is on evaluating registries.

Slide 10: Updates to 1st Edition Chapters

So, let’s go through the key chapters that had significant changes. In Chapter 2, which covers planning there are several key steps in planning a patient registry, including articulating its purpose, determining whether it is an appropriate means of addressing the research question, identifying stakeholders, defining the scope, the target population, assessing feasibility, securing funding and so on. To this chapter, we have now added a more significant discussion of public/private partnerships.

Chapter 3 also had significant changes. Chapter 3 covers registry design. A patient registry should be designed with respect to its major purpose or purposes with the understanding that different levels of rigor may be required for registries designed to address focused analytical questions versus registries that are more descriptive in nature. As you know, design and analysis go hand in hand and what we tried to do in these revisions is to align this chapter much more closely with the analysis and interpretation chapter, which is now Chapter 13.

A third chapter which changed quite a bit is Chapter 5. This is on data elements. The selection of data elements requires balancing such factors as their importance for the integrity of the registry and the analysis of the primary outcomes, as well as things like reliability, burden, risk of patient identification, and so on. What we did here was further stress data standards. We explained what data standards are. We pointed the reader to a lot more data standards because these will have direct consequences on our ability to aggregate and link registries in the future.

Slide 11: Updates to 1st Edition Chapters

Another chapter which had significant additions is Chapter 8. Chapter 8 focuses on registry ethics. This is data ownership and privacy and there are obviously critical ethical and legal issues that arise in collecting and using data for patient registries. In the First Edition, we had a pretty strong review of HIPAA and the Common Rule. In this edition, we expanded the review to include the Patient Safety and Quality Improvement Act of 2005, which covers patient safety organizations, the Genetic Information Nondiscrimination Act, which touches a little bit on biosamples, and the HITECH Act. In addition to adding a chapter on using patient registries for product safety assessment, we felt that the original chapter on adverse event detection required some updating since risk evaluation and mitigation strategies or REMs did not exist at the time the first Handbook was published. So we added sections on that.

Slide 12: Case Examples (New Examples Highlighted)

So I've listed on the next three slides the case examples. We added another 20 case examples to this edition. Many of the examples touch on the new chapters such as linking registry data or electronic health record use, but many examples were added to the older chapters such as designing a registry for Health Technology Assessment, using a collaborative approach to plan and implement the registry.

Slide 13: Case Examples (New Examples Highlighted)

And these are some more of the case examples that have been added. The case examples that have been added are in that brownish/reddish color and the original ones are in the black typeface.

Slide 14: Case Examples (New Examples Highlighted)

So just one more comment on the case examples. The purpose of their inclusion just as in the First Edition is solely to illustrate a point in the text. We don't endorse any of these registries for their quality. It's simply to use them as an example.

Slide 15: New Chapter: Linking Registry Data

So let me talk briefly about two chapters. The first chapter is linking registry data. Registry data may be linked to other data sources, for example, administrative data sources or other registries that examine questions that cannot be addressed using the registry data alone. For these projects, we need to determine what the risk of identifying patients becomes when we combine data, and what the legal and ethical requirements are in performing the linkage.

Slide 16: Linking Registry Data: Overview

Basically, the chapter is divided into two equally weighted and important sets of questions that must be addressed in the data linkage process. First, what is a feasible technical approach to linking the data; and second, is linkage legally feasible under the permissions, terms, and conditions that apply to the original formation or collection of the dataset?

Slide 17: Table 10: Technical Planning Questions

This table, which is from the chapter outlines a number of questions researchers need to ask to address the central issues in data linkage. There are many statistical techniques for linking records: deterministic matching, probabilistic matching, and so on. The choice of the technique should be guided by the types of data available, and linkage projects need to be managed by the types of data that are available, for example, records that exist only in one database, variations in units of measure, the types of things that you can imagine.

In addition, it is important to understand that linkage of de-identified data may result in accidental reidentification and risks of reidentification vary depending on the variables used and should be managed with guidance from legal and statistical experts to minimize risk and ensure compliance with HIPAA as well as the Common Rule and state statutes and so on.

Slide 18: Table 9: Legal Planning Questions

So these additional tables in that chapter walk through questions that someone who is undergoing planning a linkage should be thinking about, what's the purpose of the data linkage, what are the conditions in which the original data was collected, are there biospecimens present, what kind of data do you have, who is conducting the linkage…

Slide 19: Table 9: Legal Planning Questions (Continued)
…other laws or policies that might apply, future uses of the aggregated data, have those future uses being considered in the consenting process, and so on.

Slide 20: Linking Registry Data: Case Example

As in all the new chapters, we have added new case studies that emphasize some relevant point or feature. Here is an example of one of the case studies which is the linkage between the American College of Cardiology CathPCI Registry which lacks long-term follow-up data, and Medicare data which lacks detailed procedural information. So you can see that there is a complementarity between those two. By combining them, the registry developers in this case and the linkage experts were able to better understand outcomes for diagnostic and interventional cardiac catheterization.

Slide 21: New Chapter: Interfacing Registries with Electronic Health Records (EHRs)

The other chapter I'm going to talk about is achieving interoperability between electronic health records and registries, which will become increasingly important as both EHR adoption increases, particularly under ARRA, and patient registries use increases now that the Handbook is out and that the purposes for both are growing. In this chapter, we take the viewpoint that such interoperability should be based on open standards that enable any willing provider to interface with any applicable registry without requiring customization or permission from the EHR vendor. The chapter explores how we might achieve this in the near term.

Slide 22: Interfacing Registries & EHRs: Overview

This chapter has four main topics, the role of EHRs and patient registries in health care, the vision of EHR interoperability, some of the challenges in interoperability, and then what are partial solutions or potential solutions that can get us at least most of the way there.

Slide 23: Interfacing Registries & EHRs: Definitions

One of the things that we do early on in this chapter is distinguish clearly between an EHR and a registry because there is some confusion in the literature and as people talk about these things. What is the definition of an EHR? It is a record of health-related information on an individual. An EHR is individually-focused whereas a patient registry -- this is the definition from the Registry Handbook -- is an organized system that evaluates specified outcomes for a population. So, one is individually focused, one is population focused.

Slide 24: Interoperability Challenges

Interoperability for health information systems requires accurate and consistent data exchange, and then the use of the information that has been exchanged. There are two different realms to that, syntactic interoperability, the ability to exchange data, and semantic interoperability which is the ability to understand the exchange data. Those are the core constructs of interoperability. Both must be present in order for EHRs and registries to share data successfully. There are other important issues such as managing patient identifiers and so on. We touch on those; but not in as much detail.

The summary findings of the chapter are that getting to full complete, seamless interoperability is unlikely to be achieved for many years.

Slide 25: Partial and Potential Solutions

But there are partial solutions today that are really quite good and get us most if not very close to all of the way there.
Specifically we point to several open standard components. An open standard is one that is freely available to any system vendor who wants to use it. Many of these open standards are well tested and can provide very real and substantial benefits today. We call this functional interoperability in the chapter.

Slide 26: Building-block Approach: Retrieve Form for Data Capture/HITSP-TP50

We describe examples of groups of those standards, such as what's called the retrieve form for data capture, HITSP-TP50 which is a collection of open standards that provides enough of a framework to enable us to have real interoperability today. This diagram is showing an EHR linked to a registry or multiple registries. The clinicians sit down with the EHR to enter information on the patient. Through open standards the EHR recognizes that the patient is already enrolled in a registry or meets the criteria for a new registry. It signals to the clinician that this is the case, assuming the other permissions such as informed consent are in place, and the clinician can opt to move forward. The registry form itself actually emerges within the EHR. They don't go anywhere else, and it gets partially filled out or completely filled out by whatever data is already in the EHR. But because we don't have exact matching of data standards between registries and EHRs yet, we don't have outcomes defined for everything yet, there are always a few data elements that still need to be completed. The clinician can complete that and hit submit and the data goes on to the registry as well as into the EHR. In the case examples, I'll talk about how the practical implementation of this actually drives down the time it takes to participate in a registry with a patient -- from 15-20 minutes to go between the two systems, to a few seconds to a few minutes to do it. So it has very great value potentially for enabling this field to grow.

Slide 27: Summary

The chapter concludes that EHR registry interoperability will be increasingly important as both are growing. While a complete solution does not yet exist, there are enough open standard building blocks to significantly improve work flow and reduce duplication of effort, and these should generally be adopted. It will be helpful to all.

Slide 28: Interfacing Registries & EHRs: Case Examples

Okay, so three of the case examples are presented in the Handbook on this HITSP-TP50 model. In one case, the registry focused on the effectiveness of pain management. It was made interoperable with a commercial EHR with a dramatic reduction in work by the clinicians. In a second case called ASTER, the same model was used to facilitate adverse event reporting to the FDA. In this case, the form is triggered in the EHR by a potential adverse event. It emerges, is completed, and then transferred to the FDA via the National Health Information Network or NHIN. In a third case, the EHR is made interoperable with a quality registry from a professional society, the American College of Rheumatology as well as simultaneously meeting the needs for the Center for Medicare and Medicaid Services physician quality reporting initiatives so that a clinician entering a patient in both registries simultaneously through the same interface can actually meet requirements for both programs at the same time.

I'd like to now turn this over to my colleague and co-editor, Dr. Nancy Dreyer, and I look forward to your questions later in this session.

Slide 28: Review of Major Changes

DR. DREYER: Thanks, Rich. Before I start, I wanted to thank all of you who are in the room and watching the Webcast, who were contributors or reviewers. This Handbook represents a contribution of a lot of people. It is intended as a very practical guide. Without your help in bringing together that vast amount of experience, it wouldn't be very useful. So a heartfelt, “thank you,” on behalf of all the editors to all the contributors.

My assignment today is to talk to you about the rest of the new sections in the book, specifically, planning for the end of a registry and using a registry for product safety assessment, with some case examples. I will then take you through some of the changes to the last section which is about how to evaluate a patient registry.

Slide 30: New Section: When to End a Registry

The section on how to end a registry is within a chapter. It is a sidebar, but it is a pretty important one because if any of you have ever been on the funding side or the leading side of a registry, it seems like a great idea when you start it. You make a commitment to doing something important for public health or for whatever reason, and then it becomes quite a challenge to figure out when you have enough information to stop. For example, what is your obligation to the audience? So this section addresses some of those issues.

Slide 31: Stopping Decisions and Registry Goals

When you are planning to stop a registry, it helps when you started the registry to have put forth some measurable goals so you understand when you have achieved your purpose. Ideally, they would be codified in the protocol and you could use them as a guide to periodically assess whether you've achieved those goals or not.

On the other side of the coin, it's also important to use those goals to see if you are not making them. So a registry that fails to meet measurable goals should be considered as a candidate to be stopped.

There are also other reasons that people sometimes stop a registry. They have to do with funding, that the purpose is no longer relevant, or that the data quality is really insufficient for the goals of the registry.

Slide 32: What Happens When a Registry Stops?

Stopping is not a clear simple point. Stopping could mean no longer accruing new patients and continuing to follow those already in the registry, or it could mean stopping recruitment and folding everything down. So, regardless of how you're stopping, you need to think through exactly what would be stopped and what the processes would be.

Slide 33: What Happens When a Registry Stops?

For example, sometimes there may be a multi-sponsored registry and one sponsor wants to back out and the other sponsors are willing to take up the slack or to take full possession of the ownership and to continue to run it. In that case, it's important to go through the issues of who owns the data, confidentiality, and all of the ethical obligations attendant to making a transfer. But it's not impossible and it is often a very good way to transition a registry.

A correlated or related idea is, what do you do with the data? So if you' are folding down a registry, it is often desirable to think through issues about whether the data should be preserved and, if so, how. Sometimes data that were collected for one purpose can serve other purposes that you hadn't thought of at the time. Sometimes they are of historical importance, sometimes they are just useful for descriptive information, or for generating hypotheses. So it is important to think through what might be the potential uses whether you in fact could use the data and if so, where and how it should be stored.

Slide 34: Ending a Registry: Case Example

Here’s an example. One of the 38 case examples is about bupropion, a drug marketed as Wellbutrin and Zyban and used for depression and smoking cessation. It is a widely used drug. In this example, the manufacturer created a registry to look at teratogenic effects. This was a very large, very long-term project. It went more than ten years. They accrued data on 1,500 babies who had been born to mothers who used this product during their pregnancy. This case study is about a teratogenic signal that emerged, and the question was, “What do you do?” This was a registry that only had treated or exposed babies. You couldn't really evaluate it much further in the registry, but you could characterize it. So, they went to an external database, and in this case it was the health insurance claims database. They put together a sizable cohort, also over ten years, with comparators and looked for the presence of this teratogenic effect which was evident at birth. Based on that external validation effort, the end comparison effort, they were able to see that the signal was in fact consistent with background noise. Based on that, an advisory committee advised that the registry had served its purpose. So, this is an example of a registry that I don't think had a clear endpoint, but it generated a signal. The signal was evaluated and they decided that this was enough information.

Slide 35: New Chapter: Use of Registries for Product Safety Assessments

Another important chapter deals with registries for product safety assessments. This is becoming an increasingly important tool. When new products come to market, they have been extensively evaluated, but generally in optimal populations and they are not inclusive of the full range of patients who would use a product. So, registries are becoming particularly useful for understanding subpopulations that wouldn't have been in a clinical trial, either because of their age or their fragility or co-morbidities or other factors. So, we devoted a whole chapter to this, particularly how the data should be looked at for monitoring adverse events, how to figure out what the expectations should be so you can gauge what your observation is, what you should expect and how often you should look at it, and then what your legal and ethical responsibilities are for reporting, including informing physicians.

Slide 36: Use of Registries for Product Safety Assessments: Overview

This chapter is divided into four sections: registries that are specifically designed for safety assessments; registries that weren't designed for safety assessments but can provide important information; signal detection; and requirements for reporting. These are all hot topics. I think you'll find this chapter useful if you deal with this work.

Slide 37: Registries Designed for Safety Assessments

The utility of this chapter is that it talks about the challenges of looking at large and diverse populations over extended times, trying to parse out the effects of multiple treatments that people are using, and those types of issues. We deal a little bit about the study size questions, for example, when you are trying to detect safety events, size matters. So there is an importance to understanding when you have enough information to confidently rule out a problem as well as to confirm one.

We also talk about some of the challenges in recruiting the appropriate populations of interests, figuring out how well your recruitment has been and if you in fact do have a good representation of the target population. Then, we address some of the issues of trying to figure out timing and switching treatment and multiple therapies. I want to make it clear that this is not a methods handbook for how to do analysis of complex treatments. It is a practical guide to some of the issues, but it points you to other references. It is not everything you ever wanted to know, but it has a lot of information.

Slide 38: Registries Designed for Purposes Other than Safety

We also talk about the challenges of using a registry that was designed for purposes other than safety. Sometimes these are registries that were targeted to study the natural history of the disease, but there can be safety signals in them or safety data in them. So we talk about how you handle that.

Slide 39: Use of Registries for Product Safety Assessment: Case Example

We give an example of how registries like this are used. This example is from the British Society for Rheumatology Biologics Register. This important registry created by rheumatologists, has been used by the National Institute of Clinical Health and Excellence in the UK, a group that decides what goes on the national health formulary. It is an important decision-making body for the United Kingdom. They used this register to try to understand anti-TNF therapy; and this registry was used by decision-makers to allow a product or class of products to come on formulary.

Slide 40: Evaluating Registries

The last chapter I wanted to talk about is one that is widely cited. I am very interested in your comments on anything, but in this chapter in particular, because this is the chapter about evaluating registries. There has been some discussion about if it should it be a checklist. It's not, but I want to know if you think that that's important. What it does, is lay out two categories of quality: research quality and evidence quality. Research quality has to do with the process and evidence quality has to do with what you get.

For each of these -- research quality and evidence quality -- we have two levels of evaluation. One is a basic level that every registry should have, and the second category is potential enhancements that are really quite good, but they are not essential. So I'll take you through some of those issues.

Slide 41: Table 20: Research Quality – Basic Elements of Good Practice for Establishing and Operating Registries

This is the table that we give on the basic elements of good practice for research quality. I know the print is small, but we're talking about the basic elements of research design. It should have objectives. You should know who you are studying. You should think of finding clinically meaningful outcomes. Think about an efficient and reliable way to collect data, use validated scales when they exist. Think about the followup time you need to get enough information about what you're studying. We're not saying that you have to have enough follow-up for the ideal study, but we're saying you have to think through it and acknowledge what the desirable information is and what you can do. For example, if you are studying something rare and you think you need a million people, but you can only afford to study 10,000, you write that. Ideally we can't rule out a risk of one in a million, but we can tell you about the experiences in 10,000 people. And we talk about reporting and planning an analysis. So I think when you get a chance to look at this in a font size that you can read, you'll see that there should be no surprises here. This is basic good research practice.

Slide 42: Table 21: Research Quality – Potential Enhancements to Good Practice for Establishing and Operating Registries

This table shows potential enhancements. Here we talk about the difference between having an informal study plan and a formal study protocol and the value of having internal comparisons. One of the big challenges in registries is how to know if what you found is unusual or meaningful. There is some value to internal comparison groups, that is, information on nonexposed comparators that were selected at the same point in time. That is not always the Holy Grail, but certainly something to think about.

We also discuss the value of a formal statistical evaluation of study size requirements and the potential added value you could get from linkage with external sources. For example, if you are looking for serious side effects, you might want to collect enough data elements that allow you to link, at some point in time, to the National Death Index. That is a tremendous resource that will tell you not only who died but also their cause of death. Those are potential enhancements to a good research design.

Slide 42: Evidence Quality – Indicators of Good Evidence Quality for Registries

Now, remember, research quality refers to how good your processes were; evidence quality refers to how it all worked out. This table has been reorganized from the last edition. The major changes are that we reorganized the table in a more logical fashion, for example, here we talk about external validity and internal validity.

Slide 42: Evidence Quality – Indicators