Skip Navigation
Department of Health and Human Services www.hhs.gov
 
 

Back to "About This Webcast"

Slide 1

Use of Analytic Hierarchy Process to elicit stakeholder preferences for prioritizing research

August 27, 2012

Jerry A. Krishnan, MD, PhD (jakris@uic.edu)
Professor of Medicine and Public Health
Associate Vice President for Population Health Sciences

on behalf of the CONCERT Investigators


Slide 2

Outline

  1. Chronic obstructive pulmonary disease (COPD) as a key health condition.
  2. Rating importance and simple ranks to establish priorities.
  3. Analytic hierarchy process to establish priorities.


Slide 3

Chronic Obstructive  Pulmonary Disease (COPD)

  • Key health condition in US
    • Most common lung disorder
      • 24,000,000 persons
    • 3rd leading cause of death
      • Deaths rising
      • Heart disease, Cancer, COPD, CVA, Accidents
    • 3rd leading cause of hospital readmissions
    • $49.9 billion / yr


Picture: NHLBI Chartbook called Public Health Strategic Framework for CPOD Prevention


Slide 4

Chronic Obstructive  Pulmonary Disease (COPD)

  • Key health condition in US
  • Model complex medical condition
    • Multiple co-morbid conditions
    • Multiple healthcare providers
    • Multiple healthcare settings


Pictures: A photograph of a man coughing, a photograph of a patient with a nurse and a photograph of an elderly patient. 


Slide 5

“Setting effectiveness and translational research priorities to improve COPD care”

Year 1 (Importance, simple ranks)

  • May 21-22, 2009
  • Hard Rock Cafe
  • San Diego, CA
  • Chronic COPD care
  • Care coordination in COPD


Year 2 (AHP)

  • May 20-21, 2010
  • New Orleans
  • Acute COPD care
  • Transitions in care in COPD

 
Slide 6

Who? What? When?

  • Stakeholders
    • Patient advocacy groups
      • COPD foundation
    • Funders of health care
      • CMS, Wellpoint
    • Quality
      • Joint Commission, AHQA
    • Professional societies
      • ATS, ACP, ACCP, AARC, AACVPR,SHM,AASM, CAEM,ACEP, ASPH
    • Research funders
      • NHLBI, AHRQ, NINR


Slide 7

Who? What? When?

  • Phases of stakeholder engagement (2 years)
  • Pre-conference TCs
  • Goals, procedures
  • Elicit topics
  • Provisional voting
  • In person meeting
  • Presentations by topic experts
  • Discussion of provisional votes
  • Final ranking
  • Post-conference
  • Review / comment on priorities
  • Submit for peer review
  •  Improved Community Problem Solving


Slide 8

Importance (1 (most) to 9 (least))

  Topic Median(IQR)

1.

A

2 (1 – 3)

2.

B

3 (1 - 3)

3.

C

3 (1 - 4)

4.

D

3 (2 - 4)

5.

E

3 (2 - 4)

6.

F

3 (3 - 4)

7.

G

3 (4 – 9)

8.

H

3 (5 - 10)

9.

I

4 (3 - 5)


Slide 9

Importance (1 (most) to 9 (least))

  Topic Median(IQR)

1.

A

2 (1 - 3)

2.

B

3 (1 - 3)

3.

C

3 (1 - 4)

4.

D

3 (2 - 4)

5.

E

3 (2 - 4)

6.

F

3 (3 - 4)

7.

G

3 (4 - 9)

8.

H

3 (5 - 10)

9.

I

4 (3 - 5)

  • Several topics identified
  • Preferences variable
  • All topics important to someone
    • Simple rating of importance does not provide separation
    • Rationale (criteria) for rating unclear


Slide 10

Simple ranks

  Topic Median(IQR)

1.

A

3 (2 – 5)

2.

B

3.5 (2 – 8)

3.

C

5 (4 – 8)

4.

D

6 (2  - 7)

5.

E

6 (3 – 8)

6.

F

6.5 (4 – 9)

7.

G

6.5 (5 – 10)

8.

H

7 (5 – 8)

9.

I

7 (6 – 8)


Slide 11

Simple ranks

  Topic Median(IQR)

1.

A

3 (2 – 5)

2.

B

3.5 (2 – 8)

3.

C

5 (4 – 8)

4.

D

6 (2  - 7)

5.

E

6 (3 – 8)

6.

F

6.5 (4 – 9)

7.

G

6.5 (5 – 10)

8.

H

7 (5 – 8)

9.

I

7 (6 – 8)

  • Several topics identified
    • Preferences variable
    • Simple ranks do not measure relative importance of topics
    • Rationale (criteria) for ranking unclear


Slide 12

MCDA methods: the Analytic Hierarchy Process

  • Rating explicitly linked to criteria
  • Normalized Priority: proportion of the total importance that is attributed to a particular decision alternative.


Picture: Flowchart of the analytic hierarchy process. The top object labeled Decision objective (treatment) is linked to objects below labeled Criterion 1, Criterion 2 and Criterion 3, each of which is linked to objects labeled Alternative 1 and Alternative 2.

Courtesy of MJ IJzerman


Slide 13

MCDA methods: the Analytic Hierarchy Process

  • Series of pairwise comparisons between alternatives (research topics) for  each criterion


Picture: Flowchart of the analytic hierarchy process The top object labeled Decision objective (research topic) is linked to objects labeled Criterion 1 (1/9 to 9x as important), Criterion 2 and Criterion 3. Beneath, there are four objects labeled Topic 1, Topic 2, Topic 3 and Topic 4.


Slide 14

MCDA methods: the Analytic Hierarchy Process

Picture: Flowchart of the analytic hierarchy process The top object labeled Decision objective (research topic) is linked to objects labeled Criterion 1 (1/9 to 9x as important), Criterion 2 and Criterion 3. Beneath, there are four objects, labeled Topic 1, Topic 2, Topic 3 and Topic 4. Criterion 1 is linked to Topic 1 and Topic 2.


Slide 15

MCDA methods: the Analytic Hierarchy Process

Picture: Flowchart of the analytic hierarchy process The top object labeled Decision objective (research topic) is linked to objects labeled Criterion 1 (1/9 to 9x as important), Criterion 2 and Criterion 3. Beneath, there are four objects, labeled  Topic 1, Topic 2, Topic 3 and Topic 4. Criterion 1 is linked to Topic 1 and Topic 4.


Slide 16

MCDA methods: the Analytic Hierarchy Process

Picture: Flowchart of the analytic hierarchy process The top object labeled Decision objective (research topic) is linked to objects labeled Criterion 1 (1/9 to 9x as important), Criterion 2 and Criterion 3. Beneath, there are four objects, labeled Topic 1, Topic 2, Topic 3 and Topic 4. Criterion 1 is linked to Topic 2 and Topic 3.


Slide 17

MCDA methods: the Analytic Hierarchy Process

Picture: Flowchart of the analytic hierarchy process The top object labeled Decision objective (research topic) is linked to objects labeled Criterion 1 (1/9 to 9x as important), Criterion 2 and Criterion 3. Beneath, there are four objects, labeled Topic 1, Topic 2, Topic 3 and Topic 4. Criterion 1 is linked to Topic 2 and Topic 3.


Slide 18

MCDA methods: the Analytic Hierarchy Process

Picture: Flowchart of the analytic hierarchy process The top object labeled Decision objective (research topic) is linked to objects labeled Criterion 1 (1/9 to 9x as important), Criterion 2 and Criterion 3. Beneath, there are four objects, labeled Topic 1, Topic 2, Topic 3 and Topic 4. Criterion 1 is linked to Topic 2 and Topic 4.


Slide 19

MCDA methods: the Analytic Hierarchy Process

Picture: Flowchart of the analytic hierarchy process The top object labeled Decision objective (research topic) is linked to objects labeled Criterion 1 (1/9 to 9x as important), Criterion 2 and Criterion 3. Beneath, there are four objects, labeled Topic 1, Topic 2, Topic 3 and Topic 4. Criterion 1 is linked to Topic 3 and Topic 4.


Slide 20

MCDA methods: the Analytic Hierarchy Process

  • 6 pairwise comparisons for 4 alternatives (topics) for 1 criterion.


Picture: Flowchart of the analytic hierarchy process The top object labeled Decision objective (research topic) is linked to objects labeled Criterion 1 (1/9 to 9x as important), Criterion 2 and Criterion 3. Beneath, there are four objects, labeled Topic 1, Topic 2, Topic 3 and Topic 4. Each object labeled with a Criterion is linked to Topic 1, Topic 2, Topic 3 and Topic 4.


Slide 21

MCDA methods: the Analytic Hierarchy Process

  • 18 pairwise comparisons for 4 alternatives (topics) for 3 criteria.  What about more topics, and more criteria?


Picture: Flowchart of the analytic hierarchy process The top object labeled Decision objective (research topic) is linked to objects labeled Criterion 1 (1/9 to 9x as important) through Criterion 7. Beneath, there are 9 objects, labeled 1 through 9. Each object labeled with a Criterion is linked to each numbered object.


Slide 22

Criteria used by stakeholders

  • Uncertainty about effectiveness
  1. Impact on patient centered outcomes in efficacy studies
  2. Quality of evidence in efficacy studies
  3. Variability in care in real world settings
  4. Societal cost
  5. Feasibility of effectiveness studies
  6. Results would inform care in diverse settings


Slide 23

MCDA methods: the Analytic Hierarchy Process

Picture: Flowchart of the analytic hierarchy process The top object labeled Decision objective (research topic) is linked to objects labeled Criterion 1 through Criterion 7. Beneath, there are 9 objects, labeled 1 through 9. Each object labeled with a Criterion is linked to each numbered object.


Slide 24

MCDA methods: the Analytic Hierarchy Process

Box overlay of the previous flowchart from slide 23:.

  • For each criterion, there would be [n(n - 1)]/2pairwise comparisons, where n is the number of research topics being comparedor 9 topics, [9 (9-1)] / 2 = 36 comparisons; for 9 topics, 7 criteria, 7 X 36 = 252 comparisons


Slide 25

Modified AHP, to triage topics: 1/9 to 9x as overall important

Topic 1 2 3 4 5 6 7 8 9

1

This box is filled in

               

2

1/9

This box is filled in

             

3

1/3

 

This box is filled in

           

4

9

   

This box is filled in

         

5

2

     

This box is filled in

       

6

1/2

       

This box is filled in

     

7

3

         

This box is filled in

   

8

1/5

           

This box is filled in

 

9

4

             

This box is filled in

 

Table


Slide 26

Modified AHP, to triage topics: 1/9 to 9x as overall important

Topic Normalized priority IQR

1

 

 

2

 

0.1 – 0.4

3

0.17               

0.1 – 0.2

4

0.14   

0.06 – 0.15

5

0.12

0.04 – 0.19

6

0.07               

0.03 – 0.08

7

0.04               

0.02 – 0.05   

8

0.02

0.01 – 0.04

9

0.02

0.01 – 0.04

Table


Slide 27

Reflections on AHP for setting CER priorities

  1. Quantifies relative priorities and can be used to link voting patterns to criteria
  2. Not practical when ‘large’ # topics, criteria
    • 9 topics, 7 criteria à 252 comparisons
    • 5 topics, 5 criteria à 50 comparisons
    • 3 topics, 3 criteria à  9 comparisons
  1. CONCERT’s experience
    • Use pragmatic version of AHP (or other approach) to triage topics and criteria
    • Fully deploy AHP on highest scoring topics and most important criteria
    • Given variation in preferences, collaborate with different sets of stakeholders on separate CER topics


Slide 28

Acknowledgements

David Au - UW/VA
Shannon Carson - UNC
Jerry Krishnan, Todd Lee - UIC
Ted Naureckas - U of C
Peter Lindenauer - Baystate / Tufts
Mary Ann McBurnie - KPCHR / DCC
Richard Mularski - KPCHR

COPD
Outcomes-based
Network for
Clinical
Effectiveness and
Research
Translation

Picture: image of a map of the United States with the locations of the researchers’ institutions marked