A comparison of the Innovation and Value Initiative’s Open-Source Value Project and the Institute for Clinical and Economic Review’s value assessment framework
By Xcenda
By: Charley Hallock, PharmD; Robin Tan, PhD
Updated December 18, 2020
With the rise in high-cost treatments and growing emphasis on value-based healthcare, value assessment frameworks (VAFs) have been increasing in prominence in the United States (US). These frameworks have been recently evolving to better reflect the input of patients and various other stakeholders. Two well-known VAFs, the Institute for Clinical and Economic Review (ICER) and Innovation and Value Initiative (IVI), have both recently refined their methods in this regard; however, there remain notable differences in terms of how thoroughly their analyses reflect multistakeholder perceptions of value.
Introduction
Over the past decade, we have seen the United States (US) healthcare system begin to shift from a volume-based to a value-based reimbursement structure. Defining value is challenging because it varies depending on the stakeholder (eg, a patient may define value differently than a payer). In the US market-based system, healthcare decisions are often made in a decentralized and disparate manner, so the idea of distilling a treatment’s “value” into a single number using traditional measures of cost-effectiveness may not accurately represent value equally among different stakeholders. Rewarding initiatives focused on “value-based” care is another matter of contention; again, largely due to differences in stakeholder perspectives.
The Innovation and Value Initiative (IVI) was founded in 2016 with a goal of creating transparent and patient-centered methods of value assessment. The intention of its Open-Source Value Project (OSVP) was to create an open-source, iterative, modeling platform with consensus from various stakeholders throughout the assessment process. A stakeholder advisory group comprised of representatives from various healthcare segments (eg, patient advocacy groups, payers, healthcare systems, employers, healthcare sciences firms) guides IVI’s choice of topics and offers strategic direction for each assessment.
While IVI is an emerging entity, the Institute for Clinical and Economic Review (ICER), founded in 2006, remains arguably the most influential value assessment framework (VAF) in the US healthcare system today. Amid pressure for increased transparency from multiple stakeholder groups, ICER updated its VAF methodology in January 2020 with several notable improvements, including the addition of a “Subgroups and Heterogeneity” section, 12-month report updates, and a 24-month real-world evidence (RWE) update. Of note, updates to ICER’s Patient Engagement Program are also underway, which include: 1) a dedicated chapter on patient experience in each ICER report, 2) outcome measures informed by patients, 3) a formal patient survey, 4) a patient representative expert reviewer, and 5) a dedicated patient representative on the policy roundtable. For a more in-depth look at the Patient Engagement Program, check out our Fall 2020 Health Technology Assessment Quarterly (HTAQ) article.
The changing tides of healthcare assessment models, with an increased focus on multistakeholder engagement and transparency, carry implications for all US healthcare stakeholders. The updates IVI and ICER are making not only reflect the shift to focus on patient-centered value-based care in the US, but they may also play a role in shaping what the healthcare system will look like going forward. This article focuses on a comparison of IVI and ICER methodologies using major depressive disorder (MDD) as a use case for illustrating the similarities and differences between the 2 frameworks.
IVI OSVP
IVI’s OSVP is an innovative approach to value assessment. The stated mission of the multistakeholder initiative is “to advance the science, practice, and use of value assessment in healthcare to make it more meaningful to those who receive, provide, and pay for care.” IVI’s goal is to develop innovative, relevant, and rigorous methods and translational research in order to: 1) capture the values and preferences of patients and healthcare plan enrollees, and 2) make the measures and assessments of value transparent, flexible, and usable to all US healthcare marketplace stakeholders for decision making.
The project is comprised of 4 components: 1) developer resources, 2) disease-specific models, 3) tailored assessments, and 4) stakeholder community building. OSVP models are freely available for download and custom use, including all model materials (eg, source code). To date, 2 models, rheumatoid arthritis (RA) and non-small cell lung cancer (NSCLC), have been developed, with a third model for MDD underway. Basic and advanced value models are available for both RA and NSCLC disease states; the basic model is intended for general audience users and the advanced model is intended for clinical or technical experts and provides complete control over all modeling. Responses from patient organizations and the modeling community have been positive overall; however, feedback from US decision makers has been limited thus far.
The process of developing the models (Figure 1) and the models themselves are described as a “hands-on laboratory” designed to engage stakeholders in constructive conversations to bring consensus around measuring value. Model development typically takes a few months followed by a 3- to 6-month public feedback phase, after which the tool refinement process begins. The final output is an open-source model and report (released approximately 3 months after tool refinement). Given that IVI is continuously improving their process, timelines may differ between assessments.
Figure 1. The OSVP process
Currently, IVI is in the process of evaluating treatments for MDD, which began in June 2020. The OSVP IVI-MDD model differs slightly from the previous 2 models (RA and NSCLC), as IVI engaged multistakeholder groups at the outset and throughout the conceptualization and development process. Previously, external stakeholders were not engaged until after the initial model release (RA/NSCLC), or only patient focus groups were contacted before the initial model release (NSCLC). The anticipated timeline of the OSVP IVI-MDD model is outlined below (Figure 2). Additional model details and outputs are listed in Table 2.
Figure 2. OSVP IVI-MDD estimated timeline
ICER VAF
Typically, ICER updates its VAF methodology every 2 to 3 years, encouraging stakeholder feedback throughout the process. Additional details on ICER’s 2020 VAF updates were included in the Spring 2020 edition of HTAQ. ICER specifically requested patient input on: 1) understanding the patient experience and values, 2) incorporating patient-generated evidence, and 3) integrating dimensions of value not captured by quality-adjusted life-years (QALYs).
In January 2020, ICER released its final 2020–2023 VAF. Table 1 lists key updates to ICER’s methodology. In tandem with the 2020–2023 VAF methodology update, ICER was working on its “Valuing a Cure” initiative, which was finalized and published via a white paper in November 2019. This initiative was the product of conversations with health technology assessment (HTA) bodies (eg, National Institute for Health and Care Excellence [NICE] and Canadian Agency for Drugs and Technologies in Health [CADTH]) and US stakeholders (eg, patient advocacy groups, payers, employers, and manufacturers) and sought to better evaluate potential curative treatments and high-impact single or short-term therapies (SST) that are typically accompanied by high price tags and many potential unknowns at time of launch, including
- Unrecoverable costs
- Other aspects of value for patients and the health system
- Time divergence between costs and benefits
- Concerns about affordability and fair sharing of savings created by preventing downstream costs of chronic treatment
Moving forward, these considerations will be incorporated into future ICER VAF methodology updates.
Table 1. Key updates Included in ICER’s 2020–2023 VAF
| Topic | Details of ICER’s Methods Update |
|---|---|
| Patient engagement | Launching a formal patient engagement program to increase patient involvement |
| Real-world evidence | Augmenting efforts to use RWE, including generation of RWE to complement existing data sources |
| Heterogeneity | Adding a section on heterogeneity and subgroups to highlight subpopulations in which a treatment may work best |
| Evidence-rating categories | Expanding evidence-ratings categories to capture high certainty of at least a comparative net health benefit |
| Perspective | Promoting a modified societal perspective by including a “co-base case” when the societal costs of care for any disease are large relative to the direct healthcare costs, and the impact of treatment on these costs is substantial |
| Cost-effectiveness threshold | Standardizing $50K−$200K per QALY and per evLYG thresholds across all reports |
| Shared savings scenarios | Including hypothetical shared savings scenario analyses when there are substantial potential cost offsets (ie, all high-impact SSTs and non-SSTs with relevant and substantial potential cost offsets of >$1M over a lifetime) |
| Model uncertainty | Adding a new section on “Uncertainties and Controversies” to broaden discussion of alternative model structures and assumptions |
| Potential other benefits and voting structure | Revised and expanding the voting structure to capture important potential other benefits and contextual considerations |
| Evidence report updates | Implementing a formal process for reassessment of any new evidence 12 months after the release of Final Evidence Reports |
| RWE updates | Implementing a pilot effort for a 24-month RWE update for select drugs approved under accelerated pathways |
Key: evLYG – equal value of life-years gained; ICER – Institute for Clinical and Economic Review; RWE – real-world evidence; QALY – quality-adjusted life-year; SST – single or short-term transformative therapy; VAF – value assessment framework.
ICER’s typical assessment process from topic
announcement to final report lasts approximately 8 months (Figure 3).
Figure 3. ICER review timeline
Comparison IVI vs ICER
IVI’s MDD model is underway and ICER’s MDD report follows the 2017−2019 methodology. The table below presents a side-by-side comparison of both methods to better understand the differences between these 2 VAFs (Table 2). This table also highlights updates included in ICER’s 2020−2023 methodology.
Table 2. Comparison of MDD approach: IVI vs ICER
| Characteristic | IVI | ICER | |
|---|---|---|---|
| IVI OSVP-MDD model (2020) | ICER MDD methodology (2019) | ICER VAF updates (2020) | |
| Types of Value Assessment | Alternatives to QALY MCDA Novel elements of value |
QALY LY |
QALY evLYG |
| Model sharing | Open source (includes coding) |
Proprietary | Proprietary (shared with manufacturers behind a paywall) |
| Final output | Modifiable model + Published report/description of model and process | Published Final Evidence Report | Published Final Evidence Report 12-month new evidence reassessment 24-month RWE reassessment |
| Output details | Open Source Value Project Topic definition Purpose - Value assessment - Evaluation of scientific uncertainty Iterative process Components Population Treatment strategies Model structure - Disease model - Adverse events - Utility and healthcare sector costs - Productivity Model outcomes - Health outcomes, adverse events, and costs - Value assessment Sources data and parameter estimation - Treatment effects for transition rates - Adverse events - Utilities - Healthcare sector costs - Productivity - Value of hope Simulation and uncertainty analysis |
Introduction Summary of coverage policies and clinical guidelines Comparative clinical effectiveness Long-term cost-effectiveness Potential other benefits and contextual considerations Value-based price benchmarks Potential budget impact Summary of the votes and considerations for policy |
RWE Heterogeneity section Evidence-rating categories Perspective (co-base case) Cost-effectiveness threshold ($50K–$200K per QALY and evLYG) Shared savings scenario Health-benefit price benchmark Uncertainties and Controversies section |
| Stakeholder engagement | Stakeholder Advisory Group comprised of diverse stakeholders (eg, patients, employers, payers, clinicians, researchers, model developers) established at project outset to define the scope, inputs and purpose of the model; open comment period on protocol; open comment period on initial model; Technical Expert Panel review of public comments | Open input period; input from patients and patient advocacy groups throughout scoping and evidence development process; public comment period for draft scoping document; manufacturer input on research protocol and model analysis plan; public comment period for draft evidence report; public meeting | Open input period; input from patients and patient advocacy groups throughout scoping and evidence development process; public comment period for draft scoping document; input from patient organizations to identify, use, and generate RWE; manufacturer input on research protocol and model analysis plan; public comment period for draft evidence report; public meeting; debriefing with patient groups |
Key: evLYG – equal value of life-years gained; ICER – Institute for Clinical and Economic Review; IVI – Innovation and Value Initiative; LY – life-year; MCDA – multiple-criteria decision analysis; MDD – major depressive disorder; Q – quarter; QALY – quality-adjusted life-year; RWE – real-world evidence; VAF – value assessment framework.
There are several key differences in IVI and ICER approaches. IVI’s value assessments include alternatives to the QALY (eg, multiple-criteria decision analysis), and novel elements of value (eg, insurance value and value of hope). IVI’s outputs are open source (including model source coding) and modifiable, whereas ICER provides underlying source code to manufacturers (not publicly available) behind a paywall during the report development process. While IVI’s ongoing MDD assessment does not currently state the stage of MDD or treatment options to be assessed, IVI’s approach includes the ability to evaluate pharmacologic and nonpharmacologic treatments for MDD and cost-effective treatment-sequence strategies.
Looking at these differences through the lens of the MDD assessments, the 2019 ICER MDD assessment specifically examined esketamine for treatment-resistant depression compared to ketamine, electroconvulsive therapy, transcranial magnetic stimulation, oral antidepressants, antipsychotic augmentation, and no treatment (other than background antidepressants). ICER concluded that there was moderate certainty that the addition of esketamine to a newly initiated antidepressant was comparable or better to a newly initiated antidepressant alone, but that there was insufficient evidence to judge the net health benefit of esketamine vs ketamine or other therapies. Results from ICER’s cost-effectiveness analysis found that at its current wholesale acquisition cost, esketamine plus a background antidepressant compared to a background antidepressant alone exceeded commonly accepted thresholds for cost-effectiveness ($198,000/QALY). ICER’s potential other benefits and contextual considerations section described other potential advantages to patients, caregivers, the public, and the delivery system that were not accounted for in the comparative clinical effectiveness evaluation; however, this section was not meaningfully incorporated into the value assessment. The 2019 Final Evidence Report was lacking evLYGs, which is an element that has been added to the new 2020 framework.
Discussion
Because there are diverse views among US stakeholders in terms of defining and approaching value, it is important that value assessments incorporate different perspectives. Historically, cost-effectiveness models have been generally proprietary, static, and developed from a single perspective, but this approach is improving. In examining IVI’s OSVP and ICER’s 2020−2023 VAF methodology updates, we see a microcosm of the evolving value narrative and importance of multistakeholder involvement in assessments of value.
Key differences between IVI and ICER include additional elements of patient-centricity, transparency, and flexibility of the assessment, model, and final report. Since inception, IVI’s approach has offered a transparent and adaptable model that allows users to alter inputs and provides a range of results based on different preferences. In contrast, ICER’s MDD model provides base-case model results from a health system perspective and a modified societal perspective, as well as sensitivity analyses that highlight the key variables affecting those results; ICER also describes additional perspectives from stakeholders in a section of its report but does not appear to incorporate them meaningfully in the assessment. Of note, however, ICER recently announced the release of its Interactive Modeler, a modifiable rendering of the ICER’s models used to produce a specific evidence report exclusively hosted on Xcenda’s FormularyDecisions platform. This has been viewed as a step in the right direction for improved transparency.
The ongoing pressure from US stakeholders to expand definitions of value and incorporate multiple perspectives in value assessments should result in better incorporation of the patient perspective, more comprehensive information for decision making, and improved healthcare policy. It will be important to monitor how these changes affect the decision-making process, including how future reimbursement policies will be linked to assessments of value.
The article should be referenced as follows:
Hallock C, Tan R. A comparison of the innovation and value initiative’s open-source value project and the Institute for Clinical and Economic Review’s value assessment framework. HTA Quarterly. Spring 2021. https://www.xcenda.com/insights/htaq-spring-2021-comparison-ivi-icer
Sources:
- Bright J. IVI-MDD value model: part of the open-source value project. Accessed November 4, 2020.
https://www.thevalueinitiative.org/ivi-mdd-value-model/ - Institute for Clinical and Economic Review. 2020–2023 Value Assessment Framework. January 31, 2020. Accessed November 6, 2020.
https://icer-review.org/wp-content/uploads/2019/05/ICER_2020_2023_VAF_013120-4.pdf - Atlas SJ, Agboola F, Fazioli K, et al. Esketamine for the treatment of treatment-resistant depression:
effectiveness and value; Final Evidence Report. Institute for Clinical and Economic Review. June 20, 2019.
Accessed November 6, 2020. https://icer.org/wp-content/uploads/2020/10/ICER_TRD_Final_Evidence_Report_062019.pdf - Institute for Clinical and Economic Review. Methods update: valuing a cure. November 2019.
Accessed November 6, 2020. https://icer-review.org/assessment/valuing-a-cure-2019/ - Incerti D, Curtis JR, Shafrin J, et al. A flexible open-source decision model for value assessment of biologic treatment for rheumatoid arthritis. PharmacoEconomics. 2019;37:829-843.
- Innovation and Value Initiative. Mission and principles. 2019a. Accessed November 4, 2020.
https://www.thevalueinitiative.org/mission-and-principles/ - Innovation and Value Initiative. Open-source value project: transforming the process of value assessment. 2019b. Accessed November 4, 2020. https://www.thevalueinitiative.org/open-source-value-project/
- Innovation and Value Initiative. NSCLC value model. 2019c. Accessed November 4, 2020.
https://www.thevalueinitiative.org/ivi-nsclc-value-model/ - Innovation and Value Initiative. RA value model. 2019d. Accessed November 4, 2020.
https://www.thevalueinitiative.org/ivi-ra-value-model/ - Kanaskar A, Tan R, Watts-James J. A tale of 2 frameworks: impact of ICER and NICE updates on future HTAs and market access. HTA Quarterly. 2020. Accessed November 9, 2020.
https://www.xcenda.com/insights/htaq-spring-2020-2-frameworks-icer-nice-hta-market-access - Loos A, Tan R. Incorporating the patient voice in value assessments: the evolution of ICER’s approach to patient engagement. HTA Quarterly. Fall 2020. https://www.xcenda.com/insights/htaq-fall-2020-patient-voice-value-assessments-icer
- National Health Council. ICER’s value assessment framework: an overview of the 2020 value framework. Accessed November 9, 2020. https://nationalhealthcouncil.org/icers-value-assessment-framework-an-overview-of-the-2020-value-framework/
