Abstract
Recently, health systems internationally have begun to use cost-effectiveness research as formal inputs into decisions about which interventions and programmes should be funded from collective resources. This process has raised some important methodological questions for this area of research. This paper considers one set of issues related to the synthesis of effectiveness evidence for use in decision-analytic cost-effectiveness (CE) models, namely the need for the synthesis of all sources of available evidence, although these may not ‘fit neatly’ into a CE model.
Commonly encountered problems include the absence of head-to-head trial evidence comparing all options under comparison, the presence of multiple endpoints from trials and different follow-up periods. Full evidence synthesis for CE analysis also needs to consider treatment effects between patient subpopulations and the use of nonrandomised evidence.
Bayesian statistical methods represent a valuable set of analytical tools to utilise indirect evidence and can make a powerful contribution to the decision-analytic approach to CE analysis. This paper provides a worked example and a general overview of these methods with particular emphasis on their use in economic evaluation.
Similar content being viewed by others
References
Hjelmgren J, Berggren F, Andersson F. Health economic guidelines: similarities, differences and some implications. Value Health 2001; 4: 225–50
Sullivan S, Lyles A, Luce B, et al. AMCP guidance for submission of clinical and economic evaluation data to support formulary listing in US health plans and pharmacy benefits management organisations. J Manag Care Pharm 2001; 7: 272–82
National Institute for Clinical Excellence (NICE). Guide to the methods of technology appraisal. London: NICE, 2004
National Institute for Clinical Excellence (NICE). Guide to technology appraisal process. London: NICE, 2004
Drummond MF, Davies L. Economic analysis alongside clinical trials. Int J Technol Assess Health Care 1991; 7: 561–73
Claxton K, Sculpher M, Drummond M. A rational framework for decision making by the National Institute for Clinical Excellence. Lancet 2002; 360: 711–5
Barton P, Jobanputra P, Wilson J, et al. The use of modelling to evaluate new drugs for patients with a chronic condition: the case of antibodies against tumour necrosis factor in rheumatoid arthritis. Health Technol Assess 2004; 8 (11): 1–91
Briggs AH, Goeree R, Blackhouse G, et al. Probabilistic analysis of cost-effectiveness models: choosing between treatment strategies for gastroesophageal reflux disease. Med Decis Making 2002; 22: 290–308
Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian approaches to clinical trials and health-care evaluation. Chichester: Wiley, 2004
Fryback DG, Chinnis JO, Ulviva JW. Bayesian cost-effectiveness analysis: an example using the GUSTO trial. Int J Technol Assess Health Care 2001; 17: 83–97
O’Hagan A, Luce B. A primer on Bayesian statistics in health economics and outcomes research. Bethesda (MD): Medtap International, 2003
Felli JC, Hazen G. A Bayesian approach to sensitivity analysis. Health Econ 1999; 8: 263–8
Sutton AJ, Abrams KR. Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res 2001; 10: 277–303
Cooper NJ, Sutton AJ, Abrams KR. Decision analytical economic modelling within a Bayesian framework: application to prophylactic antibiotics use for caesarean section. Stat Methods Med Res 2002; 11: 491–512
Cooper NJ, Sutton AJ, Abrams KR, et al. Comprehensive decision analytical modelling in economic evaluation: a Bayesian approach. Health Econ 2004; 13: 203–26
Parmigiani G. Modeling in medical decision making: a Bayesian approach. Chichester: Wiley, 2002
Spiegelhalter DJ, Thomas A, Best N, et al. WinBUGS user manual: version 1.4. Cambridge (UK): MRC Biostatistics Unit, 2001
Doubilet P, Begg CB, Weinstein MC, et al. Probabilistic sensitivity analysis using Monte Carlo simulation. Med Decis Making 1985; 5: 157–77
Critchfield GC, Willard KE, Connelly DP. Probabilistic analysis of decision trees using Monte Carlo simulation. Med Decis Making 1986; 6: 85–92
Bailey KR. Inter-study differences: how should they influence the interpretation and analysis of results. Stat Med 1987; 6: 351–60
Lambert PC, Sutton AJ, Abrams KR, et al. A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. J Clin Epidemiol 2002; 55: 86–94
Shadish WR, Haddock CK. Combining estimates of effect size. In: Cooper H, Hedges LV, editors. The handbook of research synthesis. New York: Russell Sage Foundation, 1994: 261–81
DerSimonian R, Laird N. Meta-analysis of clinical trials. Control Clin Trials 1986; 7: 177–88
Prevost TC, Abrams KR, Jones DR. Hierarchical models in generalised synthesis of evidence: an example based on studies of breast cancer screening. Stat Med 2000; 19: 3359–76
Mandelblatt JS, Fryback DG, Weinstein MC, et al. Assessing the effectiveness of health interventions. In: Gold MR, Siegel JE, Russell LB, et al., editors. Cost-effectiveness in health and medicine. New York: Oxford University Press, 1996: 135–75
van Valkengoed IGM, Morré SA, van den Brute AJC, et al. Over-estimation of complication rates of chlamydia trachomatis screening programmes: implications for cost-effectiveness analyses. Int J Epidemiol 2004; 33: 416–25
Eddy DM, Hasselblad V, Shachter R. Analysis of breast cancer screening: seven controlled studies. In: Eddy DM, Hasselblad V, Shachter R, editors. Meta-analysis by the confidence profile method. Boston: Academic Press, 1982: 223–9
Hasselblad V, McCrory DC. Meta-analytic tools for medical decision making: a practical guide. Med Decis Making 1995; 15: 81–96
Ades AE. A chain of evidence with mixed comparisons: models for multi-parameter evidence synthesis and consistency of evidence. Stat Med 2003; 22: 2995–3016
Ades AE, Cliffe S. Markov Chain Monte Carlo estimation of a multi-parameter decision model: consistency of evidence and the accurate assessment of uncertainty. Med Decis Making 2002; 22: 359–71
Bucher H, Hengstler P, Schindler C, et al. Percutaneous transluminal coronary angioplasty versus medical treatment for non-acute coronary heart disease: meta-analysis of randomised controlled trials. BMJ 2000; 321: 73–7
Yazdanpanah Y, Sissoko D, Egger M, et al. Clinical efficacy of antiretroviral combination therapy based on protease inhibitors or non-nucleoside analogue reverse transcriptase inhibitors: indirect comparison of controlled trials. BMJ 2004; 328: 249–53
Song F, Altman DG, Glenny A-M, et al. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. BMJ 2003; 326: 472–5
Antithrombotic Trialists’ Collaboration. Collaborative metaanalysis of randomised trials of antiplatelet therapy for prevention of death, myocardial infarction, and stroke in high risk patients. BMJ 2002; 324: 71–86
Chalmers I, Altman DG. Systematic reviews. London: BMJ Publishing Group, 1995
Hasselblad V. Meta-analysis of multi-treatment studies. Med Decis Making 1998; 18: 37–43
Wilby J, Kainth K, Hawkins N, et al. A rapid and systematic review of the clinical effectiveness, tolerability and cost effectiveness of newer drugs for epilepsy in adults. London: National Institute for Clinical Excellence, 2003
Bridle C, Palmer S, Bagnall A-M, et al. A rapid and systematic review and economic evaluation of the clinical and costeffectiveness of newer drugs for treatment of mania associated with bipolar affective disorder. Health Technol Assess 2004; 18 (19): 1–187
Higgins JPT, Whitehead J. Borrowing strength from external trials in meta-analysis. Stat Med 1996; 15: 2733–49
Eddy DM, Hasselblad V, Shachter R. Meta-analysis by the confidence profile method. London: Academic Press, 1992
Stinnett A, Mullahy J. Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analyses. Med Decis Making 1998; 18: S68–80
Palmer S, Sculpher M, Philips Z, et al. Management of non-STelevation acute coronary syndromes: how cost-effective are glycoprotein IIb/IIIa antagonists in the UK National Health Service? Int J Cardiol 2005 Apr 20; 100 (2): 229–40
Abrams KR, Sutton AJ, Cooper NJ, et al. Populating economic decision models using meta-analysis of heterogeneously reported studies augmented with expert beliefs. Developing Economic Evaluation Methods (DEEM) workshop; 2003 Sep 4-5; Bristol
Berkey CS, Anderson JJ, Hoaglin DC. Multiple-outcome metaanalysis of clinical trials. Stat Med 1996; 15: 537–57
Berkey CS, Hoaglin DC, Antczak-Bouckoms A, et al. Metaanalysis of multiple outcomes by regression with random effects. Stat Med 1998; 17: 2537–50
Nam I-S, Mengerson K, Garthwaite P. Multivariate meta-analysis. Stat Med 2003; 22: 2309–33
Prentice RK. Surrogate endpoints in clinical trials: definition and operational criteria. Stat Med 1998; 8: 431–40
Psaty BM, Weiss NS, Furberg CD, et al. Surrogate end points, health outcomes, and the drug-approval process for the treatment of risk factors for cardiovascular disease. JAMA 1999; 282: 786–90
Pahor M, Psaty BM, Alderman MH et al. Health outcomes associated with calcium antagonists compared with other firstline hypertensive therapies: a meta-analysis of randomised controlled trials. Lancet 2000; 356: 1949–54
Fleming TR, de Mets DL. Surrogate end-points in clinical trials: are we being misled? Ann Intern Med 1996; 125: 605–13
Daniels MJ, Hughes MD. Meta-analysis for the evaluation of potential surrogate markers. Stat Med 1997; 16: 1965–82
Gail MH, Pfeiffer R, van Houwelingen HC, et al. On metaanalytic assessment of surrogate outcomes. Biostatistics 2000; 1: 231–46
Molenberghs G, Burzykowski T, Alonso A, et al. A perspective on surrogate endpoints in controlled clinical trials. Stat Methods Med Res 2004; 13: 177–206
Byar DP. Assessing apparent treatment-covariate interactions in randomized clinical trials. Stat Med 1985; 4: 255–63
Manca A, Rice N, Sculpher MJ, et al. Assessing generalisability by location in trial-based cost-effectiveness analysis: the use of multilevel models. Health Econ 2005 May; 14 (5): 471–85
Dixon DO, Simon R. Bayesian subset analysis. Biometrics 1991; 47: 871–81
Dixon DO, Simon R. Bayesian subset analysis in a colorectal cancer clinical trial. Stat Med 1992; 11: 13–22
Sharp SJ, Thompson SG. Analysing the relationship between treatment effect and underlying risk in meta-analysis: comparison and development of approaches. Stat Med 2000; 19: 3251–74
Vandenbroucke JP. When are observational studies as credible as randomised trials? Lancet 2004; 363: 1728–31
Centre for Reviews and Dissemination. CRD report 4. Undertaking systematic reviews of research on effectiveness: CRD’s guidance for carrying out or commissioning reviews. 2nd ed. York: CRD, University of York, 2001
Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. JAMA 2000; 342: 1887–92
Benson BA, Hartz AJ. A comparison of observational studies and randomized controlled trials. N Engl J Med 2000; 342: 1878–86
Britton A, McKee M, Black N, et al. Choosing between randomised and non-randomised studies: a systematic review. Health Technol Assess 1998; 2 (13): i–iv, 1-124
Reeves BC, MacLehose RR, Harvey IM, et al. Comparisons of effect size estimates derived from randomised and nonrandomised studies. In: Black N, Brazier J, Fitzpatrick R, et al., editors. Health services research methods: a guide to best practice. London: BMJ Publishing Group, 1998: 73–85
DuMouchel W. Bayesian data mining in large frequency tables, with an application to the FDA Spontaneous Reporting System (with discussion). Am Stat 1999; 53: 177–202
Sutton A, Abrams K, Jones D. Generalised synthesis of evidence and the threat of publication bias: the example of electronic fetal heart rate monitoring (EFM). J Clin Epidemiol 2002; 55: 1013–24
Droitcour F, Silberman G, Chelimsky E. Cross design synthesis: a new form of meta-analysis for combining the results from randomised clinical trials and medical-practice databases. Int J Technol Assess Health Care 1993; 9: 440–9
Spiegelhalter DJ, Best NG. Bayesian approaches to multiple sources of evidence and uncertainty in complex cost-effectiveness modelling. Stat Med 2003; 22: 3687–709
Schulz KF, Chalmers I, Hayes RJ, et al. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408–12
Li Z, Begg CB. Random effects models for combining results from controlled and uncontrolled studies in meta-analysis. J Am Stat Assoc 1994; 89: 1523–7
Carlin B. Analysis of local decisions using hierarchical modeling, applied to home radon measurement and remediation: comment. Stat Sci 1999; 14: 328–30
Samsa GP, Renter RA, Parmigiani G, et al. Performing costeffectiveness analysis by integrating randomised trial data with a comprehensive decision model: application to treatment of ischemic stroke. J Clin Epidemiol 1999; 52: 259–71
Parmigiani G, Ancukiewicz M, Matchar D. Decision models in clinical recommendations development: the stroke prevention policy model. In: Berry DA, Stangl DK, editors. Bayesian biostatistics. New York: Marcel Dekker, 1996: 207–33
Craig BA, Fryback DG, Klein R, et al. A Bayesian approach to modelling the natural history of a chronic condition from observations with intervention. Stat Med 1999; 18: 1355–71
Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med 2004 Oct 30; 23 (20): 3105–24
Gilks WR, Richardson S, Spiegelhalter DJ. Markov chain Monte Carlo in practice. London: Chapman & Hall/CRC, 1996
Welton NJ, Ades AE. A model of toxoplasmosis incidence in the UK: evidence synthesis and consistency of evidence. Appl Stat 2005; 54: 385–404
Lambert PC, Sutton AJ, Burton PR, et al. How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS. Stat Med 2005 Aug 15; 24 (15): 2401–28
Whitehead A. Meta-analysis of controlled clinical trials. Chichester: Wiley, 2002
Acknowledgements
Tony Ades and Mark Sculpher receive funding from the UK Medical Research Council as part of the Health Services Research Collaboration. Mark Sculpher is also funded through a Public Health Career Scientist Award from the UK NHS Research and Development Programme.
The authors have no conflicts of interest.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
1. Connectedness in Mixed Treatment Comparison Evidence Networks
The statistical approach to synthesising mixed treatment comparison evidence requires that the treatment comparisons are connected. For example, in a structure AB, BC, AC, EF, FG, the EFG group of treatments are not connected to the ABC group. The existence of AG trials would connect the two groups. The methods we propose cannot be applied to disconnected groups without further assumptions, and these assumptions would inevitably mean that inferences about all the relative treatment effects between the two groups would be based on nonrandomised evidence.
2. WinBUGS Programmes and Data Listing
We show here the WinBUGS code for the problem, partly because it clarifies some of the results in table II and also because it throws light on the nature of the synthesis. The use of this package requires some basic familiarity with Bayesian theory and MCMC methods, as well as careful attention to certain technical issues such as convergence of the MCMC chains,[17,76] and sensitivity to the way vague prior distributions are specified.[78] The WinBUGS code uses a neater notation, in which the d AB , d AC , d AD notation is replaced by d[2], d[3] and d[4]. This allows us to write the entire core model efficiently, regardless of how many treatments there are. The full data listing and all the programmes are available from http://www.hsrc.ac.uk/Current_research/research_programmes/mpes.htm (figure A1).
A sample data listing, shown below, shows 4 of the 50 randomised trials, indexed i, giving numerators r[i], denominators n[i], treatments t[i] and study numbers s[i]. There are 24 studies. The vector b[i] indicates which treatment is the effective trial-specific ‘baseline’ treatment in that study.
3. WinBUGS Programmes for Mixed Treatment Comparison
The core fixed and random effect programmes are shown here. The latter assumes that the degree of between-trials variation in random effect models is the same for all the pair-wise comparisons. This can be relaxed,[75] although the current data set is clearly much too sparse to obtain meaningful estimates of six different variances (figure A2).
Technically, a three-arm trial comparing, for example, A, B and C provides estimates of δj AB and δj AC which are correlated. Strictly speaking, this correlation should be reflected in the analysis,[39,75,79] although the effect will probably be small unless the proportion of multi-arm studies or their size is large. WinBUGS code that takes account of these correlations is available at the website given above.
4. Absolute Effects, Ranking of Treatments and Pair-wise Odds Ratios
Most CE analysis models require an explicit baseline effect on the absolute scale. Here we derive an estimate based on the average log odds of smoking cessation in the ‘no treatment’ condition A in the 19 studies in which A was involved. Investigators could alternatively take the baseline from a cohort study, or a single trial considered to represent a contemporary relevant estimate. Given an absolute estimate for treatment A we can derive absolute estimates of all other treatments by adding the LORs on the log odds scale and converting back to the natural scale. On each MCMC cycle the four interventions can be ranked, and an indicator variable, best[k], takes the value one when treatment k is best and zero otherwise. The average of the sequence of zeros and ones, given in the posterior summary, estimates the probability that treatment k is best. A further line of code generates posterior distributions for all the 6 pair-wise odds ratios (figure A3).
Rights and permissions
About this article
Cite this article
Ades, A.E., Sculpher, M., Sutton, A. et al. Bayesian methods for evidence synthesis in cost-effectiveness analysis. Pharmacoeconomics 24, 1–19 (2006). https://doi.org/10.2165/00019053-200624010-00001
Published:
Issue Date:
DOI: https://doi.org/10.2165/00019053-200624010-00001