Introduction

Chronic obstructive pulmonary disease (COPD) is a chronic respiratory disease distinguished by airway limitation.1,2 COPD is estimated to rank fifth worldwide in disease burden,3 and it is estimated to increase in the coming decades as a consequence of increasing exposure to environmental risk factors and an ageing population.2,3 More than 5% of the population is affected with the disease and it is associated with high mortality and morbidity.4 Because of the high prevalence and chronicity, COPD patients rely heavily on medical interventions. For stable COPD, a wide variety of pharmacological treatments are used.3 Non-pharmacological and rehabilitation strategies are also commonly used to improve the functional capacity of COPD patients.3

There are numerous treatments available for COPD and it is important for clinicians to make a decision on their use on the basis of up-to-date clinical evidence.5 Meta-analysis (MA) of randomised trials is considered to be one of the best approaches for summarising high-quality evidence on the efficacy and safety of various therapeutic options.6 Although these MAs are published continually as trustworthy sources of evidence, little attention is paid to their methodological quality. Methodological flaws in the conduct of MAs could cause biased conclusions. For example, an incomprehensive literature search could lead to failure in locating unpublished data. This omission may cause changes in the direction of effect for a particular treatment.7,8 Other methodological shortcomings that can lead to biased results include the lack of independent audit during data extraction, inappropriate combination of heterogeneous study results, and failure to consider risk of bias among primary studies. As biased results from MAs can mislead clinical practice,9 it is important to assess the methodological quality of MAs on COPD treatment. Results will provide an overview of MA quality in the field of COPD, and accordingly suggestions for future methodological improvement can be made.

AMSTAR (Assessing the Methodological Quality of Systematic Reviews) is a freely accessible validated tool for assessing the methodological quality of MAs.10,11 The full version can be found in the AMSTAR official website.10 Validation studies9,11 have shown that AMSTAR is a reliable critical appraisal tool with good agreement, construct validity and feasibility. The AMSTAR is adopted by a number of research and health technology assessment groups, such as the Canadian Agency for Drugs and Technologies in Health and The Cochrane Effective Practice and Organization of Care Group.12

Using a cross-sectional study design, this study aimed to (1) describe the bibliographical characteristics of MAs on COPD therapies, (2) evaluate the methodological quality of MAs on COPD treatments by using AMSTAR and (3) examine the association between bibliographical characteristics and methodological quality.

Materials and Methods

Sampling of meta-analyses

We sampled MAs by searching the Cochrane Database of Systematic Reviews (CDSR) and Database of Abstracts of Reviews of Effect (DARE) using the keyword ‘COPD’. CDSR enabled us to locate Cochrane reviews, whereas DARE is an extensive, weekly updated systematic review database that indexes non-Cochrane MA. The use of these databases allowed us to assemble a representative sample of MAs for appraisal.

Inclusion and exclusion criteria

MAs on COPD treatments published between January 2000 and April 2013 were eligible. Narrative reviews, systematic reviews with no meta-analyses, network meta-analyses and systematic reviews on respiratory diseases other than COPD (e.g., asthma, emphysema, chronic bronchitis, bronchiectasis and so on) were excluded. Two authors independently screened the titles, abstracts and full texts of the retrieved literature to assess their eligibility. Discrepancies were discussed and solved by discussion and consensus among the authors.

Extraction of bibliographical characteristics and critical appraisal

Bibliographical characteristics were assessed with a pre-designed questionnaire. All 11 items of the AMSTAR instrument were used for appraising methodological quality.11 Judgements were given as either ‘yes’ or ‘no’ for items 1, 2 and 5–11. For items 3 and 4, an option from ‘yes’, ‘cannot answer or not reported’ or ‘no’ was chosen. No aggregate score was calculated, as the original version of AMSTAR did not provide any recommendation on using this approach10 and there is no empirical evidence to advise on the appropriate weighting for each item when calculating a total score.9 A detailed operational guide for AMSTAR can be found in Supplementary Material 1.

Data analysis

The associations between bibliographical characteristics and scoring on AMSTAR items were analysed using multivariate logistic regression or multi-nominal logistic regression depending on the scaling of the dependent variable. Hosmer and Lemeshow tests were performed to evaluate the model fitting for all logistic regressions. All statistical analyses were conducted with SPSS 18.0, with a two-tailed significance level of 0.01 to avoid multiple test bias.

Results

MA searching and selection

The search strategy identified a total of 137 systematic reviews (73 from CDSR and 64 from DARE). No duplicates were found. After assessing the titles, abstracts and full texts, a total of 79 MAs met our inclusion criteria and were included in this study (Figure 1). A full list of included MAs can be found in Supplementary Material 2.

Figure 1
figure 1

Sampling of meta-analyses on COPD treatment: flow chart.

Full size image

Bibliographical characteristics and methodological quality

The median year of publications was 2010 (n=44, 55.7%). There were more MAs published in the field from 2010 onwards. Fifty (63.3%) MAs had an impact factor of >5. Detailed results for bibliographical characteristics and methodological quality are shown in Tables 1 and 2.

Table 1 Bibliographical characteristics of 79 included meta-analyses on COPD treatments
Full size table
Table 2 Methodological quality of 79 included meta-analyses on COPD treatments
Full size table

Association between bibliographical characteristics and methodological quality: multivariate analyses

Table 3 illustrates the results from logistic regression analyses (full results can be found in Supplementary Material 3). A higher journal impact factor was significantly associated with favourable methodological scoring on the following aspects: an ‘a priori’ design provided in the MA; duplicate study selection and data extraction; and provision of study lists. MAs published more recently were more likely to have conducted an assessment of the scientific quality of included studies, as well as an evaluation of publication bias. MAs on non-pharmacological treatments performed better in considering the scientific quality of included studies when formulating conclusions, compared with their pharmacological counterparts. Finally, MAs on non-pharmacological treatments also fared marginally better in using appropriate meta-analytic methods for combining findings. P values of Hosmer–Lemeshow tests for all logistic regression analyses were >0.05.

Table 3 Association between publication characteristics and methodological quality of MAs on COPD treatments: multivariate analyses
Full size table

Discussion

Main findings

This study assessed the methodological quality of 79 MAs on COPD treatment published between 2000 and 2013. The proportion of MAs that fulfilled each individual AMSTAR domain was similar to that of other medical fields like urology, orthodontics and nursing.13–15 An overview of results from these studies is shown in Table 4. The majority of MAs were published after 2010, demonstrating an increasing popularity of systematic review methods among researchers in the field. The quality of more recent MAs in terms of assessing and documenting the scientific quality of the included studies was higher, reflecting progress in methodological sophistication. The frequency of updating was acceptable as >60% of the included MAs represented an update of a previous report. Higher impact factor is associated with clarification of an ‘a priori’ design, and this trend could be explained by the requirement of Cochrane MA in publishing a protocol. Other MA authors should consider registering their MA protocols on PROSPERO, which is an international online platform for prospective registration of systematic review protocols.16

Table 4 Methodological quality of meta-analyses in other health-care fields assessed by AMSTAR
Full size table

Interpretation of findings in relation to previously published work

Despite these progresses, the overall methodological quality of MAs on COPD treatment can only be considered disappointing. Only 17.7% of included MAs explicitly considered the scientific quality of primary studies when formulating their conclusions. As trials with high risk of bias are more likely to report larger effect sizes,17 readers of MAs on COPD treatments may consider reported conclusions more judiciously. Even when the trial’s risk of bias had been assessed, <40% of MAs had used the Cochrane risk of bias tool, a specialised instrument developed for appraising RCTs.18 In the future, proper use of Cochrane risk of bias tools should be promoted and explicit consideration on the risk of bias of primary studies and its impact on the treatment results should be made.

These improvement measures are especially important for MAs on pharmacological treatments as it appears that they perform less well compared with non-pharmacological MAs. Less than satisfactory performance in this aspect could be related to the influence of funders, as previous studies have clearly shown the relationship between industry funding and positive results from MAs.19–21 Another area of concern is the lack of reporting on conflict of interest among MA authors: in our results, only two MAs reported the funding sources for all included primary studies, as well as for the MA itself. More than 48% of the MA authors did not declare a conflict of interest. In addition, harmful effects of treatment were also under-reported in the sampled MAs. A total of 1,099 studies were summarised in these MAs, representing data from more than 800,000 patients. Despite the wealth of data reviewed, only 58% described the harmful effects of treatment. In the future, it is important for journal editors to encourage reporting in these aspects.

A substantial proportion of included MAs (64.6%) did not assess the likelihood of publication bias. As studies with positive results are published more often and more quickly compared with negative studies regardless of methodological quality,22,23 an overestimation of COPD treatment effect could be observed among the MAs.24 Treatment effect could be overestimated if publication bias existed even when the included individual trials have a low risk of bias.25 To reduce the likelihood of overestimating results, in the future, MAs should explicitly assess publication bias using proper methods,26 and appropriate adjustment on the meta-analytic results can be considered if publication bias is present.27

In addition to publication bias, language bias also increases the risk of overestimation as positive findings are more likely to be published in English language journals.28 In our MA sample, only 10% of the MAs published had searched non-English databases. Building an international, multi-lingual team for conducting MAs could reduce English language bias by widening the coverage on non-English databases. Currently, more than half of the corresponding authors are from North America and Europe, and the review team size is rather small, with a median of four people. Adding authors from non-Western countries could be a solution for reducing language bias.

Strengths and limitations of this study

One of the strengths of this study was that we identified all MAs indexed in CDSR and DARE. The CDSR included all Cochrane Reviews prepared by Cochrane Review Groups within the Cochrane Collaboration, whereas DARE covers the majority of indexed non-Cochrane MAs.29 One limitation of the present study is that the AMSTAR appraisal process was difficult to implement when the reporting quality was poor. This could be attributed to space restrictions in non-Cochrane journals, which limit detailed reporting.30 Authors and journal editors are encouraged to adhere to the PRISMA requirement, which includes all required reporting components for authors to publish a systematic review.31

Conclusions

Our findings demonstrate that the methodological quality of MAs on COPD treatment is disappointing. Areas that require improvement in the future include the consideration of scientific quality when formulating conclusions, reporting of conflict of interest, assessment of publication bias, and prevention of language bias. Also, the harmful effects of treatment were often under-reported. Clinicians should be judicious when applying the conclusions of MA results to patients, taking into account the potential bias resulting from methodological limitations. Journal editors and peer reviewers will have a strong role in ensuring continual improvement of MA quality by upholding the methodological and reporting standards recommended by AMSTAR and PRISMA.31