Accepted Feb 14, 2020; Epub ahead of print Feb 27, 2020

J Rehabil Med 2020; 52: jrm00031

Correspondence address: Antti Malmivaara, Centre for Health and Social Economics, National Institute for Health and Welfare, Mannerheimintie 166, 00270 Helsinki, Finland. E-mail: antti.malmivaara@thl.fi

In a systematic review it is important that all characteristics of randomized controlled trials are reported, so that clinicians can determine to which patients the results of a systematic review can be applied. This study assessed how comprehensively these characteristics of randomized controlled trials were recorded in the systematic reviews published in leading general medical journals. A search of the literature found a total of 115 systematic reviews. Of these, 71% were on pharmacological interventions, 35% were on other conservative treatments, 13% were on surgical interventions, and 0% were on rehabilitation interventions. None of the systematic reviews assessed how patients were selected to the study; 35% reported relevant clinical features; 25% comorbid conditions; and 21% patients’ behavioural factors. Functioning, environmental factors, inequity-related factors; how interventions were carried out; how well the patients were followed-up; and the adequacy of statistical analyses were reported in only 0–9% of the systematic reviews. In conclusion, the reporting of study characteristics in the systematic reviews does not make it possible to assess how similar the different studies had been, or to which patients these study findings could be generalized. In future, randomized controlled trials should be described better in systematic reviews. Further studies are needed on this subject.

Regarding the scientific literature, there are guidelines for how to comprehensively describe the essential PICO (patient, intervention, control intervention/comparator, outcome) elements of randomized controlled trials (RCTs) in systematic reviews (SRs) and meta-analyses, and how to assess the risk of bias and evaluate the generalizability of evidence of RCTs in SRs and meta-analyses (1–4). In addition to a description of characteristics of the main diagnosis, it is recommended that SRs also include comprehensive reporting of other patient characteristics, including equity-related factors, and those related to the healthcare system (1, 5, 6). Even more comprehensive documentation is essential in the assessment of studies on observational effectiveness, the benchmarking controlled trials (BCTs) (7). The benchmarking method (BM) used in BCTs comprises 5 main categories and several subcategories related to PICO and statistical issues (8–13). The conceptualization and operationalization of the BM has been based on the CONSORT (Consolidated Standards of Reporting Trials) statement for RCTs, and on scientific studies on methodology, as well as the relevance of items in experimental and observational effectiveness studies. The BM referred to in this study has been used previously for assessing the generalizability of evidence from RCTs published in the leading medical journals (14).

The aims of the current paper were to evaluate how comprehensively the relevant items for the assessment of the generalizability of findings from RCTs were recorded in SRs in the 4 leading general medical journals publishing SRs; and, based on the SRs, to evaluate the clinical homogeneity of the RCT studies, the justification for meta-analysis of the RCT data, and the generalizability of evidence from the SRs.

The BM was used to assess the degree of completeness of how the SRs documented reporting of their study object, the RCTs. The BM is based on 5 categories (selection, baseline characteristics, intervention factors, outcome assessments, and statistical issues) and several subcategories (7, 14, 15), which were modified for the assessment of how comprehensively the authors of the SRs had recorded the essential features of the RCTs (Table I). Considering an item as recorded in an SR was based on whether there was any information on that particular item. For example, regarding competence, it was assessed whether the competence of the staff had been rated and documented, whether any characterization was included in the SR, and it was rated as not-documented if there was no mention of competence.

PubMed was searched to find all SRs and meta-analyses of cluster randomized and individually randomized controlled trials aiming at the assessment of the effectiveness of treatment or rehabilitation interventions, which were published in the Annals of Internal Medicine, BMJ, JAMA (The Journal of the American Medical Association) and Lancet from 1 January 2016 to 28 February 2019.

The inclusion criterion was that “systematic review” was stated in the title or in the abstract of the paper, and when the full-text article was obtained it conformed with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement definition of a SR. The New England Journal of Medicine was not included, because it does not publish SRs. Reviews based on individual patient data (IPD) were included.

The exclusion criteria were: if there were fewer than 2 RCTs; if the review assessed the effectiveness of diagnostic or screening procedures; if the review assessed the effects of healthcare-system-related interventions (e.g. the effects of resources or the organization of care); umbrella reviews, i.e. SRs of SRs. The following key words were used: SR, meta-analysis, name of each journal, and time-frame from 1 January 2016 to 28 February 2019. PubMed’s advanced search tool was used. The search was repeated to ensure that all eligible papers were included. Both literature searches were performed in March 2019. No additional papers were found in the replicated PubMed search. Any paper with potential eligibility was examined according to the title and abstract, and for the potentially eligible ones the full-text papers were retrieved. Final decisions on eligibility were based on the full-text paper.

Descriptive information was extracted on the selection of patients, completeness and validity of the data for the baseline characteristics, interventions, outcomes, and statistical analysis. For patient characteristics, the numbers of subcategories recorded in the SRs were documented separately, e.g. for the behavioural factors, the percentages of SRs documenting 1, 2, 3, or 4 of the subcategories of exercise, smoking, alcohol use, and obesity were documented. Data extraction was checked twice (3 assessments in total) to ensure the accuracy of the data. The information was gathered both from the main texts and from all supplementary material provided alongside the article. Categorization of the type of intervention was made based on both the index and control interventions.

A total of 115 SRs and meta-analyses fulfilling the inclusion criterion were identified; 36 in Annals of Internal Medicine, 32 in BMJ, 26 in JAMA, and 21 in Lancet. The included and excluded RCTs are shown in Table SI¹. Considering all the comparisons, pharmacological interventions were included in 71% of the SRs, other conservative treatments in 35%, and surgical interventions in 13%. There were no SRs on rehabilitation interventions. A meta-analysis was included in all of the SRs published in the BMJ and the Lancet, and in 75% and 92% of the SRs published in Annals of Internal Medicine and JAMA, respectively. None of the SRs addressed inequity as a study question (Table II).

A description of patient inclusion and exclusion criteria in the RCTs under review was available in 11% (range 5–16% in the 4 journals) of the SRs (Table III). A description of index and control interventions was provided in 55% (range 29–69%) and 50% (range 24–62%), respectively. Primary outcomes were described in 23% of the SRs (range 19–31%). All 4 components of the PICO in individual RCTs were described in 3 of the SRs from the BMJ, 2 from JAMA, and none from the Annals of Internal Medicine or Lancet (Table SI¹). A total of 5 out of 115 SRs reported all PICO characteristics, i.e. 4% of all SRs.

None of the 115 SRs reported a description of the patients’ paths prior to assessment of their eligibility, or the reasons for exclusion before randomization, or percentages of eligible patients declining participation (Table IV). A description of the consecutiveness of the patient recruitment, the characteristics of healthcare system features, and the competence of staff were reported in 1%, 6% and 1% of the SRs, respectively.

Demographic and disorder specific clinical data was reported in 35% (range 0–50%) of the SRs (Table IV). Functioning of the patients (at least one item describing disease-specific or generic disability or health-related quality of life) was reported in 3% of the SRs. Co-morbid conditions were reported in 25% of the SRs (range 19–48%).

Any behavioural factor was reported in 21% (range 13–29%) of the SRs; any environmental factor in 0% of the SRs; and any factor related to potential inequity in 9% of the SRs (Table IV).

The baseline comparability of patients in the index and control groups in the RCTs was assessed in 7% of the SRs.

Adherence of patients in the RCTs to the index and control intervention(s) according to the protocol was reported in 7% of the SRs (range 5–8%). Cross-over to index interventions and control interventions was reported in 0% of the SRs. The use of other healthcare services besides the experimental interventions was reported in 2% of the SRs (Table IV).

The validity of outcome assessment was reported in 5% of the SRs (range 0–12%). None of the SRs reported reasons for dropping out of the RCT. None of the SRs assessed whether power calculations were performed in the RCTs. The adequateness of statistical analysis in the RCTs was assessed in 3% of the SRs (range 0–6%) (Table IV).

The results of assessment of individual SRs are shown in Table SI¹.

The aim of this review was to determine how comprehensively items relevant to the assessment of the generalizability of findings from RCTs were assessed in SRs published in leading general medical journals. It was found that the original RCTs often do not report these features (14), but a systematic review can assess the presence or non-presence of reporting of all the items, irrespective of whether they have been reported in the RCTs.

The BM referred to in this study was originally designed for the assessment of observational effectiveness studies where there is no randomization of the comparison groups, and therefore baseline comparability between groups is usually not satisfactory and necessitates statistical adjustment. In order to be able to adjust for baseline differences between groups, a very detailed description of patient characteristics is needed (7, 15, 16). The BM is in harmony with the PRISMA recommendations; but it makes explicit the items to be considered in SRs when evaluating the RCTs comprehensively (Table I). The BM has also been used for the assessment of the validity of RCTs and the generalizability of their evidence (7).

For the current paper, some modifications were made to the BM in order to best answer the current study questions. Most of the SRs assessed the effectiveness of pharmacological therapies, which is in agreement with the majority of RCTs being focused on these interventions (14) (Table I). Surprisingly none of the SRs assessed the effectiveness of rehabilitation, although the proportion of the elderly population in need of rehabilitative interventions is increasing rapidly worldwide, particularly in lower-middle income and upper-middle income countries (17). To be considered a form of rehabilitation, the current paper considered only those interventions that unequivocally fulfilled the criterion of also using efforts besides or beyond biomedical interventions to increase the functioning of patients. Thus, for example, single physical medicine interventions were not considered to be a form of rehabilitation. None of the SRs focussed on equity (Table III), even though the PRISMA statement extension to equity has been published in 2012, with the aim of increasing the number of SRs with a focus on equity issues (6).

Reporting of the intended PICO (patients, index interventions, control interventions and outcomes) in the RCTs under study was deficient overall, and this was the case in all 4 journals. Patient inclusion and exclusion criteria were reported in approximately 10% of the SRs overall. A description of the index and control interventions was reported in approximately 50% of the SRs, but varied from 24% to 69% between the journals. Only 4% of the SRs reported all PICO characteristics, although complete reporting of PICO information is necessary for the reader to understand the aim of each individual RCT. Furthermore, without comprehensive PICO information, assessment of the clinical heterogeneity of RCTs, and of the generalizability of the findings of a SR, may not be possible.

None of the SRs reported the description of patient selection in the RCTs (i.e. the patients’ path to entering a trial, and the proportion of those declining to participate and reporting the reasons for exclusion). A description of the consecutiveness of patient recruitment was rarely reported. A lack of information on the selection of patients means that the reader is not able to make judgements about whether the study population is representative of all those patients who need help for the disorder in question, or whether the patients are selected in a way that does not allow any generalizations, or that all generalizations are uncertain (14). In essence, the lack of data concerning the selection limits the possibilities to assess the clinical homogeneity of the RCTs and the generalizability of the findings of the SRs.

Only 6% of the SRs assessed the reporting of characteristics of the healthcare systems in the RCTs, and only 1% assessed the competence of staff. Health-care systems are potential determinants for patient outcomes (16), and there is evidence to show that the safety of healthcare interventions is dependent on staff competence (18). Lack of information on healthcare systems and on the competence of staff impedes the generalizability of the findings from the SRs.

Only 35% of the SRs reported what the medical indication was in the RCTs and only 3% reported respective disability. One of the journals (BMJ) did not report this information at all. Thus, the most crucial data from the RCTs is lacking in most of the SRs. Data on comorbid conditions, often encountered in clinical practice, was lacking in 75% of the SRs. A lack of data regarding these essentials may preclude any assessment of the generalizability of the findings (14).

Reporting of any behavioural factor in the RCTs was assessed in 21% of the SRs, while 0% of the SRs reported environmental factors (work and living conditions, marital status); and only 9% of the SRs reported any factor related to inequity (education, ethnicity). None of the SRs reported a single characteristic belonging to all 3 subcategories of behavioural, environmental or inequity-related factors, although all of these categories may affect the effectiveness of treatment (10, 19). None of the SRs reported data on socioeconomic status, which has been shown to be a strong prognostic indicator for a wide spectrum of disorders (20, 21).

Overall, there is evidence to show that patient characteristics, such as functioning, co-morbidities, as well as behavioural, environmental and equity-related factors may have a direct effect on outcome in RCTs, or may modify the treatment effect (8–10, 12, 13, 19, 22, 23). Even in cases where the effects of these characteristics are assumed to be minor, their importance as modifiers of the treatment effects will remain un-known unless these features are reported and analysed in the RCTs and SRs.

Infrequent reporting of the baseline comparability of the patients in the index and control treatment arms (which was reported in only in 7% of the SRs) impairs the assessment of the validity of the effectiveness estimates of the RCTs in the SRs. Baseline comparability between treatment arms is a shared criteria for BCTs and RCTs. It is also the feature of RCTs that makes it superior to observational effectiveness studies in terms of internal validity (7, 15)

The adherence of patients in the RCTs to the index and control intervention(s) according to the protocol was reported in 7% of SRs, while 0% of the SRs reported the cross-over to index interventions and control interventions; and only 2% of the SRs reported on use of other healthcare services besides the experimental interventions. In experimental studies the interventions (the experimental interventions and interventions due to use of other healthcare services) are the causal factors for the outcomes (24). If adherence to the intervention has been very high in some RCTs, and very low in other RCTs, the latter obviously results in lower estimates of effectiveness than the former. If there is no information on the degree of adherence it is not possible to assess the clinical homogeneity of the RCTs, and neither is it possible to assess the generalizability of the findings from the SRs.

The rare assessment of reporting of the follow-up percentages in the RCTs (in only 8% of the SRs), and total lack of reporting of reasons for patients dropping out of RCTs weakens the assessment of the generalizability of findings from the SRs.

Baseline comparability (the aim of randomization), description of adherence to interventions (a causal factor), and the validity of the outcome (effect) was assessed in 7%, 7% and 5% of the SRs, respectively. As these vital validity factors of an RCT are lacking, the appraisal of internal validity of individual RCTs is seriously impaired (24, 25).

The adequateness of the statistical analysis in the RCTs was assessed in only 3% of the SRs. There is evidence that there are often errors in statistical analyses of RCTs, which may lead to biased effectiveness estimates (26). In most meta-analyses the effectiveness estimates derive from the results sections of the RCTs (an exception is with individual patient meta-analysis) and if the estimates are not valid, the estimates of the meta-analyses will be biased. Lack of reporting of whether power calculations were made in the RCTs may lead to unwarranted interpretations of no effectiveness in cases where the statistical power has been insufficient to reach statistically significant results. Only clinically homogeneous RCTs can be analysed in a meta-analysis, and only then can the SR increase the statistical power and lead to narrower confidence intervals.

Although there were differences between the 4 journals in how the SRs assessed the comprehensiveness of reporting in the RCTs, the reporting was poor in all the journals. The reporting did not allow the assessment of the clinical homogeneity of the RCTs, the justification for meta-analysis, or inferences concerning the generalizability of the findings. Even in meta-analyses based on individual patient data from each RCT, there must be appropriate information from all the trials to ensure that it is plausible to combine patient data from different trials.

Limitations

A limitation of this paper is that a single person, the author, performed the literature search and data extraction. However, the literature search was repeated to ensure the completeness of the SRs published, and the author checked the accuracy of the extracted data twice. In order to make the judgement as unequivocal as possible as to whether an item was recorded in an SR, the decision was based on whether there was any information on that particular item; for example, the competence of the staff was rated as documented, if any characterization was included in the SR, and rated as not-documented if there was no mention of competence. Moreover, in the case of patient characteristics, the numbers of subcategories recorded in the SRs were documented separately (Table IV). Table SI shows for each RCT whether each of the items was reported, making it readily possible to check the accuracy of any assessment. Although there would be errors in some of the assessments, it is highly unlikely that the majority of the assessments would not be valid after the assessments had been repeated twice. However, there is an urgent need to replicate the findings of this paper, and to assess different sets of criteria for SRs in their ability to assess clinical heterogeneity/homogeneity of the included RCTs enabling assessment of justification of a meta-analysis, and generalizability of findings in different clinical contexts. Persons having the best competence on each particular study question are the most capable of reaching valid inferences on this matter.

The number of items that are important for any particular SR may be fewer than the number of items that are potentially important and thus suggested in the BM. Many journals have word limitations, which may pose limitations for the inclusion of BM characteristics. Since it is obvious that the items important for the generalizability of results should be documented, researchers should provide the information in tables or report some items in web appendices.

Conclusions

The current paper suggests that a comprehensive description of characteristics of RCTs by using the BM would be the primary option for SRs, and in case some items are left out of reporting, the reasons for omitting these would be provided in the SR. A comprehensive description will allow generalizability to any setting, and obviously the authors of SRs cannot be aware of all of these. Documentation of the BM items requires work from researchers carrying out SRs. However, this should not be considered additional work, but as recording of the characteristics of the very study object, the RCT. Compared with the total work involved in a SR, documenting these essentials is not an unsurmountable task.

The main finding of this paper is that the reporting of the characteristics of RCTs in the SRs published in the leading medical journals is extremely poor. The relevance of each item is, to some degree, dependent on the study context. However, every SR should document at least a description of the indication and its severity, the adherence to the intervention, and the proportions of patients crossing over to other treatment arms in each RCT. Considering the vast influence that SRs have in clinical and health policy decision-making, the findings of the current paper may have implications both for research, for critical assessment of SRs, and for clinical practice and policy-making. There is a need for further research on how exactly to define the concepts of BM and how to operationalize these concepts, and whether it is feasible also to consider minimum standards, even though the variation in context dependence worldwide may make this difficult.

The aim of a SR is to provide a full description of its study object, which, in most cases, is a RCT. When assessed in a comprehensive manner, the SRs and meta-analyses published in the leading general medical journals show a lack of reporting of the essential characteristics of their study objects, i.e. the RCTs. The findings indicate that it is currently not possible to generalize evidence from any of these SRs and meta-analyses. In the future, a detailed description of the RCTs is needed to decide whether a meta-analysis is justified in SRs, and in order to ensure the generalizability of evidence for clinical practice. The comprehensive data extraction, based here on the BM, allows the assessment of the clinical homogeneity of RCTs, and the assessment of the generalizability of the evidence in SRs. There is an urgent need for further research into these questions.

The author thanks Professor Pekka Jousilahti for valuable comments in the final stages of writing the manuscript; and Gareth Attwood, PG Dip, Licentiate Dip. TESOL, and Riitta Malmivaara, BA, MA for checking the English language.

The author has no conflicts of interest to declare.