Content » Vol 50, Issue 10

Guest Editorial

Assessing the effectiveness of rehabilitation and optimizing effectiveness in routine clinical work

Antti Malmivaara, MD, PhD1,2,3

 

From the 1Centre for Health and Social Economics, National Institute for Health and Welfare and 2Orton Orthopaedic Hospital and Orton Research Institute, Orton Foundation, Helsinki, Finland, 3Cochrane Rehabilitation, Don Carlo Gnocchi Foundation, Milan, Italy. E-mail: antti.malmivaara@thl.fi

Accepted Sep 25, 2018; Epub ahead of print Oct 9, 2018

J Rehabil Med 2018; 50: 00–00

The purpose of all activities in medicine (education, research, administration, management, clinical work etc.) is to advance the health and wellbeing of patients and populations in ordinary healthcare settings (1). The real-effectiveness medicine (REM) framework recommends instituting actions on 4 levels to advance effectiveness in routine clinical work (Fig. 1). The basis of all effectiveness is the competence of the staff; without this, any benefit for the patient is unimaginable. Since the beginning of 1990s the Royal Society for Physicians and Surgeons of Canada has used a competence framework, CanMeds, which has been successfully implemented also for healthcare professionals other than physicians (2). In addition to medical expertise, the CanMeds framework assesses competence related to communication, co-operation, management, health advocacy, scientific skills and professionalism, including relevant ethical issues.

The second level of the REM framework is evidence-based medicine as a means to convey the current scientific evidence into everyday treatment decisions. Randomized controlled trials (RCTs), systematic reviews and meta-analyses, and clinical guidelines constitute the core of evidence-based medicine. The third level of the REM is to attain valid data on patients, interventions and outcomes in the real-world setting, and the fourth level is to use this data in benchmarking with peers treating similar patients (Fig. 1).


Fig. 1. The 4 levels of the real-effectiveness medicine framework for pursuing best effectiveness for patients in real-world settings. EBM: evidence-based medicine.

Obtaining valid and generalizable research data on effectiveness of rehabilitation

Medicine and rehabilitation is very much about intervention: how we are able to help patients beyond what would be their state if we were not there. Knowledge of the effectiveness of our interventions and of the most effective interventions is therefore important.

There are 2 ways to obtain scientific information about effectiveness: experiment or observation. The question is how to decide which of these is the most appropriate option for a particular study. If the study question is on effectiveness of a single intervention, then an RCT, i.e. experimental design, is usually the best option. To be valid, the RCT should include a concealed randomization protocol in order to reach maximal comparability between the intervention groups. Consequently, the between-group differences in the outcomes can be attributed to the differences in interventions in the treatment arms. The most important validity items in RCTs can be grouped into 3 categories: baseline comparability (aim of randomization), difference in the interventions between the treatment arms (the cause for outcome), and validity of the outcome (valid outcome measures, high follow-up rate) (Fig. 2).


Fig.2. The three main categories (domain, cause, effect) to be considered for the design, conduct and appraisal of randomized controlled trials (RCTs)

However, even when one is studying a single intervention, there may be ethical reasons, or issues related to the study question or the feasibility of the study, which may render observational effectiveness study, the benchmarking controlled trial (BCT) an alternative option, or even the only option (3) (Table I). If the aim is to assess the effectiveness of a clinical pathway, BCT is, in most cases, the design of choice, since cluster randomized trials are rarely feasible. If the study question is about comparing the performance of different healthcare providers treating and rehabilitating a particular group of patients, the only suitable study design, by definition, is the BCT.


Table I. Instances where a randomized controlled trial (RCT), or an observational study design, the benchmarking controlled trial (BCT), may be chosen for assessment of treatment or rehabilitation effectiveness.

There are recommendations on how to plan, conduct, report and assess observational effectiveness studies, the BCTs (4, 5). The term benchmarking is derived from the necessity to make between-peer comparisons, which makes it possible to learn from the best practices. The validity criteria on BCTs are rather similar to those of RCTs, except that there are 3 further main validity requirements for BCTs due to lack of randomization: the selection of patients must be reported, as well as the staff competence and relevant healthcare system features (4, 5).

Choosing between a double-blinded or non-blinded randomized controlled trials

As always in science, one starts with a study question, and then proceeds to find the methods that provide a valid answer. Thus, whether to blind the patients and healthcare professionals treating and rehabilitating them is dependent on the study question. If the question is on the biological (or physical) effectiveness of an intervention (intervention effect per se), one has to use a double-blinded study design. However, if the study question is to assess the effectiveness of an intervention in the non-blinded circumstances of everyday healthcare, one should neither blind the patient nor the therapist. In most cases rehabilitation interventions fall into this category, where blinding is contra-indicated.

The idea of using double-blinding only when blinding is needed to answer to the study question has important implications. As double-blinded RCTs assess only the intervention effect per se, excluding the placebo effect, it produces abstract effectiveness estimates that do not occur in ordinary health care. The true effectiveness of, for example, a drug for depression is composed of the biological effect shown by a double-blind RCT plus the placebo effect of the drug to the patient (6). Consequently, the effectiveness estimates from double-blind RCTs are not valid for calculating number needed to treat figures, or cost-effectiveness (or cost-utility) estimates for real-world circumstances.

 

Generalizability and clinical importance of the treatment effect

Appropriate description of patient, index and control interventions and outcome (PICO) characteristics that have actualized in the original study are necessary for the clinician to be able to judge whether the results are generalizable. In addition to PICO characteristics,  also those of co-morbid conditions, and patients behavioural, environmental and equity-related items are often important for making generalizations.

For a treatment to be considered effective the results must be statistically significant. An additional question is the size of the treatment effect. Often a clinically important treatment effect is determined based on the mean difference in perceived outcome between the treatment and control groups. However, it is more appropriate to assess the differences in percentages of those patients in the treatment and control arms who exceed the effect size considered clinically important. It is suggested here that the statistical significance of between-group proportions of patients receiving small, moderate and large treatment effects will be assessed, and the difference in proportions would be used and offered to clinicians for informing patients. Similarly, it is suggested that, in systematic reviews, instead of assessing the magnitude of effectiveness from standardized mean differences in the individual RCTs, the between-group differences in proportions of patients receiving, small, moderate and large treatment effects would be assessed.

 

Healthcare system features as ways to increase effective patient care

Healthcare systems may have a strong modifying effect on patient outcomes. For example, building up a well-functioning clinical pathway for fragile elderly people with a hip fracture improves the patients’ probability of returning home (7). The design of choice for assessing the impact of healthcare systems on patient outcomes is the BCT, as cluster RCTs are usually unfeasible (8).

 

Systematic reviews, meta-analyses and overviews of systematic reviews

Systematic reviews aim to provide a summary of all relevant effectiveness studies for a particular clinical question. Comprehensive description of the characteristics of patients, intended interventions (and how they actually happened during the trial) and outcomes are the cornerstones of a systematic review, and make it possible for the clinicians to assess whether the results are generalizable to their own practice. The first  of the 3 most important validity items in synthesizing evidence from original studies is, that only those trials which are clinically homogeneous are used for evidence syntheses, and only these clinically homogeneous trials will be subjected to a meta-analysis. The second essential validity criterion is that the interpretations are based on the most valid original studies. Thus, primary meta-analyses should be based on clinically homogeneous studies having best internal validity (9). The third validity item relates to the whether there may be a publication bias.

Overviews of systematic reviews aim to summarize evidence from different systematic reviews covering a particular area of interest. As the study objects are the systematic reviews, not the original publications, the overviews should aim to provide a thorough description of the patients, interventions, and outcomes in each systematic review; to assess the internal validity of the reviews, and to present the results and authors’ conclusions. The overviews should not engage in statistical assessment of the evidence itself.

The recently established Cochrane Rehabilitation provides summaries of those Cochrane systematic reviews that have assessed the effectiveness of rehabilitation, and helps rehabilitation professionals in their efforts to apply the best current evidence in their clinical praxis (10).

Optimizing the effectiveness of rehabilitation in routine clinical work

The aim of medicine and rehabilitation is to help the patient. The starting point is to find out what the diagnosis is, and what are the factors contributing to the patient’s condition. The first clinical intervention category is to convey this information to the patients, advise and support the patients in their own efforts to alleviate symptoms and increase probability of recovery (Fig. 3). The second intervention category is to assess the functioning of the patient and take actions to increase the patient’s abilities and participation (11). The third intervention category is to consider effective biological (or physical) ways of treating the patient. The way to increase the placebo effect of the biological treatments can be used to further increase the effectiveness of treatment and rehabilitation.

The author has no conflicts of interest to declare.


Fig. 3. Cumulative effectiveness in the real-life treatment and rehabilitation of patients. The degree of effectiveness is schematic and does not represent a particular context.

References
  1. Malmivaara A. Real-effectiveness medicine – pursuing the best effectiveness in the ordinary care of patients. Ann Med 2013; 45: 103–106.
    View article    Google Scholar
  2. Turner SR, White JS, Poth C, Rogers WT. Learning the CanMEDS roles in a near-peer shadowing program: a mixed methods randomized control trial. Med Teach 2012; 34: 888–892.
    View article    Google Scholar
  3. Malmivaara A. Clinical impact research – how to choose experimental or observational intervention study? Ann Med 2016; 48: 492–495.
    View article    Google Scholar
  4. Malmivaara A. Assessing validity of observational intervention studies – the benchmarking controlled trials. Ann Med 2016; 48: 440–443.
    View article    Google Scholar
  5. Malmivaara A. Benchmarking controlled trial – a novel concept covering all observational effectiveness studies. Ann Med 2015; 47: 332–340.
    View article    Google Scholar
  6. Malmivaara A. Validity and generalizability of findings of randomized controlled trials on arthroscopic partial meniscectomy of the knee. Scand J Med Sci Sports 2018; 28: 1970–1981.
    View article    Google Scholar
  7. Addo J1, Bhalla A, Crichton S, Rudd AG, McKevitt C, Wolfe CD. Provision of acute stroke care and associated factors in a multiethnic population: prospective study with the South London Stroke Register. Br Med J 2011; 342: d744.
    View article    Google Scholar
  8. Malmivaara A. System impact research – increasing public health and health care system performance. Ann Med 2016; 48: 211–215.
    View article    Google Scholar
  9. Malmivaara A. Methodological considerations of the GRADE method. Ann Med 2015 Feb; 47: 1–5.
    View article    Google Scholar
  10. Negrini S, Arienti C, Pollet J, Engkasan JP, Gimigliano F, Grubisic F, et al. Cochrane rehabilitation: report of the first year of work. Eur J Phys Rehabil Med 2018; 54: 463–465.
    View article    Google Scholar
  11. Stucki G. Olle Hook Lectureship 2015: The World Health Organization’s paradigm shift and implementation of the International Classification of Functioning, Disability and Health in rehabilitation. J Rehabil Med 2016; 48: 486–493.
    View article    Google Scholar

Comments

Do you want to comment on this paper? The comments will show up here and if appropriate the comments will also separately be forwarded to the authors. You need to login/create an account to comment on articles. Click here to login/create an account.