Accepted Jun 11, 2020; Epub ahead of print Jul 10, 2020

J Rehabil Med 2020; 52: jrm00085

Correspondence address: Roxanne Maritz, Swiss Paraplegic Research, Guido A. Zäch Strasse 4, 6207 Nottwil, Switzerland, E-Mail: roxanne.maritz@paraplegie.ch

In our study we developed a common metric serving as a neutral comparator of two assessment tools which are used for assessing activities of daily living in rehabilitation patients in Switzerland. This common metric enables clinicians to use different established assessment tools assessing the same information, while being able to compare the respective information from those different scales on a larger level e.g. comparisons across clinics using different tools. This study is based on the example of Switzerland, where rehabilitation clinics can choose one of two measurement tools assessing activities of daily living, to report their outcome quality. With the common metric, the results from all the clinics can be compared with each other, no matter what tool was used for the assessment, enabling learning and improvement processes.

Functioning is the key indicator for rehabilitation as a health strategy (1). In order to strengthen rehabilitation it is essential to integrate functioning information, through the WHO’s International Classification of Functioning, Disability and Health (ICF), into national health information systems, including reports on health-care quality (2–4). Functioning information, including information on activities of daily living (ADL), is often collected using a variety of assessment tools, which can limit comparability across patients and clinics (5). Two prominent examples of such tools are the Functional Independence Measure (FIM^TM) and the Barthel Index (BI) (6, 7). Noteworthy, of these well-established tools there are many country and rehabilitation group adapted versions, such as the Extended Barthel Index (EBI) (8), the Modified Barthel Index (9) and the United Kingdom Functional Assessment Measure (UK FIM+FAM) (10). While all of these assessment tools focus on ADLs, their items and scoring structures differ.

There are 2 options for enhancing the comparability of functioning information: first, to define a single assessment tool as the standard, and, secondly, to establish a transformation system between existing established assessment tools. The first option would be difficult to achieve, as there are various reasons for the heterogeneity of the assessment tools in use, such as clinical utility and clinic-specific standards. The latter option, in which a common metric for functioning information is developed, to enable comparison and aggregation of information collected with different tools that measure the same concept, is more feasible (5, 11–15). For this purpose, a Standardized Assessment and Reporting System (StARS) for functioning information, with an ICF-based interval-scaled neutral common metric as core element, would enable clinicians to continue using different assessment tools, while at the same time enabling the aggregation and comparison of conceptually equivalent information (5, 13). This common metric could serve as a reporting reference, e.g. for national rehabilitation quality reports, allowing for comparisons between institutions using different tools. A common metric also allows for the transformation of the score of one tool into the score of another conceptually equivalent tool. Furthermore, it provides a transformation of the ordinal-scaled ADL scores to an interval-scale level, which is required to calculate means and change scores between admission and discharge (16).

In order to demonstrate how a StARS of functioning information can be established, this paper takes Switzerland’s national rehabilitation quality reports as an example, providing a simplified illustration for the heterogeneous landscape of assessment tools. For Swiss national quality reports in musculoskeletal and neurological rehabilitation, clinics can choose to report with the FIM^TM or the EBI, but this impedes the comparison of clinics that use different assessment tools.

Earlier research by Prodinger et al. provided a score transformation of the FIM^TM 13-items motor score and the BI on the basis of the Rasch model (17). While the FIM^TM 13-item motor scale has been studied extensively (18), its 5 items of cognition have received less attention. Nevertheless, cognitive impairment is important in neurological disorders (19). The importance of cognition was the reason for development of the EBI, extending the BI with 6 cognitive items (8). The current paper therefore seeks to build on the evidence established by Prodinger et al. regarding the motor scales of these 2 assessment tools (17), using the same psychometric approach, but extending it to the 2 assessment tools’ versions, including cognitive items. Furthermore, this paper provides a concrete example of how an ICF-based and interval-scaled common metric as a core of a StARS (5), can be created, so that the outcomes of different rehabilitation clinics, using different assessment tools assessing ADLs, can be compared and aggregated.

The objective of this study was to create an ICF-based interval-scaled common metric as a core element of a StARS for functioning information, based on the example of Swiss national quality reports. The approach included the assessment of the 2 key requirements for standardized reporting of health information: (i) to determine whether the 2 assessment tools can be considered conceptually equivalent, and (ii) to examine whether a reference metric including the FIM™ and EBI can be established by applying the Rasch model (20).

Setting and subjects

In Switzerland, the National Association for Quality Development in Hospitals and Clinics (ANQ) coordinates the measurement and public reporting of outcome quality indicators for all rehabilitation clinics (21). For this purpose, functioning information from every patient in neurological and musculoskeletal rehabilitation is collected. Clinics can choose 1 of the 2 tools, either the FIM™ (18-item version) or the EBI for this part of the data collection. To overcome the issue of comparability, the ANQ has commissioned the development of an expert-based transformation algorithm of the 2 assessment tools, called the ANQ ADL Score. In order to create and validate the expert-based transformation algorithm, the Institute of Medical Sociology and Rehabilitation Science from the Charité – Universitätsmedizin Berlin, Germany conducted the respective study from 2015 to 2017 (22). The validation sample included 265 patients undergoing neurorehabilitation from 5 Swiss rehabilitation clinics, representative of the whole continuum of score ranges of the 2 assessment tools. All patients were assessed at admission with both assessment tools, effectively providing a reliable basis for scale-equating procedures, i.e. a common person design (23, 24). Both tools were assessed either in German (4 clinics) or Italian (1 clinic). The data collected in the ANQ ADL Score study, i.e. a prior research project, were used in secondary data analysis in the current study to provide the basis for the ICF-based and interval-scaled common metric. Both the ANQ ADL Score study and the current study were given ethics approval by the respective Swiss Ethics Commission.

ADL assessment tools

The FIM™ is an assessment tool administered by health professionals comprising 18 items. In order to qualify for FIM™ administration, the health professionals received training provided by the ANQ according to the respective FIM^TM policy. FIM™ consists of 13 motor items and 5 cognitive items. All items are scored from 1 (total assistance) to 7 (complete independence), summing to a total score ranging between 18 and 126 (25).

The EBI is an assessment tool with 16 items, administered by health professionals. A user manual for the administration of EBI is available, but there is no specific training. Ten motor items are based on the original BI (26), 6 items cover cognitive functioning aspects, of which 5 are derived from the cognitive FIM^TM items. One cognitive item is unique to EBI and refers to vision and neglect. All items are scored from 0 to 4, resulting in a total score of 0–64 (8). Not all EBI items contain all scoring categories (e.g. Item 1 Feeding can be scored 0, 2, 3 or 4), therefore an adapted 0–50 scoring version was proposed for Rasch analyses with EBI, which was taken as a basis for this study (27). A conversion from EBI 0–64 scores to EBI 0–50 scores, referred to as EBI50, on item basis can be found in Appendix S1¹.

Recent studies for both tools showed, that in the context of national quality reports they measure a unidimensional construct and can be reported as total scores on the interval-level, when Rasch-based transformation with bi-factor equivalent design is applied (27, 28). In these studies, neither tool showed differential item functioning for sex, age, nationality, healthcare insurance status of patients, time-point of measurement, rehabilitation group (neurological or musculoskeletal) or clinic language (German, French or Italian).

ICF linking

The first part of data analysis entailed linking each item of the respective tools to the ICF by using the ICF linking rules, an established method to enable comparability of health information (29). In order to satisfy the first requirement for standardized reporting and scale equating, i.e. to demonstrate the conceptual equivalence of the 2 assessment tools to be integrated into the common metric, the items from each assessment tool were linked to corresponding ICF categories. Furthermore, the perspectives from which information is collected and the categorization of response options were identified for both tools, in accordance with the ICF linking rules. Two researchers, with extensive linking experience (Maritz R., Selb M.), independently linked all items of the FIM^TM and the EBI to the ICF, following step by step the refined version of the ICF linking rules (29) in Microsoft® Excel. The results of the independent linking were then compared and discussed. When no agreement on linking of an item could be reached, a third researcher (Prodinger B.) was involved to give advice and reach agreement.

Rasch analysis

The second part of the data analysis was based on Rasch analysis. In order to satisfy the second requirement for standardized reporting, i.e. score equivalence, the polytomous partial credit Rasch measurement model and associated requirements for equating of  instruments were applied to derive an interval-scaled common metric from ordinal data (23, 30). These requirements include: unidimensionality; item invariance across sample subgroups, such as age or sex; and local independence, i.e. the demonstration that responses to any item should depend only on the trait (functional independence in the case of EBI and FIM^TM) and not on responses to other items (31). The analyses were conducted on the total score level of the 2 assessment tools. This was based on the reasoning that: (i) the total scores reflect the level of reporting in the national quality reports; (ii) previous findings support that the 2 tools can be reported as unidimensional metrics on the total score level (27, 28), representing the construct of functional independence, even though they incorporate both motor and cognitive items; and (iii) the recommendations for scale equating by Andrich are fullfilled (23). The data from the validation sample of the ANQ ADL Score study (described above under the subheading “Setting and subjects”) were used for Rasch analyses. Basic sample characteristics and descriptive statistics were conducted using Stata Version 14.2, Rasch analysis was conducted with RUMM2030 professional version 5.4.

The analytical focus gave reference to the following 6 key criteria, helping to judge if fit to the Rasch model and its requirements for equating of 2 instruments was achieved: (i) the class-interval based conditional test of fit, assessing the observed and the expected scores under the model conditional on each total score through a Pearson χ² test; (ii) the item-trait interaction χ² test, reflecting fit of the data to the Rasch model, also referring to observed and expected scores on the level of class-intervals; (iii) the reliability indexes, reported as Cronbach’s alpha and person separation index (PSI); (iv) differential item functioning (DIF), indicating if there is invariance for different subgroups; (v) unidimensionality, expressed as percentage of significant t-tests, using individual t-tests comparing person-ability estimates for each respondent derived from the subtest analysis; and (vi) threshold ordering, indicating whether the different scoring categories of an assessment scale are represented in a successive order (16, 23, 30). Acceptable levels of the key criteria are represented in the bottom row of the corresponding results Table I.

In order to fully examine the defined key criteria, a 2-tiered analysis was used to deal with some restrictions of the analysis software, as FIM^TM has 109 scoring options (ranging from 18 to 126) and the RUMM2030 software allows for inclusion of only 101 scoring options. First, analysis of the assessment tools’ total scores was performed, in which the FIM^TM total scores were rescaled to 0 to 100 together with the EBI50 total scores. This first step is shown in the first row of the corresponding result in Table I, allowing for a conditional test of fit in which both total scores served as 2 items. The FIM^TM score was then re-weighted by 1.09 to give the usual operational score of the FIM^TM. Secondly, the FIM^TM items were divided into 2 testlets and the EBI50 items were combined in a third testlet. Which FIM™ item was contained in which testlet, was based on a previous research project about FIM™ total scores for use in national quality reports (28) and is indicated in the legend of Table I. This second analysis step enabled values of the variance in the latent estimate to be obtained, which imply the degree of local dependency remaining in the testlets (shown in the second row of the corresponding result in Table I). Furthermore, the 2 FIM™ testlets could then be taken together as a single super-testlet in a paired t-test analysis with the EBI total score, thus addressing the software limitation described, so that the FIM^TM total scores can be reported on their original range.

Differential item functioning strategy

DIF was tested for sex, age (4 groups based on interquartile ranges), healthcare insurance status (general, semi-private, private), nationality (Swiss/other), duration of rehabilitation (4 groups based interquartile ranges) and clinic (5 rehabilitation clinics involved) using 4 class intervals. When uniform or non-uniform DIF p-values were < 0.05, DIF was considered to be present, and the respective testlets were split, starting with the highest DIF, and continuing until no further DIF was present (32). The split and unsplit solutions were then contrasted on the basis of the Rasch person estimates, anchored to each other with an unsplit testlet free of DIF. To determine whether DIF split was necessary for the transformation table, an effect size calculation was performed, based on a Cohen’s D calculation, including the mean of the person estimates, their standard deviations, and the sample size of the split and unsplit version (32). If the resulting effect size was below 0.1, DIF was considered small and no action was taken, i.e. the unsplit solution was retained (33).

Common metric

If model fit was achieved with both Rasch analysis steps, an interval-scaled common metric was created based on the Rasch location estimates. The common metric is based on a paired t-test from the second Rasch analysis step, with the EBI total score testlet on the one hand and both FIM(TM) testlets together on the other hand. Likewise, the FIM(TM) total scores can be reported on their original range (18–126) and are therefore considered more accurate. The common metric was designed to range from 0 (complete dependence in ADL) to 100 (complete independence in ADL), oriented at similar research projects (13, 17), reflecting minimum and maximum logit estimates derived from the joint analysis.

International Classification of Functioning, Disability and Health linking

An overview of the ICF linking is shown in Table II, and detailed linking at the level of the items is shown in Appendix S2¹. Both assessment tools’ items represent a dependency perspective and all items responses were categorized in the form of an intensity of this dependency (1–7 for FIM™, 0–4 for EBI). The content of the FIM^TM items was reflected in 24 ICF categories, and the content of the EBI in 26 ICF categories. Both assessment tools cover the same ICF categories, with the exception of EBI item 16 vision/neglect, linked to the ICF categories b156 Perceptual functions and b210 Seeing functions that are not reflected in the FIM^TM. Both tools covered predominantly the activities and participation categories of the ICF, with a focus on the ICF chapters d3 Communication, d4 Mobility and d5 Self-care. Due to the high level of concordance, the tools were considered conceptually equivalent, i.e. measuring the same latent trait, both covering content related to the concept of functioning.

Sample characteristics

The sample incorporated 265 patients in neurological rehabilitation in Switzerland from 2016 and 2017. Four clinics from the German-speaking and one clinic from the Italian-speaking region of Switzerland each provided between 31 and 70 cases. The sex distribution was 50.8% male (n = 123) and 49.2% female (n = 119). The mean age of the sample was 67.2 years, ranging from 18 to 92 years, while the mean age of the cases of the different clinics ranged from 62.9 to 73.9 years. The majority (90.0%) were Swiss citizens, 10.0% had other nationalities. Almost three-quarters (74.0%) had general healthcare insurance, 18.6% semi-private insurance and 7.4% private insurance. The mean rehabilitation duration time was 37.7 days, ranging from 5 to 150 days (standard deviation 25.4). Minimal attained FIM^TM score of 18 was attained by 7 cases, the maximal score of 126 was not achieved. The minimal EBI score of 0 was achieved by 2, the maximal score by 11 cases. There were missing values for the variables sex, age, healthcare insurance status, rehabilitation duration (each n = 23), and origin (n = 24).

Rasch analysis

Results of the Rasch equating analyses are shown in Table I. Fit of the data as total scores (first analysis step, first line Table I), and as 3 testlets (second analysis step with 1 EBI testlet, 2 FIM^TM super-items, second line of Table I) showed good fit to the Rasch model. The conditional test of fit in the total score analysis and the item-trait χ² in both analyses were all non-significant. Reliability was consistent with high-stakes clinical decision-making. All the common variance was subsequently included in the interval-scaled transformation, and both analyses satisfied the monotonic relationship with the functioning trait, thus showing no disordering of thresholds, again a requirement for successful test equating at the scale score level. DIF was present at the clinic level for both tools. A comparison of person estimates between a non-split and split- solution resulted in an effect size of 0.02; thus, it was considered marginal and no action was taken. Detailed information on the DIF strategy and the effect size calculation are shown in Appendix S3¹.

Common metric

Based on the second analysis step, a common metric was created (see Table III), using the total scores of FIM™ and EBI as testlets. The related transformation table for FIM™ and EBI50 total raw scores can be retrieved from the metric and is shown in Appendix S4¹. For example a FIM™ raw score of 114 can be translated to 72.9 in the common metric, or an EBI50 raw score of 47. The common metric also shows that the operational range of the FIM^TM (common metric values from 8.4 to 96.6) is larger than that of the EBI (common metric values from 9.3 to 81.1), indicating that the FIM™ covers a wider range of patient abilities and that there might be ceiling effects with the EBI. The operational ranges of the 2 assessment tools, in contrast to the common metric, are shown in Fig. 1. The fact that the operational range of the common metric is wider than the range of both assessment tools on its own, can be explained by the fact that the calculation of the common metric is based on patients assessed with both tools together, i.e. the common person design of the sample.

This study provides an ICF-based and interval-scaled common metric, including the transformation between FIM^TM and EBI for use in national rehabilitation quality reports, and thus facilitates a Standardized Assessment and Reporting System (StARS) for functioning information. In order to create the common metric, 2 steps were followed; first, ICF linking showed that the 2 assessment tools can be considered as conceptually equivalent. In the second step a common metric including the 2 tools could be established, as fit to the Rasch models’ requirements for the equating of instruments was achieved. The current study provides an example of how ADL scales assessing the same information can be made comparable through the ICF on the conceptual level and an interval-scaled common reference metric on the scale level. The methodology applied can inform further research, in which conceptually similar functioning information is collected with different tools and needs to be made comparable or aggregated. Such a system could, for example, be used in areas other than national quality reports, such as systematic reviews and meta-analyses in the field of rehabilitation (34), or country comparisons of rehabilitation outcomes with different versions of the BI or the FIM.

The common metric established in this study has 4 major advantages over other transformation systems, such as, for example, an expert-based transformation system developed in the original ANQ ADL Score study (22) or scale equating without a reference metric, such as equating using the Leunbach model (35). First, the common metric approach is based on the ICF, the global standard for conceptualizing and describing functioning information, serving as neutral and conceptual reference to compare the content of the different tools included in the common metric (4, 36). Secondly, Rasch analysis allows an interval-based scoring and transformation table to be derived to support the calculation of valid change scores of functioning, e.g. between admission and discharge, which can inform clinical practice and research about functioning change in a quantifiable way (37). Thirdly, the common metric allows us to take the operational ranges of the 2 assessment tools into account, and can therefore operate at the level of the more detailed tool (17). Fourthly, through the ICF-based common metric, other assessment tools measuring functioning could be added in future (5). Given that this study was conducted in the context of Swiss national quality reports, only 2 assessments were included. In principle, any number of instruments can be integrated into a common metric as long as they are conceptually equivalent. Examples with more than 2 scales exist and have been published (5, 13, 20).

The comparison of the operational ranges of the 2 assessment tools showed that FIM^TM covers a wider operational range than EBI. This finding can inform the choice of assessment tools and the interpretation of change scores. The basis of adding further assessment tools to the common metric would again be the linking of the respective assessment tool with the ICF and a person-equating design, in which the new tool to be added to the metric is assessed in parallel with either EBI or FIM^TM. As FIM^TM is the assessment tool with the wider operational range, and is more widely used in rehabilitation in general, and was also previously used in other scale-equating projects, e.g. with the Barthel Index or the minimal dataset (11, 17, 38), it appears to be the choice as a linking scale.

Attention should also be paid to the equating design, as indirect transformation between 2 tools, i.e. the equating of instruments via one instrument that is already included in a transformation set, have been shown to be less precise than the direct transformation, i.e. the equating of different instruments in the same study via a common person design (35). However, indirect transformations are likely to be more feasible in terms of data collection. This reflects typical challenges in the practice of quality improvement work (39). A limitation of the study is that the dataset only covers data from 2 assessment tools and includes only neurological rehabilitation patients. The common metric would also be needed for other assessment tools or rehabilitation groups, such as that of musculoskeletal patients in the example of the Swiss quality reports. As previous studies of each assessment tool showed no substantial DIF between neurological and musculoskeletal rehabilitation patients (27, 28), the transformation table could preliminarily be applied to the musculoskeletal ANQ reports. Nevertheless, this should be validated in a future equating study including musculoskeletal data. Another limitation of the study was the restrictions of the RUMM2030 software not being able to cover total score or testlets with more than 100 thresholds (e.g. a score of 0–100), leading to some adaptions in the analyses. Likewise, the Rasch analyses had to be 2-tiered in order to enable judgement about the model fit. Furthermore, the current system does not facilitate the separation of cognitive from motor performance, as it is based on the total scores of the ADL tools, which reflect the level of reporting of the Swiss quality reports in rehabilitation. However, previous studies on EBI (27) and FIM™ (28) showed that the total scores, including motor and cognitive items, can be reported as a unidimensional construct, i.e. functional independence. Another methodological challenge was that the weight of ICF linking was not clearly defined, i.e. there is no clear cut-off when conceptual equivalency is achieved. However, with the methodological approach chosen, the conclusion drawn, based at the conceptual level on the results of the ICF linking, was supported at an empirical level by the Rasch analysis.

A strength of the present study, in addition to the above-mentioned advantages of the common metric, includes the use of the Rasch model as a basis to create the common metric. In the Rasch model the raw score is a sufficient statistic, such that there is a one-to-one correspondence between the raw score and the latent estimate, which is not the case in other item response theory or classical test theory models (16). Furthermore, the resulting estimates of the Rasch model, which build the basis for the common metric, are distribution free, given the person and item parameter separation. The common metric is therefore not sample dependent and can be applied to any relevant sample (16).

Application in practice

The ICF-based and interval-scaled common metric including the FIM^TM and the EBI, as provided in this study, can be applied in national quality reports for neurological rehabilitation and, based on previous evidence with regards to DIF, for musculoskeletal rehabilitation. Application of the transformation table enables the comparison of clinics that are using different assessment tools and, at the same time, the calculation of valid change scores for patients between admission and discharge, as the transformation is based on the interval-scale level. Extension of the common metric with other functioning-related measures is possible. Possible extensions could be informed on the basis of the conducted ICF linking in comparison with the ICF Rehabilitation Set, also referred to as the ICF Generic-30, which was designed to define ICF categories as a minimal standard for reporting and assessing functioning and disability in clinical populations along the continuum of care (40).

Conclusion

The ICF-based and interval-scaled common metric provided in this study supports the assessment of patient outcomes using different ADL assessment tools (FIM™ and EBI) in clinical practice, while at the same time being able to compare the related outcomes of different clinics. It lays the groundwork for a standardized reporting system of functioning information for use in national quality improvement reports.

The authors thank Melissa Selb from the ICF Research Branch for her support in the ICF linking, Martin Brünger, Stefanie Köhn, Anna Schlumbohm and Karla Spyra from the Institute of Medical Sociology and Rehabilitation Science of the Charité – Universitätsmedizin Berlin, Germany for providing and supporting us with the data from the ANQ ADL Score Study, Luise Menzi, Head of Rehabilitation ANQ, Klaus Schmitt, former Corporate Development Director and Anke Scheel-Sailer, Head of Clinical Quality Management Research Department from the Swiss Paraplegic Centre, for their provision of valuable advice for the project. This project is part of the cumulative Dissertation of Roxanne Maritz, which is funded by the Swiss National Science Foundation’s National Research Programme “Smarter Health Care” (NRP 74) within the NRP74 StARS project.

The authors have no conflicts of interest to declare.