Carlijn H. van der Zee, MSc1, Albert Kap, MD1, Radha Rambaran Mishre, MD2, Evert J. Schouten, MD3 and Marcel W. M. Post, PhD1,4

From the 1Rudolf Magnus Institute of Neuroscience and Center of Excellence for Rehabilitation Medicine, University Medical Center Utrecht and Rehabilitation Center De Hoogstraat, Utrecht, 2Sophia Revalidatie, location Delft, 3Libra Zorggroep, location Tilburg, The Netherlands and 4 Swiss Paraplegic Research, Nottwil, Switzerland

OBJECTIVE: To assess the responsiveness of 4 participation measures.

DESIGN: Longitudinal study with repeated measurements at the start (t1) and at the end (t2) of a multidisciplinary outpatient rehabilitation programme, and at 4 months follow-up (t3).

SUBJECTS: Outpatients with different diagnoses (n = 395) from 5 rehabilitation centres in The Netherlands.

METHODS: Measures were the Frenchay Activities Index (FAI), the Participation subscale of the ICF Measure of Participation and Activities Screener (IMPACT-SP), the Participation Scale, and the Utrecht Scale for Evaluation of Rehabilitation-Participation (USER-Participation). Responsiveness was analysed using the effect size and the standardized response mean.

RESULTS: Comparing scores at t1 and t2, the standardized response mean was 0.54 for the USER-Participation Restriction scale, 0.41 for the FAI, 0.40 for the IMPACT-SP, 0.39 for the USER-Participation Satisfaction scale, –0.36 for the Participation Scale, and 0.21 for the USER-Participation Frequency scale. Effect size values were generally somewhat smaller than standardized response mean values. Effect size and standardized response mean values were negligible between t2 and t3. Responsiveness parameters varied between diagnostic groups, with participants with acquired brain injury showing the largest change and participants with neuromuscular disease or chronic pain showing least change.

CONCLUSION: Overall and across the different diagnostic groups, the USER-Participation Restriction scale showed the best responsiveness.

Key words: longitudinal studies; validation studies; social participation; outcome assessment (healthcare).

J Rehabil Med 2011; 00: 00–00

Correspondence address: Marcel W. M. Post, Rehabilitation Center De Hoogstraat, Rembrandtkade 10, NL-3583 TM Utrecht, The Netherlands. E-mail: m.post@dehoogstraat.nl

Submitted December 9, 2010; accepted July 21, 2011

INTRODUCTION

Most patients are referred to rehabilitation treatment because of chronic disabling conditions. Their treatment will be aimed at minimizing the consequences of these conditions to improve independence and ultimately social participation (1, 2). In particular in the outpatient clinic re-establishment of participation is a key aim of rehabilitation programmes. Measurement of participation outcomes is necessary to evaluate the accomplishment of this goal (3). Outcome measurement may help clinicians to document the results of their interventions, provide a valuable tool in developing standardized clinical practices, guide clinical decisions regarding care, improve organizational performance, and provide a basis for outcome standards (3). Furthermore, there is growing pressure from a variety of stakeholders, including sponsors, regulatory agencies and service recipients, to share medical performance information to improve our healthcare system (3). However, even though participation is generally considered an important goal, measuring participation as outcome of rehabilitation is not common (4, 5). This discrepancy has been related to the nature of participation as being affected by many factors outside the control of the rehabilitation team, but also to measures of participation being less developed than measures of more basic activities of daily living (6).

One major problem is the conceptualization of participation. In the International Classification of Functioning, Disability and Health (ICF) (7), “Activity” is the “execution of a task or action by an individual” and represents the individual perspective on functioning. “Participation” refers to “the involvement of an individual in a life situation” and represents the social perspective on functioning. Originally conceived of as 2 distinct components, the final version of the ICF merged the 2 taxonomies of Activities and Participation into a single series of categories in 9 chapters. Since then, no consensus has emerged on how best to distinguish between Activities and Participation, and on the aspects of functioning that comprise Participation (8–10). We largely agree with Whiteneck & Dijkers (8), who advocated a distinction between Activities and Participation, and suggested to designate the ICF categories in the chapters 1 (Learning and Applying Knowledge), 2 (General Tasks and Demands), 3 (Communication), 4 (Mobility) and 5 (Self-Care) as Activities, the categories in the chapters 7 (Interpersonal Interactions and Relationships), 8 (Major Life Areas) and 9 (Community, Social and Civic Life) as Participation, and to designate some categories in chapter 6 (Domestic Life) as Activities and other categories as Participation (the latter to be added to the Major Life Areas chapter). Earlier, we advocated a largely similar distinction between Activities (ICF chapters 1–5) and Participation (ICF chapters 6–9) (10). Furthermore, a recent review showed that all current ICF-based participation measures cover ICF chapters 6–9, whereas they diverge in the coverage of the other ICF chapters (5). This suggests that the ICF categories classified in these chapters 6–9 are the most characteristic of participation.

Another issue is that it is not specified in the ICF whether participation problems should be rated from an outsider (“objective”) or an insider (“subjective”) perspective (7). Objective rating concerns observable behaviours, such as the number of hours a week doing paid work or the frequency of visiting friends in the previous 4 weeks. Subjective rating concerns the subjective appraisal of the person involved. The difference between objective and subjective rating is of importance, as these are, usually, only weakly related (11, 12). Objective ratings of participation appear straightforward as they concern observable behaviours. However, what constitutes appropriate participation is highly culture-dependent. It is therefore difficult to determine which behaviours fall inside or outside a “normal range” of objective participation because, in modern societies, a wide variety of lifestyles exist together (8, 13). For this reason, problems with participation as perceived by the person and satisfaction about performance might be at least as relevant as actual performance for those parts of life that involve choice instead of necessity (5, 8).

Finally, like any measure, a participation measure must be valid, reproducible, and responsive to be useful as an outcome measure (14). Many participation measures have been developed since the introduction of the ICF and have been tested for validity, but their responsiveness has rarely been established (5).

In response to this lack of data, we started a prospective multi-centre study to identify a valid and responsive instrument to measure participation outcomes of outpatient rehabilitation (15). We searched for participation measures that satisfied the following criteria: (i) applicable in various diagnostic groups; (ii) feasible, being brief and suitable for self-report, for use in routine outcome monitoring; (iii) providing both objective and subjective ratings of participation; (iv) covering the ICF Participation chapters (7); and (v) having sound psychometric properties. Covering ICF Participation chapters was determined by linking the items to the ICF (16, 17). This linking was done by a single author (CZ) and, in case of any doubt, consensus was reached with the last author (MP). The few participation measures rating both objective and subjective participation unfortunately did not meet other selection criteria, and therefore, we selected 3 measures rating either objective or subjective participation and meeting other selection criteria (15). The Frenchay Activities Index (FAI) (18) was selected because it is part of the Dutch core set measurement instruments in post-acute stroke care and is therefore the only participation measure frequently used in clinical practice in The Netherlands. Eighty percent of the items were linked to categories in the ICF Participation chapters (17). Although the FAI has been developed for use in stroke patients, it has also been used in other diagnostic groups and in elderly people living in the community (4). The ICF Measure of Participation and Activities Screener (IMPACT-S) (10) was selected because it is the only participation measure that covers all Activities and Participation chapters of the ICF (7), for this study we only used the Participation subscale of the IMPACT (IMPACT-SP). All IMPACT-SP items have been linked to the Participation chapters of the ICF (5). The Participation Scale (19) was selected because it is the only participation measure that asks people to rate their participation using an explicit base of comparison, namely “the peer group”, which is defined as “those who are similar to the respondent in all respects (socio-cultural, economic and demographic) except for the disease or disability” (19). For the Participation Scale, 77.8% of the items cover the Participation chapters of the ICF (5). Finally, since we found no instrument measuring both objective and subjective participation and which satisfied most other criteria, we developed a new measure, the Utrecht Scale for Evaluation of Rehabilitation-Participation (USER-Participation) (20). The USER-Participation is based on the ICF and therefore all items cover the Participation chapters of the ICF. In an earlier study, the IMPACT-S, the Participation Scale, and the USER-Participation showed good reproducibility (21), but this is only indirect evidence for responsiveness. Evidence on responsiveness of the FAI is contradictory (4).

The aim of the present study was to compare the responsiveness of the FAI, the IMPACT-SP, the Participation Scale, and the USER-Participation in patients who took part in an outpatient rehabilitation programme by comparing effect sizes (ES) and standardized response means (SRM) of these measures in a longitudinal design. It is assumed that undergoing such a programme is associated with improved participation. This assumption enables a direct comparison between the measures, namely that the measure that shows the highest ES and SRM values can be considered the most responsive.

METHODS

Sample

Patients were selected from 5 rehabilitation centres in The Netherlands: De Hoogstraat (Utrecht), Sophia Revalidatie (The Hague and Delft), and Libra Zorggroep (Eindhoven and Tilburg). Patients were eligible to participate in the study when they: (i) started a multidisciplinary outpatient rehabilitation programme in the period from May 2008 until February 2009, which was expected to last for at least 4 consecutive weeks; (ii) were at least 18 years of age; and (iii) were able to read and write Dutch. Multidisciplinary was defined as the involvement of at least 2 different disciplines beside the physiatrist. Exclusion criteria were: (i) severe aphasia; (ii) severe cognitive impairments; or (iii) a fast progressive medical condition. Aphasia and cognitive impairments were assessed by the treating physiatrist.

Procedure

Eligible participants were informed by their physiatrist about the project and invited for the first measurement by first author. After signing informed consent, patients completed the first measurement within the first 2 weeks of the programme (t1), the second measurement at the end of the programme (t2), and the third measurement 4 months after t2 (t3). The end of the programme was defined as either completion of the programme or when the multidisciplinary programme switched to a monodisciplinary programme. The questionnaire was computer administered. T1 and t2 were completed at the centre in the presence of a research assistant. The research assistant assisted the patient in the use of the computer to complete the questionnaire, if necessary. T3 was sent by e-mail to all participants with internet access. In the e-mail was a link that could be used to log in on a website to complete the questionnaire. Participants without internet access received the questionnaire by regular post. Reminders were sent 2 weeks after the initial invitation to participants not returning the questionnaire. The study protocol was approved by the Medical Ethics Board of Rehabilitation Centre De Hoogstraat.

Instruments

The FAI measures lifestyle after stroke (18). It comprises 15 items on frequency of activities, for example “how often did you prepare the meal?” All items are rated on a scale with response options ranging from 0 to 3, in which a high score indicates a higher level of activity. The Dutch Modi-FAI was used, which has a shorter time frame of 4 weeks instead of 3–6 months and was developed to make it more suitable as an outcome measure of outpatient rehabilitation (22). A total score is calculated by adding up the item scores, with a total range from 0 to 45. Previous research showed a moderate ES (0.59) and SMR (0.5) of the FAI in stroke patients (23, 24). Cronbach’s α for the FAI ranged from 0.74 to 0.78 in the current study.

The IMPACT-S comprises items assessing experienced activity limitations and participation restrictions. Nine domain scores, 2 subtotal scores (for Activity and Participation) and 1 total score can be computed (10). In line with our conceptualization of participation, we only used the IMPACT-SP, which comprises 15 items, for example “do you experience any restrictions due to your health or disability with the acquisition of necessities?” All items are rated on a scale from 0 (cannot do that at all) to 3 (no limitations whatsoever). A total score is calculated by converting the summed score to a score on a 0–100 scale, in which a higher score indicates fewer participation restrictions. Internal consistency (α 0.92–0.93) and test-retest reliability (intraclass correlations (ICC) 0.90) of the IMPACT-SP score were excellent (10, 21). A very strong correlation (0.86) was found between the IMPACT-SP and the World Health Organization (WHO) Disability Assessment Schedule II (10). Cronbach’s α of the IMPACT-SP ranged from 0.84 to 0.89 in the current study.

The Participation Scale measures experienced participation restrictions (19). Eighteen items asses the level of participation compared with peers and, in case of a lower level of participation, the extent to which the respondent experiences this as a problem. For example “do you have opportunity to take care of yourself (appearance, nutrition, health, etc.) as well as your peers?” If sometimes or no: “how big a problem is it to you?” Both answers are combined in an item score between 0 (same level of participation) and 5 (lower level of participation and this is experienced as a large problem). A response option “not applicable” is available in case a respondent does not want or need to do the activity, and is scored and interpreted as 0. A total Participation Scale score is obtained as the sum of the item scores, ranging from 0 to 90, with a high score indicating severe participation restrictions. Originally the Participation Scale was an interviewer-administered instrument. For this study it was translated into Dutch and re-designed as a self-report measure in cooperation with the authors of this measure. The interview version of the Participation Scale has been found to be valid and reliable, with an α of 0.92, a test-retest ICC of 0.83, and inter-tester ICC of 0.80 (19). The Dutch self-report version had a good test-retest ICC of 0.82 (21). Cronbach’s α of the Participation Scale ranged from 0.90 to 0.94 in the current study.

The USER-Participation consists of 31 items and measures 3 aspects of participation: frequency of behaviours, experienced participation restrictions, and satisfaction with participation (20). The Frequency scale consists of 12 items on frequency of vocational activity (4 items) and leisure and social activity (8 items). For example “how many hours per week do you spend on household duties” or “how often did you do sports or any other physical exercise in the last 4 week?” Each item is scored from 0 (none at all) to 5 (36 h or more/19 times or more). The Restriction scale contains 10 items asking for experienced participation restrictions as a result of the health condition. For example “does your illness or condition currently limit you in doing household duties?” Each item score ranges from 0 (not possible) to 3 (without difficulty). A “not applicable” option is available for every item and can be used if the item is not relevant to the person or if experienced restrictions are not related to the person’s health status or disability. The Satisfaction scale consists of 9 items asking for satisfaction with various aspects of participation, for example “how satisfied are you with work, education or household duties?” Items are rated on a scale from 0 (very dissatistied) to 4 (very satisfied). A “not applicable” option is available for the items on vocational activity and partnership relation. The sum scores for the Frequency, Restrictions, and Satisfaction scales are based on the items that are applicable and each sum score is converted to a score on a 0–100 scale, higher scores indicating good levels of participation (higher frequency, less restrictions, higher satisfaction). The USER-Participation showed generally good reproducibility (ICC 0.65–0.85) (21). Cronbach’s α of the USER-Participation in the current study ranged from 0.60 to 0.75 for the Frequency scale, from 0.85 to 0.91 for the Restrictions scale, and from 0.78 to 0.88 for the Satisfaction scale.

Finally, physical and cognitive independence and subjective complaints (pain, fatigue, and mood) were measured with the Utrecht Scale for Evaluation of Rehabilitation (USER) (25). The physical independence scale consists of 14 items on mobility and self-care and the cognitive independence scale consists of 10 items on communication, applied cognition, and behaviour. Items are scored from 0 (with help, <50% independent) to 5 (without difficulty, without aids or adaptations). Subjective complaints consist of pain (1 item), fatigue (1 item) and mood (4 items) and are rated on a scale from 0 (not at all) to 100 (worse/most imaginable). The physical and cognitive independence score are the sum scores of all items in that scale (score range 0–70 and 0–50, respectively). The mood score is calculated by taking the mean of the 4 items. The USER is a reliable measure and the physical and cognitive independence scales showed very high correlations with the Barthel Index and Functional Independence Measure (range 0.84–0.94) (25).

Statistical analyses

All analyses were performed with the data of patients who completed all 3 measurements (n = 395). Floor and ceiling effects were considered present if 15% of respondents scored the lowest or highest score on a scale, respectively (26). Score distributions were considered normal if the skewness was between –1 and 1. Responsiveness of the measures was compared using the ES and the SRM. The ES is calculated as the mean difference score divided by the standard deviation (SD) of the baseline score (27). Following Cohen, an ES of 0.2 is considered as small, 0.5 as moderate, and 0.8 as large (28). The SRM is calculated by dividing the mean difference score by the SD of the difference score (29). Cohen’s interpretation of the ES has often been applied to the SRM (30), although SRM and ES values are statistically not equivalent (31). ES and SRM for the total group and for the different diagnostic groups were calculated for t1–t2 and t2–t3. Since a large variety of diagnoses was seen, these were categorized as: (i) musculoskeletal, including multiple trauma, amputation, hand injury, etc.; (ii) brain injury, including stroke and traumatic brain injury; (iii) other neurological disorder, including multiple sclerosis, spinal cord injury, and neuromuscular diseases; (iv) heart condition; (v) chronic pain; and (vi) other. Data were analysed using SPSS 16.0.

RESULTS

A total of 509 patients were included in the study, of whom 427 (83.9%) completed t2, and 395 (77.6%) completed the t3 measurement. Mean time between t1 and t2 was 135 days (SD = 72) and between t2 and t3 was 128 days (SD = 15). Reasons for drop-out were: early cessation of the rehabilitation programme due to hospitalization or worsening of the condition (n = 17), not finishing the multidisciplinary rehabilitation programme in time for the study (n = 21), refusal (n = 47), death (n = 2), and unknown reasons (n = 27). Two-thirds (261/395) of all participants completed the internet questionnaire at t3. Characteristics of participants who completed all 3 measurements are shown in Table I. Between t1 and t2 the participants showed a moderate improvement in physical independence (ES 0.32; SRM 0.52) and a small improvement in cognitive independence (ES 0.20; SRM 0.27).

Score distributions of all participation measures are shown in Table II. There were no floor or ceiling effects, except for the USER-Participation Restriction scale on t3 and for the IMPACT-SP on t2 and t3, which showed a slight ceiling effect (18.7%, 15.4%, and 15.0%, respectively).

Responsiveness parameters for t1–t2 and for t2–t3 for the total group are displayed in Table III. The largest values for responsiveness were found between t1 and t2. Responsiveness values between t2 and t3 were negligible. SRM values were slightly higher than ES values. The USER-Participation Restriction scale and the IMPACT-SP showed the highest SRM and the USER-Participation Frequency scale showed the lowest SRM.

Responsiveness parameters for the different diagnostic groups are shown in Table IV. The USER-Participation Restriction scale was the most responsive measure in most diagnostic groups, while the Participation Scale and the USER-Participation Frequency scale were least responsive across diagnostic groups. ES and SRM values were largest in the Brain injury group, and smallest in the Chronic pain group.

DISCUSSION

In this study, the USER-Participation Restriction scale showed the highest responsiveness, both in the whole group and in different diagnostic groups.

This is the first study providing data on responsiveness of the IMPACT-SP, the Participation Scale and the USER-Participation. A few limitations apply to this study. First, an external criterion of change in participation was not included, so that it was not possible to compute and compare minimal important change (MIC) values of these measures. Further research is necessary to establish MIC values of the various measures, although this methodology itself has not been established yet (32). Secondly, there was heterogeneity of administration at t3, as part of the questionnaires was computer-administered and others were sent by ordinary post. However, it has been found that internet and posted paper questionnaires are answered similarly (33). Finally, this study was performed in persons with mild physical and cognitive limitations. The latter was necessary because the study used a self-report questionnaire. Further research is necessary to establish the responsiveness of these measures in persons with severe disabilities.

The responsiveness of the FAI has been studied previously (23) in a stroke population and that study showed a higher responsiveness of the FAI (ES 0.59) than the present study found in the Brain injury group (0.46). However, in that study the ES was calculated using the absolute value of score differences, not taking into account that this score difference can either be a decline or an improvement. In the present study some patients showed a decrease in the FAI score, resulting in smaller mean difference scores and thereby in lower, but more realistic, responsiveness values. If we had used the absolute value of the score difference to calculate the ES, the value would have been 0.66. The reason for this value being somewhat higher compared with the study of Schepers et al. (23) might be that our sample received a rehabilitation programme compared with no rehabilitation programme in the study of Schepers et al. In contrast, a recent study in stroke survivors (24) showed a slightly lower SRM (0.5) for the FAI than this study (0.63). This might be due to the longer intervention period, probably causing larger mean differences, in the present study.

The IMPACT-S, Participation Scale, and the USER-Participation were previously tested for reproducibility, which is an indirect measure of responsiveness (21). The mean score differences of all measures in the present study were larger than the smallest detectable change at group level found in the reproducibility study, but were smaller than the smallest detectable change at individual level. This means that the responsiveness of these measures is sufficient for evaluation studies, but that caution is needed if they are applied in individual patient care.

The responsiveness data for the measures included in this study can be compared with data on responsiveness of other participation measures that did not fit our selection criteria. The responsiveness of the London Handicap Scale (34) was examined in persons with progressive multiple sclerosis in inpatient rehabilitation (35). A small ES of 0.23 was found, but the inpatient setting is not the ideal context to examine responsiveness of a participation measure, since different aspects of participation (like grocery shopping, working, and household tasks) are not applicable. The responsiveness of the Impact on Participation and Autonomy (IPA) (36) was studied in a multi-diagnostic group of 49 persons following an outpatient rehabilitation programme with measurements at the start and 3 months later. Small SRM values (range 0–0.28) were found in the whole study group (37). We did not select the IPA for this study because it measures another construct, i.e. autonomy, and because the questionnaire is relatively difficult to complete. The responsiveness of the Participation Measure for Post-Acute Care (PM-PAC) (38) was tested in a multi-diagnostic sample of 94 patients who were discharged from an inpatient rehabilitation programme and of whom 83% received some form of outpatient rehabilitation services (39). The PM-PAC showed higher SRM values (range 0.65–0.82) than the responsiveness values found in the present study (40). We did not include the PM-PAC in this study because with a total of 52 items the measure is quite long. Furthermore, the PM-PAC includes questions on performance and satisfaction, but satisfaction is only asked regarding interpersonal relationships and the satisfaction items do not make up a separate scale.

In the study group as a whole, responsiveness parameters were small to moderate. Responsiveness parameters for the scales of the USER-Participation and the USER are similar. This might be due to the heterogeneity of the population. Heterogeneity in diagnosis may result in larger SD values of baseline and change scores, and thereby in smaller ES and SRM values. Therefore, we analysed responsiveness parameters separately for different diagnostic groups. The figures were highest in the Brain injury group, and lowest in the Musculoskeletal group and the Chronic pain group. These differences might be explained by differences in the goals, contents and intensity of rehabilitation programmes for different diagnostic groups. For example, acceptance of pain and maintaining a balance between activity and rest as treatment goals in persons with chronic benign pain are more likely to result in improvement of satisfaction with participation than in a higher frequency of participation. This example underscores that more participation is not necessarily better (13).

The responsiveness parameters were largest for the rehabilitation period and negligible for follow up. In The Netherlands, there is little financial pressure to limit the duration of rehabilitation, so that treatment can continue until the desired level of functioning is reached. A further improvement of participation after finishing the rehabilitation programme was nevertheless expected because patients might have less time and energy to participate during the rehabilitation programme itself, but such an improvement was not found. In retrospect, this might be attributed to the long duration and gradually decreasing intensity of outpatient rehabilitation in The Netherlands, so that most patients might already have reached their desired level of participation and treatment frequency is already low at the moment of discharge. If so, the stability of participation scores during the follow-up period might imply that patients are able to maintain their level of participation after finishing the rehabilitation programme.

ACKNOWLEGDMENTS

We would like to thank all research assistants, Petra van Eerdewijk, Hanneke Roelse, Aja Kuin (De Hoogstraat), Magda van Duinen (Blixembosch), Judith Rink, Judit Kleijnen, Anja van der Bruggen (Leijpark), Saskia Revier, Rixt Sikkes and Annette van Dijk (Sophia Revalidatie), for their enthusiasm and effort to reach as many participants as possible and for carefully collecting data. We would like to thank Caroline Schafrat and Jolanda Hennink for their creative solutions concerning the logistics of the project.

REFERENCES