Luzius Mader, MSc1,2, Marcel W. M. Post, PhD1,3,4, Carolina S. Ballert, MSc1,2, Gisela Michel, PhD2, Gerold Stucki, MD, MS1,2 and Martin W. G. Brinkhof, PhD1,2

From the 1Swiss Paraplegic Research (SPF), Nottwil, 2Department of Health Sciences and Health Policy, University of Lucerne, Lucerne, Switzerland, 3Brain Center Rudolf Magnus and Center of Excellence for Rehabilitation Medicine, University Medical Center Utrecht and De Hoogstraat Rehabilitation, Utrecht and 4Center for Rehabilitation, University of Groningen, University Medical Center Groningen, Groningen, The Netherlands

OBJECTIVE: To examine the metric properties of the Utrecht Scale for Evaluation of Rehabilitation-Participation (USER-Participation) in persons with spinal cord injury in Switzerland from a classical and item response theory perspective.

DESIGN: Cross-sectional survey.

SUBJECTS: Persons with spinal cord injury living in the Swiss community (n = 1,549).

METHODS: Score distributions and internal consistency were evaluated using Classical Test Theory. The Restrictions and Satisfaction scales were subjected to Rasch analysis. Anchored analyses were performed to account appropriately for items with structural missing.

RESULTS: Internal consistency was good for the Restrictions (α = 0.90) and Satisfaction scales (α = 0.90), but not for the Frequency scale (α = 0.65). Rasch analyses showed acceptable model fit for the Restrictions and Satisfaction scales after collapsing response categories of some items and merging some items into testlets. Differential item functioning was small. Anchoring allowed inclusion of the item work/education in the Restrictions scale and work/education and/or partner relationship in the Satisfaction scale.

CONCLUSION: The Restrictions and Satisfaction scales of the USER-Participation showed satisfactory metric properties. The Frequency scale showed fewer optimal properties, but nonetheless provides important additional information regarding participation. Conversion tables were performed to transform USER-Participation raw scores into a 0–100 interval scale using Rasch-based ability estimates for use in epidemiological studies.

Key words: spinal cord injury; social participation; validation studies; psychometrics; Rasch analysis; disability evaluation.

J Rehabil Med 2016; 48: 165–174

Correspondence address: Martin W. G. Brinkhof, Swiss Paraplegic Research, Guido A. Zäch Strasse 4, CH-6207 Nottwil, Switzerland. E-mail address: martin.brinkhof@paraplegie.ch

Accepted Jun 22, 2015; Epub ahead of print Feb 1, 2016

INTRODUCTION

Spinal cord injury (SCI) is associated with substantial changes in motor, sensory and autonomic functioning, resulting in high levels of disability (1). However, life expectancy and community discharge among persons with SCI have increased in recent decades due to improvement in acute medical treatment and rehabilitation care (1, 2). Community integration, social participation and quality of life are currently viewed as the ultimate goals of rehabilitation (3–5). Persons with SCI may, however, still experience restrictions or barriers to participation in different domains, including employment or social-recreational activities (4, 6, 7). Inadequate levels of social participation may be related to impaired life satisfaction, emotional and physical well-being (6, 8–10). The concept of participation was introduced in the International Classification of Functioning, Disability and Health (ICF) and is defined as “involvement in a life situation” (11). Major components of social participation include home and family roles, other productive roles (e.g. work, school, volunteering), social network, and leisure activities (3, 12).

The Utrecht Scale for Evaluation of Rehabilitation-Participation (USER-Participation) is a newly developed International Classification of Functioning, Disability and Health (ICF)-based participation measure assessing objective and subjective participation (13). It covers ICF chapters 6–9, described previously as the most characteristic of participation (14). The USER-Participation consists of 32 items and assesses 3 aspects of participation: Frequency, experienced Restrictions, and Satisfaction (15). An earlier version of the USER-Participation has been shown to be a valid measure in rehabilitation outpatients in the Netherlands (13). In this study, construct, concurrent, and discriminative validity were generally good (13). These findings were supported by a validation study of the slightly modified final version of the USER-Participation in an SCI population by van der Zee et al. (15). Reproducibility and responsiveness of the USER-Participation have also been shown to be generally satisfactory (16–18). To our knowledge, the USER-Participation has only been validated in the Netherlands by using Classical Test Theory (CTT). Consequently, its metric properties have to be further examined in different settings to support the validity of the USER-Participation in an international context, including the use of Rasch analysis as an emerging standard in questionnaire development (19). In addition, no information on the usefulness of the USER-Participation in evaluating population-based data on participation in the context of epidemiological studies is available. The overall objective of the present study was to examine the metric properties of the USER-Participation from a classical and item response theory perspective. Specifically, the basic metric properties of the USER-Participation scale were evaluated by investigating their fit to the Rasch model and in order to provide conversion tables for conversion of raw scores into Rasch-derived ability scores.

METHODS

Sample and procedure

Cross-sectional data from the community survey of the Swiss Spinal Cord Injury Cohort Study (SwiSCI) were analysed. The SwiSCI community survey includes persons with a diagnosis of traumatic or non-traumatic SCI, aged 16 years or older with a permanent residency in Switzerland (20). Exclusion criteria were congenital conditions leading to SCI (e.g. spina bifida), new SCI in the context of palliative care, neuro­degenerative disorders (e.g. multiple sclerosis) and Guillain-Barré syndrome (20). The recruitment process for the SwiSCI community survey is outlined elsewhere (21). The survey contained 3 subsequent modules that were sent out with an interval of approximately 3 months (20). Participants were invited to complete the questionnaire on paper, online or via a telephone interview. The focal data for the present study were collected in the second module or Basic Module (20). In total 3,144 individuals with SCI were invited to participate in the SwiSCI community survey and a representative sample of 1,549 participated in the Basic Module, signifying a response rate of 49.3% (21). The SwiSCI community survey was formally approved by the respective regional Research Ethics Committees. All participants gave written consent for the anonymous use of their data.

Instrument

The USER-Participation is a self-report questionnaire of 32 items with 3 separate scales: Frequency, Restrictions, and Satisfaction (13, 15). The Frequency scale consists of 11 items, 4 items on vocational activities and 7 items on leisure and social activities. The 4 items on vocational activities address the number of hours spent per week, and are scored on a 6-point ordinal scale from 0 (not at all) to 5 (36 h or more). The 7 items on leisure and social activities address the frequency in the last 4 weeks scoring from 0 (not at all) to 5 (19 times or more). The Restrictions scale consists of 11 items that address activities that may be restricted by their health condition. The perceived difficulty in performing the activity is rated on a 4-point scale, ranging from 0 (not possible at all) to 3 (no difficulty at all). If any item is not relevant to the person or the restrictions are not related to the person’s health status, the option “not applicable” is available. The Satisfaction scale consists of 10 items on satisfaction with vocational, leisure and social activities. Items are rated on a scale from 0 (not satisfied at all) to 5 (very satisfied). For the items on vocational activities and partner relationship a “not applicable” option is available. The sum score of each scale is based on all applicable items and is converted to a 0–100 scale, with higher scores indicating better participation (more time spent/higher frequency, less restrictions, higher satisfaction). There is no total USER-Participation score aggregating the scores from each scale.

Statistical analyses

All analyses were performed using Stata version 13 (22) and R version 3.1.0 (23), in particular with the R-package TAM for Rasch analysis (24). We evaluated score distributions with descriptive statistics. Floor and ceiling effects were considered present if > 15% of participants achieved the minimum or maximum score on a scale (25). Applying the classical test theory (CTT) approach, we evaluated internal consistency with Cronbach’s alpha and item-rest correlations. A Cronbach’s alpha of > 0.70 was considered satisfactory (25) and item-rest correlations of 0.30 were used as the minimum standard as in previous studies (13, 15).

We established the interval properties of the Restrictions and Satisfaction scales with a model from the Rasch family, namely the Partial Credit Model (PCM) for polytomous response data (26). Rasch analysis allows the verification of important properties of measurement scales, such as monotonicity, independence, and unidimensionality. Rasch analysis substantiates whether scale properties are met or whether refinement of the scale is needed to improve internal consistency. Finally, the findings of the present Rasch analysis may contribute to the existing pool of evidence supporting use of the USER-Participation scale for measuring participation in the context of rehabilitation medicine (13). In Rasch analysis, the probability of a person scoring on an item can be modelled with a logistic function of the difference between the person’s ability and item difficulty (27). We did not perform Rasch analysis of the Frequency scale, since frequency of performance is not a good indicator of item difficulty (but rather is influenced by personal preference), and is not scaled on one dimension because different participation domains cannot be performed at the same time (15, 19).

Reliability was quantified with the Person Separation Index (PSI), which can be interpreted as Cronbach’s alpha (27). The fit of the data to the Rasch model was examined with infit and outfit mean square statistics. Infit and outfit mean square statistics ranging between 0.80 and 1.20 were considered acceptable (28). In addition, the Smith Interval was calculated to provide a sample size adjusted range of fit (29). A correct order of thresholds is present if each response option demonstrates the highest probability at a specific range of the measured construct with successive thresholds found at increasing levels of difficulty of the item. Disordered thresholds might be collapsed to refine the scale. Furthermore, targeting was investigated to verify that person abilities match the difficulties of the items, with good targeting signifying that both the mean item difficulties and the mean person abilities are centred towards zero. We analysed local item dependency (LID) to identify potential patterns in the item residuals. LID is present if standardized person-item residuals show correlations of > 0.20. To address local dependency, we added testlets that were created through the aggregation and summation of dependent items. We assessed dimensionality with principal component analysis (PCA) of residuals. Unidimensionality is present if the second eigenvalue is < 1.40 (30). Differential item functioning (DIF) was analysed to test whether item responses are influenced by group characteristics with an analysis of variance (ANOVA) of the residuals (27). The level of significance of the DIF analysis is Bonferroni corrected for repeated testing. In the present study, we investigated DIF for age and sex (male/female), as well as for type of SCI (paraplegia/tetraplegia), degree of SCI (complete/incomplete), age at injury, and time since injury. Invariance was also tested for language (German/French) and questionnaire type (non-paper-form/paper-form). We split age, age at injury, and time since injury at the median. The items work/education and partner relationships were marked as not applicable by 21.3% and 15.3% of the participants on the Restrictions scale, and by 28.4% and 19.0% on the Satisfaction scale. To account for the structural missingness in these 2 items anchored analyses were performed (31). In a first step, Rasch analysis of the Restrictions and Satisfactions scales was performed excluding the 2 items work/education and partner relationship. After the necessary adjustments to attain good fit to the Rasch model, the corresponding item difficulties and thresholds were computed. In a second step, for both scales the computed item difficulties and thresholds of these respective sets of items served as anchors to calculate the difficulties of the items work/education and partner relationship for responders to these items. This was done first separately for each item and then for responders to both items. Anchoring thereby allows a refinement of the person ability estimates for the subgroup with additional information on restriction due or satisfaction with work or partner. Conversion tables for all tested models were developed to transform responses obtained on different items into a common scale using Rasch-based ability estimates.

RESULTS

Characteristics of the study population

Demographic and SCI characteristics of the study population are shown in Table I. The mean age of our sample was 52.4 years and 71.5% were men. Two-thirds were paraplegics and the mean time since injury was 16.9 years.

Internal consistency and score distributions of items and scales

Score distributions for items and scales are presented in Table II. No scale showed floor or ceiling effects. The Frequency scale showed a Cronbach’s alpha of 0.65 with several item-rest correlations below 0.30. The Restrictions and Satisfaction scales showed good reliability and high item-rest correlations (Table II).

Rasch analysis of the Restrictions and Satisfaction scales with all items

Rasch analysis of the Restrictions and Satisfaction scales with all items revealed substantial misfit, disordered thresholds for several items and local item dependencies (data not shown). The most problematic items were work/education and partner relationship in both scales. No model including these 2 items showed acceptable fit. Both items have a substantial amount of missing values (24.2% up to 37.6%) because these were not applicable to many participants.

Rasch analysis of the anchor items of the Restrictions and Satisfaction scales

Acceptable model fit was achieved for the items of both scales when excluding work/education and partner relationship (Tables III and IV). Regarding the Restrictions scale, the response options with difficulty and no difficulty at all had to be collapsed for correct ordering of the items visits from family or friends and telephone/computer contacts (Fig. 1). No disordering was found in the Satisfaction scale (Fig. 2). The analysis of residuals identified LID between the items going out and outdoor activities (r = 0.31) of the Restrictions scale. Concerning the Satisfaction scale we found LID between the items family relationships and friends/acquaintances (r = 0.36) and outdoor activities and going out (r = 0.28). Consequently, we formed a testlet with the items outdoor activities and going out for both scales. We also formed a testlet with the items family relationships and friends/acquaintances of the Satisfaction scale. The final estimates are displayed in Tables III and IV. Reliability of the anchor items of the Restrictions and Satisfaction scales was good, with a Person Separation index (PSI) of 0.80 and 0.84, respectively (Table V). Mean person ability was –0.13 and –0.04 for the Restrictions and Satisfaction scales, with corresponding item difficulties of –1.94 and –0.84 (Table V). DIF for the anchor items was generally low (Table VI). Concerning the Restrictions scale, DIF was mainly associated with SCI characteristics and age at SCI, whereas the Satisfaction scale showed DIF for sex and language (4 items in total).

Anchored analysis of the Restrictions scale

Adding the item work/education to the previously calculated item difficulties of the anchor items for responders to this item resulted in good item fit (Table III). However, the outfit, which is a more outlier sensitive statistic, was below the acceptable boundary of 0.8 for the telephone/computer contact item in all analyses of the Restrictions scale. Also, the item visits to family and friends showed infit and outfit values that were slightly below the cut-off values in the analysis of the anchor items with the work/education item. Low infit or outfit values indicate that the items overfit to the model, which is less a threat to a scale than underfit. The response options with help from others and with difficulty of the item work/education had to be collapsed for correct ordering (Fig. 1). No LID was identified. The reliability of the anchor items and work/education was good, with a PSI of 0.84 (Table V). The DIF found in the analysis with anchor items could not be confirmed in the anchored analysis (Table VI). Only for the item housekeeping did the DIF observed in the anchor item analyses persist. No model including the item partner relationship for those participants that responded to this item showed acceptable fit statistics in this scale (results not shown). Conversion keys are provided in Table VII.

 

Fig. 1. Person-item map of the Utrecht Scale for Evaluation of Rehabilitation (USER) – Participation Restrictions scale.

Anchored analysis of the Satisfaction scale

Adding the item work/education and partner relationship (separately or together) for responders to 1 or both of those items resulted in acceptable item fit for all 3 models, i.e. anchor items plus work/education, anchor items plus partner relationship as well as for all items of the Satisfaction scale together (Table IV). Regarding the item work/education and partner relationship the response categories dissatisfied, neutral, and satisfied had to be collapsed for correct ordering (Fig. 2). All 3 models showed good fit except for 1 item (which overfit) (Table IV), good reliability (Table V) and no LID. DIF observed among the anchor items persisted only for the testlet relations and the testlet outdoors in the anchored analysis (Table VI). The testlet outdoor infit and outfit was below the acceptable boundary of 0.8 for all 3 analyses of the Satisfaction scale. Conversion keys for all models are provided in Table VIII.

 

Fig. 2. Person-item map of the Utrecht Scale for Evaluation of Rehabilitation (USER) –Participation Satisfaction scale.

Dimensionality of the USER-Participation

After refinement of the scales, unidimensionality of the USER-Participation Restrictions and Satisfaction scales were confirmed with a principal component analysis of the residuals with second factor eigenvalues of 1.38 and 1.26, respectively. The PCA of the residuals of the anchored Rasch analyses also showed unidimensionality with second eigenvalues of <1.40 for both scales.

DISCUSSION

This study evaluated the metric properties of the USER-Participation in a Swiss SCI population from a classical and item response theory perspective. Generally satisfactory basic metric properties of the USER-Participation were found. The Restrictions and Satisfaction scales had satisfactory internal consistency with excellent reliability according to De Wolf et al. (32). Only the Frequency scale showed internal consistency below the recommended 0.70 level, and low correlations between the items on vocational activities. These findings are in line with previous validation studies of the USER-Participation (13, 15). The underlying reason is probably that performing one activity competes with performing another, given the boundaries of limited time availability. High internal consistency has been described to be less relevant if items reflect different aspects of a complex construct, such as participation (15, 19, 33). Given that different participation activities are segregated, the Frequency scale is reminiscent of a formative measurement model. In formative measurement models, items jointly contribute to a latent construct, while the all-inclusive meaning is derived from the measured items (34, 35). In the case that the formative latent construct is represented by exclusive indicators reflecting heterogeneous causes (34, 36) formative variables are not necessarily correlated. The formative measurement model concept is applicable to the Frequency scale of the USER-Participation, in which all items can jointly be considered as useful indicators of the frequency of participation, while the frequency of distinct items, such as, for instance, going out and paid work, are not inevitably correlated. Regarding a formative measurement model, the Frequency scale provides important descriptive information with respect to engagement in certain activities. However, our results indicate that this scale does not provide a valid summary score for the degree of participation in the Swiss SCI population.

In the present study the anchor method was applied to appropriately account for items within the USER-Participation Restrictions and Satisfaction scales that are not applicable to some of the participants. This approach is important in Rasch analysis, because ignoring structural missing by the recoding and treatment of not applicable items as missing values may lead to bias and misleading inference regarding ability scores. Of note, this issue has been ignored in Rasch analysis of other participation measures, such as the World Health Organization Disability Assessment Schedule (WHODAS II) and the Impact on Participation and Autonomy scale, which similarly faced structural missing regarding work-related items (37, 38). By excluding such key domains of participation, content validity may be affected. Lund et al. suggested a separate scale for work-related items for the Impact on Participation and Autonomy scale (39). Regarding the Participation scale, best model fit was also found for a separate scale with work-related items (40, 41). The methodology used in the present study offers a valid alternative to the exclusion or separation of problematic (but regarding content validity, important) items.

Disordered thresholds, local item dependencies and differential item functioning were found for some items of the Restrictions and Satisfaction scales that could be solved by collapsing response categories and forming testlets. A possible explanation may be that participants were unable to differentiate between response options. Poor item fit indicates that the performance on these items cannot adequately be predicted by the person’s ability derived from the Rasch model. From a conceptual point of view, the items going out and outdoor activities as well as the items family relationships and friends/acquaintances cover similar aspects of participation and forming a testlet solved these problems. The anchoring approach worked well for the Satisfaction scale. All tested models of the Satisfaction scale, i.e. anchor items plus work/education, anchor items plus partner relationship as well as for all items, showed generally good fit (with few exceptions), good reliability, no LID and unidimensionality. Concerning the Restrictions scale, no items showed underfit in any of the analyses, and 2 items showed overfit to the model: telephone/computer contact in both analyses and visits to family and friends in the analysis of the anchor items + work/education. The anchoring approach did not work for the item partner relationship. The poor fit of the item partner relationship of the Restrictions scale may result from diverging interpretations. Participants might understand this question in the context of restrictions in creating a relationship or in maintaining a relationship. Participants might also interpret this item as a more physical problem with intercourse. However, this item has to be further validated in future studies.

The analysis of DIF revealed no systematic DIF for all models of both scales. The Restrictions scale was more sensitive to injury-related characteristics, whereas the Satisfaction varied mainly across socio-demographic or questionnaire-related variables. The suitability of DIF analyses (42, 43) regarding SCI level and SCI degree in the Restrictions scale might be questionable since the latent trait is intuitively associated with the severity of SCI. This has also been discussed extensively in a study investigating the metric properties of items included in the analysis of the functioning profile and the self-report version of the Spinal Cord Independence Measure (SCIM-SR) (44, 45) in this issue. Furthermore, future research should clarify whether DIF related to questionnaire language is related to translation issues or cultural differences.

Targeting showed a shift towards higher abilities for both scales, indicating that items do not completely match the expected participant’s abilities. This is also indicated by the higher proportion of maximum scores compared with minimum scores for both scales. Person ability and item difficulty patterns might diverge due to the heterogeneous characteristics of the study population (32). Among all participants, 68.6% were paraplegics and 57.7% had an incomplete lesion, which may partially explain the high ability level of our sample in regard to restrictions and satisfaction with participation. Social support may play a key role with regards to participation, in that individuals with insufficient access to social support may be more likely to perceive participation problems (46). Important contextual factors further include assistive devices, such as wheelchair or modern information and communication technology, which enable individuals with severe SCI to work, chat with friends or perform leisure activities.

A major strength of the present study is the use of an anchoring approach to include the items that inherently encounter structural missing, such as work/education and partner relationship. This study presents an approach to how to deal with such items and provides conversion keys to enable use of the USER-Participation for the collection of data in epidemiological studies. Furthermore, the sample size in the present study is large. The examination of properties of the USER-Participation from a Classical Test Theory and Item Response Theory perspective allowed us to go beyond the mere testing of the reliability and item total correlations and included a probabilistic approach of measurement, which takes into account the difficulty of items and ability of persons in evaluating the consistency of the metric properties of the questionnaire. DIF analysis with regard to language and questionnaire type is a further strength providing justification for the use of an instrument in a multilingual population, such as in Switzerland. Also, invariance of the mode of administration confirms that derived ability scores are unbiased with regards to the response mode chosen by participants.

In conclusion, this study showed that the basic metric properties of the USER-Participation Restrictions and Satisfaction scales are satisfactory. However, internal consistency of the Frequency scale is limited. Nevertheless, our findings revealed that the item partner relationship requires further attention. Future research should focus on the use of the USER-Participation as an outcome measure in different clinical settings, epidemiological studies and diagnostic groups.

ACKNOWLEDGEmeNTS

The authors would like to thank Veronika Lay for her assistance in data analysis.

Financial support: This study was funded by Swiss Paraplegic Research, Nottwil, Switzerland. G. Michel received structural financial support from the Domarena Foundation.

The authors declare no conflicts of interest.

REFERENCES