Annelies C. De Wolf, MPH1,2, Robyn L. Tate, PhD1,2, Natasha A. Lannin, PhD1,3,4, James Middleton, PhD1, Amanda Lane-Brown, PhD1 and Ian D. Cameron, PhD1,2

From the 1Rehabilitation Studies Unit, Northern Clinical School, Sydney Medical School, University of Sydney, 2Royal Rehabilitation Centre Sydney, Ryde, 3Faculty of Health Sciences, La Trobe University and 4Alfred Health, Melbourne, Sydney

OBJECTIVE: To evaluate the reliability and validity of WHODAS II within the spinal cord injury population.

SUBJECTS: Sixty-three people with traumatic spinal cord injury.

METHODS: The World Health Organization Disability Assessment Scale II (WHODAS II), Craig Handicap Assessment and Reporting Technique, and Medical Outcomes Study 36-item Short-Form Health Survey (MOS SF-36) were administered at 2 years post discharge from rehabilitation. Distribution, reliability, discriminant validity, and convergent/divergent validity were evaluated using classical tests. Rasch analyses were applied to assess dimensionality, item spread, and person/item reliability.

RESULTS: Cronbach’s alpha coefficients ranged from 0.61 (getting around) to 0.97 (participation). Ceiling effects were present in 4 out of 6 domains. WHODAS II discriminated between levels of impairment and work force status on ‘self-care’, ‘getting around’, ‘life activities’, and total score. Correlations with MOS SF-36 supported convergent/divergent validity. Five items didn’t fit the Rasch model. The item/person map reveald a shortage of items able to differentiate the more able person. WHODAS II demonstrated good person and item separation and reliability.

CONCLUSION: This study provides preliminary support for reliability and validity of WHODAS II in a spinal cord injured population. Limitations were noted for dimensionality and item person distribution. Findings need to be confirmed in larger samples.

Key words: spinal cord injury; participation; disability; assessment; psychometric properties; Rasch analyses.

J Rehabil Med 2012; 00: 00–00

Correspondence address: Annelies C. De Wolf, MPH, Rehabilitation Studies Unit, University of Sydney and Royal Rehabilitation Centre Sydney, 59 Charles Street, Ryde NSW 2112, Australia. E-mail: anneliesd@med.usyd.edu.au

Submitted August 28, 2011; accepted March 30, 2012

Introduction

Returning to full participation in daily life is the ultimate goal of rehabilitation. Yet, activity and participation is fraught with challenges, particularly in the spinal cord injury (SCI) population. Some of the unique challenges that present to this population are associated conditions (1), physical and/or structural barriers (2), lack of accessible transportation (3), limited work opportunities (4), limited social support (2), and secondary health complications (5, 6).

The World Health Organization Disability Assessment Scale II (WHODAS II) is an instrument that measures everyday functioning across 6 domains that correspond with the activities and participation components of the International Classification of Functioning, Disability and Health (ICF; WHO, 2001). It was designed as a generic measure, suitable for use with different health conditions, in different countries and cultures (7, 8). The WHO website advises that information from the WHODAS II can be used to identify needs, match patients to interventions, track functioning over time, and measure clinical outcomes and treatment effectiveness. Specific to the SCI population, the WHODAS II has attractive features compared to other SCI validated participation measures in that no other measure has a subscale structures matching directly with components of the ICF (9, 10).

Psychometric properties of the WHODAS II have been evaluated for a number of clinical conditions, including musculoskeletal diseases (11–14), chronic diseases (13, 15), psychiatric conditions (11, 13, 15), cancer (15), hearing loss (16), stroke (13, 15, 17), and spinal conditions (18, 19). Noonan et al. (20) recently reported on distribution, reliability, and construct validity of WHODAS II in 145 persons with SCI who were around 5 years post discharge from hospital. Dimensionality using item to scale correlations was assessed in a population of spinal conditions which included the same 145 persons with SCI, as well as persons with none neurological involved spinal degeneration and fractures (n = 400) (18). Psychometric properties for WHODAS II have generally been found adequate (21) although the presence of a ceiling effect has been identified as a concern (13, 20).

To our knowledge, only one other study examined the psychometric properties of the WHODAS II when applied to the population with SCI (20). Internal scale validity for this population has not been thoroughly examined. In our quest to find suitable measures of activity and participation for use in this clinical group, we seek to confirm and examine further whether the WHODAS II is a reliable and valid instrument to measure activity and participation in the spinal cord injury population. To determine the construct validity of the WHODAS II in this clinical group, both classical tests and item response theories were applied. Convergent/divergent validity was examined with reference to the Craig Handicap Assessment and Reporting Technique (CHART) and the Medical Outcomes Study (MOS SF-36).

Methods

Participants/study design

The data used to examine the psychometric properties of the WHODAS II were obtained from a prospective cohort study designed to evaluate community reintegration following SCI. (22) Participants (n = 75) who sustained traumatic SCI were recruited from three SCI rehabilitation units in Sydney, Australia (Prince of Wales Hospital, Royal North Shore Hospital, and Royal Rehabilitation Centre Sydney) between October 2003 and September 2007. Individuals were excluded from this study if they required permanent ventilation, or had (in addition to SCI) a severe traumatic brain injury (post traumatic amnesia > 7 days) or significant mental health disorder. The study sample consisted of 63 participants who were followed up at 2 years after discharge from the inpatient unit. The study was approved by the respective institutional ethics committees.

Outcome measures

WHODAS II is a 36-item generic measure of disability that examines difficulties in 6 domains of life during the previous 30 days (7): understanding and communicating (6 items), getting around (5 items), self-care (4 items), getting along with others (5 items), life activities (8 items), and participation in society (8 items). Each item is rated on a 5-point scale, from 1 (no difficulty) to 5 (extreme difficulty/cannot do). The instrument produces a total score (disability index) and 6 domain scores, ranging from 0 (best) to 100 (worst). The domain scores are transformed from the total raw score (sum of items) of each domain according to the following formula: Transformed score=[(actual raw score – lowest possible raw score) / (possible raw score range)] x 100. The disability score is calculated using the Statistical Package for the Social Sciences (SPSS) – (syntax available from World Health Organization). The disability score in this study was calculated from 32 items (omitting questions related to work/school because 44% of participants were not employed and/or not involved in school at 2 years post discharge). The self-administered version was used for the study. Item 2.5 ‘Walking a long distance such as a kilometer’ was reworded to ‘Walk or wheel a long distance such as a kilometer’.

The CHART is a 32-item scale specifically developed to measure handicap in people with SCI living in the community. The 6 dimensions (physical independence, mobility, occupation, societal integration, economic self-sufficiency, and cognitive independence) focus on objective, observable criteria that are easily quantifiable. 5 of the 6 domains (economic domain omitted) plus the total score of 5 domains were examined in the present study. Domain scores range from 0 to 100 (total score 0–500) with higher scores indicative of less handicap. The domain scores are calculated using a scoring formula that was calibrated on a non-disabled population (USA) where the majority receives a score of 100 (full test manual is available from www.craighospital.org). The CHART has demonstrated strong psychometric properties in the SCI population (23–25).

The Medical Outcomes Study 36-item Short Form Health Survey (MOS SF-36) is a generic, multidimensional, self-report health questionnaire. The SF-36 measures 8 health concepts or domains: physical functioning, role (physical) limitation, bodily pain, general health, vitality, social functioning, role (emotional) limitation, and mental health. Two component scores, physical and mental component score (PCS and MCS) can be derived from the domain scores (26). The domain and component scores are standardized to Australian norms with a mean score of 50 (standard deviation (SD) 10). The walk-wheel version (27) was used in the study, which replaces the walk language with ‘walk or wheeling’. The SF-36 has demonstrated acceptable psychometric properties within the SCI population (27, 28).

Data analysis

Statistical analyses for psychometric tests were performed using SPSS 18.0 for Windows. The percentage of participants with lowest and highest possible score were calculated with values greater than 15 percent considered to reflect floor and ceiling effects respectively. (29) Kolmogorov-Smirnov (K-S) tests examined the distribution of the domain and total scores. Because 4 out of 6 domains deviated from the normal distribution, non-parametric analyses were performed across all domains to allow uniformity of statistical methods. The equivalent parametric tests were carried out on the two domains and total score which were normally distributed to confirm strength of findings – results are reported when strength differs from non-parametric test results.

Internal consistency was evaluated for the total scale and for the domains using Cronbach’s alpha coefficient, where > 0.8 was considered excellent, 0.7 to 0.8 adequate, 0.6 to 0.7 questionable, and < 0.6 poor (30).

Discriminant validity was examined by i) comparing different impairment levels (high versus low), and ii) comparing those participants who were in the workforce (paid employment or volunteer) with those not in the work force. Mann-Whitney U analyses was applied and Hodges Lehman confidence intervals (31) were calculated to assist with interpretation of statistically significant between-group differences. It was hypothesized that persons with higher levels of impairment (tetraplegia AIS A-C) would have higher scores (more disability) on domains related to getting around, self-care, life activities, participation, and the total score than persons with low impairment SCI (tetraplegia AIS D and paraplegia). Likewise, we hypothesized that persons in the workforce would have lower scores (less disability) for domains related to getting around, self-care, life activities, participation, and total score.

To establish convergent/divergent validity, correlations were examined between domains of WHODAS II and domains of CHART and MOS SF-36 that were thought to have similar/dissimilar underlying constructs. Spearman correlation coefficients were interpreted according to the following criteria: > 0.80 = high, 0.61 to 0.80 = marked, 0.41 to 0.60 = moderate, 0.20 to 0.40 = fair, and < 0.20 = no relationship (17). It was hypothesized that moderate or stronger correlations would be found between domains of WHODAS II and domains of CHART and SF-36 with similar underlying constructs and that no relationship would be found between domains with dissimilar underlying constructs (Table IV). Convergent/divergent validity was rated sufficient if at least 75% of the a priori hypotheses were confirmed (29).

Construct validity was further investigated using Rasch analysis. Rasch analyses was chosen over confirmatory factor analysis (CFA) because of the information the former provides at the item level and thus allowed for a meaningful interpretation of the difficulty of the items and ability of persons. This was particularly of interest given the assessed ceiling effects. Additionally, the Rasch model (unlike CFA) has no problems accommodating the missing data from work-related items and therefore allowed the analyses to be carried out on the set of 36 items (32). Rasch rating scale analysis was conducted using the Winstep program (33) to evaluate WHODAS II in terms of the following:

i) Unidimensionality: to determine whether all items represent a similar construct. The infit mean square (MnSq) statistic was used with criterion for misfit set at > 1.4 (34). Misfit items were investigated in terms of participant characteristics and associated factors to seek an understanding for the reason that items may have misfit. No minimum MnSq fit cutoff was set because identifying overfit items (MnSq < 0.6), for the purpose of eliminating and/or replacing the item with more efficient ones, was not the objective of our study (34).

ii) Hierarchy of items: to determine the order of item difficulty in relation to the distribution of person ability and to identify item gaps the person-item map was visually inspected.

iii) Person reliability and separation: to determine the extent to which the WHODAS II distinguished among persons with different levels of functioning and participation (with separation index criterion set at ≥ 2.0, and reliability at ≥ 0.80). Discernible strata of person ability were calculated using the formula by Fisher (35): (4 x separation index + 1) / 3. Item reliability was calculated to examine the degree to which the item response categories reflected increasing levels of disability (item separation with the criterion set at ≥ 2.0, and reliability at ≥ 0.80).

Results

Demographic and injury variables of participants are presented in Table I.

Calculation of WHODAS II total and domain scores provided information about the self-rated activity and participation limitations of adults with SCI 2 years after discharge (Table II and III). The greatest limitations were found in the domains of ‘getting around’, ‘self-care’, and ‘life activities’. The least limitation was found for ‘understanding and communicating’. Within different levels of impairment, the single greatest limitation was found among individuals with tetraplegia (ASIA A–C) within the domain of ‘self-care’. The total score and two domains, ‘life activities’ and ‘participation’ were normally distributed (K-S > 0.05). The other domains were not normally distributed. Positive skewness with potential for ceiling effect (best score) were seen for ‘understanding and communicating’, ‘self care’, ‘getting along with others’, and ‘life activities’. There were no major floor effects.

Internal consistency

Internal consistency of WHODAS II was examined using the 32-item version (omitting questions involving work/school). WHODAS II total score, and the domains ‘understanding and communicating’, ‘self care’, ‘life activities’, and ‘participation’ demonstrated excellent internal consistency; internal consistency was adequate for the domain getting along, and questionable for the domain getting around.

Discriminant validity

Mann-Whitney U analyses were used to examine if the domains and total score differentiated individuals with high level impairment from low level impairment (Table III). WHODAS II discriminated as expected between high and low impairment for the domains ‘getting around’, ‘self-care’, ‘life activities’, and the WHODAS II total score. Mann-Whitney U analyses calculated a trend for the domain ‘life activities (p = 0.059) which was confirmed statistically significant using parametric equivalent t-test (mean difference = –17.8; 95% confidence interval (CI) –34.6 to –1.1; p = 0.037). Contrary to hypothesized, there was no between group difference for the domain ‘participation’.

Fig. 1 compares median scores for WHODAS II total and domain scores between persons who were in the workforce and those who were not in the workforce. WHODAS II was able to significantly discriminate (p < 0.05) between persons in the workforce and those not in the workforce in terms of the total score (median difference = 9, 95% CI: 0 to 19), and the domains ‘getting around’ (median difference = 15, 95% CI: 5 to 25) and ‘selfcare’ (Median difference = 19, CI: 0 to 50). A trend was found for domain ‘life activities’ (median difference = 15, CI: 0 to 38), p = 0.095) which was confirmed statistically significant using parametric equivalent independent t-test (mean difference = 17.5 (95% CI 2.4 to 32.6; p = 0.024). Contrary to hypothesized, there was no difference between persons working and those not working for the domain ‘participation’. As such, 4 out of 5 domains where we expected to find a difference were confirmed.

Convergent/divergent validity

Table IV presents calculated Spearman correlation coefficients between WHODAS II and CHART and MOS SF-36. Only 1 out of 7 hypothesized convergent associations between WHODAS II and CHART was confirmed (WHODAS II ‘self-care” with CHART ‘physical’) and none of the 6 hypothesized divergent associations were weak enough to present ‘no relationship’ (rs<0.20); confirming only 8 percent of a priori hypothesized associations. On the other hand, 10 out of 11 hypothesized convergent relationships between WHODAS II and MOS SF-36 were moderate and 3 out of 4 divergent associations were as expected without relationship (rs<0.20); confirming 87% of a priori hypothesized associations.

Construct validity using Rasch

Unidimensionality. Analysis of the 36 items shows that overall items were found to fit the model producing item mean infit statistics of 0.99 (SD 0.39) and mean outfit statistics of 1.00 (SD 0.51) and thus performed satisfactorily (Table V). Five of the items, appear to misfit the Rasch model (i.e. MnSq > 1.40 with concomitant Z-scores equal or greater than 2), these are items 2.1 standing long periods, 2.2 standing up from sitting down, 2.5 walking or wheeling a long distance, 3.4 staying by yourself for a few days, and 4.5 sexual activities. Table VI identifies persons with unexpected answers on misfit items. Overall, 86% of the items fit the Rasch measurement model.

Hierarchy of items. Fig. 2 shows the distribution of all items and all persons on a linear continuum, ranging from persons with less disability and items that are more challenging at the top, to persons with more disability and less challenging items at the bottom. The distribution of person abilities ranged from –2.03 to 2.40 logits, with a mean of 0.75 logits. The mean of the item calibration values was 0 logits, ranging in difficulty from as high as 2.25 logits to as low as –1.51 logits. The hierarchy of the items shows that the easiest items relate to understanding/communicating or interpersonal relationships, while the most difficult items relate to standing followed by finances and household activities. Visualization of the range of item difficulties in relation to person’s ability reveals a shortage of (fitting) items capable of differentiating the more able persons (with less disability). Since items 2.1 and 2.2 underfit the Rasch model, the difficulty a person experiences with standing for a long time or standing from sitting down has little predictive value towards the total disability score (WHODAS II).

Reliability and separation. The Rasch analysis produced a person reliability coefficient of 0.94, an indication that the items work well together to consistently reproduce a participant’s score. The person separation was 3.84 indicating that the distribution of the persons could be separated into 5 statistically distinct levels [(4 x 3.84 + 1)/3]. Item separation was 5.19, with a reliability coefficient of 0.96.

Discussion

This study has evaluated the psychometric properties of WHODAS II in the SCI population from a classical and item response theory perspective. The WHODAS II presents adequate internal consistency and construct validity in that it discriminates between different levels of impairment and workforce involvement for most domains, and presents the pattern of convergent/divergent correlation coefficients with MOS SF-36. Most items fit a unidimensional model with exception of 5 items (related to standing, standing up, walking/wheeling long distances, sexual activities, and staying alone for a few days).

The WHODAS II was able to discriminate between individuals with high and low impairment SCI in terms of getting around, self-care, life activities (household and work) and total score. Interestingly, there was no differentiation between impairment levels for the domain of ‘participation in society’, supporting the suggestion that regardless of level and completeness of neurological lesion, individuals with a SCI may perceive similar degrees of difficulty in participation in society. This domain includes items such as “How much of a problem did you have in joining in community activities?”, “How much of a problem did you have because of barriers or hindrances in the world around you?” or “How much has your health been a drain on the financial resources of you or your family?” More so than any of the other WHODAS II domains, “Participation in society” incorporates the influence of personal and environmental factors beyond control of the person with SCI, including physical access, social attitudes, public services, systems and policies, impact of health and emotional problems, family support and financial situation.

Convergent/divergent validity was established between WHODAS II and MOS SF-36, however not with CHART. The median values for the CHART domains and the mean values for the SF-36 domains were consistent with previous reported data (27, 36), indicating that our sample represented a typical SCI population. One possible explanation for the poor results with CHART may be that the quantitative approach of CHART - where scores indicative of better participation are obtained with for example a higher number of contacts and/or household members, or more days spend out of the house – is not compatible with the more qualitative approach of WHODAS II where being able to maintain a single friendship and getting along with people close to you or experiencing no difficulties with leaving the home results in low disability scores. Conversely, a number of additional strong correlation coefficients were observed between domains that were not hypothesized a priori. This is possibly the result of large number of paired correlations, increasing the chance of type 1 errors. Such findings may also be a function of the overlapping features among those domains pertaining to social outcomes, as observed by Dijkers and colleagues (37).

Rasch analysis showed that 5 of the items misfit the Rasch model. Poor fit of these items means that the performance on these items cannot be predicted comfortably by what is known about these persons’ ability based on the overall WHODAS II total score. One possible explanation could be that these questions, unlike the other 32 questions, do not represent the same construct or are confounded by other factors. Two items relate to one’s ability to use legs (standing for long periods and standing up) and a third item may be interpreted as walking or wheeling. Indeed, 81% of the participants in this sample mobilized 100% of the time in a wheelchair while the remaining 19% were able to walk with or without an assistive device to varying degrees of independence. It is therefore not surprising that in a heterogenous group of persons with a SCI, including both walking and non-walking individuals, responses to these items would diverge from the expected ability/difficulty pattern based on the overall performance on the remaining items of the WHODAS II. The majority of those with unexpected responses to these two items were people with an incomplete SCI who were able to walk, but nevertheless experienced a considerable degree of disability in everyday life. Gerhart et al. (38) report that despite being ambulant and physically independent, many “minimally impaired” individuals experience significant problems with increased spasticity, pain, bladder and bowel difficulties, sexual dysfunction and psychosocial concerns. People with SCI who walk often experience more difficulties covering long distances due to the higher energy cost compared to wheeling. Furthermore, persons with greater preserved function and less obvious SCI impairments may receive less support and have higher expectations placed on them, which increases the burden of coping and may lead to disproportionately high levels of frustration, depression and even suicide (39).

The poor fit of item 4.5 ‘sexual activities’ could potentially be related to person’s interpretation of the meaning of this question, being in the context of purely physical functioning and sexual performance versus, as the domain would suggest, the interpersonal component of sexual relations. Advances in the medical management of sexual health after SCI (40) mean that most males with a SCI are now able to sustain an erection that permits sexual intercourse. Even persons who report great difficulty in the domains of getting around, self-care, life activities, and participation may score well on this item as ultimately what one considers to be ‘sexual activities’ and finds satisfying is personally defined.

The unexpected responses to item 3.4 (staying by yourself for a few days) also have a reasonable explanation. These responses were mostly from persons with complete tetraplegia who had adjusted very well to their SCI and had returned to work, were involved in recreational activities, and managed their own transportation. Yet, each person required up to 2 h a day of paid personal care to assist with bathing, dressing, and bowel and bladder care, and may have other medical concerns such as spasticity or risk of autonomic dysreflexia, that could explain why they were unable to stay alone for more than one day, despite experiencing low levels of disability in everyday life.

Although one could argue that misfit items weaken scale unidimensionality, deletion of these items is not necessarily appropriate as all 5 items provide important information that may distinguish a person with SCI in one or more of the components of activity and participation.

Results of the Rasch analysis suggest, however, that no items of the WHODAS II instrument are able to distinguish persons with a SCI with lesser degrees of disability (low scores on WHODAS II total score). This is a potential weakness of the WHODAS II in this population, however further study is warranted with a larger sample size. Further application of Rasch analyses on other chronic disease groups is also recommended for comparison with our results. Limited ability of the WHODAS II to differentiate among people with low disability was suggested in Garin et al’s (13) larger study that included people with multiple sclerosis, stroke, traumatic brain injury, and other conditions. This suggestion however was inferred from the presence of high ceiling effects on most domains. In comparisson with Garin’s study, our SCI population presented with 10 to 25 percent less ceiling effects on all domains except for the domain understanding and communication. We found similar patterns of ceiling effects as reported by Noonan et al. (20)

The study had several limitations. The sample size was relatively small and the findings need to be confirmed with larger studies. Our data set did not allow for the evaluation of other important properties such as sensitivity to change, and responsiveness.

In conclusion, this study has provided further support for reliability and validity of the WHODAS II in a SCI population, and has for the most part been able to confirm findings from the study by Noonan et al. (20) A number of limitations of the WHODAS II are noted from our study. First, ceiling effects are present in 4 out of 7 domains and present a concern if measurement of change over time or improvement following intervention is of interest. Secondly, there is a lack of spread of items along a continuum that would allow differentiation of persons with lower levels of disability. Finally, with 5 misfitting items there is questionable unidimensionality. Future research will need to determine whether the benefit of a generic measure of disability, applicable across different cultures and disability groups that allow for comparison among studies, outweighs the marginal benefits one may hope to find by using a condition specific instrument designed for use in a spinal cord injured population. Rasch rating scale analyses on a larger sample of persons with SCI are needed to confirm our results.

Acknowledgements

The study from which data was drawn for the present paper was funded by the Motor Accidents Authority of New South Wales.

References