TOWARDS AN INSTRUMENT TARGETING MOBILITY-RELATED PARTICIPATION:

NORDIC CROSS-NATIONAL RELIABILITY

Åse Brandt, PhD1, Charlotte Löfqvist, MSc2, Inga Jónsdottir, BSc3, Terje Sund, MSc4,
Anna-Liisa Salminen, PhD5, Monica Werngren-Elgström, PhD2 and Susanne Iwarsson, PhD2

From the 1Department of Research & Development, Danish Centre for Assistive Technology, Århus, Denmark,
2Department of Health Sciences, Lund University, Lund, Sweden, 3Assistive Technology Centre, Kópavogur, Social
Insurance Administration, Reykjavik, Iceland, 4The Special Units Department, Section for Assistive Technology Centres, Norwegian Labour and Welfare Service, Oslo, Norway and 5The Social Services Division, Stakes, Helsinki, Finland

OBJECTIVE: To investigate test-retest reliability and internal consistency of a new instrument for evaluation of mobility device interventions.

Material and methods: The instrument comprised 4 scales and one summed index. Two test-retest interviews involved 147 mobility device users (mean age 60 years) with a broad range of functional limitations, living at home.

RESULTS: For 2 scales and the summed index the reliability was substantial or almost perfect (κ 0.71/ κW 0.76/intraclass correlation coefficient = 0.93; confidence interval = 0.90–0.95). The reliability of one scale was moderate (κW 0.41), but after reduction of grades and combination with another scale, it was substantial (κW 0.66). The reliability of the fourth scale was moderate (κW 0.55). The internal consistency of 3 scales varied from α 0.63 to 0.76.

CONCLUSION: Even though the test-retest reliability of all but one of the scales of the new instrument was substantial to almost perfect, this study demonstrated that revision is required. The challenges identified were probably due to the highly complex relationship between outdoor participation while using mobility devices and accessibility to the outdoor physical environment. Thus, based on the results of this study the instrument will be revised and subsequently launched as the “Nordic mobility-related participation outcome evaluation of assistive device intervention” (NOMO instrument). More research on the concept of mobility-related participation and the psychometric qualities of the instrument is required.

Key words: mobility limitation, assistive devices, outcome assessment, rehabilitation, psychometrics.

J Rehabil Med 2008; 40: 766–772

Correspondence address: Åse Brandt, Department of Research & Development, Danish Centre for Assistive Technology, P. P. Ørumsgade 11, bygn. 3, DK-8000 Århus C, Denmark. E-mail: aab@hmi.dk

Submitted January 17, 2008; accepted May 19, 2008

Introduction

The demand for evidence-based praxis in healthcare is increasing, but studies on the effectiveness of assistive technology are scarce. A large proportion of the assistive devices currently in use are mobility devices aiming at making participation in everyday life and society possible for persons with mobility impairments (1, 2). Even though several studies on mobility device outcomes have been carried out, most of them are laboratory studies investigating the efficacy of the devices focusing on functional outcomes, and only a few studies target effectiveness in real life situations, i.e. participation in everyday life and society. Such studies are vital, since they may shed light on how and to what degree mobility devices assist the users in living an active life and thereby whether the devices contribute to fulfilling political ambitions of providing people with disability equal opportunities for participation in society (3). Reasons for the lack of participation outcome studies are most likely that assistive technology outcome evaluation still is in its infancy and that only few outcome evaluation instruments are applicable (4). Furthermore, before the launch of the International Classification of Functioning, Disability and Health (ICF) (5), in which participation is a central component, the concept was generally seldom in focus for outcome evaluation.

In order to define the eventual need for a new instrument targeting participation outcomes of mobility device interventions, a search in the CINAHL, SocioFile (sociological abstracts database), PubMed, PEDro (Physiotherapy Evidence Database), Cochrane, and AMED (Allied and Complementary Medicine Database) databases was performed in 2003, resulting in no hits. Some instruments targeted participation in general, but were considered too broad for mobility device outcome evaluation, e.g. the assessments of the Life Habits (LIFE-H) (6), “Impact of participation and autonomy” (IPA) (7), and “Craig Handicap Scaling and Reporting Technique” (CHART) (8). Other instruments had been developed specifically for assessment of assistive technology interventions, e.g. the “Individually prioritised problems analysis” (IPPA) (9) and the “Functional evaluation in a wheelchair” (FEW) (10), but did not target the outcome dimension of interest.

With the intention of closing this gap a Nordic project encompassing partners from Denmark, Finland, Iceland, Norway and Sweden was initiated aimed at developing a new instrument for outcome evaluation of mobility device interventions. The project was called “The Nordic Assisted Mobility Evaluation” (NAME). The instrument developed within the project was constructed to evaluate mobility device interventions in terms of “mobility-related participation” (11), meaning that only participation aspects involving mobility were targeted. The theoretical basis for development of the instrument was the ICF framework, stating that assistive devices are environmental factors that, along with other environmental factors, personal factors, and the person’s body functions, influence participation. Thus, mobility-related participation items were selected on the basis of the ICF list of activities and participation and were subsequently qualified by discussions with groups using mobility devices, assistive technology experts and researchers. Pilot versions of the instrument underwent content validation and feasibility tests in all 5 countries, with satisfactory results. As part of the ongoing instrument optimization process, the objective of this study was to investigate the test-retest reliability and internal consistency of the instrument developed within the NAME Project.

Material and methods

The instrument

At the initiation of this study, the instrument developed within the NAME Project was available in 5 Nordic languages, and in addition to the instrument a manual was provided. The instrument had a structured interview format and consisted of a baseline questionnaire to be used just before the user gets his/her mobility device, and a follow-up questionnaire. Each version had an optional, descriptive part (Part A) and a mandatory outcomes part (Part B). Part A comprised items concerning demographics, functional limitations, living conditions, means of transportation, etc. Part B constituted the core of the instrument and was identical in the baseline- and the follow-up versions (Fig. 1). A short version of Part B of the instrument is enclosed in Appendix I. It consisted of 2 sets of items, each to be rated with 2 different scales. In addition, one of the scales was used for building up a summed index. The contents of Part B were as follows:

Four items about mobility in different environments:

• Dependence in mobility scale with “yes”/”no”/”does not know” response categories.

• Extent of assistance in mobility scale offering response options on a 5-graded ordinal scale ranging from “very little assistance” to “very much assistance”.

Twenty-two items about mobility-related participation:

• Frequency of mobility-related participation scale offering response options on a 9-graded scale going from “at least once a day” to “never”. In addition “does not know” was offered.

• Ease/difficulty in mobility during participation scale offering response options on a 5-graded ordinal scale ranging from “very easy” to “very difficult”. In addition “does not know” was offered.

• Mobility-related participation repertoire index, i.e. a sum of the number of participation aspects performed, based on the responses of the “Frequency of mobility-related participation scale”.

Participants

The study participants were mobility device users from the 5 Nordic countries, selected by municipality therapists aiming at reaching variation in terms of age, sex, mobility device use, civil status, and urban/rural environments. Inclusion criteria were: mobility device use for at least 4 months; a stable mobility device situation; at least 18 years of age; sufficient cognitive function for study participation; private residence. Exclusion criteria were: insufficient communication abilities; mobility device motive other than mobility and participation (e.g. to alleviate pain); mobility-related participation not possible (e.g. because of hospitalization).

A total of 172 persons were asked to participate. Of these, 19 declined, mostly because they generally did not wish to participate in studies. During the study 6 participants dropped out, mainly because of illness, leaving 147. The participants represented the variation aimed for concerning demographics and device use, except that only a few used walking frames or transportation wheelchair (Table I).

Data collection

Part A of the baseline version of the instrument was used only for retrieving descriptive participant information. In addition, questions about self-reported diagnosis based on the World Health Organization (WHO) International Classification of Diseases ICD-10 (12) were developed and used for descriptive purposes; the participants reported their predominant diagnoses; at most 3 could be reported.

In order to test Part B for test-retest reliability and internal consistency, 2 interviews with 7 days interval (standard deviation = 4) were accomplished at home visits, except from 7 participants who were interviewed at the interviewers’ offices, at a rehabilitation centre, or in the participant’s office. Before the second interview it was ensured that no major changes in prerequisites for mobility had occurred since the first interview. The interviews were undertaken by occupational therapists, physiotherapists, or by a person with another type of academic background. All were trained in administration of the instrument by members of the project team (author constellation). Data were entered in a SPSS 14.0 datasheet according to project-specific guidelines, followed by quality control.

Data analysis

Prior to the analysis the frequency of “does not know” responses was computed, since a high frequency might mean that the “does not know” data had to be included in the analysis, in turn implying that the ordinal scales would have to be analysed as categorical data. Since the frequency of “does not know” responses was low (0.13–0.64% of all possible responses), they were not included further in the analysis. The 22nd Part B item “Other aspects“ was not included in the analysis. Percentage agreement and kappa coefficients were calculated to examine response consistency (agreement) between the 2 interviews (13, 14). For categorical data, i.e. the Dependence in mobility scale, kappa (κ) was computed, and for the remainder scales consisting of ordered categorical data, weighted kappa (κW) was used. The intraclass correlation coefficient (ICC) with 95% confidence limits was used for the Mobility-related participation repertoire summed score. The strength of agreement was interpreted according to Landis and Koch’s guidelines (16): κ values < 0.00 = poor agreement; 0.00–0.20 = slight; 0.21–0.40 = fair; 0.41–0.60 = moderate; 0.61–0.80 = substantial; 0.81–1.00 = almost perfect agreement. The ICC was interpreted in the same way.

For analysis of internal consistency Cronbach’s alpha (α) was used. Levels of 0.70–0.90 were considered optimal, since they suggest internal consistency without redundancy of items (15, 16). The results for the 2 interviews were very similar, thus the final internal consistency analysis was based on data from both interviews. Analysis was accomplished for the entire study population and for sex and age subgroups. A country subgroup analysis was considered but not computed because of small sample sizes (17).

During the process of data analysis, the agreement of the Extent of assistance in mobility scale and the Ease/difficulty in mobility during participation scale appeared to be insufficient. In order to optimize the scales they were transformed into fewer grades step by step, followed by renewed analysis. In addition, the Extent of assistance scale was combined with the Dependence in mobility scale into 1 scale, thereby reducing the total number of scales from 4 to 3.

The SAS (for kappa) or SPSS 14.00 software (for all other analyses) were used.

Ethics

All principles in ethical guidelines for human research were followed meticulously. In Iceland formal ethical consent was sought and granted, while this was not required for this type of study in the other countries.

Results

The reliability of the 4 scales and the summed score varied from moderate to nearly perfect (Table II). That is, the reliability of the Dependence in mobility and the Frequency of mobility-related participation scales was substantial, and for the Mobility-related participation repertoire summed score, it was almost perfect. The reliability of the Extent of assistance in mobility scale was moderate, barely exceeding the cut-off to be interpreted as fair. Changing the scale into a 3-grade scale did not improve the level of agreement. However, after the construction of a new 4-grade scale based on the 3-graded version in combination with the Dependence in mobility scale, labelled Extent of dependence, the level of agreement improved to substantial. For the Ease/difficulty in mobility during participation scale the reliability was moderate. Reduction of the number of grades into 3-graded or dichotomous scales changed the κW coefficient only slightly, while the percent agreement rose to some extent (Table II).

The results for the subgroup analysis concerning sex and age were similar to those for the entire sample (Table III). To sum up, the reliability of the scales was substantial to moderate after having reduced the number of grades for one scale, which was further combined with another scale, resulting in 3 scales and a summed score with almost perfect reliability.

The levels of internal consistency of the 2 original scales Dependence in mobility and Extent of assistance in mobility were less than optimal and optimal, respectively. After merging the 2 scales into the new Extent of dependence scale, the internal consistency was optimal. It was also optimal for the Frequency of mobility-related participation scale. Because of too few cases, it was not possible to analyse internal consistency for the Ease/difficulty in mobility during participation scale (Table II).

Discussion

The present study reports the first step of investigation of a new instrument’s psychometric properties. After scale optimization it revealed mostly positive results, but also important areas for further instrument improvements. This instrument developed in a Nordic context, within the NAME Project, is intended for assessment of mobility-related participation outcomes of mobility device interventions. The structured interview format targets mobility and mobility-related outcome dimensions concerning ease/difficulty, assistance, and types and frequency of performance. That is, different scales representing dimensions that as a whole can be regarded as relevant in studies on outcomes of mobility device interventions. Other studies have found that the choice of scale for assessing disability may impact the findings substantially and that difficulty scales were the most sensitive for detecting disability, indicating that the ease/difficulty scale is a core scale of the instrument (18–20).

The reliability test-retest involved a considerable sample size, including participants with a relevant variation of functional limitations, diagnoses, and device use in the 5 Nordic countries. The fact that only a few participants used walking frames or transportation wheelchairs and that the participants’ self-rated health was lower than in the general population in the same age groups (21) is regarded as typical for the instrument’s target group. Even though the present study did not target feasibility, the data collected supported the positive results of the pilot tests accomplished prior to the current study (unpublished data). That is, the proportions of “does not know” responses were very low, indicating that the target group was able to answer the questions included in the instrument.

The study took place in a Nordic context among mobility device users representing a relevant variety of characteristics, indicating ecological validity. While the total sample size was sufficiently large for overall test-retest analysis and for age and sex subgroup analysis, it was too small for valid country subgroup analysis.

The reliability was substantial to almost perfect except for one scale, namely Ease/difficulty in mobility during participation. There may be several explanations why the reliability of this scale was only moderate. One is that κ statistics are affected by asymmetrical data distributions, that is when the prevalence of a response is very low or high, the kappa value may be artificially lowered (22, 23). Since data for most items of the Ease/difficulty in mobility during participation scale was asymmetrical, this is a possible explanation, while data inspection revealed that the items showing asymmetrical data did not present lower κ values than other items. In addition, the percent agreement was not very high, supporting the assumption that data asymmetry probably was not the correct explanation. Another explanation may be that some items consisted of more than one mobility-related participation aspect, e.g. using services such as the chemist, post office, library, bank or receiving treatment at a physician, physiotherapist or dentist, making the items ambiguous. However, these kinds of questions did not display markedly low κ values. A third explanation may be the complexity of the insufficiently explored relationship between outdoor participation and accessibility problems; a relationship that is challenged while moving about using a mobility device (2, 24). The ease/difficulty in mobility-related participation may depend on the accessibility to, and usability of, the physical environment, i.e. involving the person with his/her impairments, the mobility device and the environmental barriers at the place where participation takes place (11, 25). Given that, for example, shopping is performed in different shops and the ease/difficulty perceived depends on the environmental barriers present outside and in the shop, it may be difficult to consistently assess how easy/difficult it is to get about while shopping. Based on our experiences through the process of instrument development in this field, the complexity just described constitutes the major challenge for coming up with optimally valid and reliable instruments for assessment of mobility-related participation. This is supported by the fact that the level of test-retest reliability of several other instruments for assessment of outdoor participation or of outdoor environments is similar to that of the Ease/difficulty in mobility during participation scale. One example is the LIFE-H, which is used for assessment of life habits and handicap, concepts close to that of participation. The overall reliability for adults with spinal cord injury assessed by ICC was 0.74, but for the mobility category it was only 0.59 (26). Another example is Okochi et al. (27), who studied the test-retest reliability of ICF codes and qualifiers in geriatric care. The mean κW value for mobility in the “Activity and participation” domain was 0.50 for older people living at home, but when supplementing the instrument with illustrations of different environments, the mean κW value increased to 0.76. This supports the assumption that respondents may think of different places when asked about mobility-related participation, posing demands on specification of the environment to reduce ambiguity and improve instrument reliability. As concerns the instrument tested in the current study, adding instructions to participants to think about preferred locations for each participation aspect may increase the reliability of the Ease/difficulty in mobility during participation. Turning to instruments for self-assessment of aspects of the physical environment, challenges similar to those we experienced in the current study prevail. For example the “Usability in My Home” (UIMH) with κW ranging from 0.57 to 0.83 for 6 specific items, the lowest value concerned accessibility outdoors (28). Another example is the “Home and Community Environment” instrument. For this instrument the mean κ value for community mobility was 0.47 (range 0.20–0.64) and for home mobility 0.66 (range 0.28–1.0) at test-retest. In all, even though κ values cannot be compared directly, it seems as if it is more difficult to obtain high reliability for assessments outdoors than indoors. Summing up on reliability in instruments targeting aspects of participation outdoors, this kind of research is challenging. Based on the results of the current study, and not least on our practical experiences and discussions along the instrument development process within the NAME Project, the concept of mobility-related participation is particularly challenging. The limitations as concerns reliability must be kept in mind when using the instrument for outcome evaluation, since lower instrument reliability requires larger sample sizes (16).

The internal consistency for the Frequency of mobility-related participation scale was optimal, and so was the subsequently
developed Extent of dependence scale. Internal consistency is often regarded as a measure of reliability, but since Cronbach’s alpha describes how well a group of items focuses on a single concept, this coefficient may instead be regarded as a first expression of an instrument’s construct validity (13). Nevertheless, in order to investigate to what degree the instrument evaluates mobility-related participation as expected, and whether each scale assesses one single dimension, there is a need for in-depth studies on construct validity.

When it comes to use of the instrument for outcome measurement purposes, it should be kept in mind that the scales, like most instruments for evaluation of human performance and opinions, are ordinal. A further development of the instrument could be application of Rasch analysis, thereby changing the scales into interval data and improving the instrument’s psychometric quality (29). Also the instrument’s sensitivity should be investigated, that is its ability to detect changes between groups of mobility device users and its responsiveness, i.e. ability to detect changes resulting form mobility device interventions. Prior to further psychometric testing, the instrument (Appendix I) should be revised based on the findings of the present study. A main change required is to replace the 2 scales on dependence in mobility in different environments with one new scale. Furthermore, some items should be revised and the instructions made more specific, in particular when it comes to specification of environments in focus for the rating. In order to reflect the outcome dimension at target, the new instrument will be launched as the “Nordic mobility-related participation outcome evaluation of assistive device interventions”, NOMO 1.0.

In conclusion, based on a cross-Nordic project, this article reports an investigation of the test-retest reliability and internal consistency of the first version of a new instrument for evaluation of outcomes of mobility device interventions in terms of mobility and mobility-related participation. The reliability of the scales and the summed score building up the instrument varied from moderate to nearly perfect. We demonstrated that it was possible to improve reliability by means of scale optimization, resulting in concrete strategies for further revisions of the instrument. Subsequently, the instrument will be made available for use in practice contexts in the Nordic countries. However, further psychometric testing is required and will be planned in parallel with the experiences gained through practical application. This study contributes to the knowledge generation concerning reasons for challenges to reliability in this kind of instruments, suggesting that they may be due to the complexity of the concept of mobility-related participation. A particular challenge for instruments targeting use of mobility devices for outdoor mobility and participation is to deal with complex person-environment relationships involving mobility devices as part of the environmental component.

Acknowledgements

We wish to thank P. Kotiranta, Stakes, Finland; K. B. Laursen, Århus University, Denmark; and V. Horstmann and B. Slaug at Lund University, Sweden, for contributions in different phases of this project, and the reviewer for the suggestion to change the instrument’s title. Thanks also to consortium members participating in the first project phase and to all the mobility device users, data collectors, and the municipality staff who identified mobility device users for the study. We are grateful to the Nordic Development Centre for Rehabilitation Technology, The Danish Association of Occupational Therapists, The Swedish Research Council, The Social Insurance Administration, Iceland, The National Insurance Administration, The Swedish Handicap Institute, Stakes, Finland, and The Danish Centre for Assistive Technology for funding the research. This paper was prepared in the context of the Centre for Ageing and Supportive Environments (CASE) at Lund University, funded by The Swedish Research Council.

References

1. Brundell S, editor. Äldrestatistik Juni 2006. Äldres hälsa, funktionshinder, boende och hjälpmedel [Statistics of elderly, June 2006. Elderly’s health, disability, living and aides]. Vällingby, Sweden: Hjälpmedelsinstitutet; 2006 (in Swedish).

2. Brandt Å, Iwarsson S, Ståhl A. Older people’s use of powered wheelchairs for activity and participation. J Rehabil Med 2004; 36: 70–77.

3. United Nations. Standard rules on the equalisation of opportunities for persons with disabilities. New York, NY: United Nations Department of Public Information; 1994.

4. Rust KL, Smith RO. Assistive technology in the measurement of rehabilitation and health outcomes. Am J Phys Med Rehabil 2005; 84: 780–793.

5. WHO. International Classification of Functioning, Disability and Health: ICF. Geneva: WHO; 2001.

6. Noreau L, Fougeyrollas P, Vincent C. The LIFE-H: Assessment of the Quality of Social Participation. In: Marincek C, Bühler C, Knops H, Andrich R, editors. Assistive technology – added value to the quality of life. Amsterdam: IOS Press; 2001, p. 604–605.

7. Cardol M, de Haan RJ, van den Bos GAM, de Jong BA, de Groot IJM. The development of a handicap assessment questionnaire: the Impact on Participation and Autonomy (IPA). Clin Rehabil 1999; 13: 411–419.

8. Whiteneck GG, Charlifue SW, Gerhart KA, Overholser RD, Richardson GN. Quantifying handicap: a new measure of long-term rehabilitation outcomes. Arch Phys Med Rehabil 1992; 73: 519–526.

9. Wessels RD, de Witte L, Andrich R, Ferrario M, Persson J, Oberg B, et al. IPPA, a user-centred approach to assess effectiveness of assistive technology provision. Technol Disabil 2000; 13: 105–115.

10. Mills T, Holm MB, Trefler E, Schmeler M, Fitzgerald S, Boninger M. Development and consumer validation of the Functional Evaluation in a Wheelchair (FEW) instrument. Disabil Rehabil 2002; 24: 38–46.

11. Brandt Å, editor. Outcomes of rollator and powered wheelchair interventions. User satisfaction and participation. Dissertation. Lund, Sweden: Faculty of Medicine, Division of Occupational Therapy, Lund Universtiy; 2005.

12. WHO. ICD-10. 2007. [cited 2007 Dec 13]. Available from: URL: http:// www.who.int/classifications/icd/en

13. Fayers PM, Machin D, editors. Quality of life: assessment, analysis, and interpretation. Chichester, UK: John Wiley & Sons Ltd; 2000.

14. Bartko JJ, Carpenter WT. On the methods and theory of reliability. J Nerv Ment Dis 1976; 163: 307–317.

15. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–174.

16. Streiner DL, Norman GR, editors. Health measurement scales – a practical guide to their development and use. 3rd edn. Oxford: Oxford University Press; 2003.

17. Soeken KL, Prescott PA. Issues in the use of kappa to estimate reliability. Med Care 1986; 24: 733–741.

18. Jette AM. How measurement techniques influence estimates of disability in older populations. Soc Sci Med 1994; 38: 937–942.

19. Laditka SB, Jenkins CL. Difficulty or dependency? Effects of measurement scales on disability prevalence among older Americans. J Health Soc Policy 2001; 13: 1–13.

20. Verbrugge LM, Jette AM. The disablement process. Soc Sci Med 1994; 38: 1–14.

21. Bjørner JB, Damsgaard MT, Watt T, Bech P, Rasmussen NK,
Kristensen TS, et al., editors. Dansk manual til SF-36. Et spørgeskema om helbredsstatus. [Danish manual for the SF-36-questionnaire. A questionnaire on health status]. Copenhagen: Lif; 1997 (in Danish).

22. Feinstein AR, Cicchetti DV. High agreement but low Kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990; 43: 543–549.

23. Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol 1993; 46: 123–429.

24. Lavery I, Davey S, Ewart K. The vital role of street design and management in reducing barriers to older people’s mobility. Landsc Urban Plan 1996; 35: 181–192.

25. Kraskowsky LH, Finlayson M. Factors affecting older adults’ use of adaptive equipment. Amer J Occupat Ther 2000; 55: 303–310.

26. Fougeyrollas P, Noreau L, Bergeron H, Cloutier R, Dion S, St Michel G. Social consequences of long term impairments and disabilities: conceptual approach and assessment of handicap. Int J Rehabil Res 1998; 21: 127–141.

27. Okochi J, Utsunomiya S, Takahashi T. Health measurement using the ICF: test-retest reliability study of ICF codes and qualifiers in geriatric care. Health Qual Life Outcomes 2005; 3: 46.

28. Fänge A, Iwarsson S. Physical housing environment: development of a self-assessment instrument. Can J Occup Ther 1999; 66: 250–260.

29. Bond TG, Fox CM. Applying the Rasch model: fundamental measurement in the human sciences. Mahwah (NJ): Lawrence Erlbaum Associates; 2001.