Eva Grill, DrPH, PhD1,2, Thomas Gloor-Juzi, MScPT3, Erika Omega Huber, MAS FH3,4 and Gerold Stucki, MD, MS2,5,6

From the 1Institute for Health and Rehabilitation Sciences (IHRS), Munich, Germany, 2ICF Research Branch of WHO Collaborating Centre for the Family of International Classifications in German, Nottwil, Switzerland,
3Institute for Physical Medicine and Rehabilitation, University Hospital Zurich, Zurich, 4Swiss Physiotherapeutic
Association, Sursee, 5Swiss Paraplegic Research (SPF), Nottwil and 6Seminar of Health Sciences and Health Policy, University of Lucerne, Lucerne, Switzerland

OBJECTIVE: To operationalize items based on categories of the International Classification of Functioning, Disability and Health (ICF) relevant to patient problems that are addressed by physiotherapeutic interventions in the acute hospital, and to test the reliability of these items when applied by physiotherapists.

METHODS: A selection of 124 ICF categories was operationalized in a formal decision-making and consensus process. The reliability of the newly operationalized item list was tested with a cross-sectional study with repeated measurements.

RESULTS: The item writing process resulted in 94 dichotomous and 30 polytomous items. Data were collected in a convenience sample of 28 patients with neurological, musculoskeletal, cardiopulmonary, or internal organ conditions, requiring physical therapy in an acute hospital. Fifty-six percent of the polytomous and 68% of the dichotomous items had a raw agreement of 0.7 or above, whereas 36% of all polytomous and 34% of all dichotomous items had a kappa coefficient of 0.7 and above.

CONCLUSION: The study supports that the ICF is adaptable to professional and setting-specific needs of physiotherapists. Further research towards the development of reliable instruments for physiotherapists based on the ICF seems justified.

Key words: ICF; classification; reliability; outcome measures; health status assessment.

J Rehabil Med 2011; 43: 162–173

Correspondence address: Eva Grill, Institute for Health and Rehabilitation Sciences, Ludwig-Maximilians-Universität München, DE-81377 Munich, Germany. E-mail: eva.grill@med.uni-muenchen.de

Submitted May 25, 2010; accepted September 30, 2010

Introduction

Healthcare management in the acute hospital must reflect patient’s needs and the specific reasons for their hospitalization. The risk for loss of functioning is exacerbated by factors such as complications of intervention, pre-existing chronic conditions, co-morbidities and old age. The timely identification of these risk factors, and the consequent definition of adequate care provisions for maintaining or improving functioning, with an aim of minimizing disability, is of utmost importance, even in the acute hospital (1, 2).

Persistent loss of functioning can be prevented most effectively when early rehabilitation interventions are provided for the patient as part of the acute medical treatment (3–5). Part of the task of physiotherapists is to identify and assess acute functional impairment, and to take measures to correct or alleviate the long-term outcome (6, 7). In order to improve functioning and minimize disability, caregivers must have recourse to sound concepts and instruments for measuring functioning (8–10). This holds true for all health professionals and settings, but is particularly relevant for physiotherapists in the acute hospital setting (11–13).

The International Classification of Functioning, Disability and Health (ICF) (14) is a multipurpose classification that may constitute a suitable basis for assessment of functioning in a multidisciplinary environment. However, the ICF is comprehensive, encompassing every possible aspect of human functioning, and must consequently be tailored in several respects for the needs and purposes of its potential users (15–18). First, it must address all relevant concepts, such as those addressed by the particular interventions made by physiotherapists (19). Secondly, these concepts have to be operationalized in a way that is suitable to the setting, e.g. the acute hospital. Thirdly, any such operationalization must be of proven reliability in the hands of prospective users; here, physiotherapists. Once a reliable operationalization has been defined, the constituent items can be used to document categorical profiles of patients’ functioning. If the need for a new instrument dealing with specific domains should arise, the objectivity and validity of subsets of these items have to be established.

To date, the utility and reliability of ICF categories operationalized specifically for use by physiotherapists has not been established. The objective of the present study was therefore to operationalize items based on ICF categories relevant to patient problems addressed by physiotherapeutic interventions in the acute hospital, and to test the reliability of these items in the hands of physiotherapists.

Methods

Study design

Selection of ICF categories and operationalization. The ICF has two parts, each containing 2 separate components. Part 1 covers functioning and disability, and includes the components Body Functions (b) and Structures (s) and Activities and Participation (d). Part 2 covers contextual factors, and includes the components Environmental Factors (e) and Personal Factors.

Comprising a total of 1424 such categories, the ICF gives a comprehensive description of human functioning. Not all of those categories, however, are useful and relevant for physiotherapists. Based on a selection of categories for the acute hospital (so called ICF Core Sets) (15, 16, 18) and a Delphi exercise (19), we chose 124 categories potentially relevant for physiotherapists in the acute hospital. Both selection process (17) and Delphi exercise have been described in detail (19). In brief, the ICF Core Set development was a consensus process based on evidence gathered from preliminary studies, which included contributions of focus groups, systematic reviews and empirical data collection. The Delphi exercise was a consensus-building, 3-round, e-mail survey of a total of 263 physiotherapists in Germany and Switzerland. The first round requested lists of Body Functions, Body Structures, Activities and Participation, and Environmental Factors influenced by physical therapy intervention. The responses were then translated into ICF categories. In the second round, the participants were provided with the resulting ICF categories, along with their frequencies. The participating physiotherapists were then asked to judge whether a named ICF category fell within their professional prevue. The third round was carried out accordingly. This process resulted in the selection of 124 ICF categories describing the most common patient problems managed by physiotherapists in the acute hospital. The selection consisted of 49 categories of the component Body Functions, 18 of the component Body structures, 34 of the component Activities and Participation, and 23 of the component Environmental Factors.

All categories of the ICF are quantified using the same generic 0–4 scale, with qualifier 0 representing no problem, and qualifier 4 representing that the problem is complete or pervasive. In general, Environmental Factors can act as a facilitator or a barrier. Therefore, the categories of this component have a valence, thus ranging from –4 to +4. The qualifier “not specified” is to be used if the information available is not sufficient, and “not applicable” if the category is not applicable.

As the metric properties of this scale are not yet sufficiently evaluated, ICF categories and their qualifiers have to be converted into items. These items must be specific to the situation, to the patient group whose problems are to be observed, and to the health professional group intending to use the ICF as an assessment tool. The method used to write items for those ICF categories involved a formal decision-making and consensus process (20), which integrated evidence gathered from a systematic review (21) and expert opinion (22). Three weeks prior to the consensus conference (April 29th –May 1st, 2004), all participants received a compilation of the second level of the ICF (German edition) (23), a complete manual, as well as the results of the systematic review and information about the consensus process. At the conference, they were to decide on item definitions for the a priori selection of 124 ICF categories, overseen by 6 experts from Switzerland, all of whom had working experience in clinical physical therapy in the 3 groups of health conditions cited, and expertise in development and testing of clinical measures. The 6 experts also had previous experience in the application of the ICF model and the ICF classification.

In the first step of the process (Fig. 1), the participants had to decide on which type of scale the given ICF category should be measured. This could be either dichotomous, indicating that impairment or limitation is present or absent, or polytomous, indicating a possible grading of limitation or impairment on a qualifier scale from 0 to 4, as noted above. Next, the participants had to find definitions for the extreme anchor points of the polytomous categories, i.e. the qualifiers 0 and 4. Once these anchor points were set, the participants had to decide which clinically meaningful steps would be appropriate for the intervening qualifiers 1, 2 and 3. The item qualifiers were framed without specific attribution to medical conditions or disabling factors, but instead focused on physiotherapists’ work in an acute hospital situation. As a consequence, the qualifier 0, attributable to persons without impairment or limitation, would represent the best possible outcome or functional status obtainable by therapy in the acute hospital, notwithstanding that a patient might still have potential for further improvement. A translated non-validated version of the operationalization is given in Appendix I.

Reliability testing

The study design was cross-sectional with repeated measurements in a convenience sample of 28 patients with neurological, musculoskeletal, cardiopulmonary and internal organ conditions, requiring physiotherapy in an acute hospital. Physiotherapy was assigned on prescription by the respective ward physician, and according to predefined clinical pathways. Patients were recruited at the University Hospital Zurich between June and October 2004. Patients with neurological conditions were recruited from the Departments of Neurology and Neurosurgery, patients with musculoskeletal conditions from the Departments of Orthopedics, Rheumatology and Surgery, and patients with cardiopulmonary and internal organ conditions from the Departments of Internal Medicine, Surgery and Heart Surgery. Sample size was determined by feasibility and precision considerations. Even given a very high or very low proportion of positive ratings, a sample size of 22–33 would be sufficient to detect a moderate kappa (0.5–0.6) with a power of 0.8 (24).

Approval of the project was first obtained from the local institution ethics committee, in accordance with the Declaration of Helsinki. Informed consent was obtained from all participants.

Measures

As described above, 124 items had been written, based on the 124 ICF categories most relevant to patient problems addressed by physiotherapists in the acute hospital. Items could either be graded on a dichotomous (0/1) or polytomous (0–4) scale.

Data collection procedure

Upon recruitment, the patients were interviewed twice, at an interval of not less than 24 h and not exceeding 72 h. The two interviews were conducted by 2 different physiotherapists trained in the application and principles of the ICF. Inclusion criteria were that patients must be inpatients of at least 18 years of age, whose main health condition required treatment by a physiotherapist. All patients had to have good knowledge of the German language. We excluded patients with severe cognitive impairment who were unable to give informed consent to the study.

There were 6 interviewers with expertise in application of the ICF classification. Each interviewer worked as physiotherapist in the corresponding departments of the University Hospital Zurich.

Anonymous and standardized data collection forms with consecutive numbers were provided. Prior to the interview, each patient’s medical record sheet was checked and relevant information on socio-demographic variables, diagnoses and assessments was extracted. The interviewers were instructed to collect the data from the most reliable source, so they were expected to ask the health professionals who were best informed about the condition of the patient being interviewed. Information that could not be retrieved from the records or from the health professionals was obtained directly from the patient. We assumed that both interviewers were likely to ask the therapist in attendance who had been responsible for assessment and treatment of the patient. Consequently, intra-rater and inter-rater reliabilities are similar in this study.

Each interviewer was blind to the results of the other.

Quality assurance procedures

Interviewers were trained in the course of a structured one-day meeting, and were provided with a manual. All data forms were then re-checked by a second person for completeness and plausibility. Patients who refused to participate were asked to complete a short questionnaire on socio-demographic variables, diagnosis, and to give a reason for their refusal to participate. Data were recorded by double entry. Raw data were inspected for accuracy and outliers.

Data analysis

Reliability was analysed using the percentage of raw agreement and Cohen’s kappa for nominal scales (25). Overall raw agreement was calculated by dividing the sum of the frequencies of the main diagonal of a contingency table by the sample size. As with other measures of agreement, the maximum possible value of kappa is 1. By convention, a kappa value of 0.81–1.00 is defined as almost perfect agreement, 0.61–0.80 as substantial and 0.41–0.60 as moderate, 0.21–0.40 as fair, 0.00–0.20 as slight, and values below 0.00 as poor agreement (25). Following the recommendation of Gardner (26), we reported the percentage of items with raw agreement or kappa of 0.7 and above. Although there are situations where a kappa below 0.7 can be acceptable, this arbitrary threshold restricts our reporting to items with unambiguously high agreement. Confidence intervals for kappa were calculated using a bootstrap re-sampling methodology proposed by Vierkant (27). These analyses were carried out with SAS® v.9.1 (Cary, NC, USA).

Results

Operationalization

Of the 124 items, 94 (76%) were dichotomous and 30 (24%) polytomous. Results per ICF category are given in Tables I–IV. To give an example of the operationalization, consider the qualifier scales for the ICF category Walking (d450), which was defined as follows:

• 0: More than 100 m, inside and outside (with or without walking device).

• 1: Independent walking inside between 10 and 100 m (with or without walking device), several times a day.

• 2: Walking independently in the ward, with or without walking device (10–100 m).

• 3: Can walk independently in his/her room (up to 10 m), with or without walking device.

• 4: Incapable of walking independently.

Patients

Twenty-eight patients were included in the study. Patients’ ages ranged from 27 to 88 years, with a median age of 61 years and a mean of 59.6 years. Sixteen of the patients (57%) were female. Ten patients had neurological conditions, such as stroke, Parkinson’s disease, multiple sclerosis, and brain injury. Ten patients had musculoskeletal conditions, disease of the joints, such as primary and secondary arthrosis, and fractures of the extremities. Because of withdrawals in the second interview, we could include only 8 patients with cardiopulmonary and visceral conditions, such as ischaemic heart disease, disease of respiratory system and tumours of internal organs. Incomplete data were not included in the analysis.

Out of all items, 56% of the 94 polytomous and 68% of the 30 dichotomous items had a raw agreement of 0.7 and above. Thirty-six percent of all polytomous and 34% of all dichotomous items had a kappa coefficient of 0.7 and above. The detailed results are presented in Tables I–IV for each component; we maintain the structure of ICF in chapters, but list the categories in decreasing order of percentage of agreement.

In the component Body Functions (Table I), all 10 polytomous items (100%) had a raw agreement of 0.7 and above, and 9 had a kappa of 0.7 and above (90%). All of the 39 dichotomous items had a raw agreement of 0.7 and above, and 28 had a kappa of 0.7 and above (72%).

In the component Body Structures (Table II), all 18 items (100%) had a raw agreement of 0.7 and above, and 12 had a kappa of 0.7 and above (67%).

In the component Activities and Participation (Table III), a total of 7 of the polytomous items (50%) had a raw agreement of 0.7 and above, and 4 had a kappa of 0.7 and above (29%). Thirteen dichotomous items (65%) had a raw agreement of 0.7 and above, 2 had a kappa of 0.7 and above (10%).

A total of 3 (25%) polytomous items of the component Environmental Factors (Table IV) had a raw agreement of 0.7 and above, and 15 dichotomous items (44%) had a raw agreement of 0.7 and above. No polytomous or dichotomous item had a kappa of 0.7 and above.

Discussion

To the best of our knowledge, this is the first attempt to operationalize ICF categories in such a manner that the resulting items are useful and reliable when applied by physiotherapists in the acute hospital situation. In the components Body Functions and Body Structures more than two-thirds of the operationalized categories showed substantial agreement as measured by the kappa coefficient. It can be argued that diagnosis by physiotherapists is based mainly on their assessment of body functions (28). It is obvious that most of the categories of Body Functions and Body Structures that proved to have less than substantial agreement describe issues (such as Blood vessel functions (b415) and Haematological system functions (b430)) that are beyond the professional prevue and daily practice of physiotherapists.

In the components Activities and Participation and Environmental Factors, agreement of the operationalized categories was not entirely satisfactory, with the exception of the chapters Mobility and Self-Care, where agreement ranged from moderate to almost perfect. Again, this is probably due to the scope of professional expertise of physiotherapists, who are especially proficient in assessing mobility issues, such as Walking (d450), and activities of daily living, such as Eating (d550) or Drinking (d560). Indeed, within the chapter Mobility, polytomous items obtained even better agreement than dichotomous items, thus showing that the lack of agreement was not necessarily caused by a flawed operationalization, but arose due to item definitions that were not sufficiently clear for the user. Moreover, the ICF provides 2 different constructs for the categories of the component Activities and Participation, “Capacity” and “Performance” (14). “Capacity” pertains to a patient’s limitation with or without receiving assistance. In contrast, “Performance” describes the problem in the person’s current environment, i.e. the limitation a person is experiencing, even with the use of assistive devices. Even though the interviewers were advised always to assess the patient in the context of “Capacity”, this may have proven difficult for categories manifesting mainly beyond the acute hospital environment, such as Driving (d475), Doing housework (d640) and Acquiring, keeping and terminating a job (d845). For this reason, the ICF recommends using a standardized environment for the assessment of “Capacity”. While the acute hospital indeed constitutes a kind of standardized environment, the extent of the standardization of assessment must evidently go even further. For example, when assessing categories of the component Activities and Participation it might be necessary to evaluate both “Capacity” and “Performance”, both with and without assistance, since these are major issues for goal setting (29, 30).

Although Environmental Factors cannot be influenced directly by physiotherapists, physiotherapists are well aware of their potential impact on outcome and prognosis (19). It is therefore very important for physiotherapists to know the influence of Environmental Factors. Arguably, the ICF provides very general and broad definitions for the categories of the component Environmental Factors. To give an example, Products and technology for personal use in daily living (e115) includes all general and assistive products, such as clothes, textiles and furniture, but also prosthetic devices or remote control systems. Therefore, the reliability of this category directly depends on how explicitly the item is described. Any fundamentally reliable assessment of Environmental Factors within the ICF has to delve more deeply into the particulars of definitions and details.

In interpreting these results, it must always be borne in mind that indices of agreement, such as the kappa coefficient, are artificially lower in populations with a restricted spectrum of the measured characteristic as in the present. Nonetheless, kappa provides a good overall estimate of the chance-corrected agreement, at the risk, however, of reducing the data to a single number that can be interpreted only if the underlying contingency table is examined and the clinical context is considered (31, 32). More sophisticated methods to assess reliability, such as log-linear models, might eventually find use in larger scale investigations of this sort (33).

Our study is not without limitations. For reasons of feasibility, the sample size had to be restricted, which necessarily yielded agreement estimates with low precision. Sample size was determined to yield confidence intervals not including the null point. Arguably, it would have been equally appropriate to test the difference relative to a pre-set threshold kappa such as 0.5, but this would have required 10–50-fold greater sample sizes. Still, the precision was sufficiently high to differentiate items with good agreement from those with low agreement. A posteriori power estimation using the power diagrams by Donner & Eliasziw (34) shows that at the present sample size and at an alpha level of 0.05 test power was adequate (0.8) for detecting a substantial agreement, i.e. a kappa value of 0.6 and above. Another limitation pertains to the definition of items, as alluded to above in the context of Environmental Factors. In most categories we decided to adopt the definition of the second level, which was, however, in a few instances insufficiently specific. To give an example, the second-level category Lifting and carrying objects (d430) is defined as rising up an object or taking something from one place to another; this definition clearly mixes the 2 concepts of “Lifting” and “Carrying”. In contrast, the third-level categories differentiate fully between the 2 concepts. Most probably, their use would have resulted in the writing of more reliable items, albeit at the expense of a more complex questionnaire.

Future research directions

This study examined the reliability of single items based on the ICF. In practice, functioning is often assessed by physiotherapists with the help of composite scales. While those scales may be valid and reliable, they tend to measure only single aspects of functioning, and they are, by definition, masking the effects of single items. Moreover, most of the established measures have not been based on a comprehensive framework such as the ICF, but arose from the requirements of particular aspects of functioning. With this in mind, there are several potential future directions of research to develop ICF-based assessments. First, it must be recalled that subsets of the items presented here could be combined to form new measures. This may scarcely be necessary for all the items, since there are already a number of established measures perfectly meeting their purpose. On the other hand, there may be a need for new measures to be used in specific situations, such as the acute hospital, and by physiotherapists. Validity and objectivity of such new measures would then have to be ascertained. In addition, the present items can be used “as they are” to document patients’ categorical functioning profile and to highlight those aspects of functioning expected to improve through therapy. The usefulness and applicability of this approach has yet to be studied.

In conclusion, physiotherapists should adopt the ICF as a unifying framework in order to be able to communicate patients’ needs in a language that is understood by all health professionals. It has to be emphasized that the assessment of functioning should always be a team effort, with each profession contributing. ICF-based items should be the basis of a common language in the acute hospital setting. The World Health Organization (WHO) encourages future ICF users to develop clinical standards and to assign clinically meaningful and appropriate wording to its existing qualifier frame. We have now provided proof of principle that the ICF can be adapted to the professional and setting-specific needs of physiotherapists who can reliably use the operationalized ICF as a checklist and tool to assess patients and monitor the results of interventions. Further research is directed towards combining these items into single scales.

Acknowledgements

The authors would like to thank the Swiss Physiotherapy Association, Sursee, Switzerland for providing financial support for the operationalization conference in Ermatingen (CH). We also thank the experts involved in the conference: Ruud Knols, Esther Kramer, Roland Paillex, Barbara Rau, Veronika Schoeb and Markus Wirz. Also, Andrea Camenisch, Désirée Forster, Liz Penzkofer, Doris Rüedi, Martina Schütz, Marguerite Stauffacher, the interviewers and Hiltrud Schmid, Marie-Louise Squaratti, Astrid Zanettin and the physiotherapy staff and the head therapists of all wards involved at the University Hospital Zurich for assistance in data collection. We also thank Leanne Pobjoy and Dr Paul Cumming for their help in preparing the manuscript.

References