Accepted Oct 6, 2020; Epub ahead of print Oct 19, 2020

J Rehabil Med 2021; 53: jrm00147

Correspondence address: Tiebin Yan, Department of Rehabilitation Medicine, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, 107 Yan Jiang West Road, Guangzhou 510120, China. E-mail: yantb@mail.sysu.edu.cn

DOI: 10.2340/16501977-2756

As the Chinese International Classification of Functioning, Disability and Health Rehabilitation Set became widely used in rehabilitation institutions in China, it was found to be time-consuming and inefficient for a single rater to complete the entire evaluation in a single setting. Team evaluation provides an alternative team approach for busy professionals, especially when used to evaluate patients with complex problems or poor communication ability. The whole set of 30 evaluation categories was divided among a hypothetical team consisting of a physician, a nurse, a physiotherapist and an occupational therapist, with 6 categories assigned to the physician, 7 to the nurse, 9 to the physiotherapist, and 8 to the occupational therapist. Each professional in the team rated categories closely related to their daily work. The team evaluation approach promises to better share the evaluation workload, perhaps improving the accuracy of evaluations and strengthening interdisciplinary collaboration in the clinic.

The International Classification of Functioning, Disability and Health (ICF) is a standardized framework for describing and understanding health, functioning and disability, which was endorsed by the World Health Assembly in 2001 (1). An individual’s functioning and level of disability may be recorded by selecting the appropriate ICF category and using its corresponding code and qualifiers (2). With the increasing number of people with disability, there have been calls to strengthen rehabilitation services in the world’s health systems (3). The ICF can be used as a unifying model for the conceptual description of rehabilitation strategies and as a useful tool for quantifying the effectiveness of rehabilitation services (4, 5). In order to facilitate clinical practice, the ICF Core Sets specify shortlists of ICF categories relevant to a particular health condition or setting. They have been rigorously developed to enhance the utility of the ICF in clinical practice and research (6, 7).

The ICF Rehabilitation Set (ICF-RS), with 30 categories, is a minimal set for reporting and assessing functioning and disability in clinical rehabilitation and in research (8–10). Its domains cover body functions, activities and participation. As the ICF Rehabilitation Set is essentially a list of categories, it needs to be considered from a measurement perspective (11, 12). A Chinese version of the ICF-RS has been developed, describing how to use it to measure aspects of functioning (13). The Chinese psychometric properties have shown good reliability and validity in multi-centre research, and they have been widely used as a functional evaluation tool in China (14).

However, as the Chinese ICF-RS became widely used, some problems were observed in its application. Using it involved clinical examination and interviews. That was time-consuming. A single rater would have difficulty finishing the entire evaluation in a single sitting, especially with a patient with complex complaints or poor language expression. Some raters reported that even 30 min to complete one evaluation was too long in a busy rehabilitation department. That undoubtedly limited the instrument’s application. In addition, the categories of the ICF-RS cover a wide range of functioning, and some categories were not closely related to the job responsibilities of some of the rehabilitation professionals using it. This may have resulted in a further decrease in overall attention and enthusiasm in its use.

A team approach may be a way to increase the evaluation efficiency and facilitate the process. In such an approach the assessment would be completed by a team of professionals, with each rater handling some categories consistent with his or her routine job responsibilities in the clinic. Ottiger has reported a new multidisciplinary observation scale for stroke patients, in which neurologists, nurses, physiotherapists, occupational therapists and speech therapists each rate assigned domains (15). A group led by Catz has reported a revised version of the Spinal Cord Independence Measure evaluated by a team including an occupational therapist, a nurse and a physiotherapist (16). Both of those scales were found easier and more convenient to implement, as each professional in the team rated their own part of the scale. As yet, however, there have been no reports about a team version of the ICF Rehabilitation Core Set.

This study aimed to develop a team evaluation approach to using the Chinese version of the ICF-RS. It was designed to be applied in rehabilitation institutions in China, with the professionals in the team rating categories closely related to their daily work. The team approach was supported by a mobile app, which the team members could use to record and share their evaluation results (17).

Overview

The objective of the team approach was to divide the original 30 categories into sections, and to have each section evaluated by a different rehabilitation professional in the team: one professional for each category. The study was conducted using a modified Delphi technique and an expert panel (18, 19). The process had 3 steps, as follows. (i) To establish a panel of ICF experts, which would screen candidate professionals for participating in the rating team through discussion. (ii) A first-round Delphi survey was conducted in which the participants were asked to score the candidate professionals using a 5-point Likert Scale in terms of their suitability for evaluating particular ICF-RS categories (20, 21). Table I presents the list of categories. A first draft of the team roster was then prepared according to the results combined with discussion among the experts. (iii) A second-round survey was then conducted in which the participants were asked to rate their agreement with the category assignments. Ethics approval for this procedure was obtained from the ethics committee of Sun Yat-sen Memorial Hospital (Guangzhou, China). Written informed consent was obtained from all participants.

Screening basis

An expert panel was set up to screen professionals for the rating team. The experts were all from the Rehabilitation Department of Sun Yat-sen Memorial Hospital, Sun Yat-sen University (Guangzhou, China). They were a senior physician, a senior nurse, a senior physiotherapist, and a senior occupational therapist. All of the experts had at least 5 years of experience working in a rehabilitation department and had worked with the ICF for more than 3 years.

The candidate professionals were evaluated through item-by-item discussion of each category. Two main questions were proposed to help the screening: (i) What kinds of professionals are in contact with or pay attention to this category’s content in their daily work? (ii) What kinds of professionals are involved with this category using other scales in a rehabilitation clinic? The discussion was based on the definition and evaluation content of each category, but constrained, to some extent, by the availability of professionals in rehabilitation departments in China.

First-round Delphi survey

Participants. Purposive snowball sampling was used to enrol participants (22). The sample size of participants was not clearly defined for the Delphi survey, and many studies reported the participant numbers ranged from 10 to 50 (23–25). This study aimed to enrol 25–30 experts for the Delphi survey. The initial recruiting exploited an online expert group specialized in ICF research in China. Group members were asked to volunteer. Those who volunteered were then asked to recommend other relevant ICF experts. Some experts who had participated in a previous Delphi survey about the psychometric properties of Chinese ICF-RS (13) were also selected. The members of the expert panel did not participate in the 2 rounds of Delphi surveys.

Geographical representation was taken into account in the recruitment process. Those included were all registered senior physicians, senior nurses, attending or senior therapists. All had at least 10 years of clinical experience in hospitals, including at least 5 years of clinical experience in rehabilitation and also 2 years of experience related to the ICF. One nurse who was not at the senior level was included because the ICF-RS was the topic of her doctoral research.

Design

An invitation letter, describing the survey’s purpose, method, content and matters needing attention, was sent to the participants by e-mail or WeChat (Weixin, Beijing, Tencent Technology Co., LTD), inviting them to participate. A questionnaire was then sent to those who agreed to participate. The questionnaire had 2 parts: expert consultation for team evaluation; and personal information about the respondent.

In the expert consultation part, the respondents were asked to evaluate if a physician, a nurse, or a therapist was suitable for assessing each ICF-RS category. The sub-questions for each category were designed based on the types of candidate professionals. The respondents used a 5-point response format from “strongly unsuitable” (quantified as 1) to “strongly suitable” (quantified as 5) for each question. Only one option could be selected in response to each question. The evaluation content and evaluation options for each category of the ICF-RS were provided for the respondents’ reference (Fig. 1).

The personal information collected about the participants included their name, age, gender, education, profession, title, years using the ICF and (self-assessed) degree of familiarity with the ICF.

It took approximately 10–15 min to complete the questionnaire. The respondents were asked to return it within 2 weeks. If no reply had been received by 3 days before the deadline, the respondent was reminded by the coordinator. All of the responses collected were checked, and if there were any omissions the respondent was contacted and asked to complete the responses. Eventually, a questionnaire with missing responses was deemed invalid.

If only one type of professional was rated as “basically suitable”, “suitable” or “strongly suitable” (a score of 3, 4 or 5) for assessing a particular category by more than 75% of the respondents, and if the mean suitability score was ≥ 3.6, that type of professional was considered appropriate for that category without any further discussion (13, 26, 27). In most cases, however, more than one type of professional met that standard. In those cases, the category assignment was considered through discussion among the expert panel. The mean suitability scores, the balance of workload and the relationship with a profession’s routine work were major factors to be considered in assignment. If no profession met that standard, then the profession rated as “basically suitable”, “suitable” or “strongly suitable” (a score of 3, 4 or 5) was chosen by the majority of the respondents.

Second-round Delphi survey

The respondents who participated in the first-round survey received a second questionnaire by email or WeChat. That questionnaire reported the results of the first-round survey and the initial category assignments. The respondents were asked to consider to what extent they agreed with the assignments. They replied using a 5-point Likert scale: 1, “strongly disagree”; 2, “disagree”; 3, “doesn’t matter”; 4, “agree”; and 5, “strongly agree”. Two weeks were allowed to return their responses, and a reminder was sent 3 days before the deadline by a coordinator. According the standard for consensus generally used in such research (13, 26, 27), an assignment was accepted if it had a mean agreement score ≥ 3.6 and more than 75% of the respondents chose “agree” or “strongly agree” (Likert scores of 4 and 5). Any category with a mean agreement score below 3.6 or total agreement among the respondents of less than 75% was re-assigned in discussion among the expert panel considering the results from the first Delphi round and based on their clinical experience.

Statistical analysis

Version 20.0 of the SPSS software suite (IBM, Armonk, NY, USA) was used for statistical analysis. The suitability and expert agreement scores for each category were calculated and expressed as mean (standard deviation (SD)). The higher the mean, the better the agreement about a category assignment.

Screening of professionals

The Chinese version of the ICF-RS was generated to be used in rehabilitation institutions in China, but many of these institutions do not employ personnel across the full set of professional categories. As a result, some of the work of speech therapists, psychotherapists and social workers is, in fact, done by other types of professionals. Through discussion, it was decided that a speech therapist was only related to category “d550 Eating”. A psychotherapist was related to the category “b152 Emotional functions”, and a social worker was related to category “d850 Remunerative employment”. All of those categories could, it was decided, also be evaluated by other sorts of professionals instead. Thus, speech therapists, psychotherapists and social workers were excluded from the evaluations. Physicians, nurses, physiotherapists and occupation therapists were eventually selected to participate in the rating team.

Results of the first-round survey

Of the 31 experienced rehabilitation professionals initially recruited for the Delphi survey, 30 (96.8%) completed the questionnaire adequately (11 physicians, 9 nurses and 10 therapists). The demographics and qualifications of these professionals are shown in Table II.

Among the 30 categories, there were 22 in which only one type of professional got a suitability percentage ≥ 75% and a mean suitability score ≥ 3.6. Those selections were accepted without further discussion. Of the 22, there were 4 categories for which a physician was clearly considered the most suitable. 2 was assigned to a nurse, 9 were assigned to a physiotherapist, and 7 to an occupational therapist. There were 8 categories in which at least 2 types of professionals got a suitability percentage ≥ 75% and a suitability score ≥ 3.6. Through discussion, the expert panel assigned them to be evaluated by a physician or a nurse in an attempt to share the workload. “d240 Handling stress and other psychological demands” was assigned to the physicians. “d510 Washing oneself”, “d520 Caring for body parts”, “d530 Toileting” and “d550 Eating” were assigned to nurses. The nurses also received “d230 Carrying out daily work” and “d570 Looking after one’s health”, although 3 types of professionals were considered suitable. The category “b130 Energy and drive functions” met that standard with 4 types of professionals, but it was assigned to the physicians (Fig. 2). The details of the initial category assignments, the suitability rate and the mean suitability scores are listed in Table III.

Results of the second-round survey

The 30 respondents who participated in the first expert survey also received the second-round questionnaire. Of these, 29 (96.7%) responded adequately, including 10 physicians, 9 nurses and 10 therapists. In the second round, 29 categories received more than 75% agreement, with the mean agreement scores ranging from 3.79 to 4.72 (Table IV). Category “d230 Carrying out daily routine” received less than 70% agreement, so it was assigned to the occupational therapists in view of the fact that it was the assignment that received the highest mean score in the first-round survey.

Results of the second-round survey

The 30 respondents who participated in the first expert survey also received the second-round questionnaire. Of these, 29 (96.7%) responded adequately, including 10 physicians, 9 nurses and 10 therapists. In the second round, 29 categories received more than 75% agreement, with the mean agreement scores ranging from 3.79 to 4.72 (Table IV). Category “d230 Carrying out daily routine” received less than 70% agreement, so it was assigned to the occupational therapists in view of the fact that it was the assignment that received the highest mean score in the first-round survey.

This study attempted to develop a team approach to facilitate use of the Chinese version of the ICF-RS in evaluations. Team evaluation provides an alternative approach for busy professionals, especially when used to evaluate patients with complex problems or poor communication ability. The professional analysis assigned 6 categories to physicians, 7 to nurses, 9 to physiotherapists and 8 to occupational therapists.

The Delphi technique used in this study successfully developed a consensus among the experts contributing through repeated information exchange and feedback (28). Unlike with the classic Delphi technique, the modified Delphi method used here did not apply the conventional 4 rounds (29–31). The consultation ended after 2 rounds, as the experts had reached a consensus. A review of 30 studies using the Delphi technique reveals they reached consensus after 1–5 rounds. Among them are 14 studies which reported requiring only 2 Delphi rounds (32). They were mostly generating standards, guidelines and scales (33–35). Some reports described using intermediate face-to-face meetings between Delphi survey rounds (36) or the involvement of different expert panels in the consensus process (37). As there were no reports about how to divide the same scale into sections to generate another rating version, in this study expert panel discussion was combined with 2 rounds of Delphi survey to successfully develop the team approach.

Finally, relatively few categories were assigned to the physicians and nurses. However, as the goal of the team approach was to share out the evaluation tasks to facilitate the process while ensuring the relevance and the accuracy of the results, which was considered satisfactory. All of the assignments met the consensus standard and seemed relevant to actual job responsibilities in a clinic.

The category “d240 Handling stress and other psychological demands” refers to regulating and controlling one’s mental state in order to accomplish tasks that may be complicated by stress and/or distraction. It emphasizes completing tasks under stress rather than simple physical movement and coordination. Physicians pay more attention to patients’ psychological adjustment than therapists normally would, so assigning this category to physicians received good support in the second round of the survey. The category “b130 Energy and the drive function” refers to physical fitness and the executive initiative required to achieve general goals and meet special needs. It too emphasizes the patient’s mental state, but also physical strength and energy. Because low energy and fatigue in the rehabilitation process are often related to inconsistent training intensity, or perhaps to heart failure or other factors, physicians can better grasp the patient’s basic difficulty and its causes and carry out more comprehensive and accurate assessments. The categories “d510 Washing oneself”, “d520 Caring for body parts”, “d530 Toileting” and “d550 Eating” were developed based on the content of the Modified Barthel Index. That index is a rating scale for ability in the activities of daily living, which is commonly used in rehabilitation in China. Research has shown that Barthel Index scoring by nurses agrees well with that by physicians (38).

In the second round of the survey the assignment of category “d230 Carrying out daily routine” generated less than 70% agreement. It was eventually assigned to the occupational therapists. The category refers to planning, scheduling and completing the daily routine, including controlling one’s activity level. The goal of occupational therapy is to create meaningful and satisfying lives through targeted and well-designed activities. Much of occupational therapy relates to patients’ daily lives and work, improving their quality of life and returning them to their families and society (39). Hence, d230 was considered consistent with the scope of practice and professional responsibilities of occupational therapists.

While this study related entirely to China’s rehabilitation context, medical professionals and their skills and patients and their problems have many similarities worldwide. This study’s assignments may therefore be helpful for rehabilitation personnel attempting to apply the ICF-RS in many different settings. The professionals who completed the questionnaire all had rich experience and deep understanding of rehabilitation practice. It might, however, have been better to have provided a job responsibility guide for their reference. That might have helped them to make easier and more consistent choices. In addition, although consensus was reached in 2 rounds, a third round of Delphi surveying would perhaps have been useful for research purposes. It is possible that some deviation from the consensus might have emerged in a third round. More research is needed to demonstrate that the team evaluation approach developed here has validity consistent with that of the conventional single-rater procedure. This will be the next stage of our research.

A new team approach to applying the ICF-RS in evaluation was generated in this study. The whole set of 30 evaluation categories was divided among a hypothetical team consisting of a physician, a nurse, a physiotherapist and an occupational therapist, with 6 categories assigned to the physician, 7 to the nurse, 9 to the physiotherapist and 8 to the occupational therapist. A limited number of categories can be finished easily during a professional’s work day without undue pressure. Improving the evaluation efficiency of the individual team members inevitably facilitates the implementation of the ICF-RS in the clinic. The team evaluation approach promises to better share the evaluation workload, perhaps improving the accuracy of evaluations and strengthening interdisciplinary collaboration in the clinic.

The authors thank all the participants in the Delphi study. Funding. This study was funded as a Ministry of Education Tian Cheng Hui Zhi Innovative Education Project (number 2018A01026) and as a Guangzhou Science and Technology Plan Project (number 201704020140).

The authors have no conflicts of interest to declare.