Fieke S. Koopman, MD1, Merel A. Brehm, PhD1, Yvonne F. Heerkens, PhD2, Frans Nollet, MD, PhD1 and Anita Beelen, PhD1

From the 1Department of Rehabilitation, Academic Medical Center, University of Amsterdam, Amsterdam and 2Dutch Institute of Allied Health Professions, Amersfoort, and Department of Occupation & Health, HAN University of Applied Sciences, Nijmegen, The Netherlands

OBJECTIVES: (i) To compare the content of the Fatigue Severity Scale and the subscale “subjective experience of fatigue” of the Checklist Individual Strength, and (ii) to assess the reliability of both questionnaires in polio survivors.

DESIGN: Repeated-measures at a 3-week interval.

SUBJECTS: Consecutive series of 61 polio survivors.

METHODS: Concepts contained in the questionnaire items were linked to the International Classification of Functioning, Disability and Health (ICF), using standardized linking rules. Reliability analyses included tests of internal consistency, test-retest reliability and measurement error.

RESULTS: Questionnaires differed in the extent to which they measured other than fatigue-related aspects of functioning (represented ICF components: “Body functions”: 50% and 80%, “Activities and Participation”: 30% and 0%, for the Fatigue Severity Scale and Checklist Individual Strength, respectively). Internal consistency and test-retest reliability were considered acceptable, while measurement error was large (Cronbach’s α: 0.90 and 0.93, intraclass correlation coefficient: 0.80 and 0.85, smallest detectable change: 28.7% and 29.4% for the Fatigue Severity Scale and Checklist Individual Strength, respectively).

CONCLUSION: Considering the acceptable clinimetric properties, we conclude that both the Fatigue Severity Scale and the Checklist Individual Strength can be applied in research on post-poliomyelitis syndrome when measuring fatigue. However, because the 2 questionnaires differ in content they cannot be used interchangeably.

Key words: fatigue; post-poliomyelitis syndrome; questionnaires.

J Rehabil Med 2014; 46: 761–767

Correspondence address: Fieke S. Koopman, Department of Rehabilitation, Academic Medical Center, University of Amsterdam, PO 22660, 1100 DD Amsterdam, The Netherlands. E-mail: s.koopman@amc.uva.nl

Accepted Mar 18, 2014; Epub ahead of print Jun 17, 2014

INTRODUCTION

Fatigue, defined as an overwhelming sense of tiredness, lack of energy and feeling of exhaustion (1), is one of the most frequently reported complaints in persons with post-poliomyelitis syndrome (PPS) (2, 3). Compared with healthy individuals, persons with PPS experience high levels of fatigue (4) that persist over time (5). Because fatigue in PPS impacts negatively on quality of life (6) it is a relevant problem in this patient group.

Fatigue is commonly quantified by means of self-report questionnaires. Studies have used different questionnaires to investigate the prevalence, severity and natural course of fatigue in PPS (5), as well as to evaluate the effects of treatment to reduce fatigue (7–10). The most frequently used instrument to measure fatigue in PPS is the Fatigue Severity Scale (FSS) (11), which was originally developed and tested in multiple sclerosis and systemic lupus erythematosus (11). More recently, the Checklist Individual Strength (CIS20) has been introduced for assessing fatigue within PPS research (12). The CIS20 was formerly developed for chronic fatigue syndrome (CFS) (13), and has been recommended for research purposes by the International Chronic Fatigue Syndrome Study Group (14). Although the CIS20 consists of 4 sub-scales related to fatigue, the sub-scale subjective experience of fatigue (CIS20-F) is generally used as the primary outcome measure in studies assessing fatigue (15, 16). Over the past decade, this scale has also become increasingly popular for assessing fatigue in neuromuscular disease (17, 18).

The choice of fatigue instrument is dependent on the construct to be measured. Because fatigue is a complex, multidimensional problem (5), care should be taken in comparing the outcomes from different fatigue instruments. That is, instruments may vary greatly with regard to the International Classification of Functioning, Disability and Health (ICF) categories covered (19), indicating that they measure different constructs. At face value, the FSS and CIS20-F appear to measure different constructs (i.e. aspects of fatigue), although detailed information on their content validity is lacking. The current study therefore assessed and compared the conceptual properties of the FSS and the CIS20-F.

In addition to clarifying the conceptual properties of the FSS and CIS20-F, their clinimetric properties on reliability need to be known for valid assessment of fatigue over time, or adequately to distinguish between patients with different levels of fatigue. The reliability of the FSS has been investigated in 3 studies in PPS, and has shown acceptable to good internal consistency and test-retest reliability (20–22). The CIS20 has been validated in CFS (23), multiple sclerosis (24) and the working population (25). However, the clinimetric properties on reliability of the CIS20-F in PPS are unknown.

The aims of the current study were: (i) to perform a content comparison on the FSS and the CIS20-F questionnaires for measuring fatigue; and (ii) to assess the reliability (internal consistency, test-retest reliability and measurement error) of these questionnaires in polio survivors.

METHODS

Participants and procedures

A consecutive series of 80 polio survivors known to the outpatient clinic of Rehabilitation of the Academic Medical Centre in Amsterdam were invited to participate in this study. Written information on the study, a brief survey on demographic variables, 2 fatigue questionnaires (FSS and CIS20-F) and a stamped return envelope were sent out to these patients. To assess test-retest reliability and measurement error, a second package of questionnaires was sent (within 3 weeks) to participants who returned the first questionnaire.

According to Dutch law, studies involving human subjects must undergo medical ethics review if they are subject to the Medical Research Involving Human Subjects Act (WMO). A study involving the completion of a questionnaire does not fall within the scope of the Act and, therefore, approval from the medical ethics committee was not required for our study.

Fatigue questionnaires

The FSS (11) is a self-administered questionnaire measuring fatigue. It consists of 9 statements rated on a 7-point scale, ranging from 1 (strong disagreement) to 7 (strong agreement). The total score is calculated as the mean of the responses to the 9 statements. A score of 4 or higher indicates a moderate to high level of fatigue (26).

The CIS20 (13) is a 20-item questionnaire that contains 4 sub-scales (subjective experience of fatigue, reduction in concentration, reduction in motivation and reduction in physical activity) related to fatigue. For the purpose of this study, only the sub-scale CIS20-F was used. The CIS20-F consists of 8 statements, and participants have to indicate on a 7-point scale (ranging from 1 to 7) to what extent the particular statement applies to them. The total score is calculated as the sum of the responses to the 8 statements. A higher CIS20-F score indicates a higher degree of fatigue. Based on research with CFS patients, a score of 35 or higher indicates severe fatigue (23).

Content comparison of FSS with CIS20-F

The content of the FSS and the CIS20-F were compared by identifying, quantifying and comparing the concepts contained in both questionnaires, using the ICF (27) as a reference. This method of linking outcome measures to the ICF, based on standardized linking rules developed by Cieza et al. (28, 29), is widely used to explore the content of a measurement instrument. Following the linking rules, each meaningful concept (MC), i.e. a unit of text that conveys a single theme within an instrument item, was linked to the most precise corresponding ICF category, identified with its alphanumeric code that indicates the component of the ICF: Body Functions (b), Body Structures (s), Activities and Participation (d), Environmental Factors (e), and Personal Factors. Perceptions were not separately coded if they were inextricably bound up with other MCs (30). For example, in item 1 of the CIS20-F “I feel tired” “feeling tired” was considered as 1 MC. In item 6 of the CIS20-F “physically I feel I am in a bad condition” “feeling” and “physical condition” were considered different MCs, and therefore coded separately.

Two health professionals working in rehabilitation research (FK, MB) independently extracted and linked MCs contained in FSS and CIS20-F items. Disagreements (defined as the identification of different MCs or the assignment of different ICF categories) were resolved by a consensus procedure, if necessary, with a third author trained in the ICF linking rules (YH).

Analyses

Returned questionnaires were checked for completeness, and they were excluded from the analysis if one or more of the items were missing. Patient characteristics and mean scores of the FSS and CIS20-F were analysed with descriptive statistics.

Content analysis. For both questionnaires, the results of the linking process were presented at the item level (i.e. a presentation of MCs contained in the items of both questionnaires and the linked ICF categories) and at the ICF component level (i.e. a representation of the ICF components in both questionnaires). Furthermore, the content density and content diversity were determined for each questionnaire (31). The content density is a measure of multidimensionality within the item structure of an instrument and is calculated as the ratio of the total number of MCs and the total number of items in the questionnaire. Ratios close to 1 mean that each item contains 1 MC, while higher values show that there are several MCs contained within each item. The content diversity, which is a measure of reach or bandwidth of a questionnaire with regard to ICF categories covered, is the ratio of the number of different ICF codes and the number of MCs. Values close to 1 indicate that MCs of the measure correspond to a different ICF category. Values closer to zero indicate that several MCs in the questionnaire relate to the same ICF category.

Reliability of the FSS and CIS20-F. Internal consistency (32) was assessed using Cronbach’s α coefficient and item-total correlation (ITC). An α coefficient > 0.70 is considered sufficient, an α coefficient > 0.80 is considered good for the purpose of group comparisons, and an ITC > 0.30 is considered acceptable (33).

Test-retest reliability was assessed with the intraclass correlation coefficient (ICC) and 95% confidence intervals (95% CI) of the ICC. A 2-way mixed effects model with an absolute agreement definition was used, assuming that included patients are a random selection of the population and the raters (i.e. items) are fixed (32). An ICC value > 0.70 is considered acceptable (33) and a lower boundary of the 95% CI of the ICC > 0.75 is considered to indicate excellent reliability (34). Systematic differences between the 2 assessments together with the 95% CI were analysed for the total scores on the FSS and the CIS20-F using paired t-tests.

Measurement error was expressed with the standard error of measurement (SEM), calculated as √(varo + varres), where varo is the variance due to systematic differences between occasions and varres is the random error variance (32). From the SEM, the smallest detectable change (SDC), i.e. the amount of change between 2 assessment scores that is reliably detectable above measurement error) (35) was derived, calculated as 1.96 × √2 × SEM (32).

Analyses were carried out using the Statistical Program for Social Sciences (SPSS) for Windows, version 18. For all tests, a p-value < 0.05 was considered significant.

RESULTS

Content comparison of FSS with CIS20-F

The 2 authors (FK, MB) reached full consensus regarding identification of different MCs from the items of both questionnaires (30 MCs for both questionnaires together). There was disagreement in the assignment of ICF codes for 9 MCs (30%), and a third author (YH) determined the ICF code for these MCs.

Tables I and II summarize the results of the content comparison. The total number of identified MCs was 20 for the FSS and 10 for the CIS20-F. With a high number of MCs per item, the FSS had a higher content density ratio than the CIS20-F (2.2 and 1.25, respectively). Also, the content diversity ratio of the FSS was higher compared with the CIS20-F (0.45 and 0.30, respectively). The proportion of MCs that could be linked to an ICF category was 80% for both questionnaires. The MCs of the CIS20-F were linked to 3 different categories, all within the ICF component “Body Functions”. Ten MCs (50%) of the FSS were linked to the ICF component “Body Functions” (to 3 different categories) and 6 MCs (30%) were linked to the ICF component “Activities and Participation” (to 6 different categories). The ICF components “Environmental Factors” and “Personal Factors” were not represented in the questionnaires.

MCs in both questionnaires were found to be linked to 2 different fatigue-related ICF categories: “b1300 Energy level”, which is defined as “mental functions that produce vigour and stamina” and “b4552 Fatigability”, defined as “exercise tolerance functions related to susceptibility to fatigue, at any level of exertion”. Most items (6 out of 8) of the CIS20-F contained only 1 MC, which were all linked to 1 of the 2 fatigue-related ICF categories. Most items (8 out of 9) of the FSS contained both MCs linked to 1 of the 2 fatigue-related ICF categories, as well as 1 or more MCs linked to non-fatigue-related ICF categories (within the ICF components “Body Functions” and “Activities and Participation”). For example, item 9 of the FSS contains the fatigue-related ICF category “b1300 Energy level” as well as the non-fatigue related ICF categories “d850 Remunerative employment”, “d760 Family relationships” and “d9 Community, social and civic life”.

Descriptive data of the FSS and CIS20-F

From the 80 questionnaires that were sent out, 61 were returned (76%). Baseline characteristics of these 61 patients are shown in Table III. Test-retest response rate was 90%. On the first and second assessment, respectively, 3 (5%) FSS questionnaires and 3 (5%) CIS20-F questionnaires and 1 (2%) FSS questionnaire and 2 (4%) CIS20-F questionnaires were excluded, due to missing values on 1 or more items. Data from 49 FSS questionnaires and 47 CIS20-F questionnaires were available for test-retest analyses.

The item scores on the first assessment are shown in Table IV. The mean (standard deviation (SD)) total fatigue score was 5.3 (1.3) on the FSS and 38 (12) on the CIS20-F. The highest item scores were seen on question 6 of the FSS (“My fatigue prevents sustained physical functioning”: median score 6.5 (IQR 6.0–7.0), and on question 7 of the CIS20-F (“I get tired very quickly”; median score 6.0 (IQR 5.0-7.0)). Highest possible total scores were found in 5% and 3% of the patients on the FSS and the CIS20-F, respectively. Lowest possible total scores were only seen on the CIS20-F, in 2% of the patients.

Reliability of FSS and CIS20-F

Both the FSS and the CIS20-F showed acceptable internal consistency (Cronbach’s α = 0.90 for the FSS on the first assessment with ITCs ranging from 0.38 to 0.82; Cronbach’s α = 0.93 for the CIS20-F on the first assessment with ITCs ranging from 0.70 to 0.87).

Table V shows the results regarding test-retest reliability and measurement error. Total fatigue scores on the FSS and the CIS20-F did not differ between the first and second assessment (mean difference: –0.04 (95% CI –0.26–0.19) for FSS and 1.2 (95% CI –0.55–2.93) for CIS20-F). The ICCs (95% CI) for the FSS and the CIS20-F were 0.80 (0.67–0.88) and 0.85 (0.75–0.91), respectively, indicating acceptable and excellent test-retest reliability. Measurement error, expressed by the SDC, was 1.55 (representing 28.7% of the mean of the 2 assessments) for the FSS and 11.6 (29.4% of the mean) for the CIS20-F.

DISCUSSION

This study shows that the FSS and CIS20-F differ in content with regard to the ICF categories they represent. Our results further show that the ability of the FSS and CIS20-F to discriminate between polio survivors with different levels of fatigue is acceptable (test-retest reliability), while the measurement error of both questionnaires is large. Clinicians and researchers should be aware of the differences in content metrics of these questionnaires and the insufficient sensitivity to detect real changes beyond measurement error in individual patients.

The results of the current study with respect to content metrics (i.e. content density and content diversity) and the percentages of ICF components covered in the FSS are comparable to those reported in the study of Gradinger (19). Yet, it is important to note that, in the Gradinger study, only the content metrics and the representation of different ICF components were reported and not the results of the linking process on item level, which prevents detailed insight into the construct of the questionnaire.

The lower content density and content diversity of the CIS20-F compared with the FSS indicates less multidimensionality within the item structure of the CIS20-F, with a smaller coverage of the questionnaire with respect to different ICF categories. This study has found that the CIS20-F focuses only on fatigue-related concepts (within the ICF component “Body Functions”), while the FSS measures both fatigue-related concepts and non-fatigue related concepts (within the ICF components “Body Functions” and “Activities and Participation”). Examining the items of the FSS and the CIS20-F in more detail and taking into account the differences in content density and content diversity, one might suggest that the CIS20-F quantifies the concept of fatigue, while the FSS seems mainly to measure the interference of fatigue with functioning. However, this conclusion should be regarded with caution, because information regarding the causal relationship contained in items was not systematically analysed. This limitation has been described by the developers of the ICF linking rules (29). Nonetheless, the method of ICF linking has been shown to be helpful in clarifying the conceptual differences between the FSS and the CIS20-F.

Regarding reliability, both the FSS and the CIS20-F show acceptable internal consistency and test-retest reliability, with excellent test-retest reliability for the CIS20-F, based on the lower 95% CI boundary of the ICC. For the FSS, our results are in accordance with those reported in previous studies in polio survivors (with Cronbach’s α ranging from 0.80 to > 0.95 (20–22) and ICCs ranging from 0.83 to 0.97 (20, 22)). No previous data on reliability of the CIS20-F in polio survivors have been published, but the present results are comparable to reliability parameters found in CFS (Cronbach’s α = 0.88) (23) and multiple sclerosis (ICC = 0.84 (95% CI 0.72–0.91)) (24).

Despite acceptable internal consistency and test-retest reliability, the smallest detectable changes (SDC) found for both questionnaires were rather high. For the FSS this is in line with the results of the study by Horemans, who reported an SDC of 1.5 (27%) (20). Considering that changes in FSS scores following different therapeutic interventions in PPS were found to range from 0.6 to 2.7 (7–10) it seems that the FSS is insufficiently sensitive to detect changes beyond measurement error in single individuals. For applications at the group level, much smaller changes can be detected, since measurement error is reduced by √n for a group of n patients (32). For both questionnaires we found highest and lowest possible scores in less than 15% of the patients, which means that floor and ceiling effects do not negatively influence the reliability of the questionnaires (32).

The large measurement error might, in part, result from day-to-day fluctuations in fatigue in polio survivors over time (20), although there are no studies available to support this hypothesis. For detecting changes in fatigue in individual patients with PPS, one might consider using multiple repeated measures over time in order to overcome the problem of large measurement error (32).

Generalization of our results to PPS patients is considered to be good, since we used a consecutive series of polio survivors known to an outpatient rehabilitation clinic. The level of fatigue in our population is comparable to that in earlier studies in polio survivors, with FSS scores ranging from 5.1 to 5.4 and corresponding SDs ranging from 1.1 to 1.5 (5, 20, 21). Furthermore, according to the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN), we used a fair sample size (37). The response rate of 76% found in our study is considered sufficient and in line with response rates in previous surveys in polio survivors (varying from 74% to 88%) (4, 38, 39).

Study limitations

This study has some limitations. First, we examined the conceptual properties of only 2 fatigue questionnaires. Future research should focus on investigating the conceptual properties of other fatigue instruments commonly used in polio research, so that a core set of instruments can be selected for use in research and clinical practice. The present study might serve as an example for this purpose. Secondly, despite the standardization of the ICF linking rules, this method is not simple and straightforward (28, 29). An evaluation of our linkage process showed that agreement between the 2 coders was 70%, which is in line with the results of other studies reporting this measure (varying from 64% to 79%) (19, 30). Extensive training of the ICF linking procedure might further improve the reliability of the method. Finally, although this study gave insight into the content validity and reliability of the FSS and CIS20-F in polio survivors, studies aimed at determining the construct validity are needed to gain more insight into the internal relationships of items within the instrument and the relationships with scores of other instruments or differences between relevant groups (for the CIS20-F). The results of our content analyses provide the basis for formulating hypotheses regarding the construct validity and facilitate the interpretation of the results of this process. Furthermore, the responsiveness of both questionnaires should also be studied to determine their adequateness to detect meaningful changes in fatigue over time (32).

Conclusion

This study shows that the FSS and CIS20-F differ in content with regard to the ICF categories they represent. It is suggested that the CIS20-F primarily quantifies the concept of fatigue, while the emphasis of the FSS is on measuring the interference of fatigue with functioning. Considering the acceptable internal consistency and test-retest reliability, as well as the potential of the 2 instruments to detect changes at group level, we conclude that both the FSS and the CIS20-F can be applied in PPS research when assessing fatigue, although they cannot be used interchangeably.

REFERENCES