APPLICABILITY OF THE ICF CHECKLIST TO TRAUMATICALLY BRAIN-INJURED PATIENTS IN POST-ACUTE REHABILITATION SETTINGS

Sanna Koskinen, LicPsych, Eeva-Maija Hokkinen, PT, Jaana Sarajuuri, LicPsych and Hannu Alaranta, MD, PhD

From the Käpylä Rehabilitation Centre, Synapsia, Helsinki, Finland

OBJECTIVE: To evaluate the applicability of the International Classification of Functioning, Disability and Health (ICF) checklist in post-acute traumatically brain-injured patients in rehabilitation settings.

DESIGN: A cross-sectional study based on the written documents of an interdisciplinary rehabilitation team.

SUBJECTS: A sample of 55 patients with traumatic brain injury.

METHODS: Two raters extracted information from the patients’ medical documents using the ICF checklist. The most common ICF categories were identified and the agreement between the raters was evaluated.

RESULTS: Of the 123 checklist categories, 30 reached a prevalence of 30% or more in the ratings of both raters, and 18 further categories reached a prevalence of 30% or more in the ratings of either one rater. Seventy-five categories (61%) did not reach the cut-off point and were thus considered irrelevant. Fourteen ICF categories not included in the checklist were also considered important. Extracting the data from pre-existing documents seems to be reliable: in 86% of the most relevant categories the difference between the raters in the qualifier values was at most 1.

CONCLUSION: A checklist is a practical tool in clinical work. However, the current ICF checklist seems not to be adequate in characterizing patients with post-acute traumatic brain injury. Developing an ICF core set for these patients might prove useful.

Key words: ICF, checklist, traumatic brain injury, rehabilitation, agreement.

J Rehabil Med 2007; 39: 467–472

Correspondence address: Sanna Koskinen, Käpylä Rehabilitation Centre, Synapsia, PO Box 103, FI-00251 Helsinki, Finland. E-mail: sannakos@gmail.com

Submitted August 30, 2006; accepted February 5, 2007

INTRODUCTION

The International Classification of Functioning, Disability and Health (ICF) belongs to the “family” of international classifications developed by the World Health Organization (WHO) for application to various aspects of health (1, 2). It provides a framework for functioning and health as well as a universal and standard language to be used by different professionals in different countries and settings. The ICF is not an assessment tool and does not consist of specific assessment measures or protocols for use in evaluation. As Reed et al. (3) state, ICF coding in healthcare settings reflects the findings of clinical assessments conducted by health professionals. The professionals are encouraged to consider how the expert evaluations they already make could be translated into the framework and codes of the ICF.

In the clinical context the ICF is intended for use in needs assessment, matching interventions to specific health states, rehabilitation, and outcome evaluation. In order to facilitate the use of the ICF in clinical encounters, the WHO has developed the ICF checklist. The checklist comprises 123 categories out of more than 1400 categories of the whole ICF classification system. The checklist makes it possible to generate a profile of the patient using the most important ICF categories (4, 5). To address the issue of feasibility, the ICF core sets project was initiated in 2001 (6–8). ICF core sets have been developed for 12 burdensome chronic conditions (5, 9). Stroke has thus far been the only neurological condition with a defined core set (10). However, a first version of the ICF core set for patients with neurological conditions in the acute hospital (11) as well as in early post-acute rehabilitation facilities (12, 13) have been presented recently.

The ICF is an exciting landmark event for rehabilitation (14) and the field of rehabilitation may be the one in which the ICF will have its greatest contribution, offering a unique theoretical and practical classification model (15, 16).

Traumatic brain injury (TBI) affects people of all ages and is the leading cause of long-term disability among children and young adults (17, 18). According to a recently published review article, approximately 7.8 million persons are alive in Europe with some level of disability caused by TBI. The main external causes of the injuries are falls and motor vehicle accidents both in Europe and in Finland (19, 20). The ICF may provide a valuable framework in illustrating the sequelae of TBI with its various impairments, activity limitations and participation restrictions as well as the extensive need for rehabilitation.

However, very few studies have focused on patients with TBI in the frame of reference of the ICF. An interesting exception is a recent study on life satisfaction after TBI and the WHO model of disability (21).

The aim of the present study was to investigate the applicability of the ICF checklist in describing the functioning and health of patients with TBI in an interdisciplinary post-acute inpatient rehabilitation setting. The main objectives of the study were: (i) to identify the most common problems encountered in patients with post-acute TBI when the ICF checklist is used, and (ii) to analyse the agreement between 2 raters with different professional backgrounds.

METHODS

Subjects

A total of 305 patients with the primary diagnosis of TBI underwent their first inpatient rehabilitation period in the Käpylä Rehabilitation Centre, Helsinki, Finland, from 1 January 2002 to 31 December 2004. These patients also participated in an international multicentre quality of life study, QOLIBRI (22). The Finnish data of that study will be published later in the context of the ICF. The inclusion criteria for the QOLIBRI study and the present study were: age 18–60 years at the time of assessment, minimum age at injury 15 years, available informed consent form, diagnosis of TBI made by a physician according to ICD-10, and time since injury 3 months to 15 years. The exclusion criteria were: Glasgow outcome scale – extended (GOSE; 23) < 3, spinal cord injury, patients with past or present psychiatric conditions, ongoing severe addiction, inability to understand, co-operate and answer, and terminally ill patients.

The QOLIBRI questionnaire was completed by 124 patients. For 16 patients the Glasgow Coma Score (GCS) was unknown and these patients were excluded. Of the remaining 108 patients a systematic sample of 55 patients was formed: every second patient was picked up from the alphabetical list of patients’ surnames. The first patient on the list was drawn by lot.

ICF coding procedure

Two professionals (clinical neuropsychologist SK and physiotherapist E-MH), both having more than 20 years’ experience in rehabilitation and assessment of patients with TBI, analysed independently in detail the written documents of the 55 patients using the ICF checklist, Version 2.1a, Clinician Form (4). The checklist is a short form of the whole ICF classification presenting the 4 components: Body Functions (b), Body Structures (s), Activities and Participation (d), and Environmental Factors (e); 29 1-level chapters; and 123 2-level categories. The data were extracted from a total of 350 written documents: 55 documents from a neurologist, nurse, neuropsychologist and physiotherapist; 54 from a social worker; 53 from a speech and language pathologist; and 23 from an occupational therapist. In this study it took on average 90 minutes (range 30–245 minutes) for the rater to go through the documents of 1 patient and to complete the checklist. No specific assessment tools were developed for the ICF coding. The coding was based purely on the written documents produced by experienced professionals using regular clinical assessment methods in their everyday work. Consequently, the domains for the activities and participation component were coded using the performance qualifier, and not the capacity qualifier in a standardized environment.

The instructions for the use of the checklist (4) present guidelines for grading the qualifiers. In the categories of Body Functions, Body Structures, and Activity and Participation the qualifier code 0 refers to no impairment/difficulty. Values 1 to 4 range from 1 = mild impairment/difficulty (problem present less than 25% of the time, with an intensity a person can tolerate and which happened rarely over the last 30 days) to 4 = complete impairment/difficulty (problem present more than 95% of the time, with an intensity that is totally disrupting the persons day-to-day life and which happened every day over the last 30 days). The qualifiers of Environment are graded from 0 to +4 (0 = no barriers/facilitators to 4 = complete barrier and +4 = complete facilitator). The qualifiers 8 = not specified and 9 = not applicable were not used in this study.

Training of the raters involved familiarization with the principles of the ICF from the ICF handbook (1), the Finnish translation of the ICF handbook (2), and other literature; from 5 1-day workshops arranged by STAKES (National Research and Development Centre for Welfare and Health, Finland); from thorough discussions of the principles of coding as well as coding and analysing the documents of 3 pilot patients before the initial study. One of the raters (SK) had also earlier carried out a pilot study using the ICIDH-2 beta-2 (24) and participated in the translation process of the ICF from English to Finnish.

The statistical analyses were conducted with the SPSS statistical software (13.0 for Windows). The agreement between the raters was analysed by Cohen’s Kappa and intraclass correlation. Cohen’s Kappa is used to measure how much agreement exists beyond the amount expected by chance alone. When agreement on quantitative scales is assessed, the intraclass correlation coefficient (ICC) is employed as a measure of reliability (25). A rigorous definition for the ICC is that the ICC is the correlation between one measurement on a target and another measurement obtained on that target (26). There are numerous versions of the ICC. In this study the version ICC(3,k) was used.

RESULTS

The data describing the patients and the injury-related variables are presented in Table I. Sixty-nine percent of the patients were male and 64% of the injuries were caused by traffic. According to the GCS classification (27) 54% of the injuries were severe, 11% moderate and 35% mild.

The most common problems

Fig. 1 and Tables II–IV present the most typical problems identified by both raters in the same patients. This information is dichotomized as 0 = no problem existing and 1 = problem existing (qualifier codes 1–4) in order to present the existence of the problems.

Fig. 1 shows the impairments of body functions and structures, activity limitations and participation restrictions, as well as barriers or facilitators in the environment in the 1-level classification of the ICF. Of the 29 1-level chapters 25 (86%) were documented in the records of the patients. The most common chapters, in which the patients had problems in at least one category, were b1: mental functions (100% of the patients), b2: sensory functions and pain (91%), b7: neuro-musculoskeletal and movement related functions (65%), s1: structure of the nervous system (100%), s7: structure related to movement (51%), d1: learning and applying knowledge (64%), d2: general tasks and demands (58%), d3: communication (100%), d4: mobility (73%), d6: domestic life (76%), d7: interpersonal interactions and relationships (76%), d8: major life areas (100%), e1: products and technology (82%), e3: support and relationships (89%), and e5: services, systems and policies (100%).

Descriptions of 100 (81%) different 2-level categories out of the 123 categories in the checklist were identified from the written documents of the patients. In earlier studies (5, 13) the cut-off point of at least 30% of the patients having a problem has been used to indicate the most typical problems in a specified condition. Thirty (24%) of the categories in the checklist were identified in the documents of at least 17 patients (at least 30% of the patients) by both raters (Tables II–IV). The categories are presented in descending order based on the number of patients having the problem.

Twelve (39%) of the 31 Body Functions (b) categories in the checklist were documented with a frequency of more than 30% of the patients. The most typical problems included neuropsychological symptoms and problems with energy and drive functions, sleep, pain and vestibular functions (balance).

The only category in the component of Body Structures (s) in which more than 30% of the patients had problems was s110: brain. The next problematic categories were s750: lower extremity (18% of the patients), s760: trunk (13%), and s710: head and neck region (9%).

None of the patients were able to work at the time of the rehabilitation period. Problems with communication were common, especially in the field of expressing oneself in conversation or in speaking and, to a lesser degree, in receiving spoken messages. Problems with complex interpersonal interactions as well as with solving problems and undertaking multiple tasks were common.

All the categories in the component of Environment (e) with the prevalence of at least 30% were coded as “facilitators”. Health services and support from health professionals as well as from immediate family were the most important categories. Besides these facilitators both raters agreed that one patient had barriers in e155: Design, construction and building products and technology of buildings for private use, one patient in e225: Climate, and one patient in e250: Sound. No cases existed in which one of the raters would have rated a category as a facilitator while the other would have rated it as a barrier.

In this study 30 categories reached a prevalence of 30% in the assessments of both raters and an additional 18 categories in the assessments of either one of the raters. Thus, 75 categories of the checklist did not reach the prevalence of 30% and were considered less relevant for this sample of patients with TBI. At least 1 irrelevant category was found in every 1-level chapter. The 1-level chapters in which no category reached the prevalence of 30% in the assessments of both raters are presented in Table V.

The medical documents frequently contained information typical of patients with TBI and related to specific ICF categories, but missing from the checklist. According to the study design this information was not systematically recorded, but was noted as an additional remark. These remarks included: b126 (temperament and personality functions), b160 (thought functions), b180 (experience of self and time functions), b240 (sensations associated with hearing and vestibular function), b250 (taste function), b255 (smell function), b320 (articulation functions), b510/b5105 (swallowing), b760 (control of voluntary movement functions), d160 (focusing attention) d163 (thinking), d177 (making decisions), d230 (carrying out daily routine), and d240 (handling stress and other psychological demands).

Agreement between the 2 raters

The agreement between the 2 raters in using the qualifier codes 0–4 was analysed using Cohen’s Kappa and ICC (25, 26). The kappa statistics is used widely in clinical research in the evaluation of categorical data for the assessment of “agreement beyond chance”. However, a well-known and disturbing paradox in using the Kappa is that particularly high values of observer agreement (Po) may result to low values of Kappa (28). In our study the Kappa remained low in many ICF categories due to the homogeneity of the sample. For that reason the ICC is also presented.

Table VI presents the agreement on the level of the 4 ICF components. The agreement was highest in the component of Body Structures (s) and lowest in Activities and Participation (d).

In order to analyse the agreement in more detail, the differences between the raters in the values of the qualifiers were calculated patient by patient in the 30 categories that reached a prevalence of 30% or above. In the components of Body Functions (b), Body Structures (s), and Environment (e) more than 50% of the ratings of the 2 raters were identical. In 86% of the categories of the components b, s, d, and e the differences in the ratings were at most 1 point.

In the 12 most typical categories of Body Functions (b) 55.9% of the categories had identical qualifier values in the rating of raters A and B. In 91.4% of the categories the difference between the ratings was at most 1. The highest agreement (difference in the qualifier value between the raters at most 1) was found in the category of b164 (higher level cognitive functions) (100%) and the lowest in b152 (emotional functions) (69.1%).

The only category in the component of Body Structures (s) in which more than 30% of the patients had problems was s110: brain. Category s110 was coded according to GCS and the length of post-traumatic amnesia (PTA). Using these criteria no differences were found in the qualifiers of s110 between raters A and B.

In the 11 most typical categories of Activities and Participation (d) 43.0% of categories had identical values in the qualifiers in the rating of raters A and B, and in 82.2% of categories the difference between the ratings was at most 1. The highest agreement (100%) was found in the category d850 (remunerative employment) and the lowest (52.7%) in d220 (undertaking multiple tasks).

In the 6 most typical categories of Environment (e) 51.2% of categories had identical qualifier values in the rating of raters A and B, and in 80.0% of categories the difference between the ratings was at most 1. The highest agreement (98.2%) was found in the category e580 (health services, systems and policies) and the lowest (56.3%) in e570 (social security, services, systems and policies).

DISCUSSION

The aim of this cross-sectional study was to assess the applicability of the ICF checklist in identifying the most common problems documented in a sample of patients with post-acute TBI, and to assess the agreement between 2 raters with different professional backgrounds.

Problems related to 100/123 2-level categories of the checklist were identified reflecting the high diversity of the sequelae of TBI. Of these 100 categories, 30 were identified in the documents of at least 30% of the patients and were thus considered most relevant for this sample of patients with TBI. The distribution of the components was the same as in earlier studies with neurological patients in early post-acute rehabilitation facilities (13) and with stroke patients (5); the most common problems were found in the components of Body Functions (b) and Activities and Participation (d). This reflects the broad manifestation of injuries in body functions in neurological patients as well as the focus on activities and participation in rehabilitation settings.

With a prevalence of 30% or above, 30/123 categories of the checklist were identified by both raters, and 18 more by either one of the raters. The remaining 75 categories of the checklist may be considered less relevant in this population of patients with TBI. In the component of Body Functions (b) the main irrelevant categories were found in chapters b3, b4–b6, and b8. In contrast to our results, the issues presented in these categories have been found important in the earlier studies with neurological patients (10–13). These categories include, among others, the cardiovascular, respiratory, digestive and metabolic functions typical, for example, to stroke but not to TBI. All the categories except s110 (brain) of Body Structures (s) were considered irrelevant in our study. Restrictions in self-care (d5) were rare in our study compared with the earlier studies, while the patients with the poorest outcome were excluded. Chapter d9 (community, social and civic life) was also under-represented in our study. This may reflect the tradition in documentation: the important topics of participating into community are not documented comprehensively.

The medical documents of the patients frequently contained information related to ICF categories but missing from the checklist. This information was not recorded systematically but was noted as additional remarks. According to these remarks 14 ICF categories were identified.

The recently developed ICF core sets for patients with neurological conditions in the acute hospital (11) and in early post-acute rehabilitation facilities (12) consist of 85 and 116 ICF categories, respectively. Of these categories 60/85 and 70/116 are also included in the ICF checklist. In our study, 33 of the 48 most common ICF checklist categories identified by either one or both of the raters are also included in the core set for the early post-acute rehabilitation settings. Additionally, 9 out of the 14 ICF categories frequently reported in the medical documents but missing from the checklist were included in this core set. Most of the relevant categories of our study that were not included in the core sets were found in the component of Activities and Participation (d). These categories reflect restrictions in demanding activities (undertaking single/multiple tasks, driving, looking after one’s health, and interpersonal interactions) or activities related to later stages of recovery (domestic life, work and employment, economic life, and recreation and leisure). Most of the categories included in the core sets but assessed as irrelevant in our study were found in the component of Body Functions (b), Body Structures (s), and Self-care (d5). The differences may reflect the fact that the core sets are developed for patients in earlier stages of recovery. To summarize, 48/123 categories of the checklist and 42/116 categories of the core set for neurological conditions in early post-acute rehabilitation settings can be considered relevant for our sample of patients with TBI. This means that more than 60% of the categories of the checklist as well as of this core set can be considered irrelevant. According to our results, the ICF checklist or the core sets for neurological patients do not adequately characterize patients with TBI in the later stages of recovery. Developing an ICF core set specially for patients with TBI might prove useful.

According to our experience the use of the qualifiers was not unambiguous. However, with the instructions presented in the checklist it was possible to use the qualifier codes 0–4 as indicators of the severity of the problems. The agreement between the 2 raters was relatively high. In 86% of categories the difference in the values of the qualifiers was at most 1. The highest agreement between the raters was found in components that were either the most objective, in which the criteria for measuring was clear, or in which the tradition in documentation was systematic, such as remunerative employment (d850) or higher-level cognitive functions (b164). The lowest agreement was found in components that either required expert professional knowledge (e.g. b152: emotional functions), or were not clearly defined, documented, operationalized, or measured (e.g. d220: undertaking multiple tasks). Assessing these components required interpretation and “reading between the lines”. Exact use of the qualifiers would demand more specific measures and more detailed documentation of the patient’s management in everyday situations.

One potential limitation of the present study is that it was carried out in only one rehabilitation centre. However, the study deliberately focused on one centre in order to obtain as systematic an understanding as possible of the functioning of the patients based on one documentation tradition in an experienced neuro-rehabilitation centre. Had data from a less experienced centre been included the initial assessments might have been less accurate. The other limitation in generalizing the results to the whole post-acute TBI population is that the patients with the poorest functional outcome were excluded from this study. The aim was to focus on a group of patients who themselves were able to communicate reasonably well and to respond to complex methods of assessments in cognitive functioning and quality of life issues.

The use of the ICF checklist in this study was based on the written medical documents of the patients and no specific assessment tools were developed for this purpose. One aim of this study was to go beyond specific instruments by using data that are provided in normal clinical practice. The study is a cross-sectional study representing the patients’ functioning during their first rehabilitation period. It would be interesting in future research to examine the applicability of ICF classification to long-term outcome during the rehabilitation process and to quality of life.

ACKNOWLEDGEMENTS

We thank docent Seija Talo, PhD, from the National Research and Development Centre for Welfare and Health, Finland (STAKES), for commenting on this article and for familiarizing us in the use of the ICF over many years. We also thank docent Laura Hokkanen, PhD, from the University of Helsinki for her valuable comments.

REFERENCES

1. World Health Organization. International Classification of Functioning, Disability and Health: ICF. Geneva: World Health Organization; 2001.

2. WHO. Toimintakyvyn, toimintarajoitteiden ja terveyden kansainvälinen luokitus ICF. Stakes. Jyväskylä; 2004. (Finnish translation of the International Classification of Functioning, Disability and Health).

3. Reed GF, Lux JB, Bufka LF, Peterson DB, Threats TT, Trask C, et al. Operationalizing the International Classification of Functioning, Disability and Health in clinical settings. Rehabil Psychol 2005; 50: 122–131.

4. http://www3.who.int/icf/cheklist/icf-checlist.pdf. World Health Organization, September 2003.

5. Ewert T, Fuessl M, Cieza A, Andersen C, Chatterji S, Kostanjsek N, Stucki G. Identification of the most common patient problems in patients with chronic conditions using the ICF checklist. J Rehabil Med 2004; 36 Suppl 44: 22–29.

6. Stucki G, Cieza A, Ewert T, Kostanjsek N, Chatterji S, Üstun TB. Clinical commentary. Application of the International Classification of Functioning, Disability and Health (ICF) in clinical practice. Disabil Rehabil 2002; 24: 281–282.

7. Stucki G, Grimby G. Foreword. Applying the ICF in medicine. J Rehabil Med 2004; 36 Suppl 44: 5–6.

8. Cieza A, Ewert T, Üstun TB, Chatterji S, Kostanjsek N, Stucki G. Development of ICF core sets for patients with chronic conditions. J Rehabil Med 2004; 36 Suppl 44: 9–11.

9. Weigl M, Cieza A, Andersen C, Kollerits B, Amann E, Stucki G. Identification of relevant ICF categories in patients with chronic health conditions: a Delphi exercise. J Rehabil Med 2004; 36 Suppl 44: 12–21.

10. Geyh S, Cieza A, Schouten J, Dickson H, Frommelt P, Omar Z, et al. ICF core sets for stroke. J Rehabil Med 2004; 36 Suppl 44: 135–141.

11. Ewert T, Grill E, Bartholomeyczik S, Finger M, Mokrusch T, Kostanjsek N, Stucki G. ICF Core Set for patients with neurological conditions in the acute hospital. Disabil Rehabil 2005; 27: 367–373.

12. Stier-Jarmer M, Grill E, Ewert T, Bartholomeyczik S, Finger M, Mokrusch T, et al. ICF Core Set for patients with neurological conditions in early post-acute rehabilitation facilities. Disabil Rehabil 2005; 27: 389–395.

13. Grill E, Lipp B, Boldt C, Stucki G, Koenig E. Identification of relevant ICF categories by patients with neurological conditions in early post-acute rehabilitation facilities. Disabil Rehabil 2005; 27: 459–465.

14. Stucki G, Ewer T, Cieza A. Value and application of the ICF in rehabilitation medicine. Disabil Rehabil 2003; 25: 628–634.

15. Bilbao A, Kennedy C, Chatterji S, Üstun B, Barquero JL, Barth JT. The ICF: applications of the WHO model of functioning, disability and health to brain injury rehabilitation. NeuroRehabilitation 2003; 18: 239–250.

16. Stucki G. International Classification of Functioning, Disability, and Health (ICF). A promising framework and classification for rehabilitation medicine. Am J Phys Med Rehabil 2005; 84: 733–740.

17. NIH Consensus Development Panel on Rehabilitation of Persons with Traumatic Brain Injury. Rehabilitation of persons with traumatic brain injury. JAMA 1999; 282: 974–983.

18. Ragnarsson KJ. Results of the NIH consensus conference on “Rehabilitation of persons with traumatic brain injury”. Restor Neurol Neurosci 2002; 20: 103–108.

19. Tagliaferri F, Compagnone C, Korsic M, Servadei F, Kraus J. A systematic review of brain injury epidemiology in Europe. Acta Neurochir 2006; 148: 255–268.

20. Alaranta H, Koskinen S, Leppänen L, Palomaki H. Nationwide epidemiology of hospitalized patients with first-time traumatic brain injury with special reference to prevention. Wien Med Wschr 2000; 150: 444–448.

21. Pierce CA, Hanks RA. Life satisfaction after traumatic brain injury and the world health organization model of disability. Am J Phys Med Rehabil 2006; 85: 889–898.

22. von Steinbuechel N, Petersen C, Bullinger M. QOLIBRI Group. Assessment of health-related quality of life in persons after traumatic brain injury – development of the Qolibri, a specific measure. Acta Neurochir Suppl 2005; 93: 43–49.

23. Wilson LJT, Pettigrew LEL, Teasdale GM. Structured interviews for the Glasgow outcome scale and the Extended Glasgow outcome scale: guidelines for their use. J Neurotrauma 1998; 15: 573–585.

24. Koskinen S, Ahlblad Y, Alaranta H, Kivekäs J, Alanen E. The ICIDH-2 classification in the assessment of functioning in traumatically brain injured (TBI) patients. (English abstract) In: Talo S, editor. Functioning – from the frame of reference to assessment and measurement. Turku: The Social Insurance Institution, Finland, Social security and health reports 49; 2001, p. 179–195.

25. Fleiss J, editor. Statistical methods for rates and proportions. New York: John Wiley; 1981, p. 140–153.

26. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 1979; 86: 420–428.

27. Teasdale G, Jennett B. Assessment of coma and impaired consciousness: a practical scale. Lancet 1974; ii: 81–84.

28. Lantz CA, Nebenzahl E. Behavior and interpretation of the K statistic: resolution of the two paradoxes. J Clin Epidemiol 1996; 49: 431–434.