Evdokia Billis, PhD1,2, Christopher J. McCarthy, PhD3, John Gliatis, MD4, Matthiew Gittins, PhD2, Maria Papandreou, PhD5 and Jacqueline A. Oldham, PhD2

From the 1Department of Physiotherapy, Technological Educational Institute (T.E.I.) of Patras, Patras, Greece,
2The University of Manchester, Manchester, UK, 3Imperial College Healthcare NHS Trust, London, UK, 4Orthopaedic
Department, University Hospital of Patras, Patras, Greece and 5Department of Physiotherapy, Technological Educational Institute (T.E.I.) of Athens, Athens, Greece

OBJECTIVE: To investigate the inter-tester reliability of a non-specific low back pain examination procedure, for sub-classifying non-specific low back pain.

DESIGN: Reliability study.

PARTICIPANTS: Thirty patients with non-specific low back pain (12 males, 18 females, mean age 27.7 years, standard deviation 10.3) and 7 physiotherapists (raters).

METHODS: Based on a health professionals’ consensus via focus groups and a Delphi servey, an examination procedure was developed comprising 206 items discriminatory for non-specific low back pain, 108 of which were from the History (clinical questions) and 98 from the Physical Examination (clinical tests) section. Utilizing this procedure, each patient was examined by a blinded pair of raters.

RESULTS: Moderate to excellent agreement was obtained in 125 (61%) items (77 History and 48 Physical Examination items), 47 of which obtained substantial or excellent agreement (kappa>0.61), 37 moderate agreement (kappa between 0.41 and 0.6), and 41 excellent percentage agreements. Poor reliability (kappa < 0.41) was yielded in the remaining 81 items (31 History and 50 Physical Examination items).

CONCLUSION: Satisfactory reliability was obtained in nearly two-thirds of History and half of the Physical Examination items on a non-specific low back pain assessment list generated through consensus agreement. These findings provide clinicians and researchers with valuable information regarding which items are considered reliable and can be utilized in non-specific low back pain patient evaluation/assessment procedures, classification attempts and clinical trials.

Key words: reliability; inter-tester; clinical items; history; physical examination; non-specific low back pain.

J Rehabil Med 2012; 44: 447–453

Guarantor’s address: Evdokia Billis, Lecturer in Physiotherapy, Department of Physiotherapy, Branch Department of Aigion, Technological Educational Institute (T.E.I.) of Patras, Psaron 6, Aigion, 25100, Greece. E-mail: ebillis@teipat.gr

Submitted February 15, 2011; accepted November 22, 2011

INTRODUCTION

Classification systems are probably the most popular diagnostic approach presently used in non-specific low back pain (NSLBP). Assigning patients with NSLBP into homogenous subgroups based on their clinical presentation is believed to be the optimal way to overcome the diagnostic difficulties encountered with this heterogeneous population. Furthermore, it has been proven that classification systems enhance treatment outcomes more successfully than other management strategies (1–3).

However, despite their advantages (2–4) classification systems have recognized shortcomings in clinical practice, including their subjective nature and their unknown or questionable reliability. Most classification systems include clinical items and subgroups that have not been developed on the basis of a wider professional consensus. This may lead to bias and a system that is not clinically feasible or generalizable (3, 5, 6). In addition, although the reliability of classification systems has been addressed in some studies (7–14), the particular clinical items included are often of unknown or poor reliability. This can result in not knowing whether subgroup classifications are genuine or a product of failure in discriminatory ability resulting from poor reliability. Thus, although there is emerging evidence that classification systems are probably the optimal method for diagnosing and guiding treatment for NSLBP, basic steps towards their evaluation appear to require further elaboration. In particular, reliability is vital for improving the confidence of the clinical items included in a classification system and is also a prerequisite for validity testing (15). Thus, it is important to incorporate reliable items within the sub-grouping process.

In an attempt to improve the selection of examination items for future classification of NSLBP, a reliability study was conducted. The study investigated the inter-tester reliability of an extensive clinical list comprising items that were considered discriminatory for NSLBP, which were developed by a large consensus of health professionals (16, 17). This process is believed to improve upon previous studies by providing information on the reliability of a set of examination items considered discriminatory for NSLBP through consensus. Those items proving reliable could then be further utilized to identify patients with similar (homogenous) characteristics.

This is the first of two studies to investigate the inter-tester reliability of a consensus-agreed list of potentially discriminatory items for the clinical assessment of patients with NSLBP. The second accompanying study (18) assesses whether the clinical items proven to be the most reliable can identify homogenous subgroups in a sample of patients with NSLBP.

METHODS

Sample

A convenience sample of Greek adult patients with NSLBP recruited via local physiotherapy referrals was invited to participate in the study. Patients were excluded if their low back pain (LBP) was due to specific pathology, they had undergone lumbar surgery, were pregnant, or if they had a severe neurological condition (influencing their cognitive and motor performance). Overall, 30 patients consented to participate. In addition, 7 physiotherapists (5 men, 2 women), experienced in treating LBP agreed to perform the assessments, which took place in 3 physiotherapy clinics in Greece, situated in Athens, Patras and Lamia. Ethical approval for reliability testing was obtained from the ethics committees of the Technological Educational Institute (TEI) of Lamia, Greece and the University of Manchester, UK.

Procedure

The process of developing the items being tested for reliability has been described previously (16, 17). Briefly, clinical features considered discriminatory for assessing and sub-classifying NSLBP were developed following 3 focus groups and a 2-round Delphi servey involving 23 health professionals (physiotherapists and doctors) and a representative stratified random sample of 150 physiotherapists (PTs), respectively. These clinical features were then transformed into clinically applicable questions and tests. This process was undertaken by 2 manual therapists experienced in spinal problems, utilizing an extensive content analysis procedure. Thus, this led to the development of a comprehensive, clinically applicable and discriminatory clinical examination list for assessing NSLBP. The list was separated into 2 sections, namely History and Physical Examination. The History section comprised 108 items and the Physical Examination 98 items. These clinical features comprising the examination guide are summarized in the results presented in Table II, Tables SI–SII (available from http://www.medicaljournals.se/jrm/content/?doi=10.2340/16501977-0950). Further details regarding each question are presented in Billis (19). Prior to reliability testing, a training procedure took place to ensure a basic level of standardization and comprehension of operational definitions amongst PTs. Training lasted 4 h and was supported by a booklet, which summarized all important (key) examination points.

For reliability testing, raters were divided into pairs, in which the principal investigator was always one of the people within each pair. Thus, 6 PT pairs were formed, with each pair examining 5 patients. Every patient was simultaneously assessed in random order by the 2 therapists, with a 10–15 min break between the 2 examinations. PTs were instructed to advise their patients to rest should pain became intolerable. In addition, self-reported outcome measures for evaluating pain, physical disability, and psychosocial status were administered (summarized in Table I). Each rater was blind to the other raters’ assessments as well as each patient’s outcome measure scores. The whole procedure lasted approximately 1.5 h (approximately 30–40 min for each rater).

Data analysis

Data were analysed utilizing the kappa coefficient of concordance for nominal level data and weighted kappa coefficients with equal weighting amongst the point scales for ordinal level data (20, 21). Kappa values usually range from 0 to 1, where 0 accounts for no agreement and 1 for excellent agreement, although negative kappas may also be obtained, representing worse than chance agreement (21). Additional subcategories have been suggested, where 0 implies poor agreement, 0.01–0.2 slight agreement, 0.21–0.4 fair agreement, 0.41–0.6 moderate agreement, 0.61–0.8 substantial agreement, and 0.81–1.0 almost perfect agreement (22).

For some items, which presented almost perfect agreement and minimal variability across therapists, their kappa values could not be calculated. In cases where negative (rather than positive) findings were too high for both testers (i.e. “saddle anaesthesia” was reported in only two cases by the one therapist and in no cases by the second), the classic 2 × 2 contingency table was not formed, and subsequently, “meaningful” kappas could not be calculated (23). For such items percentage agreements on all paired ratings were calculated instead. In addition, 95% confidence interval (CI) were calculated for all items. Analysis was performed utilizing SPSS (version 15.0).

RESULTS

Thirty patients with NSLBP (12 males, 18 females) with a mean age of 27.7 years (standard deviation (SD) 10.3, range 19–58) were examined. Seventy percent of patients had back pain of less than 6 weeks’ duration. The sample’s profile is summarized in Table I.

The 7 clinical physiotherapists (5 men, 2 women) performing the examinations had a mean clinical experience of 11.8 years (range 7–19) in treating patients with LBP, and 4 of them were musculoskeletal specialists.

A total of 206 clinical items were included in the reliability analysis. Kappa values ranged from –0.050 to 1 and weighted kappa values from –0.168 to 0.665. Eighty-eight items demonstrated moderate to perfect agreement (values over 0.41), fair agreement (values between 0.21 and 0.41) was achieved in 34 items, whereas 47 items demonstrated no agreement between the pairs of therapists, 26 of which presented with negative kappas, indicating worse than chance agreement. Percentage agreements in the remaining 41 items ranged from 86.7% to 100%, indicating almost perfect agreement. Excellent, moderate and poor reliability results from History and Physical Examination sections are presented in Table II, Tables SI–SII, respectively.

DISCUSSION

This study explored the inter-tester reliability of an examination list of clinical features perceived to be discriminatory in assessing back pain (16, 17). Such a study was considered necessary for identifying which clinical items are considered reliable for inclusion in future NSLBP classification attempts (see accompanying publication). Overall, the sample utilized had comparable demographics with previous reliability studies (7, 24, 25), and can be considered representative of typical NSLBP populations. In addition, all PTs (raters) had previous clinical experience with patients with LBP.

Moderate to excellent agreement was obtained in 125 (61%) out of the 206 items. Of these, 77 (37.4%) were from History section and 48 (23.3%) from Physical Examination. Kappa coefficients were calculated in all except 41 items, which were calculated with percentage agreements. Although high percentage agreements do not automatically assume acceptable reliability, they provide an indication of the consistency achieved amongst testers, are considered an appropriate reliability alternative for categorical data (23), and have been used extensively in previous studies (7, 25–27). Poor reliability results were yielded in 81 items (39.3%); 31 (15%) from history and 50 (24.3%) from physical examination, 26 of which reported negative kappas, indicating worse than chance agreement (21).

Although two-thirds of items obtained from History demonstrated satisfactory reliability, it is interesting to note that nearly one-third were not reliable. One could assume that simple questioning would result in consistent responses; however, this was not always the case. Whether this is attributed to the patient (i.e. lack of consistency, fatigue, change of presentation between examinations) or to the rater is unknown. However, unlike specific LBP, for which there are studies that have investigated the consistency of history-taking (28, 29), within the NSLBP field there are only a few reports. Waddell et al. (26) found high percentage agreements for pain location and other symptoms, LBP onset and severity, diurnal pattern, function, disability as well as aggravating and easing factors. Pain location in the form of pain drawing was also highly repeatable in another study (30). These findings agree with the current study for the aforementioned items. In addition, in the study by McCarthy et al. (24), in which the reliability of clinical tests and questions included in international LBP guidelines were investigated in a large patient sample, several similarities to the present study were detected, with the exception of psychosocial items, which were found to be reliable in their study and unreliable in this one.

Within the Physical Examination, half of the items reported good reliability. Several previous studies have investigated the reliability of LBP physical examination items. This study’s results on postural/gait observation agree with previous research reports indicating good reliability (12, 26, 31, 32). Active movements yielded more reliable results on assessing pain reproduction than range of movement (ROM). Overall, this agrees with most studies indicating that pain provocation testing is a more reliable assessment marker than ROM (23, 25, 27, 31). Centralization (33, 34) in flexion was a more reliable clinical indicator compared with extension. Although centralization is usually considered highly reliable for both flexion and extension (9, 33), the results in the current study were characterized as satisfactory, particularly considering that none of the raters had specialized (McKenzie-type) training.

Neurological examination yielded satisfactory reliability for myotomal testing (most) dermatomal testing and straight leg raise (SLR). However, reflex testing, ROM of SLR and some dermotomal tests were unreliable. Although neurological examination is usually reliable (23, 24, 26, 35), a mixed picture whereby some parts of this examination are reliable and others are not, has also been reported (32). The lower levels of agreement observed in this study may be partly attributed to variability in the Likert-type responses (3-point for SLR, 4-point for dermatomes and 5-point for reflexes), which could have confused the therapists. Another possible explanation could be that PTs were not thoroughly familiar with these tests in clinical practice, as medical doctors are the only “first-line” primary care practitioners screening for LBP in Greece, and are thus the ones predominantly utilizing neurological examination.

Passive examination was unreliable for evaluating ROM, which is in agreement with previous literature (25, 27). Pain provocation testing by palpation and passive joint examination was reliable for some spinal levels (T12, L1, L2 and S1) and lumbosacral areas (upper lumbar and sacroiliac). This, again, agrees with previous reports (25, 32). Lumbar pain provocation tests in the form of passive intervertebral motion testing have been extensively investigated, having an acceptable level of reliability; however, criticism has been levelled at the methodological quality of most studies (36). For sacro-iliac joint testing, this study’s results agree with previous literature, indicating that reliability on individual tests is not considered good (37), and that testing should be performed on clusters of different tests (which are more reliable). Muscle testing was unreliable; whether this was attributed to the test, tester or patient (i.e. fatigue) factors is unknown.

Finally, reliability was examined in some clinical reasoning items, half of which obtained good results. From the clinical judgements on the patients’ active movements, the existence of a closing pattern1 and impairment dysfunction2 were considered reliable. It could be argued that these two movement patterns are commonly encountered (38, 39), thus PTs are confident in their assessment. The primary pain mechanism and the predominant domain of influence and prognosis for recovery were also reliable amongst the judging therapists. These outcomes agree with a previous large-scale reliability study (24), thus providing some confidence in the PTs’ clinical reasoning processes. Interestingly, behavioural signs were not deemed reliable, and this is in disagreement with McCarthy et al. (24), where similar items yielded moderate reliability.

This study has made an effort to provide a good standard reliability design by attempting to follow most of the suggestions for improving reliability studies that have been recommended by May et al. (27). However, a limitation regarding the sequential type of assessments performed by each therapist pair must be acknowledged. However common and acceptable this procedure may be (7, 24, 25, 31), it is a confounding factor in relation to the consistency of the clinical findings, in that it may either overestimate the consistency of the findings, or it may result in changes in the patient’s clinical presentation (between examinations) should the patient become fatigued or aggravated by symptoms. This “biasing” effect was, however, reduced by randomizing the order of examination between therapists and by ensuring that the gap between examinations was not too short or too long (considering the patients’ pain/disability levels) so as to dramatically change their presentation. In addition, although the sample’s clinical profile was comparable to samples used in previous reliability studies (7, 24, 25), it should be acknowledged that it consisted of relatively acute, minimally disabled patients.

The similarities obtained between this and other reliability reports provide some confidence in the study’s outcomes. In particular, the more “straightforward” aspects of the examination (i.e. history items, aggravating and easing factors, pain location, pain provocation testing, etc.) demonstrated higher reliability, similar to the findings of a number of previous studies (23–25, 27, 31). Therefore, such clinical items with established reliability across studies could be recommended for use as a standardized approach, forming the basis for future clinical evaluation and clinical trials involving patients with NSLBP. The items that were proven to be unreliable across studies (i.e. motion palpation, muscle activation tests, sacroiliac testing, etc.), seem to form more “complex” examination processes, and should either be discarded or used with caution before further research indicates whether their reliability can be improved. Continuing to apply unreliable items in clinical practice/research may distort or compromise the true outcome value of the undertaken process. It is, however, interesting that some items that presented with poor reliability in this study, are considered significant markers or valuable prognostic indicators in other studies, such as several psychosocial and functional items (40, 41). Further research is required carefully to evaluate the overall value of these items as well as considering alternative approaches.

A strength of this study is that this examination list was developed on the basis of a large consensus of experienced clinicians, thus improving the generalizability of the examination process. In addition, consensus assures a degree of face validity of the involved questions and tests, making the examination process a practical, easily applicable tool for clinicians dealing with NSLBP. This approach enhances the clinical significance of the discriminatory items for patients with NSLBP, and the accompanying reliability statistics presented in this study facilitate health professionals to adopt this evaluation approach in their practice. However, it must be acknowledged that good reliability in examination testing does not guarantee validity in developing clinical subgroups. Further research should explore subgroup analysis. The accompanying study (Billis et al.) presents an exploration of the development of homogenous subsets in NSLBP by the use of a cluster analysis approach that utilizes the clinical items deemed reliable in the current study.

In conclusion, this study explored the inter-tester reliability of an examination procedure for NSLBP obtained from a consensus amongst Greek health professionals. Satisfactory reliability was obtained in nearly two-thirds of clinical questions obtained from the History section and in nearly half of the clinical tests obtained from the Physical Examination section. These findings provide clinicians’ and researchers’ insights into the clinical items considered reliable and that are recommended for inclusion in future NSLBP assessment procedures, sub-classification processes and clinical trials.

AcknowledgEments

We thank physiotherapists Panagiotis Trigkas, Savvas Spanos, George Krekoukias and Ioannis Stathopoulos as well as all patients for participating in this study. This research was funded with the MACP’s Elsevier Science and Doctoral Awards for Research in Manipulative Physiotherapy.

References