Tina Juul, MHSc1,2, Karen Søgaard, PhD1, Ewa M. Roos, PhD1 and Aileen M. Davis, PhD3

From the 1Institute of Sports Science and Clinical Biomechanics, The University of Southern Denmark, Odense, 2CopenRehab, Section of Social Medicine, Department of Public Health, University of Copenhagen, Denmark and 3Division of Health Care and Outcomes Research, Toronto Western Research Institute, University Health Network and the University of Toronto, Toronto, Canada

OBJECTIVE: To develop a patient-reported outcome evaluating the impact of neck pain. The results of item generation and reduction and subscale structure in support of the content and construct validity of the measure are reported.

METHODS: Items were generated from the literature and through focus groups including patients with neck pain and healthcare professionals, respectively. Item reduction was based on focus groups and field-tested questionnaire data. Construct validity was assessed using exploratory factor analysis.

RESULTS: Focus groups containing 24 patients (mean age 57.2 (standard deviation (SD) 15.9) years, range 24–85 years); 19 women) and 12 healthcare professionals were conducted before data saturation was achieved. A total of 196 patients with neck pain (mean age 47.8 (SD 13.7) years), range 18–89 years; 146 women) completed the preliminary questionnaire. Overall 35 items were removed from the original 69. A multidimensional questionnaire, divided into five subscales, was developed from the remaining 34 items: mobility; symptoms; sleep disturbance; everyday activity and pain; and participation in everyday life. Exploratory factor analysis supported a 5-subscale structure.

CONCLUSION: The Neck OutcOme Score has excellent content validity and preliminary results support a 5-subscale structure. Additional work is needed to assess the reliability, further construct validity and responsiveness.

Key words: patient-reported outcome; questionnaire; content validity; construct validity; neck pain.

J Rehabil Med 2015; 47: 844–853

Correspondence address: Tina Juul, Institute of Sports Science and Clinical Biomechanics, The University of Southern Denmark, Campusvej 55, DK-5230 Odense M, Denmark. E-mail: tiju@sund.ku.dk

Accepted Jun 25, 2015; Epub ahead of print Sep 11, 2015

INTRODUCTION

Neck pain (NP) is the second most common musculoskeletal disorder (1). At any given time 12–14% of the adult population reports having NP (2), which has been related to neuromuscular deficits (3), deficits in eye movement control (4) and impaired performance of muscles in the cervical spine (5). As a result, NP often causes reduced work ability and significantly increases the risk of long-term sickness absence (6). Given the burden of NP, there is an urgent need for efficacious interventions. However, evaluation of effectiveness and efficacy require salient and robust outcome measures.

There is a lack of patient-reported outcomes (PROs) with sound psychometric properties for the evaluation of patients with NP. The Neck Disability Index (NDI) (7), one of the most applied PROs for NP, has been criticized regarding content validity (8), reliability (9, 10) and dimensionality (10). The NDI was originally developed to measure disability in activities of daily living (7). However, the development process of NDI presents shortcomings, such as a broad selection of patients, achievement of data saturation and lack of patient input in determining content validity (11). This is mainly due to the absence of clear standardized procedures (12). While current measures can be modified and their measurement properties evaluated for people with NP, the need for patient input to determine content validity (12) based on a conceptual framework (13) is often lacking or inadequate. We therefore elected to develop a PRO that evaluates the impact of NP related to impairment, activity and participation, the Neck OutcOme Score (NOOS). Prior to item generation, the concepts being measured need to be clearly defined. The International Classification of Functioning, Disability and Health (ICF) (14) bio-psycho-social framework is commonly used for the development of PROs (15) and the efficacy of a given treatment is often evaluated based on content related to the classification codes (16). The ICF includes 3 domains: body functions and structure, activity, and participation (14). While there is guidance for the content of body functions and structure, the activity and participation domains are combined, leaving the content of the individual domains open to interpretation (17). Nevertheless, the ICF is the most applied framework. The present study reports the results of item generation and reduction and subscale structure in support of the content and construct validity of the NOOS.

METHODS

The study was approved by the Regional Scientific Ethical Committee for Southern Denmark, Denmark and the Danish Data Protection Agency (ID: 30513). All participants provided informed consent, and all rights of the participants were protected.

Questionnaire development

Development of the NOOS was guided by the Consensus-based Standards for the selection of health status Measurement INstruments (COSMIN) (13) recommendations.

Target population identification

In order to represent people with a broad spectrum of NP, eligibility criteria included people over 18 years of age, who were able to communicate in written and spoken Danish, who had NP (with/without cervical radiculopathy) indicated on a body diagram (18), and who sought treatment for their NP. Patients were excluded if they had or were being investigated for cancer; had a neurological disorder (e.g. Multiple Sclerosis); lacked the ability to communicate (e.g. due to a psychotic episode) or had a history of alcohol or drug abuse.

Conceptual model

The development of the NOOS was based on a reflective model (classic test theory) in which the construct is reflected by the items (19). The ICF framework was used as the conceptual model; hence the classification system was used as guidance when assigning a given item a priori to a specific subscale. Body functions, body structure, activity and participation were defined in accordance with the ICF framework (20). In addition, the NOOS used the template of the Knee injury and Osteoarthritis Outcome Score (KOOS), a multi-dimensional measure with 5 subscales (21). This template (i.e. layout) was chosen, since it has shown good acceptability in patients with ankle, knee, hip and groin-related problems (18, 22, 23). A 5-point ordinal scale was used, with response options ranging from “no problems” (0) to having “extreme problems” (4). A normalized score (0–100, worst to best) is calculated for each subscale, using the following formula: 100–mean (itemX: itemX)/4*100, which provides a patient profile.

Item generation and reduction

Fig. 1 shows an outline of the item generation process and reduction.

Fig. 1. Item reduction and generation process. PRO: Patient Reported Outcome; QOL: Quality Of Life; NOOS: Neck OutcOme Score; Q1: item 1, neck-related quality of life; P1: item 1, pain subscale; HCP’: Health care professionals.

Items were generated through a systematic literature review and focus groups with patients and healthcare professionals (HCPs) to ensure content validity. A draft questionnaire was developed by the first author (TJ), grouping items into subscales based on the conceptual model from more than 100 items identified from existing PROs for patients with NP (e.g. 7, 24–33). For example, items regarding range of motion were grouped with similar items concerning body functions and structure (e.g. stiffness and dizziness) to form a subscale. An item such as: “I have a fair degree of difficulty concentrating” from the NDI (7) was reformulated to “Have you had difficulty concentrating?”. According to the ICF it was classified as body functions and structure and therefore added to the “symptoms” subscale. Similarly, items were assigned to either a subscale related to activities or participation (e.g. activity items such as shopping or housework, while social and leisure activities formed the participation subscale). Three members of the study group (TJ, EMR, KS) evaluated the draft questionnaire and minor changes were made to item wording. After removing repeated items, the preliminary version of the NOOS, included 69 items with 8 subscales: symptoms (9 items), general well-being (5 items), pain (13 items), physical function/daily activities (19 items), participation (7 items), function in sport and recreation (6 items), emotional impact (5 items) and neck-related quality of life (QOL) (5 items).

Focus groups were then conducted with patients and HCPs to identify additional relevant items and test the comprehensibility, relevance, acceptability and feasibility of the questionnaire (pilot-testing). Patients were recruited from 4 private physiotherapy clinics located in Copenhagen, Aarhus and Odense, Denmark. To ensure that patients were representative of the target population all patients’ with NP seeking treatment were included (e.g. patients with whiplash-associated disorder, chronic and acute NP, disc prolapses, derangement, spinal stenosis and osteoarthritis).

The groups of HCPs included physiotherapists specializing in musculoskeletal physiotherapy who treated patients with NP daily and researchers focusing on NP who had experience developing PROs or who had linguistic skills. Each focus group consisted of 2–5 informants with a mix of men and women and was facilitated by the same moderator (TJ). Focus groups were conducted until no new information was elicited (i.e. data saturation).

Focus group participants completed the preliminary version of the questionnaire. They were asked about their experiences with completing the questionnaire and identifying missing content (items) as well as commenting on problem items (e.g. lack of relevance, comprehensibility, language and readability). The patients were asked questions, such as how they interpreted the various terms? How did they choose their answers? What context did they use to answer the item and did they think about their most serious episodes or was it an average? Likewise, they were asked to comment on issues such as whether or not they would be willing to complete a questionnaire, which took approximately 10 min to complete (i.e. acceptability) and whether or not the participants would be able to complete the questionnaire themselves or would need help (feasibility).

Initial item reduction was performed based on focus group data. To ensure it was the patients who perceived a given item as meaningful, all items were included in the focus groups (Fig. 1). An item was removed if none of the patients had experienced the symptoms or had any difficulties with the given activity. Any new items or issues concerning items identified in prior focus groups were discussed in later groups before a final decision was made about adding or removing an item. Thus, the “new” draft version of the NOOS was consistently re-tested in focus groups until data saturation was achieved. After completing focus groups the modified version of the NOOS was field-tested.

An electronic version of the questionnaire, including the modifications based on focus group input, was created for subsequent development (field-testing). The electronic version was a precise reproduction of the paper-based layout. However, to ensure transferability, the electronic version was tested and assessed by 11 patients with NP and 3 HCPs (details available upon request).

Patients with the same eligibility criteria as above were recruited for field-testing from 15 private physiotherapist clinics and 1 spine centre in or near Copenhagen, Aarhus, Odense and Aalborg, Denmark. The baseline electronic data collection took place at the clinics via a designated tablet. One and 12 weeks after enrolment the participants were provided with a link to the questionnaire via email to enable questionnaire completion using a personal computer, tablet or smart-phone. SurveyXact® (©2013 Rambøll) was used for data collection and storage. If the participant did not respond, a reminder was sent after 2, 4 and 6 days.

Item reduction was performed using data from field-testing. Items were removed if they had a mean and/or a median score of < 1 (“mild” problems), and/or more than 30% of the respondents reported having “no problems” (baseline administration) and/or if they had low reliability (i.e. a test-retest reliability intraclass correlation coefficient (ICC version 2,1) (33) less than 0.50 (18)). Finally, prior to conducting principal component analysis (PCA), the correlation matrix of the remaining items was assessed. Items a priori assigned to sub-scales based on the same underlying construct, were expected to have correlations between 0.30 and 0.80. Items outside this range were considered for removal. PCA with oblique rotation was conducted with items with factor loadings on a single factor ≤ 0.30 removed (34). The clinical importance of each item was reviewed by all members of the study group (TJ, EMR, KS, AMD) before any items were removed based on the above-mentioned criteria.

Construct validity of the subscales

After completing the item reduction process, construct validity was assessed. The initial plan was to confirm the subscale structure using confirmatory factor analysis (CFA). CFA is strongly driven by theory and was therefore chosen since the ICF framework was used as a conceptual model. The items were initially arranged according to the subscales based on the ICF and confirmatory factor analysis was initiated. However, due to poor model fit, item relationships including within the assigned subscales based on correlations, debating conceptual fit, item correlations and possible issues of different item response structure were evaluated. Examination of the inter-item correlation matrix, e.g. (Table I) revealed that many items had correlations within the range 0.30–0.80 with items on other subscales, making it difficult to achieve a model with overall goodness of fit (34) and conceptual clarity. Hence, we elected to conduct an exploratory factor analysis (EFA) with promax rotation (with Kaiser normalization). Parallel analysis and the scree test were assessed to determine the number of factors retained (35). Items were considered to load on a factor if the factor loading was ≥ 0.30 (34). A sample size of 5–7 patients per item has been recommended when conducting factor analysis as long as there are at least 100 patients in the sample (36). All quantitative data analyses were conducted in SPSS v22.0 (IBM®, SPSS Statistics).

RESULTS

Item generation and content validity

A total of 11 focus groups (6 focus groups including 24 patients (19 women, 5 men (mean age 57.2 (standard deviation (SD) 15.9), range 24–85 years)) and 5 focus groups with HCP) were conducted before data saturation was achieved. The HCPs consisted of 6 physiotherapists, 2 professors, 3 associate professors and 1 PhD student. Finally, 1 linguist (professor) was interviewed separately to ensure comprehensibility of the questionnaire. All focus groups were conducted from January to April 2012.

As shown in Table II, 19 of 69 items were removed because they were deemed unimportant, irrelevant or redundant. Four items were added to the questionnaire because patients felt they were missing. These items were related to swallowing, hearing and balance difficulties (symptom subscale) and experience of limitations in one’s life as a result of neck problems (neck-related QOL subscale). In addition, “finding joy in everyday life” was moved from the “emotional impact” subscale to “neck-related QOL” subscale as patients’ perceived this an essential part of QOL. Items that patients’ perceived as symptoms associated with their neck problems (e.g. headache) were moved from the “general health” subscale to the “symptoms” subscale. NP when sitting still for a long period of time was re-phrased to incorporate sitting still for more than 1-h based on patient feedback. The last response option initially worded “extreme” was changed to “very strong” (except for items related to pain intensity) as several patients mentioned that they would never choose a word like “extreme” to describe their situation.

The HCPs added 5 items to the questionnaire: headache intensity and arm/finger pain intensity (“pain” subscale); visual disorder, difficulty tying a shoelace, writing and sleep difficulties in relation to neck problems were added to the “symptom” subscale. Four items related to pain during sitting still were merged into 1 item incorporating pain during a sitting activity, such as reading, watching TV (pain subscale). NP when “standing still for a short period of time” was re-phrased as “standing still for more than 30 min” and a similar item was removed from the “activity” subscale. Furthermore, the HCPs removed 1 item each from the “neck-related QOL” and “emotional impact” subscales. Both items were linked to daily life situations taking place prior to patients experiencing neck problems and might be subject to recall bias. Finally, 2 items about the future concerns and impact were removed, as the HCPs were concerned that these items would not change during a rehabilitation intervention. As only 2 items remained in the “emotional impact” subscale, there were concerns about the psychometric properties of a subscale containing only 2 items, and these items were therefore moved to the “neck-related QOL” subscale.

The resulting 54-item draft questionnaire with 5 subscales; symptoms (18 items), pain (14 items), physical function, daily activities (9 items), participation (7 items) and neck-related QOL (6 items), was reviewed by a linguist, who made minor changes to the syntactic structure and wording of items. After pilot-testing in 2 focus groups, no further modifications were identified and this preliminary version in electronic format was used for field-testing.

Item reduction based on the electronic survey data

A total of 196 patients (146 women, mean age 47.8 (SD 13.7), range 18–89 years) with NP completed the questionnaire. Hereof 43% also reported experiencing head pain/headache, 55% reported shoulder pain (left, right or both), 24% reported pain in the upper arm (left, right or both) and 19% reported experiencing pain in the forearm/hand (left, right or both). Due to electronic data collection, there were no missing data. Patient characteristics are presented in Table III.

After review of the patient responses, content, measurement qualities and clinical relevance by all 4 authors using the a priori criteria, there was consensus that 18 items met criteria for removal. Fig. 1 and Table IV show the items removed through the quantitative item reduction processes. As shown in Table IV, 9 items were removed from the “symptom” subscale. Item S5, S11, S12 and S13 had a high proportion of patients reported having “no problems” (≥ 51%); S4 had a low correlation with the remaining items (≤ 0.28) and, along with S15, did not meet factor loading criteria; S6 had a very low correlation with the remaining items (≤ 0.24), except for 1 (S8 = 0.30) and even though the item is clinically relevant, it was decided to remove it. All members of the study group agreed that feeling numbness or a tingling sensation in their arm or fingers represented a separate construct, which in the current questionnaire is only represented by this 1 item. S16 was highly correlated with S17 and S18 (r = > 0.74) and it had low reliability (ICC = 0.58; 95% confidence interval (95% CI): 0.44–0.72); and, S10 only correlated with S8 and S9 and 68% of the patients reported having “no problems” or “rarely” having problems. Three items were removed from the “pain” subscale: P6 had poor reliability (ICC = 0.38; 95% CI 0.09–0.59); P13 had a mean and a median < 1; and, P4 had low factor loadings (< 0.23) and > 30% of the patients reported having “no problems”. Four items were removed from the “physical function, daily activities” subscale. A2, A4, A5 and A6 were considered irrelevant since they all had mean values < 1 and >30% of the patients reported having “no problems”. PT5 was highly correlated with PT1, PT2, and PT3 (> 0.62) and 38% of the patients reported having “no problems”. The item was therefore considered irrelevant and removed from the “participation” subscale. Finally, Q6 was removed from the “neck-related QOL” subscale due to low correlations with the remaining “neck-related QOL” items (< 0.30).

After this item reduction process, the questionnaire consisted of 36 items (symptoms (9); pain (11); physical function, daily activities (5); participation (6); and, neck-related QOL (5)). These items were included in the assessment of subscale structure.

Construct validity of the subscales

The EFA was conducted after the item reduction process including 36 items, and results from the final EFA (promax rotation) are presented in Table V.

The analysis yielded 6 factors with an eigenvalue over 1 and factor loadings of 0.30 or above, explaining 64% of the variance. Parallel analysis indicated that the 6 factors should be retained and χ2 goodness-of-fit analysis revealed that the 6-factor solution fit the data, χ2 = 4676.42, p < 0.000. Item Q1 and P1 separated from the remaining items, loading on a single factor with an eigenvalue of 1.38 explaining 4% of the variance. However, both items refer to frequency (“How often do you feel NP?” and “How often are you made aware of your neck problems?”), which is significantly different from the other items referring to the degree of a given problem. Furthermore, a subscale consisting of only 2 items (which may represent a method effect given their response options) often has psychometric challenges (13). The factor analysis was repeated after omitting item P1 and Q1. It did not significantly change the remaining factor loadings. Thus, the items were removed (Table VI).

After performing factor analysis the subscales were renamed, as the initial a priori subscales (latent constructs) were perceived to be somewhat altered, and not well reflected by the initial labels. As shown in Tables V and VI, the final version of the questionnaire thus consisted of 34 items divided into the following subscales: Mobility (7 items: S1, S2, S3, S17, S18, P5, P7), Symptoms (5 items: P2, P3, S7, S8, S9), Sleep disturbance (4 items: P8, P14, S14, A1), Everyday activity and pain (8 items: P9, P10, P11, P12, A3, A7, A8, A9), Participation in everyday life (10 items: PT1, PT2, PT3, PT4, PT6, PT7, Q2, Q3, Q4, Q5). A description of the items included in the final version of the NOOS questionnaire is presented in Table V.

The Danish and English linguistically translated versions (37) are available from http://www.koos.nu/.

DISCUSSION

The development of a multidimensional PRO for patients with NP, specifically item generation, reduction and construct validity, is described here.

During the qualitative development process patients with NP uncovered issues related to their neck problems, revealing complications such as movement difficulties, headache and dizziness, but more unforeseen issues related to participating in leisure activities and experiencing limitations in one’s life were also revealed. Hence, the NOOS covers a broad spectrum of important aspects of life.

Content validity was achieved through the inclusion of stakeholders, such as patient and clinical HCPs. The development process covered the inclusion of items addressing content from existing PRO questionnaires, such as the NDI (7) and the Neck Pain and Disability Scale (29), but also inclusion of new items that emerged during focus groups, and a new subscale was formed (participation). To underpin the concepts being measured the ICF framework was used. Hence, while the identification of items pertaining to participation might have been expected, patients who were not knowledgeable about the framework, generated these new items. The emergence of new items confirms that the NOOS addresses gaps in existing PROs and stresses the need for a new measure.

The ICF framework includes 3 domains: body functions and structure, activity, and participation. In addition, it incorporates contextual factors, divided into external environmental factors (e.g. social structures) and internal personal factors (e.g. social background) (20). While the NOOS covers the 3 domains, it does not include environmental or personal factors. In the ICF, environmental factors are a single classification, while personal factors are included in the model, they are not classified (14), and hence personal factors are extremely difficult to categorize systematically. Furthermore, the aim was to develop an instrument that could evaluate rehabilitation programmes for patients with NP. It was developed in a clinical setting with patients seeking physiotherapy treatment. Personal and environmental factors are conceptually likely to be effect modifiers or may mediate the relationship between the main ICF components. As a result, the questionnaire was developed focusing on body functions and structure, activity and participation. The emergence of new items confirms that the NOOS addresses gaps in the existing PROs.

The ICF manual provides guidance regarding classification of content, but there are several aspects that have been left open awaiting further elaboration (17); for example, a clear distinction between the activity and participation domains, additional specification of the conceptual model, particularly the relationship(s) of the different components (14). From the patients’ perspective, our findings show that there was no clear distinction between the body functions and structure and activity domains when patients were asked to respond to symptoms or difficulties experienced in relation to specified activities, which might explain why many of the existing measurements have failed to detect the effects of rehabilitation interventions. However, there was clear differentiation between the activity and participation domains. Lack of empirical discrimination challenges the investigation of validity. Similar results have been found for other PROs (38). As a result, researchers have argued that discriminant validity between, for example, pain on activity and self-reported physical functions measures, is lacking (38). Additional research is needed in order for the ICF framework to have both a theoretical and an empirical foundation, e.g. clearly distinguishing the boundaries between the different domains. While our data resulted in abandoning our confirmatory factor analytical approach, the results of the EFA support 5-subscale structure of the NOOS. However, it is important that further empirical testing of the NOOS incorporates CFA.

The present study included patients across the breadth of NP and recruited patients from both private therapy clinics and the municipality; we did not include people with NP who did not seek therapy. This may limit the generalizability of the results, as there may be differences between patients not seeking and seeking therapy. However, access to rehabilitation is publically available in Denmark, which might have limited potential differences.

During the focus groups participants completed the preliminary version of the questionnaire, which was ICF-based. This could have affected the dialogue to only include issues related to the constructs (items) presented, thus limiting the opportunity for breeding new constructs. Nevertheless, great effort went into facilitating an open dialogue beyond the presented items, although it cannot be ruled out that this may have influenced the outcome. Furthermore, the HCPs removed 2 items related to future concerns and impact, which may have limited the measure as it has been shown that patient’s beliefs about the consequences of their condition can have an impact on rehabilitation (39).

In conclusion, the NOOS has excellent content validity, and preliminary results support a 5-subscale structure: Mobility (7), Symptoms (5), Sleep disturbance (4), Everyday activity and pain (8), and Participating in everyday life (10). Further work is needed to confirm test-retest reliability (measurement error, internal consistency), construct validity, responsiveness and interpretability. In addition, our data indicate ways in which the ICF framework could be developed.

Conflicts of interest

The authors declare that they have no financial affiliation (including research funding) or involvement with any commercial organization that has a direct financial interest in any matter included in this manuscript. The authors declare that they have no conflicts of interest.

Authors’ contributions

Tina Juul was involved in the planning of the study design, data acquisition, data analysis and writing the paper. Ewa M Roos, Karen Søgaard and Aileen Davis contributed to the analysis and interpretation of the data as well as study conception and design. All authors were involved in drafting the article or revising it critically for important intellectual content and all authors approved the final version of the manuscript.

ACKNOWLEDGEMENTS

This study was supported by grants from The Practice Foundation and The University of Southern Denmark. The authors would like to acknowledge all clinicians, healthcare professionals and patients for helping with the development of the questionnaire and/or the data collection.

REFERENCES