Elizabeth E. Marfeo, PhD, MPH, OTR/L1, Pengsheng Ni, MD, MPH1, Leighton Chan, MD, MPH2, Elizabeth K. Rasch, PhD, PT2, Christine M. McDonough, PhD1, Diane E. Brandt, PT, MS, PhD2, Kara Bogusz, BA1 and Alan M. Jette, PhD, PT1

From the 1Boston University School of Public Health, Health & Disability Research Institute, Boston, MA, and 2National Institutes of Health, Rehabilitation Medicine Department, Mark O. Hatfield Clinical Research Center, Bethesda, MD, USA

OBJECTIVE: To develop a system to guide interpretation of scores generated from 2 new instruments measuring work-related physical and behavioral health functioning (Work Disability – Physical Function (WD-PF) and WD – Behavioral Function (WD-BH)).

DESIGN: Cross-sectional, secondary data from 3 independent samples to develop and validate the functional levels for physical and behavioral health functioning.

SUBJECTS: Physical group: 999 general adult subjects, 1,017 disability applicants and 497 work-disabled subjects. Behavioral health group: 1,000 general adult subjects, 1,015 disability applicants and 476 work-disabled subjects.

METHODS: Three-phase analytic approach including item mapping, a modified-Delphi technique, and known-groups validation analysis were used to develop and validate cut-points for functional levels within each of the WD-PF and WD-BH instrument’s scales.

RESULTS: Four and 5 functional levels were developed for each of the scales in the WD-PF and WD-BH instruments. Distribution of the comparative samples was in the expected direction: the general adult samples consistently demonstrated scores at higher functional levels compared with the claimant and work-disabled samples.

CONCLUSION: Using an item-response theory-based methodology paired with a qualitative process appears to be a feasible and valid approach for translating the WD-BH and WD-PF scores into meaningful levels useful for interpreting a person’s work-related physical and behavioral health functioning.

Key words: outcome assessment (healthcare); disability evalua­tion; work.

J Rehabil Med 2015; 47: 00–00

Correspondence address: Elizabeth E. Marfeo, Boston University School of Public Health, Health & Disability Research Institute, 715 Albany Streeet, T5W Boston, MA 02118, USA. E-mail: emarfeo@bu.edu

Accepted Dec 10, 2014; Epub ahead of printFeb 27, 2015

INTRODUCTION

The primary federal provider of insurance and financial assistance for individuals with a work-related disability in the USA is the Social Security Administration (SSA) (1–3). Support from the SSA represents a significant resource for disabled workers and their families. Previous literature has described both conceptual and programmatic challenges that SSA faces in efficiently and comprehensively characterizing a person’s potential ability to work (4, 5). In its 2007 report, entitled, “Improving the Social Security Disability Decision Process,” an Institute of Medicine (IOM) panel noted that, as medical treatment and assistive technologies advance, the diagnostic basis for the SSA’s work disability adjudication process have become less useful as a marker of work disability (6). To address this gap in current disability assessment, we are currently developing a new instrument; the Work Disability Functional Assessment Battery (WD-FAB).

A unique feature of this new instrument is its conceptual foundation that uses principles outlined by the World Health Organization (WHO) International Classification of Functioning, Disability, and Health (ICF) (4). The ICF highlights the multifactorial nature of disability by focusing on biologic, personal, and social perspectives of disability (4, 7–10). Factors related to a person’s ability to work are complex and extend beyond disease symptoms and impairments alone, but include factors such as functional activity limitations, psychological well-being and contextual factors (11, 12). In developing both the physical and behavioral health domains of the WB-FAB, the goal was to expand the scope of SSA’s current work disability assessment by creating a measure of functional, activity-based aspects relevant to a person’s potential ability to work. This research integrates a more functional approach into the paradigm of work disability assessment, focusing on activities or tasks that relate to a person’s potential ability to participate in the workplace compared with SSA’s current definition of disability, which focuses primarily on symptoms and impairments related to their disease (4, 13).

The IOM report also recommended the development of alternative approaches, including the creation of standardized functional assessment instruments to more accurately measure work disability in the context of a large federal disability program (2, 6). Clinician-administered and/or performance-based functional assessments are time-consuming, costly, and impractical to implement in large programs such as the SSA’s work disability adjudication process. Comprehensive functional assessment strategies exist, but they are time-consuming to administer and are impractical for widespread use (14). Item-response theory (IRT) and computer adaptive test (CAT)-based assessment of function may be a promising means to provide the SSA with standardized functional assessment tools that are feasible for widespread implementation, and that when combined with other medical evidence and workplace information, can provide valuable information on a person’s ability to engage in substantial gainful employment activity (15–18). The goal of our research is to integrate a more functional approach into the process of work disability assessment among a large, national disability program, focusing on activities or tasks that relate to a person’s potential ability to participate in the workplace. This approach lends itself well to utilizing the WHO ICF, as an underlying framework for assessing the multifactorial nature of disability (4, 7).

To be consistent with modern heath outcome assessment methodologies we developed 2 new assessments for measuring work-related physical and behavioral health functioning using IRT methodologies (15, 18). IRT measurement models, a class of statistical procedures used to develop measurement scales, examine the associations between individuals’ response to a series of items designed to measure a specific outcome domain (e.g. physical functioning) (19). Data collected from samples of individuals are fit statistically to an underlying IRT model that best explains the covariance among item responses (20). When an instrument is developed using IRT methods, the items are calibrated to a common metric, which allows for a hierarchical ordering of ability level or performance. When assumptions of a particular IRT model are met, such as the graded response model, which was used in the WD-FAB development, estimates of a person’s functional ability do not strictly depend on a particular fixed set of items (21). This scaling feature allows one to compare persons along a functional outcome dimension even if they have not completed the identical set of functional items. This is a significant advantage over many current instruments developed using classical test theory whereby a fixed set of items is required, limiting the breadth of potential content coverage and score precision possible compared with IRT techniques (21).

Several measures have been developed previously to assess a range of factors that are associated with a person’s ability to work; however, many of these measures assess limited scope of the multifaceted construct of work ability, have less than optimal psychometric properties, or are disease-specific (22–24). Instruments such as the Workplace Activity Limitations Scale, Work Limitations Questionnaire, and Work Productivity and Activity Impairment Questionnaire are widely used in both clinical and research settings. A major limitation of these instruments is their limited breadth of coverage in characterizing a person’s functional abilities related to work. Comprehensively characterizing a person’s ability to work using these existing measures would require administration of many items, using several different instruments, and lead to an undue respondent burden as well as be impracticable for use in the context of a large governmental disability program such as the SSA. In addition, many of items used in these instruments fall slightly outside of the scope of item content relevant to the context of the SSA. Individuals applying for SSA disability benefits have been out of work for least 12 months; therefore items that refer to performance of specific work tasks would be outside the reasonable recall period to reliability assess. Lastly, given the heterogeneity of medical conditions for which individuals are apply for disability benefits in large, national disability programs, a more generic instrument provides advantages over administering disease condition-specific instruments that have been developed previously. For example, a major advantage is that the WD-FAB offers the opportunity to systematically and efficiently collect information about work-related functional abilities across all applicants using a common metric. Comprehensively characterizing a person’s ability to work using these existing measures would require administration of many items, using several different instruments, and lead to an undue respondent burden as well as be impracticable for use in the context a large governmental disability program such as the SSA. Because the WD-FAB was developed using modern IRT/CAT methods it offers significant advantages over current measurement techniques in its ability to efficiently and comprehensively characterize work-related function across a wide range of work disabling conditions.

The scaling feature of instruments developed using IRT provides the basis for implementing assessments using CAT. CAT programs use a simple form of artificial intelligence that selects questions tailored to the test-taker, and thereby shortens or lengthens the test to achieve the level of precision desired by a user. The combination of IRT and CAT methods allows the WD-FAB to generate highly precise scores with relatively low respondent burden (25). One challenge with using the IRT/CAT-based instruments is translating the scores into meaningful information that can be used for clinical or policy decision-making or, as with our work, guide characterization of a person’s ability to work (26, 27).

An approach to help facilitate the interpretation of IRT-based health outcomes assessment scores is called functional staging. Developing functional stages or levels allows for a brief, meaningful description of a patient’s function in various domains of activity and facilitates interpretation of assessment scores. Approaches to developing these categorizations are relatively new in the area of health outcomes assessment (27, 28).

Results from previous work, provide evidence for initial psychometric and construct validity of the work-related physical and behavioral health functioning instruments, the Work Disability – Physical Function (WD-PF) and the WD – Behavioral Function (WD-BH) (4, 15–18). The WD-PF instrument measures the domain of physical function along 5 dimensions: Whole Body Mobility, Upper Body Function, Upper Extremity Fine Motor, Changing & Maintaining Body Position. The WD-BH scales include: Self-Efficacy, Mood & Emotions, Behavioral Control, and Social Interactions.

While IRT methods allow us to place individuals on a continuum, ranging from lowest functioning to highest functioning, the score itself does not indicate what a person can or cannot do per se, but situates them along the continuum of ability level (20). Given this measurement property of these instruments, an important step to facilitate understanding a score is to provide a system to guide stakeholders’ interpretation of a score, so that meaningful conclusions may be drawn. The objective of this study is to apply a novel methodology to develop and interpret functional levels for each of the WD-PF and WD-BH scales.

METHODS

Data for this study combines samples from 3 earlier studies aimed at development and validation of 2 new instruments that measure physical and behavioral health functioning relevant to work (the WD-PF and the WD-BH). Calibration data were collected from a group of individuals applying for SSA disability benefits: claimants. Additional data were collected in order to develop norm-based scores against which to compare the claimant scores. The third sample was a sample of work-disabled individuals used for initial validation of the newly developed instruments. Subjects contributed to either the development of the WD-PF or WD-BH functional levels; there was no overlap between these 2 groups.

Subject selection and setting

The claimant sample included a group of individuals who were applying for disability benefits through the US SSA’s disability programs. Eligibility criteria required that the individual apply on his or her own behalf due to a condition that was physical, mental, or both physical and mental in nature. Additional criteria included: 21 years of age and being able to speak, read, and understand English. The general adult sample was drawn from a large internet opt-in survey pool, allowing for approximation of the sample to be representative of a US adult population matched on sex, racial/ethnic background, age, and education, weighted equally. These subjects had to be 21 years or older. Lastly, another independent sample was collected to allow for initial validation of the instruments with a sample of individuals self-reporting permanent work disability due to physical or mental conditions, the “work-disabled” sample. Subjects in all 3 samples had to provide informed consent prior to participating in any study activities. An institutional ethical review board approved all study procedures.

Data collection

Both the claimant and work-disabled samples completed either the WD-PF or WD-BH, depending on the nature of their self-reported disability (physical, mental or both). In addition, the work-disabled subjects completed legacy instruments to examine concurrent validation of the performance of each instrument. Individuals in the general adult sample completed either the WD-BH or WD-PF, selected randomly. Basic demographic information was collected for all study participants.

Instruments

Work Disability Functional Assessment Battery: Physical and Behavioral Health Measures – these instruments were developed for the purposes of characterizing a person’s physical or behavioral health functioning across domains relevant to work. The WD-PF scales were Whole Body Mobility, Upper Body Function, Upper Extremity Fine Motor, Changing & Maintaining Body Position. All of the WD-BH scales were used for developing functional levels: Self-Efficacy, Mood & Emotions, Behavioral Control, and Social Interactions). Previous work confirmed the factor structure and construct validity of the WD-FAB scales for both domains (4, 15–18). All scales demonstrated good accuracy, reliability and content coverage for assessing work-related physical and behavioral health functioning. Details of the development and initial psychometric testing of these instruments have been discussed elsewhere (4, 15–18).

Sequential analytic approach

We used a 3-phase analytic strategy that incorporated both quantitative and qualitative procedures to develop functional levels describing categories of physical functioning and behavioral health functioning relevant to the context of work. The origins of this approach come from educational settings where experts use a data-driven consensus process for setting standards for academic performance (28–30). More recently, these methods have been modified and applied in the context of healthcare to develop a technique known as functional staging (27, 31). All quantitative data analysis procedures were performed using SAS computer software (32) and Microsoft Excel was used to generate the item maps.

• Phase 1 utilized the data collected from the WD-BH and WD-PF instruments to empirically derive item maps (27, 28). Item maps are tools that help facilitate the standard setting procedure by ordering the items by difficulty level sorted from easiest to most difficult item within each scale. Estimates of item difficulty were developed using item calibrations estimated from item response theory (IRT) analysis (15, 18). The item maps were based on the general adult sample responses because this is the sample that serves as the comparator of physical and behavioral health functioning when assessing a person’s potential ability to work.

• The second phase was based on a modified-Delphi qualitative process that aims to reach consensus from experts and stakeholders in establishing the cut-points for the functional levels (33, 34, 35). These cut-points designated the threshold between one functional level and another. A panel of experts and stakeholders was convened for each of the 2 domains of interest. A total of 19 individuals participated in the modified-Delphi process (8 in the physical group, 11 in the behavioral health group). The panel included individuals with expertise in measurement development, work capacity evaluation, and vocational rehabilitation in both areas of physical and mental health. The expert’s professional backgrounds included physical therapists, occupational therapists, a rehabilitation medicine physician, psychologists, and psychometricians. The modified-Delphi technique involved 3 rounds: independent cut-point designation, feedback and summary of the independent cut-points, then finalizing the cut-points with consensus. These levels provided a means by which to initially test construct validity of the WD-FAB’s ability to differentiate between groups of individuals with various degrees of physical and behavioral health functioning.

• The last phase focused on validation of the cut-points using known-groups comparisons among 3 independent samples. We hypothesized that the general adult sample should demonstrate a greater proportion of subjects in the higher functional levels compared with the claimant and work-disabled samples; whereas, there would be a smaller proportion of individuals in the lower levels within the general adult sample compared with the claimant and work-disabled samples. Using χ2 tests at the alpha 0.05 level, we tested statistical differences in distribution of percentages of the sample within each of physical and behavioral health functional levels across 3 comparative samples. The general adult sample served as the reference group. Post-hoc sensitivity analysis of the functional level distributions was conducted to test whether the distributions were affected by age and gender. In addition, within the known groups work-disabled sample, means scores were examined across each functional level for each scale within the WD-BH and WD-PF instrument, hypothesizing that each means score should be discriminative in a monotonic pattern increasing with each incremental functional level.

RESULTS

Table I describes basic demographic information for each of the 3 samples. The physical group included 999 subjects in the general adult sample, 1,017 in the SSA claimant and 497 in the work-disabled sample. The behavioral health group comprised 1,000 general adult subjects, 1,015 SSA claimants and 476 work-disabled subjects. For each of the samples in the physical group, there were slightly more men than women, the samples were predominantly white, with the work-disabled sample mean age slightly older (56 years of age vs 49 years of age for the general adult and claimant samples). For individuals in the behavioral health group, there were slightly more males in the general adult sample (51.5%), and more females in the claimant and work-disabled samples (56.3% and 66.4%, respectively). Similar to the physical group all 3 samples in the behavioral group were predominantly white. On average, the behavioral health group’s age was slightly younger than the physical group but was similar in that the work disabled group was slightly older than the general adult and claimant samples (51 years of age vs 49 years of age for the general adult and 43 years of age for the claimant).

Phase 1: Item maps

Results from the empirical analysis allowed graphical presentation of item difficulties for each item in the WD-BH and WD-PF scales. The item maps allowed integration of the item calibrations estimated from the IRT analyses, using a graded response model (35) to facilitate the phase-2 consensus process. The item maps served to simplify the cognitive task for the expert panel to evaluate the content and difficulty levels of each individual item in establishing cut-points for each functional level.

Phase 2: WD-BH and WD-PF functional level definitions

The initial functional levels developed for the WD-BH instrument included either 4 or 5 categories of behavioral health function ranging from poor to excellent. Results of the consensus process yielded 4 functional levels for the Self-Efficacy and Social Interaction scales; 5 levels for the Mood & Emotions and Behavioral Control scales. For the WD-PF instrument, 5 levels of physical function were established ranging from lowest to highest ability to perform tasks relevant for each scale (Whole Body Mobility, Upper Body Function, Upper Extremity Fine Motor, and Changing & Maintaining Body Position). See Table II for descriptions of the functional levels for the WD-BH and WD-PF instruments.

Phase 3: Cut point Validation

Figs 1a–d illustrate the results of the Behavioral Health Functional Levels for each of the 3 samples: general adult, claimant, and work-disabled. Counts of claimants classified in each functional level, show a general pattern that the number of claimants at lower functional levels are greater in the claimant and work-disabled samples compared with the general adult sample. For most of the WD-BH scales, results supported the hypothesis that the general adult sample consistently included a higher percentage of individuals in the higher functional levels compared with both the claimant and work-disabled groups. This assumption did hold true for the general adult vs the work-disabled sample for the Self-Efficacy scale. All percentages of general adult vs claimant and general adult vs work disabled are statistically significantly different at the alpha 0.05 level with p < 0.0001. Similarly, our hypothesis was also supported for the Physical Functional Levels; the general adult sample consistently yielded significantly more individuals in the higher functioning levels compared with the claimant and work-disabled samples (Figs 2a–d). Although our general hypothesis was supported, variation by scale was observed as to how the samples were distributed within each functional level. Lastly, results from comparing the means using the WD-FAB instruments demonstrated a monotonic relationship (progressively increasing with each increase in functional level) across the functional levels for each scales for both physical and behavioral health (Table III). Mean scores across each functional level were significantly different at the alpha 0.05 level.

Fig. 1. Behavioral Health Functional levels across 3 samples: US general adult vs claimant, US general adult vs work-disabled. (A) Self-efficacy, (B) Mood and emotions, (C) Behavioral control, (D) Social interactions.

Fig. 2. Physical health functional levels across 3 samples: US general adult vs claimant, US general adult vs work-disabled. (A) Whole-body mobility, (B) Upper body function, (C) Upper extremity fine motor, (D) Changing and maintaining body position.

DISCUSSION

Findings from this study provide initial validation of functional levels designed to assist in the interpretation of physical and behavioral health functional scores relevant to work. As hypothesized, the general adult sample demonstrated a greater proportion of individuals functioning at the higher levels compared with the claimant and work-disabled samples. Applying the functional level categorizations, the mean score significantly increased within each functional level across the current WD-FAB instrument scales. Results from this study provide initial evidence that using an IRT-based approach for creating functional levels describing both work-related physical and behavioral health is both feasible and psychometrically sound.

Overall, the results of this study suggested that the general adult sample reported higher functional levels for both the physical and behavioral health functional scales compared with the claimant and work-disabled samples. However, there was variation in how the samples were distributed across the levels for each domain: physical and behavioral health. The functional levels as applied using the WD-BH scales resulted in all 3 samples with the majority of individuals in the mid-range functional levels (2–4) with a smaller proportion of each sample being in the lowest and highest functional levels (1 and 5). This is in contrast to the functional levels applied to the WD-PF scales, where the general adult sample tended to have a greater proportion of individuals in the higher functional levels (4, 5). This variation highlights the differences in how to approach measuring physical and behavioral health functioning; physical domains lend themselves well to a hierarchical scoring system, where behavioral health assessments may not fit that structure as well.

When developing new health outcomes measures, it is important to develop tools that provide information that is meaningful and interpretable to relevant stakeholders (26, 36, 37). This study represents continued work in improving the way in which work-related physical and behavioral health functioning is measured. The goal of this study was to create a system for interpreting a person’s physical and behavioral health functioning as an essential step in developing a new IRT-based measure. The WD-FAB offers advantages over current instruments, in that a highly precise score can be obtained with relatively low respondent burden. The challenge then becomes how to interpret the IRT-based scores in a clinically meaningful way. Findings from this study demonstrate the utility of the WD-BH and WD-PF beyond merely producing a score along a continuum of functioning, but outline a method to facilitate interpretation of those scores in the context of assessing work disability.

This study represents initial exploratory phases of translating the WD-BH and WD-PF scores into meaningful levels useful for interpreting a person’s work-related physical and behavioral health functioning; however, a few limitations should be noted. The sequential analytic method used in this study is new in its application to health assessment, future replication and validation of this methodology should be completed to provide additional validation and demonstration of its feasibility in facilitating meaningful interpretation of IRT-based instruments. Within the work-disabled sample, there were no subjects in the Upper Extremity Fine motor lowest functional level category. Although this was not true for the general adult and SSA claimant samples, each of which having very few (0.60% and 5.6%, respectively), but some representation of individuals in this lowest functional level. Previous work demonstrated adequate psychometric performance of this scale, excluding floor effects (18). Unexpectedly, the overall pattern of higher functioning for the general adult sample compared with the claimant and work-disabled samples did not hold for the Self-Efficacy scale. The literature suggests that self-efficacy is an important, yet complex, factor related to work performance (38). From our previous work, we found that this was one of the weaker scales in terms of item fit (16). Through a process of item replenishment we plan to continue to refine and improve of the WD-BH Self-Efficacy scale, which will enable opportunities to better understand the relationship between work disability and self-efficacy (39–41). Lastly, an opt-in internet panel of respondents may possess unique characteristics as a sample population. Efforts were made to match these individuals to the US adult population on key demographic variables, but the permanently disabled sub-set may not be representative of individuals who are unable to work more generally. This study’s findings suggest the need for future work in functional level refinement for this particular scale among a permanently disabled sample. Next steps to further validate the WD-FAB instruments and their functional levels are currently underway. Such efforts include comparison of the WD-FAB with performance-based assessments of physical and mental work capacity.

This study was performed in the context of large US disability benefits program; however we believe the potential utility of these measures may extend beyond that specific context and may be relevant to a variety of researchers and policy-makers looking to assess a person’s ability to function in the workplace. Currently, there is no universal standard for cut points to determine thresholds of work functioning that should indicate the allocation of benefits, yet many countries face challenges with rising number of individuals applying for sickness and disability benefits including the UK, the Netherlands and Scandinavian countries, and many other countries within the Organization for Economic Co-operation and Development (OECD) (42–45). Applying these WD-FAB instruments within varying populations would necessitate some level of re-examination of the scores and the relevant functional levels to be conducted as guided by the given program’s needs and objectives. One limitation that still exists with many modern measurement development techniques is sometimes there is a need to re-test the psychometric properties when the measures are applied in new populations.

The process used in establishing these initial functional levels for the WD-BH and WD-PF builds upon work done in educational settings used for academic performance standard setting (28–30). More recently, researchers have begun to adapt these methods for application in other health status measures (26, 27). In the area of health assessment, little has been published as to the methodology of moving from score to interpretation. This work presents a novel approach that combines both quantitative and qualitative methods to arrive at both a psychometrically robust instrument that yields meaningful scores, meeting the needs of interested stakeholders. Systematically documenting such processes as described in the article may prove useful for applications beyond work disability assessment, but provide guidance for interpreting scores for IRT-based health outcomes assessments.

In conclusion, this study applied a novel approach utilizing IRT-based methodologies paired with a qualitative process to develop functional levels for 2 measures of work-related physical and behavioral health functioning. Four functional levels were developed for the WD-PF scales (Whole Body Mobility, Upper Body Function, Upper Extremity Fine Motor, and Changing & Maintaining Body Position). Four levels were developed for the WD-BH scales (Mood & Emotions, Behavioral Control, Social Interactions, and Self-Efficacy). Initial validation of the cut-points for each of the functional levels was supported through using 3 independent samples to test the distribution of the subjects in each sample across the functional levels. Results from this study provide further support for the newly developed WD-BH and WD-PF instruments in measuring work-related physical and behavioral health functioning.

ACKNOWLEDGMENTS

This research was supported by Social Security Administration and National Institutes of Health (SSA-NIH) Interagency Agreements (NIH contract nos. HHSN269200900004C, HHSN269201000011C, HHSN269201100009I) and by the NIH intramural research program. The authors and co-authors have no financial or other conflicts of interest to disclose that may bias the reporting of the results presented in this study.

REFERENCES