Virginia J. Lewis, PhD1,2, Lisa Dell, PhD2 and Lynda R. Matthews, PhD3

From the 1Australian Institute for Primary Care & Ageing, La Trobe University, 2Australian Centre for Posttraumatic Mental Health, University of Melbourne, Melbourne and 3Work and Health Research Team, Ageing, Work and Health Research Unit, Faculty of Health Sciences, The University of Sydney, Sydney, Australia

OBJECTIVE: The increasing number of veterans with complex health conditions accessing rehabilitation leads to the need for an outcome measure that identifies success in areas beyond return to work. The current study was designed to assess the feasibility of Goal Attainment Scaling as a routine measure of outcomes of rehabilitation.

METHODS: Fifteen organisations contracted by the Australian Department of Veterans’ Affairs to work with veterans were invited to trial Goal Attainment Scaling. Training was provided to rehabilitation professionals, and existing documentation was modified by the Australian Department of Veterans’ Affairs to introduce the Goal Attainment Scaling approach.

RESULTS: Analysis of the use of Goal Attainment Scaling supported the feasibility and potential usefulness of the tool in a veteran population. Rehabilitation providers set goals across a range of domains including medical, psychological, social, as well as return to work. The quality of the goals and the outcome measures was generally good.

CONCLUSIONS: The Goal Attainment Scaling approach was seen to support a client-focussed approach to rehabilitation. Data obtained through the use of Goal Attainment Scaling can be summarised at different levels to be useful for clients, providers, rehabilitation coordinators, and senior executives to monitor and report on the overall success of the different types of rehabilitation provided to veteran clients.

Key words: activities of daily living; psychosocial; mental health; rehabilitation; vocational rehabilitation; veteran; PTSD.

J Rehabil Med 2013; 45: 403–409

Corresponding address: Australian Institute for Primary Care & Ageing, La Trobe University, Melbourne, 3086, Australia. E-mail: V.lewis@latrobe.edu.au

Accepted Dec 14, 2012; Epub ahead of print Mar 15, 2013

Introduction

Delivering and measuring effective rehabilitation is particularly challenging for populations that have complex conditions and symptoms (1): it is unlikely that there will be a single clear focus for interventions, and this leads to complications around defining and measuring outcomes. Veterans have been recognised as a population that tends to present with complex problems (2–4). Sandberg et al. (5) note that there are unprecedented numbers of veterans meeting criteria for various psychiatric disorders, such as posttraumatic stress disorder, and that these disorders can have a significant effect on quality of life, social functioning, and capacity to resume satisfying roles in the community. In addition, one of the distinctive characteristics of veteran rehabilitation is that veterans may experience an accumulation of symptoms and functional impairments that are not able to be traced back to a single combat-related event (2, 4).

In recognition of the complexity of the needs of the population they support and the findings of a comprehensive review of veterans’ medical, social and vocational rehabilitation undertaken by the Government of the day (6), the Australian Department of Veterans’ Affairs (DVA) identified the need to redefine their rehabilitation model. The review (6) noted that: “the objective of rehabilitation should be to restore veterans to their optimal level of function commensurate with their service-related disabilities, in order to provide them with better quality of life, maximised vocational outcomes and reduced dependency on financial disability compensation.” The resultant Military Rehabilitation and Compensation Act (MRCA) (7) involved a redefinition of rehabilitation from the previous focus on vocational rehabilitation in isolation to one of biopsychosocial rehabilitation.

In adopting a biopsychosocial definition of health and healthcare, DVA recognised that the goal of “returning to work” and the use of return-to-work statistics common elsewhere in the rehabilitation sector as a measure of success (8) were unlikely to be sufficient for it to monitor the needs and achievements of clients, or the effectiveness of rehabilitation services being purchased for clients. DVA required a measure that would monitor the outcomes of biopsychosocial rehabilitation services provided to veterans as well as return to work outcomes. Routine practice prior to the project reported in this article was for all rehabilitation cases to be identified after a preliminary assessment as “return to work” or “not return to work”. The success of cases that were identified as having a return to work goal was measured simply by reporting the proportion of cases where return to work was achieved. All other goals and any cases that did not involve returning a client to work had no outcome measure and were merely counted; there were no data to indicate whether client, provider or DVA would consider the outcome of the rehabilitation services to have been successful or not.

A literature review was undertaken to identify potential measures that could be adopted by DVA to provide an economically viable way to measure the outcomes of all clients receiving biopsychosocial rehabilitation, particularly for complex problems. Goal Attainment Scaling (GAS) emerged as a theoretically appropriate and potentially practical approach.

The potential acceptability and appropriateness of GAS as a routine outcome measure for rehabilitation is supported by the recommended method for its implementation. The collaborative “SMART” goal setting process at the core of GAS is considered part of best practice in a number of disciplines (9–12) and GAS uses this process as part of the method of determining the measurement metric (13). Goals can be completely individualised for the client’s needs and they can be changed if circumstances change.

As a measure, GAS is considered to be “a sensitive method of measuring specific outcomes on individual goals after a period of treatment” ([14] p. 218). Mailloux et al. (15) considered that the GAS process captures functional and meaningful aspects of a client’s progress, aspects which can be challenging to assess using traditional standardised measures. There is growing evidence to suggest that GAS is as sensitive an indicator, if not more so, when compared with other standard measures (16–18). Issues such as measurement floor and ceiling effects, lack of sensitivity to change/revision, and disjuncture between the clients’ concerns and administrative processes of some standard measures are all potentially avoided through the use of GAS during the rehabilitation process. In addition, the ongoing use of standardised clinical tools is not undermined, because achievement of particular target scores on relevant measures can be a specified outcome under the GAS approach.

While the issue of goal-setter independence has been raised as a potential problem for applying GAS as a routine outcome measure, the use of GAS for DVA or other organisations providing services to clients under a workers’ compensation insurance scheme introduces a level of multiple rater reliability checking: the case coordinator responsible for managing each claim has sufficient knowledge of each client’s needs to make some independent assessment of the appropriateness of goals and outcomes, and to assess whether or not the outcomes have been achieved.

GAS enables a single measure to be used across all rehabilitation cases, despite significant differences in the nature of the problems experienced, or the length of time someone has experienced their problems. The single standardised score that is derived from the GAS for each client can provide DVA or other service purchasers and providers with a simple metric to indicate whether the goals of clients have been achieved through provision of purchased services.

The aim of this project was to assess the feasibility of goal attainment scaling as a method for guiding practice and assessing outcomes of rehabilitation services delivered to DVA clients by contracted providers. DVA introduced modifications to their standard forms for contractors and invited providers to use them and participate in an assessment of their usability. Feasibility of the GAS as a method for measuring outcomes for DVA clients was assessed at a number of levels: by reviewing the content of the documentation completed in the first 12 months; by surveying participating rehabilitation providers (online) after 18 months; and by interviewing key stakeholders within DVA.

Method

Participants

Accredited rehabilitation provider companies, contracted through the Australian Government to work with veterans in 3 states of Australia (Queensland, Victoria and South Australia), were sent a letter by DVA inviting them to participate in the GAS feasibility study by adopting the modified forms. Individual private practitioners were not invited to participate as they represent only a minority of providers. Fifteen rehabilitation organisations, including those providing the majority of services to DVA, agreed to participate in the study. The study was undertaken for a period of 12 months. At the conclusion of the study there had been 75 practitioners involved at some time (attending training and/or submitting completed documentation) and documentation was received from 14 organisations.

Intervention

Goal Attainment Scaling. GAS methods required practitioners to set rehabilitation goals in collaboration with the client and family or significant others, such as a carer. For each goal, the client and practitioner developed detailed and very specific observable and quantifiable descriptions of possible outcomes (refer to Table I for examples of goal attainment scales). Consistent with the recommended procedure in the literature (19, 20), 5 outcome levels were identified, including the expected or desired level of performance or outcome, 2 levels that would be seen as less favourable and 2 levels that were more favourable.

The 5 recommended outcome levels for each goal were assigned numeric values from –2 (the least favourable outcome) to +2 (the most favourable outcome). The expected outcome or goal was assigned 0. The client and practitioner reviewed the outcome after the planned intervention or a predetermined length of time, and a score between –2 to +2 was allocated to that goal. Consistent with the recommended procedure, standardised T-scores were calculated for each client by adding the GAS score for all goals then converting this total score to a single T-score using a GAS conversion table according to the number of goals. When converted, the aim is to achieve a mean score of 50. A higher score indicates achievement of the more favourable goals, while a lower score indicates that less favourable goals were achieved. There is good evidence for the reliability, validity and sensitivity of the GAS method in rehabilitation settings (21, 22).

Measures

Rehabilitation Documentation. Modifications were made to 4 different documents required by DVA to be used by contracted rehabilitation practitioners, including a rehabilitation plan, a rehabilitation plan amendment, a rehabilitation progress report (6-month review), and a rehabilitation closure report. Clients could have more than one plan amendment and progress report as required, consistent with usual practice.

Online survey of providers. An online survey was developed to explore providers’ experiences of using GAS, their views of the potential benefits to clients, any recommendations for modification to the study forms, and their overall willingness to continue to use GAS. Questions with Likert scale (e.g., agree-disagree) or categorical response options and open-ended questions were used. Open-ended questions included: “Are there any instances where you have found it difficult to use the GAS process or where you think it may not be relevant?” Providers also had contact with the researchers throughout the trial when questions arose, and records were kept of all issues arising and their resolution.

Interviews with key stakeholders. Two key stakeholders were presented the results of the analyses and asked for their views of the extent to which the GAS approach: (1) satisfactorily met the needs of DVA for a practical routine outcome measure, (2) could be used for quality assurance and routine monitoring, and (3) compared with current practice.

Procedure

Participating rehabilitation providers were invited to a one-day training session on GAS provided by the Australian Centre for Posttraumatic Health (ACPMH) and delivered by PhD-qualified instructors. They were given an Instruction Manual for Service Providers that was developed by ACPMH to describe the study and the procedures to be followed to implement GAS, including completing the required documentation.

Providers from the participating organisations implemented the GAS procedure and completed modified documentation for each new DVA rehabilitation case referred to their service. In addition to identifying and describing the goals of the rehabilitation plan, providers were also required by DVA to describe the nature of the service/s to be delivered. This was done using pre-existing categories designed to reflect the eight areas of DVA’s biopsychosocial rehabilitation service delivery model included in the initial assessment of client need: medical (compensable), medical (non-compensable), psychosocial, home/self-care, aids/modifications, recreation, rest/sleep and vocational/training. Categorisation of the goals was not part of the GAS procedure – it was an element maintained during modification of the existing DVA forms at the request of DVA. Providers sent de-identified copies of all GAS documentation to the researchers during the study, either at the point of completion of each form or at case closure.

Six months after completion of the study, all providers who had completed at least one rehabilitation plan during the trial were invited to complete an anonymous online survey via email invitations with the Survey Monkey link. Following completion of the study, 2 senior DVA rehabilitation policy personnel participated in a telephone interview to review the project findings.

Analysis

Feasibility of the GAS as a method for measuring outcomes for DVA clients was assessed at a number of levels. First, the rehabilitation documents were reviewed to determine the nature, quality, timing and outcomes of the goals being included in rehabilitation plans. This analysis also comprised a quality assurance check undertaken by a PhD-qualified rehabilitation specialist (Author 3) to verify that the goals being developed were suitable, achievable, realistic and provided an appropriate timeframe, given clients’ current documented circumstances. Second, the GAS-related content of the completed documentation (goals, outcomes, t-scores) was recorded in an Excel data base and descriptive quantitative and thematic qualitative analyses were undertaken. Third, responses by rehabilitation providers to items in the online survey were analysed using Survey Monkey and SPSSx. Finally, comments by key stakeholders to interview questions were analysed using thematic analysis.

Results

Rehabilitation documentation

A total of 82 GAS rehabilitation plans were received from participating rehabilitation providers. Of these 82 GAS plans, 15 were completed and ‘closed’ during the period of the study. In 14 cases, there were matched ‘sets’ of open and closed plans (i.e., a plan opened and closed during the study period for the same client); in the case of one client, only a rehabilitation closure report was provided, suggesting the plan had opened before the modified forms were in use. For all clients and all plans, a total of 202 goals were formulated. A total of 27 amendment plans were received with modifications to the GAS initially described. Of these 27 amendments, 17 were due to a change to the timeline in the plan (16 extended, 1 shortened), and the remaining 10 had new goals added to the plans.

Nature of goals

Two hundred and two goals were formulated during the study. Following thematic analysis of the goals, a number of common sub-themes were identified. Table II details the themes and provides some examples of the goals developed by providers.

There were goals provided in every DVA service delivery category which suggests that broader biopsychosocial rehabilitation needs were being considered by providers, along with traditional return to work needs.

Defining a range of outcomes

Of the 202 goals that were defined by providers, the expected outcome (score of 0) for clients was clearly defined in 194 instances. The 8 goals that were missing a definition of the expected outcome included one instance where a goal was specific enough to be a surrogate outcome and a number where the goals related to the role of the provider rather than the client’s outcomes. Given rehabilitation providers were instructed to clearly define the expected outcome level (0) as a minimum, and other levels “if time permitted”, a substantial number of providers managed to define outcomes for all levels. Results in Table III shows how many times each outcome (–2, –1, 0, +1, +2) was defined. This finding reflects opinion that specifying the continuum of possible outcomes is the most difficult task in the GAS process for most people (9). Examples of goals with 5 outcome levels defined can be seen in Table IV.

Quality of plan goals

The review of plan goals confirmed that, in general, there was logical sequencing of goals developed by the client and provider from stated barriers to identified rehabilitation objectives. Goals were usually clearly recorded and defined by activities so that an expected outcome would be measurable after an established time frame.

Goal Achievement Scores

During the feasibility study, 15 rehabilitation closure reports were received. Of these 15 reports, 11 contained GAS scores; the remaining 4 had left the GAS table blank. Reasons for the lack of GAS scores in the 4 blank forms were: (i) case closed due to client relocation (n = 1), (ii) plan closed early due to unstable psychological state of client (n = 1), and (iii) DVA staff member requested the case be closed (n = 2). Consistent with the procedure described to providers in the training and the manual, t-scores for the 11 GAS scores were provided as an overall indication of achievement of all goals. The t-scores ranged from 31 to 62 with a mean score of 48.

Online survey of rehabilitation providers

A total of 19 providers (25% of all those contacted) responded to the online survey, with respondents from each of the 3 states. Informal feedback indicated that some providers who had completed GAS forms no longer worked in the participating organisations, although no details of staff changes were provided by organisations to allow a more accurate distribution of the survey. The majority of providers responding identified themselves as having trained in occupational therapy (n = 7), with the remainder indicating they had been trained in psycho­logy (n = 4), rehabilitation counselling (n = 3) and social work (n = 3). The length of time working in rehabilitation ranged from 3 months to over 20 years with a mean of 6.2 years (standard deviation [SD] = 5.1). Providers had been working with DVA clients for a mean of 5 years (although the range was broad: from 3 months to 18 years [SD = 5.0]). Seven providers indicated that they were currently working with 5 or fewer DVA clients, with a further 8 indicating that they were currently working with more than 10 DVA clients. Two providers indicated that there were not currently working with any DVA clients. With regard to total client load, most respondents (n = 12) indicated that half or fewer of their clients were DVA clients.

Responses to questions comparing the experience of the GAS procedure to previous routine practice are provided in Table V. In response to a summary question (not reported in Table V), 7 out of 13 providers thought that GAS “provided a better service to DVA clients than previous rehabilitation planning” (1 “much better”, 2 “somewhat better” and 4 “slightly better”) with the remaining 6 saying the service was the same as before and none endorsing the only negative option (“less than before”). Six out of 15 said that the GAS provided “a better service to DVA”, while the remaining 9 said the service was the same as before.

In other comments, providers reported some issues related to the GAS procedures (such as problems identifying 5 levels of outcome and the time it takes to identify multiple goals or uncertainty about how to complete the progress report forms in particular). Analysis of the characteristics of respondents providing these kinds of comments suggested that they were more likely to have limited experience working with veteran clients or had not undertaken any of the initial training on the use of the GAS tool. Some comments also reflected a poor understanding of DVA’s biopsychosocial rehabilitation model or a lack of experience or skill in rehabilitation practice. By contrast, one provider noted: “The GAS process is actually a positive shift away from the strong RTW [return to work] focus of [the rehabilitation provider], and allows us as health professionals to return to the holistic, needs based service provision that we all wished to do in the first place.” Bouwens et al. (23) observed that a reasonable level of skill was required to implement the GAS approach. These findings highlight the fact that the introduction of GAS as routine process would need to involve ongoing training and support to providers to ensure it is properly implemented.

Key stakeholder interviews

Analysis of the data from interviews with key stakeholders confirmed their support of the GAS method. Stakeholders reported that they found the forms to be clear and consistent with the information needs of DVA. The t-scores recorded on progress and case closure forms were seen as easy to interpret and summarise. The additional information that was generated through application of the GAS method (such as categories of goals) was seen as useful for routine monitoring, with potential to provide cumulative data that could be useful at a number of levels over time. It was noted that the time taken to complete a plan using the GAS procedure was likely to be longer, particularly in the early stages of implementation when the process was new to providers; however, there was a perception that this would reduce over time, and that the potential benefits to clients and providers made this acceptable.

Discussion

This study aimed to assess the feasibility of GAS as a method for setting goals and measuring outcomes for clients of DVA receiving a wide range of services through the Rehabilitation Program. Feasibility was assessed by evaluating the quality of completion of the forms used by providers to determine and assess goals and outcomes; by seeking feedback from providers using GAS through an online survey; and through key stakeholder interviews with senior DVA staff who oversight rehabilitation.

The quality of the data produced by rehabilitation providers was generally very high. This was despite the fact that the high turnover and role change among providers that is typical in the sector meant that many providers completed only one GAS form, and many did not receive face-to-face training. There were some cases where definition of outcomes could have been more precise; however, overall the feasibility of the providers implementing GAS with veteran clients was demonstrated through the study.

The majority of the small number of providers who responded to the online survey indicated that the experience of working with clients with the GAS approach was at least as good as or better than past practice. This finding reflects the claims in the literature that the process of GAS is a useful mechanism for supporting comprehensive planning, for setting clear expectations of all parties involved in the plan (i.e., client, health care providers, rehabilitation providers), and for facilitating discussions between clients and providers (10–12). Where issues were raised in relation to the GAS procedure, these related primarily to matters that could be addressed by providing ongoing training and resources, particularly when new providers start to use GAS.

The results of this study confirm that the GAS provides a single measure of overall achievement on a range of goals for DVA clients with complex conditions and that this data would be useful as a routine measure of the outcomes of services purchased by DVA and provided to their clients.

While the overall acceptability and potential usefulness of GAS was established through this study, a number of recommendations about potential modifications to the forms and procedures were made by providers to improve the prospective benefits of adopting them. In particular, a simplified approach to using GAS where straightforward services were provided (such as household services or provision of aids/appliances) was proposed. The rationale for having some kind of GAS for these services is that even simple services like these have an intended positive outcome for clients that go beyond simply receiving the service. That is, mobility equipment should be comfortable and increase the client’s independent movement; modifications to houses should allow an individual to live independently with a good quality of life. Rather than assuming that such services always lead to positive results, the recommendation is that service purchasers should be following up with clients to ensure that the intended benefit had occurred. The GAS approach draws attention to weakness in current routine practice, where the provision of the service is considered to be achievement, rather than taking a consumer-focussed perspective.

Limitations

Clients themselves were not directly asked about their experiences with, or perceptions of, the benefits of GAS. The response rate to the online survey was not very high, and the turnover of rehabilitation staff meant that not all of those who responded had very much experience with the GAS and modified forms. It was not possible to follow-up professionals who had left their employment. The high workforce turnover and role changes that are typical in this service delivery sector also highlight the limitations of providing face-to-face training. Future training could be more effectively and economically delivered through an online or CD-based training package. Only 3 states in Australia were asked to trial the modified procedure; however, key stakeholders did not consider that other states would have different issues with the GAS. Their selection was based on the fact that other states were involved in different system development activities by DVA. While only a limited number of stakeholders were interviewed, they were those personnel with the closest involvement in system development activities and a good understanding of the needs of DVA.

Conclusions

Evidence from the study of the use of GAS as a routine outcome measure for veteran rehabilitation cases supported its feasibility: the quality of completion was generally high, despite the limited experience of most staff caused by rapid staff turnover, and the lack of face-to-face training; the providers who responded to the survey reported that the GAS approach and procedures for reporting supported a client-focussed approach to rehabilitation more than current routine practice; and the key stakeholders considered the data arising from the GAS would meet their currently unmet need for information to support and monitor provision of biopsychosocial rehabilitation services to clients.

The data derived from GAS has benefits at a number of levels: it supports the provider in their professional relationship with the client; it supports DVA rehabilitation coordinators to monitor the quality of services, particularly the extent to which providers are able to set appropriate goals and support clients to achieve them; it supports DVA rehabilitation coordinators to assess the extent to which purchased services have met the needs of clients; and it supports DVA senior executives to report to their stakeholders on the overall success of rehabilitation provided to clients. DVA is currently developing plans for introducing GAS nationally in Australia.

ACKNOWLEDGEMENTS

The research reported in this paper was funded by the Australian Department of Veterans’ Affairs. The views expressed in this article are those of the authors and do not reflect the official policy or position of the Australian Department of Veterans’ Affairs. The authors have declared that no competing interests exist.

References