Anika Aakerøy Jordbru, PhD, Liv Marit Smedstad, MD, PhD, Ole Klungsøyr, PhD and Egil Wilhelm Martinsen, MD, PhD
From the Vestfold Hospital Trust, Clinic of Physical Medicine and Rehabilitation, Tønsberg, Norway
OBJECTIVE: Psychogenic gait disorder, defined as loss of ability to walk without neurological aetiologies, has poor rehabilitation options that are well documented. Left untreated these patients have substantial and long-lasting dysfunction. The present study examined the effect of a 3-week inpatient rehabilitation programme compared with a waiting list control condition, and whether eventual gains were maintained at 1-month and 1-year follow-up.
DESIGN: A cross-over design evaluated the effect of treatment, and a carry-over effect was considered as a long-lasting treatment effect. Treatment consisted of adapted physical activity within a cognitive behavioural framework, and focused on offering an alternative explanation of symptoms, positively reinforcing normal gait and not reinforcing dysfunction.
Patients: A total of 60 patients were recruited from neurological departments and were randomly assigned to immediate treatment (intervention) or treatment after 4 weeks (controls).
RESULTS: Cross-over design revealed that the mean difference between treatment vs no treatment was 8.4 Functional Independence Measure units (p < 0.001, 95% confidence interval 5.2–11.7), and 6.9 Functional Mobility Scale units (p < 0.001, 95% confidence interval 5.5–8.3). Patients significantly improved their ability to walk and their quality of life after inpatient rehabilitation compared with the untreated control group. The improvements in gait were sustained at 1-month and 1-year follow-up.
CONCLUSION: Substantial and lasting improvement can be achieved by inpatient rehabilitation of patients with psychogenic gait, and the gains are maintained during follow-up.
Key words: psychogenic gait; psychogenic motor disorder; conversion disorder; movement disorder; physical activity; rehabilitation.
J Rehabil Med 2014; 46: 00–00
Correspondence address: Anika Aakerøy Jordbru, Vestfold Hospital Trust, Clinic Physical Medicine and Rehabilitation, Post Box 2168, NO-3103 Tønsberg, Norway. E-mail: Anika.Jordbru@siv.no
Accepted Aug 22, 2013; Epub ahead of print Nov 13, 2013
Introduction
Psychogenic gait is a psychogenic motor disorder (PMD) that affects the ability to walk and is characterized by symptoms that are not explained by neurological disease. Diagnostic criteria for PMDs were first published by Fahn & Williams in 1988 (1). Other labels for the same phenomenon are medically unexplained symptoms (MUS), non-organic or functional neurological symptoms and conversion gait disturbance.
This condition is relatively common in clinical practice and represents challenges for those affected and for the healthcare system. In a recent study of more than 3,000 outpatients referred to neurological clinics in Scotland, one-third had symptoms that were not at all, or only somewhat, explained by organic disease (2). For conversion of motor type an annual incidence rate of 15–22/100,000, and a 1-year prevalence rate of 300/100,000 has been reported (3). The course of PMD tends to be long-lasting, and Stone and co-workers (4) stated that, out of 60 patients, more than 80% remained symptomatic and disabled 12 years after the original diagnosis was given.
Various forms of treatment for PMD have been proposed, but evidence supporting any treatment is scarce (5). Goldstein and co-workers (6) found that cognitive behavioural therapy was more effective than standard medical care in reducing seizure frequency in psychogenic non-epileptic seizures (PNES) . Guided self-help for functional (psychogenic) symptoms was found to be efficacious for patients with neurological symptoms not explained by organic disease (7). Three randomized controlled trials on conversion disorder have been included in a Cochrane review (8). Two examined the usefulness of hypnosis (9, 10), the third compared paradoxical intention with diazepam treatment for PNES. The review concluded that the benefit could not be established due to poor methodological quality.
Intervention studies are few. Heruti et al. (11) report that participation in active regular and integrative rehabilitation is beneficial. One prospective study without a control group assessed the usefulness of rehabilitation for 39 patients with conversion motor disorder (12). Progressive training was effective for 8 out of 9 acute patients. Thirteen out of 21 patients with chronic disorders, who underwent a strategic behavioural intervention (patients were told that full recovery indicated organic aetiology of symptoms, whereas failure to recover was a definitive proof of psychiatric disease), were symptom-free at discharge. The strategic intervention was superior to standard behavioural treatment. In a historical cohort study Czarnecki et al. (13) found that 1 week of intensive rehabilitation, based on motor reprogramming, was successful in 60 patients with functional movement disorders. Forty percent of patients had a gait disorder. The authors call for prospective, controlled clinical trials, and to our knowledge, no previous randomized controlled trial has assessed the usefulness of physical rehabilitation in psychogenic gait.
The aim of the present study was to examine: (i) the effect of a 3-week inpatient rehabilitation programme compared with a waiting list control condition for patients with psychogenic gait; (ii) whether eventual gains were maintained at 1-month and 1-year follow-ups.
Material and Methods
Methods
This was a cross-over study in which patients were randomized consecutively and equally (with a 1:1 ratio) to immediate 3 weeks of treatment or 4 weeks on a waiting list. Those on the waiting list received treatment after the waiting period.
The trial was conducted at Vestfold Hospital Trust, Clinic of Physical Medicine and Rehabilitation, which has a catchment area of approximately 400,000 from the South-Eastern region of Norway. Recruitment took place between May 2007 and October 2010.
Inclusion criteria were: disabling walking disturbance resembling psychogenic gait with no organic explanation after neurological examination; age 18–69 years; duration less than 5 years, and; willingness to participate in the study. We excluded patients who needed inpatient psychiatric treatment, those with coexistent somatic disorders (multiple sclerosis, cerebral palsy, etc.) or those who did not want to take part in active rehabilitation.
Patients
Of the 114 patients consecutively referred to the hospital out-patient clinic for gait disturbances, 54 were not included in the study (see flow-chart in Fig. 1). The most common reasons for exclusion were that: the gait disturbance was judged not so severe that it interfered with their ability to function in their daily lives (18 patients); symptoms indicating need of inpatient psychiatric treatment (9 patients); symptom duration longer than 5 years (10 patients); comorbidity with neurological disorders (9 patients); or that patients did not want to take part in a rehabilitation programme based on physical activity (8 patients).
Fig. 1. Show flow chart of patients through the randomized controlled trial.
Most of the included patients had a severe limp in one leg, often with dragging of the foot, which was rotated laterally or medially. Other frequently observed characteristics were walking with small, slow steps, as if walking on ice, or/and truncal imbalance (14).
Patients received information about the study orally and in writing. They were informed that treatment would take place within 4 weeks. Patients were randomly assigned by blocks of 4, balanced for sex, to intervention or control groups. The randomization procedure was performed at a statistical office at a site remote from where the study was conducted. The first author was blinded to information about intervention or control group, which was kept in sealed envelopes. The envelopes were allocated to patients consecutively in the same order as patients had given written informed consent. Patients in the intervention group were admitted to the hospital within a week, while those in the control group waited at home for 4 weeks. The patients from the intervention group, as well as the control group, were consecutively admitted to the ward, and the team did not know to which group the patients were allocated. The first author handled all data collection and was not involved in the treatment.
Among the 60 patients who were included, 31 were randomly assigned to the intervention and 29 to the control group. In the intervention group 1 did not attend and 1 dropped out during treatment. In the control group 4 participants did not meet for treatment after the waiting list period. Forty-six responded to 1-month and 40 to the 1-year follow-up.
Intervention
The intervention consisted of adapted physical activity (APA) with an educational and cognitive behavioural frame of reference. The treatment was carried out by an interdisciplinary team consisting of physician, physiotherapist, occupational therapist, nurse and an educator in APA. The patients were admitted to a unit with approximately 20 patients with mixed diagnoses, and among these, only 1 or 2 at the time had psychogenic gait.
The intervention consisted of 3 main elements, as follows:
Symptom explanation. At admittance the patients were given an adapted medical explanation of their functional disturbances, but no specific diagnostic label. The aim was to present an alternative understanding of their symptoms. The patients were told that there is no exact explanation of the symptoms, except that they commonly occur following stressful life events. Typically, explanations would entail telling the patients that thorough examinations have ruled out serious illness. The patients are reassured that it is common to see a disconnection between the nervous system and muscles. There are good chances for reconnection by attending multiple activities, and a quick recovery can be expected.
Positively reinforcing normal function. Following any improvement in gait or posture according to their rehabilitation goals, the team reacted with positive reinforcements. The positive responses were given both during therapy as well as during other ordinary daily activities. The treatment often consisted of daily adapted sport activities, such as riding a bicycle, ball activities, outdoor canoeing, and indoor climbing, and patients were helped to shift focus from disability to mastering of activities. Patients were admitted for pre-planned stays for 3 weeks and had contact with hospital staff on a 24-h basis, including outside training sessions, especially with nurses during the evenings and nights. The majority spent at least the first weekend at the hospital. Encouraging and reinforcing normal function was also a joint treatment strategy when patients were not in training situations. This made the institution a round-the-clock arena for treatment. Thus, we tried to convey a clear message that a person can get better by training, with focus on activities they can do in spite of their dysfunction.
Not positively reinforcing dysfunction. Whenever no improvement was seen during training sessions, the positive feedback from the team was held back. The team focused on the healthy part of the patients and their resources, and less on the symptoms and lack of function. The team aimed to minimize the attention given to sickness or illness behaviours. This attitude was much more difficult to standardize, because care and consideration are strong elements in hospital treatment. However, by emphasizing the importance of this element in treatment and drilling the staff, we believe that it was accomplished. As part of the strategy with lack of positive reinforcement of dysfunction, patients were informed that the standard length of 3 weeks in hospital would be reduced if no progress took place within the first week. Those who benefitted from this approach experienced considerable recovery during the first week of intensive training. Prolonging a hospital stay for a patient with no progress would be unfavourable for the patient, leading to the opposite of mastering.
Assessments and instruments
At baseline data on socio-demographics and symptom duration were collected (Table I). At baseline, admittance, discharge, 1-month and 1-year follow-up the participants completed the following instruments:
Table I. Demographic and clinical characteristics of all included patients at baseline (n=60) |
|||
Characteristic |
All patients |
Intervention (immediate treatment) |
Control (treatment after 4 weeks) |
Gender, n (%) |
|||
Female |
48 (80) |
25 (81) |
23 (79) |
Male |
12 (20) |
6 (19) |
6 (21) |
Gait-function, n (%) |
|||
Wheelchair-bound |
15 (25) |
5 (16) |
10 (34) |
Walking with walker/crutches |
23 (38) |
12 (39) |
11 (38) |
Walking without aids |
22 (37) |
14 (45) |
8 (28) |
Employment-level, n (%) |
|||
Full-time employed |
28 (47) |
11 (35) |
17 (59) |
Part-time employed |
10 (17) |
8 (26) |
2 (7) |
Disability pension |
12 (20) |
7 (22) |
5 (17) |
Without regular job |
5 (8) |
2 (7) |
3 (10) |
Student |
5 (8) |
3 (10) |
2 (7) |
Living alone, n (%) |
41 (68) |
20 (65) |
21 (72) |
Not smoking, n (%) |
36 (60) |
16 (52) |
20 (69) |
Age, years, mean (SD) [range] |
37.6 (11.0) [18–62] |
38.8 (12.2) [18–62] |
36.3 (9.7) [18–58] |
Years of education after public school, mean (SD) [range] |
2.0 (1.8) [0–8] |
2.1 (2.0) [0–8] |
2.1 (1.6) [0–7] |
Duration, months, mean (SD) [range] |
9.5 (12.1) [>1/48] |
8.3 (10.9) [>1/48] |
10.9 (13.3) [>1/48] |
SD: standard deviation. |
Functional Independence Measure (FIM). This 18-item scale assesses physical and cognitive disability (motor subscale, 13 items; cognitive subscale, 5 items). Items are scored to assess the level of assistance required for an individual to perform activities of daily living (15). Each item is scored on a 1–7 scale, giving a range of sum scores of 18–126; higher scores indicate more independence (16). To ensure reliable scores, every second year the current rater (AJ) underwent a test for judging precision in scoring FIM.
Functional Mobility Scale (FMS). This scale classifies functional mobility according to the need for assistance devices over 3 distinct distances: 5, 50 and 500 m (17). Assistive devices range from walkers and crutches to wheelchairs (18). For each distance a rating of 1–6 is assigned. Sum scores range from 3–18, and higher scores indicate less dependency. The FMS was scored by the first author (AJ).
Health-related quality of life. This was assessed by self-report using SF-12, which is a down-sized version of the Short-Form Health Survey (SF-36). The 12 items provide scores for domains of physical and mental health, respectively (19). SF-12 provides these scores, originally derived from SF-36, with remarkable accuracy but far less respondent burden (20). The SF-12 has been validated for use in the USA, UK and many other European countries, including Norway.
Ethical considerations
All patients were informed about the study according to the Declaration of Helsinki, and written consent was obtained. There are no other treatment options available for these patients in Norway, and not offering treatment was therefore considered unethical. But leaving the control group untreated for a period of 4 weeks was regarded as acceptable, as many had had symptoms for years. The study was approved by The Regional Committee for Ethics in Research. The trial is registered at www.ClinicalTrials.gov (ID NCT01422278).
Statistics
The cross-over design and randomization in this study made it possible to assess the causal effect of treatment with high power. The efficiency of the design is due to use of within-person information; each patient was exposed to a period of intervention and a period of no intervention. A treatment effect beyond the completion of the intervention is considered as a carry-over effect. In this design the carry-over effect can be regarded as a long-term treatment effect and is an advantage. All patients contribute to its estimation, in contrast to the simplest and less efficient cross-over design (21). Longitudinal analysis of the mean response profiles was performed to impose a minimum of restrictions on the shape of development over time within the 2 groups, and the covariance between the responses at the different time-points. Time was considered as categorical, and no structure was assumed for the covariance matrix (22). The mean response profiles in the 2 groups for the outcome measures are shown in Figs. 2, 3, 4 a–b. Residuals were inspected to assess model adequacy.
Fig. 2. Development in Functional Mobility Scale (FMS) in intervention and control groups. Time 0: baseline; Time 1: end of treatment r1, end of control r0; Time 2: end of treatment r0, one-month control r1; Time 3: one-month control r0; Time 4: one-year control r0 and r1; 95% confidence interval.
Fig. 3. Development in global Functional Independency Measurements (FIM) score in the intervention and control groups. Time 0: baseline; Time 1: end of treatment r1, end of control r0; Time 2: end of treatment r0, one-month control r1; Time 3: one-month control r0; Time 4: one-year control r0 and r1; 95% confidence interval.
Fig. 4. Development in Health Related Quality of Life (Short-Form Health Survey 12; SF-12) in the intervention and control groups. (a) Physical domain and (b) Mental domain. Time 0: baseline; Time 1: end of treatment r1, end of control r0; Time 2: end of treatment r0, one-month control r1; Time 3: one-month control r0; Time 4: one-year control r0 and r1; 95% confidence interval.
With regard to loss to follow-up, the estimation method (maximum likelihood) is consistent (unbiased for large n) under the assumption of “missing at random” (21). A violation of this assumption can create selection bias. To get an impression of the magnitude of selection bias, the analysis of the observed data is compared with a complete case analysis that consists only of those patients with observations at all time-points (n = 40).
All statistical analyses were performed with the statistical package PASW Statistics 18 (release 18.0.1).
Results
The majority of patients were female, mean age 38 years, and mean duration of symptoms 10 months (Table I). The majority needed walking aids. The most disabled had incomplete paralysis and were wheelchair users, while those who were less disabled used walkers or crutches.
Outcome
The mean duration of the no-treatment waiting time for the control group was 4 weeks. The mean level of all outcome measures improved during the intervention in both groups. This is quantified and tested for significance in Table II.
The model showed strong and significant treatment and carry-over effects for both FMS and FIM (Table II). The mean difference between treatment and no treatment was 6.9 FMS units (p < 0.001, 95% confidence interval (CI) 5.5–8.3), with treatment as reference category, and a carry-over effect of similar magnitude of 8.1 units (p < 0.001, 95% CI 5.9–10.3). The mean difference between treatment and no treatment was 8.4 FIM units (p < 0.001, 95% CI 5.2–11.7), and carry-over effect was 9.2 (p < 0.001, 95% CI 5.4–13.1). The improvements were also clinically significant. The majority were helped back to independent living, and many returned to work.
For the SF-12 physical and mental assessments the model showed significant treatment effects, but the carry-over effect was significant only for SF-12 physical (Table II). For the SF-12 physical, the mean difference in score for treatment vs no treatment was 11.7 units (p <0.001, 95% CI 7.2–16.1), with treatment as reference category, and the carry-over effect was 14.1 (p <0.001, 95% CI 5.9–22.2). For the SF-12 mental the mean difference in score for treatment vs no treatment was 6.9 units (p <0.01, 95% CI 2.1–11.8), and the carry-over effect was not significant.
No significant differences between the groups were observed, neither were there any significant changes over time that were not covered by the treatment or carry-over effects.
Selection bias was of no magnitude, shown by complete case analysis. The score of treatment effect for FMS was 6.4 (95% CI 3.9–8.9), for FIM the difference score was 6.4 (95% CI 3.9–8.9), for SF-12 physical the difference in score of treatment was 14.3 (95% CI 8.4–20.1) and for SF-12 mental it was 9.1 (95% CI 3.4–14.8).
No patients needed a wheelchair or crutches at discharge from hospital.
Table II. Statistical analysis for all included measurements |
|||||
T0 (n = 60) Mean (SD) |
T1 (n = 59) Mean (SD) |
T2 (n = 55) Mean (SD) |
T3 (n = 46) Mean (SD) |
T4 (n = 40) Mean (SD) |
|
FMS (score range 3–18) |
|||||
Intervention group |
9.6 (4.3) |
16.5 (4.6) |
17.4 (2.2) |
17.4 (2.2) |
16.2 (4.5) |
Waiting list |
9.2 (4.5) |
9.8 (4.9) |
15.4 (4.6) |
16.7 (3.4) |
17.9 (0.4) |
FIM motor (item 1–13) (score range 13–91) |
|||||
Intervention group |
80.7 (12.2) |
90.1 (2.2) |
90.4 (2.0) |
90.4 (2.0) |
90.7 (0.7) |
Waiting list |
79.3 (13) |
80.9 (12.1) |
89.6 (2.8) |
90.7 (0.8) |
90.7 (0.7) |
FIM cognitive (item 14–18) (score range 5–35) |
|||||
Intervention group |
34.45 (1.84) |
34.9 (0.2) |
34.9 (0.4) |
34.9 (0.4) |
35 (0.0) |
Waiting list |
35 (0.0) |
35 (0.0) |
34.9 (0.2) |
35 (0.0) |
35 (0.0) |
SF-12 Physical (score range 0–100) |
|||||
Intervention group |
25.7 (8.0) |
37.2 (10.8) |
35.5 (11.5) |
35.5 (11.5) |
28.6 (10.2) |
Waiting list |
28.3 (8.6) |
27.3 (8.1) |
36.6 (13.9) |
40.1 (14.2) |
44.5 (13.7) |
SF-12 Mental (score range 0–100) |
|||||
Intervention group |
47.3 (14.3) |
54.9 (9.0) |
51.6 (10.7) |
51.6 (10.7) |
49.3 (13.6) |
Waiting list |
42.9 (12.9) |
45.8 (13.5) |
54.3 (10.4) |
54.8 (9.8) |
52.1 (9.1) |
FMS: Function Mobility Scale; FIM: Functional Independence Measure; SF-12: Short-Form Health Survey-12; T0: baseline; T1: end of treatment intervention group, end of control, control group; T2: end of treatment control group, one-month control intervention group; T3: one-month control, control group; T4: one-year control, control group and intervention group; SD: standard deviation. |
Discussion
A 3-week inpatient rehabilitation programme was effective in improving psychogenic gait relative to an untreated control group, as reflected in significant changes on all measures. Patients kept their gains at 1-year follow-up, the only exception being the SF-12 mental subscale.
This study confirms the results from previous studies without control groups (12, 23), as well as a retrospective cohort study (13) and numerous case reports (24). Physical rehabilitation combined with symptom explanation and positive reinforcement is effective for improving physical function.
The treatment effect in this study is well illustrated in the response profiles (Figs 2, 3, 4 a–b). The randomized group assignment ensured that the only aspect separating groups (e.g. at the first time-point, 3 weeks) is that the intervention group has received treatment earlier than the control group. Therefore, the difference in response between the groups can be attributed to the treatment. The carry-over effect is viewed as a long-lasting treatment effect. It has the same role in the model as the treatment effect, and is of value for the patient.
There were some drop-outs at 1-month and 1-year follow-up. This is a clear limitation, as those who drop out may have worse outcome. However, comparison between analyses of observed data and complete case analysis showed a weakened treatment effect for FIM and FMS in the complete case analysis, but strengthened treatment effect for SF-12 physical and mental. This indicates that a potential selection bias does not systematically go in one direction, as might be expected, for example, if only patients with poor outcomes at baseline were lost to follow-up.
Duration of symptoms in our study varied from 1 to 48 months. Patient with acute onset had more severe dysfunction, but both those with short- or long-lasting complaints responded well to the programme. Previous literature has reported that factors associated with quick recovery are acute onset and prompt treatment (25, 26), and that recovery would be less likely after 2 years from onset (27). In our study, however, patients with duration as long as 4 years responded well.
In a comprehensive treatment programme like this, it is not possible to estimate which of the elements have caused the therapeutic effect. The joint symptom explanation from all team members is aimed at giving the patients an alternative understanding of their symptoms. We used the wording symptoms instead of an explicit diagnosis, aiming at reducing the stigma of a mental disorder. Previous studies state that patients, whose symptoms cannot be explained by physical disease, often have poor outcome after specialist consultation that focuses only on excluding organic disease (28). The literature supports the premise that offering a good explanation of the symptom to a patient is a prerequisite to successful further treatment (29). The findings in this study support the importance of presenting an alternative understanding to patients.
Positive reinforcement is a strong stimulus to change, and is an important behavioural element within a cognitive behaviour therapy (CBT) paradigm. The main behavioural element is the physical activity. This has been shown to be useful in the treatment of other mental disorders, especially depression and anxiety disorders (30, 31).
A strength of this study is the design, with random assignment to treatment and control groups. Patients were assessed by the use of well-established instruments. Both therapist-rated questionnaires (FIM and FMS) have detailed descriptions of anchoring points, which reduces the danger of rater bias. The rater had received training in the instruments before the study, and had formal competence on one of the instruments (FIM). The same person rated the patients at all time-points.
There are no validated instruments that are developed for assessing psychogenic gait, and previous studies have not used specific instruments. FMS was developed for assessment of children with cerebral palsy, but has also been used in other diagnostic groups in rehabilitation settings (32, 33). FMS has been useful for discriminating between varying levels of disabilities and functional mobility, and is sensitive to change after rehabilitation intervention.
Although the instruments are not specific for psychogenic gait, we believe that our choice of instruments is adequate.
SF-12 is a self-report instrument. The use of both therapist rating and self-report, and the consistency between these 2 kinds of measures, support the validity of the findings.
A major limitation is that the study was not blinded. The rater had access to information about whether the current patient was in the control or intervention group when the rating took place. Clinical studies like this may at best be single-blind, and future studies should include blind rating of outcome.
Another limitation is that the randomization lasted for only 4 weeks. From a scientific point of view it would be optimal if the waiting time for the control group lasted longer. For ethical reasons we determined that the control period should not be increased, and we therefore cannot exclude the possibility of some spontaneous remissions. The beneficial outcome at 1 year may be due to the intervention. However, spontaneous remissions are also possible and may contribute to the good long-term effect
Optimally we should have had a more comprehensive description of the patients’ clinical state and comorbid conditions, such as anxiety, depression and fatigue, but these data were not available.
The only outcome measure in which changes were not statistically significant, was SF-12 mental at 1-and 12-month follow-up. One explanation may be a floor effect, as the deviation from the normal on this measure was so small that significant change would be difficult to obtain. An alternative explanation is that the programme did not significantly affect these aspects. We have no other measures of mental health that might have solved this problem.
Interdisciplinary treatment in a setting of adapted physical activity and rehabilitation could be useful in other institutions. The major challenge, which was successfully taken care of in this study, was to have a whole team in a round-the-clock service meeting the patients consistently and loyal to a mutual therapeutic understanding. It would be interesting to examine whether this approach could be transferred to other patient groups, such as those with chronic pain and/or fatigue.
This study indicates that a 3-week inpatient rehabilitation programme leads to significant improvements in gait and functional independency, and that gains are sustained at 1-year follow-up. This is promising for a patient group for which evidence-based treatment has previously been scarce.
There is a need for future randomized studies with blind assessment or outcome. A closer look at factors that may predict positive outcome would also be of interest.
Acknowledgement
The authors would like to thank all of the patients in Clinic of Physical Medicine and Rehabilitation, who participated in the study.
References