Objective: To evaluate the validity of a script concordance test to assess guideline-consistent clinical reasoning by physical therapists in stroke rehabilitation, and to identify critical features of physical
therapists specializing in stroke rehabilitation.

Methods: A script concordance test was developed according to current standards. Four subgroups of physical therapists (those specializing in neurology, those focusing on neurology or geriatrics, other, and non-specialized undergraduate students) were asked to complete the test. The construct validity of the script concordance test was evaluated with 1-way analysis of variance (ANOVA) to estimate differences between subgroups. Associations between physical therapist characteristics, and script concordance test scores were analysed with bivariate regression analysis followed by multivariate analyses.

Results: The script concordance test, with 59 items, was completed by 211 physical therapists. ANOVA analysis showed statistically significant differences between the script concordance test scores of the 4 groups (p < 0.001), with higher scores by the physical therapists specializing in neurology compared with the other, non-specialized, subgroups. The multivariate analysis showed that better guideline knowledge (B = 1.07; CI = 0.48–1.65; p = <0.001), successful completion of the Dutch Neurorehabilitation course (B = 4.1; CI = 1.37–6.87; p = 0.003), and participation in professional development activities (B = 2.4; CI = 0.05–4.68; p = 0.046) were associated with higher script concordance test scores.

Conclusion: The script concordance test has good construct validity. Greater self-reported guideline knowledge, successful completion of the post-bachelor Dutch Neurorehabilitation course, as well as systematic participation in professional development activities facilitate important factors that enhance specialization. The script concordance test is a valid feedback tool for physical therapists to support professional development in the domain of stroke rehabilitation.

Accepted May 3, 2019; Epub ahead of print May 17, 2019

J Rehabil Med 2019; 51: 418–425

Correspondence address: Gert Kwakkel, VU University Medical Center Department of Rehabilitation, De Boelelaan 1117, 1081 HV Amsterdam The Netherlands. E-mail: g.kwakkel@vumc.nl

Adherence of physical therapists to scientific recommendations can improve care for stroke patients. This study developed a test to measure the use of scientific recommendations in the clinical care that physical
therapists provide to patients with stroke. The study also identified critical features of therapists who specialize in stroke rehabilitation. It was concluded that the test could indeed distinguish therapists specializing in stroke rehabilitation. The study also found that higher knowledge levels, successful completion of the Dutch Neurorehabilitation course and systematic participation in professional development activities, such as multidisciplinary team meetings and regional case meetings, stimulate the use of scientific recommendations and thus enhance specialization. The test developed in this study is a potential tool to support improvement in continued education to promote professional development of physical therapists in stroke rehabilitation.

S

everal systematic reviews have shown that the evidence and underlying key assumptions for applying neurological treatment approaches, such as Bobath, are weak (1–3). Up to 2008, several professional debates were held in the Netherlands to encourage a more eclectic, evidence-based treatment approach, using the International Classification of Functioning (ICF) as a framework and based on the current understanding of mechanisms that drive stroke recovery and functional prognosis. This eclectic approach allows us to apply new interventions (3) and innovative treatment strategies, based on recently developed theories about sensorimotor recovery for which evidence has been found in the field of stroke rehabilitation (4, 5). The growing and ever-changing amount of evidence in the field of stroke rehabilitation covers approximately 53 different interventions (2), indicating that stroke rehabilitation is a continuum of complexity, with highly complex and specific post-stroke treatments (1). This complexity challenges the use of evidence in the clinical reasoning process of physical therapists (PTs). We are of the opinion that adherence of PTs to changing clinical practice guidelines in stroke rehabilitation (6, 7), including guideline-consistent clinical reasoning, requires special skills that need to be trained in postgraduate specialization in stroke rehabilitation in order to improve care for stroke patients. Supported by a grant from the Dutch National Institute of Health, a 1-year course, called the Dutch Post-Stroke Neurorehabilitation Course, was started in the Netherlands (3).

Clinical reasoning is a vital component of clinical competence (8), and its importance is recognized in major policy documents about professional development. Higgs and co-workers (9), as well as Edwards and colleagues (10), defined clinical reasoning as a process in which the therapist, interacting with the patient and others, such as family members or other healthcare providers, helps to structure meaning, goals, and health management strategies based on clinical data, patient wishes and choices, and professional judgement and knowledge (9, 10). Systematic application of scientific evidence in the clinical reasoning process is thought to improve the effectiveness and quality of physical therapy practice. We are therefore of the opinion that assessment of the consistent use of clinical practice guidelines (CPGs) in clinical reasoning can provide feedback on the use of guidelines and can identify areas of improvement (11). However, valid measures of guideline-consistent clinical reasoning are scarce in postgraduate physical therapy practice (Box 1).

The script concordance test (SCT) is a written test, based on brief clinical scenarios, to assess clinical reasoning in a context of uncertainty. This test measures the extent to which the clinical reasoning of an individual professional matches that of a group of experts (8, 12, 13). Several studies on the assessment of clinical reasoning in the medical domain have shown that the SCT is a valid assessment tool across a variety of continuing professional development activities (12, 14). However, no research evidence is available on the validity of the SCT as a tool to assess the clinical reasoning of PTs. We are of the opinion that this tool might help us examine the use of guidelines in the clinical reasoning process of PTs in stroke rehabilitation.

This paper first describes the development of an SCT for physical therapy, focusing on the diagnosis, clinical assessment, neurological and functional prognosis for outcome, and treatment in accordance with the Dutch CPG on Stroke in physical therapy (further referred to as CPG Stroke) (2, 15). The study then evaluates the validity of the SCT, defined as the degree to which the scores (17) of the SCT are consistent with our hypothesis that there are differences between the following 4 groups: (i) PTs specializing in neurology; (ii) PTs focusing on neurology or geriatrics; (iii) other or non-specialized PTs; (iv) undergraduate physical therapy students. It was hypothesized that PTs specializing in neurology could be distinguished by their higher SCT score. Furthermore, it was identified whether knowledge, expertise and professional development of PTs were related to guideline-consistent clinical reasoning, in order to identify the critical features of PTs who specialize in stroke rehabilitation.

Development of the content. The SCT was developed according to the guidelines published by the Association for Medical Education in Europe (AMEE) for SCT construction (8, 12, 13) (Fig. 1). First, the project group carefully determined the scope of the SCT, including operationalization of its purpose and the focus of the assessment (Table I). Secondly, a test blueprint was created to bolster the content validity of the SCT. The creation of the test blueprint was a dynamic process, running in parallel with the construction of the test, based on scientific reviews of stroke rehabilitation (1, 2, 15, 16). Thirdly, clinical scenarios were constructed, also known as item vignettes or case vignettes, followed by a set of 2–3 questions, which could be scored on a 5-point Likert scale. The aim was to construct 20–25 vignettes with a total of 60–75 questions in order to achieve sufficient score reliability (12). Two authors (MM and NMO) were responsible for developing draft vignettes. The authenticity of the clinical scenarios was enhanced by studying real-life case examples, by job shadowing the first author in different work settings across the continuum of stroke care. The draft vignettes were reviewed by a development panel, consisting of 4 PTs who had participated in the development of the CPG Stroke. Consensus on relevance, clarity and content of the “cases” was reached in 2 e-mail rounds and 2 in-person consensus rounds.

Development of a web-based test. The final version of the case scenarios and items for the SCT were agreed upon by the authors and the development panel, and this version was programmed in a web-based test.

Development of a scoring algorithm. To develop the scoring algorithm, a reference panel consisting of 15 members was invited to complete the test. Members of the project group could nominate PTs from their network for the reference panel. Individuals were selected if they met all of the following 4 criteria: (i) registered in the Central Quality Register for Physical Therapy; (ii) high level of guideline knowledge and use of the CPG Stroke guideline in clinical practice, based on their participation in the development of the CPG Stroke and/or teaching a course in neurorehabilitation for PTs in which the CPG Stroke was used; (iii) consensus of at least 3 members of the project group and 3 members of the development panel about their level of expertise; and (iv) providing informed consent for participation in the study. Scores for each question were computed from the answers chosen by the reference panel, as proposed in the AMEE guideline (12). Credit for each answer was transformed proportionally to obtain a maximum score of 1 credit for the modal answer from the reference panel for each item, a score of 0 credits for an answer that was not selected by any of the reference panel and partial credit for an answer other than the modal answer. A web-based calculator developed by the University of Montreal was used to analyse the reference panel’s response and construct the scoring algorithm (16).

Optimization of the test items. In establishing the final version of an SCT, different quality assessment strategies for item optimization have been described. A stepwise quality assessment was performed based on the AMEE guideline (12) on SCT development.

First, the variability of the reference panel responses was assessed. The variability among the members of the reference panel has been shown to be a key determinant of the discriminatory power of an SCT (12). Ideally, SCT questions produce a range of expert responses clustered around a modal answer. Questions with unanimity or with a broad distribution of responses are considered to have low quality. The quality of answers was rated using criteria based on the AMEE guideline (12), as follows: (i) high quality: a range around a modal answer (maximum 3 different response categories); (ii) good quality: a maximum of 4 different response categories with only one expert choosing the extreme answer; (iii) doubtful quality: the 2 highest scoring answers are more than one response category apart; and (iv) poor quality: broad distribution. The doubtful and poor items were considered to be performing poorly in the SCT.

Secondly, the item-total correlation coefficient of the subject responses was assessed, which provided an estimate of each item’s discriminative capacity. A negative or low item-total correlation contributes minimally or not at all to the reliability of the test, although it can also reflect the heterogeneity of clinical competence or the nature of the domain tested. Therefore, it should be carefully considered whether the items with negative or below 0.05 item-total correlations should be discarded.

Thirdly, the content validity as perceived by the reference panel was assessed, rated on a 5-point Likert-scale ranging from (fully) disagree (0) to (fully) agree (5). For each item, the percentage of the members of the reference panel who judged that this item was an adequate reflection of guideline-consistent clinical reasoning, and was relevant for daily practice for patients with stroke, was calculated. An arbitrary cut-off point of 65% agreement or full agreement on this item was used. Percentage scores below this cut-off point were considered to indicate low content validity.

The items that performed poorly on 2 of the 3 levels of quality assessed were presented to the development panel. If 75% of the panel recommended removal of this item, it was discarded.

The total score of the optimized SCT was the sum of the credits of the remaining items, expressed as a percentage of the maximum score.

An undirected recruitment campaign, using an e-mail sent in February 2015 to 1,704 potential participants, was performed. Post-graduate PTs (n = 728) were approached via a Dutch national education institute for allied health professionals (Nederlands Paramedisch Instituut). This was a sample of PTs with a variety of fields of interest, as recorded by the institute, such as sports, musculoskeletal, neurology, cardiology and oncology. Physical therapy students (n = 976) at 7 universities of applied sciences with a physical therapy programme were also approached. All received a reminder e-mail in March 2015. After a positive response to the recruitment mail, a participant received a log-in code for the web-based SCT. After completing the SCT, participants were assigned to 1 of 4 groups based on specialization. Since there is no formal registry of PTs specializing in neurology in the Netherlands held by an institution or society, the authors defined a classification based on therapist characteristics. The first group consisted of a PTs specializing in neurology who met the specialization criteria stated in the CPG Stroke (i.e. treatment volume of at least 5 unique stroke patients a year, completion of the postgraduate course on stroke rehabilitation, participation in professional development activities in the field of stroke, and self-report of neurology being their main specialization). The second group consisted of PTs with a self-reported focus on geriatrics or neurology who did not meet all criteria for the group specializing in neurology. The third group consisted of PTs with other specializations (e.g. musculoskeletal or cardiovascular), or no specialization. The fourth group consisted of undergraduate physical therapy students.

The SCT was web-based and available for completion by the participants from February until June 2015. In addition, participants completed a survey to identify demographic characteristics and clinical expertise. By participating, PTs could obtain continuing education credits for the Dutch Central Quality Register for Physical Therapy. No incentive was available for the physical therapy students.

Data for all participants were entered into a computer database and analysed with the IBM SPSS statistical package (version 21.0). Descriptive statistics were used to present the characteristics of the subjects and the SCT score.

Construct validity. Construct validity was defined as the degree to which the scores (17) of the SCT were consistent with our hypothesis that there are differences between the 4 groups, so that PTs specializing in neurology could be distinguished by their higher SCT score. The construct validity of the SCT was analysed with 1-way ANOVA to estimate differences between subgroups with a Bonferroni post-hoc analysis.

Identification of critical features of PTs specializing in stroke rehabilitation. To identify physical therapy characteristics associated with guideline-consistent clinical reasoning in stroke rehabilitation, the optimized SCT score, expressed as percentages of the total score, was used as the dependent variable. To avoid random allocation of physical therapy characteristics in the regression model, 10 hypotheses were formulated based on the expertise of the project group and development panel. It was hypothesized that the following features of clinical competence would be related to a higher SCT score:

Higher level of guideline knowledge: defined as higher self-reported clinical practice guideline knowledge, using a numerical rating scale.

Being acquainted with international clinical practice guideline(s) in stroke rehabilitation. Acquaintance was positive when 1 or more international clinical practice guideline(s) were mentioned in response to the open-ended question.

More years of employment within specialized physical therapy.

Delivering care within a team, with teamwork defined as working with at least 3 different healthcare professionals, including structured multidisciplinary team meetings to discuss patients’ treatment plans.

• A higher number of unique stroke patients treated in the last 12 months, as reported by the PT.
• A higher frequency of reading scientific literature about stroke care per year, scientific literature being defined as international peer-reviewed journals.
• University-level education, defined as having successfully completed a research Master’s degree or a PhD programme.
• Having attended in-depth training on neurorehabilitation, defined as successfully having completed the Dutch Neurorehabilitation course.
• Participation in professional development activities, defined as systematic participation in teaching on neurorehabilitation, formalized community stroke networks, or case discussion meetings or peer feedback groups for neurology of the Royal Dutch Society for Physical Therapy.

Associations between physical therapy characteristics and SCT score were first analysed in a bivariate regression analysis to identify statistically significant independent determinants (p < 0.1) of the SCT score. Subsequently, multicollinearity was assessed, and only physical therapy characteristics with a correlation coefficient > 0.7, VIF> 10 or Tolerance < 0.2 were selected, to prevent over-parametrization of the prediction model. Finally, the remaining characteristics were included in a multivariate, forward-selection linear regression analysis followed by a backwards selection method. standardized and unstandardized coefficients (B) were estimated, with 95% confidence intervals (95% CI) for the unstandardized coefficients. Only physical therapy characteristics with a 2-tailed statistically significant level with a p-value < 0.05 in both methods were considered to make a significant contribution to the multivariable regression model.

The SCT developed contained 59 items, using a fixed questioning format, nested within 21 clinical scenarios. All criteria stated in the AMEE guideline (12) were met and the items were sufficiently distributed across the different domains of the test blueprint. An example item is presented in Box 2. A reference panel of 15 physical therapy experts on stroke rehabilitation developed the scoring algorithm. After the quality assessment, 12 items were assumed to perform poorly and the development panel selected 9 of these 12 items to be discarded. After this optimization procedure, 50 items in 21 clinical scenarios were used to calculate the total SCT score for each participant, expressed as a percentage of the maximum score.

A total of 211 participants fully completed the SCT, divided into 4 different groups of PTs (Fig. 2). Participants’ characteristics, experience, education and SCT scores are summarized in Table II. Complete datasets for the regression analysis were available for 210 subjects.

ANOVA analysis showed statistically significant differences in SCT scores between the 4 groups (p < 0.001). Levene’s test revealed that population variances were equal (p > 0.05). The Bonferroni test showed statistically significant differences (p < 0.01) between the mean SCT score of the group specializing in neurology and the graduated PTs with a different specialization, as well as with the physical therapy students and the group focusing on neurology or geriatrics. The mean differences between the students and the other specialized PTs did not achieve statistical significance (p = 0.3). This is illustrated in a boxplot of SCT score per group (Fig. 3)

The bivariate analysis showed associations between the SCT score and six PT characteristics: Self-reported rating of guideline knowledge, the frequency of reading scientific literature, the number of  unique stroke patients treated in the last 12 months, acquaintance with international clinical guidelines, completion of the Dutch Neurorehabilitation course, and participation in professional development activities (p < 0.1) (Table III). Since no multicollinearity was detected between the determinants, all determinants were included in the multivariate analysis

The multivariate analysis, using both forward and backward linear regression, showed that greater guideline knowledge, successful completion of the Dutch Neurorehabilitation course, and participation in professional development activities were statistically significantly associated with a higher SCT score (Table III). In this model, 22.5% (R²) of the variability of the SCT was accounted for by these 3 factors.

To our knowledge, this is the first study in which a SCT was developed for PTs working in stroke rehabilitation to discriminate between levels of specialization in terms of implementing consistent clinical reasoning. The SCT showed good construct validity, as it was able to distinguish PTs specializing in stroke rehabilitation on the basis of their SCT scores. The SCT score reflects the degree to which their use of the Stroke CPG matched that of experts. Furthermore, our data show that greater perceived knowledge of the stroke guidelines, successful completion of the Dutch Neurorehabilitation course, and systematic participation in professional development activities (i.e. knowledge about evidence-based guidelines, completion of the post-Bachelor’s Dutch Neurorehabilitation course, and participation in professional activities, such as teaching, participation in networks, case discussion meetings) did explain a substantial part of the variance of outcome. In other words, each of these 3 factors contributed significantly to higher SCT scores. This finding suggests that education and active participation in knowledge transfer are important aspects that drive specialization experts in stroke rehabilitation.

A strength of this study was the thorough and careful development of the case vignettes for the SCT test. All steps were discussed with professionals in the field of stroke rehabilitation and experts in educational sciences. In addition, the web-based design makes it easy to access and use. Another strength of this study is the measurement of clinical reasoning; the development of a valid assessment tool contributes to the body of knowledge on clinical reasoning and to the improvement of clinical reasoning, which is a vital component of clinical competence.

The current study had some limitations. First, the sample size was relatively small, mainly restricted by the limited number of participants in the group specializing in neurology. Secondly, some PTs reported that items were multi-interpretable. The literature shows that clinical reasoning strategies among healthcare professionals differ, depending on the content and the context of the clinical problem. We assume that the empirical-analytical approach of hypothetico-deductive reasoning does not always match the reasoning strategies of participants. Thirdly, the SCT we have developed is relevant only for a limited period, since evidence and healthcare evolve. Therefore, an SCT should be a “living test”, which is adjusted to fit new evidence that becomes available and new developments in healthcare, such as precision medicine. The SCT offers the opportunity to take variability into account; for instance, by presenting the same reasoning dilemmas in different individual contexts of clinical uncertainty. However, it should be investigated whether the individual variability that precision medicine accounts for is also assessed validly with the SCT. Fourthly, the SCT could have resulted in greater contrast if the variability among the answers of the reference panel could be reduced, since it seems that there was little consensus between panel members on some items. A follow-up consensus round in the reference panel to construct the scoring algorithm might reveal if outliers can be reduced and the validity increased further. We recommend this as an extra step in constructing an SCT. Furthermore, an SCT is not useful for all guidelines. The variability in the answers relates to the level of evidence, therefore guidelines that lack a high level of evidence are prone to variability in the scoring by experts and thus in the scoring algorithm. This will translate into low contrasts in SCT scores between PTs, making the SCT less valid for discrimination of experts from non-experts in the field.

Other validation studies of SCTs in the medical domain have examined the differences between students, residents and specialists (18–21). The total number of questions, and thus the time needed for completion, varied from 40 questions in 30 min to 153 questions in 3 h. These mean scores (50–80) from these studies found similar validity and is comparable to the current study. The authors of these studies concluded that the SCT is an innovative test that can distinguish between different levels of clinical reasoning, and that an SCT is a potential tool for professional development. The current study confirms that SCT is also applicable in the field of physical therapy.

These findings suggest that completion of the Dutch Neurorehabilitation course is associated with higher SCT scores and thus stimulates consistent integration of the CPG Stroke in clinical reasoning. Explicit learning methods, such as peer assessment, facilitate the integration of knowledge in clinical reasoning. This is crucial to learning how to integrate knowledge in clinical reasoning, since this is the way to cope with new rapidly growing evidence.

In contrast to physical therapy specializations, such as manual therapy and paediatric physical therapy, the Royal Dutch Society of Physical Therapy does not keep a country-wide registry of PTs working in the field of neurorehabilitation in the Netherlands. To support continued development, we suggest developing a postgraduate specialization programme for PTs specializing in neurology in the Netherlands. The programme could include active learning methods and systematic continuous professional development activities with built-in feedback and peer assessment in order to provide a full continuous quality improvement cycle. Highly motivated PTs can join the programme and make their continuous education activities transparent to different stakeholders. We believe that the construction of such a programme should be initiated by the physical therapy profession, with the aim of creating a learning environment.

Future research should focus on improving the psychometric properties of the SCT and its clinical use. It would be of interest to study whether the test is responsive to individual scores and subscores. This could reveal the components of clinical reasoning and the area of knowledge that an individual PT excels in, or identify gaps in knowledge that should be improved. This would allow PTs to give direction to and individualize their development. Ideally, the feedback should create intrinsic motivation to improve and excel. Of special interest in the future, will be the study of the relationship between the clinical reasoning of PTs and clinical and patient-reported outcomes.

We conclude that the SCT is a valid instrument for physical therapy practice, offering the potential to support an increase in guideline-consistent clinical reasoning by PTs in the domain of stroke rehabilitation, in undergraduate education, as well as postgraduate professional development.

The authors would like to thank the development panel for their contributions to this study: Barbara Harmeling–van der Wel (Erasmus MC, Rotterdam), Jip Kamphuis MSc (ViaReva, Apeldoorn), Roland van Peppen (Hogeschool Utrecht) and Janne Veerbeek (VU Medisch Centrum). For the recruitment of participants to this study we would like to thank Jacqueline Outermans (Hogeschool Utrecht), Anne Griet Brader (Hanze hogeschool Groningen), Corjan Hagendijk (Hogeschool van Rotterdam), Susy Braun (Hogeschool Zuyd), Tjarco Koppenaal (Avans Hogeschool), Miriam Wijbenga and Jaap van den Berg (Hogeschool van Amsterdam) and Marleen Buruma (Nederlands Paramedisch Instituut). Finally, we would like to thank all participating physical therapists and students.

The authors have no conflicts of interests to declare.