OBJECTIVE: There is a gap in the translation of knowledge about stroke between researchers and clinicians. This paper describes the creation and pilot testing of an evidence-based stroke rehabilitation intervention website, StrokEngine (http://www.strokengine.org), which was designed to close this gap.
DESIGN: A within-subject design was used to compare the usability and navigability of StrokEngine vs other search strategies/sites. Each participant searched a well-known stroke website, searched StrokEngine, and performed a free search, with the order of search randomized. A standard questionnaire was used to elicit information on usability and navigability across the 3 searches.
SUBJECTS: A purposive sample of 19 rehabilitation clinicians from Montreal, Quebec, with varied stroke-related treatment experience.
RESULTS: All 19 clinicians gave the highest usability score to StrokEngine (p < 0.05): StrokEngine usability score (mean 43, SD 4) vs the Cochrane Library (mean 26, SD 8), the Royal College of Physicians website (mean 20, SD 5) and a general Internet search (mean 26, SD 7).
CONCLUSION: This preliminary study on StrokEngine’s usability and navigability suggests that it has the potential to be an asset for clinicians who wish to keep abreast of information on intervention effectiveness.
Key words: cerebrovascular accident, critical appraisal, databases, factual, decision-support systems, evidence-based practice, health literacy, information dissemination, knowledge translation.
J Rehabil Med 2008; 40: 329–333
Correspondence address: Nicol Korner-Bitensky, School of Physical and Occupational Therapy, Faculty of Medicine, McGill University, 3630 Promenades Sir-William-Osler, Montreal, Quebec, Canada H3G 1Y5. E-mail: nicol. korner-bitensky@mcgill.ca
Submitted September 21, 2007; accepted January 2, 2008
INTRODUCTION
Stroke is a leading cause of serious long-term disability. In Canada alone there are more than 300,000 survivors living with impairments that impact on function and participation and cost the economy $2.7 billion annually (1). Thus it is not surprising that evidence-based knowledge regarding the effectiveness of interventions for stroke rehabilitation is rapidly emerging (2). Unfortunately there is a gap in knowledge translation (KT) between researchers and clinicians (3). This paper describes the creation and pilot testing of an evidence-based stroke rehabilitation intervention website, StrokEngine (http://www.strokengine.org), designed to close this gap.
In stroke rehabilitation there are large variations in the treatments chosen by physiotherapists (PTs) (4) and occupational therapists (OTs) (5) even within the same country. When comparing practices in different countries, variations in treatment are even more evident. For example, Ogiwara (6) compared stroke treatment performed by PTs in Japan with those in Sweden. There were major differences in the treatment approach between the 2 countries: Japanese respondents primarily chose functionally meaningful movement of the non-affected side, while Swedish respondents preferred weight-bearing on the affected side. In both cases most respondents indicated that practical courses and hands-on experience guided their treatment choices (6).
The importance of evidence-based practice (EBP) is evident in both the acute and rehabilitation phases of stroke management. A report from the Canadian Institute of Health Information, Health Care in Canada 2002, indicated notable differences in interventions used across regions of Canada and associated variations in key outcomes such as survival (7). Duncan and colleagues (8) demonstrated that adherence to post-acute stroke rehabilitation guidelines resulted in better functional outcomes.
The question arises as to whether clinicians seek out evidence of effectiveness of the interventions they use. Pringle (9), in a study of 264 OTs, found that only 16% stated that research was an important influence on current practice. Similarly, Turner & Whitfield (10) found that of 321 PTs surveyed in England and Australia, only 3% indicated that research articles led them to their current interventions. When 500 English OTs were asked to identify enablers of EBP, factors such as access to relevant resources, time to read and to find the evidence and to implement the evidence into practice, were identified (11).
Though journals are of interest to clinicians, they do not offer a readily available consolidated source of information about specific treatment interventions. Nor do clinicians necessarily have the time, or the critical appraisal abilities required to synthesize the literature. While it could be argued that the Internet offers quick and widely available access to information, this medium also presents disadvantage. For example, a recent search of the term “stroke” on the popular search engine Google (http://google.com) yielded 8 million results. A review of the first 50 revealed mostly general patient information and descriptions of specific acute medical interventions, but little information geared to health professionals regarding post-stroke rehabilitation. A search using the term “stroke” and 2 specific interventions (“balance training” and “aquatic therapy”) produced thousands of results, but with no evidence-based review geared to health professionals. A further search, using “stroke” and “evidence-based rehabilitation” also revealed thousands of websites. Of the first 50, 2 were evidence-based reviews: The Internet Stroke Center focuses primarily on medical interventions and provides general patient information and a stroke trials directory (12); while the Evidence-Based Review of Stroke Rehabilitation (EBRSR) provides an in-depth review of the latest stroke rehabilitation evidence for use by health professionals (13). It should be noted that the authors of the EBRSR have become collaborators in the creation of StrokEngine.
As there was no single website that consolidated information on stroke rehabilitation interventions in both lay terms for use by patients and their families, and professional terms for use by health professionals, it was deemed valuable to create such a site. Pilot testing of the family section of StrokEngine has already been carried out (14). The study presented here addresses 2 specific objectives related to the portion of StrokEngine aimed at health professionals. The first is to describe the process used in building this site, the second is to present the pilot study findings that evaluate, for clinicians working with a stroke clientele, perceptions of the usability and navigability of the StrokEngine prototype.
MATERIALS AND METHODS
Objectives
Objective 1: structure of StrokEngine. The StrokEngine homepage lists interventions or topic areas in alphabetical order. Each topic has been reviewed and summarized for 3 different audiences, as indicated by 3 links.
The “Quick Review” links to a 1-paragraph summary of the intervention/topic and indicates the levels of evidence (15) for its effectiveness on various outcomes. This summary has been developed primarily for health professionals, for example, the medical resident working in a stroke unit who is asked by a patient’s family about the value of acupuncture to relieve stroke pain.
The “In Depth Review” is aimed at health professionals who require in-depth knowledge of the effectiveness of specific rehabilitation interventions in order to assist them in making informed treatment decisions. The information includes relevant publications with links to the full publication, the evidence of effectiveness, a summary of the findings of each study, and any conflicting reports of effectiveness and plausible reasons.
Finally, “Patient/Family Info” provides a concise description of the topic, written at a grade 6 reading level, as described elsewhere (16).
Content creation of StrokEngine. The following methods are used to identify the evidence of effectiveness of each stroke rehabilitation intervention.
For each topic specific questions are constructed using the PICO framework (17) where P = Population, I = Intervention, C = Control, and O = Outcome. First, a question is posed – e.g. for a post-stroke patient (P) is acupuncture (I) more effective than usual therapy (C) at reducing spasticity (O)? The literature is then searched systematically to identify all articles pertaining to the question. If the article reports on a randomized controlled trial (RCT), the study is assigned a quality rating using the Physiotherapy Evidence Database (PEDro) scale (18).
Once each intervention has been exhaustively reviewed, the level of effectiveness of the intervention is determined specifically for use with a stroke clientele (19, 20). Five levels of evidence are considered: level 1a (supported by the results of 2 or more RCTs of “high” quality or a meta-analysis); level 1b (supported by a single RCT of “high” quality), level 2 (supported by at least 1 non-experimental study), level 3 (absence of RCT evidence, agreement by a group of experts on the appropriate treatment course, or pre-post designs, cohort studies), level 4 (disagreement between findings of at least 2 RCTs of equal quality) and level 5 (no published evidence regarding effectiveness).
Each StrokEngine module is updated every 6–9 months. Also, as novel interventions are introduced in clinical practice a new module is created.
Objective 2: pilot testing of StrokEngine. For our pilot study, usability and navigability of the StrokEngine prototype was compared against 2 commonly used search strategies – unstructured Internet searching and searching a well-recognized stroke website. Usability is defined as the quality of the user’s experience when interacting with a system, a combination of factors that includes ease of learning, efficiency of use, error frequency and subjective satisfaction (21). Navigability is the perceived organization of the information and the ease of finding information (22).
Subjects
A purposive sample of 19 therapists, 13 OTs and 6 PTs was identified from a listing of clinicians working in Montreal, Quebec. Therapists were chosen to reflect diversity in practice settings, stroke experience, Internet use and education. Because the website was developed in English (it is now available in French), an additional criterion was ability to read English content.
Measure – questionnaire
A questionnaire was developed based on an extensive review of the literature to identify items important to include in website evaluation. Section 1 included questions that queried clinicians regarding their: (1) current knowledge level of the stroke-related topic chosen; (2) estimation of the effectiveness of the intervention; (3) preferred method of learning; (4) usual method(s) of learning; and (5) comfort when searching the Internet for stroke-specific information. The last 2 items were measured using a 5-point Likert scale.
Section 2 contained 10 questions pertaining to website usability and navigability, each scored on a 5-point Likert scale from “very unsatisfied to “extremely satisfied,” including: overall impression of the home page; overall organization, completeness, and quality and quantity of information; ease of finding information; overall appearance; impression; ease of use; and satisfaction.
The third section elicited information on potential explanatory variables including those related to the clinician and their learning style, work environment and overall opinion on the usability of the 3 search strategies. The last questions asked participants to provide suggestions for additions, changes, and improvements to StrokEngine and their likes and dislikes.
Procedure
Ethics approval was obtained from an ad-hoc Ethics Review Committee, School of Physical and Occupational Therapy, McGill University, Montreal, Canada. Potential participants were recruited by telephone. Those who were eligible and agreed to participate were invited to attend a 2-h session in an electronic classroom. All 19 potential subjects agreed to participate.
On the day of the session, each participant signed an informed consent form. Each chose 1 of the 2 prototype StrokEngine module topics to search according to their preference, “acupuncture” or “interventions for unilateral spatial neglect (USN)”.
At baseline, participants completed Section 1 regarding their baseline knowledge of the chosen topic and preferred learning styles. Each then independently proceeded with 3 different searches. The first was a 20-min free search where they were instructed to search as they typically would. The free search was followed by completion of Section 2 related to usability and navigability. The second 20 min search, as allocated by concealed assignment, was either of StrokEngine or a recognized website that provided scientific evidence on the topic, specifically, the Royal College of Physicians website (23) for acupuncture, or the Cochrane Library website (24) for USN. Clinicians again completed Section 2. Finally, each clinician completed the third 20-min search, corresponding to the structured site that they had not yet looked at, and again completed Section 2.
Statistical analysis and sample size considerations
Descriptive statistics were used to determine the prevalence of responses to the various questions. To identify differences in perceived usability and navigability across the 3 sites, the Wilcoxon matched-pairs signed-ranks test was used to compare summed scores for each construct, specifically usability and navigability. A non-parametric test was chosen because of the small sample size. Given the matched nature of the analyses, a sample size of 19 was deemed sufficient to show differences in site preferences if indeed these differences existed.
Clinician comments and suggestions regarding additions and changes, features that they liked and disliked and their general comments were transcribed and coded using thematic groupings. This coding was done independently by 2 members of our research team and validated during discussion by the entire research team.
Results
Subjects
Clinicians worked in various settings including: acute care (6), in-patient rehabilitation (9), out-patient/community rehabilitation (2), homecare (1) and teaching hospital (1), and had varying levels of experience (10.6 years, standard deviation (SD) 10 years). Five were involved in administration work, 2 on a full-time basis. Eighteen had Bachelor’s level professional degrees; one had a Masters. All were female.
When asked to describe their baseline comfort in searching the Internet for information on effectiveness of treatments, 6 clinicians indicated “very”, 7 indicated “somewhat”, 4 indicated “a little” and 2 indicated “not at all”. When asked to describe their comfort in searching the Internet for stroke-related information 2 indicated “extremely”, 7 indicated “very”, 7 indicated “somewhat”, 2 indicated “a little” and one indicated “not at all”.
When asked to describe their preferred and typical way of learning, the most prevalent response for both was books or journals (Fig. 1). Few respondents indicated using the Internet for searching out new information.
Fig. 1. Learning preferences and actual ways of obtaining information about stroke
Website comparisons
Knowledge gained. Fig. 2 shows the respondents’ self-related perception of level of knowledge regarding the topic chosen (acupuncture or USN) at baseline and following each search. Interestingly, there was a substantial difference in the perceived knowledge gained according to which of the 2 recognized sites was used for the control search. The StrokEngine search was, on average, rated the highest for gaining new knowledge.
Fig. 2. Self-perceived mean level of knowledge according to Internet search strategy
Satisfaction with the sites. Fig. 3 indicates the respondents’ satisfaction with the various searches. The greatest overall satisfaction was associated with the StrokEngine search.
When using a free search strategy 10 of 19 clinicians indicated yes to “would you recommend this site to colleagues”: fewer indicated yes after completing the well-recognized sites (3 of 9 who searched Cochrane and 1 of 10 who searched the Royal College website). In contrast, 19 of 19 responded yes after performing the StrokEngine search.
Fig. 3. Participant satisfaction according to search strategy
Usability and navigability. All 19 clinicians gave the highest global usability score to StrokEngine with the Wilcoxon matched-pairs signed-ranks test, revealing significant differences at p < 0.05 between the StrokEngine usability score (mean 43, SD 4) and that of each of the 3 comparison sites including the Cochrane database (mean 26, SD 8), the Royal College of Physicians website (mean 20, SD 5) and the free search (mean 26, SD 7).
Two components of overall usability, “overall organization of information on the website” and “overall ease of finding information” were used to operationally define navigability. All 19 clinicians ranked the StrokEngine’s navigability above or equal to the “respected” websites, and higher than the free searches.
General comments and suggestions for improving StrokEngine. Suggested changes included requests for a French language version of StrokEngine, requests for specific modules, and links to referenced articles. Other suggestions included links to stroke organizations, StrokEngine author contact information, and more detailed explanations of the levels of evidence. Also, clinicians suggested additions that would enable customization of StrokEngine to specific regions around the world by incorporating information on stroke-related community resources by clicking on a map representing a specific region. Finally, there was a request for a site similar to StrokEngine but focused on assessment tools and their psychometric properties.
DISCUSSION
We developed a web-based knowledge translation tool to help bridge the gap between research and actual practice in stroke rehabilitation. The goal was to design a user-friendly interface and information structured in a way that permits a clinician to access varying depths of information depending on their needs. The results of this pilot testing are encouraging. The test version of our tool was consistently ranked higher than reputable websites and those found during free searches. Furthermore, the feedback provided by the clinicians has allowed us to tailor StrokEngine to the needs of health professionals. For example, as per the recommendations, additions now include: links to full articles, links to stroke organizations, a section to explain the levels of evidence, a French edition of StrokEngine, named Info-AVC, and a site focused on providing clinicians with information on the psychometric properties of screening and assessment tools, StrokEngine-Assess, which is accessible from the StrokEngine site.
As we continue to develop StrokEngine and StrokEngine-Assess there are numerous challenges. We are aware that there are clinicians who may not be comfortable using the Internet. In addition, as it is our goal to have StrokEngine used worldwide, we will collaborate with the international stroke community to insure that the site meets varying cultural and linguistic needs. We use ongoing consultation with an interdisciplinary team of experts in library science, knowledge translation, marketing, research, and clinical care to make the site valuable to the various stakeholders.
A second barrier to overcome is the sustainability of the resource. More than 75 researchers, clinicians and decision-makers have invested time and effort into the creation, evaluation and dissemination of StrokEngine and StrokEngine-Assess. We are aware that the site is valuable to clinicians only if it is continually updated with the newest evidence, and thus achieving long-term international funding is a major goal.
Potential limitations
This study was conducted in the city in which the developers of the StrokEngine site were researchers and as such there may have been some clinicians who scored the site more positively. Also, while we created the questionnaire based on existing standardized Likert scaling, the acceptability of combining the questions to create global scores of usability and navigability was not validated. Inasmuch as the clinicians appeared to understand clearly the questions and completion rates were high for all items, it is likely that the questions were valid.
To conclude, we have presented a stroke rehabilitation website, StrokEngine, developed with the goal of facilitating knowledge translation for clinicians regarding the effectiveness of interventions used in stroke rehabilitation. This preliminary study on its usability and navigability suggests that StrokEngine has the potential to be a valuable asset for stroke rehabilitation clinicians who wish to keep abreast of the latest information on intervention effectiveness.
Acknowledgements
We gratefully acknowledge the clinicians who participated. We also acknowledge the following individuals: Martine Brosseau, Nora Dun, Kathy Estevez, Genevieve Michaud, Amanda Slywynskyj and David Donnini.
This project was funded by the Canadian Stroke Network, the Heart and Stroke Foundation of Ontario, the Réseau provincial de recherche en adaptation réadaptation (REPAR) of the Fond de la recherche en santé du Québec (FRSQ), and the Centre de recherche interdisciplinaire en réadaptation du Montréal Métropolitain (CRIR). N. Korner-Bitensky is supported by a senior career award from FRSQ. A. Menon-Nair is funded by a doctoral award from the Canadian Institutes of Health Research (CIHR) on Innovations in Patient Safety and Knowledge Translation in the Faculty of Medicine, University of Toronto.
References
1. Stroke Statistics [page on the Internet]. Heart and Stroke Foundation. c2007 [updated 2002 Feb 4; cited 2007 Aug 15]. Available from: http://ww2.heartandstroke.ca/Page.asp?PageID = 33& ArticleID = 1078&Src = news&From = SubCategory
2. Knowledge translation & commercialization [homepage on the Internet]. Canadian Institutes of Health Research. c2007 [updated 2007 July 3; cited 2007 Aug 15]. Available from: http://www. cihr-irsc.gc.ca/e/29529.html
3. Sackley CM, Lincoln NB. Single-blind randomized controlled trial of visual feedback after stroke: effects on stance symmetry and function. Disabil Rehabil 1997; 19: 536–546.
4. Dumoulin C, Korner-Bitensky N, Tannenbaum C. Urinary incontinence after stroke. Identification, assessment, and intervention by rehabilitation professionals in Canada. Stroke 2007; 38: 2745–2751.
5. Walker MF, Drummond AER, Gatt J, Sackley CM. Occupational therapy for stroke patients: a survey of current practice. Br J Occup Ther 2000; 63: 367–372.
6. Ogiwara S. Physiotherapy in stroke rehabilitation: a comparison of bases for treatment between Japan and Sweden. J Phys Ther Sci 1997; 12: 63–69.
7. Health care in Canada [report on the Internet]. Ottawa: Canadian Institute for Health Information; 2002 [cited August 16, 2007]. Available from: http://secure.cihi.ca/cihiweb/dispPage.jsp?cw_page = PG_60_E&cw_topic = 60&cw_rel = AR_43_E
8. Duncan PW, Horner RD, Reker DM, Samsa GP, Hoenig H, Hamilton B, et al. Adherence to postacute rehabilitation guidelines is associated with functional recovery in stroke. Stroke 2002; 33: 167–177.
9. Pringle E. EPB: is it for me? Therapy Weekly 1999 Jun 10: 12.
10. Turner PA, Whitfield TWA. Physiotherapists’ reasons for selection of treatment techniques: a cross-national survey. Physiother Theory Pract 1999; 15: 235–246.
11. Curtin M, Jarmazovic E. Occupational therapists’ views and perceptions of evidence-based practice. Br J Ther 2001; 64: 214–222.
12. Internet Stroke Center [homepage on the Internet]. St Louis: Washington University; c1997–2007 [updated 2001 Aug 2; cited 2007 Aug 16]. Available from: http://www.strokecenter.org/
13. Evidence-based review of stroke rehabilitation [homepage on the Internet]. London, ON: St Joseph’s Health Care London; c2006 [updated 2006 Dec 18; cited 2007 Aug 16]. Available from: http://www.ebrsr.com/index_home.html
14. Rochette A, Korner-Bitensky N, Tremblay V, Kloda L. Internet information on stroke rehabilitation for clients and families: assessment of the quality of StrokEngine-Family. Disabil Rehabil. In press 2007.
15. Foley NC, Teasell RW, Bhogal SK, Speechley MR. Stroke rehabilitation evidence-based review: methodology. Top Stroke Rehabil 2003; 10: 1–7.
16. Doak CC, Doak LG, Root, JH, editors. Teaching patients with low literacy skills. 2nd edn. Philadelphia: Lippincott; 1996.
17. Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club 1995; 123: A12–A13.
18. Physiotherapy Evidence Database (PEDro) scale [page on the Internet]. Sydney, AU: Centre of Evidence-Based Physiotherapy; [updated 1999 Dec; cited 2007 Aug 16]. Available from: http://www.pedro.fhs.usyd.edu.au/scale_item.html
19. Grant H. Determining the need and provision of health services for stroke in Eastern Ontario: an epidemiological approach to needs assessment [report on the Internet]. Final report. Kingston (ON): Health Information Partnership Eastern Ontario Region and Queen’s Health Policy Research Unit; 2001 Feb. Available from: http://chspr.queensu.ca/downloads/Reports/strokereport.pdf
20. Gresham GE, Duncan PW, Statson WB. Post-stroke rehabilitation. Clinical Practice Guideline No. 16. Rockville, MD: US Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research; 1995 May. AHCPR Publication No.95–0662.
21. Williams P, Nicholas D, Huntington P. McLean F. Surfing for health: user evaluation of a health information website. Part one: background and literature review. Health Info & Libraries J 2002; 19: 98–108.
22. Zhang Y, Zhu H, Greenwood S. Website complexity metrics for measuring navigability. In: Proceedings of the Quality Software, Fourth International Conference on (QSIC’04); 2004 Sept 8–9; Braunschweig, Germany: IEEE Computer Society; 2004, p. 172–179.
23. Royal College of Physicians [homepage on the Internet]. London: Royal College of Physicians [updated 2007 Aug 1; cited 2007 Aug 16]. Available from: http://www.rcplondon.ac.uk/
24. Cochrane Library [homepage on the Internet]. The Cochrane Collaboration; c1999–2007 [cited 2007 Aug 16]. Available from: http://thecochranelibrary.com/
