Anita Menon, PhD Fellow1,2, Nicol Korner-Bitensky, PhD2,3, Mark Chignell, PhD4 and Sharon Straus, PhD1

From the 1Department of Health Policy, Management and Evaluation and 4Department of Mechanical and Industrial Engineering, University of Toronto, 2School of Physical and Occupational Therapy, McGill University and 3Centre de recherche interdisciplinaire en réadaptation du Montréal Métropolitain, Montreal, Canada

Rationale and OBJECTIVES: Rigorous usability testing of e-learning resources is an important prerequisite to their widespread use among clinicians. This study demonstrates the application of an evidence-based approach to usability testing of two stroke-related e-learning resources (StrokEngine).

METHODS: 14 stroke rehabilitation clinicians (occupational therapists and physiotherapists) from Ontario, Canada participated in a 1.5 h in-person testing session. Clinicians navigated StrokEngine in search of information to answer questions on stroke assessment/intervention. Their search patterns were observed and clinicians provided verbal/written feedback about StrokEngine. Content analysis was used to generate themes and categorize them under two broad categories: facilitators and barriers to use.

RESULTS: Five key facilitators and three key barriers to StrokEngine use were identified and related to screen format, layout/organization, ease of navigation, quality of content, likelihood of using StrokEngine in the future, and system dysfunctions. All 14 clinicians were very or extremely satisfied with the layout/organization, quality and clinical relevance of the content, stating that they were likely to use StrokEngine in the future.

CONCLUSION: All identified barriers from this study were addressed with website modifications in order to maximize the usability and navigability of StrokEngine. This rigorous methodology for usability testing can be applied during the design process of any e-learning resource.

Key words: evidence-based practice; stroke; rehabilitation; e-learning; continuing education; methodology.

J Rehabil Med 2012; 44: 338–345

Correspondence address: Anita Menon, School of Physical and Occupational Therapy, McGill University, 3630 Promenade Sir William Osler, Montreal, Quebec H3G 1Y5, Canada. E-mail: anita.menon@mcgill.ca

Submitted June 10, 2011; accepted December 19, 2011

BACKGROUND AND OBJECTIVES

E-learning enables clinicians to access the latest evidence, while giving them the freedom to reflect on and then revisit the content as necessary (1, 2). It can minimize the time required to access evidence, thus making clinicians more likely to seek and incorporate this evidence into clinical decision-making (3). Given that the lack of protected time to search/appraise the literature has been identified as the largest barrier impeding knowledge uptake by clinicians (3–5), e-learning has become a preferred (6) and effective (7) format for continuing education. Concomitantly, because there is support for e-learning (6–8), it is important to understand the methods by which an e-learning resource can be rigorously evaluated for its usability, navigability and clinical relevance.

Usability testing is a systematic process that evaluates the ease with which a clinician can use an e-learning resource to achieve their educational goals (9). Kushniruk et al. (8–12) and Patel et al. (13) have developed an evidence-based approach to usability testing, which has been widely used and recognized as a rigorous methodology (14–21). This methodology includes: 1) identifying clear objectives for usability testing; 2) eliciting feedback from participants that represent typical resource users; 3) selecting tasks and contexts used during usability testing that reflect real-life application of the resource; 4) collecting data that includes audio/video recording of computer screen activity, as well as the participant’s verbal feedback and physical interactions with the resource; 5) using both structured and/or unstructured tasks during testing; and, 6) translating the usability testing results into recommendations and/or modifications. Another important consideration when testing usability is iterative systems analysis, which involves evaluating the e-learning resource during the design phase followed by cycles of redesign and repeat usability testing (8–13).

Two interactive e-learning resources have been developed by a team of international stroke rehabilitation experts (researchers, clinicians, and decision makers), in response to a Canada-wide survey indicating that best practices are not being routinely used by stroke rehabilitation clinicians (22–26), and because adherence to stroke rehabilitation guidelines translates into improved patient outcomes (4). StrokEngine (www.strokengine.ca) provides clinicians with evidence on the effectiveness of stroke-specific rehabilitation interventions while StrokEngine-Assess summarizes the psychometric properties of standardized assessments relevant for use in a stroke clientele (27–29). These resources provide knowledge at different levels of intensity including best practice summaries that can be reviewed within seconds (Quick Review) and in-depth systematic reviews (In-depth Review). Content for these resources are based on a comprehensive, systematic review of the most recent literature and national/international best practice guidelines for stroke care. The primary goal of these resources is to facilitate knowledge uptake of best practices by stroke rehabilitation clinicians to ultimately improve patient outcomes. This resource has received international recognition from stakeholders including students, clinicians, researchers, and policy makers, those with stroke and their family/friends. This resource has also been recognized by the Canadian Cochrane Center (30) for its scientific rigor and is directly linked to its website. The term “StrokEngine” will be used hereafter when referring to both StrokEngine and StrokEngine-Assess.

The first iteration of usability testing of StrokEngine was completed during its design phase on a purposive sample of 19 clinicians working in stroke rehabilitation (27). Using a preliminary prototype of StrokEngine created through a focus group of stroke rehabilitation experts (27–29), clinicians rated usability and navigability of 3 search strategies that they were instructed to use to find information on stroke-specific treatments. The following were the three search strategies: performing an unstructured Internet search; searching a general stroke site that was not rehabilitation-specific using the URL provided; and searching StrokEngine. Clinicians consistently scored StrokEngine higher in terms of usability and navigability as compared to unstructured Internet searching or searching a general stroke site, based on results from their self-administered questionnaires (27). Recommendations made during this first usability testing were incorporated when designing the subsequent modules and since that time, numerous new features and changes have been made to StrokEngine based on feedback from the international clinical, academic and research community.

To continue this iterative process of insuring that StrokEngine has maximal usability and navigability, it was deemed important to retest the usability of StrokEngine with frontline rehabilitation clinicians. This paper describes a rigorous, evidence-based methodology to usability testing that can be applied to any health-related e-learning resource (8–21).

MATERIALS AND METHODS

Study population and recruitment

Given that the content on StrokEngine is primarily aimed at stroke rehabilitation clinicians, consisting of occupational therapists and physical therapists, the goal was to conduct usability testing with these end-users. Thus, purposive sampling was used to identify occupational and physical therapists working in stroke rehabilitation in Ontario, Canada. The College of Occupational Therapists of Ontario and College of Physiotherapists of Ontario provided contact details for clinicians working in adult neurology. Clinicians were contacted at work. Inclusion criteria included those working with a stroke clientele in an acute care hospital or in/out-patient rehabilitation site, their work site within 20 km of the testing laboratory; and, varying levels of comfort with searching the Internet. Excluded were individuals with no computer skills and those enrolled in other research studies involving StrokEngine. Ethics approval was attained from St. Michael’s Hospital and the University of Toronto, Ontario, Canada.

Sample size considerations

Eight to 10 participants is considered sufficient for identifying factors that facilitate or hinder clinicians’ use of web-based resources, and it has been found that 4 to 5 participants generally identify 80% of a website’s usability issues (31, 32). To ensure that saturation would occur across most major usability issues, and, to account for potential “no-shows”, 14 clinicians were recruited.

Procedures

Usability testing procedures were based on the framework developed by Kushniruk et al. (8–12) and Patel et al. (13), which is described in the background section of this paper. Consenting clinicians participated in an individual 1.5 h, in-person testing session, which was facilitated by a one of the two highly trained research assistants. Each clinician completed a form eliciting socio-demographic information (e.g. age, discipline, clinical experience, time spent on continuing education, access to Internet at work/home, etc.).

At the beginning of the session, the clinician was asked to read a vignette representing a typical patient seen in the clinician’s work environment. Three validated vignettes (22–26) were available to choose from – one describing a typical patient with stroke admitted to an acute care hospital, one receiving in-patient rehabilitation, and, another receiving out-patient services (see Appendix 1 for vignettes). Each vignette included salient information regarding the patient’s stroke sequelae (e.g. presence of unilateral spatial neglect, impaired balance and motor function, difficulty with ambulation, and decreased strength).

Next, the clinician was instructed to search StrokEngine for information that would help answer five clinical questions specific to the patient depicted in the vignette (Appendix 2 for questions). For example, if the question asked about the effectiveness of acupuncture in the acute phase post-stroke, the clinician would have to search StrokEngine for information to answer the question. The participant was reassured that the accuracy of their answers was not important; rather, their search patterns (i.e. sequential order of web pages visited), and time required to find the correct web pages for answering questions, as well as their physical interactions with the resource were of interest.

While searching for information, the clinician was observed by a trained research assistant. The clinician was asked to “talk out loud” (13) and provide feedback about features of the website that facilitated or hindered their ability to search for information. S/he was instructed to share their likes and dislikes about the screen format, layout and organization of information, consistency of operations, and ease of navigation for the homepage and modules. This “talk out loud” methodology facilitates a systematic assessment of usability by capturing the individual’s ongoing thought processes during task performance, and their reasoning/problem-solving skills while using a website (8, 33–35). If the participant was silent for more than 3 consecutive min, the research assistant used verbal cues (i.e. “remember to keep talking”) to encourage the clinician to verbalize his/her thoughts.

Measures of usability

Clinicians’ comments about the features of StrokEngine that facilitated or hindered their ability to search for information, along with their responses to the 5 clinical questions, were audio-recorded. For each clinical question, a defined search pattern was established which consisted of a sequential series of web pages visited in order to retrieve the answer. For example, when searching for information on the effectiveness of acupuncture for improving motor function, the clinician was expected to: 1) select “acupuncture” from the main homepage; 2) select Clinician Quick Review or Clinician In-depth Review within the acupuncture module; and, 3) find information specific to the outcome “motor function”. The clinician’s search patterns for finding information to answer each question were observed and recorded on a standard observation sheet. Any deviations from the defined search patterns, along with any physical signs of frustration/confusion while searching (e.g. sighs, frowning, grimacing, fidgeting, hand gestures, looking away from screen, etc.), were documented as observed navigational errors. Any verbal cues provided by the research assistant during the searches were also recorded, along with specific features of StrokEngine that produced the navigational errors (e.g. layout and organization of information, consistency of operations, etc.). The time required to find information for answering each of the 5 clinical questions was noted.

Given that some clinicians may not have had the opportunity to provide all of their comments during the searches and that some may have felt more comfortable providing their comments in writing, additional feedback was captured using a self-administered questionnaire (see Appendix 3). The questionnaire consisted of open and closed-ended questions that were related to: 1) screen format, layout and organization of information, and ease of navigation for the homepage and modules (4 Likert-type questions); 2) their general opinion (i.e. likes, dislikes and areas for improvement) (3 open-ended questions); and 3) their likelihood of using StrokEngine in the future (1 Likert-type question).

This mixed methods approach for data collection has been validated through extensive research by Kushniruk and colleagues (8–12). The main benefit of this rigorous methodology is that it permits the collection of rich data using multiple forms of data collection (i.e. audio recording of verbal comments; direct observation of physical interactions; self-administered questionnaires). Information can be captured in different ways depending on how a clinician prefers to communicate: some may be willing to spontaneously share their verbal feedback while interacting with the website, whereas others may prefer to provide their written feedback on a self-administered questionnaire after some reflection (15, 36).

Data analyses

Clinicians’ socio-demographic data was described using proportions for categorical variables. Clinicians’ verbal comments about the features of StrokEngine were transcribed verbatim. Content analysis was then used to identify dominant themes from clinicians’ verbal comments, their observed navigational errors (i.e. deviations from defined search patterns, physical signs of frustration/confusion), and their responses to the open-ended questions of the feedback questionnaire, and categorize them as factors that facilitated or hindered StrokEngine use (37, 38). The identification and categorization of dominant themes/factors was conducted by the principal investigator and then verified by one of the project team members. Specifically, the data were analyzed line by line and sentences that represented the same theme were grouped together and categorized as a factor that facilitated/hindered StrokEngine use. For example, if a clinician verbally commented that the “font style for the homepage is clear and easy to read”, this comment was associated with the theme “screen format” and was categorized as a factor that facilitated StrokEngine use. If a clinician deviated from the defined search pattern within a module and required cueing from the trained research assistant, this observed navigational error was associated with the theme “module layout” and was categorized as a factor that hindered use. If a clinician frowned while verbally stating “the link for StrokEngine-Assess is difficult to find on the homepage”, this observation was associated with the theme “homepage layout” and categorized as a factor that hindered use. Finally, if a clinician responded in their questionnaire that “it becomes easier to find information over time”, this written comment was associated with the theme “consistency of operations” and categorized as a factor that facilitated use.

Responses to the 10 Likert-type questions of the feedback questionnaire were described using proportions. The mean and median time required to respond to each clinical question was calculated.

RESULTS

Study demographics

Of 35 clinicians contacted, 21 did not work with a stroke population. The other 14 clinicians (13 female, 1 male) met eligibility criteria, and agreed to participate. While the majority were less than 35 years of age (n = 7 had 4–10 years, and 3 had greater than 10 years of experience. At baseline clinicians ranged in their comfort with searching the Internet from somewhat comfortable (n = 4), very comfortable (n = 5), to extremely comfortable (n =5). All participants had access to Internet at home and at work. The average time spent on stroke-related continuing education per month ranged widely with some indicating they spent less than 2 h (n = 4); others between 4–6 h (n = 4), 7–10 h (n = 4) and, greater than 15 h (n = 2). The majority perceived that their worksites were supportive of their continuing education activities (i.e. provided funds and protected work time) (n = 10). Eight clinicians worked in an acute care hospital, 5 in inpatient rehabilitation, and 1 in outpatient rehabilitation. Most had no prior experience with StrokEngine (n = 10). All identified their worksite as a teaching institution (i.e. hosts students for their clinical training).

Facilitators to use of StrokEngine: Major themes identified

Five key themes that facilitated StrokEngine use were identified and related to screen format, layout/organization of information on the homepages and within modules, ease of navigation, quality of content, and likelihood of using StrokEngine in the future. The following are details regarding the key themes identified:

a) Screen format

Ten clinicians mentioned in the questionnaire that the graphics used for the homepage and modules of StrokEngine were clear and visually appealing, such that they were very or extremely satisfied with the visual presentation of these sites. Eight clinicians verbally stated that “I like those blue buttons.they make it easier to browse between the different sections of a module”. Seven verbally commented that the topic headings listed on the homepage were “really easy to spot and select”. However, one clinician was only somewhat satisfied with the screen format of the homepages for both sites, verbally stating that “they appear slightly busy”.

b) Layout and organization of information on homepages and modules

The majority of clinicians (n = 11) pointed out verbally and on the feedback questionnaire that they were very or extremely satisfied with the overall organization and layout of information on the StrokEngine homepages and modules, verbally stating that “I like how the clinician info and patient/family info are organized separately”, “it’s nice to see how information is categorized according to the patient’s stage of stroke- acute, sub-acute”, “good layout. The site is consistent and organized for efficient searching”. Three verbally commented that they were “happy to see when modules were last updated” on the homepage (n = 3), thus speaking to the credibility of these e-learning resources. Two mentioned in the questionnaire that the categorization of tools on StrokEngine-Assess according to domain type (i.e. tools measuring cognition, balance, mobility etc.) was helpful for their searches.

c) Ease of navigation

The consistency of operations made it easy to get quickly familiarized with StrokEngine and where to search for information. One clinician verbally commented that “even for an individual with limited computer skills, the various links are easy to find on both sites”.

The time required to retrieve the answer to Question 1 (related to the benefits of acupuncture post-stroke, Appendix 1) ranged from 25 s to 8 min with a mean of 3.40 min (standard deviation (SD) 2.03) for the group as a whole. The mean time to retrieve answers decreased for the subsequent 4 questions: 2.84 min (SD 2.73); 2.73 min (SD 1.37); 1.63 min (SD 0.86); 1.24 min (SD 0.97). The median time to retrieve answers for Question 1 to 5 was 3.28 min, 2.11 min, 1.95 min, 1.25 min, and 1.01 min respectively. Almost all clinicians found it very or extremely easy to search for information on a given topic or intervention, as well as browse between different topics (n = 13), verbally stating that “it is easy to search for information once you get familiar with the site”, “StrokEngine is very user-friendly”, and “having the topics in alphabetical order makes it easier for searching”.

d) Quality of the content

The open-ended feedback indicated that all clinicians had positive comments about the quality and breadth of content on both sites and the clinical relevance, stating that “the amount of information available on this website is excellent”. Ten clinicians reported on the questionnaire that they were impressed with the number of stroke-related assessments and interventions reviewed, along with appreciation for the content that they would find difficult to retrieve and synthesize themselves (e.g. psychometric properties of tools, treatment effectiveness, patient information in lay terms, etc.). Four clinicians mentioned on the questionnaire that they “really like how you can view and download a copy of an assessment tool for free” from StrokEngine-Assess. A few verbally commented that “the Quick Review section is a great reference point, but if clinicians want to learn more, the In-Depth Review section provides them with all of the details... but in a busy clinic, I think clinicians are probably going to use the Quick Review”. Six clinicians verbally stated that the printable information for patients and families “is convenient, easy to understand and has lots of interesting pictures- great for patient education”.

e) Likelihood of using StrokEngine in the future

All participants were very or extremely likely to use this website in the future to guide clinical decision-making, commenting verbally and in the questionnaire that “I will definitely use it (StrokEngine), especially if it is free and available on the net”, “I will bring this website to my colleagues... it will also be useful for my students”, “it will help me make decisions about which assessments or treatments I should use with my patients”, “Internet-based learning is fantastic and very practical”, and “too bad I didn’t know about it sooner”.

Barriers to use of StrokEngine: Major themes identified

Three key themes that hindered StrokEngine use were identified and related to screen format, layout/organization of information within a module, and system dysfunctions. A detailed list of all barriers is presented in Table I. The following are details regarding the key themes identified:

a) Screen format

Five clinicians verbally stated that the link to navigate between the two e-learning resources- specifically the intervention site and the assessment site- was not clearly visible: “The link to go to StrokEngine-Assess from the StrokEngine homepage is hard to spot.maybe this link could be larger in size and brighter in color to draw your eye to it” and “It is a bit confusing to go between StrokEngine and StrokEngine-Assess... it’s hard to find the link.”

b) Layout and organization of information within a module

Nine clinicians commented verbally or on their questionnaire that the organization of information was slightly different between the In-Depth Review section and Quick Review section of the StrokEngine modules, and this difference was identified as a barrier to use. In the Quick Review section, information for each outcome is summarized in a table (www.strokengine.ca) but for the In-Depth Review section, information for each outcome appears on separate web pages, requiring clinicians to click between them to retrieve this information. A few of them verbally stated that “it’s hard to find information on a specific outcome... you need to click on a tab to select information for that outcome... this is kinda confusing”, “I may miss this information all together because it doesn’t appear all at once” and “it’s hard to figure out that you need to click a tab in order to get the next level of information... if the Internet service is slow, it may also increase time to get to the information”. Similarly, seven clinicians found that the content for the Clinician How-To section of the StrokEngine modules also appeared on separate web pages, requiring the clinician to click between them to retrieve information. A few of them verbally stated that “this section is confusing... I didn’t realize I have to click on this tab to get the information... I expected it to open up immediately”. While these clinicians did not have any suggestions for modification, some later reported that “after getting used to the website, I was able to find my answers better” (n = 3). They mentioned in the questionnaire that the consistency of the website made it easy to become familiarized with the site and to know where to search for information.

c) System dysfunctions

Nine clinicians tried to use the search button to retrieve information but this feature was not working properly on both resources: “It doesn’t seem to work... it keeps bringing you back to the homepage” and “I typed in prism therapy in the search box but nothing seems to come up... maybe the search box isn’t working. but I would rather use this feature than browse the site for information on prism therapy because it’s faster.” Five clinicians reported verbally and in the questionnaire that the links to stroke best practice guidelines within the Best Practices section of the StrokEngine modules were not working: “Every time I click on a link in the Best Practice section, nothing pops up... it doesn’t seem to work” and “It would be nice to have direct links to practice guidelines... these links don’t seem to take you anywhere.”

DISCUSSION

This study identified factors that facilitated or hindered stroke rehabilitation clinicians’ use of two e-learning resources (StrokEngine), as well as illustrating the application of an evidence-based methodology for usability testing. Clinicians participating in this study commented that StrokEngine provided them with the latest evidence regarding stroke-related assessment and intervention in a format that was quick and easy to review. They were satisfied with the layout and organization of content, as well as how easy it was to search for information on a given topic. The consistency of the layout for these resources made it easy to get familiarized with them and to know where to search for information. Clinicians reported on the quality of the content and clinical relevance, and all stated that they were likely to use this website in the future. Factors that hindered StrokEngine use were related to the screen format, layout/organization of information within a module, and system dysfunctions. Each identified barrier was reviewed with a web developer and necessary website modifications were completed in order to maximize usability (Table I).

This study describes a rigorous methodology and detailed process for usability testing that can be applied to any health-related e-learning resource. Use of a mixed methods approach with multiple forms of data collection (i.e. verbal feedback, written responses to Likert-type and open-ended questions, observation of physical interactions) proved to be extremely valuable for this study because we were able to generate a comprehensive list of themes, as well as identify which key themes were repeated across the various data sources. To illustrate the advantage of having multiple data sources, one clinician had no navigational errors or signs of confusion during her searches, but she verbally stated “how do I get to StrokEngine-Assess from here... can’t seem to see the link.... oh, there it is” during her testing session but failed to report this barrier in her feedback questionnaire. To illustrate how key themes were repeated across multiple data sources, one clinician not only mentioned verbally and in the feedback questionnaire that information was organized differently for In-Depth Review section and Quick Review section of the StrokEngine modules, she also made a navigational error because of this difference during her searches. By using multiple sources in order to identify barriers for StrokEngine use, we were able to capture more of the needed changes and thus make relevant website modifications and tailor these e-learning resources for maximal usability and navigability.

Early and ongoing usability testing with an understanding of the clinicians’ needs during the design process is essential to creating a sustainable and user-friendly e-learning resource (17). One time testing is insufficient. Given that “big problems often mask small problems”, iterative testing enables developers to employ user feedback to incrementally change product design. Indeed, our experience was that this second round of usability testing for StrokEngine highlighted important barriers that may have been “masked” during the first round of testing. For example, during our second round of testing, clinicians reported that the link to navigate between the two e-learning resources– specifically the intervention site and the assessment site– was not clearly visible. This barrier may have not been detected during the first round of usability testing because there were only two modules (i.e. acupuncture and unilateral spatial neglect) for the clinicians to browse on the StrokEngine prototype, whereas for this second round, clinicians were free to browse between the 36 intervention modules and 64 assessment modules of two e-learning resources. As such, we were able to comprehensively evaluate the ease with which a clinician can go back and forth between the two e-learning resources, between different modules within a given resource, and between different sections within a given module.

Ultimately, the efforts of iterative testing of these two e-learning resources allowed us to elicit clinician feedback and subsequently redesign components of the sites that will maximize the potential of StrokEngine as an important knowledge resource.

Limitations

The most important limitation of this usability testing was that the study population, consisting of occupational and physical therapists working in stroke rehabilitation, was not representative of all potential StrokEngine users (i.e. other health professionals working with a stroke clientele, such as physicians, nurses, speech therapists, etc.). Given that this study was conducted as a preamble to a larger study examining the effectiveness of StrokEngine for improving knowledge acquired by occupational therapists and physical therapists regarding stroke rehabilitation best practices, it was necessary to accrue a similar target population for usability testing of StrokEngine so that the study findings are representative. While the study sample was predominantly female, this gender imbalance is quite reflective of the population of practicing clinicians: according to 2010 membership statistics in Ontario, almost 90% of occupational and physical therapists are female (41, 42).

Conclusion

Researchers and clinicians have a responsibility to work together to develop effective tools for disseminating research evidence in a user-friendly format. This paper illustrates a systematic methodology for usability testing of health-related e-learning resources in order to optimize their KT potential.

Acknowledgements

Menon was funded by a doctoral award from the CIHR program (Innovations in Patient Safety and Knowledge Translation) at the Faculty of Medicine, University of Toronto. Korner-Bitensky was funded by a senior career award from the Fond de la recherche en santé du Québec. Straus was funded by a Canada Research Chair Award in Knowledge Translation. This project was primarily funded by a CIHR Knowledge to Action Operating Grant, with additional funds from the Canadian Stroke Network, the Réseau provincial de recherche en adaptation réadaptation, and the Centre de recherche interdisciplinaire en réadaptation du Montréal Métropolitain. We acknowledge the support of our dedicated team of research assistants including Christine Marquez, David Newton, and Gail Klein.

REFERENCES