Usability Test Procedures
for the U.Va. Library
|
Intro | Heuristics | Card Sorts | Test Development | Testing Process | Follow-up
Introduction: The Place of Usability Testing in UVA Library Site Development
The goal of usability testing is to identify potential difficulties that users may have in accomplishing their information gathering goals using UVA Library web pages and sites. A web site is usable if it assists users in performing their tasks easily and quickly; if it reflects an understanding of users’ goals and tasks; and if the “look and feel” (navigation and design) are consistent and predictable. No matter how well a site owner understands the realm of information to be conveyed in a web site, the ability to design a successful web experience is not a given; site owners and designers will learn important things from any testing that is done.
Any new or revised UVA Library web site, excluding Library exhibition web sites, will undergo usability testing as a part of its implementation. Once a site is complete, the site will be tested. A report of the test, including recommendations for changes, will be submitted to the site owner. After any changes have been made, the site will be re-tested to see if the usability of the site has been improved.
The UVA Library has a Usability Group that coordinates such testing. This group identifies the appropriate type of test – Heuristic testing, Card Sorts, Full Usability testing – and oversees the testing process.
Heuristic Usability Testing
Heuristic testing involves Web experts comparing a site to a set of 10 principles established for a well-running and well-designed site. The heuristic principles are grouped into three categories: information structure and navigation (aka, “Information Architecture”), content and design, and issues specific to search forms and data manipulation.
The Library's Heuristic Principles used in its heuristic testing are available at http://staff.lib.virginia.edu/usability/heuristics.html.
If a site is deemed to require heuristic testing, these tests are undertaken by the Usability Group. At least three members of the Usability group will review the site. The tests will be summarized and reported upon to the site owner. In some cases, only heuristic testing is done. In other cases, heuristic testing is combined with full usability testing.
Card Sorts
A Card Sort is a process through which the appropriateness of the organizing categories of a site are tested. The goal is to understand the mental models of users – what the organizing principle of a site should be, how information should be categorized and grouped, and how categories should be labeled. The results of a Card Sort test generate suggestions for the information architecture of a site, as well as suggesting labeling for navigation and menus.
The UVa Library process is a Closed Card Sort. Groups of no more than 6-8 participants are given stacks of cards, one for each of the existing or suggested set of organizing categories and each single site page. The group is asked to organize cards representing pages under the categories. We break slightly from the usual practice by additionally allowing users to suggest new categories or new wording for category labeling. One or more groups may be tested, representing multiple user constituencies.
This procedure is most often part of the review of an already existing Library site to suggest revisions to a site’s structure and labeling. At least one member of the Usability Group will oversee each Card Sort test. The tests will be summarized and reported upon to the site owner.
Development of a Usability Test
A coordinator for each site test will be named from the members of the usability group. This person will lead the process of site development, testing, and follow-up.
The process of designing a usability test starts with the site owner filling out a Site Owner questionnaire that is available as a Word document at http://staff.lib.virginia.edu/usability/SiteOwnerQuestionnaire.doc. The site owner's answers about the mission, audience, and functions of the site should help the test author create the questions and identify the groups of users that should be included in the testing.
A full usability test is one that sets a series of tasks that a test participant should be able to perform using the site that is being tested. The test should consist of 10-15 tasks/questions, or no more than can be tested in 1 hour. Smaller sites do sometimes have fewer questions.
The groups of users to be tested should reflect the constituency of users identified that use or will use the site. This may include undergraduates, graduate students, faculty, Library staff (sometimes separated into Classified and Faculty groups), members of the general public, or staff at other Libraries. The minimum number of people to test for each group is three. No more than five of any category is needed. For most Library sites, we test with three each of undergraduates, graduates, faculty, and Library staff.
The questions should require the tester to answer a question or perform a task without cluing them in that something definitely is part of the site. The questions must avoid the exact language that is used on the site to label the feature or function. The questions should be written as "Is there?" or "Can you?" questions, not "Where is x in the site?". In some tests, different question sets may be used for different test groups, if different site features or functionality are available for each group. An example is a test of the ILS web site, where a feature such as LEO is not available to students. Examples of past tests are available on the Usability web site. Examples of past tests are available at http://staff.lib.virginia.edu/usability/tests/.
Once the test has been developed, reviewed, and approved by the Usability team, a testing log is created. The template for creating a log is available at http://staff.lib.virginia.edu/usability/tests/TestLog.doc.
Once testing has been performed, recommendations are written up. The recommendations are presented to the site owner. After any or all of the recommendations have been acted upon, the site is re-tested, using the same questions, to see if the usability of the site has been improved. Improvement may be measured in a higher number of correct answers found on the first try by a tester, and in improved speed in finding the correct answers. As of 2005-2006, Metric I.2.b of the Library's Balanced Scorecard spells out the details of a metric to measure improved usability of a site through testing; see http://staff.lib.virginia.edu/usability/metric.html.
In some cases, full usability testing is not performed. Heuristic testing is a review of a site for consistency in navigation and site labeling, looking for comprehensible site architecture and labeling, and seeking out any issues with the site, such as broken links. Only the Usability team performs this testing. Details and forms are available at http://staff.lib.virginia.edu/usability/heuristics.html.
The Testing Process
A proper usability test requires a facilitator, who does the talking and asks the questions; an observer who records what is said and done; and a participant, who is a typical user of the web site being tested.
The facilitator must not be the designer, or an advocate, of the web site under review. Both the facilitator and observer should maintain neutrality and objectivity during the session. The participant is usually a student, staff or faculty member, it generally should not be an “insider” who has extensive knowledge of the workings of the organization.
Preparations
The facilitator brings to the test:
- Usability test procedures (available as a one-page cheat sheet at http://staff.lib.virginia.edu/usability/tests/TestCheatSheet.doc)
- The set of questions (tasks) for the particular test
- Logging sheet for the observer (Created by the test author, this includes the questions, space to enter the observations, and space to enter the time elapsed for each task. Participant names should not be recorded. A Word template is available at http://staff.lib.virginia.edu/usability/tests/TestLog.doc)
- Information on incentives for the participant
Before the participant arrives, the facilitator should:
- Arrange the physical space so the participant will sit comfortably in front of the computer screen, and in a position where s/he can easily speak to the facilitator. The facilitator and the observer should sit where they can easily see the screen.
- Clear the cache and history of the browser so visited links are not visible.
- Display the site being tested.
Greeting the Participant
When the participant arrives, the facilitator should:
- Introduce oneself to the participant. State your name and position within the library. Introduce the observer. Tell the participant that the observer will be recording what is said and done; this is a standard feature of usability testing.
- Make the participant feel welcome and comfortable; we want this be a pleasant experience. Thank the participant for agreeing to help us. Tell him/her about how long it will take—usually 45 to 60 minutes.
- Explain the purpose of the test: e.g., “We are NOT testing your skills, but rather the layout, design, and content of a particular web resource. Your feedback will be very helpful to us as we work to improve library services.”
- Describe the open-ended nature of the test: “We will ask you to do some specific tasks, but there are no right or wrong ways to perform the tasks. If you find that you cannot perform a task, it is all right to say so.” Stress that there may be multiple ways to answer the questions, and that no pathways are wrong.
- Encourage talking—thinking out loud. “It will be very helpful if you share your thoughts with us. As you do these tasks, we want you to think out loud. Let us know what you are thinking. It may seem unnatural, but it is very helpful to us.”
- Ask the participant if s/he has questions or comments.
- Ask the participant to sit down in front of the computer. Ask if s/he is comfortable. Let him/her look at the site for 30 seconds to one minute.
The Actual Test
- Ask the questions one by one. Say it the same way for each participant. Don’t show him/her the list of all questions; we don’t want anyone to feel overwhelmed, or to be distracted by thinking of tasks to come.
- Record the time each task took in 15 second intervals, e.g., :15, :45, 2:30, etc.
- Try to stimulate talking. As needed, give reminders to think out loud.
- Listen. Let the participant talk.
- Don’t offer help or corrections. Make notes of problems during the test, and follow up during the debriefing. Don’t stop during the actual test.
- If the participant reveals some misunderstanding of the web site or the library, do not correct him/her during the test. This is not an instruction session.
- Do not defend the site; remain neutral and objective.
- Do not lead the participant. Don’t say, “That was difficult, wasn’t it?”
- If feedback is given, make it consistent for all tasks, not just for those that are “right” or “successful.” For example, if you say “ok” after some tasks, say it for all tasks.
- If a participant asks for guidance, try to deflect the request. For example, if you are asked, “Should I click here?” Turn it back with another question: ”What do you think will happen if you click there?”
- Do not smile or nod at the observer; no inside jokes or communication.
- Prompt only when the participant is not talking. Do not rush if s/he seems to be thinking about the task. Gentle prompts include: “What are you thinking now?” “What link did you expect to see?” “What are you looking for?”
Conclusion and Debriefing
After the last question, tell the participant the test is over, and s/he did a great job. Thank them again, and explain that you want to chat a bit and get his or her opinions on the site and the test.
Questions you might ask (all are not necessary, and others might be applicable):
- What did you think of the site? What did you like? What did you not like?
- What did you expect to find, but didn’t?
- Did the site meet your expectations?
- What do you think this site is for?
- How might you use the site?
- Did you understand the terminology used on this site?
- If you had three things to tell the site manager, what would they be?
Go over any problems with the site, replay things that seemed difficult. Ask any follow-up questions
If the participant has revealed any misperceptions about the site or the library, feel free to correct them at this point.
Finally, ask “Anything else?” and allow them to share any final thoughts.
If the participant is to receive a gift or incentive, explain how to claim it.
After the Test
The results of the test must be transcribed into an electronic version of the testing log. Individual tester names need not be included. Documents must have unique and representative names so we can avoid overwriting existing files. Please use the following formulas to generate document and link names for user test results, summaries of a series of tests, and heuristic analyses.
User Test Results
ut_results_[site name]_[numerical date of test][initials of administrator].doc
Example: ut_results_central_01802ch.doc (results of user test on the Central site administered by Carol Hunter on Jan. 8, 2002)
Summaries of a series of User Tests
ut_summary_[site name].doc
Example: ut_summary_central.doc (summary of series of user tests on the Central site)
Heuristic Analyses
ha_[site name]_[numerical date of document][initials of administrator or tester].doc
Example: ha_redesign-pdf_112801fw.doc (results of heuristic analyses of the Site Redesigns (using the PDF), conducted by Frank Wilmot on Nov. 28, 2001)
Test results are uploaded to the Usability web site through the protected upload page. Links to the documents must have meaningful names that are related to the names of the files. Access to uploaded results is restricted.
Link to User Test Results
[site name], [date] ([last name of administrator])
Example: Central Web, Jan 8 2001 (Hunter)
Link to Summaries of a series of User Tests
Summary, [site name], [date of document] ([last name of administrator])
Example : Summary: Central Web, Jan 28 2002 (Whiteside)
Link to Heuristic Analyses
Heuristic Analysis, [site name], [date of document] ([last name of administrator])
Example : Heuristic Analysis, Site Redesigns (PDF), Nov. 28 2001 (Wilmot)
After testing has been completed on a site, a summary document will be created by the test coordinator for each category of user and for the testing of the site overall. Each user category summary will include basic statistics for each question ("2 of 3 testers failed to find the link"), variations in answers, and tester comments. The overall test summary will include basic statistics for the entire test, and identify trends in the pathways that users followed. Examples of summary documents are available in the restricted results area of the Usability site. The Communications department will use these summaries to write recommendations for site owners.
Follow-Up
After the recommendations report is presented to the site owner, revisions if called for, must be made to the site within one month. After the revisions are complete, the site must be re-tested to check for improvement to the site's usability.
Re-testing
When performing a follow-up test on a site, do not ask the same participants to return to re-test the site. Test the same number of participants in the same categories as in the first test. Use the same questions and the same log forms. The same rules for testing, writing up tests, uploading files, and creating summaries apply.
The last step is the comparison of the original test to the follow-up test, to see if the usability of the site has been improved. Improvement may be measured in a higher number of correct answered found on the first try by a tester, and in improved speed in finding the correct answer. This data will be used in responding to the Library Balanced Scorecard Metric for Usability testing.
This document was adapted from the usability web sites of various institutions,
including Indiana University, the Yale University Library, the California Digital Library,
and the Gelman Library of George Washington University.
University of Virginia Library
March 14, 2003
revised July 6, 2005
