The Usability Study of HRC usability test
The usability evaluation of HRC Digital Collection Page
The Harry Ransom Center (HRC) is an internationally renowned humanities research library and museum at The University of Texas at Austin. Its extensive digital collections provide unique insight into the creative process of writers and artists, deepening understanding and appreciation of literature, photography, film, art, and the performing arts. Public programs, film series, poetry readings and a lot of modes of access to materials are provided for not only scholars but also anyone. Due to its significance and popularity for scholars and students, the HRC staff expects to offer easy access and use for visitors. Improving web page usability would benefit students and professional researchers alike.
Our client, Jullianne Ballou, Digital Archive Project Librarian, is looking for an evaluation of the Harry Ransom Center’s online digital collections page. HRC uses CONTENTdm, a digital collections management tool to build its online collections. After listening to complaints of faculties and students that they have trouble searching target digital file, librarians of HRC begin to value this problem. They admit that they have never done something related to usability but it is necessary to provide easy access to the users. Ballou would like to find out how they can improve the parts of the digital collections page, becoming a top competitor with other institutions.
Therefore, our purpose is to implement an evaluation of the website of HRC to verify how useful and easy it is for students and scholars to conduct their research as well as provide suggestions on how to improve the page.
About
What We're Testing
Business Questions
Users
We are testing the usability of the Harry Ransom Center’s digital collections page.
-
Are users able to find the collections they need on this site?
-
Will an improved structure generate higher task completion than before?
-
What functions of the digital collections are confusing for users?
Students or faculty from UT Austin who use HRC’s digital collection frequently; at least once a week. Our client informed us that these users primarily come from the English, literature, humanities, art history, theater and politics department. They have basic knowledge of how to use technology and are comfortable with using the computer.
Sample Size
We are going to recruit a mix of 8-10 students and faculty who are affiliated with UT Austin to conduct the usability test.
Users
Methods
We will use moderated in-person(8-10) to operate the test.
Tasks
Metrics
Task success
Think-Aloud Protocol
Time-on-task
Eye-tracking
Metrics
Task Metrics
These task metrics will be considered in the evaluation of the usability of the HRC’s digital collections page. These metrics are what will be focusing on during the usability test, which will point out pain points as well as strengths on the site.
Task Completion:Task completion will be measured using the binary metric. When the user completes the task, their success rate will be given a 1. If they are unable to complete the task, they will be given a 0. By using the binary metric, we will be able to quantify the success rate of task completion on the site.
Task Time：For each task, we are estimating the user will need 5 minutes. Because we are utilizing the Think-Aloud Protocol, which requires the user to vocalise their actions and thoughts, time will begin when the user expresses their thoughts upon receiving the task. We are choosing to do so because we desire to know the full process of the task time which starts with the user deciding how to approach the task.
We will not evaluate the times from failed tasks, however, they will be taken into consideration to point out where users struggled on the site.
Task Difficulty: We will use a single ease question to measure task difficulty. After the completion of each task, we will provide users with the question, “Overall, how easy or difficult was the task to complete?” with a seven point rating scale, ranging from Very Difficult to Very Easy, to provide their answer.
Confidence: During and before the test, we will keep in mind that users tend to be overconfident when completing tasks. However, we will not use this measure until after the session to determine the level it may have played in the tasks.
Efficiency
Quesenberry's 5Es.
● Effective – Can I complete my goal?
● Efficient – Can I do it quickly?
● Error Tolerant – Can I do it correctly or can I easily get the help I need?
● Engaging – Do I like it?
● Easy to Learn – Can I do it correctly the first time?
We will conduct a pilot test to test the design of tasks and make some moderation according to the results.
Post-Test Questions
In addition to the test-metrics like SUS, SUPR-Q or NPS, we also like to include a few open-ended questions about what the users would improve and comments or issues they had (especially important in unmoderated studies), as well as any post-experience brand or attitudinal metrics.
The following shows several examples for post-test questions.
Operation
-
How would you rate the difficulty in finding the information you need?
-
Which function are you using or which page are you visiting the most frequently?
-
Do you think the general search function is powerful enough? helpful? Do you think the advance search function is helpful?
-
Satisfaction
-
How did the time it took to open web pages of HRC compare with what you expected?
-
How comfortable is it for you to use the website and to finish the task? Do you think the website is well organized?
-
Is there any point you are not satisfied with or think can be improved?
Evaluation Methods
The test evaluation will include a pre-test, post-task, and post-test questionnaires. The combination of quantitative and qualitative data will be compiled and analyzed and then used to develop conclusions for the Final Report.
Qualitative
-
User's comments and questions
-
User's body language and facial expressions
Quantitative
-
Time to complete task
-
Number of users who are able/unable to complete the task
-
Number of attempts for the task
-
Number of errors occurred during the task
-
Data collected from pre-test questionnaire
-
Self-reported metrics results from post-task, and post-test questionnaire
For the qualitative data collected, I will extract keywords that reflect user’s behaviors and opinions. I will code the common points mentioned by more than one person and record the times mentioned. For the quantitative data, I will use mean, median, modes, etc. to reflect the features of the results.
Appendix: Moderator Scripts
Hi, XXX. It is good to see you. I will walk you through this session today. Before we begin the test, I will read you something we ask people to finish some tasks, so we can see whether the form works well. This session will be about 5 minutes.
The first thing I want to make clear is that we testing the form not you. You can do anything wrong here. You do not have to worry about it. As you look at the form, I will ask you as much as possible to try to think-aloud to see what you are looking at, what you are thinking, what you are trying to do. It will be a good help to us. Please do not feel that it will hurt our feelings. We do this test to improve our web form. With your permission, we are going to record what we see and the conversation. Recording will be only used to improve the web form and be seen by people who work on this project.
If you want I would love to ask you to sign a single permission form. It just says we have the permission to record you. Recording will be only used to improve the web form and be seen by people who work on this project.
Before you work with the form I would like to ask you some simple questions:
What do you do all day?
How many times do you using internet all week, including searching the website and sending emails?
In what degree is your English?
Next I will bring up the form you are going to work on.
Please perform the tasks as quickly and accurate as possible. (If they asks, state that I am noting the start and finish time for each task)
Group Member: Bria Fowler, Xi Chen and Xiaomeng Mu