UX-3834, Analyze Special Collections Tree Test Results
Purpose: Assess usability and clarity of proposed Special Collections website reorganization
Stakeholders: Special Collections users and staff
Testing date: 8/27/18
Participants: 36 UVA undergraduates
Enrolled UVA undergraduates were solicited by email from our queue of volunteers to take a 10 minute test for which they received a $10 deposit to their UVA debit card. An online tree test tool by Optimal Workshop was employed to help determine the efficacy of the proposed information structure. Based on previous testing, we created a simple hierarchy of 4 top categories, each of which had between two and six subcategories (see fig. 2, Tree Structure). Undergraduate testers were asked to complete ten tasks designed to assess how closely the information structure matched their mental models (see Full Test Results, Analysis tab). The tree test results indicate how many testers found the correct path on their first try, how many found on their second or third try, and how many failed to find the correct path; as well as how fast the tester completed the task. For an overview of how to analyze tree test results, visit Atlassian’s Tree Testing for Websites.
Summary of preliminary findings
- Overall success rate was 70%. Even without visual cues, navigation, and menus, participants are finding the correct answer at least two-thirds of the time, suggesting the tree is effective but may need some tweaking.
- Overall directness (going directly to the correct area) was 48%. Some participants are struggling to find the right path but get there eventually.
- Participants usually identified what they would find under the About and Donate categories. The Destination tab on the full results site shows that large numbers of participants (the green boxes) correctly selected these categories, and few incorrectly selected these categories (the orange, red, and white boxes).
- The Donate link is readily found and understood. 31 participants found the correct policy under Donate, and the average time to complete the task (5.52 seconds) was half the average time taken to complete the other 9 tasks.
- Almost half of the participants selected Class visits and instruction to complete a task about finding a map, suggesting that we should duplicate some information about planning a visit in this category.
- The Destination tab shows us that Online reference request was incorrectly selected 16 times (although it is not necessarily a bad thing that students opt to ask for online help when they don’t know an answer).
- Participants were able to correlate the term “artifacts on display” with Collections and Exhibitions (task 5).
- Participants were able to correlate the term “give” with Donate (task 6).
- Participants found the digital camera policy under Using Special Collections/Usage Policies and correctly interpreted that this policy would probably cover taking photos with a phone.
Deeper dive: task #7: “Can you confirm that there is a substantial number of William Faulkner papers in Special Collections?”
Roughly half of the participants found the correct category, which is several levels down in the hierarchy. However, first-click data shows that an additional 34% clicked on Collections and Exhibitions but didn’t investigate Featured Collections, where they would have found the Faulkner papers. Most wound up in either Manuscript Collections or UVA Archives, so were circling in the right neighborhood but didn’t ring the right doorbell.
- Class visits and instruction seemed to resonate with undergraduates, so consider building up this popular area with duplicate links from Plan a visit.
- Online reference request is used to answer a wide spectrum of questions, so give this a high profile (possibly as a sidebar).
- About and Donate work well as top-level categories.
- Using Special Collections is a broad category that may need further refinement through discussions with Special Collections staff and/or further user research.
- Collections and Exhibitions may not resonate with undergraduates but may work for others with more familiarity with research collections. Featured Collections should be rethought to identify a current goal and purpose, which will inform how and where to present it, and may inform Collections and Exhibitions.
The UX team will factor in these recommendations to rework the information architecture.