Research Results: Library home page testing and surveys

User Experience Project ID: UX-1922, Library homepage development
Purpose: Assess usability and clarity of proposed website designs
Stakeholders: UVA Library staff and users

Test dates

  • 2/22/17 – 2/24/17 ; 9/14/17 – 9/19/17 ; 11/15/17 – 11/21/17 (usability testing with UVA undergraduate and graduate students)
  • 4/5/17 – 4/14/17 (online surveys of UVA students, staff, and website visitors)

Methodologies: Usability tests (3) and online surveys (3) to present iterations of Library home page designs to test usability and elicit feedback. Efforts were focused on search box location, carousel location and navigation, Hours page, Research page, mobile and tablet navigation, and link naming.

Summary of findings

  • Hours are easily found and testers liked its new prominence
  • Mobile version of Hours is easily navigated and was praised as “well-formatted”
  • Improved visibility of search, which is a primary or secondary task for website visitors
  • The “Ask a Librarian” brand is firmly established and the “Ask a subject specialist” link seems to be understood as another place to get help
  • Iterations of new design were usually viewed as more attractive than the previous versions
  • Mobile hamburger menu is more easily found when labeled “MENU”
  • The Research page redesign was described as clean, efficient, logical, and organized
  • Thumbnail images under the carousel image helped testers navigate content

Project status: Newly designed website theme rolled out 3/7/2018

Final reports (UVA only):

Research Results: Website Information Architecture testing

User Experience Project ID: UX-2846, Analyze Tree-Test Results

Purpose: Assess usability and clarity of proposed website top level reorganization
Stakeholders: UVA Library staff and users
Testing dates: 11/13/17 – 11/14/17
Participants: 29 UVA undergraduates (paid)

Enrolled undergraduates were solicited via a banner on the Library web page to take a 15 minute test. An online tree-test tool by Optimal Workshop was employed to help determine the efficacy of a proposed information structure based on previous testing results. We created a simple hierarchy of 5 top categories for the Library website:

  • About UVA Libraries
  • Using the Library
  • Help and requests
  • Research and instruction
  • Advanced technology

Each category had between three and nine subcategories. Undergraduate testers were asked to complete ten tasks, and were told it was possible to complete these tasks on the Library website. The tree-test results indicate how many testers found the correct path on their first try, how many found on their second or third try, and how many failed to find the correct path.

Summary of preliminary findings

  • Undergraduates correctly identified categories about spaces (libraries, study areas, room reservations)
  • Undergraduates correctly identified categories about searching for books and articles
  • The category Advanced Technology was not well understood
    • The task to find specialized software could only be successfully completed by selecting the “Advanced technology” category, but only 38% of testers selected the category immediately, and 41% of testers ultimately selected one of the other four categories
  • Color printers and scanners (equipment) were not easily found under Using the Library
    • 38% of testers found it immediately, an additional 28% found it eventually, and 34% did not find it at all

Deeper dive: task #8, “Find course materials on reserve for your class”

Only Task 8 Find course materials on reserve for your class12 testers (41%) navigated correctly to “Search course reserves” or “Search Virgo” in order to complete the task. Of the 17 that did not navigate correctly, 11 incorrectly looked for the answer under the “Help and requests” category (which may show a willingness to ask a librarian for help when they can’t find something on the website). Of those 17 failures, 10 selected “Research and instruction” (one of the correct categories) at some point, but still did not successfully complete the task. It took all 29 testers an average of 17.65 seconds to finish this task, which was considerably slower (by at least 8 seconds) than the time spent on the other tasks. Previous testing has indicated that some students are unfamiliar with the concept of course materials being held on reserve in a library. The high rate of failure coupled with the extended time it took to finish this task further suggests confusion with this concept.

Project status
The testing information gathered here continues to inform website organization projects. Some categories of information should be surfaced in order to be found by undergraduates, but other user groups (graduate students and faculty) also need to be tested.

Testing results:

Research Results: Graduate Students and Their Needs

User Experience Project ID: UX-2368, Facilitate two focus groups with graduate students

Purpose: Investigate sources of dissatisfaction among graduate students
Stakeholders: Academic Engagement, Administration & Planning, Collections, Access & Discovery and other Library areas
Focus group dates: 11/29/17, 11/30/17
Participants: 11 graduate students (including one session with five international graduate students)

Low ratings by a small subset of graduate students in the 2016 user survey were a cause for concern, but root causes were elusive. Two focus groups were offered to assist with identifying issues. One focus group was limited to international students to learn of their specific needs. Most students were recruited from usual avenues (web banner on the Library home page; contact cards left at print stations). Others had participated in the 2016 user survey and agreed to future participation in user research, and several were invited based on past feedback they’d given. UX research staff created two focus group protocols probing on problems and resolutions, confusion or difficulties, type of communication preferred, and how the Library can better contribute to graduate student work and to their overall positive experience. The international students were also asked about specific challenges they face.

Major findings

  • Communication and connectivity are important elements to a positive graduate student experience. Graduate students want us to market to them early and often, and want communications and workshops that are geared toward them.
  • Graduate students want document delivery service and they want us to be proactive, try harder, and care more about getting the paper and electronic materials they need.
  • Library systems and policies are confusing and frustrating, especially to incoming students. Many navigate with the help of their peers. They want DIY short online video tutorials for online tools that can be found and viewed at point-of-need, but like in-person assistance with subject specialists for in-depth research.
  • Graduate students want space for work and storage in libraries. Physical comforts impact willingness to use and stay in library spaces.
  • Students with mobility issues can be negatively impacted by collection decisions that split call number runs between multiple libraries.
  • International students matriculate with few expectations for library services beyond checking out books and comfortable study space; there were no notable differences in library experience with their peers from the U.S.


  • Create a communication plan, to include physical materials, for incoming and other graduate students to convey services, policies, and contacts.
  • Create workshops to engage graduate students in library services and research resources.
  • Create short online video tutorials for services and processes.
  • Consider providing scanning and electronic delivery of print journal articles to all graduate students.
  • Explore possibilities to expand dedicated graduate student space in all libraries.
  • Increase efforts to convey information about critical systems, services, and policies to graduate students, particularly incoming. Include basic information about online catalog, Interlibrary Loan, policies (borrowing, recall, course reserve), off campus access to online research materials, printing, obtaining a carrel, how to reserve a classroom, and stacks navigation.
  • Increase in-person communication with graduate students about subject liaison program.
  • Review tight stacks areas that may provide poor accessibility.
  • Consider revising collections policies to keep all materials in the same subject together to enhance accessibility for students with mobility issues.
  • Do further research to gather more complete data on the following:
    • assess how the Library should communicate with graduate students;
    • gain a deeper understanding of the specific needs of international students;
    • determine graduate student priorities for document delivery, paging from other libraries, and other delivery services;
    • assess what current graduate students wish they’d known as incoming students.

Final Report and protocols (UVA only):

Research Results: What Undergraduate Students Want from a Library Makerspace in Clemons

User Experience Project ID: UX-1905, Focus group of undergraduate students to get feedback on the future Library Makerspace

Purpose: Assess general themes and issues relating to creating a creative space in the Library

Stakeholders: Library Services and Spaces

Test dates: 4/7/17; 4/14/17; 4/18/17

Motion capture suit

Motion capture suit

Participants without Makerspace experience

  • 4th year, History and Religious Studies
  • 4th year, Nursing
  • 3rd year, Urban & Environmental Planning
  • 4th year, Economics

Participants with Makerspace experience

  • 3rd year, Computer Science
  • 3rd year, Drama
  • 4th year, Neuroscience & Biology
  • 2nd year, Philosophy & Middle East Studies and History
  • 3rd year, Computer Science & Studio Art
  • 1st year, Computing Engineering
  • 4th year, Computer Science


Lulzbot 3D printer

Lulzbot 3D printer

Three focus groups were held. One focus group had four undergraduates with no Makerspace experience. Two focus groups had seven undergraduates with Makerspace experience. Inexperienced participants were asked to give impressions of the terms “build,” “make,” and “create,” while experienced Makers were asked to give impressions of the term “Makerspace” and asked about past Makerspace experiences. All participants were asked what a Library creative space should help students do or accomplish; what makes for a good experience in a creative space; and the importance of whether or not such a space is student-run. All students also gave input on equipment, training, and design of the Clemons space.

Summary of findings

  • All groups express the need for a space that is welcoming to all students and the Library is uniquely positioned in students’ minds as a neutral space for exploration.
  • All student groups felt that first impressions of the space are important, and that a modern look and an open floorplan are important elements for the design of a Library creative space.
  • Perceptions of Makerspaces and of highly-technical equipment is informed more by students’ field of study than by their year of study. Those students familiar with Makerspaces tend to be in STEM fields but span all class years, suggesting Maker culture is well-integrated into those curriculums.
  • Students who have not used a Makerspace find them intimidating and unwelcoming, and they perceive that they lack the necessary skills to use Makerspaces. They are more likely to be from the Humanities and Social Sciences.
  • Too much high-end technology is “a turn-off” to inexperienced students and they
    Sewing machine

    Sewing machine

    want a creative space with “simple stuff.”

  • There was no strongly expressed opinion by major or class year that the Clemons space should be student-run.
  • Recommendations include:
    • Plan a range of outreach strategies for all UVA students
    • Design a well-lit space with comfortable, modern furniture with a variety in seating, tables, and desks to support group and individual work
    • Host inclusive design sessions with students to give them planning input
    • Consider having an ongoing steering committee that includes students to create policy and address operational questions
    • Consider means to monitor service quality and respond to issues as they arise
    • Plan a range of outreach strategies for students who may be interested in creating and making with non-technology items
    • Ensure that signage and supplemental information clearly demonstrates the equipment available, how to use it, and potential uses to reduce psychological barriers to using the tools
    • Arrange space so that both quiet exploration and collaborative ideation sessions can be supported
    • Design service points to be integrated into the space and easy to approach
    • Implement a variety of instruction programs to serve multiple learning styles
    • Specify (in print and on web) what equipment is available for use and for check-out
    • Partner with existing student clubs and organizations (technology, gaming, media…) to promote Makerspace services and instruction

Project status: Clemons Makerspace is expected to open in advance of the Fall 2017 semester.

Project Files (UVA only):

Research Results: Remote testing on home page prototype

User Experience Project ID UX-1687: Create, Launch, and Evaluate Online Test

  • Purpose: Assess new home page design by non-UVA users
  • Stakeholders: UX team
  • Test date: 12/7/16
  • Test participants: 3 online testers in the United States, age 18+, each using a desktop or laptop computer
  • Methodology: I received three credits to run a remote test with I wrote the test (see project files) and typed it into the online form. I was able to designate what device I wanted testers to use (desktop, smartphone, tablet), plus choose the testers’ country (UK, US, France, Germany, Netherlands), age range, and socioeconomic status. Testers were asked to view
    Prototype screen (partial)

    Prototype screen (partial)

    ototype/ and use it to complete six tasks and rate their ease at completing each task. Testers were also asked for their opinions about the site. The testers used screen cast software which recorded their voice and keystrokes, and they were told to think out loud as they worked through the test. Links to the resulting .mp4 files and a summary of the test were made available for download.


  • 3/3 testers had trouble locating subject specialist help from a Virgo screen. After a minute of clicking and scrolling 2/3 testers found the answer.
  • From the Library Hours page 2/3 testers clicked on Collections when asked to look for a book, possibly because there is no search box on the Hours page (see fig. 1). One tester used the search box at the top of the Collections page to search Virgo, and the second tester used a Virgo link under the heading, “Ways to Explore” on the Collections page (see fig. 2). The third tester used his browser back button until he found a search box on a library page. All testers were able to successfully complete the task.
Figure 1 Library Hours page

Figure 1 Library Hours page

Figure 2 Library Collections page

Figure 2 Library Collections page

  • 3/3 testers had some difficulty finding a library lab with reservable space, although each was eventually successful (in different ways). The first tester went to Reserve a Room under Spaces and Equipment, then clicked on Group Study Rooms (a link to the booking software). The second tester also selected Reserve a Room (but via the Services page). She scanned for the term “lab” but went back to the Services page when she didn’t find it. She then clicked on Digital Labs, Study Space Info, and again on Digital Labs before finding text about reserving rooms within the Scholars Lab. The third tester also went into Reserve a Room under Spaces and Equipment, then backed out to the main page and selected Explore Libraries and Labs, where he found text about reservable rooms in the Library Data Commons@Curry. It is reassuring that each tester found the answer, and it supports that we need to maintain some reciprocal links and different navigational paths.
  • All testers began new tasks without navigating back to the main page, which reinforces the need for good navigation and design from every page.
  • One tester confessed to being a retired librarian, so was quite familiar with terminology like “online catalog” and “journal finder”. Not my intended audience!

Project files: WhatUsersDo_Report

Research Results: Improving the UVA First-Year Student Experience

Project Details

  • User Experience Project IDs: UX-1609 and UX-1414, Plan and Conduct First-Year focus groups
  • Purpose: To understand changes in Library usage, attitudes, and perceptions over time; and to assess the feasibility of using a panel of committed First-Year students for this purpose.
  • Stakeholders:UX team, First Year and other AE Instructors
  • Dates: 9/29/16 and 11/11/16
  • Participants: 9 undergraduate students
  • Methodology: An open session was offered to library staff to generate questions for First-Year student focus groups. Participants determined categories for the questions and UX research staff created a focus group protocol of 10 questions. Seven FY students participated. Due to difficulty in recruiting enough students for a second focus group we instead interviewed two additional FY students with a protocol revised based on the results of the focus group.


  • FY students use libraries for quiet study, meeting friends, finding resources.
  • Librarians are useful for helping to navigate stacks and find books, help with printers, and make recommendations.
  • Librarians go out of their way to help. Tours with librarians have been very helpful. Some students don’t have much contact with librarians.
  • FY students believe that no one will touch their stuff in the library because of the Honor Code.
  • Early impressions of walking into a UVA library were that there were lots of study spaces and not many books in sight.
  • Navigating in the libraries is difficult for new students. It’s not clear how to look up a book or find a book in the stacks. Stacks are “confusing” and finding a book “takes way too long.”
  • Things FY students like about the Library:
    • The late/all-night hours
    • Can always find a place to study
    • Egg chairs
    • “Librarians are always very friendly and they just want to help.”
    • Meeting rooms
    • Areas with tables and white boards
    • The absolute silence of the Harry Potter room
  • FY students in their first semester have done virtually no research in the library
  • FY students trust information that comes from professors, subject experts, upperclassmen, librarians, parents, peers.
  • FY students frequently pack for the day and bring books, laptops, pencils, food/drink, chargers, and phones with them.
  • Most take handwritten notes in class because electronics are a distraction; easier to format and organize; “can’t do math problems on a laptop”; helps to focus better; “I’m thinking as I’m writing.”

Next Steps

  • Create survey themes and questions based on above findings.
    • Possible themes to include Spaces, Navigating, Searching, Help.
  • Seek IRB approval in order to retain the option to publish results.
  • Recruit First-Year students during summer orientation to create a panel of up to 10 students committed to participating for at least one semester.
  • Plan for 8-10 online surveys, and possibly an in-person interview, during the 2017-2018 school year.

First-Year Panel Project:

Project Files (UVA only):

Research Results: Perform and assess a guerrilla test on new home page

Project Details

  • User Experience Project ID: UX-1372, Perform and assess a guerrilla test
  • Purpose: Assess usability of the new home page design
  • Stakeholders: UX team
  • Test date: 8/25/16
  • Test participants:
    • 5 undergraduates
    • 1 faculty
  • Methodology: “Guerrilla” testing, in which we approached students entering Clemons Library and asked for 10 minutes of their time in exchange for a candy bar or banana. Testers were asked for first impressions and to rank the three most important links or areas on the newly-designed page. Testers were also asked to complete seven tasks on both laptop and mobile versions, and then were asked to rank the ease with which they completed those tasks.


  • Initial impressions from four testers were that the page is “easy to navigate” and “organized.” One tester who had already used the page was “frustrated by it, can’t find anything.” Other comments: like the photo banner, like the prominence of the hours and Ask A Librarian on the banner.
  • Most important links were varied: Research had three votes; two votes each for Hours, Ask A Librarian, Services, and Find Study Spaces; and one vote each for Virgo, Events, Reserve a Room, and Printers and Photocopiers.
  • When asked to find the hours of a digital media lab, four testers correctly used the Hours dropdown menu in the sticky header or the “View All Hours” link. Two navigated to the “Digital Labs” link under Services so eventually found the correct information, but with some difficulty.
  • Most testers were familiar with Ask A Librarian as a way to get help finding an article. AAL was found in the sticky header, the banner, and the footer.
  • When asked to book a room for a study group, three selected the “Reserve a Room” link under Spaces. Two navigated to Services, then to “Reserve and Rent a Room”. One tester said he always finds the link by Googling “Book It”.
  • Finding an event that occurs next month in Brown library was difficult for four testers, but three eventually found it. One gave up. Two tried to use the search box by typing in Brown, which apparently does not search location, so they were not successful using that method. Most testers easily found and used the >> on the calendar to bring up the next month; and half used the library filter to find events in Brown.
  • When asked to find expert help in Sociology, three testers went to Ask A Librarian. One found the Consultations heading on the Services page, and one thought maybe he’d use the staff directory, but commented, “I wouldn’t know why I would be on this [library] site for a sociology expert.” None of the testers found the “Subject Liaisons” link under Services.
  • Other impressions of new design: two commented that they liked the blue; and the design was called “nice,” “easy to navigate,” “self-explanatory,” and “modern.” One tester’s impression was “not favorable” because he finds it similar to the new UVA home page design that “benefits outsiders.”
  • On the mobile design, 4/6 testers found the hamburger menu and used it to find About.
  • On the mobile design, 5/6 testers were successful in scrolling to find a recent announcement under News.
  • On a scale of 1-5, with 5 being very easy, 5/6 testers rated their ease in completing these tasks as 4 or 5.


Mobile App Ideation Session

On Friday, February 12, the UX Team conducted a Mobile App Ideation session.

Definition of ideation

:  the capacity for or the act of forming or entertaining ideas

Merriam-Webster. Merriam-Webster. Web. 17 Feb. 2016.

The purpose of the session was to give Library Staff the opportunity to take part in the Team’s Mobile App Project. The Mobile App Project is part of a larger project already underway to develop a solid and flexible web services layer. Participants learned about ideation and the project development process. By the end of the session, the participants had created prototypes that addressed user identified problems. During the Spring semester, these prototypes will be the basis for user testing and further development.

The Challenge was to create a “minimum viable product” for the Library’s mobile app that focuses on the needs of both graduate and undergraduate students.

  • Define Stage:
    • Participants divided into teams and examined feedback gathered from student focus group interviews. Needs/wants/observations were separated into categories. The goal was to form Insights based on these categories.
    • Participants then formed “How might we” questions, based on these Insights. Four questions that best aligned with student feedback were chosen.
    • In the next exercise, each participant composed a tweet, announcing the new Library App.
  • Diverge Stage:
    • In this stage, participants explored various ideas and drew a storyboard that illustrated their most viable idea.
  • Decide and Prototype Stages:
    • In the final exercise, each team selected the best individual idea and created a storyboard for presentation to the group. The storyboards will later be used for testing and development.


The ideation session resulted in the selection of four themes. The Library app should provide

  • A customizable, fully functional version of Virgo and ILL
  • The ability to reserve group study rooms, and display seat availability throughout each location
  • Location based information for navigation
  • Real-time information about the Library

In the last exercise, participants explored these themes and created five mock-ups for a proposed app.

Storyboard Summaries

  1. For a user wanting real-time information about study space availability: the user could select a specific library and a specific room. The app would indicate either percentage full, or provide a heat map to indicate seat availability.
  2. (Assumes User has opted-in to multiple notifications)
    User is in a bookstore when the Library app reminds the user to check the Library catalog before buying a book. The user scans the ISBN, finds the book, and chooses to be reminded to get the book when she is in the Library. User visits the Library, receives notification(s).  The app provides a stacks map and offers help locating the book. Another notification asks the user if she would like to consult databases based on the call number of her selection, or talk to a subject specialist. The user receives one final notification, asking for feedback on the experience.
  3. User has three objectives: print a paper, locate a specific study group, and check-out a movie. The app would show the locations of services (such as printers) and study spaces (quiet and collaborative), as well as provide the ability to search the Library catalog. In each case, the app would provide a floor plan, indicating the user’s location and path to the desired service/room/item.
  4. User types search into Search Box → Retrieves map of the Grounds → User chooses Alderman Library → App displays catalog record showing copies available → App displays floor plan of Alderman Library, indicating You are here → App displays layout of the stacks, indicating the location of the book for which the user searched.
  5. Premise: People want all kinds of information, not limited to databases. The app would connect people who have questions with people who have answers. There would be three channels from which to choose: Ask a Librarian, AI, and Hey (shout out). The last would be for questions such as “How many people are in line at the Coffee Café?” Users might get points for answering such questions.

Videos of storyboards:


Research Results: What Users Might Want on Digital Signs

The UX Team and internal stakeholders are working together to find the best way to leverage new, integrated, interactive digital signage to enhance the user experience.

To that end, UX hosted a staff brainstorming session in April to generate a large number of ideas for content we could present to users via digital signs and the Alderman kiosk. Staff then selected their most promising ideas to present to users for their feedback.

In May, we worked with staff to deploy whiteboards in Alderman, Clemons, and Brown where students could vote and comment on staff ideas, or add their own.

The results, while not scientific, are a good “gut check” for what might resonate with users, and are as follows (including number of “checks” in favor of idea):

  1. What study space is not full: 80
  2. Videos (exercise/dance breaks): 39
  3. Display quiet and collaborative spaces: 35
  4. Information about Safe Ride, Safe Walk: 35
  5. Local and library-wide events, day of event
    promotion: 27
  6. Interactive map of Grounds (w/shuttle/bus
    schedules, walking directions): 24
  7. Instant polls and surveys: 17
  8. Library staff Who’s Who: 16

Bar graph of user votes for digital signage ideas.

Bar graph of user votes for digital signage ideas, by library

Update: In another view of this data, the results change slightly when one subtracts the votes against and idea from the votes for an idea to arrive at the net vote:

  1. What study space is not full: +76
  2. Display quiet and collaborative spaces: +35
  3. Information about Safe Ride, Safe Walk: +35
  4. Videos (exercise/dance breaks): +34
  5. Local and library-wide events, day of event promotion: +27
  6. Interactive map of Grounds (w/shuttle/bus schedules, walking directions): +21
  7. Instant polls and surveys: +12
  8. Library staff Who’s Who: +10

You can find more detailed results in Confluence, and you can review the output of the Library staff brainstorm there as well.

The purpose of this exercise was to get a rough idea of content interests to create some helpful and interesting signs for users, but was not intended to be prescriptive. We’ll pilot ideas and test results as we assess how users interact with these signs.

LibGuides v2 Transition Resources

The Library will be transitioning to LibGuides v2 this summer (hooray!). This post summarizes the important points that guide owners will need to make a smooth transition.


  • The migration will take place 6/17, but v2 guides will NOT be live at that time.
  • Guide owners have until 8/1 to make changes in preparation for Fall semester.


A cross-Area group is available to help with transitional needs. Contacts and the kinds of questions they can address are as follows:

  • Renee Reighart (UX): Usability, writing for the web, content best practices
  • Jenn Huck (I&S): Technical transition details
  • Jill Heinze (UX): Usability, supporting user research
  • Tim Morton (CAD): Revised A-Z list; Resource metadata
  • Chris Ruotolo and Michele Claibourn (AE): Managerial support


The migration group held an information session on 5/31:

A member of the migration group is reaching out to those content owners who wanted but were unable to attend.

After the 6/17 migration, training sessions will be offered that focus on refining and optimizing guides in the v2 environment. Sessions will be announced to staff once they are scheduled.


The UX has created and made available numerous resources to facilitate transition:

Get Help!

We know that this transition is a lot of work for content owners, and we want you to know you’re not alone! If you have questions or concerns and aren’t sure who to talk to, feel free to contact Jill Heinze to find the assistance you need.

Thank you, guide owners!