• Purpose: Understand use of and expectations about search from the UVA Library homepage
• Stakeholders: UVA Library homepage users and staff
• Survey dates: April 19-25 (students) and June 29-30 (faculty)
• Participants: 33 UVA undergraduate students, 19 graduate students, and 9 UVA faculty
• Methodology: Participants were recruited from our existing research queue and through emails forwarded to academic departments. Students were compensated with a $10 Amazon card for completing the survey. The online Qualtrics surveys presented examples of site search bars and search results using screenshots from Harvard and Stanford libraries. Participants were given tasks to determine clarity of designs and were asked for opinions about the preferred display of search results. See Survey for a full copy of the survey questions.

Summary of Recommendations

• Keep the Virgo search bar prominently displayed on the Library home page. Faculty and graduate students in particular use it frequently.
• The magnifying glass icon that represents site search isn’t sufficiently identifiable on its own. If used, the icon needs a label.
• There is some evidence that patrons are not expecting separate search bars for the catalog and library site search, and also evidence that patrons prefer to have a comprehensive page of search results. This strongly suggests that we should work toward having a combined search for Virgo and the Library website, at minimum. Preferably we should also include LibGuides and other major content sites within the Library domain.
• The search results page should be categorized in a familiar way (“catalog,” “articles,” “library information,” “LibGuides”) and should have a clean, clear, and modern design. Breadcrumbs will help patrons navigate this complex search results page.

A Closer Look: Survey Question #4

“Here is a screenshot of another library homepage.  Click or tap once on the area where you would perform a search for information about this library’s food and drink policies.”

Harvard home page screenshot

Only 10/52 of students and 3/7 faculty correctly identified the magnifying glass icon as the place to find site search. Almost half of the students (22/52) misidentified the Hollis catalog search bar as site search. Participants may have interpreted “perform a search” to be synonymous with clicking on links. Student comments indicate that many chose the Hollis search bar because of its size and central location. Students may have expected the main search bar to provide this information or they weren’t expecting a second search bar. “Hollis” may not be identifiable as an online catalog, leading to incorrect use. Or there may be an expectation that a search bar will search everything Library-related.

When asked to explain why they chose the area they did:

• “Food and drink policies wouldn’t be in the catalog, it would be on the library site”
• “That’s where I would normally search in other databases”
• “It is a search box”
• “Very in my face; I don’t have to search for the search box”
• “It is at the center of the page”
• “It looks like a regular search bar”
• “It may be easiest just to ask someone”
• “No good choices”
• “Unsure where to search”

A Closer Look: Survey Question #9

“Which option do you prefer for the display of search results?”

Option 1 [Harvard] search results screenshot

Option 2 (Stanford) search results screenshot

Option 2 [Stanford]

Combined totals are roughly split between Options 1 and 2, but the majority of graduate students (13/17) preferred Option 2 (Stanford). Faculty were evenly split between Option 1 and Option 2. Even when considering which students indicated that they currently use UVA Library site search at least once a month, the preference between Option 1 and Option 2 is evenly split.

The comments give us a good idea of what we should take away from each design. The Harvard design (option 1) was thought to be sleek, clean, and easy to read and navigate. The Stanford design (option 2) received high marks for breaking down search results into familiar categories like articles, catalog, and library information. The Stanford design was considered organized and complete in its presentation of results; but also criticized for being “outdated” and “cluttered.”

“Do you have any further comments about the display of search results?”

Sampling of comments from undergraduate students:

• “I like the subcategorization [of Option 2]. Helps me quickly eliminate places I need to look.”
• “I think the descriptiveness of option 2 is great but option 1 layout is cleaner.”

Survey

Faculty and students were given surveys with only slight differences. This is the survey as presented to students. Some questions were removed from the final analysis due to problematic wording.

Please help the Library evaluate our website.

The Library is reviewing the search bar function on our web pages and we would like your help. You will be asked to view several screenshots from library pages at other institutions and asked questions about them. This survey should take no more than 10 minutes of your time. It is necessary to use a desktop or a laptop or a tablet in landscape view to complete this survey.

If you have any questions about this survey please contact Melinda Baumann, User Experience Librarian, at baumann@virginia.edu.

Thank you!

Q1. Which best describes your relationship with the University of Virginia?

• UVA Faculty or Staff
• UVA Undergraduate Student
• UVA Graduate Student
• UVA Alum
• Other

Please tell us about your experiences and expectations when searching for information on the Library’s web page.

Q2. How many times a month do you start a Virgo (Library catalog) search from the Library home page?

UVA Library header screenshot

____ 0 ____ 1 – 5  ____ More than 5

Q3. How many times a month do you search the Library website using the search bar shown here?

Library site search screenshot

____ 0 ____ 1 – 5  ____ More than 5

Here is a screenshot of another library homepage.

Q4. Click or tap once on the area where you would perform a search for information about this library’s food and drink policies.

Harvard home page screenshot

What prompted you to choose this area?

______________________________________________________

The remaining survey questions relate to the display of search results.

Here is a screenshot of another library’s search results page.

Q5. Click or tap once on the area of this screenshot where you might expect to find information about that library’s food and drink policies.

Search results screenshot from another library

Please rate your confidence that you might find information on food and drink policies here, where 1 would be not confident, 3 would be neutral, and 5 would be very confident.

Q6. Click or tap once in the area where you might search the library catalog to find resources about different food and drink policies. [Question removed from final analysis]

Search results screenshot from another library

Please rate your reaction to these search results, with 1 would be confusing or frustrating, 3 would be neutral, and 5 would be clear and easy to read. [Question removed from final analysis]

Do you have other comments about this search results page? [Question removed from final analysis]

______________________________________________________

Here is a screenshot of another library’s search results page.

Q7. Click or tap once in the area of this screenshot where you might expect to find information about that library’s food and drink policies.

Search results screenshot from another library

Please rate your confidence that you might find information on food and drink policies here, where 1 would be not confident, 3 would be neutral, and 5 would be very confident.

Q8. Click or tap once in the area where you might search the library catalog to find resources about different food and drink policies. [Question removed from final analysis]

Search results screenshot from another library

Please rate your reaction to these search results, with 1 would be confusing or frustrating, 3 would be neutral, and 5 would be clear and easy to read. [Question removed from final analysis]

Do you have other comments about this search results page? [Question removed from final analysis]

______________________________________________________

Q9. Which option do you prefer for the display of search results?

Option 1 [Harvard] search results screenshot

Option 2 (Stanford) search results screenshot

____ Option 2 [Stanford]

____ No preference

Do you have other comments about this search results page?

______________________________________________________

Thank you for helping to improve the Library web site!

Screenshot of UVA Library homepage, 1/25/21

• Purpose:  Update research on use of and expectations about UVA Library homepage
• Stakeholders:  UVA Library homepage users and staff
• Survey dates:  March 15-17 (students) and April 5-19 (faculty)
• Participants:  58 UVA undergraduate and graduate students and 21 UVA faculty
• Methodology: Participants were recruited from our existing research queue, homepage banners, and emails to Library liaisons forwarded to academic departments. Two online Qualtrics surveys were written based on data gathered four years ago when the homepage was newly-designed, and based on current research questions. Students were remunerated with a $10 gift card.

Highlights of Findings

Survey participants were asked about the three links or areas were most important to them (Q1). Faculty selected Virgo or Advanced Search Virgo as their first choice by a more than 2:1 margin, and 85% of faculty chose Virgo as one of their top three links. LEO/ILL and Research (tied at 40%), Databases and Services (tied at 35%), and Hours (25%) rounded out faculty’s top link choices.

Final tally of most important links (without regard to whether the choice was first, second, or third)

Students also selected Virgo as their first choice by a more than 2:1 margin. Other top choice links were Research (18%), Hours (13%), Spaces (11%), and Databases and Subject Guides (tied at 5%). Hours were more important to faculty than students, which may be an anomaly caused by the closing of most libraries due to the COVID pandemic; analytics from March and April 2021 show that the Hours page is still a top performer (second only to the Research page).

Faculty described the Library homepage as “informative,” “organized,” and “comprehensive” (Q2). Third terms provided by faculty were more likely to be negative

Wordcloud combining faculty and student top descriptive terms

(“busy,” “cluttered, “overwhelming”). Students used descriptors such as “clean,” “informative,” “organized,” and “helpful.”

Survey participants were asked to rank the importance of nine types of information available on the About the Library page (Q5). Both students and faculty gave Library Collections top importance (86% of faculty and 77% of students ranked it 1, 2, or 3). Then opinions diverged for students, who chose Directions and Maps next, followed by Plan a Visit, Mission and Policies, and Staff Directories. Faculty ranked Staff Directories more highly than Directions and Maps, followed by Mission and Policies and Plan a Visit. Both faculty and students ranked Tours, Library Assessment and Statistics, Jobs, and Ways to Financially Support the Library as having less importance to them.

Survey participants were asked to evaluate the Cornell University Library Hours page (Q8). About a third of the students liked the green and red colors representing Open and Closed: “The color coding makes it easier to find libraries that are open and closed when quickly scrolling.” Almost as many students liked the contact information (email, phone numbers). Seven students liked that services such as making a research appointment and picking up materials were listed: “I like that it provides links immediately for further steps.” Five students appreciated the thumbnail photos of libraries, and one commented that they might provide a necessary visual cue: “I like how it shows pictures for some of the libraries. If possible, that would be great because we don’t always call the libraries by their proper name so it can be confusing.” Faculty had similar responses: They approved of the color-coding, contact information, and links to services. Neither group mentioned the Libraries map button in the upper right corner.

Recommendations

• Virgo search bar, ILL/LEO, Research, and Hours should all be prominent on the new homepage design.
• Terms used to describe the homepage include “informative” and “organized” that reflect the UVA Library brand. Students first described the homepage as “clean” which, while not a negative term, may not be what we envision. Refresh this data with more testing after we’ve determined what terms we’d like to represent the Library brand.
• Faculty and students expect to find Collections on the About page; we’ve already put it there in the new navigation. The remaining top areas will either be on the About the Library page or a subpage. Plan a Visit ranked in the middle with faculty and students but should prove to be more important to other types of users, meriting its place of prominence in the new About navigation.
• When viewing another institution’s Hours page, the color-coded Open and Closed buttons were favored by students and faculty, as were the contact information and the links to primary services to make appointments and retrieve materials. Some respondents cautioned about adding more information to our already-complex Hours page. Until we take on a complete redesign of the Hours page we should heed the latter advice.

Survey

Faculty and students were given surveys that differed slightly in question order. Faculty were not asked two questions about Hours as marked below. This is the survey as presented to students.

Please help the Library evaluate our website!

The Library is refreshing the look of its webpages and we would like your help. You will be asked to review several current pages of the Library website and a screenshot from a library website at another institution, and asked questions about them. This survey should take no more than 10 minutes of your time.

If you have any questions about this survey please contact Melinda Baumann, User Experience Librarian, at baumann@virginia.edu.

Thank you!

Which best describes your relationship with the University of Virginia?

• UVA Faculty or Staff
• UVA Undergraduate Student
• UVA Graduate Student
• UVA Alum
• Community Member or Other

Please open the Library homepage (https://www.library.virginia.edu) in a web browser, and then answer the following questions.

Q1. As you think about using the Library homepage, what are the three links or areas on this page that are most important to you?

1st link or area ________________________________________________

2nd link or area ________________________________________________

3rd link or area ________________________________________________

Q2. What are three words or phrases you would use to describe this homepage?

1st descriptive word or phrase _______________________________________________

2nd descriptive word or phrase ______________________________________________

3rd descriptive word or phrase _______________________________________________

Please tell us more about your expectations when using the Library homepage.

Q3. If you were to click or tap on the About link in the Library website header, what types of information would you expect to find on the resulting page?

_______________________________________________

Q4. Now please open the Library About page (https://www.library.virginia.edu/about-uva-library) in a web browser.

Is this what you expected to find? Check the box that best describes your answer.

Q5. Here are the types of information about the Library that you might find on an About page. Please rank them in order of importance to you, with 1 being most important and 9 being least important.

______ Information to plan a visit

______ Directions and maps

______ Tours

______ Library collections

______ Library mission and policies

______ Library assessment and statistics

______ Jobs

______ Staff directories

______ Ways to financially support the Library

Q6. (Students only) If you were to click or tap on the Hours link in the Library website header, what types of information would you expect to find on the resulting page?

_______________________________________________

Q7. (Students only) Go to the Library Hours page (https://www.library.virginia.edu/hours) in another browser tab.

Is this what you expected to find? Check the box that best describes your answer.

Q8. Please compare the UVA Library Hours page to the screenshot from another library hours web page, below. Are there any features on this hours page that you like?

Please tell us about your experiences and expectations when searching for information on the Library’s web page.

Q9. If you were to use the search bar in the Library website header above, what type of search results would you expect to see? This question refers to https://www.library.virginia.edu/

• Results from Virgo, the Library’s online catalog
• Results from the Library website (site search)
• Results from both Virgo and site search
• None of these results

Thank you for helping to improve the Library homepage!

OBJECTIVE: Identify and assess mental models for Learning Resources web page via card sort research

Stakeholders: UVA students; Teaching & Learning staff
Testing dates: 10/7-10/9/20
Participants: 8 graduate and 24 undergraduate students

Methodology: Enrolled UVA graduate and undergraduate students from our research queue were solicited by email to take a fifteen minute online test for which they received a $15 VISA card. We used a card sort tool by Optimal Workshop to map how students think about organizing online resources. T&L staff created a list of 40 learning resources (or “cards”). The 32 participants were asked to sort the cards into groups that made sense to them (see Full Test Results, Analysis tab). The data were then analyzed to determine patterns in mental models to inform the organization of the Learning Resources web page. For more information see Nielsen Norman Group’s overview of the card sort methodology.

Summary of findings

• There was considerable agreement on the grouping of cards relating to multimedia, technology, and training. Nine cards were grouped together from 84% to 100% of the time, indicating that participants saw a strong relationship between these cards.
• A second grouping occurred with cards that related to conducting research, research methods, and definitions of scholarly terms.
• A third grouping related to discrete tasks one would accomplish in the Library and things a new Library user might find helpful to know.
• Cards that related to getting help were more often put into other topic-specific categories; there were few mental models that included a generic “Help” grouping.
• Although the term “Virgo” was used on four cards there was little attempt to create a specific “Virgo” grouping (more likely to be a librarian’s mental model of categorizing by specific research tools). These and other cards relating to acquiring books and journal articles were distributed in categories about doing research and about accomplishing Library-related tasks.
• After completing the card sort participants were asked to give a name to the web page the resources might appear on. Although suggestions were provided to steer people away from the terms “learning” and “resources,” the majority of participants (21/32) still used at least one of those terms, and of those, 11 used both terms.

Recommendations

• The learning resources can be grouped into three task-based categories that align with students’ mental models.
• Each category should have subcategories for Getting Started; Help; and other subcategories that will allow students to drill down to more specific tasks and resources.
• Further usability testing on these categories may result in some fine-tuning of these terms. At this writing, the following categories and subcategories are recommended:
  • Doing Research
    • Getting Started
    • Help
    • Citations
    • Searching for Books and Articles
  • Training and Multimedia
    • Getting Started
    • Help
    • Tutorials
  • Using the Library
    • Getting Started
    • Help
    • Accessing Materials

40 Learning Resources cards

• 3D Printing Studio Training-MakerBot Playlist
• Accessing a Book Through the Hathi Trust Emergency Temporary Access Service
• Audacity Basic Training Playlist
• Booking a Room at a UVA Library
• Boolean Search Tips
• Canon C200 Camera Training Playlist
• DaVinci Resolve 16: Basic Training Playlist
• DaVinci Resolve 16: Beginner’s Guide
• Deeper dive into databases
• Finding a Book in the UVA Library
• Getting Help
• Getting Started with Virgo, the Library’s catalog
• How Do I Find What I Need? – searching UVA resources
• How Do I Start My Research?
• How Do I Use Information Correctly?
• How Is Information Created? – about peer review and scholarship
• How to Make a Request in Virgo
• How to Make Video Tutorials Playlist
• iMovie Essentials Training Playlist
• Information Ethics & Citations
• Interlibrary Loan – when UVA Library doesn’t own it
• Introduction to databases
• Navigating the Library Website
• Peer Review in 3 Minutes
• Placing an Interlibrary Loan Request
• Podcasting in Audacity
• Recalling a Book
• Scholarly vs. Popular Sources Checklist
• Searching Virgo
• Thinking Tool
• Using Academic Search Complete for articles
• Using UVA Library Subject Guides
• Virgo Help
• Walking Tour of Grounds
• What is an Academic Library?
• What is Authority? – how to determine an authoritative source
• What Type of Source Do I Need?
• Who is a Scholar and How Do I Become One? – foundations of scholarship
• Word Press Training
• Zotero Walk-Through – citation manager

Objectives

• Gather feedback from graduate student population on overhauled online catalog tool in development
• Explore overall impressions of the design, utility, and clarity of the new interface
• Obtain high-level guidance on how to approach future design/development
• Expose areas of design/development consideration we may not have explored or that may elaborate on user stories
• Identify potential usability issues and/or ideas we should explore in upcoming testing
• Surface and discuss design/branding questions and ideas

Method

• Focus group with 6 UVA graduate students on October 25, 2019
• Set ground rules:
  • We want you to do the talking
  • No right/wrong
  • All answers are confidential (but we are recording and transcribing for a limited number of project participants)
• What we’re interested in: Reactions to this work-in-progress
  • Look-and-feel
  • Organization of info
  • Functionality
  • Presentation/understandability of results
  • Ideas for improvement: Generating ideas and figuring out mental models. Answers the question: “How would that solve a problem for you?”
  • Paper for sketching, jotting notes, plus white board and markers provided

Themes

• Provide more information sooner (full screen display of records; provide more details on records)
  • Want to get to info in fewer clicks; don’t add unnecessary steps
  • Prefer list display of search results to see more at once
• Groupings, “see all”, availability, filters/facets/values all caused initial confusion but were learnable to grads
• Very much liked that it was possible to search multiple sources at once and find things they weren’t looking for
  • Also want WorldCat, special collections, digital humanities, Google Scholar pools
• Grads would use customization if available to set default search options (sources, filters) and display views
  • Really like bookmarks that extend over sessions (unlike current catalog’s ‘starring’ option)
• Important to be able to limit by date, language, availability before searching
• Subject headings should be hyperlinks to aid research
• Missed having facets on basic search
• Comment: “This looks like a blog”
• Biggest problem with VIRGO currently is the search algorithm that produces results that are not always relevant (books and articles)

UX-4517, Gather feedback from advisory group and staff about proposed Confluence Information Architecture

Purpose: Assess usability and clarity of staff website migrated from wordpress to confluence (Wiki)

Stakeholders: Library staff
Testing dates: 6/14/19
Participants: 6 Library staff

Methodology: library staff were recruited to spend 10 minutes reviewing a new interface and arrangement of intranet content and asked to perform tasks designed to assess usability and clarity of key areas.

Objectives were to assess first impressions and to obtain feedback about what is missing or confusing with the new organization of content, and to determine what improvements are needed. Major categories of information were informed by analytics and other data. Tasks were to find organizational information; contact Library Staff Council; update one’s own directory information; receive reimbursement for work-related travel; and subscribe to a library listserv.

Intranet as tested with library staff on 6/14/19

Findings

• Top of the page needs something visually interesting.
• The term “workflow,” used in the first task, confused some people which may have reduced successful completion.
• Add an alpha list of departments with a “More…” link to complete list.
• “Our Organization” was either overlooked or not fully understood. Clarify with more links or teaser info about what’s in this area.
• “Space Shortcuts” should be relabeled as “Forms,” but the wiki software won’t allow. Need to make it clearer that these are all forms.
• Add Library Staff Council link on main page for higher visibility.
• “Listserv” is not a universally-understood term, so add names of listservs for identification: “Library listservs: Libtalk, Lib-reftalk, Libstaff…”
• The purpose and function of the left menu caused some confusion but proved to be learnable.
• Add a “Communication” category to include info about signing up for listservs and linking to committee and project meeting minutes to answer the question: “What is this group/project up to?”
• With the creation of a “Communication” category the “Working at the Library” category could more closely map to Human Resources and professional development-related items to narrow focus.
• Highlight the search bar to make more visually prominent.
• Review all Quick Links and group similar links together.
• Link to Confluence Help under Get Help.

Protocol

UX-4276, User research on proposed IA structure

Purpose: Assess usability and clarity of proposed public website reorganization
Stakeholders: Library users and staff
Testing dates: 4/3/19-4/4/19
Participants: 15 graduate and 27 undergraduate students (all uva)

Full results: https://www.optimalworkshop.com/treejack/LibUX/uvalibraryweb2019/shared-results/265545lxvt6r83471igax43i7w25jio8

Methodology: Enrolled UVA graduate and undergraduate students were solicited by email from our research queue to take a ten minute test for which they received a $10 deposit to their Cavalier Advantage card. An online tree test tool by Optimal Workshop was employed to help determine the efficacy of the proposed information structure. Based on previous testing and analytics, we created a simple hierarchy of four top categories, each of which had between two and five subcategories (see Tree Structure). Participants were asked to complete 12 tasks designed to assess how closely the information structure matched their mental models (see Full Test Results, Analysis tab). The tree test results indicate how many participants found the correct path on their first try, how many found on their second or third try, and how many failed to find the correct path; as well as how fast the participant completed the task. For an overview of how to analyze tree test results, visit Atlassian’s Tree Testing for Websites.

Summary of Findings

• Across all 12 tasks, 79% of participants ended up at the expected answer and 77% of answers were chosen without backtracking. Even without visual cues, navigation, and menus, participants are finding the correct answer four out of five times.
• Seven out of 12 tasks had a high rate of success (between 81% and 95% participants got to the correct answer) and completing each task took an average of 5.29 seconds.
• All 15 graduate students ended up at the expected answer on 5 tasks.
• Half of the participants first clicked on ABOUT and most immediately found and correctly selected Plan a Visit. Another quarter of the participants looked first under USING THE LIBRARY but all found their way to ABOUT/Plan a Visit (task 3).
• Participants were able to correlate the terms “library study areas” and “reserve a study space” with Library Spaces (tasks 1 and 7).
• Participants were able to correlate the phrase “find a book” with Search, Borrow, Request (task 4).
• Participants seemed to have no trouble identifying “Special Collections” as a library and locating it under Libraries & Collections (task 8); although possibly using the term “collections” in the task was a false identifier.
• Five out of 15 graduate students incorrectly selected TEACHING & PUBLICATION SUPPORT/Teaching Support to look for course reserves (task 9).
• Five participants incorrectly selected Accessibility Services as the location for getting help with an off-grounds access problem. 26 participants went directly to GETTING HELP/Ask a Librarian (task 10).
• Five tasks from the previous tree test in 2017 were largely duplicated in this latest test. In each case, the results improved. See 2017 test results.

Deeper dive: task #2: “A librarian visited your Economics class last week to talk about doing research. Where can you find his name?”

Participants looked in all four categories for the answer, and only eight got there. 20 participants selected ABOUT/Staff Directory, which is logical if a name is known (and probably indicates a poorly-worded task). We intend to link the subject specialist listing with the staff directory and usability testing will determine if this helps in getting students to subject experts. The remaining 8 participants selected GETTING HELP/Ask a Librarian, and it is not necessarily a bad thing that students opt to ask for online help when they don’t know an answer.

Recommendations

• The overall high rate of success indicates a strong IA that aligns with the mental model of students.
• Plan a Visit should remain under ABOUT.
• Library Spaces and Search, Borrow, Request seem to work well under USING THE LIBRARY. Usability testing with a wireframe or prototype will help confirm.
• Whether or not Special Collections can be found under Libraries & Collections is inconclusive and would benefit from further testing.
• Teaching Support resonated with some participants as a logical location for course reserves, so consider adding a search for course reserves there.
• Add further description to Accessibility Services to define it more clearly.
• The plan to link the subject specialist directory with the staff directory will improve people searching.

Tree structure as tested

Tasks and answers