“Letting User Search Behavior Lead Us: An Analysis of Search Queries & Relevancy in USC’s Web-Scale Discovery Tool” – Beth Namei and Christal Young – notes from the session

This session presented out on findings from research done on USC’s recent implementation of the discovery layer Summon.  The presenters did a review based on relevance of returned results for a smaller sub-set of searches, and compared them to both Google and Google Scholar.  Some really unexpected findings, and some really great strategies for transforming the iffy performance of our tools into new approaches for teaching information literacy in our organizations.  Read on!

  • Web scale discovery tools

  • prehosted – preindexed content – faster

  • federated search tools – slowness problems – live searches in all these databases

  • most discovery tools have faceted searching

  • Have we chosen wisely?

    • bringing students back from Google – providing a more intuitive experience for novice searchers

  • Why USC got Summon –

    • 1. provide better discoverability of large number of subscription DBs

    • 2. provide relevant results to users

    • 3. provide better user experience with library’s website

  • Motivation for our study – anecdotal complaints, get evidence about how successful Summon was in leading users to relevant sources

  • learn more about user search behavior when using single search box

  • Transaction Log analysis – history of actions executed in a system

    • advantages – unobtrusive, large quantities of data, low cost

    • disadvantages – leaves gaps & questions, does not capture contextual info, does not capture demographic info, does not capture user satisfaction

  • Of 1.2 million searches in Fall 2013, about 180K unique searches, they chose 384 searches

  • Defining “successful search” – relevant results

    • it’s about more than just the simple search box – it’s also about what you get as a result

  • consistently the message of being overwhelmed

    • “relevancy has made a difference in all of our search lives”

  • Summon’s relevancy ranking – looks at term proximity, term frequency, field weighting

    • also content types (journal articles, books weighted more), currency, citation counts, local collections,

  • A “success engine” – automatically looking for synonyms and term-stemming

  • systems-oriented relevancy rating – did the topic in the search match the results that were delivered

  • figuring out inter-rater reliability – challenging process

  • Known item searches & keyword / topic / exploratory searches

  • Rating known items – relevant (first item) / partially relevant (2nd-10th item) / not relevant (not listed in first 10 results, or no results retrieved)

  • Rating keyword searches – relevant (all search terms in title or record) / partially relevant (3 of first 5 results appear to be related) / not relevant (at least 4 of first 5 items appear to be false hits)

  • wanted also to conduct the searches in Google & Google Scholar as well

    • how does summon measure up?

    • used same rubric

  • Findings

  • top 100 most frequent queries – 73% keyword searches, 25% database or publication names

  • top 100 unique searches make up 13% of total searches / 87% of total searches were unique (the ‘long tail’)

  • Keywords – 62% / Known Item Searches – 36%

  • Summon’s report card – F! – 54% of all sample searches retrieved relevant results

    • after the curve – including ‘partially relevant’ brought it up to 73%

  • Google’s report card – B – 85% received relevant results

    • after the curve – 95% retrieved relevant & partially relevant

  • Google Scholar – F! – also 54%

    • after the curve – B as well

  • Failed searches in Summon – 32% of all known item searches did not locate item being searched for

    • 33% of these failed searches are for items USC does not own

  • Of the items we do own

    • 57% didn’t show up because of user error

    • 40% of the time it should have found something

    • 3% had an irregular character that glitched the system

  • relevancy of academic known item searches that USC owns – everybody gets better

  • REVISED scorecard – Summon gets a C plus, but Google does too

  • User errors – 18% of the searches in the sample had a user input error

  • “Did you mean?” – appeared 24 times – 83% was triggered by user input errors

    • 42% of the time the did you mean links took user to relevant results

  • unlike Summon, Google automatically redirects to the search that it thinks you mean

    • summon lets the user decide if to proceed with the correction

    • people are not going to learn the skill of revising their search

    • Google redirection leads to more user success

  • does the number of words affect search success? – ideal search has 2-4 words

  • Duplicates – only 11% of known item searches, and 22% of keyword

  • anectodal biggest complaint – linking to full text , but that wasn’t their experience in the search data → only 3 bad links out of 55

  • What we learned – Summon has some work to do

  • comparing usage data between catalog and Summon – ppl now think that Summon is the catalog

    • when they look for a book, they will find it

  • Winning! sort of = iffy on relevancy, but intuitive tool and ease of use is great

  • Leadership strategies

    • #1 – learn and use your library’s discovery tool

      • some of USC’s instruction librarians weren’t using it

    • #2 – change what and how we teach

      • reconsider teaching advanced search features – teach more effective critical thinking & evaluative skills

      • less about where to click and more about helping students understand how to interpret search results

    • #3 – be a squeaky wheel – we can’t depend on people to report problems

    • #4 – improvise and fail in front of students

      • show students how to troubleshoot a failed search

      • show searches with typos or how to revise a search that gets no results

    • #5 – stop thinking like a librarian

      • unlearn what we’ve been taught

      • are we in the biz of promoting library databases, or helping facilitate effective search?

    • #6 – empathize with users AND colleagues

      • try not to criticize or judge

      • invote skeptical librarians into your classes to watch you teach with the discovery service

      • showing vs. telling – talking can only take you so far

  • Common complaints

    • cheating, imprecision, homogenization of information, laziness

  • complaining is not a strategy – people want to create new information, why should they work so hard to find what they need to do that?

    • #7 – solicit feedback and then listen

    • #8 – turn negativity to your advantage

    • #9 – Gather evidence – test your assumptions, test colleagues assumptions, study user behavior, assess the tool and then assess again

    • #10 – redefining ourselves – perpetual beta – we don’t have to be the same way forever – fast-moving changing world

  • Questions – deciding when to teach Summon or not in lower/upper division orientations

    • showing what quick search is/isn’t good for – knowing more about the format of sources

    • student’s mental model of searching – intuitive means that they don’t know how it works, it just works! – developing as researchers means that they get more sophisticated

    • discovery tool is like training wheels