Systematic DCG Evaluations of Search Algorithm Components

Abstract

Many of the search algorithms used in LexisNexis applications are composed of individual components that each contribute to the relevance scores assigned to the members of result sets. In Lexis Advance, there are several such components called Levers, which can be enabled or disabled via query plans to alter the behavior of the search engine. Thirteen Levers are currently enabled in the Production search engine. These Levers were largely implemented and deployed before the current era of search accuracy testing using the STF tool. To determine how much each Lever individually contributes to the overall accuracy of the search engine, a series of experiments were conducted in STF in which the results of turning off each lever were compared with the system as it is configured now. From these experiments, we were able to determine which Levers had the most impact (whether positive or negative) and thus we able to target future efforts in improving the overall search algorithm. One of the major outcomes of this effort was obtaining the query-document pairs that needed to be rated by human testers in order to improve the coverage of tests. An overview of the testing apparatus is provided so that other organizations within the RELX Group may conduct similar assessments.

Keywords: search, relevancy, accuracy, metrics, algorithm, hrt, dcg, hdcg, evaluation