An Introduction to Query Token Density (QTD) Search Algorithms

Abstract

A new class of search algorithms and metrics based on an idea called Query Token Density (QTD) has been created and implemented in the Lexis Nexis search engine. QTD is based on the notion that dense, high-frequency co-occurrence of query terms within a document is a proxy for high user relevance. QTD has proven to be highly successful at improving Discounted Cumulative Gain (DCG) based relevance precision for multi-token Boolean and Natural Language queries, though results are significantly better for Natural Language searches.

We discuss the computation and testing of the QTD algorithm including the initial implementations, testing of the algorithm via human subject matter expert relevance testing completed on thousands of queries and results, end-customer facing A/B testing, and performance testing in a cloud-based environment. We also discuss some additional notes on implementation, possible extensions for various content types, algorithm implementation improvements, and some variants of the algorithm to be explored in future work.

Keywords: Search Engine Algorithm, Search Algorithms, Query Token Density, QTD, algorithm implementation, algorithm testing, Natural Language search, Boolean search