Evaluating Exploratory Visualization Systems: A User Study on How Clustering-Based Systems Supporting Information Seeking from Large Document Collections

Abstract

Assisting iterative, opportunistic and evolving visual sensemaking has been an important research topic as information overload continues. Such exploratory systems (ESs) maximize users’ information gain through learning and have been widely used in scientific discovery and decision making contexts. Although many ESs have been developed recently, there is a lack of general guidance on how to evaluate such systems. Researchers are facing challenges such as understanding the underlying cognitive learning process of these systems. In this paper, we propose an evaluation framework designed specifically for ESs. The new framework is built based on cognitive load theory (CLT) and takes the user as well as the system as the focus of evaluation. With the thorough examination of users’ cognitive process, carefully designed procedures detangle the interwoven, interactive process of ESs as well as the variability that comes with human subjects. To demonstrate the usage of the framework, we present an evaluation conducted on Newdle, a clustering-based ES for large news documents. This study was a successful use case of the framework. It revealed how and why clustering-based ES benefited (or not) users in a variety of information seeking tasks. We also report the different information seeking strategies observed from the users and summarize leverage points for designing clustering-based ES.