Evaluation challenges and directions for information-seeking support systems

Article Kelly, D., Dumais, S., & Pedersen, J. O. (2009). Evaluation challenges and directions for information-seeking support systems. Computer, 42(3), 60-66. Information Seeking Support Systems are tools that support search for open ended sorts of tasks. Evaluation The basic objective of IR system evaluation is to assess search performance, which is usually tied directly to how effectively the system retrieves and ranks relevant information objects. Common measures for information retrieval evaluation are precision, recall, mean average precision, mean reciprocal rank, and discounted cumulative gain. For interactive information retrieval (IIR), evaluation uses searchers to use one or more experimental IR system to find information described by a small number of prescribed topics... their interactions with systems are logged and at various points they provide feedback via questionnaires or other self reporting. Performance is often measured by comparing searchers with baselines. IR and IIR models rely on labratory evaluation. Ehtnographic evaluations can lead to a more detailed understanding of searchers real information needs, Current models for evaluating ISSSs are insufficient because the user and task models in traditional IR evaluation don't capture all types of information seeking tasks, activities, and situations the web's dynamic nature continually changes the base of objects available for retrieval over time information seeking tasks are often complex and evolve without having stable definable end points information seeking occurs over sustained periods, implying the need for longitudinal evaluation designs that measure change. Evaluation Directions One way to create a research community that includes participation from the broad range of disciplines needed to develop and evaluate IS systems is to create shareable resources that facilitate and enable participation. These resources should contain datasets, search components, data collection tools, methods and measures, and operational search environments in which to explore new ideas.