Living Labs for Online Evaluation

Experimental evaluation has always been central to Information Retrieval research. The field is increasingly moving towards online evaluation, which involves experimenting with real, unsuspecting users in their natural task environments. Essentially, the production search engine operates as a "living lab." For a long time, this type of evaluation was only available to those working within organizations that operate a search engine. This, however, is about to change as pioneering efforts to realize the idea of living labs in practice are now in place and are available to the community. Specifically, with the recent introduction of the Living Labs for IR Evaluation initiative at CLEF and the OpenSearch track at TREC, researchers can now have direct access to such labs. In this talk, I will provide an overview of the underlying theory, introduce the living labs for IR evaluation platform, and discuss the specific use-cases studied at CLEF and TREC.

Krisztian Balog is an associate professor at the University of Stavanger where he leads the Information Access & Interaction research group. His general research interests lie in the use and development of information retrieval, information extraction, and machine learning techniques for intelligent information access tasks. His current research concerns entity-oriented and semantic search, and novel evaluation methodologies. He has been a co-organizer of international workshops on these themes (SIGIR 2008, 2011 and 2012; CIKM 2013 and 2015), and of world-wide information retrieval benchmarking efforts (TREC Entity track 2009-2011; CLEF Living Labs for IR Evaluation 2015-2016; TREC Open Search track 2016).