CLEF

Conference and Labs of the Evaluation Forum

Get Started

Unlocking Information Access

The CLEF Initiative (Conference and Labs of the Evaluation Forum) is structured in two main parts:

  1. a series of Evaluation Labs, i.e. laboratories to conduct evaluation of information access systems and workshops to discuss and pilot innovative evaluation activities;
  2. a peer-reviewed Conference on a broad range of issues, including
    • investigation continuing the activities of the Evaluation Labs;
    • experiments using multilingual and multimodal data; in particular, but not only, data resulting from CLEF activities;
    • research in evaluation methodologies and challenges.

The CLEF Initiative promotes research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure. CLEF promotes research and development by providing an infrastructure for:

  • multilingual and multimodal system testing, tuning and evaluation;
  • investigation of the use of unstructured, semi-structured, highly-structured, and semantically enriched data in information access;
  • creation of reusable test collections for benchmarking;
  • exploration of new evaluation methodologies and innovative ways of using experimental data;
  • discussion of results, comparison of approaches, exchange of ideas, and transfer of knowledge.
Template for Bids

Bids to host CLEF 2026 are open

Deadline Wednesday, December 20th, 2023

Guidelines on submitting you bid can be found in the Template for Bids.

Submit your bid by email to the Steering Committee Chair at chair@clef-initiative.eu.

CLEF 2022 LNCS proceedings cover page

CLEF 2023 LNCS Proceedings

Volume 14163

The CLEF 2023 LNCS proceedings are available at https://doi.org/10.1007/978-3-031-42448-9.

CLEF 2022 CEUR-WS Working Notes page

CLEF 2023 CEUR-WS Working Notes

Volume 3497

The CLEF 2023 CEUR-WS Working Notes are available at https://ceur-ws.org/Vol-3497/.

CLEF 2024 logo

CLEF 2024 Conference

9-12 September 2024, Grenoble, France

Please, visit the CLEF 2024 site for more information.

CLEF book cover page

Book on 20 years of CLEF

The Information Retrieval Series, volume 4

The book "Information Retrieval Evaluation in a Changing World. Lessons Learned from 20 Years of CLEF" is available at https://doi.org/10.1007/978-3-030-22948-1.

Publications

The CLEF Initiative publishes two series of peer-reviewed papers:

  • the conference proceedings, containing conference papers, condensed lab overviews, and best of labs, under the Springer Lecture Notes in Computer Science (LNCS) series;
  • the working notes, containing extended lab overviews and lab participant papers, under the CEUR Workshop Proceedings (CEUR-WS.org) series.

Moreover, over the years the CLEF Initiative has published a few books related to multilingual and multimodal information access.

According to Google Scholar Metrics CLEF is among the top 20 venues for the "Databases and Information Systems" area.

  • Recent

    6 publications

  • LNCS Proceedings

    24 publications

  • CEUR-WS Working Notes

    23 publications

  • Books

    4 publications

  • All

    51 publications

CLEF 2023 LNCS proceedings cover page

CLEF 2023 LNCS Proceedings

Volume 14163

CLEF 2023 CEUR-WS Working Notes page

CLEF 2023 CEUR-WS Working Notes

Volume 3497

CLEF 2022 LNCS proceedings cover page

CLEF 2022 LNCS Proceedings

Volume 13390

CLEF 2022 CEUR-WS Working Notes page

CLEF 2022 CEUR-WS Working Notes

Volume 3180

CLEF 2021 LNCS proceedings cover page

CLEF 2021 LNCS Proceedings

Volume 12880

CLEF 2021 CEUR-WS Working Notes page

CLEF 2021 CEUR-WS Working Notes

Volume 2936

CLEF 2020 LNCS proceedings cover page

CLEF 2020 LNCS Proceedings

Volume 12260

CLEF 2020 CEUR-WS Working Notes page

CLEF 2020 CEUR-WS Working Notes

Volume 2696

CLEF 2019 LNCS proceedings cover page

CLEF 2019 LNCS Proceedings

Volume 11696

CLEF 2019 CEUR-WS Working Notes page

CLEF 2019 CEUR-WS Working Notes

Volume 2380

CLEF 2018 LNCS proceedings cover page

CLEF 2018 LNCS Proceedings

Volume 11018

CLEF 2018 CEUR-WS Working Notes page

CLEF 2018 CEUR-WS Working Notes

Volume 2125

CLEF 2017 LNCS proceedings cover page

CLEF 2017 LNCS Proceedings

Volume 10456

CLEF 2017 CEUR-WS Working Notes page

CLEF 2017 CEUR-WS Working Notes

Volume 1866

CLEF 2016 LNCS proceedings cover page

CLEF 2016 LNCS Proceedings

Volume 9822

CLEF 2016 CEUR-WS Working Notes page

CLEF 2016 CEUR-WS Working Notes

Volume 1609

CLEF 2015 LNCS proceedings cover page

CLEF 2015 LNCS Proceedings

Volume 9283

CLEF 2015 CEUR-WS Working Notes page

CLEF 2015 CEUR-WS Working Notes

Volume 1391

CLEF 2014 LNCS proceedings cover page

CLEF 2014 LNCS Proceedings

Volume 8685

CLEF 2014 CEUR-WS Working Notes page

CLEF 2014 CEUR-WS Working Notes

Volume 1180

CLEF 2013 LNCS proceedings cover page

CLEF 2013 LNCS Proceedings

Volume 8138

CLEF 2013 CEUR-WS Working Notes page

CLEF 2013 CEUR-WS Working Notes

Volume 1179

CLEF 2012 LNCS proceedings cover page

CLEF 2012 LNCS Proceedings

Volume 7488

CLEF 2012 CEUR-WS Working Notes page

CLEF 2012 CEUR-WS Working Notes

Volume 1178

CLEF 2011 LNCS proceedings cover page

CLEF 2011 LNCS Proceedings

Volume 6941

CLEF 2011 CEUR-WS Working Notes page

CLEF 2011 CEUR-WS Working Notes

Volume 1177

CLEF 2010 LNCS proceedings cover page

CLEF 2010 LNCS Proceedings

Volume 6360

CLEF 2010 CEUR-WS Working Notes page

CLEF 2010 CEUR-WS Working Notes

Volume 1176

CLEF 2009 LNCS proceedings cover page

CLEF 2009 LNCS Proceedings - Part I

Volume 6241

CLEF 2009 LNCS proceedings cover page

CLEF 2009 LNCS Proceedings - Part II

Volume 6242

CLEF 2009 CEUR-WS Working Notes page

CLEF 2009 CEUR-WS Working Notes

Volume 1175

CLEF 2008 LNCS proceedings cover page

CLEF 2008 LNCS Proceedings

Volume 5706

CLEF 2008 CEUR-WS Working Notes page

CLEF 2008 CEUR-WS Working Notes

Volume 1174

CLEF 2007 LNCS proceedings cover page

CLEF 2007 LNCS Proceedings

Volume 5152

CLEF 2007 CEUR-WS Working Notes page

CLEF 2007 CEUR-WS Working Notes

Volume 1173

CLEF 2006 LNCS proceedings cover page

CLEF 2006 LNCS Proceedings

Volume 4730

CLEF 2006 CEUR-WS Working Notes page

CLEF 2006 CEUR-WS Working Notes

Volume 1172

CLEF 2005 LNCS proceedings cover page

CLEF 2005 LNCS Proceedings

Volume 4022

CLEF 2005 CEUR-WS Working Notes page

CLEF 2005 CEUR-WS Working Notes

Volume 1171

CLEF 2004 LNCS proceedings cover page

CLEF 2004 LNCS Proceedings

Volume 3491

CLEF 2004 CEUR-WS Working Notes page

CLEF 2004 CEUR-WS Working Notes

Volume 1170

CLEF 2003 LNCS proceedings cover page

CLEF 2003 LNCS Proceedings

Volume 3237

CLEF 2003 CEUR-WS Working Notes page

CLEF 2003 CEUR-WS Working Notes

Volume 1169

CLEF 2002 LNCS proceedings cover page

CLEF 2002 LNCS Proceedings

Volume 2785

CLEF 2002 CEUR-WS Working Notes page

CLEF 2002 CEUR-WS Working Notes

Volume 1168

CLEF 2001 LNCS proceedings cover page

CLEF 2001 LNCS Proceedings

Volume 2406

CLEF 2001 CEUR-WS Working Notes page

CLEF 2001 CEUR-WS Working Notes

Volume 1167

CLEF 2000 LNCS proceedings cover page

CLEF 2000 LNCS Proceedings

Volume 2069

CLEF 2000 CEUR-WS Working Notes page

CLEF 2000 CEUR-WS Working Notes

Volume 1166

Early Detection of Mental Health Disorders by Social Media Monitoring. The First Five Years of the eRisk Project

Early Detection of Mental Health Disorders by Social Media Monitoring. The First Five Years of the eRisk Project

Studies in Computational Intelligence, volume 1018, 2022

Information Retrieval Evaluation in a Changing World. Lessons Learned from 20 Years of CLEF cover page

Information Retrieval Evaluation in a Changing World. Lessons Learned from 20 Years of CLEF

The Information Retrieval Series, volume 41, 2019

Multilingual Information Retrieval. From Research To Practice cover page

Multilingual Information Retrieval. From Research To Practice

text book, 2012

mageCLEF, Experimental Evaluation in Visual Information Retrieval cover page

ImageCLEF. Experimental Evaluation in Visual Information Retrieval

The Information Retrieval Series, volume 32, 2010

Editions

CLEF started in 2000, under the name Cross-Language Evaluation Forum, as an event in conjunction with the European Conference for Digital Libraries (ECDL), now called Theory and Practice on Digital Libraries (TPDL).

Since 2010, CLEF has taken the form of an independent event, under the name Conference and Labs of the Evaluation Forum.

CLEF 2024

9-12 September 2024
Grenoble, France

CLEF 2023

18-21 September 2023
Thessaloniki, Greece

CLEF 2022

5-8 September 2022
Bologna, Italy

CLEF 2021

21-24 September 2021
Bucharest, Romania (virtual)

CLEF 2020

22-25 September 2020
Thessaloniki, Greece (virtual)

CLEF 2019

9-12 September 2019
Lugano, Switzerland

CLEF 2018

10-14 September 2018
Avignon, France

CLEF 2017

11-14 September 2017
Dublin, Ireland

CLEF 2016

5-8 September 2016
Évora, Portugal

CLEF 2015

8-11 September 2015
Toulouse, France

CLEF 2014

15-18 September 2014
Sheffield, UK

CLEF 2013

23-26 September 2013
Valencia, Spain

CLEF 2012

17-20 September 2012
Rome, Italy

CLEF 2011

19-22 September 2011
Amsterdam, The Netherlands

CLEF 2010

20-23 September 2010
Padua, Italy

CLEF 2009

30 September-2 October 2009
Corfu, Greece

CLEF 2008

17-19 September 2008
Aarhus, Denmark

CLEF 2007

19-21 September 2007
Budapest, Hungary

CLEF 2006

20-22 September 2006
Alicante, Spain

CLEF 2005

21-23 September 2005
Vienna, Austria

CLEF 2004

15-17 September 2004
Bath, UK

CLEF 2003

21-22 August 2003
Trondheim, Norway

CLEF 2002

19-20 September 2002
Rome, Italy

CLEF 2001

3-4 September 2001
Darmstadt, Germany

CLEF 2000

21-22 September 2000
Lisbon, Portugal

Steering Commitee

The main purpose of the Steering Committee is to ensure the continuity, the growth, and the quality of the CLEF Initiative.

Nicola Ferro

Nicola Ferro

University of Padua, Italy

Chair
Paolo Rosso

Paolo Rosso

Universitat Politècnica de València, Spain

Deputy Chair for the Conference
Martin Braschler

Martin Braschler

Zurich University of Applied Sciences (ZHAW), Switzerland

Deputy Chair for the Labs
Avi Arampatzis

Avi Arampatzis

Democritus University of Thrace, Greece

Member
Alberto Barrón-Cedeño

Alberto Barrón-Cedeño

University of Bologna, Italy

Member
Khalid Choukri

Khalid Choukri

Evaluations and Language resources Distribution Agency (ELDA), France

Member
Fabio Crestani

Fabio Crestani

Università della Svizzera italiana, Switzerland

Member
Carsten Eickhoff

Carsten Eickhoff

University of Tübingen, Germany

Member
Norbert Fuhr

Norbert Fuhr

University of Duisburg-Essen, Germany

Member
Anastasia Giachanou

Anastasia Giachanou

Utrecht University, The Netherlands

Member
Lorraine Goeuriot

Lorraine Goeuriot

Université Grenoble Alpes, France

Member
Julio Gonzalo

Julio Gonzalo

National Distance Education University (UNED), Spain

Member
Donna Harman

Donna Harman

National Institute for Standards and Technology (NIST), USA

Member
Bogdan Ionescu

Bogdan Ionescu

University Politehnica of Bucharest, Romania

Member
Evangelos Kanoulas

Evangelos Kanoulas

University of Amsterdam, The Netherlands

Member
Birger Larsen

Birger Larsen

University of Aalborg, Denmark

Member
David E. Losada

David E. Losada

Universidade de Santiago de Compostela, Spain

Member
Mihai Lupu

Mihai Lupu

Field consultant, Vienna, Austria

Member
Maria Maistro

Maria Maistro

University of Copenhagen, Denmark

Member
Josiane Mothe

Josiane Mothe

IRIT, Université de Toulouse, France

Member
Henning Müller

Henning Müller

University of Applied Sciences Western Switzerland (HES-SO), Switzerland

Member
Jian-Yun Nie

Jian-Yun Nie

Université de Montréal, Canada

Member
Gabriella Pasi

Gabriella Pasi

University of Milano-Bicocca, Italy

Member
Eric SanJuan

Eric SanJuan

University of Avignon, France

Member
Giuseppe Santucci

Giuseppe Santucci

Sapienza University of Rome, Italy

Member
Laure Soulier

Laure Soulier

Pierre and Marie Curie University (Paris 6), France

Member
Theodora Tsikrika

Theodora Tsikrika

Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), Greece

Member
Christa Womser-Hacker

Christa Womser-Hacker

University of Hildesheim, Germany

Member

Past Members

Paul Clough

Paul Clough

University of Sheffield, United Kingdom

Member
Djoerd Hiemstra

Djoerd Hiemstra

Radboud University, The Netherlands

Member
Jaana Kekäläinen

Jaana Kekäläinen

University of Tampere, Finland

Member
Seamus Lawless

Seamus Lawless

Trinity College Dublin, Ireland

Member
Carol Peters

Carol Peters

ISTI, National Council of Research (CNR), Italy

Chair (2000-2009)
Founder of CLEF
Emanuele Pianta

Emanuele Pianta

Centre for the Evaluation of Language and Communication Technologies (CELCT), Italy

Member
Maarten de Rijke

Maarten de Rijke

University of Amsterdam, The Netherlands

Member
Jacques Savoy

Jacques Savoy

University of Neuchêtel, Switzerland

Member
Alan Smeaton

Alan Smeaton

Dublin City University, Ireland

Member

Association

What is the CLEF Association?

The CLEF Association is an independent no-profit legal entity, established in October 2013 as a result of activity of the PROMISE Network of Excellence which backed CLEF from 2010 to 2013.

What is the mission of the CLEF Association?

The CLEF Association has scientific, cultural and educational objectives and operates in the field of information access systems and their evaluation. Its mission is:

  • to foster critical thinking about advancing information access and use from a technical, economic and societal perspective;

  • to promote access to information and use of evaluation.

What are the objectives of the CLEF Association?

Within its areas of interest, the CLEF Association aims at a better understanding of the use and access to information and how to improve this. The areas of interest stated in the the above mission translate into the following objectives:

  • clustering stakeholders with multidisciplinary competences and different needs, including academia, industry, education and other societal institutions;

  • facilitating medium/long-term research in information access and use and its evaluation;

  • increasing, transferring and applying expertise.

What are the activities of the CLEF Association?

The CLEF Association pursues its mission and objectives via the following pillar activities:

  • CLEF: sustains and promotes the popular CLEF evaluation series as well as providing support for its coordination, organisation, and running;

  • collections and experimental data: fosters the adoption and exploitation of large-scale shared experimental collections, makes them available under appropriate conditions and trusted channels, and shares experimental results and scientific data for comparison with state-of-the-art and for reuse;

  • infrastructure: supports the adoption and deployment of software and hardware infrastructures which facilitate the experimental evaluation process, the sharing of experimental collections and results, and interaction with and understanding of experimental data;

  • education and knowledge transfer: organises educational events, such as summer schools, and knowledge transfer activities, such as workshops, aimed not only at spreading know-how about information access and use but also at raising awareness and stimulating alternative viewpoints about the technical, economic, and societal implications.

What is the statute of the CLEF Association?

The complete statute of the CLEF Association is available in the following PDF document.

Other Initiatives

What are the other Evaluation Initiatives?

There are other "sister" evaluation initiatives around the world in the information access area, namely:

  • TREC, the Text REtrieval Conference in the United States;                 

  • NTCIR, the NII Testbeds and Community for Information access Research in Japan and Asia;

  • FIRE, the Forum for Information Retrieval Evaluation in India;

  • MediaEval, the Benchmarking Initiative for Multimedia Evaluation in Europe.

Contact

If you need any additional information about CLEF, please email us at
chairclef-initiative.eu