CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

We are proud to introduce CommonLID, a web-focused language identification benchmark spanning 109 languages, created in partnership with open source organizations and language communities. The CommonLID dataset is available through Hugging Face, and the preprint paper is available on arXiv.

At the Common Crawl Foundation, we want to make our open data as comprehensive and representative as possible.  As part of this, one of our long-term research goals has been to improve the language coverage of our crawls.  We have been exploring several approaches towards this aim, but the most important for this blog post are our efforts to improve automatic language identification.

While detecting the language in which a text is written seems like a simple task, prior research has shown that current language identification models can struggle, particularly with underrepresented languages.  This problem is even more challenging when detecting languages in web data.  The way people write on the web is often very different to how ‘formal’ text is written, meaning we need models which are tested on web data specifically.

For instance, these are some examples of web text whose language is mislabelled by existing LID systems:

However, there is a general lack of quality datasets available for testing language identification models, especially those which cover a wide range of languages. Those which do exist don’t include web data, making it hard to assess performance and limiting researchers’ ability to make progress towards better language detection models for the web.

To close this gap, we are pleased to announce the release of CommonLID, a language identification benchmark dataset for the web which covers 109 languages.  This dataset was nearly two years in the making, involving significant collaboration with language communities and multiple research organisations.  We created CommonLID as part of a shared task at the Workshop for Multilingual Data Quality Signals, hosted at COLM in 2025.  The first step was building an annotation platform  in partnership with MLCommons and Factored AI, so that participants could view and mark-up data.  In collaboration with EleutherAI, we then hosted multiple hackathons with language community organisations like Masakhane and SEACrowd during which participants contributed language labels for a selection of Common Crawl’s web data.  Finally, we curated the final dataset and used it to evaluate multiple existing language identification models, shedding light on the current state of the art.

Everyone who contributed to the project was invited to be on the paper, and we would like to thank all 97(!) co-authors for their hard work in making this project happen.  We hope to expand CommonLID in the future to include data for more languages through continued community-led work.  We also hope to use CommonLID, and its future versions, to help develop and maintain a new generation of open source LID models, allowing us and our community to better curate multilingual corpora from our crawl data.