Common Crawl - Open Repository of Web Crawl Data

Latest Blog Post

Introducing warc2zip

We are pleased to introduce warc2zip, a tool that allows anyone to turn WARC archives into simpler, zipped files that can be inspected with ordinary tools.

Luca Foppiano

Luca Foppiano is a Senior Engineer at the Common Crawl Foundation.

Common Crawl maintains a free, open repository of web crawl data that can be used by anyone.

Common Crawl is a 501(c)(3) non–profit founded in 2007.‍We make wholesale extraction, transformation, and analysis of open web data accessible to researchers.

Over 300 billion pages spanning 15 years.

Free and open corpus since 2007.

Cited in over 10,000 research papers.

3–5 billion new pages added each month.

Featured Papers

Adam Visokay et al.

The uneven rise of AI-generated web content

Nathan Godey, et al.

Gaperon: A Peppered English-French Generative Language Model Suite

Sheriff Issaka, et al.

The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP

Badal Nyalang

Stereotyped by Silence: How LLMs Erase Northeast Indian Languages Through Omission and Orthographic Corruption

Xiangyu Wu, et al.

MirrorCAPTCHA: Wild CAPTCHA, Wild Distribution, Wild Web-based Platform Meet Multimodal LLM Agents

Michael Paris, Hande Celikkanat, Luca Foppiano

Measuring What the Crawler Sees: Discovery Curves, Core Persistence, and Shell Dynamics in Longitudinal Web Crawls

Julio Garbers, Terry Gregory

The Diffusion of Artificial Intelligence Across Firms: Evidence from Europe

Pedro Ortiz Suarez, et al.

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Latest Blog Post

Introducing warc2zip

The Data

Resources

Community

About

Common Crawl is a 501(c)(3) non–profit founded in 2007.
‍
We make wholesale extraction, transformation, and analysis of open web data accessible to researchers.