The crawl archive for February 2025 is now available.
The data was crawled between February 6th and February 20th, and contains 2.6 billion web pages (or 402 TiB of uncompressed content). Page captures are from 47.6 million hosts or 38.5 million registered domains and include 1 billion new URLs, not visited in any of our prior crawls.
Archive Location & Download
The February 2025 crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2025-08/.
To assist with exploring and using the dataset, we provide gzip compressed files which list all segments, WARC, WAT, and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively, please see Get Started for detailed instructions.
Let us know about your results via Common Crawl's Google Group, or on our Discord server.
Erratum:
Redirect target URL in URL indexes may be a relative URL
When the HTTP “Location” header includes a relative URL, the corresponding “redirect” field in the CDX index and “fetch_redirect” field in the columnar index will also store a relative URL. In all other cases, redirect targets in the URL indexes should be recorded as absolute URLs.
Additional information about this issue and the fix is available in this GitHub issue.
Erratum:
Content is truncated
Some archived content is truncated due to fetch size limits imposed during crawling. This is necessary to handle infinite or exceptionally large data streams (e.g., radio streams). Prior to March 2025 (CC-MAIN-2025-13), the truncation threshold was 1 MiB. From the March 2025 crawl onwards, this limit has been increased to 5 MiB.
For more details, see our truncation analysis notebook.
Erratum:
Nodes in Domain-Level Webgraphs Not Sorted and May Include Duplicates
The nodes in domain-level Web Graphs may not be properly sorted lexicographically by node label (reversed domain name). It's also possible that few nodes are duplicated, that is two nodes share the same label. For more details, see the Issue Report in the cc-webgraph repository.
The issue affects all domain-level Web Graphs until the issue has been fixed for the May, June/July, August 2022 Web Graph (cc-main-2022-may-jun-aug-domain) and the following Web Graph releases.
.svg)
