Web Archiving File Formats Explained
March 1, 2024
In the ever–evolving landscape of digital archiving and data analysis, it is helpful to understand the various file formats used for web crawling. From the early ARC format to the more advanced WARC, and the specialised WET and WAT files, each plays an important role in the field of web archiving. In this post, we explain these formats, exploring their unique features, applications, and the enhancements they offer.
Read More...A Further Look Into the Prevalence of Various ML Opt–Out Protocols
February 22, 2024
This post details some experiments that we have done regarding Machine Learning Opt–Out protocols. We decided to investigate the prevalence of some of these protocols, by taking a deeper look at our WARC files, and finding which proportions of domains are using which opt–out protocols.
Read More...Balancing Discovery and Privacy: A Look Into Opt–Out Protocols
February 13, 2024
What opt–out protocols are, their importance, how you can use them, how we respect them, and what the emerging initiatives are that surround them.
Read More...Host- and Domain-Level Web Graphs May/Sep/Nov 2023
December 22, 2023
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, September, and November of 2023.
Read More...November/December 2023 Crawl Archive Now Available
December 15, 2023
The crawl archive for November/December 2023 is now available. The data was crawled between November 28th and December 12th, and contains 3.35 billion web pages (or 454 TiB of uncompressed content).
Read More...Oct/Nov 2023 Performance Issues
November 15, 2023
Our datasets have become very popular over time, with downloads doubling every 6 months for several years in a row. This post details some steps to take if you are impacted by performance issues.
Read More...Host- and Domain-Level Web Graphs Mar/May/Oct 2023
October 18, 2023
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, May, and October 2023. The host-level graph consists of 378.7 million nodes and 2.6 billion edges, and the domain-level graph has 94.2 million nodes and 1.7 billion edges.
Read More...September/October 2023 crawl archive now available
October 12, 2023
The crawl archive for September/October 2023 is now available! The data was crawled Sept 21 – October 5 and contains 3.4 billion web pages or 456 TiB of uncompressed content.
Read More...Bridging Digital Exploration and Scientific Frontiers
October 10, 2023
This month Common Crawl Foundation members had the privilege of attending 5th International Open Search Symposium at CERN in Geneva, Switzerland.
Read More...May/June 2023 crawl archive now available
June 21, 2023
The crawl archive for May/June 2023 is now available! The data was crawled May 27 – June 11 and contains 3.1 billion web pages or 390 TiB of uncompressed content. Page captures are from 44 million hosts or 35 million registered domains and include 1.0 billion new URLs, not visited in any of our prior crawls.
Read More...March/April 2023 crawl archive now available
April 6, 2023
The crawl archive for March/April 2023 is now available! The data was crawled March 20 – April 2 and contains 3.1 billion web pages or 400 TiB of uncompressed content. Page captures are from 43 million hosts or 34 million registered domains and include 1.2 billion new URLs, not visited in any of our prior crawls.
Read More...Host- and Domain-Level Web Graphs September/October, November/December 2022 and January/February 2023
March 15, 2023
We are pleased to announce a new release of host-level and domain-level web graphs based on the September/October, November/December 2022 and January/February 2023 crawls. For more information about the data formats and the processing pipeline, please see the announcements of previous webgraph releases.
Read More...January/February 2023 crawl archive now available
February 16, 2023
The crawl archive for January/February 2023 is now available! The data was crawled January 26 – February 9 and contains 3.15 billion web pages or 400 TiB of uncompressed content. Page captures are from 40 million hosts or 33 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls.
Read More...November/December 2022 crawl archive now available
December 14, 2022
The crawl archive for November/December 2022 is now available! The data was crawled November 26 – December 10 and contains 3.35 billion web pages or 420 TiB of uncompressed content. Page captures are from 44 million hosts or 34 million registered domains and include 1.2 billion new URLs, not visited in any of our prior crawls.
Read More...September/October 2022 crawl archive now available
October 11, 2022
The crawl archive for September/October 2022 is now available! The data was crawled September 24 – October 8 and contains 3.15 billion web pages or 380 TiB of uncompressed content. Page captures are from 44 million hosts or 34 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls. This crawl includes improvements made in extracting clean text in WET files and WAT anchor texts.
Read More...Host- and Domain-Level Web Graphs May, June/July and August 2022
September 23, 2022
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June/July and August 2022. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Read More...August 2022 crawl archive now available
August 22, 2022
The crawl archive for August 2022 is now available! The data was crawled August 7 – 20 and contains 2.55 billion web pages or 295 TiB of uncompressed content. Page captures are from 46 million hosts or 37 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls.
Read More...June/July 2022 crawl archive now available
July 13, 2022
The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content. Page captures are from 44 million hosts or 35 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.
Read More...May 2022 crawl archive now available
June 2, 2022
The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content. Page captures are from 45 million hosts or 36 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.
Read More...Host- and Domain-Level Web Graphs October, November/December 2021 and January 2022
March 16, 2022
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2021 and January 2022. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Read More...Introducing CloudFront as a new way to access Common Crawl data as part of Amazon Web Services’ registry of open data
March 1, 2022
Ten years ago(!) Common Crawl joined AWS’s Open Data Sponsorships program, hosted on S3, with free access to everyone. Since then, the dataset has expanded (by petabytes!) and our community of users has seen extraordinary growth.
Read More...January 2022 crawl archive now available
February 2, 2022
The crawl archive for January 2022 is now available! The data was crawled January 16 – 29 and contains 2.95 billion web pages or 320 TiB of uncompressed content. It includes page captures of 1.35 billion new URLs, not visited in any of our prior crawls.
Read More...November/December 2021 crawl archive now available
December 14, 2021
The crawl archive for November/December 2021 is now available! The data was crawled Nov 26 – Dec 9 and contains 2.5 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls.
Read More...October 2021 crawl archive now available
November 1, 2021
The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.
Read More...Host- and Domain-Level Web Graphs June, July/August and September 2021
October 8, 2021
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of June, July/August and September 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Read More...
.svg)
