December 2017 Crawl Archive Now Available
December 22, 2017
The crawl archive for December 2017 is now available! The archive contains 2.9 billion web pages and over 240 TiB of uncompressed content.
Read More...November 2017 Crawl Archive Now Available
November 29, 2017
The crawl archive for November 2017 is now available! The archive contains 3.2 billion web pages and 260 TiB of uncompressed content.
Read More...Host- and Domain-Level Web Graphs Aug/Sept/Oct 2017
November 27, 2017
We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September, and October 2017. These graphs, along with ranked lists of hosts and domains, follow the first (February, March, April 2017) and second (May, June, July 2017) web graph releases.
Read More...October 2017 Crawl Archive Now Available
October 29, 2017
The crawl archive for October 2017 is now available! The archive contains 3.65 billion web pages and over 300 TiB of uncompressed content.
Read More...September 2017 Crawl Archive Now Available
September 29, 2017
The crawl archive for September 2017 is now available! The archive contains 3.01 billion web pages and over 250 TiB of uncompressed content.
Read More...August 2017 Crawl Archive Now Available
August 28, 2017
The crawl archive for August 2017 is now available! The archive contains 3.28 billion+ web pages and over 280 TiB of uncompressed content.
Read More...Now Available: Host- and Domain-Level Web Graphs
August 18, 2017
We are pleased to announce the release of host-level and domain-level web graphs based on the published crawls of May, June, and July 2017. These graphs, along with ranked lists of hosts and domains, follow on our first host-level web graph (February, March, April 2017).
Read More...July 2017 Crawl Archive Now Available
July 31, 2017
The crawl archive for July 2017 is now available! The archive contains 2.89 billion+ web pages and over 240 TiB of uncompressed content.
Read More...June 2017 Crawl Archive Now Available
July 4, 2017
The crawl archive for June 2017 is now available! The archive contains 3.16 billion+ web pages and over 260 TiB of uncompressed content.
Read More...May 2017 Crawl Archive Now Available
June 5, 2017
The crawl archive for May 2017 is now available! The archive contains 2.96 billion+ web pages and over 250 TiB of uncompressed content.
Read More...Common Crawl's First In-House Web Graph
May 22, 2017
We are pleased to announce the release of a host-level web graph of recent monthly crawls (February, March, April 2017). The graph consists of 385 million nodes and 2.5 billion edges.
Read More...April 2017 Crawl Archive Now Available
May 9, 2017
The crawl archive for April 2017 is now available! The archive contains 2.94 billion+ web pages and over 250 TiB of uncompressed content.
Read More...March 2017 Crawl Archive Now Available
April 4, 2017
The crawl archive for March 2017 is now available! The archive contains 3.07 billion+ web pages and over 250 TiB of uncompressed content.
Read More...February 2017 Crawl Archive Now Available
March 10, 2017
The crawl archive for February 2017 is now available! The archive contains 3.08 billion+ web pages and over 250 TiB of uncompressed content.
Read More...January 2017 Crawl Archive Now Available
February 1, 2017
The crawl archive for January 2017 is now available! The archive contains more than 3.14 billion web pages and about 250 TiB of uncompressed content.
Read More...December 2016 Crawl Archive Now Available
December 16, 2016
The crawl archive for December 2016 is now available! The archive contains more than 2.85 billion web pages.
Read More...October 2016 Crawl Archive Now Available
November 7, 2016
The crawl archive for October 2016 is now available! The archive contains more than 3.25 billion web pages.
Read More...September 2016 Crawl Archive Now Available
October 7, 2016
The crawl archive for September 2016 is now available! The archive contains more than 1.72 billion web pages.
Read More...News Dataset Available
October 4, 2016
We are pleased to announce the release of a new dataset containing news articles from news sites all over the world.
Read More...Data Sets Containing Robots.txt Files and Non-200 Responses
September 16, 2016
Together with the crawl archive for August 2016 we release two data sets containing robots.txt files and server responses with HTTP status code other than 200 (404s, redirects, etc.) The data may be useful to anyone interested in web science, with various applications in the field.
Read More...August 2016 Crawl Archive Now Available
September 16, 2016
The crawl archive for August 2016 is now available! The archive contains more than 1.61 billion web pages.
Read More...July 2016 Crawl Archive Now Available
August 9, 2016
The crawl archive for July 2016 is now available! The archive contains more than 1.73 billion web pages.
Read More...June 2016 Crawl Archive Now Available
July 14, 2016
The crawl archive for June 2016 is now available! The archive contains more than 1.23 billion web pages.
Read More...May 2016 Crawl Archive Now Available
June 19, 2016
The crawl archive for May 2016 is now available! More than 1.46 billion web pages are in the archive.
Read More...April 2016 Crawl Archive Now Available
May 24, 2016
The crawl archive for April 2016 is now available! More than 1.33 billion webpages are in the archive.
Read More...
.svg)
