November/December 2020 crawl archive now available
December 10, 2020
The crawl archive for November/December 2020 is now available! The data was crawled between November 23 and December 6 and contains 2.64 billion web pages or 270 TiB of uncompressed content. It includes page captures of 1.4 billion new URLs, not visited in any of our prior crawls.
Read More...October 2020 crawl archive now available
November 7, 2020
The crawl archive for October 2020 is now available! The data was crawled between October 19th and November 1st and contains 2.71 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.5 billion new URLs, not visited in any of our prior crawls.
Read More...Interactive Webgraph Statistics Notebook Released
October 28, 2020
We are pleased to announce the release of an interactive Jupyter notebook that is used to provide visualization of webgraph statistics, and a way to interact with the webgraph.
Read More...Host- and Domain-Level Web Graphs Jul/Aug/Sep 2020
October 16, 2020
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Read More...September 2020 crawl archive now available
October 7, 2020
The crawl archive for September 2020 is now available! The data was crawled between September 18th and October 2nd and contains 3.45 billion web pages or 345 TiB of uncompressed content. It includes page captures of 1.5 billion new URLs, not visited in any of our prior crawls.
Read More...August 2020 crawl archive now available
August 19, 2020
The crawl archive for August 2020 is now available! It contains 2.45 billion web pages or 235 TiB of uncompressed content, crawled between August 2nd and 15th. It includes page captures of 940 million URLs unknown in any of our prior crawl archives.
Read More...July 2020 crawl archive now available
July 20, 2020
The crawl archive for July 2020 is now available! It contains 3.14 billion web pages or 300 TiB of uncompressed content, crawled between July 2nd and 16th. It includes page captures of 1.1 billion URLs unknown in any of our prior crawl archives.
Read More...Host- and Domain-Level Web Graphs Feb/Mar/May 2020
June 16, 2020
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, March/April and May/June 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Read More...May/June 2020 crawl archive now available
June 10, 2020
The crawl archive for May/June 2020 is now available! It contains 2.75 billion web pages or 255 TiB of uncompressed content, crawled between May 24th and June 7th. It includes page captures of 1.2 billion URLs unknown in any of our prior crawl archives.
Read More...March/April 2020 crawl archive now available
April 14, 2020
The crawl archive for March/April 2020 is now available! It contains 2.85 billion web pages or 280 TiB of uncompressed content, crawled between March 28th and April 10th. It includes page captures of 1 billion URLs unknown in any of our prior crawl archives.
Read More...February 2020 crawl archive now available
March 4, 2020
The crawl archive for February 2020 is now available! It contains 2.6 billion web pages or 240 TiB of uncompressed content, crawled between February 16th and 29th. It includes page captures of 1 billion URLs unknown in any of our prior crawl archives.
Read More...Host- and Domain-Level Web Graphs Nov/Dec/Jan 2019 – 2020
February 10, 2020
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2019 and January 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Read More...January 2020 crawl archive now available
February 3, 2020
The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. It includes page captures of 960 million URLs not contained in any crawl archive before.
Read More...December 2019 crawl archive now available
December 19, 2019
The crawl archive for December 2019 is now available! It contains 2.45 billion web pages or 234 TiB of uncompressed content, crawled between December 5th and 16th. It includes page captures of 850 million URLs not contained in any crawl archive before.
Read More...November 2019 crawl archive now available
November 27, 2019
The crawl archive for November 2019 is now available! It contains 2.55 billion web pages or 250 TiB of uncompressed content, crawled between November 11th and 23rd with a short operational break on Nov 16th. It includes page captures of 1.1 billion URLs not contained in any crawl archive before.
Read More...Host- and Domain-Level Web Graphs Aug/Sep/Oct 2019
November 12, 2019
We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September and October 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Read More...October 2019 crawl archive now available
October 29, 2019
The crawl archive for October 2019 is now available! It contains 3.0 billion web pages or 280 TiB of uncompressed content, crawled between October 13th and 24th. It includes page captures of 1.1 billion URLs not contained in any crawl archive before.
Read More...September 2019 crawl archive now available
September 28, 2019
The crawl archive for September 2019 is now available! It contains 2.55 billion web pages or 240 TiB of uncompressed content, crawled between September 15th and 24th. It includes page captures of 1.0 billion URLs not contained in any crawl archive before. The other 1.5 billion pages have been already captured in prior crawls and are now revisited.
Read More...August 2019 crawl archive now available
August 30, 2019
The crawl archive for August 2019 is now available! It contains 2.95 billion web pages or 260 TiB of uncompressed content, crawled between August 17th and 26th.
Read More...Host- and Domain-Level Web Graphs May/June/July 2019
August 8, 2019
We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of May, June and July 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Read More...July 2019 crawl archive now available
July 30, 2019
The crawl archive for July 2019 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between July 15th and 24th.
Read More...June 2019 crawl archive now available
July 2, 2019
The crawl archive for June 2019 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between June 16th and 27th with an operational break from 21st to 24th.
Read More...May 2019 crawl archive now available
May 31, 2019
The crawl archive for May 2019 is now available! It contains 2.65 billion web pages or 220 TiB of uncompressed content, crawled between May 19th and 27th.
Read More...Host- and Domain-Level Web Graphs Feb/Mar/Apr 2019
May 9, 2019
We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of February, March and April 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Read More...April 2019 crawl archive now available
April 30, 2019
The crawl archive for April 2019 is now available! It contains 2.5 billion web pages or 198 TiB of uncompressed content, crawled between April 18th and 26th.
Read More...