We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September and October 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.
What's new?
The following improvements have been made for this webgraph release:
- the graphs now also included edges stemming from HTTP 303 "See Other" redirects (in addition to other HTTP redirect status codes)
- the Common Crawl robots.txt WARC files are used to get additional host-level redirects including hosts which exclude the entire content in their robots.txt
- links from robots.txt files to sitemaps are now extracted directly from the robots.txt WARC files, see the Feb/Mar/Apr 2018 web graph announcement for more details about this type of host-level links
Host-level graph
The graph consists of 820 million nodes and 4.55 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 752 million dangling nodes (92%) and the largest strongly connected component contains 50 million (6%) nodes.
You can download the graph and the ranks of all 820 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/host/ as prefix to access the files from everywhere.
Domain-level graph
The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.
The domain-level graph has 92.7 million nodes and 2.4 billion edges. 52% or 48 million nodes are dangling nodes, the largest strongly connected component covers 36 million or 40% of the nodes.
All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/domain/.
Download files of the Common Crawl Aug/Sep/Oct 2019 domain-level Webgraph
Credits
Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible. We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl's Google Group!
.svg)
