We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June/July and August 2022. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks. See below for a summary of changes and improvements implemented for the current web graph release.
Changes, improvements and bug fixes
- Unicode internationalized domain names are always converted into their ASCII equivalents (IDNA). This is now ensured for node labels in the host-level webgraph (see cc-pyspark#35) and consequently also for the domain-level webgraph where non-ASCII characters were replaced by question marks (see cc-webgraph#6)
- The nodes of the domain graph are now strictly sorted lexicographically by node label (the reverse domain name). This should allow for more efficient compression of the list of domain nodes.
- The strict sorting was implemented to address a bug (cc-webgraph#3) which may cause duplicated nodes (two or more nodes with the same label) in the domain graph.
- The domain graph includes domain names equal to multi-part public suffixes. Previously the assumption was that names of registered domains are exactly one level below any ICANN suffix in the public suffix list and host names which are equal to multi-part suffixes (including at least one dot) were excluded. Such host names are now included, eg. gov.uk, freight.aero or altoadige.it. No further validation (eg. DNS lookup) is performed, so also invalid domain names may be included. Generally, except for a valid domain name string with a valid TLD or public suffix, no further validation is performed for any domain name. For more details, see cc-webgraph#1.
Host-level graph
The graph consists of 449 million nodes and 2.69 billion edges. Both hyperlinks and HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including pure "technical" ones pointing to images, JavaScript libraries, web fonts, etc. However, only host names with a valid IANA TLD are used. Consequently, URLs with an IP address as host component are not taken into account for building the host-level graph.
There are 389 million dangling nodes (86.6%) and the largest strongly connected component contains 46.4 million (10.3%) nodes. Dangling nodes stem from
- hosts that have not been crawled, yet are pointed to from a link on a crawled page
- hosts without any links pointing to a different host name
- or hosts which did only return an error page (eg. HTTP 404)
Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.
You can download the graph and the ranks of all 449 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/host/ (this requires an account on AWS). Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/host/ as prefix to access the files from everywhere.
Please note that the text representation of the host-level graph is shipped in 96 gzip-compressed files listed in two path listings - one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using "gzip -d" or "gunzip". By adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing you get the list of URLs to download the entire graph.
Download files of the Common Crawl May/Jun/Aug 2022 host-level Webgraph
Domain-level graph
The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) e5ff0c7 of the public suffix list was used (commit date 2022-09-15).
The domain-level graph has 91 million nodes and 1.57 billion edges. 50% or 45 million nodes are dangling nodes, the largest strongly connected component covers 37 million or 40% of the nodes.
All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/ or on https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/.
Download files of the Common Crawl May/Jun/Aug 2022 domain-level Webgraph
Credits
Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible. We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl's Google Group!
.svg)
