< Back to Blog
May 7, 2018

Host- and Domain-Level Web Graphs Feb/Mar/Apr 2018

Note: this post has been marked as obsolete.
We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of February, March and April 2018. Additional information about data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Sebastian Nagel
Sebastian Nagel
Sebastian is a Distinguished Engineer with Common Crawl.

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of February, March and April 2018. Additional information about data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs).

What's new?

The graphs now contain links from sitemap announcements in robots.txt files. This small addition of 2.5 million inter-host links is motivated by the fact that sitemaps directives are sometimes (see example 1, 2, 3) used for link spam or aggressive SEO, often in combination with excessive use of inter-host hyperlinks on HTML pages. We hope that this addition helps to improve the detection rate of link spam detection algorithms.

Host-level graph

The graph consists of 2.14 billion nodes and 10.15 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 2.02 billion dangling nodes (94%) and the largest strongly connected component contains only 77 million (3.6%) nodes. The host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 2 billion hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2018-feb-mar-apr/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2018-feb-mar-apr/host/ as prefix to access the files from everywhere.

The following files and formats are provided:

Download files of the Common Crawl Feb/Mar/Apr 2018 host-level Webgraph

Size File Description
12.45 GB cc-main-2018-feb-mar-apr-host-vertices.paths.gz nodes ⟨id, rev host⟩, paths of 80 vertices files
50.22 GB cc-main-2018-feb-mar-apr-host-edges.paths.gz edges ⟨from_id, to_id⟩, paths of 160 edges files
20.68 GB cc-main-2018-feb-mar-apr-host.graph graph in BVGraph format
2 kB cc-main-2018-feb-mar-apr-host.properties
24.82 GB cc-main-2018-feb-mar-apr-host-t.graph transpose of the graph (outlinks inverted to inlinks)
2 kB cc-main-2018-feb-mar-apr-host-t.properties
1 kB cc-main-2018-feb-mar-apr-host.stats WebGraph statistics
28.84 GB cc-main-2018-feb-mar-apr-host-ranks.txt.gz harmonic centrality and pagerank

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 98 million nodes and 1.5 billion edges. 57% or 56 million nodes are dangling nodes, the largest strongly connected component covers 37 million or 38% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2018-feb-mar-apr/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2018-feb-mar-apr/domain/.

Download files of the Common Crawl Feb/Mar/Apr 2018 domain-level Webgraph

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl's Google Group!

This release was authored by:
No items found.