August Crawl Archive Introduces Language Annotations

The crawl archive for August 2018 is now available! It contains 2.65 billion web pages and 220 TiB of uncompressed content, crawled between August 14th and 22th.

‍

Together with an upgrade of the crawler software we've plugged in a language detector and now provide as annotation the language a web page is written in.

‍Please note that the WARC files of August 2018 (CC-MAIN-2018-34) are affected by a WARC format error and contain an extra \r\n between HTTP header and payload content. Also the given "Content-Length" is off by 2 bytes. For more information about this bug see this post on our user forum.

Language Annotations

We now run the Compact Language Detector 2 (CLD2) on HTML pages to identify the language of a document. CLD2 is able to identify 160 different languages and up to 3 languages per document. The detected languages resp. the ISO-639-3 code are shown in the URL index as a new field, e.g., "languages": "zho,eng". The WARC metadata records contain the full CLD2 response including scores and text coverage:
‍

‍
On github you'll find the Java bindings to the CLD2 native library and the distribution of the primary document languages as part of our crawl statistics. Please note that the columnar index does not contain the detected languages for now. This requires a change of the table schema. We plan to add the new fields later after we've verified that an update of the schema does not break common tools (e.g., Spark or Presto/Athena) used to process the table.

Crawler Software Upgrade and Minor Changes to WARC Files

Our crawler has been upgraded and is now based on the most recent version of Apache Nutch (1.15). The source code can be found on github in our Nutch fork.

In conjunction with the crawler upgrade we made the following minor changes affecting the WARC record format of the crawl archives:

• "HTTP 304 notmodified" responses are now stored as WARC revisit records in the "crawldiagnostics" subset along with 404s, redirects and other non-200 responses. For now the revisit records contain a payload digest although there is no payload sent together with HTTP 304 responses.  The stupid reason is that the columnar index requires the digest field and we want to make sure that all tools continue to work as expected. The SHA-1 digest of an empty payload (zero bytes) is used for the revisit records.
• All HTTP response headers are now preserved. As before, if the page content is truncated or was compressed or chunked during transfer, the headers "Content-Encoding", "Transfer-Encoding" and "Content-Length" need to be rewritten, otherwise WARC readers may fail reading the record payload. E.g., a page compressed on the HTTP protocol layer may have the following headers – the original headers are prefixed with X-Crawler-:

    X-Crawler-Content-Encoding: gzip
    X-Crawler-Content-Length: 2010
    Content-Length: 16125

• The crawler may now also store pages fetched partially because of a network disconnect. These captures are marked as WARC-Truncated: disconnect in the WARC record header. Note that the crawler may also truncate the page payload because of a content limit (we store only 1 MB per page) or a time limit (after 10 minutes a page download is canceled).
• the WARC record headers indicate still "WARC/1.0" although we follow the WARC specification, v1.1. While testing various WARC reader libraries we've found that at least two of them fail on records with a "WARC/1.1" header.

‍

Please note that due to a bug the first two crawled segments are without robots.txt captures.

Archive Location and Download

The August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2018-34/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files. By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2018-34/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact info@commoncrawl.org for sponsorship information.