Introducing CloudFront as a new way to access Common Crawl data as part of Amazon Web Services’ registry of open data

Ten years ago(!) Common Crawl joined AWS’s Open Data Sponsorships program, hosted on S3, with free access to everyone. Since then, the dataset has expanded (by petabytes!) and our community of users has seen extraordinary growth. With growth comes change, therefore…Following are new measures to accommodate the number and volume of requests for Common Crawl data, and to ensure efficient and stable access to the data:

‍

1. Common Crawl data is now available on CloudFront under the URL base prefixes*:

• https://data.commoncrawl.org/ or
• https://ds5q9oxwqwsfj.cloudfront.net/

* Data is accessible via http:// or https:// as of Monday, April 4, 2022.

‍

‍2. Access to data from the Amazon cloud using the S3 API will be restricted to authenticated AWS users, and unsigned access to s3://commoncrawl/ will be disabled. See Q&A for further details.

‍

3. To access data from outside the Amazon cloud, via HTTP(S), a new URL prefix – https://data.commoncrawl.org/ – must be used instead of https://commoncrawl.s3.amazonaws.com/.

Q & A

Q: How can I identify whether my code is using unauthenticated S3 access?

A: Authenticated access is the default, but you should verify that your code is not configured otherwise; code fragments requesting unauthenticated access could be (but are not limited to):

• AWS CLI with the command-line option --no-sign-request:

aws --no-sign-request s3 cp s3://commoncrawl/...

• Python using boto3 and botocore.UNSIGNED:

import boto3
import botocore
s3client = boto3.client('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED))

• Hadoop or Spark (various programming languages): usage of AnonymousAWSCredentialsProvider

‍

Q: Where can I find up-to-date documentation and examples that are consistent with these new protocols?

A: We will update our documentation and examples to read data either via HTTP(S) using the new CloudFront access, or via S3 and authenticated access. We will also update all data download links. It will take a few days until everything is updated. Thanks for your patience!

‍

Q: Are range requests supported?

A: Yes, range requests are supported by CloudFront same as for S3, see the CloudFront docs about RangeGETs.

‍

Q: What is the recommended access method on AWS but in a different region (not us-east-1)?

A: We recommend that you run your computing workload in the same region (us-east-1) as the Common Crawl dataset whenever possible. If you have a specific, ongoing need to run computing workloads using Common Crawl in another AWS region, the AWS Open Data team would like to hear more about your use case (opendata@amazon.com).

‍

Q: What are typical error messages indicating that unauthenticated access is used?

A: The HTTP response status code is a 403 Forbidden. However, also a restrictive IAM policy on the user's side could deny access to s3://commoncrawl/ using the S3 API.

Two examples for error messages related to unauthenticated access to s3://commoncrawl/:

> curl https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2022-05/warc.paths.gz
<!--?xml version="1.0" encoding="UTF-8"?-->
<error><code>AccessDenied</code><message>Access Denied</message>...</error>

> aws --no-sign-request s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2022-05/warc.paths.gz .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden