Hyperlink Graph from Web Data Commons
November 13, 2013
The talented team at Web Data Commons recently extracted and analyzed the hyperlink graph within the Common Crawl 2012 corpus. Altogether, they found 128 billion hyperlinks connecting 3.5 billion pages.
Read More...Startup Profile: SwiftKey’s Head Data Scientist on the Value of Common Crawl’s Open Data
August 14, 2013
Sebastian Spiegler is the head of the data team and SwiftKey and a volunteer at Common Crawl. Yesterday we posted Sebastian’s statistical analysis of the 2012 Common Crawl corpus. Today we are following it up with a great video featuring Sebastian talking about why crawl data is valuable, his research, and why open data is important.
Read More...A Look Inside Our 210TB 2012 Web Corpus
August 13, 2013
Want to know more detail about what data is in the 2012 Common Crawl corpus without running a job? Now you can thanks to Sebastian Spiegler!
Read More...Professor Jim Hendler Joins the Common Crawl Advisory Board!
March 22, 2013
We are extremely happy to announce that Professor Jim Hendler has joined the Common Crawl Advisory Board. Professor Hendler is the Head of the Computer Science Department at Rensselaer Polytechnic Institute (RPI) and also serves as the Professor of Computer and Cognitive Science at RPI’s Tetherless World Constellation.
Read More...URL Search Tool!
March 5, 2013
A couple months ago we announced the creation of the Common Crawl URL Index and followed it up with a guest post by Jason Ronallo describing how he had used the URL Index. Today we are happy to announce a tool that makes it even easier for you to take advantage of the URL Index!
Read More...The Winners of The Norvig Web Data Science Award
February 25, 2013
We are very excited to announce that the winners of the Norvig Web Data Science Award Lesley Wevers, Oliver Jundt, and Wanno Drijfhout from the University of Twente!
Read More...Analysis of the NCSU Library URLs in the Common Crawl Index
January 16, 2013
Last week we announced the Common Crawl URL Index. The index has already proven useful to many people and we would like to share an interesting use of the index that was very well described in a great blog post by Jason Ronallo.
Read More...Common Crawl URL Index
January 8, 2013
We are thrilled to announce that Common Crawl now has a URL index! Scott Robertson, founder of triv.io graciously donated his time and skills to creating this valuable tool.
Read More...blekko donates search data to Common Crawl
December 17, 2012
We are very excited to announce that blekko is donating search data to Common Crawl! Founded in 2007, blekko has created a new type of search experience that enlists human editors in its efforts to eliminate spam and personalize search.
Read More...The Norvig Web Data Science Award
November 15, 2012
We are very excited to announce the Norvig Web Data Science Award! Common Crawl and SARA created the award to encourage research in web data science.
Read More...Towards Social Discovery - New Content Models; New Data; New Toolsets
October 30, 2012
This is a guest blog post by Matthew Berk, Founder of Lucky Oyster. Matthew has been on the front lines of search technology for the past decade.
Read More...Winners of the Code Contest!
September 18, 2012
We’re very excited to announce the winners of the First Ever Common Crawl Code Contest! We were thrilled by the response to the contest and the many great entries. Several people let us know that they were not able to complete their project in time to submit to the contest. We’re currently working with them to finish the projects outside of the contest and we’ll be showcasing some of those projects in the near future!
Read More...Common Crawl Code Contest Extended Through the Holiday Weekend
August 30, 2012
Do you have a project that you are working on for the Common Crawl Code Contest that is not quite ready? If so, you are not the only one. A few people have emailed us to let us know their code is almost ready but they are worried about the deadline, so we have decided to extend the deadline through the holiday weekend.
Read More...TalentBin Adds Prizes To The Code Contest
August 14, 2012
The prize package for the Common Crawl Code Contest now includes three Nexus 7 tablets thanks to TalentBin!
Read More...Amazon Web Services sponsoring $50 in credit to all contest entrants!
August 10, 2012
Did you know that every entry to the First Ever Common Crawl Code Contest gets $50 in Amazon Web Services (AWS) credits? If you're a developer interested in big datasets and learning new platforms like Hadoop, you truly have no reason not to try your hand at creating an entry to the code contest!
Read More...Still time to participate in the Common Crawl code contest
August 7, 2012
There is still plenty of time left to participate in the Common Crawl code contest! The contest is accepting entries until August 30th, why not spend some time this week playing around with the Common Crawl corpus and then submit your work to the contest?
Read More...Strata Conference + Hadoop World
August 3, 2012
This year's Strata Conference teams up with Hadoop World for what promises to be a powerhouse convening in NYC from October 23-25. Check out their full announcement below and secure your spot today.
Read More...Mat Kelcey Joins The Common Crawl Advisory Board
August 1, 2012
We are excited to announce that Mat Kelcey has joined the Common Crawl Board of Advisors! Mat has been extremely helpful to Common Crawl over the last several months and we are very happy to have him as an official Advisor to the organization.
Read More...Common Crawl's Brand Spanking New Video and First Ever Code Contest!
July 18, 2012
At Common Crawl we've been busy recently! After announcing the release of 2012 data and other enhancements, we are now excited to share with you this short video that explains why we here at Common Crawl are working hard to bring web crawl data to anyone who wants to use it.
Read More...2012 Crawl Data Now Available
July 16, 2012
I am very happy to announce that Common Crawl has released 2012 crawl data as well as a number of significant enhancements to our example library and help pages.
Read More...The Open Cloud Consortium’s Open Science Data Cloud
July 3, 2012
Common Crawl has started talking with the Open Cloud Consortium (OCC) about working together. If you haven’t already heard of the OCC, it is an awesome nonprofit organization managing and operating cloud computing infrastructure that supports scientific, environmental, medical and health care research.
Read More...OSCON 2012
June 19, 2012
We're just one month away from one of the biggest and most exciting events of the year, O'Reilly's Open Source Convention (OSCON). This year's conference will be held July 16th-20th in Portland, Oregon.
Read More...Learn Hadoop and get a paper published
May 9, 2012
We're looking for students who want to try out the Apache Hadoop platform and get a technical report published.
Read More...Big Data Week: meetups in SF and around the world
April 13, 2012
Big Data Week aims to connect data enthusiasts, technologists, and professionals across the globe through a series of meet-ups. The idea is to build community among groups working on big data and to spur conversations about relevant topics ranging from technology to commercial use cases.
Read More...Data 2.0 Summit
March 28, 2012
Next week a few members of the Common Crawl team are going the Data 2.0 Summit in San Francisco.
Read More...
.svg)
