Explore @commoncrawl Open Source projects

Common Crawl (@commoncrawl)

commoncrawl

Stars
2,284
Global Org. Rank 8,605 (Top 3 %)
Registered almost 13 years ago
Most used languages

Java
43.8 %

Python
31.3 %

Jupyter Notebook
12.5 %

Shell
6.3 %

C++
6.3 %

commoncrawl

Common Crawl support library to access 2008-2012 crawl archives (ARC files)

cc-pyspark

Process Common Crawl data with Python and Spark

news-crawl

News crawling with StormCrawler - stores content as WARC

commoncrawl-crawler

The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)

cc-crawl-statistics

Statistics of Common Crawl monthly archives mined from URL index files

cc-index-table

Index Common Crawl archives in tabular format

cc-webgraph

Tools to construct and process webgraphs from Common Crawl data

commoncrawl-examples

A library of examples showing how to use the Common Crawl corpus (2008-2012, ARC format)

example-warc-java

cc-notebooks

Various Jupyter notebooks about Common Crawl data

Jupyter Notebook

language-detection-cld2

Natural language detection, Java bindings for CLD2

cc-citations

Scientific articles using or citing Common Crawl data

Jupyter Notebook

ml-opt-out-experiments

A series of experiments into ML opt–out protocols

cc-nutch-example

Apache Nutch example project to archive content in WARC files

whirlwind-python

A whilrlwind tour of Common Crawl's data using Python

cc-monitoring

Code that monitors Common Crawl infrastructure

cc-legal

Repository for legal documentation at the Common Crawl Foundation