DigitalPebble/behemoth-commoncrawl

This repository has been archived on 10/Jul/2019
Stars
4
Rank 3,304,323 (Top 66 %)
Language
Java
Created over 12 years ago
Updated over 9 years ago

DigitalPebble/behemoth-commoncrawl

DigitalPebble

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Support for old (pre 2013) CommonCrawl dataset in Behemoth

storm-crawler

A scalable, mature and versatile web crawler based on Apache Storm

behemoth

Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.

TextClassification

A Text Classification API in Java originally developed by DigitalPebble Ltd. The API is independent from the ML implementations used and can be used as a front end to various ML algorithms. libSVM and liblinear are currently embedded.

textclassification-examples

Use cases for DigitalPebble's TextClassification API

stormcrawlerfight

Crawl configurations for benchmarking / testing StormCrawler

stormcrawler-docker

Resources for running StormCrawler with Docker services

ansible-storm

Ansible playbook for deploying a Storm cluster

TextClassificationPlugin

GATE Processing Resource wrapping DigitalPebble's TextClassification API

ngrams-api

Java API for querying a N-Grams corpus. Uses Lucene for searching and indexing from the Google Web-1T format

tescobank

Setup for crawling tescobank with SC

NutchFight

Resources for comparison between 1.8 and 2.x of Apache Nutch

sc-warc

WARC resources for StormCrawler

behemoth-textclassification

Module for classifying Behemoth documents with a model from our Text Classification API

crawlurlfrontier

Crawl config used to test URL Frontier on a large scale and produce WARCs for CommonCrawl.

behemoth-elasticsearch

ElasticSearch module for Behemoth

urlfrontier-client

URLFrontier client written in Rust (mostly as a way of learning Rust)