Discover DigitalPebble/ngrams-api Open Source project

This repository has been archived on 14/Jan/2021
Stars
4
Rank 3,304,323 (Top 66 %)
Language
Java
License
Other
Created over 12 years ago
Updated over 12 years ago

DigitalPebble

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Java API for querying a N-Grams corpus. Uses Lucene for searching and indexing from the Google Web-1T format

storm-crawler

A scalable, mature and versatile web crawler based on Apache Storm

HTML

842

behemoth

Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.

Java

282

TextClassification

A Text Classification API in Java originally developed by DigitalPebble Ltd. The API is independent from the ML implementations used and can be used as a front end to various ML algorithms. libSVM and liblinear are currently embedded.

Java

textclassification-examples

Use cases for DigitalPebble's TextClassification API

Java

stormcrawlerfight

Crawl configurations for benchmarking / testing StormCrawler

Shell

stormcrawler-docker

Resources for running StormCrawler with Docker services

Dockerfile

ansible-storm

Ansible playbook for deploying a Storm cluster

TextClassificationPlugin

GATE Processing Resource wrapping DigitalPebble's TextClassification API

Java

behemoth-commoncrawl

Support for old (pre 2013) CommonCrawl dataset in Behemoth

Java

tescobank

Setup for crawling tescobank with SC

Java

NutchFight

Resources for comparison between 1.8 and 2.x of Apache Nutch

Java

sc-warc

WARC resources for StormCrawler

behemoth-textclassification

Module for classifying Behemoth documents with a model from our Text Classification API

Java

crawlurlfrontier

Crawl config used to test URL Frontier on a large scale and produce WARCs for CommonCrawl.

FLUX

behemoth-elasticsearch

ElasticSearch module for Behemoth

Java

urlfrontier-client

URLFrontier client written in Rust (mostly as a way of learning Rust)

Rust

DigitalPebble/ngrams-api

DigitalPebble

Reviews

Repository Details

More Repositories