Top Rating
- Top Contributors
  Discover the Top Open Source contributors by country or by language
- Interviews
  Discover real stories from Open Source developers
Discover

Discover your Favorite Language
Discover the top trending repositories and projects on Github. Explore the latest trends in your preferred languages.

R

C++

C

Java

Dart

C#

Objective-C

Emacs Lisp

More Languages
Awesome

Awesome repositories
Discover the most awesome repositories and projects of your favorite languages. Inspired by the Awesome-* lists trend in GitHub.

Nix

F#

PHP

Kotlin

Lua

Swift

Zig

C#

More Languages
By Country

Rankings by Country
Discover the community of talented open source contributors in each country.

🇦🇩 Andorra

🇾🇹 Mayotte

🇧🇿 Belize

🇸🇹 São Tomé and Príncipe

🇱🇺 Luxembourg

🇬🇮 Gibraltar

🇹🇭 Thailand

🇱🇷 Liberia

All Countries Compare Countries

DigitalPebble/crawlurlfrontier

Stars
1
Language FLUX
Created almost 4 years ago
Updated 8 months ago

DigitalPebble/crawlurlfrontier

DigitalPebble

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Crawl config used to test URL Frontier on a large scale and produce WARCs for CommonCrawl.

storm-crawler

A scalable, mature and versatile web crawler based on Apache Storm

behemoth

Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.

TextClassification

A Text Classification API in Java originally developed by DigitalPebble Ltd. The API is independent from the ML implementations used and can be used as a front end to various ML algorithms. libSVM and liblinear are currently embedded.

textclassification-examples

Use cases for DigitalPebble's TextClassification API

stormcrawlerfight

Crawl configurations for benchmarking / testing StormCrawler

stormcrawler-docker

Resources for running StormCrawler with Docker services

ansible-storm

Ansible playbook for deploying a Storm cluster

TextClassificationPlugin

GATE Processing Resource wrapping DigitalPebble's TextClassification API

ngrams-api

Java API for querying a N-Grams corpus. Uses Lucene for searching and indexing from the Google Web-1T format

behemoth-commoncrawl

Support for old (pre 2013) CommonCrawl dataset in Behemoth

tescobank

Setup for crawling tescobank with SC

NutchFight

Resources for comparison between 1.8 and 2.x of Apache Nutch

sc-warc

WARC resources for StormCrawler

behemoth-textclassification

Module for classifying Behemoth documents with a model from our Text Classification API

behemoth-elasticsearch

ElasticSearch module for Behemoth

urlfrontier-client

URLFrontier client written in Rust (mostly as a way of learning Rust)