Discover seomoz/simhash-cluster Open Source project by Moz (@seomoz)

Simhash and near-duplicate detection

664

simhash-py

Queue / Pipeline Management

377

qless

Fast Redis Bloom Filters in Python

292

pyreBloom

A toolkit for working with API endpoint definition files, giving you a stub app, a schema validation middleware, and browsable documentation.

286

interpol

HTML

187

word2gauss

Gaussian word embeddings

Modern robots.txt Parser for Python

186

reppy

URL Transformation, Sanitization

178

SEOmozAPISamples

Mozscape API sample code

simhash-cpp

url-py

Core Lua Scripts for qless

102

qless-core

Python API for Various DB-Backed Simhash Clusters

simhash-db-py

Python Bindings for qless

qless-py

qdr

Query-Document Relevance

Training/test data for Dragnet

dragnet_data

Shell

publicsuffix-elixir

Elixir library providing public suffix logic based on publicsuffix.org data

Elixir

linkscape-gem

Provides an interface to SEOmoz's suite of APIs, including the free and site intelligence APIs.

Social-Authority-SDK

Your Friendly Asynchronous S3 Upload Protocol Droid

s3po

Analysis of Google Webmaster Tools search data

GWT-keyword-analysis

Gevent Crawling in Python, with Utilities

g-crawl-py

Data science tools from Moz

mozsci

C++ bindings for url parsing and sanitization

url-cpp

vocab

Vocabulary using n-grams

A fast URI parser that wraps Google's chromium URL canonicalization library

uri_parser

Fetch urls quickly and asynchronously with Twisted, honoring politeness.

downpour

Robot exclusion protocol in C++

rep-cpp

mltk - Moz Language Tool Kit

mltk

Easily create job pipelines out of declared job dependencies using Qless.

plines

awssh

AWSSH Config

A complete mesos cluster setup with automatic load balancing

roger-mesos

Python Bindings for Linkscape's API

linkscape-py

Node.js bindings for qless

qless-js

JavaScript

roger-bamboo

Roger's internal load balancer and frontend proxy. Based on https://github.com/QubitProducts/bamboo

gzippy

Gzip files in python

asis

Lightweight As-Is Server

awscpp

AWS C++ Bindings

Rack middleware that handles basic auth and HMAC auth

rack-authenticate

Some elasticsearch utilities I've put together / been using in investigating elasticsearch performance

elasticsearch-utils

Python bindings to libJudy

pyjudy

A Resque plugin for processing queues from random jobs based on queue weightings. Inspired by resque-fairly.

resque-unfairly

Monitoring stack for RogerOS

roger-monitoring

A Curio Cabinet of the Odd Behaviors We've Seen on the Internet

crawl-curio-cabinet

HTML

qless-docker

Create a qless docker image!

robots.txt file inspection

irobot

Simple and fast Bloom filter

bloomfilter-py

Search for tags and such in zendesk

docker-sortdb

Docker setup for SortDB

Shell

qless-java

qless java binding

Java

zendesk-search

JavaScript

deb-swift

fiji

Cell schemas and schema versioning for HBase

HTML

p5-Webservice-Followerwonk-SocialAuthority

Perl Client for The Followerwonk Social Authority API

Perl

qless-util-py

Utilities for use with qless-py

Implements a dictionary that is scoped to a process tree for Erlang and Elixir.

process_tree_dictionary

Elixir

moz_nav

DEPRECATED. Common navigation and layout across all SEOmoz applications

Stuff for reading crawler log files. Probably not of much interest to those outside of SeoMOZ.

logtools