Discover scrapinghub/httpation Open Source project

Stars
3
Rank 3,963,521 (Top 79 %)
Language
Erlang
License
MIT License
Created about 8 years ago
Updated over 7 years ago

scrapinghub

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

portia

Visual scraping for Scrapy

Python

8,991

splash

Lightweight, scriptable browser as a service with an HTTP API

Python

3,898

dateparser

python parser for human readable dates

Python

2,525

frontera

A scalable frontier for web crawlers

Python

1,288

slackbot

A chat bot for Slack (https://slack.com).

Python

1,263

extruct

Extract embedded metadata from HTML markup

Python

832

scrapyrt

HTTP API for Scrapy spiders

Python

829

python-crfsuite

A python binding for crfsuite

Python

770

spidermon

Scrapy Extension for monitoring spiders execution.

Python

530

price-parser

Extract price amount and currency symbol from a raw text string

Python

307

article-extraction-benchmark

Article extraction benchmark: dataset and evaluation scripts

Python

268

webstruct

NER toolkit for HTML data

HTML

252

python-scrapinghub

A client interface for Scrapinghub's API

Python

195

adblockparser

Python parser for Adblock Plus filters

Python

187

js2xml

Convert Javascript code to an XML document

Python

186

testspiders

Useful test spiders for Scrapy

Python

183

scrapy-training

Scrapy Training companion code

Python

171

skinfer

Skinfer is a tool for inferring and merging JSON schemas

Python

140

sample-projects

Sample projects showcasing Scrapinghub tech

Python

137

shub

Scrapinghub Command Line Client

Python

125

python-simhash

An efficient simhash implementation for python

122

scrapy-poet

Page Object pattern for Scrapy

Python

119

number-parser

Parse numbers written in natural language

Python

108

mdr

A python library detect and extract listing data from HTML page.

106

web-poet

Web scraping Page Objects core library

Python

aile

Automatic Item List Extraction

HTML

wappalyzer-python

UNMAINTAINED Python wrapper for Wappalyzer (utility that uncovers the technologies used on websites)

Python

pydepta

A python implementation of DEPTA

scrapinghub-stack-scrapy

Software stack with latest Scrapy and updated deps

Dockerfile

aduana

Frontera backend to guide a crawl using PageRank, HITS or other ranking algorithms based on the link structure of the web graph, even when making big crawls (one billion pages).

scrapy-autoextract

Zyte Automatic Extraction integration for Scrapy

Python

scrapy-autounit

Automatic unit test generation for Scrapy.

Python

learn.scrapinghub.com

Scrapinghub Learning Center. Report issues in Jira: Report issues in Jira: https://scrapinghub.atlassian.net/projects/WEB

portia2code

arche

scmongo

MongoDB extensions for Scrapy

Python

exporters

Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations

Python

webpager

Paginating the web

scrapy-frontera

More flexible and featured Frontera scheduler for Scrapy

Python

page_clustering

A simple algorithm for clustering web pages, suitable for crawlers

HTML

flatson

Tool to flatten stream of JSON-like objects, configured via schema

Python

scaws

Extensions for using Scrapy on Amazon AWS

Python

docker-images

Dockerfile

scrapylib

Collection of Scrapy utilities (extensions, middlewares, pipelines, etc)

Python

pycon-speakers

Speakers Spider (PyCon 2014 sprint)

Python

docker-devpi

pypi caching service using devpi and docker

Shell

crawlera-tools

Crawlera tools

Python

scrapinghub-entrypoint-scrapy

Scrapy entrypoint for Scrapinghub job runner

Python

scrapy-mosquitera

Restrict crawl and scraping scope using matchers.

Python

andi

Library for annotation-based dependency injection

Python

kafka-scanner

High Level Kafka Scanner

Python

autoextract-spiders

Pre-built Scrapy spiders for AutoExtract

Python

python-cld2

Python bindings for CLD2.

Python

product-extraction-benchmark

Jupyter Notebook

python-hubstorage

Deprecated HubStorage client library - please use python-scrapinghub>=1.9.0 instead

Python

shublang

Pluggable DSL that uses pipes to perform a series of linear transformations to extract data

Python

shub-workflow

Python

shubc

Go bindings for Scrapinghub HTTP API and a sweet command line tool for Scrapy Cloud

scrapinghub-stack-portia

Software stack used to run Portia spiders in Scrapinghub cloud

Python

tutorials

Python

pastebin

Python

navscraper

Vanguard ETF NAV scraper

Python

varanus

A command line spider monitoring tool

Python

hcf-backend

Crawl Frontier HCF backend

Python

pydatanyc

Python

autoextract-poet

web-poet definitions for AutoExtract

Python

collection-scanner

HubStorage collection scanner library

Python

locode

Python

adblockgoparser

Golang parser for Adblock Plus filters

autoextract-examples

Jupyter Notebook

webstruct-demo

HTTP demo for https://github.com/scrapinghub/webstruct

Python

shub-image

Deprecated client side tool to prepare docker images to run crawlers in Scrapinghub - please use shub>=2.5.0 instead

Python

docker-cloudera-manager

Run Cloudera Manager in docker

Dockerfile

custom-images-examples

Examples of custom images running on Scrapinghub platform

hubstorage-frontera

Hubstorage crawl frontier backend for Frontera

Python

xpathcsstutorial

[Work in progress] XPath & CSS for web scraping tutorial

Jupyter Notebook

epmdless_dist

Erlang

egraylog

Erlang

scrapinghub-conda-recipes

Conda packages for scrapinghub channel

Shell

pydaybot

Demo bot for Python Day Uruguay 2011

Python

erl-iputils

Erlang

jupyterhub-stacks

A docker images for jhub cluster

Python

cld2

Compact Language Detector 2

C++

scrapinghub-stack-hworker

[DEPRECATED] Software stack fully compatible with Scrapy Cloud 1.0

Python

crawlera.com

crawlera.com website

HTML

discourse-sso-google

Use Google as Single-Sign-On provider for Discourse

Python

pkg-opengrok

Ubuntu packaging for OpenGrok

Shell

scrapinghub/httpation

scrapinghub

Reviews

Repository Details

More Repositories