• Stars
    star
    8,991
  • Rank 3,798 (Top 0.08 %)
  • Language
    Python
  • License
    BSD 3-Clause "New...
  • Created about 10 years ago
  • Updated 8 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Visual scraping for Scrapy

Portia

Portia is a tool that allows you to visually scrape websites without any programming knowledge required. With Portia you can annotate a web page to identify the data you wish to extract, and Portia will understand based on these annotations how to scrape data from similar pages.

Running Portia

The easiest way to run Portia is using Docker:

You can run Portia using Docker & official Portia-image by running:

docker run -v ~/portia_projects:/app/data/projects:rw -p 9001:9001 scrapinghub/portia

You can also set up a local instance with Docker-compose by cloning this repo & running from the root of the folder:

docker-compose up

For more detailed instructions, and alternatives to using Docker, see the Installation docs.

Documentation

Documentation can be found from Read the docs. Source files can be found in the docs directory.

More Repositories

1

splash

Lightweight, scriptable browser as a service with an HTTP API
Python
3,898
star
2

dateparser

python parser for human readable dates
Python
2,445
star
3

frontera

A scalable frontier for web crawlers
Python
1,274
star
4

slackbot

A chat bot for Slack (https://slack.com).
Python
1,249
star
5

scrapyrt

HTTP API for Scrapy spiders
Python
810
star
6

extruct

Extract embedded metadata from HTML markup
Python
801
star
7

python-crfsuite

A python binding for crfsuite
Python
763
star
8

spidermon

Scrapy Extension for monitoring spiders execution.
Python
510
star
9

price-parser

Extract price amount and currency symbol from a raw text string
Python
296
star
10

webstruct

NER toolkit for HTML data
HTML
252
star
11

article-extraction-benchmark

Article extraction benchmark: dataset and evaluation scripts
Python
235
star
12

python-scrapinghub

A client interface for Scrapinghub's API
Python
195
star
13

adblockparser

Python parser for Adblock Plus filters
Python
187
star
14

js2xml

Convert Javascript code to an XML document
Python
185
star
15

testspiders

Useful test spiders for Scrapy
Python
183
star
16

scrapy-training

Scrapy Training companion code
Python
171
star
17

skinfer

Skinfer is a tool for inferring and merging JSON schemas
Python
139
star
18

sample-projects

Sample projects showcasing Scrapinghub tech
Python
136
star
19

shub

Scrapinghub Command Line Client
Python
125
star
20

python-simhash

An efficient simhash implementation for python
C
121
star
21

scrapy-poet

Page Object pattern for Scrapy
Python
109
star
22

mdr

A python library detect and extract listing data from HTML page.
C
106
star
23

number-parser

Parse numbers written in natural language
Python
103
star
24

aile

Automatic Item List Extraction
HTML
88
star
25

web-poet

Web scraping Page Objects core library
Python
87
star
26

wappalyzer-python

UNMAINTAINED Python wrapper for Wappalyzer (utility that uncovers the technologies used on websites)
Python
82
star
27

pydepta

A python implementation of DEPTA
C
80
star
28

scrapinghub-stack-scrapy

Software stack with latest Scrapy and updated deps
Dockerfile
62
star
29

scrapy-autounit

Automatic unit test generation for Scrapy.
Python
55
star
30

learn.scrapinghub.com

Scrapinghub Learning Center. Report issues in Jira: Report issues in Jira: https://scrapinghub.atlassian.net/projects/WEB
CSS
55
star
31

aduana

Frontera backend to guide a crawl using PageRank, HITS or other ranking algorithms based on the link structure of the web graph, even when making big crawls (one billion pages).
C
53
star
32

scrapy-autoextract

Zyte Automatic Extraction integration for Scrapy
Python
53
star
33

portia2code

Python
49
star
34

arche

Analyze scraped data
Python
47
star
35

scmongo

MongoDB extensions for Scrapy
Python
44
star
36

exporters

Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
Python
40
star
37

page_clustering

A simple algorithm for clustering web pages, suitable for crawlers
HTML
35
star
38

webpager

Paginating the web
C
35
star
39

scrapy-frontera

More flexible and featured Frontera scheduler for Scrapy
Python
34
star
40

scaws

Extensions for using Scrapy on Amazon AWS
Python
32
star
41

flatson

Tool to flatten stream of JSON-like objects, configured via schema
Python
32
star
42

docker-images

Dockerfile
32
star
43

scrapylib

Collection of Scrapy utilities (extensions, middlewares, pipelines, etc)
Python
31
star
44

pycon-speakers

Speakers Spider (PyCon 2014 sprint)
Python
30
star
45

docker-devpi

pypi caching service using devpi and docker
Shell
28
star
46

crawlera-tools

Crawlera tools
Python
26
star
47

scrapinghub-entrypoint-scrapy

Scrapy entrypoint for Scrapinghub job runner
Python
25
star
48

scrapy-mosquitera

Restrict crawl and scraping scope using matchers.
Python
25
star
49

autoextract-spiders

Pre-built Scrapy spiders for AutoExtract
Python
20
star
50

kafka-scanner

High Level Kafka Scanner
Python
19
star
51

python-cld2

Python bindings for CLD2.
Python
17
star
52

andi

Library for annotation-based dependency injection
Python
17
star
53

python-hubstorage

Deprecated HubStorage client library - please use python-scrapinghub>=1.9.0 instead
Python
16
star
54

shublang

Pluggable DSL that uses pipes to perform a series of linear transformations to extract data
Python
15
star
55

product-extraction-benchmark

Jupyter Notebook
15
star
56

shubc

Go bindings for Scrapinghub HTTP API and a sweet command line tool for Scrapy Cloud
Go
13
star
57

shub-workflow

Python
12
star
58

scrapinghub-stack-portia

Software stack used to run Portia spiders in Scrapinghub cloud
Python
10
star
59

navscraper

Vanguard ETF NAV scraper
Python
8
star
60

tutorials

Python
7
star
61

pastebin

Python
7
star
62

hcf-backend

Crawl Frontier HCF backend
Python
7
star
63

varanus

A command line spider monitoring tool
Python
7
star
64

pydatanyc

Python
7
star
65

autoextract-poet

web-poet definitions for AutoExtract
Python
6
star
66

collection-scanner

HubStorage collection scanner library
Python
5
star
67

locode

Python
5
star
68

autoextract-examples

Jupyter Notebook
4
star
69

webstruct-demo

HTTP demo for https://github.com/scrapinghub/webstruct
Python
4
star
70

shub-image

Deprecated client side tool to prepare docker images to run crawlers in Scrapinghub - please use shub>=2.5.0 instead
Python
4
star
71

docker-cloudera-manager

Run Cloudera Manager in docker
Dockerfile
3
star
72

adblockgoparser

Golang parser for Adblock Plus filters
Go
3
star
73

hubstorage-frontera

Hubstorage crawl frontier backend for Frontera
Python
3
star
74

httpation

Erlang
3
star
75

xpathcsstutorial

[Work in progress] XPath & CSS for web scraping tutorial
Jupyter Notebook
3
star
76

custom-images-examples

Examples of custom images running on Scrapinghub platform
3
star
77

epmdless_dist

Erlang
2
star
78

egraylog

Erlang
2
star
79

scrapinghub-conda-recipes

Conda packages for scrapinghub channel
Shell
2
star
80

pydaybot

Demo bot for Python Day Uruguay 2011
Python
2
star
81

erl-iputils

Erlang
1
star
82

jupyterhub-stacks

A docker images for jhub cluster
Python
1
star
83

scrapinghub-stack-hworker

[DEPRECATED] Software stack fully compatible with Scrapy Cloud 1.0
Python
1
star
84

cld2

Compact Language Detector 2
C++
1
star
85

pkg-opengrok

Ubuntu packaging for OpenGrok
Shell
1
star
86

crawlera.com

crawlera.com website
HTML
1
star
87

discourse-sso-google

Use Google as Single-Sign-On provider for Discourse
Python
1
star