alash3al/scraply alash3al/scraply

  • Stars
    star
    122
  • Rank 292,031 (Top 6 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 5 years ago
  • Updated over 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
alash3al/scraply
github

alash3al

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Scraply a simple dom scraper to fetch information from any html based website

Scraply

Scraply, is a very simple html scraping tool, if you know css & jQuery then you can use it!, scraply should be simple and tiny as well it could be used as a component in a large system something like this use-case

Overview

you can use scraply within your stack via cli or http.

# here is the CLI usage

# extracting the title and the description from scraply github repo page
$ scraply extract \
    -u "https://github.com/alash3al/scraply" \
    -x title='$("title").text()' \
    -x description='$("meta[name=description]").attr("content")'

# same thing but with custom user agent
$ scraply extract \
    -u "https://github.com/alash3al/scraply" \
    -ua "OptionalCustomUserAgent"\
    -x title='$("title").text()' \
    -x description='$("meta[name=description]").attr("content")'

# same thing but with asking scraply to return the response body for debugging purposes
$ scraply extract \
    --return-body \
    -u "https://github.com/alash3al/scraply" \
    -x title='$("title").text()' \
    -x description='$("meta[name=description]").attr("content")'

for http usage, we will run the http server then using any http client to interact with it.

# running the http server
# by default it listens on address ":8010" which equals to "0.0.0.0:8010"
# for more information execute `$ scraply help`
$ scraply serve

# then in another shell let's execute the following curl 
$ curl http://localhost:8010/extract \
    -H "Content-Type: application/json" \
    -s \
    -d '{"url": "https://github.com/alash3al/scraply", "extractors": {"title": "$(\"title\").text()"}, "return_body": false, "user_agent": "CustomeUserAgent"}'

for debugging, there is shell

$ scraply shell -u https://github.com/alash3al/scraply
➜ (scraply) > $("title").text()
GitHub - alash3al/scraply: Scraply a simple dom scraper to fetch information from any html based website and convert that info to JSON APIs

➜ (scraply) > request.url
https://github.com/alash3al/scraply

➜ (scraply) > response.status_code
200

➜ (scraply) > response.url
https://github.com/alash3al/scraply

➜ (scraply) > response.body
<html>.....

Download ?

you can go to the releases page and pick the latest version. or you can $ docker run --rm -it ghcr.io/alash3al/scraply scraply help

Contribution ?

for sure you can contribute, how?

  • clone the repo
  • create your fix/feature branch
  • create a pull request

nothing else, enjoy!

About

I'm Mohamed Al Ashaal, a software engineer :)

More Repositories

1

sqler

write APIs using direct SQL queries with no hassle, let's rethink about SQL
Go
2,031
star
2

redix

a very simple pure key => value storage system that speaks Redis protocol with Postgres as storage engine and more
Go
1,166
star
3

wsify

Just a tiny, simple and real-time self-hosted pub/sub messaging service
Go
504
star
4

smtp2http

A tiny software that receive a smtp request (email) and send it to the specified webhook as a http post request
Go
410
star
5

lightify

a reverse proxy that boosts the web app performance!
Go
191
star
6

srchx

A standalone lightweight full-text search engine built on top of blevesearch and Go with multiple storage (scorch, boltdb, leveldb, badger)
Go
151
star
7

httpsify

a transparent HTTPS termination proxy using letsencrypt with auto certification renewal
Go
109
star
8

go-pubsub

An in-process simple publish/subscribe library for golang
Go
103
star
9

go-smtpsrv

a tiny package that implements SMTP server for Go projects
Go
95
star
10

re-txt

converts text-formats from one to another, it is very useful if you want to re-format a json file to yaml, toml to yaml, csv to yaml, ... etc
Go
64
star
11

dockssh

Dockssh, ssh into any container from anywhere with the help of redis for password storage
Go
61
star
12

bbadger

adding badger support to blevesearch
Go
59
star
13

xyr

Query any data source using SQL, works with the local filesystem, s3, and more. It should be a very tiny and lightweight alternative to AWS Athena, Presto ... etc.
Go
58
star
14

phoo

a very simple high performance PHP application server and php-fpm supervisor
Go
58
star
15

Plus

Asynchronous I/O environment in pure PHP
PHP
57
star
16

Horus

a minimal event-driven, flexible, portable and micro PHP web application framework
PHP
56
star
17

goukv

a key-value store with multiple backends including leveldb, badgerdb, postgresql
Go
52
star
18

scrapyr

a simple & tiny scrapy clustering solution, considered a drop-in replacement for scrapyd
Go
50
star
19

droxy

a transparent standalone http reverse proxy for docker containers
Go
45
star
20

sql2slack

send sql results periodically to slack
Go
34
star
21

xerver

a tiny static and fastcgi reverse proxy written in golang
Go
34
star
22

pxml-tpl

PHTML a smart tiny oo PHP HTML style template system, no more regex and no more large code, packags, ...
PHP
33
star
23

arp2http

notify the specified webhook when a device is connected/disconnected
Go
30
star
24

mailux

Experimental Passwordless Login even without sending emails/sms to users
Shell
29
star
25

goemitter

an event-emitter for golang
Go
22
star
26

katch

headless chrome as a service for generating PDF, PNG, JPEG and HTML of any web page
Go
16
star
27

aggrex

a crazy API gateway aggregation using javascript as a language and go as a runtime
Go
15
star
28

exeq

painless task queue manager for shell commands with an intuitive cli interface (execute shell commands in distributed cloud-native queue manager).
Go
13
star
29

axync

a smart cooperative multitasking kernel for php7
PHP
13
star
30

olive-go

Just a lightweight golang web application middleware
Go
13
star
31

img2pdf

convert a directory of images to a PDF file
Go
11
star
32

vidutils

a very simple, tiny and intuitive ffmpeg wrapper with a cli interface for inspecting & transforming media files supported by the original ffmpeg software
Go
11
star
33

go-schemaless

a schemaless implementation based on postgres and jsonb
Go
10
star
34

tix

a super simple stupid event-loop kernel in pure PHP
PHP
9
star
35

libsrchx

a simple wrapper for blevesearch created for srchx the tiny full-text search engine
Go
9
star
36

xerve

fast, tiny static file server with auto gzip & minify
Go
8
star
37

mylitedb

a mysql server backed by sqlite!
Go
8
star
38

funker

a platform for function as a service based javascript written in Go.
Go
7
star
39

go-univush

a universal push notification library written in go based on apns2 and firebase
Go
7
star
40

tlx

a dummy tls reverse proxy written in go
Go
6
star
41

dir2webp

convert images in directory/directories to webp format using the power of concurrent programming
Go
6
star
42

dart-path-selector

extract values from Map/List using dot-seprated strings you don't have to cast multiple times to fetch a simple values, this is very useful while working with i.e json data
Dart
5
star
43

uwatch

notify email(s) that a url is now online
Go
5
star
44

mysql2sqlite

a command-line utility to move data from mysql to sqlite
Go
5
star
45

xonsole

A tiny cli framework based on laravel\console, so you can use it without full laravel installation
PHP
4
star
46

go-fastcgi-client

A FastCGI client library for Golang projects
Go
4
star
47

HorusJS

A lightweight NodeJS microframework
JavaScript
3
star
48

cgify

a for fun script server (custom cgi)
Go
3
star
49

xtore

a document store based on cockroachdb
Go
3
star
50

wp-allowed-hosts

a plugin that protects your wp site from the CVE-2017-8295 vulnerability
PHP
2
star
51

alash3al

WHO AM I
2
star
52

xmg

a tiny microservice for getting perceptual hash of an image or faces in the image
Go
2
star
53

rsty

a tiny RESTful service middleware for Golang applications
Go
2
star
54

googly

a simple command line tool to search google with simple options
Go
2
star
55

goring

A simple consistent hashing ring implementation in Golang
Go
2
star
56

aof

A simple concurrency safe Append-Only-File for storage purposes
Go
2
star
57

olive.js

a lightweight micro javascript web application framework
JavaScript
1
star
58

php-myRouter

A PHP Router (/class/method/parm1/parm2/parm3/parm4/.....................)
PHP
1
star
59

z-pharo

Z-Pharo PHP CMS Framework
1
star
60

isup

check whether a url is up or down, and notify you whenever it become available
Go
1
star
61

LiteJS

A very tiny full featured template engine for javascript for both client and server sides, you need no-learning curve
JavaScript
1
star
62

odb

just a for-fun only-value database
Go
1
star