• Stars
    star
    1,307
  • Rank 35,997 (Top 0.8 %)
  • Language
    Ruby
  • License
    MIT License
  • Created about 13 years ago
  • Updated 10 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

Wombat

Gem Version Code Climate Coverage Status FOSSA Status

Web scraper with an elegant DSL that parses structured data from web pages.

Usage:

gem install wombat

Scraping a page:

The simplest way to use Wombat is by calling Wombat.crawl and passing it a block:

require 'wombat'

Wombat.crawl do
  base_url "https://www.github.com"
  path "/"

  headline xpath: "//h1"
  subheading css: "p.alt-lead"

  what_is({ css: ".one-fourth h4" }, :list)

  links do
    explore xpath: '/html/body/header/div/div/nav[1]/a[4]' do |e|
      e.gsub(/Explore/, "Love")
    end

    features css: '.nav-item-opensource'
    business css: '.nav-item-business'
  end
end
The code above is gonna return the following hash:
{
  "headline"=>"How people build software",
  "subheading"=>"Millions of developers use GitHub to build personal projects, support their businesses, and work together on open source technologies.",
  "what_is"=>[
    "For everything you build",
    "A better way to work",
    "Millions of projects",
    "One platform, from start to finish"
  ],
  "links"=>{
    "explore"=>"Love",
    "features"=>"Open source",
    "business"=>"Business"
  }
}

This is just a sneak peek of what Wombat can do. For the complete documentation, please check the links below:

Contributing to Wombat

  • Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
  • Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
  • Fork the project
  • Start a feature/bugfix branch
  • Commit and push until you are happy with your contribution
  • Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
  • Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.

Contributors

Copyright

Copyright (c) 2019 Felipe Lima. See LICENSE.txt for further details.

License

FOSSA Status

More Repositories

1

AsymmetricGridView

Android ListView that mimics a GridView with asymmetric items. Supports items with row span and column span
Java
1,838
star
2

GifImageView

Android ImageView that handles animated GIF images
Kotlin
1,165
star
3

ktnes

A multiplatform NES emulator written in Kotlin
Kotlin
304
star
4

QuickReturn

Android ListView that implements the QuickReturn UI pattern. Written from scratch with focus on performance.
Java
191
star
5

kales

Kotlin on Rails
HTML
86
star
6

random-maze-generator

Find the shortest path out of a generated random maze. Rendered with canvas, tested with Jasmine.
JavaScript
62
star
7

brazil.db

🇧🇷 Tabelas Postgres/MySQL com todas as cidades, estados, ruas, latitude, longitude e CEPs do Brasil
46
star
8

roberthood

Unofficial robinhood.com web UI
JavaScript
41
star
9

AbsListViewHelper

Really simple attacher class to add a header and/or a footer to an Android ListView or GridView
Java
37
star
10

Android-ImageManager

[DEPRECATED] An image downloading and caching library for Android
Java
26
star
11

churras

🥩 DeFi wallet dashboard dApp
TypeScript
23
star
12

cs-exercises

A collection coding problems and solutions to stay in shape
JavaScript
17
star
13

6502Android

A 6502 Emulator for Android written in Kotlin
Kotlin
14
star
14

saasquatch

Rails boilerplate project for Saas websites
Ruby
6
star
15

show-do-milhao

💰 A command line Show do Milhão app
Haskell
6
star
16

obgen

Observable (reactive streams) pattern implemented using es2015 in Typescript
TypeScript
6
star
17

brownian-tree-webgl

Brownian Tree rendered with WebGL with Rust + WASM
Rust
4
star
18

hackerrank

My solutions to some hackerrank challenges
Kotlin
3
star
19

android-bookstore

Java
3
star
20

kt-micrograd

Kotlin port of karpathy/micrograd
Kotlin
3
star
21

felipecsl.com

My personal website and blog
HTML
3
star
22

rusty-chess

A chess game written in Rust compiled for the web with WASM
Rust
3
star
23

tree-viz

Render a Tree data structure in a HTML canvas with next.js
TypeScript
3
star
24

imdb-browser

Next.js app for browsing IMDB data for movies and tv shows
TypeScript
2
star
25

dover

CMS as a service written in ASP.NET MVC
C#
2
star
26

hexaco-viz

Visualization tool for HEXACO personality test results
JavaScript
2
star
27

julipe.org

JuLipe Estate Inc.
JavaScript
1
star
28

dover-csharp-api

CSharp Library for integration with Dover
C#
1
star
29

bin-packing

A collection of common bin packing algorithms in Kotlin
Kotlin
1
star
30

old-felipecsl.com

My Blog website on Octopress
JavaScript
1
star
31

mensagei.ro

A simple web based instant messenger
1
star
32

felipecsl

1
star
33

kapt-classloader-bug

Proof of concept for a bug with the Kotlin kapt compiler as of version 1.2.40
Java
1
star