Harvestman
Harvestman is a very simple, lightweight web crawler for Quick'n'Dirtyâ„¢ web scraping.
It's quite useful for scraping search result pages:
require 'harvestman'
Harvestman.crawl 'http://www.foo.com/bars?page=*', (1..5) do
price = css 'div.item-price a'
...
end
[!] Warning: this gem is in alpha stage (no tests), don't use it for anything serious.
Installation
Via command line:
$ gem install harvestman
Basic usage
Harvestman is fairly simple to use: you specify the URL to crawl and pass in a block.
Inside the block you can call the css
(or xpath
) method to search the HTML document and get the inner text inside each node.
See Nokogiri for more information.
Perhaps this is best understood with an example:
Harvestman.crawl "http://www.24pullrequests.com" do
headline = xpath "//h3"
catchy_phrase = css "div.visible-phone h3"
puts "Headline: #{headline}"
puts "Catchy phrase: #{catchy_phrase}"
end
One node to rule them all
Harvestman assumes there's only one node at the path you passed to the css
.
If there is more than one node at that path, you can pass in an additional block.
Another example:
Harvestman.crawl 'http://en.wikipedia.org/wiki/Main_Page' do
# Print today's featured article
tfa = css "div#mp-tfa"
puts "Today's featured article: #{tfa}"
# Print all the sister projects
sister_projects = []
css "div#mp-sister b" do
sister_projects << css("a")
end
puts "Sister projects:"
sister_projects.each { |sp| puts "- #{sp}" }
end
Note that inside the block we use css("a")
and not css("div#mp-sister b a")
. Calls to css
or xpath
here assume div#mp-sister b
is the parent node.
Pages / Search results
If you want to crawl a group of similar pages (eg: search results, as shown above), you can insert a *
somewhere in the URL string and it will be replaced by each element in the second argument.
Final example:
require 'harvestman'
Harvestman.crawl 'http://www.etsy.com/browse/vintage-category/electronics/*', (1..3) do
css "div.listing-hover" do
title = css "div.title a"
price = css "span.listing-price"
puts "* #{title} (#{price})"
end
end
The above code is going to crawl Etsy's electronics category pages (from 1 to 3) and output every item's title and price. Here we're using a range (1..3)
but you could've passed an array with search queries:
"http://www.site.com?query=*", ["dogs", "cats", "birds"]
Performance
When using the * feature described above, each page is run inside a separate thread. You can disable multithreading by passing an additional argument :plain
to the crawl
method, like this:
require 'harvestman'
Harvestman.crawl 'http://www.store.com/products?page=*', (1..99), :plain do
...
end
Needless to say, this will greatly decrease performance.
License
See LICENSE.txt