scrapy-webdriver
Scrape using Selenium webdriver.
Not well tested. Has probably lots of bugs, unhandled situations, and scrapy integration problems. Use at your own risk.
That being said, it works for me, in all its non-blocking glory.
Installation
For now it's not on pypi, but this should work:
pip install https://github.com/sosign/scrapy-webdriver/archive/master.zip
Or something like this, in setup.py:
setup(
install_requires=[
'scrapy_webdriver',
...,
],
dependency_links=[
'https://github.com/sosign/scrapy-webdriver/archive/master.zip#egg=scrapy_webdriver',
],
...,
)
Configuration
Add something like this in your scrapy project settings:
DOWNLOAD_HANDLERS = {
'http': 'scrapy_webdriver.download.WebdriverDownloadHandler',
'https': 'scrapy_webdriver.download.WebdriverDownloadHandler',
}
SPIDER_MIDDLEWARES = {
'scrapy_webdriver.middlewares.WebdriverSpiderMiddleware': 543,
}
WEBDRIVER_BROWSER = 'PhantomJS' # Or any other from selenium.webdriver
# or 'your_package.CustomWebdriverClass'
# or an actual class instead of a string.
# Optional passing of parameters to the webdriver
WEBDRIVER_OPTIONS = {
'service_args': ['--debug=true', '--load-images=false', '--webdriver-loglevel=debug']
}
Usage
In order to have webdriver handle your downloads, use the provided
class scrapy_webdriver.http.WebdriverRequest in place of the stock scrapy
Request, like so:
from scrapy_webdriver.http import WebdriverRequest
yield WebdriverRequest('http://www.example.com')
Parameters not supported (yet?) are: method, body, headers, cookies.
Hacking
Pull requests much welcome. Just make sure the tests still pass, and add to them as necessary:
python setup.py test