Scrapy Dynamic Configurable
A dynamic configurable news crawler based Scrapy. See the blogs for more detail.
##Requirements
- Scrapy
- MySQL
- Redis
- SQLAlchemy
##Install in development
Mac OS X, use Homebrew
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew install mysql
$ brew install redis
$ sudo pip install scrapy
$ sudo pip install SQLAlchemy
$ sudo pip install redisUbuntu*
$ sudo apt-get install redis-server mysql-server mysql-client
$ pip install scrapy
$ pip install SQLAlchemy
$ pip install redisand restore datatables from 'db.sql'
It is for scrapy 1.0 . It is for scrapy 1.0 .