scrapy-mongodb

MongoDB pipeline for Scrapy. This library supports both MongoDB in standalone setups and replica sets. It will insert items to MongoDB as soon as your spider finds data to extract. scrapy-mongodb can also buffer objects if you prefer to write chunks of data to MongoDB rather than one write per document (see MONGODB_BUFFER_DATA option for details).

INSTALLATION

Dependencies

Read more here.

Instructions

Install via pip:

pip install -r requirements.txt
pip install scrapy-mongodb

CONFIGURATION

Basic configuration

Add these options to settings.py file:

ITEM_PIPELINES = {
    ...
    'scrapy_mongodb.MongoDBPipeline': 300,
    ...
}

MONGODB_URI = 'mongodb://localhost:27017'
MONGODB_DATABASE = 'scrapy'
MONGODB_COLLECTION = 'my_items'

If you want a unique key in your database, name the key with this option:

MONGODB_UNIQUE_KEY = 'url'

Replica sets

You can configure scrapy-mongodb to support MongoDB replica sets by adding MONGODB_REPLICA_SET option and specify additional replica set hosts in MONGODB_URI:

MONGODB_REPLICA_SET = 'myReplicaSetName'
MONGODB_URI = 'mongodb://host1.example.com:27017,host2.example.com:27017,host3.example.com:27017'

If you need to ensure that your data has been replicated, use the MONGODB_REPLICA_SET_W option. It is an implementation of the w parameter in pymongo. Details from the pymongo documentation:

Write operations will block until they have been replicated to the specified number or tagged set of servers. w=<int> always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to two secondaries). Passing w=0 disables write acknowledgement and all other write concern options.

Data buffering

To ease the load on MongoDB, scrapy-mongodb has a buffering feature. You can enable it by setting MONGODB_BUFFER_DATA to the buffer size you want. E.g: scrapy-mongodb will write 10 documents at a time to the database if you set:

MONGODB_BUFFER_DATA = 10

It is not possible to combine this feature with MONGODB_UNIQUE_KEY. Technically due to that the update method in pymongo doesn't support multi-doc updates.

Timestamps

scrapy-mongodb can append a timestamp to your item when inserting it to the database. Enable this feature with:

MONGODB_ADD_TIMESTAMP = True

This will modify the document to something like this:

{
    ...
    'scrapy-mongodb': {
        'ts': ISODate("2013-01-10T07:43:56.797Z")
    }
    ...
}

The timestamp is in UTC.

One collection per spider

It's possible to write data to 1 collection per spider. To enable that feature, set this environment variable:

MONGODB_SEPARATE_COLLECTIONS = True

Full list of available options

Deprecated options

Since scrapy-mongodb 0.5.0

PUBLISHING TO PYPI

make release

CHANGELOG

Read more here.

AUTHOR

This project is maintained by: Sebastian Dahlgren (GitHub | Twitter | LinkedIn).

LICENSE

Read more here.