đ¤ English | đ įŽäŊä¸æ
ScrapydWeb: Web app for Scrapyd cluster management, with support for Scrapy log analysis & visualization.
Scrapyd â ScrapydWeb â LogParser
đ Recommended Reading
đ How to efficiently manage your distributed web scraping projects
đ How to set up Scrapyd cluster on Heroku
đ Demo
â Features
View contents
-
đ Scrapyd Cluster Management
- đ¯ All Scrapyd JSON API Supported
- âī¸ Group, filter and select any number of nodes
- đąī¸ Execute command on multinodes with just a few clicks
-
đ Scrapy Log Analysis
- đ Stats collection
- đ Progress visualization
- đ Logs categorization
-
đ Enhancements
- đĻ Auto packaging
- đĩī¸ââī¸ Integrated with đ LogParser
- â° Timer tasks
- đ§ Monitor & Alert
- đą Mobile UI
- đ Basic auth for web UI
đģ Getting Started
View contents
â ī¸ Prerequisites
â Make sure that đ Scrapyd has been installed and started on all of your hosts.
âŦī¸ Install
- Use pip:
pip install scrapydwebâ Note that you may need to execute python -m pip install --upgrade pip first in order to get the latest version of scrapydweb, or download the tar.gz file from https://pypi.org/project/scrapydweb/#files and get it installed via pip install scrapydweb-x.x.x.tar.gz
- Use git:
pip install --upgrade git+https://github.com/my8100/scrapydweb.gitOr:
git clone https://github.com/my8100/scrapydweb.git
cd scrapydweb
python setup.py installâļī¸ Start
- Start ScrapydWeb via command
scrapydweb. (a config file would be generated for customizing settings at the first startup.) - Visit http://127.0.0.1:5000 (It's recommended to use Google Chrome for a better experience.)
đ Browser Support
The latest version of Google Chrome, Firefox, and Safari.
âī¸ Running the tests
View contents
$ git clone https://github.com/my8100/scrapydweb.git
$ cd scrapydweb
# To create isolated Python environments
$ pip install virtualenv
$ virtualenv venv/scrapydweb
# Or specify your Python interpreter: $ virtualenv -p /usr/local/bin/python3.7 venv/scrapydweb
$ source venv/scrapydweb/bin/activate
# Install dependent libraries
(scrapydweb) $ python setup.py install
(scrapydweb) $ pip install pytest
(scrapydweb) $ pip install coverage
# Make sure Scrapyd has been installed and started, then update the custom_settings item in tests/conftest.py
(scrapydweb) $ vi tests/conftest.py
(scrapydweb) $ curl http://127.0.0.1:6800
# '-x': stop on first failure
(scrapydweb) $ coverage run --source=scrapydweb -m pytest tests/test_a_factory.py -s -vv -x
(scrapydweb) $ coverage run --source=scrapydweb -m pytest tests -s -vv --disable-warnings
(scrapydweb) $ coverage report
# To create an HTML report, check out htmlcov/index.html
(scrapydweb) $ coverage htmlđī¸ Built With
đ Changelog
Detailed changes for each release are documented in the đ HISTORY.md.
đ¨âđģ Author
 my8100 |
|---|
đĨ Contributors
 Kaisla |
|---|
Šī¸ License
This project is licensed under the GNU General Public License v3.0 - see the đ LICENSE file for details.