spider-rs/spider

Stars
562
Rank 79,281 (Top 2 %)
Language
Rust
License
MIT License
Created almost 7 years ago
Updated 8 months ago

spider-rs/spider

spider-rs

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

The fastest web crawler written in Rust. Maintained by @a11ywatch.

Spider

The fastest web crawler and indexer.

Getting Started

View the spider or CLI directory to get started.

Benchmarks

See BENCHMARKS.

Examples

Contributing

See CONTRIBUTING.

proxier

Auto proxy whitelisting for 3rd parties