fff
The Fairly Fast Fetcher. Requests a bunch of URLs provided on stdin fairly quickly.
The main idea is to launch a new request every n milliseconds, without waiting
for the last request to finish first. This makes for consistently fast fetching,
but can be hard on system resources (e.g. you might run out of file descriptors).
The advantage though, is that hitting a bunch of very slow URLs or URLs that
result in timeouts doesn't slow the overall progress very much.
Install
â–¶ go get -u github.com/tomnomnom/fff
Usage
Basic usage:
â–¶ cat urls.txt | fff
Options:
â–¶ fff --help
Request URLs provided on stdin fairly frickin' fast
Options:
-b, --body <data> Request body
-d, --delay <delay> Delay between issuing requests (ms)
-H, --header <header> Add a header to the request (can be specified multiple times)
-k, --keep-alive Use HTTP Keep-Alive
-m, --method HTTP method to use (default: GET, or POST if body is specified)
-o, --output <dir> Directory to save responses in (will be created)
-s, --save-status <code> Save responses with given status code (can be specified multiple times)
-S, --save Save all responses
-x, --proxy <proxyURL> Use the provided HTTP proxy
Tuning
You might want to increase your open file descriptor limit before doing anything crazy:
â–¶ ulimit -n 16384
TODO
- Create an index file in the output directory