DuckDuckGo Tracker Radar Collector
How do I use it?
Use it from the command line
- Clone this project locally (
git clone [email protected]:duckduckgo/tracker-radar-collector.git
) - Install all dependencies (
npm i
) - Run the command line tool:
npm run crawl -- -u "https://example.com" -o ./data/ -v
Available options:
-o, --output <path>
- (required) output folder where output files will be created-u, --url <url>
- single URL to crawl-i, --input-list <path>
- path to a text file with list of URLs to crawl (each in a separate line)-d, --data-collectors <list>
- comma separated list (e.g-d 'requests,cookies'
) of data collectors that should be used (all by default)-c, --crawlers <number>
- override the default number of concurrent crawlers (default number is picked based on the number of CPU cores)--reporters <list>
- comma separated list (e.g.--reporters 'cli,file,html'
) of reporters to be used ('cli' by default)-v, --verbose
- instructs reporters to log additional information (e.g. for "cli" reporter progress bar will not be shown when verbose logging is enabled)-l, --log-path <path>
- instructs reporters where all logs should be written to-f, --force-overwrite
- overwrite existing output files (by default entries with existing output files are skipped)-3, --only-3p
- don't save any first-party data (e.g. requests, API calls for the same eTLD+1 as the main document)-m, --mobile
- emulate a mobile device when crawling-p, --proxy-config <host>
- optional SOCKS proxy host-r, --region-code <region>
- optional 2 letter region code. For metadata only-a, --disable-anti-bot
- disable simple build-in anti bot detection script injected to every frame--chromium-version <version_number>
- use custom version of Chromium (e.g. "843427") instead of using the default--config <path>
- path to a config file that allows to set all the above settings (and more). Note that CLI flags have a higher priority than settings passed via config. You can find a sample config file intests/cli/sampleConfig.json
.--autoconsent-action <action>
- automatic autoconsent action (requires thecmps
collector). Possible values: optIn, optOut
Use it as a module
-
Install this project as a dependency (
npm i git+https://github.com:duckduckgo/tracker-radar-collector.git
). -
Import it:
// you can either import a "crawlerConductor" that runs multiple crawlers for you
const {crawlerConductor} = require('tracker-radar-collector');
// or a single crawler
const {crawler} = require('tracker-radar-collector');
// you will also need some data collectors (/collectors/ folder contains all build-in collectors)
const {RequestCollector, CookieCollector, โฆ} = require('tracker-radar-collector');
- Use it:
crawlerConductor({
// required โ
urls: ['https://example.com', {url: 'https://duck.com', dataCollectors: [new ScreenshotCollector()]}, โฆ], // two formats available: first format will use default collectors set below, second format will use custom set of collectors for this one url
dataCallback: (url, result) => {โฆ},
// optional โ
dataCollectors: [new RequestCollector(), new CookieCollector()],
failureCallback: (url, error) => {โฆ},
numberOfCrawlers: 12,// custom number of crawlers (there is a hard limit of 38 though)
logFunction: (...msg) => {โฆ},// custom logging function
filterOutFirstParty: true,// don't save any first-party data (false by default)
emulateMobile: true,// emulate a mobile device (false by default)
proxyHost: 'socks5://myproxy:8080',// SOCKS proxy host (none by default)
antiBotDetection: true,// if anti bot detection script should be injected (true by default)
chromiumVersion: '843427',// Chromium version that should be downloaded and used instead of the default one
maxLoadTimeMs: 30000,// how long should crawlers wait for the page to load, defaults to 30s
extraExecutionTimeMs: 2500,// how long should crawlers wait after page loads before collecting data, defaults to 2.5s
});
OR (if you prefer to run a single crawler)
// crawler will throw an exception if crawl fails
const data = await crawler(new URL('https://example.com'), {
// optional โ
collectors: [new RequestCollector(), new CookieCollector(), โฆ],
log: (...msg) => {โฆ},
urlFilter: (url) => {โฆ},// function that, for each request URL, decides if its data should be stored or not
emulateMobile: false,
emulateUserAgent: false,// don't use the default puppeteer UA (default true)
proxyHost: 'socks5://myproxy:8080',
browserContext: context,// if you prefer to create the browser context yourself (to e.g. use other browser or non-incognito context) you can pass it here (by default crawler will create an incognito context using standard chromium for you)
runInEveryFrame: () => {window.alert('injected')},// function that should be executed in every frame (main + all subframes)
executablePath: '/some/path/Chromium.app/Contents/MacOS/Chromium',// path to a custom Chromium installation that should be used instead of the default one
maxLoadTimeMs: 30000,// how long should the crawler wait for the page to load, defaults to 30s
extraExecutionTimeMs: 2500,// how long should crawler wait after page loads before collecting data, defaults to 2.5s
});
crawl-cli.js
and crawlerConductor.js
to see how crawlerConductor
and crawler
are used in the wild.
Output format
Each successfully crawled website will create a separate file named after the website (when using the CLI tool). Output data format is specified in crawler.js
(see CollectResult
type definition).
Additionally, for each crawl metadata.json
file will be created containing crawl configuration, system configuration and some high-level stats.
Data post-processing
Example post-processing script, that can be used as a template, can be found in post-processing/summary.js
. Execute it from the command line like this:
node ./post-processing/summary.js -i ./collected-data/ -o ./result.json
node --max_old_space_size=4096
.
Creating new collectors
Each collector needs to extend the BaseCollector
and has to override following methods:
id()
which returns name of the collector (e.g. 'cookies')getData(options)
which should return collected data.options
have following properties:finalUrl
- final URL of the main document (after all redirects) that you may want to use,filterFunction
which, if provided, takes an URL and returns a boolean telling you if given piece of data should be returned or filtered out based on its origin.
Additionally, each collector can override following methods:
init(options)
which is called before the crawl beginsaddTarget(targetInfo)
which is called whenever new target is created (main page, iframe, web worker etc.)postLoad()
which is called after the page has loaded. This is the place for executing heavy page interactions (extraExecutionTimeMs
is applied after this hook).
There are couple of built-in collectors in the collectors/
folder. CookieCollector
is the simplest one and can be used as a template.
Each new collector has to be added in two places to be discoverable:
crawlerConductor.js
- so thatcrawlerConductor
knows about it (and it can be used in the CLI tool)main.js
- so that the new collector can be imported by other projects
You can also add types to define the structure of the data exported by your collector. These should be added to the CollectorData
type in collectorsList.js
. This will add type hints to all places where the data is used in the code.