trandoshan-io/crawler trandoshan-io/crawler

  • This repository has been archived on 10/Aug/2020
  • Stars
    star
    150
  • Rank 247,323 (Top 5 %)
  • Language
    Go
  • License
    GNU General Publi...
  • Created over 5 years ago
  • Updated almost 5 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
trandoshan-io/crawler
github

trandoshan-io

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Go process used to crawl websites

crawler

Build Status Go Report Card Maintainability

Crawler is a Go written program designed to crawl website

features

  • use tor SOCKS proxy to crawl hidden services
  • fast, built using valyala/fasthttp (up to 10x faster than net/http)
  • extract both absolute and relative URLs
  • use scalable messaging protocol (nats)

how it work

  • The Crawler process connect to a nats server (specified by env variable NATS_URI) and set-up a subscriber for message with tag todoSubject
  • When an URL is received the crawler start crawling
  • When crawling is done, the crawler will publish content to nats server with subject contentSubject and found urls with subject doneSubject

More Repositories

1

k8s

Kubernetes configuration file to boostrap trandoshan cluster
Shell
46
star
2

trandoshan-parent

Parent repository used to centralize tools to run the application, manage source code, ...
Shell
21
star
3

dashboard

Web UI to manage a Trandoshan instance
TypeScript
11
star
4

persister

Go process used to abstract database interaction. It consumes messages from queue and put result in database allowing asynchronous saving operation
Go
7
star
5

scheduler

Go orchestrator process used to schedule website parsing
Go
6
star
6

api

REST API used by other processes to gather data
Go
5
star
7

feeder

Oneshot process used to put an URL in the todo queue. Useful to start crawling process
Go
2
star