• Stars
    star
    10
  • Rank 1,807,489 (Top 36 %)
  • Language
    Go
  • License
    MIT License
  • Created about 1 year ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Go(od) Job is a simple job scheduler that supports task retries, logging, and task sharding.

Go(od) Job

Go Reference Go Report Card codecov

gojob is a simple job scheduler.

Install

go get github.com/WangYihang/gojob

Usage

Create a job scheduler with a worker pool of size 32. To do this, you need to implement the Task interface.

// Task is an interface that defines a task
type Task interface {
	// Do starts the task, returns error if failed
	// If an error is returned, the task will be retried until MaxRetries
	// You can set MaxRetries by calling SetMaxRetries on the scheduler
	Do() error
}

The whole code looks like this (try it online).

package main

import (
	"fmt"
	"net/http"

	"github.com/WangYihang/gojob"
)

type MyTask struct {
	Url        string `json:"url"`
	StatusCode int    `json:"status_code"`
}

func New(url string) *MyTask {
	return &MyTask{
		Url: url,
	}
}

func (t *MyTask) Do() error {
	response, err := http.Get(t.Url)
	if err != nil {
		return err
	}
	t.StatusCode = response.StatusCode
	defer response.Body.Close()
	return nil
}

func main() {
	var numTotalTasks int64 = 256
	scheduler := gojob.New(
		gojob.WithNumWorkers(8),
		gojob.WithMaxRetries(4),
		gojob.WithMaxRuntimePerTaskSeconds(16),
		gojob.WithNumShards(4),
		gojob.WithShard(0),
		gojob.WithTotalTasks(numTotalTasks),
		gojob.WithStatusFilePath("status.json"),
		gojob.WithResultFilePath("result.json"),
		gojob.WithMetadataFilePath("metadata.json"),
	).
		Start()
	for i := range numTotalTasks {
		scheduler.Submit(New(fmt.Sprintf("https://httpbin.org/task/%d", i)))
	}
	scheduler.Wait()
}

Use Case

http-crawler

Let's say you have a bunch of URLs that you want to crawl and save the HTTP response to a file. You can use gojob to do that. Check it out for details.

Try it out using the following command.

$ go run github.com/WangYihang/gojob/examples/complex-http-crawler@latest --help
Usage:
  main [OPTIONS]

Application Options:
  -i, --input=                        input file path
  -o, --output=                       output file path
  -r, --max-retries=                  max retries (default: 3)
  -t, --max-runtime-per-task-seconds= max runtime per task seconds (default: 60)
  -n, --num-workers=                  number of workers (default: 32)

Help Options:
  -h, --help                          Show this help message
$ cat urls.txt
https://www.google.com/
https://www.facebook.com/
https://www.youtube.com/
$ go run github.com/WangYihang/gojob/examples/complex-http-crawler@latest -i input.txt -o output.txt -n 4
$ tail -n 1 output.txt
{
    "started_at": 1708934911909748,
    "finished_at": 1708934913160935,
    "num_tries": 1,
    "task": {
        "url": "https://www.google.com/",
        "http": {
            "request": {
                "method": "HEAD",
                "url": "https://www.google.com/",
                "host": "www.google.com",
            	// details omitted for simplicity
            },
            "response": {
                "status": "200 OK",
                "proto": "HTTP/2.0",
                "header": {
                    "Alt-Svc": [
                        "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
                    ],
                },
            	// details omitted for simplicity
                "body": "",
            }
        }
    },
    "error": ""
}

Integration with Prometheus

gojob provides metrics (num_total, num_finshed, num_succeed, num_finished) for Prometheus. You can use the following code to expose the metrics.

package main

func main() {
	scheduler := gojob.New(
		// All you need to do is just adding the following option to the scheduler constructor
		gojob.WithPrometheusPushGateway("http://localhost:9091", "gojob"),
	).Start()
}

Coverage

codecov-graph

More Repositories

1

Platypus

🔨 A modern multiple reverse shell sessions manager written in go
Go
1,503
star
2

GitHacker

🕷️ A `.git` folder exploiting tool that is able to restore the entire Git repository, including stash, common branches and common tags.
Python
1,408
star
3

Webshell-Sniper

🔨 Manage your website via terminal
Python
419
star
4

SourceLeakHacker

🐛 A multi threads web application source leak scanner
Python
379
star
5

ccupp

基于社会工程学的弱口令密码字典生成工具
Python
340
star
6

UsbKeyboardDataHacker

USB键盘流量包取证工具 , 用于恢复用户的击键信息
Python
320
star
7

Reverse-Shell-Manager

🔨 A multiple reverse shell session/client manager via terminal
Python
237
star
8

USB-Mouse-Pcap-Visualizer

USB mouse traffic packet forensic tool, mainly used to draw mouse movements and dragging trajectories
JavaScript
233
star
9

Exploit-Framework

🔥 An Exploit framework for Web Vulnerabilities written in Python
Python
170
star
10

Apache-HTTP-Server-Module-Backdoor

👺 A Simple Backdoor For Apache HTTP Server
C
151
star
11

MIT-6.031-Readings-zh-cn

麻省理工大学-18年春季学期-软件构造(6.031)课程阅读中文版
84
star
12

awesome-web-security

📓 Some notes and impressive articles of Web Security
74
star
13

Codiad-Remote-Code-Execute-Exploit

A simple exploit to execute system command on codiad
Python
64
star
14

Find-PHP-Vulnerabilities

🐛 A plug-in of sublime 2/3 which is able to find PHP vulnerabilities
Python
56
star
15

PwnMe

二进制渗透题目汇总
Python
54
star
16

IdiomsSolitaire

成语接龙
Python
48
star
17

sqli-labs

WriteUp of sqli-labs (GitBook : https://www.gitbook.com/book/wangyihang/sqli-labs/details)
39
star
18

12306

12306网站抢票Python脚本
Python
28
star
19

WebShellCracker

WebShell密码爆破工具
Python
19
star
20

LinuxShellScript

LinuxShell编程笔记
Shell
15
star
21

SQL-Hacker

简单SQL注入工具
Python
14
star
22

XorShellcode

Shellcode异或加密工具
Python
12
star
23

Subdomain-Crawler

A program for collecting subdomains of a list of given second-level domains (SLD)
Go
12
star
24

ShellcodeSpider

Shellcode Spider of Exploit-DB
C
12
star
25

HIT-Courses-Calendar

哈尔滨工业大学教务处课表Excel转换iCal脚本
Python
10
star
26

Proxy-Verifier

A set of tools designed to efficiently and effectively locate publicly available proxy server resources.
Go
9
star
27

PPT-Generator

Generate PPT via a simple summary
Python
9
star
28

tplayer

一个Linux终端播放器 , 使用字符绘制图片/视频 , 并按照帧率播放
Python
8
star
29

Presentations

8
star
30

t3sec-network-flow-analysis

6
star
31

acw-sc-v2.js

`acw_sc__v2` cookie generator
HTML
5
star
32

Docker-Container-Exposer

Expose docker containers to public network
Shell
5
star
33

pickle-pickle

A arbitary python code executer via python pickle
Python
5
star
34

CrackMe

CrackMe 汇总
Python
5
star
35

Platypus-Python

Python
5
star
36

SimpleEncrypter

简单shellcode加密工具(存在 0 字节)
Python
4
star
37

Markdown-URL-to-Title

Python
4
star
38

DBLP-Spider

A spider tool for downloading the DBLP search results into local BibTeX files.
Python
4
star
39

MovieSearcher

电影资源搜索工具
Python
3
star
40

Image-LSB-Stego

Python
3
star
41

http-grab

Go
2
star
42

tranco-go-package

Go
2
star
43

acw-sc-v2-py

Python requests.HTTPAdapter for `acw_sc__v2`
Python
2
star
44

dns-grab

Go
2
star
45

PrintableShellcodeCreator

可打印shellcode生成工具
C
2
star
46

DBAPPSecurity-Unified-Security-Management-Python-Connector

Python Connector for DBAPPSecurity Unified Security Management | 明御®运维审计与风险控制系统(堡垒机)
Python
1
star
47

bgphenet

Go
1
star
48

acw-sc-v2-go

Go
1
star
49

ModifyHeadersForChrome

ModifyHeadersForChrome
JavaScript
1
star
50

ProcessInjector

C
1
star
51

JBrowserWithPulgins

Java实现的一个简单的Web浏览器 , 提供了插件功能 , 目前插件有下载地址分析器
Java
1
star