sysulq/httpmq sysulq/httpmq

  • Stars
    star
    308
  • Rank 135,712 (Top 3 %)
  • Language
    Go
  • License
    MIT License
  • Created about 10 years ago
  • Updated 4 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
sysulq/httpmq
github

sysulq

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A simple HTTP message queue written in Go with goleveldb, just like httpsqs written in C with Tokyo Cabinet.

httpmq

Build Status Docker Pulls Go Report Card

Httpmq is a simple HTTP message queue written in Go with goleveldb, just like httpsqs wriiten in C with Tokyo Cabinet.

Also, you can refer to github.com/hnlq715/httpmq-rs#benchmark for the benchmark of Rust implementation.

Feature

  • Very simple, less than 300 lines Go code.
  • Very fast, more than 10000 requests/sec.
  • High concurrency, support the tens of thousands of concurrent connections.
  • Multiple queue.
  • Low memory consumption, mass data storage, storage dozens of GB of data takes less than 100MB of physical memory buffer.
  • Convenient to change the maximum queue length of per-queue.
  • Queue status view.
  • Be able to view the contents of the specified queue ID.
  • Multi-Character encoding support.

Usage

Docker

  docker run -d -it -p 1218:1218 sophos/httpmq

Binary

Usage of ./httpmq:
-auth string
  	auth password to access httpmq
-buffer int
  	write buffer(MB) (default 32)
-cache int
  	cache size(MB) (default 64)
-cpu int
  	cpu number for httpmq (default 4)
-db string
  	database path (default "level.db")
-ip string
  	ip address to listen on (default "0.0.0.0")
-k int
  	keepalive timeout for httpmq (default 60)
-maxqueue int
  	the max queue length (default 1000000)
-port string
  	port to listen on (default "1218")
  1. PUT text message into a queue

HTTP GET protocol (Using curl for example):

curl "http://host:port/?name=your_queue_name&opt=put&data=url_encoded_text_message&auth=mypass123"

HTTP POST protocol (Using curl for example):

curl -d "url_encoded_text_message" "http://host:port/?name=your_queue_name&opt=put&auth=mypass123"
  1. GET text message from a queue

HTTP GET protocol (Using curl for example):

curl "http://host:port/?charset=utf-8&name=your_queue_name&opt=get&auth=mypass123"
  1. View queue status

HTTP GET protocol (Using curl for example):

curl "http://host:port/?name=your_queue_name&opt=status&auth=mypass123"
  1. View queue details

HTTP GET protocol (Using curl for example):

curl "http://host:port/?name=your_queue_name&opt=view&pos=1&auth=mypass123"
  1. Reset queue

HTTP GET protocol (Using curl for example):

curl "http://host:port/?name=your_queue_name&opt=reset&pos=1&auth=mypass123"

Benchmark

Test machine(Hackintosh):

                    'c.          
                 ,xNMM.          ----------------------- 
               .OMMMMo           OS: macOS 11.6.1 20G224 x86_64 
               OMMM0,            Host: Hackintosh (SMBIOS: iMac20,1) 
     .;loddo:' loolloddol;.      Kernel: 20.6.0 
   cKMMMMMMMMMMNWMMMMMMMMMM0:    Uptime: 13 hours, 16 mins 
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.    Packages: 45 (brew) 
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: zsh 5.8 
;MMMMMMMMMMMMMMMMMMMMMMMM:       Resolution: 1920x1080@2x 
:MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua 
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM: Quartz Compositor 
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    WM Theme: Blue (Dark) 
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   Terminal: vscode 
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   CPU: Intel i5-10600K (12) @ 4.10GHz 
    kMMMMMMMMMMMMMMMMMMMMMMd     GPU: Radeon Pro W5500X 
     ;KMMMMMMMWXXWMMMMMMMk.      Memory: 17549MiB / 32768MiB 
       .cooc,.    .,coo:.

fasthttp

PUT queue:

wrk -c 10 -t 2 -d 10s "http://127.0.0.1:1218/?name=xoyo&opt=put&data=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
Running 10s test @ http://127.0.0.1:1218/?name=xoyo&opt=put&data=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   211.87us  169.58us   4.97ms   98.28%
    Req/Sec    24.36k     2.23k   27.87k    64.36%
  489524 requests in 10.10s, 82.64MB read
Requests/sec:  48459.16
Transfer/sec:      8.18MB

GET queue:

wrk -c 10 -t 2 -d 10s "http://127.0.0.1:1218/?name=xoyo&opt=get"
Running 10s test @ http://127.0.0.1:1218/?name=xoyo&opt=get
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   218.26us  251.84us  11.56ms   99.22%
    Req/Sec    23.37k     1.06k   24.70k    90.59%
  469685 requests in 10.10s, 303.14MB read
Requests/sec:  46504.02
Transfer/sec:     30.01MB

More Repositories

1

nginx-vts-exporter

Simple server that scrapes Nginx vts stats and exports them via HTTP for Prometheus consumption
Go
638
star
2

doggy

Lightweight, idiomatic and stable for building Go 1.7+ HTTP services
Go
283
star
3

gofluent

(Not Maintained) Something acting like fluentd rewritten in Go.
Go
180
star
4

nginx-prometheus-metrics

A production demo to collect prometheus metrics for nginx with lua embedded.
Lua
95
star
5

gobreak

Latency and fault tolerance library like Netflix's Hystrix with prometheus and gobreaker.
Go
44
star
6

go-loadbalance

A well designed loadbalance API with dozens of implementations for go-grpc loadbalance.
Go
38
star
7

goroutine-pool

A simple goroutine pool which can create and release goroutine dynamically, inspired by fasthttp.
Go
32
star
8

nginx_upstream_check_module

(Not maintained) This module can be used in pure nginx-1.4.7 and nginx-1.6.2 to check upstream servers, with several patches applied.
C
17
star
9

status-nginx-module

A http status module for pure nginx, which is in production already.
C
14
star
10

graphql-grpc-gateway

Go
10
star
11

zrpc

A lightweight distributed RPC framework powered by pure C language and based on ZeroMQ and pbc.
C
9
star
12

struct2interface

Development helper program that generates a Golang interface by inspecting the structure methods of an existing .go file.
Go
7
star
13

rsa

RAS crypto algorithm for Decrypt with Public Key and Encrypt with Private Key.
Go
6
star
14

httpmq-rs

A simple HTTP message queue written in Rust with rocksdb, just like httpmq written in Go with leveldb.
Rust
3
star
15

lua-resty-fluent

A simple fluent client based on openresty.
Lua
3
star
16

practice-to-intergrate-k8s

A practice to integrate golang, microservice and so on... with k8s
Go
2
star
17

data_struct

Common data structures
C
1
star
18

srt2fcpxml

Go
1
star
19

architecture-best-practice

Best practice from the ground up.
1
star
20

github-trending

A toy to play with weixin mp and github.
Go
1
star
21

proto

Simplify developing HTTP API through Protobuf way.
Go
1
star
22

High-Performance-Browser-Networking-cn

中文翻译
1
star
23

awesome-nginx

A collection of awsome resources around nginx.
1
star
24

ratelimitx

A simple ratelimit for golang, implemented with memcache and gobreak, aims on high availability.
Go
1
star
25

enhanced-limit-conn-nginx-module

An enhanced limit conn nginx module.
C
1
star
26

jumpcutter

An auto jump cutter, implemented with opencv and silence detect.
Python
1
star