• Stars
    star
    360
  • Rank 117,500 (Top 3 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created almost 10 years ago
  • Updated over 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Access log profiler based on response time

Kataribe

Access log profiler based on response time

Prerequisites

Apache

Add %D to LogFormat.

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D" with_time
CustomLog logs/access_log with_time

Nginx

Add $request_time to log_format.

log_format with_time '$remote_addr - $remote_user [$time_local] '
                     '"$request" $status $body_bytes_sent '
                     '"$http_referer" "$http_user_agent" $request_time';
access_log /var/log/nginx/access.log with_time;

H2O

Add %{duration}x to access-log directive

access-log:
  path: /var/log/h2o/access.log
  format: "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %{duration}x"

Envoy

Use default format

Varnishncsa

Add %D to varnishncsa -F option.

varnishncsa -a -w $logfile -D -P $pidfile -F '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" %D'

Rack

Add Rack::CommonLogger to config.ru.

logger = Logger.new("/tmp/app.log")
use Rack::CommonLogger, logger

Usage

Install via go command or download release file

# go >=1.16
go install github.com/matsuu/kataribe@latest

# go <1.16
go get -u github.com/matsuu/kataribe

Generate kataribe.toml

kataribe -generate

Edit kataribe.toml

${EDITOR} kataribe.toml

Pass access log to kataribe by stdin

cat /path/to/access.log | kataribe

Example

Sort By Count
Count   Total      Mean    Stddev    Min    P50    P90    P95    P99    Max    2xx   3xx  4xx  5xx  Request
17238   0.000  0.000000  0.000000  0.000  0.000  0.000  0.000  0.000  0.000  17238     0    0    0  GET /stylesheets/*
 5746   0.000  0.000000  0.000000  0.000  0.000  0.000  0.000  0.000  0.000   5746     0    0    0  GET /images/*
 5198  12.449  0.002395  0.002292  0.001  0.002  0.003  0.005  0.009  0.069   5198     0    0    0  GET / HTTP/1.1
 2873  22.753  0.007920  0.007529  0.004  0.006  0.011  0.015  0.035  0.193      0  2873    0    0  POST /login HTTP/1.1
  548   2.851  0.005203  0.004015  0.003  0.004  0.007  0.009  0.021  0.066    548     0    0    0  GET /mypage HTTP/1.1
    1   0.303  0.303000  0.000000  0.303  0.303  0.303  0.303  0.303  0.303      0     0    0    1  GET /report HTTP/1.1

Sort By Total
Count   Total      Mean    Stddev    Min    P50    P90    P95    P99    Max    2xx   3xx  4xx  5xx  Request
 2873  22.753  0.007920  0.007529  0.004  0.006  0.011  0.015  0.035  0.193      0  2873    0    0  POST /login HTTP/1.1
 5198  12.449  0.002395  0.002292  0.001  0.002  0.003  0.005  0.009  0.069   5198     0    0    0  GET / HTTP/1.1
  548   2.851  0.005203  0.004015  0.003  0.004  0.007  0.009  0.021  0.066    548     0    0    0  GET /mypage HTTP/1.1
    1   0.303  0.303000  0.000000  0.303  0.303  0.303  0.303  0.303  0.303      0     0    0    1  GET /report HTTP/1.1
17238   0.000  0.000000  0.000000  0.000  0.000  0.000  0.000  0.000  0.000  17238     0    0    0  GET /stylesheets/*
 5746   0.000  0.000000  0.000000  0.000  0.000  0.000  0.000  0.000  0.000   5746     0    0    0  GET /images/*

Sort By Mean
Count   Total      Mean    Stddev    Min    P50    P90    P95    P99    Max    2xx   3xx  4xx  5xx  Request
    1   0.303  0.303000  0.000000  0.303  0.303  0.303  0.303  0.303  0.303      0     0    0    1  GET /report HTTP/1.1
 2873  22.753  0.007920  0.007529  0.004  0.006  0.011  0.015  0.035  0.193      0  2873    0    0  POST /login HTTP/1.1
  548   2.851  0.005203  0.004015  0.003  0.004  0.007  0.009  0.021  0.066    548     0    0    0  GET /mypage HTTP/1.1
 5198  12.449  0.002395  0.002292  0.001  0.002  0.003  0.005  0.009  0.069   5198     0    0    0  GET / HTTP/1.1
17238   0.000  0.000000  0.000000  0.000  0.000  0.000  0.000  0.000  0.000  17238     0    0    0  GET /stylesheets/*
 5746   0.000  0.000000  0.000000  0.000  0.000  0.000  0.000  0.000  0.000   5746     0    0    0  GET /images/*

Sort By Standard Deviation
Count   Total      Mean    Stddev    Min    P50    P90    P95    P99    Max    2xx   3xx  4xx  5xx  Request
 2873  22.753  0.007920  0.007529  0.004  0.006  0.011  0.015  0.035  0.193      0  2873    0    0  POST /login HTTP/1.1
  548   2.851  0.005203  0.004015  0.003  0.004  0.007  0.009  0.021  0.066    548     0    0    0  GET /mypage HTTP/1.1
 5198  12.449  0.002395  0.002292  0.001  0.002  0.003  0.005  0.009  0.069   5198     0    0    0  GET / HTTP/1.1
    1   0.303  0.303000  0.000000  0.303  0.303  0.303  0.303  0.303  0.303      0     0    0    1  GET /report HTTP/1.1
17238   0.000  0.000000  0.000000  0.000  0.000  0.000  0.000  0.000  0.000  17238     0    0    0  GET /stylesheets/*
 5746   0.000  0.000000  0.000000  0.000  0.000  0.000  0.000  0.000  0.000   5746     0    0    0  GET /images/*

Sort By Maximum(100 Percentile)
Count   Total      Mean    Stddev    Min    P50    P90    P95    P99    Max    2xx   3xx  4xx  5xx  Request
    1   0.303  0.303000  0.000000  0.303  0.303  0.303  0.303  0.303  0.303      0     0    0    1  GET /report HTTP/1.1
 2873  22.753  0.007920  0.007529  0.004  0.006  0.011  0.015  0.035  0.193      0  2873    0    0  POST /login HTTP/1.1
 5198  12.449  0.002395  0.002292  0.001  0.002  0.003  0.005  0.009  0.069   5198     0    0    0  GET / HTTP/1.1
  548   2.851  0.005203  0.004015  0.003  0.004  0.007  0.009  0.021  0.066    548     0    0    0  GET /mypage HTTP/1.1
17238   0.000  0.000000  0.000000  0.000  0.000  0.000  0.000  0.000  0.000  17238     0    0    0  GET /stylesheets/*
 5746   0.000  0.000000  0.000000  0.000  0.000  0.000  0.000  0.000  0.000   5746     0    0    0  GET /images/*

TOP 37 Slow Requests
 1  0.303  GET /report HTTP/1.1
 2  0.193  POST /login HTTP/1.1
 3  0.149  POST /login HTTP/1.1
 4  0.108  POST /login HTTP/1.1
 5  0.105  POST /login HTTP/1.1
 6  0.101  POST /login HTTP/1.1
 7  0.084  POST /login HTTP/1.1
 8  0.080  POST /login HTTP/1.1
 9  0.080  POST /login HTTP/1.1
10  0.069  GET / HTTP/1.1
11  0.066  GET /mypage HTTP/1.1
12  0.063  POST /login HTTP/1.1
13  0.063  POST /login HTTP/1.1
14  0.057  POST /login HTTP/1.1
15  0.056  POST /login HTTP/1.1
16  0.056  GET / HTTP/1.1
17  0.054  POST /login HTTP/1.1
18  0.054  POST /login HTTP/1.1
19  0.048  POST /login HTTP/1.1
20  0.046  GET /mypage HTTP/1.1
21  0.045  POST /login HTTP/1.1
22  0.045  POST /login HTTP/1.1
23  0.044  POST /login HTTP/1.1
24  0.042  POST /login HTTP/1.1
25  0.041  GET / HTTP/1.1
26  0.040  POST /login HTTP/1.1
27  0.039  GET / HTTP/1.1
28  0.038  POST /login HTTP/1.1
29  0.038  GET / HTTP/1.1
30  0.037  POST /login HTTP/1.1
31  0.037  POST /login HTTP/1.1
32  0.036  GET / HTTP/1.1
33  0.036  POST /login HTTP/1.1
34  0.036  GET / HTTP/1.1
35  0.036  POST /login HTTP/1.1
36  0.035  POST /login HTTP/1.1
37  0.035  POST /login HTTP/1.1

License

Apache-2.0

Author

matsuu

More Repositories

1

vagrant-isucon

ISUCON過去問を構築するためのVagrantfile集
445
star
2

aws-isucon

ISUCON過去問環境をAWSで再現するための一式まとめ
HCL
223
star
3

cloud-init-isucon

ISUCON過去問環境を構築するためのcloud-config集
96
star
4

docker-isucon

Dockerfile for isucon
Dockerfile
81
star
5

ansible-isucon

Ansible playbooks for ISUCON
Shell
75
star
6

neovim-aarch64-appimage

build nvim.appimage for aarch64(arm64) arch
Dockerfile
55
star
7

isucon12q

Java
44
star
8

azure-isucon-templates

Microsoft Azure Templates for ISUCON
Shell
44
star
9

auwifispot-client

au Wi-Fi SPOT CLI client
Perl
35
star
10

namazu

緊急地震速報(予報)を受け取ってSNSに投げるプログラム一式
Go
33
star
11

go-mysql-query-digest

Alternative to pt-query-digest in Golang
Go
31
star
12

terraform-templates

Template files for Terraform
21
star
13

docker-azure-cli

Dockerfile for azure-cli on Alpine Linux
Dockerfile
20
star
14

docker-nethack

Dockerfile for Nethack 3.6.7
Dockerfile
17
star
15

docker-fc2blog

Dockerfiles for fc2blog
Shell
16
star
16

wsl-isucon

ISUCON過去問と同等の環境をWSL2上で構築するためのスクリプト
Shell
13
star
17

docker-gentoo-distccd

Dockerfile for distccd server on gentoo
Shell
12
star
18

isucon13f

isucon13で8時間戦った記録
PHP
12
star
19

docker-nethack-server

Dockerfile for Public NetHack server
Dockerfile
11
star
20

docker-pt-query-digest

Dockerfile for pt-query-digest(Percona Toolkit)
Dockerfile
10
star
21

oci-arm-isucon

Oracle Cloudの無料枠でISUCON過去問を構築するTerraform一式
HCL
10
star
22

vagrant-yisucon

Vagrantfile for building Yahoo! JAPAN ISUCON environment
9
star
23

docker-pgbadger

Dockerfile for pgBadger
Dockerfile
8
star
24

isucon11q

isucon11予選時のコミットログ
Perl
7
star
25

terraform-isucon

Terraform configuration files for building ISUCON environment
HCL
6
star
26

slowquery2tsv

MySQLのパフォーマンススキーマやPostgreSQLのpg_stat_statementsの内容を整形した上でTSV形式に出力するツール
Go
6
star
27

docker-pixiv-isucon2016

Dockerfile for building Pixiv ISUCON 2016 environment
6
star
28

vagrant-pixiv-isucon2016

Vagrantfile for building Pixiv ISUCON 2016 environment
6
star
29

isucon2-cloudformation

AWS CloudFormation templates for isucon2
5
star
30

docker-jnethack

Dockerfile for JNetHack
Dockerfile
4
star
31

docker-jnethack-server

Dockerfile for Public JNetHack server
3
star
32

imsettings

Input Method selection framework
C
3
star
33

docker-devhub

Dockerfile for DevHub
Shell
3
star
34

powertop

for japanese translation
C
3
star
35

weechat

Fork of git://git.sv.gnu.org/weechat.git
C
3
star
36

terraform-pixiv-isucon2016

Terraform configuration files for building Pixiv ISUCON 2016 environment
HCL
3
star
37

vagrant-isucon11-prior

ISUCON 事前講習2021 ハンズオンのVagrantfile
2
star
38

vagrant-isucon-pass

ISUCONの過去問を予選通過スコアまでチューニングするVagrantfile
Ruby
2
star
39

dart3d

dart 3d with webgl
Dart
2
star
40

gnupg

Fork of upstream at git://git.gnupg.org/gnupg.git
C
2
star
41

isucon12f

ISUCON12本選
PHP
2
star
42

g-cpan

g-cpan
1
star
43

docker-hengband

Dockerfile for 変愚蛮怒(hengband)
Dockerfile
1
star
44

sparkleshare

1
star
45

matsuu.github.com

1
star
46

wsl-isucon11-final

ISUCON11本選の環境をWSL2上に構築するスクリプト
Shell
1
star
47

wsl-isucon9-qualifier

ISUCON9予選の環境をWSL2上に構築するスクリプト
Shell
1
star
48

konoha

1
star
49

try_git

1
star
50

ccs-tools

TOMOYO Linux tools
C
1
star
51

wsl-isucon10-qualifier

ISUCON10予選の環境をWSL2上に構築するスクリプト
Shell
1
star
52

hackernews

hackernews RSS description generator
Python
1
star
53

wsl-isucon11-qualifier

ISUCON11予選の環境をWSL2上に構築するスクリプト
Shell
1
star
54

KoboRoot

telnetd enabled KoboRoot
Shell
1
star
55

docker-webscalesql

Dockerfile fo webscalesql
Shell
1
star
56

vagrant-kayac-isucon-2022

カヤック社内ISUCONとほぼ同じ環境を構築するためのVagrantfile
1
star
57

ansible-isucon-pass

ISUCONで予選を突破できるスコアまでチューニングを施すansible playbook
Go
1
star
58

packer-isucon

1
star