• Stars
    star
    195
  • Rank 199,374 (Top 4 %)
  • Language
    C
  • License
    GNU General Publi...
  • Created over 4 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Interrupts-off or softirqs-off latency tracer

Trace-irqoff

我们的需求是什么

在实际问题中,业务经常会遇到网络延迟高问题,这种问题分析下来。基本是如下几种可能原因:

  • 中断关闭时间太长
  • softirq 关闭时间太长

以上是我们根据经验猜测可能出现的原因,实际问题中,我迫切的需要确定是否以上原因导致问题。如果是的话,具体是什么原因导致以上两种情况发生呢?因此,我们迫切需要定位具体的元凶,使其现形。所以,我们的需求是开发一个工具可以追踪和定位中断或者软中断关闭的时间。这款工具现在已经开发完成,名为:Interrupts-off or softirqs-off latency tracer,简称 trace-irqoff。

如何安装

安装 trace-irqoff 工具很简单,git clone代码后执行如下命令即可安装。

make -j8
make install

如何使用

安装 trace-irqoff 工具成功后。会创建如下 /proc/trace_irqoff 目录。

root@n18-061-206:/proc/trace_irqoff# ls
distribute  enable  sampling_period  trace_latency

/proc/trace_irqoff 目录下存在 4 个文件,分别:distribute, enable, sampling_period 和 trace_latency。工具安装后,默认是关闭状态,我们需要手动打开 trace。

1. 打开 trace
echo 1 > /proc/trace_irqoff/enable
2. 关闭 trace
echo 0 > /proc/trace_irqoff/enable
3. 设置 trace 阈值

trace-irqoff 工具只会针对关闭中断或者软中断时间超过阈值的情况下记录堆栈信息。因此我们可以通过如下命令查看当前 trace 的阈值:

cat /proc/trace_irqoff/trace_latency
trace_irqoff_latency: 50ms
 hardirq:
 softirq:

默认阈值是 50ms,如第 2 行所示。第 4 行输出 hardirq: 代表下面的栈是可能关闭中断超过阈值的栈。同理,第 6 行是软中断关闭时间超过阈值的栈。

如果需要修改阈值至 100ms 可通过如下命令(写入值单位是 ms):

echo 100 > /proc/trace_irqoff/trace_latency
4. 清除栈信息

当然如果需要清除 /proc/trace_irqoff 记录的栈信息。可以执行如下命令(不会修改阈值为 0):

echo 0 > /proc/trace_irqoff/trace_latency
5. 查看中断关闭次数的统计信息

如果我们需要知道中断被关闭一定的时间的次数,可以通过如下命令获取统计信息。

root@n18-061-206:/proc/trace_irqoff# cat distribute
hardirq-off:
     msecs      : count   distribution
    20 -> 39    : 1     |**********                              |
    40 -> 79    : 0     |                                        |
    80 -> 159   : 4     |****************************************|
   160 -> 319   : 2     |********************                    |
   320 -> 639   : 1     |**********                              |
softirq-off:
     msecs      : count   distribution
    20 -> 39    : 0     |                                        |
    40 -> 79    : 0     |                                        |
    80 -> 159   : 0     |                                        |
   160 -> 319   : 1     |****************************************|

在这个例子中,我们看到hardirq被关闭时间x ∈ [80, 159] ms,次数4次。softirq被关闭时间x ∈ [160, 319] ms,次数1次

如果没有任何信息输出,这说明没有任何地方关闭中断时间超过20ms。

6. 修改采样周期

从上面一节我们可以看到,中断关闭时间分布图最小粒度是 20ms。这是因为采样周期是 10ms。根据采样定理,大于等于 2 倍采样周期时间才能反映真实情况。如果需要提高统计粒度,可修改采样周期时间。例如修改采样周期为 1ms,可执行如下命令(必须在 tracer 关闭的情况下操作有效):

# 单位 ms,可设置最小的采样周期是 1ms。
echo 1 > /proc/trace_irqoff/sampling_period

案例分析

1. hardirq 关闭

我们使用如下示意测试程序,关闭中断 100ms。查看 trace_irqoff 文件内容。

static void disable_hardirq(unsigned long latency)
{
    local_irq_disable();
    mdelay(latency);
    local_irq_enable();
}

通过模块测试以上代码,然后查看栈信息。

cat /proc/trace_irqoff/trace_latency
trace_irqoff_latency: 50ms
 hardirq:
 cpu: 17
   COMMAND: bash PID: 22840 LATENCY: 107ms
   trace_irqoff_hrtimer_handler+0x39/0x99 [trace_irqoff]
   __hrtimer_run_queues+0xfa/0x270
   hrtimer_interrupt+0x101/0x240
   smp_apic_timer_interrupt+0x5e/0x120
   apic_timer_interrupt+0xf/0x20
   disable_hardirq+0x5b/0x70
   proc_reg_write+0x36/0x60
   __vfs_write+0x33/0x190
   vfs_write+0xb0/0x190
   ksys_write+0x52/0xc0
   do_syscall_64+0x4f/0xe0
   entry_SYSCALL_64_after_hwframe+0x44/0xa9
 softirq:

我们可以看到 hardirq 一栏记录 cpu17 执行 bash 命令,关闭中断 107ms(误差 10ms 之内)。其栈信息对应disable_hardirq() 函数中。第 20 行 softirq 一栏没有信息,说明没有记录 softirq 被关闭的栈。

2. softirq 关闭

我们使用如下示意测试程序,关闭 softirq 100ms。查看 trace_irqoff 文件内容。

static void disable_softirq(unsigned long latency)
{
    local_bh_disable();
    mdelay(latency);
    local_bh_enable();
}

通过模块测试以上代码,然后查看栈信息。

cat /proc/trace_irqoff/trace_latency
trace_irqoff_latency: 50ms
 hardirq:
 softirq:
 cpu: 17
   COMMAND: bash PID: 22840 LATENCY: 51+ms
   trace_irqoff_hrtimer_handler+0x97/0x99 [trace_irqoff]
   __hrtimer_run_queues+0xfa/0x270
   hrtimer_interrupt+0x101/0x240
   smp_apic_timer_interrupt+0x5e/0x120
   apic_timer_interrupt+0xf/0x20
   delay_tsc+0x3c/0x50
   disable_softirq+0x4b/0x80
   proc_reg_write+0x36/0x60
   __vfs_write+0x33/0x190
   vfs_write+0xb0/0x190
   ksys_write+0x52/0xc0
   do_syscall_64+0x4f/0xe0
   entry_SYSCALL_64_after_hwframe+0x44/0xa9

   COMMAND: bash PID: 22840 LATENCY: 106ms
   trace_irqoff_timer_handler+0x3a/0x60 [trace_irqoff]
   call_timer_fn+0x29/0x120
   run_timer_softirq+0x16c/0x400
   __do_softirq+0x108/0x2b8
   do_softirq_own_stack+0x2a/0x40
   do_softirq.part.21+0x56/0x60
   __local_bh_enable_ip+0x60/0x70
   disable_softirq+0x62/0x80
   proc_reg_write+0x36/0x60
   __vfs_write+0x33/0x190
   vfs_write+0xb0/0x190
   ksys_write+0x52/0xc0
   do_syscall_64+0x4f/0xe0
   entry_SYSCALL_64_after_hwframe+0x44/0xa9

针对 softirq 关闭情况,有 2 个栈与之对应。我们注意到第 9 行的函数名称和第 24 行的函数名称是不一样的。第 9 行的栈是硬件中断 handler 捕捉到软中断关闭,第 24 行是软中断 handler 捕捉到软中断被关闭。正常情况下,我们以 24 行开始的栈为分析目标即可。当 24 行的栈是无效的时候,可以看第 9 行的栈。这里注意:第 9 行的 lantency 提示信息 51+ms 是阈值信息。并非实际 latency(所以我在后面添加一个'+'字符,表示latency大于51ms)。实际的 latency 是第 24 行显示的 106ms。下面就看下为什么 2 个栈是有必要的。

3. ksoftirqd 延迟

我们看一个曾经处理的一个实际问题。

cat /proc/trace_irqoff/trace_latency
trace_irqoff_latency: 300ms
 hardirq:
 softirq:
 cpu: 4
   COMMAND: lxcfs PID: 4058797 LATENCY: 303+ms
   trace_irqoff_record+0x12b/0x1b0 [trace_irqoff]
   trace_irqoff_hrtimer_handler+0x97/0x99 [trace_irqoff]
   __hrtimer_run_queues+0xdc/0x220
   hrtimer_interrupt+0xa6/0x1f0
   smp_apic_timer_interrupt+0x62/0x120
   apic_timer_interrupt+0x7d/0x90
   memcg_sum_events.isra.26+0x3f/0x60
   memcg_stat_show+0x323/0x460
   seq_read+0x11f/0x3f0
   __vfs_read+0x33/0x160
   vfs_read+0x91/0x130
   SyS_read+0x52/0xc0
   do_syscall_64+0x68/0x100
   entry_SYSCALL_64_after_hwframe+0x3d/0xa2

   COMMAND: ksoftirqd/4 PID: 34 LATENCY: 409ms
   trace_irqoff_record+0x12b/0x1b0 [trace_irqoff]
   trace_irqoff_timer_handler+0x3a/0x60 [trace_irqoff]
   call_timer_fn+0x2e/0x130
   run_timer_softirq+0x1d4/0x420
   __do_softirq+0x108/0x2a9
   run_ksoftirqd+0x1e/0x40
   smpboot_thread_fn+0xfe/0x150
   kthread+0xfc/0x130
   ret_from_fork+0x1f/0x30

我们看到下面的进程 ksoftirqd/4 的栈,延迟时间是 409ms。ksoftirqd 进程是 kernel 中处理 softirq 的进程。因此这段栈对我们是没有意义的,因为元凶已经错过了。所以此时,我们可以借鉴上面的栈信息,我们看到当 softirq 被延迟 303ms 的时候,当前 CPU 正在执行的进程是 lxcfs。并且栈是 memory cgroup 相关。因此,我们基本可以判断 lxcfs 进程执行时间过长,由于 kernel 态不支持抢占,因此导致 ksoftirqd 进程没有机会得到运行。

More Repositories

1

IconPark

🍎Transform an SVG icon into multiple themes, and generate React icons,Vue icons,svg icons
TypeScript
8,298
star
2

xgplayer

A HTML5 video player with a parser that saves traffic
JavaScript
8,260
star
3

sonic

A blazingly fast JSON serializing & deserializing library
Assembly
6,870
star
4

monoio

Rust async runtime based on io-uring.
Rust
3,864
star
5

byteps

A high performance and generic framework for distributed DNN training
Python
3,603
star
6

lightseq

LightSeq: A High Performance Library for Sequence Processing and Generation
C++
3,193
star
7

ByteX

ByteX is a bytecode plugin platform based on Android Gradle Transform API and ASM. 字节码插件开发平台
Java
2,865
star
8

Elkeid

Elkeid is an open source solution that can meet the security requirements of various workloads such as hosts, containers and K8s, and serverless. It is derived from ByteDance's internal best practices.
Go
2,226
star
9

AlphaPlayer

AlphaPlayer is a video animation engine.
Java
2,181
star
10

scene

Android Single Activity Framework compatible with Fragment.
Java
2,097
star
11

bhook

🔥 ByteHook is an Android PLT hook library which supports armeabi-v7a, arm64-v8a, x86 and x86_64.
C
2,073
star
12

flutter_ume

UME is an in-app debug kits platform for Flutter. Produced by Flutter Infra team of ByteDance
Dart
2,053
star
13

terarkdb

A RocksDB compatible KV storage engine with better performance
C++
2,044
star
14

btrace

🔥🔥 btrace(AKA RheaTrace) is a high performance Android trace tool which is based on Perfetto, it support to define custom events automatically during building apk and using bhook to provider more native events like Render/Binder/IO etc.
Kotlin
1,913
star
15

gopkg

Universal Utilities for Go
Go
1,704
star
16

android-inline-hook

🔥 ShadowHook is an Android inline hook library which supports thumb, arm32 and arm64.
C
1,660
star
17

bitsail

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
Java
1,627
star
18

go-tagexpr

An interesting go struct tag expression syntax for field validation, etc.
Go
1,470
star
19

GiantMIDI-Piano

Python
1,431
star
20

appshark

Appshark is a static taint analysis platform to scan vulnerabilities in an Android app.
Kotlin
1,363
star
21

AabResGuard

The tool of obfuscated aab resources.(Android app bundle资源混淆工具)
Java
1,307
star
22

piano_transcription

Python
1,247
star
23

CodeLocator

Kotlin
1,163
star
24

BoostMultiDex

BoostMultiDex is a solution for quickly loading multiple dex files on low Android version devices (4.X and below, SDK <21).
Java
1,106
star
25

music_source_separation

Python
1,039
star
26

Fastbot_Android

Fastbot(2.0) is a model-based testing tool for modeling GUI transitions to discover app stability problems
C++
1,031
star
27

SALMONN

SALMONN: Speech Audio Language Music Open Neural Network
Python
1,000
star
28

memory-leak-detector

C
919
star
29

fedlearner

A multi-party collaborative machine learning framework
Python
892
star
30

monolith

ByteDance's Recommendation System
Python
844
star
31

sonic-cpp

A fast JSON serializing & deserializing library, accelerated by SIMD.
C++
811
star
32

godlp

sensitive information protection toolkit
Go
770
star
33

MVDream

Multi-view Diffusion for 3D Generation
Python
744
star
34

res-adapter

Official implementation of "ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models".
Python
724
star
35

bytemd

ByteMD v1 repository
TypeScript
679
star
36

tailor

C
669
star
37

ibot

iBOT 🤖: Image BERT Pre-Training with Online Tokenizer (ICLR 2022)
Jupyter Notebook
663
star
38

RealRichText

A Tricky Solution for Implementing Inline-Image-In-Text Feature in Flutter.
Dart
658
star
39

guide

A new feature guide component by react 🧭
TypeScript
651
star
40

mockey

a simple and easy-to-use golang mock library
Go
622
star
41

magic-microservices

Make Web Components easier and powerful!😘
TypeScript
570
star
42

Fastbot_iOS

About Fastbot(2.0) is a model-based testing tool for modeling GUI transitions to discover app stability problems
Objective-C
553
star
43

flow-builder

A highly customizable streaming flow builder.
TypeScript
526
star
44

MVDream-threestudio

3D generation code for MVDream
Python
473
star
45

effective_transformer

Running BERT without Padding
C++
457
star
46

ByteTransformer

optimized BERT transformer inference on NVIDIA GPU. https://arxiv.org/abs/2210.03052
C++
449
star
47

Next-ViT

Python
426
star
48

matxscript

A high-performance, extensible Python AOT compiler.
C++
408
star
49

byteir

A model compilation solution for various hardware
MLIR
362
star
50

syllepsis

Syllepsis is an out-of-the-box rich text editor.
TypeScript
355
star
51

uss

This is the PyTorch implementation of the Universal Source Separation with Weakly labelled Data.
Python
324
star
52

OMGD

Online Multi-Granularity Distillation for GAN Compression (ICCV2021)
Python
323
star
53

neurst

Neural end-to-end Speech Translation Toolkit
Python
298
star
54

danmu.js

HTML5 danmu (danmaku) plugin for any DOM element
JavaScript
292
star
55

vArmor

vArmor is a cloud native container sandbox system based on AppArmor/BPF/Seccomp. It also includes multiple built-in protection rules that are ready to use out of the box.
Go
263
star
56

particle-sfm

ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild. ECCV 2022.
C++
263
star
57

CloudShuffleService

Cloud Shuffle Service(CSS) is a general purpose remote shuffle solution for compute engines, including Spark/Flink/MapReduce.
Java
245
star
58

lynx-llm

paper: https://arxiv.org/abs/2307.02469 page: https://lynx-llm.github.io/
Python
227
star
59

g3

Enterprise-oriented Generic Proxy Solutions
Rust
227
star
60

xgplayer-vue

Vue component for xgplayer, a HTML5 video player with a parser that saves traffic
JavaScript
219
star
61

DEADiff

[CVPR 2024] Official implementation of "DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations"
Python
209
star
62

flux

A fast communication-overlapping library for tensor parallelism on GPUs.
C++
201
star
63

ParaGen

ParaGen is a PyTorch deep learning framework for parallel sequence generation.
Python
186
star
64

ByteMLPerf

AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and versatility of software and hardware.
Python
181
star
65

MoMA

MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation
Jupyter Notebook
177
star
66

AWERTL

An non-invasive iOS framework for quickly adapting Right-To-Left style UI
Objective-C
175
star
67

Bytedance-UnionAD

Ruby
170
star
68

keyhouse

Keyhouse is a skeleton of general-purpose Key Management System written in Rust.
Rust
163
star
69

react-model

The next generation state management library for React
TypeScript
162
star
70

LargeBatchCTR

Large batch training of CTR models based on DeepCTR with CowClip.
Python
162
star
71

ic_flow_platform

IFP (ic flow platform) is an integrated circuit design flow platform, mainly used for IC process specification management and data flow contral.
Python
154
star
72

DanmakuRenderEngine

DanmakuRenderEngine is a lightweight and scalable Android danmaku library. 轻量级高扩展安卓弹幕渲染引擎
Kotlin
149
star
73

primus

Java
148
star
74

diat

A CLI tool to help with diagnosing Node.js processes basing on inspector.
JavaScript
146
star
75

coconut_cvpr2024

Jupyter Notebook
143
star
76

Hammer

An efficient toolkit for training deep models.
Python
138
star
77

ns-x

An easy-to-use, flexible network simulator library in Go.
Go
116
star
78

pv3d

Python
113
star
79

fc-clip

This repo contains the code for our paper Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
Python
109
star
80

RLFN

Winner of runtime track in NTIRE 2022 challenge on Efficient Super-Resolution
Python
106
star
81

DCFrame

DCFrame is a Swift UI collection framework, which can easily create complex UI.
Swift
100
star
82

trace-noschedule

Trace noschedule thread
C
99
star
83

decoupleQ

A quantization algorithm for LLM
Cuda
99
star
84

tar-wasm

A faster experimental wasm-based tar implementation for browsers.
Rust
95
star
85

TWIST

Official codes: Self-Supervised Learning by Estimating Twin Class Distribution
Python
95
star
86

magic-portal

⚡ A blazing fast micro-component and micro-frontend solution uses web-components under the hood.
TypeScript
91
star
87

xgplayer-react

React component for xgplayer, a HTML5 video player with a parser that saves traffic
JavaScript
84
star
88

fe-foundation

UI Foundation for React Hooks and Vue Composition Api
TypeScript
80
star
89

nnproxy

Scalable NameNode RPC Proxy for HDFS Federation
Java
79
star
90

dbatman

Go
74
star
91

Elkeid-HUB

Elkeid HUB is a rule/event processing engine maintained by the Elkeid Team that supports streaming/offline (not yet supported by the community edition) data processing. The original intention is to solve complex data/event processing and external system linkage requirements through standardized rules.
Python
74
star
92

FreeSeg

Python
69
star
93

pull_to_refresh

Flutter pull_to_refresh widget
Dart
67
star
94

Jeddak-DPSQL

DPSQL (Privacy Protection SQL Query Service) - This project is a microservice Middleware located between the database engine ( Hive , Clickhouse , etc.) and the application system. It provides transparent SQL query result desensitization capabilities.
Python
62
star
95

terark-zip

A data structure and algorithm library built for TerarkDB
C++
62
star
96

trace-runqlat

C
61
star
97

ipmb

An interprocess message bus system built in Rust.
Rust
60
star
98

X-Portrait

Source code for the SIGGRAPH 2024 paper "X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention"
Python
59
star
99

kernel

ByteDance kernel for use on cloud.
C
57
star
100

scroll_kit

Dart
56
star