• Stars
    star
    816
  • Rank 55,866 (Top 2 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 5 years ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

An implementation of failpoints for Golang.

failpoint

LICENSE Language Go Report Card Build Status Coverage Status Mentioned in Awesome Go

An implementation of failpoints for Golang. Fail points are used to add code points where errors may be injected in a user controlled fashion. Fail point is a code snippet that is only executed when the corresponding failpoint is active.

Quick Start (use failpoint-ctl)

  1. Build failpoint-ctl from source

    git clone https://github.com/pingcap/failpoint.git
    cd failpoint
    make
    ls bin/failpoint-ctl
  2. Inject failpoints to your program, eg:

    package main
    
    import "github.com/pingcap/failpoint"
    
    func main() {
        failpoint.Inject("testPanic", func() {
            panic("failpoint triggerd")
        })
    }
  3. Transfrom your code with failpoint-ctl enable

  4. Build with go build

  5. Enable failpoints with GO_FAILPOINTS environment variable

    GO_FAILPOINTS="main/testPanic=return(true)" ./your-program
  6. If you use go run to run the test, don't forget to add the generated binding__failpoint_binding__.go in your command, like:

    GO_FAILPOINTS="main/testPanic=return(true)" go run your-program.go binding__failpoint_binding__.go

Quick Start (use failpoint-toolexec)

  1. Build failpoint-toolexec from source

    git clone https://github.com/pingcap/failpoint.git
    cd failpoint
    make
    ls bin/failpoint-toolexec
  2. Inject failpoints to your program, eg:

    package main
    
    import "github.com/pingcap/failpoint"
    
    func main() {
        failpoint.Inject("testPanic", func() {
            panic("failpoint triggerd")
        })
    }
  3. Use a separate build cache to avoid mixing caches without failpoint-toolexec, and build

    GOCACHE=/tmp/failpoint-cache go build -toolexec path/to/failpoint-toolexec

  4. Enable failpoints with GO_FAILPOINTS environment variable

    GO_FAILPOINTS="main/testPanic=return(true)" ./your-program
  5. You can also use go run or go test, like:

    GOCACHE=/tmp/failpoint-cache GO_FAILPOINTS="main/testPanic=return(true)" go run -toolexec path/to/failpoint-toolexec your-program.go

Design principles

  • Define failpoint in valid Golang code, not comments or anything else

  • Failpoint does not have any extra cost

    • Will not take effect on regular logic
    • Will not cause regular code performance regression
    • Failpoint code will not appear in the final binary
  • Failpoint routine is writable/readable and should be checked by a compiler

  • Generated code by failpoint definition is easy to read

  • Keep the line numbers same with the injecting codes(easier to debug)

  • Support parallel tests with context.Context

Key concepts

  • Failpoint

    Faillpoint is a code snippet that is only executed when the corresponding failpoint is active. The closure will never be executed if failpoint.Disable("failpoint-name-for-demo") is executed.

    var outerVar = "declare in outer scope"
    failpoint.Inject("failpoint-name-for-demo", func(val failpoint.Value) {
        fmt.Println("unit-test", val, outerVar)
    })
  • Marker functions

    • It is just an empty function

      • To hint the rewriter to rewrite with an equality statement
      • To receive some parameters as the rewrite rule
      • It will be inline in the compiling time and emit nothing to binary (zero cost)
      • The variables in external scope can be accessed in closure by capturing, and the converted code is still legal because all the captured-variables location in outer scope of IF statement.
    • It is easy to write/read

    • Introduce a compiler check for failpoints which cannot compile in the regular mode if failpoint code is invalid

  • Marker funtion list

    • func Inject(fpname string, fpblock func(val Value)) {}
    • func InjectContext(fpname string, ctx context.Context, fpblock func(val Value)) {}
    • func Break(label ...string) {}
    • func Goto(label string) {}
    • func Continue(label ...string) {}
    • func Fallthrough() {}
    • func Return(results ...interface{}) {}
    • func Label(label string) {}
  • Supported failpoint environment variable

    failpoint can be enabled by export environment variables with the following patten, which is quite similar to freebsd failpoint SYSCTL VARIABLES

    [<percent>%][<count>*]<type>[(args...)][-><more terms>]

    The argument specifies which action to take; it can be one of:

    • off: Take no action (does not trigger failpoint code)
    • return: Trigger failpoint with specified argument
    • sleep: Sleep the specified number of milliseconds
    • panic: Panic
    • break: Execute gdb and break into debugger
    • print: Print failpoint path for inject variable
    • pause: Pause will pause until the failpoint is disabled

How to inject a failpoint to your program

  • You can call failpoint.Inject to inject a failpoint to the call site, where failpoint-name is used to trigger the failpoint and failpoint-closure will be expanded as the body of the IF statement.

    failpoint.Inject("failpoint-name", func(val failpoint.Value) {
        failpoint.Return("unit-test", val)
    })

    The converted code looks like:

    if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
        return "unit-test", val
    }
  • failpoint.Value is the value that passes by failpoint.Enable("failpoint-name", "return(5)") which can be ignored.

    failpoint.Inject("failpoint-name", func(_ failpoint.Value) {
        fmt.Println("unit-test")
    })

    OR

    failpoint.Inject("failpoint-name", func() {
        fmt.Println("unit-test")
    })

    And the converted code looks like:

    if _, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
        fmt.Println("unit-test")
    }
  • Also, the failpoint closure can be a function which takes context.Context. You can do some customized things with context.Context like controlling whether a failpoint is active in parallel tests or other cases. For example,

    failpoint.InjectContext(ctx, "failpoint-name", func(val failpoint.Value) {
        fmt.Println("unit-test", val)
    })

    The converted code looks like:

    if val, _err_ := failpoint.EvalContext(ctx, _curpkg_("failpoint-name")); _err_ == nil {
        fmt.Println("unit-test", val)
    }
  • You can ignore context.Context, and this will generate the same code as above non-context version. For example,

    failpoint.InjectContext(nil, "failpoint-name", func(val failpoint.Value) {
        fmt.Println("unit-test", val)
    })

    Becomes

    if val, _err_ := failpoint.EvalContext(nil, _curpkg_("failpoint-name")); _err_ == nil {
        fmt.Println("unit-test", val)
    }
  • You can control a failpoint by failpoint.WithHook

    func (s *dmlSuite) TestCRUDParallel() {
        sctx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {
            return ctx.Value(fpname) != nil // Determine by ctx key
        })
        insertFailpoints = map[string]struct{} {
            "insert-record-fp": {},
            "insert-index-fp": {},
            "on-duplicate-fp": {},
        }
        ictx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {
            _, found := insertFailpoints[fpname] // Only enables some failpoints.
            return found
        })
        deleteFailpoints = map[string]struct{} {
            "tikv-is-busy-fp": {},
            "fetch-tso-timeout": {},
        }
        dctx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {
            _, found := deleteFailpoints[fpname] // Only disables failpoints. 
            return !found
        })
        // other DML parallel test cases.
        s.RunParallel(buildSelectTests(sctx))
        s.RunParallel(buildInsertTests(ictx))
        s.RunParallel(buildDeleteTests(dctx))
    }
  • If you use a failpoint in the loop context, maybe you will use other marker functions.

    failpoint.Label("outer")
    for i := 0; i < 100; i++ {
        inner:
            for j := 0; j < 1000; j++ {
                switch rand.Intn(j) + i {
                case j / 5:
                    failpoint.Break()
                case j / 7:
                    failpoint.Continue("outer")
                case j / 9:
                    failpoint.Fallthrough()
                case j / 10:
                    failpoint.Goto("outer")
                default:
                    failpoint.Inject("failpoint-name", func(val failpoint.Value) {
                        fmt.Println("unit-test", val.(int))
                        if val == j/11 {
                            failpoint.Break("inner")
                        } else {
                            failpoint.Goto("outer")
                        }
                    })
            }
        }
    }

    The above code block will generate the following code:

    outer:
        for i := 0; i < 100; i++ {
        inner:
            for j := 0; j < 1000; j++ {
                switch rand.Intn(j) + i {
                case j / 5:
                    break
                case j / 7:
                    continue outer
                case j / 9:
                    fallthrough
                case j / 10:
                    goto outer
                default:
                    if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
                        fmt.Println("unit-test", val.(int))
                        if val == j/11 {
                            break inner
                        } else {
                            goto outer
                        }
                    }
                }
            }
        }
  • You may doubt why we do not use label, break, continue, and fallthrough directly instead of using failpoint marker functions.

    • Any unused symbol like an ident or a label is not permitted in Golang. It will be invalid if some label is only used in the failpoint closure. For example,

      label1: // compiler error: unused label1
          failpoint.Inject("failpoint-name", func(val failpoint.Value) {
              if val.(int) == 1000 {
                  goto label1 // illegal to use goto here
              }
              fmt.Println("unit-test", val)
          })
    • break and continue can only be used in the loop context, which is not legal in the Golang code if we use them in closure directly.

Some complicated failpoints demo

  • Inject a failpoint to the IF INITIAL statement or CONDITIONAL expression

    if a, b := func() {
        failpoint.Inject("failpoint-name", func(val failpoint.Value) {
            fmt.Println("unit-test", val)
        })
    }, func() int { return rand.Intn(200) }(); b > func() int {
        failpoint.Inject("failpoint-name", func(val failpoint.Value) int {
            return val.(int)
        })
        return rand.Intn(3000)
    }() && b < func() int {
        failpoint.Inject("failpoint-name-2", func(val failpoint.Value) {
            return rand.Intn(val.(int))
        })
        return rand.Intn(6000)
    }() {
        a()
        failpoint.Inject("failpoint-name-3", func(val failpoint.Value) {
            fmt.Println("unit-test", val)
        })
    }

    The above code block will generate something like this:

    if a, b := func() {
        if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
            fmt.Println("unit-test", val)
        }
    }, func() int { return rand.Intn(200) }(); b > func() int {
        if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
            return val.(int)
        }
        return rand.Intn(3000)
    }() && b < func() int {
        if val, ok := failpoint.Eval(_curpkg_("failpoint-name-2")); ok {
            return rand.Intn(val.(int))
        }
        return rand.Intn(6000)
    }() {
        a()
        if val, ok := failpoint.Eval(_curpkg_("failpoint-name-3")); ok {
            fmt.Println("unit-test", val)
        }
    }
  • Inject a failpoint to the SELECT statement to make it block one CASE if the failpoint is active

    func (s *StoreService) ExecuteStoreTask() {
        select {
        case <-func() chan *StoreTask {
            failpoint.Inject("priority-fp", func(_ failpoint.Value) {
                return make(chan *StoreTask)
            })
            return s.priorityHighCh
        }():
            fmt.Println("execute high priority task")
    
        case <- s.priorityNormalCh:
            fmt.Println("execute normal priority task")
    
        case <- s.priorityLowCh:
            fmt.Println("execute normal low task")
        }
    }

    The above code block will generate something like this:

    func (s *StoreService) ExecuteStoreTask() {
        select {
        case <-func() chan *StoreTask {
            if _, ok := failpoint.Eval(_curpkg_("priority-fp")); ok {
                return make(chan *StoreTask)
            })
            return s.priorityHighCh
        }():
            fmt.Println("execute high priority task")
    
        case <- s.priorityNormalCh:
            fmt.Println("execute normal priority task")
    
        case <- s.priorityLowCh:
            fmt.Println("execute normal low task")
        }
    }
  • Inject a failpoint to dynamically extend SWITCH CASE arms

    switch opType := operator.Type(); {
    case opType == "balance-leader":
        fmt.Println("create balance leader steps")
    
    case opType == "balance-region":
        fmt.Println("create balance region steps")
    
    case opType == "scatter-region":
        fmt.Println("create scatter region steps")
    
    case func() bool {
        failpoint.Inject("dynamic-op-type", func(val failpoint.Value) bool {
            return strings.Contains(val.(string), opType)
        })
        return false
    }():
        fmt.Println("do something")
    
    default:
        panic("unsupported operator type")
    }

    The above code block will generate something like this:

    switch opType := operator.Type(); {
    case opType == "balance-leader":
        fmt.Println("create balance leader steps")
    
    case opType == "balance-region":
        fmt.Println("create balance region steps")
    
    case opType == "scatter-region":
        fmt.Println("create scatter region steps")
    
    case func() bool {
        if val, ok := failpoint.Eval(_curpkg_("dynamic-op-type")); ok {
            return strings.Contains(val.(string), opType)
        }
        return false
    }():
        fmt.Println("do something")
    
    default:
        panic("unsupported operator type")
    }
  • More complicated failpoints

    • There are more complicated failpoint sites that can be injected to
      • for the loop INITIAL statement, CONDITIONAL expression and POST statement
      • for the RANGE statement
      • SWITCH INITIAL statement
    • Anywhere you can call a function

Failpoint name best practice

As you see above, _curpkg_ will automatically wrap the original failpoint name in failpoint.Eval call. You can think of _curpkg_ as a macro that automatically prepends the current package path to the failpoint name. For example,

package ddl // which parent package is `github.com/pingcap/tidb`

func demo() {
	// _curpkg_("the-original-failpoint-name") will be expanded as `github.com/pingcap/tidb/ddl/the-original-failpoint-name`
	if val, ok := failpoint.Eval(_curpkg_("the-original-failpoint-name")); ok {...}
}

You do not need to care about _curpkg_ in your application. It is automatically generated after running failpoint-ctl enable and is deleted with failpoint-ctl disable.

Because all failpoints in a package share the same namespace, we need to be careful to avoid name conflict. There are some recommended naming rules to improve this situation.

  • Keep name unique in current subpackage

  • Use a self-explanatory name for the failpoint

    You can enable failpoints by environment variables

    GO_FAILPOINTS="github.com/pingcap/tidb/ddl/renameTableErr=return(100);github.com/pingcap/tidb/planner/core/illegalPushDown=return(true);github.com/pingcap/pd/server/schedulers/balanceLeaderFailed=return(true)"

Implementation details

  1. Define a group of marker functions
  2. Parse imports and prune a source file which does not import a failpoint
  3. Traverse AST to find marker function calls
  4. Marker function calls will be rewritten with an IF statement, which calls failpoint.Eval to determine whether a failpoint is active and executes failpoint code if the failpoint is enabled

rewrite-demo

Acknowledgments

  • Thanks gofail to provide initial implementation.

More Repositories

1

tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
Go
37,055
star
2

talent-plan

open source training courses about distributed database and distributed systems
Rust
10,112
star
3

awesome-database-learning

A list of learning materials to understand databases internals
9,339
star
4

docs-cn

TiDB/TiKV/PD 中文文档
Shell
1,811
star
5

ossinsight

Analysis, Comparison, Trends, Rankings of Open Source Software, you can also get insight from more than 7 billion with natural language (powered by OpenAI). Follow us on Twitter: https://twitter.com/ossinsight
TypeScript
1,744
star
6

parser

A MySQL Compatible SQL Parser
Go
1,409
star
7

tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
Go
1,221
star
8

tiflash

The analytical engine for TiDB and TiDB Cloud. Try free: https://tidbcloud.com/free-trial
C++
944
star
9

tispark

TiSpark is built for running Apache Spark on top of TiDB/TiKV
Scala
883
star
10

go-ycsb

A Go port of Yahoo! Cloud Serving Benchmark (YCSB)
Go
594
star
11

docs

TiDB database documentation. TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
Python
584
star
12

dm

Data Migration Platform
Go
456
star
13

tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Go
426
star
14

tiup

A component manager for TiDB
Go
417
star
15

ossinsight-lite

🚧[WIP] Yet another customizable free GitHub stats dashboard based on TiDB Serverless: https://ossinsight-lite.vercel.app, hand-drawn style.
TypeScript
383
star
16

presentations

367
star
17

tidb-docker-compose

Python
352
star
18

tidb-ansible

Python
326
star
19

tidb-binlog

A tool used to collect and merge tidb's binlog for real-time data backup and synchronization.
Go
292
star
20

tla-plus

TLA
292
star
21

tidb-tools

tidb-tools are some useful tool collections for TiDB.
Go
286
star
22

dumpling

Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).
Go
281
star
23

community

TiDB community content
260
star
24

chaos

A test framework for linearizability check with Go
Go
206
star
25

tidb.ai

https://TiDB.AI is a Graph RAG based and conversational knowledge base tool built with TiDB Serverless Vector Storage and LlamaIndex. Open source and free to use.
TypeScript
186
star
26

go-tpc

A toolbox to benchmark TPC workloads in Go
Go
177
star
27

tidb-dashboard

A Web UI for monitoring, diagnosing and managing the TiDB cluster.
TypeScript
175
star
28

kvproto

Protocol buffer files for TiKV
CMake
152
star
29

tidb-lightning

This repository has been moved to https://github.com/pingcap/br
Go
142
star
30

tipocket

A toolkit for testing TiDB
Go
141
star
31

blog-cn

Shell
126
star
32

br

A command-line tool for distributed backup and restoration of the TiDB cluster data
Go
123
star
33

tidb-dev-guide

A comprehensive development guide to help you be more and more familiar with the TiDB community and become an expert finally.
118
star
34

tidb-bench

A Simple Benchmark For TiDB
C
108
star
35

gdocwiki

A wiki based on Google Doc / Drive
TypeScript
102
star
36

tidb-map

A series of maps to help users and contributors
95
star
37

tipb

TiDB protobuf
CMake
92
star
38

style-guide

Style guide for PingCAP and TiKV code
80
star
39

benchmarksql

Unofficial mirror of benchmarksql on github
Java
79
star
40

go-randgen

a QA tool to random generate sql by bnf pattern
Go
75
star
41

mysql-tester

A Golang implementation of MySQL Test Framework
Go
63
star
42

weekly

57
star
43

tidb-prisma-vercel-demo

Virtual online bookstore application demo which you can find books of various categories and rate the books.
TypeScript
56
star
44

tiproxy

Go
56
star
45

advanced-statefulset

Go
55
star
46

blog

Python
49
star
47

docs-tidb-operator

Documentation for TiDB on Kubernetes. TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
Python
47
star
48

tikv-client-lib-java

TiKV Java client library
Java
44
star
49

tidiff

A toolset to improve efficiency
Go
41
star
50

meetup

37
star
51

fn

Go
35
star
52

tidb-vector-python

TiDB Vector SDK for Python, including code examples. Join our Discord: https://discord.gg/XzSW23Jg9p
Python
35
star
53

tiunimanager

TiUniManager
Go
34
star
54

thirdparty-ops

This repo is used for the operation and maintenance of third party tools.
Python
32
star
55

dead-mans-switch

A bypass monitoring prober
Go
32
star
56

ng-monitoring

Go
31
star
57

django-tidb

TiDB dialect for Django
Python
30
star
58

tidb-course-201-lab

Lab scripts for the PingCAP training course: TiDB SQL for Developers.
Python
30
star
59

tidb-vision

TiDB data visualization
JavaScript
28
star
60

tidb-inspect-tools

Python
27
star
61

monitoring

Shell
26
star
62

activerecord-tidb-adapter

TiDB adapter for ActiveRecord, allows the use of TiDB as a backend for ActiveRecord and Rails apps.
Ruby
24
star
63

diag

A tool to collect diagnostic data from TiDB Clusters
Go
24
star
64

docs-dm

Documentation for the TiDB Data Migration (DM) tool in both English and Chinese.
Python
23
star
65

LinguFlow

LinguFlow, a low-code tool designed for LLM application development, simplifies the building, debugging, and deployment process for developers.
TypeScript
23
star
66

website-docs

The next generation of PingCAP Docs. Powered by Gatsby ⚛️.
TypeScript
22
star
67

book.tidb.net

JavaScript
22
star
68

kdt

Kernel Debug Toolkit
Shell
20
star
69

log

Go
17
star
70

octopus

A toolkit including many powerful distributed test tools
Go
15
star
71

Auto-GPT-TiDB-Serverless-Plugin

Python
15
star
72

errcode

Go
14
star
73

dbt-tidb

A dbt adapter for TiDB
Python
14
star
74

tidb_workload_analysis

Go
14
star
75

k8s-fluent-bit-stackdriver

Shell
11
star
76

website

The website of PingCAP. Powered by Gatsby ⚛️ and Rocket 🚀.
JavaScript
11
star
77

tpcc-mysql

forked from https://code.launchpad.net/~percona-dev/perconatools/tpcc-mysql
C
11
star
78

tidb-insight

Python
11
star
79

tidb-loadbalance

Java
10
star
80

tso

Timestamp Oracle
Go
9
star
81

tiunimanager-ui

A web UI for TiUniManager
TypeScript
9
star
82

tidb-ctl

TiDB Controller
Go
9
star
83

hackernews-insight

Chat to query Hacker News database, based on Auto-GPT and TiDB Cloud Serverless Database
TypeScript
9
star
84

tidb-cloud-backup

Go
8
star
85

wordpress-tidb-plugin

PHP
8
star
86

docs-appdev

Python
7
star
87

wordpress-tidb-docker

WordPress x TiDB Serverless Tier Cluster
Shell
7
star
88

tidb-academy-labs

6
star
89

etcdv3-gateway

Gateway for etcdv3
Go
6
star
90

tispark-test

C
6
star
91

sysutil

sysutil is a library which implementats the gRPC service Diagnostics and shares the diagnostics functions between TiDB and PD.
Go
6
star
92

sqlalchemy-tidb

Python
5
star
93

oasis

Python
5
star
94

homebrew-brew

Homebrew taps for TiDB
Ruby
5
star
95

mysqlrelay

Go
4
star
96

tidb-lmdb

lmdb as storage engine for tidb
Go
4
star
97

cloud-assets-utils

Cloud assets utils by PingCAP FE.
OCaml
4
star
98

mpdriver

MySQL Protocol Driver, used to record MySQL query commands..
Go
4
star
99

tidb-helper

Shell
3
star
100

vldb-boss-2018

Slides and links for VLDB BOSS 2018
3
star