Discover k1LoW/runn Open Source project by Ken’ichiro Oyama (@k1LoW)

runn ( means "Run N". is pronounced /rʌ́n én/. ) is a package/tool for running operations following a scenario.

Key features of runn are:

As a tool for scenario based testing.
As a test helper package for the Go language.
As a tool for workflow automation.
Support HTTP request, gRPC request, DB query, Chrome DevTools Protocol, and SSH/Local command execution
OpenAPI Document-like syntax for HTTP request testing.
Single binary = CI-Friendly.

Online book

runn cookbook (Japanese)

Quickstart

You can use the runn new command to quickly start creating scenarios (runbooks).

🚀 Create and run scenario using curl or grpcurl commands:

Command details

$ curl https://httpbin.org/json -H "accept: application/json"
{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Overview",
        "type": "all"
      }
    ],
    "title": "Sample Slide Show"
  }
}
$ runn new --and-run --desc 'httpbin.org GET' --out http.yml -- curl https://httpbin.org/json -H "accept: application/json"
$ grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello
{
  "reply": "hello alice"
}
$ runn new --and-run --desc 'grpcb.in Call' --out grpc.yml -- grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello
$ runn list *.yml
  Desc             Path      If
---------------------------------
  grpcb.in Call    grpc.yml
  httpbin.org GET  http.yml
$ runn run *.yml
..

2 scenarios, 0 skipped, 0 failures

🚀 Create scenario using access log:

Command details

$ cat access_log
183.87.255.54 - - [18/May/2019:05:37:09 +0200] "GET /?post=%3script%3ealert(1); HTTP/1.0" 200 42433
62.109.16.162 - - [18/May/2019:05:37:12 +0200] "GET /core/files/js/editor.js/?form=\xeb\x2a\x5e\x89\x76\x08\xc6\x46\x07\x00\xc7\x46\x0c\x00\x00\x00\x80\xe8\xdc\xff\xff\xff/bin/sh HTTP/1.0" 200 81956
87.251.81.179 - - [18/May/2019:05:37:13 +0200] "GET /login.php/?user=admin&amount=100000 HTTP/1.0" 400 4797
103.36.79.144 - - [18/May/2019:05:37:14 +0200] "GET /authorize.php/.well-known/assetlinks.json HTTP/1.0" 200 9436
$ cat access_log| runn new --out axslog.yml
$ cat axslog.yml| yq
desc: Generated by `runn new`
runners:
  req: https://dummy.example.com
steps:
  - req:
      /?post=%3script%3ealert(1);:
        get:
          body: null
  - req:
      /core/files/js/editor.js/?form=xebx2ax5ex89x76x08xc6x46x07x00xc7x46x0cx00x00x00x80xe8xdcxffxffxff/bin/sh:
        get:
          body: null
  - req:
      /login.php/?user=admin&amount=100000:
        get:
          body: null
  - req:
      /authorize.php/.well-known/assetlinks.json:
        get:
          body: null
$

Usage

runn can run a multi-step scenario following a runbook written in YAML format.

As a tool for scenario based testing / As a tool for automation.

runn can run one or more runbooks as a CLI tool.

$ runn list path/to/**/*.yml
  id:      desc:             if:       steps:  path
-------------------------------------------------------------------------
  a1b7b02  Only if included  included       2  p/t/only_if_included.yml
  85ccd5f  List projects.                   4  p/t/p/list.yml
  47d7ef7  List users.                      3  p/t/u/list.yml
  97f9884  Login                            2  p/t/u/login.yml
  2249d1b  Logout                           3  p/t/u/logout.yml
$ runn run path/to/**/*.yml
S....

5 scenarios, 1 skipped, 0 failures

As a test helper package for the Go language.

runn can also behave as a test helper for the Go language.

Run N runbooks using httptest.Server and sql.DB

func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	ts := httptest.NewServer(NewRouter(db))
	t.Cleanup(func() {
		ts.Close()
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Runner("req", ts.URL),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.Load("testdata/books/**/*.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}

Run single runbook using httptest.Server and sql.DB

func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	ts := httptest.NewServer(NewRouter(db))
	t.Cleanup(func() {
		ts.Close()
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Book("testdata/books/login.yml"),
		runn.Runner("req", ts.URL),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.New(opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.Run(ctx); err != nil {
		t.Fatal(err)
	}
}

Run N runbooks using grpc.Server

func TestServer(t *testing.T) {
	addr := "127.0.0.1:8080"
	l, err := net.Listen("tcp", addr)
	if err != nil {
		t.Fatal(err)
	}
	ts := grpc.NewServer()
	myapppb.RegisterMyappServiceServer(s, NewMyappServer())
	reflection.Register(s)
	go func() {
		s.Serve(l)
	}()
	t.Cleanup(func() {
		ts.GracefulStop()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Runner("greq", fmt.Sprintf("grpc://%s", addr),
	}
	o, err := runn.Load("testdata/books/**/*.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}

Run N runbooks with http.Handler and sql.DB

func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	t.Cleanup(func() {
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.HTTPRunnerWithHandler("req", NewRouter(db)),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.Load("testdata/books/**/*.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}

Examples

See the details

Runbook ( runn scenario file )

The runbook file has the following format.

step: section accepts list or ordered map.

List:

desc: Login and get projects.
runners:
  req: https://example.com/api/v1
  db: mysql://root:mypass@localhost:3306/testdb
vars:
  username: alice
  password: ${TEST_PASS}
steps:
  -
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  -
    req:
      /login:
        post:
          body:
            application/json:
              email: "{{ steps[0].rows[0].email }}"
              password: "{{ vars.password }}"
    test: steps[1].res.status == 200
  -
    req:
      /projects:
        get:
          headers:
            Authorization: "token {{ steps[1].res.body.session_token }}"
          body: null
    test: steps[2].res.status == 200
  -
    test: len(steps[2].res.body.projects) > 0

Map:

desc: Login and get projects.
runners:
  req: https://example.com/api/v1
  db: mysql://root:mypass@localhost:3306/testdb
vars:
  username: alice
  password: ${TEST_PASS}
steps:
  find_user:
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  login:
    req:
      /login:
        post:
          body:
            application/json:
              email: "{{ steps.find_user.rows[0].email }}"
              password: "{{ vars.password }}"
    test: steps.login.res.status == 200
  list_projects:
    req:
      /projects:
        get:
          headers:
            Authorization: "token {{ steps.login.res.body.session_token }}"
          body: null
    test: steps.list_projects.res.status == 200
  count_projects:
    test: len(steps.list_projects.res.body.projects) > 0

Grouping of related parts by color

List:

Map:

`desc:`

Description of runbook.

`runners:`

Mapping of runners that run steps: of runbook.

In the steps: section, call the runner with the key specified in the runners: section.

Built-in runners such as test runner do not need to be specified in this section.

runners:
  ghapi: ${GITHUB_API_ENDPOINT}
  idp: https://auth.example.com
  db: my:dbuser:${DB_PASS}@hostname:3306/dbname

In the example, each runner can be called by ghapi:, idp: or db: in steps:.

`vars:`

Mapping of variables available in the steps: of runbook.

vars:
  username: [email protected]
  token: ${SECRET_TOKEN}

In the example, each variable can be used in {{ vars.username }} or {{ vars.token }} in steps:.

`debug:`

Enable debug output for runn.

debug: true

`if:`

Conditions for skip all steps.

if: included # Run steps only if included

`skipTest:`

Skip all test: sections

skipTest: true

`force:`

Force all steps to run.

force: true

`loop:`

Loop setting for runbook.

Simple loop runbook

loop: 10
steps:
  [...]

loop:
  count: 10
steps:
  [...]

Retry runbook

It can be used as a retry mechanism by setting a condition in the until: section.

If the condition of until: is met, the loop is broken without waiting for the number of count: to be run.

Also, if the run of the number of count: completes but does not satisfy the condition of until:, then the step is considered to be failed.

loop:
  count: 10
  until: 'outcome == "success"' # until the runbook outcome is successful.
  minInterval: 0.5 # sec
  maxInterval: 10  # sec
  # jitter: 0.0
  # interval: 5
  # multiplier: 1.5
steps:
  waitingroom:
    req:
      /cart/in:
        post:
          body:
[...]

outcome ... the result of a completed (success, failure, skipped).

`concurrency:`

Runbooks with the same key are assured of a single run at the same time.

concurrency: use-shared-db

`steps:`

Steps to run in runbook.

The steps are invoked in order from top to bottom.

Any return values are recorded for each step.

When steps: is array, recorded values can be retrieved with {{ steps[*].* }}.

steps:
  -
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  -
    req:
      /users/{{ steps[0].rows[0].id }}:
        get:
          body: null

When steps: is map, recorded values can be retrieved with {{ steps.<key>.* }}.

steps:
  find_user:
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  user_info:
    req:
      /users/{{ steps.find_user.rows[0].id }}:
        get:
          body: null

`steps[*].desc:` `steps.<key>.desc:`

Description of step.

`steps[*].if:` `steps.<key>.if:`

Conditions for skip step.

steps:
  login:
    if: 'len(vars.token) == 0' # Run step only if var.token is not set
    req:
      /login:
        post:
          body:
[...]

`steps[*].loop:` `steps.<key>.loop:`

Loop settings for steps.

Simple loop step

steps:
  multicartin:
    loop: 10
    req:
      /cart/in:
        post:
          body:
            application/json:
              product_id: "{{ i }}" # The loop count (0..9) is assigned to `i`.
[...]

steps:
  multicartin:
    loop:
      count: 10
    req:
      /cart/in:
        post:
          body:
            application/json:
              product_id: "{{ i }}" # The loop count (0..9) is assigned to `i`.
[...]

Retry step

It can be used as a retry mechanism by setting a condition in the until: section.

If the condition of until: is met, the loop is broken without waiting for the number of count: to be run.

Also, if the run of the number of count: completes but does not satisfy the condition of until:, then the step is considered to be failed.

steps:
  waitingroom:
    loop:
      count: 10
      until: 'steps.waitingroom.res.status == "201"' # Store values of latest loop
      minInterval: 500ms
      maxInterval: 10 # sec
      # jitter: 0.0
      # interval: 5
      # multiplier: 1.5
    req:
      /cart/in:
        post:
          body:
[...]

( steps[*].retry: steps.<key>.retry: are deprecated )

Variables to be stored

runn can use variables and functions when running step.

Also, after step runs, HTTP responses, DB query results, etc. are automatically stored in variables.

The values are stored in predefined variables.

Variable name	Description
`vars`	Values set in the `vars:` section
`steps`	Return values for each step
`i`	Loop index (only in `loop:` section)
`env`	Environment variables
`current`	Return values of current step
`previous`	Return values of previous step
`parent`	Variables of parent runbook (only included)

Runner

HTTP Runner: Do HTTP request

Use https:// or http:// scheme to specify HTTP Runner.

When the step is invoked, it sends the specified HTTP Request and records the response.

runners:
  req: https://example.com
steps:
  -
    desc: Post /users                     # description of step
    req:                                  # key to identify the runner. In this case, it is HTTP Runner.
      /users:                             # path of http request
        post:                             # method of http request
          headers:                        # headers of http request
            Authorization: 'Bearer xxxxx'
          body:                           # body of http request
            application/json:             # Content-Type specification. In this case, it is "Content-Type: application/json"
              username: alice
              password: passw0rd
    test: |                               # test for current step
      current.res.status == 201

See testdata/book/http.yml and testdata/book/http_multipart.yml.

Structure of recorded responses

The following response

HTTP/1.1 200 OK
Content-Length: 29
Content-Type: application/json
Date: Wed, 07 Sep 2022 06:28:20 GMT
Set-Cookie: cookie-name=cookie-value

{"data":{"username":"alice"}}

is recorded with the following structure.

[`step key` or `current` or `previous`]:
  res:
    status: 200                              # current.res.status
    headers:
      Content-Length:
        - '29'                               # current.res.headers["Content-Length"][0]
      Content-Type:
        - 'application/json'                 # current.res.headers["Content-Type"][0]
      Date:
        - 'Wed, 07 Sep 2022 06:28:20 GMT'    # current.res.headers["Date"][0]
      Set-Cookie:
        - 'cookie-name=cookie-value'         # current.res.headers["Set-Cookie"][0]
    cookies:
      cookie-name: *http.Cookie              # current.res.cookies["cookie-name"].Value
    body:
      data:
        username: 'alice'                    # current.res.body.data.username
    rawBody: '{"data":{"username":"alice"}}' # current.res.rawBody

Do not follow redirect

The HTTP Runner interprets HTTP responses and automatically redirects. To disable this, set notFollowRedirect to true.

runners:
  req:
    endpoint: https://example.com
    notFollowRedirect: true

Enable Cookie Sending

The HTTP Runner automatically saves cookies by interpreting HTTP responses. To enable cookie sending during requests, set useCookie to true.

runners:
  req:
    endpoint: https://example.com
    useCookie: true

See testdata/book/cookie.yml and testdata/book/cookie_in_requests_automatically.yml.

Validation of HTTP request and HTTP response

HTTP requests sent by runn and their HTTP responses can be validated.

OpenAPI v3:

runners:
  myapi:
    endpoint: https://api.github.com
    openapi3: path/to/openapi.yaml
    # skipValidateRequest: false
    # skipValidateResponse: false

Custom CA and Certificates

runners:
  myapi:
    endpoint: https://api.github.com
    cacert: path/to/cacert.pem
    cert: path/to/cert.pem
    key: path/to/key.pem
    # skipVerify: false

gRPC Runner: Do gRPC request

Use grpc:// scheme to specify gRPC Runner.

When the step is invoked, it sends the specified gRPC Request and records the response.

runners:
  greq: grpc://grpc.example.com:80
steps:
  -
    desc: Request using Unary RPC                     # description of step
    greq:                                             # key to identify the runner. In this case, it is gRPC Runner.
      grpctest.GrpcTestService/Hello:                 # package.Service/Method of rpc
        headers:                                      # headers of rpc
          authentication: tokenhello
        message:                                      # message of rpc
          name: alice
          num: 3
          request_time: 2022-06-25T05:24:43.861872Z
  -
    desc: Request using Server streaming RPC
    greq:
      grpctest.GrpcTestService/ListHello:
        headers:
          authentication: tokenlisthello
        message:
          name: bob
          num: 4
          request_time: 2022-06-25T05:24:43.861872Z
        timeout: 3sec                                 # timeout for rpc
    test: |
      steps.server_streaming.res.status == 0 && len(steps.server_streaming.res.messages) > 0
  -
    desc: Request using Client streaming RPC
    greq:
      grpctest.GrpcTestService/MultiHello:
        headers:
          authentication: tokenmultihello
        messages:                                     # messages of rpc
          -
            name: alice
            num: 5
            request_time: 2022-06-25T05:24:43.861872Z
          -
            name: bob
            num: 6
            request_time: 2022-06-25T05:24:43.861872Z

runners:
  greq:
    addr: grpc.example.com:8080
    tls: true
    cacert: path/to/cacert.pem
    cert: path/to/cert.pem
    key: path/to/key.pem
    # skipVerify: false
    # protos:
    #   - general/health.proto
    #   - myapp/**/*.proto
    # importPaths:
    #   - protobuf/proto

See testdata/book/grpc.yml.

Structure of recorded responses

The following response

message HelloResponse {
  string message = 1;

  int32 num = 2;

  google.protobuf.Timestamp create_time = 3;
}

{"create_time":"2022-06-25T05:24:43.861872Z","message":"hello","num":32}

and headers

content-type: ["application/grpc"]
hello: ["this is header"]

and trailers

hello: ["this is trailer"]

are recorded with the following structure.

[`step key` or `current` or `previous`]:
  res:
    status: 0                                      # current.res.status
    headers:
      content-type:
        - 'application/grpc'                       # current.res.headers[0].content-type
      hello:
        - 'this is header'                         # current.res.headers[0].hello
    trailers:
      hello:
        - 'this is trailer'                        # current.res.trailers[0].hello
    message:
      create_time: '2022-06-25T05:24:43.861872Z'   # current.res.message.create_time
      message: 'hello'                             # current.res.message.message
      num: 32                                      # current.res.message.num
    messages:
      -
        create_time: '2022-06-25T05:24:43.861872Z' # current.res.messages[0].create_time
        message: 'hello'                           # current.res.messages[0].message
        num: 32                                    # current.res.messages[0].num

DB Runner: Query a database

Use dsn (Data Source Name) to specify DB Runner.

When step is invoked, it executes the specified query the database.

runners:
  db: postgres://dbuser:dbpass@hostname:5432/dbname
steps:
  -
    desc: Select users            # description of step
    db:                           # key to identify the runner. In this case, it is DB Runner.
      query: SELECT * FROM users; # query to execute

See testdata/book/db.yml.

Structure of recorded responses

If the query is a SELECT clause, it records the selected rows,

[`step key` or `current` or `previous`]:
  rows:
    -
      id: 1                           # current.rows[0].id
      username: 'alice'               # current.rows[0].username
      password: 'passw0rd'            # current.rows[0].password
      email: '[email protected]'      # current.rows[0].email
      created: '2017-12-05T00:00:00Z' # current.rows[0].created
    -
      id: 2                           # current.rows[1].id
      username: 'bob'                 # current.rows[1].username
      password: 'passw0rd'            # current.rows[1].password
      email: '[email protected]'        # current.rows[1].email
      created: '2022-02-22T00:00:00Z' # current.rows[1].created

otherwise it records last_insert_id and rows_affected .

[`step key` or `current` or `previous`]:
  last_insert_id: 3 # current.last_insert_id
  rows_affected: 1  # current.rows_affected

Support Databases

PostgreSQL:

runners:
  mydb: postgres://dbuser:dbpass@hostname:5432/dbname

runners:
  db: pg://dbuser:dbpass@hostname:5432/dbname

MySQL:

runners:
  testdb: mysql://dbuser:dbpass@hostname:3306/dbname

runners:
  db: my://dbuser:dbpass@hostname:3306/dbname

SQLite3:

runners:
  db: sqlite:///path/to/dbname.db

runners:
  local: sq://dbname.db

Cloud Spanner:

runners:
  testdb: spanner://test-project/test-instance/test-database

runners:
  db: sp://test-project/test-instance/test-database

CDP Runner: Control browser using Chrome DevTools Protocol (CDP)

Use cdp:// or chrome:// scheme to specify CDP Runner.

When the step is invoked, it controls browser via Chrome DevTools Protocol.

runners:
  cc: chrome://new
steps:
  -
    desc: Navigate, click and get h1 using CDP  # description of step
    cc:                                         # key to identify the runner. In this case, it is CDP Runner.
      actions:                                  # actions to control browser
        - navigate: https://pkg.go.dev/time
        - click: 'body > header > div.go-Header-inner > nav > div > ul > li:nth-child(2) > a'
        - waitVisible: 'body > footer'
        - text: 'h1'
  -
    test: |
      previous.text == 'Install the latest version of Go'

See testdata/book/cdp.yml.

Functions for action to control browser

attributes (aliases: getAttributes, attrs, getAttrs)

Get the element attributes for the first element node matching the selector (sel).

actions:
  - attributes:
      sel: 'h1'
# record to current.attrs:

actions:
  - attributes: 'h1'

click

Send a mouse click event to the first element node matching the selector (sel).

actions:
  - click:
      sel: 'nav > div > a'

actions:
  - click: 'nav > div > a'

doubleClick

Send a mouse double click event to the first element node matching the selector (sel).

actions:
  - doubleClick:
      sel: 'nav > div > li'

actions:
  - doubleClick: 'nav > div > li'

evaluate (aliases: eval)

Evaluate the Javascript expression (expr).

actions:
  - evaluate:
      expr: 'document.querySelector("h1").textContent = "hello"'

actions:
  - evaluate: 'document.querySelector("h1").textContent = "hello"'

fullHTML (aliases: getFullHTML, getHTML, html)

Get the full html of page.

actions:
  - fullHTML
# record to current.html:

innerHTML (aliases: getInnerHTML)

Get the inner html of the first element node matching the selector (sel).

actions:
  - innerHTML:
      sel: 'h1'
# record to current.html:

actions:
  - innerHTML: 'h1'

latestTab (aliases: latestTarget)

Change current frame to latest tab.

actions:
  - latestTab

localStorage (aliases: getLocalStorage)

Get localStorage items.

actions:
  - localStorage:
      origin: 'https://github.com'
# record to current.items:

actions:
  - localStorage: 'https://github.com'

location (aliases: getLocation)

Get the document location.

actions:
  - location
# record to current.url:

navigate

Navigate the current frame to url page.

actions:
  - navigate:
      url: 'https://pkg.go.dev/time'

actions:
  - navigate: 'https://pkg.go.dev/time'

outerHTML (aliases: getOuterHTML)

Get the outer html of the first element node matching the selector (sel).

actions:
  - outerHTML:
      sel: 'h1'
# record to current.html:

actions:
  - outerHTML: 'h1'

screenshot (aliases: getScreenshot)

Take a full screenshot of the entire browser viewport.

actions:
  - screenshot
# record to current.png:

scroll (aliases: scrollIntoView)

Scroll the window to the first element node matching the selector (sel).

actions:
  - scroll:
      sel: 'body > footer'

actions:
  - scroll: 'body > footer'

sendKeys

Send keys (value) to the first element node matching the selector (sel).

actions:
  - sendKeys:
      sel: 'input[name=username]'
      value: '[email protected]'

sessionStorage (aliases: getSessionStorage)

Get sessionStorage items.

actions:
  - sessionStorage:
      origin: 'https://github.com'
# record to current.items:

actions:
  - sessionStorage: 'https://github.com'

setUploadFile (aliases: setUpload)

Set upload file (path) to the first element node matching the selector (sel).

actions:
  - setUploadFile:
      sel: 'input[name=avator]'
      path: '/path/to/image.png'

setUserAgent (aliases: setUA, ua, userAgent)

Set the default User-Agent

actions:
  - setUserAgent:
      userAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'

actions:
  - setUserAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'

submit

Submit the parent form of the first element node matching the selector (sel).

actions:
  - submit:
      sel: 'form.login'

actions:
  - submit: 'form.login'

text (aliases: getText)

Get the visible text of the first element node matching the selector (sel).

actions:
  - text:
      sel: 'h1'
# record to current.text:

actions:
  - text: 'h1'

textContent (aliases: getTextContent)

Get the text content of the first element node matching the selector (sel).

actions:
  - textContent:
      sel: 'h1'
# record to current.text:

actions:
  - textContent: 'h1'

title (aliases: getTitle)

Get the document title.

actions:
  - title
# record to current.title:

value (aliases: getValue)

Get the Javascript value field of the first element node matching the selector (sel).

actions:
  - value:
      sel: 'input[name=address]'
# record to current.value:

actions:
  - value: 'input[name=address]'

wait (aliases: sleep)

Wait for the specified time.

actions:
  - wait:
      time: '10sec'

actions:
  - wait: '10sec'

waitReady

Wait until the element matching the selector (sel) is ready.

actions:
  - waitReady:
      sel: 'body > footer'

actions:
  - waitReady: 'body > footer'

waitVisible

Wait until the element matching the selector (sel) is visible.

actions:
  - waitVisible:
      sel: 'body > footer'

actions:
  - waitVisible: 'body > footer'

SSH Runner: execute commands on a remote server connected via SSH

Use ssh:// scheme to specify SSH Runner.

When step is invoked, it executes commands on a remote server connected via SSH.

runners:
  sc: ssh://username@hostname:port
steps:
  -
    desc: 'execute `hostname`' # description of step
    sc:
      command: hostname

runners:
  sc:
    hostname: hostname
    user: username
    port: 22
    # host: myserver
    # sshConfig: path/to/ssh_config
    # keepSession: false
    # localForward: '33306:127.0.0.1:3306'
    # keyboardInteractive:
    #   - match: Username
    #     answer: k1low
    #   - match: OTP
    #     answer: ${MY_OTP}

See testdata/book/sshd.yml.

Structure of recorded responses

The response to the run command is always stdout and stderr.

[`step key` or `current` or `previous`]:
  stdout: 'hello world' # current.stdout
  stderr: ''            # current.stderr

Exec Runner: execute command

The exec runner is a built-in runner, so there is no need to specify it in the runners: section.

It execute command using command: and stdin:

-
  exec:
    command: grep hello
    stdin: '{{ steps[3].res.rawBody }}'

See testdata/book/exec.yml.

Structure of recorded responses

The response to the run command is always stdout, stderr and exit_code.

[`step key` or `current` or `previous`]:
  stdout: 'hello world' # current.stdout
  stderr: ''            # current.stderr
  exit_code: 0          # current.exit_code

Test Runner: test using recorded values

The test runner is a built-in runner, so there is no need to specify it in the runners: section.

It evaluates the conditional expression using the recorded values.

-
  test: steps[3].res.status == 200

The test runner can run in the same steps as the other runners.

Dump Runner: dump recorded values

The dump runner is a built-in runner, so there is no need to specify it in the runners: section.

It dumps the specified recorded values.

-
  dump: steps[4].rows

-
  dump:
    expr: steps[4].rows
    out: path/to/dump.out

The dump runner can run in the same steps as the other runners.

Include Runner: include other runbook

The include runner is a built-in runner, so there is no need to specify it in the runners: section.

Include runner reads and runs the runbook in the specified path.

Recorded values are nested.

-
  include: path/to/get_token.yml

It is also possible to override vars: of included runbook.

-
  include:
    path: path/to/login.yml
    vars:
      username: alice
      password: alicepass
-
  include:
    path: path/to/login.yml
    vars:
      username: bob
      password: bobpass

It is also possible to skip all test: sections in the included runbook.

-
  include:
    path: path/to/signup.yml
    skipTest: true

It is also possible to force all steps in the included runbook to run.

-
  include:
    path: path/to/signup.yml
    force: true

Bind Runner: bind variables

The bind runner is a built-in runner, so there is no need to specify it in the runners: section.

It bind runner binds any values with another key.

  -
    req:
      /users/k1low:
        get:
          body: null
  -
    bind:
      user_id: steps[0].res.body.data.id
  -
    dump: user_id

The bind runner can run in the same steps as the other runners.

Expression evaluation engine

runn has embedded antonmedv/expr as the evaluation engine for the expression.

See Language Definition.

Additional built-in functions

urlencode ... url.QueryEscape
bool ... cast.ToBool
compare ... Compare two values ( func(x, y any, ignoreKeys ...string) bool ).
diff ... Difference between two values ( func(x, y any, ignoreKeys ...string) string ).
input ... prompter.Prompt
intersect ... Find the intersection of two iterable values ( func(x, y any) any ).
secret ... prompter.Password
select ... prompter.Choose
basename ... filepath.Base
faker.* ... Generate fake data using Faker ).

Option

See https://pkg.go.dev/github.com/k1LoW/runn#Option

Example: Run as a test helper ( func `T` )

https://pkg.go.dev/github.com/k1LoW/runn#T

o, err := runn.Load("testdata/**/*.yml", runn.T(t))
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}

Example: Add custom function ( func `Func` )

https://pkg.go.dev/github.com/k1LoW/runn#Func

desc: Test using GitHub
runners:
  req:
    endpoint: https://github.com
steps:
  -
    req:
      /search?l={{ urlencode('C++') }}&q=runn&type=Repositories:
        get:
          body:
            application/json:
              null
    test: 'steps[0].res.status == 200'

o, err := runn.Load("testdata/**/*.yml", runn.Func("urlencode", url.QueryEscape))
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}

Filter runbooks to be executed by the environment variable `RUNN_RUN`

Run only runbooks matching the filename "login".

$ env RUNN_RUN=login go test ./... -run TestRouter

Measure elapsed time as profile

opts := []runn.Option{
	runn.T(t),
	runn.Book("testdata/books/login.yml"),
	runn.Profile(true)
}
o, err := runn.New(opts...)
if err != nil {
	t.Fatal(err)
}
if err := o.Run(ctx); err != nil {
	t.Fatal(err)
}
f, err := os.Open("profile.json")
if err != nil {
	t.Fatal(err)
}
if err := o.DumpProfile(f); err != nil {
	t.Fatal(err)
}

$ runn run testdata/books/login.yml --profile

The runbook run profile can be read with runn rprof command.

$ runn rprof runn.prof
  runbook[login site](t/b/login.yml)           2995.72ms
    steps[0].req                                747.67ms
    steps[1].req                                185.69ms
    steps[2].req                                192.65ms
    steps[3].req                                188.23ms
    steps[4].req                                569.53ms
    steps[5].req                                299.88ms
    steps[6].test                                 0.14ms
    steps[7].include                            620.88ms
      runbook[include](t/b/login_include.yml)   605.56ms
        steps[0].req                            605.54ms
    steps[8].req                                190.92ms
  [total]                                      2995.84ms

Capture runbook runs

opts := []runn.Option{
	runn.T(t),
	runn.Capture(capture.Runbook("path/to/dir")),
}
o, err := runn.Load("testdata/books/**/*.yml", opts...)
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}

$ runn run path/to/**/*.yml --capture path/to/dir

Load test using runbooks

You can use the runn loadt command for load testing using runbooks.

$ runn loadt --load-concurrent 2 --max-rps 0 path/to/*.yml

Number of runbooks per RunN....: 15
Warm up time (--warm-up).......: 5s
Duration (--duration)..........: 10s
Concurrent (--load-concurrent).: 2
Max RunN per second (--max-rps): 0

Total..........................: 12
Succeeded......................: 12
Failed.........................: 0
Error rate.....................: 0%
RunN per seconds...............: 1.2
Latency .......................: max=1,835.1ms min=1,451.3ms avg=1,627.8ms med=1,619.8ms p(90)=1,741.5ms p(99)=1,788.4ms

It also checks the results of the load test with the --threshold option. If the condition is not met, it returns exit status 1.

$ runn loadt --load-concurrent 2 --max-rps 0 --threshold 'error_rate < 10' path/to/*.yml

Number of runbooks per RunN...: 15
Warm up time (--warm-up)......: 5s
Duration (--duration).........: 10s
Concurrent (--load-concurrent): 2

Total.........................: 13
Succeeded.....................: 12
Failed........................: 1
Error rate....................: 7.6%
RunN per seconds..............: 1.3
Latency ......................: max=1,790.2ms min=95.0ms avg=1,541.4ms med=1,640.4ms p(90)=1,749.7ms p(99)=1,786.5ms

Error: (error_rate < 10) is not true
error_rate < 10
├── error_rate => 14.285714285714285
└── 10 => 10

Variables for threshold

Variable name	Type	Description
`total`	`int`	Total
`succeeded`	`int`	Succeeded
`failed`	`int`	Failed
`error_rate`	`float`	Error rate
`rps`	`float`	RunN per seconds
`max`	`float`	Latency max (ms)
`mid`	`float`	Latency mid (ms)
`min`	`float`	Latency min (ms)
`p90`	`float`	Latency p(90) (ms)
`p99`	`float`	Latency p(99) (ms)
`avg`	`float`	Latency avg (ms)

Install

As a CLI tool

deb:

$ export RUNN_VERSION=X.X.X
$ curl -o runn.deb -L https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.deb
$ dpkg -i runn.deb

RPM:

$ export RUNN_VERSION=X.X.X
$ yum install https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.rpm

apk:

$ export RUNN_VERSION=X.X.X
$ curl -o runn.apk -L https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.apk
$ apk add runn.apk

homebrew tap:

$ brew install k1LoW/tap/runn

manually:

Download binary from releases page

docker:

$ docker container run -it --rm --name runn -v $PWD:/books ghcr.io/k1low/runn:latest list /books/*.yml

go install:

$ go install github.com/k1LoW/runn/cmd/runn@latest

As a test helper

$ go get github.com/k1LoW/runn

Alternatives

zoncoen/scenarigo: An end-to-end scenario testing tool for HTTP/gRPC server.

References

zoncoen/scenarigo: An end-to-end scenario testing tool for HTTP/gRPC server.
fullstorydev/grpcurl: Like cURL, but for gRPC: Command-line tool for interacting with gRPC servers
ktr0731/evans: Evans: more expressive universal gRPC client

k1LoW/runn

k1LoW

Reviews

Repository Details

Online book

Quickstart

Usage

As a tool for scenario based testing / As a tool for automation.

As a test helper package for the Go language.

Run N runbooks using httptest.Server and sql.DB

Run single runbook using httptest.Server and sql.DB

Run N runbooks using grpc.Server

Run N runbooks with http.Handler and sql.DB

Examples

Runbook ( runn scenario file )

Grouping of related parts by color

desc:

runners:

vars:

debug:

if:

skipTest:

force:

loop:

Simple loop runbook

Retry runbook

concurrency:

steps:

steps[*].desc: steps.<key>.desc:

steps[*].if: steps.<key>.if:

steps[*].loop: steps.<key>.loop:

Simple loop step

Retry step

Variables to be stored

Runner

HTTP Runner: Do HTTP request

Structure of recorded responses

Do not follow redirect

Enable Cookie Sending

Validation of HTTP request and HTTP response

Custom CA and Certificates

gRPC Runner: Do gRPC request

Structure of recorded responses

DB Runner: Query a database

Structure of recorded responses

Support Databases

CDP Runner: Control browser using Chrome DevTools Protocol (CDP)

Functions for action to control browser

SSH Runner: execute commands on a remote server connected via SSH

Structure of recorded responses

Exec Runner: execute command

Structure of recorded responses

Test Runner: test using recorded values

Dump Runner: dump recorded values

Include Runner: include other runbook

Bind Runner: bind variables

Expression evaluation engine

Additional built-in functions

Option

Example: Run as a test helper ( func T )

Example: Add custom function ( func Func )

Filter runbooks to be executed by the environment variable RUNN_RUN

Measure elapsed time as profile

Capture runbook runs

Load test using runbooks

Variables for threshold

Install

As a CLI tool

As a test helper

Alternatives

References

More Repositories

`desc:`

`runners:`

`vars:`

`debug:`

`if:`

`skipTest:`

`force:`

`loop:`

`concurrency:`

`steps:`

`steps[*].desc:` `steps.<key>.desc:`

`steps[*].if:` `steps.<key>.if:`

`steps[*].loop:` `steps.<key>.loop:`

Example: Run as a test helper ( func `T` )

Example: Add custom function ( func `Func` )

Filter runbooks to be executed by the environment variable `RUNN_RUN`