• Stars
    star
    185
  • Rank 200,764 (Top 5 %)
  • Language
    Python
  • License
    MIT License
  • Created over 4 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

pytest-timeout

python version anaconda ci pre-commit

Warning

Please read this README carefully and only use this plugin if you understand the consequences. This plugin is designed to catch excessively long test durations like deadlocked or hanging tests, it is not designed for precise timings or performance regressions. Remember your test suite should aim to be fast, with timeouts being a last resort, not an expected failure mode.

This plugin will time each test and terminate it when it takes too long. Termination may or may not be graceful, please see below, but when aborting it will show a stack dump of all thread running at the time. This is useful when running tests under a continuous integration server or simply if you don't know why the test suite hangs.

Note

While by default on POSIX systems pytest will continue to execute the tests after a test has timed out this is not always possible. Often the only sure way to interrupt a hanging test is by terminating the entire process. As this is a hard termination (os._exit()) it will result in no teardown, JUnit XML output etc. But the plugin will ensure you will have the debugging output on stderr nevertheless, which is the most important part at this stage. See below for detailed information on the timeout methods and their side-effects.

The pytest-timeout plugin has been tested on Python 3.6 and higher, including PyPy3. See tox.ini for currently tested versions.

Usage

Install is as simple as e.g.:

pip install pytest-timeout

Now you can run the test suite while setting a timeout in seconds, any individual test which takes longer than the given duration will be terminated:

pytest --timeout=300

Furthermore you can also use a decorator to set the timeout for an individual test. If combined with the --timeout flag this will override the timeout for this individual test:

@pytest.mark.timeout(60)
def test_foo():
    pass

By default the plugin will not time out any tests, you must specify a valid timeout for the plugin to interrupt long-running tests. A timeout is always specified as a number of seconds, and can be defined in a number of ways, from low to high priority:

  1. You can set a global timeout in the pytest configuration file using the timeout option. E.g.:

    [pytest]
    timeout = 300
  2. The PYTEST_TIMEOUT environment variable sets a global timeout overriding a possible value in the configuration file.
  3. The --timeout command line option sets a global timeout overriding both the environment variable and configuration option.
  4. Using the timeout marker on test items you can specify timeouts on a per-item basis:

    @pytest.mark.timeout(300)
    def test_foo():
        pass

Setting a timeout to 0 seconds disables the timeout, so if you have a global timeout set you can still disable the timeout by using the mark.

Timeout Methods

Interrupting tests which hang is not always as simple and can be platform dependent. Furthermore some methods of terminating a test might conflict with the code under test itself. The pytest-timeout plugin tries to pick the most suitable method based on your platform, but occasionally you may need to specify a specific timeout method explicitly.

If a timeout method does not work your safest bet is to use the thread method.

thread

This is the surest and most portable method. It is also the default on systems not supporting the signal method. For each test item the pytest-timeout plugin starts a timer thread which will terminate the whole process after the specified timeout. When a test item finishes this timer thread is cancelled and the test run continues.

The downsides of this method are that there is a relatively large overhead for running each test and that test runs are not completed. This means that other pytest features, like e.g. JUnit XML output or fixture teardown, will not function normally. The second issue might be alleviated by using the --boxed option of the pytest-xdist plugin.

The benefit of this method is that it will always work. Furthermore it will still provide you debugging information by printing the stacks of all the threads in the application to stderr.

signal

If the system supports the SIGALRM signal the signal method will be used by default. This method schedules an alarm when the test item starts and cancels the alarm when the test finishes. If the alarm expires during the test the signal handler will dump the stack of any other threads running to stderr and use pytest.fail() to interrupt the test.

The benefit of this method is that the pytest process is not terminated and the test run can complete normally.

The main issue to look out for with this method is that it may interfere with the code under test. If the code under test uses SIGALRM itself things will go wrong and you will have to choose the thread method.

Specifying the Timeout Method

The timeout method can be specified by using the timeout_method option in the pytest configuration file, the --timeout_method command line parameter or the timeout marker. Simply set their value to the string thread or signal to override the default method. On a marker this is done using the method keyword:

@pytest.mark.timeout(method="thread")
def test_foo():
    pass

The timeout Marker API

The full signature of the timeout marker is:

pytest.mark.timeout(timeout=0, method=DEFAULT_METHOD)

You can use either positional or keyword arguments for both the timeout and the method. Neither needs to be present.

See the marker api documentation and examples for the various ways markers can be applied to test items.

Timeouts in Fixture Teardown

The plugin will happily terminate timeouts in the finalisers of fixtures. The timeout specified applies to the entire process of setting up fixtures, running the tests and finalising the fixtures. However when a timeout occurs in a fixture finaliser and the test suite continues, i.e. the signal method is used, it must be realised that subsequent fixtures which need to be finalised might not have been executed, which could result in a broken test-suite anyway. In case of doubt the thread method which terminates the entire process might result in clearer output.

Avoiding timeouts in Fixtures

The timeout applies to the entire test including any fixtures which may need to be setup or torn down for the test (the exact affected fixtures depends on which scope they are and whether other tests will still use the same fixture). If the timeouts really are too short to include fixture durations, firstly make the timeouts larger ;). If this really isn't an option a timeout_func_only boolean setting exists which can be set in the pytest ini configuration file, as documented in pytest --help.

For the decorated function, a decorator will override timeout_func_only = true in the pytest ini file to the default value. If you need to keep this option for a decorated test, you must specify the option explicitly again:

@pytest.mark.timeout(60, func_only=True)
def test_foo():
    pass

Debugger Detection

This plugin tries to avoid triggering the timeout when a debugger is detected. This is mostly a convenience so you do not need to remember to disable the timeout when interactively debugging.

The way this plugin detects whether or not a debugging session is active is by checking if a trace function is set and if one is, it check to see if the module it belongs to is present in a set of known debugging frameworks modules OR if pytest itself drops you into a pdb session using --pdb or similar.

This functionality can be disabled with the --disable-debugger-detection flag or the corresponding timeout_disable_debugger_detection ini setting / environment variable.

Extending pytest-timeout with plugins

pytest-timeout provides two hooks that can be used for extending the tool. These hooks are used for setting the timeout timer and cancelling it if the timeout is not reached.

For example, pytest-asyncio can provide asyncio-specific code that generates better traceback and points on timed out await instead of the running loop iteration.

See pytest hooks documentation for more info regarding to use custom hooks.

pytest_timeout_set_timer

@pytest.hookspec(firstresult=True)
def pytest_timeout_set_timer(item, settings):
    """Called at timeout setup.

    'item' is a pytest node to setup timeout for.

    'settings' is Settings namedtuple (described below).

    Can be overridden by plugins for alternative timeout implementation strategies.
    """

Settings

When pytest_timeout_set_timer is called, settings argument is passed.

The argument has Settings namedtuple type with the following fields:

Attribute Index Value
timeout 0 timeout in seconds or None for no timeout
method 1 Method mechanism, 'signal' and 'thread' are supported by default
func_only 2
Apply timeout to test function only if True,

wrap all test function and its fixtures otherwise

pytest_timeout_cancel_timer

@pytest.hookspec(firstresult=True)
def pytest_timeout_cancel_timer(item):
    """Called at timeout teardown.

    'item' is a pytest node which was used for timeout setup.

    Can be overridden by plugins for alternative timeout implementation strategies.
    """

is_debugging

When the timeout occurs, user can open the debugger session. In this case, the timeout should be discarded. A custom hook can check this case by calling is_debugging() function:

import pytest
import pytest_timeout


def on_timeout():
    if pytest_timeout.is_debugging():
        return
    pytest.fail("+++ Timeout +++")

Session Timeout

The above mentioned timeouts are all per test function. The "per test function" timeouts will stop an individual test from taking too long. We may also want to limit the time of the entire set of tests running in one session. A session all of the tests that will be run with one invokation of pytest.

A session timeout is set with --session-timeout and is in seconds.

The following example shows a session timeout of 10 minutes (600 seconds):

pytest --session-timeout=600

You can also set the session timeout the pytest configuration file using the session_timeout option:

[pytest]
session_timeout = 600

Cooperative timeouts

Session timeouts are cooperative timeouts. pytest-timeout checks the session time at the end of each test function, and stops further tests from running if the session timeout is exceeded. The session will results in a test failure if this occurs.

In particular this means if a test does not finish of itself, it will only be interrupted if there is also a function timeout set. A session timeout is not enough to ensure that a test-suite is guaranteed to finish.

Combining session and function timeouts

It works fine to combine both session and function timeouts. In fact when using a session timeout it is recommended to also provide a function timeout.

For example, to limit test functions to 5 seconds and the full session to 100 seconds:

pytest --timeout=5 --session-timeout=100

Changelog

2.3.1

  • Fixup some build errors, mostly README syntax which stopped twine from uploading.

2.3.0

  • Fix debugger detection for recent VSCode, this compiles pydevd using cython which is now correctly detected. Thanks Adrian Gielniewski.
  • Switched to using Pytest's TerminalReporter instead of writing directly to sys.{stdout,stderr}. This change also switches all output from sys.stderr to sys.stdout. Thanks Pedro Algarvio.
  • Pytest 7.0.0 is now the minimum supported version. Thanks Pedro Algarvio.
  • Add --session-timeout option and session_timeout setting. Thanks Brian Okken.

2.2.0

  • Add --timeout-disable-debugger-detection flag, thanks Michael Peters

2.1.0

  • Get terminal width from shutil instead of deprecated py, thanks Andrew Svetlov.
  • Add an API for extending pytest-timeout functionality with third-party plugins, thanks Andrew Svetlov.

2.0.2

  • Fix debugger detection on OSX, thanks Alexander Pacha.

2.0.1

  • Fix Python 2 removal, thanks Nicusor Picatureanu.

2.0.0

  • Increase pytest requirement to >=5.0.0. Thanks Dominic Davis-Foster.
  • Use thread timeout method when plugin is not called from main thread to avoid crash.
  • Fix pycharm debugger detection so timeouts are not triggered during debugger usage.
  • Dropped support for Python 2, minimum pytest version supported is 5.0.0.

1.4.2

  • Fix compatibility when run with pytest pre-releases, thanks Bruno Oliveira,
  • Fix detection of third-party debuggers, thanks Bruno Oliveira.

1.4.1

  • Fix coverage compatibility which was broken by 1.4.0.

1.4.0

  • Better detection of when we are debugging, thanks Mattwmaster58.

1.3.4

  • Give the threads a name to help debugging, thanks Thomas Grainger.
  • Changed location to https://github.com/pytest-dev/pytest-timeout because bitbucket is dropping mercurial support. Thanks Thomas Grainger and Bruno Oliveira.

1.3.3

  • Fix support for pytest >= 3.10.

1.3.2

  • This changelog was omitted for the 1.3.2 release and was added afterwards. Apologies for the confusion.
  • Fix pytest 3.7.3 compatibility. The capture API had changed slightly and this needed fixing. Thanks Bruno Oliveira for the contribution.

1.3.1

  • Fix deprecation warning on Python 3.6. Thanks Mickaël Schoentgen
  • Create a valid tag for the release. Somehow this didn't happen for 1.3.0, that tag points to a non-existing commit.

1.3.0

  • Make it possible to only run the timeout timer on the test function and not the whole fixture setup + test + teardown duration. Thanks Pedro Algarvio for the work!
  • Use the new pytest marker API, Thanks Pedro Algarvio for the work!

1.2.1

  • Fix for pytest 3.3, thanks Bruno Oliveira.
  • Update supported python versions:
    • Add CPython 3.6.
    • Drop CPyhon 2.6 (as did pytest 3.3)
    • Drop CPyhon 3.3
    • Drop CPyhon 3.4

1.2.0

  • Allow using floats as timeout instead of only integers, thanks Tom Myers.

1.1.0

  • Report (default) timeout duration in header, thanks Holger Krekel.

1.0.0

  • Bump version to 1.0 to commit to semantic versioning.
  • Fix issue #12: Now compatible with pytest 2.8, thanks Holger Krekel.
  • No longer test with pexpect on py26 as it is no longer supported
  • Require pytest 2.8 and use new hookimpl decorator

0.5

  • Timeouts will no longer be triggered when inside an interactive pdb session started by pytest.set_trace() / pdb.set_trace().
  • Add pypy3 environment to tox.ini.
  • Transfer repository to pytest-dev team account.

0.4

  • Support timeouts happening in (session scoped) finalizers.
  • Change command line option --timeout_method into --timeout-method for consistency with pytest

0.3

  • Added the PYTEST_TIMEOUT environment variable as a way of specifying the timeout (closes issue #2).
  • More flexible marker argument parsing: you can now specify the method using a positional argument.
  • The plugin is now enabled by default. There is no longer a need to specify timeout=0 in the configuration file or on the command line simply so that a marker would work.

0.2

  • Add a marker to modify the timeout delay using a @pytest.timeout(N) syntax, thanks to Laurant Brack for the initial code.
  • Allow the timeout marker to select the timeout method using the method keyword argument.
  • Rename the --nosigalrm option to --method=thread to future proof support for eventlet and gevent. Thanks to Ronny Pfannschmidt for the hint.
  • Add timeout and timeout_method items to the configuration file so you can enable and configure the plugin using the ini file. Thanks to Holger Krekel and Ronny Pfannschmidt for the hints.
  • Tested (and fixed) for python 2.6, 2.7 and 3.2.

More Repositories

1

pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
Python
11,150
star
2

pytest-testinfra

Testinfra test your infrastructures
Python
2,306
star
3

pytest-mock

Thin-wrapper around the mock package for easier use with pytest
Python
1,725
star
4

pytest-cov

Coverage plugin for pytest.
Python
1,647
star
5

pytest-xdist

pytest plugin for distributed testing and loop-on-failures testing modes.
Python
1,318
star
6

pytest-asyncio

Asyncio support for pytest
Python
1,293
star
7

pytest-django

A Django plugin for pytest.
Python
1,286
star
8

pytest-bdd

BDD library for the py.test runner
Python
1,248
star
9

pluggy

A minimalist production ready plugin system
Python
1,136
star
10

pytest-html

Plugin for generating HTML reports for pytest results
Python
649
star
11

pyfakefs

Provides a fake file system that mocks the Python file system modules.
Python
596
star
12

pytest-randomly

🎲 Pytest plugin to randomly order tests and control random.seed
Python
573
star
13

pytest-flask

A set of pytest fixtures to test Flask applications
Python
471
star
14

pytest-qt

pytest plugin for Qt (PyQt5/PyQt6 and PySide2/PySide6) application testing
Python
376
star
15

pytest-rerunfailures

a pytest plugin that re-runs failed tests up to -n times to eliminate flakey failures
Python
351
star
16

pytest-factoryboy

factory_boy integration the pytest runner
Python
348
star
17

pytest-selenium

Plugin for running Selenium with pytest
Python
325
star
18

cookiecutter-pytest-plugin

A Cookiecutter template for pytest plugins 💻
Python
281
star
19

pytest-splinter

pytest splinter and selenium integration for anyone interested in browser interaction in tests
Python
250
star
20

pytest-describe

Describe-style plugin for the pytest framework
Python
201
star
21

pytest-subtests

unittest subTest() support and subtests fixture
Python
177
star
22

pytest-repeat

pytest plugin for repeating test execution
Python
158
star
23

pytest-order

pytest plugin that allows to customize the test execution order
Python
145
star
24

pytest-instafail

py.test plugin to show failures instantly
Python
130
star
25

unittest2pytest

helps rewriting Python `unittest` test-cases into `pytest` test-cases
Python
125
star
26

pytest-cpp

Use pytest's runner to discover and execute C++ tests
C++
111
star
27

pytest-github-actions-annotate-failures

Pytest plugin to annotate failed tests with a workflow command for GitHub Actions
Python
109
star
28

pytest-env

pytest plugin to set environment variables in pytest.ini or pyproject.toml file
Python
103
star
29

pytest-xprocess

pytest external process plugin
Python
94
star
30

pytest-reportlog

Replacement for the --resultlog option, focused in simplicity and extensibility
Python
75
star
31

pytest-variables

Plugin for providing variables to pytest tests/fixtures
Python
73
star
32

execnet

distributed Python deployment and communication
Python
72
star
33

pytest-play

pytest plugin that let you automate actions and assertions with test metrics reporting executing plain YAML files
Python
68
star
34

pytest-messenger

Pytest-messenger report plugin for all popular messengers like: Slack, DingTalk, Telegram
Python
67
star
35

py

Python development support library (note: maintenance only)
Python
66
star
36

pytest-random-order

pytest plugin to randomise the order of tests with some control over the randomness
Python
64
star
37

pytest-print

pytest-print adds the printer fixture you can use to print messages to the user (directly to the pytest runner, not stdout)
Python
63
star
38

pytest-mimesis

Mimesis integration with the pytest test runner. This plugin provider useful fixtures based on providers from Mimesis.
Python
62
star
39

pytest-services

Collection of fixtures and utility functions to run service processes for your tests
Python
58
star
40

pytest-forked

extracted --boxed from pytest-xdist to ensure backward compat
Python
58
star
41

pytest-runner

Python
57
star
42

pytest-metadata

Plugin for accessing test session metadata
Python
55
star
43

iniconfig

Python
49
star
44

apipkg

Python
48
star
45

pytest-twisted

test twisted code with pytest
Python
45
star
46

pytest-stress

A Pytest plugin that allows you to loop tests for a user defined amount of time.
Python
40
star
47

pytest-incremental

py-test plugin: an incremental test runner
Python
40
star
48

pytest-faker

faker integration the pytest test runner
Python
38
star
49

nose2pytest

Scripts to convert Python Nose tests to PyTest
Python
35
star
50

pytest-base-url

pytest plugin for URL based tests
Python
35
star
51

pytest-cloud

Distributed tests planner plugin for pytest testing framework.
Python
35
star
52

pytest-freezer

Pytest plugin providing a fixture interface for spulec/freezegun
Python
35
star
53

plugincompat

Test execution and compatibility checks for pytest plugins
CSS
34
star
54

pytest-fixture-tools

Pytest fixture tools
Python
32
star
55

pytest-faulthandler

py.test plugin that activates the fault handler module during testing
Python
27
star
56

pytest-echo

pytest plugin to dump environment variables, package version and generic attributes.
Python
21
star
57

pytest-localserver

py.test plugin to test server connections locally. This repository was migrated from Bitbucket.
Python
19
star
58

pytest-inline

pytest-inline is a pytest plugin for writing inline tests.
Python
14
star
59

pytest-nunit

An Nunit output plugin for Pytest
Python
10
star
60

design

Graphic design for Pytest project
9
star
61

pytest-plus

pytest-plus adds new features to pytest
Python
9
star
62

pytest-bpdb

pytest plugin for dropping to bpdb on test failures
Python
6
star
63

blog.pytest.org

Repository for the official pytest blog
Python
4
star
64

pytest-tcpclient

pytest mocking of TCP clients
Python
3
star
65

meta

Used for generic pytest organization issues, stuff that can impact multiple projects.
2
star
66

pytest-talks

public pytest talks and workshops - meant to help user groups spin up talks and workshops
HTML
1
star
67

regendoc

Python
1
star
68

sprint

pytest development sprint 2024
1
star