python-client-benchmarks

Benchmarking different python HTTP clients, using a small, high-performance sample API.

Why?

To answer a StackOverflow question: how much does performance differ, for python HTTP clients?

This provides a Python client library benchmarking tool and a simple but high-performance API (HTTP and HTTPS) to use in benchmarking.

Oh, and of course, results in a controlled environment!

How do I run the test server?

In one terminal:

• docker run --rm -i -t -w /tmp -p 4443:443 -p 8080:80 svanoort/client-flask-demo:2.0
• OR: (sudo) ./build-docker.sh && ./run-docker.sh

The test server will expose several API endpoints, with HTTP responses on port 8080 & HTTPS (self-signed cert) on port 4443:

• /ping - returns "pong"
• /bigger - returns a fixed ~585B JSON response
• /length/<integer> - returns a fixed-length random binary response of $integer bytes. The first time this is generated and cached, then it is served from cache.

How do I run the benchmark client?

It's a command line utility, and has help via "python benchmark.py -h" python benchmark.py http://host:port/

How

1. On one server or system

2. In one terminal, build & run the docker image to start a test server: sh build-docker.sh && sh run-docker.sh

3. In another terminal, run: python benchmark.py --cycles 10000

4. Tests take a few minutes minute to finish, then you can close process in terminal 1.

Short Summary of Results

PyCurl is much faster than Requests (or other HTTP client libraries), generally completing smaller requests 2-3x as fast, and requires 3-10x less CPU time. This is most visible with connection creation for small requests; given large enough requests (above 100 kB/s), Requests can still saturate a fast connection quite easily even without concurrent HTTP requests (assuming that all post-request processing is done separately) on a fast CPU.

On this system:

• pycurl takes about 73 CPU-microseconds to issue a request when reusing a connection
• requests takes about 526 CPU-microseconds to issue a request when reusing a connection
• pycurl takes about 165 CPU-microseconds to open a new connection and issue a request (no connection reuse), or ~92 microseconds to open
• requests takes about 1078 CPU-microseconds to open a new connection and issue a request (no connection reuse), or ~552 microseconds to open
• CPU-limited peak throughput with PyCurl is roughly 50%-100% better than requests with very large request sizes, assuming an extremely fast network connection relative to CPU (in this case several GBit)

Benchmark Setup

• Testing with two c4.large instances in AWS, one running a client, and the other running the server.
• All tests issued 10000 sequential requests to minimize the impact of noise
• Timing was done using timeit, to separate initial setup & library loading from actual request time.
• For loopback tests, both are run on the same (client) system, to measure performance without bandwidth limits.
• Logs were collected from each benchmark run, along with vmstat dumps to verify that the CPU was never fully loaded (and never a bottleneck).

Specs:

• Amazon Linux (Fedora-based), latest update and all updates installed as of 26-Feb-2016.
• Enhanced networking, one source suggests the c4.large has ~517 Mbps of bandwidth per host.
• Apache Benchmark confirms that ~65 MB is roughly the limit (there are rare cases where 100 MB/s connections may be observed though)
• 2 vCPUs, Intel Xeon E5-2666 v3 or equivalent (8 ECU)
• 3.75 GB of RAM
• 8 GB GP2 EBS volume

Graphs

Detailed Results

Loopback Test, running on 1 c4.large and making 1 kB requests against a local server

Detailed CPU TIME Loopback Test, running on 1 c4.large with different request sizes

Full data for server-to-server tests: available in the Google Sheet under the AWS-to-AWS benchmark tab.

Also available in the new-aws-results folder.

Miscellanea

build-docker.sh was used to generate the docker image ./run-docker.sh will launch the container

Caveats

• This is only one system type and one operating system tested
• I am only trying GET requests to a sample API
• I've taken pains to generate data in as scientific and clean a manner as I can reasonably manage (reporting stats over 10k requests), but do not collect per-execution data so there are no error bars
• HTTPS performance should be taken with an extra grain of salt and only used for rough comparison:
  • Many different options and configurable settings exist for HTTPS, all with varying performance
  • There are also multiple TLS/SSL implementations available for libcurl, for sanity's sake I am only testing the default one.
  • For similar reasons I'm not testing pyopenssl use in requests (in addition to base requests settings)