RevSys Python Builds

What’s wrong with stock Python images?

The main image variety is too big when the user is operating in a context that involves multiple private registries.

It’s base layer is an Ubuntu image (with a few exceptions, this provides an as-installed-by-Ubuntu environment) which is, essentially, what one gets with a base minimal install of Ubuntu 16.04.

The alpine images are delightfully wee. But, alas, musl. It is awesome. I mean, it exists. If you don’t have build problems with your modules that have a C or C++ component, use it. It is small, fast and well vetted.

Note: There are workarounds for some of the more common non-pure-python module build failures but usually involve much shenaniganry that I have neither the time nor inclination to pursue.

The slim images are similar to the ubuntu images. Rather than an Ubuntu base image, they use a Debian 8 base.

We settled on a Debian 8 image provided by google’s GCE service. It is smaller by way of excluding systemd-related bits while building the initial bootstrap chroot.

The resulting images are smaller than -slim but larger than alpine.

Optimized Python images

Three builds are available per Python version:

• :wee - stock build without compile or link-time optimizations enabled
• :wee-optimized - build with profile-guided optimization flag
• :wee-optimized-lto - build with PGO + Link-Time-Optimization flags

Building

The registry is included from /.versions

make targets:

• build-image: docker build...
• push-image: docker push ...
• image: runs build-image then push-image

stock build

:# LTO= OPTIMAL= PYTHON_VERSION=3.5.3 make image

PGO optimizations

:# LTO= OPTIMAL=1 PYTHON_VERSION=3.5.1 make image

LTO Optimizations

:# LTO=1 OPTIMAL=1 PYTHON_VERSION=3.6.3 make image

Python 2 builds

The build arguments have stayed fairly consistent at least since Python 2.6. The build will not fail if the Python 3-specific environment variables remain set; however, the image will get a misleading tag. For Python 2 builds, use:

:# LTO= OPTIMAL= PYTHON_VERSION=2.7.13 make image

Build Optimization

Build times are radically reduced by using a local proxy. You can find an example squid 3/4 configuration in ./examples.

After setting the cache_dir path the command, squid -f /path/to/config -N -z to initialize the cache filesystem storage.

To kick off the cache service: squid -f /path/to/config -N -a 3128.

Once the proxy is running, set the environment variable export PKG_PROXY=http://10.129.1.1:3128. Docker transparently passes the HTTP_PROXY and HTTPS_PROXY environment variables into each build container if they are configured with the --build-args flag.

The transparency is significant in that whether or not your build is leveraging a proxy will not affect the validity of the build host’s cache.

Origin Notes

Based in large part on the contents of the Docker Library Python repository.

1. uses GNU make and environment variables to set options
   1. LTO: if present and not empty, activate link-time-optimization functionality
   2. OPTIMAL: if present and not empty, activate profiler-guided-optimization functionality
   3. `PYTHON_VERSION’: e.g.: 3.6.3, 3.5.1
2. Does not include TCL/TK. This is on purpose. If you neeed IDLE or pexpect you’ll need to customize the Dockerfile and rebuild.
3. Uses multi-layer Dockerfile syntax

Keep in touch!

If you have a question about this project, please open a GitHub issue. If you love us and want to keep track of our goings-on, here's where you can find us online: