httpimport

Python's missing feature!

The feature has been suggested in Python Mailing List

Remote, in-memory Python package/module importing through HTTP/S

A feature that Python misses and has become popular in other languages is the remote loading of packages/modules.

httpimport lets Python packages and modules to be installed and imported directly in Python interpreter's process memory, through remote URIs, and more...

Python2 support has been discontinued. Last version that supports Python2 is 1.1.0.

Basic Usage

Load package/module accessible through any HTTP/S location

with httpimport.remote_repo('http://my-codes.example.com/python_packages'):
  import package1

Load a module from PyPI:

with httpimport.pypi_repo():
  import distlib # https://pypi.org/project/distlib/

print(distlib.__version__)
# '0.3.6' <-- currently latest (https://github.com/pypa/distlib/blob/0.3.6/distlib/__init__.py#L9)

Load directly from a GitHub/BitBucket/GitLab repo

with httpimport.github_repo('operatorequals', 'httpimport', ref='master'):
  import httpimport as httpimport_upstream
  # Also works with 'bitbucket_repo' and 'gitlab_repo'

Load a Python module from a Github Gist (using this gist):

url = "https://gist.githubusercontent.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace/raw/e55fa867d3fb350f70b2897bb415f410027dd7e4"

with httpimport.remote_repo(url):
  import hello

hello.hello()
# Hello world

Load a package/module directly to a variable

# From HTTP/S URL
http_module = httpimport.load('package1', 'https://my-codes.example.com/python_packages')
print(http_module)
<module 'package1' from 'https://my-codes.example.com/python_packages/package1/__init__.py'>

# From PyPI
pypi_module = httpimport.load('distlib', importer_class=httpimport.PyPIImporter)
print(pypi_module)
<module 'distlib' from 'https://files.pythonhosted.org/packages/76/cb/6bbd2b10170ed991cf64e8c8b85e01f2fb38f95d1bc77617569e0b0b26ac/distlib-0.3.6-py2.py3-none-any.whl#distlib/__init__.py'>

Load Python packages from archives served through HTTP/S

No file is touching the disk in the process

# with httpimport.remote_repo('https://example.com/packages.tar'):
# with httpimport.remote_repo('https://example.com/packages.tar.bz2'):
# with httpimport.remote_repo('https://example.com/packages.tar.gz'):
# with httpimport.remote_repo('https://example.com/packages.tar.xz'):
with httpimport.remote_repo('https://example.com/packages.zip'):
  import test_package

Serving a package through HTTP/S

Any package can be served for httpimport using a simple HTTP/S Server:

echo 'print("Hello httpimport!")' > module.py
python -m http.server
Serving HTTP on 0.0.0.0 port 8000 ...

>>> import httpimport
>>> with httpimport.remote_repo("http://127.0.0.1:8000"):
...   import module
...
Hello httpimport!

Profiles

After v1.0.0 it is possible to set HTTP Authentication, Custom Headers, Proxies and several other things using URL and Named Profiles!

URL Profiles

URL Profiles are INI configurations, setting specific per-URL options, as below:

[http://127.0.0.1:8000]
allow-plaintext: yes ; also 'true' and '1' evaluate to True

[https://example.com]
proxy-url: https://127.0.0.1:8080 ; values must not be in quotes (')

Now, requests to http://127.0.0.1:8000 will be allowed (HTTP URLs do not work by default) and requests to https://example.com will be sent to an HTTP Proxy.

with httpimport.remote_repo("https://example.com"): # URL matches the URL profile
  import module_accessed_through_proxy

Named Profiles

Named Profiles are like URL profiles but do not specify a URL and need to be explicitly used:

[github]
headers:
  Authorization: token <Github-Token>

And the above can be used as follows:

with httpimport.github_repo('operatorequals','httpimport-private-test', profile='github'):
  import secret_module

Github Tokens look like github_pat_<gibberish> and can be issued here: https://github.com/settings/tokens/new

Profiles for PyPI

When importing from PyPI extra options can be used, as described in the profile below:

[pypi]

# The location of a 'requirements.txt' file
# to use for PyPI project versions
requirements-file: requirements-dev.txt

# Inline 'requirements.txt' syntax appended
requirements:
  distlib==0.3.5
  sampleproject==3.0.0

# Only version pinning notation is supported ('==')
# with 'requirements' and 'requirements-file' options

# A map that contains 'module': 'PyPI project' tuples
# i.e: 'import sample' --> search 'sample' module at 'sampleproject' PyPI Project:
# https://pypi.org/project/sampleproject/
project-names:
  sample: sampleproject

The PyPI Profiles can be used exactly like all Named Profiles:

with httpimport.pypi_repo(profile='pypi'):
  import distlib
  import sample
distlib.__version__
# '0.3.5' <-- pinned in the profile 'requirements' option
sample.__url__
# 'https://files.pythonhosted.org/packages/ec/a8/5ec62d18adde798d33a170e7f72930357aa69a60839194c93eb0fb05e59c/sampleproject-3.0.0-py3-none-any.whl#sample/__init__.py' <-- loaded from 'sampleproject'

Additionally, all other options cascade to PyPI profiles, such as HTTPS Proxy (HTTP proxies won't work, as PyPI is hosted with HTTPS), headers, etc.

NOTE: The values in Profiles MUST NOT be quoted (',")

Profile Creation

Profiles can be provided as INI strings to the set_profile function and used in all httpimport functions:

httpimport.set_profile("""
[profile1]

proxy-url: https://my-proxy.example.com
headers:
  Authorization: Basic ...
  X-Hello-From: httpimport
  X-Some-Other: HTTP header
""")
with httpimport.remote_repo("https://code.example.com", profile='profile1'):
  import module_accessed_through_proxy

Advanced

Profiles are INI configuration strings parsed using Python configparser module.

The ConfigParser object for httpimport is the global variable httpimport.CONFIG and can be used freely:

import httpimport
httpimport.CONFIG.read('github.ini') # Read profiles from a file

with httpimport.github_repo('operatorequals','httpimport-private-test', profile='github'):
  import secret_module

Default Profiles

The httpimport module automatically loads Profiles found in $HOME/.httpimport.ini and under the $HOME/.httpimport/ directory. Profiles under $HOME/.httpimport/ override ones found in $HOME/.httpimport.ini.

Profile Options:

Supported

HTTP options

• zip-password - v1.0.0
• proxy-url - v1.0.0
• headers - v1.0.0
• allow-plaintext - v1.0.0
• ca-verify - v1.3.0
• ca-file - v1.3.0

PyPI-only options

• project-names - v1.2.0
• requirements - v1.2.0
• requirements-file - v1.2.0

Not yet (subject to change)

• allow-compiled

• auth

• auth-type

• tls-cert

• tls-key

• tls-passphrase

Debugging...

import httpimport
import logging

logging.getLogger('httpimport').setLevel(logging.DEBUG)

Beware: Huge Security Implications!

Using the httpimport with plain HTTP URLs is highly discouraged

As HTTP traffic can be read and changed from all intermediate hosts (unlike HTTPS), it is possible for a remote adversary to alter the HTTP responses consumed by httpimport and add arbitrary Python code to the downloaded packages/modules. This directly results in arbitrary Remote Code Execution on your current user's context of your host!

In other words, using plain HTTP through the Internet can compromise your host without a way to notice it.

You have been warned! Use only HTTPS URLs with httpimport!

Contributors

• ldsink - The RELOAD flag and Bug Fixes
• lavvy - the load() function
• superloach - Deprecation of imp module in Python3 in favour of importlib
• yanliakos - Bug Fix
• rkbennett - Relative Imports fix, Proxy support

Donations

In case my work helped you, you can always buy me a beer or a liter of gas through the Internet or in case you meet me personally.

In the second case we can talk about any video of Internet Historian or Ordinary Things, while listening to a Lofi Girl Playlist, like the citizens of the Internet that we are.