• Stars
    star
    233
  • Rank 172,230 (Top 4 %)
  • Language
    Python
  • License
    GNU Lesser Genera...
  • Created almost 5 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A lightweight wrapper for pip to support requirements.txt, Pipenv and Poetry lock files or converting them to pip-tools compatible output. Designed for containerized Python applications but not limited to them.

µPipenv

GitHub tag (latest by date) https://travis-ci.com/thoth-station/micropipenv.svg?branch=master PyPI - Python Version PyPI - License PyPI - Downloads

A lightweight wrapper for pip to support requirements.txt, Pipenv and Poetry lock files or converting them to pip-tools compatible output. Designed for containerized Python applications but not limited to them.

For a brief video preview, check this demo (the micropipenv part starts at 9:00).

See also micropipenv: Installing Python dependencies in containerized applications for more info about this tool and a blog post published about it.

What's the difference in comparision to pip when using requirements.txt?

  • if requirements.txt state all the packages in a pinned version with hashes (e.g. pip-tools), micropipenv installs packages with a possible fallback if the installation order is relevant
    • you don't need to care about the installation and maintain correct order or requirements in requirements.txt
    • best effort installation - try until there is a possibility to succeed
  • if requirements.txt do not state all the packages in a pinned form
    • pip's resolver algorithm is used and it's left on pip to resolve requirements
    • the same behavior as micropipenv would not be used

What's the difference in comparision to Poetry?

  • a lightweight addition to Poetry, not a Poetry replacement
    • micropipenv does not substitute Poetry it rather complements it for containerized deployments where the size of the container image and software shipped with it matters
  • no release management to Python package indexes
  • micropipenv does not implement resolver, it uses already resolved stack that application is shipped with based on poetry.lock and pyproject.toml
  • no virtual environment management
    • virtual environment management is left on user, if needed

What's the difference in comparision to Pipenv?

  • a lightweight addition to Pipenv, not a Pipenv replacement
    • micropipenv does not substitute Pipenv it rather complements it for containerized deployments where the size of the container image and software shipped with it matters
  • it does not vendor all the dependencies as Pipenv
  • micropipenv does not implement resolver, it uses already resolved stack that application is shipped with Pipfile.lock
  • no virtual environment management
    • virtual environment management is left on user, if needed

micropipenv use cases

Why should I use micropipenv instead of Pipenv or Poetry?

  • I would like to have a tool that "rules them all" - one lightweight tool to support all Python dependency lock file managers (pip-tools, Poetry, Pipenv) and lets users decide what they want to use when deploying Python applications in containerized environments (e.g. Kubernetes, OpenShift, ...).
  • I would like to have a fast and minimalistic tool to install software packages in CI.
  • I would like to have containerized Python applications as small as possible with minimum software shipped and required to build and run the Python application in production.
  • I would like to convert files produced by Pipenv/Poetry to a pip-tools compatible output.
  • I don't want to install Pipenv/Poetry, but I would like to run a project that uses Pipenv/Poetry for dependency management (e.g. restricted environments).
  • My Pipenv installation is broken and Pipenv upstream did not issue any new Pipenv release.
  • I would like to deploy my application into a production environment and my application dependencies are managed by Pipenv/Poetry (dependencies are already resolved), but I don't want to run Pipenv/Poetry in production (e.g. OpenShift's s2i build process).

micropipenv install

The tool supports installing dependencies of the following formats:

  • Pipenv style lock format - files Pipfile and Pipfile.lock
  • Poetry style lock format - files pyproject.toml and poetry.lock
  • pip-tools style lock format - file requirements.txt
  • raw requirements.txt as used by pip (not a lock file)

In case of Pipenv, Poetry and pip-tools style format, the tool performs automatic recovery if the installation order of dependencies is relevant (one dependency fails to install as it depends on an another one).

To enforce the installation method used, specify --method option to the install subcommand. By default, micropipenv traverses the filesystem up from the current working directory and looks for the relevant files in the following order:

  1. Pipfile.lock and optionally Pipfile (if --deploy set)
  2. poetry.lock and pyproject.toml
  3. requirements.txt for pip-tools and raw pip requirements

To install dependencies issue the following command:

micropipenv install --dev  # --dev is optional

You can supply additional positional arguments that will be passed to pip. Use double dashes to distinguish pip options from micropipenv options.

# issue `pip install --user'
micropipenv install -- --user

micropipenv does not create any virtual environment as in case of Pipenv/Poetry. It rather directly talks to pip, if necessary, and constructs arguments out of the lock file used.

To create a virtual environment to be used by micropipenv:

python3 -m venv venv/ && . venv/bin/activate

micropipenv install --deploy

If you wish to mimic pipenv --deploy functionality, you can do so:

micropipenv install --deploy

Note however, there is a need to parse Pipfile and verify its content corresponds to Pipefile.lock used (digest computed on Pipfile content). micropipenv requires toml extras for this functionality, so you will need to install micropipenv[toml] (see installation instructions bellow).

The --deploy option takes no effect for Poetry and requirements installation methods.

micropipenv install --dev

Installation of "development" dependencies can be acomplished using the --dev flag. This flag has no effect when requirements.txt file is used.

micropipenv requirements / micropipenv req

To generate output compatible with pip-tools, you can issue the following command:

micropipenv requirements

This applies to conversion from Poetry and Pipenv specific lock files.

Additional configuration options can limit what is present in the output (e.g. --no-dev to remove development dependencies).

A special option --only-direct makes micropipenv work on Pipfile instead of Pipfile.lock. This requires toml extras, so install micropipenv[toml] for this functionality (see installation instructions bellow). To get direct dependencies of an application and store them in requirements.txt file:

micropipenv requirements --only-direct > requirements.txt

For a setup that follows pip-tools convention with requirements.in and requirements.txt

micropipenv requirements --no-dev > requirements.txt
micropipenv requirements --no-dev --only-direct > requirements.in
micropipenv requirements --no-default > dev-requirements.txt
micropipenv requirements --no-default --only-direct > dev-requirements.in

See micropipenv requirements --help for more info.

micropipenv as a library

micropipenv exposes some core functionality on top of Pipfile/Pipfile.lock. You can import its functions and use micropipenv as a lightweight library for Pipfile/Pipfile.lock and pyproject.toml/poetry.lock manipulation.

Adjusting options using environment variables

All options can be triggered using environment variables - the name of an environment variable is always prefixed with MICROPIPENV_ and consists of the name of the option converted to uppercase, dashes are replaced with underscores (example --no-dev is mapped to MICROPIPENV_NO_DEV). All environment variables corresponding to flags are parsed as integers and subsequently casted to a boolean. For example, to turn --no-dev flag on, set MICROPIPENV_NO_DEV=1 (0 disables the flag). Parameters supplied to CLI take precedence over environment variables.

A special environment variable MICROPIPENV_PIP_BIN can point to an alternate pip binary.

To run this tool in a verbose mode, you can set the MICROPIPENV_DEBUG=1 (the same behavior can be achieved with multiple --verbose supplied).

The tool prints software stack information to the standard error output. This was designed for Thoth to capture information about installed dependencies as a useful source of information for Thoth's build analyzers. This behaviour can be suppressed by setting MICROPIPENV_NO_LOCKFILE_PRINT=1 environment variable.

Besides printing, the tool also writes the content of Pipfile.lock (if a locked software stack is used) to the directory where lock files are present (for Pipenv files, the Pipfile.lock is kept untouched). This behaviour can be suppressed by providing MICROPIPENV_NO_LOCKFILE_WRITE=1 environment variable.

Example usage

Install dependencies managed by Poetry as pip install --user would do (option --method is optional, auto-discovery is performed if omitted):

$ ls
poetry.lock pyproject.toml project.py
$ micropipenv install --method poetry -- --user

Install dependencies (both main and develop) managed by Poetry into a virtual environment:

$ ls
poetry.lock pyproject.toml project.py
$ python3 -m venv venv/
$ . venv/bin/activate
(venv) $ micropipenv install --dev

Install dependencies managed by Pipenv (both main and develop) into a virtual environment (option --method is optional, auto-discovery is performed if omitted):

$ ls
Pipfile Pipfile.lock src/
$ python3 -m venv venv/
$ . venv/bin/activate
(venv) $ micropipenv install --dev

Perform deployment of an application as Pipenv would do with Python interpreter version check and Pipfile file hash check (you can create virtual environment only if necessary):

$ ls
Pipfile Pipfile.lock src/
$ python3 -m venv venv/
$ . venv/bin/activate
(venv) $ micropipenv install --deploy

Generate pip-tools compliant requirements.in, dev-requirements.in, requirements.txt and dev-requirements.txt out of Pipfile and Pipfile.lock - project dependencies managed by Pipenv:

$ ls
Pipfile Pipfile.lock src/
$ micropipenv requirements --no-dev > requirements.txt
$ micropipenv requirements --no-dev --only-direct > requirements.in
$ micropipenv requirements --no-default > dev-requirements.txt
$ micropipenv requirements --no-default --only-direct > dev-requirements.in

Generate pip-tools complaint requirements.in, dev-requirements.in, requirements.txt and dev-requirements.txt out of pyproject.toml and poetry.lock - project dependencies managed by Poetry:

$ ls
poetry.lock pyproject.toml src/
$ micropipenv requirements --no-dev > requirements.txt
$ micropipenv requirements --no-dev --only-direct > requirements.in
$ micropipenv requirements --no-default > dev-requirements.txt
$ micropipenv requirements --no-default --only-direct > dev-requirements.in

For OpenShift's s2i integration, check this repo with a demo.

Installation

The project is hosted on PyPI so installing it using pip works as expected:

pip install micropipenv

The default installation does not bring any dependencies so its just micropipenv that gets installed. However, the default installation supports only Pipfile.lock management. If you would like to manipulate also with Pipfile or Poetry specific lock files, you will need to install micropipenv with TOML support (TOML is not in the standard Python library):

pip install micropipenv[toml]

Once the project gets installed, you can browse the help message by invoking the micropipenv CLI:

micropipenv --help

If you wish to install micropipenv on your Fedora system:

dnf install -y micropipenv

See available RPM packages.

No installation

You can run micropipenv without actually installing it - simply download the file and execute it. If you do not wish to save micropipenv.py file to disk, you can issue:

curl https://raw.githubusercontent.com/thoth-station/micropipenv/master/micropipenv.py | python3 - --help

Anything after python3 - will be passed as an argument to micropipenv.py so installing packages can be simply performed using:

curl https://raw.githubusercontent.com/thoth-station/micropipenv/master/micropipenv.py | python3 - install -- --user

All arguments after -- will be passed to pip as options.

OpenShift s2i (Source-To-Image)

micropipenv is available in UBI, Fedora and RHEL based container images. To enable micropipenv and benefit from its features, you need to export ENABLE_MICROPIPENV=1 environment variable in more recent Python 3 container images. See sclorg/s2i-python-container repo for more information.

License and copying

This project is licensed under the terms of the GNU Lesser General Public License v3 or later. See LICENSE-LGPL and LICENSE-GPL files for the license terms.

Copyright (C) 2020-2022 Project Thoth; Red Hat Inc.

Original author:
Maintainers:

More Repositories

1

adviser

The recommendation engine for Python software stacks and Dependency Monkey in project Thoth.
Python
31
star
2

core

Using Artificial Intelligence to analyse and recommend Software Stacks for Artificial Intelligence applications.
Go
28
star
3

kebechet

I'm Kebechet bot, goddess of freshness - I will keep your source code fresh and up-to-date
Python
23
star
4

invectio

Statically analyze sources and extract information about called or exported library functions in Python applications
Python
20
star
5

prescriptions

⚕️💊 Prescriptions to heal your applications and application dependencies 💊⚕️
Python
17
star
6

jupyterlab-requirements

JupyterLab Extension for dependency management and optimization
Jupyter Notebook
16
star
7

solver

Dependency solver for the Thoth project
Python
16
star
8

jupyter-nbrequirements

Dependency management and optimization in Jupyter Notebooks
TypeScript
16
star
9

thamos

A CLI tool and library for communicating with Thoth
Python
14
star
10

thoth

Platform for analyzing and recommending Python packages and Python software stacks not only for AI/ML applications
Python
14
star
11

storages

Storage and database adapters for project Thoth
Python
14
star
12

thoth-application

Thoth-Station ArgoCD Applications
Open Policy Agent
13
star
13

glyph

Generate CHANGELOG entries out of commit messages using AI/ML techniques
Python
11
star
14

user-api

Web Service exposing Thoth functions to the RESTful worlds
Python
8
star
15

s2i-minimal-notebook

A minimal Jupyter Notebook S2I image
Shell
8
star
16

amun-api

Thoth's execution engine for inspecting quality, performance, and usability of software and software stacks in a cluster.
Dockerfile
8
star
17

template-project

This is a Template for any Python based project
Python
7
star
18

ansible-role-argo-workflows

Ansible Role - Argo Workflows
7
star
19

search

Visualize Python package metadata and browse advisory results.
TypeScript
6
star
20

mi

an experiment on Source Operation Metrics
Python
6
star
21

s2i-thoth

Thoth's addition to OpenShift's s2i Python builds to benefit from Thoth's recommendations in your application
Dockerfile
6
star
22

talks

Talks, presentations and research papers published about Thoth
5
star
23

datasets

Thoth Datasets to share Thoth heritage and create new ideas...
Jupyter Notebook
5
star
24

python

Library for Python related code
Python
4
star
25

si-bandit

Thoth's Security Indicator using bandit
Python
4
star
26

common

A common library for the Thoth project
Python
4
star
27

integration-tests

Integration tests for the Thoth project to make sure deployment works as expected
Python
4
star
28

si-aggregator

Thoth's Security Indicator Aggregator
Python
4
star
29

notebooks

A set of notebooks to experiment with data in project Thoth
Jupyter Notebook
4
star
30

investigator

Thoth investigator is a Kafka based component that consumes all messages produced by Thoth components.
Python
4
star
31

meteor-operator

Project Meteor Operator for OpenShift
Go
4
star
32

report-processing

Library for processing all outputs/reports produced by Thoth components.
Python
4
star
33

cli-examples

Examples of Thoth's recommendations using Thamos CLI
Python
4
star
34

workflow-helpers

All code required to help workflows (e.g Argo).
Python
3
star
35

management-api

An API service used for administration of deployed Thoth
Python
3
star
36

s2i-example

An example of Thoth's recommendations in OpenShift's s2i builds for a TensorFlow stack
Python
3
star
37

cve-update-job

Update Thoth's database of CVEs
Python
3
star
38

sentry-openshift

Deployment of Sentry onto OpenShift for monitoring your OpenShift application
3
star
39

jupyternb-build-pipeline

Openshift-pipelines and tekton based pipeline for packaging jupyternb in to image.
3
star
40

messaging

This is Thoth messaging, I will send my knowledge to the Universe!
Python
3
star
41

thoth-station.github.io

Project Thoth website
JavaScript
3
star
42

thoth-toolbox

This is a Thoth tool box! A toolbox of Thoth tools...
Dockerfile
3
star
43

license-solver

Detect license information from Python package metadata as provided by thoth-solver
Python
3
star
44

thoth-pre-commit-hook

Thoth adviser pre-commit hooks.
Python
3
star
45

s2i-generic-data-science-notebook

Shell
2
star
46

zuul-config

Python
2
star
47

metrics-exporter

This is a Prometheus exporter for Thoth.
Python
2
star
48

fext

Fast CPython extensions to Python standard library with focus on performance.
C++
2
star
49

dependency-monkey

The "Dependency Monkey" is a service for validating of package dependencies within an application stack.
Python
2
star
50

graph-sync-job

Job for syncing data from object store to a graph database
Python
2
star
51

jupyter-notebook

Jupyter notebook s2i build for project Thoth
Shell
2
star
52

package-update-job

Aggregate updates from Packages hosted in Indices...
Python
2
star
53

jupyter-notebook-s2i

Jupyter notebook S2I builder image.
Shell
2
star
54

goblinoid

Create a graph database schema and indexes from source code automatically
Python
2
star
55

sync-job

A one time job to sync all Thoth data
Python
2
star
56

package-releases-job

A job for monitoring new releases on PyPI and creating relevant entries in the graph database
Python
2
star
57

opendatahub-cnbi

A repository for the OpenDataHub CNBi feature.
2
star
58

source-management

Thoth's interaction library for different source management systems
Python
2
star
59

s2i-example-flask

Example of an Python's s2i application consuming Thoth's recommendations
Shell
2
star
60

workflows

Argo workflows used to get certain Thoth jobs done.
Shell
2
star
61

buildlog-parser

Parse OpenShift's s2i build log into its structured representation with all the metadata available
Python
2
star
62

amun-client

A library for communicating with Amun API
Python
2
star
63

stub-api

This is a project demonstrating the basic structure of a API Service as used by the Thoth-Station.
Python
2
star
64

slo-reporter

Provide report on Thoth Service Level Objectives (SLO) to stakeholders.
Python
2
star
65

si-cloc

Thoth's Security Indicator using cloc
Python
2
star
66

qeb-hwt

I'm Kebechet bot, goddess of freshness - I will keep your dependencies fresh and up-to-date
Python
2
star
67

revsolver

A reverse solver implementation
Python
1
star
68

socrates

Experimental project on reviewer recommendation based on source code and srcops analysis.
Python
1
star
69

ansible-role-core-templates

This Ansible role will create an OpenShift Templates required to deploy Thoth.
1
star
70

init-job

Thoth's initialization job that automatically initializes a fresh deployment
Python
1
star
71

prescriptions-refresh-job

A periodic job to refresh Thoth's prescriptions
Python
1
star
72

solver-errors-reporter

Cluster solver errors collected by Thoth Dependency Solver and report back to take actions.
Jupyter Notebook
1
star
73

reporter

Thoth Reporter to provide information on status of thoth services from analysis of documents stored on Ceph.
Python
1
star
74

pulp-metrics-exporter

Metrics exporter exporting information about Pulp
Python
1
star
75

tensorflow-serving-build

Tensorflow serving binary build
Shell
1
star
76

ansible-role-core-imagestreams

A set of Ansible Roles used by Thoth
Shell
1
star
77

external-janusgraph-service

This Ansible role will create an OpenShift Service object to integrate an external JanusGraph Server.
1
star
78

nepthys

Bringing your code back to live as an interactive documentation automatically
Shell
1
star
79

aicoe-ci-pulp-upload-example

A small example demonstrating Python package releases uploaded using AICoE-CI to Operate First Pulp instance.
Python
1
star
80

ansible-role-postgresql

Ansible role to provision PostgreSQL into Thoth
1
star
81

cleanup-job

A job for cleaning up old OpenShift objects created by one-time analyses
Python
1
star
82

srcops-testing

This is a SrcOps test repository
Python
1
star
83

isis-api

API exposing algorithms on top of project2vec
Python
1
star
84

ps-nlp

This is a repository for a Predictive Stack for Natural Language Processing (NLP)
Shell
1
star
85

image-pusher

Push images to an external registry from internal OpenShift registry
Python
1
star
86

s2i

Tooling and a library for Thoth's Python Source-To-Image (s2i) applications
Python
1
star
87

s2i-example-migration

An example repository for a project to be migrated to use Thoth (demo)
Python
1
star
88

performance

Scripts to benchmark libraries used by Amun
Python
1
star
89

osiris

Build log aggregation service which integrates with OpenShift and Ceph storage.
Python
1
star
90

ansible-role-cronjob

Create a Thoth CronJob in an OpenShift Project/namespace.
1
star
91

selinon-worker

Selinon worker used for data gathering, data cleansing and experiments in project Thoth
Python
1
star
92

build-watcher

Watch for builds done in OpenShift and automatically submit new images to Thoth's image analysis
Python
1
star
93

graph-refresh-job

A job for scheduling solver to resolve dependency graphs of new packages
Python
1
star
94

package-extract

Manipulation with package dependencies in containers and install logs
Python
1
star