Dependency management and optimization in Jupyter Notebooks.

This extension provides control over the notebook dependencies.

The main goals of the project are the following:

• manage notebook requirements without leaving the notebook
• provide a unique and optimized* environment for each notebook

*The requirements are optimized using the Thoth resolution engine

pip install jupyter-nbrequirements

And enable the required extensions (might not be needed with the latest version, but to be sure..)

jupyter nbextension install --user --py jupyter_nbrequirements

Since v0.4.0, we've introduced a new UI! Check it out, interact with it and see what it can offer you!

Our development efforts will from now on focus primarily on improving the UI.

The Jupyter magic is in sync with the UI, so don't worry old schoolers, you can still run the commands manually and the existing notebooks will work!

Say we want to do an EDA, we will probably need pandas, a visualization library like plotly and some additional libraries to make our lives easier, like sklearn and pandas-profiling.

In a Jupyter notebook cell:

%dep add pandas --version ">=0.24.0"
%dep add plotly
%dep add sklearn
%dep add pandas-profiling

And perhaps our code would need some refactoring and linter checks later on, so let's add a dev dependency.

%dep add --dev black

You can now check the requirements that your notebook has by issuing %requirements (or %dep, which is just an alias for it) command:

%requirements

[packages]
pandas = ">=0.24.0"
plotly = "*"
sklearn = "*"
pandas-profiling = "*"

[dev-packages]
black = "*"

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.6"

Up to this point, we've been working only with the metadata. In order to create the environment and actually install the dependencies, you run the %dep ensure command (insipired by the golang's dep, for those familiar with Golang).

%dep ensure

Since this project is still under development and it uses the Thoth resolution engine to optimize the notebook dependencies (which is also still under development as well), in case something goes wrong, ensure accepts the engine parameter, which can be set to pipenv

%dep ensure --engine pipenv

Check out the examples for more info.

See the Project Board.

Author: Marek Cermak [email protected], @AICoE - Project Thoth