InvertibleNetworks.jl

Documentation	Build Status

Building blocks for invertible neural networks in the Julia programming language.

• Memory efficient building blocks for invertible neural networks
• Hand-derived gradients, Jacobians $J$ , and $\log |J|$
• Flux integration
• Support for Zygote and ChainRules
• GPU support
• Includes various examples of invertible neural networks, normalizing flows, variational inference, and uncertainty quantification

Installation

InvertibleNetworks is registered and can be added like any standard julia package with the command:

] add InvertibleNetworks

Papers

The following publications use InvertibleNetworks.jl:

• "Reliable amortized variational inference with physics-based latent distribution correction"

  • paper: https://arxiv.org/abs/2207.11640
  • presentation
  • code: ReliableAVI.jl

• "Learning by example: fast reliability-aware seismic imaging with normalizing flows"

  • paper: https://arxiv.org/abs/2104.06255
  • presentation
  • code: ReliabilityAwareImaging.jl

• "Enabling uncertainty quantification for seismic data pre-processing using normalizing flows (NF)—an interpolation example"

  • paper
  • code: WavefieldRecoveryUQ.jl

• "Preconditioned training of normalizing flows for variational inference in inverse problems"

  • paper: https://arxiv.org/abs/2101.03709
  • presentation
  • code: FastApproximateInference.jl

• "Parameterizing uncertainty by deep invertible networks, an application to reservoir characterization"

  • paper: https://arxiv.org/abs/2004.07871
  • presentation
  • code: https://github.com/slimgroup/Software.SEG2020

• "Generalized Minkowski sets for the regularization of inverse problems"

  • paper: http://arxiv.org/abs/1903.03942
  • code: SetIntersectionProjection.jl

Building blocks

• 1x1 Convolutions using Householder transformations (example)

• Residual block (example)

• Invertible coupling layer from Dinh et al. (2017) (example)

• Invertible hyperbolic layer from Lensink et al. (2019) (example)

• Invertible coupling layer from Putzky and Welling (2019) (example)

• Invertible recursive coupling layer HINT from Kruse et al. (2020) (example)

• Activation normalization (Kingma and Dhariwal, 2018) (example)

• Various activation functions (Sigmoid, ReLU, leaky ReLU, GaLU)

• Objective and misfit functions (mean squared error, log-likelihood)

• Dimensionality manipulation: squeeze/unsqueeze (column, patch, checkerboard), split/cat

• Squeeze/unsqueeze using the wavelet transform

Examples

• Invertible recurrent inference machines (Putzky and Welling, 2019) (generic example)

• Generative models with maximum likelihood via the change of variable formula (example)

• Glow: Generative flow with invertible 1x1 convolutions (Kingma and Dhariwal, 2018) (generic example, source)

GPU support

GPU support is supported via Flux/CuArray. To use the GPU, move the input and the network layer to GPU via |> gpu

using InvertibleNetworks, Flux

# Input
nx = 64
ny = 64
k = 10
batchsize = 4

# Input image: nx x ny x k x batchsize
X = randn(Float32, nx, ny, k, batchsize) |> gpu

# Activation normalization
AN = ActNorm(k; logdet=true) |> gpu

# Test invertibility
Y_, logdet = AN.forward(X)

Acknowledgments

This package uses functions from NNlib.jl, Flux.jl and Wavelets.jl

References

• Yann Dauphin, Angela Fan, Michael Auli and David Grangier, "Language modeling with gated convolutional networks", Proceedings of the 34th International Conference on Machine Learning, 2017. https://arxiv.org/pdf/1612.08083.pdf

• Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio, "Density estimation using Real NVP", International Conference on Learning Representations, 2017, https://arxiv.org/abs/1605.08803

• Diederik P. Kingma and Prafulla Dhariwal, "Glow: Generative Flow with Invertible 1x1 Convolutions", Conference on Neural Information Processing Systems, 2018. https://arxiv.org/abs/1807.03039

• Keegan Lensink, Eldad Haber and Bas Peters, "Fully Hyperbolic Convolutional Neural Networks", arXiv Computer Vision and Pattern Recognition, 2019. https://arxiv.org/abs/1905.10484

• Patrick Putzky and Max Welling, "Invert to learn to invert", Advances in Neural Information Processing Systems, 2019. https://arxiv.org/abs/1911.10914

• Jakob Kruse, Gianluca Detommaso, Robert Scheichl and Ullrich Köthe, "HINT: Hierarchical Invertible Neural Transport for Density Estimation and Bayesian Inference", arXiv Statistics and Machine Learning, 2020. https://arxiv.org/abs/1905.10687

Authors

• Philipp Witte, Georgia Institute of Technology (now Microsoft)

• Gabrio Rizzuti, Utrecht University

• Mathias Louboutin, Georgia Institute of Technology

• Rafael Orozco, Georgia Institute of Technology

• Ali Siahkoohi, Georgia Institute of Technology