Denoising diffusion probabilistic models

These tutorials explores the new class of generative models based on diffusion probabilistic models [ 1 ] . This class of models is inspired by considerations from thermodynamics [ 2 ] , but also bears strong ressemblence to denoising score matching [ 3 ] , Langevin dynamics and autoregressive decoding. We will also discuss the more recent development of denoising diffusion implicit models [ 4 ] , which bypass the need for a Markov chain to accelerate the sampling. Stemming from this work, we will also discuss the wavegrad model [ 5 ] , which is based on the same core principles but applies this class of models for audio data.

In order to fully understand the inner workings of diffusion model, we will review all of the correlated topics through tutorial notebooks. These notebooks are available in Pytorch or in JAX (in the jax_tutorials/ folder), thanks to the great contribution of Cristian Garcia.

We split the explanation between four detailed notebooks.

1. Score matching and Langevin dynamics.
2. Diffusion probabilistic models and denoising
3. Applications to waveforms with WaveGrad
4. Implicit models to accelerate inference

[1] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239.

[2] Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv:1503.03585.

[3] Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural computation, 23(7), 1661-1674.

[4] Song, J., Meng, C., & Ermon, S. (2020). Denoising Diffusion Implicit Models. arXiv preprint arXiv:2010.02502.

[5] Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., & Chan, W. (2020). WaveGrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713.