mattiasvillani/BayesLearnCourse

Contents

The course gives a gentle but solid introduction to Bayesian statistical inference, with special emphasis on models and methods in computational statistics and machine learning.

• We will get off to a shocking start by introducing a very different probability concept than the one you are probably used to: subjective probability.
• We will then move on to the mathematics of the prior-to-posterior updating in basic statistical models, such as the Bernoulli, normal and multinomial models.
• Bayesian prediction and decision making under uncertainty is carefully explained, and you will hopefully see why Bayesian methods are so useful in modern application where so much focuses on prediction and decision making.
• The really interesting stuff starts to happen when we study regression and classification. You will learn how prior distributions can be seen as regularization that allows us to use flexible models with many parameters without overfitting the data and improving predictive performance.
• A new world will open up when we learn how complex models can be analyzed with simulation methods like Markov Chain Monte Carlo (MCMC), Hamiltonian Monte Carlo (HMC) and approximate optimization methods like Variational Inference (VI).
• You will also get a taste for probabilistic programming languages for Bayesian learning, in particular the popular Stan language in R.
• Finally, we'll round off with an introduction to Bayesian model selection and Bayesian variable selection.

Course literature and Schedule

The course will use the following book as the main course literature:

• Gelman, Carlin, Stern, Dunson, Vehtari, Rubin (2014). Bayesian Data Analysis (BDA). Chapman & Hall/CRC: Boca Raton, Florida. 3rd edition. Here is the book webpage and a free PDF version.
• Villani, M. (2022). Bayesian Learning (BL). Approximately half of a book that I am writing is here. This is work in progress, but currently covers L1-L6 and L11.
• Additional course material linked from this page, such as articles and tutorial.

The course schedule on TimeEdit is here: Schedule.

Computer labs

• The three computer labs are central to the course. Expect to allocate substantial time for each lab, not just the scheduled time for computer exercises. Many of the exam questions will be computer based, so working on the labs will also help you with the exam.

• The labs should be done in R, since only R is available at the computer exam and your lab solutions will be available for you at the exam.

• The labs should be done in pairs of students.

• Each lab report should be submitted as a PDF along with the .R file with code. Submission is done through Athena.

• There is only two hours of supervised time allocated to each lab. The idea is that you:

  • should start working on the lab before the computer session
  • so that you are in a position to ask questions at the session
  • and then finish up the report after the lab session.

• The datasets used in the labs are included in the SUdatasets R package which is available on github. There is also a course package bayeslearn with some distributions and functions for regression, logistic regression etc. To install the packages, you first need to install the remotes package.

# To install and load the package with the data
install.packages("remotes") # only one time
library(remotes)  
install_github("StatisticsSU/SUdatasets") # only one time
library(SUdatasets)
install_github("StatisticsSU/bayeslearn") # only one time
library(bayeslearn)

# For a list of all available data sets
help(, "SUdatasets")

# For information about specific data sets
help(name_of_dataset)

Teachers

Mattias Villani, Lecturer
Professor, Natural Born Bayesian 😜
Department of Statistics, Stockholm University \

Oscar Oelrich, Math exercises and Computer labs
PhD in Statistics, specializing in Bayesian Statistics
Department of Statistics, Stockholm University

Lectures

Lecturer: Mattias Villani

Lecture 1 - Subjective probability. Bayesian updating. Bernoulli model. Gaussian model.
Reading: BDA Ch. 1, 2.1-2.4 | BL Ch. 1-2, 4 | Slides
Notebooks: Analyzing email spam data with a Bernoulli model | Analyzing internet download speeds with a Gaussian model
Widgets: gaussian known variance | bayesian updating - widget

Lecture 2 - Poisson model. Summarizing posteriors. Conjugate priors. Prior elicitation.
Reading: BDA Ch. 2.-2.9 | BL Ch. 2 and 4 | Slides
Notebooks: Analyzing number of eBay bidders with an iid Poisson model
Widget: Autoregressive processes

Lecture 3 - Multi-parameter models. Marginalization. Monte Carlo simulation. Multinomial model.
Reading: BDA Ch. 3 | BL Ch. 3 | Slides
Notebooks: Analyzing mobile phone survey data with a multinomial model

Lecture 4 - Linear Regression. Prediction. Making Decisions
Reading: BDA Ch. 14, Ch. 9.1-9.2 | BL Ch. 5 and 6 | Slides\
Code: Prediction with two-parameter Gaussian model
Widgets: linear regression | prediction iid Gaussian

Lecture 5 - Classification. Large-sample properties and Posterior approximation.
Reading: BDA Ch. 16.1-16.3, 4.1-4.2 | BL 7-8 and appendix on Taylor approximations | Slides
Notebook: Logistic regression
Code: bayeslearn package - Logistic and Probit regression for Titanic data
Extras: Video on Taylor approximation

Lecture 6 - Nonlinear regression and Classification. Regularization.
Reading: BDA Ch. 20.1-20.2 | Slides

Lecture 7 - Gibbs sampling and Data augmentation
Reading: BDA Ch. 10-11 | Slides
Code: Gibbs sampling for a bivariate normal | Gibbs sampling for a mixture of normals
Widget: mixture of normals
Extras: Illustration of Gibbs sampling when parameters are strongly correlated.

Lecture 8 - MCMC and Metropolis-Hastings
Reading: BDA Ch. 11 | Slides
Code: Simulating Markov Chains

Lecture 9 - HMC and Variational inference
Reading: BDA Ch. 12.4 and Ch. 13.7 | Slides
Extras: HMC animations | HarlMCMC shake | visual explantion of HMC algorithm tuning | Radford Neal's handbook chapter on HMC

Lecture 10 - Probabilistic programming for Bayesian learning
Reading: RStan vignette | Linear regression in RStan | Prediction in RStan | Slides
RStan Code: Bernoulli | Normal model | Logistic regression | Logistic regression with random effects | Poisson model | Three plants data
Turing code (Julia): Bernoulli model | Normal model

Lecture 11 - Bayesian model comparison
Reading: BDA Ch. 7 | BL Ch. 12 | Slides
Code: Comparing models for count data

Lecture 12 - Bayesian variable selection and model averaging
Reading: Article on Bayesian variable selection | Slides

Mathematical exercises

Assistant: Oscar Oelrich

Exercise session 1: Problem set

Exercise session 2: Problem set

Computer labs

Assistant: Oscar Oelrich

Computer Lab 1 - Exploring posterior distributions in one-parameter models by simulation and direct numerical evaluation.
Lab: Lab 1
Submission: Athena.

Computer Lab 2 - Polynomial regression and classification with logistic regression.
Lab: Lab 2
Submission: Athena.

Computer Lab 3 - Gibbs sampling, Metropolis-Hastings and Stan.
Lab: Lab 3 | Note on how to code R functions which take functions as arguments
Submission: Athena.

Examination

The course examination consists of:

• Written lab reports (deadlines given in Athena)
• Exam (pen and paper + computer)

The written exam will be in a room with computer access and R will be installed. Roughly 1/4 of the exam will be pen-and-paper math problems, and 3/4 computer based problems. You will be given the opportunity to submit your lab solutions as a PDF so that they will be available to you during the exam. The uploaded lab solutions can be polished, but are only allowed to contain material related to the lab solutions. The uploaded lab solutions will be screened before the exam. You will also have the following PDF with distributions available during the exam Distributions from BDA3.

Some old exams with solutions:

• Old exams

R

• The main page with links to downloads for the programming language R
• RStudio - a very nice developing environment for R.
• An introduction to R | John Cook's intro to R for programmers | base-R cheat sheet .