Contents

Computational Statistics for Bayesian Inference with PyMC3

This series of notebooks and material is being put together by Dr. Srijith Rajamohan with the introductory lectures on the foundations of Probability and the Bayes Theorem being offered by Dr. Robert Settlage. The purpose of this series is to teach the basics of Bayesian statistics for the purpose of performing Inference. This is not intended to be a comprehensive course that teaches the basics of statistics and probability nor does it cover Frequentist statistical techniques based on the Null Hypothesis Significance Testing (NHST). What it does cover is:

  • The basics of Bayesian probability

  • Understanding Bayesian inference and how it works

  • The bare-minimum set of tools and a body of knowledge required to perform Bayesian inference in Python, i.e. the PyData stack of NumPy, Pandas, Scipy, Matplotlib, Seaborn and Plot.ly

  • A scalable Python-based framework for performing Bayesian inference, i.e. PyMC3

With this goal in mind, the content is divided into the following three main sections (courses).

  • Introduction to Bayesian Statistics

  • Introduction to Monte Carlo Methods

  • PyMC3 for Bayesian Modeling and Inference

Please read the section titled ‘The What, Why and Whom…’.

Note

This draws a lot of inspiration from some brilliant people in this field and I will list those names here as this work is being developed.

PyMC3 can be found here.

Course Outline

Course 1 - Introduction to Bayesian Statistics

  • The Foundations of Probability

  • Distributions, Central Tendencies and Shape Parameters

  • Parameter Estimation

  • Introduction to the Bayes Theorem

  • Inference and Decisions

  • Bayesian and Frequentist approach

  • Distributions

    • Generate data and Parameter estimation

    • Gaussian Mixture Models

    • Information Criterion

  • Non-parametric Methods and Kernel Density Estimation

  • Introduction to Sampling

  • Sampling from Discrete Distributions

  • Inverse Transform Method

  • Rejection Sampling Method

  • Importance Sampling Method

Course 2 - Introduction to Monte Carlo Methods

  • R2 and Explained Variance

  • Underfitting vs. Overfitting, Simplicity vs. Accuracy

  • Cross Validation

  • Log-likelihood and Deviance

  • AIC and WAIC

  • Entropy

  • KL Divergence

  • Model Averaging

  • Stationarity and Ergodicity

  • Building blocks - Markov Chains

  • Building blocks - Why does it work?

  • Foundations of Bayesian Inference

  • Outline of the Metropolis Algorithm

  • Building the Inferred Distribution

  • Python Code for the Metropolis Algorithm

  • Introduction to the Metropolis-Hastings Algorithm

  • Introduction to Gibbs Sampling

  • Details of the Gibbs Sampling algorithm 1

  • Details of the Gibbs Sampling algorithm 2

  • Hamiltonian Monte Carlo

  • Characteristics of MCMC

Course 3 - PyMC3 for Bayesian Modeling and Inference

  • Introduction to PyMC3

  • Introduction to PyMC3 with Linear Regression

  • Introduction to PyMC3 - Traces

  • Composition of Distributions for Uncertainty

  • Highest Posterior Density and Region of Practical Equivalence

  • Credible Intervals and Confidence Intervals

  • Modeling with a Gaussian Distribution

  • Using PyMC3 to Model a Phenomenon with a Gaussian distribution

  • Posterior Predictive Checks

  • Robust Models with a Student’s t-Distribution

  • Hierarchical/Multilevel Models

  • Hierarchical Models - Shrinkage

  • Linear Regression

  • Mean-center for Linear Regression

  • Hierarchical Linear Regression

  • Polynomial Regression for Non-linear Data

  • Multiple Linear Regression

  • Logistic Regression

  • Multiple Logistic Regression

  • Multiclass Classification

  • Inferring Rate Change with a Poisson Distribution

  • Tuning

  • Mixing and Potential Scale Reduction Factor

  • Centered vs. Non-centered Parameterization

  • Autocorrelation and Effective Sample Size

  • Monte Carlo Error

  • Divergence

  • Revisiting the Multiclass Classification problem

  • PyMC3 metrics

  • Diagnosing and Debugging MCMC with PyMC3 (7 min)

  • ArViz Data Representation

  • PyMC3 COVID project