Recent years have seen considerable advances in generative models, which learn distributions from data and also generate new data instances from the learned distribution; and dynamical models, which model systems with a dynamical or temporal component. Both of these developments have been leveraging advances in deep learning. The course will cover key advances in generative and dynamical models, including variational auto-encoders, normalizing flows, generative adversarial networks, diffusion models, neural differential equations, learning operators, among other topics.
| Date | Topic+Papers | Slides | Scribe Notes |
01. | Tue, Jan 17 |
Course Overview |
|
|
02. | Thu, Jan 19 |
RBM 1: Energy Based Models, (Restricted) Boltzmann Machines (RBMs), Sigmoid Belief Networks
- 2007 (Notes) Geoffrey Hinton, Boltzmann Machines, Scholarpedia, 2007.
- 2022 (arXiv) Benyamin Ghojogh, et al., Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey, arXiv, 2022.
- 1990 (TR) Radford Neal, Learning Stochastic Feedforward Networks, Tech Report, 1990.
|
Boltzmann Machines |
|
03. | Tue, Jan 24 |
RBM 2: Learning in (Restricted) Boltzmann Machines, Deep Boltzmann Machines, Contrastive Divergence
- 2005 (AISTATS) Miguel Carreira-Perpignan and Geoffrey Hinton, On contrastive divergence learning, AISTATS, 2005.
- 2012 (TR) Geoffrey Hinton, A Practical Guide to Training Restricted Boltzmann Machines, Univ of Toronto Tech Report, 2012.
- 2012 (NC) Ruslan Salakhutdinov and Geoffrey Hinton, An Efficient Learning Procedure for Deep Boltzmann Machines, Neural Computation, 2012.
- 2010 (AISTATS) Ruslan Salakhutdinov and Hugo Larochelle, Efficient Learning of Deep Boltzmann Machines, AISTATS, 2010.
|
Learning (R)BMs |
|
04. | Thu, Jan 26 |
Inf 1: Markov Chain Monte Carlo, Gibbs Sampling, Langevin Dynamics
|
Sampling, Inference |
|
05. | Tue, Jan 31 |
Inf 2: Hamiltonian Monte Carlo, Variational Inference
|
(included above) |
|
06. | Thu, Feb 2 |
AR 1: Neural Autoregressive Models
- 2011 (AISTATS): H. Larochelle, I. Murray, The Neural Autoregressive Distribution Estimator, AISTATS, 2011.
- 2015 (ICML): M. Germain, K. Gregor, I. Murray, H. Larochelle, Made: Masked autoencoder for distribution estimation, ICML, 2015.
|
NADE, MADE |
|
07. | Tue, Feb 7 |
AR 2: PixelRNN/CNN, Quantiles
- 2016 (ICML): A. van den Oord, N. Kalchbrenner, K. Kavukcuoglu, Pixel recurrent neural networks, ICML, 2016.
- 2016 (NeurIPS): A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, K. Kavukcuoglu, Conditional image generation with PixelCNN decoders, NeurIPS, 2016.
- 2018 (ICML): G. Ostrovski, W. Dabney, R. Munos, Autoregressive Quantile Networks for Generative Modeling, ICML, 2018.
|
PixelRNN, CNN Quantiles |
AR2 notes |
08. | Thu, Feb 9 |
AR 3: Transformers
- 2017 (NeurIPS): A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, NeurIPS, 2017.
- 2019 (arXiv): R. Child, S. Gray, A. Radford, I. Sutskever, Generating Long Sequences with Sparse Transformers, arXiv, 2019.
- 2021 (ICLR): A. Dosovitskiy, et al.: An Image is worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR, 2021
|
Transformers |
AR3 notes |
09. | Tue, Feb 14 |
VAE 1: Variational Autoencoders, Evidence Lower Bound, Importance Weighting
- 2014 (ICLR): D. Kingma, M. Welling, Auto-Encoding Variational Bayes, ICLR, 2014.
- 2014 (ICML): D. Rezende, S. Mohamed, D. Wierstra, Stochastic Backpropagation and ApproximateIInference in Deep Generative Models, ICML, 2014.
- 2019 (FTML): D. Kingma and M. Welling, An Introduction to Variational Autoencoders, FTML, 2019.
- 2016 (ICLR): Y. Burda, R. Grosse, R. Salakhutdinov. Importance Weighted Autoencoders. ICLR, 2016.
|
VAE basics |
VAE1 notes 1
VAE1 notes 2 |
10. | Thu, Feb 16 |
VAE 2: Improved Inference, Information Theoretic Perspective, Representation
- 2018 (ICML): T. Rainforth, A. Kosiorek, T. Le, C. Maddison, M. Igl, F. Wood, Y. Teh, Tighter Variational Bounds are Not Necessarily Better. ICML, 2018.
- 2018 (ICML): A. Alemi, B. Poole, I. Fischer, J. Dillon, R. Saurous, K. Murphy. Fixing a broken ELBO. ICML, 2018.
- 2019 (ICLR): B. Dai, D. Wipf. Diagnosing and Enhancing VAE Models. ICLR, 2019.
|
ELBO, Rate Distortion
Diagnosing VAEs |
VAE2 notes 1
VAE2 notes 2 |
11. | Tue, Feb 21 |
VAE 3: VQ VAE, Deep VAE
|
VQ VAE, Deep VAE |
VAE3 notes 1
VAE3 notes 2 |
12. | Thu, Feb 23 |
VAE 4: Disentanglement
- 2017 (ICLR): I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, A. Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. ICLR, 2017.
- 2018 (NeurIPS): T. Chen, X. Li, R. Grosse, D. Duvenaud. Isolating sources of disentanglement in variational autoencoders. NeurIPS, 2018.
- 2019 (ICML): Emile Mathieu, Tom Rainforth, N Siddharth, Yee Whye Teh Disentangling Disentanglement in Variational Autoencoders ICML, 2019.
- 2019 (ICML): F. Locatello, S. Bauer, M. Lucic, G. Ratsch, S. Gelly, B. Scholkopf, O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. ICML, 2019.
|
Disentanglement 1
Disentanglement 2 |
VAE4 notes 1
VAE4 notes 2 |
13. | Tue, Feb 28 |
DiffM1: Score Matching, Diffusion Models
|
Background, Early Development |
DiffM1 notes 1
DiffM1 notes 2 |
14. | Thu, Mar 2 |
DiffM2: Score Matching, Diffusion Models
- 2019 (UAI): Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon, Sliced score matching: A scalable approach to density and score estimation UAI, 2019.
- 2019 (NeurIPS): Y. Song, S. Ermon. Generative modeling by estimating gradients of the data distribution NeurIPS, 2019.
- 2020 (NeurIPS): J. Ho, A. Jain, P. Abbeel. Denoising diffusion probabilistic models NeurIPS, 2020.
- 2021 (ICLR): Y. Song, J. Sohl-Dickstein, D. Kingma, A. Kumar, S. Ermon, B. Poole. Score-based generative modeling through stochastic differential equations ICLR, 2021.
|
15. | Tue, Mar 7 |
DiffM3: Connections
- 2007 (IEEE Trans NN): Aapo Hyvarinen, Connections between score matching, contrastive divergence, and pseudolikelihood for continuous-valued variables IEEE Transactions on neural networks, 18(5):1529-1531, 2007.
- 2012 (JMLR): Michael Gutmann and Aapo Hyvarinen, Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics, JMLR, 2012.
- 2011 (NC): Pascal Vincent, A connection between score matching and denoising autoencoders, Neural Computation, 23(7):1661-1674, 2011.
- 2021 (arXiv): Yang Song, Diederik Kingma, How to Train Your Energy-Based Models arXiv, 2021.
|
Connections |
DiffM3 notes |
16. | Thu, Mar 9 |
DiffM4: Convergence, Stable Diffusion
- 2014 (Book): Ramon Van Handel, Probability in high dimension, Chapter 2, Princeton Univ, 2014.
- 2022 (CVPR): Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Bjorn Ommer, High-Resolution Image Synthesis with Latent Diffusion Models, CVPR, 2022.
|
Convergence, Stable Diffusion |
DiffM4 notes 1
DiffM4 notes 2 |
| Tue, Mar 14 |
Spring Break |
|
|
| Thu, Mar 16 |
Spring Break |
|
|
17. | Tue, Mar 21 |
NF 1: Normalizing Flows
- 2016 (ICLR): L. Dinh, D. Krueger, Y. Bengio. NICE: Non-linear Independent Components Estimation. ICLR Workshop, 2015.
- 2015 (ICML): D. Rezende, S. Mohamed, Variational Inference with Normalizing Flows, ICML, 2015.
- 2017 (ICLR): L. Dinh, J. Sohl-Dickstein, S. Bengio. Density Estimation using Real NVP. ICLR, 2017.
- 2018 (NeurIPS): D. P. Kingma, P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. NeurIPS, 2018.
|
Normalizing Flows |
NF1 notes |
18. | Thu, Mar 23 |
NF 2: Autoregressive Flows
|
Autoregressive Flows |
NF 2 notes |
19. | Tue, Mar 28 |
NF 3: Improved, Neural, Residual Flows
- 2019 (ICML): J. Ho, X. Chen, A. Srinivas, Y. Duan, P. Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design.. ICML, 2019.
- 2018 (ICML): C.-W. Huang, D. Krueger, A. Lacoste, A. Courville. Neural Autoregressive Flows. ICML, 2018.
- 2019 (NeurIPS): R. Chen, J. Behrmann, D. Duvenaud, J. Jacobsen. Residual Flows for Invertible Generative Modeling. NeurIPS, 2019.
|
Improved Flows |
NF3 notes |
20. | Thu, Mar 30 |
GAN 1: Basics, Info-GAN, f-GAN
- 2014 (NeurIPS): I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative Adversarial Networks, NeurIPS, 2014.
- 2014 (arXiv): Mehdi Mirza, Simon Osindero, Conditional Generative Adversarial Nets, arXiv, 2014.
- 2016 (NeurIPS): X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. NeurIPS, 2016.
- 2010 (IEEE Info Theory): XuanLong Nguyen, Martin J. Wainwright, Michael I. Jordan, Estimating divergence functionals and the likelihood ratio by convex risk minimization, IEEE Transactions on Information Theory, 2010.
- 2016 (NeurIPS): S. Nowozin, B. Cseke, R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. NeurIPS, 2016.
|
Basics, Info-GAN
f-GAN |
GAN1 notes
GAN2 notes |
21. | Tue, Apr 4 |
GAN 2: DC-GAN, Cycle-GAN, Style-GAN
- 2015 (arXiv): A. Radford, L. Metz, S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
- 2017 (ICCV): J. Zhu, T. Park, P. Isola, A. Efros. Unpaired image-to image translation using cycle-consistent adversarial networks. ICCV, 2017.
- 2017 (ICCV): Xun Huang, Serge Belongie, Arbitrary Style Transfer in Real-Time With Adaptive Instance Normalization, ICCV, 2017.
- 2019 (CVPR): Tero Karras, Samuli Laine, Timo Aila, A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR, 2019.
|
DC-GAN, Cycle-GAN
Style-GAN |
GAN2 notes |
22. | Tue, Apr 6 |
GAN 3: Wasserstein GANs
- 2017 (ICLR): M. Arjovsky, L. Bottou. Towards principled methods for training generative adversarial networks. In ICLR, 2017.
- 2017 (ICML): M. Arjovsky, S. Chintala, L. Bottou. Wasserstein GAN. In ICML, 2017.
- 2017 (NeurIPS): Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. C. Courville. Improved training of Wasserstein GANs. NeurIPS, 2017.
|
WGAN 1
WGAN 2 |
GAN3 notes |
23. | Thu, Apr 11 |
GAN 4: Optimization
|
GAN Optimization |
GAN4 notes |
24. | Thu, Apr 13 |
NODE 1: Neural Ordinary Differential Equations 1
|
Neural ODE |
NODE1 notes |
25. | Tue, Apr 18 |
NODE 2: Neural Ordinary Differential Equations 2
|
Augmented ODE |
NODE2 notes |
26. | Thu, Apr 20 |
LOP 1: Deep Operator Networks
|
PINNs
DeepONets |
LOP1 notes |
27. | Tue, Apr 25 |
LOP 2: Fourier Neural Operators
- 2021 (ICLR): Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A.Anandkumar, Fourier neural operator for parametric partial differential equations ICLR, 2021.
- 2022 (JMLR): Nikola Kovachki, Samuel Lanthaler, Siddhartha Mishra, On universal approximation and error bounds for Fourier Neural Operators, JMLR, 2022.
|
FNOs
FNO theory |
LOP2 notes |
28. | Thu, Apr 27 |
PP 1: Project Presentations 1
|
|
|
29. | Tue, May 2 |
PP 2: Project Presentations 2
|
|
|