Recent years have seen considerable advances in generative models, which learn distributions from data and also generate new data instances from the learned distribution; and dynamical models, which model systems with a dynamical or temporal component. Both of these developments have been leveraging advances in deep learning. The course will cover key advances in generative and dynamical models, including variational auto-encoders, normalizing flows, generative adversarial networks, neural differential equations, physics guided machine learning, among other topics.

	Date	Topic+Papers	Slides+Notes	Presenter(s)
01.	Tue, Aug 24	Course Overview	Overview	Arindam
02.	Thu, Aug 26	Probabilistic Models: Introduction	Prob. Models: Intro	Arindam
03.	Tue, Aug 31	Intro to Deep Generative: Part 1 2014 (ICLR): D. Kingma, M. Welling, Auto-Encoding Variational Bayes, ICLR, 2014. 2014 (ICML): D. Rezende, S. Mohamed, D. Wierstra, Stochastic Backpropagation and Approximate Inference in Deep Generative Models, ICML, 2014. 2019 (FTML): D. Kingma and M. Welling, An Introduction to Variational Autoencoders, FTML, 2019.	Intro to VAEs	Arindam
04.	Thu, Sept 02	Intro to Deep Generative: Part 2 2015 (ICML): D. Rezende, S. Mohamed, Variational Inference with Normalizing Flows, ICML, 2015. 2014 (NeurIPS): I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative Adversarial Networks, NeurIPS, 2014.	Intro to NFs, GANs	Arindam
05.	Tue, Sept 07	AR 1: Deep Autoregressive Models 2011 (AISTATS): H. Larochelle, I. Murray, The Neural Autoregressive Distribution Estimator, AISTATS, 2011. 2015 (ICML): M. Germain, K. Gregor, I. Murray, H. Larochelle, Made: Masked autoencoder for distribution estimation, ICML, 2015.	NADE MADE	Varun Arindam
06.	Thu, Sept 09	AR 2: PixelRNN/CNN, Quantiles 2016 (ICML): A. van den Oord, N. Kalchbrenner, K. Kavukcuoglu, Pixel recurrent neural networks, ICML, 2016. 2016 (NeurIPS): A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, K. Kavukcuoglu, Conditional image generation with PixelCNN decoders, NeurIPS, 2016. 2018 (ICML): G. Ostrovski, W. Dabney, R. Munos, Autoregressive Quantile Networks for Generative Modeling, ICML, 2018.	Pixel RNN/CNN AR Quantile	Sanchit Dachun
07.	Tue, Sept 14	AR 3: Transformers 2017 (NeurIPS): A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, NeurIPS, 2017. 2019 (arXiv): R. Child, S. Gray, A. Radford, I. Sutskever, Generating Long Sequences with Sparse Transformers, arXiv, 2019. 2021 (ICLR): A. Dosovitskiy, et al.: An Image is worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR, 2021	Transformers Sparsity, Vision apps	Carl Sanchit
08.	Thu, Sept 16	VAE 1: Evidence Lower Bound 2016 (ICLR): Y. Burda, R. Grosse, R. Salakhutdinov. Importance Weighted Autoencoders. ICLR, 2016. 2018 (ICML): T. Rainforth, A. Kosiorek, T. Le, C. Maddison, M. Igl, F. Wood, Y. Teh, Tighter Variational Bounds are Not Necessarily Better. ICML, 2018. 2018 (ICML): A. Alemi, B. Poole, I. Fischer, J. Dillon, R. Saurous, K. Murphy. Fixing a broken ELBO. ICML, 2018.	Importance Weighting SNR Rate Distortion	Shengyu Varun
09.	Tue, Sept 21	VAE 2: Improved Inference, Representation 2016 (NeurIPS): C. Sonderby, T. Raiko, L. Maaloe, S. Sonderby, O. Winther. Ladder Variational Autoencoders. NeurIPS, 2016. 2019 (ICLR): B. Dai, D. Wipf. Diagnosing and Enhancing VAE Models. ICLR, 2019. 2017 (NeurIPS): A. van den Oord, O. Vinyals, K. Kavukcuoglu. Neural Discrete Representation Learning. NeurIPS, 2017.	LVAE VAE Opt VQ-VAE	Hantao Minhao Hanhtao
10.	Thu, Sept 23	VAE 3: Deep Hierarchical 2019 (NeurIPS): L. Maaloe, M. Fraccaro, V. Lievin, O. Winther. BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling. NeurIPS, 2019. 2020 (NeurIPS): A. Vahdat, J. Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. NeurIPS, 2020. 2021 (ICLR): R. Child. Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images. ICLR, 2021.	BIVA, NVAE Very Deep VAE	Yuancheng Xiaoyang
11.	Tue, Sept 28	VAE 4: Disentanglement 2017 (ICLR): I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, A. Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. ICLR, 2017. 2018 (NeurIPS): T. Chen, X. Li, R. Grosse, D. Duvenaud. Isolating sources of disentanglement in variational autoencoders. NeurIPS, 2018. 2019 (ICML): F. Locatello, S. Bauer, M. Lucic, G. Ratsch, S. Gelly, B. Scholkopf, O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. ICML, 2019.	beta-VAE, variants Challenges	Dachun Shengyu
12.	Thu, Sept 30	NF 1: Improved Flows 2016 (ICLR): L. Dinh, D. Krueger, Y. Bengio. NICE: Non-linear Independent Components Estimation. ICLR Workshop, 2015. 2017 (ICLR): L. Dinh, J. Sohl-Dickstein, S. Bengio. Density Estimation using Real NVP. ICLR, 2017. 2018 (NeurIPS): D. P. Kingma, P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. NeurIPS, 2018.	Flow based models	Arindam
13.	Tue, Oct 05	NF 2: Autoregressive Flows 2016 (NeurIPS): D. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, M. Welling. Improved Variational Inference with Inverse Autoregressive Flow. NeurIPS, 2016. 2017 (NeurIPS): G. Papamakarios, T. Pavlakou, I. Murray. Masked Autoregressive Flow for Density Estimation. NeurIPS, 2017.	Autoregressive flows	Arindam
14.	Thu, Oct 07	NF 3: Improved, Neural, Residual Flows 2019 (ICML): J. Ho, X. Chen, A. Srinivas, Y. Duan, P. Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design.. ICML, 2019. 2018 (ICML): C.-W. Huang, D. Krueger, A. Lacoste, A. Courville. Neural Autoregressive Flows. ICML, 2018. 2019 (NeurIPS): R. Chen, J. Behrmann, D. Duvenaud, J. Jacobsen. Residual Flows for Invertible Generative Modeling. NeurIPS, 2019.	Improved Flows	Arindam
15.	Tue, Oct 12	NF 4: Discrete+Mixed Models 2019 (NeurIPS): D. Tran, K. Vafa, K. Agrawal, L. Dinh, B. Poole. Discrete flows: Invertible generative models of discrete data. NeurIPS, 2019.. 2019 (NeurIPS): E. Hoogeboom, J. Peters, R. van den Berg, M. Welling. Integer discrete flows and lossless compression. NeurIPS, 2019. 2020 (NeurIPS): D. Nielsen, P. Jaini, E. Hoogeboom, O. Winther, M. Welling, SurVAE flows: Surjections to bridge the gap between VAEs and flows, NeurIPS, 2020.	Discrete Discrete, Mixed	Carl Arindam
16.	Thu, Oct 14	GAN 1: Info-GAN, f-GAN 2016 (NeurIPS): X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. NeurIPS, 2016. 2016 (NeurIPS): S. Nowozin, B. Cseke, R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. NeurIPS, 2016.	InfoGAN f-GAN	Yuancheng Vivek
17.	Tue, Oct 19	GAN 2: Cycle-GAN, DC-GAN 2015 (arXiv): A. Radford, L. Metz, S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 2017 (ICCV): J. Zhu, T. Park, P. Isola, A. Efros. Unpaired image-to image translation using cycle-consistent adversarial networks. ICCV, 2017.	DCGAN Cycle-GAN	Brandon Nengyu
18.	Thu, Oct 21	GAN 3: Wasserstein GANs 2017 (ICLR): M. Arjovsky, L. Bottou. Towards principled methods for training generative adversarial networks. In ICLR, 2017. 2017 (ICML): M. Arjovsky, S. Chintala, L. Bottou. Wasserstein GAN. In ICML, 2017. 2017 (NeurIPS): Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. C. Courville. Improved training of Wasserstein GANs. NeurIPS, 2017.	GAN Training WGAN	Hantao Xiaoyang
19.	Tue, Oct 26	GAN 4: Optimization 2018 (ICML): L. Mescheder, A. Geiger, S. Nowozin. Which training methods for gans do actually converge? ICML, 2018. 2018 (ICLR): A. Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard GAN. ICLR, 2018. 2020 (NeurIPS): R. Sun, T. Fang, A. Schwing. Towards a better global loss landscape of GANs. NeurIPS, 2020.	Convergence, Relativistic Better Loss Landscapes	Minhao Nengyu
20.	Thu, Oct 28	GAN 5: Generalization 2017 (ICML): S. Arora, R. Ge, Y. Liang, T. Ma, Y. Zhang. Generalization and equilibrium in generative adversarial nets (GANs). ICML, 2017. 2019 (NeurIPS): B. Wu, S. Zhao, C. Chen, H. Xu, L. Wang, L., X. Zhang, G. Sun, J. Zhou. Generalization in generative adversarial networks: A novel perspective from privacy protection. NeurIPS, 2019.	Generalization, Equilibrium Privacy	Vivek Brandon
21.	Tue, Nov 02	NODE 1: Neural Ordinary Differential Equations 1 2018 (NeurIPS): R. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud. Neural ordinary differential equations NeurIPS, 2018. 2020 (ICML): C. Finlay, J. Jacobsen, L. Nurbekyan, A. Oberman. How to train your neural ODE ICML, 2020.	Neural ODE Regularization, Transport	Anurendra Arindam
22.	Thu, Nov 04	NODE 2: Neural Ordinary Differential Equations 2 2019 (NeurIPS): E. Dupont, A. Doucet, Y. W. Teh. Augmented Neural ODEs NeurIPS, 2019. 2020 (ICML): H. Zhang, X. Gao, J. Unterman, T. Arodz. Approximation Capabilities of Neural ODEs and Invertible Residual Networks ICML, 2020.	Augmented Neural ODE Universal Approximator	Bhavesh Arindam
23.	Tue, Nov 09	NSDE 1: Stochastic Dynamics 2015 (ICML): T. Salimans, A. Diederik, D. P. Kingma, M. Welling. Markov Chain Monte Carlo and Variational Inference: Bridging the Gap ICML, 2015. 2015 (ICML): J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics ICML, 2015.	Stochastic Dynamics	Arindam
24.	Thu, Nov 11	SBM 1: Score-based Models 1 2015 (JMLR): A. Hyvarinen. Estimation of non-normalized statistical models by score matching JMLR, 2005. 2019 (NeurIPS): Y. Song, S. Ermon. Generative modeling by estimating gradients of the data distribution NeurIPS, 2019.	Score-based Models 1	Arindam
25.	Tue, Nov 16	SBM 2: Score-based Models 2 2020 (NeurIPS): J. Ho, A. Jain, P. Abbeel. Denoising diffusion probabilistic models NeurIPS, 2020. 2021 (ICLR): Y. Song, J. Sohl-Dickstein, D. Kingma, A. Kumar, S. Ermon, B. Poole. Score-based generative modeling through stochastic differential equations ICLR, 2021.	Score-based Models 2	Arindam
26.	Thu, Nov 18	DS 1: Physics+ML 2019 (JCP): M. Raissi, P. Perdikaris, G. E. Karniadakis Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations J. Comp. Phy., 2019. 2016 (PNAS): S. L. Brunton, J. L. Proctor, J. N. Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems PNAS, 2016.	PINN Discovering Equations	Bhavesh Anurendra
27.	Tue, Nov 30	DS 2: Learning Operators 2021 (ICLR): Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A.Anandkumar, Fourier neural operator for parametric partial differential equations ICLR, 2021. 2021 (NMI): L. Lu, R. Jin, G.Pang, Z. Zhang, G. E. Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators NMI, 2021.	Learning Operators	Arindam
28.	Thu, Dec 02	PP1: Project Presentations 1
29.	Tue, Dec 07	PP2: Project Presentations 2

Date

Topic+Papers

Slides+Notes

Presenter(s)

01.

Tue, Aug 24

Course Overview

Overview

Arindam

02.

Thu, Aug 26

Probabilistic Models: Introduction

Prob. Models: Intro

Arindam

03.

Tue, Aug 31

Intro to Deep Generative: Part 1

2014 (ICLR): D. Kingma, M. Welling, Auto-Encoding Variational Bayes, ICLR, 2014.
2014 (ICML): D. Rezende, S. Mohamed, D. Wierstra, Stochastic Backpropagation and Approximate Inference in Deep Generative Models, ICML, 2014.
2019 (FTML): D. Kingma and M. Welling, An Introduction to Variational Autoencoders, FTML, 2019.

Intro to VAEs

Arindam

04.

Thu, Sept 02

Intro to Deep Generative: Part 2

2015 (ICML): D. Rezende, S. Mohamed, Variational Inference with Normalizing Flows, ICML, 2015.
2014 (NeurIPS): I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative Adversarial Networks, NeurIPS, 2014.

Intro to NFs, GANs

Arindam

05.

Tue, Sept 07

AR 1: Deep Autoregressive Models

2011 (AISTATS): H. Larochelle, I. Murray, The Neural Autoregressive Distribution Estimator, AISTATS, 2011.
2015 (ICML): M. Germain, K. Gregor, I. Murray, H. Larochelle, Made: Masked autoencoder for distribution estimation, ICML, 2015.

NADE

MADE

Varun

Arindam

06.

Thu, Sept 09

AR 2: PixelRNN/CNN, Quantiles

2016 (ICML): A. van den Oord, N. Kalchbrenner, K. Kavukcuoglu, Pixel recurrent neural networks, ICML, 2016.
2016 (NeurIPS): A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, K. Kavukcuoglu, Conditional image generation with PixelCNN decoders, NeurIPS, 2016.
2018 (ICML): G. Ostrovski, W. Dabney, R. Munos, Autoregressive Quantile Networks for Generative Modeling, ICML, 2018.

Pixel RNN/CNN

AR Quantile

Sanchit

Dachun

07.

Tue, Sept 14

AR 3: Transformers

2017 (NeurIPS): A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, NeurIPS, 2017.
2019 (arXiv): R. Child, S. Gray, A. Radford, I. Sutskever, Generating Long Sequences with Sparse Transformers, arXiv, 2019.
2021 (ICLR): A. Dosovitskiy, et al.: An Image is worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR, 2021

Transformers

Sparsity, Vision apps

Carl

Sanchit

08.

Thu, Sept 16

VAE 1: Evidence Lower Bound

2016 (ICLR): Y. Burda, R. Grosse, R. Salakhutdinov. Importance Weighted Autoencoders. ICLR, 2016.
2018 (ICML): T. Rainforth, A. Kosiorek, T. Le, C. Maddison, M. Igl, F. Wood, Y. Teh, Tighter Variational Bounds are Not Necessarily Better. ICML, 2018.
2018 (ICML): A. Alemi, B. Poole, I. Fischer, J. Dillon, R. Saurous, K. Murphy. Fixing a broken ELBO. ICML, 2018.

Importance Weighting

SNR
Rate Distortion

Shengyu

Varun

09.

Tue, Sept 21

VAE 2: Improved Inference, Representation

2016 (NeurIPS): C. Sonderby, T. Raiko, L. Maaloe, S. Sonderby, O. Winther. Ladder Variational Autoencoders. NeurIPS, 2016.
2019 (ICLR): B. Dai, D. Wipf. Diagnosing and Enhancing VAE Models. ICLR, 2019.
2017 (NeurIPS): A. van den Oord, O. Vinyals, K. Kavukcuoglu. Neural Discrete Representation Learning. NeurIPS, 2017.

LVAE

VAE Opt

VQ-VAE

Hantao

Minhao

Hanhtao

10.

Thu, Sept 23

VAE 3: Deep Hierarchical

2019 (NeurIPS): L. Maaloe, M. Fraccaro, V. Lievin, O. Winther. BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling. NeurIPS, 2019.
2020 (NeurIPS): A. Vahdat, J. Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. NeurIPS, 2020.
2021 (ICLR): R. Child. Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images. ICLR, 2021.

BIVA, NVAE

Very Deep VAE

Yuancheng

Xiaoyang

11.

Tue, Sept 28

VAE 4: Disentanglement

2017 (ICLR): I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, A. Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. ICLR, 2017.
2018 (NeurIPS): T. Chen, X. Li, R. Grosse, D. Duvenaud. Isolating sources of disentanglement in variational autoencoders. NeurIPS, 2018.
2019 (ICML): F. Locatello, S. Bauer, M. Lucic, G. Ratsch, S. Gelly, B. Scholkopf, O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. ICML, 2019.

beta-VAE, variants

Challenges

Dachun

Shengyu

12.

Thu, Sept 30

NF 1: Improved Flows

2016 (ICLR): L. Dinh, D. Krueger, Y. Bengio. NICE: Non-linear Independent Components Estimation. ICLR Workshop, 2015.
2017 (ICLR): L. Dinh, J. Sohl-Dickstein, S. Bengio. Density Estimation using Real NVP. ICLR, 2017.
2018 (NeurIPS): D. P. Kingma, P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. NeurIPS, 2018.

Flow based models

Arindam

13.

Tue, Oct 05

NF 2: Autoregressive Flows

2016 (NeurIPS): D. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, M. Welling. Improved Variational Inference with Inverse Autoregressive Flow. NeurIPS, 2016.
2017 (NeurIPS): G. Papamakarios, T. Pavlakou, I. Murray. Masked Autoregressive Flow for Density Estimation. NeurIPS, 2017.

Autoregressive flows

Arindam

14.

Thu, Oct 07

NF 3: Improved, Neural, Residual Flows

2019 (ICML): J. Ho, X. Chen, A. Srinivas, Y. Duan, P. Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design.. ICML, 2019.
2018 (ICML): C.-W. Huang, D. Krueger, A. Lacoste, A. Courville. Neural Autoregressive Flows. ICML, 2018.
2019 (NeurIPS): R. Chen, J. Behrmann, D. Duvenaud, J. Jacobsen. Residual Flows for Invertible Generative Modeling. NeurIPS, 2019.

Improved Flows

Arindam

15.

Tue, Oct 12

NF 4: Discrete+Mixed Models

2019 (NeurIPS): D. Tran, K. Vafa, K. Agrawal, L. Dinh, B. Poole. Discrete flows: Invertible generative models of discrete data. NeurIPS, 2019..
2019 (NeurIPS): E. Hoogeboom, J. Peters, R. van den Berg, M. Welling. Integer discrete flows and lossless compression. NeurIPS, 2019.
2020 (NeurIPS): D. Nielsen, P. Jaini, E. Hoogeboom, O. Winther, M. Welling, SurVAE flows: Surjections to bridge the gap between VAEs and flows, NeurIPS, 2020.

Discrete

Discrete, Mixed

Carl

Arindam

16.

Thu, Oct 14

GAN 1: Info-GAN, f-GAN

2016 (NeurIPS): X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. NeurIPS, 2016.
2016 (NeurIPS): S. Nowozin, B. Cseke, R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. NeurIPS, 2016.

InfoGAN

f-GAN

Yuancheng

Vivek

17.

Tue, Oct 19

GAN 2: Cycle-GAN, DC-GAN

2015 (arXiv): A. Radford, L. Metz, S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
2017 (ICCV): J. Zhu, T. Park, P. Isola, A. Efros. Unpaired image-to image translation using cycle-consistent adversarial networks. ICCV, 2017.

DCGAN

Cycle-GAN

Brandon

Nengyu

18.

Thu, Oct 21

GAN 3: Wasserstein GANs

2017 (ICLR): M. Arjovsky, L. Bottou. Towards principled methods for training generative adversarial networks. In ICLR, 2017.
2017 (ICML): M. Arjovsky, S. Chintala, L. Bottou. Wasserstein GAN. In ICML, 2017.
2017 (NeurIPS): Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. C. Courville. Improved training of Wasserstein GANs. NeurIPS, 2017.

GAN Training

WGAN

Hantao

Xiaoyang

19.

Tue, Oct 26

GAN 4: Optimization

2018 (ICML): L. Mescheder, A. Geiger, S. Nowozin. Which training methods for gans do actually converge? ICML, 2018.
2018 (ICLR): A. Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard GAN. ICLR, 2018.
2020 (NeurIPS): R. Sun, T. Fang, A. Schwing. Towards a better global loss landscape of GANs. NeurIPS, 2020.

Convergence, Relativistic

Better Loss Landscapes

Minhao

Nengyu

20.

Thu, Oct 28

GAN 5: Generalization

2017 (ICML): S. Arora, R. Ge, Y. Liang, T. Ma, Y. Zhang. Generalization and equilibrium in generative adversarial nets (GANs). ICML, 2017.
2019 (NeurIPS): B. Wu, S. Zhao, C. Chen, H. Xu, L. Wang, L., X. Zhang, G. Sun, J. Zhou. Generalization in generative adversarial networks: A novel perspective from privacy protection. NeurIPS, 2019.

Generalization, Equilibrium

Privacy

Vivek

Brandon

21.

Tue, Nov 02

NODE 1: Neural Ordinary Differential Equations 1

2018 (NeurIPS): R. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud. Neural ordinary differential equations NeurIPS, 2018.
2020 (ICML): C. Finlay, J. Jacobsen, L. Nurbekyan, A. Oberman. How to train your neural ODE ICML, 2020.

Neural ODE

Regularization, Transport

Anurendra

Arindam

22.

Thu, Nov 04

NODE 2: Neural Ordinary Differential Equations 2

2019 (NeurIPS): E. Dupont, A. Doucet, Y. W. Teh. Augmented Neural ODEs NeurIPS, 2019.
2020 (ICML): H. Zhang, X. Gao, J. Unterman, T. Arodz. Approximation Capabilities of Neural ODEs and Invertible Residual Networks ICML, 2020.

Augmented Neural ODE

Universal Approximator

Bhavesh

Arindam

23.

Tue, Nov 09

NSDE 1: Stochastic Dynamics

2015 (ICML): T. Salimans, A. Diederik, D. P. Kingma, M. Welling. Markov Chain Monte Carlo and Variational Inference: Bridging the Gap ICML, 2015.
2015 (ICML): J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics ICML, 2015.

Stochastic Dynamics

Arindam

24.

Thu, Nov 11

SBM 1: Score-based Models 1

2015 (JMLR): A. Hyvarinen. Estimation of non-normalized statistical models by score matching JMLR, 2005.
2019 (NeurIPS): Y. Song, S. Ermon. Generative modeling by estimating gradients of the data distribution NeurIPS, 2019.

Score-based Models 1

Arindam

25.

Tue, Nov 16

SBM 2: Score-based Models 2

2020 (NeurIPS): J. Ho, A. Jain, P. Abbeel. Denoising diffusion probabilistic models NeurIPS, 2020.
2021 (ICLR): Y. Song, J. Sohl-Dickstein, D. Kingma, A. Kumar, S. Ermon, B. Poole. Score-based generative modeling through stochastic differential equations ICLR, 2021.

Score-based Models 2

Arindam

26.

Thu, Nov 18

DS 1: Physics+ML

2019 (JCP): M. Raissi, P. Perdikaris, G. E. Karniadakis Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations J. Comp. Phy., 2019.
2016 (PNAS): S. L. Brunton, J. L. Proctor, J. N. Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems PNAS, 2016.

PINN

Discovering Equations

Bhavesh

Anurendra

27.

Tue, Nov 30

DS 2: Learning Operators

2021 (ICLR): Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A.Anandkumar, Fourier neural operator for parametric partial differential equations ICLR, 2021.
2021 (NMI): L. Lu, R. Jin, G.Pang, Z. Zhang, G. E. Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators NMI, 2021.

Learning Operators

Arindam

28.

Thu, Dec 02

PP1: Project Presentations 1

29.

Tue, Dec 07

PP2: Project Presentations 2

Following University policy, all students are required to engage in appropriate behavior to protect the health and safety of the community. Students are also required to follow the campus COVID-19 protocols.

Students who feel ill must not come to class. In addition, students who test positive for COVID-19 or have had an exposure that requires testing and/or quarantine must not attend class. The University will provide information to the instructor, in a manner that complies with privacy laws, about students in these latter categories. These students are judged to have excused absences for the class period and should contact the instructor via email about making up the work.

Students who fail to abide by these rules will first be asked to comply; if they refuse, they will be required to leave the classroom immediately. If a student is asked to leave the classroom, the non-compliant student will be judged to have an unexcused absence and reported to the Office for Student Conflict Resolution for disciplinary action. Accumulation of non-compliance complaints against a student may result in dismissal from the University.

All students, faculty, staff, and visitors are required to wear face coverings in classrooms and university spaces. This is in accordance with CDC guidance and University policy and expected in this class.

Please refer to the University of Illinois Urbana-Champaign's COVID-19 website for further information on face coverings. Thank you for respecting all of our well-being so we can learn and interact together productively.

In order to implement COVID-19-related guidelines and policies affecting university operations, instructional faculty members may ask students in the classroom to show their Building Access Status in the Safer Illinois app or the Boarding Pass. Staff members may ask students in university offices to show their Building Access Status in the Safer Illinois app or the Boarding Pass. If the Building Access Status says “Granted,” that means the individual is compliant with the university’s COVID-19 policies—either with a university-approved COVID-19 vaccine or with the on-campus COVID-19 testing program for unvaccinated students.

Students are required to show only the Building Access Screen, which shows compliance without specifying whether it was through COVID-19 vaccination or regular on-campus testing. To protect personal health information, this screen does not say if a person is vaccinated or not. Students are not required to show anyone the screen that displays their vaccination status. No university official, including faculty members, may ask students why they are not vaccinated or any other questions seeking personal health information.

Diminished mental health, including significant stress, mood changes, excessive worry, substance/alcohol abuse, or problems with eating and/or sleeping can interfere with optimal academic performance, social development, and emotional wellbeing. The University of Illinois offers a variety of confidential services including individual and group counseling, crisis intervention, psychiatric services, and specialized screenings at no additional cost. If you or someone you know experiences any of the above mental health concerns, it is strongly encouraged to contact or visit any of the University’s resources provided below. Getting help is a smart and courageous thing to do -- for yourself and for those who care about you.

Counseling Center: 217-333-3704, 610 East John Street Champaign, IL 61820

McKinley Health Center:217-333-2700, 1109 South Lincoln Avenue, Urbana, Illinois 61801

Sexual Misconduct Reporting Obligation

The University of Illinois is committed to combating sexual misconduct. Faculty and staff members are required to report any instances of sexual misconduct to the University’s Title IX Office. In turn, an individual with the Title IX Office will provide information about rights and options, including accommodations, support services, the campus disciplinary process, and law enforcement options.

A list of the designated University employees who, as counselors, confidential advisors, and medical professionals, do not have this reporting responsibility and can maintain confidentiality, can be found here: http://wecare.illinois.edu/resources/students/#confidential

Other information about resources and reporting is available here: http://wecare.illinois.edu

Academic Integrity

The University of Illinois at Urbana-Champaign Student Code should also be considered as a part of this syllabus. Students should pay particular attention to Article 1, Part 4: Academic Integrity. Read the Code at the following URL: http://studentcode.illinois.edu/

Academic dishonesty may result in a failing grade. Every student is expected to review and abide by the Academic Integrity Policy: https://studentcode.illinois.edu/article1/part4/1-401/. Ignorance is not an excuse for any academic dishonesty. It is your responsibility to read this policy to avoid any misunderstanding. Do not hesitate to ask the instructor(s) if you are ever in doubt about what constitutes plagiarism, cheating, or any other breach of academic integrity.

Illinois law requires the University to reasonably accommodate its students' religious beliefs, observances, and practices in regard to admissions, class attendance, and the scheduling of examinations and work requirements. You should examine this syllabus at the beginning of the semester for potential conflicts between course deadlines and any of yourreligious observances. If a conflict exists, you should notify your instructor of the conflict and follow the procedure at
https://odos.illinois.edu/community-ofcare/resources/students/religious-observances/
to request appropriate accommodations. This should be done in the first two weeks of classes.

To obtain disability-related academic adjustments and/or auxiliary aids, students with disabilities must contact the course instructor and the Disability Resources and Educational Services (DRES) as soon as possible. To contact DRES, you may visit 1207 S. Oak St., Champaign, call 333-4603, email disability@illinois.edu or go to https://www.disability.illinois.edu. If you are concerned you have a disability-related condition that is impacting your academic progress, there are academic screening appointments available that can help diagnosis a previously undiagnosed disability. You may access these by visiting the DRES website and selecting “Request an Academic Screening” at the bottom of the page.