10-423 / 10-623 / 10-723Generative AI

Lecture 1: RNN LMs / Autodiff Lecture 2: Transformer LMs Lecture 3: Learning LLMs / Decoding Lecture 4: Pre-training, fine-tuning / Modern Transformers Lecture 5: Computer Vision: CNNs / Encoder-only Transformers / Vision Transformers Lecture 6: Generative Adversarial Networks (GANs) / PGM Lecture 7: Diffusion models (Part I)Lecture 8: Diffusion models (Part II)Lecture 9: Variational Autoencoders (VAEs)Lecture 10: Parameter-efficient fine tuning Lecture 11: In-Context Learning / Prompt Engineering / Instruction Fine-tuning / Reinforcement learning with human feedback (RLHF)Lecture 12: Direct Preference Optimization (DPO) / Text-to-image generation / Latent diffusion model Lecture 13: Vision-language models Lecture 14: Cross-Attention / Diffusion Transformer / Prompt-to-Prompt Lecture 15: Querying Transformer / Scaling Laws Lecture 16: Mixture of Experts Lecture 17: Distributed training Lecture 18: Flash Attention / Efficient decoding strategies Lecture 19: Long Context in LLM Lecture 20: Reasoning Models Lecture 21: State Space Models / Hybrid Models Lecture 22: Real-world Issues and Considerations / What can go wrong?Lecture 23: Code Generation / Autonomous Agents Lecture 24: Audio understanding and synthesis Lecture 25: Generative Models for Videos Lecture 26: Interactive World Models + Science of Alignment

Scaling Up › Mixture of Experts

Lecture 16: Mixture of Experts

Mon, Mar 16

Lecturer: Matt Gormley

Unit: Scaling Up

Loading PDF…

Loading PDF…