Generative models of text › Pre-training, fine-tuning / Modern Transformers
Lecture 4: Pre-training, fine-tuning / Modern Transformers
Wed, Sep 2
Readings
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Ainslie et al. (2023). EMNLP.
- Longformer: The Long-Document Transformer. Beltagy et al. (2020).
- RoFormer: Enhanced Transformer with Rotary Position Embedding. Su et al. (2021).
Unit: Generative models of text
PDF not available
PDF not available