Elective course I took in Q4 of my first year while at Twente. It’s basically a in-depth continuation of the GenAI course. A foundation model is anything that gives you meaningful re-usable features. Foundation Models are trained on massive datasets and perform well in zero-shot (no fine-tuning) scenarios (so like ChatGPT, Gemini, all these LLMs).
Topics covered
- Introduction to Foundation Models
- Transformers, Introduction to Transformers in Deep Learning
- Transformers in depth
- Efficient FoMos, Mixture of Experts (MOE), Flash Attention
- CLIP, DINO
- Self-Supervision Objectives for FoMo Pre-training
- Training Foundation Models
- Improving the Generalization of ViTs for Action Understanding with VLM Pre-Training
- FoMo Post-training and Adaption
- Beyond attention-based methods — Structured State Space Sequence Models (SSM)
