We will show you how to build intuition around training performance with a focus on GPUpoor fine tuning.

Part 2 of this talk:    • Napkin Math For Fine Tuning Pt. 2 w/J...  

More resources available here:
https://parlancelabs.com/education/f...


00:00 Introduction
Johno introduces the topic "Napkin Math for Fine Tuning," aiming to answer common questions related to model training, especially for beginners in finetuning large existing models.

01:23 About Johno and AnswerAI
Johno shares his background and his work at AnswerAI, an applied R&D lab focusing on the societal benefits of AI.

03:18 Plan for the Talk
Johno outlines the structure of the talk, including objectives, running experiments, and live napkin math to estimate memory use.

04:40 Training and FineTuning Loop
Description of the training loop: feeding data through a model, measuring accuracy, updating the model, and repeating the process.

09:05 Hardware Considerations
Discussion on the different hardware components (CPU, GPU, RAM) and how they affect training performance.

12:28 Tricks for Efficient Training
Overview of various techniques to optimize training efficiency, including LoRa, quantization, and CPU offloading.

13:12 Full FineTuning
Describes the parameters and memory involved with full finetuning

18:14 LoRA
Detailed explanation of full finetuning versus parameterefficient finetuning techniques like LoRa.

21:04 Quantization and Memory Savings
Discussion on quantization methods to reduce memory usage and enable training of larger models.

23:10 Combining Techniques
Combining different techniques like quantization and LoRa to maximize training efficiency.

22:55 Running Experiments
Importance of running controlled experiments to understand the impact of various training parameters.

25:46 CPU Offloading
How CPU offloading works and the tradeoffs.

28:31 RealWorld Example
Demo of memory optimization and problemsolving during model training, with code. This also includes pragmatic ways to profile your code.

45:44 Case Study: QLoRA + FSDP
Discussion of QLorA with FSDP, along with a discussion of tradeoffs.

54:25 Recap / Conclusion
Johno summarizes the key points of his talk.