April 11, 2024
Speaker: Hyung Won Chung, OpenAI

Shaping the Future of AI from the History of Transformer

AI is developing at such an overwhelming pace that it is hard to keep up. Instead of spending all our energy catching up with the latest development, I argue that we should study the change itself. First step is to identify and understand the driving force behind the change. For AI, it is the exponentially cheaper compute and associated scaling. I will provide a highlyopinionated view on the early history of Transformer architectures, focusing on what motivated each development and how each became less relevant with more compute. This analysis will help us connect the past and present in a unified perspective, which in turn makes it more manageable to project where the field is heading. Slides here: https://docs.google.com/presentation/...

0:00 Introduction
2:05 Identifying and understanding the dominant driving force behind AI.
15:18 Overview of Transformer architectures: encoderdecoder, encoderonly and decoderonly
23:29 Differences between encoderdecoder and decoderonly, and rationale for encoderdecoder’s additional structures from the perspective of scaling.

About the speaker:
Hyung Won Chung is a research scientist at OpenAI ChatGPT team. He has worked on various aspects of Large Language Models: pretraining, instruction finetuning, reinforcement learning with human feedback, reasoning, multilinguality, parallelism strategies, etc. Some of the notable work includes scaling Flan paper (FlanT5, FlanPaLM) and T5X, the training framework used to train the PaLM language model. Before OpenAI, he was at Google Brain and before that he received a PhD from MIT.

More about the course can be found here: https://web.stanford.edu/class/cs25/

View the entire CS25 Transformers United playlist:    • Stanford CS25  Transformers United