“Building OpenSource Generalist Robot Policies”
ABSTRACT
Generalist robot policies, trained on large and diverse robot datasets, have the potential to transform how robot learning research is done: in the same way that current models in NLP are almost universally derived from pretrained large language models, future robot policies might be initialized from generalist robot models and finetuned with only modest amounts of target domain data.
In this talk I will discuss our efforts on building such generalist robot policies. I will focus on two key ingredients: data and models. On the data side, I will discuss our recent works on building the largest opensource real robot manipulation datasets to date, the Open XEmbodiment dataset and DROID, with a total of 2M+ robot trajectories. On the model side, I will summarize our learnings from building RTX and Octo, the first generalist robot policies trained on the Open XEmbodiment dataset. I will discuss their current limitations and outline important steps for future research towards ubiquitous robot foundation models.
PRESENTER
Karl Pertsch is a postdoc at UC Berkeley and Stanford, jointly advised by Sergey Levine and Chelsea Finn, where he focuses on building robot foundation models. Karl obtained his PhD from USC, advised by Joseph Lim. During his PhD he interned at MetaAI and Google Brain. He is a recipient of a Fulbright Scholarship and the CoRL’20 best paper presentation award.