Today we’re joined by Roland Memisevic, a senior director at Qualcomm AI Research. In our conversation with Roland, we discuss the significance of language in humanlike AI systems and the advantages and limitations of autoregressive models like Transformers in building them. We cover the current and future role of recurrence in LLM reasoning and the significance of improving grounding in AI—including the potential of developing a sense of self in agents. Along the way, we discuss Fitness Ally, a fitness coach trained on a visually grounded large language model, which has served as a platform for Roland’s research into neural reasoning, as well as recent research that explores topics like visual grounding for large language models and stateaugmented architectures for AI agents.

Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confi...


CONNECT WITH US!
===============================
Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/
Join our Slack Community: https://twimlai.com/community/
Subscribe to our newsletter: https://twimlai.com/newsletter/
Want to get in touch? Send us a message: https://twimlai.com/contact/


CHAPTERS
===============================
00:00 Intro
03:26 Language as a key ingredient for humanlike AI
09:00 Fitness Alley background
14:00 Transition from RNNs to attentionbased models for better language capabilities
18:42 GPTlike models lack recurrency; Recurrence can address length generalization
26:09 Autoregressive models are crucial for building intelligent and agentic AI systems
31:15 Language is crucial in reasoning; language models lack understanding of code generation but when linked with perceptual input, can improve reasoning capabilities
39:04 Look, Remember and Reason paper
41:15 Model architecture; combining language and vision models for reasoning with a topdown approach
46:33 Combining vision and reasoning develop common sense gaps in AI
50:56 Situated chat paper
53:51 Painter paper
01:01:11 Missing pieces in AI: recurrence, memory, and a sense of self in agents


LINKS & RESOURCES
===============================
Look, Remember and Reason: Visual Reasoning with Grounded Rationales https://arxiv.org/pdf/2306.17778.pdf
Situated Realtime Interaction with a Virtually Embodied Avatar https://embodiedai.org/papers/2023/1...
Painter: Teaching Autoregressive Language Models to Draw Sketches https://arxiv.org/abs/2308.08520
Learning “Common Sense” and Physical Concepts with Roland Memisevic #111 https://twimlai.com/podcast/twimlai/l...
Pixels to Concepts with Backpropagation with Roland Memisevic #427 https://twimlai.com/podcast/twimlai/p...

For a COMPLETE LIST of links and references, head over to https://twimlai.com/go/646.


Camera: https://amzn.to/3TQ3zsg
Microphone: https://amzn.to/3t5zXeV
Lights: https://amzn.to/3TQlX49
Audio Interface: https://amzn.to/3TVFAIq
Stream Deck: https://amzn.to/3zzm7F5