Abstract:

Deep neural networks trained on highvariation tasks ("goals”) have had immense success as predictive models of the human and nonhuman primate visual pathways. More specifically, a positive relationship has been observed between model performance on ImageNet categorization and neural predictivity. Past a point, however, improved categorization performance on ImageNet does not yield improved neural predictivity, even between very different architectures. In this talk, I will present two case studies in both rodents and primates, that demonstrate a more general correspondence between selfsupervised learning of visual representations relevant to highdimensional embodied control and increased gains in neural predictivity.

In the first study, we develop the (currently) most precise model of the mouse visual system, and show that selfsupervised, contrastive algorithms outperform supervised approaches in capturing neural response variance across visual areas. By “implanting” these visual networks into a biomechanicallyrealistic rodent body to navigate to rewards in a novel maze environment, we observe that the artificial rodent with a contrastivelyoptimized visual system is able to obtain more reward across episodes compared to its supervised counterpart. The second case study examines mental simulations in primates, wherein we show that selfsupervised video foundation models that predict the future state of their environment in latent spaces that can support a wide range of sensorimotor tasks, align most closely with human error patterns and macaque frontal cortex neural dynamics. Taken together, our findings suggest that selfsupervised learning of visual representations that are reusable for downstream Embodied AI tasks may be a promising way forward to study the evolutionary constraints of neural circuits in multiple species.

Timestamps:

0:00 Guangyu Robert Yang Introduction
1:17 Introduction
10:54 Mouse Visual Cortex as a TaskGeneral, Limited Resource System
29:13 Reusable Latent Representations for Primate Mental Simulation
51:45 Heuristics for Interrogating Natural Intelligence

Papers Discussed:
1. A. Nayebi*, N. C. Kong*, C. Zhuang, J. L. Gardner, A. M. Norcia, & D. L. Yamins. Mouse visual cortex as a limited resource system that selflearns an ecologicallygeneral representation. PLOS Computational Biology, 19(10), e1011506. 2023. https://doi.org/10.1371/journal.pcbi....

2. A. Nayebi, R. Rajalingham, M. Jazayeri, G.R. Yang. “Neural foundations of mental simulation: future prediction of latent representations on dynamic scenes”. Advances in Neural Information Processing Systems (NeurIPS), Volume 36 (2023): 70548–70561. https://arxiv.org/abs/2305.11772