Sign Language: Towards Sign Understanding for Robot Autonomy

Speaker: Dr. Nicky Zimmerman, National University of Singapore

Abstract: Navigational signs are common aids for human wayfinding and scene understanding, but are underutilized by robots. We argue that they benefit robot navigation and scene understanding, by directly encoding privileged information on actions, spatial regions, and relations. Interpreting signs in open-world settings remains a challenge owing to the complexity of scenes and signs, but recent advances in vision-language models (VLMs) make this feasible. To advance progress in this area, we introduce the task of navigational sign understanding which parses locations and associated directions from signs. Additionally, we present a baseline approach using VLMs, and demonstrate their promise on navigational sign understanding. We also outline different applications, such as localization and navigation, which benefit from the spatial-symbolic information encoded by navigational signs.

Biography: Nicky is a Postdoctoral Research Fellow in National University of Singapore working with Prof. David Hsu. Her research focuses on open world navigation. She graduated as a PhD in Computer Science from Università della Svizzera italiana in 2024, where she also completed her MSc. Previously, she worked for General Motors and Intel.

Host: Prof. Kai