Spatial LLM Agents for Zero-Shot Visual Positioning

Traditional robotic navigation relies on dense metric maps, pre-compiled navigation meshes, or classical graph-based Visual Positioning Systems (VPS).
Open-vocabulary 3D scene graphs have recently emerged, allowing robots to query environments semantically.
Pushing massive, unstructured point cloud data directly into multimodal Large Language Models to perform abstract spatial reasoning and zero-shot routing — without traditional pathfinding algorithms — remains largely unexplored.
This thesis bypasses navigation meshes by feeding point cloud semantics and spatial coordinates directly into multimodal LLMs (e.g., GPT-4V, LLaVA-3D).
The goal is an agent that interprets highly abstract human queries, grounds them in the provided spatial context, and outputs explicit, navigable routing paths across unstructured environments.

Spatial representation: Formulate an efficient tokenization or embedding strategy to represent large-scale 3D spatial data within LLM context windows.
Zero-shot VPS: Develop LangChain-based prompting mechanisms allowing the agent to deduce its location and plan a path from surrounding semantics.
Benchmarking: Evaluate the agent’s point-to-point routing accuracy against classical metric-semantic baselines on open datasets.

Literature research on 3D-LLMs, scene graphs, and vision-and-language navigation
Design and implementation of the spatial tokenization / scene abstraction module
Implementation of the agentic routing pipeline
Quantitative benchmarking against classical navigation baselines

Hong, Y., et al. (2023). “3D-LLM: Injecting the 3D World into Large Language Models.” NeurIPS.
Gu, J., et al. (2024). “ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning.” ICRA.
Zhou, G., et al. (2023). “NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models.” arXiv.

Ashwin Nedungadi