Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures

With the advancement of LLMs, their applications have expanded from text generation to complex problems like advanced mathematical reasoning and multimodal agents. Many tasks require spatial reasoning, whether to solve geometry problems or understand verbal commands about surroundings as an embodied agent. Existing approaches incorporate additional modalities through vision encoders, projections of point cloud data, or more specialized representations.

The GSU dataset aims to evaluate LLMs' spatial reasoning capabilities over grids, particularly in embodied settings and coordinate structures. The study finds that while most models grasp basic grid concepts, they struggle with frames of reference relative to an embodied agent and identifying 3D shapes from coordinate lists.

GSU is the first text-only grid dataset for evaluating LLMs' spatial reasoning capabilities. By removing visual inputs, the study isolates spatial reasoning from perception, focusing on models' performance in reference frames and 3D shape identification from coordinate lists.

• �� Navigation Task: Generate or follow instructions on 2D and 3D grids, testing recognition of 1D directional differences and rotating/updating spatial reference frames.
• �� Object Localization Task: Test where targets are relative to a viewer or separate spatial reference.
• �� Structure Composition Task: Translate sets of coordinates into basic shapes, testing ability to link coordinates to real-world shapes.

The study tested five broad categories of models, including small text-only models, small vision-language models, larger models, and frontier models. Experimental design included generating test and training sets for each task and using 1-shot and 3-shot prompting for evaluation.

Latest frontier models perform well on provided tasks but struggle with harder variants. Small LLMs show potential to match frontier model performance through full fine-tuning or LORA fine-tuning. Exposure to visual modality does not provide a usable understanding of 3D space.

The GSU dataset can be used to evaluate LLMs' performance in non-visual spatial tasks, particularly in embodied settings. This offers new directions for developing specialized embodied agents.

Models perform poorly in embodied reference frames, especially when rotating and updating spatial reference frames. Additionally, difficulty in identifying 3D shapes from coordinate lists. Future research could explore enhancing LLMs' performance in non-visual spatial tasks.