3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

In recent years, multimodal large language models (MLLMs) have rapidly transformed the field of artificial intelligence, demonstrating unprecedented capabilities in reasoning, generation, and multi-modality integration. Existing models such as ChatGPT-5, Qwen3, and LLaVA-Plus have shown that language-centric architectures can be adapted for cross-modality understanding. However, these models excel primarily in small-scale or object-centric scenarios, and their potential in 3D city-scale environments remains largely unexplored. Diverse city environments introduce a new level of complexity for multi-modality perception and understanding. Unlike indoor benchmarks that involve a limited number of objects, a city scene usually contains thousands of entities with heterogeneous attributes and intricate spatial relationships.

The core problem of multimodal perception and understanding in 3D city-scale scenes involves numerous heterogeneous objects and their complex spatial relationships. Existing multimodal large language models often lack comprehensive understanding of inter-object relationships and global scenes when handling such large-scale environments. Answering queries like 'Which hospital is closest to the railway station? And where is its emergency department located?' requires understanding object categories, precise spatial coordinates, relational proximity, and city scene layout. These tasks highlight the need for a unified framework that can simultaneously perform 3D object perception, relationship calculation, and holistic scene understanding.

3DCity-LLM's core innovations include its coarse-to-fine feature encoding strategy and large-scale high-quality dataset. First, the model employs three parallel branches to achieve feature encoding for target objects, inter-object relationships, and global scenes, addressing limitations of existing models in handling complex urban environments. Second, the introduction of the 3DCity-LLM-1.2M dataset provides rich training resources, covering seven task categories from fine-grained object analysis to multi-faceted scene planning. Additionally, the study proposes a multi-dimensional evaluation protocol, combining traditional text-similarity metrics with LLM-based semantic assessments, ensuring comprehensive evaluations of open-ended city-scale tasks.

• �� 3DCity-LLM employs a coarse-to-fine feature encoding strategy, comprising three parallel branches for target object, inter-object relationship, and global scene.

• �� It is trained on the 3DCity-LLM-1.2M dataset, which includes approximately 1.2 million high-quality samples across seven task categories, from fine-grained object analysis to multi-faceted scene planning.

• �� A multi-dimensional protocol based on text-similarity metrics and LLM-based semantic assessment ensures accurate evaluations.

• �� Through task-driven instruction tuning, 3DCity-LLM can handle diverse tasks ranging from fine-grained object analysis to complex scene analysis and goal-oriented planning.

In the experimental design, two benchmarks were used to validate the performance of 3DCity-LLM. By comparing with existing state-of-the-art methods, the study demonstrates significant improvements in BLEU-4, METEOR, and reliability metrics. The experiments also include ablation studies to confirm the effectiveness of the coarse-to-fine feature encoding strategy, particularly in enhancing spatial reasoning capabilities when handling large-scale urban scenes.

The experimental results show that 3DCity-LLM significantly outperforms existing state-of-the-art methods in two benchmarks, with improvements ranging from 0.50 to 8.40 in BLEU-4, 1.07 to 10.69 in METEOR, and 0.16 to 1.51 in reliability. Particularly in complex tasks such as object analysis, relationship computation, and scene planning, 3DCity-LLM excels, demonstrating robust perception and understanding capabilities in complex urban environments.

Application scenarios for 3DCity-LLM include urban planning, intelligent transportation, and urban safety. By providing comprehensive understanding of city-scale scenes, the model can support decision-making in urban planning, optimize traffic flow, and enhance the efficiency of urban safety monitoring. In intelligent transportation, 3DCity-LLM can be used for real-time traffic flow analysis and route planning, improving traffic efficiency.

Despite 3DCity-LLM's excellent performance in multiple tasks, it may face computational resource constraints when processing real-time city scenes, particularly in handling large-scale urban data in real-time. Additionally, in extremely complex urban environments, the model may overlook certain details, leading to incomplete understanding. Future research could focus on enhancing the model's real-time processing capabilities and expanding the dataset to cover more diverse urban scenarios.