StreetSenser: a novel approach to sensing street view via a fine-tuned multimodal large language model

Street view imagery (SVI) can capture urban physical space features, residents’ sentiments, and soundscape characteristics, providing insights into complex relationships between human activities and the built environment. However, existing SVI-based urban perception models have limitations, including insufficient model generality, neglect of element interactions, and lack of logical reasoning capabilities. The study proposes a multimodal large language model, StreetSenser, with powerful SVI analysis and general capabilities learned from human perception. First, an optimized dataset was designed and constructed, encompassing diverse street scenarios and annotations from human perception. Subsequently, StreetSenser was developed through a two-stage Chain-of-Perception (CoP) fine-tuning method, with the ‘Object - Attribute - Relation’ framework applied to the Qwen2.5-VL model to enable perception of complex information across visual, emotional and acoustic dimensions. Results revealed that modules such as SVI description and CoP enhance urban perception capacity of StreetSenser significantly. Furthermore, with only seven billion parameters, StreetSenser demonstrates a performance comparable to or exceeding that of large models such as GPT-4o across various tasks, while also approaching or surpassing traditional models. In addition, the compact model size enables efficient and cost-effective deployment for large-scale urban applications. The results validate the potential of StreetSenser as a key decision-support tool for urban planning.