OmniTester: Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles

Abstract The generation of corner cases has become increasingly crucial for efficiently testing autonomous vehicles prior to road deployment. However, existing methods struggle to accommodate diverse testing requirements and often lack the ability to generalize to unseen situations, thereby reducing the convenience and usability of the generated scenarios. A method that facilitates easily controllable scenario generation for efficient autonomous vehicles (AV) testing with realistic and challenging situations is greatly needed. To address this, OmniTester is proposed as a multimodal Large Language Model (LLM) based framework that fully leverages the extensive world knowledge and reasoning capabilities of LLMs. OmniTester is designed to generate realistic and diverse scenarios within a simulation environment, offering a robust solution for testing and evaluating AVs. In addition to prompt engineering, OmniTester employs tools from Simulation of Urban Mobility to simplify the complexity of codes generated by LLMs. It further incorporates Retrieval-Augmented Generation and a self-improvement mechanism to enhance the LLM's understanding of scenarios, thereby increasing its ability to produce more realistic scenes. Experiment results demonstrated the controllability and realism of the proposed approaches in generating three types of challenging and complex scenarios. Additionally, OmniTester effectively reconstructs novel scenarios described in crash reports, driven by the generalization capability of LLMs.