← Back to Library

LLM-based Realistic Safety-Critical Driving Video Generation

Authors: Yongjie Fu, Ruijian Zha, Pei Tian, Xuan Di

Published: 2025-07-02

arXiv ID: 2507.01264v1

Added to Library: 2025-07-03 04:01 UTC

Safety

📄 Abstract

Designing diverse and safety-critical driving scenarios is essential for evaluating autonomous driving systems. In this paper, we propose a novel framework that leverages Large Language Models (LLMs) for few-shot code generation to automatically synthesize driving scenarios within the CARLA simulator, which has flexibility in scenario scripting, efficient code-based control of traffic participants, and enforcement of realistic physical dynamics. Given a few example prompts and code samples, the LLM generates safety-critical scenario scripts that specify the behavior and placement of traffic participants, with a particular focus on collision events. To bridge the gap between simulation and real-world appearance, we integrate a video generation pipeline using Cosmos-Transfer1 with ControlNet, which converts rendered scenes into realistic driving videos. Our approach enables controllable scenario generation and facilitates the creation of rare but critical edge cases, such as pedestrian crossings under occlusion or sudden vehicle cut-ins. Experimental results demonstrate the effectiveness of our method in generating a wide range of realistic, diverse, and safety-critical scenarios, offering a promising tool for simulation-based testing of autonomous vehicles.

🔍 Key Points

  • Proposes a framework that integrates Large Language Models (LLMs) for few-shot code generation in synthesizing safety-critical driving scenarios within the CARLA simulator.
  • Utilizes Cosmos-Transfer1 for realistic video generation, enhancing the visual fidelity of simulated scenarios and bridging the gap between virtual simulations and real-world appearances.
  • Enables the generation of rare but critical edge case scenarios that challenge autonomous driving systems, aiding comprehensive testing and validation processes.
  • Offers a natural language interface for non-programmers to specify complex driving scenarios, thereby democratizing access to simulation tools for autonomous vehicle developers.
  • Demonstrates significant improvements in visual quality and realism in generated videos, as evidenced by comparative analyses against other state-of-the-art models.

💡 Why This Paper Matters

This paper presents a significant advancement in the field of autonomous vehicle testing by effectively combining LLMs with a novel video generation approach. The ability to create diverse and realistic safety-critical scenarios offers valuable tools for the evaluation and improvement of autonomous driving systems, which is crucial for ensuring safety in real-world applications. The findings suggest a promising direction for future research, where the framework could evolve to include varied sensor modalities and learning techniques.

🎯 Why It's Interesting for AI Security Researchers

The paper is highly relevant to AI security researchers as it addresses the generation of safety-critical scenarios that can expose vulnerabilities in autonomous driving systems. By simulating rare and extreme situations, researchers can better understand potential failure modes and enhance the robustness of AI systems against adversarial conditions. Furthermore, the incorporation of LLMs for scenario generation allows for rapid iteration and adaptation, which can be crucial in developing secure and reliable autonomous technologies.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF