We study how vision-language models (VLMs) trained on web-scale data can be integrated into end-to-end driving systems to boost generalization and enable interactivity with human users. While recent approaches adapt VLMs to driving via single-round visual question answering (VQA), human drivers reason about decisions in multiple steps. Starting from the localization of key objects, humans estimate object interactions before taking actions. The key insight is that with our proposed task, Graph VQA, where we model graph-structured reasoning through perception, prediction and planning question-answer pairs, we obtain a suitable proxy task to mimic the human reasoning process. We instantiate datasets (DriveLM-Data) built upon nuScenes and CARLA, and propose a VLM-based baseline approach (DriveLM-Agent) for jointly performing Graph VQA and end-to-end driving. The experiments demonstrate that Graph VQA provides a simple, principled framework for reasoning about a driving scene, and DriveLM-Data provides a challenging benchmark for this task. Our DriveLM-Agent baseline performs end-to-end autonomous driving competitively in comparison to state-of-the-art driving-specific architectures. Notably, its benefits are pronounced when it is evaluated zero-shot on unseen sensor configurations. Our question-wise ablation study shows that the performance gain comes from the rich annotation of prediction and planning QA pairs in the graph structure. All data, models and an official evaluation server are available at https://github.com/OpenDriveLab/DriveLM.

End-to-end driving systems have made rapid progress, but have so far not been applied to the challenging new CARLA Leaderboard 2.0. Furthermore, while there is a large body of literature on end-to-end architectures and training strategies, the impact of the training dataset is often overlooked. In this work, we make a first attempt at end-to-end driving for Leaderboard 2.0. Instead of investigating architectures, we systematically analyze the training dataset, leading to new insights: (1) Expert style significantly affects downstream policy performance. (2) On complex driving datasets, frames should not be weighted based on simplistic criteria such as class frequencies. (3) Instead, estimating whether a frame changes the target labels compared to previous frames can reduce dataset size without losing important information. By incorporating these findings, our model ranks first and second respectively on the map and sensors tracks of the 2024 CARLA Challenge.

Autonomous urban driving exposes numerous corner cases that previous planning approaches struggle to handle robustly. The CARLA Leaderboard 2.0 simulator introduces 38 complex scenarios involving high-speed highway driving, traffic violations, and dynamic obstacles, posing a demanding closed-loop benchmark for autonomous systems. We propose PDM-Lite, the first rule-based expert system to successfully navigate all scenarios in this realistic environment. Integrating components like the Intelligent Driver Model, kinematic bicycle predictions, and dynamic lane changing for obstacle handling, PDM-Lite demonstrates state-of-the-art performance through extensive evaluations. It achieves near-perfect route completion rates, low infractions per kilometer, and driving scores improving upon the previous best method Kyber-E2E by +17.05 points. PDM-Lite establishes a robust and interpretable baseline on the complex scenarios, enabling high-quality data collection for sample-efficient imitation learning. We release PDM-Lite as an open-source system to accelerate research in this critical domain.