Leap Icon

Continuously Learning, Adapting and Improving:
A Dual-Process Approach to Autonomous Driving LeapAD


Jianbiao Mei1,2,*;, Yukai Ma1,2,*; Xuemeng Yang2 , Licheng Wen2 , Xinyu Cai2 , Xin Li2,4 , Daocheng Fu2 , Bo Zhang2 , Pinlong Cai2 , Min Dou2 , Botian Shi2,† , Liang He3 , Yong Liu1,† , Yu Qiao2 ,
1Zhejiang University
, 2Shanghai Artificial Intelligence Laboratory
, 3East China Normal University
, 4Shanghai Jiao Tong University
*Indicates Equal Contributors †Indicates Corresponding Authors
Code arXiv

Abstract

Autonomous driving has advanced significantly due to sensors, machine learning, and artificial intelligence improvements. However, prevailing methods struggle with intricate scenarios and causal relationships, hindering adaptability and interpretability in varied environments. To address the above problems, we introduce LeapAD, a novel paradigm for autonomous driving inspired by the human cognitive process. Specifically, LeapAD emulates human attention by selecting critical objects relevant to driving decisions, simplifying environmental interpretation, and mitigating decision-making complexities. Additionally, LeapAD incorporates an innovative dual-process decision-making module, which consists of an Analytic Process (System-II) for thorough analysis and reasoning, along with a Heuristic Process (System-I) for swift and empirical processing. The Analytic Process leverages its logical reasoning to accumulate linguistic driving experience, which is then transferred to the Heuristic Process by supervised fine-tuning. Through reflection mechanisms and a growing memory bank, LeapAD continuously improves itself from past mistakes in a closed-loop environment. Closed-loop testing in CARLA shows that LeapAD outperforms all methods relying solely on camera input, requiring 1-2 orders of magnitude less labeled data. Experiments also demonstrate that as the memory bank expands, the Heuristic Process with only 1.8B parameters can inherit the knowledge from a GPT-4 powered Analytic Process and achieve continuous performance improvement.

LeapAD Architecture

In LeapAD, the Scene Understanding module mimics human driving attention, analyzing surrounding images and selectively focusing on key objects influencing decision-making. It provides detailed descriptions and behavioral reasoning of these objects, facilitating informed subsequent decisions. The Analytic Process then uses an LLM to accumulate experience in driving analysis and decision-making, conducting reflections on accidents. The experience is stored in memory and transferred to a lightweight language model, forming our Heuristic Process for quick responses and continuous learning.

The Heuristic Process in LeapAD performs closed-loop decision-making using a few-shot strategy. It is designed for enabling instant decision-making within the vehicle. The Heuristic Process relies on the knowledge transferred from the Analytic Process to make quick and efficient decisions. This lightweight language model ensures rapid responses and adaptability in various driving scenarios, maintaining a high level of performance with minimal computational resources.

The Analytic Process in LeapAD is designed for thorough analysis and reasoning. It processes more complex scenarios and builds a comprehensive memory bank of high-quality driving decisions. The Analytic Process accumulates experience through continuous learning and self-reflection, analyzing accidents, and updating the memory bank. This accumulated knowledge is then transferred to the Heuristic Process through supervised fine-tuning (SFT), ensuring the entire LeapAD continuously improves and adapts to new driving environments and challenges.

Case Studies

BibTeX

@article{mei2024continuously,
          title={Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving},
          author={Mei, Jianbiao and Ma, Yukai and Yang, Xuemeng and Wen, Licheng and Cai, Xinyu and Li, Xin and Fu, Daocheng and Zhang, Bo and Cai, Pinlong and Dou, Min and others},
          journal={arXiv preprint arXiv:2405.15324},
          year={2024}
        }