OASim: an Open and Adaptive Simulator based on Neural Rendering for Autonomous Driving

Shanghai AI Laboratory, Shanghai, China

Corresponding author

Abstract

With deep learning and computer vision technology development, autonomous driving provides new solutions to improve traffic safety and efficiency. The importance of building high-quality datasets is self-evident, especially with the rise of end-to-end autonomous driving algorithms in recent years. Data plays a core role in the algorithm closed-loop system. However, collecting real-world data is expensive, time-consuming, and unsafe. With the development of implicit rendering technology and in-depth research on using generative models to produce data at scale, we propose OASim, an open and adaptive simulator and autonomous driving data generator based on implicit neural rendering. It has the following characteristics: (1) High-quality scene reconstruction through neural implicit surface reconstruction technology. (2) Trajectory editing of the ego vehicle and participating vehicles. (3) Rich vehicle model library that can be freely selected and inserted into the scene. (4) Rich sensors model library where you can select specified sensors to generate data. (5) A highly customizable data generation system can generate data according to user needs. We demonstrate the high quality and fidelity of the generated data through perception performance evaluation on the Carla simulator and real-world data acquisition.

Framework

OASim focuses on generating high-fidelity and customizable autonomous driving data through neural implicit reconstruction and rendering techniques. The pipeline of OASim is shown as in the figure above. The hierarchical structure can be divided into four layers. The first data layer converts the input data into the format we require, including data cleaning and labeling. The processed sensory data and labeled HD map are then input into the back-end layer. This layer is the core of the system and implements 3D reconstruction, traffic flow simulation and novel data synthesis. The front-end layer provides an interactive interface for users to conveniently change the vehicle route and sensor configurations. The newly synthesized data can be used for multiple downstream tasks such as perception, planning, etc.

Sensor Simulator

OASim allows flexible configuration of the agent's sensor suite, including cameras, LiDAR, Radar, etc. We support modifying the sensor model by changing the intrinsic and extrinsic parameters. The position and orientation of the sensor relative to the vehicle body are represented by extrinsic parameters. Some commonly used intrinsic combinations are preset in the system to facilitate user selection. Once the sensor configuration is complete, it will be used to generate and preview data.

Results

Novel view synthesis results

Rendered images of different camera focal length

Qualitative image rendering results

Rendered point clouds of different LiDAR models

Non-rigid pedestrian rendering results

Simulation results of different traffic flow environments

BibTeX

@misc{yan2024oasim,
      title={OASim: an Open and Adaptive Simulator based on Neural Rendering for Autonomous Driving}, 
      author={Guohang Yan and Jiahao Pi and Jianfei Guo and Zhaotong Luo and Min Dou and Nianchen Deng and Qiusheng Huang and Daocheng Fu and Licheng Wen and Pinlong Cai and Xing Gao and Xinyu Cai and Bo Zhang and Xuemeng Yang and Yeqi Bai and Hongbin Zhou and Botian Shi},
      year={2024},
      eprint={2402.03830},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

@article{guo2023streetsurf,
  title = {StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Views},
  author = {Guo, Jianfei and Deng, Nianchen and Li, Xinyang and Bai, Yeqi and Shi, Botian and Wang, Chiyu and Ding, Chenjing and Wang, Dongliang and Li, Yikang},
  journal = {arXiv preprint arXiv:2306.04988},
  year = {2023}
}

@misc{wen2023limsim,
  title={LimSim: A Long-term Interactive Multi-scenario Traffic Simulator}, 
  author={Licheng Wen and Daocheng Fu and Song Mao and Pinlong Cai and Min Dou and Yikang Li and Yu Qiao},
  year={2023},
  eprint={2307.06648},
  archivePrefix={arXiv},
  primaryClass={eess.SY}
}