Awesome Self-Driving Cars

Introduction

A curated list of all awesome things related to self-driving car.

Table of Contents

• Opensource code
  • Fullstack
  • Library
• Courses
• Papers & Blogs
  • Overview
  • HD Map
  • Simulation & Data generation
  • Localization
  • Calibration
  • Perception
  • Prediction
  • Planning
  • Control
  • End-to-End
  • Misc
• Datasets and Benchmarks
• Algorithms
  • Overview
  • HD Map
  • Simulation & Data generation
  • Localization
  • Calibration
  • Perception
  • Prediction
  • Planning
  • Control
  • End-to-End
  • Misc
• Systems
  • RTOS
• Cloud service
  • Simulation Service
  • HD Map Service
  • Data Service
  • Monitor Service
  • OTA
• Hardware
  • Computing Unit
  • sensors
    • GPS/IMU
    • Camera
    • LiDAR
    • RADAR
    • Ultrasonic Sensor
  • CAN card
  • Drive by wire
  • V2X
  • HMI Device
  • Black Box
• Big Players

Autonomous driving technology stack

First, let's look at the technology stack for autonomous driving. In order to understand the full stack of autonomous driving. After that, you can learn the corresponding skill tree.

Opensource

Fullstack

• apollo - Apollo is an open source autopilot platform that contains almost everything. Including hardware, systems, vehicle platforms, cloud services, etc. You can quickly build a self-driving system of your own by Apollo.
• autoware - The original Autoware project built on ROS 1. Launched as a research and development platform for autonomous driving technology.
• openpilot - Openpilot is an open source driver agent. Use the iphone to control the car, which provides adaptive cruise control (ACC) and lane keeping assist (LKAS).

Library

• ROS - The Robot Operating System (ROS) is a flexible framework for writing robot software. It is a collection of tools, libraries, and conventions that aim to simplify the task of creating complex and robust robot behavior across a wide variety of robotic platforms.
• OpenCV library - OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products.
• Point Cloud Library - Point Cloud Library (PCL) is a standalone, large open project for 2D / 3D imagery and point cloud processing. Widely used to process laser point cloud data.
• TensorFlow - TensorFlow is an open source software library for numerical computation using data flow graphs. Used for automatic driving perception and prediction.
• ompl - The Open Motion Planning Library.

Courses

• Udacity Self-Driving Car Nanodegree - Udacity's flagship program is sponsored by many self-driving car hiring partners. The nanodegree program includes 3 terms: 1) Term 1: Computer Vision and Deep Learning, 2) Term 2: Sensor Fusion, Localization, and Control, and 3) Term 3: Path Planning, Elective, and Systems. Each term costs $800.
• MIT 6.S094: Deep Learning for Self-Driving Cars - This class is an introduction to the practice of deep learning through the applied theme of building a self-driving car. It is open to beginners and is designed for those who are new to machine learning, but it can also benefit advanced researchers in the field looking for a practical overview of deep learning methods and their application. By the way, it's free!
• BitTiger Build Your Own Autonomous Vehicle Mastery Program - Two weeks of live classes in Bay Area taught by engineers from Vector.ai on building a self-driving mini car from ground up. Topics include deep learning, ROS, sensors, computer vision, localization, mapping and control. The program costs $7,000.
• Apollo Autopilot Introduction - Note: it's a Chinese tutorial. This is an open class between Baidu and Peking University. It is very comprehensive and detailed, which is a good introductory course.

Papers & Blogs

Overview

• A Survey of Autonomous Driving: Common Practices and Emerging Technologies - 2020
• A Survey of Deep Learning Techniques for Autonomous Driving - 2020
• Self-Driving Cars: A Survey - 2019
• Towards Fully Autonomous Driving: Systems and Algorithms - 2011

HD Map

• Highly Efficient HD Map Creation: Accelerating Mapping Process with GPUs - An introduction PPT (2017)

Simulation & Data generation

Localization

• A survey of the state-of-the-art localization techniques and their potentials for autonomous vehicle applications - JIOT 2017
• Robust and Precise Vehicle Localization based on Multi-sensor Fusion in Diverse City Scenes - ICRA 2018
• Map-Based Precision Vehicle Localization in Urban Environments -
• Robust Vehicle Localization in Urban Environments Using Probabilistic Maps -

SLAM

• Simultaneous localization and mapping: A survey of current trends in autonomous driving - 2017
• Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age - 2016
• A critique of current developments in simultaneous localization and mapping - 2016

Perception

Survey

• Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art - CVPR 2017

Object Detection

• Object Detection in 20 Years: A Survey - CVPR 2019
• Object Detection With Deep Learning: A Review - CVPR 2018
• Deep Learning for Generic Object Detection: A Survey - CVPR 2018
• 50 Years of object recognition: Directions forward - 2013

3D Object Detection

• 3D Object Detection for Autonomous Driving: A Survey - 2022

• 3D Object Detection for Autonomous Driving: A Comprehensive Survey - 2022

• A survey on 3d object detection methods for autonomous driving applications - 2019

• Deep Learning-based Image 3D Object Detection for Autonomous Driving - 2023

• 3D Object Detection from Images for Autonomous Driving: A Survey - 2022

• A survey on deep learning based methods and datasets for monocular 3D object detection - 2021

• Monocular 3d object detection for autonomous driving - 2016

• A survey of robust 3d object detection methods in point clouds - 2022

• Point-cloud based 3D object detection and classification methods for self-driving applications: A survey and taxonomy - 2021

• A comprehensive survey of LIDAR-based 3D object detection methods with deep learning for autonomous driving -2021

• Deep 3D object detection networks using LiDAR data: A review - 2020

• Deep learning for 3d point clouds: A survey - 2020

• Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges - 2020

• Multi-modal 3d object detection in autonomous driving: a survey - 2021

• 3D object recognition and classification: a systematic literature review - 2019

• A review and comparative study on probabilistic object detection in autonomous driving - 2021

Object Tracking

• Multiple Object Tracking: A Literature Review - CVPR 2014
• Deep Learning in Video Multi-Object Tracking: A Survey - Neurocomputing 2019
• Deep Learning for Multi-Object Tracking: A Survey - 2019

Lane Detection

• Recent progress in road and lane detection: a survey - 2014

Data Fusion

• Multisensor data fusion: A review of the state-of-the-art
• A Review of Data Fusion Techniques
• A Comprehensive Review of The Multi-sensor Data Fusion Architectures
• A Survey of Multisensor Fusion Techniques, Architectures and Methodologies

Prediction

• A Review of Tracking, Prediction and Decision Making Methods for Autonomous Driving - LG 2019

Planning

Survey

• Annual Review of Control, Robotics, and Autonomous Systems - 2018
• A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles - Robotics 2016
• A Review of Motion Planning Techniques for Automated Vehicles - 2016
• Real-time motion planning methods for autonomous on-road driving: State-of-the-art and future research directions - 2015

Other

• ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst - Waymo's paper about how to train a policy for autonomous driving via imitation learning that is robust enough to drive a real vehicle.
• Baidu Apollo EM Motion Planner - A real-time motion planning system based on the Baidu Apollo (open source) autonomous driving platform.

Control

• A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles - Robotics 2016

End-to-End

• End to End Learning for Self-Driving Cars - 2016 NVIDIA

Misc

• An Introduction to LIDAR - Awesome introduction by Voyage about the key sensor of self-driving cars.
• Learning a Driving Simulator - comma.ai's approach for self-driving cars is based on an agent that learns to clone driver behaviors and plans maneuvers by simulating future events in the road. This paper investigates variational autoencoders with classical and learned cost functions using generative adversarial networks for embedding road frames. A transition model is learned in the embedded space using action conditioned Recurrent Neural Networks (RNNs).
• 16 Questions About Self-Driving Cars - a16z's Frank Chen goes over the 16 most commonly asked questions, technical and non-technical, about self-driving cars.
• Ways to think about cars - Awesome blog post by a16z's Benedict Evans on electric cars, on-demand car services, and self-driving cars.
• The Third Transportation Revolution - Awesome blog post by John Zimmer on Lyft's vision for self-driving cars. Spoiler alert, John predicts self-driving cars will account for the majority of Lyft rides within 5 years. And by 2025, private car ownership will all-but end in major U.S. cities.
• Cars and second order consequences - Benedict Evans on the impact of electric and autonomy on cars and beyond.

Datasets and Benchmarks

• KITTI Vision Benchmark Suite [Images] - Large vision benchmark dataset with objection detection evaluation training/testing images and leaderboard on cars and pedestrians.
• Cityscapes - Semantic, instance-wise, dense pixel annotations of 30 classes.
• comma.ai's Driving Dataset [Videos] - Seven and a quarter hours (~ 80 GB) of largely highway driving. With this dataset, comma.ai's founder George Hotz trained a self-driving car all by himself.
• Udacity's Driving Dataset [Videos] - Eight hours (over 280 GB) of driving data collected for their open source self-driving car challenges. Udacity also provides convenient scripts to port the data.
• Washington DC's Lidar Data - Lidar point cloud of the entire Washington DC area is made available by the District of Columbia’s Office of the Chief Technology Officer (OCTO).
• Apolloscape - Apolloscape provides images with 10x higher resolution and pixel-level annotation. And also Provides multiple levels of scene complexity.
• nuScenes - The nuScenes dataset (pronounced /nuːsiːnz/) is a public large-scale dataset for autonomous driving provided by nuTonomy-Aptiv.
• Waymo Open Dataset - The Waymo Open Dataset is comprised of high-resolution sensor data collected by Waymo self-driving cars in a wide variety of conditions.

Traffic Sign

• STSD - More than 20 000 images with 20% labeled, Contains 3488 traffic signs.
• LISA - 7855 annotations on 6610 frames.
• Tsinghua-Tencent 100K - 100000 images containing 30000 traffic-sign instances.
• German Traffic Sign [Images] - More than 50,000 images and 40 classes of traffic signs.
• Swedish Traffic Sign - A dataset with traffic signs recorded on 350 km of Swedish roads, consisting of 20 000 images with 20% of annotations.

Algorithms

Systems

RTOS

Cloud service

Simulation Service

• Udacity's Self-Driving Car Simulator - This simulator is built for Udacity's Self-Driving Car Nanodegree to teach students how to train cars how to navigate road courses using deep learning. It is used for the project of Behavioral Cloning.
• Microsoft's AirSim - An open-source and cross platform simulator built for drones and other vehicles. AirSim is designed as a platform for AI research to experiment with deep learning, computer vision and reinforcement learning algorithms for autonomous vehicles.
• MIT's Moral Machine - Moral machine provides a "platform for 1) building a crowd-sourced picture of human opinion on how machines should make decisions when faced with moral dilemmas, and 2) crowd-sourcing assembly and discussion of potential scenarios of moral consequence". If you are a fan of the trolley problem, you can't miss this.
• MIT's Google Self-Driving Car Simulator - Self-driving car simulated completely by visual programming language Scratch.
• Carla - CARLA has been developed from the ground up to support development, training, and validation of autonomous driving systems.
• Lgsvl - The LGSVL Simulator is a simulator that facilitates testing and development of autonomous driving software systems. The LGSVL simulator is developed by the Advanced Platform Lab at the LG Electronics America R&D Center, formerly the LG Silicon Valley Lab.

HD Map Service

Data Service

Monitor Service

OTA

Hardware

Computing Unit

Sensors

GPS/IMU

Camera

LiDAR

RADAR

Ultrasonic Sensor

CAN card

Drive by wire

V2X

HMI Device

Black Box

Big Players

If I have seen further it is by standing on ye sholders of Giants.
                                                                - Isaac Newton