LSTR: Lane Shape Prediction with Transformers
๐ End-to-end architecture: Directly output lane shape parameters.โก Super lightweight: The number of model parameters is only 765,787.โก Super low complexity: The number of MACs (1 MAC = 2 FLOP) is only 574.280M.๐ Training friendly: Lower GPU memory cost. Input (360, 640, 3) with batch_size 16 uses 1245MiB GPU usages.
PyTorch(1.5.0) training, evaluating and pretrained models for LSTR (Lane Shape Prediction with Transformers). We streamline the lane detection to a single-stage framework by proposing a novel lane shape model that achieves 96.18 TuSimple accuracy.
For details see End-to-end Lane Shape Prediction with Transformers by Ruijin Liu, Zejian Yuan, Tie Liu, Zhiliang Xiong.
Updates!!
-
ใ2021/12/03ใ
๐ฅ Our new work Learning to Predict 3D Lane Shape and Camera Pose from a Single Image via Geometry Constraints by Ruijin Liu, Dapeng Chen, Tie Liu, Zhiliang Xiong, Zejian Yuan is accepted by AAAI2022! The preprint paper and codes will be released soon! -
ใ2021/11/23ใ We now support Train and Test Custom Data. Tutorial: Train and Test Your Custom Data.
-
ใ2021/11/16ใ We fix the Multi-GPU Training.
-
ใ2020/12/06ใ We now support CULane Dataset.
Comming soon
- LSTR-nano(New backbone): 96.33 TuSimple accuracy with only 40% MACs (229.419M) and 40% #Params (302,546) of LSTR.
- Mosaic Augmentation.
- Loguru based logger module.
- Geometry based loss functions.
- Segmentation prior.
Model Zoo
We provide the baseline LSTR model file (trained on TuSimple train and val sets after 500000 iterations) in the ./cache/nnet/LSTR/LSTR_500000.pkl (~3.1MB).
Data Preparation
Download and extract TuSimple train, val and test with annotations from TuSimple. We expect the directory structure to be the following:
TuSimple/
LaneDetection/
clips/
label_data_0313.json
label_data_0531.json
label_data_0601.json
test_label.json
LSTR/
Set Envirionment
- Linux ubuntu 16.04
conda env create --name lstr --file environment.txt
After you create the environment, activate it
conda activate lstr
Then
pip install -r requirements.txt
Training and Evaluation
To train a model:
(if you only want to use the train set, please see ./config/LSTR.json and set "train_split": "train")
python train.py LSTR
- Visualized images are in ./results during training.
- Saved model files (every 5000 iterations) are in ./cache during training.
To train a model from a snapshot model file:
python train.py LSTR --iter 500000
To evaluate (GPU 603MiB usage when evaluating single image iteratively), then you will see the paper's result:
python test.py LSTR --testiter 500000 --modality eval --split testing
To evaluate FPS (set --batch to maximum the FPS, GPU 877MiB usage if you repeat each image 16 times):
python test.py LSTR --testiter 500000 --modality eval --split testing --batch 16
To evaluate and save detected images in ./results/LSTR/500000/testing/lane_debug:
python test.py LSTR --testiter 500000 --modality eval --split testing --debug
- Demo (displayed parameters are rounded to three significant figures.)
To evaluate and save decoder attention maps (store --debugEnc to visualize encoder attention maps):
python test.py LSTR --testiter 500000 --modality eval --split testing --debug --debugDec
To evaluate on a set of images (store your images in ./images, then the detected results will be saved in ./detections):
python test.py LSTR --testiter 500000 --modality images --image_root ./ --debug
Citation
@InProceedings{LSTR,
author = {Ruijin Liu and Zejian Yuan and Tie Liu and Zhiliang Xiong},
title = {End-to-end Lane Shape Prediction with Transformers},
booktitle = {WACV},
year = {2021}
}
License
LSTR is released under BSD 3-Clause License. Please see LICENSE file for more information.
Contributing
We actively welcome your pull requests!