HybridNets: End2End Perception Network

HybridNets: End-to-End Perception Network

by Dat Vu, Bao Ngo, Hung Phan ^📧 FPT University

(^📧) corresponding author.

arXiv technical report (arXiv 2203.09035)

About The Project

HybridNets is an end2end perception network for multi-tasks. Our work focused on traffic object detection, drivable area segmentation and lane detection. HybridNets can run real-time on embedded systems, and obtains SOTA Object Detection, Lane Detection on BDD100K Dataset.

Project Structure

HybridNets
│   backbone.py                   # Model configuration
|   export.py                     # UPDATED 10/2022: onnx weight with accompanying .npy anchors
│   hubconf.py                    # Pytorch Hub entrypoint
│   hybridnets_test.py            # Image inference
│   hybridnets_test_videos.py     # Video inference
│   train.py                      # Train script
│   train_ddp.py                  # DistributedDataParallel training (Multi GPUs)
│   val.py                        # Validate script
│   val_ddp.py                    # DistributedDataParralel validating (Multi GPUs)
│
├───encoders                      # https://github.com/qubvel/segmentation_models.pytorch/tree/master/segmentation_models_pytorch/encoders
│       ...
│
├───hybridnets
│       autoanchor.py             # Generate new anchors by k-means
│       dataset.py                # BDD100K dataset
│       loss.py                   # Focal, tversky (dice)
│       model.py                  # Model blocks
│
├───projects
│       bdd100k.yml               # Project configuration
│
├───ros                           # C++ ROS Package for path planning
│       ...
│
└───utils
    |   constants.py
    │   plot.py                   # Draw bounding box
    │   smp_metrics.py            # https://github.com/qubvel/segmentation_models.pytorch/blob/master/segmentation_models_pytorch/metrics/functional.py
    │   utils.py                  # Various helper functions (preprocess, postprocess, eval...)

Getting Started

Installation

The project was developed with Python>=3.7 and Pytorch>=1.10.

git clone https://github.com/datvuthanh/HybridNets
cd HybridNets
pip install -r requirements.txt

Demo

# Download end-to-end weights
curl --create-dirs -L -o weights/hybridnets.pth https://github.com/datvuthanh/HybridNets/releases/download/v1.0/hybridnets.pth

# Image inference
python hybridnets_test.py -w weights/hybridnets.pth --source demo/image --output demo_result --imshow False --imwrite True

# Video inference
python hybridnets_test_videos.py -w weights/hybridnets.pth --source demo/video --output demo_result

# Result is saved in a new folder called demo_result

Usage

Data Preparation

Recommended dataset structure:

HybridNets
└───datasets
    ├───imgs
    │   ├───train
    │   └───val
    ├───det_annot
    │   ├───train
    │   └───val
    ├───da_seg_annot
    │   ├───train
    │   └───val
    └───ll_seg_annot
        ├───train
        └───val

Update your dataset paths in projects/your_project_name.yml.

For BDD100K:

• imgs
• det_annot
• da_seg_annot
• ll_seg_annot

Training

1) Edit or create a new project configuration, using bdd100k.yml as a template. Augmentation params are here.

# mean and std of dataset in RGB order
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]

# bdd100k anchors
anchors_scales: '[2**0, 2**0.70, 2**1.32]'
anchors_ratios: '[(0.62, 1.58), (1.0, 1.0), (1.58, 0.62)]'

# BDD100K officially supports 10 classes
# obj_list: ['person', 'rider', 'car', 'truck', 'bus', 'train', 'motorcycle', 'bicycle', 'traffic light', 'traffic sign']
obj_list: ['car']
obj_combine: ['car', 'bus', 'truck', 'train']  # if single class, combine these classes into 1 single class in obj_list
                                               # leave as empty list ([]) to not combine classes

seg_list: ['road',
          'lane']
seg_multilabel: false  # a pixel can belong to multiple labels (i.e. lane line + underlying road)

dataset:
  dataroot: path/to/imgs
  labelroot: path/to/det_annot
  segroot:
  # must be in correct order with seg_list
  - path/to/da_seg_annot
  - path/to/ll_seg_annot
  fliplr: 0.5
  flipud: 0.0
  hsv_h: 0.015
  hsv_s: 0.7
  hsv_v: 0.4
...

2) Train

python train.py -p bdd100k        # your_project_name
                -c 3              # coefficient of effnet backbone, result from paper is 3
            OR  -bb repvgg_b0     # change your backbone with timm
                -n 4              # num_workers
                -b 8              # batch_size per gpu
                -w path/to/weight # use 'last' to resume training from previous session
                --freeze_det      # freeze detection head, others: --freeze_backbone, --freeze_seg
                --lr 1e-5         # learning rate
                --optim adamw     # adamw | sgd
                --num_epochs 200

Please check python train.py --help for cheat codes.

IMPORTANT (deprecated): If you want to train on multiple gpus, use train_ddp.py. Tested on NVIDIA DGX with 8xA100 40GB.
Why didn't we combine DDP into the already existing train.py script?

1. Lots of if-else.
2. Don't want to break functioning stuffs.
3. Lazy.

Update 24/06/2022: train_ddp.py broke because we have a lot of things changed. Therefore, we decided to write a merged train.py with DDP support for easier maintainance. In the meantime, please clone this commit with a working train_ddp.py script if you really have to.

3) Evaluate

python val.py -w checkpoints/weight.pth

Again, check python val.py --help for god mode.

Validation process got killed! What do I do? => This is because we use a default confidence threshold of 0.001 to compare with other networks. So when calculating metrics, it has to handle a large amount of bounding boxes, leading to out-of-memory, and finally exploding the program before the next epoch.

That being said, there are multiple ways to circumvent this problem, choose the best that suit you:

• Train on a high-RAM instance (RAM as in main memory, not VRAM in GPU). For your reference, we can only val the combined car class with 64GB RAM.
• Train with python train.py --cal_map False to not calculate metrics when validating. This option will only print validation losses. When the losses seem to flatten and the weather is nice, rent a high-RAM instance to validate the best weight with python val.py -w checkpoints/xxx_best.pth. We actually did this to save on cost.
• Reduce the confidence threshold with python train.py --conf_thres 0.5 or python val.py --conf_thres 0.5, depending on your application and end goals. You don't have to get best recall unless you're either helping us by experimenting 🥰 or competing with us 😠.

4) Export

python export.py -w checkpoints/weight.pth --width 640 --height 384

This automatically creates an ONNX weight and an anchor_{H}_{W}.npy file to use in postprocessing. Refer to the ROS section for usage example.

Training Tips

Anchors ⚓

If your dataset is intrinsically different from COCO or BDD100K, or the metrics of detection after training are not as high as expected, you could try enabling autoanchor in project.yml:

...
model:
  image_size:
  - 640
  - 384
need_autoanchor: true  # set to true to run autoanchor
pin_memory: false
...

This automatically finds the best combination of anchor scales and anchor ratios for your dataset. Then you can manually edit them project.yml and disable autoanchor.

If you're feeling lucky, maybe mess around with base_anchor_scale in backbone.py:

class HybridNetsBackbone(nn.Module):
  ...
  self.pyramid_levels = [5, 5, 5, 5, 5, 5, 5, 5, 6]
  self.anchor_scale = [1.25,1.25,1.25,1.25,1.25,1.25,1.25,1.25,1.25,]
  self.aspect_ratios = kwargs.get('ratios', [(1.0, 1.0), (1.4, 0.7), (0.7, 1.4)])
  ...

and model.py:

class Anchors(nn.Module):
  ...
  for scale, ratio in itertools.product(self.scales, self.ratios):
    base_anchor_size = self.anchor_scale * stride * scale
    anchor_size_x_2 = base_anchor_size * ratio[0] / 2.0
    anchor_size_y_2 = base_anchor_size * ratio[1] / 2.0
  ...

to get a grasp on how anchor boxes work.

And because a picture is worth a thousand words, you can visualize your anchor boxes in Anchor Computation Tool.

Training stages

We experimented with training stages and found that this settings achieved the best results:

1. --freeze_seg True ~ 200 epochs
2. --freeze_backbone True --freeze_det True ~ 50 epochs
3. Train end-to-end ~ 50 epochs

The reason being detection head is harder to converge early on, so we basically skipped segmentation head to focus on detection first.

Results

Traffic Object Detection

Drivable Area Segmentation

Lane Line Detection

License

Distributed under the MIT License. See LICENSE for more information.

Acknowledgements

Our work would not be complete without the wonderful work of the following authors:

• EfficientDet
• YOLOv5
• YOLOP
• KMeans Anchors Ratios
• Anchor Computation Tool

Citation

If you find our paper and code useful for your research, please consider giving a star ⭐ and citation 📝 :

@misc{vu2022hybridnets,
      title={HybridNets: End-to-End Perception Network}, 
      author={Dat Vu and Bao Ngo and Hung Phan},
      year={2022},
      eprint={2203.09035},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}