You Only Look Once for Panopitic Driving Perception.(MIR2022)
Alternatives To Yolop
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
4 days ago48apache-2.0Python
OpenPCDet Toolbox for LiDAR-based 3D Object Detection.
a day ago89apache-2.0Python
[ECCV 2022] This is the official implementation of BEVFormer, a camera-only framework for autonomous driving perception, e.g., 3D object detection and semantic map segmentation.
3d Pointcloud1,551
10 hours ago2Python
Papers and Datasets about Point Cloud.
2 months ago121mitPython
You Only Look Once for Panopitic Driving Perception.(MIR2022)
4 months ago11apache-2.0Python
A general 3D object detection codebse.
Awesome Yolo Object Detection596
7 days ago
🚀🚀🚀 A collection of some awesome public YOLO object detection series projects.
4 years ago55C++
Caffe implementation of our multi-scale object detection framework
3 years ago39apache-2.0
Ia Ssd295
a year ago7apache-2.0Python
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (CVPR 2022, Oral)
Awesome Automanous 3d Detection Methods265
a year ago4
Alternatives To Yolop
Select To Compare

Alternative Project Comparisons

You Only 👀 Once for Panoptic 🚗 Perception

You Only Look at Once for Panoptic driving Perception

by Dong Wu, Manwen Liao, Weitian Zhang, Xinggang Wang ✉️, Xiang Bai, Wenqing Cheng, Wenyu Liu School of EIC, HUST

(✉️) corresponding author.

arXiv technical report (Machine Intelligence Research2022)

The Illustration of YOLOP



  • We put forward an efficient multi-task network that can jointly handle three crucial tasks in autonomous driving: object detection, drivable area segmentation and lane detection to save computational costs, reduce inference time as well as improve the performance of each task. Our work is the first to reach real-time on embedded devices while maintaining state-of-the-art level performance on the BDD100Kdataset.

  • We design the ablative experiments to verify the effectiveness of our multi-tasking scheme. It is proved that the three tasks can be learned jointly without tedious alternating optimization.

  • We design the ablative experiments to prove that the grid-based prediction mechanism of detection task is more related to that of semantic segmentation task, which is believed to provide reference for other relevant multi-task learning research works.



Traffic Object Detection Result

Model Recall(%) mAP50(%) Speed(fps)
Multinet 81.3 60.2 8.6
DLT-Net 89.4 68.4 9.3
Faster R-CNN 81.2 64.9 8.8
YOLOv5s 86.8 77.2 82
YOLOP(ours) 89.2 76.5 41

Drivable Area Segmentation Result

Model mIOU(%) Speed(fps)
Multinet 71.6 8.6
DLT-Net 71.3 9.3
PSPNet 89.6 11.1
YOLOP(ours) 91.5 41

Lane Detection Result:

Model mIOU(%) IOU(%)
ENet 34.12 14.64
SCNN 35.79 15.84
ENet-SAD 36.56 16.02
YOLOP(ours) 70.50 26.20

Ablation Studies 1: End-to-end v.s. Step-by-step:

Training_method Recall(%) AP(%) mIoU(%) Accuracy(%) IoU(%)
ES-W 87.0 75.3 90.4 66.8 26.2
ED-W 87.3 76.0 91.6 71.2 26.1
ES-D-W 87.0 75.1 91.7 68.6 27.0
ED-S-W 87.5 76.1 91.6 68.0 26.8
End-to-end 89.2 76.5 91.5 70.5 26.2

Ablation Studies 2: Multi-task v.s. Single task:

Training_method Recall(%) AP(%) mIoU(%) Accuracy(%) IoU(%) Speed(ms/frame)
Det(only) 88.2 76.9 - - - 15.7
Da-Seg(only) - - 92.0 - - 14.8
Ll-Seg(only) - - - 79.6 27.9 14.8
Multitask 89.2 76.5 91.5 70.5 26.2 24.4

Ablation Studies 3: Grid-based v.s. Region-based:

Training_method Recall(%) AP(%) mIoU(%) Accuracy(%) IoU(%) Speed(ms/frame)
R-CNNP Det(only) 79.0 67.3 - - - -
R-CNNP Seg(only) - - 90.2 59.5 24.0 -
R-CNNP Multitask 77.2(-1.8) 62.6(-4.7) 86.8(-3.4) 49.8(-9.7) 21.5(-2.5) 103.3
YOLOP Det(only) 88.2 76.9 - - - -
YOLOP Seg(only) - - 91.6 69.9 26.5 -
YOLOP Multitask 89.2(+1.0) 76.5(-0.4) 91.5(-0.1) 70.5(+0.6) 26.2(-0.3) 24.4


  • The works we has use for reference including Multinet (paper,code,DLT-Net (paper,Faster R-CNN (paper,code,YOLOv5scode) ,PSPNet(paper,code) ,ENet(paper,code) SCNN(paper,code) SAD-ENet(paper,code). Thanks for their wonderful works.
  • In table 4, E, D, S and W refer to Encoder, Detect head, two Segment heads and whole network. So the Algorithm (First, we only train Encoder and Detect head. Then we freeze the Encoder and Detect head as well as train two Segmentation heads. Finally, the entire network is trained jointly for all three tasks.) can be marked as ED-S-W, and the same for others.


Traffic Object Detection Result

detect result

Drivable Area Segmentation Result

Lane Detection Result


  • The visualization of lane detection result has been post processed by quadratic fitting.

Project Structure

 images   # inference images
 output   # inference result
 config/default   # configuration of training and validation
  activations.py   # activation function
  evaluate.py   # calculation of metric
  function.py   # training and validation of model
  general.py   #calculation of metricnmsconversion of data-formatvisualization
  loss.py   # loss function
  postprocess.py   # postprocess(refine da-seg and ll-seg, unrelated to paper)
  AutoDriveDataset.py   # Superclass datasetgeneral function
  bdd.py   # Subclass datasetspecific function
  hust.py   # Subclass dataset(Campus scene, unrelated to paper)
  DemoDataset.py   # demo dataset(image, video and stream)
  YOLOP.py    # Setup and Configuration of model
  light.py    # Model lightweightunrelated to paper, zwt)
  commom.py   # calculation module
  augmentations.py    # data augumentation
  autoanchor.py   # auto anchor(k-means)
  split_dataset.py  # (Campus scene, unrelated to paper)
  utils.py  # loggingdevice_selecttime_measureoptimizer_selectmodel_save&initialize Distributed training
  dataset/training time  # Visualization, logging and model_save
  demo.py    # demo(foldercamera)
  deploy    # Deployment of model
  datapre    # Generation of gt(mask) for drivable area segmentation task
weights    # Pretraining model


This codebase has been developed with python version 3.7, PyTorch 1.7+ and torchvision 0.8+:

conda install pytorch==1.7.0 torchvision==0.8.0 cudatoolkit=10.2 -c pytorch

See requirements.txt for additional dependencies and version requirements.

pip install -r requirements.txt

Data preparation


We recommend the dataset directory structure to be the following:

# The id represent the correspondence relation
dataset root

Update the your dataset path in the ./lib/config/default.py.


You can set the training configuration in the ./lib/config/default.py. (Including: the loading of preliminary model, loss, data augmentation, optimizer, warm-up and cosine annealing, auto-anchor, training epochs, batch_size).

If you want try alternating optimization or train model for single task, please modify the corresponding configuration in ./lib/config/default.py to True. (As following, all configurations is False, which means training multiple tasks end to end).

# Alternating optimization
_C.TRAIN.SEG_ONLY = False           # Only train two segmentation branchs
_C.TRAIN.DET_ONLY = False           # Only train detection branch
_C.TRAIN.ENC_SEG_ONLY = False       # Only train encoder and two segmentation branchs
_C.TRAIN.ENC_DET_ONLY = False       # Only train encoder and detection branch

# Single task 
_C.TRAIN.DRIVABLE_ONLY = False      # Only train da_segmentation task
_C.TRAIN.LANE_ONLY = False          # Only train ll_segmentation task
_C.TRAIN.DET_ONLY = False          # Only train detection task

Start training:

python tools/train.py

Multi GPU mode:

python -m torch.distributed.launch --nproc_per_node=N tools/train.py  # N: the number of GPUs


You can set the evaluation configuration in the ./lib/config/default.py. (Including batch_size and threshold value for nms).

Start evaluating:

python tools/test.py --weights weights/End-to-end.pth

Demo Test

We provide two testing method.


You can store the image or video in --source, and then save the reasoning result to --save-dir

python tools/demo.py --source inference/images


If there are any camera connected to your computer, you can set the source as the camera number(The default is 0).

python tools/demo.py --source 0


input output


Our model can reason in real-time on Jetson Tx2, with Zed Camera to capture image. We use TensorRT tool for speeding up. We provide code for deployment and reasoning of model in ./toolkits/deploy.

Segmentation Label(Mask) Generation

You can generate the label for drivable area segmentation task by running

python toolkits/datasetpre/gen_bdd_seglabel.py

Model Transfer

Before reasoning with TensorRT C++ API, you need to transfer the .pth file into binary file which can be read by C++.

python toolkits/deploy/gen_wts.py

After running the above command, you obtain a binary file named yolop.wts.

Running Inference

TensorRT needs an engine file for inference. Building an engine is time-consuming. It is convenient to save an engine file so that you can reuse it every time you run the inference. The process is integrated in main.cpp. It can determine whether to build an engine according to the existence of your engine file.

Third Parties Resource


If you find our paper and code useful for your research, please consider giving a star ⭐️ and citation 📝 :

  title={Yolop: You only look once for panoptic driving perception},
  author={Wu, Dong and Liao, Man-Wen and Zhang, Wei-Tian and Wang, Xing-Gang and Bai, Xiang and Cheng, Wen-Qing and Liu, Wen-Yu},
  journal={Machine Intelligence Research},
Popular Object Detection Projects
Popular Autonomous Driving Projects
Popular Artificial Intelligence Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Object Detection
Autonomous Driving