BEVFusion（北大&阿里）环境搭建教程

BEVFusion环境搭建

论文题目：BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework

论文地址：https://arxiv.org/pdf/2205.13790.pdf

代码地址：ADLab-AutoDrive/BEVFusion: Offical PyTorch implementation of “BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework” (github.com)

前言：这是今年新发的一篇论文，我在第一次阅读时，代码还未开源，前几天发现开源了，而且这个框架做的很清晰，可以基于这个工作熟悉一下融合方案，也能稍稍改进一下。

笔者环境：

sys.platform: linux

Python: 3.8.3 | packaged by conda-forge | (default, Jun 1 2020, 17:43:00) [GCC 7.5.0]

CUDA available: True

GPU 0,1: NVIDIA GeForce RTX 3090

CUDA_HOME: /usr

NVCC: Build cuda_11.5.r11.5/compiler.30672275_0

GCC: gcc (Ubuntu 7.5.0-6ubuntu2) 7.5.0

PyTorch: 1.8.0+cu111

PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel® Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel® 64 architecture applications
• Intel® MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
• CuDNN 8.0.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.9.0+cu111

OpenCV: 4.6.0

MMCV: 1.4.0

MMCV Compiler: GCC 7.3

MMCV CUDA Compiler: 11.1

MMDetection: 2.11.0

MMDetection3D: 0.11.0+9d3e162

说人话：torch1.8.0+cuda111，显卡是3090，其余环境根据根据官方文档配的

一、安装

1、克隆BEVFusion代码

git clone https://github.com/ADLab-AutoDrive/BEVFusion.git

2、新建虚拟环境

conda create -n bevfusion python=3.8.3
conda activate bevfusion #激活

3、根据cuda版本安装torch

尽量选择1.7或者1.8，cuda版本要对应

2022年11月22日更新：尽量选1.7吧，1.8的bug导致训练有些问题

2022年12月2日更新：最近太忙，前几天发现1.7的torch不适合在这个项目上用，具体原因和3090显卡有关，缺少相关库（3090最低支持cuda11.1，torch1.7最高支持cuda11.0）

pip install torch=1.8.0+cu111 torchvision=0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

4、安装mmcv-full

这里要根据自己的torch和cuda来，另外，BEVFusion官方文档推荐1.4.0版本

从以下网址找到自己合适的版本

https://download.openmmlab.com/mmcv/dist/{cu_version}/{torch_version}/index.html

其中cu_version代表cuda版本，torch_version代表torch版本，

例如，我的是torch1.8.0+cuda111，所以我的地址是：

https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html

因为创建的环境是python3.8，所以选择cp38，别忘了mmcv-full版本是1.4.0，下载whl文件，并传到服务器上，然后pip whl文件（不再赘述pip安装whl了，百度一下）

5、安装MMDetection

下面是BEVFusion官方推荐的环境

python=3.8.3

mmdet=2.11.0 (please install mmdet in mmdetection-2.11.0)

mmcv=1.4.0

mmdet3d=0.11.0

numpy=1.19.2

torch=1.7.0

torchvision=0.8.0

注意please install mmdet in mmdetection-2.11.0，作者已经在仓库中放了这个文件夹，我们直接进入这个文件夹

然后执行：

pip install -r requirements/build.txt
pip install -v -e .  # or "python setup.py develop"(推荐执行后者)

6、安装MMDetection3D

退回到BEVFusion根目录下，执行下面指令

pip install -v -e .  # or "python setup.py develop"（推荐执行后者）

至此配置完毕

以下是我的环境

Package Version Editable project location

---

absl-py 1.3.0

addict 2.4.0

anyio 3.6.2

argon2-cffi 21.3.0

argon2-cffi-bindings 21.2.0

asttokens 2.1.0

attrs 22.1.0

backcall 0.2.0

beautifulsoup4 4.11.1

black 22.10.0

bleach 5.0.1

cachetools 5.2.0

certifi 2022.9.24

cffi 1.15.1

charset-normalizer 2.1.1

click 8.1.3

contourpy 1.0.6

cycler 0.11.0

Cython 0.29.32

debugpy 1.6.3

decorator 5.1.1

defusedxml 0.7.1

depthwise-conv2d-implicit-gemm 0.0.0

descartes 1.1.0

entrypoints 0.4

exceptiongroup 1.0.4

executing 1.2.0

fastjsonschema 2.16.2

filelock 3.8.0

fire 0.4.0

flake8 5.0.4

fonttools 4.38.0

google-auth 2.14.1

google-auth-oauthlib 0.4.6

grpcio 1.50.0

h5py 3.7.0

huggingface-hub 0.11.0

idna 3.4

imageio 2.22.4

importlib-metadata 5.0.0

importlib-resources 5.10.0

iniconfig 1.1.1

ipykernel 6.17.1

ipython 8.6.0

ipython-genutils 0.2.0

ipywidgets 8.0.2

jedi 0.18.1

Jinja2 3.1.2

joblib 1.2.0

jsonschema 4.17.0

jupyter 1.0.0

jupyter_client 7.4.6

jupyter-console 6.4.4

jupyter_core 5.0.0

jupyter-server 1.23.2

jupyterlab-pygments 0.2.2

jupyterlab-widgets 3.0.3

kiwisolver 1.4.4

llvmlite 0.31.0

loguru 0.6.0

lyft-dataset-sdk 0.0.8

Markdown 3.4.1

MarkupSafe 2.1.1

matplotlib 3.6.2

matplotlib-inline 0.1.6

mccabe 0.7.0

mistune 2.0.4

mmcls 0.24.1

mmcv-full 1.4.0

mmdet 2.11.0 /home/wistful/work/my_bevfusion/mmdetection-2.11.0

mmdet3d 0.11.0 /home/wistful/work/my_bevfusion

mmpycocotools 12.0.3

msgpack 1.0.4

msgpack-numpy 0.4.8

multimethod 1.9

mypy-extensions 0.4.3

nbclassic 0.4.8

nbclient 0.7.0

nbconvert 7.2.5

nbformat 5.7.0

nest-asyncio 1.5.6

networkx 2.2

ninja 1.11.1

notebook 6.5.2

notebook_shim 0.2.2

numba 0.48.0

numpy 1.23.4

nuscenes-devkit 1.1.9

oauthlib 3.2.2

opencv-python 4.6.0.66

packaging 21.3

pandas 1.4.4

pandocfilters 1.5.0

parso 0.8.3

pathspec 0.10.2

pexpect 4.8.0

pickleshare 0.7.5

Pillow 9.3.0

pip 22.3.1

pkgutil_resolve_name 1.3.10

platformdirs 2.5.4

plotly 5.11.0

pluggy 1.0.0

plyfile 0.7.4

prettytable 3.5.0

prometheus-client 0.15.0

prompt-toolkit 3.0.32

protobuf 3.20.3

psutil 5.9.4

ptyprocess 0.7.0

pure-eval 0.2.2

pyasn1 0.4.8

pyasn1-modules 0.2.8

pycocotools 2.0.6

pycodestyle 2.9.1

pycparser 2.21

pyflakes 2.5.0

Pygments 2.13.0

pyparsing 3.0.9

pyquaternion 0.9.9

pyrsistent 0.19.2

pytest 7.2.0

python-dateutil 2.8.2

pytz 2022.6

PyWavelets 1.4.1

PyYAML 6.0

pyzmq 24.0.1

qtconsole 5.4.0

QtPy 2.3.0

requests 2.28.1

requests-oauthlib 1.3.1

rsa 4.9

scikit-image 0.19.3

scikit-learn 1.1.3

scipy 1.9.3

Send2Trash 1.8.0

setuptools 65.5.1

Shapely 1.8.5.post1

six 1.16.0

sniffio 1.3.0

soupsieve 2.3.2.post1

stack-data 0.6.1

tabulate 0.9.0

tenacity 8.1.0

tensorboard 2.11.0

tensorboard-data-server 0.6.1

tensorboard-plugin-wit 1.8.1

tensorpack 0.11

termcolor 2.1.0

terminado 0.17.0

terminaltables 3.1.10

threadpoolctl 3.1.0

tifffile 2022.10.10

timm 0.6.11

tinycss2 1.2.1

toml 0.10.2

tomli 2.0.1

torch 1.8.0+cu111

torchaudio 0.8.0

torchpack 0.3.1

torchvision 0.9.0+cu111

tornado 6.2

tqdm 4.64.1

traitlets 5.5.0

trimesh 2.35.39

typing_extensions 4.4.0

urllib3 1.26.12

wcwidth 0.2.5

webencodings 0.5.1

websocket-client 1.4.2

Werkzeug 2.2.2

wheel 0.38.4

widgetsnbextension 4.0.3

yapf 0.32.0

zipp 3.10.0

---

sys.platform: linux

Python: 3.8.3 | packaged by conda-forge | (default, Jun 1 2020, 17:43:00) [GCC 7.5.0]

CUDA available: True

GPU 0,1: NVIDIA GeForce RTX 3090

CUDA_HOME: /usr/local/cuda

NVCC: Build cuda_11.1.TC455_06.29069683_0

GCC: gcc (Ubuntu 7.5.0-6ubuntu2) 7.5.0

PyTorch: 1.8.0+cu111

PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel® Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel® 64 architecture applications
• Intel® MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
• CuDNN 8.0.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.9.0+cu111

OpenCV: 4.6.0

MMCV: 1.4.0

MMCV Compiler: GCC 7.3

MMCV CUDA Compiler: 11.1

MMDetection: 2.20.0

MMDetection3D: 0.11.0+9d3e162

二、组织数据集

注意，一般来说使用mmdetection3d框架的工作，要把数据集组织到mmdetection3d/data下，BEVFusion整个目录就是mmdetection3d的组织格式。所以要把数据集组织到BEVFusion/data下

It is recommended to symlink the dataset root to $MMDETECTION3D/data. If your folder structure is different from the following, you may need to change the corresponding paths in config files.

数据集组织我就不再赘述了，按照官方说明的组织形式组织好

mmdetection3d

├── mmdet3d

├── tools

├── configs

├── data

│ ├── nuscenes

│ │ ├── maps

│ │ ├── samples

│ │ ├── sweeps

│ │ ├── v1.0-test

| | ├── v1.0-trainval

│ ├── kitti

│ │ ├── ImageSets

│ │ ├── testing

│ │ │ ├── calib

│ │ │ ├── image_2

│ │ │ ├── velodyne

│ │ ├── training

│ │ │ ├── calib

│ │ │ ├── image_2

│ │ │ ├── label_2

│ │ │ ├── velodyne

│ ├── waymo

│ │ ├── waymo_format

│ │ │ ├── training

│ │ │ ├── validation

│ │ │ ├── testing

│ │ │ ├── gt.bin

│ │ ├── kitti_format

│ │ │ ├── ImageSets

│ ├── lyft

│ │ ├── v1.01-train

│ │ │ ├── v1.01-train (train_data)

│ │ │ ├── lidar (train_lidar)

│ │ │ ├── images (train_images)

│ │ │ ├── maps (train_maps)

│ │ ├── v1.01-test

│ │ │ ├── v1.01-test (test_data)

│ │ │ ├── lidar (test_lidar)

│ │ │ ├── images (test_images)

│ │ │ ├── maps (test_maps)

│ │ ├── train.txt

│ │ ├── val.txt

│ │ ├── test.txt

│ │ ├── sample_submission.csv

│ ├── scannet

│ │ ├── meta_data

│ │ ├── scans

│ │ ├── batch_load_scannet_data.py

│ │ ├── load_scannet_data.py

│ │ ├── scannet_utils.py

│ │ ├── README.md

│ ├── sunrgbd

│ │ ├── OFFICIAL_SUNRGBD

│ │ ├── matlab

│ │ ├── sunrgbd_data.py

│ │ ├── sunrgbd_utils.py

│ │ ├── README.md

（以上数据集不用全需要的，自己需要哪个组织哪个）

然后利用BEVFusion/tools/create_data.py脚本组织文件就行了，放一下组织nuScenes数据集成功的截图吧

三、训练与测试

1、分布式训练

# first train camera stream
./tools/dist_train.sh configs/bevfusion/cam_stream/bevf_pp_4x8_2x_nusc_cam.py 8
# then train LiDAR stream
./tools/dist_train.sh configs/bevfusion/lidar_stream/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d.py 8
# then train BEVFusion
./tools/dist_train.sh configs/bevfusion/bevf_pp_2x8_1x_nusc.py 8

./tools/dist_train.sh configs/bevfusion/cam_stream/bevf_pp_4x8_2x_nusc_cam.py 8中，8代表GPU个数

使用sh文件进行训练和测试时，可能会遇到无权限的问题，使用chmod 777 tools/dist_train.sh进行权限更改即可

其中，配置文件我就不再叙述，可以使用官方的简单测试，如果需要使用自己的配置文件，请参考mmdetection3d官方文档

值得注意的是，我在使用分布式和非分布式方法做测试的时候，非分布式运行正常，分布式仍然提示找不到数据集，我搜寻资料得知是因为工作路径的原因导致的，我用print(os.getcwd())打印了一下，发现使用非分布式训练时，工作路径/home/wistful/work/my_bevfusion/mmdetection3d/tools下，即在BEVFusion/mmdetection3d/tools下，而使用分布式训练时，工作路径在/home/wistful/work/my_bevfusion下，即在BEVFusion根目录下。所以我使用os.chdir(‘/home/wistful/work/my_bevfusion/mmdetection3d/tools’)将工作路径改变了一下，问题解决

2、非分布式训练

直接在pycharm中根据各参数设置一下，训练即可，不再叙述。

3、测试

./tools/dist_test.sh configs/bevfusion/bevf_pp_2x8_1x_nusc.py ./work_dirs/bevfusion_pp.pth 8 --eval bbox
 # 前者是配置文件，后者是GPU个数

# 如需可视化
./tools/dist_test.sh configs/bevfusion/bevf_pp_2x8_1x_nusc.py ./work_dirs/bevfusion_pp.pth 8 --eval bbox --show-dir YourVisualResultsDir --show --eval bbox #根据test.py的参数看一下

可视化结果会保存在你指定的目录下，如下所示

导入CloudCampare查看