Jetson Nano - FaceBook Detectron2 installation

What is Detectron2?

Detectron is an object detection system released by Facebook in 2018. Detectron development started in 2016, so it didn't support Facebook's PyTorch and was developed in Caffe2. Detectron2 is the second generation of the Detectron library, with important enhancements for both research and production use. Detectron shows quite good accuracy, but there are always speed issues to use with the Jetson Nano. The purpose of this article is to find out how much processing speed has been improved along with the accuracy of the Dettectron2.

The following is the content of detectron2 introduced in gitgub of facebookresearch that developed detectron2.

Detectron2 is Facebook AI Research's next generation software system that implements state-of-the-art object detection algorithms. It is a ground-up rewrite of the previous version, Detectron, and it originates from maskrcnn-benchmark.

What's New

• It is powered by the PyTorch deep learning framework.
• Includes more features such as panoptic segmentation, densepose, Cascade R-CNN, rotated bounding boxes, etc.
• Can be used as a library to support different projects on top of it. We'll open source more research projects in this way.
• It trains much faster.

Install Detectron2

Prerequisites

There seems to be a lot of people having trouble installing Detectron2 in the Jetson series. Detectron2 is a system made on Facebook, so you must install Pytorch and torchvision in advance. Installation is not difficult using the newly released JetPack 4.4 DP (Developer Preview) from NVidia.

See my other blog https://spyjetson.blogspot.com/2020/06/jetson-nano-jetpack-44-dp-and-pytorch.html for how to install Pytorch 1.5, torchvision 0.6.0 on Jetpack4.4dp Please do.

After installing the Pytorch, torchvision,  install these packages. If successful, you can see the Success message.

pip3 install cython 
pip3 install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'


..............

Successfully installed cython-0.29.20 matplotlib-3.2.1 numpy-1.18.5 pycocotools-2.0 python-dateutil-2.8.1 setuptools-47.3.0 six-1.15.0

Build Detectron2 from Source

Now it's time to install Detectron2. The homepage https://github.com/facebookresearch/detectron2/blob/master/INSTALL.md introduces how to install a pre-made version without a build, but it doesn't work well with Jetson Nano. So I will build the source code.

This is time consuming. You can have a coffee break time until the build is over.

python3 -m pip install 'git+https://github.com/facebookresearch/detectron2.git'


..............

Successfully installed absl-py-0.9.0 cachetools-4.1.0 cloudpickle-1.4.1 detectron2-0.1.3 fvcore-0.1.1.post20200616 google-auth-1.17.2 google-auth-oauthlib-0.4.1 grpcio-1.29.0 importlib-metadata-1.6.1 markdown-3.2.2 mock-4.0.2 oauthlib-3.1.0 portalocker-1.7.0 protobuf-3.12.2 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydot-1.4.1 requests-2.23.0 requests-oauthlib-1.3.0 rsa-4.6 tabulate-0.8.7 tensorboard-2.2.2 tensorboard-plugin-wit-1.6.0.post3 termcolor-1.1.0 tqdm-4.46.1 werkzeug-1.0.1 yacs-0.1.7 zipp-3.1.0

When the build is finished, test it as follows.  When I import  the detectron2, pre-installed yaml version does not match the version required by Detectron2.

root@jetpack-4:~# python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import detectron2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/detectron2/__init__.py", line 5, in <module>
    setup_environment()
  File "/usr/local/lib/python3.6/dist-packages/detectron2/utils/env.py", line 98, in setup_environment
    _configure_libraries()
  File "/usr/local/lib/python3.6/dist-packages/detectron2/utils/env.py", line 80, in _configure_libraries
    assert get_version(yaml) >= (5, 1), "Requires pyyaml>=5.1"
AssertionError: Requires pyyaml>=5.1

>>> import yaml
>>> print(yaml.__version__)
3.12
 

Detectron2 requires 5.1 or higher, but version 3.12 is currently installed. Therefore, you need to upgrade the yaml version.

root@jetpack-4:~# pip3 install pyyaml --upgrade
Collecting pyyaml
  Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB)
    100% |████████████████████████████████| 276kB 204kB/s
Building wheels for collected packages: pyyaml
  Running setup.py bdist_wheel for pyyaml ... done
  Stored in directory: /root/.cache/pip/wheels/a7/c1/ea/cf5bd31012e735dc1dfea3131a2d5eae7978b251083d6247bd
Successfully built pyyaml
Installing collected packages: pyyaml
  Found existing installation: PyYAML 3.12
    Not uninstalling pyyaml at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed pyyaml-5.3.1
root@jetpack-4:~# python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import detectron2
>>> from detectron2 import model_zoo
>>> from detectron2.engine import DefaultPredictor
>>> from detectron2.config import get_cfg
>>> from detectron2.utils.visualizer import Visualizer
>>> from detectron2.data import MetadataCatalog

After upgrading the pyyaml ​​package to 5.3.1 and testing again, you can see that detectron2 is imported without error .

Test the Detectron2

The following example is a modification of the Jupyter Notebook example provided by Facebook on CoLab.

from detectron2 import model_zoo
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog
import cv2
import numpy as np
import requests, sys, time, os
from PIL import Image

url = 'http://images.cocodataset.org/val2017/000000439715.jpg'

img = Image.open(requests.get(url, stream=True).raw)
im = np.asarray(img, dtype="uint8")
height, width, channels = im.shape
if channels == 3:
    im = cv2.cvtColor(im, cv2.COLOR_RGB2BGR)
else:
    im = cv2.cvtColor(im, cv2.COLOR_RGBA2BGR)

print('image W:%d H:%d'%(width, height))

cfg = get_cfg()
# add project-specific config (e.g., TensorMask) here if you're not running a model in detectron2's core library
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model
# Find a model from detectron2's model zoo. You can use the https://dl.fbaipublicfiles... url as well
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
predictor = DefaultPredictor(cfg)


for i in range (5):
    fps_time  = time.perf_counter()
    outputs = predictor(im)

    fps = 1.0 / (time.perf_counter() - fps_time)
    print("Net FPS: %f" % (fps))

    #print(outputs["instances"].pred_classes)
    #print(outputs["instances"].pred_boxes)


    v = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
    out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
    
cv2.imwrite("detectron2_result.jpg", out.get_image()[:, :, ::-1])
cv2.imwrite("./source_image.jpg", im)

<test_detectron2.py>

This test python file downloads a file from the Internet, runs the Instance segmentation model, and saves the result to the detectron2_result.jpg file. It is repeated 5 times for one input image. The reason is that for all network models, the first inference task is always time consuming. Therefore, you can run more than once and refer to the working time after the second. In the next blog, I will discuss instance segmentation in detail.

root@jetpack-4:/usr/local/src/detectron2# python3 test_detectron2.py
image W:640 H:480
Net FPS: 0.139404
Net FPS: 0.240988
Net FPS: 0.241667
Net FPS: 0.241672
Net FPS: 0.241247
Segmentation fault (core dumped)

It can be seen that approximately 0.24 FPS time is required to process the 640X480 size image. It is not so satisfactory. Personally, I judge 10 FPS as actual availability. And the last Segmentation fault was explained in a previous blog.

<original image and result image>

Wrapping up

I tested it on Jetson Nano, if you use Jetpack 4.4 DP version or higher, there will be no problem in TX2 or Xavier. In the next article, I will look at various features and performance of Detectron2.