2021년 2월 13일 토요일

Jetpack 4.5 - Install the latest version of OpenPose on Jetson Xavier NX

 In the previous article, I installed OpenPose 1.7.0 on the Jetson Nano. This time, I will install OpenPose 1.7.0 on Xavier NX. Since both Nano and Xavier NX will use JetPack 4.5, there is no significant difference in installation.

However, Xavier NX can use SSD. By installing an SSD that provides better performance than an SD card, I will try to work faster.

The working directory on the Nano was /usr/local/src. In Xavier NX, the directory will be used by mounting the SSD (/dev/nvme0n1) to /usr/local/src.


Prerequisites

For how to use SSD in Xavier NX, refer to the following.

You can see that the NVME SSD is mounted in /usr/local/src. Therefore, performance improvement can be expected rather than working on an SD card.

root@spypiggy-nx:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p1   59G   14G   43G  24% /
none            3.5G     0  3.5G   0% /dev
tmpfs           3.8G   40K  3.8G   1% /dev/shm
tmpfs           3.8G   30M  3.8G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
tmpfs           777M  4.0K  777M   1% /run/user/0
tmpfs           777M   20K  777M   1% /run/user/120
tmpfs           777M  140K  777M   1% /run/user/1000
/dev/nvme0n1    117G  1.3G  110G   2% /usr/local/src


Refer to the following for how to install OpenPose 1.7.0 in Jetson Nano. It is helpful to read this information in advance.


CMake version check and update

Refer to "Jetpack 4.5-Install the latest version of OpenPose on Jetson Nano" and install the latest version of cmake in advance. 

The installation process is almost the same as the Jetson Nano. However, the working directory have changed. On Jetson Nano, I will work in the /usr/local/src directory as the root user, but in Xavier NX you will work in the SSD that is mounted at /usr/local/src directory as the root user.

First, I checked the mounted SSD information. And check the power mode. I had to use 15W, 6 Core to use the maximum performance of the Xavier NX.

spypiggy@spypiggy-nx:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p1   59G   13G   44G  23% /
none            3.5G     0  3.5G   0% /dev
tmpfs           3.8G  4.0K  3.8G   1% /dev/shm
tmpfs           3.8G   21M  3.8G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/nvme0n1    117G  427M  111G   1% /home/spypiggy
tmpfs           777M   16K  777M   1% /run/user/120
tmpfs           777M     0  777M   0% /run/user/1000


spypiggy@spypiggy-nx:~$ sudo nvpmodel -q
NV Fan Mode:quiet
NV Power Mode: MODE_15W_6CORE
2


Then build the latest cmake.

root@spypiggy-nx:~# apt-get update
root@spypiggy-nx:~# apt-get -y install libssl-dev libcurl4-openssl-dev qt5-default
root@spypiggy-nx:~# apt-get install -y build-essential libboost-all-dev libboost-dev libhdf5-dev libatlas-base-dev
root@spypiggy-nx:~# apt-get install -y python3-dev python3-pip
root@spypiggy-nx:~# apt-get remove -y cmake
root@spypiggy-nx:~# cd /usr/local/src
root@spypiggy-nx:/usr/local/src# wget https://github.com/Kitware/CMake/releases/download/v3.19.4/cmake-3.19.4.tar.gz
root@spypiggy-nx:/usr/local/src# tar -xvzf cmake-3.19.4.tar.gz
root@spypiggy-nx:/usr/local/src# cd cmake-3.19.4
root@spypiggy-nx:/usr/local/src/cmake-3.19.4# ./bootstrap --qt-gui
root@spypiggy-nx:/usr/local/src/cmake-3.19.4# make -j6
root@spypiggy-nx:/usr/local/src/cmake-3.19.4# make install

If you use ssh, you must restart your ssh session. The cmake and cmake-gui are created in the /cmake-3.19.4/bin directory. If the version of cmake or cmake-gui has not been updated in a new ssh or local terminal after make install, copy the cmake and cmake-gui files from the above directory to the /usr/bin directory.


Install OpenPose for Xavier NX

Install Protocol Buffer

First, install Google's protocol buffer. 

root@spypiggy-nx:~# apt-get install -y libprotobuf-dev protobuf-compiler libgflags-dev libgoogle-glog-dev
 

Install OpenPose 1.7.0

As of now, the latest version of OpenPose is 1.7.0, released on November 17th, 2020. 

Download the source code. 

root@spypiggy-nx:~# cd /usr/local/src
root@spypiggy-nx:/usr/local/src# wget https://github.com/CMU-Perceptual-Computing-Lab/openpose/archive/v1.7.0.tar.gz root@spypiggy-nx:/usr/local/src# tar -xvzf v1.7.0.tar.gz root@spypiggy-nx:/usr/local/src# cd openpose-1.7.0/ root@spypiggy-nx:/usr/local/src/openpose-1.7.0# cd 3rdparty root@spypiggy-nx:/usr/local/src/openpose-1.7.0# git clone https://github.com/CMU-Perceptual-Computing-Lab/caffe.git root@spypiggy-nx:/usr/local/src/openpose-1.7.0# git clone https://github.com/pybind/pybind11.git

<Download Openpose source code>



step1. Enable BUILD_PYTHON OPTION 

The following is the procedure to create a Makefile using cmake. Run "sudo cmake-gui" and specify the source code directory and build directory as below. Be sure to check the BUILD_PYTHON option.



step2. Disable USE_CUDNN option


The use of CUDNN must be excluded in the option. If you build OpenPose 1.7.0 from JetPack 4.5 and you use this option, the build is fine, but phenomena such as system down or segmentation fault(core dump) may occur when running. 
I do not yet know the exact cause of this phenomenon. For now, this option is excluded to prevent severe errors at runtime.

step3. Generate Makefile 

After checking the options, click the Configure button again. Maybe the red area will turn white. And finally, click Generate button to generate Makefile. 



When the Makefile is properly created, run make command.

root@spypiggy-nx:/usr/local/src/openpose-1.7.0# cd build
root@spypiggy-nx:/usr/local/src/openpose-1.7.0/build# make -j6
root@spypiggy-nx:/usr/local/src/openpose-1.7.0/build# make install
root@spypiggy-nx:/usr/local/src/openpose-1.7.0/build# cd python
root@spypiggy-nx:/usr/local/src/openpose-1.7.0/build/python# make

It's time to copy our OpenPose python modules to python3.6 path directory(/usr/lib/python3.6/dist-packages). Then you can import openpose without path problem.

cp -r /usr/local/src/openpose-1.7.0/build/python/openpose/ /usr/lib/python3.6/dist-packages

Let's test your openpose python library.

root@spypiggy-nx:/usr/local/src/openpose-1.7.0# python3
Python 3.6.9 (default, Oct  8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import openpose

Yes, I finally installed the OpenPose 1.7.0 version successfully on the Jetson Xavier NX.


Run a sample program

Let's run a sample program to test whether the OpenPose is properly installed.

root@spypiggy-nx:/usr/local/src/openpose-1.7.0# ./build/examples/openpose/openpose.bin --video ./examples/media/video.avi
Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
OpenPose demo successfully finished. Total time: 162.277487 seconds.

And maybe you can watch the next video.


This sample program records 1.3 fps with video.avi. Although this value is higher than the Jetson Nano's FPS, it is still not satisfactory. 

You can control the processing speed with the --net_resolution option.

Increasing the resolution of this option value increases the accuracy instead of increasing the processing speed to some extent.

However, if this option value is too high than the resolution used for learning in OpenPose, the recognition rate will decrease. Conversely, lowering this option lowers the recognition rate and increases the processing speed.

The following figure is the result of testing after setting --net_resolution "256x128".
<net_resolution 256x128>

Wrapping up

In my previous blog, I installed OpenPose 1.7.0 on Jetson Nano, and this time I installed OpenPose 1.7.0 on Xavier NX. Unlike the Jetson Nano, the Xavier NX, which has 8GB of memory, did not show a forced termination of the program. However, since the FPS value is not satisfactory, it seems good to find an appropriate combination of processing speed and recognition rate by adjusting the net_resolution option.














2021년 2월 12일 금요일

Jetpack 4.5 - Install the latest version of OpenPose on Jetson Nano

 OpenPose(https://github.com/CMU-Perceptual-Computing-Lab/openpose) is one of the most popular pose estimation framework. 




There was no problem in installing OpenPose up to the Jetpack 4.4DP version, but when OpenPose was installed in the JetPack 4.4 Production Release, an error occurred. The cause of this problem occurred when some of the CUDA functions used in Caffe were missing. This is explained in detail in Jetson Xavier NX-Human Pose estimation using OpenPose.

However, OpenPose version 1.7.0 released on November 17th, 2020 seems to have solved this problem. If you look at the OpenPose installation documentation, you will find the following contents.

OpenPose uses a custom fork of Caffe (rather than the official Caffe master). Our custom fork is only updated if it works on our machines, but we try to keep it updated with the latest Caffe version. This version works on a newly formatted machine (Ubuntu 16.04 LTS) and in all our machines (CUDA 8 and 10 tested). The default GPU version is the master branch, which it is also compatible with CUDA 10 without changes (official Caffe version might require some changes for it). We also use the OpenCL and CPU tags if their CMake flags are selected. We only modified some Caffe compilation flags and minor details.

Alternatively, you can use your own Caffe distribution on Ubuntu/Mac by 1) disabling BUILD_CAFFE, 2) setting Caffe_INCLUDE_DIRS to {CAFFE_PATH}/include/caffe, and 3) setting Caffe_LIBS to {CAFFE_PATH}/build/lib/libcaffe.so, as shown in the image below. Note that cuDNN-compatible Caffe version is required in order to get the maximum possible accuracy in OpenPose.


I will install the latest version of OpenPose once again in JetPack 4.5 of the newly installed Jetson Nano.


Prerequisites

For reference, I installed OpenPose while upgrading to OpenCV 4.5. But I think it will be fine with OpenCV 4.1.1 provided by JetPack 4.5.


cmake version check

    To build Openpose on the Jetson Nano, you should have cmake version 3.12.2 or higher. First check the cmake version.

    cmake --version
    


    If your Jetson Nano's cmake version is lower than 3.12.2, remove the old cmake and rebuild from source codes. Check the latest version at https://github.com/Kitware/CMake/releases . At this time(2021.02.9), Ver 3.19.4 is the latest cmake version.  I used the --qt-gui option in bootstrap to build cmake-gui together.

    apt-get update
    apt-get install -y libssl-dev libcurl4-openssl-dev qt5-default
    apt-get install -y build-essential libboost-all-dev libboost-dev libhdf5-dev libatlas-base-dev
    apt-get install -y python3-dev python3-pip
    apt-get remove -y cmake
    cd /usr/local/src
    wget https://github.com/Kitware/CMake/releases/download/v3.19.4/cmake-3.19.4.tar.gz
    tar -xvzf cmake-3.19.4.tar.gz
    cd cmake-3.19.4
    ./bootstrap --qt-gui
    make -j4
    make install


    Then you must restart your ssh session. The cmake and cmake-gui are created in the /cmake-3.19.4/bin directory. If the version of cmake or cmake-gui has not been updated in a new ssh or local terminal after make install, copy the cmake and cmake-gui files from the above directory to the /usr/bin directory.


    Install OpenPose for JetPack 4.5


    Install Protocol Buffer

    First, install Google's protocol buffer. 

    apt-get install -y libprotobuf-dev protobuf-compiler libgflags-dev libgoogle-glog-dev
    <Protocol Buffer installation>

    Install OpenPose 1.7.0

    As of now, the latest version of OpenPose is 1.7.0, released on November 17th, 2020. 
    Download the source code. 

    root@spypiggy-nano:/#cd /usr/local/src
    root@spypiggy-nano:/usr/local/src#wget https://github.com/CMU-Perceptual-Computing-Lab/openpose/archive/v1.7.0.tar.gz
    root@spypiggy-nano:/usr/local/src#tar -xvzf v1.7.0.tar.gz
    root@spypiggy-nano:/usr/local/src#cd openpose-1.7.0/
    root@spypiggy-nano:/usr/local/src/openpose-1.7.0#cd 3rdparty
    
    root@spypiggy-nano:/usr/local/src/openpose-1.7.0/3rdparty#git clone https://github.com/CMU-Perceptual-Computing-Lab/caffe.git
    root@spypiggy-nano:/usr/local/src/openpose-1.7.0/3rdparty#git clone https://github.com/pybind/pybind11.git


    <Download Openpose source code>

    step1. Enable BUILD_PYTHON OPTION 

    The following is the procedure to create a Makefile using cmake. This time, I will use cmake-gui. You can also use cmake in the shell. Run cmake-gui and specify the source code directory and build directory as below. Be sure to check the BUILD_PYTHON option.


    step2. Disable USE_CUDNN option


    <cmake-gui build option>


    The use of CUDNN must be excluded in the option. If you build OpenPose 1.7.0 from JetPack 4.5 and you use this option, the build is fine, but phenomena such as system down or segmentation fault(core dump) may occur when running. 
    I do not yet know the exact cause of this phenomenon. For now, this option is excluded to prevent severe errors at runtime.

    step3. Generate Makefile 

    After checking the options, click the Configure button again. Maybe the red area will turn white. Perhaps this step will be time consuming. And finally, click Generate button to generate Makefile.



    Now, a build directory contains files necessary for build including Makefile.
    When the Makefile is properly created, run make command.

    root@spypiggy-nano:/usr/local/src/openpose-1.7.0/build# make -j4
    
    root@spypiggy-nano:/usr/local/src/openpose-1.7.0/build# make install
    root@spypiggy-nano:/usr/local/src/openpose-1.7.0/build# cd python
    root@spypiggy-nano:/usr/local/src/openpose-1.7.0/build/python# make -j4


    Python import Problem 

    Before you run python samples, first check the openpose python file. Yes it's under the "openpose installation directory/build/python/openpose".

    It's time to copy our OpenPose python modules to python3.6 path directory(/usr/lib/python3.6/dist-packages). Then you can import openpose without path problem.

    cp -r /usr/local/src/openpose-1.7.0/build/python/openpose/ /usr/lib/python3.6/dist-packages


    Let's test your openpose python library.

    root@spypiggy-nano:/usr/local/src/openpose-1.7.0/build/python/openpose# python3
    Python 3.6.9 (default, Oct  8 2020, 12:12:24)
    [GCC 8.4.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import openpose
    >>>
    

    Yes, I finally installed the OpenPose 1.7.0 version successfully.


    Run a sample program 

    Let's run a sample program to test whether the OpenPose is properly installed.

    root@spypiggy-nano:/usr/local/src/openpose-1.7.0# ./build/examples/openpose/openpose.bin --video ./examples/media/video.avi  --net_resolution "256x128"
    Starting OpenPose demo...
    Configuring OpenPose...
    Starting thread(s)...
    Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
    

    If you see this output, it's a successfull installation.


    You have now confirmed that OpenPose is working properly on Jetson Nano.

    Be Careful : The Jetson Nano has 4GB of memory. 4GB of memory is not enough to run OpenPose. In the example above, I used the --net_resolution "256x128" option. The reason for using this option is related to memory usage. As the resolution of this option value increases, the processing speed decreases and the memory usage increases. If the resolution is increased above a certain level, the process is forcibly terminated due to insufficient memory. 

    If you don't use the --net_resolution option, OpenPose will find and use the optimal value. For a 16:9 ratio image, the default value of "656x368" is used. This value must be a multiple of 16.

    root@spypiggy-nano:/usr/local/src/openpose-1.7.0# ./build/examples/openpose/openpose.bin --video ./examples/media/video.avi
    Starting OpenPose demo...
    Configuring OpenPose...
    Starting thread(s)...
    Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
    Killed

    <openpose process is killed due to insufficient memory>

    JetPack's Ubuntu desktop takes up a lot of memory. Therefore, if you replace the desktop GUI with a lightweight LXDE, you can get an additional 500MB to 1GB of memory. How to acquire additional memory using LXDE is described below:


    Wrapping up 

    Refer to the articles below for how to use OpenPose in the Jetson Nano, Xavier NX, and TX2 series.


    If you want to install OpenPose 1.7.0 on the Xavier NX, refer to this article:


    In the next article, I'll cover how to use Python with OpenPose 1.7.0.


    2021년 2월 9일 화요일

    JetPack 4.5 - OpenCV 4.1 vs. OpenCV 4.5

     Jetpack 4.5 is provided with OpenCV 4.1.1 installed. OpenCV 4.1.1 is a version provided from Jetpack 4.3, released in April 2020. The current (2021.02) OpenCV latest version is 4.5.1. Let's see how to upgrade OpenCV 4.1.1 to 4.5.1 in JetPack 4.5.. And we will see if there is a need to upgrade from OpenCV 4.1.1 to 4.5.1.


    OpenCV 4.1.1

    First, let's look at the information of OpenCV installed in JetPack 4.5.

    root@spypiggy-nano:~# python3
    Python 3.6.9 (default, Oct  8 2020, 12:12:24)
    [GCC 8.4.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import cv2
    >>> print(cv2.getBuildInformation())
    
    General configuration for OpenCV 4.1.1 =====================================
      Version control:               4.1.1-2-gd5a58aa75
    
      Platform:
        Timestamp:                   2019-12-13T17:25:11Z
        Host:                        Linux 4.9.140-tegra aarch64
        CMake:                       3.10.2
        CMake generator:             Unix Makefiles
        CMake build tool:            /usr/bin/make
        Configuration:               Release
    
      CPU/HW features:
        Baseline:                    NEON FP16
          required:                  NEON
          disabled:                  VFPV3
    
      C/C++:
        Built as dynamic libs?:      YES
        C++ Compiler:                /usr/bin/c++  (ver 7.4.0)
        C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
        C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
        C Compiler:                  /usr/bin/cc
        C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
        C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
        Linker flags (Release):      -Wl,--gc-sections
        Linker flags (Debug):        -Wl,--gc-sections
        ccache:                      NO
        Precompiled headers:         NO
        Extra dependencies:          dl m pthread rt
        3rdparty dependencies:
    
      OpenCV modules:
        To be built:                 calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo python2 python3 stitching ts video videoio
        Disabled:                    world
        Disabled by dependency:      -
        Unavailable:                 java js
        Applications:                tests perf_tests examples apps
        Documentation:               NO
        Non-free algorithms:         NO
    
      GUI:
        GTK+:                        YES (ver 2.24.32)
          GThread :                  YES (ver 2.56.4)
          GtkGlExt:                  NO
    
      Media I/O:
        ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
        JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
        WEBP:                        build (ver encoder: 0x020e)
        PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.34)
        TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.9)
        JPEG 2000:                   build (ver 1.900.1)
        HDR:                         YES
        SUNRASTER:                   YES
        PXM:                         YES
        PFM:                         YES
    
      Video I/O:
        FFMPEG:                      YES
          avcodec:                   YES (57.107.100)
          avformat:                  YES (57.83.100)
          avutil:                    YES (55.78.100)
          swscale:                   YES (4.8.100)
          avresample:                NO
        GStreamer:                   YES (1.14.5)
        v4l/v4l2:                    YES (linux/videodev2.h)
    
      Parallel framework:            TBB (ver 2017.0 interface 9107)
    
      Trace:                         YES (with Intel ITT)
    
      Other third-party libraries:
        Lapack:                      NO
        Eigen:                       YES (ver 3.3.4)
        Custom HAL:                  YES (carotene (ver 0.0.1))
        Protobuf:                    build (3.5.1)
    
      Python 2:
        Interpreter:                 /usr/bin/python2.7 (ver 2.7.15)
        Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.15+)
        numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.3)
        install path:                lib/python2.7/dist-packages/cv2/python-2.7
    
      Python 3:
        Interpreter:                 /usr/bin/python3 (ver 3.6.9)
        Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.6m.so (ver 3.6.9)
        numpy:                       /usr/lib/python3/dist-packages/numpy/core/include (ver 1.13.3)
        install path:                lib/python3.6/dist-packages/cv2/python-3.6
    
      Python (for build):            /usr/bin/python2.7
    
      Java:
        ant:                         NO
        JNI:                         NO
        Java wrappers:               NO
        Java tests:                  NO
    
      Install to:                    /usr
    -----------------------------------------------------------------
    


    The above is summarized as follows.

    Item Description Etc
    Python support both Python 2.7.15 and 3.6.9
    C/C++ support yes
    Java support No
    world module No Ability to build multiple modules in opencv into one world file
    Platform Linux 4.9.140-tegra aarch64, CMake Ver:3.10.2
    CMake version Linux 4.9.140-tegra aarch64
    GUI support GTK+
    Media ZLib, JPEG, WEBP, PNG, TIFF, JPEG 2000, HDR, SUNRASTER, PXM, PFM
    Video FFMPEG, GStreamer, v4l/v4l2
    Third Party Eigen, Custom HAL, Protobuf

    If you look carefully at the above, there is something strange. There is no information about CUDA support. All Jetson series are equipped with NVidia GPU, and JetPack supports CUDA library to use this GPU. However, there is no information on CUDA support in OpenCV.

    Let's check whether CUDA is supported once again.

    root@spypiggy-nano:~# python3
    Python 3.6.9 (default, Oct  8 2020, 12:12:24)
    [GCC 8.4.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import cv2
    >>> print(cv2.cuda.getCudaEnabledDeviceCount())
    0
    

    It can be confirmed once again that CUDA is not supported. So why didn't the NVidia JetPack team support CUDA from OpenCV?

    The reason for this can be found in OpenCV 4.1.1 in Jetpack 4.3 is built without CUDA (GPU support).

    If you look at dusty_nv's reply on this page, he says:

    There are issues with the CUDA code in OpenCV (the CUDA code is actually in opencv_contrib) which result in tests not passing. You are free to build with it enabled if you like, but we are not able to support it.

    That is, it was not applied because of an error that occurred during the build process. If it is possible to build without errors, you can build it yourself.


    Is CUDA Support Required in OpenCV?

    If you mainly use Tensorflow, PyTorch, or TensorRT, there is no reason to use CUDA in OpenCV. This is because these machine learning frameworks support CUDA. Perhaps you will simply use OpenCV to open images, resize them, and reformat images. OpenCV rarely requires CUDA to do this.

    However, if you want to load a machine learning network model directly from OpenCV, input an inference image and get the result, you must use OpenCV dnn module that supports CUDA. In addition, the ability to directly load models of machine learning frameworks such as Tensorflow and PyTorch is supported from OpenCV 4.2. So you must upgrade OpenCV 4.1.1 to 4.2.X or to higher version.

    And in OpenCV, images are objects of class Mat. If you are using C++, use the cv::Mat class, and if you are using Python, use the cv2.Mat class.

    In addition, OpenCV provides additional cv::gpu::GpuMat and cv2.cuda_GpuMat classes that operate in CUDA memory so that objects of the Mat class can be processed quickly in CUDA. 

    Therefore, if you want to use the cv::gpu::GpuMat and cv2.cuda_GpuMat classes, you need to upgrade to OpenCV which also supports CUDA.


    A brief introduction to OpenCV CUDA.

    The following is a test after upgrading OpenCV to 4.5.1 on Jetson Nano's JetPack 4.5. OpenCV 4.1.1 provided by Jetpack 4.5 does not support CUDA, so OpenCV has been upgraded to 4.5.1 for performance comparison. The upgrade method will be explained later.

    OpenCV's CUDA module is still being improved. Therefore, the functions of the existing CPU mode are not 100% usable in CUDA. Since some functions are not implemented, in some cases you may need to use functions that run on the CPU.

    A description of the development background of OpenCV CUDA is available at https://opencv.org/platforms/cuda/. This page provides a table comparing the performance of Intel i5 CPU and Tesla C2050 CUDA. In Primitive Image Processing, it can be seen that the performance difference is up to 30 times. However, there is not a difference in performance in all operations using OpenCV. There are many performance differences in image processing filters that require repetitive numerical calculations. There are cases in which there is little difference in performance in tasks such as changing the image format, or if CUDA is used, the performance is inferior.


    <Performance comparison>


    How to use CUDA in OpenCV requires the following three steps.

    1. upload: Move data to CUDA memory
    2. processing: Use cuda function to proceed with the desired operation
    3. download: Move the finished data to be used again in CPU

    Let's see how to use CUDA in OpenCV with the following simple example code. In this example, the upload process is performed for CUDA processing, but the last download process is omitted. If you want to save the processed video frame result, you can save it through the download process.

    import cv2
    import time
    import argparse
    
    parser = argparse.ArgumentParser()
    parser.add_argument('--gpu', type=int, default = 0)
    opt = parser.parse_args()
    
    video = './see.mp4'
    vsize = (1024,768)
    process_time = 0
    num_frames = 0
    def cuda_process():
        global process_time, num_frames
        count = 1
        vod = cv2.VideoCapture(video)
        num_frames = vod.get(cv2.CAP_PROP_FRAME_COUNT)
        
        ret, frame = vod.read()
        # set scale of resized image
        gpu_frame = cv2.cuda_GpuMat()
        while ret:
            s = time.time()
            gpu_frame.upload(frame)
            resized = cv2.cuda.resize(gpu_frame, vsize)
            luv = cv2.cuda.cvtColor(resized, cv2.COLOR_BGR2LUV)
            hsv = cv2.cuda.cvtColor(resized, cv2.COLOR_BGR2HSV)
            gray = cv2.cuda.cvtColor(resized, cv2.COLOR_BGR2GRAY)
            process_time += (time.time() - s)
            ret, frame = vod.read()
            #print('frame :%d'%count)
            count += 1
        vod.release()
       
    def cpu_process():    
        global process_time, num_frames
        count = 1
        vod = cv2.VideoCapture(video)
        num_frames = vod.get(cv2.CAP_PROP_FRAME_COUNT)
        ret, frame = vod.read()
        while ret:
            s = time.time()
            resized = cv2.resize(frame, vsize)
            luv = cv2.cvtColor(resized, cv2.COLOR_BGR2LUV)
            hsv = cv2.cvtColor(resized, cv2.COLOR_BGR2HSV)
            gray = cv2.cvtColor(resized, cv2.COLOR_BGR2GRAY)
            process_time += (time.time() - s)
            ret, frame = vod.read()
            #print('frame :%d'%count)
            count += 1
        vod.release()
     
    
    if __name__ == "__main__":
        start = time.time()
        if opt.gpu:
            cuda_process()
        else:
            cpu_process()
        end = time.time()
        print('processing time:%6.2f'%(process_time))
        print('total frame[%d] time[%6.2f]'%(num_frames, end - start))
     
    

    <compare.py>

    Let's run the example.

    root@spypiggy-nano:/usr/local/src/study# python3 compare.py --gpu=0
    processing time: 12.90
    total frame[365] time[ 30.55]
    root@spypiggy-nano:/usr/local/src/study# python3 compare.py --gpu=1
    processing time: 15.61
    total frame[365] time[ 33.11]

    Contrary to your expectations, it took a little longer to process if you used CUDA. Perhaps if you added the download process to save the video frame, the time difference could have been a bit more. Even though CUDA was used like this, the reason it took more time is probably because the functions we used such as resize and cvtColor did not show a big difference in performance between CUDA and non-CUDA. Rather, the time difference would have occurred because the upload process was added to the CUDA memory.

    However, there are some functions that can benefit greatly when using CUDA. The following is the second example. This example uses the goodFeaturesToTrack filter. This filter is a filter that finds the coordinates of an edge in an image.

    <image with goodFeaturesToTrack filter applied>


    import cv2
    import time
    import argparse
    
    parser = argparse.ArgumentParser()
    parser.add_argument('--gpu', type=int, default = 0)
    opt = parser.parse_args()
    scale = 0.5
    video = './see.mp4'
    vsize = (1024,768)
    process_time = 0
    num_frames = 0
    def cuda_process():
        global process_time, num_frames
        count = 1
        vod = cv2.VideoCapture(video)
        num_frames = vod.get(cv2.CAP_PROP_FRAME_COUNT)
        ret, frame = vod.read()
        # set scale of resized image
        gpu_frame = cv2.cuda_GpuMat()
        detector = cv2.cuda.createCannyEdgeDetector(100, 200)
        trackdetector = cv2.cuda.createGoodFeaturesToTrackDetector(cv2.CV_8UC1)
        while ret:
            gpu_frame.upload(frame)
            gpu_resized = cv2.cuda.resize(gpu_frame, vsize)
            gpu_gray = cv2.cuda.cvtColor(gpu_resized, cv2.COLOR_BGR2GRAY)
            s = time.time()
            #canny = detector.detect(gpu_gray)
            gpu_coners = trackdetector.detect(gpu_gray)
            process_time += (time.time() - s)
            #cpu_canny = canny.download()
            cpu_coners = gpu_coners.download()
            '''
            cv2.imshow("original", gpu_resized.download())
            cv2.imshow("canny", cpu_canny)
            k = cv2.waitKey(1)
            if k == 27:
                break        
            '''    
            ret, frame = vod.read()
            #print('frame :%d'%count)
            count += 1
        vod.release()
       
    def cpu_process():    
        global process_time, num_frames
        count = 1
        vod = cv2.VideoCapture(video)
        num_frames = vod.get(cv2.CAP_PROP_FRAME_COUNT)
        ret, frame = vod.read()
        while ret:
            resized = cv2.resize(frame, vsize)
            gray = cv2.cvtColor(resized, cv2.COLOR_BGR2GRAY)
            s = time.time()
            #canny = cv2.Canny(gray, 100,200)
            corners = cv2.goodFeaturesToTrack(gray,25,0.01,10)
            process_time += (time.time() - s)
            '''
            cv2.imshow("original", resized)
            cv2.imshow("canny", canny)
            k = cv2.waitKey(1)
            if k == 27:
                break        
            '''    
            ret, frame = vod.read()
            #print('frame :%d'%count)
            count += 1
        vod.release()
     
    
    if __name__ == "__main__":
        start = time.time()
        if opt.gpu:
            cuda_process()
        else:
            cpu_process()
        end = time.time()
        cv2.destroyAllWindows()
        print('processing time:%6.2f'%(process_time))
        print('total frame[%d] time[%6.2f]'%(num_frames, end - start))
     
    

    <compare2.py>


    Let's run the second example.

    root@spypiggy-nano:/usr/local/src/study# python3 compare2.py --gpu=0
    processing time: 17.78
    total frame[365] time[ 41.17]
    root@spypiggy-nano:/usr/local/src/study# python3 compare2.py --gpu=1
    processing time:  8.92
    total frame[365] time[ 34.45]

    Unlike the previous example, you can see that there is a significant performance improvement. In filter processing time, processing time has improved a lot, but it can be seen that some performance loss occurs because upload and download process between CUDA memory and CPU memory is required in total processing time. As you can see from the examples above, using CUDA does not necessarily improve the processing speed. Depending on the OpenCV function you use and the image size, the results will vary a lot.


    Installing OpenCV 4.5.1

    At this point, I will install the latest version of OpenCV, 4.5.1 which supports CUDA, into JetPack 4.5.


    Increase swap memory(Jetson Nano Users)

    If you are using Xavier NX, there is no need to increase the swap memory. When you build a large software packages like openCV, you may experience an out of memory phenomenon. Increasing the swap file size can prevent this malfunction.


    git clone https://github.com/JetsonHacksNano/installSwapfile
    cd installSwapfile
    ./installSwapfile.sh
    

    Above script file will increase 6GB swap files. You can change the swap file size by modifying the scripts. If you want to uninstall the swap setting, open the fstab file and delete the swap file line and reboot. 


     The description from now on can also be applied to Jetson Nano, Xavier NX and TX2 using JetPack 4.5.

     One thing to note is -D CUDA_ARCH_BIN="7.2", which specifies CUDA Architecture in the process of creating Makefile using cmake. Nano, TX2, and Xavier have different values. These values are described in JetsonNano-Useful tips before using Nano

    Compute capability (version)Micro-architectureGPUsTegra,Jetson
    3.2KeplerGK20ATegra K1, Jetson TK1
    5.3MaxwellGM20BTegra X1, Jetson TX1, Jetson Nano
    6.2PascalGP10BTegra X2, Jetson TX2
    7.2VoltaGV10BTegra Xavier, Jetson Xavier NX, Jetson AGX Xavier

    <table from JetsonNano-Useful tips before using Nano>

     
     The following is a script file that can be easily installed if you use Jetson Nano.


    #!/bin/bash
    #
    # Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
    #
    # NVIDIA Corporation and its licensors retain all intellectual property
    # and proprietary rights in and to this software, related documentation
    # and any modifications thereto.  Any use, reproduction, disclosure or
    # distribution of this software and related documentation without an express
    # license agreement from NVIDIA Corporation is strictly prohibited.
    
    # spyjetson(https://spyjetson.blogspot.com/) modified
    
    if [ "$#" -ne 1 ]; then
        echo "Usage: $0 <Install Folder>"
        exit
    fi
    folder="$1"
    user="nvidia"
    passwd="nvidia"
    
    echo "** Remove OpenCV4.1 first"
    sudo sudo apt-get purge *libopencv*
    
    echo "** Install requirement"
    sudo apt-get update
    sudo apt-get install -y build-essential cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
    sudo apt-get install -y libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev
    sudo apt-get install -y python2.7-dev python3-dev python-dev python-numpy python3-numpy
    sudo apt-get install -y libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev
    sudo apt-get install -y libv4l-dev v4l-utils qv4l2 v4l2ucp
    sudo apt-get install -y curl
    sudo apt-get update
    
    echo "** Download opencv-4.5.1"
    cd $folder
    curl -L https://github.com/opencv/opencv/archive/4.5.1.zip -o opencv-4.5.1.zip
    curl -L https://github.com/opencv/opencv_contrib/archive/4.5.1.zip -o opencv_contrib-4.5.1.zip
    unzip opencv-4.5.1.zip 
    unzip opencv_contrib-4.5.1.zip 
    cd opencv-4.5.1/
    
    echo "** Building..."
    mkdir release
    cd release/
    cmake -D WITH_CUDA=ON -D ENABLE_PRECOMPILED_HEADERS=OFF  -D CUDA_ARCH_BIN="5.3" -D CUDA_ARCH_PTX="" -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-4.5.1/modules -D WITH_GSTREAMER=ON -D WITH_LIBV4L=ON -D BUILD_opencv_python2=ON -D BUILD_opencv_python3=ON -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_EXAMPLES=OFF -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local ..
    make -j4
    sudo make install
    
    echo "** Install opencv-4.5.1 successfully"
    echo "** Bye :)"
    

    <opencv4.5_jetsonnano.sh>

    I put the script file on my github. And since the script that changed CUDA_ARCH_BIN value for Jetson Nano and TX2 is also uploaded, you can use the script suitable for the Jetson series you are using.

    Now run the script.

    root@spypiggy-nano:/usr/local/src# ./opencv4.5_jetsonnano.sh ./
    

    The script file needs a parameter for source code download directory. The script file takes a lot of time to build OpenCV 4.5.1. If you are not the root user, you may be asked to enter a password to obtain sudo privileges in the middle.


    root@spypiggy-nano:/usr/local/src# python3
    Python 3.6.9 (default, Oct  8 2020, 12:12:24)
    [GCC 8.4.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import cv2
    >>> print(cv2.getBuildInformation())
    
    General configuration for OpenCV 4.5.1 =====================================
      Version control:               unknown
    
      Extra modules:
        Location (extra):            /usr/local/src/opencv_contrib-4.5.1/modules
        Version control (extra):     unknown
    
      Platform:
        Timestamp:                   2021-02-08T10:22:39Z
        Host:                        Linux 4.9.201-tegra aarch64
        CMake:                       3.10.2
        CMake generator:             Unix Makefiles
        CMake build tool:            /usr/bin/make
        Configuration:               RELEASE
    
      CPU/HW features:
        Baseline:                    NEON FP16
    
      C/C++:
        Built as dynamic libs?:      YES
        C++ standard:                11
        C++ Compiler:                /usr/bin/c++  (ver 7.5.0)
        C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
        C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
        C Compiler:                  /usr/bin/cc
        C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
        C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
        Linker flags (Release):      -Wl,--gc-sections -Wl,--as-needed
        Linker flags (Debug):        -Wl,--gc-sections -Wl,--as-needed
        ccache:                      NO
        Precompiled headers:         NO
        Extra dependencies:          m pthread cudart_static dl rt nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
        3rdparty dependencies:
    
      OpenCV modules:
        To be built:                 alphamat aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python2 python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
        Disabled:                    world
        Disabled by dependency:      -
        Unavailable:                 cnn_3dobj cvv hdf java julia matlab ovis sfm ts viz
        Applications:                apps
        Documentation:               NO
        Non-free algorithms:         NO
    
      GUI:
        GTK+:                        YES (ver 2.24.32)
          GThread :                  YES (ver 2.56.4)
          GtkGlExt:                  NO
        VTK support:                 NO
    
      Media I/O:
        ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
        JPEG:                        libjpeg-turbo (ver 2.0.6-62)
        WEBP:                        build (ver encoder: 0x020f)
        PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.34)
        TIFF:                        build (ver 42 - 4.0.10)
        JPEG 2000:                   build (ver 2.3.1)
        OpenEXR:                     build (ver 2.3.0)
        HDR:                         YES
        SUNRASTER:                   YES
        PXM:                         YES
        PFM:                         YES
    
      Video I/O:
        DC1394:                      NO
        FFMPEG:                      YES
          avcodec:                   YES (57.107.100)
          avformat:                  YES (57.83.100)
          avutil:                    YES (55.78.100)
          swscale:                   YES (4.8.100)
          avresample:                NO
        GStreamer:                   YES (1.14.5)
        v4l/v4l2:                    YES (linux/videodev2.h)
    
      Parallel framework:            pthreads
    
      Trace:                         YES (with Intel ITT)
    
      Other third-party libraries:
        Lapack:                      NO
        Eigen:                       YES (ver 3.3.4)
        Custom HAL:                  YES (carotene (ver 0.0.1))
        Protobuf:                    build (3.5.1)
    
      NVIDIA CUDA:                   YES (ver 10.2, CUFFT CUBLAS)
        NVIDIA GPU arch:             53
        NVIDIA PTX archs:
    
      cuDNN:                         YES (ver 8.0.0)
    
      OpenCL:                        YES (no extra features)
        Include path:                /usr/local/src/opencv-4.5.1/3rdparty/include/opencl/1.2
        Link libraries:              Dynamic load
    
      Python 2:
        Interpreter:                 /usr/bin/python2.7 (ver 2.7.17)
        Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.17)
        numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.3)
        install path:                lib/python2.7/dist-packages/cv2/python-2.7
    
      Python 3:
        Interpreter:                 /usr/bin/python3 (ver 3.6.9)
        Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.6m.so (ver 3.6.9)
        numpy:                       /usr/lib/python3/dist-packages/numpy/core/include (ver 1.13.3)
        install path:                lib/python3.6/dist-packages/cv2/python-3.6
    
      Python (for build):            /usr/bin/python2.7
    
      Java:
        ant:                         NO
        JNI:                         NO
        Java wrappers:               NO
        Java tests:                  NO
    
      Install to:                    /usr/local
    -----------------------------------------------------------------
    


    You can see that the newly built OpenCV 4.5.1 supports CUDA.


    Wrapping Up

     It is not difficult to upgrade OpenCV to OpenCV 4.5.1 on JetPack 4.5. If you need to use CUDA or use the dnn module in OpenCV, it is recommended to upgrade to 4.5.1. However, if you mainly use machine learning frameworks such as Tensorflow, PyTorch and use OpenCV for simple purposes for image input/output, it is sufficient to use version 4.1.1 without upgrading.