레이블이 OpenCV인 게시물을 표시합니다. 모든 게시물 표시
레이블이 OpenCV인 게시물을 표시합니다. 모든 게시물 표시

2023년 3월 13일 월요일

Installing the Latest Version of OpenCV (ver 4.7) on Xavier NX Anaconda Enviromment(JetPack 5.1)

 

JetPack 5.1 comes with OpenCV 4.5 installed. 

However, many developers, including myself, prefer Python virtual environments using Anaconda. New OpenCV can be installed by creating a new virtual environment in Anaconda. 

The advantage of this method is that existing OpenCV 4.5 can be used outside the virtual environment. 

In this article, I will explain how to install the fosine version of OpenCV in anaconda virtual environment while maintaining the existing OpenCV 4.5. As of March 2023, the latest version of OpenCV is 4.7. However, it is not difficult for you to update the existing OpenCV 4.5 to the newly built OpenCV 4.7. I will explain this method as well.


Build the latest version of OpenCV in Anaconda

Prerequisites

You should install JetPack 5.1 in advance. And anaconda should also be installed.


Creating a Virtual Environment for OpenCV Installation

First, create a Python virtual environment. I will build and install a new OpenCV in this virtual environment.


# Then Create the env again
spypiggy@spypiggy-NX:~ $ conda create -n yolov8 python=3.8.16
spypiggy@spypiggy-NX:~ $ conda activate yolov8


Then execute the following script. This script stores the required package installation and the process of downloading, compiling, and installing OpenCV 4.7 source code in one file.

You can download this script at my repo(https://github.com/raspberry-pi-maker/NVIDIA-Jetson/tree/master/useful_scripts)

# part of the script to be modified
# 1. opencv version to install. The current version is 4.7.0
# 2. python version of anaconda virtual environment. The current version is 3.8
# 3. CUDA_ARCH_BIN. The Current value is "7.2" for Xavier NX


echo "OpenCV 4.7 Installation script for Jetson Xavier NX (Jetpack 5.1)"

if [ "$#" -ne 1 ]; then
    echo "Usage: $0 <Install Folder>"
    exit
fi
folder="$1"

#You don't have to remove pre-installed OpenCV 4.5
#echo "** Remove Old OpenCV first"
#sudo apt-get purge *libopencv*

echo "** Install requirement"
sudo apt-get update
sudo apt-get install -y build-essential cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
sudo apt-get install -y libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev
sudo apt-get install -y libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev
sudo apt-get install -y libopenblas-dev libatlas-base-dev libblas-dev 
sudo apt-get install -y libeigen3-dev libhdf5-dev libgflags-dev
sudo apt-get install -y libv4l-dev v4l-utils qv4l2 v4l2ucp
sudo apt-get update
conda install -y numpy

echo "** Download opencv-4.7.0"
cd $folder
wget https://github.com/opencv/opencv/archive/4.7.0.zip 
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.7.0.zip 

unzip 4.7.0.zip 
unzip opencv_contrib.zip 
cd opencv-4.7.0/

echo "** Building..."
mkdir release
cd release
cmake \
-D ENABLE_PRECOMPILED_HEADERS=0  \
-D CUDA_ARCH_BIN="7.2" \
-D CUDA_ARCH_PTX="" \
-D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-4.7.0/modules \
-D WITH_OPENCL=0 \
-D WITH_CUDA=1 \
-D OPENCV_DNN_CUDA=1 \
-D EIGEN_INCLUDE_PATH=/usr/include/eigen3 \
-D WITH_EIGEN=1 \
-D WITH_CUBLAS=1 \
-D WITH_FFMPEG=1 \
-D WITH_GSTREAMER=1 \
-D WITH_V4L=1 \
-D WITH_LIBV4L=1 \
-D BUILD_opencv_python2=0 \
-D BUILD_opencv_python3=1 \
-D BUILD_TESTS=0 \
-D BUILD_PERF_TESTS=0 \
-D BUILD_EXAMPLES=0 \
-D OPENCV_GENERATE_PKGCONFIG=1 \
-D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
-D PYTHON3_LIBRARY=$CONDA_PREFIX/lib/python3.8 \
-D PYTHON3_INCLUDE_DIR=$CONDA_PREFIX/include/python3.8 \
-D PYTHON3_EXECUTABLE=$CONDA_PREFIX/bin/python \
-D PYTHON3_PACKAGES_PATH=$CONDA_PREFIX/lib/python3.8/site-packages \
..

make -j6
sudo make install

echo "** Install opencv-4.7.0 successfully"
echo "** Bye :)"

<install_opencv4.7_NX.sh>

When using an anaconda virtual environment, you need to know the Python path used. I created a virtual environment named yolov8. Then there are execution Python and installation packages in the following directories. The virtual environment Python path is stored in $CONDA_PREFIX.

(base) spypiggy@spypiggy-NX:/usr/lib/aarch64-linux-gnu$ cd ~/src
(base) spypiggy@spypiggy-NX:~/src$ conda activate yolov8
(yolov8) spypiggy@spypiggy-NX:~$ echo $CONDA_PREFIX
/home/spypiggy/anaconda3/envs/yolov8


  • Python executable path: /home/spypiggy/anaconda3/env/yolov8/bin/python
  • Python packages path: /home/spypiggy/anaconda3/env/yolov8/lib/python3.8/site-packages


The part of the script above that you should pay attention to is the red text.

This is the part where you set the anaconda python related build environment variable while creating a makefile with the cmake command. These environment variables should specify Anaconda's virtual environment Python.

Now run the script for building OpenCV 4.7 for Anaconda Virtual Environment!

I copied install_opencv4.7_NX.sh to ~/src directory first. This build process will take more than an hour.

(yolov8) spypiggy@spypiggy-NX:~$ cd ~/src
(yolov8) spypiggy@spypiggy-NX:~/src$ bash install_opencv4.7_NX.sh


If it ended well without errors, it is anacorn as follows. The cv2 directory should be created in the site-packages directory of the virtual environment.

(yolov8) spypiggy@spypiggy-NX:~/anaconda3/envs/yolov8/lib/python3.8/site-packages$ pwd
/home/spypiggy/anaconda3/envs/yolov8/lib/python3.8/site-packages
(yolov8) spypiggy@spypiggy-NX:~/anaconda3/envs/yolov8/lib/python3.8/site-packages$ ls -ald cv2
drwxr-xr-x 7 root root 4096  3월 13 08:33 cv2


libffi patch

Now, let's test the cv2 package to see if it works well.

(yolov8) spypiggy@spypiggy-NX:~$ python
Python 3.8.16 (default, Mar  2 2023, 03:16:31)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/spypiggy/anaconda3/envs/yolov8/lib/python3.8/site-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/home/spypiggy/anaconda3/envs/yolov8/lib/python3.8/site-packages/cv2/__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
  File "/home/spypiggy/anaconda3/envs/yolov8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /lib/aarch64-linux-gnu/libp11-kit.so.0: undefined symbol: ffi_type_pointer, version LIBFFI_BASE_7.0
>>>

Something went wrong. After several googling, libffi is the cause of this error.I could see that it was related to so.

Look up libffi related files in the /home/spypiggy/anaconda3/envs/yolov8/lib/ directory. You can see that the libffi.7.so file and the libffi.so.7 file are linked to version 8. You'll feel a little strange.

(yolov8) spypiggy@spypiggy-NX:~/anaconda3$ ls -al envs/yolov8/lib/libffi*
lrwxrwxrwx 1 spypiggy spypiggy     15  3월 12 16:22 envs/yolov8/lib/libffi.7.so -> libffi.so.8.1.0
lrwxrwxrwx 1 spypiggy spypiggy     15  3월 12 16:22 envs/yolov8/lib/libffi.8.so -> libffi.so.8.1.0
-rw-rw-r-- 2 spypiggy spypiggy 114688 11월 22 23:17 envs/yolov8/lib/libffi.a
lrwxrwxrwx 1 spypiggy spypiggy     15  3월 12 16:22 envs/yolov8/lib/libffi.so -> libffi.so.8.1.0
lrwxrwxrwx 1 spypiggy spypiggy     15  3월 12 16:22 envs/yolov8/lib/libffi.so.7 -> libffi.so.8.1.0
lrwxrwxrwx 1 spypiggy spypiggy     15  3월 12 16:22 envs/yolov8/lib/libffi.so.8 -> libffi.so.8.1.0
-rwxrwxr-x 2 spypiggy spypiggy 104400 11월 22 23:17 envs/yolov8/lib/libffi.so.8.1.0
(yolov8) spypiggy@spypiggy-NX:~/anaconda3


This time, let's look for these files in the /home/spypiggy/anaconda3/lib directory.

(yolov8) spypiggy@spypiggy-NX:~/anaconda3/lib$ ls -al  libffi*
lrwxrwxrwx 1 spypiggy spypiggy    15  3월  9 20:54 libffi.so -> libffi.so.7.1.0*
lrwxrwxrwx 1 spypiggy spypiggy    15  3월  9 20:54 libffi.so.6 -> libffi.so.7.1.0*
lrwxrwxrwx 1 spypiggy spypiggy    15  3월  9 20:54 libffi.so.7 -> libffi.so.7.1.0*
-rwxrwxr-x 2 spypiggy spypiggy 50376  3월  7  2021 libffi.so.7.1.0*

The libffi version 7 exists in this directory, and the symbolic link is also linked to the version 7.

It is not known what causes the symbolic links to change, but restoring these links solves the problem.

Copy the /home/spypiggy/anaconda3/lib/libffi.so.7.1.0 file to the /home/spypiggy/anaconda3/envs/yolov8/lib/ directory, and then recreate the symbolic link.

lrwxrwxrwx 1 spypiggy spypiggy     15  3월 13 17:09 libffi.7.1.0 -> libffi.so.7.1.0
lrwxrwxrwx 1 spypiggy spypiggy     15  3월 13 18:03 libffi.7.so -> libffi.so.7.1.0
lrwxrwxrwx 1 spypiggy spypiggy     15  3월 12 16:22 libffi.8.so -> libffi.so.8.1.0
-rw-rw-r-- 2 spypiggy spypiggy 114688 11월 22 23:17 libffi.a
lrwxrwxrwx 1 spypiggy spypiggy     15  3월 12 16:22 libffi.so -> libffi.so.8.1.0
-rw-r--r-- 1 spypiggy spypiggy  50376  3월 13 23:06 libffi.so.7.1.0
lrwxrwxrwx 1 spypiggy spypiggy     15  3월 13 22:50 libffi.so.8 -> libffi.so.8.1.0
-rw-rw-r-- 2 spypiggy spypiggy  35144  3월 13 17:02 libffi.so.8.1.0

<after correction>

The libffi issue may be related to the version of OpenCV being built. When a lower version than the latest version 4.7 is built, the above error may not occur.

Now let's check out the built cv2 again.

(yolov8) spypiggy@spypiggy-NX:~/src/yolov8$ python
Python 3.8.16 (default, Mar  2 2023, 03:16:31)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.7.0'
>>> print(cv2.getBuildInformation())

General configuration for OpenCV 4.7.0 =====================================
  Version control:               unknown

  Extra modules:
    Location (extra):            /home/spypiggy/src/opencv_contrib-4.7.0/modules
    Version control (extra):     unknown

  Platform:
    Timestamp:                   2023-03-12T16:05:37Z
    Host:                        Linux 5.10.104-tegra aarch64
    CMake:                       3.16.3
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    NEON FP16

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                /usr/bin/c++  (ver 9.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined
    Linker flags (Debug):        -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          m pthread cudart_static dl rt nppc nppial nppicc nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 alphamat aruco barcode bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cvv java julia matlab ovis python2 sfm ts viz
    Applications:                apps
    Documentation:               NO
    Non-free algorithms:         NO

  GUI:                           GTK2
    GTK+:                        YES (ver 2.24.32)
      GThread :                  YES (ver 2.64.6)
      GtkGlExt:                  NO
    VTK support:                 NO

  Media I/O:
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.37)
    TIFF:                        build (ver 42 - 4.2.0)
    JPEG 2000:                   build (ver 2.4.0)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES
      avcodec:                   YES (58.54.100)
      avformat:                  YES (58.29.100)
      avutil:                    YES (56.31.100)
      swscale:                   YES (5.5.100)
      avresample:                NO
    GStreamer:                   YES (1.16.3)
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            pthreads

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Lapack:                      NO
    Eigen:                       YES (ver 3.3.7)
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.19.1)

  NVIDIA CUDA:                   YES (ver 11.4, CUFFT CUBLAS)
    NVIDIA GPU arch:             72
    NVIDIA PTX archs:

  cuDNN:                         YES (ver 8.6.0)

  Python 3:
    Interpreter:                 /home/spypiggy/anaconda3/envs/yolov8/bin/python (ver 3.8.16)
    Libraries:                   /home/spypiggy/anaconda3/envs/yolov8/lib/python3.8 (ver 3.8.16)
    numpy:                       /home/spypiggy/anaconda3/envs/yolov8/lib/python3.8/site-packages/numpy/core/include (ver 1.23.5)
    install path:                /home/spypiggy/anaconda3/envs/yolov8/lib/python3.8/site-packages/cv2/python-3.8

  Python (for build):            /usr/bin/python2.7

  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /home/spypiggy/anaconda3/envs/yolov8
-----------------------------------------------------------------

You can see that it is neatly built. The red text shows that ffmpeg is available for video processing and NVidia CUDA GPU is available.

And finally, you can see that Python 3 is linked to the Anaconda virtual environment Python we wanted.


Wrapping up

Because Xavier NX can add NVME SSDs, it can configure enough storage space compared to Jetson Nano. It is advantageous to use the Anaconda virtual environment to fully utilize the storage secured using SSD. After creating multiple virtual environments on SSDs, you can install and use ML frameworks such as PyTorch, Tensorflow, and YOLO in the desired virtual environment.

However, OpenCV provided by Anaconda excludes ffmpeg from video processing. And CUDA acceleration is also not supported. To solve this problem, it is recommended to build an OpenCV that can be used in Python in an anaconda virtual environment.

Please refer to the above and build your own OpenCV for Anaconda optimized for Xavier NX.


2023년 3월 10일 금요일

Xavier NX - YOLOv8 Object Detection (JetPack 5.1)

 

In the last two posts, I explained the process of installing Jetpack 5.1 on Xavier NX and installing YOLOv8.


Prerequsites


YOLOv8 is still being updated. Ultralytics, which released VOLOv8, is continuously releasing updated VOLOv8 on github. Therefore, it is recommended that you also update it from time to time.
The update method is simple. Since we installed the ultralytics package using pip in the anaconda virtual environment, you can also use the pip command to update.

pip install ultrslytics --upgrade


YOLOv8 Object Detection Models

VOLOv8 provides pre-trained models. These models were trained using the COCO dataset.

<YOLOv8 pre-trained models>

As shown in the table above, the YOLOv8n model is the lightest model and the YOLOv8x model is the heaviest. The lighter the model, the less memory is used and the processing speed is faster. But the accuracy is poor.

You can collect training data and create your own custom models, but in this article I will use pre-trained models.

COCO Dataset

COCO is a large-scale object detection, segmentation, and captioning dataset. COCO has several features:

  • Object segmentation
  • Recognition in context
  • Superpixel stuff segmentation
  • 330K images (>200K labeled)
  • 1.5 million object instances
  • 80 object categories
  • 91 stuff categories
  • 5 captions per image
  • 250,000 people with keypoints
Many machine learning models are trained using the COCO dataset for performance evaluation. The pre-trained models of YOLOv8 are also models trained using the COCO dataset.
Object Detection of the COCO model is classified into a total of 80.


<COCO model 80 labels>


The YOLOv8 pre-trained model can also find 80 objects according to this classification.
Labeling values can be checked in the coco.yaml file.


YOLOv8 Object Detection

YOLOv8 provides two methods of object detection. The first method is to use the cli command and the second method is to use the Python API.

The document about YOLOv8 object recognition is https://docs.ultralytics.com/tasks/detection/.


CLI command

This is a method using the yolo command provided by VOLOv8. You can use the yolo command to perform the same functions as the Python API, such as learning, validation, and prediction. 

yolo detect predict model=yolov8n.pt source="https://ultralytics.com/images/bus.jpg"  # predict with official model
yolo detect predict model=path/to/best.pt source="https://ultralytics.com/images/bus.jpg"  # predict with custom model

The basic CLI command usage is as follows. We will use the above line because I will be using an pre-trained model.

If you test using ssh without X11 forwarding, it is better to save the result because it is difficult to check the image directly.


(base) spypiggy@spypiggy-NX:~$ conda activate yolov8
(yolov8) spypiggy@spypiggy-NX:~$ yolo detect predict model=yolov8n.pt source="https://ultralytics.com/images/bus.jpg" save=True show=False
Ultralytics YOLOv8.0.51 🚀 Python-3.8.16 torch-1.14.0a0+44dac51c.nv23.02 CUDA:0 (Xavier, 6857MiB)
YOLOv8n summary (fused): 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs

Downloading https://ultralytics.com/images/bus.jpg to bus.jpg...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 476k/476k [00:00<00:00, 3.94MB/s]
image 1/1 /home/spypiggy/bus.jpg: 640x480 4 persons, 1 bus, 1 stop sign, 86.2ms
Speed: 2.4ms preprocess, 86.2ms inference, 23.2ms postprocess per image at shape (1, 3, 640, 640)
Results saved to runs/detect/predict


You can see that the resulting image is stored in ./runs/detect/predict/bus.jpg. If you open the file, you can see the image like this.

<YOLOv8 detected image - /runs/detect/predict/bus.jpg>

If you test in the GUI environment of Xavier NX, you can directly check the result by changing the show option to True.


Python API

Here's how to use the Python API. Using the Python API, various application programs can be developed. 

The following is the simplest example using the Python API. Presumably all the important predictions are stored in results.

from ultralytics import YOLO

# Load a model
model = YOLO("yolov8n.pt")  # load an official model

# Predict with the model
results = model("https://ultralytics.com/images/bus.jpg")  # predict on an image


The results stores as many results as the number of input inferences. Therefore, if only one image is input, it is a list structure with one result value. And the element of the list type is 'ultralytics.yolo.engine.results.Results'.

The following code outputs box coordinates, object classification, confidence, etc. from the detection result.

from ultralytics import YOLO

# Load a model
model = YOLO("yolov8n.pt")  # load an official model
# Predict with the model
results = model("https://ultralytics.com/images/bus.jpg")  # predict on an image

for result in results:
    for box in result.boxes.data:
        print("x1:%f y1:%f  x2[%f] y2[%f] Conf[%f] Label[%f]"%(box[0], box[1], box[2], box[3], box[4], box[5]))

<sample_detect.py>


If you run the code, you can see the box coordinates, confidence, and label index.

(yolov8) spypiggy@spypiggy-NX:~/src/yolov8$ python sample_detect.py

Found https://ultralytics.com/images/bus.jpg locally at bus.jpg
image 1/1 /home/spypiggy/src/yolov8/bus.jpg: 640x480 4 persons, 1 bus, 1 stop sign, 85.5ms
Speed: 2.3ms preprocess, 85.5ms inference, 9.1ms postprocess per image at shape (1, 3, 640, 640)
x1:17.000000 y1:231.000000  x2[801.000000] y2[769.000000] Conf[0.870380] Label[5.000000]
x1:49.000000 y1:399.000000  x2[244.000000] y2[903.000000] Conf[0.868917] Label[0.000000]
x1:670.000000 y1:380.000000  x2[810.000000] y2[875.000000] Conf[0.852670] Label[0.000000]
x1:221.000000 y1:406.000000  x2[345.000000] y2[857.000000] Conf[0.818634] Label[0.000000]
x1:0.000000 y1:255.000000  x2[32.000000] y2[325.000000] Conf[0.347606] Label[11.000000]
x1:0.000000 y1:551.000000  x2[67.000000] y2[874.000000] Conf[0.281894] Label[0.000000]

The most important data is boxes.data, where coordinates, confidence, and label information are all stored.

Now let's draw these values to the image like the yolo CLI command. 

The ultralytics.yolo.engine.results.Results object contains original image information in orig_img. Since this value is a numpy array type, it can be used directly as a Mat type in OpenCV.

from ultralytics import YOLO
import cv2

colors = [(255,0 , 0), (0,255,0), (0,0,255)]
font = cv2.FONT_HERSHEY_SIMPLEX   
def draw(img, boxes):
    index = 0
    for box in boxes.data:
        p1 =  (int(box[0].item()), int(box[1].item()))
        p2 =  (int(box[2].item()), int(box[3].item()))
        img = cv2.rectangle(img, p1, p2, colors[index % len(colors)], 3)
        text = label_map[int(box[5].item())] + " %4.2f"%(box[4].item()) 
        cv2.putText(img, text, (p1[0], p1[1] - 10), font, fontScale = 1, color = colors[index % len(colors)], thickness = 2)
        index += 1
    cv2.imwrite("./result.jpg", img)    
    # cv2.imshow("draw", img)
    # cv2.waitKey(0)
    # cv2.destroyAllWindows()


# Load a model
model = YOLO("yolov8n.pt")  # load an official model
label_map = model.names
# Predict with the model
results = model("https://ultralytics.com/images/bus.jpg")  # predict on an image

count = len(results)

for result in results:
    draw(result.orig_img, result.boxes)

<sample_detect2.py>


Let's run and check the output result.jpg.

<result.jpg>

Finally, VOLOv8 using Python also got the same result!


Python API and YOLOv8 inference type

In the previous examples, the result was obtained by directly inserting the image file into the YOLO model.
But since I use OpenCV a lot, most of the time I want to open an image using OpenCV and then pass the Mat object to the YOLO model. 


from ultralytics import YOLO
import cv2

colors = [(255,0 , 0), (0,255,0), (0,0,255)]
font = cv2.FONT_HERSHEY_SIMPLEX   
def draw(img, boxes):
    index = 0
    for box in boxes.data:
        p1 =  (int(box[0].item()), int(box[1].item()))
        p2 =  (int(box[2].item()), int(box[3].item()))
        img = cv2.rectangle(img, p1, p2, colors[index % len(colors)], 3)
        text = label_map[int(box[5].item())] + " %4.2f"%(box[4].item()) 
        cv2.putText(img, text, (p1[0], p1[1] - 10), font, fontScale = 1, color = colors[index % len(colors)], thickness = 2)
        index += 1
    cv2.imwrite("./result2.jpg", img)    
    # cv2.imshow("draw", img)
    # cv2.waitKey(0)
    # cv2.destroyAllWindows()


# Load a model
model = YOLO("yolov8n.pt")  # load an official model
label_map = model.names

img = cv2.imread("./bus.jpg", cv2.IMREAD_COLOR)
results = model(img)  # predict on an OpenCV mat object

for result in results:
    draw(result.orig_img, result.boxes)

<simple_detect3.py>


As you can see from the code above, you can pass an OpenCV mat object as an input parameter instead of a file name, and the result is the same as passing a file name.


The types of input sources that can be received in YOLOv8 are as follows. You can see that it provides various input sources such as Python PIL, OpenCV, and numpy as well as file names.


Running YOLOv8 models using torchvision

In the above document, one of the input sources is a torch tensor. However, as of March 2023, this feature does not appear to have been implemented.

The ultralytics Githib issue page has the following article:


Therefore, if you use torchvision, you must convert the tensor type image to PIL or np.array format until YOLOV8 properly supports tensor type source. In the example below, I open an image file using torchvision and then convert the image tensor to PIL format and feed it to the model.


import torch
import torchvision as tv
from ultralytics import YOLO
import cv2
import torchvision.transforms as T

colors = [(255,0 , 0), (0,255,0), (0,0,255)]
font = cv2.FONT_HERSHEY_SIMPLEX   
def draw(img, boxes):
    index = 0
    for box in boxes.data:
        p1 =  (int(box[0].item()), int(box[1].item()))
        p2 =  (int(box[2].item()), int(box[3].item()))
        img = cv2.rectangle(img, p1, p2, colors[index % len(colors)], 3)
        text = label_map[int(box[5].item())] + " %4.2f"%(box[4].item()) 
        cv2.putText(img, text, (p1[0], p1[1] - 10), font, fontScale = 1, color = colors[index % len(colors)], thickness = 2)
        index += 1
    cv2.imwrite("./result3.jpg", img)    


# Load a model
model = YOLO("yolov8n.pt")  # load an official model
label_map = model.names


img = tv.io.read_image("./bus.jpg")
img = T.ToPILImage()(img)
results = model(img)  # predict on an image

for result in results:
    draw(result.orig_img, result.boxes)

<simple_detect4.py>

If you open and check the result3.jpg file, you can see that an image like result2.jpg has been created.


Running YOLOv8 models directly from OpenCV

In the 2021 article "Running OpenPose models directly from OpenCV", it was explained that since OpenCV 4.2, various network models can be used directly in OpenCV. 

I installed OpenCV using Anaconda. You should first check whether the version of OpenCV you are using can use the dnn function to directly load the network model. Unfortunately, the OpenCV of Anaconda we installed does not support the dnn function.

You can check it using OpenCV's cv2.getBuildInformation() function.

  ......
OpenCV modules:
    To be built:                 alphamat aruco bgsegm bioinspired calib3d ccalib core cvv datasets dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      barcode dnn_objdetect dnn_superres mcc text wechat_qrcode
    Unavailable:                 cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev dnn java julia matlab ovis python2 sfm ts viz
    Applications:                -
    Documentation:               NO
    Non-free algorithms:         NO
......

<cv2.getBuildInformation() output of anaconda opencv>


And this is output of Xavier NX built in OpenCV's output.

  ......
 OpenCV modules:
    To be built:                 calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo python2 python3 stitching ts video videoio
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 java
    Applications:                tests perf_tests examples apps
    Documentation:               NO
    Non-free algorithms:         NO
......

<cv2.getBuildInformation() output of Jetson built in opencv>

It can be seen that OpenCV's dnn cannot be used in the anaconda environment we are currently using.


Wrapping up

In the previous article, we learned about installing YOLOv8 on Xavier NX, and in this article, we learned how to use the CLI commands and Python API provided by YOLOv8.

In the next article, we will look at YOLOv8 processing speed comparison and video processing in Xavier NX.

You can downlaod the source code at my Github.