2022년 3월 13일 일요일

OpenCV camera control on Raspberry Pi BullsEye OS

 Fall 2021 Raspberry Pi OS moved from Buster to BullsEye. This was done in line with the upgrade of the Debian OS, which is the basis of the Raspberry Pi OS. And finally, in February 2022, the official version of the Raspberry Pi OS 64-bit was released. However, when the Raspberry Pi OS changed to BullsEye, the camera control part changed too radically. In the article introducing the 64-bit version, I mentioned the precautions for changing the camera control part, but let's take a closer look here and see how to use OpenCV in a new way. If you are using a USB webcam, this article is not of much help. This article is for users who use CSI cameras.


CSI Camera Control on Raspberry Pi OS BullsEye

The Raspberry Pi OS refers to the CSI camera as a Legacy camera. And this camera control was developed for SoC of Broadcom, the chipset supplier of Raspberry Pi. So it's a fairly closed structure. As the Raspberry Pi Foundation changed the OS to BullsEye, the camera control was replaced with an open libcamera, causing compatibility issues with existing software. I wonder if maybe they're trying to change their chipset vendor from Broadcom to something else. The following is the official page for the Raspberry Pi camera. https://www.raspberrypi.com/documentation/accessories/camera.html. A verbose account of why we're moving to the open source libcamera. 

The PiCamera module, which was used for existing Python development, can no longer be used right now. This module is of course not available in the new libcamera as it uses the old closed legacy camera control. The Raspberry Pi Foundation is making PiCamera2 for libcamera. You can download the beta version of picamera2 developed up to now (2022.03) from github (https://github.com/raspberrypi/picamera2) and test it. However, since it is impossible to keep the API 100% the same, it is unavoidable to modify an existing Python program for libcamera.  Since there are libcamera programs corresponding to the previously used raspicam program, there is no problem with previewing, taking pictures, and taking videos in the command window.

The following are libcamera programs that can replace the existing raspistill and raspivid.

  • libcamera-hello A simple "hello world" application which starts a camera preview stream and displays it on the screen.

  • libcamera-jpeg A simple application to run a preview window and then capture high resolution still images.

  • libcamera-still A more complex still image capture application which emulates more of the features of raspistill.

  • libcamera-vid A video capture application.

  • libcamera-raw A basic application for capturing raw (unprocessed Bayer) frames directly from the sensor.

  • libcamera-detect This application is not built by default, but users can build it if they have TensorFlow Lite installed on their Raspberry Pi. It captures JPEG images when certain objects are detected.


But for developers, it's a completely different story. In a situation that made all existing programs unusable, the Raspberry Pi Foundation had no choice but to improve raspi-config so that BullsEye can use the existing Legacy camera. Note that libcamera and legacy cameras cannot be used at the same time. Enabling Interface Options/Legacy Camera in raspi-config will revert to legacy legacy camera mode. Disable camera, not enable option as before. Therefore, this option should not be enabled when using libcamera.

<YouTube explaining how to return to Legacy mode>


If you are using a camera solution, we recommend that you proceed with caution when upgrading to BullsEye OS.


OpenCV on BullsEye

BullsEye exists in both 32-bit and 64-bit OS, but in the test below, we will proceed with the 64-bit standard. In the introduction of 64-bit OS, I mentioned the advantages of using a 64-bit OS, and it is recommended to use a 64-bit OS if possible.


Install OpenCV using pip

There is an easy way to install OpenCV from BullsEye. How to use the python pip command. As of March 2022, the latest version 4.5.5 is being installed.

pi@raspberrypi8GB:~ $ pip3 install opencv-python
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting opencv-python
  Downloading opencv_python-4.5.5.64-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (39.2 MB)
     |████████████████████████████████| 39.2 MB 17 kB/s 
Requirement already satisfied: numpy>=1.19.3 in /usr/lib/python3/dist-packages (from opencv-python) (1.19.5)
Installing collected packages: opencv-python
Successfully installed opencv-python-4.5.5.64


And let's look at the build information of the packages installed using pip. The cv2.getBuildInformation() function tries to output build information.


pi@raspberrypi8GB:~ $ python
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print(cv2.getBuildInformation())

General configuration for OpenCV 4.5.5 =====================================
  Version control:               4.5.5-dirty

  Platform:
    Timestamp:                   2022-03-04T10:02:17Z
    Host:                        Linux 4.9.140-tegra aarch64
    CMake:                       3.22.2
    CMake generator:             Unix Makefiles
    CMake build tool:            /bin/gmake
    Configuration:               Release

  CPU/HW features:
    Baseline:                    NEON FP16

  C/C++:
    Built as dynamic libs?:      NO
    C++ standard:                11
    C++ Compiler:                /opt/rh/devtoolset-10/root/usr/bin/c++  (ver 10.2.1)
    C++ flags (Release):         -Wl,-strip-all   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -Wl,-strip-all   -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /opt/rh/devtoolset-10/root/usr/bin/cc
    C flags (Release):           -Wl,-strip-all   -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -Wl,-strip-all   -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -L/root/ffmpeg_build/lib  -Wl,--gc-sections -Wl,--as-needed  
    Linker flags (Debug):        -L/root/ffmpeg_build/lib  -Wl,--gc-sections -Wl,--as-needed  
    ccache:                      YES
    Precompiled headers:         NO
    Extra dependencies:          /lib64/libopenblas.so Qt5::Core Qt5::Gui Qt5::Widgets Qt5::Test Qt5::Concurrent /lib64/libz.so dl m pthread rt
    3rdparty dependencies:       libprotobuf ade ittnotify libjpeg-turbo libwebp libpng libtiff libopenjp2 IlmImf quirc tegra_hal

  OpenCV modules:
    To be built:                 calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo python3 stitching video videoio
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 java python2 ts
    Applications:                -
    Documentation:               NO
    Non-free algorithms:         NO

  GUI:                           QT5
    QT:                          YES (ver 5.15.0 )
      QT OpenGL support:         NO
    GTK+:                        NO
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /lib64/libz.so (ver 1.2.7)
    JPEG:                        libjpeg-turbo (ver 2.1.2-62)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         build (ver 1.6.37)
    TIFF:                        build (ver 42 - 4.2.0)
    JPEG 2000:                   build (ver 2.4.0)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES
      avcodec:                   YES (58.134.100)
      avformat:                  YES (58.76.100)
      avutil:                    YES (56.70.100)
      swscale:                   YES (5.9.100)
      avresample:                NO
    GStreamer:               NO
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            pthreads

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Lapack:                      YES (/lib64/libopenblas.so)
    Eigen:                       NO
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.19.1)

  OpenCL:                        YES (no extra features)
    Include path:                /io/opencv/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 3:
    Interpreter:                 /opt/python/cp36-cp36m/bin/python3.6 (ver 3.6.15)
    Libraries:                   libpython3.6m.a (ver 3.6.15)
    numpy:                       /opt/python/cp36-cp36m/lib/python3.6/site-packages/numpy/core/include (ver 1.19.3)
    install path:                python/cv2/python-3

  Python (for build):            /bin/python2.7

  Java:                          
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /io/_skbuild/linux-aarch64-3.6/cmake-install
-----------------------------------------------------------------

There are a few things to watch out for above. GStreamer values from Video I/O. For OpenCV 4.5.5 installed with pip, this value is No. That is, I didn't use "-D WITH_GSTREAMER=ON" in the OpenCV build options. Therefore, this package cannot be used with GStreamer.

Now, we will use this package to control the Raspberry Pi CSI camera. I have only connected one CSI camera to the Raspberry Pi, and the sample code is a simple camera preview python code. This simple Python code uses the cv2.VideoCapture(0) function to open a CSI camera and then displays it on the screen.

import cv2
import sys
cap = cv2.VideoCapture(0)
if cap.isOpened() == False:
    print('camera open Failed')
    sys.exit(0)

cap.set(cv2.CAP_PROP_FRAME_WIDTH,640);
cap.set(cv2.CAP_PROP_FRAME_HEIGHT,480);

while True:

    succes, img = cap.read()
    if succes == False:
        print('camera read Failed')
        sys.exit(0)
    k = cv2.waitKey(1)
    if k == ord('q'):
        break
    cv2.imshow('Img',img)
cap.release()
cv2.destroyAllWindows()

<preview.py>

And before testing the Python program, let's check the camera. /dev/video0 exists.

pi@raspberrypi8GB:~ $ ls  /dev/video*
/dev/video0  /dev/video10  /dev/video11  /dev/video12  /dev/video13  /dev/video14  /dev/video15  /dev/video16  /dev/video18  /dev/video20  /dev/video21  /dev/video22  /dev/video23


There seems to be no problem at all using the cv2.VideoCapture(0) function. Now run the python program. But an error occurs. The camera connection works normally when tested with the libcamera-hello command.

pi@raspberrypi8GB:~/src $ python3 preview.py 
camera read Failed

This problem is caused by not handling libcamera properly in OpenCV being used. If you enable Legacy Camera in raspi-config, reboot and test this program again. You can check that it works normally. In the end, it wasn't a problem with the Python code, it was a problem caused by OpenCV's inability to handle libcamera.

<Operation screen after restoring to Legacy mode>


The bottom line is that the cv2.VideoCapture(0) function, which opens the camera in OpenCV, works fine with the legacy camera stack, but not with the new libcamera stack.


GStreamer and OpenCV

The SBC (Single Board Computer) I mainly use are Raspberry Pi, Jetson Nano, Jetson Xavier NX, and Korean Odroid. Raspberry Pi and Jetson series are often used for development. However, in the Jetson series, it is possible to use the camera module V2 for Raspberry Pi. And if you look at the OpenCV code that controls the camera in the Jetson series, GStreamer is used. 

Here is part of my blog post Camera - CSI Camera (Raspberry Pi camera V2). This article explains how to use the Raspberry Pi camera with the Jetson series.

def gstreamer_pipeline(
    capture_width=1280,
    capture_height=720,
    display_width=1280,
    display_height=720,
    framerate=60,
    flip_method=0,
):
    return (
        "nvarguscamerasrc ! "
        "video/x-raw(memory:NVMM), "
        "width=(int)%d, height=(int)%d, "
        "format=(string)NV12, framerate=(fraction)%d/1 ! "
        "nvvidconv flip-method=%d ! "
        "video/x-raw, width=(int)%d, height=(int)%d, format=(string)BGRx ! "
        "videoconvert ! "
        "video/x-raw, format=(string)BGR ! appsink"
        % (
            capture_width,
            capture_height,
            framerate,
            flip_method,
            display_width,
            display_height,
        )
    )

  ..... skip ......
  cap = cv2.VideoCapture(gstreamer_pipeline(flip_method=0), cv2.CAP_GSTREAMER)

I'm using a pipelined string that I pass to gstreamer instead of the number I normally use in cv2.VideoCapture function. I skipped this part too, but after seeing the changed Raspberry Pi camera stack structure in BullsEye, I understood why NVidia used this method. Not only is Gstreamer a good implementation of the open source camera stack libcamera, but it also made it easy to control the camera by creating and adding a pipeline component ("nvarguscamerasrc" in the above string) for NVidia's SoC.

The conclustion is that if we want to use OpenCV with BullsEye's new camera stack, libcamera, we must also use GStreamer. I confirmed earlier that OpenCV installed with pip does not support GStreamer. This led me to the another conclusion that I had to build my own OpenCV to support GStreamer.


Build GStreamer Support OpenCV on BullsEye 64-bit

OpenCV is a pretty big package. On Raspberry Pi 4, it takes about 1-2 hours to build. And since the build requires a lot of memory, it is recommended that those who use the 2GB model secure virtual memory and increase the memory before building.


increase virtual memory

Applies to 2GB model users only. There are two ways to increase virtual memory. In the past, the swap file was used a lot, but recently there is a tendency to recommend using zram, which increases memory by compressing RAM. No matter which method you choose, there is no big problem. You can use both.

use swap file

$ sudo nano /etc/dphys-swapfile

#Find "CONF_MAXSWAP" and set 4096 
CONF_MAXSWAP=4096

# reboot
$ sudo reboot


use zram

$ git clone https://github.com/StuartIanNaylor/zram-swap-config \
&& cd zram-swap-config
$ sudo ./install.sh

$ sudo nano /etc/zram-swap-config.conf
#Find below lines and set the values
MEM_FACTOR=40
DRIVE_FACTOR=300
COMP_ALG=lz4
SWAP_DEVICES=1
SWAP_PRI=75
PAGE_CLUSTER=0
SWAPPINESS=90

# reboot
$ sudo reboot

Now, after rebooting, if you check the memory with the free -m command, you can see that the swap part has increased a lot.


Build OpenCV on BullsEye OS (32-bit, 64-bit common)

Here is the build, this script was introduced in Install OpenCV 4.5 on Raspberry Pi 4. If you replace 4.5.5 with the version you want in the script below, the code of that version is built.

#!/bin/bash
set -e
echo "Installing OpenCV 4.5.5 on your Raspberry Pi 64-bit OS"
echo "It will take minimal 2.0 hour !"
cd ~
# install the dependencies
sudo apt-get install -y build-essential cmake git unzip pkg-config
sudo apt-get install -y libjpeg-dev libtiff-dev libpng-dev
sudo apt-get install -y libavcodec-dev libavformat-dev libswscale-dev
sudo apt-get install -y libgtk2.0-dev libcanberra-gtk* libgtk-3-dev
sudo apt-get install -y libgstreamer1.0-dev gstreamer1.0-gtk3
sudo apt-get install -y libgstreamer-plugins-base1.0-dev gstreamer1.0-gl
sudo apt-get install -y libxvidcore-dev libx264-dev
sudo apt-get install -y python3-dev python3-numpy python3-pip
sudo apt-get install -y libtbb2 libtbb-dev libdc1394-22-dev
sudo apt-get install -y libv4l-dev v4l-utils
sudo apt-get install -y libopenblas-dev libatlas-base-dev libblas-dev
sudo apt-get install -y liblapack-dev gfortran libhdf5-dev
sudo apt-get install -y libprotobuf-dev libgoogle-glog-dev libgflags-dev
sudo apt-get install -y protobuf-compiler

# download the latest version
cd ~ 
sudo rm -rf opencv*
wget -O opencv.zip https://github.com/opencv/opencv/archive/4.5.5.zip 
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.5.5.zip 
# unpack
unzip opencv.zip 
unzip opencv_contrib.zip 
# some administration to make live easier later on
mv opencv-4.5.5 opencv
mv opencv_contrib-4.5.5 opencv_contrib
# clean up the zip files
rm opencv.zip
rm opencv_contrib.zip

# set install dir
cd ~/opencv
mkdir build
cd build

# run cmake
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-D ENABLE_NEON=ON \
-D WITH_OPENMP=ON \
-D WITH_OPENCL=OFF \
-D BUILD_TIFF=ON \
-D WITH_FFMPEG=ON \
-D WITH_TBB=ON \
-D BUILD_TBB=ON \
-D WITH_GSTREAMER=ON \
-D BUILD_TESTS=OFF \
-D WITH_EIGEN=OFF \
-D WITH_V4L=ON \
-D WITH_LIBV4L=ON \
-D WITH_VTK=OFF \
-D WITH_QT=OFF \
-D OPENCV_ENABLE_NONFREE=ON \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=OFF \
-D PYTHON3_PACKAGES_PATH=/usr/lib/python3/dist-packages \
-D OPENCV_GENERATE_PKGCONFIG=ON \
-D BUILD_EXAMPLES=OFF ..

# run make
make -j4
sudo make install
sudo ldconfig

# cleaning (frees 300 MB)
make clean
sudo apt-get update

echo "Congratulations!"
echo "You've successfully installed OpenCV 4.5.5 on your Raspberry Pi 64-bit OS"

<install_opencv.sh>


You can create and run the install_opencv.sh file with the above contents. After the build is finished, the script is made to install it automatically. And most importantly, "-D WITH_GSTREAMER=ON " is added to the build options. A build time of about an hour or more is required.

# If you already installed OpenCV using pip, remove the preinstalled OpenCV.
$ pip3 uninstall opencv-python
$ chmod 755 install_opencv.sh
$ ./install_opencv.sh


Check GStreamer Support OpenCV

The build script even does the installation. If the script finishes successfully, now check the build information again.


pi@raspberrypi:~/src $ python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print(cv2.getBuildInformation())

General configuration for OpenCV 4.5.5 =====================================
  Version control:               unknown

  Extra modules:
    Location (extra):            /home/pi/opencv_contrib/modules
    Version control (extra):     unknown

  Platform:
    Timestamp:                   2022-03-11T01:35:23Z
    Host:                        Linux 5.10.103-v8+ aarch64
    CMake:                       3.18.4
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/gmake
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                /usr/bin/c++  (ver 10.2.1)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fopenmp -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fopenmp -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -Wl,--gc-sections -Wl,--as-needed  
    Linker flags (Debug):        -Wl,--gc-sections -Wl,--as-needed  
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          dl m pthread rt
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 aruco barcode bgsegm bioinspired calib3d ccalib core datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 alphamat cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv java julia matlab ovis python2 sfm viz
    Applications:                perf_tests apps
    Documentation:               NO
    Non-free algorithms:         YES

  GUI:                           GTK3
    GTK+:                        YES (ver 3.24.24)
      GThread :                  YES (ver 2.66.8)
      GtkGlExt:                  NO

  Media I/O: 
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 62)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.37)
    TIFF:                        build (ver 42 - 4.2.0)
    JPEG 2000:                   build (ver 2.4.0)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      YES (2.2.6)
    FFMPEG:                      YES
      avcodec:                   YES (58.91.100)
      avformat:                  YES (58.45.100)
      avutil:                    YES (56.51.100)
      swscale:                   YES (5.7.100)
      avresample:                NO
    GStreamer:               YES (1.18.4)
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            TBB (ver 2020.2 interface 11102)

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Lapack:                      NO
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.19.1)

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.9.2)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.9.so (ver 3.9.2)
    numpy:                       /usr/lib/python3/dist-packages/numpy/core/include (ver 1.19.5)
    install path:                /usr/lib/python3/dist-packages/cv2/python-3.9

  Python (for build):            /usr/bin/python3

  Java:                          
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /usr/local
-----------------------------------------------------------------


You can see that GStreamer has changed to YES (1.18.4) in Video I/O. Now the OpenCV we built can use GStreamer. BullsEye has some GStreamer-related packages installed, but to use GStreamer in OpenCV, you need to install some additional programs.

# install a missing dependency
$ sudo apt-get install libx264-dev libjpeg-dev
# install the remaining plugins
$ sudo apt-get install libgstreamer1.0-dev \
     libgstreamer-plugins-base1.0-dev \
     libgstreamer-plugins-bad1.0-dev \
     gstreamer1.0-plugins-ugly \
     gstreamer1.0-tools
# install some optional plugins
$ sudo apt-get install gstreamer1.0-gl gstreamer1.0-gtk3
# if you have Qt5 install this plugin
$ sudo apt-get install gstreamer1.0-qt5


Now let's check if GStreamer properly controls the camera in the BullsEye libcamera stack environment. You can see that the GStreamer pipeline starts at libcamerasrc.


pi@raspberrypi:~ $ gst-launch-1.0 libcamerasrc  ! video/x-raw, width=1280, height=720, framerate=30/1 ! videoconvert ! videoscale ! clockoverlay time-format="%D %H:%M:%S" ! video/x-raw, width=640, height=360 ! autovideosink
Setting pipeline to PAUSED ...
[0:50:02.423348871] [1666]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3424-e68e0f1e
[0:50:02.436735159] [1669] ERROR CameraSensor camera_sensor.cpp:551 'ov5647 10-0036': Camera sensor does not support test pattern modes.
[0:50:02.454558105] [1669]  INFO RPI raspberrypi.cpp:1317 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media1
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[0:50:02.467184089] [1672]  INFO Camera camera.cpp:1028 configuring streams: (0) 1280x720-NV21
[0:50:02.469238300] [1669]  INFO RPI raspberrypi.cpp:747 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 1920x1080-SGBRG10_1X10 - Selected unicam format: 1920x1080-pGAA
WARNING: from element /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstXvImageSink:autovideosink0-actual-sink-xvimage: Pipeline construction is invalid, please add queues.
Additional debug info:
../libs/gst/base/gstbasesink.c(1249): gst_base_sink_query_latency (): /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstXvImageSink:autovideosink0-actual-sink-xvimage:
Not enough buffering available for  the processing deadline of 0:00:00.015000000, add enough queues to buffer  0:00:00.015000000 additional data. Shortening processing latency to 0:00:00.000000000.

And the camera screen is displayed properly on the screen as follows. For reference, in Raspberry Pi OS, Legacy Camera is disabled. You can see that GStreamer takes good control of the camera in the libcamera stack environment.

<GStreamer screen working normally in libcamera stack>

I have verified that GStreamer handles CSI camera properly using libcamera. Now let's finally open the camera using the GStreamer pipeline in OpenCV.


Camera control using libcamera stack in OpenCV

Here is the Python OpenCV code using the GStreamer pipeline.


import cv2
import numpy as np
import sys
connstr = 'libcamerasrc ! video/x-raw, width=640, height=480, framerate=30/1 ! videoconvert ! videoscale ! clockoverlay time-format="%D %H:%M:%S" ! appsink'
cap = cv2.VideoCapture(connstr, cv2.CAP_GSTREAMER)
if cap.isOpened() == False:
    print('camera open Failed')
    sys.exit(0)


while True:

    succes, img = cap.read()
    if succes == False:
        print('camera read Failed')
        sys.exit(0)

    k = cv2.waitKey(1)
    if k == ord('q'):
        break

    cv2.imshow('Img',img)

cap.release()
cv2.destroyAllWindows()

<preview_gstreamer.py>

It is almost identical to the preview.py we created earlier. The biggest difference is that the parameters of the cv2.VideoCapture function have changed a lot. 

The string used as a parameter is almost identical to the pipeline used in the GStreamer test using gst-launch-1.0 earlier. Only the last sink part has been replaced with appsink.


Now let's run it.

pi@raspberrypi:~/src/camera $ python3 preview_gstreamer.py 
[1:30:40.993962062] [1859]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3424-e68e0f1e
[1:30:41.007467606] [1863] ERROR CameraSensor camera_sensor.cpp:551 'ov5647 10-0036': Camera sensor does not support test pattern modes.
[1:30:41.025695799] [1863]  INFO RPI raspberrypi.cpp:1317 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media1
[1:30:41.036632907] [1866]  INFO Camera camera.cpp:1028 configuring streams: (0) 640x480-NV21
[1:30:41.038899151] [1863]  INFO RPI raspberrypi.cpp:747 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
[ WARN:0@0.648] global /home/pi/opencv/modules/videoio/src/cap_gstreamer.cpp (1374) open OpenCV | GStreamer warning: unable to query duration of stream
[ WARN:0@0.648] global /home/pi/opencv/modules/videoio/src/cap_gstreamer.cpp (1405) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1

There is an error message, but it is working normally. This error message also occurs in the libcamera-still program, but it does not cause any problems. I think it will be improved when the libcamera stack is updated. The camera preview window appears successfully as shown in the following figure.

<OpenCV screen working normally in libcamera stack>


Wrapping up

With the Raspberry Pi OS updated to BullsEye, the camera control part has changed radically, causing a lot of confusion for developers using CSI cameras. Also, it seems that they switched the camera stack from Legacy to libcamera too quickly without even the Picamera package for Python being properly prepared. If you are a developer using the Picamera package, we recommend that you use BullsEye's camera stack back to Legacy.

And if you are a developer using OpenCV, you can solve it by building OpenCV as described above and then using GStreamer. GStreamer has great flexibility because it connects the desired functionality in a pipelined way. If you habitually use 0 index to control the camera with cv2.VideoCapture(0), please pay attention to how to use GStreamer. As you can see from the example above, even date and time information can be easily printed in the preview window without the help of OpenCV.

The source code can be downloaded from My Github.


댓글 없음:

댓글 쓰기