DeepStream, as the name implies, is NVIDIA's platform for processing continuous frames through video or cameras. In a previous blog, I implemented object tracking using YOLOv4 and DeepSORT. However, the Jetson Nano had a performance as low as 0.3 FPS. At this processing speed, the moving distance of the object is too large at a frame interval that takes about 3.3 seconds, which greatly reduces accuracy. Therefore, it is necessary to optimize the video stream processing. In the next few blogs, I'll look at DeepStream, which is optimized for video and camera processing, and explore how to improve the performance of the object tracking function.
Prerequisites
I use python virtual environment in Xavier NX. Also, Tensorflow and PyTorch, which are frequently used frameworks, have been installed in the virtual environment in advance. Please proceed with the above process in advance and then follow the instructions below.
- Jetson Xavier NX - JetPack 4.4(production release) headless setup
- Jetson Xavier NX - Python virtual environment and ML platforms(tensorflow, Pytorch) installation
What is DeepStream?
NVIDIA's official site for the DeepStream SDK is https://developer.nvidia.com/deepstream-sdk.
DeepStream has the following features.
- NVIDIA DeepStream simplifies the development of scalable intelligent video analytics (IVA) applications
- Developers can now use this to quickly build new applications to transform video into valuable insight.
- Applications for the DeepStream SDK include image classification, scene understanding, video categorization, content filtering etc.
<DeepStream Architecture>
NVIDIA’s DeepStream SDK delivers a complete streaming analytics toolkit for AI-based multi-sensor processing, video and image understanding.
DeepStream is for vision AI developers, software partners, startups and OEMs building IVA apps and services.
The following table is NVIDIA's DeepStream frame processing speed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
* FP16 inference on Jetson Nano
Running on the DLAs for AGX Xavier and NX frees up GPU for other tasks
Xavier NX records from a minimum of 30 FPS to a maximum of 1080 FPS. If the processing speed exceeds 10 FPS, it can be considered to be used in commercial systems.
Tip : DeepStream 5.0, installed on JetPack 4.4 and earlier, has properly implemented Python bindings. In JetPack 4.4 + DeepStream 5.0, installing DeepStream5.0 automatically installs the Python bindings, making it easy to implement DeepStream in Python. Therefore, users of previous version must upgrade to version 5.0 or later.
DeepStream 5.0 on the Xavier NX
DeepStream 5.0 was first released in April 2020 and is the DP (Developer Preview) version until June. And DeepStream 5.0 only works with JetPack 4.4 and above. The DeepStream 5.0 General Availability version was now released. You should use General Availability version not DP versoin.
The following are the contents published by https://docs.nvidia.com/metropolis/deepstream/DeepStream_5.0_Release_Notes.pdf and are not solved by DeepStream 5.0. Previous versions of Deepstream were difficult to implement in Python and had to build the pyds.so (python deepstream) file through a separate binding build. This file connects Python code and Deepstream implemented in native c/c++.
Limitations
- With V4L2 codecs only MAX 1024 (decode + encode) instances are provided. The maximum number of instances can be increased by doing changes in open source code.
- detected-min-w and detected-min-h must be set to values larger than 32 in the primary inference configuration file (config_infer_primary.txt) for gstdsexample on Jetson.
- The Kafka protocol adapter sometimes does not automatically reconnect when the Kafka Broker to which it is connected goes down and comes back up, thereby requiring application restart.
- If the nvds log file, ds.log, has been deleted, then to restart logging you must delete the file /run/rsyslogd.pid within the container before reenabling logging by running the setup_nvds_logger.sh script as described in the “nvds_logger” section of the DeepStream Plugin Development Guide.
- On NVIDIA® Jetson AGX Xavier™, more than 50 instances of certain 1080p H.265 streams are not working due to limited memory for the decoder.
- On Jetson, running a DeepStream application over SSH (via putty) with X11 forwarding does not work.
- DeepStream currently expects model network width to be a multiple of 4 and network height to be a multiple of 2.
- Triton Inference Server implementation in DeepStream currently supports a single GPU. The models need to be configured to use a single GPU.
- For some models sometime output in DeepStream is not exactly same as observed in Transfer Learning Toolkit. This is due to input scaling algorithm differences.
- DRC support is Alpha quality.
- On the fly Model update only supports same type of Model with same Network parameters.
Installing DeepStream on Jetson Xavier NX
I will explain in the assumption that your Xavier NX has the General Availability version JetPack 4.4 installed, not the DP version. If you need information on installing JetPack 4.4 on Xavier NX, see Jetson Xavier NX-JetPack 4.4 (production release) headless setup.
First download the DeepStream SDK from https://developer.nvidia.com/deepstream-getting-started. To download the SDK, you will probably need an NVidia developer account. I will download and install the deb file.
<download page https://developer.nvidia.com/deepstream-getting-started>
After downloading the file, copy it to Xavier NX. Then install the downloaded deepstream_sdk_5.0_arm64.deb file.
spypiggy@XavierNX:~/download$ sudo apt-get install libgstrtspserver-1.0-0
spypiggy@XavierNX:~/download$ sudo dpkg -i deepstream_sdk_5.0_arm64.deb
......
......
--------------------------------------------------------------------------------------- NOTE: sources and samples folders will be found in /opt/nvidia/deepstream/deepstream-5.0 ---------------------------------------------------------------------------------------
You can see that it is properly installed in the /opt/nvidia/deepstream/deepstream-5.0 directory. The pyds.so file, which is essential for Python bindings, is located in the following directory.
spypiggy@XavierNX:~$ ls -al /opt/nvidia/deepstream/deepstream-5.0/lib/pyds.so -rw-r--r-- 1 root root 1023352 Jul 27 04:19 /opt/nvidia/deepstream/deepstream-5.0/lib/pyds.so
Installing deepstream_python_apps
spypiggy@XavierNX:~$ source /home/spypiggy/python/bin/activate (python) spypiggy@XavierNX:~$ (python) spypiggy@XavierNX:~$ cd ~/src (python) spypiggy@XavierNX:~/src$ git clone https://github.com/NVIDIA-AI-IOT/deepstream_python_apps.git (python) spypiggy@XavierNX:~/src$ cd deepstream_python_apps/ (python) spypiggy@XavierNX:~/src/deepstream_python_apps$ ls -al total 1420 drwxrwxr-x 5 spypiggy spypiggy 4096 Aug 6 06:57 . drwxr-xr-x 12 spypiggy spypiggy 4096 Aug 6 06:57 .. drwxrwxr-x 11 spypiggy spypiggy 4096 Aug 6 06:57 apps -rw-rw-r-- 1 spypiggy spypiggy 5410 Aug 6 06:57 FAQ.md drwxrwxr-x 8 spypiggy spypiggy 4096 Aug 6 06:57 .git -rw-rw-r-- 1 spypiggy spypiggy 9674 Aug 6 06:57 HOWTO.md -rw-rw-r-- 1 spypiggy spypiggy 1097 Aug 6 06:57 LICENSE drwxrwxr-x 2 spypiggy spypiggy 4096 Aug 6 06:57 notebooks -rw-rw-r-- 1 spypiggy spypiggy 316866 Aug 6 06:57 .python-app-pipeline.png -rw-rw-r-- 1 spypiggy spypiggy 2818 Aug 6 06:57 README.md -rw-rw-r-- 1 spypiggy spypiggy 1081940 Aug 6 06:57 .test3-app.png
Sample files are in the apps directory. You can simply check if DeepStream 5.0 integration works using Python.
(python) spypiggy@XavierNX:~/src/deepstream_python_apps$ cd apps/ (python) spypiggy@XavierNX:~/src/deepstream_python_apps/apps$ python3 Python 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.append('../') >>> import platform >>> import configparser >>> >>> import gi >>> gi.require_version('Gst', '1.0') >>> from gi.repository import GObject, Gst >>> from common.is_aarch_64 import is_aarch64 >>> from common.bus_call import bus_call >>> >>> import pyds
>>>
If the libraries for using DeepStream 5.0 in Python are imported without errors as above, the installation is successful.
Wrapping Up
DeepStream is a framework optimized by TensorRT from NVidia to process video streams with high performance. If you've been hungry for performance while using video, cameras, etc. in the Jetson series, DeepStream is definitely a good choice. And since DeepStream is an important part of NVidia's Metropolis framework that processes video streams from a large number of IoT devices at high speed, performance will continue to be improved and updated in the future.
Important features such as Python bindings have been properly built into the framework since DeepStream 5.0. Therefore, if you are starting for the first time, be sure to start using the latest DeepStream 5.0 from JetPack 4.4 General Availability version. Existing users are also recommended to upgrade to 5.0. And, whenever the latest version is released in the future, please check for new features and keep your system up to date with the latest DeepStream whenever possible.
In the next blog I will continue to look at examples of using DeepStream in Python.
댓글 없음:
댓글 쓰기