Previous blogs showed how to use Tensorflow, Pytorch Detectron2, DETR, and NVIDIA DNN Vision Library.
And recently I also looked at object detection using YOLOv4.
So far, Object Detection has been performed using still images, video files, or webcams.
When using video and webcams, the main concerns were whether the recognition was correct on a frame-by-frame basis and processing speed (FPS).
When using video and webcams, the main concerns were whether the recognition was correct on a frame-by-frame basis and processing speed (FPS).
Object Tracking is mainly used in video. Object tracking is to determine whether objects recognized in consecutive frames are the same or not, and to track the path of movement in the case of the same objects.
For example, Object Tracking is essential to count floating populations on the street.
With Object Detection technology, the number of people in one frame can be counted, but the number of people who have passed a distance for an hour cannot be counted without using Object Tracking technology. Object Tracking is diverse. There are many algorithms and many methods like "Centroid Tracking", "SORT(Simple Online and Realtime Tracking)". Among them, SORT is a popular object tracking technology. SORT is highly accurate because it uses a Kalman filter that removes noise when predicting the moving path of an object. Recently, DeepSORT, which adds Deep Learning technology to SORT, is also widely used.
With Object Detection technology, the number of people in one frame can be counted, but the number of people who have passed a distance for an hour cannot be counted without using Object Tracking technology. Object Tracking is diverse. There are many algorithms and many methods like "Centroid Tracking", "SORT(Simple Online and Realtime Tracking)". Among them, SORT is a popular object tracking technology. SORT is highly accurate because it uses a Kalman filter that removes noise when predicting the moving path of an object. Recently, DeepSORT, which adds Deep Learning technology to SORT, is also widely used.
SORT and DeepSORT
SORT consists of Object Detection, KalmanFilter, and Hungarian algorithm.
- Object Detection : You can use various frameworks such as YOLO, PyTorch Detectron, Tensorflow Object Detection.
- KalmanFilter : After removing noise components from the moving path and speed of the previous object, predict the next position.
- Hungarian algorithm : The Hungarian algorithm determines the movement of the same object, the appearance of new objects, and the disappearance of existing objects by calculating the positions predicted by the Kalman filter and the positions of real objects.
<SORT >
However, there are weaknesses in the Hungarian algorithm. If the positions of two objects overlap, the object recognized in the next frame is reversed. This is because the Hungarian algorithm does not take into account the features of things. Because humans understand both the location and features of objects perceived by the eyes, even if the overlapping objects become farther away, they can be accurately identified and tracked without confusion. DeepSORT complements this weakness. DeepSORT is an improvement that enables the tracking of objects more accurately by reflecting the features of objects in the Hungarian algorithm.
<DeepSORT >
DeepSORT Object Tracking using Tensorflow, YOLOv4
As mentioned before, it is not necessary to use the YOLOv4 Object Detection model to use DeepSORT. You can also use PyTorch Detectron2 or NVIDIA vision library. Here, we will convert the YOLOv4 model for Tensorflow.
There are many examples of DeepSORT on github. Most cover similar content. In this article, I'll use yehengchen's github (https://github.com/yehengchen/Object-Detection-and-Tracking), modified for Jetson Nano.
Prerequisites
And this project requires a lot of memory. Jetson Nano's 4GB of memory is quite scarce. Therefore, if the memory is not secured as much as possible, an error may occur due to insufficient memory in the process.
The Jetson Nano's Ubuntu desktop occupies about 1.5 GB of memory. Therefore, only 2.5 GB of memory is actually available.
Please delete Ubuntu Desktop and replace it with LXDE. Then, you can free up about 1 GB of memory.
Securing additional memory by changing the desktop was described in "Use more memory by changing Ububtu desktop to LXDE".
Please delete Ubuntu Desktop and replace it with LXDE. Then, you can free up about 1 GB of memory.
Securing additional memory by changing the desktop was described in "Use more memory by changing Ububtu desktop to LXDE".
Why should I install TensorFlow?
YOLO is sufficient to implement Object Detection. However, we will use DeepSORT tracking technology. Models needed for DeepSORT are in the model_data directory.
The extension pb files, market1501.pb, mars.pb, and mars-small128.pb files are models required for DeepSORT. These files work with Tensorflow.
Therefore, Tensorflow must be installed for DeepSORT. If you are using the model for DeepSORT for PyTorch, you might need to install PyTorch instead of Tensorflow.
The extension pb files, market1501.pb, mars.pb, and mars-small128.pb files are models required for DeepSORT. These files work with Tensorflow.
Therefore, Tensorflow must be installed for DeepSORT. If you are using the model for DeepSORT for PyTorch, you might need to install PyTorch instead of Tensorflow.
Install Tensorflow
In the Jetson series, tensorflow should not be built according to the contents of the Tensorflow homepage. Download and install Tensorflow for Jetson provided by NVidia. We will be using Tensorflow version 1.5 for JetPack 4.4.
To install Tensorflow for Jetpack 4.4, follow the instructions in "JetsonNano-Installing Tensorflow".
Install necessary packages
Keras, scipy, scikit-learn, etc. used by yehengchen's github are version sensitive. Therefore, please keep the version indicated below.
apt-get install liblapack-dev libatlas-base-dev gfotran #numpy to 1.19.0 pip3 install --upgrade numpy pip3 install Keras==2.3.1 pip3 install scikit-learn==0.21.2 pip3 install scipy #scipy to 1.5.0 pip3 install --upgrade scipy
Install yehengchen's github
Now that you're ready, download the source code .
cd /usr/local/src git clone https://github.com/yehengchen/Object-Detection-and-Tracking.git
## Download the YOLOv4 model
cd Object-Detection-and-Tracking/OneStage/yolo/deep_sort_yolov4/ wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights
cd model_data/
cp "/usr/local/src/Object-Detection-and-Tracking/OneStage/yolo/Train-a-YOLOv4-model/cfg/yolov4.cfg" ./
cp yolo_anchors.txt yolo4_anchors.txt
## Convert the YOLOv4 models to Keras model
## If there are not enough memory, the conversion process might fail.
cd ..
python3 convert.py "model_data/yolov4.cfg" "model_data/yolov4.weights" "model_data/yolo.h5"
##Download the test video clip
wget https://git.kpi.fei.tuke.sk/ml163fe/atvi/-/raw/4f70d8fd9c263b5a90dcdbc7a94b1176a520124c/python_objects_detection/TownCentreXVID.avi -O test_video/TownCentreXVID.avi
Object tracking with sample video
Now let's check whether it works properly using the video clip downloaded earlier.
I modified the main.py file to print the FPS value. The FPS values were divided into two FPS values, including the time to process the inference frame in the model and the time to output the screen and write it to a new video file. The modified main.py file can be downloaded from my github.
This command, TownCentreXVID.avi file has the following properties.
- Total 11475 frames, 15 frames/second, 1920 X 1080 frame size.
Therefore, it takes a lot of time to process this video clip. If you want to abort, press Ctrl+C during execution.
root@jetpack-4:/usr/local/src/Object-Detection-and-Tracking/OneStage/yolo/deep_sort_yolov4# python3 main.py -c person -i "./test_video/TownCentreXVID.avi" Using TensorFlow backend. 2020-06-30 18:40:19.078024: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2 /usr/local/lib/python3.6/dist-packages/sklearn/utils/linear_assignment_.py:21: DeprecationWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead. DeprecationWarning) WARNING:tensorflow:From main.py:27: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. 2020-06-30 18:40:31.516170: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency .......... 2020-06-30 18:41:59.919461: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8 2020-06-30 18:42:07.822707: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.75GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-06-30 18:42:07.823401: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2020-06-30 18:42:15.321959: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.21GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-06-30 18:42:16.545443: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2020-06-30 18:42:17.661731: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. .......... Net FPS:0.016420 FPS:0.015778 Net FPS:0.257968 FPS:0.234274 Net FPS:0.321014 FPS:0.291949 Net FPS:0.314152 FPS:0.287686
The following image captures a frame from the output.avi file created in the output directory.
<output.avi>
You can see the 0.27 FPS value displayed at the top left. You can see that tracking works by playing the created video file. However, the 0.3 FPS value is problematic. Because video frames are processed sequentially, the accuracy is the same except for the time-consuming problem, but it is a big problem in real-time video capture.
If it takes about 3 seconds or more to process 1 frame, the accuracy of tracking is too low when the position change value of the object (person) is too large when trying to process the next frame.
Wrapping up
I implemented object tracking using YOLOv4 and Deep SORT in Jetson Nano.
Unfortunately, it achieved a low performance of 0.3 FPS. Applying this value to a real-time camera makes object tracking accuracy too low. Therefore, it is inappropriate to use YOLOv4 + DeepSORT in Jetson Nano. Sooner or later, I will make an opportunity to find out how to apply the YOLOv4 tiny model or run this example on the Jetson Xavier NX to speed up.
Unfortunately, it achieved a low performance of 0.3 FPS. Applying this value to a real-time camera makes object tracking accuracy too low. Therefore, it is inappropriate to use YOLOv4 + DeepSORT in Jetson Nano. Sooner or later, I will make an opportunity to find out how to apply the YOLOv4 tiny model or run this example on the Jetson Xavier NX to speed up.
댓글 없음:
댓글 쓰기