JetsonNano - Installing the latest Pytorch 1.9 and Pose Estimation

 Wednesday, October 16, 2019, I've described how to implement PoseEstimation in PyTorch.

• JetsonNano - Human Pose estimation using Pytorch, torchvision

It was explained based on JetPack 4.4 and PyTorch 1.6. But now some links are not complete and contents are outdated. Therefore, I will try to implement PoseEstimation again after installing PyTorch 1.9 from JetPack 4.6, the most recent version as of August 2021. In terms of content, there is no significant difference from the previous article.

Prerequisites

Before you build Pytorch, torchvision, you must pre install these packages.

apt-get install libjpeg-dev zlib1g-dev

Install PyTorch

PyTorch should not be downloaded from the PyTorch website, but must be downloaded from the link below and installed. The installation file below is built for NVidia Jetson series.

Before installing pytorch , visit this site to check the latest pytorch version.

Before installing torchvision , visit this site to check the latest torchvision version.

The latest version at this time is PyTorch 1.9. Download and install the file below.

Delete old versions of PyTorch

First check pre-installed PyTorch. If there is no PyTorch version already installed, proceed to the next step.

root@spypiggy-nano:/usr/local/src/detr# pip3 freeze|grep torch
torch==1.1.0
torchvision==0.3.0

root@spypiggy-nano:/usr/local/src/detr# pip3 uninstall  torchvision==0.3.0
root@spypiggy-nano:/usr/local/src/detr# pip3 uninstall  torch==1.1.0

Download and install Pytorch whl file 

We always use Python 3.X. Therefore, download the whl file that can be used in Python 3.6. And install the necessary packages as follows.

apt-get install python3-pip libopenblas-base libopenmpi-dev 
pip3 install Cython
wget -O torch-1.9.0-cp36-cp36m-linux_aarch64.whl https://nvidia.box.com/shared/static/h1z9sw4bb1ybi0rm3tu8qdj8hs05ljbm.whl
pip3 install torch-1.9.0-cp36-cp36m-linux_aarch64.whl

Download and install Pytorch whl file

If you have successfully installed PyTorch 1.9.0, install Torchvision 0.10.0. The latest version of torchvision can be found at https://github.com/pytorch/vision/releases.

sudo apt-get install libjpeg-dev zlib1g-dev libfreetype6-dev
wget https://github.com/pytorch/vision/archive/v0.10.0.tar.gz
tar -xvzf v0.10.0.tar.gz
cd vision-0.10.0
#This takes very long time, have a coffee time
sudo python3 setup.py install

Let's check whether the installation is correct. If you see the screen like this, the installation is successful.

root@spypiggyNano:/usr/local/src# python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import torchvision
>>> torch.__version__
'1.9.0'
>>> torchvision.__version__
'0.10.0a0'

Be Careful : You must use python3, pip3 commands. PyTorch 1.5 and later, Python 2 is no longer supported.

Installation Sample Codes for Pose Estimation

Now we have finished installing Pytorch, torchvision. It's time to install sample python codes to proceed. I'll use the codes from  https://github.com/kairess/torchvision_walkthrough.git .

cd /usr/local/src
git clone https://github.com/kairess/torchvision_walkthrough.git
cd /usr/local/src//torchvision_walkthrough 

Now you can find several sample files to test. some files are jupyter notebook files. The author of these codes use a MacBook. So the sample codes do not take GPU(cuda) into account. I'm going to modify the sample codes to sue CUDA. Using CUDA in pytorch is about 10 times faster!

Keypoint detection comparison of performance with or without cuda

The example code below is slightly modified to use CUDA.

import torch
import torchvision
from torchvision import models
import torchvision.transforms as T

import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
from matplotlib.path import Path
import matplotlib.patches as patches
import argparse
import sys, time

IMG_SIZE = 480
THRESHOLD = 0.95


parser = argparse.ArgumentParser(description="Keypoint detection. - Pytorch")
parser.add_argument("--cuda", action="store_true")
args = parser.parse_args()

if True == torch.cuda.is_available():
    print('pytorch:%s GPU support'% torch.__version__)
else:
    print('pytorch:%s GPU Not support ==> Error:Jetson should support cuda'% torch.__version__)
    sys.exit()
print('torchvision', torchvision.__version__)

model = models.detection.keypointrcnn_resnet50_fpn(pretrained=True).eval()
if(args.cuda):
    model = model.cuda()

#img = Image.open('imgs/07.jpg')
img = Image.open('imgs/apink1.jpg')
img = img.resize((IMG_SIZE, int(img.height * IMG_SIZE / img.width)))

plt.figure(figsize=(16, 16))
plt.imshow(img)


trf = T.Compose([
        T.ToTensor()
        ])

input_img = trf(img)
print(input_img.shape)
if(args.cuda):
    input_img = input_img.cuda()

#The first result is time consuming. After the second, check the processing time with the result.
model([input_img])
fps_time  = time.perf_counter()
out = model([input_img])[0]print(out.keys())


codes = [
    Path.MOVETO,
    Path.LINETO,
    Path.LINETO
]

fig, ax = plt.subplots(1, figsize=(16, 16))
ax.imshow(img)

for box, score, keypoints in zip(out['boxes'], out['scores'], out['keypoints']):
    if(args.cuda):
        score = score.cpu().detach().numpy()
    else:        
        score = score.detach().numpy()

    if score < THRESHOLD:
        continue

    if(args.cuda):
        box = box.to(torch.int16).cpu().numpy()
        keypoints = keypoints.to(torch.int16).cpu().numpy()[:, :2]
    else:
        box = box.detach().numpy()
        keypoints = keypoints.detach().numpy()[:, :2]

    rect = patches.Rectangle((box[0], box[1]), box[2]-box[0], box[3]-box[1], linewidth=2, edgecolor='b', facecolor='none')
    ax.add_patch(rect)

    # 17 keypoints
    for k in keypoints:
        circle = patches.Circle((k[0], k[1]), radius=2, facecolor='r')
        ax.add_patch(circle)
    
    # draw path
    # left arm
    path = Path(keypoints[5:10:2], codes)
    line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
    ax.add_patch(line)
    
    # right arm
    path = Path(keypoints[6:11:2], codes)
    line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
    ax.add_patch(line)
    
    # left leg
    path = Path(keypoints[11:16:2], codes)
    line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
    ax.add_patch(line)
    
    # right leg
    path = Path(keypoints[12:17:2], codes)
    line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
    ax.add_patch(line)

plt.savefig('result.jpg')
fps = 1.0 / (time.perf_counter() - fps_time)

if(args.cuda):
    print('FPS(cuda support):%f'%(fps))
else:    
    print('FPS(cuda not support):%f'%(fps))

<keypoints2.py>

Let's run above code without --cuda options and with --cuda options.

spypiggy@spypiggyNano:/usr/local/src/torchvision_walkthrough$ sudo python3 keypoints2.py
pytorch:1.9.0 GPU support
torchvision 0.10.0a0
torch.Size([3, 335, 480])
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /media/nvidia/NVME/pytorch/pytorch-v1.9.0/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
dict_keys(['boxes', 'labels', 'scores', 'keypoints', 'keypoints_scores'])
FPS(cuda not support):0.021109
spypiggy@spypiggyNano:/usr/local/src/torchvision_walkthrough$ sudo python3 keypoints2.py --cuda
pytorch:1.9.0 GPU support
torchvision 0.10.0a0
torch.Size([3, 335, 480])
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /media/nvidia/NVME/pytorch/pytorch-v1.9.0/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
dict_keys(['boxes', 'labels', 'scores', 'keypoints', 'keypoints_scores'])
FPS(cuda support):0.158586

Be Careful : Note that the model is processed twice in the source code. FPS is calculated as the processing time of the second model. The reason is that the first execution result after loading the model takes a lot of time.

The saved result.jpg file is as follows.

<result.jpg>

Even with CUDA, the speed is only 0.15 FPS. At this speed, it takes about 6.7 seconds to process one frame, making it unsuitable for real-time video stream processing. The main reason is that the resnet50 model used in this article is quite heavy instead of recording excellent accuracy.

Under the hood

Now let's dig deeper.

Torchvision keypoint number and human parts

Torchvision's keypoint numbering is different from OpenPose or Tensorflow's models.

The values are like this.

COCO_PERSON_KEYPOINT_NAMES = [
    'nose', 
    'left_eye',
    'right_eye',
    'left_ear',
    'right_ear',
    'left_shoulder',
    'right_shoulder',
    'left_elbow',
    'right_elbow',
    'left_wrist',
    'right_wrist',
    'left_hip',
    'right_hip',
    'left_knee',
    'right_knee',
    'left_ankle',
    'right_ankle'
]

Result from model

Unlike TensorFlow 1.X, Pytorch is so intuitive that it makes code easier to understand.

Only three lines of code are enough.

First convert a numpy image to tensor, move the variable to cuda if using GPU.

Then insert the tensor to model, the return value is the list of dictionary type. As I inserted one image to model, index 0 of the list is sufficient.

input_img = trf(img)    # Make image to Pytorch tensor
input_img = input_img.to(device)
out = model([input_img])[0]

If you print the out variable's dictionary keys, you can see these key values.

print(out.keys())

dict_keys(['boxes', 'labels', 'scores', 'keypoints', 'keypoints_scores'])

You can see these values' explanations at  https://pytorch.org/vision/stable/models.html#object-detection-instance-segmentation-and-person-keypoint-detection

But the document is incomplete. They don't explain the keypoints_scores. The remaining values ​​are explained as follows.

• boxes (FloatTensor[N, 4]): the ground-truth boxes in [x1, y1, x2, y2] format, with values between 0 and H and 0 and W
• labels (Int64Tensor[N]): the class label for each ground-truth box
• keypoints (FloatTensor[N, K, 3]): the K keypoints location for each of the N instances, in the format [x, y, visibility], where visibility=0 means that the keypoint is not visible.

Be careful : keypoints visibility value seems to be not correct. 1 means the point is visible, 0 means the point is invisible(hidden human parts). However, this value is always 1. Therefore, this value has no meaning so far. It is a good way to determine the accuracy of the keypoint region with the keypoints_scores value.

If you input image inference to the network model(keypointrcnn_resnet50_fpn), you can get output dictionary value.  In this code, 'for loop' iterates for human counts, and prints keypoints and keypoint_score values.

out = model([input_img])[0]
for box, score, keypoints, kscores in zip(out['boxes'], out['scores'], out['keypoints'], out['keypoints_scores'] ):
    score = score.cpu().detach().numpy()
    box = box.cpu().detach().numpy()
    points = keypoints.cpu().detach().numpy()
    kscores = kscores.cpu().detach().numpy()
    print(kscores)
    print(points)

Let's check the keypoints_score and keypoints of this picture. This is an inference image. The lower body is not visible in this picture.

<03.jpg>

Run this command.

spypiggy@spypiggyNano:/usr/local/src/torchvision_walkthrough$ sudo python3 keypoints2.py --image=./imgs/03.jpg --cuda
pytorch:1.9.0 GPU support
torchvision 0.10.0a0
torch.Size([3, 719, 480])
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /media/nvidia/NVME/pytorch/pytorch-v1.9.0/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
dict_keys(['boxes', 'labels', 'scores', 'keypoints', 'keypoints_scores'])
scores: 0.998314380645752
kscores: [  1.25098372   0.82812518   2.83464313  10.02956772  10.867136
   7.50433922   9.46248245   5.73677111   4.82222462  -1.08437061
  -2.38515258  -2.25649142  -2.69200468  -3.09163022  -2.31611848
  -1.32797825  -1.90648782]
keypoints: [[386 295]
 [193 245]
 [368 264]
 [224 298]
 [352 298]
 [127 452]
 [412 470]
 [112 706]
 [420 706]
 [174 409]
 [419 706]
 [205 706]
 [354 706]
 [217 706]
 [434 578]
 [113 706]
 [421 706]]
FPS(cuda support):0.148066

Let's compare the keypoints_score and it's name. The score at the lower wrist is very low. If you see the above picture, you might understand enough.

nose                 1.250984
left_eye             0.828125
right_eye            2.834643
left_ear             10.029568
right_ear            10.867136
left_shoulder        7.504339
right_shoulder       9.462482
left_elbow           5.736771
right_elbow          4.822225
left_wrist           -1.084371
right_wrist          -2.385153
left_hip             -2.256491
right_hip            -2.692005
left_knee            -3.091630
right_knee           -2.316118
left_ankle           -1.327978
right_ankle          -1.906488

The kscore values of the lower body key points are negative. And looking at the output image result.jpg, the line connecting the feet from the waist was drawn strangely because the -value kscore was not taken into account. The line from the elbow to the wrist was also drawn strangely.

<result.jpg>

Example reflecting kscores

The code below is an improvement not to draw a line when the score of the connection keypoint is - when connecting the arm and leg joints.

import torch
import torchvision
from torchvision import models
import torchvision.transforms as T

import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
from matplotlib.path import Path
import matplotlib.patches as patches
import argparse
import sys, time

IMG_SIZE = 480


parser = argparse.ArgumentParser(description="Keypoint detection. - Pytorch")
parser.add_argument('--image', type=str, default="./imgs/03.jpg", help='inference image')
parser.add_argument('--accuracy', type=float, default=0.9, help='accuracy. default=0.6')
parser.add_argument("--cuda", action="store_true")
args = parser.parse_args()

if True == torch.cuda.is_available():
    print('pytorch:%s GPU support'% torch.__version__)
else:
    print('pytorch:%s GPU Not support ==> Error:Jetson should support cuda'% torch.__version__)
    sys.exit()
print('torchvision', torchvision.__version__)

model = models.detection.keypointrcnn_resnet50_fpn(pretrained=True).eval()
if(args.cuda):
    model = model.cuda()

img = Image.open(args.image)
#img = Image.open('imgs/apink1.jpg')
img = img.resize((IMG_SIZE, int(img.height * IMG_SIZE / img.width)))

plt.figure(figsize=(16, 16))
plt.imshow(img)


trf = T.Compose([
        T.ToTensor()
        ])

input_img = trf(img)
print(input_img.shape)
if(args.cuda):
    input_img = input_img.cuda()


#The first result is time consuming. After the second, check the processing time with the result.
#model([input_img])
fps_time  = time.perf_counter()
out = model([input_img])[0]
print(out.keys())
t_human = 0
r_human = 0



codes = [
    Path.MOVETO,
    #Path.LINETO,
    Path.LINETO
]

fig, ax = plt.subplots(1, figsize=(16, 16))
ax.imshow(img)
t_human = 0
r_human = 0
for box, score, keypoints, kscores  in zip(out['boxes'], out['scores'], out['keypoints'], out['keypoints_scores'] ):
    if(args.cuda):
        score = score.cpu().detach().numpy()
        kscores = kscores.cpu().detach().numpy()    
        box = box.to(torch.int16).cpu().numpy()
        keypoints = keypoints.to(torch.int16).cpu().numpy()[:, :2]
    else:        
        score = score.detach().numpy()
        box = box.detach().numpy()
        keypoints = keypoints.detach().numpy()[:, :2]
        kscores = kscores.detach().numpy()    

    t_human += 1
    if score < args.accuracy:
        continue
    r_human += 1

    rect = patches.Rectangle((box[0], box[1]), box[2]-box[0], box[3]-box[1], linewidth=2, edgecolor='b', facecolor='none')
    ax.add_patch(rect)

    # 17 keypoints
    #for k in keypoints:
    for x in range(len(keypoints)):
        k = keypoints[x]
        if kscores[x] > 0:
            if x == 5:
                circle = patches.Circle((k[0], k[1]), radius=4, facecolor='r')
            else:
                circle = patches.Circle((k[0], k[1]), radius=2, facecolor='r')
            ax.add_patch(circle)
    
    # draw path
    # left arm
    if kscores[5] > 0 and kscores[7] > 0:
        path = Path(keypoints[5:8:2], codes)
        line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
        ax.add_patch(line)
    if kscores[7] > 0 and kscores[9] > 0:
        path = Path(keypoints[7:10:2], codes)
        line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
        ax.add_patch(line)
    
    # right arm
    if kscores[6] > 0 and kscores[8] > 0:
        path = Path(keypoints[6:9:2], codes)
        line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
        ax.add_patch(line)
    if kscores[8] > 0 and kscores[10] > 0:
        path = Path(keypoints[8:11:2], codes)
        line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
        ax.add_patch(line)


    # left leg
    if kscores[11] > 0 and kscores[13] > 0:
        path = Path(keypoints[11:14:2], codes)
        line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
        ax.add_patch(line)
    if kscores[13] > 0  and kscores[15] > 0:
        path = Path(keypoints[13:16:2], codes)
        line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
        ax.add_patch(line)
    

    # right leg
    if kscores[12] > 0 and kscores[14] > 0:
        path = Path(keypoints[12:15:2], codes)
        line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
        ax.add_patch(line)
    if kscores[14] > 0 and kscores[16] > 0:
        path = Path(keypoints[14:17:2], codes)
        line = patches.PathPatch(path, linewidth=2, facecolor='none', edgecolor='r')
        ax.add_patch(line)

plt.savefig('result.jpg')
fps = 1.0 / (time.perf_counter() - fps_time)
print('total human:%d  real human:%d'%(t_human, r_human))

if(args.cuda):
    print('FPS(cuda support):%f'%(fps))
else:    
    print('FPS(cuda not support):%f'%(fps))

The result of executing the above code is as follows.

Wrapping up

After about two years since October 2019, I looked at keypoint recognition in PyTorch again. PyTorch has been upgraded in the meantime, but there doesn't seem to be much change in the model and documentation for keypoint recognition.

This Resnet50-based model has high accuracy, but the processing speed is too low to be suitable for real-time video processing on the Jetson Nano. It is well worth considering for image file processing purposes. If you want to detect keypoints in real-time video files on Jetson Nano, please refer to the following links.

• JetsonNano - NVIDIA AI IOT - Human Pose estimation using TensorRT
• Xavier NX - NVIDIA AI IOT - Human Pose estimation using TensorRT

The source code of this text can be downloaded from my github.

Running OpenPose models directly from OpenCV

This article has some references from Deep Learning based Human Pose Estimation using OpenCV.

 So far, I have written a lot about OpenPose. Most of the articles have implemented the Pose Estimation function using the Python functions provided by OpenPose. In this article, I will compare how to use the Python module provided by OpenPose and the Caffe model provided by OpenCV's dnn module.

Prerequisites

• Jetpack 4.5 - Install the latest version of OpenPose on Jetson Nano

Pose Models

There are three models that extract body keypoints from OpenPose. Among these, if the model is not specified, body_25 is used by default. However, when using OpenCV dnn, this value must be specified correctly.

The three models are as follows.

• COCO : 18 keypoints. 
• MPI : 15 keypoints. least accurate model but fastest on CPU
• BODY_25 : fastest for CUDA version, most accurate, and includes foot keypoints

The position of the Pose model is as follows.

root@spypiggy-nx:/usr/local/src/openpose-1.7.0/models/pose# pwd
/usr/local/src/openpose-1.7.0/models/pose
root@spypiggy-nx:/usr/local/src/openpose-1.7.0/models/pose# tree
.
├── body_25
│   ├── pose_deploy.prototxt
│   └── pose_iter_584000.caffemodel
├── coco
│   ├── pose_deploy_linevec.prototxt
│   └── pose_iter_440000.caffemodel
└── mpi
    ├── pose_deploy_linevec_faster_4_stages.prototxt
    ├── pose_deploy_linevec.prototxt
    └── pose_iter_160000.caffemodel

3 directories, 7 files

Tips : In the Pose model, only the default BODY_25 might be installed. If other models do not exist, run the getmodels.sh command in the models directory to download the rest of the models.

Common way to run OpenPose

The following example is a simple example using the OpenPose Python module.

import cv2
from openpose import pyopenpose as op

params = dict()
params["model_folder"] = "/usr/local/src/openpose-1.7.0/models/"
params["net_resolution"] = "320x256"  #inference resolution

opWrapper = op.WrapperPython()
opWrapper.configure(params)
opWrapper.start()


datum = op.Datum()
imageToProcess = cv2.imread('/usr/local/src/image/blackpink/blackpink.png')
datum.cvInputData = imageToProcess
opWrapper.emplaceAndPop(op.VectorDatum([datum]))
newImage = datum.cvOutputData[:, :, :]
cv2.imwrite("/tmp/result.jpg", newImage)

<original.py>

Now run the sample code.

root@spypiggy-nx:/usr/local/src/study/opencv_dnn# python3 sample.py
Starting OpenPose Python Wrapper...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
[ WARN:0] global /usr/local/src/opencv-4.5.1/modules/core/src/matrix_expressions.cpp (1334) assign OpenCV/MatExpr: processing of multi-channel arrays might be changed in the future: https://github.com/opencv/opencv/issues/16739

</tmp/result.jpg>

However, there is also a way to use OpenCV's dnn module. Models created using popular machine learning frameworks are read directly from OpenCV and processed. Models that can be directly processed in OpenCV are as follows.

• PyTorch
• Tensorflow
• Darknet(YOLO)
• Caffe
• ONNIX

OpenPose uses models learned using the Caffe framework. Therefore, you can follow the method of using the Caffe model provided by OpenCV.

Running the OpenPose model in OpenCV

Various network models have been supported since OpenCV 3.3. However, in versions prior to 4.2, the acceleration capabilities using NVidia GPU is not provided. Therefore, if you use OpenCV 4.1 provided by JetPack 4.5, you can use only the CPU without the GPU acceleration capabilities.

A simple way to use OpenCV's dnn is as follows.

• Load network model
• Read Image and Prepare blob
• Make prediction(forward)
• Parse results(key points)

Let's create the sample python program in a way that uses OpenCV dnn.

In the case of using the OpenPose module, you could directly receive the image linking keypoints from the module. However, when using the OpenCV dnn module, only the accuracy values of each pixel are received. (These values can also be obtained when using the OpenPose module.) Using these values, I need to draw the keypoint of the image.

Load network model

I am using OpenPose models trained on Caffe Framework. Caffe models have 2 files –

• .prototxt file which specifies the architecture of the neural network – how the different layers are arranged etc.
• .caffemodel file which stores the weights of the trained model

I'm going to use OpenPose default body_25 models and for OpenCV 4.1 compatibility, I will use CPU mode first.

protoFile = "/usr/local/src/openpose-1.7.0/models/pose/body_25/pose_deploy.prototxt"
weightsFile = "/usr/local/src/openpose-1.7.0/models/pose/body_25/pose_iter_584000.caffemodel"
net = cv2.dnn.readNetFromCaffe(protoFile, weightsFile)
net.setPreferableBackend(cv2.dnn.DNN_TARGET_CPU)

Read Image and Prepare blob

The input image that I read using OpenCV should be converted to a input blob ( like Caffe ) so that it can be fed to the network. The blobFromImage do the job which converts the image from OpenCV format to Caffe blob format. 

img = cv2.imread('/usr/local/src/image/blackpink/blackpink.png')
frameWidth = img.shape[1]
frameHeight = img.shape[0]

inHeight = 368
inWidth = int((inHeight/frameHeight)*frameWidth)
Blob = cv2.dnn.blobFromImage(img, 1.0 / 255, (inWidth, inHeight),
                          (0, 0, 0), swapRB=False, crop=False)

Make Prediction

Once the image is passed to the model, the predictions can be made using a single line of code. The forward method for the DNN class in OpenCV makes a forward pass through the network and its output is the prediction results. 

net.setInput(Blob)
output = net.forward()

Parse results(key points)

The output is a 4D matrix :

• The first dimension being the image ID. If the inference data is a single image, this value must be 1.
• The second dimension indicates the index of a keypoint. The model produces Confidence Maps and Part Affinity maps which are all concatenated. For BODY_25 model it consists of 78 parts – 25 keypoint confidence Maps + 1 background + 26*2 Part Affinity Maps. Similarly, for MPI, it produces 44 points. I will use only the first 25 points for body_25 model which correspond to Keypoints.  This value defines the corresponding keypoint in the image (W x H). If this demention value is 0, you can find the nose, if it is 1, the neck, and so on. If the pose model is not BODY_25 but COCO or MPI, the index number and the corresponding body part will be different.
• The third dimension is the height of the output map.
• The fourth dimension is the width of the output map. I check whether each keypoint is present in the image or not. I get the location of the keypoint by finding the maxima of the confidence map of that keypoint. I also use a threshold to reduce false detections.

<output matrix of body-25 model> 

Be Careful: If you use other models like COCO, MPI, the output's 4D matrix should change. When using COCO, Keypoint detection channel might be 19(18 + background image) and PAF image might be 38(19 X 2).

Probability image extraction 

The probability distribution image for each part can be extracted as follows.

for index in range(25):
    probMap = output[0,index,:,:]

probMap is a two-dimensional matrix of H x W. It is similar to a gray scale image. The pixel value is a probability value of the existence of a corresponding key point, and has a probability value between 0.0 and 1.0. Therefore, even if you check this probMap using imshow or imwrite function, it is difficult to check with the naked eye.

To check visually, multiply by 255 and change it to an image pixel value range of 0 to 255, and then use the imshow and imwrite functions.

key_points = {
    0:  "Nose", 1:  "Neck", 2:  "RShoulder", 3:  "RElbow", 4:  "RWrist", 5:  "LShoulder", 6:  "LElbow",
    7:  "LWrist", 8:  "MidHip", 9:  "RHip", 10: "RKnee", 11: "RAnkle", 12: "LHip", 13: "LKnee",
    14: "LAnkle", 15: "REye", 16: "LEye", 17: "REar", 18: "LEar", 19: "LBigToe", 20: "LSmallToe",
    21: "LHeel", 22: "RBigToe", 23: "RSmallToe", 24: "RHeel", 25: "Background"
}
for index in range(25):
    probMap = output[0,index,:,:] * 255
    cv2.imwrite('/tmp/proMap_%s.jpg'%key_points[index], probMap)

If you run the code above, you can visually check the probability value for each part as shown in the following figure. Since the white point is a value close to 255, it is close to 1 as a probability value, and there is a high possibility that a keypoint exists.

If you want to accurately compare the position of the original image, you can use alpha blending as follows.

alpha = 0.3

for index in range(26):
    probMap = output[0,index,:,:] * 255
    probMap = cv2.resize(probMap, (img.shape[1], img.shape[0]))
    probMap = np.asarray(probMap, np.uint8)
    probMap = cv2.cvtColor(probMap,cv2.COLOR_GRAY2BGR)
    dst = cv2.addWeighted(img, alpha, probMap, (1-alpha), 0)
    cv2.imwrite('/tmp/combined_%s.jpg'%key_points[index], dst)

If you run the code above, you can get the next image with the proMap and the original image blended.

<combined_Nose.jpg>

PAF(Part Affinity Field) image extraction 

As can be seen from the probability image, when using OpenCV's dnn module, multiple specific keypoints all appear in one image. In the case of a single person image, there is no problem, but in the case of an image containing multiple people, it is necessary to connect the key points to each person.

This is an unnecessary process using OpenPose's Python module.

Soon I face the following problems. When trying to connect the nose to the throat in an image of multiple people, it is not easy to make a valid connection.

<which neck should I connect?>

The information necessary to effectively connect key points on a per-person basis can be obtained using PAF. As described above, the first 25 output values have a probability distribution for each of 25 body parts. And the remaining 52 contain information for linking these body parts information. 

PAF

PAF is a vector representation of the relationship with the next most effective keypoint when connecting keypoints.

The figure below shows PAF vectors when keypoints 1 to 7 connect to the next keypoints.

<image from Implementation of PAF (Openpose) Pose Detection Network & its Training Accelerations on GCP>

This image is a PAF image of the neck and nose joint. There are always two PAFs. One is, if the key points constituting the joint are A and B, there are vectors heading to A->B and vectors heading to B->A.

<PAF image of joint pair(neck and nose)>

Connecting valid pairs

The process of extracting vectors from PAF images to create valid joints is quite complex. Multi-Person Pose Estimation in OpenCV using OpenPose provides an excellent example, so I will take it and use it. Unfortunately, the examples provided in this article only work for the COCO model. So I modified this example to make it work on the BODY-25 model as well.

The key is the numpy dot operation. The np.dot function creates the largest value if two vectors are in the same direction, and 0 is returned if the two vectors make up 90 degrees. Therefore, if the vector consisting of two keypoint coordinates constituting the joint and the PAF vector are in the same direction, the largest value is returned. This algorithm is used to figure out a valid keypoint joint.

OpenPose does not use a top-down method for finding people and then detecting the keypoint. It finds all key pines and then connect a valid pair, and then use the Bottom Up method to calculate the number of people.

def getKeypoints(probMap, threshold=0.1):

    mapSmooth = cv2.GaussianBlur(probMap,(3,3),0,0)

    mapMask = np.uint8(mapSmooth>threshold)
    keypoints = []

    #find the blobs
    contours, _ = cv2.findContours(mapMask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

    #for each blob find the maxima
    for cnt in contours:
        blobMask = np.zeros(mapMask.shape)
        blobMask = cv2.fillConvexPoly(blobMask, cnt, 1)
        maskedProbMap = mapSmooth * blobMask
        _, maxVal, _, maxLoc = cv2.minMaxLoc(maskedProbMap)
        keypoints.append(maxLoc + (probMap[maxLoc[1], maxLoc[0]],))

    return keypoints



# Find valid connections between the different joints of a all persons present
def getValidPairs(output):
    valid_pairs = []
    invalid_pairs = []
    n_interp_samples = 10
    paf_score_th = 0.1
    conf_th = 0.7
    # loop for every POSE_PAIR
    for k in range(len(mapIdx)):
        # A->B constitute a limb
        pafA = output[0, mapIdx[k][0], :, :]
        pafB = output[0, mapIdx[k][1], :, :]
        pafA = cv2.resize(pafA, (frameWidth, frameHeight))
        pafB = cv2.resize(pafB, (frameWidth, frameHeight))

        # Find the keypoints for the first and second limb
        candA = detected_keypoints[POSE_PAIRS[k][0]]
        candB = detected_keypoints[POSE_PAIRS[k][1]]
        nA = len(candA)
        nB = len(candB)

        # If keypoints for the joint-pair is detected
        # check every joint in candA with every joint in candB
        # Calculate the distance vector between the two joints
        # Find the PAF values at a set of interpolated points between the joints
        # Use the above formula to compute a score to mark the connection valid

        if( nA != 0 and nB != 0):
            valid_pair = np.zeros((0,3))
            for i in range(nA):
                max_j=-1
                maxScore = -1
                found = 0
                for j in range(nB):
                    # Find d_ij
                    d_ij = np.subtract(candB[j][:2], candA[i][:2])
                    norm = np.linalg.norm(d_ij)
                    if norm:
                        d_ij = d_ij / norm
                    else:
                        continue
                    # Find p(u)
                    interp_coord = list(zip(np.linspace(candA[i][0], candB[j][0], num=n_interp_samples),
                                            np.linspace(candA[i][1], candB[j][1], num=n_interp_samples)))
                    # Find L(p(u))
                    paf_interp = []
                    for k in range(len(interp_coord)):
                        paf_interp.append([pafA[int(round(interp_coord[k][1])), int(round(interp_coord[k][0]))],
                                           pafB[int(round(interp_coord[k][1])), int(round(interp_coord[k][0]))] ])
                    # Find E
                    paf_scores = np.dot(paf_interp, d_ij)
                    avg_paf_score = sum(paf_scores)/len(paf_scores)

                    # Check if the connection is valid
                    # If the fraction of interpolated vectors aligned with PAF is higher then threshold -> Valid Pair
                    if ( len(np.where(paf_scores > paf_score_th)[0]) / n_interp_samples ) > conf_th :
                        if avg_paf_score > maxScore:
                            max_j = j
                            maxScore = avg_paf_score
                            found = 1
                # Append the connection to the list
                if found:
                    valid_pair = np.append(valid_pair, [[candA[i][3], candB[max_j][3], maxScore]], axis=0)

            # Append the detected connections to the global list
            valid_pairs.append(valid_pair)
        else: # If no keypoints are detected
            print("No Connection : k = {}".format(k))
            invalid_pairs.append(k)
            valid_pairs.append([])
    return valid_pairs, invalid_pairs



# This function creates a list of keypoints belonging to each person
# For each detected valid pair, it assigns the joint(s) to a person
def getPersonwiseKeypoints(valid_pairs, invalid_pairs):
    # the last number in each row is the overall score
    personwiseKeypoints = -1 * np.ones((0, nPoints + 1))

    for k in range(len(mapIdx)):
        if k not in invalid_pairs:
            partAs = valid_pairs[k][:,0]
            partBs = valid_pairs[k][:,1]
            indexA, indexB = np.array(POSE_PAIRS[k])

            for i in range(len(valid_pairs[k])):
                found = 0
                person_idx = -1
                for j in range(len(personwiseKeypoints)):
                    if personwiseKeypoints[j][indexA] == partAs[i]:
                        person_idx = j
                        found = 1
                        break

                if found:
                    personwiseKeypoints[person_idx][indexB] = partBs[i]
                    personwiseKeypoints[person_idx][-1] += keypoints_list[partBs[i].astype(int), 2] + valid_pairs[k][i][2]

                # if find no partA in the subset, create a new subset
                elif not found and k < (nPoints - 1):
                    row = -1 * np.ones(nPoints + 1)
                    row[indexA] = partAs[i]
                    row[indexB] = partBs[i]
                    # add the keypoint_scores for the two keypoints and the paf_score
                    row[-1] = sum(keypoints_list[valid_pairs[k][i,:2].astype(int), 2]) + valid_pairs[k][i][2]
                    personwiseKeypoints = np.vstack([personwiseKeypoints, row])
    return personwiseKeypoints

<Part of a code that finds a valid pairs>

You can download the entire source code from my Github.

Now run the program and check the results.

root@spypiggy-nx:/usr/local/src/study/opencv_dnn# python3 op_cv.py --image=/usr/local/src/image/walking.jpg --model=body25          
root@spypiggy-nx:/usr/local/src/study/opencv_dnn# python3 op_cv.py --image=/usr/local/src/image/walking.jpg --model=coco

<body-25 model result and coco model result>

So far, I have briefly seen how to use OpenPose in OpenCV's dnn module. Next, I will install OpenCV 4.5 and test the speed to use the GPU acceleration function of the OpenCV dnn module.

Install OpenCV 4.5 and rebuild OpenPose

Jetpack 4.5 provides OpenCV 4.1 by default. Therefore, to implement dnn using GPU in OpenCV, it is necessary to upgrade to OpenCV 4.5(It is possible if the version is 4.2 or higher).

For the OpenCV 4.5 upgrade, refer to the following article.

• JetPack 4.5 - OpenCV 4.1 vs. OpenCV 4.5

And if you installed OpenPose 1.7 with OpenCV 4.1 installed, it is recommended to rebuild OpenPose 1.7 after installing OpenCV 4.5. Unless you rebuild OpenPose 1.7, there is no problem using OpenCV's dnn module, which is introduced in this article. However, if you use the Python module provided by OpenPose, an error occurs because of the OpenCV version as follows.

root@spypiggy-nx:/usr/local/src/study/opencv_dnn# python3 original.py
Traceback (most recent call last):
  File "sample.py", line 2, in <module>
    from openpose import pyopenpose as op
  File "/usr/lib/python3.6/dist-packages/openpose/__init__.py", line 1, in <module>
    from . import pyopenpose as pyopenpose
ImportError: libopencv_highgui.so.4.1: cannot open shared object file: No such file or directory

This error occurs because OpenCV was upgraded to 4.5. So, if you rebuild OpenPose with OpenCV 4.5 installed, this error will disappear. 

For the OpenPose1.7 installation, refer to the following article.

• Jetpack 4.5 - Install the latest version of OpenPose on Jetson Nano

If you have not installed OpenPose, you can install OpenPose after upgrading OpenCV to 4.5.

GPU mode and CPU mode speed comparison

For testing, I used about 10 seconds in Chaplin's movie.

<Charlie Chaplin's Modern Times>

I measured the time it took to process a total of 238 frames in cpu mode and gpu mode.

root@spypiggy-nx:/usr/local/src/study/opencv_dnn# python3 op_cv_video.py --video=/usr/local/src/image/chaplin.mp4 --model=coco --device=cpu
......
......

Frame[238] processed time[15.89]
Total processed time[3821.73]
avg frame processing rate :16.06 

root@spypiggy-nx:/usr/local/src/study/opencv_dnn# python3 op_cv_video.py --video=/usr/local/src/image/chaplin.mp4 --model=coco --device=gpu
......
......

Frame[238] processed time[0.96]
Total processed time[241.08]
avg frame processing rate :1.01 

The processing time was reduced to 1/16, and it showed amazing performance. And you can see that the keypoints are being displayed normally in the output video.

<gpu-coco-output.mp4>

If you are using OpenCV dnn in the Jetson series, this is why you should upgrade OpenCV to 4.2 or higher. If you use frameworks such as Tensorflow, Caffe, PyTorch, YOLO, etc. other than OpenPose in the Jetson series, this is also a reason to upgrade OpenCV.

OpenPose built in python module vs. OpenCV dnn

Let's compare the performance of the Python module provided by OpenPose with the dnn module of OpenCV tested earlier. 

For testing, let's simplify an example program using OpenPose's Python module and test it with the same Charlie Chaplin video.

import cv2
import time
import numpy as np
from random import randint
import argparse
import sys, time
from openpose import pyopenpose as op

parser = argparse.ArgumentParser(description='Run keypoint detection')
parser.add_argument("--device", default="gpu", help="Device to inference on")
parser.add_argument("--video", default="/usr/local/src/image/chaplin.mp4", help="Input video")
parser.add_argument("--model", default="body25", help="model : body25 or coco")
args = parser.parse_args()



threshold = 0.2

if args.model == 'body25':
    #Body_25 model use 25 points
    key_points = {
        0:  "Nose", 1:  "Neck", 2:  "RShoulder", 3:  "RElbow", 4:  "RWrist", 5:  "LShoulder", 6:  "LElbow",
        7:  "LWrist", 8:  "MidHip", 9:  "RHip", 10: "RKnee", 11: "RAnkle", 12: "LHip", 13: "LKnee",
        14: "LAnkle", 15: "REye", 16: "LEye", 17: "REar", 18: "LEar", 19: "LBigToe", 20: "LSmallToe",
        21: "LHeel", 22: "RBigToe", 23: "RSmallToe", 24: "RHeel", 25: "Background"
    }

    #Body_25 keypoint pairs 
    POSE_PAIRS = [[1,2], [1,5], [2,3], [3,4], [5,6], [6,7],     #arm, shoulder line
                [1,8], [8,9], [9,10], [10,11], [8,12], [12,13], [13,14],  #2 leg
                [11,24], [11,22], [22,23], [14,21],[14,19],[19,20],    #2 foot  
                [1,0], [0,15], [15,17], [0,16], [16,18], #face
                [2,17], [5,18]
                ]  
                
    nPoints = 25

else:
    key_points = {
        0:  "Nose", 1:  "Neck", 2:  "RShoulder", 3:  "RElbow", 4:  "RWrist", 5:  "LShoulder", 6:  "LElbow",
        7:  "LWrist", 8:  "RHip", 9:  "RKnee", 10: "R-Ank", 11: "LHip", 12: "LKnee", 13: "LKnee", 14: "LAnkle", 
        15: "REye", 16: "LEye", 17: "REar", 18: "LEar"
    }
    POSE_PAIRS = [[1,2], [1,5], [2,3], [3,4], [5,6], [6,7],
                [1,8], [8,9], [9,10], [1,11], [11,12], [12,13],
                [1,0], [0,14], [14,16], [0,15], [15,17],
                [2,17], [5,16] ]
    nPoints = 18

alpha = 0.3

colors = [ [0,100,255], [0,100,255], [0,255,255], [0,100,255], [0,255,255], [0,100,255],
         [0,255,0], [255,200,100], [255,0,255], [0,255,0], [255,200,100], [255,0,255],
         [0,0,255], [255,0,0], [200,200,0], [255,0,0], [200,200,0], [0,0,0]]


cap = cv2.VideoCapture(args.video)
ret, img = cap.read()
if ret == False:
    print('Video File Read Error')    
    sys.exit(0)
frameHeight, frameWidth, c = img.shape

fourcc = cv2.VideoWriter_fourcc('m', 'p', '4', 'v')
out_video = cv2.VideoWriter('/tmp/%s-%s-output.mp4'%(args.model, args.device), fourcc, cap.get(cv2.CAP_PROP_FPS), (frameWidth,frameHeight))
frame = 0
inHeight = 368
t_elapsed = 0.0

params = dict()
params["model_folder"] = "/usr/local/src/openpose-1.7.0/models/"
params["net_resolution"] = "368x-1" 
params["display"] = "0"     #speed up the processing time

opWrapper = op.WrapperPython()
opWrapper.configure(params)
opWrapper.start()


while cap.isOpened():
    f_st = time.time()
    ret, img = cap.read()
    if ret == False:
        break
    frame += 1    

    datum = op.Datum()
    datum.cvInputData = img
    opWrapper.emplaceAndPop(op.VectorDatum([datum]))
    human_count = len(datum.poseKeypoints)

    frameClone = img.copy()

    #draw keypoint circle
    for human in range(human_count):
        for j in range(nPoints):
            if datum.poseKeypoints[human][j][2] > threshold:
                center = (int(datum.poseKeypoints[human][j][0]) ,  int(datum.poseKeypoints[human][j][1]))
                #cv2.circle(frameClone, datum.poseKeypoints[human][j][0:2], 5, colors[j % 17], -1, cv2.LINE_AA)
                cv2.circle(img, center, 3, colors[j % 17], -1, cv2.LINE_AA)

    #draw line
    for human in range(human_count):
        for pair in POSE_PAIRS:
            if datum.poseKeypoints[human][pair[0]][2] > threshold and datum.poseKeypoints[human][pair[1]][2] > threshold:
                S = (int(datum.poseKeypoints[human][pair[0]][0]), int(datum.poseKeypoints[human][pair[0]][1]))
                T = (int(datum.poseKeypoints[human][pair[1]][0]), int(datum.poseKeypoints[human][pair[1]][1]))
                center = (int(datum.poseKeypoints[human][j][0]) ,  int(datum.poseKeypoints[human][j][1]))
                cv2.line(frameClone, S, T, colors[pair[0] % 17], 3, cv2.LINE_AA)


    out_video.write(frameClone)
    f_elapsed = time.time() - f_st 
    t_elapsed += f_elapsed
    print('Frame[%d] processed time[%4.2f]'%(frame, f_elapsed))


print('Total processed time[%4.2f]'%(t_elapsed))
print('avg frame processing rate :%4.2f'%(t_elapsed / frame))
cap.release()
out_video.release()

<op_video.py using OpenPose's Python module>

Now run the code and check the performance.

root@spypiggy-nx:/usr/local/src/study/opencv_dnn# python3 op_video.py --model=coco
Starting OpenPose Python Wrapper...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
......
......

Frame[238] processed time[0.34]
Total processed time[88.42]
avg frame processing rate :0.37

When using OpenPose's Python module, the performance is almost three times that of using OpenCV dnn GPU mode. 

Perhaps the reason for this difference in performance is that the part that connects ketpoint pairs using PAF is handled in C/C++ code in OpenPose, while in Python in the example using OpenCV dnn introduced earlier.

And while the OpenCV dnn module produced 26X3 output images, OpenPose omitting this process would have had an effect.

Wrapping up

I implemented OpenCV's keypoint recognition using OpenCV's dnn module. Since this method loads and processes Caffe models directly from OpenCV, there is an advantage that you do not need to install OpenPose if you only download the model.

In the case of using the OpenPose module, it was easy to apply because it provides the result value by classifying key points by person. However, in the case of using the OpenCV dnn module, only the keypoint value is provided, so there is a difficulty in connecting valid keypoints in units of people again using a PAF vector.

You can implement it yourself by referring to the example introduced above or the example provided on learnopencv's github site, but it takes a lot of study to understand the whole content. So, if you are only interested in keypoint extraction rather than using PAF vectors, it is much easier to use the Python module provided by OpenPose. When using OpenPose's Python module, the performance is also 3 times better than when using OpenCV dnn.

The source code can be downloaded from my github.