OpenPose 1.7 Python Programming on Jetson Series #2(video)

 This description is based on Jetpack 4.5 and OpenPose 1.7.0 (released on November 17, 2020). The Jetson series has been tested on the Nano and Xavier NX, but will work the same on the Xavier and TX2.

In the previous article, we saw how to detect the body keypoint of OpenPose using Python.

This time, we will look at how to use video files and webcam cameras.

Prerequisites

• Jetpack 4.5 - Install the latest version of OpenPose on Jetson Nano
• OpenPose 1.7 Programming on Jetson Series using CommandLine tools
• OpenPose 1.7 Python Programming on Jetson Series #1

Traditional method using OpenCV

There are many known ways to frame a video or webcam using OpenCV, and I've also used it in many articles. Extracts images frame by frame from a video or webcam. And as in the previous article, you can process it in the same way as a still image.

# From Python
# It requires OpenCV installed for Python
import sys
import cv2
import os, time
from sys import platform
import argparse
from openpose import pyopenpose as op
from datetime import datetime

try:
    # Flags
    parser = argparse.ArgumentParser()
    parser.add_argument("--video_path", default="/usr/local/src/openpose-1.7.0/examples/media/video.avi", help="Process an video. ")
    args = parser.parse_known_args()

    # Custom Params (refer to include/openpose/flags.hpp for more parameters)
    params = dict()
    params["model_folder"] = "/usr/local/src/openpose-1.7.0/models/"
    params["net_resolution"] = "320x-1" 
    # Add others in path?
    for i in range(0, len(args[1])):
        curr_item = args[1][i]
        if i != len(args[1])-1: next_item = args[1][i+1]
        else: next_item = "1"
        if "--" in curr_item and "--" in next_item:
            key = curr_item.replace('-','')
            if key not in params:  params[key] = "1"
        elif "--" in curr_item and "--" not in next_item:
            key = curr_item.replace('-','')
            if key not in params: params[key] = next_item

    # Starting OpenPose
    opWrapper = op.WrapperPython()
    opWrapper.configure(params)
    opWrapper.start()
    
    # Process video
    cap = cv2.VideoCapture(args[0].video_path)
    color = (0,0,255) #BGR
    thickness = -1      #draw inner space of circle
    font = cv2.FONT_HERSHEY_SIMPLEX
    while True:
        s = datetime.now() 
        ret,img = cap.read()
        if ret == False:
            break
        datum = op.Datum()
        datum.cvInputData = img
        opWrapper.emplaceAndPop(op.VectorDatum([datum]))
        human_count = len(datum.poseKeypoints)
        # Display Image
        
        for human in range(human_count):
            for j in range(25):
                if datum.poseKeypoints[human][j][2] > 0.01:
                    center = (int(datum.poseKeypoints[human][j][0]) ,  int(datum.poseKeypoints[human][j][1]))
                    cv2.circle(img, center, 3, color, thickness)
        e = datetime.now()
        delta = e - s
        sec = delta.total_seconds()   
        
        cv2.putText(img,'FPS[%5.2f] %d person detected'%(1/( sec),human_count),(20,30), font, 1,(255,255,255),1,cv2.LINE_AA)
        cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", img)
        cv2.waitKey(1)

except Exception as e:
    print(e)
    sys.exit(-1)
    
cap.release()
cv2.destroyAllWindows()
    

<01_2_body_from_video.py>

I circled the original image's keypoint coordinates to differentiate it from the default output image rendered by OpenPose.

root@spypiggy-nx:/usr/local/src/study# python3 01_2_body_from_video.py
Starting OpenPose Python Wrapper...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.

You should probably see output like this:

If you are using a webcam, change the parameter of cv2.VideoCapture to a number, and you do not need to modify the rest of the code.

cap = cv2.VideoCapture(0) #First Webcam, 

Caffe Method

This method is shown in the OpenPose example.

However, it is difficult to explain in detail because the documentation provided by the OpenPose team is insufficient. Personally, I prefer to use the OpenCV introduced earlier.

# From Python
# It requires OpenCV installed for Python
import sys
import cv2
import os, time
from sys import platform
import argparse
from openpose import pyopenpose as op
from datetime import datetime

def display(sec, datums):
    datum = datums[0]
    img = datum.cvInputData[:, :, :]
    human_count = len(datum.poseKeypoints)
    color = (0,0,255) #BGR
    thickness = -1      #draw inner space of circle
    font = cv2.FONT_HERSHEY_SIMPLEX

    for human in range(human_count):
        for j in range(25):
            if datum.poseKeypoints[human][j][2] > 0.01:
                center = (int(datum.poseKeypoints[human][j][0]) ,  int(datum.poseKeypoints[human][j][1]))
                cv2.circle(img, center, 3, color, thickness)

    cv2.putText(img,'FPS[%6.2f] %d person detected'%(1.0/( sec),human_count),(20,30), font, 1,(255,0,0),1,cv2.LINE_AA)
    cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", img)
    key = cv2.waitKey(1)
    return (key == 27)


def printKeypoints(datums):
    datum = datums[0]
    print("Body keypoints: \n" + str(datum.poseKeypoints))
    print("Face keypoints: \n" + str(datum.faceKeypoints))
    print("Left hand keypoints: \n" + str(datum.handKeypoints[0]))
    print("Right hand keypoints: \n" + str(datum.handKeypoints[1]))


try:
    # Flags
    parser = argparse.ArgumentParser()
    parser.add_argument("--no_display", action="store_true", help="Disable display.")
    args = parser.parse_known_args()

    # Custom Params (refer to include/openpose/flags.hpp for more parameters)
    params = dict()
    params["model_folder"] = "/usr/local/src/openpose-1.7.0/models/"
    params["net_resolution"] = "320x256" 
    params["camera_resolution"] = "640x480" 

    # Add others in path?
    for i in range(0, len(args[1])):
        curr_item = args[1][i]
        if i != len(args[1])-1: next_item = args[1][i+1]
        else: next_item = "1"
        if "--" in curr_item and "--" in next_item:
            key = curr_item.replace('-','')
            if key not in params:  params[key] = "1"
        elif "--" in curr_item and "--" not in next_item:
            key = curr_item.replace('-','')
            if key not in params: params[key] = next_item

    # Construct it from system arguments
    # op.init_argv(args[1])
    # oppython = op.OpenposePython()

    # Starting OpenPose
    opWrapper = op.WrapperPython(op.ThreadManagerMode.AsynchronousOut)
    opWrapper.configure(params)
    opWrapper.start()

    # Main loop
    userWantsToExit = False
    while not userWantsToExit:
        # Pop frame
        s = datetime.now() 
        datumProcessed = op.VectorDatum()
        if opWrapper.waitAndPop(datumProcessed):
            e = datetime.now()
            delta = e - s
            sec = delta.total_seconds()    
            if not args[0].no_display:
                # Display image
                userWantsToExit = display(sec, datumProcessed)
            print('FPS:%6.2f Total [%d] frames return'%(1.0 / (sec), len(datumProcessed)))    
            #printKeypoints(datumProcessed)
        else:
            break

except Exception as e:
    print(e)
    sys.exit(-1)

<12_1_asynchronous_custom_output.py>

I set the camera resolution to '640x480'.

root@spypiggy-nx:/usr/local/src/study# python3 12_1_asynchronous_custom_output.py
Starting OpenPose Python Wrapper...
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
Desired webcam resolution 640x480 could not being set. Final resolution: 2304x1536 in /usr/local/src/openpose-1.7.0/src/openpose/producer/webcamReader.cpp:WebcamReader():37
Auto-detecting camera index... Detected and opened camera 0.
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.

You should probably see output like this:

Everything is processed in the next two lines. However, there is not enough documentation on this part, so if you want to learn more, you have to look directly into the source code.

        datumProcessed = op.VectorDatum()
        if opWrapper.waitAndPop(datumProcessed):

You can check the comments on opWrapper.waitAndPop in the /include/openpose/wrapper/wrapper.hpp file.

The display function of the example provided by OpenPose outputs datum.cvOutputData of the image rendered by OpenPose, but I decided to draw the keypoint coordinates on the original captured image as in the previous example using OpenCV. The original image is datum.cvInputData.

I have seen occasional malfunctions when the webcam is set to high resolution and then output to the screen. The FPS value was output abnormally or the process was terminated. This phenomenon does not seem to be related to the previously described memory shortage. If there is anyone who knows the exact cause, please let me know.

Wrapping up

Processing video or webcams in OpenPose is done frame by frame, like most machine learning frameworks. You can use the traditional method using OpenCV or the API provided by OpenPose. Personally, I would like the OpenPose team to provide a more detailed manual on the API.

And as of now, I don't have enough understanding of the API, I find it much more comfortable and natural to use OpenCV's VideoCapture function.

The source code can be downloaded from my github.

OpenPose 1.7 Python Programming on Jetson Series #1(image)

This description is based on Jetpack 4.5 and OpenPose 1.7.0 (released on November 17, 2020). The Jetson series has been tested on the Nano and Xavier NX, but will work the same on the Xavier and TX2.

Prerequisites

Install OpenPose 1.7.0 and basic usage.

• Jetpack 4.5 - Install the latest version of OpenPose on Jetson Nano
• OpenPose 1.7 Programming on Jetson Series using CommandLine tools

Under the hood

 When you install OpenPose, there are Python sample files in the openpose-1.7.0/examples/tutorial_api_python directory. These files are good resources for OpenPose programming with Python.

I'll start working on these files.

root@spypiggy-nx:/usr/local/src/openpose-1.7.0/examples/tutorial_api_python# ls -al
total 60
drwxrwxr-x  2 root root 4096 11월 17 14:48 .
drwxrwxr-x 11 root root 4096 11월 17 14:48 ..
-rw-rw-r--  1 root root 2900 11월 17 14:48 01_body_from_image.py
-rw-rw-r--  1 root root 3146 11월 17 14:48 02_whole_body_from_image.py
-rw-rw-r--  1 root root 3362 11월 17 14:48 04_keypoints_from_images.py
-rw-rw-r--  1 root root 4276 11월 17 14:48 05_keypoints_from_images_multi_gpu.py
-rw-rw-r--  1 root root 3330 11월 17 14:48 06_face_from_image.py
-rw-rw-r--  1 root root 3751 11월 17 14:48 07_hand_from_image.py
-rw-rw-r--  1 root root 3675 11월 17 14:48 08_heatmaps_from_image.py
-rw-rw-r--  1 root root 3377 11월 17 14:48 09_keypoints_from_heatmaps.py
-rw-rw-r--  1 root root 3345 11월 17 14:48 12_asynchronous_custom_output.py
-rw-rw-r--  1 root root  634 11월 17 14:48 CMakeLists.txt
-rw-rw-r--  1 root root 2572 11월 17 14:48 openpose_python.py
-rw-rw-r--  1 root root  174 11월 17 14:48 README.md

Python import path Problem

All of the above examples contain the following code.

    # Import Openpose (Windows/Ubuntu/OSX)
    dir_path = os.path.dirname(os.path.realpath(__file__))
    try:
        # Windows Import
        if platform == "win32":
            # Change these variables to point to the correct folder (Release/x64 etc.)
            sys.path.append(dir_path + '/../../python/openpose/Release');
            os.environ['PATH']  = os.environ['PATH'] + ';' + dir_path + '/../../x64/Release;' +  dir_path + '/../../bin;'
            import pyopenpose as op
        else:
            # Change these variables to point to the correct folder (Release/x64 etc.)
            sys.path.append('../../python');
            # If you run `make install` (default path is `/usr/local/python` for Ubuntu), you can also access the OpenPose/python module from there. This will install OpenPose and the python library at your desired installation path. Ensure that this is in your python path in order to use it.
            # sys.path.append('/usr/local/python')
            from openpose import pyopenpose as op
    except ImportError as e:
        print('Error: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?')
        raise e

The purpose of this code is because of the Python import path problem.

However, since we copied the openpose Python package to the /usr/lib/python3.6/dist-packages directory during the installation process, these codes are unnecessary. So it doesn't matter if you erase it.

Image path, model path problem

The image path and model path name used in the test in the example code are as follows.

    parser = argparse.ArgumentParser()
    parser.add_argument("--image_path", default="../../../examples/media/COCO_val2014_000000000192.jpg", help="Process an image. Read all standard formats (jpg, png, bmp, etc.).")
    args = parser.parse_known_args()

    # Custom Params (refer to include/openpose/flags.hpp for more parameters)
    params = dict()
    params["model_folder"] = "../../../models/"

You can see that the path doesn't fit. Not "../../../examples/..." but "../../examples/..." is correct. "../../../models/" also "../../models/" is correct.

However, I recommend using absolute paths rather than relative paths. If you write new Python code in a new directory, there is an inconvenience of having to change these relative paths every time. Therefore, it is recommended to use an absolute path name. I will use the absolute path.

Memory Problem

Although the Xavier NX has 8GB of memory, it is much less than the machine learning desktop. In the case of the Jetson Nano, it's even worse. OpenPose requires more memory than you think. Therefore, it is necessary to properly adjust the network input size.

The option to adjust this value in OpenPose is net_resolution. Since OpenPose is an image-targeted operation, net_resolution has a "WxH" format. W and H values are multiples of 16. Keeping the ratio of W and H close to the aspect ratio of the input image helps to output good results.

The example code doesn't use the net_resolution value, but I'll use it appropriately.

    # Flags
    parser = argparse.ArgumentParser()
    parser.add_argument("--image_dir", default="/usr/local/src/openpose-1.7.0//examples/media/", help="Process a directory of images. Read all standard formats (jpg, png, bmp, etc.).")
    parser.add_argument("--no_display", default=False, help="Enable to disable the visual display.")
    args = parser.parse_known_args()

    # Custom Params (refer to include/openpose/flags.hpp for more parameters)
    params = dict()
    params["model_folder"] = "/usr/local/src/openpose-1.7.0/models/"
    params["net_resolution"] = "320x256"

To specify the net_resolution option in OpenPose Python, simply add it to the params dictionary.

I will copy the example files to /usr/local/src/study and edit them one by one.

cp /usr/local/src/openpose-1.7.0/examples/tutorial_api_python/* /usr/local/src/study/

01_body_from_image.py

Some of the files were modified with the above-mentioned content. The last cv2.waitKey(0) was modified for screen capture, so it is not a necessary modification.

# From Python
# It requires OpenCV installed for Python
import sys
import cv2
import os
from sys import platform
import argparse
from openpose import pyopenpose as op


try:
    # Flags
    parser = argparse.ArgumentParser()
    parser.add_argument("--image_path", default="/usr/local/src/openpose-1.7.0/examples/media/COCO_val2014_000000000192.jpg", help="Process an image. Read all standard formats (jpg, png, bmp, etc.).")
    args = parser.parse_known_args()

    # Custom Params (refer to include/openpose/flags.hpp for more parameters)
    params = dict()
    params["model_folder"] = "/usr/local/src/openpose-1.7.0/models/"
    params["net_resolution"] = "320x256" #COCO_val2014_000000000192.jpg image is landscape mode, so 320x256 is a good choice       
    # Add others in path?
    for i in range(0, len(args[1])):
        curr_item = args[1][i]
        if i != len(args[1])-1: next_item = args[1][i+1]
        else: next_item = "1"
        if "--" in curr_item and "--" in next_item:
            key = curr_item.replace('-','')
            if key not in params:  params[key] = "1"
        elif "--" in curr_item and "--" not in next_item:
            key = curr_item.replace('-','')
            if key not in params: params[key] = next_item

    # Starting OpenPose
    opWrapper = op.WrapperPython()
    opWrapper.configure(params)
    opWrapper.start()

    # Process Image
    datum = op.Datum()
    imageToProcess = cv2.imread(args[0].image_path)
    datum.cvInputData = imageToProcess
    opWrapper.emplaceAndPop(op.VectorDatum([datum]))
    human_count = len(datum.poseKeypoints)
    # Display Image
    for human in range(human_count):
        print(datum.poseKeypoints[human])
    print("Total %d human detected"%human_count)
    cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", datum.cvOutputData)
    k = 0
    while k != 27:
        k = cv2.waitKey(0) & 0xFF
except Exception as e:
    print(e)
    sys.exit(-1)

<01_body_from_image.py>

Now let's run the code. As the file name suggests, it extracts the human keypoints from the image and prints the keypoints information to the shell. And it also displays the image file to the screen.

root@spypiggy-nx:/usr/local/src/study# python3 01_body_from_image.py
Starting OpenPose Python Wrapper...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
Body keypoints:
[[[  3.29139008e+02   2.12978333e+02   7.70797849e-01]
  [  3.25197052e+02   2.16946854e+02   9.44534898e-01]
  [  2.97181213e+02   2.20953629e+02   8.90540242e-01]
  [  2.79063293e+02   2.47102036e+02   9.40675259e-01]
  [  2.95104279e+02   2.65098389e+02   8.69051158e-01]

<output>

Keypoint analysis

As you can see from the Python code, all the important data is in datum.poseKeypoints.

This value is a Python nested list as shown below.

And each keypoint sequence represents these parts. 

// Result for BODY_25 (25 body parts consisting of COCO + foot)
// const std::map<unsigned int, std::string> POSE_BODY_25_BODY_PARTS {
//     {0,  "Nose"},
//     {1,  "Neck"},
//     {2,  "RShoulder"},
//     {3,  "RElbow"},
//     {4,  "RWrist"},
//     {5,  "LShoulder"},
//     {6,  "LElbow"},
//     {7,  "LWrist"},
//     {8,  "MidHip"},
//     {9,  "RHip"},
//     {10, "RKnee"},
//     {11, "RAnkle"},
//     {12, "LHip"},
//     {13, "LKnee"},
//     {14, "LAnkle"},
//     {15, "REye"},
//     {16, "LEye"},
//     {17, "REar"},
//     {18, "LEar"},
//     {19, "LBigToe"},
//     {20, "LSmallToe"},
//     {21, "LHeel"},
//     {22, "RBigToe"},
//     {23, "RSmallToe"},
//     {24, "RHeel"},
//     {25, "Background"}
// };

You can find human counts using len() function.

human_count = len(datum.poseKeypoints) 

Next let's search all keypoints of all humans

for human in range(human_count):
    print(datum.poseKeypoints[human])

Improved 01_body_from_image.py

Using the first Python example, I simply tested the body keypoints. This time, we will improve the code and modify it to draw the number at the keypoint.

Modify a part of the code as follows.

After copying the image created in OpenPose, it was modified to print the body part number at 25 coordinates.

    # Process Image
    datum = op.Datum()
    imageToProcess = cv2.imread(args[0].image_path)
    datum.cvInputData = imageToProcess
    opWrapper.emplaceAndPop(op.VectorDatum([datum]))
    newImage = datum.cvOutputData[:, :, :]
    human_count = len(datum.poseKeypoints)
   
    font = cv2.FONT_HERSHEY_SIMPLEX
    for human in range(human_count):
        for j in range(25):
            if datum.poseKeypoints[human][j][2] > 0.01:
                cv2.putText(newImage, str(j),  ( int(datum.poseKeypoints[human][j][0]) + 10,  int(datum.poseKeypoints[human][j][1])), font, 0.5, (0,255,0), 2) 
        print(datum.poseKeypoints[human])    
    
    # Display Image
    for human in range(human_count):
        print(datum.poseKeypoints[human])
    print("Total %d human detected"%human_count)
    cv2.imshow("OpenPose 1.7.0 - Tutorial Python API", newImage)

<01_1_body_from_image.py part of the modified code>

Now let's run the code.

root@spypiggy-nx:/usr/local/src/study# python3 01_1_body_from_image.py

You will get the following screen output.

<output>

Wrapping up

Using the Python example file provided by OpenPose, I modified the Python code that extracts body keypoints optimized for the Jetson series.

The source code can be downloaded from my github.

Next time I'll continue to explore how to use video and cameras in OpenPose.

    

OpenPose 1.7 Programming on Jetson Series using CommandLine tools

 When OpenPose is built, openpose.bin or openpose.exe is automatically created. This program provides most of OpenPose's features.

Therefore, if you know how to use this program well, you can implement OpenPose functions in conjunction with this program in Python or C/C++.

Detailed usage of openpose.bin can be found in OpenPose Doc-Demo.

Prerequisites

Install OpenPose 1.7.0.

• Jetpack 4.5 - Install the latest version of OpenPose on Jetson Nano

Sample Image and video files

Image and video files to be used for testing exist in the openpose-1.7.0/examples/media directory. These files are automatically created when building OpenPose.

root@spypiggy-nx:/usr/local/src/openpose-1.7.0/examples/media# ls -al
total 4032
drwxrwxr-x  2 root root    4096 11월 17 14:48 .
drwxrwxr-x 11 root root    4096 11월 17 14:48 ..
-rwxrwxr-x  1 root root  230425 11월 17 14:48 COCO_val2014_000000000192.jpg
-rwxrwxr-x  1 root root  107714 11월 17 14:48 COCO_val2014_000000000241.jpg
-rwxrwxr-x  1 root root  208764 11월 17 14:48 COCO_val2014_000000000257.jpg
-rwxrwxr-x  1 root root   75635 11월 17 14:48 COCO_val2014_000000000294.jpg
-rwxrwxr-x  1 root root  159712 11월 17 14:48 COCO_val2014_000000000328.jpg
-rwxrwxr-x  1 root root   81713 11월 17 14:48 COCO_val2014_000000000338.jpg
-rwxrwxr-x  1 root root  131457 11월 17 14:48 COCO_val2014_000000000357.jpg
-rwxrwxr-x  1 root root  118630 11월 17 14:48 COCO_val2014_000000000360.jpg
-rwxrwxr-x  1 root root  246526 11월 17 14:48 COCO_val2014_000000000395.jpg
-rwxrwxr-x  1 root root  118150 11월 17 14:48 COCO_val2014_000000000415.jpg
-rwxrwxr-x  1 root root  102365 11월 17 14:48 COCO_val2014_000000000428.jpg
-rwxrwxr-x  1 root root  194898 11월 17 14:48 COCO_val2014_000000000459.jpg
-rwxrwxr-x  1 root root  131485 11월 17 14:48 COCO_val2014_000000000474.jpg
-rwxrwxr-x  1 root root  105570 11월 17 14:48 COCO_val2014_000000000488.jpg
-rwxrwxr-x  1 root root   22611 11월 17 14:48 COCO_val2014_000000000536.jpg
-rwxrwxr-x  1 root root  188065 11월 17 14:48 COCO_val2014_000000000544.jpg
-rwxrwxr-x  1 root root  130509 11월 17 14:48 COCO_val2014_000000000564.jpg
-rwxrwxr-x  1 root root  129689 11월 17 14:48 COCO_val2014_000000000569.jpg
-rwxrwxr-x  1 root root   96802 11월 17 14:48 COCO_val2014_000000000589.jpg
-rwxrwxr-x  1 root root  112452 11월 17 14:48 COCO_val2014_000000000623.jpg
-rw-rw-r--  1 root root 1395096 11월 17 14:48 video.avi

OpenPose models

During the build process using cmake, we have already downloaded several models.

With the 3 models we downloaded, most of the work is possible.

Model Name	Description	Installed
BODY_25_MODEL	Openpose's basic model for body keypoint detection	Y
BODY_COCO_MODEL	Microsoft's COCO(Common Objects in Context) model for keypoint detection. Refer to "JetsonNano-Human Pose estimation using tensorflow" for a description of the key points used in this model.	N
BODY_MPI_MODEL	mpii (max planck institut informatik) model for keypoint detection. Refer to "Jetson Xavier NX - Human Pose estimation using tensorflow (mpii)" for a description of the key points used in this model.	N
BODY_FACE_MODEL	Openpose's basic model for face keypoint detection	Y
BODY_HAND_MODEL	Openpose's basic model for hand keypoint detection	Y

Model files exist in the openpose-1.7.0/models directory as follows.

root@spypiggy-nx:/usr/local/src/openpose-1.7.0/models# ls -al hand/ pose/ face/
face/:
total 150816
drwxrwxr-x 2 root root      4096  2월 13 20:54 .
drwxrwxr-x 6 root root      4096 11월 17 14:48 ..
-rw-rw-r-- 1 root root    676709 11월 17 14:48 haarcascade_frontalface_alt.xml
-rw-rw-r-- 1 root root     25962 11월 17 14:48 pose_deploy.prototxt
-rw-r--r-- 1 root root 153717332  2월 13 20:54 pose_iter_116000.caffemodel

hand/:
total 143928
drwxrwxr-x 2 root root      4096  2월 13 20:54 .
drwxrwxr-x 6 root root      4096 11월 17 14:48 ..
-rwxrwxr-x 1 root root     26452 11월 17 14:48 pose_deploy.prototxt
-rw-r--r-- 1 root root 147344024  2월 13 20:55 pose_iter_102000.caffemodel

pose/:
total 20
drwxrwxr-x 5 root root 4096 11월 17 14:48 .
drwxrwxr-x 6 root root 4096 11월 17 14:48 ..
drwxrwxr-x 2 root root 4096  2월 13 20:54 body_25
drwxrwxr-x 2 root root 4096 11월 17 14:48 coco
drwxrwxr-x 2 root root 4096 11월 17 14:48 mpi

Body Keypoints

The location of the coordinates used in the most commonly used BODY-25-MODEL is as follows.

<image from https://github.com/infocom-tpo/SwiftOpenPose/issues/14>

Whenever in doubt, always use the --help option. This option describes in detail all the features openpose.bin provides. Among the many options, important options that are frequently used are as follows. Options not described here can also be checked using the --help option.

keypoint detection using openpose.bin

Body Keypoint detection

Let's use openpose.bin to detect keypoints in the image file. Make the /usr/local/src/output directory in advance.

./build/examples/openpose/openpose.bin --image_dir ./examples/media --write_images /usr/local/src/output  --write_json /usr/local/src/output --net_resolution "320x224"

Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
OpenPose demo successfully finished. Total time: 10.629069 seconds.

Perhaps the following files were created in the /usr/local/src/output directory.

option	Description
image_dir	The directory path where OpenPose will work. Save image files to work in this directory.
write_dir	The path to the directory where OpenPose's job results are stored.
write_json	Directory to write OpenPose output in JSON format. It includes body, hand, and face pose keypoints (2-D and 3-D), as well as pose candidates
net_resolution	Multiples of 16. If it is increased, the accuracy potentially increases. If it is decreased, the speed increases. For maximum speed-accuracy balance, it should keep the closest aspect ratio possible to the images or videos to be processed. Using `-1` in any of the dimensions, OP will choose the optimal aspect ratio depending on the user's input value. Jetson Nano, Xavier NX does not have enough memory. Therefore, it is recommended to use the appropriate value. If this value is not specified, the process may be forcibly terminated in the Jetson series.

If you are a programmer you are more interested in json files than output images. The json file stores keypoint information detected by OpenPose. 

The following is a comparison between the output image and the json file. It can be seen that the keypoint information of the three people detected in the image is included in the json file. The way to read the json file is to divide it by 3 numbers. The first two are coordinates, and the last is accuracy. 

Since we are marking 25 body keypoints, all 25 X 3 = 75 numbers will be displayed per person. If the keypoint is not detected, all 3 numbers are displayed as 0.

<COCO_val2014_000000000328_rendered.png>

<COCO_val2014_000000000328_keypoints.json>

Body, Hand, Face Keypoint detection

Key points of the hand face can be easily added with the --face and --hand options.

./build/examples/openpose/openpose.bin --image_dir /usr/local/src/image \

--write_images /usr/local/src/output  \

--write_json /usr/local/src/output \

--net_resolution "368x320" \

--hand \
--hand_net_resolution "256x256" \
--face \
--face_net_resolution "256x256"

Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
OpenPose demo successfully finished. Total time: 15.909267 seconds.

Perhaps the following files were created in the /usr/local/src/output directory.

<output image files>

This time, you can see that the json file also includes face information and hand information.

<part of json file>

option	Description
hand	Whether the hand keypoint is detected
hand_net_resolution	Multiples of 16 and squared. Analogous to `net_resolution` but applied to the hand keypoint detector. This value must be squared rectangle. Jetson Nano, Xavier NX does not have enough memory. Therefore, it is recommended to use the appropriate value. If this value is not specified, the process may be forcibly terminated in the Jetson series.
face	Whether the face keypoint is detected
face_net_resolution	Multiples of 16 and squared. Analogous to `net_resolution` but applied to the face keypoint detector. This value must be squared rectangle. Jetson Nano, Xavier NX does not have enough memory. Therefore, it is recommended to use the appropriate value. If this value is not specified, the process may be forcibly terminated in the Jetson series.

If you use default hand_detector, each hand consists of 21 key points. Like body, three values are one key point information, and it consists of two coordinates and one probability value.

<21 keypoint hand model>

If you use default face_detector, the face consists of 70 key points. Like body, three values are one key point information, and it consists of two coordinates and one probability value. The 70 keypoint is the same as the 68 keypoint used in many face recognition such as dlib. However, only two coordinates of the iris part of both eyes were added.

<70 keypoint face model>

Detecting keypoints in video files

You can detect keypoints in video files by specifying the path to the video file using --video instead of --image_dir.

./build/examples/openpose/openpose.bin \
--video ./examples/media/video.avi \
--net_resolution "256x128" \
--write_json /usr/local/src/output \
--write_video /usr/local/src/output/result.avi

Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
OpenPose demo successfully finished. Total time: 65.039817 seconds.

This command saves the output as a result.avi video file. And it saves the keypoint information extracted from every frame as a json file. The sample video file consists of 205 frames. Therefore, 205 json files are created.

root@spypiggy-nx:/usr/local/src/output# ls -l
total 17088
-rw-r--r-- 1 root root 16582436  2월 14 20:38 result.avi
-rw-r--r-- 1 root root     2996  2월 14 20:37 video_000000000000_keypoints.json
-rw-r--r-- 1 root root     2379  2월 14 20:37 video_000000000001_keypoints.json
-rw-r--r-- 1 root root     3053  2월 14 20:37 video_000000000002_keypoints.json
-rw-r--r-- 1 root root     3259  2월 14 20:37 video_000000000003_keypoints.json

......

......

-rw-r--r-- 1 root root     4093  2월 14 20:38 video_000000000202_keypoints.json
-rw-r--r-- 1 root root     4174  2월 14 20:38 video_000000000203_keypoints.json
-rw-r--r-- 1 root root     4197  2월 14 20:38 video_000000000204_keypoints.json

However, if you use openpose.bin to process video files or camera frames, it is very difficult to process in real time because the analysis result for the video frame is output as a file.

Useful options

option	Description
logging_level	Integer in the range [0, 255]. 0 will output any opLog() message, while 255 will not output any. Current OpenPose library messages are in the range 0-4: 1 for low priority messages and 4 for important ones.) type: int32 default: 3
display	Display mode: -1 for automatic selection; 0 for no display (useful if there is no X server and/or to slightly speed up the processing if visual output is not required); 2 for 2-D display; 3 for 3-D display (if `--3d` enabled); and 1 for both 2-D and 3-D display.) type: int32 default: -1 If you use -1, It can achieve a performance improvement of about 30%.
render_pose	if you are not using --write_video or --write_images, you can speed up the processing using -render_pose 0

Wrapping up

The openpose.bin is made to perform various functions through options. Therefore, if you are familiar with how to use this program, you can implement OpenPose through control using a familiar programming language.

Controlling openpose.bin using Python

The following simple Python code is the execution of the following command using Python.

./build/examples/openpose/openpose.bin --image_dir /usr/local/src/image --write_images /usr/local/src/output  --write_json /usr/local/src/output --net_resolution "320x224"

import subprocess, os, sys, time

exec = '/usr/local/src/openpose-1.7.0/build/examples/openpose/openpose.bin'
working_dir = '/usr/local/src/openpose-1.7.0/'
image_dir_info = "/usr/local/src/image"
write_images_info = "/usr/local/src/output"
write_json_info = "/usr/local/src/output"
net_resolution_info = "320x224"

params = []
params.append(exec)
params.append("--image_dir")
params.append(image_dir_info)
params.append("--write_images")
params.append(write_images_info)
params.append("--write_json")
params.append(write_json_info)
params.append("--net_resolution")
params.append(net_resolution_info)
#no display will speed up the processing time.
params.append("--display")
params.append("0")

#chdir is necessary. If not set, searching model path may fail
os.chdir(working_dir)
s = time.time()
process = subprocess.Popen(params, stdout=subprocess.PIPE)
output, err = process.communicate()
exit_code = process.wait()
e = time.time()
output_str = output.decode('utf-8')
print("Python Logging popen exit code :%d"%exit_code)
print("Python Logging popen return :%s"%output_str)
print("Python Logging Total Processing time:%6.2f"%(e - s))

<op_control.py>

When the Python program is executed, the option value is passed along with the call to openpose.bin. And the output image we want and the json file will be saved in the /usr/local/src/output directory. It's up to you to modify your Python code to do the following with the json files stored in this directory.

root@spypiggy-nx:/usr/local/src/study# python3 op_control.py
Python Logging popen exit code :0
Python Logging popen return :Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
OpenPose demo successfully finished. Total time: 5.128867 seconds.

Python Logging Total Processing time:  5.52

The openpose.bin program alone can implement many functions, but in the next article, we will look at how to implement openpose directly in Python code. In the previous post JetsonNano-Human Pose estimation using OpenPose, some codes do not work as the API is changed on OpenPose 1.7. I'll go ahead and fix some code that doesn't work.

The source code can be downloaded from https://github.com/raspberry-pi-maker/NVIDIA-Jetson.