This article assumes that you have read the following two blogs. So, if you haven't seen the two blogs below, I recommend reading them in advance.
Although this article explains what works on Xavier NX, if you use JetPack 4.4 or higher, it will work on TX2, Nano as well.
In "Xavier NX-Remote Python Development using VSCode" I introduced you how to debug Xavier NX's python code on a remote computer.
It is also possible to debug by installing VSCode directly on Xavier NX or Nano. However, I do most of the work using the Jetson series remotely using ssh on my WIndows laptop. While this way of working is familiar to me, debugging Python is inconvenient. Simple debugging can be done using the print function. However, the variable data of the Python code related to AI needs to check the value of one variable, not just the value of a single variable, but a large multidimensional data moment by moment. Because of this, there is a limit to using the print function.
In this article, I will look at how to remotely debug the Python code introduced in "Xavier NX-NVIDIA AI IOT-Human Pose estimation using TensorRT" directly using VSCode.
Run VSCode and connect to Xavier NX
The following figure is the screen described in "Xavier NX-Remote Python Development using VSCode". After reading the contents of the article "Xavier NX-Remote Python Development using VSCode", please follow this screen.
<Screen connected to remote Xavier NX using VSCode>
Let's run the example of "Xavier NX-NVIDIA AI IOT-Human Pose estimation using TensorRT" in the VSCode terminal to see if it works.
The "Unable to init server" message is a GUI related warning caused by using OpenCV and PIL. I can ignore this warning because I didn't put the code to print the image file to the screen in the example code. If you look at the picture above, you can see that it works normally after warning.
Debug trt_pose detect_image2.py
Now let's debug the TensorRT Pose Estimation example.
First, open the detect_image2.py file, then click the Debug button on the left side of VSCode. Then the "create a launch.json file" screen appears as shown in the following figure.
<creating launch.json file>
Click "create a launch.json file" to display the Debug Configuration option. Here you select the option to debug the currently open Python file.
The following launch.json file will appear.
Since our Python code uses runtime parameters, we add an args entry to the launch.json file. And it's a good idea to specify the working directory where the code will run. The reason is that you open the human_pose.json file in the detect_image2.py file, and if you don't use an absolute path in your Python code, you need to specify the working directory to avoid an error.
<launch.json>
Now is the time to finally debug.
Set a breakpoint in the code you want to examine. To set it up, simply click next to the line number. Then a red dot appears.
<set breakpoint>
Once you have set the breakpoint, press the green triangle button to run the code.
If everything is OK, code execution will stop at that line as shown in the following figure. And the current values of variables are shown in real time on the left pane.
You can debug while moving in units of lines or functions by using the debugging button at the top of the VSCode screen. This is a great feature that can reduce your debugging time!
If you move the mouse cursor to the variable you want to investigate while proceeding the code line by line, you can check the value of the current variable in real time.
<Real-time variable value check using mouse cursor>
Measuring the confidence of trt_pose keypoints
Recently, there was someone who asked me if I could check the accuracy of the keypoints detected using trt_pose. Let's check if we can find the confidence value of the keypoint using the debugging function introduced in this article.
Since the confidence value will vary from keypoint to keypoint, if confidence value exists it will have to exist with the keypoint coordinates.
<keypoint value check>
Keypoint values are taken from the peaks variable. Examining these peaks values reveals that there are only two values. This value is a coordinate value indicating the location of the key point. Therefore, it can be seen that the confidence value of the keypoint is not currently provided by trt_pose.
Providing keypoint extraction threshold in trt_pose
The confidence value of the keypoint is unknown, but the threshold value can be provided when extracting the keypoints.
<ParseObjects check>
Examining parse_objects shows the values of cmap_threshold and link_threshold. These values are the threshold values used when extracting key points. Adjusting these values makes the keypoint extraction slightly different.
Change these thresholds to 0.3, then test. The way to change the threshold is to provide this value as a parameter in the code that creates parse_objects.
Let's change the code as above and compare the created image files.
<threshold 0.1> <threshold 0.3>
At the threshold of 0.3, it can be seen that the keypoint is detected a little more strictly.
Wrapping up
I used VSCode to debug the Jetson series' trt_pose example on a remote computer. The method introduced in this article can be useful for all Python development such as tensorflow, pytorch, opencv, etc.
