TensorRT(High speed inference engine) - 2. Reduce loading time for TensrRT models

I used JetsonSeries(Nano, TX2), Ubuntu 18.04 Official image with root account.
This article introduces the contents of JK Jung's blog. It takes a lot of time to load the network model in TensorFlow. More time(extreamely long time) is required to load the TensorRT model in Tensorflow.

He saids
"
When I first tried out TensorRT integration in TensorFlow (TF-TRT) a few months ago, I encountered this “extremely long model loading time problem” with tensorflow versions 1.9, 1.10, 1.11 and 1.12. This problem has since been reported multiple times on NVIDIA Developer Forum: for example, here, here, and here. As a result, I was forced to use an older version of tensorflow which could suffer from incompatibility with models trained with newer version of tensorflow…
Thanks to Dariusz, one of the readers, I now have a solution to the problem.
"

How to fix the “extremely long model loading time problem of TF-TRT”

The root cause of the problem is: the default ‘python implementation’ of python3 ‘protobuf’ module runs too slowly on the Jetson platforms. And the solution is simply to replace it with ‘cpp implementaion’ of that same module.

First check the protobuf version of your Jetson Series. In JK Jung's blog, his protobuf version is 3.6.1. But Jetpack is constantly being upgraded, so you need to check the protobuf version of your Jetson at this time.

spypiggy@spypiggy-desktop:~$ pip3 show protobuf
Name: protobuf
Version: 3.9.1
Summary: Protocol Buffers
Home-page: https://developers.google.com/protocol-buffers/
Author: None
Author-email: None
License: 3-Clause BSD License
Location: /usr/local/lib/python3.6/dist-packages
Requires: six, setuptools

Download the script file and modify the version

First download the script file and modify the script file to replace the version number.

$ wget https://raw.githubusercontent.com/jkjung-avt/jetson_nano/master/install_protobuf-3.6.1.sh -O install_protobuf.sh
$ vim install_protobuf.sh

In my case, version is 3.9.1, so I replaced "3.6.1" to "3.9.1".

#!/bin/bash

set -e

folder=${HOME}/src
mkdir -p $folder

echo "** Download protobuf-3.9.1 sources"
cd $folder
if [ ! -f protobuf-python-3.9.1.zip ]; then
  wget https://github.com/protocolbuffers/protobuf/releases/download/v3.9.1/protobuf-python-3.9.1.zip
fi
if [ ! -f protoc-3.9.1-linux-aarch_64.zip ]; then
  wget https://github.com/protocolbuffers/protobuf/releases/download/v3.9.1/protoc-3.9.1-linux-aarch_64.zip
fi

echo "** Install protoc"
unzip protobuf-python-3.9.1.zip
unzip protoc-3.9.1-linux-aarch_64.zip -d protoc-3.9.1
sudo cp protoc-3.9.1/bin/protoc /usr/local/bin/protoc

echo "** Build and install protobuf-3.9.1 libraries"
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
cd protobuf-3.9.1/
./autogen.sh
./configure --prefix=/usr/local
make
make check
sudo make install
sudo ldconfig

echo "** Update python3 protobuf module"
# remove previous installation of python3 protobuf module
sudo pip3 uninstall -y protobuf
sudo pip3 install Cython
cd python/
# force compilation with c++11 standard
sed -i '205s/if v:/if True:/' setup.py
python3 setup.py build --cpp_implementation
python3 setup.py test --cpp_implementation
sudo python3 setup.py install --cpp_implementation

echo "** Build protobuf-3.9.1 successfully"

Run the script

./install_protobuf.sh

The script would take a while to finish, take a coffee break time.

Wrapping up

With the python code, it just takes very long to deserialize the trt pb file. The solution is to use “C++ implementation (with Cython wrapper)” of the python3 protobuf module.
With the solution applied, the optimized tensorrt pb file load time can be shortened.