Today, I will introduce image processing using CartoonGAN, which is introduced at http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2205.pdf.
In fact, what I'm introducing today is more useful on a desktop with better memory and GPU performance than a Jetson Nano .Today's introduction is from https://github.com/Yijunmaverick/CartoonGAN-Test-Pytorch-Torch.
Prerequisites
This example requires PyTorch. To install PyTorch, see https://spyjetson.blogspot.com/2019/12/jetsonnano-human-pose-estimation-using.html.This link explains how to install the latest PyTorch 1.4 on Jetpack 4.3. And TorchVision requires PIL as a base. However, the error message "cannot import name 'PILLOW_VERSION'" occurs when using the latest Pillow 7.0. This error is due to the deletion of the PILLOW_VERSION definition in Pillow 7.0. Perhaps sooner or later PyTorch will release a new version that fixes this bug. However, we need to avoid this error right now, so we need to install Pillow version 6.X.
I already have 7.0 installed, but it was newly installed as a lower version with the below command.
root@spypiggy-nano:/usr/local/src/CartoonGAN-Test-Pytorch-Torch# pip3 install "pillow<7"Collecting pillow<7Downloading https://files.pythonhosted.org/packages/b3/d0/a20d8440b71adfbf133452d4f6e0fe80de2df7c2578c9b498fb8120 83383/Pillow-6.2.2.tar.gz (37.8MB)
100%|████████████████████████████████|37.8MB14kB/s
Building wheels for collected packages:pillowRunningsetup.pybdist_wheelforpillow...doneStoredindirectory:/root/.cache/pip/wheels/f6/0a/7c/5e6567101a10388b915c4ebf73edb849f73908ad154e9eb9bc
Successfully built pillow
Installing collected packages:pillowFoundexistinginstallation:Pillow7.0.0UninstallingPillow-7.0.0:SuccessfullyuninstalledPillow-7.0.0Successfully installed pillow-6.2.2
Install CartoonGAN
And download the source code.
cd /usr/local/src
git clone https://github.com/Yijunmaverick/CartoonGAN-Test-Pytorch-Torch.git
Then download the pre-trained models using the script files.
cd /usr/local/src/CartoonGAN-Test-Pytorch-Torch
sh pretrained_model/download_pth.sh
Test
Some of the example source code provided has been modified to fit the latest PyTorch.
And Jetson Nano's 4GB of memory isn't enough to run PyTorch. Therefore, if the load_size value is 450, the process may suddenly stop due to insufficient memory while processing some images. If this happens, reduce this value and test.
If you use --style = Hayao, --style = Hosoda, --style = Paprika in the above Python command, you can test by changing the model.
You can download the source code at https://github.com/raspberry-pi-maker/NVIDIA-Jetson
I used Jetson Nano, JetPack 4.3 Official image with root account.
What is BeautyGAN?
BeautyGAN is Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network. Official website(http://liusi-group.com/projects/BeautyGAN) summarizes BeautyGAN as follows.
Facial makeup transfer aims to translate the makeup style from a given reference makeup face image to another non-makeup one while preserving face identity. Such an instance-level transfer problem is more challenging than conventional domain-level transfer tasks, especially when paired data is unavailable. Makeup style is also different from global styles (e.g., paintings) in that it consists of several local styles/cosmetics, including eye shadow, lip-stick, foundation, and so on. Extracting and transferring such local and delicate makeup information is infeasible for existing style transfer methods. We address the issue by incorporating both global domain-level loss and local instance-level loss in an dual input/output Generative Adversarial Network, called BeautyGAN. Specically, the domain-level transfer is ensured by discriminators that distinguish generated images from domains’ real samples. The instance-level loss is calculated by pixel-level histogram loss on separate local facial regions. We further introduce perceptual loss and cycle consistency loss to generate high quality faces and preserve identity. The overall objective function enables the network to learn translation on instance-level through unsupervised adversarial learning. We also build up a new makeup dataset that consists of 3834 high-resolution face images. Extensive experiments show
The following figure is a schematic diagram of the operation flow of BeautyGAN.
Figure : Framework of the proposed BeautyGAN. The upper pipeline shows the overall system.Gaccpets two images asinputs: non-makeup imageIsr c, reference makeup imageIr e f, and generates two outputs: transferred makeup imageIBsr c, anti-makeup imageIAr e f. The generated images are fed into the sameGto build up reconstruction results:Ir ecsr c,Ir ecr e f. There are fourloss terms for trainingG: cycle consistency loss, perceptual loss, adversarial loss (denoted asDAandDB) and makeup loss. Thelower pipeline shows the details of makeup loss. It consists of three local histogram loss terms acted on face, eye shadow andlips, respectively. We first utilize face parsing model to separate each cosmetic region ofIsr c,Ir e f,IBsr c. Then, for each region,we employ histogram matching betweenIsr candIr e fto obtain a histogram remapping facial region as ground truth. The localloss term calculates pixel-level differences between such ground truth and corresponding cosmetic region ofIBsr c.
Tensorflow is required to operate BeautyGAN. If TensorFlow is not installed on your Jetson Nano, install TensorFlow first. And you need OpenCV to run GitHub's example code that introduced BeautyGAN. If you are using JetPack4.3, OpenCV 4.1 will be installed by default. If you are using JetPack 4.2 or earlier, you must install OpenCV separately.
And you may need to install imageio packages to run main.py example.
pip3 install imageio
Install BeautyGAN
Select and download one of the two Githubs introduced above. I will download https://github.com/kairess/BeautyGAN. This site provides excellent Jupyter Notebook examples and additional Youtube videos.
<Youtube video provided by kairess>
cd /usr/local/src
git clone https://github.com/kairess/BeautyGAN.git
Download the pre-trained models
You can download models that have already been trained from the following two sites.
Download these files and copy them to the "/usr/local/src/BeautyGAN/models" directory.
Testing main.py
BeautyGAN provides main.py for testing. With this program, you can take a brief look at the operation of BeautyGAN.
Be Careful: If you downloaded source codes from https://github.com/kairess/BeautyGAN, you need to slightly modify the model path in the main.py file. Honlan's original source code uses the model directory name, while kairess's fork version uses the models directory. Therefore, change the "model" directory of the main.py file to "models".
# -*- coding: utf-8 -*-importtensorflowastfimportnumpyasnpimportosimportglobfromimageioimportimread,imsaveimportcv2importargparseparser=argparse.ArgumentParser()parser.add_argument('--no_makeup',type=str,default=os.path.join('imgs','no_makeup','xfsy_0068.png'),help='path to the no_makeup image')args=parser.parse_args()defpreprocess(img):return(img/255.-0.5)*2defdeprocess(img):return(img+1)/2batch_size=1img_size=256no_makeup=cv2.resize(imread(args.no_makeup),(img_size,img_size))X_img=np.expand_dims(preprocess(no_makeup),0)makeups=glob.glob(os.path.join('imgs','makeup','*.*'))result=np.ones((2*img_size,(len(makeups)+1)*img_size,3))result[img_size:2*img_size,:img_size]=no_makeup/255.tf.reset_default_graph()sess=tf.Session()sess.run(tf.global_variables_initializer())# I modified model ->modelssaver=tf.train.import_meta_graph(os.path.join('models','model.meta'))saver.restore(sess,tf.train.latest_checkpoint('models'))graph=tf.get_default_graph()X=graph.get_tensor_by_name('X:0')Y=graph.get_tensor_by_name('Y:0')Xs=graph.get_tensor_by_name('generator/xs:0')foriinrange(len(makeups)):makeup=cv2.resize(imread(makeups[i]),(img_size,img_size))Y_img=np.expand_dims(preprocess(makeup),0)Xs_=sess.run(Xs,feed_dict={X:X_img,Y:Y_img})Xs_=deprocess(Xs_)result[:img_size,(i+1)*img_size:(i+2)*img_size]=makeup/255.result[img_size:2*img_size,(i+1)*img_size:(i+2)*img_size]=Xs_[0]imsave('result.jpg',result)
<main.py>
Before going through the code, let's run it once to see the results.
Tips: The TensorFlow example above requires a lot of memory. If an error occurs due to insufficient memory, changing Ubuntu Desktop to LXDE can secure an additional 1 GB of memory. I also got the correct results after using this method. Please refer to my other post https://spyjetson.blogspot.com/2019/09/jetson-nano-useful-tips-before-you.html for LXDE Desktop changes.
root@spypiggy-nano:/usr/local/src/BeautyGAN# python3 main.py
2020-04-1515:50:11.401865:Itensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0WARNING:tensorflow:Frommain.py:29:Thenametf.reset_default_graphisdeprecated.Pleaseusetf.compat.v1.reset_default_graphinstead.WARNING:tensorflow:Frommain.py:30:Thenametf.Sessionisdeprecated.Pleaseusetf.compat.v1.Sessioninstead.2020-04-1515:50:23.173272:Itensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1..............2020-04-1515:51:27.574461:Itensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.72020-04-1515:51:51.405994:Itensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0Lossy conversion from float64 to uint8.Range [0, 1].Convert image to uint8 prior to saving to suppress this warning.
If it ended without error, the result.jpg file would be created in the current working directory.
<result.jpg>
The image in the top row is a makeup image, and the images are stored in the imgs / makeup directory. The bottom left image is the original image without makeup. This file can be changed with the --no_makeup option. The default is imgs / no_makeup / xfsy_0068.png file. And the rest of the lower images are the result of applying the upper makeup image. You can see that it has become a fairly natural makeup.
defpreprocess(img):return(img/255.-0.5)*2
The pixels of the image have RGB values from 0 to 255 (uint 8). The preprocess function adjusts the range of pixel values from 0 to 255 to a range between -1 and 1. BeautyGAN models use values from this value range.
defdeprocess(img):return(img+1)/2
The ndarray that has passed the preprocess function is output to the makeup ndarray using the model. The range of this value is also between -1 and 1. Again, we need to change these values from 0 to 255. However, the deprocess function does not convert to uint8 from 0 to 255, but to a float value between 0 and 1. The reason for this conversion is that when the imageio module's imsave function receives an ndarray with a float value between 0 and 1, it is automatically converted to 0 to 255 uint8 and stored. However, this method is confusing and the postprocess function used by kairess is much easier to understand.
<new deprocess function that is easy to understand>
The part of reading the model already learned in the tensor flow is omitted because there are many examples on the Internet. Also, it is easy to understand the arrangement of the output image after makeup on the result ndarray by reading the code.
Make up your face
Now, let's select a random input image and a desired makeup model and output the makeup image.
The makeup image exists in the imgs / makeup directory.
I removed imageio package from the source code and used only OpenCV. OpenCV uses BGR color model, so before pass the image to the model, convert to RGB model. And before saving, convert RGB color to BGR model once more.
importtensorflowastfimportnumpyasnpimportosimportglobfromimageioimportimread,imsaveimportcv2importargparseparser=argparse.ArgumentParser()parser.add_argument('--no_makeup',type=str,default=os.path.join('imgs','no_makeup','xfsy_0068.png'),help='path to the no_makeup image')parser.add_argument('--makeup',type=str,default=os.path.join('imgs','makeup','XMY-014.png'),help='path to the makeup image')args=parser.parse_args()defpreprocess(img):return(img/255.-0.5)*2defdeprocess(img):return((img+1.)*127.5).astype(np.uint8)batch_size=1img_size=256no_makeup=cv2.resize(cv2.imread(args.no_makeup,cv2.IMREAD_COLOR),(img_size,img_size))no_makeup=cv2.cvtColor(no_makeup,cv2.COLOR_BGR2RGB)X_img=np.expand_dims(preprocess(no_makeup),0)makeup_image=args.makeuptf.reset_default_graph()sess=tf.Session()sess.run(tf.global_variables_initializer())saver=tf.train.import_meta_graph(os.path.join('models','model.meta'))saver.restore(sess,tf.train.latest_checkpoint('models'))graph=tf.get_default_graph()X=graph.get_tensor_by_name('X:0')Y=graph.get_tensor_by_name('Y:0')Xs=graph.get_tensor_by_name('generator/xs:0')makeup=cv2.resize(imread(makeup_image),(img_size,img_size))Y_img=np.expand_dims(preprocess(makeup),0)Xs_=sess.run(Xs,feed_dict={X:X_img,Y:Y_img})Xs_=deprocess(Xs_)img=cv2.cvtColor(Xs_[0],cv2.COLOR_RGB2BGR)cv2.imwrite('makeup.jpg',img)
<makeup.py>
Run the code, and you will get the makeup.jpg files.
This is a good example to learn by using a Jupyter Nodebook.
Wrapping up
Currently, only trained models provided by BeautyGAN can use 256X256X3 images. Therefore, it is impossible to create a high-resolution makeup image at this time. I hope that the source code related to the training has been released so that this part can be modified and improved by many people.
