JetsonNano - Facial Makeup Transfer with Deep GAN(Generative Adversarial Network)

I used Jetson Nano, JetPack 4.3 Official image with root account.

What is BeautyGAN?

 BeautyGAN is Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network. Official website(http://liusi-group.com/projects/BeautyGAN) summarizes BeautyGAN as follows.

Facial makeup transfer aims to translate the makeup style from a given reference makeup face image to another non-makeup one while preserving face identity. Such an instance-level transfer problem is more challenging than conventional domain-level transfer tasks, especially when paired data is unavailable. Makeup style is also different from global styles (e.g., paintings) in that it consists of several local styles/cosmetics, including eye shadow, lip-stick, foundation, and so on. Extracting and transferring such local and delicate makeup information is infeasible for existing style transfer methods. We address the issue by incorporating both global domain-level loss and local instance-level loss in an dual input/output Generative Adversarial Network, called BeautyGAN. Specically, the domain-level transfer is ensured by discriminators that distinguish generated images from domains’ real samples. The instance-level loss is calculated by pixel-level histogram loss on separate local facial regions. We further introduce perceptual loss and cycle consistency loss to generate high quality faces and preserve identity. The overall objective function enables the network to learn translation on instance-level through unsupervised adversarial learning. We also build up a new makeup dataset that consists of 3834 high-resolution face images. Extensive experiments show 

 The following figure is a schematic diagram of the operation flow of BeautyGAN.

Figure : Framework of the proposed BeautyGAN. The upper pipeline shows the overall system.Gaccpets two images asinputs: non-makeup imageIsr c, reference makeup imageIr e f, and generates two outputs: transferred makeup imageIBsr c, anti-makeup imageIAr e f. The generated images are fed into the sameGto build up reconstruction results:Ir ecsr c,Ir ecr e f. There are fourloss terms for trainingG: cycle consistency loss, perceptual loss, adversarial loss (denoted asDAandDB) and makeup loss. Thelower pipeline shows the details of makeup loss. It consists of three local histogram loss terms acted on face, eye shadow andlips, respectively. We first utilize face parsing model to separate each cosmetic region ofIsr c,Ir e f,IBsr c. Then, for each region,we employ histogram matching betweenIsr candIr e fto obtain a histogram remapping facial region as ground truth. The localloss term calculates pixel-level differences between such ground truth and corresponding cosmetic region ofIBsr c.


If you want to know more about BeautyGAN's theory, please refer to the article on this page(http://liusi-group.com/pdf/BeautyGAN-camera-ready_2.pdf).

And I have referenced a lot of pages at https://github.com/kairess/BeautyGAN which forked the github site (https://github.com/Honlan/BeautyGAN) that introduced BeautyGAN.

Prerequisites

Tensorflow is required to operate BeautyGAN. If TensorFlow is not installed on your Jetson Nano, install TensorFlow first. And you need OpenCV to run GitHub's example code that introduced BeautyGAN. If you are using JetPack4.3, OpenCV 4.1 will be installed by default. If you are using JetPack 4.2 or earlier, you must install OpenCV separately.

Install OpenCV

If you use JetPack 4.3, OpenCV 4.1 is preinstalled so skip this process.
If you are using JetPack 4.2 or below, see my other post on OpenCV installation https://spyjetson.blogspot.com/2019/09/jetsonnano-opencv-411-build.html.

Install Tensorflow

Do not follow the installation process of the TensorFlow homepage. Nvidia provides the optimized Tensorflow package available in JetPack. To install Tensorflow provided by Nvidia, see my other post https://spyjetson.blogspot.com/2019/09/jetsonnano-installing-tensorflow-114.html.

If you want to run Taehee Lee kairess 's code, install these packages too.

• matplotlib
• dlib

Installing dlib is described at https://spyjetson.blogspot.com/2019/11/jetsonnano-face-recognition-using-dlib.html.

To install matplotlib,

pip3 install -U matplotlib

And you may need to install imageio packages to run main.py example.

pip3 install imageio

Install BeautyGAN

Select and download one of the two Githubs introduced above. I will download https://github.com/kairess/BeautyGAN. This site provides excellent Jupyter Notebook examples and additional Youtube videos.

<Youtube video provided by kairess>

cd /usr/local/src
git clone https://github.com/kairess/BeautyGAN.git

Download the pre-trained models

You can download models that have already been trained from the following two sites.

• https://pan.baidu.com/s/1wngvgT0qzcKJ5LfLMO7m8A，7lip
• https://drive.google.com/drive/folders/1pgVqnF2-rnOxcUQ3SO4JwHUFTdiSe5t9

Download these files and copy them to the "/usr/local/src/BeautyGAN/models" directory.

Testing main.py

BeautyGAN provides main.py for testing. With this program, you can take a brief look at the operation of BeautyGAN.

Be Careful: If you downloaded source codes from https://github.com/kairess/BeautyGAN, you need to slightly modify the model path in the main.py file. Honlan's original source code uses the model directory name, while kairess's fork version uses the models directory. Therefore, change the "model" directory of the main.py file to "models".

# -*- coding: utf-8 -*-

import tensorflow as tf
import numpy as np
import os
import glob
from imageio import imread, imsave
import cv2
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--no_makeup', type=str, default=os.path.join('imgs', 'no_makeup', 'xfsy_0068.png'), help='path to the no_makeup image')
args = parser.parse_args()

def preprocess(img):
    return (img / 255. - 0.5) * 2

def deprocess(img):
    return (img + 1) / 2

batch_size = 1
img_size = 256
no_makeup = cv2.resize(imread(args.no_makeup), (img_size, img_size))
X_img = np.expand_dims(preprocess(no_makeup), 0)
makeups = glob.glob(os.path.join('imgs', 'makeup', '*.*'))
result = np.ones((2 * img_size, (len(makeups) + 1) * img_size, 3))
result[img_size: 2 *  img_size, :img_size] = no_makeup / 255.

tf.reset_default_graph()
sess = tf.Session()
sess.run(tf.global_variables_initializer())

# I modified model ->models
saver = tf.train.import_meta_graph(os.path.join('models', 'model.meta'))
saver.restore(sess, tf.train.latest_checkpoint('models'))

graph = tf.get_default_graph()
X = graph.get_tensor_by_name('X:0')
Y = graph.get_tensor_by_name('Y:0')
Xs = graph.get_tensor_by_name('generator/xs:0')

for i in range(len(makeups)):
    makeup = cv2.resize(imread(makeups[i]), (img_size, img_size))
    Y_img = np.expand_dims(preprocess(makeup), 0)
    Xs_ = sess.run(Xs, feed_dict={X: X_img, Y: Y_img})
    Xs_ = deprocess(Xs_)
    result[:img_size, (i + 1) * img_size: (i + 2) * img_size] = makeup / 255.
    result[img_size: 2 * img_size, (i + 1) * img_size: (i + 2) * img_size] = Xs_[0]

imsave('result.jpg', result)

<main.py>

Before going through the code, let's run it once to see the results.

Tips: The TensorFlow example above requires a lot of memory. If an error occurs due to insufficient memory, changing Ubuntu Desktop to LXDE can secure an additional 1 GB of memory. I also got the correct results after using this method. Please refer to my other post https://spyjetson.blogspot.com/2019/09/jetson-nano-useful-tips-before-you.html for LXDE Desktop changes.

root@spypiggy-nano:/usr/local/src/BeautyGAN# python3 main.py
2020-04-15 15:50:11.401865: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
WARNING:tensorflow:From main.py:29: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.
WARNING:tensorflow:From main.py:30: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2020-04-15 15:50:23.173272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
.......
.......
2020-04-15 15:51:27.574461: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-04-15 15:51:51.405994: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
Lossy conversion from float64 to uint8. Range [0, 1]. Convert image to uint8 prior to saving to suppress this warning.

If it ended without error, the result.jpg file would be created in the current working directory.

<result.jpg>

The image in the top row is a makeup image, and the images are stored in the imgs / makeup directory. The bottom left image is the original image without makeup. This file can be changed with the --no_makeup option. The default is imgs / no_makeup / xfsy_0068.png file. And the rest of the lower images are the result of applying the upper makeup image. You can see that it has become a fairly natural makeup.

def preprocess(img):
    return (img / 255. - 0.5) * 2

The pixels of the image have RGB values ​​from 0 to 255 (uint 8). The preprocess function adjusts the range of pixel values ​​from 0 to 255 to a range between -1 and 1. BeautyGAN models use values ​​from this value range.

def deprocess(img):
    return (img + 1) / 2

The ndarray that has passed the preprocess function is output to the makeup ndarray using the model. The range of this value is also between -1 and 1. Again, we need to change these values ​​from 0 to 255. However, the deprocess function does not convert to uint8 from 0 to 255, but to a float value between 0 and 1. The reason for this conversion is that when the imageio module's imsave function receives an ndarray with a float value between 0 and 1, it is automatically converted to 0 to 255 uint8 and stored. However, this method is confusing and the postprocess function used by kairess is much easier to understand.

def deprocess(img):
    return ((img + 1.) * 127.5).astype(np.uint8)

<new deprocess function that is easy to understand>

The part of reading the model already learned in the tensor flow is omitted because there are many examples on the Internet. Also, it is easy to understand the arrangement of the output image after makeup on the result ndarray by reading the code.

Make up your face

Now, let's select a random input image and a desired makeup model and output the makeup image.

The makeup image exists in the imgs / makeup directory.

root@spypiggy-nano:/usr/local/src/BeautyGAN/imgs# ls -al makeup/
total 1852
drwxr-xr-x 2 root root   4096  4월 15 01:59 .
drwxr-xr-x 4 root root   4096  4월 15 16:41 ..
-rw-r--r-- 1 root root 228499  4월 15 01:59 vFG112.png
-rw-r--r-- 1 root root 195287  4월 15 01:59 vFG137.png
-rw-r--r-- 1 root root 187376  4월 15 01:59 vFG56.png
-rw-r--r-- 1 root root 254934  4월 15 01:59 vFG756.png
-rw-r--r-- 1 root root 197081  4월 15 01:59 vRX916.png
-rw-r--r-- 1 root root 208965  4월 15 01:59 XMY-014.png
-rw-r--r-- 1 root root 230706  4월 15 01:59 XMY-074.png
-rw-r--r-- 1 root root 189209  4월 15 01:59 XMY-136.png
-rw-r--r-- 1 root root 172506  4월 15 01:59 XMY-266.png

I removed imageio package from the source code and used only OpenCV. OpenCV uses BGR color model, so before pass the image to the model, convert to RGB model. And before saving, convert RGB color to BGR model once more.

import tensorflow as tf
import numpy as np
import os
import glob
from imageio import imread, imsave
import cv2
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--no_makeup', type=str, default=os.path.join('imgs', 'no_makeup', 'xfsy_0068.png'), help='path to the no_makeup image')
parser.add_argument('--makeup', type=str, default=os.path.join('imgs', 'makeup', 'XMY-014.png'), help='path to the makeup image')
args = parser.parse_args()

def preprocess(img):
    return (img / 255. - 0.5) * 2

def deprocess(img):
    return ((img + 1.) * 127.5).astype(np.uint8)

batch_size = 1
img_size = 256
no_makeup = cv2.resize(cv2.imread(args.no_makeup, cv2.IMREAD_COLOR), (img_size, img_size))
no_makeup = cv2.cvtColor(no_makeup, cv2.COLOR_BGR2RGB)

X_img = np.expand_dims(preprocess(no_makeup), 0)
makeup_image = args.makeup

tf.reset_default_graph()
sess = tf.Session()
sess.run(tf.global_variables_initializer())

saver = tf.train.import_meta_graph(os.path.join('models', 'model.meta'))
saver.restore(sess, tf.train.latest_checkpoint('models'))

graph = tf.get_default_graph()
X = graph.get_tensor_by_name('X:0')
Y = graph.get_tensor_by_name('Y:0')
Xs = graph.get_tensor_by_name('generator/xs:0')

makeup = cv2.resize(imread(makeup_image), (img_size, img_size))
Y_img = np.expand_dims(preprocess(makeup), 0)
Xs_ = sess.run(Xs, feed_dict={X: X_img, Y: Y_img})
Xs_ = deprocess(Xs_)
img = cv2.cvtColor(Xs_[0], cv2.COLOR_RGB2BGR)
cv2.imwrite('makeup.jpg', img)

<makeup.py>

Run the code, and you will get the makeup.jpg files.

python3 makeup.py --no_makeup=./imgs/no_makeup/lee.jpg

Cropping the face and apply the BeautyGAN

https://github.com/kairess/BeautyGAN/blob/master/test.ipynb file introduces an example of applying BeautyGAN with horizontal alignment after trimming the face area from the image.

<face cropping and horizontal alignment>

This is a good example to learn by using a Jupyter Nodebook.

Wrapping up

Currently, only trained models provided by BeautyGAN can use 256X256X3 images. Therefore, it is impossible to create a high-resolution makeup image at this time. I hope that the source code related to the training has been released so that this part can be modified and improved by many people.

You can download the source code at https://github.com/raspberry-pi-maker/NVIDIA-Jetson