This repository is about automatic line art colorization with deep learning. In addition to training the neural network with line arts only, this repository aims to colorize with several types of hints. There are mainly three types of hints.
There are many kinds of line extraction methods, such as XDoG or SketchKeras. If we train the model on only one type of line art, trained model comes to overfit and cannot colorize another type of line art properly. Therefore, like Tag2Pix, various kinds of line arts are used as the input of neural network.
I use mainly three types of line art.
An example obtained by these line extraction methods is as follows.
Moreover, I add two types of data augmenation to line arts in order to avoid overfitting.
First of all, I needed to confirm that methods based on neural networks can colorize without hint precisely and diversely. The training of mapping from line arts to color images is difficult because of variations in color. Therefore, I hypothesized that neural networks trained without hints would come to colorize single color in any regions. In addition to content loss, I tried adversarial loss because this loss function enables neural networks to match data distribution adequately.
|pix2pix & pix2pixHD|
Considering the application systems of colorization, we need to colorize with designated color. Therefore, I tried some methods that take the hint, named atari, as input of neural network.
I also consider taking the hint, named reference, as input of neural network. At first, I had tried to implement style2paints V1. However, I had difficulities reproducing the results because training came to collapse. Then, I decided to seek for substitutes for style2paints V1.