Portrait Shadow Manipulation

Portrait Shadow Manipulation
Alternatives To Portrait Shadow Manipulation
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Insightface13,716142 days ago24January 29, 20221,227mitPython
State-of-the-art 2D and 3D Face Analysis Project
3 years ago83otherPython
Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.
4 months ago54mitPython
Real-time face detection and emotion/gender classification using fer2013/imdb datasets with a keras CNN model and openCV.
Ffhq Dataset2,842
4 months ago5otherPython
Flickr-Faces-HQ Dataset (FFHQ)
3 months ago69otherPython
Github of the FaceForensics dataset
Few Shot Vid2vid1,638
a year ago22otherPython
Pytorch implementation for few-shot photorealistic video-to-video translation.
Celebamask Hq1,456
a year ago49Python
A large-scale face dataset for face parsing, recognition, generation and editing.
17 days ago25apache-2.0Python
Code for the paper: Detecting Photoshopped Faces by Scripting Photoshop
Face Mask Detection1,355
4 months ago20mitJupyter Notebook
Face Mask Detection system based on computer vision and deep learning using OpenCV and Tensorflow/Keras
a year ago129mitJupyter Notebook
Pytorch0.4.1 codes for InsightFace
Alternatives To Portrait Shadow Manipulation
Select To Compare

Alternative Project Comparisons

Portrait Shadow Manipulation (SIGGRAPH 2020)

UC Berkeley, Google & Google Research

Project Website

Tensorflow 1.15.2 has been tested for the following implementation.

This repo contains the following contents:

│   README.md
│   utils.py: foreign shadow synthesis steps
│	  including color jitter, subsurface scattering approximation,
│	  spatially-varying blur and per-pixel variations, and common util functions.
│   datasets.py: build the computational graph for training data generation.
│   mkdataset.py: offline data generatin script, saving generated images to disk.
│   mkfilelist.py: generate training / evaluation / test txt file for training.
│	test.txt: an example txt for training.
└───input: portrait image that is free of foreign shadow (ok to contain facial shadow)
│   │   000.png
│   │   001.png
│   │	  ...
└───input_mask_ind: segmentation and bounding box for each individual face
│   │   000_0.png
│   │   000.txt
│   │   001_0.png
│   │   001_1.png
│   │   001.txt
│   │   ...
│   │   ...

Prepare your (foreign shadows) training dataset

If you want to get a quick idea how the images are generated, you can jump to step 4 and refer to the jupyter notebook.

Step 0 (pre-req):

For each image, you need to have a portarit segmentation and face bounding box for each face in the image. We used the model from this paper, but there are other publicly avaialble portrait segmentation and face detectors that you can use.

We provide some exmples in this repo, see step 1.

Step 1:

Here's the structure of our datasets:

input/: folder that contains input images (no foreign shadows but may contain facial shadows)

input_mask_ind/: folder that contains individual face segmentation (.png) and bounding box (i.e. bbox) stored in .txt (each line corresponds to one face bbox). For example, for an input image 0.png in input/, 0_0.png denotes the first face in 0.png and 0_1.png denotes the second face in 0.png. 0.txt will contain two lines corresponding to the two bbox of the faces in 0.png.

silhouette/: We randomly choose to generate shadow using either perlin noise or silhouette, and this folder is to store all the random silhouette images.

Step 2:

Generate train / eval txt file, run mkfilelist.py. For example:

python3 mkfilelist.py \
	--image_base_paths ${your_parent_path}/input \
	--silhouette_paths ${your_parent_path}/silhouette \
	--file_types png \
	--output_dir ${your_parent_path}/test.txt

example.txt is an example .txt file that this script generates. During training or offline data generation, it reads in this .txt file.

Step 3:

Decide how you want to use the training data. There are two ways (both are provided) to use the training dataset:

  1. Offline: you can use the script (mkdataset.py) to generate and save out generated image pairs to disk.
  2. On-the-fly: you can use the datasets.py to build the data generation graph as part of the training computation graph, and generate data on the fly while training.

All the key synthesis functions described in paper Section 3 are implemented in utils.py* with detailed function documentation. To generate image pairs offline, run mkdataset.py. For example:

python3 mkdataset.py \
	--mode train \
	--trainwildpaths ${your_parent_path}/train.txt \
	--silhouette_paths ${your_parent_path}/eval.txt \
	--out_dir ${your_parent_path}/shadow_data \

Example ouput can be found under ./example_out/. You should expect generated image pairs to be similar to this.

*All synthesis functions in utils.py are implemented in tensorflow and it was meant for on-the-fly training data generation. If you use the script only for offline data generation and you don't have hardware for parallelization, some of the operation may not be computational efficient.

Step 4 (optional):

For a quick demo of each synthesis step, we also provide a jupyter notebook (foreign_syn.ipynb) that goes through the entire synthesis process. A few control parameters can be visualized interactively. For example, the strength of subsurface scattering approximation and spatially varying blur size:

ss_map sv_map


To download all test and evaluation dataset, please see project website.

Some of the test images come from:

Helen dataset: Face dataset with landmarks

HDR+ Burst Photography Dataset: Test images with faces

At this point, we cannot make the trained model public due to legal reasons.

Relevant links

These resources help me directly and indirectly understand this problem better:

Marc Levoy Lecture: Photographic lighting

cambridgeincolour: Introduction to portrai lighting with interactive demo

Lighting setup taxonomy: Pro portrait lighting setups with illustration

Canon tutorial: Natural Light Portraiture


We are grateful for people who agree to appear in all the photographs. Please use these photos only for academic purpose, and use with respect. Thank you.

Popular Dataset Projects
Popular Face Projects
Popular Data Processing Categories

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Jupyter Notebook