• Stars
    star
    278
  • Rank 145,020 (Top 3 %)
  • Language
    Python
  • License
    MIT License
  • Created almost 3 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[ACM MM 2021] Privacy-Preserving Portrait Matting

Privacy-Preserving Portrait Matting [ACM MM-21]

This is the official repository of the paper Privacy-Preserving Portrait Matting.

Jizhizi Liโˆ—, Sihan Maโˆ—, Jing Zhang, and Dacheng Tao

Introduction | PPT and P3M-10k | P3M-Net | Benchmark | Results | Train and Test | Inference code | Statement


๐Ÿ“ฎ News

[2023-03-28]: The extended paper Rethinking Portrait Matting with Privacy Preserving has been accepted by the International Journal of Computer Vision (IJCV).

[2022-03-31]: Publish the extended version paper "Rethinking Portrait Matting with Privacy Preserving". The code, dataset, and models are available at Github.

[2021-11-21]: Publish the dataset P3M-10k (the largest privacy-preserving portrait matting dataset, contains 10421 high-resolution real-world face-blurred portrait images and the manually labeled alpha mattes.), the train code and the test code. The dataset P3M-10k can be accessed from the following link, please make sure that you have read and agreed to the agreement. The train code and test code can be viewed from this code-base page.

[2021-12-06]: Publish the face mask of the training set and P3M-500-P validation set of P3M-10k dataset.

Dataset

Dataset Link
(Google Drive)

Dataset Link
(Baidu Wangpan ็™พๅบฆ็ฝ‘็›˜)

Dataset Release Agreement
P3M-10k Link Link (pw: fgmc) Agreement (MIT License)
P3M-10k facemask (optional) Link Link (pw: f772) Agreement (MIT License)

[2021-11-20]: Publish the inference code and the pretrained model (Google Drive | Baidu Wangpan (pw: 2308)) that can be used to test on your own privacy-preserving or normal portrait images. Some test results on P3M-10k can be viewed from this demo page.

Introduction

Recently, there has been an increasing concern about the privacy issue raised by using personally identifiable information in machine learning. However, previous portrait matting methods were all based on identifiable portrait images.

To fill the gap, we present P3M-10k in this paper, which is the first large-scale anonymized benchmark for Privacy-Preserving Portrait Matting. P3M-10k consists of 10,000 high-resolution face-blurred portrait images along with high-quality alpha mattes. We systematically evaluate both trimap-free and trimap-based matting methods on P3M-10k and find that existing matting methods show different generalization capabilities when following the Privacy-Preserving Training (PPT) setting, ๐‘–.๐‘’., training on face-blurred images and testing on arbitrary images.

To devise a better trimap-free portrait matting model, we propose P3M-Net, which leverages the power of a unified framework for both semantic perception and detail matting, and specifically emphasizes the interaction between them and the encoder to facilitate the matting process. Extensive experiments on P3M-10k demonstrate that P3M-Net outperforms the state-of-the-art methods in terms of both objective metrics and subjective visual quality. Besides, it shows good generalization capacity under the PPT setting, confirming the value of P3M-10k for facilitating future research and enabling potential real-world applications.

PPT Setting and P3M-10k Dataset

PPT Setting: Due to the privacy concern, we propose the Privacy-Preserving Training (PPT) setting in portrait matting, ๐‘–.๐‘’., training on privacy-preserved images (๐‘’.๐‘”., processed by face obfuscation) and testing on arbitraty images with or without privacy content. As an initial step towards privacy-preserving portrait matting problem, we only define the identifiable faces in frontal and some profile portrait images as the private content in this work.

P3M-10k Dataset: To further explore the effect of PPT setting, we establish the first large-scale privacy-preserving portrait matting benchmark named P3M-10k. It contains 10,000 annonymized high-resolution portrait images by face obfuscation along with high-quality ground truth alpha mattes. Specifically, we carefully collect, filter, and annotate about 10,000 high-resolution images from the Internet with free use license. There are 9,421 images in the training set and 500 images in the test set, denoted as P3M-500-P. In addition, we also collect and annotate another 500 public celebrity images from the Internet without face obfuscation, to evaluate the performance of matting models under the PPT setting on normal portrait images, denoted as P3M-500-NP. We show some examples as below, where (a) is from the training set, (b) is from P3M-500-P, and (c) is from P3M-500-NP.

P3M-10k and the facemask are now published!! You can get access to it from the following links, please make sure that you have read and agreed to the agreement. Note that the facemask is not used in our work. So it's optional to download it.

Dataset

Dataset Link
(Google Drive)

Dataset Link
(Baidu Wangpan ็™พๅบฆ็ฝ‘็›˜)

Dataset Release Agreement
P3M-10k Link Link (pw: fgmc) Agreement (MIT License)
P3M-10k facemask (optional) Link Link (pw: f772) Agreement (MIT License)

P3M-Net

Our proposed P3M-Net consists of four parts

  • A Multi-task Framework: To enable benefits from explicitly modeling both semantic segmentation and detail matting tasks and jointly optimizing for trimap-free matting, we follow [1] and [2], adopt a multi-task framework based on a modified version of ResNet-34, the model pretrained on ImageNet will be listed as follows;

  • TFI: Tripartite-Feature Integration: TFI module is used in each matting decoder block to model the interaction between encoder, segmentation decoder, and the matting decoder. TFI has three inputs, the feature map of the previous matting decoder block, the feature map from the same level semantic decoder block, and the feature map from the symmetrical encoder block. TFI passes them through a projection layer, concats the outputs and feeds into a convolutional block to generate the output feature;

  • sBFI: Shallow Bipartite-Feature Integration: sBFI module is used to model the interaction between the encoder and matting decoder. sBFI adopts the feature map from the first encoder block as a guidance to refine the output feature map from previous matting decoder block since shallow layers in the encoder contain many details and local structural information;

  • dBFI: Deep Bipartite-Feature Integration: dBFI module is used to model the interaction between the encoder and segmentation decoder. dBFI adopts the feature map from the last encoder block as a guidance for the semantic decoder since it contains abundant global semantics. Specifically, dBFI fuses the feature map from the last encoder with the ones from semantic decoder to improve the feature representation ability for the high-level semantic segmentation task.

Here we provide the model we pretrained on P3M-10k and the backbone we pretrained on ImageNet.

Model Pretrained Backbone on ImageNet Pretrained P3M-NET on P3M-10k
Google Drive Link Link

Baidu Wangpan
(็™พๅบฆ็ฝ‘็›˜)

Link
(pw: 2v1t)

Link
(pw: 2308)

Benchmark

A systematic evaluation of the existing trimap-based and trimap-free matting methods on P3M-10k is conducted to investigate the impact of the privacy-preserving training (PPT) setting on different matting models and gain some useful insights. Part of the results are shown as below. Please refer to the paper for full tables.

In the following tables, "B" denotes the blurred images, and "N" denotes the normal images. "B:N" denotes training on blurred images while testing on normal images, vice versa.

Table 1. Results of trimap-based deep learning methods on P3M-500-P.
Setting B:B B:N N:B N:N
Method SAD MSE SAD MSE SAD MSE SAD MSE
DIM 4.8906 0.0115 4.8940 0.0116 4.8050 0.0116 4.7941 0.0116
AlphaGAN 5.2669 0.0112 5.2367 0.0112 5.7060 0.0120 5.6696 0.0119
GCA 4.3593 0.0088 4.3469 0.0089 4.4068 0.0089 4.4002 0.0089
IndexNet 5.1959 0.0156 5.2188 0.0158 5.8267 0.0202 5.8509 0.0204
FBA 4.1330 0.0088 4.1267 0.0088 4.1666 0.0086 4.1544 0.0086
Table 2. Results of trimap-free methods on P3M-500-P.
Setting B:B B:N N:B N:N
Method SAD MSE SAD MSE SAD MSE SAD MSE
SHM 21.56 0.0100 24.33 0.0116 23.91 0.0115 17.13 0.0075
LF 42.95 0.0191 30.84 0.0129 41.01 0.0174 31.22 0.0123
HATT 25.99 0.0054 26.5 0.0055 35.02 0.0103 22.93 0.0040
GFM 13.20 0.0050 13.08 0.0050 13.54 0.0048 10.73 0.0033
BASIC 15.13 0.0058 15.52 0.0060 24.38 0.0109 14.52 0.0054
P3M-Net (Ours) 8.73 0.0026 9.22 0.0028 11.22 0.0040 9.06 0.0028

Results

We test our network on our proposed P3M-500-P and P3M-500-NP and compare with previous SOTA methods, we list the results as below. More results on P3M-10k test set can be found here.

Inference Code - How to Test on Your Images

Here we provide the procedure of testing on sample images by our pretrained model:

  1. Setup environment following this instruction page;

  2. Insert the path REPOSITORY_ROOT_PATH in the file core/config.py;

  3. Download the pretrained P3M-Net model from here (Google Drive|Baidu Wangpan (pw: 2308)) and unzip to the folder models/pretrained/;

  4. Save your sample images in folder samples/original/.;

  5. Setup parameters in the file scripts/test_samples.sh and run by:

    chmod +x scripts/test_samples.sh

    scripts/test_samples.sh;

  6. The results of alpha matte and transparent color image will be saved in folder samples/result_alpha/. and samples/result_color/..

We show some sample images, the predicted alpha mattes, and their transparent results as below. We use the pretrained model from section Network with Hybrid (1 & 1/2) test strategy.

Statement

If you are interested in our work, please consider citing the following:

@inproceedings{10.1145/3474085.3475512,
author = {Li, Jizhizi and Ma, Sihan and Zhang, Jing and Tao, Dacheng},
title = {Privacy-Preserving Portrait Matting},
year = {2021},
isbn = {9781450386517},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3474085.3475512},
doi = {10.1145/3474085.3475512},
booktitle = {Proceedings of the 29th ACM International Conference on Multimedia},
pages = {3501โ€“3509},
numpages = {9},
keywords = {trimap, benchmark, portrait matting, deep learning, semantic segmentation, privacy-preserving},
location = {Virtual Event, China},
series = {MM '21}
}

This project is under MIT licence.

For further questions, please contact Jizhizi Li at [email protected] or Sihan Ma at [email protected].

Relevant Projects

[1] Deep Automatic Natural Image Matting, IJCAI, 2021 | Paper | Github
โ€‚ โ€‚ โ€‚Jizhizi Li, Jing Zhang, and Dacheng Tao

[2] Bridging Composite and Real: Towards End-to-end Deep Image Matting, IJCV, 2022 | Paper | Github
โ€‚ โ€‚ โ€‚Jizhizi Liโˆ—, Jing Zhangโˆ—, Stephen J. Maybank, Dacheng Tao

[3] Referring Image Matting, CVPR, 2023 | Paper | Github
โ€‚ โ€‚ โ€‚Jizhizi Li, Jing Zhang, and Dacheng Tao

[4] Rethinking Portrait Matting with Privacy Preserving, IJCV, 2023 | Paper | Github
โ€‚ โ€‚ โ€‚Sihan Maโˆ—, Jizhizi Liโˆ—, Jing Zhang, He Zhang, Dacheng Tao

[5] Deep Image Matting: A Comprehensive Survey, ArXiv, 2023 | Paper | Github
โ€‚ โ€‚ โ€‚Jizhizi Li, Jing Zhang, and Dacheng Tao

More Repositories

1

GFM

[IJCV 2022] Bridging Composite and Real: Towards End-to-end Deep Image Matting
Python
908
star
2

AIM

[IJCAI'21] Deep Automatic Natural Image Matting
Python
377
star
3

RIM

[CVPR 2023] Referring Image Matting
200
star
4

matting-survey

Deep Image Matting: A Comprehensive Survey
159
star
5

animal-matting

33
star
6

pytorch-sss

python & pytorch implementation of paper Semantic Soft Segmentation
Jupyter Notebook
21
star
7

cnn-cell-counting

A machine learning model using CNN to count number of cells in a medical image
Python
11
star
8

homography-transformation-python

Performs homography transformation between two images, write in python
Python
5
star
9

all-kinds-crawling-tools

This repository provides all kinds of crawling tools, e.g. image-crawler, paper-crawler
Python
5
star
10

person-detector

A machine learning model built on flask framework, mostly uses openCV to detect person in photo, can be used in single or batch images
Python
4
star
11

Cell_Image_Processing-Cell_Sequence_Tracking

MATLAB
4
star
12

generate-close-view

Generate close view and paste on original image with any ratio, can be used in writing papers or making slides
Python
3
star
13

codebrew2015

Java
2
star
14

files

2
star
15

ml-algorithms

Some common used machine learning algorithms
Python
2
star
16

Approximate-Matching

Python
2
star
17

MyTime

android app for sharing/uploading/editing photos
Java
1
star
18

GOY-similarBrand

a simple nlp algorithm to find similar brand for Good On You app
Python
1
star
19

Online-Folder-Backup

1
star
20

BetterTrip

CSS
1
star
21

Pairwise-Relationship-Prediction

MATLAB
1
star
22

Fashion-Hackday-20171112

code used in fashion-hackday @20171112
Python
1
star
23

CellCounting_CNN

CNN model build to count the cell from microscopy image(Master's Thesis)
Jupyter Notebook
1
star