paper | project website | video results

In this paper, we introduce DiffusionMat, a novel image matting framework that employs a diffusion model for the transition from coarse to refined alpha mattes. Diverging from conventional methods that utilize trimaps merely as loose guidance for alpha matte prediction, our approach treats image matting as a sequential refinement learning process. This process begins with the addition of noise to trimaps and iteratively denoises them using a pre-trained diffusion model, which incrementally guides the prediction towards a clean alpha matte. The key innovation of our framework is a correction module that adjusts the output at each denoising step, ensuring that the final result is consistent with the input image's structures. We also introduce the Alpha Reliability Propagation, a novel technique designed to maximize the utility of available guidance by selectively enhancing the trimap regions with confident alpha information, thus simplifying the correction task. To train the correction module, we devise specialized loss functions that target the accuracy of the alpha matte's edges and the consistency of its opaque and transparent regions. We evaluate our model across several image matting benchmarks, and the results indicate that DiffusionMat consistently outperforms existing methods.

git clone https://github.com/cnnlstm/DiffusionMat.git
cd DiffusionMat

The environment can be set up from the provided diffusionmat.yaml:

conda env create -f diffusionmat.yaml

Please download our pre-trained models and put in ./pretrained_models.

We provide 4 samples from Composition-1k dataset for the quick inference:

python inference.py --exp samples/alphas_pred  --config matte.yml --delta_config deltablock.yml --sample -i images --t 250 --sample_step 5 --ni

The whole testset of Composition-1k dataset can be downloaded at: Composition-1k-Testset

P3M dataset can be downloaded at: P3M Dataset

Rememer to modifying the testset path at here

Evaluate Composition-1k's results by the official evaluation MATLAB code ./DIM_evaluation_code/evaluate.m (provided by Deep Image Matting)

Evaluate P3M results by the official evaluation Python code

For obtain the trainset of Composition-1k dataset, please refer to: Matteformer

Please modify the trainingset path at here

python train.py --exp training_dir --config matte.yml --delta_config deltablock.yml --sample -i images --t 250 --sample_step 5 --ni

If you find this work useful for your research, please cite:

@article{xu2023diffusionmat,
title={DiffusionMat: Alpha Matting as Sequential Refinement Learning},
author={Xu, Yangyang and He, Shengfeng and Shao, Wenqi and Wong, Kwan-Yee K and Qiao, Yu and Luo, Ping},
journal={arXiv preprint arXiv:2311.13535},
year={2023}
}

Our Codes are mainly originated from SDEdit.