Awesome Image Harmonization

A curated list of resources including papers, datasets, and relevant links pertaining to image harmonization.

Contributing

Contributions are welcome. If you wish to contribute, feel free to send a pull request. If you have suggestions for new sections to be included, please raise an issue and discuss before sending a pull request.

Table of Contents

• Survey
• Online Demo
• Leaderboard
• Color Transfer
• Papers
• Datasets
• Other Resources
• Related Topics

Survey

A brief review on image harmonization can be found in the following survey on image composition:

Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, Liqing Zhang: "Making Images Real Again: A Comprehensive Survey on Deep Image Composition." arXiv preprint arXiv:2106.14490 (2021). [arXiv]

Online Demo

Try this online demo for image harmonization and have fun!

Leaderboard

The leaderboard of SOTA image harmonization methods can be found here.

Color Transfer

We summarize different color transfer strategies which could be used for image harmonization task here.

Papers

Supervised deep learning methods

• Ke Wang, Michaël Gharbi, He Zhang, Zhihao Xia, Eli Shechtman: "Semi-supervised Parametric Real-world Image Harmonization." CVPR (2023) [arXiv] [code]
• Sheng Liu, Cong Phuoc Huynh, Cong Chen, Maxim Arap, Raffay Hamid: "LEMaRT: Label-Efficient Masked Region Transform for Image Harmonization." CVPR (2023) [pdf]
• Julian Jorge Andrade Guerreiro, Mitsuru Nakazawa, Björn Stenger: "PCT-Net: Full Resolution Image Harmonization Using Pixel-Wise Color Transformations." CVPR (2023) [pdf] [supp] [code]
• Xun Cai, Qingjie Shi, Yanbo Gao, Shuai Li, Wei Hua, Tian Xie: "A Structure-Preserving and Illumination-Consistent Cycle Framework for Image Harmonization." T-MM (2022) [pdf]
• Yazhou Xing, Yu Li, Xintao Wang, Ye Zhu, Qifeng Chen: "Composite Photograph Harmonization with Complete Background Cues." ACM MM (2022) [pdf]
• Junyan Cao, Wenyan Cong, Li Niu, Jianfu Zhang, Liqing Zhang: "Deep Image Harmonization by Bridging the Reality Gap." BMVC (2022) [arXiv] [dataset] (rendered images)
• Zonghui Guo, Zhaorui Gu, Bing Zheng, Junyu Dong, Haiyong Zheng: "Transformer for Image Harmonization and Beyond." T-PAMI (2022) [pdf]
• Ben Xue, Shenghui Ran, Quan Chen, Rongfei Jia, Binqiang Zhao, Binqiang Zhao: "DCCF: Deep Comprehensible Color Filter Learning Framework for High-Resolution Image Harmonization." ECCV (2022) [arXiv] [code]
• Jingtang Liang, Xiaodong Cun, and Chi-Man Pun: "Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization." ECCV (2022) [arXiv] [code]
• Xuqian Ren, Yifan Liu: "Semantic-guided Multi-Mask Image Harmonization." ECCV (2022) [arXiv] [code]
• Zhanghan Ke, Chunyi Sun, Lei Zhu, Ke Xu, Rynson W.H. Lau: "Harmonizer: Learning to Perform White-Box Image and Video Harmonization." ECCV (2022) [arXiv] [code]
• Jeya Maria Jose Valanarasu, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Jose Echevarria, Yinglan Ma, Zijun Wei, Kalyan Sunkavalli, Vishal M. Patel: "Interactive Portrait Harmonization." ICLR (2023) [arXiv]
• Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, Liqing Zhang: "High-Resolution Image Harmonization via Collaborative Dual Transformations." CVPR (2022) [arXiv] [dataset]
• Zhongyun Bao, Chengjiang Long, Gang Fu, Daquan Liu, Yuanzhen Li, Jiaming Wu, Chunxia Xiao: "Deep Image-based Illumination Harmonization." CVPR (2022) [arXiv] (rendered images)
• Yucheng Hang, Bin Xia, Wenming Yang, Qingmin Liao: "SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonization." CVPR (2022) [arXiv] [code]
• Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, Junyu Dong: "Image Harmonization with Transformer." ICCV (2021) [pdf] [supp] [code]
• Yifan Jiang, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi, Sarah Kong, Zhangyang Wang: "SSH: A Self-Supervised Framework for Image Harmonization." ICCV (2021) [pdf] [supp] [arXiv] [code]
• Jun Ling, Han Xue, Li Song, Rong Xie, Xiao Gu: "Region-Aware Adaptive Instance Normalization for Image Harmonization." CVPR (2021) [pdf] [supp] [arXiv] [code]
• Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, Bing Zheng: "Intrinsic Image Harmonization." CVPR (2021) [pdf] [supp] [code]
• Wenyan Cong, Li Niu, Jianfu Zhang, Jing Liang, Liqing Zhang: "BargainNet: Background-Guided Domain Translation for Image Harmonization." ICME (2021) [arXiv] [code]
• Konstantin Sofiiuk, Polina Popenova, Anton Konushin: "Foreground-aware Semantic Representations for Image Harmonization." WACV (2021) [pdf] [supp] [arXiv] [code]
• Guoqing Hao, Satoshi Iizuka, Kazuhiro Fukui: "Image Harmonization with Attention-based Deep Feature Modulation." BMVC (2020) [pdf] [supp] [code]
• Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, Liqing Zhang: "DoveNet: Deep Image Harmonization via Domain Verification." CVPR (2020) [pdf] [supp] [arXiv] [code].
• Xiaodong Cun, Chi-Man Pun: "Improving the Harmony of the Composite Image by Spatial-Separated Attention Module." T-IP (2020) [pdf] [arXiv] [code]
• Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, Ming-Hsuan Yang: "Deep Image Harmonization." CVPR (2017) [pdf] [supp] [arXiv] [code]

Unsupervised deep learning methods

• Anand Bhattad, David A. Forsyth: "Cut-and-Paste Neural Rendering." arXiv preprint arXiv: 2010.05907 (2020) [arXiv] [supp]
• Fangneng Zhan, Shijian Lu, Changgong Zhang, Feiying Ma, Xuansong Xie:"Adversarial Image Composition with Auxiliary Illumination." ACCV (2020) [pdf] [arXiv]
• Bor-Chun Chen, Andrew Kae: "Toward Realistic Image Compositing With Adversarial Learning." CVPR (2019) [pdf]
• Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, Alexei A. Efros: "Learning a Discriminative Model for the Perception of Realism in Composite Images." ICCV (2015) [pdf] [arXiv] [code]

Traditional methods

• Shuangbing Song, Fan Zhong, Xueying Qin, Changhe Tu: "Illumination Harmonization with Gray Mean Scale." Advances in Computer Graphics. CGI (2020) [pdf]
• Su Xue, Aseem Agarwala, Julie Dorsey, Holly E. Rushmeier: "Understanding and improving the realism of image composites." ACM Trans. Graph. 31(4): 84:1-84:10 (2012) [pdf]
• Kalyan Sunkavalli, Micah K. Johnson, Wojciech Matusik, Hanspeter Pfister: "Multi-scale image harmonization." ACM Trans. Graph. 29, 4 (2010) [pdf]
• Jean-François Lalonde, Alexei A. Efros: "Using Color Compatibility for Assessing Image Realism." ICCV (2007) [pdf] [code]

Datasets

• iHarmony4: It contains four subdatasets: HCOCO, HAdobe5k, HFlickr, Hday2night, with a total of 73,146 pairs of unharmonized images and harmonized images. [pdf] [link]
• GMSDataset: It contains 183 images with image resolution of 1940*1440. It consists of 16 different objects and for each object, one source image and 11 target images in different background scenes and illumination conditions are captured. [pdf] [link] (access code: ekn2)
• HVIDIT: A dataset built upon VIDIT (Virtual Image Dataset for Illumination Transfer) dataset for image harmonization. It contains 3007 images of 276 scenes for training and 329 images of 24 scenes for testing. [pdf] [link]
• RdHarmony: A rendered image harmonization dataset, which contains 25,000 ground-truth rendered images with 11 novel categories, and has the potential to generate 225,000 composite rendered images. [pdf] [link]
• RealHM: A Real-world HarMonization dataset, which contains 216 real composite images with manually harmonized outputs. [pdf] [link]
• ccHarmony: An image harmonization dataset constructed based on the images captured with color checker (cc). It contains 10 synthetic composite images for each of 426 foregrounds from 350 real images, leading to 4260 pairs of synthetic composite images and ground-truth real images. [pdf] [link]

Related Topics

Painterly/Style image harmonization

• Junyan Cao, Yan Hong, Li Niu: "Painterly Image Harmonization in Dual Domains." AAAI (2023) [code]
• Xiao Yan, Yang Lu, Juncheng Shuai, Sanyuan Zhang: "Style Image Harmonization via Global-Local Style Mutual Guided." ACCV (2022) [paper]
• Hwai-Jin Peng, Chia-Ming Wang, Yu-Chiang Frank Wang: "Element-Embedded Style Transfer Networks for Style Harmonization." BMVC (2019) [paper]
• Fujun Luan, Sylvain Paris, Eli Shechtman, Kavita Bala: "Deep Painterly Harmonization." Computer graphics forum (2018) [arXiv] [code]

Inharmonious region localization

• Penghao Wu, Li Niu, Liqing Zhang: "Inharmonious Region Localization with Auxiliary Style Feature." BMVC (2022)
• Jing Liang, Li Niu, Penghao Wu, Fengjun Guo, Teng Long: "Inharmonious Region Localization by Magnifying Domain Discrepancy." AAAI (2022) [arXiv] [code]
• Jing Liang, Li Niu, Liqing Zhang: "Inharmonious Region Localization." ICME (2021) [arXiv] [code]

Video harmonization

• Xinyuan Lu, Shengyuan Huang, Li Niu, Wenyan Cong, Liqing Zhang: "Deep Video Harmonization with Color Mapping Consistency." IJCAI (2022) [arXiv] [dataset&code]
• Haozhi Huang, Senzhe Xu, Junxiong Cai, Wei Liu, Shimin Hu: "Temporally Coherent Video Harmonization Using Adversarial Networks." IEEE Trans. Image Process. 29: 214-224 (2020) [pdf] [arXiv]

Other Resources

• Awesome-ImageHarmonization
• Awesome-Image-Composition