SIHR: a MATLAB/GNU Octave toolbox for single image highlight removal

Citation

@article{Ramos2020,
  doi = {10.21105/joss.01822},
  url = {https://doi.org/10.21105/joss.01822},
  year = {2020},
  month = jan,
  publisher = {The Open Journal},
  volume = {5},
  number = {45},
  pages = {1822},
  author = {V{\'{\i}}tor Ramos},
  title = {{SIHR}: a {MATLAB}/{GNU} {Octave} toolbox for single image highlight removal},
  journal = {Journal of Open Source Software}
}

Summary

An ongoing effort of developing new and implementing established single image highlight removal (SIHR) methods on MATLAB/GNU Octave.

Highlight, specularity, or specular reflection removal (see ¹ for a proper Web of Science expression, see [1] for a reference work entry, see [2], [3] for survey on this problem) concerns the following decomposition.

I welcome and encourage contributions to this project upon review. Please check CONTRIBUTING.md for more details.

Disclaimer 1: this repository is intended for research purposes only.
Disclaimer 2: some most of these methods are based on chromaticity analysis, they fail miserably for grayscale images.

¹ ((remov* NEAR/1 (highlight* OR specular*)) OR (separat* NEAR/1 (reflect* OR specular*)))

Raison d'être

I started out this repository by implementing, translating and collecting code snippets from the rare available^2,3,4,5 codes. Oftentimes papers are cryptical, codes are in C/C++ (requires compilation and major source code modification for general testing), or are just unavailable. See, e.g. this CSDN post⁶ that has no valid links at all.

In this context, this repository aims to be a continous algorithmic aid for ongoing research and development of SIHR methods.

² Tan and Ikeuchi. [Online]. Available: http://tanrobby.github.io/code/highlight.zip
³ Shen et al. [Online]. Available: http://ivlab.org/publications/PR2008_code.zip
⁴ Yang et al. [Online]. Available: http://www6.cityu.edu.hk/stfprofile/qiyang.htm
⁵ Shen and Zheng. [Online]. Available: http://ivlab.org/publications/AO2013_code.zip
⁶ https://blog.csdn.net/nvidiacuda/article/details/8078167

Usage (API)

Calling this toolbox's functions is very straightforward:

I_d = AuthorYEAR(I); % I is a double-valued input image of dimension
                     % m×n×3 containing linear RGB values and
                     % I_d is the calculated diffuse reflection
                     % using AuthorYEAR method.
                     % The specular component is simply
                     % I_s = I - I_d;

Methods

The following methods are available.

The following improvement is available.

Environment

The environment this repository is being developed is:

• MATLAB
  • Image Processing Toolbox
• GNU Octave
  • Image package
    • pkg install -forge image

Tested environments

Octave 4.2                                Ubuntu 18.04
Octave 5.1 (latest)    Windows 10 1903
MATLAB 9.1 (R2016b)    Windows 10 1903
MATLAB 9.6 (R2019a)    Windows 10 1903    Ubuntu 16.04 (MATLAB Online)

Installation

1. git clone https://github.com/vitorsr/SIHR.git or download a copy of the repository.
2. Start Octave or MATLAB.
   1. cd('path/to/SIHR'), i.e. change current folder to SIHR root (where SIHR.m is located).
   2. run SIHR.m for session path setup.
   3. help SIHR or doc SIHR provides a summary of the methods available.

Additional Debian/Ubuntu installation

To install the image package from Octave Forge, build-essential and liboctave-dev need to be present. Install them via apt, then proceed with package installation.

sudo apt-get install -qq -y build-essential liboctave-dev
octave --eval "pkg install -forge image"

Performance

This section aims to clarify how well (or not) the methods reproduced in this project were at reproducing results in literature.

Note: Akashi and Okatani's [10] method has highly fluctuating results because of random initialization.

Dataset

In technical literature, there exist two ground truth datasets commonly used right now. One by Shen and Zheng [9] which is distributed alongside their code, and one by Grosse et al. [12] in a dedicated page⁷.

Other test images are included alongside the code for Shen et al. [6] and Yang et al. [8].

Follow the instructions in images in order to download a local copy of these images from the respective authors' pages.

⁷ Grosse et al. [Online]. Available: http://www.cs.toronto.edu/~rgrosse/intrinsic/

Quality

Quantitative results reported are usually regarding the quality of the recovered diffuse component with respect to the ground truth available in the Shen and Zheng [9] test image set.

Automated testing

Reproduced results below are available in the utils/automated_testing.m script.

Note: ssim is not available in Octave Forge image.

Highest (self and peer-reported | reproduced) PSNR results (in dB)

Highest (self and peer-reported | reproduced) SSIM results

Expected running time (in seconds)

Note: results for MATLAB R2019b, Intel i5-8250U CPU, and 24 GB DDR4 2400 MHz RAM.

References

1. R. T. Tan, “Specularity, Specular Reflectance,” in Computer Vision, Springer US, 2014, pp. 750–752 [Online]. Available: http://dx.doi.org/10.1007/978-0-387-31439-6_538
2. A. Artusi, F. Banterle, and D. Chetverikov, “A Survey of Specularity Removal Methods,” Computer Graphics Forum, vol. 30, no. 8, pp. 2208–2230, Aug. 2011 [Online]. Available: http://dx.doi.org/10.1111/J.1467-8659.2011.01971.X
3. H. A. Khan, J.-B. Thomas, and J. Y. Hardeberg, “Analytical Survey of Highlight Detection in Color and Spectral Images,” in Lecture Notes in Computer Science, Springer International Publishing, 2017, pp. 197–208 [Online]. Available: http://dx.doi.org/10.1007/978-3-319-56010-6_17
4. R. T. Tan and K. Ikeuchi, “Separating reflection components of textured surfaces using a single image,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 2, pp. 178–193, Feb. 2005 [Online]. Available: http://dx.doi.org/10.1109/TPAMI.2005.36
5. K. Yoon, Y. Choi, and I. S. Kweon, “Fast Separation of Reflection Components using a Specularity-Invariant Image Representation,” in 2006 International Conference on Image Processing, 2006 [Online]. Available: http://dx.doi.org/10.1109/ICIP.2006.312650
6. H.-L. Shen, H.-G. Zhang, S.-J. Shao, and J. H. Xin, “Chromaticity-based separation of reflection components in a single image,” Pattern Recognition, vol. 41, no. 8, pp. 2461–2469, Aug. 2008 [Online]. Available: http://dx.doi.org/10.1016/J.PATCOG.2008.01.026
7. H.-L. Shen and Q.-Y. Cai, “Simple and efficient method for specularity removal in an image,” Applied Optics, vol. 48, no. 14, p. 2711, May 2009 [Online]. Available: http://dx.doi.org/10.1364/AO.48.002711
8. Q. Yang, S. Wang, and N. Ahuja, “Real-Time Specular Highlight Removal Using Bilateral Filtering,” in Computer Vision – ECCV 2010, Springer Berlin Heidelberg, 2010, pp. 87–100 [Online]. Available: http://dx.doi.org/10.1007/978-3-642-15561-1_7
9. H.-L. Shen and Z.-H. Zheng, “Real-time highlight removal using intensity ratio,” Applied Optics, vol. 52, no. 19, p. 4483, Jun. 2013 [Online]. Available: http://dx.doi.org/10.1364/AO.52.004483
10. Y. Akashi and T. Okatani, “Separation of reflection components by sparse non-negative matrix factorization,” Computer Vision and Image Understanding, vol. 146, pp. 77–85, May 2016 [Online]. Available: http://dx.doi.org/10.1016/j.cviu.2015.09.001
11. T. Yamamoto and A. Nakazawa, “General Improvement Method of Specular Component Separation Using High-Emphasis Filter and Similarity Function,” ITE Transactions on Media Technology and Applications, vol. 7, no. 2, pp. 92–102, 2019 [Online]. Available: http://dx.doi.org/10.3169/mta.7.92
12. R. Grosse, M. K. Johnson, E. H. Adelson, and W. T. Freeman, “Ground truth dataset and baseline evaluations for intrinsic image algorithms,” in 2009 IEEE 12th International Conference on Computer Vision, 2009 [Online]. Available: http://dx.doi.org/10.1109/ICCV.2009.5459428