Implicit Generative Models Evaluation

Qualitative Evaluation

Probabilistic Measures

Image Quality Assessment Metrics

Image quality assessment provides a measure of the quality of an image in reference to the original image or not. We here review some metrics that have been used in works on generative methods for remote sensing (Wang et al. 2019, Grohnfeldt et al. 2018)

PSNR (Peak Signal to Noise Ratio)

Compares the power of a clean image y to the power of corrupting noise from its corrupted version x as :

Pros :

Cons : High sensitivity towards biases in brightness

SAM (Spectral Angle Mapper, Boardman et al. 1993)

Estimates spectra similarity by comparing band similarities.

Given a pair of NxNxd images x and y, we have :

Variations :

• Kernel-SAM : use kernel trick on base SAM expression

Pros :

Cons :

SSIM (Structural Similarity Index, Wang et al. 2004)

Estimates structural disparities based on luminosity, constrast and structure for a pair of image windows x and y as :

see here for luminosity, contrast and structure expressions

Pros : Finds large-scale mode collapse reliably

Cons : Fails to diagnose smaller effects such as loss of variations in colors and textures + does not assess quality in terms of similarity to the dataset

Variations :

• ESSIM: adds edge information
• MS-SSIM: multi-scale comparison
• FSIM: compares phase congruency and gradient magnitude
• CW-SSIM: compares complex wavelet transform (deals with issues of image scaling, translation and rotation)

Sharpness Difference (SD)

Pretty self-explaining ?

where

Pros :

Cons :

Sanity Check at Mastered Tasks

References

• Pros and Cons of GAN Evaluation Measures, Borji 2018
• A note on the evaluation of generative models, Theis et al. 2015
• More to come