[BUG] Perturbation.perturbate_regions for RGB images

[tszoldra]

Perturbation.perturbate_regions perturbates only color channel 0 (red), while it should perturbate all color channels 0, 1, 2. Notice that ranks of the regions to perturb are insensitive to color channel through self.reduce_function which by default applies averaging of analysis score over all channels.

This bug is not visible in examples/mnist_perturbation.ipynb as it uses pictures with one color channel.
On RGB pictures, if I set self.perturbation_function="zeros", the perturbed region in the picture appears red or blueish and not totally black as would be desired. Also, peturbing with gaussian noise either produces red or blueish points and not all colors.

Steps to reproduce the bug

I am performing perturbation on 128 3-color channel pictures of size (224, 224). Perturbation region size is (28, 28), hence there are 8x8 regions to perturb. Since the code to replicate the pipeline is very long, below I show what I found while debugging Perturbation.perturbate_regions:

  def perturbate_regions(self, x, perturbation_mask_regions):
      # Perturbate every region in tensor.
      # A single region (at region_x, region_y in sample)
      # should be in mask[sample, channel, region_x, :, region_y, :]
      
     # x.shape = (128, 3, 224, 224) - batch_size 128, 3 colors, images 224x224

      x_perturbated = self.reshape_to_regions(x)   # x_perturbated.shape = (128, 3, 8, 28, 8, 28) - as expected
       
      # perturbation_mask_regions.shape = (128, 1, 8, 8) - only one channel for ordering of perturbations

      for sample_idx, channel_idx, region_row, region_col in np.ndindex(
          perturbation_mask_regions.shape
      ):
          region = x_perturbated[
              sample_idx, channel_idx, region_row, :, region_col, :
          ]
          # region.shape = (28, 28)
          # Notice that channel_idx never reaches 1, 2 as perturbation_mask_regions.shape = (128, 1, 8, 8) in the loop range - BUG

          region_mask = perturbation_mask_regions[
              sample_idx, channel_idx, region_row, region_col
          ]
          if region_mask:
              x_perturbated[
                  sample_idx, channel_idx, region_row, :, region_col, :
              ] = self.perturbation_function(region)

              if self.value_range is not None:
                  np.clip(
                      x_perturbated,
                      self.value_range[0],
                      self.value_range[1],
                      x_perturbated,
                  )
      x_perturbated = self.reshape_region_pixels(x_perturbated, x.shape)
      
      # x_perturbated.shape=(128, 3, 224, 224) - as expected but only the first channel has been modified
      
return x_perturbated

Expected behavior

All channels 0, 1, 2 are perturbed instead of only channel 0.

Screenshots

The image below is on grayscale but encoded in the RGB format, i.e. all colors have equal values on the RGB scale.
Before fix:

After fix:

Fix

Modify the region to be perturbed to contain all channels.

  def perturbate_regions(self, x, perturbation_mask_regions):
      # Perturbate every region in tensor.
      # A single region (at region_x, region_y in sample)
      # should be in mask[sample, channel, region_x, :, region_y, :]

      x_perturbated = self.reshape_to_regions(x)
      
      for sample_idx, channel_idx, region_row, region_col in np.ndindex(
          perturbation_mask_regions.shape
      ):
          region = x_perturbated[
              sample_idx, :, region_row, :, region_col, :
          ]
 
          region_mask = perturbation_mask_regions[
              sample_idx, channel_idx, region_row, region_col
          ]
          if region_mask:
              x_perturbated[
                  sample_idx, :, region_row, :, region_col, :
              ] = self.perturbation_function(region)

              if self.value_range is not None:
                  np.clip(
                      x_perturbated,
                      self.value_range[0],
                      self.value_range[1],
                      x_perturbated,
                  )
      x_perturbated = self.reshape_region_pixels(x_perturbated, x.shape)

      return x_perturbated

[adrhill]

Hi @tszoldra, thanks for the thorough issue as well as the fix!

While this does solve the problem for perturbation_function == "zeros", I'm wondering whether other perturbation functions like "mean" should average over all color channels or be applied individually to each channel.
@sebastian-lapuschkin do you have an opinion on this?