DCGAN_CAN Metrics.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"g_W-wJsiW75P"},"source":["# Quantitative Evaluation:"],"id":"g_W-wJsiW75P"},{"cell_type":"code","source":["from google.colab import drive\n","drive.mount('/content/drive')"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"tfMws_QFvvrW","executionInfo":{"status":"ok","timestamp":1686111273780,"user_tz":420,"elapsed":25136,"user":{"displayName":"TERRY MING","userId":"03637908485946588412"}},"outputId":"aaa4b7b8-83bd-44ea-89a4-b76628ea1bb0"},"id":"tfMws_QFvvrW","execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Mounted at /content/drive\n"]}]},{"cell_type":"code","execution_count":null,"metadata":{"id":"93Yqdgxvybd_"},"outputs":[],"source":["import numpy as np\n","import torch\n","import os\n","from PIL import Image\n","\n","path = '/content/drive/MyDrive/Pic 16B/CAN'"],"id":"93Yqdgxvybd_"},{"cell_type":"markdown","metadata":{"id":"CzyM6P1hXGQA"},"source":["### a) Fréchet Inception Distance (FID)"],"id":"CzyM6P1hXGQA"},{"cell_type":"markdown","metadata":{"id":"JLOGeDxTKwd2"},"source":["FID is a metric specifically used to quantitatively assess the quality of an image produced by a generative model, which improves upon the *inception score* by comparing the generated images with real images as opposed to only evaluating how well the generated images can be classified by a model (Inception v3) as a known object.\n"],"id":"JLOGeDxTKwd2"},{"cell_type":"markdown","metadata":{"id":"xPUYt6wFMWQK"},"source":["\n","$$d^2((m, C), (m_w, C_w)) = ||m-m_w||_2^{2} + Tr(C+C_w - 2(CC_w)^{1/2}$$\n","\n","Source: *GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (Heusel et al. 2017)*"],"id":"xPUYt6wFMWQK"},{"cell_type":"markdown","metadata":{"id":"lkHlCgDRMYGt"},"source":["Essentially, FID captures the difference between the two Gaussian distributions underlying the synthetic and real images (m = feature-wise mean, C = covariance matrix, _w = real-world data). As it is a \"spin-off\" of inception score, it too is related to the Inception network—it assumes the two distributions are the activations of the pool_3 layer of InceptionNet for generated and real samples.\n","\n","\n","*Remark: It is recommended to use a minimum sample size of 10,000 to calculate the FID; otherwise the true FID of the generator is underestimated.* In fact, I think > 50,000 is preferred.\n","\n","*Remark 2: FID scores depend largely on the number of samples (fewer samples = larger score), so it is crucial to use the same number of samples for each.*"],"id":"lkHlCgDRMYGt"},{"cell_type":"markdown","metadata":{"id":"FPSExz1bRj2j"},"source":["Implementation: Complicated. Requires loading pre-trained InceptionV3, resizing images, extracting its activations on both real and generated images, calculating mean and covariance of each, then feeding it into the formula above:"],"id":"FPSExz1bRj2j"},{"cell_type":"code","execution_count":null,"metadata":{"id":"KlDXY4d3R2ae"},"outputs":[],"source":["# from pytorch_fid.inception import InceptionV3\n","# model = InceptionV3().to(device)\n","# model.eval(),\n","\n","# with torch.no_grad():\n","#   pred = model(images)\n","\n","# act = (get activations somehow)\n","# mu = np.mean(act, axis=0)\n","# sigma = np.cov(act, rowvar=False)\n","\n","# etc..."],"id":"KlDXY4d3R2ae"},{"cell_type":"markdown","metadata":{"id":"f1a7qRNfSDam"},"source":["Instead, we will import a handy package to do this for us:"],"id":"f1a7qRNfSDam"},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4845,"status":"ok","timestamp":1686156997759,"user":{"displayName":"TERRY MING","userId":"03637908485946588412"},"user_tz":420},"id":"Toz_lzzeSIhi","outputId":"1400fa57-c7f6-4fdf-e54f-46e593eefe8e"},"outputs":[{"name":"stdout","output_type":"stream","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Collecting clean-fid\n","  Downloading clean_fid-0.1.35-py3-none-any.whl (26 kB)\n","Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from clean-fid) (2.0.1+cu118)\n","Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from clean-fid) (0.15.2+cu118)\n","Requirement already satisfied: numpy>=1.14.3 in /usr/local/lib/python3.10/dist-packages (from clean-fid) (1.22.4)\n","Requirement already satisfied: scipy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from clean-fid) (1.10.1)\n","Requirement already satisfied: tqdm>=4.28.1 in /usr/local/lib/python3.10/dist-packages (from clean-fid) (4.65.0)\n","Requirement already satisfied: pillow>=8.1 in /usr/local/lib/python3.10/dist-packages (from clean-fid) (8.4.0)\n","Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from clean-fid) (2.27.1)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->clean-fid) (1.26.15)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->clean-fid) (2022.12.7)\n","Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->clean-fid) (2.0.12)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->clean-fid) (3.4)\n","Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->clean-fid) (3.12.0)\n","Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch->clean-fid) (4.5.0)\n","Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->clean-fid) (1.11.1)\n","Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->clean-fid) (3.1)\n","Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->clean-fid) (3.1.2)\n","Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch->clean-fid) (2.0.0)\n","Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch->clean-fid) (3.25.2)\n","Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch->clean-fid) (16.0.5)\n","Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->clean-fid) (2.1.2)\n","Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->clean-fid) (1.3.0)\n","Installing collected packages: clean-fid\n","Successfully installed clean-fid-0.1.35\n"]}],"source":["!pip install clean-fid"],"id":"Toz_lzzeSIhi"},{"cell_type":"markdown","metadata":{"id":"Ok_RXLwZbW75"},"source":["#### remove some duplicates (already done, not necessary to run again)\n","(formmated as fake_img_x (1).png)"],"id":"Ok_RXLwZbW75"},{"cell_type":"code","execution_count":null,"metadata":{"id":"0aH21upCbZe9"},"outputs":[],"source":["for file in os.listdir(f'{path}/fake_images'):\n","  if file.endswith(').png'):\n","    os.remove(f'{path}/fake_images/{file}')"],"id":"0aH21upCbZe9"},{"cell_type":"markdown","metadata":{"id":"yP3-0FOu5gMe"},"source":["#### remove style from wikiart_real images (already done, not necessary to run again)"],"id":"yP3-0FOu5gMe"},{"cell_type":"code","execution_count":null,"metadata":{"id":"u3u_DLhI5lXw"},"outputs":[],"source":["for file in os.listdir(f'{path}/wikiart_real'):\n","  new = \"_\".join(file.split(\"_\", 2)[:2])\n","  os.rename(f'{path}/wikiart_real/{file}', f'{path}/wikiart_real/{new}.png')"],"id":"u3u_DLhI5lXw"},{"cell_type":"markdown","metadata":{"id":"X2qzZXgubnSz"},"source":["### calculate FID"],"id":"X2qzZXgubnSz"},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":519816,"status":"ok","timestamp":1686157555839,"user":{"displayName":"TERRY MING","userId":"03637908485946588412"},"user_tz":420},"id":"dFmMpsCySNBL","outputId":"148e9bf4-eda9-4111-face-93bcd057f5be"},"outputs":[{"name":"stdout","output_type":"stream","text":["compute FID between two folders\n"]},{"name":"stderr","output_type":"stream","text":["/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 12 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n","  warnings.warn(_create_warning_msg(\n"]},{"name":"stdout","output_type":"stream","text":["Found 10000 images in the folder /content/drive/MyDrive/Pic 16B/CAN/fake_images\n"]},{"name":"stderr","output_type":"stream","text":["FID fake_images : 100%|██████████| 313/313 [03:53<00:00,  1.34it/s]\n"]},{"name":"stdout","output_type":"stream","text":["Found 10000 images in the folder /content/drive/MyDrive/Pic 16B/CAN/wikiart_real\n"]},{"name":"stderr","output_type":"stream","text":["FID wikiart_real : 100%|██████████| 313/313 [03:19<00:00,  1.57it/s]\n"]}],"source":["from cleanfid import fid\n","dcgan_score = fid.compute_fid(f'{path}/fake_images', f'{path}/wikiart_real')"],"id":"dFmMpsCySNBL"},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":330245,"status":"ok","timestamp":1686157886064,"user":{"displayName":"TERRY MING","userId":"03637908485946588412"},"user_tz":420},"id":"NfHGBjXTbqq2","outputId":"71f56e46-7099-4005-f99c-05d8c2b930b4"},"outputs":[{"name":"stdout","output_type":"stream","text":["compute FID between two folders\n","Found 10000 images in the folder /content/drive/MyDrive/Pic 16B/CAN/CAN_images\n"]},{"name":"stderr","output_type":"stream","text":["FID CAN_images : 100%|██████████| 313/313 [03:42<00:00,  1.41it/s]\n"]},{"name":"stdout","output_type":"stream","text":["Found 10000 images in the folder /content/drive/MyDrive/Pic 16B/CAN/wikiart_real\n"]},{"name":"stderr","output_type":"stream","text":["FID wikiart_real : 100%|██████████| 313/313 [01:20<00:00,  3.90it/s]\n"]}],"source":["can_score = fid.compute_fid(f'{path}/CAN_images', f'{path}/wikiart_real')"],"id":"NfHGBjXTbqq2"},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":20,"status":"ok","timestamp":1686157886065,"user":{"displayName":"TERRY MING","userId":"03637908485946588412"},"user_tz":420},"id":"kAxDYMFJbrHX","outputId":"6968c0a3-21f7-4bd0-ee2f-c6840fbba80e"},"outputs":[{"data":{"text/plain":["(185.15256074979845, 370.78927324817784)"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["dcgan_score, can_score"],"id":"kAxDYMFJbrHX"},{"cell_type":"markdown","metadata":{"id":"eGSQcdDIXHhj"},"source":["### b) Traditional image processing metrics"],"id":"eGSQcdDIXHhj"},{"cell_type":"markdown","metadata":{"id":"ehwauNDBVC89"},"source":["*Source: PyTorch Image Quality: Metrics for Image Quality Assessment*"],"id":"ehwauNDBVC89"},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":16789,"status":"ok","timestamp":1686197140179,"user":{"displayName":"TERRY MING","userId":"03637908485946588412"},"user_tz":420},"id":"kG3kTi2XULP-","outputId":"ef4c6ac2-d767-40c5-8dc1-e44c89b75057"},"outputs":[{"name":"stdout","output_type":"stream","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Requirement already satisfied: piq in /usr/local/lib/python3.10/dist-packages (0.7.1)\n","Requirement already satisfied: torchvision!=0.9.0,>=0.6.1 in /usr/local/lib/python3.10/dist-packages (from piq) (0.15.2+cu118)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchvision!=0.9.0,>=0.6.1->piq) (1.22.4)\n","Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from torchvision!=0.9.0,>=0.6.1->piq) (2.27.1)\n","Requirement already satisfied: torch==2.0.1 in /usr/local/lib/python3.10/dist-packages (from torchvision!=0.9.0,>=0.6.1->piq) (2.0.1+cu118)\n","Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision!=0.9.0,>=0.6.1->piq) (8.4.0)\n","Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->torchvision!=0.9.0,>=0.6.1->piq) (3.12.0)\n","Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->torchvision!=0.9.0,>=0.6.1->piq) (4.5.0)\n","Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->torchvision!=0.9.0,>=0.6.1->piq) (1.11.1)\n","Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->torchvision!=0.9.0,>=0.6.1->piq) (3.1)\n","Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->torchvision!=0.9.0,>=0.6.1->piq) (3.1.2)\n","Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->torchvision!=0.9.0,>=0.6.1->piq) (2.0.0)\n","Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch==2.0.1->torchvision!=0.9.0,>=0.6.1->piq) (3.25.2)\n","Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch==2.0.1->torchvision!=0.9.0,>=0.6.1->piq) (16.0.5)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->torchvision!=0.9.0,>=0.6.1->piq) (1.26.15)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->torchvision!=0.9.0,>=0.6.1->piq) (2022.12.7)\n","Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->torchvision!=0.9.0,>=0.6.1->piq) (2.0.12)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->torchvision!=0.9.0,>=0.6.1->piq) (3.4)\n","Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.0.1->torchvision!=0.9.0,>=0.6.1->piq) (2.1.2)\n","Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.0.1->torchvision!=0.9.0,>=0.6.1->piq) (1.3.0)\n","Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Requirement already satisfied: lpips in /usr/local/lib/python3.10/dist-packages (0.1.4)\n","Requirement already satisfied: torch>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from lpips) (2.0.1+cu118)\n","Requirement already satisfied: torchvision>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from lpips) (0.15.2+cu118)\n","Requirement already satisfied: numpy>=1.14.3 in /usr/local/lib/python3.10/dist-packages (from lpips) (1.22.4)\n","Requirement already satisfied: scipy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from lpips) (1.10.1)\n","Requirement already satisfied: tqdm>=4.28.1 in /usr/local/lib/python3.10/dist-packages (from lpips) (4.65.0)\n","Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=0.4.0->lpips) (3.12.0)\n","Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=0.4.0->lpips) (4.5.0)\n","Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=0.4.0->lpips) (1.11.1)\n","Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=0.4.0->lpips) (3.1)\n","Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=0.4.0->lpips) (3.1.2)\n","Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=0.4.0->lpips) (2.0.0)\n","Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=0.4.0->lpips) (3.25.2)\n","Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=0.4.0->lpips) (16.0.5)\n","Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from torchvision>=0.2.1->lpips) (2.27.1)\n","Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision>=0.2.1->lpips) (8.4.0)\n","Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=0.4.0->lpips) (2.1.2)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->torchvision>=0.2.1->lpips) (1.26.15)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->torchvision>=0.2.1->lpips) (2022.12.7)\n","Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->torchvision>=0.2.1->lpips) (2.0.12)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->torchvision>=0.2.1->lpips) (3.4)\n","Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=0.4.0->lpips) (1.3.0)\n"]}],"source":["import locale\n","locale.getpreferredencoding = lambda: \"UTF-8\"\n","\n","!pip install piq\n","!pip install lpips"],"id":"kG3kTi2XULP-"},{"cell_type":"code","execution_count":null,"metadata":{"id":"T97TNjiI4MI-"},"outputs":[],"source":["# Helper functions to retrieve images, resize if needed, and convert to np array\n","\n","def get_dcgan_fake_img_batch(idx):\n","  imgs = []\n","  for j in range(idx, idx+10):\n","      im_frame = Image.open(f'{path}/fake_images/fake_img_{j}.png')\n","      im_frame = im_frame.resize((128, 128))\n","      img = np.array(im_frame)\n","      imgs.append(img)\n","  return imgs\n","\n","def get_can_fake_img_batch(idx):\n","  imgs = []\n","  for j in range(idx, idx+10):\n","      im_frame = Image.open(f'{path}/CAN_images/gen_image_{j}.png')\n","      img = np.array(im_frame)\n","      imgs.append(img)\n","  return imgs\n","\n","def get_real_img_batch(idx):\n","  imgs = []\n","  for j in range(idx, idx+10):\n","      im_frame = Image.open(f'{path}/wikiart_real/image_{j}.png')\n","      img = np.array(im_frame)\n","      imgs.append(img)\n","  return imgs"],"id":"T97TNjiI4MI-"},{"cell_type":"markdown","metadata":{"id":"X2Ls-64la1vt"},"source":["SSIM (Structural Similarity Index Measure), PSNR (Peak Signal-to-Noise Ratio), etc. These are full-reference metrics (they compare the image to an initial uncompressed/distortion-free reference of the same image), and so they are not necessarily the most appropriate for generative image quality. They are better suited for image compression/restoration in comparison to the original image, but we can try."],"id":"X2Ls-64la1vt"},{"cell_type":"markdown","metadata":{"id":"CLIW4q0YX21G"},"source":["### Structural Similarity Index Measure (SSIM)\n","\n","-Calculates perceived structural differences, based on the idea that spatially close pixels have strong inter-dependencies\n","-A weighted combination of three comparison measurements—luminance, contrast, and structure using a sliding Gaussian window\n","-A value of 1 indicates perfect similarity, 0 = no similarity, and -1 = perfect anti-correlation\n","\n","$$l(x,y) = \\frac{2\\mu_x\\mu_y + c_1}{\\mu_x^2+\\mu_y^2+c_1}$$\n","$$c(x,y) = \\frac{2\\sigma_x\\sigma_y + c_2}{\\sigma_x^2+\\sigma_y^2+c_2}$$\n","$$s(x,y) = \\frac{\\sigma_{xy} + c_3}{\\sigma_x+\\sigma_y+c_3}$$\n","\n","where $\\mu$ is the pixel sample mean, $\\sigma^2$ is the variance,\n","$\\sigma_{xy}$ is the covariance,\n","$c_1 = (k_1L)^2$ and $c_2 = (k_2L)^2$ are stablizing coefficients where $L$ is the dynamic range of the pixel values and $k_1=0.01$ and $k_2=0.03$ are constants, and $c_3 = c_2/2$.\n","\n","$$SSIM(x,y) = l(x,y)^\\alpha c(x,y)^\\beta s(x,y)^\\gamma$$\n"],"id":"CLIW4q0YX21G"},{"cell_type":"markdown","metadata":{"id":"yvnxYH-YyvUp"},"source":["To calculate SSIM, we will take our 10,000 images of each (real, DCGAN, CAN) in batches of 10, pair them and calculate SSIM for each batch pair, then finally compute the mean SSIM."],"id":"yvnxYH-YyvUp"},{"cell_type":"code","execution_count":null,"metadata":{"id":"oqm6ELsr2TLx"},"outputs":[],"source":["from piq import ssim"],"id":"oqm6ELsr2TLx"},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":7261756,"status":"ok","timestamp":1686204401928,"user":{"displayName":"TERRY MING","userId":"03637908485946588412"},"user_tz":420},"id":"PmjbUMSd1454","outputId":"efca08f7-b1d1-4864-940e-f2798b6c0917"},"outputs":[{"name":"stdout","output_type":"stream","text":["Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off]\n"]},{"name":"stderr","output_type":"stream","text":["/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.\n","  warnings.warn(\n","/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.\n","  warnings.warn(msg)\n"]},{"name":"stdout","output_type":"stream","text":["Loading model from: /usr/local/lib/python3.10/dist-packages/lpips/weights/v0.1/alex.pth\n"]},{"name":"stderr","output_type":"stream","text":["100%|██████████| 1000/1000 [2:01:00<00:00,  7.26s/it]"]},{"name":"stdout","output_type":"stream","text":["Mean SSIM:\n","DCGAN: 0.08275356287509203, CAN: 0.011302134850993753\n","Mean LPIPS:\n","DCGAN: 0.3396139343678951, CAN: 0.8619114523530006\n"]},{"name":"stderr","output_type":"stream","text":["\n"]}],"source":["from tqdm import tqdm\n","\n","# We also calculate Learned Perceptual Image Patch Similarity (LPIPS),\n","# a common image quality metric often used as a loss for training GANs. More formally, it\n","# measures perceptual distance in the feature space of the AlexNet model (or VGG, depends on which you use)\n","# A lower score indicates that the compared images are more similar\n","import lpips\n","lpips_alex = lpips.LPIPS(net='alex') # best forward scores\n","\n","dcgan_ssims = []\n","can_ssims = []\n","dcgan_lpipses = []\n","can_lpipses = []\n","\n","for i in tqdm(range(0, 10000, 10)):\n","  dcgan_imgs = get_dcgan_fake_img_batch(i)\n","  dcgan_imgs = np.stack(dcgan_imgs)\n","  dcgan_imgs = np.transpose(dcgan_imgs, (0, 3, 1, 2))\n","\n","  can_imgs = get_can_fake_img_batch(i)\n","  can_imgs = np.stack(can_imgs)\n","  can_imgs = np.transpose(can_imgs, (0, 3, 1, 2))\n","\n","  real_imgs = get_real_img_batch(i)\n","  real_imgs = np.stack(real_imgs)\n","  real_imgs = np.transpose(real_imgs, (0, 3, 1, 2))\n","\n","  dcgan_ssim = ssim(torch.Tensor(dcgan_imgs/255), torch.Tensor(real_imgs/255)).item()\n","  can_ssim = ssim(torch.Tensor(can_imgs/255), torch.Tensor(real_imgs/255)).item()\n","  dcgan_ssims.append(dcgan_ssim)\n","  can_ssims.append(can_ssim)\n","\n","  dcgan_lpips = lpips_alex(torch.Tensor(dcgan_imgs), torch.Tensor(real_imgs)).mean().item()\n","  can_lpips = lpips_alex(torch.Tensor(can_imgs), torch.Tensor(real_imgs)).mean().item()\n","  dcgan_lpipses.append(dcgan_lpips)\n","  can_lpipses.append(can_lpips)\n","\n","print(\"\\nMean SSIM:\")\n","print(f'DCGAN: {np.mean(dcgan_ssims)}, CAN: {np.mean(can_ssims)}')\n","print(\"Mean LPIPS:\")\n","print(f'DCGAN: {np.mean(dcgan_lpipses)}, CAN: {np.mean(can_lpipses)}')"],"id":"PmjbUMSd1454"},{"cell_type":"markdown","metadata":{"id":"RVMZejbpbSiA"},"source":["In contrast, we can also try using no-reference image quality metrics such as\n","BRISQUE (Blind/Referenceless Image Spatial Quality Evaluator), NIQE (Natural Image Quality Evaluator), or PIQE (Perception based Image Quality Evaluator). These are directly performed on the generated images with no reference to the real images, and they use statistical features to evaluate image quality."],"id":"RVMZejbpbSiA"},{"cell_type":"markdown","metadata":{"id":"ZDn4qrz3Z1Pt"},"source":["#### iii) Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE)"],"id":"ZDn4qrz3Z1Pt"},{"cell_type":"markdown","metadata":{"id":"ZqN7GWSRiy7p"},"source":["-Extracts 36 point-wise statistical features of locally normalized luminance coefficients to measure deviations from a Natural Scene\n","Statistics (NSS)-based model\n","-Smaller scores indicate higher perceptual quality\n","\n"],"id":"ZqN7GWSRiy7p"},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":20071,"status":"ok","timestamp":1686253015163,"user":{"displayName":"TERRY MING","userId":"03637908485946588412"},"user_tz":420},"id":"8C5_Vv46gTBr","outputId":"ab0e2ec0-92df-4fc3-9106-d112e6a51723"},"outputs":[{"name":"stdout","output_type":"stream","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Requirement already satisfied: brisque in /usr/local/lib/python3.10/dist-packages (0.0.15)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from brisque) (1.22.4)\n","Requirement already satisfied: scikit-image in /usr/local/lib/python3.10/dist-packages (from brisque) (0.19.3)\n","Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from brisque) (1.10.1)\n","Requirement already satisfied: opencv-python in /usr/local/lib/python3.10/dist-packages (from brisque) (4.7.0.72)\n","Requirement already satisfied: libsvm in /usr/local/lib/python3.10/dist-packages (from brisque) (3.23.0.4)\n","Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.10/dist-packages (from scikit-image->brisque) (3.1)\n","Requirement already satisfied: pillow!=7.1.0,!=7.1.1,!=8.3.0,>=6.1.0 in /usr/local/lib/python3.10/dist-packages (from scikit-image->brisque) (8.4.0)\n","Requirement already satisfied: imageio>=2.4.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image->brisque) (2.25.1)\n","Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.10/dist-packages (from scikit-image->brisque) (2023.4.12)\n","Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image->brisque) (1.4.1)\n","Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from scikit-image->brisque) (23.1)\n"]}],"source":["!pip install brisque"],"id":"8C5_Vv46gTBr"},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"background_save":true,"base_uri":"https://localhost:8080/"},"id":"77X-A5H2gVOv","outputId":"750cc390-22f6-4182-d976-0eef3d878dab"},"outputs":[{"name":"stderr","output_type":"stream","text":["100%|██████████| 1000/1000 [4:07:41<00:00, 14.86s/it]"]},{"name":"stdout","output_type":"stream","text":["\n","Mean BRISQUE scores:\n","DCGAN: 50.716437383869525, CAN: 88.80208471716486, Real: 18.88789619882305\n"]},{"name":"stderr","output_type":"stream","text":["\n"]}],"source":["from brisque import BRISQUE\n","\n","obj = BRISQUE(url=False)\n","\n","from PIL import Image\n","\n","dcgan_brisques = []\n","can_brisques = []\n","real_brisques = []\n","for i in tqdm(range(0, 10000, 10)):\n","  dcgan_imgs = get_dcgan_fake_img_batch(i)\n","  can_imgs = get_can_fake_img_batch(i)\n","  real_imgs = get_real_img_batch(i)\n","\n","  dcgan_brisques.extend([obj.score(img) for img in dcgan_imgs])\n","  can_brisques.extend([obj.score(img) for img in can_imgs])\n","  real_brisques.extend([obj.score(img) for img in real_imgs])\n","\n","print(\"\\nMean BRISQUE scores:\")\n","print(f'DCGAN: {np.mean(dcgan_brisques)}, CAN: {np.mean(can_brisques)}, Real: {np.mean(real_brisques)}')"],"id":"77X-A5H2gVOv"}],"metadata":{"accelerator":"GPU","colab":{"provenance":[],"gpuType":"T4"},"gpuClass":"standard","kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.10"}},"nbformat":4,"nbformat_minor":5}