Information Bottleneck for Deep Learning

Dataset: fashion mnist

Model 1: MLP with Batch Normalization

Refactored and reused my previous code for this implementation.

URL: link

Model 2: CNN

Implemented in PyTorch

URL: link

Model extension:

1. MLP with more than 3 layers (computationally expensive, in progress)
2. MLP with weights intialization via denoised autoencoder (in progress)

Other parameters:

number of bins for MI: 10

Papers

0. Opening the black box of Deep Neural Networks via Information
1. Deep Learning and the Information Bottleneck Principle
2. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
3. Regularization of Neural Networks using DropConnect