diff --git a/.gitignore b/.gitignore new file mode 100755 index 0000000..fd039b0 --- /dev/null +++ b/.gitignore @@ -0,0 +1,57 @@ +## General + +# Editor temporaries +*.swp +*~ + +# Sublime Text settings +*.sublime-workspace +*.sublime-project + +# OS files +.DS_Store +.DS_Store? +._* +.Spotlight-V100 +.Trashes +ehthumbs.db +Thumbs.db + +## Caffe +caffe +caffe_rbdn +caffe_colorization +resources +training_log +testing_log +snapshot + +# Data and models +data +Data +models +results +*.caffemodel +*.caffemodel.h5 +*.solverstate +*.solverstate.h5 +*.binaryproto +*leveldb +*lmdb + +## Miscellaneous + +# Compiled python +*.pyc + +# Compiled MATLAB +*.mex* + +# Archives +*.tar +*.tar.* +*.tgz +*.zip +*.rar +*.7z + diff --git a/LICENSE b/LICENSE new file mode 100755 index 0000000..8f5b0e8 --- /dev/null +++ b/LICENSE @@ -0,0 +1,38 @@ +Copyright (c) 2017, Venkataraman Santhanam, Vlad I. Morariu, Larry S. Davis +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* If you use the work in scientific research or as part of a larger software + system, you are requested to cite the use in any related publications or + technical documentation. The work is based upon: + + Santhanam, Venkataraman, Vlad I. Morariu, and Larry S. Davis. + "Generalized Deep Image to Image Regression." + arXiv preprint arXiv:1612.03268 (2016). + + @article{santhanam2016generalized, + title={Generalized Deep Image to Image Regression}, + author={Santhanam, Venkataraman and Morariu, Vlad I and Davis, Larry S}, + journal={arXiv preprint arXiv:1612.03268}, + year={2016} + } + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/README.md b/README.md new file mode 100755 index 0000000..9afe82d --- /dev/null +++ b/README.md @@ -0,0 +1,54 @@ +# RBDN (Recursively Branched Deconvolutional Network) +![placeholder](https://github.com/venkai/RBDN/blob/gh-pages/assets/placeholder.png) +**RBDN** is an architecture for [**Generalized Deep Image to Image Regression**](https://arxiv.org/abs/1612.03268) which features +* a memory-efficient recursive branched scheme with extensive parameter sharing that computes an early learnable multi-context representation of the input, +* end-to-end preservation of local correspondences from input to output and +* ability to choose context-vs-locality based on task as well as apply a per-pixel multi-context non-linearity. + +## Architecture +![pipeline](https://github.com/venkai/RBDN/blob/gh-pages/assets/pipeline.png) +**RBDN** gives state-of-the-art performance on 3 diverse *image-to-image regression* tasks: **Denoising**, **Relighting**, **Colorization**. + +## Installation & Usage + +- **Clone:** Run `git clone -b master --single-branch https://github.com/venkai/RBDN.git` + +- **Setup:** Go to repository `cd RBDN` and run `./setup.sh`. This will fetch caffe, download pretrained caffe models for all 3 experiments (**denoising/relighting/colorization**) and inference data, as well as set up the directory structure and symbolic links for all the training/inference scripts. + +- **Install Caffe:** Note that `setup.sh` pulls 2 different branches of caffe into 2 separate directories: namely `caffe_colorization` used for **colorization** and `caffe_rbdn` which is used for both **denoising/relighting** experiments. Both these branches will eventually be merged with the master branch in [**venkai/caffe**](https://github.com/venkai/caffe). However for now, you would have to separately install both these caffe versions if you want to perform all 3 experiments. + +- **Data:** + + - Inference data is automatically downloaded by `setup.sh`. + + - Training data/imglist for **relighting** experiment can be downloaded from either of these mirrors: [**[1]**](https://drive.google.com/file/d/0B3PoH3B39H2reWxzd3VDZDFVSlE/view?usp=sharing)/[**[2]**](https://drive.google.com/file/d/0B4c0dYlyY36JY3EwWUo3Y2MtNm8/view?usp=sharing) + This downloads the file `multipie.tar.gz`. Move it to `./data/training` and run `tar xvzf multipie.tar.gz && rm multipie.tar.gz` + + - **Denoising/colorization** experiments use the same training data/imglist: which is every single *unresized* train & validation image from both [**ImageNet ILSVRC2012**](https://arxiv.org/abs/1409.0575) and [**MS-COCO2014**](http://mscoco.org/) whose smallest spatial dimension is greater than **128** (**~1.7 million** images in total). You can simply download these datasets from their respective sources and place/symlink them within `./data/training/` without any preprocessing whatsoever. Place the appropriate imglist in `./data/training/imgset/train.txt` with the image-paths in `train.txt` being relative to `./data/training` + + - Note that data folders are not tracked by git. + +- **Inference:** Each experiment (**denoising/relighting/colorization**) has its own folder in `./inference` that contains an experiment specific MATLAB inference script `get_pred.m` which uses the [**Matcaffe**](http://caffe.berkeleyvision.org/tutorial/interfaces.html#matlab) interface to evaluate pretrained models in `./models`. The script `./inference/run_matcaffe.sh` can be used to load caffe dependencies to `LD_LIBRARY_PATH` and then start MATLAB interactively. + +- **Training:** Each experiment (**denoising/relighting/colorization**) has its own folder in `./training` that contain 2 key experiment specific scripts: + - `start_train.sh`: This starts training an **RBDN** model, either from scratch or from the most recent snapshot in the `snapshot` directory. You can pause training at any moment with `Ctrl+C` and most recent snapshot will be saved in `./snapshot/trn_iter_[*].solverstate`. Running `./start_train.sh` again will automatically resume from that snapshot. + - `run_bn.sh`: This takes the most recent snapshot in `./snapshot` and prepares it for inference by passing training data through the network and computing global mean/variance for all the *batch-normalization* layers in the network. The resulting inference-ready model is saved as `./tst_[ITER].caffemodel`, where `ITER` is the iteration corresponding to the most recent snapshot. + +# License & Citation +**RBDN** is released under a variant of the [BSD 2-Clause license](https://github.com/venkai/RBDN/blob/master/LICENSE). + +If you find **RBDN** useful in your research, please consider citing our paper: + +``` +@article{santhanam2016generalized, + title={Generalized Deep Image to Image Regression}, + author={Santhanam, Venkataraman and Morariu, Vlad I and Davis, Larry S}, + journal={arXiv preprint arXiv:1612.03268}, + year={2016} +} +``` + +# Acknowledgments +* We would like to thank [Yangqing Jia](http://daggerfs.com/), [Evan Shelhamer](http://imaginarynumber.net/) and the [**BVLC/BAIR**](http://bair.berkeley.edu/) team for creating & maintaining [**caffe**](http://caffe.berkeleyvision.org/), [Richard Zhang](https://richzhang.github.io/) for [colorization layers in caffe](https://github.com/richzhang/colorization) and [Hyeonwoo Noh](http://cvlab.postech.ac.kr/~hyeonwoonoh/), [Seunghoon Hong](http://cvlab.postech.ac.kr/~maga33/), [Dmytro Mishkin](https://github.com/ducha-aiki) for several useful caffe layers, all of which were instrumental in creating **RBDN**. + +* This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 2014-14071600012. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. \ No newline at end of file diff --git a/inference/colorization/get_pred.m b/inference/colorization/get_pred.m new file mode 100755 index 0000000..f500de9 --- /dev/null +++ b/inference/colorization/get_pred.m @@ -0,0 +1,57 @@ +function get_pred + +config=create_config; +net = caffe.Net(config.model,config.weights,'test'); +imgdir='../Data/colorization'; +resdir='./results'; +if ~isdir(resdir), mkdir(resdir); end +% Get image list +d=dir(strcat(imgdir,'/*')); +d={d.name}'; d=d(3:end); +for i=1:length(d) + imgfile=d{i}; + dt=strfind(imgfile,'.'); dt=dt(end); + resfile=strcat(resdir,'/',imgfile(1:dt),'png'); + %if exist(resfile,'file')==2, continue; end; + I=imread(strcat(imgdir,'/',imgfile)); + % If input image is larger that 800*800 (i.e. # of pixels > 640000), + % then network doesn't fit in GPU memory. Of course this depends on how much + % memory you have and if you are operating in CPU/GPU mode. + if numel(I(:,:,1))>640000 + if size(I,1)>=size(I,2) + I=imresize(I,[800,NaN],'bicubic'); + else + I=imresize(I,[NaN,800],'bicubic'); + end + end + if size(I,3)==1, I=repmat(I,[1,1,3]); end + %imshow(I); + [h,w,~]=size(I); + I=padarray(I,[64,64],'symmetric'); + I=I(1:end-rem(end,32),1:end-rem(end,32),:); + img_lab=rgb2lab(I); + img=permute(img_lab(:,:,1),[2,1,3])-50; + fprintf('[%d/%d] Processing %s: [%d x %d]\n',i,length(d),imgfile,size(img,1),size(img,2)); + net.blobs('data').reshape([size(img,1) size(img,2) 1 1]); % reshape blob 'data' + net.reshape(); %Reshape remaining blobs accordingly + cnn_input={single(img)}; + pred=net.forward(cnn_input); + pred=imresize(permute(pred{1},[2,1,3]),4); + pred_lab=zeros(size(img_lab)); + pred_lab(:,:,1)=img_lab(:,:,1); + pred_lab(:,:,2:3)=pred; + pred_rgb=(lab2rgb(pred_lab)); + pred_rgb=pred_rgb(65:h+64,65:w+64,:); + imwrite(pred_rgb,resfile); +end +caffe.reset_all(); + +function config=create_config +config.model='./test.prototxt'; +config.weights='../../models/rbdn_colorization.caffemodel'; +config.caffe_root = '../caffe_colorization'; +fprintf('initializing caffe..\n'); +addpath(fullfile(config.caffe_root, 'matlab')); +config.gpuNum=0; caffe.set_mode_gpu(); caffe.set_device(config.gpuNum); +%caffe.set_mode_cpu(); +caffe.reset_all(); diff --git a/inference/colorization/get_pred_resize.m b/inference/colorization/get_pred_resize.m new file mode 100755 index 0000000..fe6b879 --- /dev/null +++ b/inference/colorization/get_pred_resize.m @@ -0,0 +1,51 @@ +function get_pred_resize(new_size) + +% Use this instead of get_pred.m for colorizing very high-res images. +% All results in the paper were however generated with get_pred.m. + + +if nargin < 1, new_size=[224, 224]; end; +if numel(new_size)==1, new_size=[new_size, new_size]; end; + +config=create_config; +net = caffe.Net(config.model,config.weights,'test'); +imgdir='../Data/colorization'; +resdir='./results'; +if ~isdir(resdir), mkdir(resdir); end +% Get image list +d=dir(strcat(imgdir,'/*')); +d={d.name}'; d=d(3:end); +for i=1:length(d) + imgfile=d{i}; + dt=strfind(imgfile,'.'); dt=dt(end); + resfile=strcat(resdir,'/',imgfile(1:dt),'png'); + I=imread(strcat(imgdir,'/',imgfile)); + if size(I,3)==1, I=repmat(I,[1,1,3]); end + img_lab=rgb2lab(I); + I_rz=rgb2lab(imresize(I,new_size,'bicubic')); + img=permute(I_rz(:,:,1),[2,1,3])-50; % H*W -> W*H + img=img(1:end-rem(end,32),1:end-rem(end,32)); + fprintf('Processing %s: [%d x %d]\n',imgfile,size(img,1),size(img,2)); + % Reshape blob 'data' + net.blobs('data').reshape([size(img,1) size(img,2) 1 1]); + net.reshape(); %Reshape remaining blobs accordingly + cnn_input={single(img)}; + pred=net.forward(cnn_input); + pred=imresize(permute(pred{1},[2,1,3]),[size(I,1),size(I,2)]); + pred_lab=zeros(size(img_lab)); + pred_lab(:,:,1)=img_lab(:,:,1); + pred_lab(:,:,2:3)=pred; + pred_rgb=(lab2rgb(pred_lab)); + imwrite(pred_rgb,resfile); +end +caffe.reset_all(); + +function config=create_config +config.model='./test.prototxt'; +config.weights='../../models/rbdn_colorization.caffemodel'; +config.caffe_root = '../caffe_colorization'; +fprintf('initializing caffe..\n'); +addpath(fullfile(config.caffe_root, 'matlab')); +config.gpuNum=0; caffe.set_mode_gpu(); caffe.set_device(config.gpuNum); +%caffe.set_mode_cpu(); +caffe.reset_all(); diff --git a/inference/colorization/test.prototxt b/inference/colorization/test.prototxt new file mode 100755 index 0000000..4a90d6d --- /dev/null +++ b/inference/colorization/test.prototxt @@ -0,0 +1,650 @@ +name: "base_colorization_test" + +input: "data" +input_dim: 1 +input_dim: 1 +input_dim: 512 +input_dim: 512 + + +# H x W x 1 + +# CONVOLUTION (conv1) +layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W x 64 + +# BN (bn1) +layer { name: "bn1" type: "BN" bottom: "conv1" top: "conv1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu1) +layer { name: "relu1" type: "ReLU" bottom: "conv1" top: "conv1" } + +# POOLING (pool1) +layer { + name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/2) x (W/2) X 64 + +# Introduce Branch 1 Here + +# CONVOLUTION (convB1_1) +layer { name: "convB1_1" type: "Convolution" bottom: "pool1" top: "convB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bnB1_1) +layer { name: "bnB1_1" type: "BN" bottom: "convB1_1" top: "convB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB1_1) +layer { name: "reluB1_1" type: "ReLU" bottom: "convB1_1" top: "convB1_1" } + +# POOLING (poolB1) +layer { + name: "poolB1" type: "Pooling" bottom: "convB1_1" top: "poolB1" top: "poolB1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/4) x (W/4) X 64 + +# Introduce Branch 2 Here + +# CONVOLUTION (convB2_1) +layer { name: "convB2_1" type: "Convolution" bottom: "poolB1" top: "convB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB2_1) +layer { name: "bnB2_1" type: "BN" bottom: "convB2_1" top: "convB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB2_1) +layer { name: "reluB2_1" type: "ReLU" bottom: "convB2_1" top: "convB2_1" } + +# POOLING (poolB2) +layer { + name: "poolB2" type: "Pooling" bottom: "convB2_1" top: "poolB2" top: "poolB2_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/8) x (W/8) X 64 + +# Introduce Branch 3 Here + +# CONVOLUTION (convB3_1) +layer { name: "convB3_1" type: "Convolution" bottom: "poolB2" top: "convB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB3_1) +layer { name: "bnB3_1" type: "BN" bottom: "convB3_1" top: "convB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB3_1) +layer { name: "reluB3_1" type: "ReLU" bottom: "convB3_1" top: "convB3_1" } + +# POOLING (poolB3) +layer { + name: "poolB3" type: "Pooling" bottom: "convB3_1" top: "poolB3" top: "poolB3_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/16) x (W/16) X 64 + + +# Introduce Branch 4 Here + +# CONVOLUTION (convB4_1) +layer { name: "convB4_1" type: "Convolution" bottom: "poolB3" top: "convB4_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB4_1) +layer { name: "bnB4_1" type: "BN" bottom: "convB4_1" top: "convB4_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB4_1) +layer { name: "reluB4_1" type: "ReLU" bottom: "convB4_1" top: "convB4_1" } + +# POOLING (poolB4) +layer { + name: "poolB4" type: "Pooling" bottom: "convB4_1" top: "poolB4" top: "poolB4_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/32) x (W/32) X 64 + +# CONVOLUTION (convB4_2) +layer { name: "convB4_2" type: "Convolution" bottom: "poolB4" top: "convB4_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/32) x (W/32) X 64 + +# BN (bnB4_2) +layer { name: "bnB4_2" type: "BN" bottom: "convB4_2" top: "convB4_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB4_2) +layer { name: "reluB4_2" type: "ReLU" bottom: "convB4_2" top: "convB4_2" } + + +# UNPOOLING (unpoolB4) +layer { + name: "unpoolB4" type: "Unpooling" bottom: "convB4_2" bottom: "poolB4_mask" top: "unpoolB4" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/16) x (W/16) X 64 + +# DECONVOLUTION (deconvB4_1) +layer { name: "deconvB4_1" type: "Deconvolution" bottom: "unpoolB4" top: "deconvB4_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (debnB4_1) +layer { name: "debnB4_1" type: "BN" bottom: "deconvB4_1" top: "deconvB4_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB4_1) +layer { name: "dereluB4_1" type: "ReLU" bottom: "deconvB4_1" top: "deconvB4_1" } + + + +# Merge Branch 4 Here +# CONCAT (mergeB4) +layer { + name: "mergeB4" + bottom: "poolB3" + bottom: "deconvB4_1" + top: "mergeB4" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/16) x (W/16) X 128 + + +# CONVOLUTION (convB3_2) +layer { name: "convB3_2" type: "Convolution" bottom: "mergeB4" top: "convB3_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB3_2) +layer { name: "bnB3_2" type: "BN" bottom: "convB3_2" top: "convB3_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB3_2) +layer { name: "reluB3_2" type: "ReLU" bottom: "convB3_2" top: "convB3_2" } + + +# UNPOOLING (unpoolB3) +layer { + name: "unpoolB3" type: "Unpooling" bottom: "convB3_2" bottom: "poolB3_mask" top: "unpoolB3" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/8) x (W/8) X 64 + +# DECONVOLUTION (deconvB3_1) +layer { name: "deconvB3_1" type: "Deconvolution" bottom: "unpoolB3" top: "deconvB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (debnB3_1) +layer { name: "debnB3_1" type: "BN" bottom: "deconvB3_1" top: "deconvB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB3_1) +layer { name: "dereluB3_1" type: "ReLU" bottom: "deconvB3_1" top: "deconvB3_1" } + + + +# Merge Branch 3 Here +# CONCAT (mergeB3) +layer { + name: "mergeB3" + bottom: "poolB2" + bottom: "deconvB3_1" + top: "mergeB3" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/8) x (W/8) X 128 + +# CONVOLUTION (convB2_2) +layer { name: "convB2_2" type: "Convolution" bottom: "mergeB3" top: "convB2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB2_2) +layer { name: "bnB2_2" type: "BN" bottom: "convB2_2" top: "convB2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB2_2) +layer { name: "reluB2_2" type: "ReLU" bottom: "convB2_2" top: "convB2_2" } + + +# UNPOOLING (unpoolB2) +layer { + name: "unpoolB2" type: "Unpooling" bottom: "convB2_2" bottom: "poolB2_mask" top: "unpoolB2" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/4) x (W/4) X 64 + +# DECONVOLUTION (deconvB2_1) +layer { name: "deconvB2_1" type: "Deconvolution" bottom: "unpoolB2" top: "deconvB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (debnB2_1) +layer { name: "debnB2_1" type: "BN" bottom: "deconvB2_1" top: "deconvB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB2_1) +layer { name: "dereluB2_1" type: "ReLU" bottom: "deconvB2_1" top: "deconvB2_1" } + + + +# Merge Branch 2 Here +# CONCAT (mergeB2) +layer { + name: "mergeB2" + bottom: "poolB1" + bottom: "deconvB2_1" + top: "mergeB2" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/4) x (W/4) X 128 + +# CONVOLUTION (convB1_2) +layer { name: "convB1_2" type: "Convolution" bottom: "mergeB2" top: "convB1_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB1_2) +layer { name: "bnB1_2" type: "BN" bottom: "convB1_2" top: "convB1_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB1_2) +layer { name: "reluB1_2" type: "ReLU" bottom: "convB1_2" top: "convB1_2" } + + +# UNPOOLING (unpoolB1) +layer { + name: "unpoolB1" type: "Unpooling" bottom: "convB1_2" bottom: "poolB1_mask" top: "unpoolB1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/2) x (W/2) X 64 + +# DECONVOLUTION (deconvB1_1) +layer { name: "deconvB1_1" type: "Deconvolution" bottom: "unpoolB1" top: "deconvB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (debnB1_1) +layer { name: "debnB1_1" type: "BN" bottom: "deconvB1_1" top: "deconvB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB1_1) +layer { name: "dereluB1_1" type: "ReLU" bottom: "deconvB1_1" top: "deconvB1_1" } + + +# Merge Branch 1 Here +# CONCAT (mergeB1) +layer { + name: "mergeB1" + bottom: "pool1" + bottom: "deconvB1_1" + top: "mergeB1" + type: "Concat" + concat_param { + axis: 1 + } +} + +# (H/2) x (W/2) X 128 + +# CONVOLUTION (conv2_1) +layer { name: "conv2_1" type: "Convolution" bottom: "mergeB1" top: "conv2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_1) +layer { name: "bn2_1" type: "BN" bottom: "conv2_1" top: "conv2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_1) +layer { name: "relu2_1" type: "ReLU" bottom: "conv2_1" top: "conv2_1" } + +# CONVOLUTION (conv2_2) +layer { name: "conv2_2" type: "Convolution" bottom: "conv2_1" top: "conv2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_2) +layer { name: "bn2_2" type: "BN" bottom: "conv2_2" top: "conv2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_2) +layer { name: "relu2_2" type: "ReLU" bottom: "conv2_2" top: "conv2_2" } + + +# CONVOLUTION (conv2_3) +layer { name: "conv2_3" type: "Convolution" bottom: "conv2_2" top: "conv2_3" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_3) +layer { name: "bn2_3" type: "BN" bottom: "conv2_3" top: "conv2_3" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_3) +layer { name: "relu2_3" type: "ReLU" bottom: "conv2_3" top: "conv2_3" } + + +# CONVOLUTION (conv2_4) +layer { name: "conv2_4" type: "Convolution" bottom: "conv2_3" top: "conv2_4" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_4) +layer { name: "bn2_4" type: "BN" bottom: "conv2_4" top: "conv2_4" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_4) +layer { name: "relu2_4" type: "ReLU" bottom: "conv2_4" top: "conv2_4" } + + +# CONVOLUTION (conv2_5) +layer { name: "conv2_5" type: "Convolution" bottom: "conv2_4" top: "conv2_5" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_5) +layer { name: "bn2_5" type: "BN" bottom: "conv2_5" top: "conv2_5" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_5) +layer { name: "relu2_5" type: "ReLU" bottom: "conv2_5" top: "conv2_5" } + + +# CONVOLUTION (conv2_6) +layer { name: "conv2_6" type: "Convolution" bottom: "conv2_5" top: "conv2_6" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_6) +layer { name: "bn2_6" type: "BN" bottom: "conv2_6" top: "conv2_6" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_6) +layer { name: "relu2_6" type: "ReLU" bottom: "conv2_6" top: "conv2_6" } + + +# CONVOLUTION (conv2_7) +layer { name: "conv2_7" type: "Convolution" bottom: "conv2_6" top: "conv2_7" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_7) +layer { name: "bn2_7" type: "BN" bottom: "conv2_7" top: "conv2_7" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_7) +layer { name: "relu2_7" type: "ReLU" bottom: "conv2_7" top: "conv2_7" } + + +# CONVOLUTION (conv2_8) +layer { name: "conv2_8" type: "Convolution" bottom: "conv2_7" top: "conv2_8" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_8) +layer { name: "bn2_8" type: "BN" bottom: "conv2_8" top: "conv2_8" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_8) +layer { name: "relu2_8" type: "ReLU" bottom: "conv2_8" top: "conv2_8" } + + +# CONVOLUTION (conv2_9) +layer { name: "conv2_9" type: "Convolution" bottom: "conv2_8" top: "conv2_9" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_9) +layer { name: "bn2_9" type: "BN" bottom: "conv2_9" top: "conv2_9" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_9) +layer { name: "relu2_9" type: "ReLU" bottom: "conv2_9" top: "conv2_9" } + + +# H/2 x W/2 x 64 + +# DECONVOLUTION (deconv1) +layer { name: "deconv1" type: "Deconvolution" bottom: "conv2_9" top: "deconv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 2 kernel_size: 5 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H/2 x W/2 X 64 + +# BN (debn1) +layer { name: "debn1" type: "BN" bottom: "deconv1" top: "deconv1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (derelu1) +layer { name: "derelu1" type: "ReLU" bottom: "deconv1" top: "deconv1" } + +# H/2 x W/2 x 64 + +# **************************** +# ***** Unary prediction ***** +# **************************** + +# CONVOLUTION (conv_313) +layer { name: "conv_313" type: "Convolution" bottom: "deconv1" top: "conv_313" + param { lr_mult: 0.1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 313 pad: 1 kernel_size: 4 stride: 2 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H/4 x W/4 X 313 + +layer { + name: "conv_313_rh" + type: "Scale" + bottom: "conv_313" + top: "conv_313" + scale_param { + bias_term: false + filler { type: "constant" value: 2.606 } + } +} +layer { + name: "class_313_rh" + type: "Softmax" + bottom: "conv_313" + top: "class_313_rh" +} +# ******************** +# ***** Decoding ***** +# ******************** +layer { + name: "class_ab" + type: "Convolution" + bottom: "class_313_rh" + top: "class_ab" + convolution_param { + num_output: 2 + kernel_size: 1 + stride: 1 + } +} \ No newline at end of file diff --git a/inference/denoising/compute_psnr.m b/inference/denoising/compute_psnr.m new file mode 100755 index 0000000..b60b655 --- /dev/null +++ b/inference/denoising/compute_psnr.m @@ -0,0 +1,38 @@ +function mPSNRs=compute_psnr + +mPSNRs=zeros(1,11); +curr=0; +resdir='./psnr/'; +if ~isdir(resdir), mkdir(resdir); end +for nlevel=10:5:60 + resfile=sprintf('%s/n%d_psnr.mat',resdir,nlevel); + if exist(resfile,'file')==2, + load(resfile,'mPSNR'); + else + mPSNR=tmp_psnr_fixnoise(nlevel); + end + curr=curr+1; + mPSNRs(curr)=mPSNR; + fprintf('Test Noise std %0.2d: mPSNR=%0.2f\n',nlevel,mPSNR); +end +save(strcat(resdir,'/mPSNR.mat'),'mPSNRs'); +fprintf('ns=%0.2d\t',10:5:60); fprintf('\n'); +fprintf('%0.2f\t',mPSNRs); fprintf('\n'); + +function mPSNR=tmp_psnr_fixnoise(nlevel) + +if nlevel<=25, ntype='low_noise'; else ntype='high_noise'; end +preddir=sprintf('./results/%s/test/noise_std_%d/noisy_image',ntype,nlevel); +gtdir=sprintf('../Data/denoising/%s/test/clean_images/clean_image',ntype); +% inputdir=sprintf('../Data/denoising/%s/test/noise_std_%d/noisy_image',ntype,nlevel); +resfile=sprintf('./psnr/n%d_psnr.mat',nlevel); +PSNR=zeros(300,1); +for i=1:300 + GT=imread(sprintf('%s_%d.png',gtdir,i)); + P=imread(sprintf('%s_%d.png',preddir,i)); + % I=imread(sprintf('%s_%d.png',inputdir,i)); + PSNR(i,1)=psnr(P,GT); + % PSNR(i,2)=psnr(I,GT); +end +mPSNR=mean(PSNR); +save(resfile,'PSNR','mPSNR'); \ No newline at end of file diff --git a/inference/denoising/get_pred.m b/inference/denoising/get_pred.m new file mode 100755 index 0000000..8dbf0a8 --- /dev/null +++ b/inference/denoising/get_pred.m @@ -0,0 +1,97 @@ +function get_pred + +rng(9); +config=create_config; +config.resdir='./results/'; +config.datadir='../Data/denoising'; +if ~isdir(config.resdir), mkdir(config.resdir); end + +fid=fopen(sprintf('%s/imgset/test.txt',config.datadir),'r'); +imglist=textscan(fid,'%s%s'); fclose(fid); + +% Note that the network is fully convolutional and can be reshaped to +% handle variable input size. Sort test images in descending order of size +% in order to minimize calls to net.reshape(). +rs_indices=[]; +load(sprintf('%s/imgset/imglist_dn_test_sorted.mat',config.datadir)); +imglist=imglist{1}(rs_indices); + + +config.imglist=imglist; +config.net=load_net(config); +hcurr=512; wcurr=512; +for ind=1:1:length(imglist) + fprintf('[%d/%d] %s\n',ind,length(imglist),imglist{ind}); + [hcurr,wcurr]=save_pred_oneimg(ind,config,hcurr,wcurr); +end +caffe.reset_all(); +fprintf('\n\n------------\n'); +fprintf('Computing Peak Signal to Noise Ratio for Results\n') +fprintf('------------\n\n'); +compute_psnr; + +function[hnew,wnew]=save_pred_oneimg(ind,config,h_old,w_old) +net=config.net; +imgrelpath=config.imglist{ind}; +imgpath=sprintf('%s%s',config.datadir,imgrelpath); +resfile=sprintf('%s%s',config.resdir,imgrelpath); +% Try to maintain same directory structure for resdir as in datadir and +% create sub-directories in resdir as neccessary +sl_ind=strfind(resfile,'/'); sl_ind=sl_ind(end)-1; +resdir=resfile(1:sl_ind); +if ~isdir(resdir), mkdir(resdir); end +% Check if resfile exists +if exist(resfile,'file')==2 + fprintf('RESULT %s already exists\n',resfile); + hnew=h_old; wnew=w_old; + return; +end +% Do forward pass with image +I=imread(imgpath); +[h,w,~]=size(I); +% Note that for a K-branch RBDN, we have K+1 downsampling layers. +% So height and width must be a multiple of 2^(K+1). +% If this is not the case, we slide the network across the image +% and stitch (average) the predictions later. +fac=2^(config.branches+1); +sh=rem(h,fac); sw=rem(w,fac); +hnew=h-sh; wnew=w-sw; +O=zeros(size(I)); % Per-pixel sum of predictions +C=zeros(size(I)); % Per-pixel count of # of additions +if h_old~=hnew || w_old~=wnew % Reshape only when required + fprintf('Reshaping Network from [%dx%d] to [%dx%d]\n',h_old,w_old,hnew,wnew); + net.blobs('data').reshape([wnew hnew 1 1]); % Reshape blob 'data' + net.reshape(); % Reshape remaining blobs accordingly +end +fprintf('Stitching %dx%d=%d dense reconstructions\n',sh+1,sw+1,(sh+1)*(sw+1)); +for i=0:sh + for j=0:sw + img=I(1+i:hnew+i,1+j:wnew+j,:); + img=permute(img,[2,1,3]); % H*W -> W*H + cnn_input={single(img)}; + pred=net.forward(cnn_input); + pred=permute(pred{1},[2,1,3]); % W*H -> H*W + O(1+i:hnew+i,1+j:wnew+j,:)=O(1+i:hnew+i,1+j:wnew+j,:)+pred; + C(1+i:hnew+i,1+j:wnew+j,:)=C(1+i:hnew+i,1+j:wnew+j,:)+1; + end +end +% Take average over overlapping predictions +O=O./C; +O=uint8(O); +imwrite(O,resfile); + +function net = load_net(config) + +net = caffe.Net(config.model,config.weights,'test'); +fprintf('\n\nModel: %s\nWeights: %s\n\n',config.model,config.weights); + +function config=create_config +config.model='./test.prototxt'; +config.weights='../../models/rbdn_denoising.caffemodel'; +config.branches=3; + +config.caffe_root = '../caffe'; +fprintf('initializing caffe..\n'); +addpath(fullfile(config.caffe_root, 'matlab')); +config.gpuNum=0; caffe.set_mode_gpu(); caffe.set_device(config.gpuNum); +% caffe.set_mode_cpu(); diff --git a/inference/denoising/test.prototxt b/inference/denoising/test.prototxt new file mode 100755 index 0000000..8ebdfc6 --- /dev/null +++ b/inference/denoising/test.prototxt @@ -0,0 +1,510 @@ +name: "base_dn_test" + +input: "data" +input_dim: 1 +input_dim: 1 +input_dim: 512 +input_dim: 512 + + +# H x W x 3 OR H x W x 1 + +# CONVOLUTION (conv1) +layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W x 64 + +# BN (bn1) +layer { name: "bn1" type: "BN" bottom: "conv1" top: "conv1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu1) +layer { name: "relu1" type: "ReLU" bottom: "conv1" top: "conv1" } + +# POOLING (pool1) +layer { + name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" top: "pool1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/2) x (W/2) X 64 + +# Introduce Branch 1 Here + +# CONVOLUTION (convB1_1) +layer { name: "convB1_1" type: "Convolution" bottom: "pool1" top: "convB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bnB1_1) +layer { name: "bnB1_1" type: "BN" bottom: "convB1_1" top: "convB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB1_1) +layer { name: "reluB1_1" type: "ReLU" bottom: "convB1_1" top: "convB1_1" } + +# POOLING (poolB1) +layer { + name: "poolB1" type: "Pooling" bottom: "convB1_1" top: "poolB1" top: "poolB1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/4) x (W/4) X 64 + +# Introduce Branch 2 Here + +# CONVOLUTION (convB2_1) +layer { name: "convB2_1" type: "Convolution" bottom: "poolB1" top: "convB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB2_1) +layer { name: "bnB2_1" type: "BN" bottom: "convB2_1" top: "convB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB2_1) +layer { name: "reluB2_1" type: "ReLU" bottom: "convB2_1" top: "convB2_1" } + +# POOLING (poolB2) +layer { + name: "poolB2" type: "Pooling" bottom: "convB2_1" top: "poolB2" top: "poolB2_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/8) x (W/8) X 64 + +# Introduce Branch 3 Here + +# CONVOLUTION (convB3_1) +layer { name: "convB3_1" type: "Convolution" bottom: "poolB2" top: "convB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB3_1) +layer { name: "bnB3_1" type: "BN" bottom: "convB3_1" top: "convB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB3_1) +layer { name: "reluB3_1" type: "ReLU" bottom: "convB3_1" top: "convB3_1" } + +# POOLING (poolB3) +layer { + name: "poolB3" type: "Pooling" bottom: "convB3_1" top: "poolB3" top: "poolB3_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/16) x (W/16) X 64 + +# CONVOLUTION (convB3_2) +layer { name: "convB3_2" type: "Convolution" bottom: "poolB3" top: "convB3_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB3_2) +layer { name: "bnB3_2" type: "BN" bottom: "convB3_2" top: "convB3_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB3_2) +layer { name: "reluB3_2" type: "ReLU" bottom: "convB3_2" top: "convB3_2" } + + +# UNPOOLING (unpoolB3) +layer { + name: "unpoolB3" type: "Unpooling" bottom: "convB3_2" bottom: "poolB3_mask" top: "unpoolB3" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/8) x (W/8) X 64 + +# DECONVOLUTION (deconvB3_1) +layer { name: "deconvB3_1" type: "Deconvolution" bottom: "unpoolB3" top: "deconvB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (debnB3_1) +layer { name: "debnB3_1" type: "BN" bottom: "deconvB3_1" top: "deconvB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB3_1) +layer { name: "dereluB3_1" type: "ReLU" bottom: "deconvB3_1" top: "deconvB3_1" } + + + +# Merge Branch 3 Here +# CONCAT (mergeB3) +layer { + name: "mergeB3" + bottom: "poolB2" + bottom: "deconvB3_1" + top: "mergeB3" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/8) x (W/8) X 128 + +# CONVOLUTION (convB2_2) +layer { name: "convB2_2" type: "Convolution" bottom: "mergeB3" top: "convB2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB2_2) +layer { name: "bnB2_2" type: "BN" bottom: "convB2_2" top: "convB2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB2_2) +layer { name: "reluB2_2" type: "ReLU" bottom: "convB2_2" top: "convB2_2" } + + +# UNPOOLING (unpoolB2) +layer { + name: "unpoolB2" type: "Unpooling" bottom: "convB2_2" bottom: "poolB2_mask" top: "unpoolB2" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/4) x (W/4) X 64 + +# DECONVOLUTION (deconvB2_1) +layer { name: "deconvB2_1" type: "Deconvolution" bottom: "unpoolB2" top: "deconvB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (debnB2_1) +layer { name: "debnB2_1" type: "BN" bottom: "deconvB2_1" top: "deconvB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB2_1) +layer { name: "dereluB2_1" type: "ReLU" bottom: "deconvB2_1" top: "deconvB2_1" } + + + +# Merge Branch 2 Here +# CONCAT (mergeB2) +layer { + name: "mergeB2" + bottom: "poolB1" + bottom: "deconvB2_1" + top: "mergeB2" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/4) x (W/4) X 128 + +# CONVOLUTION (convB1_2) +layer { name: "convB1_2" type: "Convolution" bottom: "mergeB2" top: "convB1_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB1_2) +layer { name: "bnB1_2" type: "BN" bottom: "convB1_2" top: "convB1_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB1_2) +layer { name: "reluB1_2" type: "ReLU" bottom: "convB1_2" top: "convB1_2" } + + +# UNPOOLING (unpoolB1) +layer { + name: "unpoolB1" type: "Unpooling" bottom: "convB1_2" bottom: "poolB1_mask" top: "unpoolB1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/2) x (W/2) X 64 + +# DECONVOLUTION (deconvB1_1) +layer { name: "deconvB1_1" type: "Deconvolution" bottom: "unpoolB1" top: "deconvB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (debnB1_1) +layer { name: "debnB1_1" type: "BN" bottom: "deconvB1_1" top: "deconvB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB1_1) +layer { name: "dereluB1_1" type: "ReLU" bottom: "deconvB1_1" top: "deconvB1_1" } + + +# Merge Branch 1 Here +# CONCAT (mergeB1) +layer { + name: "mergeB1" + bottom: "pool1" + bottom: "deconvB1_1" + top: "mergeB1" + type: "Concat" + concat_param { + axis: 1 + } +} + +# (H/2) x (W/2) X 128 + +# CONVOLUTION (conv2_1) +layer { name: "conv2_1" type: "Convolution" bottom: "mergeB1" top: "conv2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_1) +layer { name: "bn2_1" type: "BN" bottom: "conv2_1" top: "conv2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_1) +layer { name: "relu2_1" type: "ReLU" bottom: "conv2_1" top: "conv2_1" } + +# CONVOLUTION (conv2_2) +layer { name: "conv2_2" type: "Convolution" bottom: "conv2_1" top: "conv2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_2) +layer { name: "bn2_2" type: "BN" bottom: "conv2_2" top: "conv2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_2) +layer { name: "relu2_2" type: "ReLU" bottom: "conv2_2" top: "conv2_2" } + + +# CONVOLUTION (conv2_3) +layer { name: "conv2_3" type: "Convolution" bottom: "conv2_2" top: "conv2_3" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_3) +layer { name: "bn2_3" type: "BN" bottom: "conv2_3" top: "conv2_3" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_3) +layer { name: "relu2_3" type: "ReLU" bottom: "conv2_3" top: "conv2_3" } + + +# CONVOLUTION (conv2_4) +layer { name: "conv2_4" type: "Convolution" bottom: "conv2_3" top: "conv2_4" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_4) +layer { name: "bn2_4" type: "BN" bottom: "conv2_4" top: "conv2_4" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_4) +layer { name: "relu2_4" type: "ReLU" bottom: "conv2_4" top: "conv2_4" } + + +# CONVOLUTION (conv2_5) +layer { name: "conv2_5" type: "Convolution" bottom: "conv2_4" top: "conv2_5" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_5) +layer { name: "bn2_5" type: "BN" bottom: "conv2_5" top: "conv2_5" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_5) +layer { name: "relu2_5" type: "ReLU" bottom: "conv2_5" top: "conv2_5" } + + +# CONVOLUTION (conv2_6) +layer { name: "conv2_6" type: "Convolution" bottom: "conv2_5" top: "conv2_6" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_6) +layer { name: "bn2_6" type: "BN" bottom: "conv2_6" top: "conv2_6" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_6) +layer { name: "relu2_6" type: "ReLU" bottom: "conv2_6" top: "conv2_6" } + + +# CONVOLUTION (conv2_7) +layer { name: "conv2_7" type: "Convolution" bottom: "conv2_6" top: "conv2_7" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_7) +layer { name: "bn2_7" type: "BN" bottom: "conv2_7" top: "conv2_7" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_7) +layer { name: "relu2_7" type: "ReLU" bottom: "conv2_7" top: "conv2_7" } + + +# CONVOLUTION (conv2_8) +layer { name: "conv2_8" type: "Convolution" bottom: "conv2_7" top: "conv2_8" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_8) +layer { name: "bn2_8" type: "BN" bottom: "conv2_8" top: "conv2_8" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_8) +layer { name: "relu2_8" type: "ReLU" bottom: "conv2_8" top: "conv2_8" } + + +# CONVOLUTION (conv2_9) +layer { name: "conv2_9" type: "Convolution" bottom: "conv2_8" top: "conv2_9" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_9) +layer { name: "bn2_9" type: "BN" bottom: "conv2_9" top: "conv2_9" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_9) +layer { name: "relu2_9" type: "ReLU" bottom: "conv2_9" top: "conv2_9" } + + +# UNPOOLING (unpool1) +layer { + name: "unpool1" type: "Unpooling" bottom: "conv2_9" bottom: "pool1_mask" top: "unpool1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + + +# H x W x 64 + + + +# DECONVOLUTION (deconv1) +layer { name: "deconv1" type: "Deconvolution" bottom: "unpool1" top: "pred" + param { lr_mult: 0.1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 1 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W X 1 + diff --git a/inference/relighting/get_pred.m b/inference/relighting/get_pred.m new file mode 100755 index 0000000..71f526b --- /dev/null +++ b/inference/relighting/get_pred.m @@ -0,0 +1,61 @@ +function get_pred + +%-- +% Create network and data config +config=create_config; +%-- + +% Prepare imagelist +d=dir(sprintf('%s/*',config.datadir)); +d={d.name}; d=d(3:end); +config.imglist=d; +% Load RBDN network from config +config.net=load_net(config); +for ind=1:length(d) + fprintf('[%d/%d] %s\n',ind,length(d),d{ind}); + save_pred_oneimg(ind,config); +end +caffe.reset_all(); + +function save_pred_oneimg(ind,config) +net=config.net; +imgrelpath=config.imglist{ind}; +imgpath=sprintf('%s/%s',config.datadir,imgrelpath); +resfile=sprintf('%s/%s',config.resdir,strrep(imgrelpath,'.jpg','.png')); + +% Do forward pass with image +img=imread(imgpath); +% Force color if bw +if size(img,3)==1, img=repmat(img,[1,1,3]); end +img=imresize(img,[224,224]); +% Convert RGB to BGR +img=img(:,:,[3,2,1]); +% Convert H*W to W*H +img=permute(img,[2,1,3]); +cnn_input={single(img)}; +% Forward pass +pred=net.forward(cnn_input); +% Convert W*H to H*W +pred=permute(pred{1},[2,1,3]); +% Convert BGR to RGB +pred=uint8(pred(:,:,[3,2,1])); +imwrite(pred,resfile); + + +function net = load_net(config) + +model = './test.prototxt'; +weights = '../../models/rbdn_relighting.caffemodel'; +net = caffe.Net(model,weights,'test'); + +function config=create_config + +config.datadir = '../Data/relighting'; +config.resdir = './results'; +if ~isdir(config.resdir), mkdir(config.resdir); end; +config.caffe_root = '../caffe'; +fprintf('initializing caffe..\n'); +addpath(fullfile(config.caffe_root, 'matlab')); +config.gpuNum=0; caffe.set_mode_gpu(); caffe.set_device(config.gpuNum); +%caffe.set_mode_cpu(); +caffe.reset_all(); \ No newline at end of file diff --git a/inference/relighting/test.prototxt b/inference/relighting/test.prototxt new file mode 100755 index 0000000..e6ab18f --- /dev/null +++ b/inference/relighting/test.prototxt @@ -0,0 +1,508 @@ +name: "base_training" + +input: "data" +input_dim: 1 +input_dim: 3 +input_dim: 224 +input_dim: 224 + +# 224 x 224 x 3 + +# CONVOLUTION (conv1) +layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 224 x 224 x 64 + +# BN (bn1) +layer { name: "bn1" type: "BN" bottom: "conv1" top: "conv1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu1) +layer { name: "relu1" type: "ReLU" bottom: "conv1" top: "conv1" } + +# POOLING (pool1) +layer { + name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" top: "pool1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# 112 x 112 X 64 + +# Introduce Branch 1 Here + +# CONVOLUTION (convB1_1) +layer { name: "convB1_1" type: "Convolution" bottom: "pool1" top: "convB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (bnB1_1) +layer { name: "bnB1_1" type: "BN" bottom: "convB1_1" top: "convB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB1_1) +layer { name: "reluB1_1" type: "ReLU" bottom: "convB1_1" top: "convB1_1" } + +# POOLING (poolB1) +layer { + name: "poolB1" type: "Pooling" bottom: "convB1_1" top: "poolB1" top: "poolB1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# 56 x 56 X 64 + +# Introduce Branch 2 Here + +# CONVOLUTION (convB2_1) +layer { name: "convB2_1" type: "Convolution" bottom: "poolB1" top: "convB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 56 x 56 X 64 + +# BN (bnB2_1) +layer { name: "bnB2_1" type: "BN" bottom: "convB2_1" top: "convB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB2_1) +layer { name: "reluB2_1" type: "ReLU" bottom: "convB2_1" top: "convB2_1" } + +# POOLING (poolB2) +layer { + name: "poolB2" type: "Pooling" bottom: "convB2_1" top: "poolB2" top: "poolB2_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# 28 x 28 X 64 + +# Introduce Branch 3 Here + +# CONVOLUTION (convB3_1) +layer { name: "convB3_1" type: "Convolution" bottom: "poolB2" top: "convB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 28 x 28 X 64 + +# BN (bnB3_1) +layer { name: "bnB3_1" type: "BN" bottom: "convB3_1" top: "convB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB3_1) +layer { name: "reluB3_1" type: "ReLU" bottom: "convB3_1" top: "convB3_1" } + +# POOLING (poolB3) +layer { + name: "poolB3" type: "Pooling" bottom: "convB3_1" top: "poolB3" top: "poolB3_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# 14 x 14 X 64 + +# CONVOLUTION (convB3_2) +layer { name: "convB3_2" type: "Convolution" bottom: "poolB3" top: "convB3_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 14 x 14 X 64 + +# BN (bnB3_2) +layer { name: "bnB3_2" type: "BN" bottom: "convB3_2" top: "convB3_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB3_2) +layer { name: "reluB3_2" type: "ReLU" bottom: "convB3_2" top: "convB3_2" } + + +# UNPOOLING (unpoolB3) +layer { + name: "unpoolB3" type: "Unpooling" bottom: "convB3_2" bottom: "poolB3_mask" top: "unpoolB3" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# 28 x 28 X 64 + +# DECONVOLUTION (deconvB3_1) +layer { name: "deconvB3_1" type: "Deconvolution" bottom: "unpoolB3" top: "deconvB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 28 x 28 X 64 + +# BN (debnB3_1) +layer { name: "debnB3_1" type: "BN" bottom: "deconvB3_1" top: "deconvB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB3_1) +layer { name: "dereluB3_1" type: "ReLU" bottom: "deconvB3_1" top: "deconvB3_1" } + + + +# Merge Branch 3 Here +# CONCAT (mergeB3) +layer { + name: "mergeB3" + bottom: "poolB2" + bottom: "deconvB3_1" + top: "mergeB3" + type: "Concat" + concat_param { + axis: 1 + } +} +# 28 x 28 X 128 + +# CONVOLUTION (convB2_2) +layer { name: "convB2_2" type: "Convolution" bottom: "mergeB3" top: "convB2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 28 x 28 X 64 + +# BN (bnB2_2) +layer { name: "bnB2_2" type: "BN" bottom: "convB2_2" top: "convB2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB2_2) +layer { name: "reluB2_2" type: "ReLU" bottom: "convB2_2" top: "convB2_2" } + + +# UNPOOLING (unpoolB2) +layer { + name: "unpoolB2" type: "Unpooling" bottom: "convB2_2" bottom: "poolB2_mask" top: "unpoolB2" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# 56 x 56 X 64 + +# DECONVOLUTION (deconvB2_1) +layer { name: "deconvB2_1" type: "Deconvolution" bottom: "unpoolB2" top: "deconvB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 56 x 56 X 64 + +# BN (debnB2_1) +layer { name: "debnB2_1" type: "BN" bottom: "deconvB2_1" top: "deconvB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB2_1) +layer { name: "dereluB2_1" type: "ReLU" bottom: "deconvB2_1" top: "deconvB2_1" } + + + +# Merge Branch 2 Here +# CONCAT (mergeB2) +layer { + name: "mergeB2" + bottom: "poolB1" + bottom: "deconvB2_1" + top: "mergeB2" + type: "Concat" + concat_param { + axis: 1 + } +} +# 56 x 56 X 128 + +# CONVOLUTION (convB1_2) +layer { name: "convB1_2" type: "Convolution" bottom: "mergeB2" top: "convB1_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 56 x 56 X 64 + +# BN (bnB1_2) +layer { name: "bnB1_2" type: "BN" bottom: "convB1_2" top: "convB1_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB1_2) +layer { name: "reluB1_2" type: "ReLU" bottom: "convB1_2" top: "convB1_2" } + + +# UNPOOLING (unpoolB1) +layer { + name: "unpoolB1" type: "Unpooling" bottom: "convB1_2" bottom: "poolB1_mask" top: "unpoolB1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# 112 x 112 X 64 + +# DECONVOLUTION (deconvB1_1) +layer { name: "deconvB1_1" type: "Deconvolution" bottom: "unpoolB1" top: "deconvB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (debnB1_1) +layer { name: "debnB1_1" type: "BN" bottom: "deconvB1_1" top: "deconvB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB1_1) +layer { name: "dereluB1_1" type: "ReLU" bottom: "deconvB1_1" top: "deconvB1_1" } + + +# Merge Branch 1 Here +# CONCAT (mergeB1) +layer { + name: "mergeB1" + bottom: "pool1" + bottom: "deconvB1_1" + top: "mergeB1" + type: "Concat" + concat_param { + axis: 1 + } +} + +# 112 x 112 X 128 + +# CONVOLUTION (conv2_1) +layer { name: "conv2_1" type: "Convolution" bottom: "mergeB1" top: "conv2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (bn2_1) +layer { name: "bn2_1" type: "BN" bottom: "conv2_1" top: "conv2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_1) +layer { name: "relu2_1" type: "ReLU" bottom: "conv2_1" top: "conv2_1" } + +# CONVOLUTION (conv2_2) +layer { name: "conv2_2" type: "Convolution" bottom: "conv2_1" top: "conv2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (bn2_2) +layer { name: "bn2_2" type: "BN" bottom: "conv2_2" top: "conv2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_2) +layer { name: "relu2_2" type: "ReLU" bottom: "conv2_2" top: "conv2_2" } + + +# CONVOLUTION (conv2_3) +layer { name: "conv2_3" type: "Convolution" bottom: "conv2_2" top: "conv2_3" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (bn2_3) +layer { name: "bn2_3" type: "BN" bottom: "conv2_3" top: "conv2_3" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_3) +layer { name: "relu2_3" type: "ReLU" bottom: "conv2_3" top: "conv2_3" } + + +# CONVOLUTION (conv2_4) +layer { name: "conv2_4" type: "Convolution" bottom: "conv2_3" top: "conv2_4" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (bn2_4) +layer { name: "bn2_4" type: "BN" bottom: "conv2_4" top: "conv2_4" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_4) +layer { name: "relu2_4" type: "ReLU" bottom: "conv2_4" top: "conv2_4" } + + +# CONVOLUTION (conv2_5) +layer { name: "conv2_5" type: "Convolution" bottom: "conv2_4" top: "conv2_5" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (bn2_5) +layer { name: "bn2_5" type: "BN" bottom: "conv2_5" top: "conv2_5" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_5) +layer { name: "relu2_5" type: "ReLU" bottom: "conv2_5" top: "conv2_5" } + + +# CONVOLUTION (conv2_6) +layer { name: "conv2_6" type: "Convolution" bottom: "conv2_5" top: "conv2_6" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (bn2_6) +layer { name: "bn2_6" type: "BN" bottom: "conv2_6" top: "conv2_6" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_6) +layer { name: "relu2_6" type: "ReLU" bottom: "conv2_6" top: "conv2_6" } + + +# CONVOLUTION (conv2_7) +layer { name: "conv2_7" type: "Convolution" bottom: "conv2_6" top: "conv2_7" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (bn2_7) +layer { name: "bn2_7" type: "BN" bottom: "conv2_7" top: "conv2_7" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_7) +layer { name: "relu2_7" type: "ReLU" bottom: "conv2_7" top: "conv2_7" } + + +# CONVOLUTION (conv2_8) +layer { name: "conv2_8" type: "Convolution" bottom: "conv2_7" top: "conv2_8" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (bn2_8) +layer { name: "bn2_8" type: "BN" bottom: "conv2_8" top: "conv2_8" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_8) +layer { name: "relu2_8" type: "ReLU" bottom: "conv2_8" top: "conv2_8" } + + +# CONVOLUTION (conv2_9) +layer { name: "conv2_9" type: "Convolution" bottom: "conv2_8" top: "conv2_9" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 112 x 112 X 64 + +# BN (bn2_9) +layer { name: "bn2_9" type: "BN" bottom: "conv2_9" top: "conv2_9" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_9) +layer { name: "relu2_9" type: "ReLU" bottom: "conv2_9" top: "conv2_9" } + + +# UNPOOLING (unpool1) +layer { + name: "unpool1" type: "Unpooling" bottom: "conv2_9" bottom: "pool1_mask" top: "unpool1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + + +# 224 x 224 x 64 + +# DECONVOLUTION (deconv1) +layer { name: "deconv1" type: "Deconvolution" bottom: "unpool1" top: "pred" + param { lr_mult: 0.1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 3 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# 224 x 224 X 3 + + diff --git a/inference/run_matcaffe.sh b/inference/run_matcaffe.sh new file mode 100755 index 0000000..e4cb4b1 --- /dev/null +++ b/inference/run_matcaffe.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +# This will open an interactive matlab session after loading caffe dependencies. +# You can then navigate to any of the 3 experiment folders from within matlab: +# "relighting/denoising/colorization" and run the inference script "get_pred.m" +# Each experiment folder EXP has its own "get_pred.m", "test.prototxt" +# EXP/get_pred.m saves results in EXP/results (not tracked by git) + + +# LOAD CAFFE DEPENDENCIES TO LD_LIBRARY_PATH (this also sets env-variable MATLAB_DIR) +source ./caffe/load_caffe_dependencies.sh + +: "${MATLAB_DIR:?needs to be set in \"./caffe/load_caffe_dependencies.sh\". +This is the same path you specified in caffe\'s Makefile.config +Please see \"./caffe/load_caffe_dependencies.sh.example\" for reference.}" +echo $MATLAB_DIR + +# If you don't want an interactive matlab session, you can uncomment the line below +# and replace run_demo.m with the inference script of your choice. + +# ${MATLAB_DIR}/bin/matlab -nodisplay -nosplash -r "run('./run_demo.m'); exit" + +${MATLAB_DIR}/bin/matlab diff --git a/setup.sh b/setup.sh new file mode 100755 index 0000000..f34033e --- /dev/null +++ b/setup.sh @@ -0,0 +1,61 @@ +#!/bin/bash + +# This will fetch caffe, pretrained models, inference data and setup directory +# structure & symlinks for all of the training/inference scripts. +# (Nothing that is created here will be tracked by git) + +############################################################################## +# ---------------------- Fetch caffe for RBDN -------------------------------- +# (caffe folders will not be tracked by git) +echo "fetch caffe for RBDN" +mkdir -p caffe_rbdn && cd caffe_rbdn +git clone -b rbdn --single-branch https://github.com/venkai/caffe.git +# You should compile this caffe library with your own Makefile.config file) +cd .. +mkdir -p caffe_colorization && cd caffe_colorization +git clone -b colorization --single-branch https://github.com/venkai/caffe.git +# You should compile this caffe library with your own Makefile.config file) +cd .. + + +############################################################################## +# ----------------------- Set up inference data ------------------------------ +# (data folders will not be tracked by git) +echo "fetch inference data for denoising, relighting, colorization experiments" +mkdir -p data/training && cd data +wget --no-check-certificate http://www.umiacs.umd.edu/~venkai/rbdn/data/inference.tar.gz +tar xvzf inference.tar.gz && rm -f inference.tar.gz +cd .. + + +############################################################################## +# ----------------------- Set up pretrained models --------------------------- +# (models folder will not be tracked by git) +echo "fetch pretrained models for denoising, relighting, colorization experiments" +wget --no-check-certificate http://www.umiacs.umd.edu/~venkai/rbdn/models/models.tar.gz +tar xvzf models.tar.gz && rm -f models.tar.gz + + +############################################################################## +# ----------- Set up directory structure and symlinks ------------------------ +echo "set up directory structure and symlinks" +for F in ./inference/*/ +do + rm -rf ${F}results && mkdir -p ${F}results +done + +for F in ./training/*/ +do + rm -rf ${F}snapshot && mkdir -p ${F}snapshot + rm -rf ${F}training_log && mkdir -p ${F}training_log +done + +rm -f ./inference/caffe && ln -sf ../caffe_rbdn/caffe ./inference/caffe +rm -f ./training/caffe && ln -sf ../caffe_rbdn/caffe ./training/caffe +rm -f ./training/caffe_colorization && \ +ln -sf ../caffe_colorization/caffe ./training/caffe_colorization +rm -f ./inference/caffe_colorization && \ +ln -sf ../caffe_colorization/caffe ./inference/caffe_colorization + +rm -f ./inference/Data && ln -sf ../data/inference ./inference/Data +rm -f ./training/Data && ln -sf ../data/training ./training/Data diff --git a/training/colorization/add_wt_classab.py b/training/colorization/add_wt_classab.py new file mode 100755 index 0000000..70b8c70 --- /dev/null +++ b/training/colorization/add_wt_classab.py @@ -0,0 +1,46 @@ + +### User Configuration + +iteration = 1000 +gpu_id_num = 0 + +## path configuration +caffe_root = '../caffe_colorization' +script_path = '.' +caffe_model = script_path + '/test.prototxt' +caffe_weight = script_path + '/train_curr_inference.caffemodel' +caffe_inference_weight = script_path + '/train_curr_inference.caffemodel' + + +### start generate caffemodel + +print 'start generating BN-testable caffemodel' +print 'caffe_root: %s' % caffe_root +print 'script_path: %s' % script_path +print 'caffe_model: %s' % caffe_model +print 'caffe_weight: %s' % caffe_weight +print 'caffe_inference_weight: %s' % caffe_inference_weight + +import numpy as np + +import sys +sys.path.append(caffe_root+'/python') +import caffe +from caffe.proto import caffe_pb2 + +# caffe.set_mode_cpu() +caffe.set_mode_gpu() +caffe.set_device(gpu_id_num) +net = caffe.Net(caffe_model, caffe_weight, caffe.TEST) +pts_in_hull = np.load('./resources/pts_in_hull.npy') # load cluster centers +net.params['class_ab'][0].data[:,:,0,0] = pts_in_hull.transpose((1,0)) # populate cluster centers as 1x1 convolution kernel +print 'Annealed-Mean Parameters populated' +print net.params.keys() +print 'start saving model' + +net.save(caffe_inference_weight) + +print 'done' + + + diff --git a/training/colorization/make_bn.prototxt b/training/colorization/make_bn.prototxt new file mode 100755 index 0000000..2d6e756 --- /dev/null +++ b/training/colorization/make_bn.prototxt @@ -0,0 +1,698 @@ +name: "exp_colorization" + +layer { name: "data" type: "ImageRegressionData" top: "data_orig" + transform_param { mirror: true crop_size: 128 } + image_data_param { + source: "../Data/imgset/train.txt" + root_folder: "../Data" + shuffle: true batch_size: 32 is_color: true label_type: NONE + } +} + +# H x W x 3 + + +# PYTHON LAYERS for BGR2Yab and soft encoding of GT + +# **************************** +# ***** Color Conversion ***** +# **************************** +layer { type: 'Python' name: 'img_lab' bottom: "data_orig" + # image in Lab space + top: "data_orig" + # top: "img_lab" + python_param { module: 'caffe_traininglayers' layer: 'BGR2LabLayer' } +} +# Split Y and ab +layer { name: "img_slice" type: "Slice" bottom: "data_orig" + # [0,100] + top: "data" + # [-110,110] + top: "data_ab" + propagate_down: false slice_param { axis: 1 slice_point: 1 } +} +# 0-center data channel +layer { name: "data_l_meansub" type: "Scale" bottom: "data" + # [-50,50] + top: "data" propagate_down: false + param {lr_mult: 0 decay_mult: 0} param {lr_mult: 0 decay_mult: 0} + scale_param { bias_term: true filler { type: 'constant' value: 1 } + bias_filler { type: 'constant' value: -50 } } +} + +# **************************** +# ***** PROCESS LABELS ******* +# **************************** + +# subsample ab +layer { name: 'data_ab_ss' type: 'Convolution' bottom: "data_ab" + # subsampled colors + top: "data_ab_ss" + param {lr_mult: 0 decay_mult: 0} param {lr_mult: 0 decay_mult: 0} + convolution_param { num_output: 2 kernel_size: 1 stride: 4 group: 2 + weight_filler { type: 'constant' value: 1 } } +} + +# encode +layer { type: 'Python' name: 'ab_enc' bottom: "data_ab_ss" + # quantized gt colors + top: "gt_ab_313" + python_param { module: 'caffe_traininglayers' layer: 'NNEncLayer' } +} +# compute gray mask +layer { type: 'Python' name: 'nongray_mask' bottom: "data_ab_ss" + # mask out grayscale images + top: "nongray_mask" + python_param { module: 'caffe_traininglayers' layer: 'NonGrayMaskLayer' } +} +# compute prior boost +layer { type: 'Python' name: 'prior_boost' bottom: "gt_ab_313" + # gradient boosting factors + top: "prior_boost" + python_param { module: 'caffe_traininglayers' layer: 'PriorBoostLayer' } +} +# multiply nongray mask and prior boost +layer { type: 'Eltwise' name: 'prior_boost_nongray' bottom: "prior_boost" + bottom: "nongray_mask" top: "prior_boost_nongray" eltwise_param { operation: 0 } +} + +#-------- + +# H x W x 1 + +# CONVOLUTION (conv1) +layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W x 64 + +# BN (bn1) +layer { name: "bn1" type: "BN" bottom: "conv1" top: "conv1" + top: "bn1-mean" top: "bn1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu1) +layer { name: "relu1" type: "ReLU" bottom: "conv1" top: "conv1" } + +# POOLING (pool1) +layer { + name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/2) x (W/2) X 64 + +# Introduce Branch 1 Here + +# CONVOLUTION (convB1_1) +layer { name: "convB1_1" type: "Convolution" bottom: "pool1" top: "convB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bnB1_1) +layer { name: "bnB1_1" type: "BN" bottom: "convB1_1" top: "convB1_1" + top: "bnB1_1-mean" top: "bnB1_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_1) +layer { name: "reluB1_1" type: "ReLU" bottom: "convB1_1" top: "convB1_1" } + +# POOLING (poolB1) +layer { + name: "poolB1" type: "Pooling" bottom: "convB1_1" top: "poolB1" top: "poolB1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/4) x (W/4) X 64 + +# Introduce Branch 2 Here + +# CONVOLUTION (convB2_1) +layer { name: "convB2_1" type: "Convolution" bottom: "poolB1" top: "convB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB2_1) +layer { name: "bnB2_1" type: "BN" bottom: "convB2_1" top: "convB2_1" + top: "bnB2_1-mean" top: "bnB2_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_1) +layer { name: "reluB2_1" type: "ReLU" bottom: "convB2_1" top: "convB2_1" } + +# POOLING (poolB2) +layer { + name: "poolB2" type: "Pooling" bottom: "convB2_1" top: "poolB2" top: "poolB2_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/8) x (W/8) X 64 + +# Introduce Branch 3 Here + +# CONVOLUTION (convB3_1) +layer { name: "convB3_1" type: "Convolution" bottom: "poolB2" top: "convB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB3_1) +layer { name: "bnB3_1" type: "BN" bottom: "convB3_1" top: "convB3_1" + top: "bnB3_1-mean" top: "bnB3_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_1) +layer { name: "reluB3_1" type: "ReLU" bottom: "convB3_1" top: "convB3_1" } + +# POOLING (poolB3) +layer { + name: "poolB3" type: "Pooling" bottom: "convB3_1" top: "poolB3" top: "poolB3_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/16) x (W/16) X 64 + + +# Introduce Branch 4 Here + +# CONVOLUTION (convB4_1) +layer { name: "convB4_1" type: "Convolution" bottom: "poolB3" top: "convB4_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB4_1) +layer { name: "bnB4_1" type: "BN" bottom: "convB4_1" top: "convB4_1" + top: "bnB4_1-mean" top: "bnB4_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB4_1) +layer { name: "reluB4_1" type: "ReLU" bottom: "convB4_1" top: "convB4_1" } + +# POOLING (poolB4) +layer { + name: "poolB4" type: "Pooling" bottom: "convB4_1" top: "poolB4" top: "poolB4_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/32) x (W/32) X 64 + +# CONVOLUTION (convB4_2) +layer { name: "convB4_2" type: "Convolution" bottom: "poolB4" top: "convB4_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/32) x (W/32) X 64 + +# BN (bnB4_2) +layer { name: "bnB4_2" type: "BN" bottom: "convB4_2" top: "convB4_2" + top: "bnB4_2-mean" top: "bnB4_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB4_2) +layer { name: "reluB4_2" type: "ReLU" bottom: "convB4_2" top: "convB4_2" } + + +# UNPOOLING (unpoolB4) +layer { + name: "unpoolB4" type: "Unpooling" bottom: "convB4_2" bottom: "poolB4_mask" top: "unpoolB4" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/16) x (W/16) X 64 + +# DECONVOLUTION (deconvB4_1) +layer { name: "deconvB4_1" type: "Deconvolution" bottom: "unpoolB4" top: "deconvB4_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (debnB4_1) +layer { name: "debnB4_1" type: "BN" bottom: "deconvB4_1" top: "deconvB4_1" + top: "debnB4_1-mean" top: "debnB4_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB4_1) +layer { name: "dereluB4_1" type: "ReLU" bottom: "deconvB4_1" top: "deconvB4_1" } + + + +# Merge Branch 4 Here +# CONCAT (mergeB4) +layer { + name: "mergeB4" + bottom: "poolB3" + bottom: "deconvB4_1" + top: "mergeB4" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/16) x (W/16) X 128 + + +# CONVOLUTION (convB3_2) +layer { name: "convB3_2" type: "Convolution" bottom: "mergeB4" top: "convB3_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB3_2) +layer { name: "bnB3_2" type: "BN" bottom: "convB3_2" top: "convB3_2" + top: "bnB3_2-mean" top: "bnB3_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_2) +layer { name: "reluB3_2" type: "ReLU" bottom: "convB3_2" top: "convB3_2" } + + +# UNPOOLING (unpoolB3) +layer { + name: "unpoolB3" type: "Unpooling" bottom: "convB3_2" bottom: "poolB3_mask" top: "unpoolB3" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/8) x (W/8) X 64 + +# DECONVOLUTION (deconvB3_1) +layer { name: "deconvB3_1" type: "Deconvolution" bottom: "unpoolB3" top: "deconvB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (debnB3_1) +layer { name: "debnB3_1" type: "BN" bottom: "deconvB3_1" top: "deconvB3_1" + top: "debnB3_1-mean" top: "debnB3_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB3_1) +layer { name: "dereluB3_1" type: "ReLU" bottom: "deconvB3_1" top: "deconvB3_1" } + + + +# Merge Branch 3 Here +# CONCAT (mergeB3) +layer { + name: "mergeB3" + bottom: "poolB2" + bottom: "deconvB3_1" + top: "mergeB3" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/8) x (W/8) X 128 + +# CONVOLUTION (convB2_2) +layer { name: "convB2_2" type: "Convolution" bottom: "mergeB3" top: "convB2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB2_2) +layer { name: "bnB2_2" type: "BN" bottom: "convB2_2" top: "convB2_2" + top: "bnB2_2-mean" top: "bnB2_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_2) +layer { name: "reluB2_2" type: "ReLU" bottom: "convB2_2" top: "convB2_2" } + + +# UNPOOLING (unpoolB2) +layer { + name: "unpoolB2" type: "Unpooling" bottom: "convB2_2" bottom: "poolB2_mask" top: "unpoolB2" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/4) x (W/4) X 64 + +# DECONVOLUTION (deconvB2_1) +layer { name: "deconvB2_1" type: "Deconvolution" bottom: "unpoolB2" top: "deconvB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (debnB2_1) +layer { name: "debnB2_1" type: "BN" bottom: "deconvB2_1" top: "deconvB2_1" + top: "debnB2_1-mean" top: "debnB2_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB2_1) +layer { name: "dereluB2_1" type: "ReLU" bottom: "deconvB2_1" top: "deconvB2_1" } + + + +# Merge Branch 2 Here +# CONCAT (mergeB2) +layer { + name: "mergeB2" + bottom: "poolB1" + bottom: "deconvB2_1" + top: "mergeB2" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/4) x (W/4) X 128 + +# CONVOLUTION (convB1_2) +layer { name: "convB1_2" type: "Convolution" bottom: "mergeB2" top: "convB1_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB1_2) +layer { name: "bnB1_2" type: "BN" bottom: "convB1_2" top: "convB1_2" + top: "bnB1_2-mean" top: "bnB1_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_2) +layer { name: "reluB1_2" type: "ReLU" bottom: "convB1_2" top: "convB1_2" } + + +# UNPOOLING (unpoolB1) +layer { + name: "unpoolB1" type: "Unpooling" bottom: "convB1_2" bottom: "poolB1_mask" top: "unpoolB1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/2) x (W/2) X 64 + +# DECONVOLUTION (deconvB1_1) +layer { name: "deconvB1_1" type: "Deconvolution" bottom: "unpoolB1" top: "deconvB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (debnB1_1) +layer { name: "debnB1_1" type: "BN" bottom: "deconvB1_1" top: "deconvB1_1" + top: "debnB1_1-mean" top: "debnB1_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB1_1) +layer { name: "dereluB1_1" type: "ReLU" bottom: "deconvB1_1" top: "deconvB1_1" } + + +# Merge Branch 1 Here +# CONCAT (mergeB1) +layer { + name: "mergeB1" + bottom: "pool1" + bottom: "deconvB1_1" + top: "mergeB1" + type: "Concat" + concat_param { + axis: 1 + } +} + +# (H/2) x (W/2) X 128 + +# CONVOLUTION (conv2_1) +layer { name: "conv2_1" type: "Convolution" bottom: "mergeB1" top: "conv2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_1) +layer { name: "bn2_1" type: "BN" bottom: "conv2_1" top: "conv2_1" + top: "bn2_1-mean" top: "bn2_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_1) +layer { name: "relu2_1" type: "ReLU" bottom: "conv2_1" top: "conv2_1" } + +# CONVOLUTION (conv2_2) +layer { name: "conv2_2" type: "Convolution" bottom: "conv2_1" top: "conv2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_2) +layer { name: "bn2_2" type: "BN" bottom: "conv2_2" top: "conv2_2" + top: "bn2_2-mean" top: "bn2_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_2) +layer { name: "relu2_2" type: "ReLU" bottom: "conv2_2" top: "conv2_2" } + + +# CONVOLUTION (conv2_3) +layer { name: "conv2_3" type: "Convolution" bottom: "conv2_2" top: "conv2_3" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_3) +layer { name: "bn2_3" type: "BN" bottom: "conv2_3" top: "conv2_3" + top: "bn2_3-mean" top: "bn2_3-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_3) +layer { name: "relu2_3" type: "ReLU" bottom: "conv2_3" top: "conv2_3" } + + +# CONVOLUTION (conv2_4) +layer { name: "conv2_4" type: "Convolution" bottom: "conv2_3" top: "conv2_4" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_4) +layer { name: "bn2_4" type: "BN" bottom: "conv2_4" top: "conv2_4" + top: "bn2_4-mean" top: "bn2_4-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_4) +layer { name: "relu2_4" type: "ReLU" bottom: "conv2_4" top: "conv2_4" } + + +# CONVOLUTION (conv2_5) +layer { name: "conv2_5" type: "Convolution" bottom: "conv2_4" top: "conv2_5" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_5) +layer { name: "bn2_5" type: "BN" bottom: "conv2_5" top: "conv2_5" + top: "bn2_5-mean" top: "bn2_5-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_5) +layer { name: "relu2_5" type: "ReLU" bottom: "conv2_5" top: "conv2_5" } + + +# CONVOLUTION (conv2_6) +layer { name: "conv2_6" type: "Convolution" bottom: "conv2_5" top: "conv2_6" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_6) +layer { name: "bn2_6" type: "BN" bottom: "conv2_6" top: "conv2_6" + top: "bn2_6-mean" top: "bn2_6-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_6) +layer { name: "relu2_6" type: "ReLU" bottom: "conv2_6" top: "conv2_6" } + + +# CONVOLUTION (conv2_7) +layer { name: "conv2_7" type: "Convolution" bottom: "conv2_6" top: "conv2_7" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_7) +layer { name: "bn2_7" type: "BN" bottom: "conv2_7" top: "conv2_7" + top: "bn2_7-mean" top: "bn2_7-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_7) +layer { name: "relu2_7" type: "ReLU" bottom: "conv2_7" top: "conv2_7" } + + +# CONVOLUTION (conv2_8) +layer { name: "conv2_8" type: "Convolution" bottom: "conv2_7" top: "conv2_8" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_8) +layer { name: "bn2_8" type: "BN" bottom: "conv2_8" top: "conv2_8" + top: "bn2_8-mean" top: "bn2_8-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_8) +layer { name: "relu2_8" type: "ReLU" bottom: "conv2_8" top: "conv2_8" } + + +# CONVOLUTION (conv2_9) +layer { name: "conv2_9" type: "Convolution" bottom: "conv2_8" top: "conv2_9" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_9) +layer { name: "bn2_9" type: "BN" bottom: "conv2_9" top: "conv2_9" + top: "bn2_9-mean" top: "bn2_9-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_9) +layer { name: "relu2_9" type: "ReLU" bottom: "conv2_9" top: "conv2_9" } + + +# H x W x 64 + +# DECONVOLUTION (deconv1) +layer { name: "deconv1" type: "Deconvolution" bottom: "conv2_9" top: "deconv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 2 kernel_size: 5 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# BN (debn1) +layer { name: "debn1" type: "BN" bottom: "deconv1" top: "deconv1" + top: "debn1-mean" top: "debn1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (derelu1) +layer { name: "derelu1" type: "ReLU" bottom: "deconv1" top: "deconv1" } + +# H x W x 64 + +# **************************** +# ***** Unary prediction ***** +# **************************** + +# CONVOLUTION (conv_313) +layer { name: "conv_313" type: "Convolution" bottom: "deconv1" top: "conv_313" + param { lr_mult: 0.1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 313 pad: 1 kernel_size: 4 stride: 2 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W X 313 + +# *************************** +# ***** Boosting priors ***** +# *************************** +layer { name: "conv_313_boost" type: "Python" bottom: "conv_313" + bottom: "prior_boost_nongray" top: "conv_313" + python_param { module: 'caffe_traininglayers' layer: 'ClassRebalanceMultLayer' } +} + +# H x W X 313 \ No newline at end of file diff --git a/training/colorization/make_bn.py b/training/colorization/make_bn.py new file mode 100755 index 0000000..df0aa86 --- /dev/null +++ b/training/colorization/make_bn.py @@ -0,0 +1,200 @@ + +### User Configuration + +iteration = 1000 +gpu_id_num = 0 + +## path configuration +caffe_root = '../caffe_colorization' +script_path = '.' +caffe_model = script_path + '/make_bn.prototxt' +caffe_weight = script_path + '/train_curr.caffemodel' +caffe_inference_weight = script_path + '/train_curr_inference.caffemodel' + +## modify this definition according to model definition +bn_blobs= ['conv1', + 'convB1_1', + 'convB2_1', + 'convB3_1', + 'convB4_1', + 'convB4_2', + 'deconvB4_1', + 'convB3_2', + 'deconvB3_1', + 'convB2_2', + 'deconvB2_1', + 'convB1_2', + 'deconvB1_1', + 'conv2_1', + 'conv2_2', + 'conv2_3', + 'conv2_4', + 'conv2_5', + 'conv2_6', + 'conv2_7', + 'conv2_8', + 'conv2_9', + 'deconv1'] +bn_layers=['bn1', + 'bnB1_1', + 'bnB2_1', + 'bnB3_1', + 'bnB4_1', + 'bnB4_2', + 'debnB4_1', + 'bnB3_2', + 'debnB3_1', + 'bnB2_2', + 'debnB2_1', + 'bnB1_2', + 'debnB1_1', + 'bn2_1', + 'bn2_2', + 'bn2_3', + 'bn2_4', + 'bn2_5', + 'bn2_6', + 'bn2_7', + 'bn2_8', + 'bn2_9', + 'debn1'] +bn_means= ['bn1-mean', + 'bnB1_1-mean', + 'bnB2_1-mean', + 'bnB3_1-mean', + 'bnB4_1-mean', + 'bnB4_2-mean', + 'debnB4_1-mean', + 'bnB3_2-mean', + 'debnB3_1-mean', + 'bnB2_2-mean', + 'debnB2_1-mean', + 'bnB1_2-mean', + 'debnB1_1-mean', + 'bn2_1-mean', + 'bn2_2-mean', + 'bn2_3-mean', + 'bn2_4-mean', + 'bn2_5-mean', + 'bn2_6-mean', + 'bn2_7-mean', + 'bn2_8-mean', + 'bn2_9-mean', + 'debn1-mean'] +bn_vars = ['bn1-var', + 'bnB1_1-var', + 'bnB2_1-var', + 'bnB3_1-var', + 'bnB4_1-var', + 'bnB4_2-var', + 'debnB4_1-var', + 'bnB3_2-var', + 'debnB3_1-var', + 'bnB2_2-var', + 'debnB2_1-var', + 'bnB1_2-var', + 'debnB1_1-var', + 'bn2_1-var', + 'bn2_2-var', + 'bn2_3-var', + 'bn2_4-var', + 'bn2_5-var', + 'bn2_6-var', + 'bn2_7-var', + 'bn2_8-var', + 'bn2_9-var', + 'debn1-var'] + + + +### start generate caffemodel + +print 'start generating BN-testable caffemodel' +print 'caffe_root: %s' % caffe_root +print 'script_path: %s' % script_path +print 'caffe_model: %s' % caffe_model +print 'caffe_weight: %s' % caffe_weight +print 'caffe_inference_weight: %s' % caffe_inference_weight + +import numpy as np + +import sys +sys.path.append(caffe_root+'/python') +import caffe +from caffe.proto import caffe_pb2 + +# caffe.set_mode_cpu() +caffe.set_mode_gpu() +caffe.set_device(gpu_id_num) +net = caffe.Net(caffe_model, caffe_weight, caffe.TEST) + +def forward_once(net): + start_ind = 0 + end_ind = len(net.layers) - 1 + net._forward(start_ind, end_ind) + # print net.outputs + return {out: net.blobs[out].data for out in net.outputs} + +print net.params.keys() + +res = forward_once(net) + +bn_avg_mean = {bn_mean: np.squeeze(res[bn_mean]).copy() for bn_mean in bn_means} +bn_avg_var = {bn_var: np.squeeze(res[bn_var]).copy() for bn_var in bn_vars} + +cnt = 1 + +for i in range(0, iteration): + res = forward_once(net) + for bn_mean in bn_means: + bn_avg_mean[bn_mean] = bn_avg_mean[bn_mean] + np.squeeze(res[bn_mean]) + for bn_var in bn_vars: + bn_avg_var[bn_var] = bn_avg_var[bn_var] + np.squeeze(res[bn_var]) + + cnt += 1 + print 'progress: %d/%d' % (i, iteration) + +## compute average +for bn_mean in bn_means: + bn_avg_mean[bn_mean] /= cnt +for bn_var in bn_vars: + bn_avg_var[bn_var] /= cnt + +for i in range(0, len(bn_vars)): + m = np.prod(net.blobs[bn_blobs[i]].data.shape) / np.prod(bn_avg_var[bn_vars[i]].shape) + bn_avg_var[bn_vars[i]] *= (m/(m-1)) + +scale_data = {bn_layer: np.squeeze(net.params[bn_layer][0].data) for bn_layer in bn_layers} +shift_data = {bn_layer: np.squeeze(net.params[bn_layer][1].data) for bn_layer in bn_layers} + +var_eps = 1e-9 + +new_scale_data = {} +new_shift_data = {} +for i in range(0, len(bn_layers)): + gamma = scale_data[bn_layers[i]] + beta = shift_data[bn_layers[i]] + Ex = bn_avg_mean[bn_means[i]] + Varx = bn_avg_var[bn_vars[i]] + new_gamma = gamma / np.sqrt(Varx + var_eps) + new_beta = beta - (gamma * Ex / np.sqrt(Varx + var_eps)) + + new_scale_data[bn_layers[i]] = new_gamma + new_shift_data[bn_layers[i]] = new_beta + +print new_scale_data.keys() +print new_shift_data.keys() + +## assign computed new scale and shift values to net.params +for i in range(0, len(bn_layers)): + net.params[bn_layers[i]][0].data[...] = new_scale_data[bn_layers[i]].reshape(net.params[bn_layers[i]][0].data.shape) + net.params[bn_layers[i]][1].data[...] = new_shift_data[bn_layers[i]].reshape(net.params[bn_layers[i]][1].data.shape) + +print 'start saving model' + +net.save(caffe_inference_weight) + +print 'done' + + + diff --git a/training/colorization/run_bn.sh b/training/colorization/run_bn.sh new file mode 100755 index 0000000..d02e485 --- /dev/null +++ b/training/colorization/run_bn.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +# Pull most recent snapshot and prepare it for inference by computing global +# mean/variance for all the batch-normalization layers by passing training +# data through the network. If the most recent snapshot is in +# ./snapshot/trn_iter_[N].caffemodel, then the final inference-ready weights +# will be saved in ./tst_[N].caffemodel +# Training Data is expected in ../Data (can be a symlink) +# ImageList is expected in ../Data/imgset/train.txt +# (relative to root ../Data) + +# Choose most recent snapshot for inference +CAFFEDIR="../caffe_colorization" +CURR_SNAPSHOT=`ls ./snapshot/trn_iter_*.caffemodel -t | head -n 1` +F=${CURR_SNAPSHOT/"./snapshot/trn_iter_"/} +ITER=${F/".caffemodel"/} +ITER=${ITER/"*"/} +P_ITER=`printf "%08g\n" ${ITER}` + +WEIGHTSA_TMP=./train_curr.caffemodel +WEIGHTSA_INF_TMP=./train_curr_inference.caffemodel +WEIGHTSA_INF=./tst_${P_ITER}.caffemodel + +echo $WEIGHTSA +echo $WEIGHTSA_INF + +# LOAD CAFFE DEPENDENCIES TO LD_LIBRARY_PATH +source ${CAFFEDIR}/load_caffe_dependencies.sh + +rm -f $WEIGHTSA_TMP && cp -f $CURR_SNAPSHOT $WEIGHTSA_TMP + +# ADJUST BATCH NORMALIZATION FOR INFERENCE +export PYTHONPATH=${CAFFEDIR}/resources:${PYTHONPATH} +python make_bn.py +python add_wt_classab.py + +rm -f ${WEIGHTSA_TMP} +mv ${WEIGHTSA_INF_TMP} ${WEIGHTSA_INF} diff --git a/training/colorization/solver.prototxt b/training/colorization/solver.prototxt new file mode 100755 index 0000000..9b611b7 --- /dev/null +++ b/training/colorization/solver.prototxt @@ -0,0 +1,15 @@ +net: "./train.prototxt" +base_lr: 3.16e-3 +lr_policy: "step" +gamma: 0.316 +stepsize: 45000 +display: 10 +max_iter: 184000 +momentum: 0.9 +momentum2: 0.99 +weight_decay: 0.001 +snapshot: 1000 +snapshot_prefix: "./snapshot/trn" +solver_mode: GPU +solver_type: ADAM +average_loss: 1000 diff --git a/training/colorization/start_train.sh b/training/colorization/start_train.sh new file mode 100755 index 0000000..b0c6c9f --- /dev/null +++ b/training/colorization/start_train.sh @@ -0,0 +1,33 @@ +#!/bin/bash + +# Start training an RBDN model for colorization. You can pause training at any moment +# with Ctrl+C and most recent snapshot will be saved in +# ./snapshot/trn_iter_[*].solverstate +# Running ./start_train.sh again will automatically resume from that snapshot. +# Training Data is expected in ../Data (can be a symlink) +# ImageList is expected in ../Data/imgset/train.txt +# (relative to root ../Data) + +CAFFEDIR="../caffe_colorization" +LOGDIR=./training_log +CAFFE=${CAFFEDIR}/build/tools/caffe +SOLVER=./solver.prototxt + +# LOAD CAFFE DEPENDENCIES TO LD_LIBRARY_PATH +source ${CAFFEDIR}/load_caffe_dependencies.sh + +mkdir -p $LOGDIR +mkdir -p snapshot +rm -f resources && ln -sf ${CAFFEDIR}/resources + +export PYTHONPATH=${CAFFEDIR}/resources:${PYTHONPATH} + +# Check if snapshot exists; if not train from scratch. +if [ "$(ls -A ./snapshot/)" ]; then + CURR_SNAPSHOT=`ls ./snapshot/*solverstate -t | head -n 1` + echo "Resuming training from snapshot ${CURR_SNAPSHOT}" + echo "If you want to train from scratch, delete everything in ./snapshot" + GLOG_log_dir=$LOGDIR $CAFFE train -snapshot $CURR_SNAPSHOT -solver $SOLVER -gpu 0 +else + GLOG_log_dir=$LOGDIR $CAFFE train -solver $SOLVER -gpu 0 +fi \ No newline at end of file diff --git a/training/colorization/test.prototxt b/training/colorization/test.prototxt new file mode 100755 index 0000000..f711797 --- /dev/null +++ b/training/colorization/test.prototxt @@ -0,0 +1,650 @@ +name: "exp_colorization" + +input: "data" +input_dim: 1 +input_dim: 1 +input_dim: 256 +input_dim: 256 + + +# H x W x 1 + +# CONVOLUTION (conv1) +layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W x 64 + +# BN (bn1) +layer { name: "bn1" type: "BN" bottom: "conv1" top: "conv1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu1) +layer { name: "relu1" type: "ReLU" bottom: "conv1" top: "conv1" } + +# POOLING (pool1) +layer { + name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/2) x (W/2) X 64 + +# Introduce Branch 1 Here + +# CONVOLUTION (convB1_1) +layer { name: "convB1_1" type: "Convolution" bottom: "pool1" top: "convB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bnB1_1) +layer { name: "bnB1_1" type: "BN" bottom: "convB1_1" top: "convB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB1_1) +layer { name: "reluB1_1" type: "ReLU" bottom: "convB1_1" top: "convB1_1" } + +# POOLING (poolB1) +layer { + name: "poolB1" type: "Pooling" bottom: "convB1_1" top: "poolB1" top: "poolB1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/4) x (W/4) X 64 + +# Introduce Branch 2 Here + +# CONVOLUTION (convB2_1) +layer { name: "convB2_1" type: "Convolution" bottom: "poolB1" top: "convB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB2_1) +layer { name: "bnB2_1" type: "BN" bottom: "convB2_1" top: "convB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB2_1) +layer { name: "reluB2_1" type: "ReLU" bottom: "convB2_1" top: "convB2_1" } + +# POOLING (poolB2) +layer { + name: "poolB2" type: "Pooling" bottom: "convB2_1" top: "poolB2" top: "poolB2_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/8) x (W/8) X 64 + +# Introduce Branch 3 Here + +# CONVOLUTION (convB3_1) +layer { name: "convB3_1" type: "Convolution" bottom: "poolB2" top: "convB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB3_1) +layer { name: "bnB3_1" type: "BN" bottom: "convB3_1" top: "convB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB3_1) +layer { name: "reluB3_1" type: "ReLU" bottom: "convB3_1" top: "convB3_1" } + +# POOLING (poolB3) +layer { + name: "poolB3" type: "Pooling" bottom: "convB3_1" top: "poolB3" top: "poolB3_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/16) x (W/16) X 64 + + +# Introduce Branch 4 Here + +# CONVOLUTION (convB4_1) +layer { name: "convB4_1" type: "Convolution" bottom: "poolB3" top: "convB4_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB4_1) +layer { name: "bnB4_1" type: "BN" bottom: "convB4_1" top: "convB4_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB4_1) +layer { name: "reluB4_1" type: "ReLU" bottom: "convB4_1" top: "convB4_1" } + +# POOLING (poolB4) +layer { + name: "poolB4" type: "Pooling" bottom: "convB4_1" top: "poolB4" top: "poolB4_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/32) x (W/32) X 64 + +# CONVOLUTION (convB4_2) +layer { name: "convB4_2" type: "Convolution" bottom: "poolB4" top: "convB4_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/32) x (W/32) X 64 + +# BN (bnB4_2) +layer { name: "bnB4_2" type: "BN" bottom: "convB4_2" top: "convB4_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB4_2) +layer { name: "reluB4_2" type: "ReLU" bottom: "convB4_2" top: "convB4_2" } + + +# UNPOOLING (unpoolB4) +layer { + name: "unpoolB4" type: "Unpooling" bottom: "convB4_2" bottom: "poolB4_mask" top: "unpoolB4" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/16) x (W/16) X 64 + +# DECONVOLUTION (deconvB4_1) +layer { name: "deconvB4_1" type: "Deconvolution" bottom: "unpoolB4" top: "deconvB4_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (debnB4_1) +layer { name: "debnB4_1" type: "BN" bottom: "deconvB4_1" top: "deconvB4_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB4_1) +layer { name: "dereluB4_1" type: "ReLU" bottom: "deconvB4_1" top: "deconvB4_1" } + + + +# Merge Branch 4 Here +# CONCAT (mergeB4) +layer { + name: "mergeB4" + bottom: "poolB3" + bottom: "deconvB4_1" + top: "mergeB4" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/16) x (W/16) X 128 + + +# CONVOLUTION (convB3_2) +layer { name: "convB3_2" type: "Convolution" bottom: "mergeB4" top: "convB3_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB3_2) +layer { name: "bnB3_2" type: "BN" bottom: "convB3_2" top: "convB3_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB3_2) +layer { name: "reluB3_2" type: "ReLU" bottom: "convB3_2" top: "convB3_2" } + + +# UNPOOLING (unpoolB3) +layer { + name: "unpoolB3" type: "Unpooling" bottom: "convB3_2" bottom: "poolB3_mask" top: "unpoolB3" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/8) x (W/8) X 64 + +# DECONVOLUTION (deconvB3_1) +layer { name: "deconvB3_1" type: "Deconvolution" bottom: "unpoolB3" top: "deconvB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (debnB3_1) +layer { name: "debnB3_1" type: "BN" bottom: "deconvB3_1" top: "deconvB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB3_1) +layer { name: "dereluB3_1" type: "ReLU" bottom: "deconvB3_1" top: "deconvB3_1" } + + + +# Merge Branch 3 Here +# CONCAT (mergeB3) +layer { + name: "mergeB3" + bottom: "poolB2" + bottom: "deconvB3_1" + top: "mergeB3" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/8) x (W/8) X 128 + +# CONVOLUTION (convB2_2) +layer { name: "convB2_2" type: "Convolution" bottom: "mergeB3" top: "convB2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB2_2) +layer { name: "bnB2_2" type: "BN" bottom: "convB2_2" top: "convB2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB2_2) +layer { name: "reluB2_2" type: "ReLU" bottom: "convB2_2" top: "convB2_2" } + + +# UNPOOLING (unpoolB2) +layer { + name: "unpoolB2" type: "Unpooling" bottom: "convB2_2" bottom: "poolB2_mask" top: "unpoolB2" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/4) x (W/4) X 64 + +# DECONVOLUTION (deconvB2_1) +layer { name: "deconvB2_1" type: "Deconvolution" bottom: "unpoolB2" top: "deconvB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (debnB2_1) +layer { name: "debnB2_1" type: "BN" bottom: "deconvB2_1" top: "deconvB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB2_1) +layer { name: "dereluB2_1" type: "ReLU" bottom: "deconvB2_1" top: "deconvB2_1" } + + + +# Merge Branch 2 Here +# CONCAT (mergeB2) +layer { + name: "mergeB2" + bottom: "poolB1" + bottom: "deconvB2_1" + top: "mergeB2" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/4) x (W/4) X 128 + +# CONVOLUTION (convB1_2) +layer { name: "convB1_2" type: "Convolution" bottom: "mergeB2" top: "convB1_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB1_2) +layer { name: "bnB1_2" type: "BN" bottom: "convB1_2" top: "convB1_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (reluB1_2) +layer { name: "reluB1_2" type: "ReLU" bottom: "convB1_2" top: "convB1_2" } + + +# UNPOOLING (unpoolB1) +layer { + name: "unpoolB1" type: "Unpooling" bottom: "convB1_2" bottom: "poolB1_mask" top: "unpoolB1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/2) x (W/2) X 64 + +# DECONVOLUTION (deconvB1_1) +layer { name: "deconvB1_1" type: "Deconvolution" bottom: "unpoolB1" top: "deconvB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (debnB1_1) +layer { name: "debnB1_1" type: "BN" bottom: "deconvB1_1" top: "deconvB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (dereluB1_1) +layer { name: "dereluB1_1" type: "ReLU" bottom: "deconvB1_1" top: "deconvB1_1" } + + +# Merge Branch 1 Here +# CONCAT (mergeB1) +layer { + name: "mergeB1" + bottom: "pool1" + bottom: "deconvB1_1" + top: "mergeB1" + type: "Concat" + concat_param { + axis: 1 + } +} + +# (H/2) x (W/2) X 128 + +# CONVOLUTION (conv2_1) +layer { name: "conv2_1" type: "Convolution" bottom: "mergeB1" top: "conv2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_1) +layer { name: "bn2_1" type: "BN" bottom: "conv2_1" top: "conv2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_1) +layer { name: "relu2_1" type: "ReLU" bottom: "conv2_1" top: "conv2_1" } + +# CONVOLUTION (conv2_2) +layer { name: "conv2_2" type: "Convolution" bottom: "conv2_1" top: "conv2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_2) +layer { name: "bn2_2" type: "BN" bottom: "conv2_2" top: "conv2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_2) +layer { name: "relu2_2" type: "ReLU" bottom: "conv2_2" top: "conv2_2" } + + +# CONVOLUTION (conv2_3) +layer { name: "conv2_3" type: "Convolution" bottom: "conv2_2" top: "conv2_3" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_3) +layer { name: "bn2_3" type: "BN" bottom: "conv2_3" top: "conv2_3" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_3) +layer { name: "relu2_3" type: "ReLU" bottom: "conv2_3" top: "conv2_3" } + + +# CONVOLUTION (conv2_4) +layer { name: "conv2_4" type: "Convolution" bottom: "conv2_3" top: "conv2_4" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_4) +layer { name: "bn2_4" type: "BN" bottom: "conv2_4" top: "conv2_4" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_4) +layer { name: "relu2_4" type: "ReLU" bottom: "conv2_4" top: "conv2_4" } + + +# CONVOLUTION (conv2_5) +layer { name: "conv2_5" type: "Convolution" bottom: "conv2_4" top: "conv2_5" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_5) +layer { name: "bn2_5" type: "BN" bottom: "conv2_5" top: "conv2_5" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_5) +layer { name: "relu2_5" type: "ReLU" bottom: "conv2_5" top: "conv2_5" } + + +# CONVOLUTION (conv2_6) +layer { name: "conv2_6" type: "Convolution" bottom: "conv2_5" top: "conv2_6" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_6) +layer { name: "bn2_6" type: "BN" bottom: "conv2_6" top: "conv2_6" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_6) +layer { name: "relu2_6" type: "ReLU" bottom: "conv2_6" top: "conv2_6" } + + +# CONVOLUTION (conv2_7) +layer { name: "conv2_7" type: "Convolution" bottom: "conv2_6" top: "conv2_7" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_7) +layer { name: "bn2_7" type: "BN" bottom: "conv2_7" top: "conv2_7" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_7) +layer { name: "relu2_7" type: "ReLU" bottom: "conv2_7" top: "conv2_7" } + + +# CONVOLUTION (conv2_8) +layer { name: "conv2_8" type: "Convolution" bottom: "conv2_7" top: "conv2_8" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_8) +layer { name: "bn2_8" type: "BN" bottom: "conv2_8" top: "conv2_8" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_8) +layer { name: "relu2_8" type: "ReLU" bottom: "conv2_8" top: "conv2_8" } + + +# CONVOLUTION (conv2_9) +layer { name: "conv2_9" type: "Convolution" bottom: "conv2_8" top: "conv2_9" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_9) +layer { name: "bn2_9" type: "BN" bottom: "conv2_9" top: "conv2_9" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (relu2_9) +layer { name: "relu2_9" type: "ReLU" bottom: "conv2_9" top: "conv2_9" } + + +# H/2 x W/2 x 64 + +# DECONVOLUTION (deconv1) +layer { name: "deconv1" type: "Deconvolution" bottom: "conv2_9" top: "deconv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 2 kernel_size: 5 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H/2 x W/2 X 64 + +# BN (debn1) +layer { name: "debn1" type: "BN" bottom: "deconv1" top: "deconv1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } + bn_mode: INFERENCE } } + +# RELU (derelu1) +layer { name: "derelu1" type: "ReLU" bottom: "deconv1" top: "deconv1" } + +# H/2 x W/2 x 64 + +# **************************** +# ***** Unary prediction ***** +# **************************** + +# CONVOLUTION (conv_313) +layer { name: "conv_313" type: "Convolution" bottom: "deconv1" top: "conv_313" + param { lr_mult: 0.1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 313 pad: 1 kernel_size: 4 stride: 2 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H/4 x W/4 X 313 + +layer { + name: "conv_313_rh" + type: "Scale" + bottom: "conv_313" + top: "conv_313" + scale_param { + bias_term: false + filler { type: "constant" value: 2.606 } + } +} +layer { + name: "class_313_rh" + type: "Softmax" + bottom: "conv_313" + top: "class_313_rh" +} +# ******************** +# ***** Decoding ***** +# ******************** +layer { + name: "class_ab" + type: "Convolution" + bottom: "class_313_rh" + top: "class_ab" + convolution_param { + num_output: 2 + kernel_size: 1 + stride: 1 + } +} \ No newline at end of file diff --git a/training/colorization/train.prototxt b/training/colorization/train.prototxt new file mode 100755 index 0000000..353c4a2 --- /dev/null +++ b/training/colorization/train.prototxt @@ -0,0 +1,689 @@ +name: "exp_colorization" + +layer { name: "data" type: "ImageRegressionData" top: "data_orig" + transform_param { mirror: true crop_size: 128 } + image_data_param { + source: "../Data/imgset/train.txt" + root_folder: "../Data" + shuffle: true batch_size: 128 is_color: true label_type: NONE + } +} + +# PYTHON LAYERS for BGR2Yab and soft encoding of GT + +# **************************** +# ***** Color Conversion ***** +# **************************** +layer { type: 'Python' name: 'img_lab' bottom: "data_orig" + # image in Lab space + top: "data_orig" + # top: "img_lab" + python_param { module: 'caffe_traininglayers' layer: 'BGR2LabLayer' } +} +# Split Y and ab +layer { name: "img_slice" type: "Slice" bottom: "data_orig" + # [0,100] + top: "data" + # [-110,110] + top: "data_ab" + propagate_down: false slice_param { axis: 1 slice_point: 1 } +} +# 0-center data channel +layer { name: "data_l_meansub" type: "Scale" bottom: "data" + # [-50,50] + top: "data" propagate_down: false + param {lr_mult: 0 decay_mult: 0} param {lr_mult: 0 decay_mult: 0} + scale_param { bias_term: true filler { type: 'constant' value: 1 } + bias_filler { type: 'constant' value: -50 } } +} + +# **************************** +# ***** PROCESS LABELS ******* +# **************************** + +# subsample ab +layer { name: 'data_ab_ss' type: 'Convolution' bottom: "data_ab" + # subsampled colors + top: "data_ab_ss" + param {lr_mult: 0 decay_mult: 0} param {lr_mult: 0 decay_mult: 0} + convolution_param { num_output: 2 kernel_size: 1 stride: 4 group: 2 + weight_filler { type: 'constant' value: 1 } } +} + +# encode +layer { type: 'Python' name: 'ab_enc' bottom: "data_ab_ss" + # quantized gt colors + top: "gt_ab_313" + python_param { module: 'caffe_traininglayers' layer: 'NNEncLayer' } +} +# compute gray mask +layer { type: 'Python' name: 'nongray_mask' bottom: "data_ab_ss" + # mask out grayscale images + top: "nongray_mask" + python_param { module: 'caffe_traininglayers' layer: 'NonGrayMaskLayer' } +} +# compute prior boost +layer { type: 'Python' name: 'prior_boost' bottom: "gt_ab_313" + # gradient boosting factors + top: "prior_boost" + python_param { module: 'caffe_traininglayers' layer: 'PriorBoostLayer' } +} +# multiply nongray mask and prior boost +layer { type: 'Eltwise' name: 'prior_boost_nongray' bottom: "prior_boost" + bottom: "nongray_mask" top: "prior_boost_nongray" eltwise_param { operation: 0 } +} + +#-------- + +# H x W x 1 + +# CONVOLUTION (conv1) +layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W x 64 + +# BN (bn1) +layer { name: "bn1" type: "BN" bottom: "conv1" top: "conv1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu1) +layer { name: "relu1" type: "ReLU" bottom: "conv1" top: "conv1" } + +# POOLING (pool1) +layer { + name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" + # top: "pool1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/2) x (W/2) X 64 + +# Introduce Branch 1 Here + +# CONVOLUTION (convB1_1) +layer { name: "convB1_1" type: "Convolution" bottom: "pool1" top: "convB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bnB1_1) +layer { name: "bnB1_1" type: "BN" bottom: "convB1_1" top: "convB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_1) +layer { name: "reluB1_1" type: "ReLU" bottom: "convB1_1" top: "convB1_1" } + +# POOLING (poolB1) +layer { + name: "poolB1" type: "Pooling" bottom: "convB1_1" top: "poolB1" top: "poolB1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/4) x (W/4) X 64 + +# Introduce Branch 2 Here + +# CONVOLUTION (convB2_1) +layer { name: "convB2_1" type: "Convolution" bottom: "poolB1" top: "convB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB2_1) +layer { name: "bnB2_1" type: "BN" bottom: "convB2_1" top: "convB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_1) +layer { name: "reluB2_1" type: "ReLU" bottom: "convB2_1" top: "convB2_1" } + +# POOLING (poolB2) +layer { + name: "poolB2" type: "Pooling" bottom: "convB2_1" top: "poolB2" top: "poolB2_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/8) x (W/8) X 64 + +# Introduce Branch 3 Here + +# CONVOLUTION (convB3_1) +layer { name: "convB3_1" type: "Convolution" bottom: "poolB2" top: "convB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB3_1) +layer { name: "bnB3_1" type: "BN" bottom: "convB3_1" top: "convB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_1) +layer { name: "reluB3_1" type: "ReLU" bottom: "convB3_1" top: "convB3_1" } + +# POOLING (poolB3) +layer { + name: "poolB3" type: "Pooling" bottom: "convB3_1" top: "poolB3" top: "poolB3_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/16) x (W/16) X 64 + + +# Introduce Branch 4 Here + +# CONVOLUTION (convB4_1) +layer { name: "convB4_1" type: "Convolution" bottom: "poolB3" top: "convB4_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB4_1) +layer { name: "bnB4_1" type: "BN" bottom: "convB4_1" top: "convB4_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB4_1) +layer { name: "reluB4_1" type: "ReLU" bottom: "convB4_1" top: "convB4_1" } + +# POOLING (poolB4) +layer { + name: "poolB4" type: "Pooling" bottom: "convB4_1" top: "poolB4" top: "poolB4_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/32) x (W/32) X 64 + +# CONVOLUTION (convB4_2) +layer { name: "convB4_2" type: "Convolution" bottom: "poolB4" top: "convB4_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/32) x (W/32) X 64 + +# BN (bnB4_2) +layer { name: "bnB4_2" type: "BN" bottom: "convB4_2" top: "convB4_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB4_2) +layer { name: "reluB4_2" type: "ReLU" bottom: "convB4_2" top: "convB4_2" } + + +# UNPOOLING (unpoolB4) +layer { + name: "unpoolB4" type: "Unpooling" bottom: "convB4_2" bottom: "poolB4_mask" top: "unpoolB4" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/16) x (W/16) X 64 + +# DECONVOLUTION (deconvB4_1) +layer { name: "deconvB4_1" type: "Deconvolution" bottom: "unpoolB4" top: "deconvB4_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (debnB4_1) +layer { name: "debnB4_1" type: "BN" bottom: "deconvB4_1" top: "deconvB4_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB4_1) +layer { name: "dereluB4_1" type: "ReLU" bottom: "deconvB4_1" top: "deconvB4_1" } + + + +# Merge Branch 4 Here +# CONCAT (mergeB4) +layer { + name: "mergeB4" + bottom: "poolB3" + bottom: "deconvB4_1" + top: "mergeB4" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/16) x (W/16) X 128 + + +# CONVOLUTION (convB3_2) +layer { name: "convB3_2" type: "Convolution" bottom: "mergeB4" top: "convB3_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB3_2) +layer { name: "bnB3_2" type: "BN" bottom: "convB3_2" top: "convB3_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_2) +layer { name: "reluB3_2" type: "ReLU" bottom: "convB3_2" top: "convB3_2" } + + +# UNPOOLING (unpoolB3) +layer { + name: "unpoolB3" type: "Unpooling" bottom: "convB3_2" bottom: "poolB3_mask" top: "unpoolB3" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/8) x (W/8) X 64 + +# DECONVOLUTION (deconvB3_1) +layer { name: "deconvB3_1" type: "Deconvolution" bottom: "unpoolB3" top: "deconvB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (debnB3_1) +layer { name: "debnB3_1" type: "BN" bottom: "deconvB3_1" top: "deconvB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB3_1) +layer { name: "dereluB3_1" type: "ReLU" bottom: "deconvB3_1" top: "deconvB3_1" } + + + +# Merge Branch 3 Here +# CONCAT (mergeB3) +layer { + name: "mergeB3" + bottom: "poolB2" + bottom: "deconvB3_1" + top: "mergeB3" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/8) x (W/8) X 128 + +# CONVOLUTION (convB2_2) +layer { name: "convB2_2" type: "Convolution" bottom: "mergeB3" top: "convB2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB2_2) +layer { name: "bnB2_2" type: "BN" bottom: "convB2_2" top: "convB2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_2) +layer { name: "reluB2_2" type: "ReLU" bottom: "convB2_2" top: "convB2_2" } + + +# UNPOOLING (unpoolB2) +layer { + name: "unpoolB2" type: "Unpooling" bottom: "convB2_2" bottom: "poolB2_mask" top: "unpoolB2" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/4) x (W/4) X 64 + +# DECONVOLUTION (deconvB2_1) +layer { name: "deconvB2_1" type: "Deconvolution" bottom: "unpoolB2" top: "deconvB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (debnB2_1) +layer { name: "debnB2_1" type: "BN" bottom: "deconvB2_1" top: "deconvB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB2_1) +layer { name: "dereluB2_1" type: "ReLU" bottom: "deconvB2_1" top: "deconvB2_1" } + + + +# Merge Branch 2 Here +# CONCAT (mergeB2) +layer { + name: "mergeB2" + bottom: "poolB1" + bottom: "deconvB2_1" + top: "mergeB2" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/4) x (W/4) X 128 + +# CONVOLUTION (convB1_2) +layer { name: "convB1_2" type: "Convolution" bottom: "mergeB2" top: "convB1_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB1_2) +layer { name: "bnB1_2" type: "BN" bottom: "convB1_2" top: "convB1_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_2) +layer { name: "reluB1_2" type: "ReLU" bottom: "convB1_2" top: "convB1_2" } + + +# UNPOOLING (unpoolB1) +layer { + name: "unpoolB1" type: "Unpooling" bottom: "convB1_2" bottom: "poolB1_mask" top: "unpoolB1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/2) x (W/2) X 64 + +# DECONVOLUTION (deconvB1_1) +layer { name: "deconvB1_1" type: "Deconvolution" bottom: "unpoolB1" top: "deconvB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (debnB1_1) +layer { name: "debnB1_1" type: "BN" bottom: "deconvB1_1" top: "deconvB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB1_1) +layer { name: "dereluB1_1" type: "ReLU" bottom: "deconvB1_1" top: "deconvB1_1" } + + +# Merge Branch 1 Here +# CONCAT (mergeB1) +layer { + name: "mergeB1" + bottom: "pool1" + bottom: "deconvB1_1" + top: "mergeB1" + type: "Concat" + concat_param { + axis: 1 + } +} + +# (H/2) x (W/2) X 128 + +# CONVOLUTION (conv2_1) +layer { name: "conv2_1" type: "Convolution" bottom: "mergeB1" top: "conv2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_1) +layer { name: "bn2_1" type: "BN" bottom: "conv2_1" top: "conv2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_1) +layer { name: "relu2_1" type: "ReLU" bottom: "conv2_1" top: "conv2_1" } + +# CONVOLUTION (conv2_2) +layer { name: "conv2_2" type: "Convolution" bottom: "conv2_1" top: "conv2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_2) +layer { name: "bn2_2" type: "BN" bottom: "conv2_2" top: "conv2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_2) +layer { name: "relu2_2" type: "ReLU" bottom: "conv2_2" top: "conv2_2" } + + +# CONVOLUTION (conv2_3) +layer { name: "conv2_3" type: "Convolution" bottom: "conv2_2" top: "conv2_3" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_3) +layer { name: "bn2_3" type: "BN" bottom: "conv2_3" top: "conv2_3" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_3) +layer { name: "relu2_3" type: "ReLU" bottom: "conv2_3" top: "conv2_3" } + + +# CONVOLUTION (conv2_4) +layer { name: "conv2_4" type: "Convolution" bottom: "conv2_3" top: "conv2_4" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_4) +layer { name: "bn2_4" type: "BN" bottom: "conv2_4" top: "conv2_4" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_4) +layer { name: "relu2_4" type: "ReLU" bottom: "conv2_4" top: "conv2_4" } + + +# CONVOLUTION (conv2_5) +layer { name: "conv2_5" type: "Convolution" bottom: "conv2_4" top: "conv2_5" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_5) +layer { name: "bn2_5" type: "BN" bottom: "conv2_5" top: "conv2_5" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_5) +layer { name: "relu2_5" type: "ReLU" bottom: "conv2_5" top: "conv2_5" } + + +# CONVOLUTION (conv2_6) +layer { name: "conv2_6" type: "Convolution" bottom: "conv2_5" top: "conv2_6" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_6) +layer { name: "bn2_6" type: "BN" bottom: "conv2_6" top: "conv2_6" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_6) +layer { name: "relu2_6" type: "ReLU" bottom: "conv2_6" top: "conv2_6" } + + +# CONVOLUTION (conv2_7) +layer { name: "conv2_7" type: "Convolution" bottom: "conv2_6" top: "conv2_7" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_7) +layer { name: "bn2_7" type: "BN" bottom: "conv2_7" top: "conv2_7" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_7) +layer { name: "relu2_7" type: "ReLU" bottom: "conv2_7" top: "conv2_7" } + + +# CONVOLUTION (conv2_8) +layer { name: "conv2_8" type: "Convolution" bottom: "conv2_7" top: "conv2_8" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_8) +layer { name: "bn2_8" type: "BN" bottom: "conv2_8" top: "conv2_8" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_8) +layer { name: "relu2_8" type: "ReLU" bottom: "conv2_8" top: "conv2_8" } + + +# CONVOLUTION (conv2_9) +layer { name: "conv2_9" type: "Convolution" bottom: "conv2_8" top: "conv2_9" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_9) +layer { name: "bn2_9" type: "BN" bottom: "conv2_9" top: "conv2_9" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_9) +layer { name: "relu2_9" type: "ReLU" bottom: "conv2_9" top: "conv2_9" } + + +# # UNPOOLING (unpool1) +# layer { +# name: "unpool1" type: "Unpooling" bottom: "conv2_9" bottom: "pool1_mask" top: "unpool1" +# unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + + +# H x W x 64 + +# DECONVOLUTION (deconv1) +layer { name: "deconv1" type: "Deconvolution" bottom: "conv2_9" top: "deconv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 2 kernel_size: 5 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# BN (debn1) +layer { name: "debn1" type: "BN" bottom: "deconv1" top: "deconv1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (derelu1) +layer { name: "derelu1" type: "ReLU" bottom: "deconv1" top: "deconv1" } + +# H x W x 64 + +# **************************** +# ***** Unary prediction ***** +# **************************** + +# CONVOLUTION (conv_313) +layer { name: "conv_313" type: "Convolution" bottom: "deconv1" top: "conv_313" + param { lr_mult: 0.1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 313 pad: 1 kernel_size: 4 stride: 2 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W X 313 + +# *************************** +# ***** Boosting priors ***** +# *************************** +layer { name: "conv_313_boost" type: "Python" bottom: "conv_313" + bottom: "prior_boost_nongray" top: "conv_313" + python_param { module: 'caffe_traininglayers' layer: 'ClassRebalanceMultLayer' } +} + +# H x W X 313 + +# SOFTMAX CROSS-ENTROPY LOSS +layer { + name: "loss_313" + type: "SoftmaxCrossEntropyLoss" + bottom: "conv_313" + bottom: "gt_ab_313" + top: "loss_313" + loss_weight: 1.0 +} diff --git a/training/denoising/make_bn.prototxt b/training/denoising/make_bn.prototxt new file mode 100755 index 0000000..362d5b0 --- /dev/null +++ b/training/denoising/make_bn.prototxt @@ -0,0 +1,512 @@ +name: "exp_denoising" + +layer { name: "data" type: "ImageRegressionData" top: "data" top: "label" + transform_param { mirror: true crop_size: 128 + wgn_param { min_noise_std: 8 max_noise_std: 50 } + } + image_data_param { + source: "../Data/imgset/train.txt" + root_folder: "../Data" + shuffle: true + batch_size: 128 is_color: false is_label_color: false label_type: NONE + } +} + +# H x W x 3 OR H x W x 1 + +# CONVOLUTION (conv1) +layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W x 64 + +# BN (bn1) +layer { name: "bn1" type: "BN" bottom: "conv1" top: "conv1" + top: "bn1-mean" top: "bn1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu1) +layer { name: "relu1" type: "ReLU" bottom: "conv1" top: "conv1" } + +# POOLING (pool1) +layer { + name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" top: "pool1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/2) x (W/2) X 64 + +# Introduce Branch 1 Here + +# CONVOLUTION (convB1_1) +layer { name: "convB1_1" type: "Convolution" bottom: "pool1" top: "convB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bnB1_1) +layer { name: "bnB1_1" type: "BN" bottom: "convB1_1" top: "convB1_1" + top: "bnB1_1-mean" top: "bnB1_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_1) +layer { name: "reluB1_1" type: "ReLU" bottom: "convB1_1" top: "convB1_1" } + +# POOLING (poolB1) +layer { + name: "poolB1" type: "Pooling" bottom: "convB1_1" top: "poolB1" top: "poolB1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/4) x (W/4) X 64 + +# Introduce Branch 2 Here + +# CONVOLUTION (convB2_1) +layer { name: "convB2_1" type: "Convolution" bottom: "poolB1" top: "convB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB2_1) +layer { name: "bnB2_1" type: "BN" bottom: "convB2_1" top: "convB2_1" + top: "bnB2_1-mean" top: "bnB2_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_1) +layer { name: "reluB2_1" type: "ReLU" bottom: "convB2_1" top: "convB2_1" } + +# POOLING (poolB2) +layer { + name: "poolB2" type: "Pooling" bottom: "convB2_1" top: "poolB2" top: "poolB2_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/8) x (W/8) X 64 + +# Introduce Branch 3 Here + +# CONVOLUTION (convB3_1) +layer { name: "convB3_1" type: "Convolution" bottom: "poolB2" top: "convB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB3_1) +layer { name: "bnB3_1" type: "BN" bottom: "convB3_1" top: "convB3_1" + top: "bnB3_1-mean" top: "bnB3_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_1) +layer { name: "reluB3_1" type: "ReLU" bottom: "convB3_1" top: "convB3_1" } + +# POOLING (poolB3) +layer { + name: "poolB3" type: "Pooling" bottom: "convB3_1" top: "poolB3" top: "poolB3_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/16) x (W/16) X 64 + +# CONVOLUTION (convB3_2) +layer { name: "convB3_2" type: "Convolution" bottom: "poolB3" top: "convB3_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB3_2) +layer { name: "bnB3_2" type: "BN" bottom: "convB3_2" top: "convB3_2" + top: "bnB3_2-mean" top: "bnB3_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_2) +layer { name: "reluB3_2" type: "ReLU" bottom: "convB3_2" top: "convB3_2" } + + +# UNPOOLING (unpoolB3) +layer { + name: "unpoolB3" type: "Unpooling" bottom: "convB3_2" bottom: "poolB3_mask" top: "unpoolB3" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/8) x (W/8) X 64 + +# DECONVOLUTION (deconvB3_1) +layer { name: "deconvB3_1" type: "Deconvolution" bottom: "unpoolB3" top: "deconvB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (debnB3_1) +layer { name: "debnB3_1" type: "BN" bottom: "deconvB3_1" top: "deconvB3_1" + top: "debnB3_1-mean" top: "debnB3_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB3_1) +layer { name: "dereluB3_1" type: "ReLU" bottom: "deconvB3_1" top: "deconvB3_1" } + + + +# Merge Branch 3 Here +# CONCAT (mergeB3) +layer { + name: "mergeB3" + bottom: "poolB2" + bottom: "deconvB3_1" + top: "mergeB3" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/8) x (W/8) X 128 + +# CONVOLUTION (convB2_2) +layer { name: "convB2_2" type: "Convolution" bottom: "mergeB3" top: "convB2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB2_2) +layer { name: "bnB2_2" type: "BN" bottom: "convB2_2" top: "convB2_2" + top: "bnB2_2-mean" top: "bnB2_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_2) +layer { name: "reluB2_2" type: "ReLU" bottom: "convB2_2" top: "convB2_2" } + + +# UNPOOLING (unpoolB2) +layer { + name: "unpoolB2" type: "Unpooling" bottom: "convB2_2" bottom: "poolB2_mask" top: "unpoolB2" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/4) x (W/4) X 64 + +# DECONVOLUTION (deconvB2_1) +layer { name: "deconvB2_1" type: "Deconvolution" bottom: "unpoolB2" top: "deconvB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (debnB2_1) +layer { name: "debnB2_1" type: "BN" bottom: "deconvB2_1" top: "deconvB2_1" + top: "debnB2_1-mean" top: "debnB2_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB2_1) +layer { name: "dereluB2_1" type: "ReLU" bottom: "deconvB2_1" top: "deconvB2_1" } + + + +# Merge Branch 2 Here +# CONCAT (mergeB2) +layer { + name: "mergeB2" + bottom: "poolB1" + bottom: "deconvB2_1" + top: "mergeB2" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/4) x (W/4) X 128 + +# CONVOLUTION (convB1_2) +layer { name: "convB1_2" type: "Convolution" bottom: "mergeB2" top: "convB1_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB1_2) +layer { name: "bnB1_2" type: "BN" bottom: "convB1_2" top: "convB1_2" + top: "bnB1_2-mean" top: "bnB1_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_2) +layer { name: "reluB1_2" type: "ReLU" bottom: "convB1_2" top: "convB1_2" } + + +# UNPOOLING (unpoolB1) +layer { + name: "unpoolB1" type: "Unpooling" bottom: "convB1_2" bottom: "poolB1_mask" top: "unpoolB1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/2) x (W/2) X 64 + +# DECONVOLUTION (deconvB1_1) +layer { name: "deconvB1_1" type: "Deconvolution" bottom: "unpoolB1" top: "deconvB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (debnB1_1) +layer { name: "debnB1_1" type: "BN" bottom: "deconvB1_1" top: "deconvB1_1" + top: "debnB1_1-mean" top: "debnB1_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB1_1) +layer { name: "dereluB1_1" type: "ReLU" bottom: "deconvB1_1" top: "deconvB1_1" } + + +# Merge Branch 1 Here +# CONCAT (mergeB1) +layer { + name: "mergeB1" + bottom: "pool1" + bottom: "deconvB1_1" + top: "mergeB1" + type: "Concat" + concat_param { + axis: 1 + } +} + +# (H/2) x (W/2) X 128 + +# CONVOLUTION (conv2_1) +layer { name: "conv2_1" type: "Convolution" bottom: "mergeB1" top: "conv2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_1) +layer { name: "bn2_1" type: "BN" bottom: "conv2_1" top: "conv2_1" + top: "bn2_1-mean" top: "bn2_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_1) +layer { name: "relu2_1" type: "ReLU" bottom: "conv2_1" top: "conv2_1" } + +# CONVOLUTION (conv2_2) +layer { name: "conv2_2" type: "Convolution" bottom: "conv2_1" top: "conv2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_2) +layer { name: "bn2_2" type: "BN" bottom: "conv2_2" top: "conv2_2" + top: "bn2_2-mean" top: "bn2_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_2) +layer { name: "relu2_2" type: "ReLU" bottom: "conv2_2" top: "conv2_2" } + + +# CONVOLUTION (conv2_3) +layer { name: "conv2_3" type: "Convolution" bottom: "conv2_2" top: "conv2_3" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_3) +layer { name: "bn2_3" type: "BN" bottom: "conv2_3" top: "conv2_3" + top: "bn2_3-mean" top: "bn2_3-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_3) +layer { name: "relu2_3" type: "ReLU" bottom: "conv2_3" top: "conv2_3" } + + +# CONVOLUTION (conv2_4) +layer { name: "conv2_4" type: "Convolution" bottom: "conv2_3" top: "conv2_4" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_4) +layer { name: "bn2_4" type: "BN" bottom: "conv2_4" top: "conv2_4" + top: "bn2_4-mean" top: "bn2_4-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_4) +layer { name: "relu2_4" type: "ReLU" bottom: "conv2_4" top: "conv2_4" } + + +# CONVOLUTION (conv2_5) +layer { name: "conv2_5" type: "Convolution" bottom: "conv2_4" top: "conv2_5" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_5) +layer { name: "bn2_5" type: "BN" bottom: "conv2_5" top: "conv2_5" + top: "bn2_5-mean" top: "bn2_5-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_5) +layer { name: "relu2_5" type: "ReLU" bottom: "conv2_5" top: "conv2_5" } + + +# CONVOLUTION (conv2_6) +layer { name: "conv2_6" type: "Convolution" bottom: "conv2_5" top: "conv2_6" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_6) +layer { name: "bn2_6" type: "BN" bottom: "conv2_6" top: "conv2_6" + top: "bn2_6-mean" top: "bn2_6-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_6) +layer { name: "relu2_6" type: "ReLU" bottom: "conv2_6" top: "conv2_6" } + + +# CONVOLUTION (conv2_7) +layer { name: "conv2_7" type: "Convolution" bottom: "conv2_6" top: "conv2_7" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_7) +layer { name: "bn2_7" type: "BN" bottom: "conv2_7" top: "conv2_7" + top: "bn2_7-mean" top: "bn2_7-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_7) +layer { name: "relu2_7" type: "ReLU" bottom: "conv2_7" top: "conv2_7" } + + +# CONVOLUTION (conv2_8) +layer { name: "conv2_8" type: "Convolution" bottom: "conv2_7" top: "conv2_8" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_8) +layer { name: "bn2_8" type: "BN" bottom: "conv2_8" top: "conv2_8" + top: "bn2_8-mean" top: "bn2_8-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_8) +layer { name: "relu2_8" type: "ReLU" bottom: "conv2_8" top: "conv2_8" } + + +# CONVOLUTION (conv2_9) +layer { name: "conv2_9" type: "Convolution" bottom: "conv2_8" top: "conv2_9" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_9) +layer { name: "bn2_9" type: "BN" bottom: "conv2_9" top: "conv2_9" + top: "bn2_9-mean" top: "bn2_9-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_9) +layer { name: "relu2_9" type: "ReLU" bottom: "conv2_9" top: "conv2_9" } + + +# UNPOOLING (unpool1) +layer { + name: "unpool1" type: "Unpooling" bottom: "conv2_9" bottom: "pool1_mask" top: "unpool1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + + +# H x W x 64 + +# DECONVOLUTION (deconv1) +layer { name: "deconv1" type: "Deconvolution" bottom: "unpool1" top: "pred" + param { lr_mult: 0.1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 1 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W X 1 diff --git a/training/denoising/make_bn.py b/training/denoising/make_bn.py new file mode 100755 index 0000000..918dd3c --- /dev/null +++ b/training/denoising/make_bn.py @@ -0,0 +1,182 @@ + +### User Configuration + +iteration = 1000 +gpu_id_num = 0 + +## path configuration +caffe_root = '../caffe' +script_path = '.' +caffe_model = script_path + '/make_bn.prototxt' +caffe_weight = script_path + '/train_curr.caffemodel' +caffe_inference_weight = script_path + '/train_curr_inference.caffemodel' + +## modify this definition according to model definition +bn_blobs= ['conv1', + 'convB1_1', + 'convB2_1', + 'convB3_1', + 'convB3_2', + 'deconvB3_1', + 'convB2_2', + 'deconvB2_1', + 'convB1_2', + 'deconvB1_1', + 'conv2_1', + 'conv2_2', + 'conv2_3', + 'conv2_4', + 'conv2_5', + 'conv2_6', + 'conv2_7', + 'conv2_8', + 'conv2_9'] +bn_layers=['bn1', + 'bnB1_1', + 'bnB2_1', + 'bnB3_1', + 'bnB3_2', + 'debnB3_1', + 'bnB2_2', + 'debnB2_1', + 'bnB1_2', + 'debnB1_1', + 'bn2_1', + 'bn2_2', + 'bn2_3', + 'bn2_4', + 'bn2_5', + 'bn2_6', + 'bn2_7', + 'bn2_8', + 'bn2_9'] +bn_means= ['bn1-mean', + 'bnB1_1-mean', + 'bnB2_1-mean', + 'bnB3_1-mean', + 'bnB3_2-mean', + 'debnB3_1-mean', + 'bnB2_2-mean', + 'debnB2_1-mean', + 'bnB1_2-mean', + 'debnB1_1-mean', + 'bn2_1-mean', + 'bn2_2-mean', + 'bn2_3-mean', + 'bn2_4-mean', + 'bn2_5-mean', + 'bn2_6-mean', + 'bn2_7-mean', + 'bn2_8-mean', + 'bn2_9-mean'] +bn_vars = ['bn1-var', + 'bnB1_1-var', + 'bnB2_1-var', + 'bnB3_1-var', + 'bnB3_2-var', + 'debnB3_1-var', + 'bnB2_2-var', + 'debnB2_1-var', + 'bnB1_2-var', + 'debnB1_1-var', + 'bn2_1-var', + 'bn2_2-var', + 'bn2_3-var', + 'bn2_4-var', + 'bn2_5-var', + 'bn2_6-var', + 'bn2_7-var', + 'bn2_8-var', + 'bn2_9-var'] + + +### start generate caffemodel + +print 'start generating BN-testable caffemodel' +print 'caffe_root: %s' % caffe_root +print 'script_path: %s' % script_path +print 'caffe_model: %s' % caffe_model +print 'caffe_weight: %s' % caffe_weight +print 'caffe_inference_weight: %s' % caffe_inference_weight + +import numpy as np + +import sys +sys.path.append(caffe_root+'/python') +import caffe +from caffe.proto import caffe_pb2 + +# caffe.set_mode_cpu() +caffe.set_mode_gpu() +caffe.set_device(gpu_id_num) +net = caffe.Net(caffe_model, caffe_weight, caffe.TEST) + +def forward_once(net): + start_ind = 0 + end_ind = len(net.layers) - 1 + net._forward(start_ind, end_ind) + return {out: net.blobs[out].data for out in net.outputs} + +print net.params.keys() + +res = forward_once(net) + +bn_avg_mean = {bn_mean: np.squeeze(res[bn_mean]).copy() for bn_mean in bn_means} +bn_avg_var = {bn_var: np.squeeze(res[bn_var]).copy() for bn_var in bn_vars} + +cnt = 1 + +for i in range(0, iteration): + res = forward_once(net) + for bn_mean in bn_means: + bn_avg_mean[bn_mean] = bn_avg_mean[bn_mean] + np.squeeze(res[bn_mean]) + for bn_var in bn_vars: + bn_avg_var[bn_var] = bn_avg_var[bn_var] + np.squeeze(res[bn_var]) + + cnt += 1 + print 'progress: %d/%d' % (i, iteration) + +## compute average +for bn_mean in bn_means: + bn_avg_mean[bn_mean] /= cnt +for bn_var in bn_vars: + bn_avg_var[bn_var] /= cnt + +for i in range(0, len(bn_vars)): + m = np.prod(net.blobs[bn_blobs[i]].data.shape) / np.prod(bn_avg_var[bn_vars[i]].shape) + bn_avg_var[bn_vars[i]] *= (m/(m-1)) + +scale_data = {bn_layer: np.squeeze(net.params[bn_layer][0].data) for bn_layer in bn_layers} +shift_data = {bn_layer: np.squeeze(net.params[bn_layer][1].data) for bn_layer in bn_layers} + +var_eps = 1e-9 + +new_scale_data = {} +new_shift_data = {} +for i in range(0, len(bn_layers)): + gamma = scale_data[bn_layers[i]] + beta = shift_data[bn_layers[i]] + Ex = bn_avg_mean[bn_means[i]] + Varx = bn_avg_var[bn_vars[i]] + new_gamma = gamma / np.sqrt(Varx + var_eps) + new_beta = beta - (gamma * Ex / np.sqrt(Varx + var_eps)) + + new_scale_data[bn_layers[i]] = new_gamma + new_shift_data[bn_layers[i]] = new_beta + +print new_scale_data.keys() +print new_shift_data.keys() + +## assign computed new scale and shift values to net.params +for i in range(0, len(bn_layers)): + net.params[bn_layers[i]][0].data[...] = new_scale_data[bn_layers[i]].reshape(net.params[bn_layers[i]][0].data.shape) + net.params[bn_layers[i]][1].data[...] = new_shift_data[bn_layers[i]].reshape(net.params[bn_layers[i]][1].data.shape) + +print 'start saving model' + +net.save(caffe_inference_weight) + +print 'done' + + + diff --git a/training/denoising/run_bn.sh b/training/denoising/run_bn.sh new file mode 100755 index 0000000..b8cc0f3 --- /dev/null +++ b/training/denoising/run_bn.sh @@ -0,0 +1,40 @@ +#!/bin/bash + +# Pull most recent snapshot and prepare it for inference by computing global +# mean/variance for all the batch-normalization layers by passing training +# data through the network. If the most recent snapshot is in +# ./snapshot/trn_iter_[N].caffemodel, then the final inference-ready weights +# will be saved in ./tst_[N].caffemodel +# Training Data is expected in ../Data (can be a symlink) +# ImageList is expected in ../Data/imgset/train.txt (relative to root ../Data) + +# Choose most recent snapshot for inference +CAFFEDIR="../caffe" +CURR_SNAPSHOT=`ls ./snapshot/trn_iter_*.caffemodel -t | head -n 1` +F=${CURR_SNAPSHOT/"./snapshot/trn_iter_"/} +ITER=${F/".caffemodel"/} +ITER=${ITER/"*"/} +P_ITER=`printf "%08g\n" ${ITER}` + +WEIGHTSA_TMP=./train_curr.caffemodel +WEIGHTSA_INF_TMP=./train_curr_inference.caffemodel +WEIGHTSA_INF=./tst_${P_ITER}.caffemodel + +echo $WEIGHTSA +echo $WEIGHTSA_INF + +# LOAD CAFFE DEPENDENCIES TO LD_LIBRARY_PATH +source ${CAFFEDIR}/load_caffe_dependencies.sh + +rm -f $WEIGHTSA_TMP && cp -f $CURR_SNAPSHOT $WEIGHTSA_TMP + +# ADJUST BATCH NORMALIZATION FOR INFERENCE + +# Start python virtual env +# source /scratch0/venkai/dependencies/env/bin/activate +python make_bn.py +# deactivate + +rm -f ${WEIGHTSA_TMP} +mv ${WEIGHTSA_INF_TMP} ${WEIGHTSA_INF} + diff --git a/training/denoising/solver.prototxt b/training/denoising/solver.prototxt new file mode 100755 index 0000000..d8f0a08 --- /dev/null +++ b/training/denoising/solver.prototxt @@ -0,0 +1,14 @@ +net: "./train.prototxt" +base_lr: 0.0000001 +lr_policy: "step" +stepsize: 100000 +gamma: 0.1 +iter_size: 1 +display: 10 +max_iter: 500000 +momentum: 0.9 +weight_decay: 0.0005 +snapshot: 1000 +snapshot_prefix: "./snapshot/trn" +solver_mode: GPU + diff --git a/training/denoising/start_train.sh b/training/denoising/start_train.sh new file mode 100755 index 0000000..d560075 --- /dev/null +++ b/training/denoising/start_train.sh @@ -0,0 +1,29 @@ +#!/bin/bash + +# Start training an RBDN model for denoising. You can pause training at any moment +# with Ctrl+C and most recent snapshot will be saved in +# ./snapshot/trn_iter_[*].solverstate +# Running ./start_train.sh again will automatically resume from that snapshot. +# Training Data is expected in ../Data (can be a symlink) +# ImageList is expected in ../Data/imgset/train.txt (relative to root ../Data) + +CAFFEDIR="../caffe" +LOGDIR=./training_log +CAFFE=${CAFFEDIR}/build/tools/caffe +SOLVER=./solver.prototxt + +# LOAD CAFFE DEPENDENCIES TO LD_LIBRARY_PATH +source ${CAFFEDIR}/load_caffe_dependencies.sh + +mkdir -p $LOGDIR +mkdir -p snapshot + +# Check if snapshot exists; if not train from scratch. +if [ "$(ls -A ./snapshot/)" ]; then + CURR_SNAPSHOT=`ls ./snapshot/*solverstate -t | head -n 1` + echo "Resuming training from snapshot ${CURR_SNAPSHOT}" + echo "If you want to train from scratch, delete everything in ./snapshot" + GLOG_log_dir=$LOGDIR $CAFFE train -snapshot $CURR_SNAPSHOT -solver $SOLVER -gpu 0 +else + GLOG_log_dir=$LOGDIR $CAFFE train -solver $SOLVER -gpu 0 +fi diff --git a/training/denoising/train.prototxt b/training/denoising/train.prototxt new file mode 100755 index 0000000..05f8381 --- /dev/null +++ b/training/denoising/train.prototxt @@ -0,0 +1,503 @@ +name: "exp_denoising" + +layer { name: "data" type: "ImageRegressionData" top: "data" top: "label" + include { phase: TRAIN } + transform_param { mirror: true crop_size: 128 + wgn_param { min_noise_std: 8 max_noise_std: 50 } + } + image_data_param { + source: "../Data/imgset/train.txt" + root_folder: "../Data" + shuffle: true + batch_size: 64 is_color: false is_label_color: false label_type: NONE + } +} + +# H x W x 1 + +# CONVOLUTION (conv1) +layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W x 64 + +# BN (bn1) +layer { name: "bn1" type: "BN" bottom: "conv1" top: "conv1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu1) +layer { name: "relu1" type: "ReLU" bottom: "conv1" top: "conv1" } + +# POOLING (pool1) +layer { + name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" top: "pool1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/2) x (W/2) X 64 + +# Introduce Branch 1 Here + +# CONVOLUTION (convB1_1) +layer { name: "convB1_1" type: "Convolution" bottom: "pool1" top: "convB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bnB1_1) +layer { name: "bnB1_1" type: "BN" bottom: "convB1_1" top: "convB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_1) +layer { name: "reluB1_1" type: "ReLU" bottom: "convB1_1" top: "convB1_1" } + +# POOLING (poolB1) +layer { + name: "poolB1" type: "Pooling" bottom: "convB1_1" top: "poolB1" top: "poolB1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/4) x (W/4) X 64 + +# Introduce Branch 2 Here + +# CONVOLUTION (convB2_1) +layer { name: "convB2_1" type: "Convolution" bottom: "poolB1" top: "convB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB2_1) +layer { name: "bnB2_1" type: "BN" bottom: "convB2_1" top: "convB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_1) +layer { name: "reluB2_1" type: "ReLU" bottom: "convB2_1" top: "convB2_1" } + +# POOLING (poolB2) +layer { + name: "poolB2" type: "Pooling" bottom: "convB2_1" top: "poolB2" top: "poolB2_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/8) x (W/8) X 64 + +# Introduce Branch 3 Here + +# CONVOLUTION (convB3_1) +layer { name: "convB3_1" type: "Convolution" bottom: "poolB2" top: "convB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB3_1) +layer { name: "bnB3_1" type: "BN" bottom: "convB3_1" top: "convB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_1) +layer { name: "reluB3_1" type: "ReLU" bottom: "convB3_1" top: "convB3_1" } + +# POOLING (poolB3) +layer { + name: "poolB3" type: "Pooling" bottom: "convB3_1" top: "poolB3" top: "poolB3_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/16) x (W/16) X 64 + +# CONVOLUTION (convB3_2) +layer { name: "convB3_2" type: "Convolution" bottom: "poolB3" top: "convB3_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB3_2) +layer { name: "bnB3_2" type: "BN" bottom: "convB3_2" top: "convB3_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_2) +layer { name: "reluB3_2" type: "ReLU" bottom: "convB3_2" top: "convB3_2" } + + +# UNPOOLING (unpoolB3) +layer { + name: "unpoolB3" type: "Unpooling" bottom: "convB3_2" bottom: "poolB3_mask" top: "unpoolB3" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/8) x (W/8) X 64 + +# DECONVOLUTION (deconvB3_1) +layer { name: "deconvB3_1" type: "Deconvolution" bottom: "unpoolB3" top: "deconvB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (debnB3_1) +layer { name: "debnB3_1" type: "BN" bottom: "deconvB3_1" top: "deconvB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB3_1) +layer { name: "dereluB3_1" type: "ReLU" bottom: "deconvB3_1" top: "deconvB3_1" } + + + +# Merge Branch 3 Here +# CONCAT (mergeB3) +layer { + name: "mergeB3" + bottom: "poolB2" + bottom: "deconvB3_1" + top: "mergeB3" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/8) x (W/8) X 128 + +# CONVOLUTION (convB2_2) +layer { name: "convB2_2" type: "Convolution" bottom: "mergeB3" top: "convB2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB2_2) +layer { name: "bnB2_2" type: "BN" bottom: "convB2_2" top: "convB2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_2) +layer { name: "reluB2_2" type: "ReLU" bottom: "convB2_2" top: "convB2_2" } + + +# UNPOOLING (unpoolB2) +layer { + name: "unpoolB2" type: "Unpooling" bottom: "convB2_2" bottom: "poolB2_mask" top: "unpoolB2" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/4) x (W/4) X 64 + +# DECONVOLUTION (deconvB2_1) +layer { name: "deconvB2_1" type: "Deconvolution" bottom: "unpoolB2" top: "deconvB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (debnB2_1) +layer { name: "debnB2_1" type: "BN" bottom: "deconvB2_1" top: "deconvB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB2_1) +layer { name: "dereluB2_1" type: "ReLU" bottom: "deconvB2_1" top: "deconvB2_1" } + + + +# Merge Branch 2 Here +# CONCAT (mergeB2) +layer { + name: "mergeB2" + bottom: "poolB1" + bottom: "deconvB2_1" + top: "mergeB2" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/4) x (W/4) X 128 + +# CONVOLUTION (convB1_2) +layer { name: "convB1_2" type: "Convolution" bottom: "mergeB2" top: "convB1_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB1_2) +layer { name: "bnB1_2" type: "BN" bottom: "convB1_2" top: "convB1_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_2) +layer { name: "reluB1_2" type: "ReLU" bottom: "convB1_2" top: "convB1_2" } + + +# UNPOOLING (unpoolB1) +layer { + name: "unpoolB1" type: "Unpooling" bottom: "convB1_2" bottom: "poolB1_mask" top: "unpoolB1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/2) x (W/2) X 64 + +# DECONVOLUTION (deconvB1_1) +layer { name: "deconvB1_1" type: "Deconvolution" bottom: "unpoolB1" top: "deconvB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (debnB1_1) +layer { name: "debnB1_1" type: "BN" bottom: "deconvB1_1" top: "deconvB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB1_1) +layer { name: "dereluB1_1" type: "ReLU" bottom: "deconvB1_1" top: "deconvB1_1" } + + +# Merge Branch 1 Here +# CONCAT (mergeB1) +layer { + name: "mergeB1" + bottom: "pool1" + bottom: "deconvB1_1" + top: "mergeB1" + type: "Concat" + concat_param { + axis: 1 + } +} + +# (H/2) x (W/2) X 128 + +# CONVOLUTION (conv2_1) +layer { name: "conv2_1" type: "Convolution" bottom: "mergeB1" top: "conv2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_1) +layer { name: "bn2_1" type: "BN" bottom: "conv2_1" top: "conv2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_1) +layer { name: "relu2_1" type: "ReLU" bottom: "conv2_1" top: "conv2_1" } + +# CONVOLUTION (conv2_2) +layer { name: "conv2_2" type: "Convolution" bottom: "conv2_1" top: "conv2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_2) +layer { name: "bn2_2" type: "BN" bottom: "conv2_2" top: "conv2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_2) +layer { name: "relu2_2" type: "ReLU" bottom: "conv2_2" top: "conv2_2" } + + +# CONVOLUTION (conv2_3) +layer { name: "conv2_3" type: "Convolution" bottom: "conv2_2" top: "conv2_3" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_3) +layer { name: "bn2_3" type: "BN" bottom: "conv2_3" top: "conv2_3" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_3) +layer { name: "relu2_3" type: "ReLU" bottom: "conv2_3" top: "conv2_3" } + + +# CONVOLUTION (conv2_4) +layer { name: "conv2_4" type: "Convolution" bottom: "conv2_3" top: "conv2_4" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_4) +layer { name: "bn2_4" type: "BN" bottom: "conv2_4" top: "conv2_4" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_4) +layer { name: "relu2_4" type: "ReLU" bottom: "conv2_4" top: "conv2_4" } + + +# CONVOLUTION (conv2_5) +layer { name: "conv2_5" type: "Convolution" bottom: "conv2_4" top: "conv2_5" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_5) +layer { name: "bn2_5" type: "BN" bottom: "conv2_5" top: "conv2_5" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_5) +layer { name: "relu2_5" type: "ReLU" bottom: "conv2_5" top: "conv2_5" } + + +# CONVOLUTION (conv2_6) +layer { name: "conv2_6" type: "Convolution" bottom: "conv2_5" top: "conv2_6" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_6) +layer { name: "bn2_6" type: "BN" bottom: "conv2_6" top: "conv2_6" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_6) +layer { name: "relu2_6" type: "ReLU" bottom: "conv2_6" top: "conv2_6" } + + +# CONVOLUTION (conv2_7) +layer { name: "conv2_7" type: "Convolution" bottom: "conv2_6" top: "conv2_7" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_7) +layer { name: "bn2_7" type: "BN" bottom: "conv2_7" top: "conv2_7" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_7) +layer { name: "relu2_7" type: "ReLU" bottom: "conv2_7" top: "conv2_7" } + + +# CONVOLUTION (conv2_8) +layer { name: "conv2_8" type: "Convolution" bottom: "conv2_7" top: "conv2_8" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_8) +layer { name: "bn2_8" type: "BN" bottom: "conv2_8" top: "conv2_8" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_8) +layer { name: "relu2_8" type: "ReLU" bottom: "conv2_8" top: "conv2_8" } + + +# CONVOLUTION (conv2_9) +layer { name: "conv2_9" type: "Convolution" bottom: "conv2_8" top: "conv2_9" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_9) +layer { name: "bn2_9" type: "BN" bottom: "conv2_9" top: "conv2_9" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_9) +layer { name: "relu2_9" type: "ReLU" bottom: "conv2_9" top: "conv2_9" } + +# UNPOOLING (unpool1) +layer { + name: "unpool1" type: "Unpooling" bottom: "conv2_9" bottom: "pool1_mask" top: "unpool1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# H x W x 64 + +# DECONVOLUTION (deconv1) +layer { name: "deconv1" type: "Deconvolution" bottom: "unpool1" top: "pred" + param { lr_mult: 0.1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 1 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W X 1 + + +### L2 LOSS ### + +layer{ + name: "rec_error" + type: "EuclideanLoss" + bottom: "pred" + bottom: "label" + top: "L2_loss" +} \ No newline at end of file diff --git a/training/relighting/make_bn.prototxt b/training/relighting/make_bn.prototxt new file mode 100755 index 0000000..4d41af6 --- /dev/null +++ b/training/relighting/make_bn.prototxt @@ -0,0 +1,522 @@ +name: "exp_relighting" + +layer { + name: "data" + type: "ImageSegData" + top: "data" + top: "label" + transform_param { + mirror: true + crop_size: 224 + } + image_data_param { + source: "../Data/multipie/imgset/train_relighting.txt" + batch_size: 38 + shuffle: true + new_height: 250 + new_width: 250 + root_folder: "../Data/multipie" + is_color: true + is_label_color: true + label_type: PIXEL + } +} + +# H x W x 3 + +# CONVOLUTION (conv1) +layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W x 64 + +# BN (bn1) +layer { name: "bn1" type: "BN" bottom: "conv1" top: "conv1" + top: "bn1-mean" top: "bn1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu1) +layer { name: "relu1" type: "ReLU" bottom: "conv1" top: "conv1" } + +# POOLING (pool1) +layer { + name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" top: "pool1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/2) x (W/2) X 64 + +# Introduce Branch 1 Here + +# CONVOLUTION (convB1_1) +layer { name: "convB1_1" type: "Convolution" bottom: "pool1" top: "convB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bnB1_1) +layer { name: "bnB1_1" type: "BN" bottom: "convB1_1" top: "convB1_1" + top: "bnB1_1-mean" top: "bnB1_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_1) +layer { name: "reluB1_1" type: "ReLU" bottom: "convB1_1" top: "convB1_1" } + +# POOLING (poolB1) +layer { + name: "poolB1" type: "Pooling" bottom: "convB1_1" top: "poolB1" top: "poolB1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/4) x (W/4) X 64 + +# Introduce Branch 2 Here + +# CONVOLUTION (convB2_1) +layer { name: "convB2_1" type: "Convolution" bottom: "poolB1" top: "convB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB2_1) +layer { name: "bnB2_1" type: "BN" bottom: "convB2_1" top: "convB2_1" + top: "bnB2_1-mean" top: "bnB2_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_1) +layer { name: "reluB2_1" type: "ReLU" bottom: "convB2_1" top: "convB2_1" } + +# POOLING (poolB2) +layer { + name: "poolB2" type: "Pooling" bottom: "convB2_1" top: "poolB2" top: "poolB2_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/8) x (W/8) X 64 + +# Introduce Branch 3 Here + +# CONVOLUTION (convB3_1) +layer { name: "convB3_1" type: "Convolution" bottom: "poolB2" top: "convB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB3_1) +layer { name: "bnB3_1" type: "BN" bottom: "convB3_1" top: "convB3_1" + top: "bnB3_1-mean" top: "bnB3_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_1) +layer { name: "reluB3_1" type: "ReLU" bottom: "convB3_1" top: "convB3_1" } + +# POOLING (poolB3) +layer { + name: "poolB3" type: "Pooling" bottom: "convB3_1" top: "poolB3" top: "poolB3_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/16) x (W/16) X 64 + +# CONVOLUTION (convB3_2) +layer { name: "convB3_2" type: "Convolution" bottom: "poolB3" top: "convB3_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB3_2) +layer { name: "bnB3_2" type: "BN" bottom: "convB3_2" top: "convB3_2" + top: "bnB3_2-mean" top: "bnB3_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_2) +layer { name: "reluB3_2" type: "ReLU" bottom: "convB3_2" top: "convB3_2" } + + +# UNPOOLING (unpoolB3) +layer { + name: "unpoolB3" type: "Unpooling" bottom: "convB3_2" bottom: "poolB3_mask" top: "unpoolB3" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/8) x (W/8) X 64 + +# DECONVOLUTION (deconvB3_1) +layer { name: "deconvB3_1" type: "Deconvolution" bottom: "unpoolB3" top: "deconvB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (debnB3_1) +layer { name: "debnB3_1" type: "BN" bottom: "deconvB3_1" top: "deconvB3_1" + top: "debnB3_1-mean" top: "debnB3_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB3_1) +layer { name: "dereluB3_1" type: "ReLU" bottom: "deconvB3_1" top: "deconvB3_1" } + + + +# Merge Branch 3 Here +# CONCAT (mergeB3) +layer { + name: "mergeB3" + bottom: "poolB2" + bottom: "deconvB3_1" + top: "mergeB3" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/8) x (W/8) X 128 + +# CONVOLUTION (convB2_2) +layer { name: "convB2_2" type: "Convolution" bottom: "mergeB3" top: "convB2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB2_2) +layer { name: "bnB2_2" type: "BN" bottom: "convB2_2" top: "convB2_2" + top: "bnB2_2-mean" top: "bnB2_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_2) +layer { name: "reluB2_2" type: "ReLU" bottom: "convB2_2" top: "convB2_2" } + + +# UNPOOLING (unpoolB2) +layer { + name: "unpoolB2" type: "Unpooling" bottom: "convB2_2" bottom: "poolB2_mask" top: "unpoolB2" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/4) x (W/4) X 64 + +# DECONVOLUTION (deconvB2_1) +layer { name: "deconvB2_1" type: "Deconvolution" bottom: "unpoolB2" top: "deconvB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (debnB2_1) +layer { name: "debnB2_1" type: "BN" bottom: "deconvB2_1" top: "deconvB2_1" + top: "debnB2_1-mean" top: "debnB2_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB2_1) +layer { name: "dereluB2_1" type: "ReLU" bottom: "deconvB2_1" top: "deconvB2_1" } + + + +# Merge Branch 2 Here +# CONCAT (mergeB2) +layer { + name: "mergeB2" + bottom: "poolB1" + bottom: "deconvB2_1" + top: "mergeB2" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/4) x (W/4) X 128 + +# CONVOLUTION (convB1_2) +layer { name: "convB1_2" type: "Convolution" bottom: "mergeB2" top: "convB1_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB1_2) +layer { name: "bnB1_2" type: "BN" bottom: "convB1_2" top: "convB1_2" + top: "bnB1_2-mean" top: "bnB1_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_2) +layer { name: "reluB1_2" type: "ReLU" bottom: "convB1_2" top: "convB1_2" } + + +# UNPOOLING (unpoolB1) +layer { + name: "unpoolB1" type: "Unpooling" bottom: "convB1_2" bottom: "poolB1_mask" top: "unpoolB1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/2) x (W/2) X 64 + +# DECONVOLUTION (deconvB1_1) +layer { name: "deconvB1_1" type: "Deconvolution" bottom: "unpoolB1" top: "deconvB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (debnB1_1) +layer { name: "debnB1_1" type: "BN" bottom: "deconvB1_1" top: "deconvB1_1" + top: "debnB1_1-mean" top: "debnB1_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB1_1) +layer { name: "dereluB1_1" type: "ReLU" bottom: "deconvB1_1" top: "deconvB1_1" } + + +# Merge Branch 1 Here +# CONCAT (mergeB1) +layer { + name: "mergeB1" + bottom: "pool1" + bottom: "deconvB1_1" + top: "mergeB1" + type: "Concat" + concat_param { + axis: 1 + } +} + +# (H/2) x (W/2) X 128 + +# CONVOLUTION (conv2_1) +layer { name: "conv2_1" type: "Convolution" bottom: "mergeB1" top: "conv2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_1) +layer { name: "bn2_1" type: "BN" bottom: "conv2_1" top: "conv2_1" + top: "bn2_1-mean" top: "bn2_1-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_1) +layer { name: "relu2_1" type: "ReLU" bottom: "conv2_1" top: "conv2_1" } + +# CONVOLUTION (conv2_2) +layer { name: "conv2_2" type: "Convolution" bottom: "conv2_1" top: "conv2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_2) +layer { name: "bn2_2" type: "BN" bottom: "conv2_2" top: "conv2_2" + top: "bn2_2-mean" top: "bn2_2-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_2) +layer { name: "relu2_2" type: "ReLU" bottom: "conv2_2" top: "conv2_2" } + + +# CONVOLUTION (conv2_3) +layer { name: "conv2_3" type: "Convolution" bottom: "conv2_2" top: "conv2_3" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_3) +layer { name: "bn2_3" type: "BN" bottom: "conv2_3" top: "conv2_3" + top: "bn2_3-mean" top: "bn2_3-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_3) +layer { name: "relu2_3" type: "ReLU" bottom: "conv2_3" top: "conv2_3" } + + +# CONVOLUTION (conv2_4) +layer { name: "conv2_4" type: "Convolution" bottom: "conv2_3" top: "conv2_4" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_4) +layer { name: "bn2_4" type: "BN" bottom: "conv2_4" top: "conv2_4" + top: "bn2_4-mean" top: "bn2_4-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_4) +layer { name: "relu2_4" type: "ReLU" bottom: "conv2_4" top: "conv2_4" } + + +# CONVOLUTION (conv2_5) +layer { name: "conv2_5" type: "Convolution" bottom: "conv2_4" top: "conv2_5" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_5) +layer { name: "bn2_5" type: "BN" bottom: "conv2_5" top: "conv2_5" + top: "bn2_5-mean" top: "bn2_5-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_5) +layer { name: "relu2_5" type: "ReLU" bottom: "conv2_5" top: "conv2_5" } + + +# CONVOLUTION (conv2_6) +layer { name: "conv2_6" type: "Convolution" bottom: "conv2_5" top: "conv2_6" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_6) +layer { name: "bn2_6" type: "BN" bottom: "conv2_6" top: "conv2_6" + top: "bn2_6-mean" top: "bn2_6-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_6) +layer { name: "relu2_6" type: "ReLU" bottom: "conv2_6" top: "conv2_6" } + + +# CONVOLUTION (conv2_7) +layer { name: "conv2_7" type: "Convolution" bottom: "conv2_6" top: "conv2_7" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_7) +layer { name: "bn2_7" type: "BN" bottom: "conv2_7" top: "conv2_7" + top: "bn2_7-mean" top: "bn2_7-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_7) +layer { name: "relu2_7" type: "ReLU" bottom: "conv2_7" top: "conv2_7" } + + +# CONVOLUTION (conv2_8) +layer { name: "conv2_8" type: "Convolution" bottom: "conv2_7" top: "conv2_8" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_8) +layer { name: "bn2_8" type: "BN" bottom: "conv2_8" top: "conv2_8" + top: "bn2_8-mean" top: "bn2_8-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_8) +layer { name: "relu2_8" type: "ReLU" bottom: "conv2_8" top: "conv2_8" } + + +# CONVOLUTION (conv2_9) +layer { name: "conv2_9" type: "Convolution" bottom: "conv2_8" top: "conv2_9" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_9) +layer { name: "bn2_9" type: "BN" bottom: "conv2_9" top: "conv2_9" + top: "bn2_9-mean" top: "bn2_9-var" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_9) +layer { name: "relu2_9" type: "ReLU" bottom: "conv2_9" top: "conv2_9" } + + +# UNPOOLING (unpool1) +layer { + name: "unpool1" type: "Unpooling" bottom: "conv2_9" bottom: "pool1_mask" top: "unpool1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + + +# H x W x 64 + +# DECONVOLUTION (deconv1) +layer { name: "deconv1" type: "Deconvolution" bottom: "unpool1" top: "pred" + param { lr_mult: 0.1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 3 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W X 3 diff --git a/training/relighting/make_bn.py b/training/relighting/make_bn.py new file mode 100755 index 0000000..a10b50d --- /dev/null +++ b/training/relighting/make_bn.py @@ -0,0 +1,182 @@ + +### User Configuration + +iteration = 475 +gpu_id_num = 0 + +## path configuration +caffe_root = '../caffe' +script_path = '.' +caffe_model = script_path + '/make_bn.prototxt' +caffe_weight = script_path + '/train_curr.caffemodel' +caffe_inference_weight = script_path + '/train_curr_inference.caffemodel' + +## modify this definition according to model definition +bn_blobs= ['conv1', + 'convB1_1', + 'convB2_1', + 'convB3_1', + 'convB3_2', + 'deconvB3_1', + 'convB2_2', + 'deconvB2_1', + 'convB1_2', + 'deconvB1_1', + 'conv2_1', + 'conv2_2', + 'conv2_3', + 'conv2_4', + 'conv2_5', + 'conv2_6', + 'conv2_7', + 'conv2_8', + 'conv2_9'] +bn_layers=['bn1', + 'bnB1_1', + 'bnB2_1', + 'bnB3_1', + 'bnB3_2', + 'debnB3_1', + 'bnB2_2', + 'debnB2_1', + 'bnB1_2', + 'debnB1_1', + 'bn2_1', + 'bn2_2', + 'bn2_3', + 'bn2_4', + 'bn2_5', + 'bn2_6', + 'bn2_7', + 'bn2_8', + 'bn2_9'] +bn_means= ['bn1-mean', + 'bnB1_1-mean', + 'bnB2_1-mean', + 'bnB3_1-mean', + 'bnB3_2-mean', + 'debnB3_1-mean', + 'bnB2_2-mean', + 'debnB2_1-mean', + 'bnB1_2-mean', + 'debnB1_1-mean', + 'bn2_1-mean', + 'bn2_2-mean', + 'bn2_3-mean', + 'bn2_4-mean', + 'bn2_5-mean', + 'bn2_6-mean', + 'bn2_7-mean', + 'bn2_8-mean', + 'bn2_9-mean'] +bn_vars = ['bn1-var', + 'bnB1_1-var', + 'bnB2_1-var', + 'bnB3_1-var', + 'bnB3_2-var', + 'debnB3_1-var', + 'bnB2_2-var', + 'debnB2_1-var', + 'bnB1_2-var', + 'debnB1_1-var', + 'bn2_1-var', + 'bn2_2-var', + 'bn2_3-var', + 'bn2_4-var', + 'bn2_5-var', + 'bn2_6-var', + 'bn2_7-var', + 'bn2_8-var', + 'bn2_9-var'] + + +### start generate caffemodel + +print 'start generating BN-testable caffemodel' +print 'caffe_root: %s' % caffe_root +print 'script_path: %s' % script_path +print 'caffe_model: %s' % caffe_model +print 'caffe_weight: %s' % caffe_weight +print 'caffe_inference_weight: %s' % caffe_inference_weight + +import numpy as np + +import sys +sys.path.append(caffe_root+'/python') +import caffe +from caffe.proto import caffe_pb2 + +# caffe.set_mode_cpu() +caffe.set_mode_gpu() +caffe.set_device(gpu_id_num) +net = caffe.Net(caffe_model, caffe_weight, caffe.TEST) + +def forward_once(net): + start_ind = 0 + end_ind = len(net.layers) - 1 + net._forward(start_ind, end_ind) + return {out: net.blobs[out].data for out in net.outputs} + +print net.params.keys() + +res = forward_once(net) + +bn_avg_mean = {bn_mean: np.squeeze(res[bn_mean]).copy() for bn_mean in bn_means} +bn_avg_var = {bn_var: np.squeeze(res[bn_var]).copy() for bn_var in bn_vars} + +cnt = 1 + +for i in range(0, iteration): + res = forward_once(net) + for bn_mean in bn_means: + bn_avg_mean[bn_mean] = bn_avg_mean[bn_mean] + np.squeeze(res[bn_mean]) + for bn_var in bn_vars: + bn_avg_var[bn_var] = bn_avg_var[bn_var] + np.squeeze(res[bn_var]) + + cnt += 1 + print 'progress: %d/%d' % (i, iteration) + +## compute average +for bn_mean in bn_means: + bn_avg_mean[bn_mean] /= cnt +for bn_var in bn_vars: + bn_avg_var[bn_var] /= cnt + +for i in range(0, len(bn_vars)): + m = np.prod(net.blobs[bn_blobs[i]].data.shape) / np.prod(bn_avg_var[bn_vars[i]].shape) + bn_avg_var[bn_vars[i]] *= (m/(m-1)) + +scale_data = {bn_layer: np.squeeze(net.params[bn_layer][0].data) for bn_layer in bn_layers} +shift_data = {bn_layer: np.squeeze(net.params[bn_layer][1].data) for bn_layer in bn_layers} + +var_eps = 1e-9 + +new_scale_data = {} +new_shift_data = {} +for i in range(0, len(bn_layers)): + gamma = scale_data[bn_layers[i]] + beta = shift_data[bn_layers[i]] + Ex = bn_avg_mean[bn_means[i]] + Varx = bn_avg_var[bn_vars[i]] + new_gamma = gamma / np.sqrt(Varx + var_eps) + new_beta = beta - (gamma * Ex / np.sqrt(Varx + var_eps)) + + new_scale_data[bn_layers[i]] = new_gamma + new_shift_data[bn_layers[i]] = new_beta + +print new_scale_data.keys() +print new_shift_data.keys() + +## assign computed new scale and shift values to net.params +for i in range(0, len(bn_layers)): + net.params[bn_layers[i]][0].data[...] = new_scale_data[bn_layers[i]].reshape(net.params[bn_layers[i]][0].data.shape) + net.params[bn_layers[i]][1].data[...] = new_shift_data[bn_layers[i]].reshape(net.params[bn_layers[i]][1].data.shape) + +print 'start saving model' + +net.save(caffe_inference_weight) + +print 'done' + + + diff --git a/training/relighting/run_bn.sh b/training/relighting/run_bn.sh new file mode 100755 index 0000000..e4afde2 --- /dev/null +++ b/training/relighting/run_bn.sh @@ -0,0 +1,41 @@ +#!/bin/bash + +# Pull most recent snapshot and prepare it for inference by computing global +# mean/variance for all the batch-normalization layers by passing training +# data through the network. If the most recent snapshot is in +# ./snapshot/trn_iter_[N].caffemodel, then the final inference-ready weights +# will be saved in ./tst_[N].caffemodel +# Training Data is expected in ../Data/multipie (can be a symlink) +# ImageList is expected in ../Data/multipie/imgset/train_relighting.txt +# (relative to root ../Data/multipie) + +CAFFEDIR="../caffe" +# Choose most recent snapshot for inference +CURR_SNAPSHOT=`ls ./snapshot/trn_iter_*.caffemodel -t | head -n 1` +F=${CURR_SNAPSHOT/"./snapshot/trn_iter_"/} +ITER=${F/".caffemodel"/} +ITER=${ITER/"*"/} +P_ITER=`printf "%08g\n" ${ITER}` + +WEIGHTSA_TMP=./train_curr.caffemodel +WEIGHTSA_INF_TMP=./train_curr_inference.caffemodel +WEIGHTSA_INF=./tst_${P_ITER}.caffemodel + +echo $WEIGHTSA +echo $WEIGHTSA_INF + +# LOAD CAFFE DEPENDENCIES TO LD_LIBRARY_PATH +source ${CAFFEDIR}/load_caffe_dependencies.sh + +rm -f $WEIGHTSA_TMP && cp -f $CURR_SNAPSHOT $WEIGHTSA_TMP + +# ADJUST BATCH NORMALIZATION FOR INFERENCE + +# Start python virtual env +# source /scratch0/venkai/dependencies/env/bin/activate +python make_bn.py +# deactivate + +rm -f ${WEIGHTSA_TMP} +mv ${WEIGHTSA_INF_TMP} ${WEIGHTSA_INF} + diff --git a/training/relighting/solver.prototxt b/training/relighting/solver.prototxt new file mode 100755 index 0000000..59d01b5 --- /dev/null +++ b/training/relighting/solver.prototxt @@ -0,0 +1,13 @@ +net: "./train.prototxt" +base_lr: 0.0000001 +lr_policy: "step" +stepsize: 12000 +gamma: 0.1 +iter_size: 2 +display: 10 +max_iter: 40000 +momentum: 0.9 +weight_decay: 0.0005 +snapshot: 500 +snapshot_prefix: "./snapshot/trn" +solver_mode: GPU diff --git a/training/relighting/start_train.sh b/training/relighting/start_train.sh new file mode 100755 index 0000000..9bceb28 --- /dev/null +++ b/training/relighting/start_train.sh @@ -0,0 +1,30 @@ +#!/bin/bash + +# Start training an RBDN model for relighting. You can pause training at any moment +# with Ctrl+C and most recent snapshot will be saved in +# ./snapshot/trn_iter_[*].solverstate +# Running ./start_train.sh again will automatically resume from that snapshot. +# Training Data is expected in ../Data/multipie (can be a symlink) +# ImageList is expected in ../Data/multipie/imgset/train_relighting.txt +# (relative to root ../Data/multipie) + +CAFFEDIR="../caffe" +LOGDIR=./training_log +CAFFE=${CAFFEDIR}/build/tools/caffe +SOLVER=./solver.prototxt + +# LOAD CAFFE DEPENDENCIES TO LD_LIBRARY_PATH +source ${CAFFEDIR}/load_caffe_dependencies.sh + +mkdir -p $LOGDIR +mkdir -p snapshot + +# Check if snapshot exists; if not train from scratch. +if [ "$(ls -A ./snapshot/)" ]; then + CURR_SNAPSHOT=`ls ./snapshot/*solverstate -t | head -n 1` + echo "Resuming training from snapshot ${CURR_SNAPSHOT}" + echo "If you want to train from scratch, delete everything in ./snapshot" + GLOG_log_dir=$LOGDIR $CAFFE train -snapshot $CURR_SNAPSHOT -solver $SOLVER -gpu 0 +else + GLOG_log_dir=$LOGDIR $CAFFE train -solver $SOLVER -gpu 0 +fi diff --git a/training/relighting/train.prototxt b/training/relighting/train.prototxt new file mode 100755 index 0000000..ea45803 --- /dev/null +++ b/training/relighting/train.prototxt @@ -0,0 +1,515 @@ +name: "exp_relighting" + +layer { + name: "data" + type: "ImageSegData" + top: "data" + top: "label" + include { + phase: TRAIN + } + transform_param { + mirror: true + crop_size: 224 + } + image_data_param { + source: "../Data/multipie/imgset/train_relighting.txt" + batch_size: 32 + shuffle: true + new_height: 250 + new_width: 250 + root_folder: "../Data/multipie" + is_color: true + is_label_color: true + label_type: PIXEL + } +} + +# H x W x 3 + +# CONVOLUTION (conv1) +layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W x 64 + +# BN (bn1) +layer { name: "bn1" type: "BN" bottom: "conv1" top: "conv1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu1) +layer { name: "relu1" type: "ReLU" bottom: "conv1" top: "conv1" } + +# POOLING (pool1) +layer { + name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" top: "pool1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/2) x (W/2) X 64 + +# Introduce Branch 1 Here + +# CONVOLUTION (convB1_1) +layer { name: "convB1_1" type: "Convolution" bottom: "pool1" top: "convB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bnB1_1) +layer { name: "bnB1_1" type: "BN" bottom: "convB1_1" top: "convB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_1) +layer { name: "reluB1_1" type: "ReLU" bottom: "convB1_1" top: "convB1_1" } + +# POOLING (poolB1) +layer { + name: "poolB1" type: "Pooling" bottom: "convB1_1" top: "poolB1" top: "poolB1_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/4) x (W/4) X 64 + +# Introduce Branch 2 Here + +# CONVOLUTION (convB2_1) +layer { name: "convB2_1" type: "Convolution" bottom: "poolB1" top: "convB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB2_1) +layer { name: "bnB2_1" type: "BN" bottom: "convB2_1" top: "convB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_1) +layer { name: "reluB2_1" type: "ReLU" bottom: "convB2_1" top: "convB2_1" } + +# POOLING (poolB2) +layer { + name: "poolB2" type: "Pooling" bottom: "convB2_1" top: "poolB2" top: "poolB2_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/8) x (W/8) X 64 + +# Introduce Branch 3 Here + +# CONVOLUTION (convB3_1) +layer { name: "convB3_1" type: "Convolution" bottom: "poolB2" top: "convB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB3_1) +layer { name: "bnB3_1" type: "BN" bottom: "convB3_1" top: "convB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_1) +layer { name: "reluB3_1" type: "ReLU" bottom: "convB3_1" top: "convB3_1" } + +# POOLING (poolB3) +layer { + name: "poolB3" type: "Pooling" bottom: "convB3_1" top: "poolB3" top: "poolB3_mask" + pooling_param { pool: MAX kernel_size: 2 stride: 2 } +} + +# (H/16) x (W/16) X 64 + +# CONVOLUTION (convB3_2) +layer { name: "convB3_2" type: "Convolution" bottom: "poolB3" top: "convB3_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/16) x (W/16) X 64 + +# BN (bnB3_2) +layer { name: "bnB3_2" type: "BN" bottom: "convB3_2" top: "convB3_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB3_2) +layer { name: "reluB3_2" type: "ReLU" bottom: "convB3_2" top: "convB3_2" } + + +# UNPOOLING (unpoolB3) +layer { + name: "unpoolB3" type: "Unpooling" bottom: "convB3_2" bottom: "poolB3_mask" top: "unpoolB3" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/8) x (W/8) X 64 + +# DECONVOLUTION (deconvB3_1) +layer { name: "deconvB3_1" type: "Deconvolution" bottom: "unpoolB3" top: "deconvB3_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (debnB3_1) +layer { name: "debnB3_1" type: "BN" bottom: "deconvB3_1" top: "deconvB3_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB3_1) +layer { name: "dereluB3_1" type: "ReLU" bottom: "deconvB3_1" top: "deconvB3_1" } + + + +# Merge Branch 3 Here +# CONCAT (mergeB3) +layer { + name: "mergeB3" + bottom: "poolB2" + bottom: "deconvB3_1" + top: "mergeB3" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/8) x (W/8) X 128 + +# CONVOLUTION (convB2_2) +layer { name: "convB2_2" type: "Convolution" bottom: "mergeB3" top: "convB2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/8) x (W/8) X 64 + +# BN (bnB2_2) +layer { name: "bnB2_2" type: "BN" bottom: "convB2_2" top: "convB2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB2_2) +layer { name: "reluB2_2" type: "ReLU" bottom: "convB2_2" top: "convB2_2" } + + +# UNPOOLING (unpoolB2) +layer { + name: "unpoolB2" type: "Unpooling" bottom: "convB2_2" bottom: "poolB2_mask" top: "unpoolB2" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/4) x (W/4) X 64 + +# DECONVOLUTION (deconvB2_1) +layer { name: "deconvB2_1" type: "Deconvolution" bottom: "unpoolB2" top: "deconvB2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (debnB2_1) +layer { name: "debnB2_1" type: "BN" bottom: "deconvB2_1" top: "deconvB2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB2_1) +layer { name: "dereluB2_1" type: "ReLU" bottom: "deconvB2_1" top: "deconvB2_1" } + + + +# Merge Branch 2 Here +# CONCAT (mergeB2) +layer { + name: "mergeB2" + bottom: "poolB1" + bottom: "deconvB2_1" + top: "mergeB2" + type: "Concat" + concat_param { + axis: 1 + } +} +# (H/4) x (W/4) X 128 + +# CONVOLUTION (convB1_2) +layer { name: "convB1_2" type: "Convolution" bottom: "mergeB2" top: "convB1_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/4) x (W/4) X 64 + +# BN (bnB1_2) +layer { name: "bnB1_2" type: "BN" bottom: "convB1_2" top: "convB1_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (reluB1_2) +layer { name: "reluB1_2" type: "ReLU" bottom: "convB1_2" top: "convB1_2" } + + +# UNPOOLING (unpoolB1) +layer { + name: "unpoolB1" type: "Unpooling" bottom: "convB1_2" bottom: "poolB1_mask" top: "unpoolB1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# (H/2) x (W/2) X 64 + +# DECONVOLUTION (deconvB1_1) +layer { name: "deconvB1_1" type: "Deconvolution" bottom: "unpoolB1" top: "deconvB1_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (debnB1_1) +layer { name: "debnB1_1" type: "BN" bottom: "deconvB1_1" top: "deconvB1_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (dereluB1_1) +layer { name: "dereluB1_1" type: "ReLU" bottom: "deconvB1_1" top: "deconvB1_1" } + + +# Merge Branch 1 Here +# CONCAT (mergeB1) +layer { + name: "mergeB1" + bottom: "pool1" + bottom: "deconvB1_1" + top: "mergeB1" + type: "Concat" + concat_param { + axis: 1 + } +} + +# (H/2) x (W/2) X 128 + +# CONVOLUTION (conv2_1) +layer { name: "conv2_1" type: "Convolution" bottom: "mergeB1" top: "conv2_1" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_1) +layer { name: "bn2_1" type: "BN" bottom: "conv2_1" top: "conv2_1" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_1) +layer { name: "relu2_1" type: "ReLU" bottom: "conv2_1" top: "conv2_1" } + +# CONVOLUTION (conv2_2) +layer { name: "conv2_2" type: "Convolution" bottom: "conv2_1" top: "conv2_2" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_2) +layer { name: "bn2_2" type: "BN" bottom: "conv2_2" top: "conv2_2" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_2) +layer { name: "relu2_2" type: "ReLU" bottom: "conv2_2" top: "conv2_2" } + + +# CONVOLUTION (conv2_3) +layer { name: "conv2_3" type: "Convolution" bottom: "conv2_2" top: "conv2_3" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_3) +layer { name: "bn2_3" type: "BN" bottom: "conv2_3" top: "conv2_3" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_3) +layer { name: "relu2_3" type: "ReLU" bottom: "conv2_3" top: "conv2_3" } + + +# CONVOLUTION (conv2_4) +layer { name: "conv2_4" type: "Convolution" bottom: "conv2_3" top: "conv2_4" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_4) +layer { name: "bn2_4" type: "BN" bottom: "conv2_4" top: "conv2_4" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_4) +layer { name: "relu2_4" type: "ReLU" bottom: "conv2_4" top: "conv2_4" } + + +# CONVOLUTION (conv2_5) +layer { name: "conv2_5" type: "Convolution" bottom: "conv2_4" top: "conv2_5" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_5) +layer { name: "bn2_5" type: "BN" bottom: "conv2_5" top: "conv2_5" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_5) +layer { name: "relu2_5" type: "ReLU" bottom: "conv2_5" top: "conv2_5" } + + +# CONVOLUTION (conv2_6) +layer { name: "conv2_6" type: "Convolution" bottom: "conv2_5" top: "conv2_6" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_6) +layer { name: "bn2_6" type: "BN" bottom: "conv2_6" top: "conv2_6" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_6) +layer { name: "relu2_6" type: "ReLU" bottom: "conv2_6" top: "conv2_6" } + + +# CONVOLUTION (conv2_7) +layer { name: "conv2_7" type: "Convolution" bottom: "conv2_6" top: "conv2_7" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_7) +layer { name: "bn2_7" type: "BN" bottom: "conv2_7" top: "conv2_7" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_7) +layer { name: "relu2_7" type: "ReLU" bottom: "conv2_7" top: "conv2_7" } + + +# CONVOLUTION (conv2_8) +layer { name: "conv2_8" type: "Convolution" bottom: "conv2_7" top: "conv2_8" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_8) +layer { name: "bn2_8" type: "BN" bottom: "conv2_8" top: "conv2_8" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_8) +layer { name: "relu2_8" type: "ReLU" bottom: "conv2_8" top: "conv2_8" } + + +# CONVOLUTION (conv2_9) +layer { name: "conv2_9" type: "Convolution" bottom: "conv2_8" top: "conv2_9" + param { lr_mult: 1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 64 pad: 1 kernel_size: 3 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# (H/2) x (W/2) X 64 + +# BN (bn2_9) +layer { name: "bn2_9" type: "BN" bottom: "conv2_9" top: "conv2_9" + bn_param { scale_filler { type: "constant" value: 1 } + shift_filler { type: "constant" value: 0.001 } } } + +# RELU (relu2_9) +layer { name: "relu2_9" type: "ReLU" bottom: "conv2_9" top: "conv2_9" } + +# UNPOOLING (unpool1) +layer { + name: "unpool1" type: "Unpooling" bottom: "conv2_9" bottom: "pool1_mask" top: "unpool1" + unpooling_param { unpool: MAX kernel_size: 2 stride: 2 } } + +# H x W x 64 + +# DECONVOLUTION (deconv1) +layer { name: "deconv1" type: "Deconvolution" bottom: "unpool1" top: "pred" + param { lr_mult: 0.1 decay_mult: 1 } + param { lr_mult: 0.1 decay_mult: 1 } + convolution_param { num_output: 3 pad: 4 kernel_size: 9 + weight_filler { type: "gaussian" std: 0.001 } + bias_filler { type: "constant" } } } + +# H x W X 3 + + +### L2 LOSS ### + +layer{ + name: "rec_error" + type: "EuclideanLoss" + bottom: "pred" + bottom: "label" + top: "L2_loss" +} \ No newline at end of file