filterbasis.tex

% !TEX root = thesis.tex
\documentclass[thesis]{subfiles}

\begin{document}
	\chapter{Inter-Filter Connectivity}\label{deeproots}
	\begin{chapquote}{Marvin Minsky, \textit{Prologue: A View from 1988, Perceptrons}}
	``The marvelous powers of the brain emerge not from any single, uniformly structured connectionist network but from highly evolved arrangements of smaller, specialized networks which are interconnected in very specific ways.''
	\end{chapquote}
	%Learning Filters with Limited Channel Extents
	%It is well understood that the structure of a neural network is critical to its ability to learn from labelled training data and to generalize. Whilst it has been shown that an infinitely wide hidden layer is a universal approximator~\citep{hornik89a}, in practice wide shallow networks do not learn as well as thinner deeper ones --- as shown by recent research~\citep{Krizhevsky2012,Szegedy2014going,Simonyan2014verydeep,He2015}.
	%This does not appear to be a limitation of finite capacity, since it has been proposed that at least some \glspl{dnn}\index{DNN} are by representable by shallow networks~\citep{Ba2013dothey}. Rather it seems to be a consequence of limitations in our methods of learning the weights of \glspl{dnn}\index{DNN}.
	
	With few exceptions, state-of-the-art \glspl{cnn}\index{CNN} for image recognition are largely monolithic, with each filter operating on the \glspl{featuremap}\index{feature map} of all filters on a previous layer. Interestingly, this is in stark contrast to what we understand of biological neural networks\index{neural network}, where we see ``highly evolved arrangements of smaller, specialized networks which are interconnected in very specific ways''~\citep{minsky1988perceptrons}.
	
	Yet it has been shown that a large proportion of the learned weights in \glspl{dnn}\index{DNN} are redundant~\citep{Denil2013predicting}, a property that has been widely exploited to make neural networks\index{neural network} smaller and more computationally efficient~\citep{Szegedy2014going,Denton2014efficient}. 
	%It is unsurprising then that regularization is a critical part of training such networks using large datasets~\citep{Krizhevsky2012}. Without regularization \glspl{dnn}\index{DNN} are susceptible to overfitting. Regularization may be achieved by weight decay\index{weight decay} or dropout~\citep{Hinton2012}. Furthermore
	A carefully designed sparse network connection structure can have a regularizing effect. \Glspl{cnn}~\citep{Fuk80,Lecun1998} embody this idea, using a sparse convolutional connection structure to exploit the locality of natural image structure. In consequence, they are easier to train.
	
	In \cref{lowrankfilters} learning a low-rank spatial basis for filters was found to improve generalization while reducing the computational complexity and model size of a \gls{cnn} with only full rank filters. However, this work addressed only the spatial extents of the convolutional filters (\ie $h$ and $w$ in \cref{fig:normalconv}). In this work we will show that a similar idea can be applied to the channel extents --- \ie filter inter-connectivity --- by using \emph{filter groups\index{filter groups}}~\citep{Krizhevsky2012}. 
	
	%We show that simple alterations to state-of-the-art \gls{cnn} architectures can drastically reduce computational cost and model size without compromising accuracy.
	
	% Try to find a creative commons version of this
	%\begin{figure}[t]
	%\centering
	%\includegraphics[width=0.25\columnwidth, page=1]{figs/tree_root.jpg}
	%\caption{This work introduces the idea of using hierarchical filter group\index{filter groups}ing to create \glsfmttext{cnn} connection structures that resemble tree roots. Applying the technique to existing state-of-the-art networks allows improved efficiency without compromising accuracy.}
	%\label{fig:tree}
	%\end{figure}
	
	% Maybe we need something like this here too, don't know
	%It has been shown that reducing the co-adaption of filters is beneficial to generalization in \glspl{dnn}\index{DNN}~\citep{Hinton2012,goodfellow2013maxout,Cogswell2016}. Co-adaption occurs when hidden layer filters (or neurons\index{neuron}) rely on one another to fit training data, such that their outputs become highly correlated. However, instead of using a modified loss, regularization penalty, or randomized network connectivity during training to prevent co-adaption of features, we take a much more direct approach. We use hierarchical filter groups\index{filter groups} (see \cref{regularizingstructure}) to allow the network itself to learn independent filters. By restricting the connectivity between filters on subsequent layers the network is forced to learn filters of limited dependence. We allow the standard network optimization (\eg \gls{sgd}) to learn appropriate weights within this restricted architecture.
	
	In this chapter we show that simple alterations to the architecture of state-of-the-art \glspl{cnn}\index{CNN} for image recognition can drastically reduce computational cost and model size while maintaining (or even increasing) accuracy, through a novel structural prior reducing the connectivity in monolithic networks to reflect more closely the sparse, localized filter co-dependencies within a trained network.
	
	\section{Related Work}\label{previouswork}
	Most previous work on reducing the computational complexity of \glspl{cnn}\index{CNN} has focused on approximating convolutional filters in the spatial (as opposed to the channel) domain, either by using low-rank approximations~\citep{mamalet2012simplifying,journals/corr/JaderbergVZ14, conf/cvpr/RigamontiSLF13, journals/corr/LebedevGROL14}, or Fourier transform based convolution~\citep{mathieu2013fast, rippel2015spectral}. More general methods have used reduced precision number representations~\citep{1502.02551v1} or compression of previously trained models~\citep{Chen2015,Kim2016}. Here we explore methods that reduce the computational impact of the large number of filter channels within state-of-the art networks. Specifically, we consider decreasing the number of incoming connections to neurons.
	
	\begin{figure}[tbp]
		\begin{subfigure}[b]{0.95\textwidth}
			\centering
			\includegraphics[width=0.66\textwidth, page=1]{Figs/PDF/groupfig}
			\caption{Convolution}\label{fig:normalconv}
		\end{subfigure}
		~
		\begin{subfigure}[b]{0.95\textwidth}
			\centering
			\includegraphics[width=0.66\textwidth, page=2]{Figs/PDF/groupfig}
			\caption{Convolution with filter groups\index{filter groups}}\label{fig:groupedconv}
		\end{subfigure}
		\caption[Convolutional filter groups\index{filter groups}]{\textbf{Convolutional Filter groups\index{filter groups}.} (a) Convolutional filters (yellow) typically have the same channel dimension $c_1$ as the input \glspl{featuremap}\index{feature map} (gray) on which they operate. However, (b) with filter group\index{filter groups}ing, $g$ independent groups of $c_2/g$ filters operate on a fraction $c_1/g$ of the input \gls{featuremap}\index{feature map} channels, reducing filter dimensions from $h$$\times$$w$$\times$$c_1$ to $h$$\times$$w$$\times$$c_1/g$. This change does not affect the dimensions of the input and output \glspl{featuremap}\index{feature map} but significantly reduces computational complexity and the number of model parameters.}\label{fig:groupconfig}
	\end{figure}
	\begin{figure}[tbp]
		\centering
		\pgfplotstableread[col sep=comma]{rootdata/alexnetma.csv}\datatable
		\pgfplotsset{major grid style={dotted,red}}
		
		\begin{tikzpicture}
		\begin{axis}[
			width=0.95\textwidth,
			height=0.33\textheight,
			axis x line=bottom,
			ylabel=Top-5 Val.\ Error,
			xlabel=Model Parameters (\# floats),
			axis lines=left,
			enlarge x limits=0.12,
			enlarge y limits=0.1,
			grid=major,
			%xmin=0,
			ytick={0.01,0.02,...,0.21},
			ymin=0.18,ymax=0.2,
			yticklabel={\pgfmathparse{\tick*100}\pgfmathprintnumber{\pgfmathresult}\%},style={
				/pgf/number format/fixed,
				/pgf/number format/precision=1,
			},
			legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
			legend columns=2,
			\setplotcyclecat{2},
			every axis plot/.append style={fill},
		]
		\addplot+[mark=*,nodes near coords,only marks,
		point meta=explicit symbolic,
		%   x filter/.code={
		%       \ifnum\coordindex>2\def\pgfmathresult{}\fi
		%   },
		] table[meta=Network,x=Param.,y expr={1 - \thisrow{Top-5 Acc.} },]{\datatable};
		%\addplot[mark=square*,mark options={fill=blue},nodes near coords,only marks,
		%   point meta=explicit symbolic,
		%   x filter/.code={
		%       \ifnum\coordindex<3\def\pgfmathresult{}\fi
		%   },
		%] table[meta=Network,x=Param.,y expr={1 - \thisrow{Top-5 Acc.} },]{\datatable};
		%\legend{Standard AlexNet}%, Alternate Filter Grouping}
		\end{axis}
		\end{tikzpicture}
		\caption[AlexNet performance and filter groups\index{filter groups}]{\textbf{AlexNet Performance and Filter Groups.} Model Parameters \vs top-5 error for variants of the AlexNet model on \gls{ilsvrc} image classification dataset. Models with moderate numbers of filter groups\index{filter groups} have far fewer parameters, yet surprisingly maintain comparable error.}\label{fig:alexnetplots}
	\end{figure}
	
	\paragraph{AlexNet Filter groups\index{filter groups}}\label{alexnetfiltergroups} Amongst the seminal contributions made by \citet{Krizhevsky2012}~is the use of `filter groups\index{filter groups|textbf}' in the convolutional layers of a \gls{cnn} (see \cref{fig:groupconfig}). While their use of filter groups\index{filter groups} was necessitated by the practical need to sub-divide the work of training a large network across multiple \gls{gpu}s, the side effects are somewhat surprising. Specifically, the authors observe that independent filter groups\index{filter groups} learn a separation of responsibility (colour features \vs texture features) that is consistent over different random initializations. Also surprising, and not explicitly stated by \citet{Krizhevsky2012}, is the fact that the AlexNet network has approximately 57\% fewer connection weights than the corresponding network without filter groups\index{filter groups}. This is due to the reduction in the input channel dimension of the grouped convolution filters (see \cref{fig:alexnetplots}).
	Despite the large difference in the number of parameters between the models, both achieve comparable accuracy on \gls{ilsvrc} --- in fact the smaller grouped network gets $\approx1$\% lower top-5 validation error. This paper builds upon these findings and extends them to state-of-the-art networks.
	
	\paragraph{Low-dimensional Embeddings}
	\citet{Lin2013NiN} proposed a method to reduce the dimensionality of convolutional \glspl{featuremap}\index{feature map}. 
	By using relatively cheap `1$\times$1' convolutional layers (\ie layers comprising $d$ filters of size 1$\times$1$\times$$c$, where $d<c$), they learn to map \glspl{featuremap}\index{feature map} into lower-dimensional spaces, \ie to new \glspl{featuremap}\index{feature map} with fewer channels. Subsequent spatial filters operating on this lower dimensional input space require significantly less computation. This method is used in most state-of-the-art networks for image classification to reduce computation~\citep{Szegedy2014going,He2015}. Our method is complementary.
	
	\paragraph{GoogLeNet} In contrast to much other work, \citet{Szegedy2014going} propose a \gls{cnn} architecture that is highly optimized for computational efficiency. GoogLeNet uses, as a basic building block, a mixture of low-dimensional embeddings~\citep{Lin2013NiN} and heterogeneously-sized spatial filters --- collectively an \Gls{inception} module. 
	There are two distinct forms of convolutional layers in the \gls{inception}\index{inception} module, low-dimensional embeddings (1$\times$1) and spatial (3$\times$3, 5$\times $5). GoogLeNet keeps large, expensive spatial convolutions (\ie 5$\times$5) to a minimum by using few of these filters, using more 3$\times$3 convolutions, and even more 1$\times$1 filters. The motivation is that most of the convolutional filters respond to localized patterns in a small receptive field, with few requiring a larger receptive field. The number of filters in each successive \gls{inception}\index{inception} module increases slowly with decreasing \gls{featuremap}\index{feature map} size, in order to maintain computational performance. GoogLeNet is by far the most efficient state-of-the-art network for \gls{ilsvrc}, achieving near state-of-the-art accuracy with the lowest computation/model size. However, we will show that even such an efficient and optimized network architecture benefits from our method.
	
	\paragraph{Low-Rank Approximations}
	Various authors have suggested approximating learned convolutional filters using tensor decomposition \citep{journals/corr/JaderbergVZ14,journals/corr/LebedevGROL14,Kim2016}. For example, \citet{journals/corr/JaderbergVZ14} propose approximating the convolutional filters in a trained network with representations that are low-rank both in the spatial and the channel domains. This approach significantly decreases computational complexity, albeit at the expense of a small amount of accuracy. In this chapter we are not approximating an existing model's weights but creating a new network architecture with explicit structural sparsity, which is then trained from scratch.
	
	\paragraph{Learning a Basis for Filters}
	\begin{figure}[tbp]
		\centering
		\includegraphics[width=0.95\textwidth, page=3]{Figs/PDF/sparsification}
		\caption[Learning a spatial basis for filters]{\textbf{Learning a spatial basis for filters.} Learning a linear combination of mostly small, heterogeneously-sized spatial filters, as proposed in \cref{lowrankfilters}. Note that all filters operate on all $c$ channels of the input \gls{featuremap}\index{feature map}.}
		\label{fig:spatialbasis}
	\end{figure}
	Our approach is connected with that presented in \cref{lowrankfilters}, where we showed that replacing 3$\times$3$\times$c filters with linear combinations of filters with smaller spatial extent (\eg 1$\times$3$\times c$, 3$\times$1$\times c$ filters, see \cref{fig:spatialbasis}) could reduce the model size and computational complexity of state-of-the-art \glspl{cnn}\index{CNN}, while maintaining or even increasing accuracy. However, that work did not address the channel extent of the filters.
	\section{Root Architectures}
	\label{method}
	In this section we present the main contribution of our work: the use of novel sparsely-connected architectures resembling tree roots --- to decrease computational complexity and model size compared to state-of-the-art \glspl{dnn}\index{DNN} for image recognition.
	
	\paragraph{Learning a Basis for Filter Dependencies}
	It is unlikely that every filter (or neuron\index{neuron}) in a deep neural network\index{neural network} needs to depend on the output of all the filters in the previous layer. In fact, reducing filter co-dependence in \glspl{dnn}\index{DNN} has been shown to benefit generalization. For example, \citet{Hinton2012} introduced {\em dropout} for regularization of \glspl{dnn}\index{DNN}. When training a network layer with dropout, a random subset of neurons\index{neuron} is excluded from both the forward and backward pass for each mini-batch.  Furthermore, \citet{Cogswell2016} observe a correlation between the covariance of hidden unit activations and overfitting. To explicitly reduce the covariance of hidden activations, they train networks with a loss function, based on the covariance matrix of the activations in a hidden layer. 
	
	Instead of using a modified loss, regularization penalty, or randomized network connectivity during training to prevent co-adaption of features, we take a much more direct approach. We use filter groups\index{filter groups} (see \cref{fig:groupconfig}) to force the network to learn filters with only limited dependence on previous layers. Each of the filters in the filter groups\index{filter groups} is smaller in the channel extent, since it operates on only a subset of the channels of the input \gls{featuremap}\index{feature map}. 
	
	\begin{figure}[tbp]
		\centering
		\begin{subfigure}[b]{0.95\textwidth}
			\centering
			\includegraphics[width=\textwidth, page=4]{Figs/PDF/groupfig}
			\caption{Convolution with $d$ filters of shape $h\times w\times c$.}
			\label{fig:normalresnet}
		\end{subfigure}\\
		\begin{subfigure}[b]{0.95\textwidth}
			\includegraphics[width=\textwidth, page=5]{Figs/PDF/groupfig}
			\caption{Root-2 Module: Convolution with $d$ filters in $g = 2$ filter groups\index{filter groups}, of shape $h\times w\times c/2$.}
			\label{fig:rootresnet2}
		\end{subfigure}
		\begin{subfigure}[b]{0.95\textwidth}
			\includegraphics[width=\textwidth, page=6]{Figs/PDF/groupfig}
			\caption{Root-4 Module: Convolution with $d$ filters in $g = 4$ filter groups\index{filter groups}, of shape $h\times w\times c/4$.}
			\label{fig:rootresnet4}
		\end{subfigure}
		\caption[Root modules: learning a channel basis for filters]{\textbf{Root Modules.} Root modules (b), (c) compared to a typical set of convolutional layers (a) found in ResNet and other modern architectures. Grey blocks represent the \glspl{featuremap}\index{feature map} over which a layer's filters operate, while colored blocks represent the filters of each layer. 
		}
		\label{fig:rootmodule}
	\end{figure}
	This reduced connectivity also reduces computational complexity and model size since the size of filters in filter groups\index{filter groups} are reduced drastically, as is evident in \cref{fig:rootmodule}. Unlike methods for increasing the efficiency of \glspl{dnn}\index{DNN} by approximating pre-trained existing networks (see \cref{previouswork}), our models are trained from random initialization using stochastic gradient descent. This means that our method can also speed up training and, since we are not merely approximating an existing model's weights, the accuracy of the existing model is not an upper bound on accuracy of the modified model.
	
	\paragraph{Root Module}
	The basic element of our network architecture, a \emph{root module}, is shown in \cref{fig:rootmodule}. A root module has a given number of filter groups\index{filter groups}, the more filter groups\index{filter groups}, the fewer the number of connections to the previous layer's outputs. Each spatial convolutional layer is followed by a low-dimensional embedding (1$\times$1 convolution). Like in \cref{lowrankfilters}, this configuration  learns a linear combination of the basis filters (filter groups\index{filter groups}), implicitly representing a filter of full channel depth, but with limited filter dependence.
	
	\section{Results}
	Here we present image classification results obtained by replacing spatial convolutional layers within existing state-of-the-art network architectures with root modules (described in \cref{method}) .
	
	\subsection{Improving \Glsfmttext{nin} on \Glsfmttext{cifar10}}
	\Gls{nin}~\citep{Lin2013NiN} is a near state-of-the-art network for \gls{cifar10}~\citep{CIFAR10}. It is composed of 3 spatial (5$\times$5, 3$\times$3) convolutional layers with a large number of filters (192), interspersed with pairs of low-dimensional embedding (1$\times$1) layers. As a baseline, we replicated the standard NiN network architecture as described by \citet{Lin2013NiN} but used state-of-the-art training methods. We trained using random 32$\times$32 cropped and mirrored images from 4-pixel zero-padded  mean-subtracted images, as used by \citet{goodfellow2013maxout, He2015}. We also used the initialization of \citet{He2015b} and batch normalization\index{batch normalization}~\citep{Ioffe2015}. With this configuration, \gls{zca} whitening was not required to reproduce validation accuracies obtained in~\citep{Lin2013NiN}. We also did not use dropout, having found it to have little effect, presumably due to our use of batch normalization\index{batch normalization}, as suggested by \citet{Ioffe2015}.
	
	\begin{table}[tbp]
		\caption[\Glsfmttext{nin} root architectures]{\textbf{\Glsfmttext{nin} Root Architectures}. Filter groups\index{filter groups} in each convolutional layer.}
		\label{table:ninconfig}
		\centering
		%\resizebox{\textwidth}{!}{
		\begin{tabular}{@{}lccccccccc@{}}
			\toprule
			Model & \multicolumn{3}{c}{conv1} & \multicolumn{3}{c}{conv2} & \multicolumn{3}{c}{conv3} \\
			& \textit{\footnotesize a} & \textit{\footnotesize b} & \textit{\footnotesize c} & \textit{\footnotesize a} & \textit{\footnotesize b} & \textit{\footnotesize c} & \textit{\footnotesize a} & \textit{\footnotesize b} & \textit{\footnotesize c} \\
			& \textit{\footnotesize5$\times$5} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize5$\times$5} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize3$\times$3} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize1$\times$1} \\
			Orig. & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1\\
			\midrule
			root-2 & 1 & 1 & 1 & 2 & 1 & 1 & 1 & 1 & 1\\
			root-4 & 1 & 1 & 1 & 4 & 1 & 1 & 2 & 1 & 1\\
			root-8 & 1 & 1 & 1 & 8 & 1 & 1 & 4 & 1 & 1\\
			root-16 & 1 & 1 & 1 & 16 & 1 & 1 & 8 & 1 & 1\\
			\bottomrule
		\end{tabular}
		%}
	\end{table}
	\begin{table}[tp]
		\caption[\Glsfmttext{nin} \gls{cifar10} results]{\textbf{\Glsfmttext{nin} \glsfmttext{cifar10} Results}}
		\label{table:nincifarresults}
		\centering
		\pgfplotstableread[col sep=comma]{rootdata/nincifar.csv}\data
		\pgfplotstableread[col sep=comma]{rootdata/nincifar_root_s.csv}\codata
		\pgfplotstableread[col sep=comma]{rootdata/nincifar_tree_s.csv}\tdatatable
		\pgfplotstableread[col sep=comma]{rootdata/nincifar_col_s.csv}\cdatatable
		\pgfplotstablevertcat{\data}{\codata}
		\pgfplotstablevertcat{\data}{\tdatatable}
		\pgfplotstablevertcat{\data}{\cdatatable}
		%\resizebox{\textwidth}{!}{
		\pgfplotstabletypeset[
		every head row/.style={before row=\toprule,after row=\midrule},
		every last row/.style={after row=\bottomrule},
		every row no 0/.style={after row=\midrule}, 
		every nth row={4}{after row=\midrule}, 
		fixed zerofill,     % Fill numbers with zeros
		columns={full name, ma, param, accuracy, cpu, gpu},
		columns/full name/.style={
			column name=Model,
			string type
		},
		columns/ma/.style={
			column name=\gls{flops} {\small $\times 10^{8}$},
			preproc/expr={{##1/1e8}}
		},
		columns/param/.style={
			column name=Param. {\small $\times 10^{5}$},
			preproc/expr={{##1/1e5}}
		},
		columns/accuracy/.style={
			column name=Accuracy,
			precision=4
		},
		columns/gpu/.style={
			column name=\gls{gpu} (ms),
			precision=3
		},
		columns/cpu/.style={
			column name=\gls{cpu} (ms),
			precision=1
		},
		%    column type/.add={@{}lrrrrrr@{}}{},
		column type/.add={@{}lcccccc@{}}{},
		highlight col max ={\data}{accuracy},
		highlight col min ={\data}{param}, 
		highlight col min ={\data}{ma}, 
		col sep=comma]{\data}
		%}
	\end{table}
	To assess the efficacy of our method, we replaced the spatial convolutional layers of the original \gls{nin} network with root modules (as described in \cref{method}). We preserved the original number of filters per layer but subdivided them into groups as shown in \cref{table:ninconfig}. We considered the first of the pair of existing 1$\times$1 layers to be part of our root modules. 
	We did not group filters in the first convolutional layer --- since it operates on the three-channel image space, it is of limited computational impact compared to other layers.
	\begin{figure}[tbp]
		\centering
		\begin{subfigure}[b]{0.95\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/nincifar.csv}\datatable
			\pgfplotstableread[col sep=comma]{rootdata/nincifar_root_s.csv}\rdatatable
			\pgfplotstableread[col sep=comma]{rootdata/nincifar_tree_s.csv}\tdatatable
			\pgfplotstableread[col sep=comma]{rootdata/nincifar_col_s.csv}\cdatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\centering
			\begin{tikzpicture}
			%\tikzstyle{every node}=[font=\footnotesize]
			\begin{axis}[
				width=0.95\textwidth,
				height=0.66\textwidth,
				axis x line=bottom,
				ylabel=Error,
				xlabel=Model Parameters,
				axis lines=left,
				enlarge x limits=0.05,
				enlarge y limits=0.05,
				grid=major,
				%xmin=0,
				ytick={0.002,0.004,...,1.0},
				%ymin=0.075,ymax=0.09,
				xticklabel style={
					/pgf/number format/fixed,
					/pgf/number format/precision=1
				},
				yticklabel={\pgfmathparse{\tick*1}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed zerofill,
					/pgf/number format/precision=1
				},
				legend style={at={(1,1.1)}, anchor=south east, column sep=0.2em, font=\small\sffamily\sansmath},
				legend columns=4,
				\setplotcyclecat{4},
				every axis plot/.append style={fill},
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
				error bars/y dir=both,
				error bars/y fixed=0.00131497782,
			] table[meta=name,x=param,y expr={1 - \thisrow{accuracy} },]{\datatable};
			\addplot+[mark=square*,
				nodes near coords, only marks,
				every node near coord/.append style={inner sep=4pt},
				point meta=explicit symbolic,
			] table[meta=name,x=param,y expr={1 - \thisrow{accuracy} },]{\rdatatable};
			\addplot+[mark=triangle*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				point meta=explicit symbolic,
			] table[meta=name,x=param,y expr={1 - \thisrow{accuracy} },]{\tdatatable};
			\addplot+[mark=diamond*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				point meta=explicit symbolic,
			] table[meta=name,x=param,y expr={1 - \thisrow{accuracy} },]{\cdatatable};
			\legend{NiN, Root, Tree, Column}
			\end{axis}
			\end{tikzpicture}
			%\caption{\textbf{Model Parameters \vs Error.}}
			%\label{fig:nincifarparamconvonly}
		\end{subfigure}
		~
		\begin{subfigure}[b]{0.95\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/nincifar.csv}\datatable
			\pgfplotstableread[col sep=comma]{rootdata/nincifar_root_s.csv}\rdatatable
			\pgfplotstableread[col sep=comma]{rootdata/nincifar_tree_s.csv}\tdatatable
			\pgfplotstableread[col sep=comma]{rootdata/nincifar_col_s.csv}\cdatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\centering
			\begin{tikzpicture}
			%\tikzstyle{every node}=[font=\footnotesize]
			\begin{axis}[
				width=0.95\textwidth,
				height=0.66\textwidth,
				axis x line=bottom,
				ylabel=Error,
				xlabel=\gls{flops} (Multiply-Add),
				axis lines=left,
				enlarge x limits=0.05,
				enlarge y limits=0.05,
				grid=major,
				%xmin=0,
				ytick={0.002,0.004,...,1.0},
				%ymin=0.075,ymax=0.09,
				xticklabel style={
					/pgf/number format/fixed zerofill,
					/pgf/number format/precision=1
				},
				yticklabel={\pgfmathparse{\tick*1}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed zerofill,
					/pgf/number format/precision=1
				},
				\setplotcyclecat{4},
				every axis plot/.append style={fill},
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
				error bars/y dir=both,
				error bars/y fixed=0.00131497782,
			] table[meta=name,x=ma,y expr={1 - \thisrow{accuracy} },]{\datatable};
			\addplot+[mark=square*,
				nodes near coords, only marks,
				every node near coord/.append style={inner sep=4pt},
				point meta=explicit symbolic,
			] table[meta=name,x=ma,y expr={1 - \thisrow{accuracy} },]{\rdatatable};
			\addplot+[mark=triangle*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				point meta=explicit symbolic,
			] table[meta=name,x=ma,y expr={1 - \thisrow{accuracy} },]{\tdatatable};
			\addplot+[mark=diamond*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				point meta=explicit symbolic,
			] table[meta=name,x=ma,y expr={1 - \thisrow{accuracy} },]{\cdatatable};
			\end{axis}
			\end{tikzpicture}
			%\caption{\textbf{\glsfmttext{flops} (Multiply-Add) \vs Error.}}
			%\label{fig:nincifarmaconvonly}
		\end{subfigure}
		
		\caption[\Glsfmttext{nin} \glsfmttext{cifar10} results]{\textbf{\Glsfmttext{nin} \glsfmttext{cifar10} Results.} Spatial filters (3$\times$3, 5$\times$5) are grouped hierarchically. The best models are closest to the origin. For the standard network, the mean and standard deviation (error bars) are shown over 5 different random initializations.
			%(left) Parameters \vs Error, (right) \gls{flops} \vs Error.
		}\label{fig:nincifarplotsconvonly}
	\end{figure}
	Results are shown in \cref{table:nincifarresults} and \cref{fig:nincifarplotsconvonly} for various network architectures\footnote{Here (and subsequently unless stated otherwise) timings are per image for a forward pass computed on a large batch. Networks were implemented using Caffe (with \gls{cudnn} v2 and \gls{mkl}) and run on an Nvidia Titan Z \gls{gpu} and 2 10-core Intel Xeon E5-2680 v2 \gls{cpu}s.}. Compared to the baseline architecture, the root variants achieve a significant reduction in computation and model size without a significant reduction in accuracy. For example, the root-8 architecture gives equivalent accuracy with only 46\% of the \gls{flops}, 33\% of the model parameters of the original network, and approximately 37\% and 23\% faster \gls{cpu} and \gls{gpu} timings (see \cref{gpuexplanation} for an explanation of the \gls{gpu} timing disparity).
	
	\newcommand{\covarlabels}[5]{%
		\begin{tikzpicture}[anchor=south west]
		\node [inner sep=0pt] (c)
		{
			#5
		};
		\ifx\covarwidth\undefined
		\newlength{\covarwidth}
		\newlength{\covarheight}
		\fi
		\settowidth{\covarwidth}{#5}
		\settoheight{\covarheight}{#5}
		\path[use as bounding box] (c.south west) rectangle (c.north east);
		%\node [anchor=south west, xshift=-0.3em, yshift=-0.3em, rotate=45] at (c.north west) {\tiny 0};
		\node [anchor=south east, xshift=\covarwidth, yshift=-0.2em] at (c.north west) {\tiny #4};
		\node [anchor=south west, xshift=0.25em, yshift=-1.05\covarheight, rotate=90] at (c.north west) {\tiny #2};
		\node [anchor=south, xshift=0.2\covarwidth, yshift=-0.2em] at (c.north west) {\scriptsize\texttt{#3}};
		\node [anchor=south, xshift=0.2em, yshift=-0.2\covarheight, rotate=90] at (c.north west) {\scriptsize\texttt{#1}};
		\end{tikzpicture}%
	}
	
	%\afterpage{
	%\begin{landscape}
		\begin{figure}[p]
			\centering
			\begin{subfigure}[b]{0.28\textheight}
				\centering
				\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=\textwidth]{Figs/PDF/msrc-cifar-nin-4pad-conv8-corr}}
				\caption{Standard}\label{fig:normalcovartestfull}
			\end{subfigure}
			\hspace{1em}
			\begin{subfigure}[b]{0.28\textheight}
				\centering
				\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=\textwidth]{Figs/PDF/msrc-cifar-nin-4pad-funnel2-convonly-conv8-corr}}
				\caption{Root-2 (1 group)}\label{fig:root2corrfull}
			\end{subfigure}
			\hspace{1em}
			\begin{subfigure}[b]{0.28\textheight}
				\centering
				\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=\textwidth]{Figs/PDF/msrc-cifar-nin-4pad-funnel4-convonly-conv8-corr}}
				\caption{Root-4 (2 groups)}\label{fig:root4corrfull}
			\end{subfigure}
			\hspace{1em}
			\begin{subfigure}[b]{0.28\textheight}
				\centering
				\vspace*{0.6em}
				\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=\textwidth]{Figs/PDF/msrc-cifar-nin-4pad-funnel8-convonly-conv8-corr}}
				\caption{Root-8 (4 groups)}\label{fig:root8corrfull}
			\end{subfigure}
			\hspace{1em}
			\begin{subfigure}[b]{0.28\textheight}
				\centering
				\vspace*{0.6em}
				\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=\textwidth]{Figs/PDF/msrc-cifar-nin-4pad-funnel16-convonly-conv8-corr}}
				\caption{Root-16 (8 groups)}\label{fig:root16corrfull}
			\end{subfigure}
			\hspace{1em}
			\begin{subfigure}[b]{0.28\textheight}
				\centering
				\vspace*{0.6em}
				\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=\textwidth]{Figs/PDF/msrc-cifar-nin-4pad-funnel32-convonly-conv8-corr}}
				\caption{Root-32 (16 groups)}\label{fig:root32corrfull}
			\end{subfigure}
			\caption[Inter-layer filter covariance conv2c--conv3a]{\textbf{Inter-layer filter covariance conv2c--conv3a.} The block-diagonal sparsity learned by a root-unit is visible in the correlation of filters on layers \texttt{conv3a} and \texttt{conv2c} in the NiN network as observed on the \glsfmttext{cifar10} training data.}\label{fig:covar}
		\end{figure}
	%\end{landscape}}
	
	\Cref{fig:covar} shows the inter-layer covariance between the adjacent filter layers \texttt{conv2c} and \texttt{conv3a} in the network architectures outlined in \cref{table:ninconfig} as evaluated on the CIFAR training set. The block-diagonalization enforced by the filter group\index{filter groups} structure (as illustrated in \cref{fig:groupconfig}) is visible, more so with larger number of filter groups\index{filter groups}. This shows that the network learns an organization of filters such that the sparsely distributed strong filter relations, visible in \cref{fig:normalcovartestfull} as brighter pixels, are grouped into a denser block-diagonal structure, leaving a visibly darker, low-correlated background.
	
	\subsection{Inter-Layer Covariance}
	\label{interlayercovar}
	To show the relationships between filters between adjacent convolutional layers, as illustrated in \cref{fig:normalconv}, we calculate the covariance of the responses from two adjacent \glspl{featuremap}, the outputs of convolutional layers with $\gls{c}_1$ and $\gls{c}_2$ filters.
	
	Let $\gls{X}_i = [\gls{vectorx}_{i,1}; \gls{vectorx}_{i,2}; \ldots ; \gls{vectorx}_{i,N}]$ be the matrix of $N$ samples $\gls{vectorx}_{i,n}$ from the $c_i$ dimensional \gls{featuremap} for layer $i$. We consider each pixel across the two \glspl{featuremap} to be a sample, and thus each vector $\gls{vectorx}_{i,n}$ is a single pixel filter response of dimension $c_i$. If two \glspl{featuremap} have different spatial dimensions, due to pooling, we up-sample the smaller \gls{featuremap} (with nearest neighbor interpolation) such that there are the same number of pixels (and thus samples) in each \gls{featuremap}.
	
	Given two samples $\gls{X}_1, \gls{X}_2$ with zero mean (\ie mean subtracted) for two adjacent \glspl{featuremap}, we calculate the inter-layer covariance,
	
	\begin{align}
	\gls{covar}(\gls{X}_1, \gls{X}_2) &= \gls{expected}\left[\gls{X}_1\,\gls{X}_2^\textrm{T}\right],\\
	&= \frac{1}{N-1} \gls{X}_1\,\gls{X}_2^\textrm{T}.
	\end{align}
	
	\begin{figure}[tbp]
		\centering
		\begin{subfigure}[b]{0.3\textheight}
			\centering
			\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=\textwidth]{Figs/PDF/ninroot32/train/layercovar_conv8}}
			\caption{Non-whitened responses}
			\label{fig:notwhitened}
		\end{subfigure}
		\hspace{1em}
		\begin{subfigure}[b]{0.3\textheight}
			\centering
			\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=\textwidth]{Figs/PDF/ninroot32/train/layercovarwhite_conv8}}
			\caption{Whitened responses}
			\label{fig:whitened}
		\end{subfigure}
		\caption[Inter-layer covariance with/without whitened responses]{Covariance for between two layers in the root-32 \glsfmttext{nin} model with and without whitened responses.}
		\label{fig:whitevsnot}
	\end{figure}
	
	While this shows the covariance between layers, it is conflated with the inherent covariances within $\gls{X}_1$ and $\gls{X}_2$ from the data (as shown in \cref{fig:notwhitened}). We can more clearly show the covariance between layers by first whitening (using \gls{zca}~\citep{CIFAR10}) the samples in $\gls{X}_1$ and $\gls{X}_2$. For a covariance matrix,
	
	\begin{equation}
	\gls{covar}(\gls{X}, \gls{X}) = \frac{1}{N-1} \gls{X}\gls{X}^\textrm{T},
	\end{equation}
	The \gls{zca} whitening transformation is given by,
	\begin{equation}
	W = \sqrt{N-1}\left(\gls{X}\gls{X}^\textrm{T}\right)^{-\frac{1}{2}}.
	\end{equation}
	Since the covariance matrix is symmetric, it is easily diagonalizable (\ie \gls{pca}),
	\begin{align}
	\gls{covar}(\gls{X}, \gls{X}) &= \frac{1}{N-1} PDP^\textrm{T},\\
	\end{align}
	where $P$ is a orthogonal matrix and $D$ a diagonal matrix. This diagonalization allows a simplified calculation of the whitening transformation (see the derivation in \citet[Appendix A]{CIFAR10}),
	
	\begin{equation}
	W = \sqrt{N-1}PD^{\circ-\frac{1}{2}}P^\textrm{T},
	\end{equation}
	where $D^{\circ-\frac{1}{2}}$ is simply $D$ with an element-wise power of $-\frac{1}{2}$.
	
	The covariance between the whitened \gls{featuremap} responses is then,
	\begin{equation}
	\gls{covar}(W_1 \gls{X}_1, W_2 \gls{X}_2) = E\left[(W_1\gls{X}_1)\,(W_2\gls{X}_2)^\textrm{T}\right].
	\end{equation}
	\afterpage{%
	    \begin{landscape}
			\begin{figure}[p]
				\begin{subfigure}[c]{0.75\paperheight}
					\centering
    				\vspace{0.6em}
					\covarlabels{conv1a}{192}{conv1a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/train/corrcoef_conv0}}
					\hfill
					\covarlabels{conv1b}{160}{conv1b}{160}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/train/corrcoef_conv1}}
					\hfill
					\covarlabels{conv1c}{96}{conv1c}{96}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/train/corrcoef_conv2}}
					\hfill
					\covarlabels{conv2a}{192}{conv2a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/train/corrcoef_conv4}}
					\hfill
					\covarlabels{conv2b}{192}{conv2b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/train/corrcoef_conv5}}
					\hfill
					\covarlabels{conv2c}{192}{conv2c}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/train/corrcoef_conv6}}
					\hfill
					\covarlabels{conv3a}{192}{conv3a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/train/corrcoef_conv8}}
					\hfill
					\covarlabels{conv3b}{192}{conv3b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/train/corrcoef_conv9}}
					\caption{\Glsfmttext{nin}}\label{fig:corrroot1}
    				\vspace{0.6em}
				\end{subfigure}\\
				\begin{subfigure}[c]{0.75\paperheight}
					\centering
    				\vspace{0.6em}
					\covarlabels{conv1a}{192}{conv1a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/train/corrcoef_conv0}}
					\hfill
					\covarlabels{conv1b}{160}{conv1b}{160}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/train/corrcoef_conv1}}
					\hfill
					\covarlabels{conv1c}{96}{conv1c}{96}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/train/corrcoef_conv2}}
					\hfill
					\covarlabels{conv2a}{192}{conv2a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/train/corrcoef_conv4}}
					\hfill
					\covarlabels{conv2b}{192}{conv2b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/train/corrcoef_conv5}}
					\hfill
					\covarlabels{conv2c}{192}{conv2c}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/train/corrcoef_conv6}}
					\hfill
					\covarlabels{conv3a}{192}{conv3a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/train/corrcoef_conv8}}
					\hfill
					\covarlabels{conv3b}{192}{conv3b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/train/corrcoef_conv9}}
					\caption{Root-4}\label{fig:corrroot4}
					\vspace*{0.6em}
				\end{subfigure}\\
				\begin{subfigure}[c]{0.75\paperheight}
					\centering
    				\vspace{0.6em}
					\covarlabels{conv1a}{192}{conv1a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/train/corrcoef_conv0}}
					\hfill
					\covarlabels{conv1b}{160}{conv1b}{160}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/train/corrcoef_conv1}}
					\hfill
					\covarlabels{conv1c}{96}{conv1c}{96}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/train/corrcoef_conv2}}
					\hfill
					\covarlabels{conv2a}{192}{conv2a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/train/corrcoef_conv4}}
					\hfill
					\covarlabels{conv2b}{192}{conv2b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/train/corrcoef_conv5}}
					\hfill
					\covarlabels{conv2c}{192}{conv2c}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/train/corrcoef_conv6}}
					\hfill
					\covarlabels{conv3a}{192}{conv3a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/train/corrcoef_conv8}}
					\hfill
					\covarlabels{conv3b}{192}{conv3b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/train/corrcoef_conv9}}
					\caption{Root-8}\label{fig:corrroot8}
					\vspace*{0.6em}
				\end{subfigure}\\
				\begin{subfigure}[c]{0.75\paperheight}
					\centering
    				\vspace{0.6em}
					\covarlabels{conv1a}{192}{conv1a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/train/corrcoef_conv0}}
					\hfill
					\covarlabels{conv1b}{160}{conv1b}{160}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/train/corrcoef_conv1}}
					\hfill
					\covarlabels{conv1c}{96}{conv1c}{96}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/train/corrcoef_conv2}}
					\hfill
					\covarlabels{conv2a}{192}{conv2a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/train/corrcoef_conv4}}
					\hfill
					\covarlabels{conv2b}{192}{conv2b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/train/corrcoef_conv5}}
					\hfill
					\covarlabels{conv2c}{192}{conv2c}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/train/corrcoef_conv6}}
					\hfill
					\covarlabels{conv3a}{192}{conv3a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/train/corrcoef_conv8}}
					\hfill
					\covarlabels{conv3b}{192}{conv3b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/train/corrcoef_conv9}}
					\caption{Root-32}\label{fig:corrroot32}
					\vspace*{0.6em}
				\end{subfigure}\\
				\begin{subfigure}[c]{0.77\paperheight}
    				\centering
	    			\includegraphics[width=0.4\linewidth]{Figs/PDF/colorbar}
				\end{subfigure}
				\caption[Intra-layer filter correlation (train)]{\textbf{\Glsfmttext{nin} Intra-Layer Correlation (Train).} Absolute correlation of filters within each layer of a NiN model variant on the training data.}
				\label{fig:nincorr}
			\end{figure}
		\end{landscape}
	}%afterpage
	\afterpage{
	    \begin{landscape}
			\begin{figure}[p]
				\begin{subfigure}[c]{0.75\paperheight}
					\centering
    				\vspace{0.6em}
					\covarlabels{conv1a}{192}{conv1a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/test/corrcoef_conv0}}
					\hfill
					\covarlabels{conv1b}{160}{conv1b}{160}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/test/corrcoef_conv1}}
					\hfill
					\covarlabels{conv1c}{96}{conv1c}{96}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/test/corrcoef_conv2}}
					\hfill
					\covarlabels{conv2a}{192}{conv2a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/test/corrcoef_conv4}}
					\hfill
					\covarlabels{conv2b}{192}{conv2b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/test/corrcoef_conv5}}
					\hfill
					\covarlabels{conv2c}{192}{conv2c}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/test/corrcoef_conv6}}
					\hfill
					\covarlabels{conv3a}{192}{conv3a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/test/corrcoef_conv8}}
					\hfill
					\covarlabels{conv3b}{192}{conv3b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/nin/test/corrcoef_conv9}}
					\caption{\Glsfmttext{nin}}\label{fig:corrroot1test}
    				\vspace{0.6em}
				\end{subfigure}\\
				\begin{subfigure}[c]{0.75\paperheight}
					\centering
    				\vspace{0.6em}
					\covarlabels{conv1a}{192}{conv1a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/test/corrcoef_conv0}}
					\hfill
					\covarlabels{conv1b}{160}{conv1b}{160}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/test/corrcoef_conv1}}
					\hfill
					\covarlabels{conv1c}{96}{conv1c}{96}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/test/corrcoef_conv2}}
					\hfill
					\covarlabels{conv2a}{192}{conv2a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/test/corrcoef_conv4}}
					\hfill
					\covarlabels{conv2b}{192}{conv2b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/test/corrcoef_conv5}}
					\hfill
					\covarlabels{conv2c}{192}{conv2c}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/test/corrcoef_conv6}}
					\hfill
					\covarlabels{conv3a}{192}{conv3a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/test/corrcoef_conv8}}
					\hfill
					\covarlabels{conv3b}{192}{conv3b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot4/test/corrcoef_conv9}}
					\caption{Root-4}\label{fig:corrroot4test}
					\vspace*{0.6em}
				\end{subfigure}\\
				\begin{subfigure}[c]{0.75\paperheight}
					\centering
    				\vspace{0.6em}
					\covarlabels{conv1a}{192}{conv1a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/test/corrcoef_conv0}}
					\hfill
					\covarlabels{conv1b}{160}{conv1b}{160}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/test/corrcoef_conv1}}
					\hfill
					\covarlabels{conv1c}{96}{conv1c}{96}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/test/corrcoef_conv2}}
					\hfill
					\covarlabels{conv2a}{192}{conv2a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/test/corrcoef_conv4}}
					\hfill
					\covarlabels{conv2b}{192}{conv2b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/test/corrcoef_conv5}}
					\hfill
					\covarlabels{conv2c}{192}{conv2c}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/test/corrcoef_conv6}}
					\hfill
					\covarlabels{conv3a}{192}{conv3a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/test/corrcoef_conv8}}
					\hfill
					\covarlabels{conv3b}{192}{conv3b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot8/test/corrcoef_conv9}}
					\caption{Root-8}\label{fig:corrroot8test}
					\vspace*{0.6em}
				\end{subfigure}\\
				\begin{subfigure}[c]{0.75\paperheight}
					\centering
    				\vspace{0.6em}
					\covarlabels{conv1a}{192}{conv1a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/test/corrcoef_conv0}}
					\hfill
					\covarlabels{conv1b}{160}{conv1b}{160}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/test/corrcoef_conv1}}
					\hfill
					\covarlabels{conv1c}{96}{conv1c}{96}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/test/corrcoef_conv2}}
					\hfill
					\covarlabels{conv2a}{192}{conv2a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/test/corrcoef_conv4}}
					\hfill
					\covarlabels{conv2b}{192}{conv2b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/test/corrcoef_conv5}}
					\hfill
					\covarlabels{conv2c}{192}{conv2c}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/test/corrcoef_conv6}}
					\hfill
					\covarlabels{conv3a}{192}{conv3a}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/test/corrcoef_conv8}}
					\hfill
					\covarlabels{conv3b}{192}{conv3b}{192}{\includegraphics[width=0.11\linewidth]{Figs/PDF/ninroot32/test/corrcoef_conv9}}
					\caption{Root-32}\label{fig:corrroot32test}
					\vspace*{0.6em}
				\end{subfigure}\\
				\begin{subfigure}[c]{0.77\paperheight}
    				\centering
	    			\includegraphics[width=0.4\linewidth]{Figs/PDF/colorbar}
				\end{subfigure}
				\caption[Intra-layer filter correlation (test)]{\textbf{\Glsfmttext{nin} Intra-Layer Correlation (Test).} Absolute correlation of filters within each layer of a NiN model variant on the test data.}
				\label{fig:nincorrtest}
			\end{figure}
		\end{landscape}
		}%afterpage
		\afterpage{
		\begin{landscape}
			\begin{figure}[p]
				\begin{subfigure}[c]{0.75\paperheight}
					%\centering
					\covarlabels{conv1a}{192}{conv1b}{160}{\includegraphics[width=0.1\linewidth]{Figs/PDF/nin/train/layercovarwhite_conv1}}
					\hfill
					\covarlabels{conv1b}{160}{conv1c}{96}{\includegraphics[width=0.05\linewidth]{Figs/PDF/nin/train/layercovarwhite_conv2}}
					\hfill
					\covarlabels{conv1c}{96}{conv2a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/nin/train/layercovarwhite_conv4}}
					\hfill
					\covarlabels{conv2a}{192}{conv2b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/nin/train/layercovarwhite_conv5}}
					\hfill
					\covarlabels{conv2b}{192}{conv2c}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/nin/train/layercovarwhite_conv6}}
					\hfill
					\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/nin/train/layercovarwhite_conv8}}
					\hfill
					\covarlabels{conv3a}{192}{conv3b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/nin/train/layercovarwhite_conv9}}
					\hfill
					\covarlabels{}{192}{}{10}{\includegraphics[height=0.12\linewidth]{Figs/PDF/nin/train/layercovarwhite_conv10}}
					\caption{\Glsfmttext{nin}}
					\vspace*{0.6em}
					\label{fig:covaroot1}
				\end{subfigure}
				\begin{subfigure}[c]{0.75\paperheight}
					%\centering
					\covarlabels{conv1a}{192}{conv1b}{160}{\includegraphics[width=0.1\linewidth]{Figs/PDF/ninroot4/train/layercovarwhite_conv1}}
					\hfill
					\covarlabels{conv1b}{160}{conv1c}{96}{\includegraphics[width=0.05\linewidth]{Figs/PDF/ninroot4/train/layercovarwhite_conv2}}
					\hfill
					\covarlabels{conv1c}{96}{conv2a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot4/train/layercovarwhite_conv4}}
					\hfill
					\covarlabels{conv2a}{192}{conv2b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot4/train/layercovarwhite_conv5}}
					\hfill
					\covarlabels{conv2b}{192}{conv2c}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot4/train/layercovarwhite_conv6}}
					\hfill
					\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot4/train/layercovarwhite_conv8}}
					\hfill
					\covarlabels{conv3a}{192}{conv3b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot4/train/layercovarwhite_conv9}}
					\hfill
					\covarlabels{}{192}{}{10}{\includegraphics[height=0.12\linewidth]{Figs/PDF/ninroot4/train/layercovarwhite_conv10}}
					\caption{Root-4}
					\vspace*{0.6em}
					\label{fig:covarroot4}
				\end{subfigure}
				\begin{subfigure}[c]{0.75\paperheight}
					%\centering
					\covarlabels{conv1a}{192}{conv1b}{160}{\includegraphics[width=0.1\linewidth]{Figs/PDF/ninroot8/train/layercovarwhite_conv1}}
					\hfill
					\covarlabels{conv1b}{160}{conv1c}{96}{\includegraphics[width=0.05\linewidth]{Figs/PDF/ninroot8/train/layercovarwhite_conv2}}
					\hfill
					\covarlabels{conv1c}{96}{conv2a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot8/train/layercovarwhite_conv4}}
					\hfill
					\covarlabels{conv2a}{192}{conv2b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot8/train/layercovarwhite_conv5}}
					\hfill
					\covarlabels{conv2b}{192}{conv2c}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot8/train/layercovarwhite_conv6}}
					\hfill
					\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot8/train/layercovarwhite_conv8}}
					\hfill
					\covarlabels{conv3a}{192}{conv3b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot8/train/layercovarwhite_conv9}}
					\hfill
					\covarlabels{}{192}{}{10}{\includegraphics[height=0.12\linewidth]{Figs/PDF/ninroot8/train/layercovarwhite_conv10}}
					\caption{Root-8}
					\vspace*{0.6em}
					\label{fig:covarroot8}
				\end{subfigure}
				\begin{subfigure}[c]{0.75\paperheight}
					%\centering
					\covarlabels{conv1a}{192}{conv1b}{160}{\includegraphics[width=0.1\linewidth]{Figs/PDF/ninroot32/train/layercovarwhite_conv1}}
					\hfill
					\covarlabels{conv1b}{160}{conv1c}{96}{\includegraphics[width=0.05\linewidth]{Figs/PDF/ninroot32/train/layercovarwhite_conv2}}
					\hfill
					\covarlabels{conv1c}{96}{conv2a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot32/train/layercovarwhite_conv4}}
					\hfill
					\covarlabels{conv2a}{192}{conv2b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot32/train/layercovarwhite_conv5}}
					\hfill
					\covarlabels{conv2b}{192}{conv2c}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot32/train/layercovarwhite_conv6}}
					\hfill
					\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot32/train/layercovarwhite_conv8}}
					\hfill
					\covarlabels{conv3a}{192}{conv3b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot32/train/layercovarwhite_conv9}}
					\hfill
					\covarlabels{}{192}{}{10}{\includegraphics[height=0.12\linewidth]{Figs/PDF/ninroot32/train/layercovarwhite_conv10}}
					\caption{Root-32}
					\label{fig:covarroot32}
				\end{subfigure}
				\caption[Inter-layer covariance all layers (train)]{\textbf{\Glsfmttext{nin} Inter-layer Covariance (Train).} The inter-layer covariance for all layers in variants of the NiN network}
				\label{fig:suppcovariances}
			\end{figure}
		\end{landscape}
	}%afterpage
	\afterpage{
		\begin{landscape}
			\begin{figure}[p]
				\begin{subfigure}[c]{0.75\paperheight}
					%\centering
					\covarlabels{conv1a}{192}{conv1b}{160}{\includegraphics[width=0.1\linewidth]{Figs/PDF/nin/test/layercovarwhite_conv1}}
					\hfill
					\covarlabels{conv1b}{160}{conv1c}{96}{\includegraphics[width=0.05\linewidth]{Figs/PDF/nin/test/layercovarwhite_conv2}}
					\hfill
					\covarlabels{conv1c}{96}{conv2a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/nin/test/layercovarwhite_conv4}}
					\hfill
					\covarlabels{conv2a}{192}{conv2b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/nin/test/layercovarwhite_conv5}}
					\hfill
					\covarlabels{conv2b}{192}{conv2c}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/nin/test/layercovarwhite_conv6}}
					\hfill
					\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/nin/test/layercovarwhite_conv8}}
					\hfill
					\covarlabels{conv3a}{192}{conv3b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/nin/test/layercovarwhite_conv9}}
					\hfill
					\covarlabels{}{192}{}{10}{\includegraphics[height=0.12\linewidth]{Figs/PDF/nin/test/layercovarwhite_conv10}}
					\caption{\Glsfmttext{nin}}
					\vspace*{0.6em}
					\label{fig:covaroot1test}
				\end{subfigure}
				\begin{subfigure}[c]{0.75\paperheight}
					%\centering
					\covarlabels{conv1a}{192}{conv1b}{160}{\includegraphics[width=0.1\linewidth]{Figs/PDF/ninroot4/test/layercovarwhite_conv1}}
					\hfill
					\covarlabels{conv1b}{160}{conv1c}{96}{\includegraphics[width=0.05\linewidth]{Figs/PDF/ninroot4/test/layercovarwhite_conv2}}
					\hfill
					\covarlabels{conv1c}{96}{conv2a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot4/test/layercovarwhite_conv4}}
					\hfill
					\covarlabels{conv2a}{192}{conv2b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot4/test/layercovarwhite_conv5}}
					\hfill
					\covarlabels{conv2b}{192}{conv2c}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot4/test/layercovarwhite_conv6}}
					\hfill
					\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot4/test/layercovarwhite_conv8}}
					\hfill
					\covarlabels{conv3a}{192}{conv3b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot4/test/layercovarwhite_conv9}}
					\hfill
					\covarlabels{}{192}{}{10}{\includegraphics[height=0.12\linewidth]{Figs/PDF/ninroot4/test/layercovarwhite_conv10}}
					\caption{Root-4}
					\vspace*{0.6em}
					\label{fig:covarroot4test}
				\end{subfigure}
				\begin{subfigure}[c]{0.75\paperheight}
					%\centering
					\covarlabels{conv1a}{192}{conv1b}{160}{\includegraphics[width=0.1\linewidth]{Figs/PDF/ninroot8/test/layercovarwhite_conv1}}
					\hfill
					\covarlabels{conv1b}{160}{conv1c}{96}{\includegraphics[width=0.05\linewidth]{Figs/PDF/ninroot8/test/layercovarwhite_conv2}}
					\hfill
					\covarlabels{conv1c}{96}{conv2a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot8/test/layercovarwhite_conv4}}
					\hfill
					\covarlabels{conv2a}{192}{conv2b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot8/test/layercovarwhite_conv5}}
					\hfill
					\covarlabels{conv2b}{192}{conv2c}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot8/test/layercovarwhite_conv6}}
					\hfill
					\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot8/test/layercovarwhite_conv8}}
					\hfill
					\covarlabels{conv3a}{192}{conv3b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot8/test/layercovarwhite_conv9}}
					\hfill
					\covarlabels{}{192}{}{10}{\includegraphics[height=0.12\linewidth]{Figs/PDF/ninroot8/test/layercovarwhite_conv10}}
					\caption{Root-8}
					\vspace*{0.6em}
					\label{fig:covarroot8test}
				\end{subfigure}
				\begin{subfigure}[c]{0.75\paperheight}
					%\centering
					\covarlabels{conv1a}{192}{conv1b}{160}{\includegraphics[width=0.1\linewidth]{Figs/PDF/ninroot32/test/layercovarwhite_conv1}}
					\hfill
					\covarlabels{conv1b}{160}{conv1c}{96}{\includegraphics[width=0.05\linewidth]{Figs/PDF/ninroot32/test/layercovarwhite_conv2}}
					\hfill
					\covarlabels{conv1c}{96}{conv2a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot32/test/layercovarwhite_conv4}}
					\hfill
					\covarlabels{conv2a}{192}{conv2b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot32/test/layercovarwhite_conv5}}
					\hfill
					\covarlabels{conv2b}{192}{conv2c}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot32/test/layercovarwhite_conv6}}
					\hfill
					\covarlabels{conv2c}{192}{conv3a}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot32/test/layercovarwhite_conv8}}
					\hfill
					\covarlabels{conv3a}{192}{conv3b}{192}{\includegraphics[width=0.12\linewidth]{Figs/PDF/ninroot32/test/layercovarwhite_conv9}}
					\hfill
					\covarlabels{}{192}{}{10}{\includegraphics[height=0.12\linewidth]{Figs/PDF/ninroot32/test/layercovarwhite_conv10}}
					\caption{Root-32}
					\label{fig:covarroot32test}
				\end{subfigure}
				\caption[Inter-layer covariance all layers (test)]{\textbf{\Glsfmttext{nin} Inter-layer Covariance (Test).} The inter-layer covariance for all layers in variants of the NiN network}
				\label{fig:suppcovariancestest}
			\end{figure}
		\end{landscape}
	}%afterpage
	\Cref{fig:nincorr} shows the per-layer (intra-layer) filter correlation. This shows the correlation of filters is more structured in root-networks, filters are learned to be linearly combined into useful filters by the root module, and thus filters are often grouped together with other filters with which they correlate strongly.
	
	\Cref{fig:covar} shows the inter-layer filter covariances between layers \texttt{conv3a} and \texttt{conv2c}. \Cref{fig:suppcovariances} shows the full set of inter-layer covariances between all convolutional layers in the NiN models. Block-diagonal sparsity is visible on the layers with filter groups\index{filter groups}, \texttt{conv2a} and \texttt{conv3a}. This block-diagonal is shown for all variants in more detail in \cref{fig:suppcovariances}.
	
	\subsection{Grouping Degree with Network Depth}
	\afterpage{
	\begin{landscape}
	\begin{figure}[p]
		\begin{subfigure}[b]{0.98\linewidth}
			%\centering
			\begin{tikzpicture}[ampersand replacement=\&]
			\begin{scope}[]
			\matrix[column sep=0em]{
				\node (1a) {
					\raisebox{-0.5\height}{\includegraphics[height=0.07\linewidth, page=15]{groupfig}}
				};\&
				\node (1b) {
					\raisebox{-0.5\height}{\includegraphics[height=0.1\linewidth, page=17]{groupfig}}
				};\&
				\node (1c) {
					\raisebox{-0.5\height}{\includegraphics[height=0.1\linewidth, page=17]{groupfig}}
				};\&
				\node (2a) {
					\raisebox{-0.5\height}{\includegraphics[height=0.08\linewidth, page=16]{groupfig}}
				};\&
				\node (2b) {
					\raisebox{-0.5\height}{\includegraphics[height=0.1\linewidth, page=17]{groupfig}}
				};\&
				\node (2c) {
					\raisebox{-0.5\height}{\includegraphics[height=0.1\linewidth, page=17]{groupfig}}
				};\&
				\node (3a) {
					\raisebox{-0.5\height}{\includegraphics[height=0.08\linewidth, page=16]{groupfig}}
				};\&
				\node (3b) {
					\raisebox{-0.5\height}{\includegraphics[height=0.1\linewidth, page=17]{groupfig}}
				};\&
				\node (3c) {
					\raisebox{-0.5\height}{\includegraphics[height=0.1\linewidth, page=17]{groupfig}}
				};\&
				\node (7) {
					{\large $\cdots$}
				};\\
				\draw node{{\footnotesize \textit{image}}/{\footnotesize \textit{conv1a}}};\&
				\draw node{\footnotesize \textit{conv1b}};\&
				\draw node{\footnotesize \textit{conv1c}};\&
				\draw node{\footnotesize \textit{conv2a}};\&
				\draw node{\footnotesize \textit{conv2b}};\&
				\draw node{\footnotesize \textit{conv2c}};\&
				\draw node{\footnotesize \textit{conv3a}};\&
				\draw node{\footnotesize \textit{conv3b}};\&
				\draw node{\footnotesize \textit{conv3c}};\\
			};
			\end{scope}
			\end{tikzpicture}
			\caption{Standard}
			\label{fig:standardtopology}
		\end{subfigure}\\
		\begin{subfigure}[b]{0.98\linewidth}
			%\centering
			\begin{tikzpicture}[ampersand replacement=\&]
			\begin{scope}[]
			\matrix[column sep=0em]{
				\node (1a) {
					\includegraphics[height=0.07\linewidth, page=15]{groupfig}
				};\&
				\node (1b) {
					\includegraphics[height=0.1\linewidth, page=17]{groupfig}
				};\& 
				\node (1c) {
					\includegraphics[height=0.1\linewidth, page=17]{groupfig}
				};\& 
				\node (2a) {
					\includegraphics[height=0.13\linewidth, page=19]{groupfig}
				};\&
				\node (2b) {
					\includegraphics[height=0.1\linewidth, page=17]{groupfig}
				};\&
				\node (2c) {
					\includegraphics[height=0.1\linewidth, page=17]{groupfig}
				};\&
				\node (3a) {
					\includegraphics[height=0.08\linewidth, page=18]{groupfig}
				};\&
				\node (3b) {
					\includegraphics[height=0.1\linewidth, page=17]{groupfig}
				};\&
				\node (3c) {
					\includegraphics[height=0.1\linewidth, page=17]{groupfig}
				};\&
				\node (4) {
					{\large $\cdots$}
				};\\
			};
			\draw[decorate,decoration={brace,mirror},](2a.south west) -- node[below=3pt] {\small root-4 module} ++(3.5, 0);
			\draw[decorate,decoration={brace,mirror},yshift=-2em](3a.south west) + (0, -0.5) -- node[below=3pt] {\small root-2 module} ++(3.5, -0.5);
			\end{scope}
			\end{tikzpicture}
			\caption{Root-4 Architecture}
			\label{fig:root4topology}
		\end{subfigure}
		\caption[\Glsfmttext{nin} standard \vs root architecture]{\textbf{\Glsfmttext{nin} Root Architecture.} The Root-4 architecture as compared to the original architecture for all the convolutional layers. Colored blocks represent the filters of each layer. Here we don't show the intermediate \glspl{featuremap}\index{feature map} over which a layer's filters operate, or the final fully-connected layer, out of space considerations (see \cref{fig:networkinnetwork,fig:rootmodule}). The decreasing degree of grouping in successive root modules means that our network architectures somewhat resemble tree roots, hence the name root.
		}\label{fig:networktopology}
	\end{figure}
	\end{landscape}
	}
	An interesting question concerns how the degree of grouping in our root modules should be varied as a function of depth in the network. For the \gls{nin}-like architectures described earlier, we might consider having the degree of grouping:
	\begin{enumerate*}[label= (\textbf{\roman*})]
		\item decrease with depth after the first convolutional layer, \eg 1--8--4 (`root');
		\item remain constant with depth after the first convolutional layer, \eg 1--4--4 (`column'); or
		\item increase with depth, \eg 1--4--8 (`tree').
	\end{enumerate*}
	
	To determine which approach is best, we created variants of the NiN architecture with different degrees of grouping per layer. Results are shown in \cref{fig:nincifarplotsconvonly}. The results show that the so-called root topology (illustrated in \cref{fig:networktopology}) gives the best performance, providing the smallest reduction in accuracy for a given reduction in model size and computational complexity. Similar experiments with deeper network architectures have delivered similar results and so we have reported results for root topologies. This aligns with the intuition of \glspl{dnn}\index{DNN} for image recognition subsuming the deformable parts model. If we assume that filter responses identify parts (or more elemental features), then there should be more filter dependence with depth, as more parts (filter responses) are assembled into complex concepts.
	
	\subsection{Improving Residual Networks on \Glsfmttext{ilsvrc}}
	Residual networks (\gls{resnet}\index{ResNet}s)~\citep{He2015} are the state-of-the-art network for \gls{ilsvrc}\@. \Glspl{resnet}\index{ResNet} are more computationally efficient than the VGG architecture~\citep{Simonyan2014verydeep} on which they are based, due to the use of low-dimensional embeddings~\citep{Lin2013NiN}. ResNets are also more accurate and quicker to converge due to the use of identity mappings.
	
	\subsubsection{ResNet 50}
	\label{resnet50results}
	As a baseline, we used the \gls{resnet}\index{ResNet} 50 model~\citep{He2015} (the largest residual network model to fit onto 8 \gls{gpu}s with Caffe). \gls{resnet}\index{ResNet} 50 has 50 convolutional layers, of which one-third are spatial convolutions (non-1$\times$1). We did not use any training augmentation aside from random cropping and mirroring. 
	For training, we used the initialization scheme described by~\citep{He2015b} modified for compound layers, as presented in \cref{initialization}, and batch normalization\index{batch normalization}~\citep{Ioffe2015}.
	\begin{table}[tp]
		\caption[ResNet 50 root architectures]{\textbf{ResNet 50.} Filter groups\index{filter groups} in each conv.\ layer.}
		\label{table:resnet50config}
		\centering
		\begin{tabular}{@{}lcccccccccccc@{}}
			\toprule
			Model & conv1 & \multicolumn{2}{c}{res2\{a--c\}} & \multicolumn{2}{c}{res3\{a--d\}} & \multicolumn{2}{c}{res4\{a--f\}} & \multicolumn{2}{c}{res5\{a--c\}} \\
			& \textit{\footnotesize7$\times$7} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize3$\times$3} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize3$\times$3} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize3$\times$3} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize3$\times$3} \\
			Orig. & 1 & 1 & 1 & 1 &  1 & 1 &  1 & 1 & 1 \\
			\midrule
			root-2 & 1 & 1 & 2 & 1 &  1 & 1 &  1 & 1 & 1 \\
			root-4 & 1 & 1 & 4 & 1 &  2 & 1 &  1 & 1 & 1 \\
			root-8 & 1 & 1 & 8 & 1 &  4 & 1 &  2 & 1 & 1 \\
			root-16 & 1 & 1 & 16 & 1 &  8 & 1 &  4 & 1 & 2 \\
			root-32 & 1 & 1 & 32 & 1 & 16 & 1 &  8 & 1 & 4 \\
			root-64 & 1 & 1 & 64 & 1 & 32 & 1 & 16 & 1 & 8 \\
			\bottomrule
		\end{tabular}
	\end{table}
	To assess the efficacy of our method, we replaced the spatial convolutional layers of the original network with root modules (as described in \cref{method}). We preserved the original number of filters per layer but subdivided them into groups as shown in \cref{table:resnet50config}. We considered the first of the existing 1$\times$1 layers subsequent to each spatial convolution to be part of our root modules. 
	\begin{table}[tp]
		\caption[ResNet 50 \Glsfmttext{ilsvrc} Results]{\textbf{ResNet 50 Results.}}
		\label{table:resnet50imagenetresults}
		%\resizebox{\textwidth}{!}{
		\centering
		\pgfplotstableread[col sep=comma]{rootdata/resnet50ma.csv}\data
		%\pgfplotstableread[col sep=comma]{rootdata/resnet50maall.csv}\alldata
		\pgfplotstableread[col sep=comma]{rootdata/resnet50maconvonly.csv}\codata
		%\pgfplotstablevertcat{\data}{\alldata}
		\pgfplotstablevertcat{\data}{\codata}
		\pgfplotstableset{
			create on use/singlegpu/.style={
				create col/expr={\thisrow{GPU Forward} / \thisrow{Batch Size}}},
		}
		\pgfplotstableset{
			create on use/singlecpu/.style={
				create col/expr={\thisrow{CPU Forward} / \thisrow{Batch Size}}},
		}
		\pgfplotstabletypeset[
		every head row/.style={
			before row=\toprule,after row=\midrule},
		every last row/.style={
			after row=\bottomrule},
		every first row/.style={
			after row=\midrule}, 
		fixed zerofill,     % Fill numbers with zeros
		columns={Full Name, Multiply-Acc., Param., Top-1 Acc., Top-5 Acc., singlecpu, singlegpu},
		columns/Full Name/.style={
			column name=Model,
			string type
		},
		columns/singlegpu/.style={
			column name=GPU (ms),
			precision=1
		},
		columns/singlecpu/.style={
			column name=\gls{cpu} (ms),
			precision=0
		},
		columns/Multiply-Acc./.style={
			column name=\gls{flops} {\small $\times 10^{9}$},
			preproc/expr={{##1/1e9}}
		},
		columns/Param./.style={
			column name=Param. {\small $\times 10^{7}$},
			preproc/expr={{##1/1e7}}
		},
		columns/Top-1 Acc./.style={precision=3},
		columns/Top-5 Acc./.style={precision=3},
		highlight col max ={\data}{Top-1 Acc.},
		highlight col max ={\data}{Top-5 Acc.}, 
		highlight col min ={\data}{Param.}, 
		highlight col min ={\data}{Multiply-Acc.},
		column type/.add={@{}llp{3em}lp{3em}rrrr@{}}{},
		%column type/.add={@{}lp{3em}p{3em}p{3em}p{3em}p{3em}p{3em}@{}}{},
		col sep=comma]{\data}
		%}
	\end{table}
	\begin{figure}[p]
		\centering
		\begin{subfigure}[b]{\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/resnet50ma.csv}\gdatatable
			\pgfplotstableread[col sep=comma]{rootdata/resnet50maconvonly.csv}\codatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\pgfplotstableset{
				create on use/singlecpu/.style={
					create col/expr={\thisrow{\gls{cpu} Forward} / \thisrow{Batch Size}}},
			}
			
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.95\textwidth,
				height=0.2\textheight,
				axis x line=bottom,
				ylabel=Top-5 Error,
				xlabel=Model Parameters (\# Floats),
				axis lines=left,
				enlarge x limits=0.05,
				%enlarge y limits=0.1,
				grid=major,
				%xmin=0,
				ytick={0.01,0.02,...,0.2},
				ymin=0.07,ymax=0.1,
				xticklabel style={
					/pgf/number format/fixed,
					/pgf/number format/precision=3
				},
				yticklabel={\pgfmathparse{\tick*100}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed,
					/pgf/number format/precision=1
				},
				\setplotcyclecat{2},
				every axis plot/.append style={fill},
				legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
				legend columns=3,
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,x=Param.,y expr={1 - \thisrow{Top-5 Acc.} },]{\gdatatable};
			\addplot+[mark=square*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,x=Param.,y expr={1 - \thisrow{Top-5 Acc.} },]{\codatatable};
			\legend{ResNet 50, Roots}
			\end{axis}
			\end{tikzpicture}
			\caption{Model Parameters \vs Top-5 Error}
			\label{fig:resnet50param}
		\end{subfigure}
		~
		\begin{subfigure}[b]{\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/resnet50ma.csv}\gdatatable
			\pgfplotstableread[col sep=comma]{rootdata/resnet50maconvonly.csv}\codatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.95\textwidth,
				height=0.2\textheight,
				axis x line=bottom,
				ylabel=Top-5 Error,
				xlabel=\gls{flops} (Multiply-Add),
				axis lines=left,
				enlarge x limits=0.05,
				%enlarge y limits=0.1,
				grid=major,
				%xmin=0,
				ytick={0.01,0.02,...,0.2},
				ymin=0.07,ymax=0.1,
				xticklabel style={
					/pgf/number format/fixed,
					/pgf/number format/precision=3
				},
				yticklabel={\pgfmathparse{\tick*100}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed,
					/pgf/number format/precision=1
				},
				\setplotcyclecat{2},
				every axis plot/.append style={fill},
				legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
				legend columns=3,
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,x=Multiply-Acc.,y expr={1 - \thisrow{Top-5 Acc.} },]{\gdatatable};
			\addplot+[mark=square*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,x=Multiply-Acc.,y expr={1 - \thisrow{Top-5 Acc.} },]{\codatatable};
			\end{axis}
			\end{tikzpicture}
			\caption{\Glsfmttext{flops} (Multiply-Add) \vs Top-5 Error}
			\label{fig:resnet50ma}
		\end{subfigure}
		~
		\begin{subfigure}[b]{\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/resnet50ma.csv}\gdatatable
			\pgfplotstableread[col sep=comma]{rootdata/resnet50maconvonly.csv}\codatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.95\textwidth,
				height=0.2\textheight,
				axis x line=bottom,
				ylabel=Top-5 Error,
				xlabel=\gls{gpu} Forward (ms),
				axis lines=left,
				enlarge x limits=0.05,
				%enlarge y limits=0.1,
				grid=major,
				%xmin=0,
				ytick={0.01,0.02,...,0.2},
				ymin=0.07,ymax=0.1,
				xticklabel style={
					/pgf/number format/fixed,
					/pgf/number format/precision=3
				},
				yticklabel={\pgfmathparse{\tick*100}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed,
					/pgf/number format/precision=1
				},
				\setplotcyclecat{2},
				every axis plot/.append style={fill},
				legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
				legend columns=3,
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,
			x expr={\thisrow{GPU Forward} / \thisrow{Batch Size}},
			y expr={1 - \thisrow{Top-5 Acc.} }
			]{\gdatatable};
			\addplot+[mark=square*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,
			x expr={\thisrow{GPU Forward} / \thisrow{Batch Size}},
			y expr={1 - \thisrow{Top-5 Acc.} },
			]{\codatatable};
			%\legend{ResNet 50, All Filters, Spatial Filters}
			\end{axis}
			\end{tikzpicture}
			\caption{\Glsfmttext{gpu} Forward Time \vs Top-5 Error}
			\label{fig:resnet50gpuforward}
		\end{subfigure}
		~
		\begin{subfigure}[b]{\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/resnet50ma.csv}\gdatatable
			\pgfplotstableread[col sep=comma]{rootdata/resnet50maconvonly.csv}\codatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.95\textwidth,
				height=0.2\textheight,
				axis x line=bottom,
				ylabel=Top-5 Error,
				xlabel=\gls{cpu} Forward (ms),
				axis lines=left,
				enlarge x limits=0.05,
				%enlarge y limits=0.1,
				grid=major,
				%xmin=0,
				ytick={0.01,0.02,...,0.2},
				ymin=0.07,ymax=0.1,
				xticklabel style={
					/pgf/number format/fixed,
					/pgf/number format/precision=3
				},
				yticklabel={\pgfmathparse{\tick*100}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed,
					/pgf/number format/precision=1
				},
				\setplotcyclecat{2},
				every axis plot/.append style={fill},
				legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
				legend columns=3,
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
				] table[meta=Network,
				x expr={\thisrow{CPU Forward} / \thisrow{Batch Size}},
				y expr={1 - \thisrow{Top-5 Acc.} },
			]{\gdatatable};
			\addplot+[mark=square*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,
			x expr={\thisrow{CPU Forward} / \thisrow{Batch Size}},
			y expr={1 - \thisrow{Top-5 Acc.} },
			]{\codatatable};
			%\legend{ResNet 50, All Filters, Spatial Filters}
			\end{axis}
			\end{tikzpicture}
			\caption{CPU Forward Time \vs Top-5 Error}
			\label{fig:resnet50cpuforward}
		\end{subfigure}
		\caption[ResNet-50 \Glsfmttext{ilsvrc} results]{\textbf{ResNet-50 \glsfmttext{ilsvrc} Results.} Models with filter groups\index{filter groups} have fewer parameters, and less floating point operations, while maintaining error comparable to the baseline.}
		\label{fig:resnet50plots}
	\end{figure}
	Results are shown in \cref{table:resnet50imagenetresults} and \cref{fig:resnet50plots} for various network architectures. Compared to the baseline architecture, the root variants achieve a significant reduction in computation and model size without a significant reduction in accuracy. For example, the best result (root-16) exceeds the baseline accuracy by 0.2\% while reducing the model size by 27\% and floating-point operations (multiply-add) by 37\%. \gls{cpu} timings were 23\% faster, while \gls{gpu} timings were 13\% faster. With a drop in accuracy of only 0.1\% however, the root-64 model reduces the model size by 40\%, and reduces the floating point operations by 45\%. \gls{cpu} timings were 31\% faster, while \gls{gpu} timings were 12\% faster. 
	
	\subsubsection{ResNet 200}\label{resnet200results}
	\begin{table}[tp]
		\caption[ResNet 200 \Glsfmttext{ilsvrc} results]{\textbf{ResNet-200 \glsfmttext{ilsvrc} Results}}
		\label{table:resnet200imagenetresults}
		%\resizebox{\columnwidth}{!}{
		\centering
		\pgfplotstableread[col sep=comma]{rootdata/resnet200.csv}\data
		\pgfplotstabletypeset[
		every head row/.style={
			before row=\toprule,after row=\midrule},
		every last row/.style={
			after row=\bottomrule},
		every first row/.style={
			after row=\midrule}, 
		columns={full name, ma, param, top1, top5},
		columns/full name/.style={
			column name=Model,
			string type
		},
		columns/ma/.style={
			column name=\gls{flops}~{\small$\times 10^{12}$},
			fixed zerofill,
			preproc/expr={{##1/1e12}},
		},
		columns/param/.style={
			column name=Param.~{\small$\times 10^{7}$},
			fixed zerofill,
			preproc/expr={{##1/1e7}},
		},
		columns/top1/.style={
			precision=4,
			column name=Top-1 Err.,
			fixed zerofill,
		},
		columns/top5/.style={
			precision=4,
			column name=Top-5 Err.,
			fixed zerofill,
		},
		column type/.add={@{}lrrrrrr@{}}{},
		highlight col min ={\data}{top1},
		highlight col min ={\data}{top5}, 
		highlight col min ={\data}{param}, 
		highlight col min ={\data}{ma}, 
		col sep=comma]{\data}
		%}
	\end{table}
	\begin{figure}[tbp]
		\centering
		\begin{subfigure}[b]{\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/resnet200orig.csv}\gdatatable
			\pgfplotstableread[col sep=comma]{rootdata/resnet200roots.csv}\codatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\pgfplotstableset{
				create on use/singlecpu/.style={
					create col/expr={\thisrow{\gls{cpu} Forward} / \thisrow{Batch Size}}},
			}
			
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.95\textwidth,
				height=0.22\textheight,
				axis x line=bottom,
				ylabel=Top-5 Error,
				xlabel=Model Parameters (\# Floats),
				axis lines=left,
				enlarge x limits=0.05,
				%enlarge y limits=0.1,
				grid=major,
				%xmin=0,
				%ytick={0.01,0.02,...,0.2},
				ymin=5.6,ymax=6.6,
				xticklabel style={
					/pgf/number format/fixed zerofill,
					/pgf/number format/precision=1
				},
				yticklabel={\pgfmathparse{\tick*1}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed zerofill,
					/pgf/number format/precision=1
				},
				\setplotcyclecat{2},
				every axis plot/.append style={fill},
				legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
				legend columns=3,
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
			] table[meta=name,x=ma,y expr={\thisrow{top5}*100},]{\gdatatable};
			\addplot+[mark=square*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				only marks,
				point meta=explicit symbolic,
			] table[meta=name,x=ma,y expr={\thisrow{top5}*100},]{\codatatable};
			\legend{ResNet 50, Roots}
			\end{axis}
			\end{tikzpicture}
			\caption{Model Parameters \vs Top-5 Error}
			\label{fig:resnet200param}
		\end{subfigure}
		~
		\begin{subfigure}[b]{\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/resnet200orig.csv}\gdatatable
			\pgfplotstableread[col sep=comma]{rootdata/resnet200roots.csv}\codatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.95\textwidth,
				height=0.22\textheight,
				axis x line=bottom,
				ylabel=Top-5 Error,
				xlabel=\gls{flops} (Multiply-Add),
				axis lines=left,
				enlarge x limits=0.05,
				%enlarge y limits=0.1,
				grid=major,
				%xmin=0,
				%ytick={0.01,0.02,...,0.2},
				ymin=5.6,ymax=6.6,
				xticklabel style={
					/pgf/number format/fixed zerofill,
					/pgf/number format/precision=1
				},
				yticklabel={\pgfmathparse{\tick*1}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed zerofill,
					/pgf/number format/precision=1
				},
				\setplotcyclecat{2},
				every axis plot/.append style={fill},
				legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
				legend columns=3,
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
			] table[meta=name,x=ma,y expr={\thisrow{top5}*100},]{\gdatatable};
			\addplot+[mark=square*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				only marks,
				point meta=explicit symbolic,
			] table[meta=name,x=ma,y expr={\thisrow{top5}*100},]{\codatatable};
			\end{axis}
			\end{tikzpicture}
			\caption{\Glsfmttext{flops} (Multiply-Add) \vs Top-5 Error.}
			\label{fig:resnet200ma}
		\end{subfigure}
		\caption[ResNet-200 \Glsfmttext{ilsvrc} results]{\textbf{ResNet-200 \glsfmttext{ilsvrc} Results.} Models with filter groups\index{filter groups} have fewer parameters, and less floating point operations, while maintaining error comparable to the baseline.}
		\label{fig:resnet200plots}
	\end{figure}
	To show that the method applies to deeper architectures, we also applied our method to \gls{resnet}\index{ResNet} 200, the deepest network for ILSVRC 2012. To provide a baseline we used code implementing full training augmentation to achieve state-of-the-art results\footnote{\url{https://github.com/facebook/fb.resnet.torch}}. \Cref{table:resnet200imagenetresults} and \cref{fig:resnet200plots} show the results, top-1 and top-5 error are for center cropped images. The models trained with roots have comparable or lower error, with fewer parameters and less computation. The root-64 model has 27\% fewer FLOPS and 48\% fewer parameters than \gls{resnet}\index{ResNet} 200.

	\subsection{Improving GoogLeNet on \Glsfmttext{ilsvrc}}
	\label{googlenet50results}
	\begin{table}[tp]
		\caption[GoogLeNet \Glsfmttext{ilsvrc} results]{\textbf{GoogLeNet \glsfmttext{ilsvrc} Results.}}
		\label{table:googlenetimagenetresults}
		%\resizebox{\columnwidth}{!}{
		\centering
		\pgfplotstableread[col sep=comma]{rootdata/googlenetma.csv}\data
		\pgfplotstableread[col sep=comma]{rootdata/googlenetmaconvonly.csv}\codata
		\pgfplotstablevertcat{\data}{\codata}
		\pgfplotstableset{
			create on use/singlegpu/.style={
				create col/expr={\thisrow{GPU Forward} / \thisrow{Batch Size}}},
		}
		\pgfplotstableset{
			create on use/singlecpu/.style={
				create col/expr={\thisrow{CPU Forward} / \thisrow{Batch Size}}},
		}
		\pgfplotstabletypeset[
		every head row/.style={
			before row=\toprule,after row=\midrule},
		every last row/.style={
			after row=\bottomrule},
		every first row/.style={
			after row=\midrule}, 
		fixed zerofill,     % Fill numbers with zeros
		columns={Full Name, Multiply-Acc., Param., Top-1 Acc., Top-5 Acc., singlecpu, singlegpu},
		columns/Full Name/.style={
			column name=Model,
			string type
		},
		columns/singlegpu/.style={
			column name=\gls{gpu} (ms),
			precision=2
		},
		columns/singlecpu/.style={
			column name=\gls{cpu} (ms),
			precision=0
		},
		columns/Multiply-Acc./.style={
			column name=\gls{flops} {\small $\times 10^{9}$},
			preproc/expr={{##1/1e9}}
		},
		columns/Param./.style={
			column name=Param. {\small $\times 10^{7}$},
			preproc/expr={{##1/1e7}}
		},
		columns/Top-1 Acc./.style={precision=3},
		columns/Top-5 Acc./.style={precision=3},
		highlight col max ={\data}{Top-1 Acc.},
		highlight col max ={\data}{Top-5 Acc.}, 
		highlight col min ={\data}{Param.}, 
		highlight col min ={\data}{Multiply-Acc.}, 
		column type/.add={@{}lp{4em}p{4em}rrrr@{}}{},
		%column type/.add={@{}lp{3em}p{3em}p{3em}p{3em}p{3em}p{3em}@{}}{},
		col sep=comma]{\data}
		%}
	\end{table}
	
	\begin{table}[tp]
		\caption[GoogLeNet root architectures]{\textbf{GoogLeNet}. Filter groups\index{filter groups} in each conv.\ layer and \Gls{inception}\index{inception} module (\textit{incp}.)}
		\label{table:googlenetconfig}
		\centering
		\begin{tabular}{@{}lcccccccccccc@{}}
			\toprule
			Model & conv1 & \multicolumn{2}{c}{conv2} & \multicolumn{3}{c}{incp.~3\{a,b\}} & \multicolumn{3}{c}{incp.~4\{a--e\}} & \multicolumn{3}{c}{incp.~5\{a,b\}} \\
			& \textit{\footnotesize7$\times$7} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize3$\times$3} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize3$\times$3} & \textit{\footnotesize5$\times$5} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize3$\times$3} & \textit{\footnotesize5$\times$5} & \textit{\footnotesize1$\times$1} & \textit{\footnotesize3$\times$3} & \textit{\footnotesize5$\times$5} \\
			Orig. & 1 &  1 &  1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1\\ 
			\midrule
			root-2 & 1 &  1 &  2 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1\\
			root-4 & 1 &  1 &  4 & 1 & 2 & 2 & 1 & 1 & 1 & 1 & 1 & 1\\
			root-8 & 1 &  1 &  8 & 1 & 4 & 4 & 1 & 2 & 2 & 1 & 1 & 1\\
			root-16 & 1 &  1 & 16 & 1 & 8 & 8 & 1 & 4 & 4 & 1 & 2 & 2\\
			\bottomrule
		\end{tabular}
	\end{table}
	
	We replicated the network as described by \citet{Szegedy2014going}, with the exception of not using any training augmentation aside from random crops and mirroring, as supported by Caffe~\citep{Jia2014}). To train we used the initialization of~\citep{He2015b} modified for compound layers, as described in \cref{initialization} and batch normalization\index{batch normalization} without the scale and bias~\citep{Ioffe2015}. At test time we only evaluate the center crop image.
	
	While preserving the original number of filters per layer, we trained networks with various degrees of filter group\index{filter groups}ing, as described in \cref{table:googlenetconfig}.  While the \gls{inception}\index{inception} architecture is relatively complex, for simplicity, we always use the same number of groups within each of the groups of different filter sizes, despite them having different cardinality. For all of the networks, we only grouped filters within each of the `spatial' convolutions (3$\times$3, 5$\times$5).
	\begin{figure}[p]
		\centering
		\begin{subfigure}[b]{\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/googlenetma.csv}\gdatatable
			\pgfplotstableread[col sep=comma]{rootdata/googlenetmaconvonly.csv}\codatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.95\textwidth,
				height=0.2\textheight,
				axis x line=bottom,
				ylabel=Top-5 Error,
				xlabel=Model Parameters (\# Floats),
				axis lines=left,
				enlarge x limits=0.05,
				grid=major,
				%xmin=0,
				ytick={0.01,0.02,...,0.2},
				ymin=0.095,ymax=0.125,
				xticklabel style={
					/pgf/number format/fixed,
					/pgf/number format/precision=3
				},
				yticklabel={\pgfmathparse{\tick*100}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed,
					/pgf/number format/precision=1
				},
				legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
				legend columns=3,
				\setplotcyclecat{2},
				every axis plot/.append style={fill},
			]
			\addplot+[mark=*,
			%nodes near coords,
			only marks,
			point meta=explicit symbolic,
			] table[meta=Network,x=Param.,y expr={1 - \thisrow{Top-5 Acc.} },]{\gdatatable};
			\addplot+[mark=square*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,x=Param.,y expr={1 - \thisrow{Top-5 Acc.} },]{\codatatable};
			\legend{GoogLeNet, Roots}
			\end{axis}
			\end{tikzpicture}
			\caption{\textbf{Model Parameters \vs Top-5 Error.}}
			\label{fig:googlenet50param}
		\end{subfigure}
		~
		\begin{subfigure}[b]{\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/googlenetma.csv}\gdatatable
			\pgfplotstableread[col sep=comma]{rootdata/googlenetmaconvonly.csv}\codatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.95\textwidth,
				height=0.2\textheight,
				axis x line=bottom,
				ylabel=Top-5 Error,
				xlabel=\gls{flops} (Multiply-Add),
				axis lines=left,
				enlarge x limits=0.05,
				grid=major,
				%xmin=0,
				ytick={0.01,0.02,...,0.2},
				ymin=0.095,ymax=0.125,
				xticklabel style={
					/pgf/number format/fixed,
					/pgf/number format/precision=3
				},
				yticklabel={\pgfmathparse{\tick*100}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed,
					/pgf/number format/precision=1
				},
				legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
				legend columns=3,
				\setplotcyclecat{2},
				every axis plot/.append style={fill},
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,x=Multiply-Acc.,y expr={1 - \thisrow{Top-5 Acc.} },]{\gdatatable};
			\addplot+[mark=square*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,x=Multiply-Acc.,y expr={1 - \thisrow{Top-5 Acc.} },]{\codatatable};
			\end{axis}
			\end{tikzpicture}
			\caption{\textbf{\Glsfmttext{flops} (Multiply-Add) \vs Top-5 Error.}}
			\label{fig:googlenet50ma}
		\end{subfigure}
		~
		\begin{subfigure}[b]{\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/googlenetma.csv}\gdatatable
			\pgfplotstableread[col sep=comma]{rootdata/googlenetmaconvonly.csv}\codatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.95\textwidth,
				height=0.2\textheight,
				axis x line=bottom,
				ylabel=Top-5 Error,
				xlabel=\gls{gpu} Forward (ms),
				axis lines=left,
				enlarge x limits=0.05,
				grid=major,
				%xmin=0,
				ytick={0.01,0.02,...,0.2},
				ymin=0.095,ymax=0.125,
				xticklabel style={
					/pgf/number format/fixed,
					/pgf/number format/precision=3
				},
				yticklabel={\pgfmathparse{\tick*100}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed,
					/pgf/number format/precision=1
				},
				\setplotcyclecat{2},
				every axis plot/.append style={fill},
				legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
				legend columns=3,
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
				] table[meta=Network,
				x expr={\thisrow{GPU Forward} / \thisrow{Batch Size}},
				y expr={1 - \thisrow{Top-5 Acc.} },]{\gdatatable};
			\addplot+[mark=square*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,
			x expr={\thisrow{GPU Forward} / \thisrow{Batch Size}},
			y expr={1 - \thisrow{Top-5 Acc.} },]{\codatatable};
			%\legend{GoogLeNet, All Filters, Spatial Filters}
			\end{axis}
			\end{tikzpicture}
			\caption{\textbf{\Glsfmttext{gpu} Forward Time \vs Top-5 Error.}}
			\label{fig:googlenet50gpuforward}
		\end{subfigure}
		~
		\begin{subfigure}[b]{\textwidth}
			\pgfplotstableread[col sep=comma]{rootdata/googlenetma.csv}\gdatatable
			\pgfplotstableread[col sep=comma]{rootdata/googlenetmaconvonly.csv}\codatatable
			\pgfplotsset{major grid style={dotted,red}}
			
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.95\textwidth,
				height=0.2\textheight,
				axis x line=bottom,
				ylabel=Top-5 Error,
				xlabel=\gls{cpu} Forward (ms),
				axis lines=left,
				enlarge x limits=0.05,
				grid=major,
				%xmin=0,
				ytick={0.01,0.02,...,0.2},
				ymin=0.095,ymax=0.125,
				xticklabel style={
					/pgf/number format/fixed,
					/pgf/number format/precision=3
				},
				yticklabel={\pgfmathparse{\tick*100}\pgfmathprintnumber{\pgfmathresult}\%},style={
					/pgf/number format/fixed,
					/pgf/number format/precision=1
				},
				\setplotcyclecat{2},
				every axis plot/.append style={fill},
				legend style={at={(0.98,0.98)}, anchor=north east, column sep=0.5em},
				legend columns=3,
			]
			\addplot+[mark=*,
				%nodes near coords,
				only marks,
				point meta=explicit symbolic,
				] table[meta=Network,
				x expr={\thisrow{CPU Forward} / \thisrow{Batch Size}},
				y expr={1 - \thisrow{Top-5 Acc.} },]{\gdatatable};
			\addplot+[mark=square*,
				nodes near coords, nodes near coords align = {below}, only marks,
				every node near coord/.append style={inner sep=4pt},
				only marks,
				point meta=explicit symbolic,
			] table[meta=Network,
			x expr={\thisrow{CPU Forward} / \thisrow{Batch Size}},
			y expr={1 - \thisrow{Top-5 Acc.} },]{\codatatable};
			%\legend{GoogLeNet, All Filters, Spatial Filters}
			\end{axis}
			\end{tikzpicture}
			\caption{\textbf{\Glsfmttext{cpu} Forward Time \vs Top-5 Error.}}
			\label{fig:googlenet50cpuforward}
		\end{subfigure}
		
		\caption[GoogLeNet \Glsfmttext{ilsvrc} results]{\textbf{GoogLeNet \glsfmttext{ilsvrc} Results.} Models with filter groups\index{filter groups} have fewer parameters, and less floating point operations, while maintaining error comparable to the baseline.}
		\label{fig:googlenet50plots}
	\end{figure}
	
	As shown in \cref{table:googlenetimagenetresults}, and plotted in \cref{fig:googlenet50plots}, our method shows significant reduction in computational complexity --- as measured in \gls{flops} (multiply-adds), \gls{cpu} and \gls{gpu} timings --- and model size, as measured in the number of floating point parameters. For many of the configurations the top-5 accuracy remains within 0.5\% of the baseline model. 
	The highest accuracy result, is 0.1\% off the top-5 accuracy of the baseline model, but has a 0.1\% higher top-1 accuracy --- within the error bounds resulting from training with different random initializations. While maintaining the same accuracy, this network has 9\% faster \gls{cpu} and \gls{gpu} timings. However, a model with only 0.3\% lower top-5 accuracy than the baseline has much higher gains in computational efficiency --- 44\% fewer floating point operations (multiply-add), 7\% fewer model parameters, 21\% faster \gls{cpu} and 16\% faster \gls{gpu} timings.
	
	While these results may seem modest compared to the results for \gls{resnet}\index{ResNet}, \gls{googlenet}\index{GoogLeNet} is by far the smallest and fastest near state-of-the-art model \gls{ilsvrc} model. We believe that more experimentation in using different cardinalities of filter group\index{filter groups}ing in the heterogeneously-sized filter groups\index{filter groups} within each \gls{inception}\index{inception} module would improve results further.
	
	\subsection{The Effect on Image-level Filters of Root Modules}
	\begin{figure}[tb]
		\centering
		\begin{subfigure}[b]{0.4\textwidth}
			\centering
			\includegraphics[width=\textwidth]{Figs/Raster/msrc-resnet-50-conv1}
			\caption{Standard}
			\label{fig:resnet50normalconv0}
		\end{subfigure}
		~
		\begin{subfigure}[b]{0.4\textwidth}
			\centering
			\includegraphics[width=\textwidth]{Figs/Raster/msrc-resnet-50-conv1-root4-convonly}
			\caption{Root-2}
			\label{fig:resnet50root2conv0}
		\end{subfigure}
		~
		\begin{subfigure}[b]{0.4\textwidth}
			\centering
			\includegraphics[width=\textwidth]{Figs/Raster/msrc-resnet-50-conv1-root8-convonly}
			\caption{Root-4}
			\label{fig:resnet50root4conv0}
		\end{subfigure}
		~
		\begin{subfigure}[b]{0.4\textwidth}
			\centering
			\includegraphics[width=\textwidth]{Figs/Raster/msrc-resnet-50-conv1-root16-convonly}
			\caption{Root-8}
			\label{fig:resnet50root8conv0}
		\end{subfigure}
		~
		\begin{subfigure}[b]{0.4\textwidth}
			\centering
			\includegraphics[width=\textwidth]{Figs/Raster/msrc-resnet-50-conv1-root32-convonly}
			\caption{Root-16}
			\label{fig:resnet50root16conv0}
		\end{subfigure}
		~
		\begin{subfigure}[b]{0.4\textwidth}
			\centering
			\includegraphics[width=\textwidth]{Figs/Raster/msrc-resnet-50-conv1-root64-convonly}
			\caption{Root-32}
			\label{fig:resnet50root32conv0}
		\end{subfigure}
		\caption[ResNet 50 conv1 filters]{\textbf{ResNet 50 \texttt{conv1} filters.} With filter groups\index{filter groups} directly after \texttt{conv1}, in \texttt{conv2}, some of the organization of filters can be directly observed, and give us intuition as to what is happening in root networks.}
		\label{fig:resnet50conv0}
	\end{figure}
	In the \gls{resnet}\index{ResNet} root models, filter groups\index{filter groups} are used in \texttt{conv2},  directly after the image level filters of \texttt{conv1} some of the organization of filters can be directly observed, and give us intuition as to what is happening in root networks. \Cref{fig:resnet50conv0} shows the \texttt{conv0} filters learned for each of the \gls{resnet}\index{ResNet} 50 models. It is apparent that the filters learned in these networks are very similar to those learned in the original model, although sometimes inverted or with a different ordering. This ordering is somewhat consistent in models with filter groups\index{filter groups} however, even with different random initializations. This is because filter groups\index{filter groups} cause filters with strong mutual information to be grouped adjacent to each other.
	
	For example, in the root-8 network (\cref{fig:resnet50root8conv0}), each row of filters corresponds to the input of an independent filter group\index{filter groups} in \texttt{conv2}. We can see that the first row primarily is composed of filters giving various directions of the same color gradient. These filters can be combined in the next layer to produce color edges easily. Due to the shortcut layer and the learned combinations of filters however, not all filter group\index{filter groups}ings are so obvious.
	
	\subsection{Layer-wise Compute/Parameter Savings}
	\afterpage{
	\begin{landscape}
	\begin{figure}[tbp]
		\centering
		\begin{subfigure}[b]{0.97\linewidth}	
			\pgfplotstableread[col sep=comma]{rootdata/resnet50-layerma.csv}\datatable
			\pgfplotstableread[col sep=comma]{rootdata/resnet50-root64-layerma.csv}\datatablea
			\begin{tikzpicture}
			\begin{axis}[
				axis x line=bottom,
				axis y line=left,
				ybar=0pt,
				bar width=0.3em,
				width=0.95\linewidth,
				height=0.4\textheight,
				enlarge x limits=0.02,
				ylabel=\gls{flops},
				y label style={at={(axis description cs:-0.04,.5)},anchor=south},
				y tick label style={
					/pgf/number format/.cd,
					fixed,
					fixed zerofill,
					precision=1,
					/tikz/.cd
				},
				ymin=0,
				xticklabels from table={\datatable}{layer},
				xticklabel style = {rotate = 90, xshift = -0.8ex, anchor = mid east, font=\footnotesize\sffamily\sansmath},
				xtick=data,
				%xmajorticks=false,
				legend style={at={(0.5,1.2)},
				draw=none, anchor=north,legend columns=-1},
				area legend,
				every axis plot/.append style={fill, draw=none, opacity=0.7},
				\setplotcyclecat{2},
			]
			\addplot+ table [x expr=\coordindex,y=MA]{\datatable};
			\addplot+ table [x expr=\coordindex,y=MA]{\datatablea};
			\legend{ResNet 50, Root-64};
			\end{axis}
			\end{tikzpicture}	
		\end{subfigure}
		~
		\begin{subfigure}[b]{0.97\linewidth}
			\pgfplotstableread[col sep=comma]{rootdata/resnet50-layerparam.csv}\datatable
			\pgfplotstableread[col sep=comma]{rootdata/resnet50-root64-layerparam.csv}\datatablea
			\begin{tikzpicture}
			\begin{axis}[
				axis x line=bottom,
				axis y line=left,
				ybar=0pt,
				bar width=0.3em,
				width=0.95\linewidth,
				height=0.4\textheight,
				enlarge x limits=0.02,
				ylabel=Parameters,
				y label style={at={(axis description cs:-0.04,.5)},anchor=south},
				y tick label style={
					/pgf/number format/.cd,
					fixed,
					fixed zerofill,
					precision=1,
					/tikz/.cd
				},
				ymin=0,
				xticklabels from table={\datatable}{layer},
				xticklabel style = {rotate = 90, xshift = -0.8ex, anchor = mid east, font=\footnotesize\sffamily\sansmath},
				xtick=data,
				every axis plot/.append style={fill, draw=none, opacity=0.7},
				\setplotcyclecat{2},
			]
			\addplot+ table [x expr=\coordindex,y=param,]{\datatable};
			\addplot+ table [x expr=\coordindex,y=param,]{\datatablea};
			\end{axis}
			\end{tikzpicture}
			
		\end{subfigure}
		\caption[ResNet 50 layer-wise \glsfmttext{flops}/parameters]{\textbf{ResNet 50 Layer-wise \glsfmttext{flops}/Parameters.} 
		}
		
		\label{fig:resnet50layerwisema}
	\end{figure}
	\end{landscape}
	}
	\Cref{fig:resnet50layerwisema} shows the difference in compute and parameters for each layer in a standard \gls{resnet}\index{ResNet} 50 model and a root-64 variant. The layers in the original networks with the highest computational complexity are clearly the spatial convolutional layers, \ie layers with 3$\times$3 spatial filters. When instead a root-module is used, the computational complexity of these layers is reduced dramatically. While the low dimensional embedding layers (1$\times$1) are not changed, these have less than half the compute of the spatial convolution layers. The number of parameters in spatial convolution layers with large numbers of input channels, which increase towards the end of the network, are similarly reduced.
	
	\section{GPU Implementation}\label{gpuexplanation}
	Our experiments show that our method can achieve a significant reduction in \gls{cpu} and \gls{gpu} runtimes for state-of-the-art \glspl{cnn}\index{CNN} without compromising accuracy. However, the reductions in \gls{gpu} runtime were smaller than might have been expected based on theoretical predictions of computational complexity (FLOPs). We believe this is largely a consequence of the optimization of Caffe for existing network architectures (particularly AlexNet and GoogLeNet) that do not use a high degree of filter group\index{filter groups}ing.
	
	Caffe presently parallelizes over filter groups\index{filter groups} by using multiple \gls{cuda} streams to run multiple \gls{cublas} matrix multiplications simultaneously. However, with a large degree of filter group\index{filter groups}ing, and hence more, smaller matrix multiplications, the overhead associated with calling \gls{cublas} from the host can take approximately as long as the matrix computation itself. To avoid this overhead, \gls{cublas} provides batched methods (\eg \texttt{cublasXgemmBatched}), where many small matrix multiplications can be batched together in one call. \citet{Jhurani2015} explore in depth the problem of using \gls{gpu}s to accelerate the multiplication of very small matrices (smaller than 16$\times$16), and show it is possible to achieve high throughput with large batches, by implementing a more efficient interface than that used in the \gls{cublas} batched calls.
	We have modified Caffe to use \gls{cublas} batched calls, and achieved significant speedups for our  root-like network architectures compared to vanilla Caffe without \gls{cudnn}, e.g. a 25\%\ speed up on our root-16 modified version of the GoogleNet architecture. However, our optimized implementation still is not as fast as Caffe with \gls{cudnn} (which was used to generate the results in this chapter), presumably because of other unrelated optimizations in the (proprietary) \gls{cudnn} library. Therefore we suggest that direct integration of \gls{cublas}-style batching into \gls{cudnn}\footnote{note that in August 2017, approximately a month before the submission of this disseration, the latest version of \gls{cudnn}, version 7, now supports the acceleration of filter groups.} could improve the performance of filter groups\index{filter groups} significantly.
	
	\section{Discussion}
	We explored the effect of using complex hierarchical arrangements of filter groups\index{filter groups} in \glspl{cnn}\index{CNN} and show that imposing a structured decrease in the degree of filter group\index{filter groups}ing with depth --- a `root' (inverse tree) topology --- can allow us to obtain more efficient variants of state-of-the-art networks without compromising accuracy. Our method appears to be complementary to existing methods, such as low-dimensional embeddings, and can be used more efficiently to train \glspl{dnn}\index{DNN} than methods that only approximate a pre-trained model's weights.
	
	We validated our method by using it to create more efficient variants of state-of-the-art \gls{nin}, Googlenet, and \gls{resnet}\index{ResNet} architectures, which were evaluated on the \gls{cifar10} and \gls{ilsvrc} datasets. Our results show comparable accuracy with the baseline architecture with fewer parameters and much less compute (as measured by \gls{cpu} and \gls{gpu} timings). For \gls{nin} on \gls{cifar10}, our model has 47\% of the parameters of the original network, and approximately 22\% faster \gls{cpu} and \gls{gpu} timings. For \gls{resnet}\index{ResNet} 50, our model has 27\% fewer parameters, and was 24\% (11\%) faster on a \gls{cpu} (\gls{gpu}).  For \gls{resnet}\index{ResNet} 200 our model has 27\% fewer FLOPS and 48\% fewer parameters. Even for the most efficient of the near state-of-the-art ILSVRC network, GoogLeNet, our model uses 7\% fewer parameters and is 21\% (16\%) faster on a CPU (GPU). Even for the most efficient of the near state-of-the-art \gls{ilsvrc} network, GoogLeNet, our model uses 7\% fewer parameters and is 21\% (16\%) faster on a \gls{cpu} (\gls{gpu}).
\end{document}