Chapter 06 - Builders' Guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0769f37c",
   "metadata": {},
   "source": [
    "# Chapter 6: Builder's Guide\n",
    "\n",
    "### Dhuvi Karthikeyan\n",
    "\n",
    "2/07/2023"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbbe5eac",
   "metadata": {},
   "source": [
    "## 6.1 Layers and Modules\n",
    "\n",
    "Organizational Hierarchy:\n",
    "\n",
    "* Module (Organized and repeating blocks of layers): Can describe a single layer, multiple layers, or the entire model itself. It is an abstraction of a functional unit of a neural network.\n",
    "    * Layer: Consists of multiple neurons. Neurons of the same layer receieve input in the same time step of information propagation. They also take a set of inputs and return a set of outputs.   \n",
    "        * Neuron: Takes a set of inputs and returns a deterministic scalar output based on internal state variables (parameters)\n",
    "        \n",
    "**In practice:**\n",
    "\n",
    "* Module is a class-level entity:\n",
    "    1. Must take input data for forward pass\n",
    "    2. Must generate output using forward call\n",
    "    3. Must calculate gradients of output with respect to the input, and parameters (Autograd)\n",
    "    4. Must initialize model parameters robustly as needed\n",
    "    5. Must store parameters and make them accessible and updateable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3202b422",
   "metadata": {},
   "source": [
    "### 6.1.3 Executing Code in the Forward Propagation Method\n",
    "\n",
    "The forward pass may not only be a mathematical transformation. It can also include control flow in it as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9d3aaf36",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "class FixedHiddenMLP(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # Random weight parameters that will not compute gradients and\n",
    "        # therefore keep constant during training\n",
    "        self.rand_weight = torch.rand((20, 20))\n",
    "        self.linear = nn.LazyLinear(20)\n",
    "\n",
    "    def forward(self, X):\n",
    "        X = self.linear(X)\n",
    "        X = F.relu(X @ self.rand_weight + 1)\n",
    "        # Reuse the fully connected layer. This is equivalent to sharing\n",
    "        # parameters with two fully connected layers\n",
    "        X = self.linear(X)\n",
    "        # Control flow\n",
    "        while X.abs().sum() > 1:\n",
    "            X /= 2\n",
    "        return X.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "625eaf58",
   "metadata": {},
   "source": [
    "How does backpropagation through the while loop work?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0cbab56",
   "metadata": {},
   "source": [
    "## 6.2 Parameter Management\n",
    "\n",
    "Most of the time one doesn't need to worry about writing parameters to memory and shuttling parameters from CPU to GPU or vice versa. The following in a minimum viable approach to understanding parameter management for non-standard use cases."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ed8dfa0",
   "metadata": {},
   "source": [
    "### 6.2.1 Parameter Accesses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "755d60cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 1])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = nn.Sequential(nn.LazyLinear(8), nn.ReLU(), nn.LazyLinear(1)) # Dummy Net\n",
    "X = torch.rand(size=(2, 4))\n",
    "net(X).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bc2977df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('weight',\n",
       "              tensor([[-0.1544,  0.1493,  0.0143,  0.1825, -0.1538,  0.1383, -0.1592,  0.2035]])),\n",
       "             ('bias', tensor([0.2956]))])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net[2].state_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "14e72c10",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.2956])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net[2].bias.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "60784251",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('0.weight', torch.Size([8, 4])),\n",
       " ('0.bias', torch.Size([8])),\n",
       " ('2.weight', torch.Size([1, 8])),\n",
       " ('2.bias', torch.Size([1]))]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get all the parameters all at once\n",
    "[(name, param.shape) for name, param in net.named_parameters()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6674419e",
   "metadata": {},
   "source": [
    "### 6.2.2 Sharing parameters\n",
    "\n",
    "The book talks about sharing parameters by simpling reusing a layer. Unsure why this would be a good move? Parameter sharing also shares updates but does it necessarily share the same gradients?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c05a7343",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0942],\n",
       "        [0.0942]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shared = nn.LazyLinear(8)\n",
    "net = nn.Sequential(nn.LazyLinear(8), nn.ReLU(),\n",
    "                    shared, nn.ReLU(),\n",
    "                    shared, nn.ReLU(),\n",
    "                    nn.LazyLinear(1))\n",
    "net(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3fd50fe5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([True, True, True, True, True, True, True, True])\n"
     ]
    }
   ],
   "source": [
    "print(net[2].weight.data[0] == net[4].weight.data[0]) # Check that they are in fact the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "44964155",
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 0",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[20], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[43mnet\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mweight\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdata\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m==\u001b[39;49m\u001b[43m \u001b[49m\u001b[43mnet\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m4\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mweight\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdata\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m]\u001b[49m)\n",
      "\u001b[0;31mRuntimeError\u001b[0m: The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 0"
     ]
    }
   ],
   "source": [
    "print(net[0].weight.data[0] == net[4].weight.data[0]) # Returns error due to lazy loading dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "91f3fa6c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 9.9000e+01,  2.3116e-01, -2.0159e-01,  2.5524e-01,  3.3988e-01,\n",
       "         -1.3143e-01,  9.9637e-02, -2.7399e-02],\n",
       "        [ 3.5148e-01, -2.9227e-01,  1.0891e-01,  2.8594e-01, -2.7712e-01,\n",
       "         -2.2687e-01,  3.1859e-02, -2.6861e-01],\n",
       "        [ 2.4074e-01, -2.5918e-01,  2.1355e-01, -9.4026e-02, -1.7284e-01,\n",
       "         -1.0327e-01,  2.5735e-01, -1.5697e-01],\n",
       "        [ 2.0667e-01, -3.9887e-02, -1.7813e-01, -8.4779e-02, -1.9416e-01,\n",
       "         -1.3511e-01, -1.8886e-01, -2.6019e-01],\n",
       "        [ 3.3085e-02,  3.4689e-02,  2.8782e-01, -1.2055e-01, -2.3878e-01,\n",
       "         -3.4340e-01, -2.4142e-01, -2.9684e-01],\n",
       "        [-2.1810e-01,  3.2054e-01, -1.6799e-01, -6.0757e-02,  2.9624e-01,\n",
       "         -2.5178e-01,  3.6727e-02,  2.9169e-01],\n",
       "        [ 3.4839e-01, -7.2290e-02, -2.2480e-02,  3.3316e-01,  3.0976e-01,\n",
       "         -3.4173e-01,  2.6596e-01, -1.0933e-01],\n",
       "        [ 1.7213e-01, -1.9290e-01, -2.5482e-01, -9.4157e-02,  3.4947e-01,\n",
       "          1.1196e-01,  2.1353e-02, -1.2128e-01]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net[2].weight.data[0,0] = 99 #Change a value in net[2]\n",
    "net[2].weight.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ed369830",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 9.9000e+01,  2.3116e-01, -2.0159e-01,  2.5524e-01,  3.3988e-01,\n",
       "         -1.3143e-01,  9.9637e-02, -2.7399e-02],\n",
       "        [ 3.5148e-01, -2.9227e-01,  1.0891e-01,  2.8594e-01, -2.7712e-01,\n",
       "         -2.2687e-01,  3.1859e-02, -2.6861e-01],\n",
       "        [ 2.4074e-01, -2.5918e-01,  2.1355e-01, -9.4026e-02, -1.7284e-01,\n",
       "         -1.0327e-01,  2.5735e-01, -1.5697e-01],\n",
       "        [ 2.0667e-01, -3.9887e-02, -1.7813e-01, -8.4779e-02, -1.9416e-01,\n",
       "         -1.3511e-01, -1.8886e-01, -2.6019e-01],\n",
       "        [ 3.3085e-02,  3.4689e-02,  2.8782e-01, -1.2055e-01, -2.3878e-01,\n",
       "         -3.4340e-01, -2.4142e-01, -2.9684e-01],\n",
       "        [-2.1810e-01,  3.2054e-01, -1.6799e-01, -6.0757e-02,  2.9624e-01,\n",
       "         -2.5178e-01,  3.6727e-02,  2.9169e-01],\n",
       "        [ 3.4839e-01, -7.2290e-02, -2.2480e-02,  3.3316e-01,  3.0976e-01,\n",
       "         -3.4173e-01,  2.6596e-01, -1.0933e-01],\n",
       "        [ 1.7213e-01, -1.9290e-01, -2.5482e-01, -9.4157e-02,  3.4947e-01,\n",
       "          1.1196e-01,  2.1353e-02, -1.2128e-01]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net[4].weight.data # The change has been reflected here since it is the same object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e44f1352",
   "metadata": {},
   "source": [
    "What happens during backprop to the gradients? Since gradients are additive, they are added together and distributed to the layer object (which makes sense because it is contributing twice essentially). It makes sense why you don't just take the gradients of one of them and multiply it by two."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "291dba23",
   "metadata": {},
   "source": [
    "## 6.3 Parameter Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e242d398",
   "metadata": {},
   "source": [
    "### Default Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "879331c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 0.0011, -0.0130, -0.0026,  0.0050]), tensor(0.))"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def init_normal(module):\n",
    "    # Check for linear layers\n",
    "    if type(module) == nn.Linear:\n",
    "        #nn.init.normal_ initialize obj to normal distribution with mean and std\n",
    "        nn.init.normal_(module.weight, mean=0, std=0.01)\n",
    "        #nn.init.constant_ initialize obj to constant\n",
    "        # nn.init.constant_(module.bias, 1)\n",
    "        #nn.init.zeros_ initialize the object to zeros\n",
    "        nn.init.zeros_(module.bias)\n",
    "        \n",
    "# .apply a function \n",
    "net.apply(init_normal)\n",
    "net[0].weight.data[0], net[0].bias.data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caddb419",
   "metadata": {},
   "source": [
    "### Custom Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "d9e31c24",
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_xavier(module):\n",
    "    if type(module) == nn.Linear:\n",
    "        nn.init.xavier_uniform_(module.weight)\n",
    "def init_42(module):\n",
    "    if type(module) == nn.Linear:\n",
    "        nn.init.constant_(module.weight, 42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "2c1fe83a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 0.2216, -0.3519,  0.1610,  0.2502])\n",
      "tensor([[42., 42., 42., 42., 42., 42., 42., 42.],\n",
      "        [42., 42., 42., 42., 42., 42., 42., 42.],\n",
      "        [42., 42., 42., 42., 42., 42., 42., 42.],\n",
      "        [42., 42., 42., 42., 42., 42., 42., 42.],\n",
      "        [42., 42., 42., 42., 42., 42., 42., 42.],\n",
      "        [42., 42., 42., 42., 42., 42., 42., 42.],\n",
      "        [42., 42., 42., 42., 42., 42., 42., 42.],\n",
      "        [42., 42., 42., 42., 42., 42., 42., 42.]])\n"
     ]
    }
   ],
   "source": [
    "net[0].apply(init_xavier)\n",
    "net[2].apply(init_42)\n",
    "print(net[0].weight.data[0])\n",
    "print(net[2].weight.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c314d47f",
   "metadata": {},
   "source": [
    "## 6.4 Lazy Initialization\n",
    "\n",
    "Lazy loading initializes parameters on the fly after the first forward pass. While this may seem \"lazy\" its particularly useful in various contexts. Take for example a scenario where a Convnet is applied to image data, but the data contains images of an unknown resolution. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e387ea9b",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'class_method' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[29], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;129m@class_method\u001b[39m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mapply_init\u001b[39m(\u001b[38;5;28mself\u001b[39m, inputs, init\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m):\n\u001b[1;32m      3\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mforward(\u001b[38;5;241m*\u001b[39minputs)\n\u001b[1;32m      4\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m init \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
      "\u001b[0;31mNameError\u001b[0m: name 'class_method' is not defined"
     ]
    }
   ],
   "source": [
    "@class_method\n",
    "def apply_init(self, inputs, init=None):\n",
    "    self.forward(*inputs)\n",
    "    if init is not None:\n",
    "        self.net.apply(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "740decad",
   "metadata": {},
   "source": [
    "## 6.5 Custom Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0af9a76f",
   "metadata": {},
   "source": [
    "### 6.5.1 Layers without Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "4f53adb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CenteredLayer(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, X):\n",
    "        return X - X.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad1abcba",
   "metadata": {},
   "source": [
    "### 6.5.2 Layers with Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "bd50d65f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyLinear(nn.Module):\n",
    "    def __init__(self, in_units, units):\n",
    "        super().__init__()\n",
    "        self.weight = nn.Parameter(torch.randn(in_units, units))\n",
    "        self.bias = nn.Parameter(torch.randn(units,))\n",
    "\n",
    "    def forward(self, X):\n",
    "        linear = torch.matmul(X, self.weight.data) + self.bias.data\n",
    "        return F.relu(linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1487d51e",
   "metadata": {},
   "source": [
    "## 6.6 File I/O"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4528c91",
   "metadata": {},
   "source": [
    "### 6.6.1 Loading and Saving Tensors\n",
    "\n",
    "x = torch.arange(4)\n",
    "\n",
    "y = torch.zeros(4)\n",
    "\n",
    "torch.save([x, y],'x-files')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9af64f6e",
   "metadata": {},
   "source": [
    "### 6.6.2 Loading and Saving Model Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "77c0fc81",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'network' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[32], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m torch\u001b[38;5;241m.\u001b[39msave(\u001b[43mnetwork\u001b[49m\u001b[38;5;241m.\u001b[39mstate_dict())\n\u001b[1;32m      2\u001b[0m torch\u001b[38;5;241m.\u001b[39mload(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mfile\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m      3\u001b[0m network\u001b[38;5;241m.\u001b[39mload_state_dict(torch\u001b[38;5;241m.\u001b[39mload(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstatedict\u001b[39m\u001b[38;5;124m'\u001b[39m))\n",
      "\u001b[0;31mNameError\u001b[0m: name 'network' is not defined"
     ]
    }
   ],
   "source": [
    "torch.save(network.state_dict())\n",
    "torch.load('file')\n",
    "network.load_state_dict(torch.load('statedict'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25e69e79",
   "metadata": {},
   "source": [
    "## 6.7 GPUs\n",
    "\n",
    "GPU performance has increased by a factor of 1000 every decade since 2000."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a9dc35b",
   "metadata": {},
   "source": [
    "### 6.7.1 Computing Devices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "c80e8f39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cuda.device_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b344c473",
   "metadata": {},
   "source": [
    "### 6.7.2 Tensors and GPUs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "d6c4e568",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([1,2,3])\n",
    "x.device # Defaults to CPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "0d94b9e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cuda', index=0)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = torch.tensor([1,2,3], device='cuda:0')\n",
    "y.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "17ec1c1a",
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[38], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mx\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m+\u001b[39;49m\u001b[43m \u001b[49m\u001b[43my\u001b[49m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!"
     ]
    }
   ],
   "source": [
    "x + y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "98b2934b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2, 4, 6], device='cuda:0')"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = x.cuda(0) # Copy the tensor onto GPU\n",
    "y + z"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25155352",
   "metadata": {},
   "source": [
    "**Note**: Transferring between devices is often much slower than the actual computation. In order to prevent this make sure to instantiate variables mindfully so as to not copy between devices too often. \n",
    "\n",
    "    * Many small operations are actually worse than one big operation\n",
    "    * Several operations done at a time are better than interspersing the operations through the code\n",
    "    * Printing tensors (which implicitly converts to NumPy) results in additional overhead\n",
    "    * As long as data and params are on the same device we can expect training to occur efficiently"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}