deep_learning/time_series_sequences/Simple_Dynamic_Seq2Seq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic RNN\n",
    "\n",
    "![](./tensorflow-seq2seq-tutorials/pictures/1-seq2seq.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation\n",
    "\n",
    "* Length issues:\n",
    "    * Static Unrolling: multiple graphs with different time lengths and separate the dataset into different buckets.\n",
    "    * Dynamic Unrolling: use control flow opts to process sequence step by step. relative to static one, dynamic type allows different lengths, require less space and therefore more efficient (more fast.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vocabulary\n",
    "\n",
    "First, we will reconstruct the sequence into another integer-based sequence.\n",
    "\n",
    "Input data consists of sequences of integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = [[10, 20, 21], [6, 4], [10, 20], [3]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sequence above is convenient to the human, but rnn prefers another type as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"tensorflow-seq2seq-tutorials\")\n",
    "\n",
    "import helpers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xt, xlen = helpers.batch(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[10, 20, 21], [6, 4], [10, 20], [3]]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[10,  6, 10,  3],\n",
       "       [20,  4, 20,  0],\n",
       "       [21,  0,  0,  0]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 4)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xt.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shape of the matrix stands for `[max_time, batch_size]`. Sequences shorter than the longest one are padded with '0' toward to the end. Such layer is called `time-major`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3, 2, 2, 1]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xlen   # represents for the time series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building the Model\n",
    "\n",
    "## Encoder\n",
    "* Starts with empty state.\n",
    "* Run through the input sequence.\n",
    "* Not interested in encoder's output, but its `final_state`.\n",
    "\n",
    "## Decoder\n",
    "* Use encoder's `final_state` as its `initial_state`.\n",
    "* The `input` are a batch-sized matrix with `<EOS>` token at the first time step.\n",
    "* The `output` are mapped onto the output space in `[hidden_units, output_vocab_size]` projection layer. `hidden_units` is like the intelligence. `output_vocab_size` represents the target space growing with the size of the dictionary.\n",
    "    \n",
    "Such type of encoder-decoder is forced to learn fixed-length representation (specificially to the size of  `hidden_units`) of the variable-length input sequence and restore output sequence only from this representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import helpers\n",
    "\n",
    "tf.reset_default_graph()\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorflow version: 1.11.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Tensorflow version: {}\".format(tf.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vocabulary size\n",
    "\n",
    "Dynamic RNN models can be adapted to (1) different batch size, (2) sequence length without retraining. For example, load serializing paramters and graph definitions via `tf.train.Saver`. But, changing vocabulary size (here, the size is the space for the whole target words.) requires a retraining."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PAD = 0\n",
    "EOS = 1\n",
    "\n",
    "vocab_size = 10\n",
    "input_embeddning_size = 20\n",
    "\n",
    "encoder_hidden_units = 20\n",
    "decoder_hidden_units = encoder_hidden_units"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inputs\n",
    "\n",
    "\n",
    "* `encoder_inputs`: a **int32** tensor shaped [encoder_max_time, batch_size]\n",
    "* `decoder_targets`: a **int32** tensor shaped [decoder_max_time, batch_size]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_inputs = tf.placeholder(shape=(None, None), dtype=tf.int32, name=\"encoder_inputs\")\n",
    "decoder_targets = tf.placeholder(shape=(None, None), dtype=tf.int32, name=\"decoder_targets\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* (Add an additional placeholder tensor) `decoder_inputs`: a **int32** tensor shaped [decoder_max_time, batch_size]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "decoder_inputs = tf.placeholder(shape=(None, None), dtype=tf.int32, name=\"decoder_inputs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't need to feed `decoder_inputs` manually. They are a function of either `decoder_targets` or previous decoder outputs during rollout.\n",
    "\n",
    "During training, `decoder_inputs` will consist of `<EOS>` token concatenated with `decoder_targets` along time axis.\n",
    "\n",
    "Notice that all shapes are specificied with `None`s (means `dynamically`). We can use batches of any size with any numbers of timesteps. But there are serveral constraints:\n",
    "\n",
    "* Feed values for the whole tensor should have the same `batch_size`.\n",
    "* Decoder `inputs` (decoder_inputs) and `outputs` (decoder_targets) should have the same `decoder_max_time`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encoder and decoder RNNs expect dense vector representation of words, e.g. `[max_time, batch_size, input_embedding_size]`. We can convert one to another by using `embeddings`.\n",
    "\n",
    "We rely on end-to-end training to learn vector representation for words jointly with encoder and decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "embeddings = tf.get_variable(\"embeddings\", \\\n",
    "        dtype=tf.float32, \\\n",
    "        shape=[vocab_size, input_embeddning_size], \\\n",
    "        initializer=tf.random_uniform_initializer(-1.0, 1.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `tf.nn.embedding_lookup` to index embedding matrix; if we given word 4, it means 4th column of the embedding matrix. Wy only need to compute gradients at 4th column.\n",
    "\n",
    "In the case, the encoder and decoder share the same embedding. (In real NLP applications, embedding matrix can be very large.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_input_embedded = tf.nn.embedding_lookup(embeddings, encoder_inputs)\n",
    "decoder_input_embedded = tf.nn.embedding_lookup(embeddings, decoder_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_cell = tf.contrib.rnn.LSTMCell(encoder_hidden_units)\n",
    "\n",
    "# _ is the encoder output, but here we don't need\n",
    "_, encoder_final_state = tf.nn.dynamic_rnn(encoder_cell, encoder_input_embedded, dtype=tf.float32, time_major=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`encoder_final_state` is the state of LSTM's hidden cells at the last moment of the Encoder rollout. `encoder_final_state` is also called `thought vector`. We use it as the input of the decoder. It is the only input to the decoder in the model without attention mechanism."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LSTMStateTuple(c=<tf.Tensor 'rnn/while/Exit_3:0' shape=(?, 20) dtype=float32>, h=<tf.Tensor 'rnn/while/Exit_4:0' shape=(?, 20) dtype=float32>)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder_final_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensorflow LSTM implementation stores state as the a tuple of tensors.\n",
    "* `encoder_final_state.h`: activation of hidden layer of LSTM cell\n",
    "* `encoder_final_state.c`: final output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "decoder_cell = tf.contrib.rnn.LSTMCell(decoder_hidden_units)\n",
    "\n",
    "decoder_outputs, decoder_final_state = tf.nn.dynamic_rnn(decoder_cell, decoder_input_embedded, \\\n",
    "                                                         initial_state=encoder_final_state, dtype=tf.float32, \\\n",
    "                                                        time_major=True, scope=\"plain_decoder\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`initial_state=encoder_final_state` requires:\n",
    "* the same cell type (here is LSTMCell)\n",
    "* the same amount of `hidden_units`\n",
    "* the same amount of layers (single layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'plain_decoder/TensorArrayStack/TensorArrayGatherV3:0' shape=(?, ?, 20) dtype=float32>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`decoder_outpute` is a `hidden_units` sized vector at every timestep. For training or prediction we need logits of size `vocab_size`. Here we have to put linear layer (fc layer without activation function) on top of LSTM output to get non-normalized logits. This layer is called projection layer by convention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "decoder_logits  = tf.contrib.layers.linear(decoder_outputs, vocab_size)\n",
    "decoder_prediction = tf.argmax(decoder_logits, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'fully_connected/BiasAdd:0' shape=(?, ?, 10) dtype=float32>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RNN outputs tensor of shape `[max_time, batch_size, hidden_units]` which projection layer maps onto `[max_time, batch_size, vocab_size]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-21-05f6ec0fbbf9>:3: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "\n",
      "Future major versions of TensorFlow will allow gradients to flow\n",
      "into the labels input on backprop by default.\n",
      "\n",
      "See `tf.nn.softmax_cross_entropy_with_logits_v2`.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "stepwise_cross_entropy = tf.nn.softmax_cross_entropy_with_logits(\n",
    "    labels=tf.one_hot(decoder_targets, depth=vocab_size, dtype=tf.float32),\n",
    "    logits=decoder_logits,\n",
    ")\n",
    "\n",
    "loss = tf.reduce_mean(stepwise_cross_entropy)\n",
    "train_op = tf.train.AdamOptimizer().minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test forward pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch_encoded:\n",
      "[[6 3 7]\n",
      " [0 7 4]\n",
      " [0 0 1]]\n",
      "decoder inputs:\n",
      "[[1 1 1]\n",
      " [0 0 0]\n",
      " [0 0 0]\n",
      " [0 0 0]]\n",
      "decoder prediction:\n",
      "[[9 9 0]\n",
      " [9 9 0]\n",
      " [9 9 0]\n",
      " [3 3 3]]\n"
     ]
    }
   ],
   "source": [
    "batch_ = [[6], [3, 7], [7, 4, 1]]\n",
    "\n",
    "batch_, batch_length = helpers.batch(batch_)\n",
    "print('batch_encoded:\\n' + str(batch_))\n",
    "\n",
    "din_, dlen = helpers.batch(np.ones(shape=(3,1), dtype=np.int32), max_sequence_length=4)\n",
    "print('decoder inputs:\\n' + str(din_))\n",
    "\n",
    "pred_ = sess.run(decoder_prediction, feed_dict={encoder_inputs: batch_, decoder_inputs: din_})\n",
    "print('decoder prediction:\\n' + str(pred_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training on the toy task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we teach our model to memorize and reproduce input sequence. Sequences will be random, with varying length. The model will simply encode sequence in a thought vector, and then decode from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "head of the batch: \n",
      "[5, 3, 5, 9, 3, 7, 6, 3]\n",
      "[4, 5, 9, 4, 2]\n",
      "[8, 9, 7, 8, 7, 3, 6]\n",
      "[7, 8, 6, 3, 2, 9, 7, 9]\n",
      "[7, 6, 7, 7, 4]\n",
      "[4, 9, 2, 6]\n",
      "[3, 8, 3, 9]\n",
      "[4, 9, 4]\n",
      "[3, 3, 5, 2, 5, 8, 4]\n",
      "[5, 4, 6, 9]\n"
     ]
    }
   ],
   "source": [
    "batch_size = 100\n",
    "\n",
    "batches = helpers.random_sequences(length_from=3, length_to=8, vocab_lower=2, vocab_upper=10, batch_size=batch_size)\n",
    "print('head of the batch: ')\n",
    "\n",
    "for idx in next(batches)[:10]:\n",
    "    print(idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def next_feed():\n",
    "    batch = next(batches)\n",
    "    encoder_inputs_, _ = helpers.batch(batch)\n",
    "    decoder_targets_, _ = helpers.batch([(sequence) + [EOS] for sequence in batch])\n",
    "    decoder_inputs_, _ = helpers.batch([[EOS] + (sequence) for sequence in batch])\n",
    "    return { encoder_inputs: encoder_inputs_, decoder_inputs: decoder_inputs_, decoder_targets: decoder_targets_ }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given encoder_inputs `[5,6,7]`, decoder_targets would be `[5,6,7,1]`, and the decoder_input would be `[1,5,6,7]`.\n",
    "\n",
    "decoder_inputs are lagged by 1 step, passing previous token as input at current step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss_track = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch 0\n",
      "  minibatch loss: 2.2997896671295166\n",
      "  sample 1:\n",
      "    input     > [9 5 6 4 8 5 5 0]\n",
      "    predicted > [9 1 9 9 1 1 5 3 3]\n",
      "  sample 2:\n",
      "    input     > [3 3 2 0 0 0 0 0]\n",
      "    predicted > [9 9 9 9 9 9 3 3 3]\n",
      "  sample 3:\n",
      "    input     > [2 3 4 0 0 0 0 0]\n",
      "    predicted > [9 9 9 7 9 9 3 3 3]\n",
      "\n",
      "batch 1000\n",
      "  minibatch loss: 0.29770025610923767\n",
      "  sample 1:\n",
      "    input     > [4 8 5 3 7 8 0 0]\n",
      "    predicted > [4 8 5 7 7 8 1 0 0]\n",
      "  sample 2:\n",
      "    input     > [2 7 9 2 4 0 0 0]\n",
      "    predicted > [2 9 9 2 4 1 0 0 0]\n",
      "  sample 3:\n",
      "    input     > [2 5 9 4 4 3 0 0]\n",
      "    predicted > [2 5 4 4 4 3 1 0 0]\n",
      "\n",
      "batch 2000\n",
      "  minibatch loss: 0.16550947725772858\n",
      "  sample 1:\n",
      "    input     > [9 6 4 4 7 4 6 0]\n",
      "    predicted > [9 6 4 4 7 6 6 1 0]\n",
      "  sample 2:\n",
      "    input     > [4 2 8 2 3 3 6 0]\n",
      "    predicted > [2 2 2 2 3 3 6 1 0]\n",
      "  sample 3:\n",
      "    input     > [9 7 9 4 9 3 0 0]\n",
      "    predicted > [9 7 9 4 9 3 1 0 0]\n",
      "\n",
      "batch 3000\n",
      "  minibatch loss: 0.08369956910610199\n",
      "  sample 1:\n",
      "    input     > [8 2 7 0 0 0 0 0]\n",
      "    predicted > [8 2 7 1 0 0 0 0 0]\n",
      "  sample 2:\n",
      "    input     > [6 4 5 3 0 0 0 0]\n",
      "    predicted > [6 4 5 3 1 0 0 0 0]\n",
      "  sample 3:\n",
      "    input     > [4 9 3 3 0 0 0 0]\n",
      "    predicted > [4 9 3 3 1 0 0 0 0]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "max_batches = 3001\n",
    "batches_in_epoch = 1000\n",
    "\n",
    "try:\n",
    "    for batch in range(max_batches):\n",
    "        fd = next_feed()\n",
    "        _, l = sess.run([train_op, loss], fd)\n",
    "        loss_track.append(l)\n",
    "\n",
    "        if batch == 0 or batch % batches_in_epoch == 0:\n",
    "            print('batch {}'.format(batch))\n",
    "            print('  minibatch loss: {}'.format(sess.run(loss, fd)))\n",
    "            predict_ = sess.run(decoder_prediction, fd)\n",
    "            for i, (inp, pred) in enumerate(zip(fd[encoder_inputs].T, predict_.T)):\n",
    "                print('  sample {}:'.format(i + 1))\n",
    "                print('    input     > {}'.format(inp))\n",
    "                print('    predicted > {}'.format(pred))\n",
    "                if i >= 2:\n",
    "                    break\n",
    "            print()\n",
    "except KeyboardInterrupt:\n",
    "    print('training interrupted')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 0.0794\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhIAAAFkCAYAAAB1rtL+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XecVNX9//H3h44o2Gg2ELsxAVlj76gkYu9EIxF/2JJv\nDMaoMbFEY4nGHlvsiG5iFxQVwYKKgtm1RAMaKdKLKAsCy8Lu+f1xdnJnZmd2Z+5On9fz8ZjH3HLu\n3M+eHbifPffcc8w5JwAAgDDa5DsAAABQvEgkAABAaCQSAAAgNBIJAAAQGokEAAAIjUQCAACERiIB\nAABCI5EAAAChkUgAAIDQSCQAAEBoaSUSZvZ7M5tqZivMbLGZPW9mO7ZwzEFm1hD3qjezHq0LHQAA\n5Fu6LRIHSLpL0l6SDpPUXtJ4M+vcwnFO0g6SejW+ejvnlqR5bgAAUGCsNZN2mdnmkpZIOtA5926S\nMgdJekPSJs65FaFPBgAACk5r+0hsLN/a8G0L5UzSx2a2wMzGm9m+rTwvAAAoAKFbJMzMJI2VtJFz\n7qBmyu0o6SBJ/5LUUdIIST+XtKdz7uMkx2wmabCk2ZJqQwUIAEB56iSpr6TXnHPLsn2y1iQS98pf\n7Pdzzi1M89i3JH3tnBuWZP/PJD0RKjAAACBJpzvnnsz2SdqFOcjM/ibpSEkHpJtENJoqab9m9s+W\npNGjR2uXXXYJ8fHlaeTIkbrtttvyHUbRod7SR52FQ72ljzpL37Rp03TGGWdIjdfSbEs7kWhMIo6V\ndJBzbk7I8w6Q1FwCUitJu+yyiwYOHBjyFOWnW7du1FcI1Fv6qLNwqLf0UWetkpOuAWklEmZ2j6Sh\nko6RtMrMejbuqnHO1TaWuV7SlpHbFmZ2oaRZkj6Xv28zQtIhkg7PyE8AAADyJt0WifPkn9J4K277\nWZJGNS73lrR11L4Okm6RtIWk1ZI+lTTIOTcp3WABAEBhSSuRcM61+Lioc+6suPWbJd2cZlwAAKAI\nMNdGCRk6dGi+QyhK1Fv6qLNwqLf0UWeFr1UjW2aLmQ2UVFVVVUUnGwAA0lBdXa2KigpJqnDOVWf7\nfLRIAACA0EgkAABAaCQSAAAgNBIJAAAQGokEAAAIjUQCAACERiIBAABCI5EAAAChkUgAAIDQSCQA\nAEBoJBIAACA0EgkAABAaiQQAAAiNRAIAAIRGIgEAAEIjkQAAAKGRSAAAgNBIJAAAQGgkEgAAIDQS\nCQAAEBqJBAAACI1EAgAAhFbQicSqVfmOAAAANKegE4nPPst3BAAAoDkFnUgsWZLvCAAAQHMKOpGo\nqcl3BAAAoDkFnUi8/nq+IwAAAM0p6ESCPhIAABS2gk4kAABAYSv4RKK+Pt8RAACAZAo+kZg3L98R\nAACAZAo+kfjFL/IdAQAASKbgE4m33sp3BAAAIJmCTyQAAEDhKopEoqEh3xEAAIBECjqR2HJL/15X\nl984AABAYgWdSPzyl/593br8xgEAABIr6ESiXTv/TiIBAEBhKuhEon17/04iAQBAYSroRIIWCQAA\nChuJBAAACI1EAgAAhFbQicQmm/j3WbPyGwcAAEisoBOJHj38+3ff5TcOAACQWEEnEh06+PeRI/Mb\nBwAASKygE4lIH4lFi/IbBwAASKygEwmzYPnFF/MXBwAASKygE4lof/pTviMAAADxiiaRWLMm3xEA\nAIB4aSUSZvZ7M5tqZivMbLGZPW9mO6Zw3MFmVmVmtWb2pZkNS/Wc11zj36dPZzwJAAAKTbotEgdI\nukvSXpIOk9Re0ngz65zsADPrK+klSRMl9Zd0h6QHzezwVE7YrVuwTCIBAEBhaZdOYefckdHrZvYL\nSUskVUh6N8lh50ua6Zy7pHH9CzPbX9JISa+3dM6GhmC5vj6daAEAQLa1to/ExpKcpG+bKbO3pAlx\n216TtE8qJ9hww2CZFgkAAApL6ETCzEzS7ZLedc79p5mivSQtjtu2WFJXM+vY0nlOOy1YXr8+/TgB\nAED2pHVrI849knaVtF+GYmli5MiR6tatm/r0kb7+WjrjDOmss4Zq6NCh2TolAABFo7KyUpWVlTHb\nampqchqDOefSP8jsb5KOlnSAc25OC2XfllTlnLsoatsvJN3mnNskyTEDJVVVVVVp4MCBGj9eGjzY\nJxPbbJN2uAAAlI3q6mpVVFRIUoVzrjrb50v71kZjEnGspENaSiIavS9pUNy2Ixq3pyQyVDa3NgAA\nKCzpjiNxj6TTJf1M0ioz69n46hRV5nozeyzqsPsk9TOzv5jZTmZ2gaSTJN2a6nnbt/fvP/hBOtEC\nAIBsS7dF4jxJXSW9JWlB1OuUqDK9JW0dWXHOzZY0RH7ciY/lH/s82zkX/yRHUpEWidraNKMFAABZ\nle44Ei0mHs65sxJsmyQ/1kQokRYJAABQWIpiro1tt813BAAAIJGiSCQ22yzfEQAAgESKIpGIFve4\nLAAAyKOiSyR+9rN8RwAAACKKJpEYMiRYDjGGFgAAyIKiSSQeeCBYzvHonwAAIImiSSR69w6W58/P\nXxwAACBQNIlEtFWr8h0BAACQijSRWLcu3xEAAACpSBOJurp8RwAAAKQiSyRGjfLvtEgAAFAYiiqR\nOOww/z54cH7jAAAAXlElEh065DsCAAAQragSCWYBBQCgsBRVIrHhhsFyQ0P+4gAAAF5RJRJtoqKt\nrc1fHAAAwCuqRCLamjX5jgAAABRtIkGLBAAA+Vd0icS4cf6dibsAAMi/okskNtnEv+++e37jAAAA\nRZhIdO7s3xkmGwCA/CvaREIimQAAIN+KOpFYtCh/cQAAgCJMJDbdNFhevz5/cQAAgCJMJDbYIFiu\nr89fHAAAoAgTCbNgmT4SAADkV9ElEtEY3RIAgPwq6kRi3rx8RwAAQHkr6kTiu+/yHQEAAOWtKBOJ\ntm39+/Dh+Y0DAIByV5SJRLSFC/MdAQAA5asoE4mnnw6Wt9hCevXV/MUCAEA5K8pE4vjjY9enTctP\nHAAAlLuiTCTiXXddviMAAKA8lUQisWyZtG5dvqMAAKD8lEQiIUmTJuU7AgAAyk/JJBKHHZbvCAAA\nKD9Fm0hEz7kRUVub+zgAAChnRZtIjB/fdNuMGbmPAwCAcla0icTmmzfd5lzu4wAAoJwVbSKRyNdf\n5zsCAADKS9EmEolaH446KvdxAABQzoo2kUhm6dJ8RwAAQPkouUSiR498RwAAQPkouUQCAADkTtEm\nEjvtlO8IAABA0SYSG2zA454AAORb0SYSzXnnHWnt2nxHAQBA6SvJROLAA6XLL893FAAAlL6STCQk\nadasfEcAAEDpK9lEgv4TAABkX9qJhJkdYGZjzGy+mTWY2TEtlD+osVz0q97MGPEBAIAiF6ZFoouk\njyVdICnVv/udpB0k9Wp89XbOLQlxbgAAUEDapXuAc+5VSa9KkplZGocudc6tSPd8AACgcOWqj4RJ\n+tjMFpjZeDPbN5Mfvv/+TbfRRwIAgOzLRSKxUNK5kk6UdIKkuZLeMrMBmfjwJUuk8eOlQw/NxKcB\nAIB0pH1rI13OuS8lfRm16QMz207SSEnDmjt25MiR6tatW8y2oUOHaujQof9b797dv0+cKH3yiTQg\nI+kJAACFr7KyUpWVlTHbampqchqDuVbcAzCzBknHOefGpHncTZL2c87tl2T/QElVVVVVGjhwYMqf\n+8UX0s47B+vLlkmbbppOZAAAFLfq6mpVVFRIUoVzrjrb58vXOBID5G95ZFSnTrHrX32V6TMAAIBo\nad/aMLMukraX70ApSf3MrL+kb51zc83sBklbOOeGNZa/UNIsSZ9L6iRphKRDJB2egfhjdO4cu96h\nQ6bPAAAAooXpI7GHpDflx4Zwkm5p3P6YpOHy40RsHVW+Q2OZLSStlvSppEHOuUkhY04qvkWifftM\nnwEAAEQLM47E22rmlohz7qy49Zsl3Zx+aOmLTyRuuEEaPToXZwYAoDyV1Fwb8S0QTzwhLV+en1gA\nACgHJZVIJBpnc5NNpLq63McCAEA5KKlEIpl9MzqOJgAAiCiLRKKqKt8RAABQmsoikQAAANlBIgEA\nAEIruURi9Wpp3bp8RwEAQHkouUSic2epXYLRMa66Svr229zHAwBAKSu5RCIiMitoxDXXSH/4g9Sv\nn/Tgg/mJCQCAUlOyicSSJdKUKbHb1q6VZs2Sfve7/MQEAECpKdlEQpI6doxdX706P3EAAFCqSjqR\nqK+PXf/nP/MTBwAApaqkE4mePfMdAQAApa2kE4ktt8x3BAAAlLaSTiSScS7fEQAAUBrKMpGoqcl3\nBAAAlIayTCQAAEBmkEgAAIDQSj6RWLw48fbPPsttHAAAlKKSTyR69Ei8/Yc/zG0cAACUopJPJAAA\nQPaQSAAAgNDKOpFYvz7fEQAAUNzKOpFYuzbfEQAAUNzKOpFYty7fEQAAUNzKIpF4//3E2+vqchsH\nAAClpiwSib33lnr3brp99uychwIAQEkpi0RCkr78Ulq2LHbbXntJ48ZJr74qNTTkJy4AAIpZ2SQS\nG24obbqpNGJE7PYhQ6Sf/lS67LL8xAUAQDErm0Qiok2Sn7iyMrdxAABQCsoukXAu8fbFiyUzacqU\n3MYDAEAxI5FoFHkUdNSo3MUCAECxK7tE4tprpV/9Kvn++vrcxQIAQLEru0SiZ0/prruS7+fpDQAA\nUld2iURLaJEAACB1JBJxaJEAACB1JBJx5s7NdwQAABQPEok4EyfmOwIAAIpH2SYSzz+ffN+ttwbL\n220nXXxx9uMBAKAYlW0icdxxvj/ERRc13ffb3wbLM2dKt9ySu7gAACgmZZtISH4kyyOPzHcUAAAU\nr3b5DiDfks29sXIlT3AAANCSsm6RkKS2bRNvv+UWaeONm27/6CNp4MBgSG0AAMpZ2ScSW22VePuf\n/pR4+/XX+2RiyZLsxQQAQLEo+0SiXz9p4ULppz/NdyQAABSfsk8kJKlXL2nsWOmgg/IdCQAAxYVE\nolHbtjzmCQBAukgkomyzTeplncteHAAAFAsSiSgdO+Y7AgAAiguJRJQOHZLvM5MWLAjWaZEAAIBE\nIkbHjtK110qVlYn3b7mlNGWKXyaRAAAgRCJhZgeY2Rgzm29mDWZ2TArHHGxmVWZWa2ZfmtmwcOFm\nl5n0xz9Kp52WvExkmnFGvQQAIFyLRBdJH0u6QFKLf5ebWV9JL0maKKm/pDskPWhmh4c4d8FYtIhW\nCQAA0k4knHOvOueudM69KMlSOOR8STOdc5c4575wzt0t6RlJI9M9dy5de61/32ijxPv32UcaNSp3\n8QAAUIhy0Udib0kT4ra9JmmfHJw7tJNO8u9ffJG8zD33SHV13OYAAJSvXCQSvSQtjtu2WFJXMyvY\nBy533tnfuujdWxo/PnGZqVN9B80jjshtbAAAFIqCnkZ85MiR6tatW8y2oUOHaujQoTmN4/AWenNM\nnCi9+qr0k5/kJh4AACSpsrJSlXGPGtbU1OQ0hlwkEosk9Yzb1lPSCufc2uYOvO222zRw4MCsBZZJ\nL79MIgEAyK1Ef1xXV1eroqIiZzHk4tbG+5IGxW07onF7yVi9Ot8RAACQe2HGkehiZv3NbEDjpn6N\n61s37r/BzB6LOuS+xjJ/MbOdzOwCSSdJurXV0QMAgLwK0yKxh6SPJFXJjyNxi6RqSX9q3N9L0taR\nws652ZKGSDpMfvyJkZLOds7FP8lR0KZNa36/pfIgLAAAJSbtPhLOubfVTALinDsrwbZJknJ3wyYL\ndt5ZOuQQ6c03E++fNcsPUtWrV27jAgAgn5hrIw1vvNH8vt69/XJdXW7iAQAg30gkQhg+PPk+Mz+2\nRGRODgAAShmJRJrmzZPuvbflcv/9b/ZjAQAg3wp6QKpCtOWWqZVbsya7cQAAUAhokciSe+7JdwQA\nAGQfiUSWtKOtBwBQBkgksmTxYummm6SVK/MdCQAA2UMi0UoLFybePmWKdOmlUteu0rvvSpMn5zYu\nAABygQb4VkplAKoDDvDvVVVSkcxBBgBASmiRyKFB8VOXAQBQ5EgkQtpzTz/wVDqWL89OLAAA5AuJ\nREgffBBu6vD586X16zMfDwAA+UAiEZKZ1CZJ7VVWJj9uq62kSy7JTkwAAOQaiUQGzZ0rzZ4tOdd8\nuUmTchIOAABZx1MbGVBVJX3yiW9tkPzjngAAlAMSiQwYODD2sc6GhubLV1VlNx4AAHKFWxtZ0FIi\nIUn77OPfnZOee67l2yEAABQiEoksSNYJM9oHH/iWiTFjpBNPlM45J/txAQCQaSQSWXDqqdIJJ0hv\nv918uT32kI47zi8/+CCtEgCA4kMfiSzo0EF69lm/PGOGtN12qR23bp0/FgCAYkGLRJb16yd9/rk0\nZEjLZWtrg+V33vFjVTAaJgCgkJFI5MCuu0o339xyuTVrguXIoFazZkmPPy7de292YgMAoDVIJHKk\nU6eWy5x/vrRqlV828+/OSWeeKV1wQfZiAwAgLBKJHEnlSY7nn5f23dcvL1vm3+mACQAoZCQSOVJf\nn1q5Tz/1j4b+859+PTqRcE765pvMxwYAQFgkEjmSyiBVEZHBquKPu+suqXt3qaYmc3EBANAaJBI5\n0r69f7/wQqmuLvXjohOJyLgUYaYvBwAgGxhHIkf69JGeeko66qggqZCkzp1jn9aIF31rI51WDQAA\ncoEWiRw6+WSfOEjSM8/498sua/6YTz4JliMJx4cfZj42AADCMFeAjwWY2UBJVVVVVRoYPa1miYo8\n6pmO9ev9kyBhjgUAlK7q6mpVVFRIUoVzrjrb56NFokhtsIGfq6MA80AAQBkhkSgADz6Y/jF1dVJ1\ntXT//ZmPBwCAVJFIFICzz5Z22CHcsVOm+NsbV1yR2ZgAAEgFiUSB+PJL6aWX0j8uMqbEnXdmNh4A\nAFJBIlFAUpkhNN7zz/v36EdKAQDIFRKJArPhhuGO69Ah8XbnpBUrwscDAEBzSCQKTNu2/v2WW6SP\nPpKOPDK149q397dHvv46dvujj0rdugWTgAEAkEmMbFlgIuNCXHSRf3/55djtybRvL+20k1+OfiT0\njTf8+3ffSZttlrk4AQCQaJEoOOecE+64+PEk3nlHuvTSYHsq05gDAJAuLi8F5i9/CTfI1Pr1seuH\nHy7ddFMwPwcjYAIAsoFbG0ViyhSpUyefGOy+e9P9c+YEy99/H/S1YORLAEA2kUgUiT33DJZPO036\nxz+Sl91oo2A50iKxYoVUXx8kGAAAZAK3NorQHXekXjYyY+iAAdL552cnHgBA+SKRKELdu6deduzY\nxMsAAGQCiUQRMpMeeST94zp3lk46SRozJvMxAQDKE4lEkRo2LFiO7hPRnLZtpWeflc44IzsxAQDK\nD4lEkTKTliyRfvc7P6JlKr76yr936pS9uAAA5YVEooh17+7Hikh3fo6lS2MfC50zR6qry2xsAIDy\nQCJRAjp3Tv+Y+fOlyy7zLRt9+khDh2Y+LgBA6WMciRIQZmyIrbeOXX/uuWD5vfekrl2lH/6wdXEB\nAEofLRIloq5O+vGPY7eNGpXeZ5j51/77Sz/6UeZiAwCUrlCJhJn90sxmmdkaM/vAzH7cTNmDzKwh\n7lVvZj3Ch4147dtLU6dK8+ZJo0f76cP79/f79tsv3Gd+803GwgMAlKi0b22Y2amSbpF0jqSpkkZK\nes3MdnTOJbv0OEk7Slr5vw3OLUk/XLRkyy2l00/3y/X10oUXSpdfLrVrl/404t27+06ZDQ3S6tXp\nd+oEAJS+MC0SIyXd75wb5ZybLuk8SaslDW/huKXOuSWRV4jzIk1t20q33y716CFtumm4z1i/Xrrr\nLj9WxdKlmY0PAFD80kokzKy9pApJEyPbnHNO0gRJ+zR3qKSPzWyBmY03s33DBIvWef/91MeciGjf\nXvrNb/xyjx7SSy9lPi4AQPFKt0Vic0ltJS2O275YUq8kxyyUdK6kEyWdIGmupLfMbECa50Yr7b23\ntMMO0pVXhv+Mo4/2HTJXrpRqa6Xq6szFBwAoPll//NM596Wk6L+DPzCz7eRvkQxLfJQ3cuRIdevW\nLWbb0KFDNZRBD1rllFOka66J3XbFFdK116b+GYsXSw89JN14o/Tdd9LGG2c2RgBAyyorK1VZWRmz\nraamJqcxmIse4rClwv7WxmpJJzrnxkRtf1RSN+fc8Sl+zk2S9nPOJXyewMwGSqqqqqrSwIEDU44P\nqbvkEunmm4P1jz/2U42n6ssvpZ/8RJo5U7r/fqlnT+nYYzMfJwAgPdXV1aqoqJCkCudc1tuN07q1\n4ZxbJ6lK0qDINjOzxvXJaXzUAPlbHsiTM8+MXe/f3z+hEdcAlNSgQT6JkKRzz5WOO076/vvMxggA\nKHxhntq4VdIIMzvTzHaWdJ+kDSQ9KklmdoOZPRYpbGYXmtkxZradmf3AzG6XdIikv7U+fIS1227S\niBHS1VdLn38ebO/bN7Xj585tum2jjWI/6+abw426CQAoHmn3kXDOPWVmm0u6RlJPSR9LGuycizwc\n2EtS9ADMHeTHndhC/rbIp5IGOecmtSZwtN7f/95021lnBU9phLHbbn6K8wkT/IBWDQ3hPwsAUPjS\n6iORK/SRyB/n/NgRHTpk9jMBALlR0H0kUPrM/NgRb78trVjhWxXOOUf6979b97n77y8ddZQfHTPH\nHYoBAFnE7J9I6MADg+X772/dZ731lp9RNGLqVOnww5OX/+IL3yrygx+07rwAgOyjRQJZd8ghsevz\n5kk33ODnAklk5519XwsAQOEjkUDKPvggM59z/fV+IrEnnvCDWUW88IK0fHlmzgEAyA0SCaRsr71i\n1wcPDvc5X33l34cNk3bd1ffLmDBBOv546aCDgnK33irtvru/zQEAKEz0kUBaHnpI2mor6YgjpLo6\nac2a1g2PvWiRf4/0mfj002Dfb3/r31eskJYt8+NU9Eo2owsAIC9okUBahg/3SYTkHxFNNBLm7rtL\nzz6buXOuXy/tuKPUu7dvvTBrWmbpUmmXXaQFCzJ3XgBAy0gkkHEXXCCdcIL0xz/6Aa5aq2fP5Pu+\n+UZavVoaP16aPl165pnWnw8AkDoSCbTav/7VtP+E5GcTbc2U5ano3l3q0cOPTyFJq1Zl93wAgFj0\nkUCrVVRIe+8tTZni17t2Dfb17SutXSt17JjZcx59tLRunV9etSoYMGvNmsyeBwDQPFokkBGRfgsX\nXSSdfHLsvg4dpM6dg/Xp04Plfv2kp59O/3wvvSS99lqwHmmJiO4/cddd0sKF0sMPB506AQCZRSKB\njOjSxb+fdlrizpDTpvn3TTeVdtpJeuABPyjVjBnSkCGtP/+NN/p356TaWt9v4te/lrbYQjr7bOnu\nu4Oye+7ph+x+/vlw55o7V7ruOuYQAQCJSbuQIatW+dlEf/ObxImE5LdvuKG0cmXTfatXSz/7mfTi\ni9mLsV8/6fvvpSVLgm3PPeeH8O7bVzruOGnbbVv+nEMPld58099G6dQpW9ECQDi5nrSLRAI5c9NN\n0qBBvk9FMn/+s3TFFbmLKV5dnZ+0rDmRRGnVKmmDDbIfEwCkg9k/UbIuuaT5JELyj4zmsz/D6NHN\n74+eH4QRNwGARAIFaKON/Pu220qjRvmLd64azoYP90+YxA985Zz06qu+/0VE5KmR6dP9vCEAUI54\n/BMFZ4MN/KBWp5wSO/dGxNtvJ96eKXV1wfL06b4fxK9/LY0d62OK2Hxzvy+SXBx7rB/Ke6utpLZt\n/baVK31i0qFD9uIFgHyiRQIF6e67kycLBx4o/eQnsduSDUQ1aVLr4thlF98yMnasX3/qqdj90S0U\nDz/sO21eeaVvwdhhBz+mxlFHtS6GePvtJ515ZmY/EwDCIpFA0Xj0Uf/IqCS98orU0OCfFDn0UN+K\n8eGHTY854IDcxffAA/79+uulNm2CWU5ff71p2aVL/cifYW7ZTJ4sPf54+DgBIJNIJFA0hg2TLrss\nWDeTRoyQJk7063vsEVs+MmDVjBnSt99mP77PPku+L3Lhv/hinxCNHOlbLqqrpTfeCMotX+5vjbz3\nXsvno7MngEJAIoGSMmCAbw2orw9mKe3XT9pkk/zGNWKENHiwdMstfiKzxYv99mOP9Y/ERqZPnzpV\nmj/fD5h10knB8W+95fuGRPvkk5yEDgDNIpFASfnwQz9QVJsk3+y+ff08Hbm2dq2foTRiwgT/Pn++\nf+/fXzrxRN9ZM+LZZ6Xf/156913pkEOkgw9mNE0AhYdEAiWlXbvkT0jMmCFVVQVTm0+YIP3f/wWP\nm6bq0kulFStaF2cizz3nR/eMduONsf08olsl9thDOuaYlj+3rs4P6R15XBUAMolEAmWjXz8/18fx\nx/u/7AcNku68U3rnnaDMxx/7yb4iEs1a+sc/pp98ZMohh8Sujx0rnXOO9N//+sG+Fi2SFizwCceI\nEf7n69jRx1xZGXvs5MnSmDG5ix1AaWIcCZS9/v39xXbtWr/cv79/nPSyy/yF+NxzfbnBg30Hzg03\n9Otbb+0n8JL8pGWRR1BPOMHPOVJXJx12WPbjf+ABfzunutpPpx7pGxIvenwMyT9GKmXmdkldHWNl\nAOWKFglAfnbS6Mc0e/Xy7926+ScoFizwf/1HTzj2/vt+W12dnwws0vfin//0tyMGDfKdJnMhMrR3\nsr4hkn9cNlVVVbE/6+rV/tZQIo895hOxN99M/fML1Zo10vnn+98ngNSQSADyg05Ftx6cfrp0xx2+\nA+S++0q9e/vJvCKtEZK05ZZ+sKnIJF9PPeUTjnZR7Xzjx/snM3KluanRzz03GPo7+qJ/xRXSxhtL\nO+8cDAu+xx6+DiKGDpW2394nTSec4FtnHn1U6t5duv9+X+bQQ/2Q5pIf32PSpOLrl/HMM9J99/mf\nDUBqmP0TyLJ//MNfiHfdVfrPf/wF+bbbfBKSaMr1I4+Uxo3LfZwRS5f6BGGjjXwy9aMf+SRr+fL0\njo+YMMG3ziTinJ+n5NRTk8+6Oneu1LmzH5K8uto/Inveeen9TKkaPVr6+c+D2CTf78Q5n0wCxYDZ\nP4ESc9pp/iL8+efS1VdLL76YeNjsK66QHnzQd4Bcs8b/ZbznnsH+ESOaHnP44ZmPN5IErFzpJzHb\nY4/Uk4jGAtjMAAASHElEQVTo4yMOOyy2H8a4ccGon5Mm+Qv3Pfck/7xttpH69JFOPtl3KD3//GCw\nsUxLlNj17i1tsUV2zgeUAlokgDyKXLiS/TM88ED/VMns2f5iuuOOvlPnggV+/+WX+1aOF17ISbih\n9enjf4Z165J3ynRO+u47n8BMneo7jv7pT4kv7m3bZm5kz2nT/G2qrl2lJ58MbunMmuXHHWnpdwQU\nGlokAPzP44/70TD79PHr06dLc+YEA1r98pe+X8SQIbHH/b//F8xAOnly7uJN5uuvfYtKc092TJsm\n7b570PpwzTXJp2eP7lS6erWft6S+3rfovPCCb/2pqfH7nZOuusonMjffHDy98uSTvjPtrrsG9Red\ntPz739LMmcF6skSipsbHet55viUJKDvOuYJ7SRooyVVVVTmglL31lnNPPZWZz1q+3LkvvnBu5Uq/\n/tlnzlVUOLdsmXP+Mhj72npr5/r1i93WrVvisoX4qq937i9/CdZ79Yrdf9BBzr30knNjx8Zu//3v\nnfvjH2O3tWnj6+yCC4JtF17o3G9/G6yPH+/LvPiic0uWBPV+/vlBmX/8I/HvZv361H6H9fU+vvfe\n83EDYVRVVTlJTtJAl4trdi5OknZQJBJARvXt6/+1mzk3ZYpffukl55Yujb2gzpzp3OuvB+XL6XXr\nrc3vf/ZZ52pr/fIxxwR1e/LJQZkBA5zbZhtfr++84+v6jTf8vueea/53VFvr3GuvxZ6zWHz1lY93\n3rx8RwLncp9IcGsDKAOzZkm1tb5/xZ57+svUkCH+SYjI+BKDBgWPwU6d6vtmRD8C+ve/N3+OZ57J\nXvy5cNFFze8/55zgNlHkFsZee0lPPx2U+fhjf+vptdf8WCJ77eUfi5Wke++NHcvjvff8rZTIsOin\nn+4fq41mJu2yi//9JbJwoS9z882p/YwrVwaDqGXSSy/59/iJ5QrZ/PmF8ZjvunWpzfZb0HKRraT7\nEi0SQE5VVjq3eHHifYsWOffww355wAD3v2b+u+927oAD/PqvfuX3R/6SfuAB30SfqdaCVaty30LR\n0qtLl/SPufRS5/7856bbly9v+djo2ybffx9b34laL9av9y0F0fbay5c99NDmvw/r1zt3ww3OrV6d\neP/q1c6tWxes3367/9wnnkhc/r//bf58+bDnnonrLdeuvtrHMXdu5j6TWxuORAIoVLW1QR8M5/wF\n5cADnZsxw69HLmr19X79ooucGz482H7TTT5pWbSo+YvmwQf75GGzzZz74Q/9ZyW6AJfT63e/c+6y\ny4L1a6+N3b92bezv6vrr/fZvv3VuzRrnPv00cfmGBueOOCLoA/LKK4nP/+WXwWdLzh13nH+/+27n\n7rjDL48e3fQ788wzfl91dbDtmGOc69Ah/PcwEyI/V6r9V7Ll7LN9HP/+d+Y+k0TCkUgAxSrZX8dz\n5vgLWrRJk5ybONFf0CLHXXyxf3/6aV+mtrbpBbKlC27kL7xUXttvn70Lf65fm23m3Lhxvo7eece5\nXXf122fMcO688xIfM3VqbIfSljrbNjQ0/R3st59zd97plx9/PPg9XXml/6v/qKP8vjFjWv6eOOfc\n22879/77iff99Ke+s2u6Fi8OEoZ582LjX7Uq+XF//3vyONPxzjv+c5Yvb7ov0lk3k5c7EglHIgEU\nq3btwv3HG/lP/e67/XuyC4lzzp1zjnN/+ENwzPXX+46MkeOdc27yZP8XePRnR786dvQXxYcf9uuP\nPOLcqac2LRed5BTLK3L7IhuvQYOatlhEJx8XX9z0dxp5vfiiv8XRqVOwLd6ECcG+mTOdW7DAuRUr\nnLv3XucGDw72DR4cJDXDhzv36KPBZzQ0OHfddc7Nn+/X6+uD4z791Cew0XEtW5b4ezZjRlAmvtXi\n9deb3jZKJvIdk3znW+d8y43k3LHHOveb3/jlyZNT+7xUkEg4EgmgWC1fnvivrpZMmuT/Ilu3LvXH\nHidOdO6hh/wx9fXO/e1vztXVNS138MHBRfDCC/3yJpv4fatX+9sFtbVB+eiLTPRFKJ3XiBHZu5gX\n+mvvvf3FP5U6eeQR52pqYi/a8a/ttku+b+TIYLmuzr8i/U2OOsr//iK3ViKvY49t+jnTpsV+ZyJP\noURed97p3Cef+PfId6RjR7/8xBPOvfxy4u/ounWxnzNwYNPvWKQVbtKk2GPr633dOOfcwoWJ/11N\nn+6fNoq2cqVzN95IIkEiASBrrr66+Wbk2bN9U/Rnn/n1ffYJ/tNvaHDuF79IfFF79NFg2bnkSUiX\nLr4/wujRzScc112XfB+vll+7755e+TFj/Hdjt92a7hs2LPZ3m+h453xCOmFC8F2qrEz9/BMnxn4P\nI7ecFi8OyixaFOz/4IPYc0+e7BOgM85wTiKRIJEAUDDWrvW3C+I7w0VfBObO9ReR886L7QsS2T9n\njr9QzZwZezFwzrn77vNlttoquPhFOqt2756diyyv8K9Iv4n41zHHBAnHtts69+ST6X/2ggU+WV2w\nIHmZv/616bY+feK35TaRYK4NAAjBzM9YOmqUn5gtkepqP4T3/vsn/5zqaj8ZWXW1H1Pgr3/1U9JL\nfnKzHXZoeszYsdKUKX5m1nvukd56y293zs/a+v33QdlXXvGfs+++fhyJK6/0Q5HfcUfzP9/550uP\nPebjR+60beuHe2+dakm5m2sj65lKmJdokQBQ4L7/vukTJWFFj8kQ7euv/V+YXbv6e/T33edvvcRb\nuNCPVOqcvyUzapQ/buedk59z/Xrfl+Hzz4PxQN5+O/irdv58fw8+sj58uB+6u6W/qiOPM0rJRws9\n+uj0/1qPvE48Mfyx5fPi1gaJBAA0eughf8FP1zffBANXtaS2NvmAZHPmxK4vW+Yfw+zVyycW8Y+W\nOufc0KH+Xr1zwfbttvPjilx2md/etm2w7+c/9+/Rw41Hv557LphX5aOPYp/aOe44f9uouQvrokVB\nUhYd56GHhr9YX355MGR64b1ym0i0y3qTBwAgtOHDwx232Wapl+3YUerRI/G+rbeOXd90U2ncuGB9\n33398OEDBwZln3wy2N+/v/TJJ3521/btg+2RWVhnzJC22UZasEC68UZp2DA/S+xuu0nvvivdfrt0\n1FH+2Esu8ccMGCBtt52vm0MO8fsjttpKmjdPOvNMf5unZ0//kvzsuevWBUOOjx0rffihdPDBfn3u\nXD/seOfO0llnSf/6lx/q/OyzpRUrpF//OjhPp06+3iLna0mXLn6I+nh33SX93/+1fHxBy0W2ku5L\ntEgAQFFZsSLx4E5XXRW0AGTa7NnBeBLduzvXo0e4z3nhBT+se7T6et9hMvL5ES+/7H+eyGPK557b\ntEXgxReD5QULfOtFpBPup586t8EG/hbUtGlNHxE9/vhgedw4/wTRRRf5IdIjI4hGXnfd1fTc/ikj\nOlvS2RIASkRDg/9LfKONsnue2lp/Ke3cObvnkXwLxFZb+eX166WaGmnRImnLLX3LSZcuvjPuWWdJ\nDz/c8ueNG+fjrqvzP8dxx/nWkIqKxOXr66U2baRly6Tu3f22Nm389nXrpA4dctvZklsbAICsadMm\n+0mE5G815EokiZCkdu38baT4W0nRM7225Mgjg2Xn/K2Xvn2Tl2/b1r9vvnnQFhE5X7t20m23SSNH\npn7+1iKRAAAgw8zCH9dcEpHsmEhyYSYdeGC4c4fVJrenAwAApYREooRUVlbmO4SiRL2ljzoLh3pL\nH3VW+EIlEmb2SzObZWZrzOwDM/txC+UPNrMqM6s1sy/NbFi4cNEc/sGFQ72ljzoLh3pLH3VW+NJO\nJMzsVEm3SLpK0u6SPpH0mpltnqR8X0kvSZooqb+kOyQ9aGaHhwsZAAAUijAtEiMl3e+cG+Wcmy7p\nPEmrJSUbNuV8STOdc5c4575wzt0t6ZnGzwEAAEUsrUTCzNrLP5w6MbLN+YEoJkjaJ8lhezfuj/Za\nM+UBAECRSPfxz80ltZW0OG77Ykk7JTmmV5LyXc2so3NubYJjOknStGnT0gyvvNXU1Ki6OvsTvZUa\n6i191Fk41Fv6qLP0RV07czK6RqGOI9FXks4444w8h1F8KpINhYZmUW/po87Cod7SR52F1lfS5Gyf\nJN1E4htJ9ZJ6xm3vKWlRkmMWJSm/IklrhORvfZwuabak2jRjBACgnHWSTyJey8XJ0koknHPrzKxK\n0iBJYyTJzKxx/c4kh70v6adx245o3J7sPMskPZlsPwAAaFbWWyIiwjy1caukEWZ2ppntLOk+SRtI\nelSSzOwGM3ssqvx9kvqZ2V/MbCczu0DSSY2fAwAAiljafSScc081jhlxjfwtio8lDXbOLW0s0kvS\n1lHlZ5vZEEm3Sfq1pHmSznbOxT/JAQAAikxBTiMOAACKA3NtAACA0EgkAABAaAWXSKQ7IVgpM7Or\nzKwh7vWfuDLXmNkCM1ttZq+b2fZx+zua2d1m9o2ZrTSzZ8ysR25/kuwxswPMbIyZzW+sn2MSlGl1\nHZnZJmb2hJnVmNl3ZvagmXXJ9s+XLS3Vm5k9kuC7Ny6uTFnVm5n93symmtkKM1tsZs+b2Y4JyvF9\na5RKnfFda8rMzjOzTxp/lhozm2xmP4krUzDfs4JKJCzNCcHKxGfynVp7Nb72j+wws0sl/UrSOZL2\nlLRKvr46RB1/u6Qhkk6UdKCkLSQ9m5PIc6OLfIffCyQ16fCTwTp6UtIu8o86D2ksd38mf5Aca7be\nGr2i2O/e0Lj95VZvB0i6S9Jekg6T1F7SeDPrHCnA962JFuusEd+1WHMlXSppoPy0FG9IetHMdpEK\n8HvmnCuYl6QPJN0RtW7yT3lcku/Y8lQfV0mqbmb/Akkjo9a7Sloj6ZSo9bWSjo8qs5OkBkl75vvn\ny0J9NUg6JtN11PgPrUHS7lFlBktaL6lXvn/uLNXbI5Kea+YY6s1PGdAgaX++b62qM75rqdXdMkln\nFeL3rGBaJCzchGDlYIfG5ucZZjbazLaWJDPbVj5zj66vFZKmKKivPeQf8Y0u84WkOSqDOs1gHe0t\n6Tvn3EdRHz9B/i/5vbIVfwE4uLE5erqZ3WNmm0btqxD1trH8z/KtxPctRTF1FoXvWhJm1sbMTpMf\nr2lyIX7PCiaRUPMTgvXKfTgF4QNJv5DPEs+TtK2kSY33sHrJ/8Kbq6+ekuoav2TJypSyTNVRL0lL\nonc65+rl/zMs1Xp8RdKZkg6VdImkgySNMzNr3N9LZVxvjfVwu6R3nXORfkt835qRpM4kvmsJmdlu\nZrZSvmXhHvnWhS9UgN+zQp20C5Kcc9HjpH9mZlMlfS3pFEnT8xMVyoFz7qmo1c/N7N+SZkg6WNKb\neQmqsNwjaVdJ++U7kCKSsM74riU1XVJ/Sd3kR4MeZWYH5jekxAqpRSLMhGBlxTlXI+lLSdvL14mp\n+fpaJKmDmXVtpkwpy1QdLZIU39u5raRNVR71KOfcLPl/o5Ge4WVbb2b2N0lHSjrYObcwahfftySa\nqbMm+K55zrn1zrmZzrmPnHN/kH/44EIV4PesYBIJ59w6SZEJwSTFTAiWs8lHCpmZbSj/j2tB4z+2\nRYqtr67y97Yi9VUl33EmusxOkrZRM5OmlYoM1tH7kjY2s92jPn6Q/D/mKdmKv5CY2VaSNpMUuQiU\nZb01XhCPlXSIc25O9D6+b4k1V2dJyvNdS6yNpI4F+T3Ld0/UuF6pp0haLX+/bGf5x1CWSeqe79jy\nVB83yz+O00fSvpJel7/HtVnj/ksa6+doST+U9IKk/0rqEPUZ90iaJd9MWCHpPUnv5Ptny2AddZFv\n/hsg3wP5N43rW2eyjiSNk/QvST+Wb5r9QtLj+f75s1Fvjftukv+PqU/jfy7/kjRNUvtyrbfGn/c7\n+Ucae0a9OkWV4fuWRp3xXUtab9c31lkfSbtJukE+MTi0EL9nea+wBBV4gaTZ8o+yvC9pj3zHlMe6\nqJR//HWNfG/bJyVtG1fmavlHgVbLzz2/fdz+jvLPcX8jaaWkpyX1yPfPlsE6Okj+Qlgf93o4k3Uk\n39t8tKSaxv8YH5C0Qb5//mzUm6ROkl6V/6unVtJMSfcqLqEvt3pLUl/1ks6MK8f3LcU647uWtN4e\nbKyLNY11M16NSUQhfs+YtAsAAIRWMH0kAABA8SGRAAAAoZFIAACA0EgkAABAaCQSAAAgNBIJAAAQ\nGokEAAAIjUQCAACERiIBAABCI5EAAAChkUgAAIDQ/j+t6ZyfPTJP5wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2113556ff60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot(loss_track)\n",
    "print('loss: {:.4f}'.format(loss_track[-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}