Sequence Classification with LSTM Recurrent Neural Networks with Keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence Classification with LSTM Recurrent Neural Networks with Keras\n",
    "\n",
    "Sequence classification is a type of predictive modelling problem whereby you have some sequence of inputs and the task is to predict a category that the sequence belongs to.\n",
    "\n",
    "The challenge of such a problem is the varying sequence lengths. There could be sequences with very large vocabulary of input symbols and as such the model then has to learn long term context and/or dependencies between the symbols in the input sequence. In this project, I develop an LSTM recurrent neural network model for sequence classification problems in Python, using keras.\n",
    "\n",
    "What makes this problem difficult is that the sequences can vary in length, be comprised of a very large vocabulary of input symbols and may require the model to learn the long term context or dependencies between symbols in the input sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Framing the problem\n",
    "\n",
    "The problem used in this report will be based on the [IMDB movie review sentiment classification problem](http://ai.stanford.edu/~amaas/data/sentiment/). In this problem, each movie review is a variable sequence of words, and the sentiment of the movie reviews has to be classified.\n",
    "\n",
    "The Large Movie Review Dataset (often referred to as the IMDB dataset) contains 25,000 highly-polar movie reviews (good or bad) for training and the same amount again for testing. The problem is thus to determine whether a given movie review has a positive or negative sentiment.\n",
    "\n",
    "Keras provides access to the IMDB dataset built-in. The `imdb.load_data()` function is leveraged for this. The words have been replaced by integers that indicate the ordered frequency of each word in the dataset. The sentences in each review are therefore comprised of a sequence of integers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word Embedding\n",
    "\n",
    "Each movie review gets projected into a different space (a real vector domain) onto a representation, using word embedding. This is where the words are encoded as real - valued vectors in a high dimensional space, to capture the notion of similarity between words in a (Euclidean) distance setting.\n",
    "\n",
    "Keras provides a convenient way to convert positive integer representations of words into a word embedding by an Embedding layer. Each word gets projected onto a 32 length real - valued vector. Also, for computational reasons, we limit the total number of words we model to the 5000 most frequent words (first 5000 of the dataset, due to the token ordering method). And, the sequence length in each review varies, so we constrain each review to be maximum 500 words, and the shorter reviews are padded with `0` values.\n",
    "\n",
    "We can now develop an LSTM model to classify the sentiment of movie reviews."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple LSTM model, for Sequence Classification\n",
    "\n",
    "This is a basic, small LSTM for the IMDB problem and achieves good accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy\n",
    "from keras.datasets import imdb\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM\n",
    "from keras.layers import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "\n",
    "# fix random seed for reproducibility\n",
    "numpy.random.seed(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are constraining the dataset to the top 5,000 words. We also split the dataset into train (50%) and test (50%) sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The `nb_words` argument in `load_data` has been renamed `num_words`.\n"
     ]
    }
   ],
   "source": [
    "# load the dataset but only keep the top n words, zero the rest\n",
    "top_words = 5000\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=top_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we truncate and pad the input sequences so that they are all the same length for modelling purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# truncate and pad input sequences\n",
    "max_review_length = 500\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=max_review_length)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=max_review_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now define, compile and fit our LSTM model.\n",
    "\n",
    "Of our model, the first layer is the Embedding layer, using (real - valued) vectors of length 32 to represent each word. Following this layer is the LSTM layer with 100 memory units (smart neurons). Then, the final layer is a dense/linear output layer with just one neuron. The activation function used is a sigmoid function, that makes binary predictions for the two classes in the problem.\n",
    "\n",
    "In terms of the optimiser and loss function used, as this is a binary classification problem, the log loss is used as the loss function (which is `binary_crossentropy` in Keras), and the `Adam` optimisation is utilised. In this model, only 2 epochs are used, as the model overfits fairly quickly. The batch size is 64."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_1 (Embedding)     (None, 500, 32)           160000    \n",
      "                                                                 \n",
      " lstm_1 (LSTM)               (None, 100)               53200     \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 1)                 101       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 213301 (833.21 KB)\n",
      "Trainable params: 213301 (833.21 KB)\n",
      "Non-trainable params: 0 (0.00 Byte)\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/2\n",
      "391/391 [==============================] - 314s 800ms/step - loss: 0.4670 - accuracy: 0.7705 - val_loss: 0.3476 - val_accuracy: 0.8564\n",
      "Epoch 2/2\n",
      "391/391 [==============================] - 310s 792ms/step - loss: 0.3090 - accuracy: 0.8794 - val_loss: 0.3057 - val_accuracy: 0.8748\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x1e521e62850>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create the model\n",
    "embedding_vecor_length = 32\n",
    "model = Sequential()\n",
    "model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))\n",
    "model.add(LSTM(100))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "print(model.summary())\n",
    "model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=2, batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is now fit, and we can see how the model performs on the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 87.48%\n"
     ]
    }
   ],
   "source": [
    "# Final evaluation of the model\n",
    "scores = model.evaluate(X_test, y_test, verbose=0)\n",
    "print(\"Accuracy: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This simple LSTM with a small amount of tuning achieves excellent results on the IMDB problem. This is the foundation for more complex models that can be explored."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSTM for Sequence Classification With Dropout\n",
    "\n",
    "Typically, recurrent neural networks, such as LSTM have the problem of overfitting.\n",
    "\n",
    "In order to combat this, dropout is applied between the layers. In Keras this is done smoothly. We simply add new dropout layers between the embedding and LSTM layers, and between the LSTM and dense output layers. Furthermore, we add dropout to the input of the embeded layer, just using a dropout parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# LSTM with Dropout for sequence classification in the IMDB dataset\n",
    "import numpy\n",
    "from keras.datasets import imdb\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM\n",
    "from keras.layers import Dropout\n",
    "from keras.layers import Embedding\n",
    "from keras.preprocessing import sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The `nb_words` argument in `load_data` has been renamed `num_words`.\n"
     ]
    }
   ],
   "source": [
    "# fix random seed for reproducibility\n",
    "numpy.random.seed(7)\n",
    "\n",
    "# load the dataset but only keep the top n words, zero the rest\n",
    "top_words = 5000\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=top_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_3 (Embedding)     (None, 500, 32)           160000    \n",
      "                                                                 \n",
      " dropout_2 (Dropout)         (None, 500, 32)           0         \n",
      "                                                                 \n",
      " lstm_3 (LSTM)               (None, 100)               53200     \n",
      "                                                                 \n",
      " dropout_3 (Dropout)         (None, 100)               0         \n",
      "                                                                 \n",
      " dense_3 (Dense)             (None, 1)                 101       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 213301 (833.21 KB)\n",
      "Trainable params: 213301 (833.21 KB)\n",
      "Non-trainable params: 0 (0.00 Byte)\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/2\n",
      "391/391 [==============================] - 258s 654ms/step - loss: 0.4588 - accuracy: 0.7780\n",
      "Epoch 2/2\n",
      "391/391 [==============================] - 256s 654ms/step - loss: 0.3150 - accuracy: 0.8738\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x1e52149a280>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# truncate and pad input sequences\n",
    "max_review_length = 500\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=max_review_length)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=max_review_length)\n",
    "\n",
    "# create the model\n",
    "embedding_vecor_length = 32\n",
    "model = Sequential()\n",
    "model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(LSTM(100))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "print(model.summary())\n",
    "model.fit(X_train, y_train, epochs=2, batch_size=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 87.56%\n"
     ]
    }
   ],
   "source": [
    "# Final evaluation of the model\n",
    "scores = model.evaluate(X_test, y_test, verbose=0)\n",
    "print(\"Accuracy: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSTM and Convolutional Neural Networks for Sequence Classification\n",
    "\n",
    "Convolutional neural networks are strong at ascertaining spatial structure within input data.\n",
    "\n",
    "For this dataset, the IMDB review data does indeed have a one - dimensional spatial structure. The CNN may be able to identify latent features for good and bad sentiment. The LSTM layer can then learn these latent features during training.\n",
    "\n",
    "We can then add a one - dimensional CNN as well as max pooling layers, after the embedding layer. These feed into the consolidated features, onto the LSTM. We use a small set of 32 features, with a filter length of 3. The pooling layer uses the standard legnth of 2, in order to reduce the feature map size by half, achieving a succesful dimensional reduction from the feature map.\n",
    "\n",
    "We can easily add a one-dimensional CNN and max pooling layers after the Embedding layer which then feed the consolidated features to the LSTM. We can use a smallish set of 32 features with a small filter length of 3. The pooling layer can use the standard length of 2 to halve the feature map size.\n",
    "\n",
    "The full code listing with a CNN and LSTM layers is listed below for completeness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LSTM and CNN for sequence classification in the IMDB dataset\n",
    "import numpy\n",
    "from keras.datasets import imdb\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM\n",
    "from keras.layers import Conv1D, MaxPooling1D\n",
    "from keras.layers import Embedding\n",
    "from keras.preprocessing import sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The `nb_words` argument in `load_data` has been renamed `num_words`.\n"
     ]
    }
   ],
   "source": [
    "# load the dataset but only keep the top n words, zero the rest\n",
    "top_words = 5000\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=top_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# truncate and pad input sequences\n",
    "max_review_length = 500\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=max_review_length)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=max_review_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_13\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_11 (Embedding)    (None, 500, 32)           160000    \n",
      "                                                                 \n",
      " conv1d_4 (Conv1D)           (None, 500, 32)           3104      \n",
      "                                                                 \n",
      " max_pooling1d_3 (MaxPoolin  (None, 250, 32)           0         \n",
      " g1D)                                                            \n",
      "                                                                 \n",
      " lstm_7 (LSTM)               (None, 100)               53200     \n",
      "                                                                 \n",
      " dense_7 (Dense)             (None, 1)                 101       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 216405 (845.33 KB)\n",
      "Trainable params: 216405 (845.33 KB)\n",
      "Non-trainable params: 0 (0.00 Byte)\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/3\n",
      "391/391 [==============================] - 98s 244ms/step - loss: 0.4008 - accuracy: 0.8051\n",
      "Epoch 2/3\n",
      "391/391 [==============================] - 98s 251ms/step - loss: 0.2380 - accuracy: 0.9065\n",
      "Epoch 3/3\n",
      "391/391 [==============================] - 105s 269ms/step - loss: 0.1957 - accuracy: 0.9258\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x1e514910940>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create the model\n",
    "embedding_vecor_length = 32\n",
    "model = Sequential()\n",
    "model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))\n",
    "model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))\n",
    "model.add(MaxPooling1D(pool_size=2))\n",
    "model.add(LSTM(100))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "print(model.summary())\n",
    "model.fit(X_train, y_train, epochs=3, batch_size=64) #I would choose 3, but due to computational reasons onl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 88.72%\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Final evaluation of the model\n",
    "scores = model.evaluate(X_test, y_test, verbose=0)\n",
    "print(\"Accuracy: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that we achieve similar results to the first example although with less weights and faster training time.\n",
    "\n",
    "I would expect that even better results could be achieved if this example was further extended to use dropout."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}