text data Quora Question Pairs EDA & XGBoost.ipynb

{
  "cells": [
    {
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true
      },
      "cell_type": "markdown",
      "source": "# About dataset\nHere goal is to identify which questions asked on Quora, a quasi-forum website with over 100 million visitors a month, are duplicates of questions that have already been asked. This could be useful, for example, to instantly provide answers to questions that have already been answered. \n\n\nHere our task is predicting whether a pair of questions are duplicates or not, and submitting a binary prediction against the logloss metric.\n### loading libraries\n"
    },
    {
      "metadata": {
        "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
        "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
        "trusted": true
      },
      "cell_type": "code",
      "source": "import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport gc\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\npal = sns.color_palette()\n\nprint('# File sizes')\nfor f in os.listdir('../input'):\n    if 'zip' not in f:\n        print(f.ljust(30) + str(round(os.path.getsize('../input/' + f) / 1000000, 2)) + 'MB')\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "fac9310d31d28b2bcb0a28103e45293c9c8a8250"
      },
      "cell_type": "markdown",
      "source": "#### traing dataset"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "94cd248324c05c5b90b650445c9398efc5a36c9c"
      },
      "cell_type": "code",
      "source": "df_train = pd.read_csv('../input/train.csv')\ndf_train.head()\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "e0b58b77ddc4d798259a4edfd24b20e42744d74f"
      },
      "cell_type": "markdown",
      "source": "Here we have--\nid: Looks like a simple rowID\nqid{1, 2}: The unique ID of each question in the pair\nquestion{1, 2}: The actual textual contents of the questions.\nis_duplicate: The label that we are trying to predict - whether the two questions are duplicates of each other."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b5a8a5af4dd10e6f053c92c6e8f32913ba92c530"
      },
      "cell_type": "code",
      "source": "print('Total number of question pairs for training: {}'.format(len(df_train)))\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "fcccda852fda51c8f9ae50d06e56f95e607f0da8"
      },
      "cell_type": "code",
      "source": "\nprint('Duplicate pairs: {}%'.format(round(df_train['is_duplicate'].mean()*100, 2)))\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "845aa865e7f1ca064e7f5c0afe16cc330acf1b4c"
      },
      "cell_type": "code",
      "source": "\nqids = pd.Series(df_train['qid1'].tolist() + df_train['qid2'].tolist())\n\nprint('Total number of questions in the training data: {}'.format(len(\n    np.unique(qids))))\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "053a9ece9887119a6776e548b08737f43d75c269"
      },
      "cell_type": "code",
      "source": "\nprint('Number of questions that appear multiple times: {}'.format(np.sum(qids.value_counts() > 1)))\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "bf36b65139a8f9cce7f920b062904792dc299684"
      },
      "cell_type": "code",
      "source": "\nplt.figure(figsize=(12, 5))\nplt.hist(qids.value_counts(), bins=50)\nplt.yscale('log', nonposy='clip')\nplt.title('Log-Histogram of question appearance counts')\nplt.xlabel('Number of occurences of question')\nplt.ylabel('Number of questions')\nprint()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "a31dce8774647a2e55a5182a5f763b302f7d3473"
      },
      "cell_type": "markdown",
      "source": "Most questions only appear a few times, with very few questions appearing several times (and a few questions appearing many times). One question appears more than 160 times, but this is an outlier."
    },
    {
      "metadata": {
        "_uuid": "1237ccfc75380d40708bfc39578eebd602c6db59"
      },
      "cell_type": "markdown",
      "source": "### Test Submission"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b4e72ef667eabc1e944661bba2f2fbb0709f88d6"
      },
      "cell_type": "code",
      "source": "\nfrom sklearn.metrics import log_loss\n\np = df_train['is_duplicate'].mean() # Our predicted probability\nprint('Predicted score:', log_loss(df_train['is_duplicate'], np.zeros_like(df_train['is_duplicate']) + p))\n\ndf_test = pd.read_csv('../input/test.csv')\nsub = pd.DataFrame({'test_id': df_test['test_id'], 'is_duplicate': p})\nsub.to_csv('naive_submission.csv', index=False)\nsub.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "68ac10567ec86cfd5910f73c946fbc433ba30ff8"
      },
      "cell_type": "markdown",
      "source": "The discrepancy between our local score and the LB one indicates that the distribution of values on the leaderboard is very different to what we have here, which could cause problems with validation later on in the competition.\n"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "f3f22abb730a20d881dc84193adb37f9b0a5816f"
      },
      "cell_type": "markdown",
      "source": "## Test Set\nNow concentrating on the test data"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b5488f0be98688e31e6eeb16d06596d84d19ce22"
      },
      "cell_type": "code",
      "source": "df_test = pd.read_csv('../input/test.csv')\ndf_test.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "66a52b93c066f93f7e1df4316ab2ed6da74b2761"
      },
      "cell_type": "code",
      "source": "print('Total number of question pairs for training: {}'.format(len(df_test)))\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "a5a9708e95f3fa5d736db4d14b7c3529fcd86755"
      },
      "cell_type": "code",
      "source": "\nplt.figure(figsize=(12, 5))\nplt.hist(qids.value_counts(), bins=50)\nplt.yscale('log', nonposy='clip')\nplt.title('Log-Histogram of question appearance counts')\nplt.xlabel('Number of occurences of question')\nplt.ylabel('Number of questions')\nprint()\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0a33fb0295d787130312939a2dcce51698c6ca0b"
      },
      "cell_type": "markdown",
      "source": "### Text analysis\nFirst off, some quick histograms to understand what we're looking at. \n\n\nWe are doing most analysis here will be only on the training set, to avoid the auto-generated questions\n"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "a5dd11ba8206b10e686c1c7e0e2fcbe57e1828d4"
      },
      "cell_type": "code",
      "source": "\ntrain_qs = pd.Series(df_train['question1'].tolist() + df_train['question2'].tolist()).astype(str)\n\ntest_qs = pd.Series(df_test['question1'].tolist() + df_test['question2'].tolist()).astype(str)\n\ndist_train = train_qs.apply(len)\ndist_test = test_qs.apply(len)\nplt.figure(figsize=(15, 10))\nplt.hist(dist_train, bins=200, range=[0, 200], color=pal[2], normed=True, label='train')\nplt.hist(dist_test, bins=200, range=[0, 200], color=pal[1], normed=True, alpha=0.5, label='test')\nplt.title('Normalised histogram of character count in questions', fontsize=15)\nplt.legend()\nplt.xlabel('Number of characters', fontsize=15)\nplt.ylabel('Probability', fontsize=15)\n\nprint('mean-train {:.2f} std-train {:.2f} mean-test {:.2f} std-test {:.2f} max-train {:.2f} max-test {:.2f}'.format(dist_train.mean(), \n                          dist_train.std(), dist_test.mean(), dist_test.std(), dist_train.max(), dist_test.max()))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "b10cb46abba68e42b5a557c9cab1be45214c1211"
      },
      "cell_type": "markdown",
      "source": "We can see that most questions have anywhere from 15 to 150 characters in them. It seems that the test distribution is a little different from the train one, but not too much "
    },
    {
      "metadata": {
        "_uuid": "419aee598783ddef9c4e8efd7073374966ca47e4"
      },
      "cell_type": "markdown",
      "source": "### Plotting word count for train and test dataset"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "4ebb30553eabb7001dc9182fdf463978d2bbb11f"
      },
      "cell_type": "code",
      "source": "#Word count for train and test data\ndist_train = train_qs.apply(lambda x: len(x.split(' ')))\ndist_test = test_qs.apply(lambda x: len(x.split(' ')))\n\nplt.figure(figsize=(15, 10))\nplt.hist(dist_train, bins=50, range=[0, 50], color=pal[2], normed=True, label='train')\nplt.hist(dist_test, bins=50, range=[0, 50], color=pal[1], normed=True, alpha=0.5, label='test')\nplt.title('Normalised histogram of word count in questions', fontsize=15)\nplt.legend()\nplt.xlabel('Number of words', fontsize=15)\nplt.ylabel('Probability', fontsize=15)\n\nprint('mean-train {:.2f} \\n std-train {:.2f} \\n mean-test {:.2f} \\n std-test {:.2f} \\n max-train {:.2f} \\n max-test {:.2f}\\n'.format(dist_train.mean(), \n                          dist_train.std(), dist_test.mean(), dist_test.std(), dist_train.max(), dist_test.max()))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "db41ebdc72d5c792532fd209249918c32dc447e7"
      },
      "cell_type": "markdown",
      "source": "#### Here most questions being about 10 words long"
    },
    {
      "metadata": {
        "_uuid": "e8f332db005ffba8acc1addd758df2a233817abd"
      },
      "cell_type": "markdown",
      "source": "### Let's see the word cloud of most common word"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "9a2508d33064f75ff8ab3c9e22b294043e104d86"
      },
      "cell_type": "code",
      "source": "#So what are the most common words? Let's take a look at a word cloud.\n\nfrom wordcloud import WordCloud\ncloud = WordCloud(width=1440, height=1080).generate(\" \".join(train_qs.astype(str)))\nplt.figure(figsize=(20, 15))\nplt.imshow(cloud)\nplt.axis('off')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "e669fd0a385863a9bbaf6d712a26f169df0e2d97"
      },
      "cell_type": "markdown",
      "source": "## Initial Feature Analysis\nBefore we create a model, we should take a look at how powerful some features are. I will start off with the word share feature from the benchmark model.\n"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6fc5ebbea919cdc31d4607627c9dd29e5b8ff4a4"
      },
      "cell_type": "code",
      "source": "\nfrom nltk.corpus import stopwords\n\nstops = set(stopwords.words(\"english\"))\n\ndef word_match_share(row):\n    q1words = {}\n    q2words = {}\n    for word in str(row['question1']).lower().split():\n        if word not in stops:\n            q1words[word] = 1\n    for word in str(row['question2']).lower().split():\n        if word not in stops:\n            q2words[word] = 1\n    if len(q1words) == 0 or len(q2words) == 0:\n        # The computer-generated chaff includes a few questions that are nothing but stopwords\n        return 0\n    shared_words_in_q1 = [w for w in q1words.keys() if w in q2words]\n    shared_words_in_q2 = [w for w in q2words.keys() if w in q1words]\n    R = (len(shared_words_in_q1) + len(shared_words_in_q2))/(len(q1words) + len(q2words))\n    return R\n\nplt.figure(figsize=(15, 5))\ntrain_word_match = df_train.apply(word_match_share, axis=1, raw=True)\nplt.hist(train_word_match[df_train['is_duplicate'] == 0], bins=20, normed=True, label='Not Duplicate')\nplt.hist(train_word_match[df_train['is_duplicate'] == 1], bins=20, normed=True, alpha=0.7, label='Duplicate')\nplt.legend()\nplt.title('Label distribution over word_match_share', fontsize=15)\nplt.xlabel('word_match_share', fontsize=15)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "586b379125a8162897713f38f2da11f0be3e6cce"
      },
      "cell_type": "markdown",
      "source": "Here we can see that this feature has quite a lot of predictive power, as it is good at separating the duplicate questions from the non-duplicate ones. Interestingly, it seems very good at identifying questions which are definitely different, but is not so great at finding questions which are definitely duplicates.\n\n"
    },
    {
      "metadata": {
        "_uuid": "2db1f274fac9f757d78d8e23eb87002140670731"
      },
      "cell_type": "markdown",
      "source": "## TF-IDF\nNow time  to improve this feature, by using something called TF-IDF (term-frequency-inverse-document-frequency). \nThis means that we weigh the terms by how uncommon they are, meaning that we care more about rare words existing in both questions than common one. This makes sense, \nas for example we care more about whether the word \"exercise\" appears in both than the word \"and\" - as uncommon words will be more indicative of the content.\n\n"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6a14f00f3639be340fae37c00d6cb7872c8d2af1"
      },
      "cell_type": "code",
      "source": "from collections import Counter\n\n# If a word appears only once, we ignore it completely (likely a typo)\n# Epsilon defines a smoothing constant, which makes the effect of extremely rare words smaller\ndef get_weight(count, eps=10000, min_count=2):\n    if count < min_count:\n        return 0\n    else:\n        return 1 / (count + eps)\n\neps = 5000 \nwords = (\" \".join(train_qs)).lower().split()\ncounts = Counter(words)\nweights = {word: get_weight(count) for word, count in counts.items()}",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "7cfd8618d8fca0d8668dd9ccdd0d115cfbb62a8c"
      },
      "cell_type": "code",
      "source": "print('Most common words and weights: \\n')\nprint(sorted(weights.items(), key=lambda x: x[1] if x[1] > 0 else 9999)[:10])\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6b5482fd60d5eaa880a964f0bc6e5901bbd0b5de"
      },
      "cell_type": "code",
      "source": "print('\\nLeast common words and weights: ')\n(sorted(weights.items(), key=lambda x: x[1], reverse=True)[:10])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "42a406eae5563e9c72be807746fee67e08cb4967"
      },
      "cell_type": "code",
      "source": "",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ac07b106391a961d0f060ae5f32db0243482162b"
      },
      "cell_type": "code",
      "source": "",
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.6",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}