BPA_Demo_v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sensitivity/Specificity analysis demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we configure several variables we are using during this demo: project id, S3 profile and VM path to save data files from the S3 bucket: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "project = 'bpa-ThermoFisher_P0001_T1'\n",
    "profile = 'bloodpac'\n",
    "path    = 'files/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Querying data from graphQL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are some examples of queries to directly interact with the API to get some metadata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "credentials = 'credentials.json'    \n",
    "json_data=open(credentials).read()\n",
    "keys = json.loads(json_data)\n",
    "auth = requests.post('https://data.bloodpac.org/user/credentials/cdis/access_token', json=keys)   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A) Query general API to get projects included in BloodPAC:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print requests.get('https://data.bloodpac.org/api/v0/submission/bpa/', \n",
    "                   headers={'Authorization': 'bearer '+ auth.json()['access_token']}).text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**B) Example graphQL query:** List submitted aligned read files (BAMs) for a specific project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data = {'query': \"\"\"{sample (project_id: \"bpa-ThermoFisher_P0001_T1\") {   \n",
    "                                 _aliquots_count \n",
    "                                 aliquots {  \n",
    "                                     analytes{\n",
    "                                         _read_groups_count \n",
    "                                         read_groups {\n",
    "                                            _submitted_aligned_reads_files_count\n",
    "                                            submitted_aligned_reads_files {\n",
    "                                                file_name\n",
    "                                            }\n",
    "                                         } \n",
    "                                     }\n",
    "                                }\n",
    "                            }\n",
    "                    } \"\"\"};\n",
    "\n",
    "print requests.post('https://data.bloodpac.org/api/v0/submission/graphql/', \n",
    "                    headers={'Authorization': 'bearer '+ auth.json()['access_token']}, \n",
    "                    json=data).text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Exploring metadata through a high level Python library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the Python library with functionality to explore metadata and make some data analyses. Also, add our authentication file to the library. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import bpa_analysis_functions_v2 as bp\n",
    "bp.add_keys('credentials.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we show some functions to explore information in the project's metadata: \n",
    "\n",
    "**A) List samples for this project:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "samples = bp.list_samples(project)\n",
    "bp.arrayTable(samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**B) List files in one project by type:** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vcf_files = bp.list_files_by_type(project,'VCF')\n",
    "bp.SummaryTable(vcf_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**C) Get summary metrics for one field in a specific node:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get expected mutations from contrived_expectation metadata and primary diagnosis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary = bp.query_summary_field(\"contrived_expectation\", \"expected_mutation_gene\")\n",
    "summary = bp.query_summary_field(\"diagnosis\", \"primary_diagnosis\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Data handle and analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we use additional functions to show how we can analyze data from one specific project in BloodPAC containing expected mutations from contrived samples and resulting VCF files. \n",
    "\n",
    "**A) Transfer files from S3 bucket to the VPC virtual machine:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "bp.get_files_from_bucket(project, profile, path, '*.vcf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**B) Calculate quality metrics (sensitivity/sensitivity)**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sample = samples[1]\n",
    "vcf_file = vcf_files[sample][8]\n",
    "metrics = bp.calculate_metrics_vcf(project, path, vcf_file)\n",
    "metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate sensitivity and sensitivity values for all VCFs in the project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "table_metrics, data_metrics = bp.calculate_metrics_all_vcf(project, path, vcf_files, samples[0])\n",
    "table_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove potential germline mutations using a baseline VCF:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "baseline_vcf = 'TFS.HZ.0.0.perc_20ng_TSVC_IonCodeTag_0101.vcf'\n",
    "table_filter_metrics, data_filter_metrics = bp.calculate_metrics_all_vcf(project, path, vcf_files, samples[0], baseline_vcf)\n",
    "table_filter_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Some results are finally visualized in barplots:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "bp.plot_metrics(data_metrics, data_filter_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}