Search - Limits.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Limits\n",
    "RDP Search does impose limits on the size of the result set when requesting for large data sets.  The following examples provide some useful techniques when dealing with results reaching the upper limits imposed by the backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<refinitiv.dataplatform.core.session.desktop_session.DesktopSession at 0x19ab2048>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import refinitiv.dataplatform as rdp\n",
    "import pandas as pd\n",
    "\n",
    "rdp.open_desktop_session('Your API Key here')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.0.0a7.post1'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', 140)\n",
    "rdp.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Grouping\n",
    "There may be instances where the result set contains groups of values for properties based on your request.  For example, if I'm interested in retrieving all exchanges within the USA, I can execute this request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4189010"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = rdp.Search.search(\n",
    "    view = rdp.SearchViews.EquityQuotes,\n",
    "    filter = \"RCSExchangeCountryLeaf eq 'United States'\",\n",
    "    top = 10000,\n",
    "    select = \"ExchangeCode, RIC\"\n",
    ")\n",
    "response.data.total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ExchangeCode</th>\n",
       "      <th>RIC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>IOM</td>\n",
       "      <td>EScv1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>IOM</td>\n",
       "      <td>NQcv1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>IOM</td>\n",
       "      <td>ESc1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>IOM</td>\n",
       "      <td>SPc1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>IOM</td>\n",
       "      <td>NQc1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9995</th>\n",
       "      <td>PNK</td>\n",
       "      <td>MGMB.PK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9996</th>\n",
       "      <td>PNK</td>\n",
       "      <td>VCTM.PK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9997</th>\n",
       "      <td>PNK</td>\n",
       "      <td>DSWR.PK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td>PNK</td>\n",
       "      <td>GHAR.PK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td>PNK</td>\n",
       "      <td>DNXG.PK</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     ExchangeCode      RIC\n",
       "0             IOM    EScv1\n",
       "1             IOM    NQcv1\n",
       "2             IOM     ESc1\n",
       "3             IOM     SPc1\n",
       "4             IOM     NQc1\n",
       "...           ...      ...\n",
       "9995          PNK  MGMB.PK\n",
       "9996          PNK  VCTM.PK\n",
       "9997          PNK  DSWR.PK\n",
       "9998          PNK  GHAR.PK\n",
       "9999          PNK  DNXG.PK\n",
       "\n",
       "[10000 rows x 2 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.data.df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above example, you can see the total available documents is over 4,000,000.  However, due to the nature of the data set, the exchange codes have been repeated which brought back the upper limit of documents within the result set.  **Note**: At the time of this writing, the upper limit has been defined as 10000 result sets.\n",
    "\n",
    "Instead of performing multiple calls and pulling out the unique codes within each result set, I can apply the grouping features offered by Search to significantly reduce the result set returned.  For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ExchangeCode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>IOM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CBT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CBF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NSQ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NYQ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>CCE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>CMX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>CSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>HAM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>NYC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>146 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    ExchangeCode\n",
       "0            IOM\n",
       "1            CBT\n",
       "2            CBF\n",
       "3            NSQ\n",
       "4            NYQ\n",
       "..           ...\n",
       "141          CCE\n",
       "142          CMX\n",
       "143          CSC\n",
       "144          HAM\n",
       "145          NYC\n",
       "\n",
       "[146 rows x 1 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdp.search(\n",
    "    view = rdp.SearchViews.EquityQuotes,\n",
    "    filter = \"RCSExchangeCountryLeaf eq 'United States'\",\n",
    "    top = 10000,\n",
    "    select = \"ExchangeCode\",\n",
    "    group_by = \"ExchangeCode\",    # Exchange codes can be grouped\n",
    "    group_count = 1               # Then limited to 1 for each to create uniqueness\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, I've significantly reduced the result set by grouping which now allows the result set using a single API call.  Using the 'grouping' technique to pull out the unique exchange codes is very useful if you wish to return many other properties as part of your results.  However, if you are stricly after the list of exchange codes, the preferred approach is to use Navigators."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Navigators\n",
    "If the goal of your search is to simply capture the list of exchange codes, then the preferred approach in this case is to use Navigators.  A navigator allows the ability to categorize and summarize properties within the result set.  For example, I can provide a simple navigator where I want to bucket all the exchange codes found within the result set.  You can do this using the following request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "response=rdp.Search.search(\n",
    "    view = rdp.SearchViews.EquityQuotes,\n",
    "    filter = \"RCSExchangeCountryLeaf eq 'United States'\",\n",
    "    top = 0,\n",
    "    navigators = \"ExchangeCode(buckets:1000)\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total exchange codes found: 146\n"
     ]
    }
   ],
   "source": [
    "codes=response.data.raw[\"Navigators\"][\"ExchangeCode\"][\"Buckets\"]\n",
    "print(f'Total exchange codes found: {len(codes)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'Label': 'ONE', 'Count': 1441128},\n",
       " {'Label': 'OPQ', 'Count': 1271714},\n",
       " {'Label': 'IOM', 'Count': 821526},\n",
       " {'Label': 'PNK', 'Count': 70284},\n",
       " {'Label': 'CBT', 'Count': 57871},\n",
       " {'Label': 'OBB', 'Count': 32934},\n",
       " {'Label': 'OTC', 'Count': 22418},\n",
       " {'Label': 'BOS', 'Count': 18419},\n",
       " {'Label': 'THM', 'Count': 17951},\n",
       " {'Label': 'XPH', 'Count': 15776},\n",
       " {'Label': 'MID', 'Count': 14700},\n",
       " {'Label': 'PSE', 'Count': 14630},\n",
       " {'Label': 'NYS', 'Count': 14232},\n",
       " {'Label': 'NYQ', 'Count': 12716},\n",
       " {'Label': 'CIN', 'Count': 12680},\n",
       " {'Label': 'NMS', 'Count': 10987},\n",
       " {'Label': 'NAS', 'Count': 10159},\n",
       " {'Label': 'NTV', 'Count': 10142},\n",
       " {'Label': 'BZX', 'Count': 10035},\n",
       " {'Label': 'BYX', 'Count': 10034},\n",
       " {'Label': 'WCB', 'Count': 9915},\n",
       " {'Label': 'IUS', 'Count': 9377},\n",
       " {'Label': 'BT1', 'Count': 8304},\n",
       " {'Label': 'ADC', 'Count': 8272},\n",
       " {'Label': 'ARC', 'Count': 7735},\n",
       " {'Label': 'ASE', 'Count': 7254},\n",
       " {'Label': 'BAT', 'Count': 6976},\n",
       " {'Label': 'DEA', 'Count': 6188},\n",
       " {'Label': 'DEX', 'Count': 6188},\n",
       " {'Label': 'BTY', 'Count': 6078},\n",
       " {'Label': 'CME', 'Count': 6015},\n",
       " {'Label': 'NXB', 'Count': 5294},\n",
       " {'Label': 'CBF', 'Count': 4963},\n",
       " {'Label': 'NBN', 'Count': 4758},\n",
       " {'Label': 'ISS', 'Count': 4400},\n",
       " {'Label': 'GSX', 'Count': 4355},\n",
       " {'Label': 'IEX', 'Count': 4287},\n",
       " {'Label': 'GSB', 'Count': 4174},\n",
       " {'Label': 'BEX', 'Count': 3631},\n",
       " {'Label': 'LCB', 'Count': 3621},\n",
       " {'Label': 'MPO', 'Count': 3319},\n",
       " {'Label': 'SCC', 'Count': 3284},\n",
       " {'Label': 'LTE', 'Count': 3279},\n",
       " {'Label': 'MMX', 'Count': 3275},\n",
       " {'Label': 'SCB', 'Count': 3253},\n",
       " {'Label': 'MPE', 'Count': 3220},\n",
       " {'Label': 'ADS', 'Count': 3176},\n",
       " {'Label': 'NSM', 'Count': 3111},\n",
       " {'Label': 'GSC', 'Count': 3104},\n",
       " {'Label': 'GSP', 'Count': 3092},\n",
       " {'Label': 'GSD', 'Count': 3088},\n",
       " {'Label': 'GSM', 'Count': 3087},\n",
       " {'Label': 'NSQ', 'Count': 3084},\n",
       " {'Label': 'LCC', 'Count': 3068},\n",
       " {'Label': 'SCP', 'Count': 2993},\n",
       " {'Label': 'SCX', 'Count': 2941},\n",
       " {'Label': 'GCZ', 'Count': 2871},\n",
       " {'Label': 'ISJ', 'Count': 2864},\n",
       " {'Label': 'GSA', 'Count': 2789},\n",
       " {'Label': 'GCD', 'Count': 2747},\n",
       " {'Label': 'GDA', 'Count': 2747},\n",
       " {'Label': 'ADF', 'Count': 2730},\n",
       " {'Label': 'GCY', 'Count': 2726},\n",
       " {'Label': 'ASQ', 'Count': 2696},\n",
       " {'Label': 'LCX', 'Count': 2592},\n",
       " {'Label': 'SCM', 'Count': 2590},\n",
       " {'Label': 'NMQ', 'Count': 2493},\n",
       " {'Label': 'LCP', 'Count': 2483},\n",
       " {'Label': 'NAQ', 'Count': 2431},\n",
       " {'Label': 'SCZ', 'Count': 2413},\n",
       " {'Label': 'SDA', 'Count': 2234},\n",
       " {'Label': 'SCD', 'Count': 2233},\n",
       " {'Label': 'GSW', 'Count': 2213},\n",
       " {'Label': 'SCY', 'Count': 2212},\n",
       " {'Label': 'IES', 'Count': 2088},\n",
       " {'Label': 'GSN', 'Count': 1916},\n",
       " {'Label': 'SCA', 'Count': 1830},\n",
       " {'Label': 'SIE', 'Count': 1795},\n",
       " {'Label': 'LCM', 'Count': 1757},\n",
       " {'Label': 'SCN', 'Count': 1667},\n",
       " {'Label': 'GSI', 'Count': 1660},\n",
       " {'Label': 'LTS', 'Count': 1660},\n",
       " {'Label': 'XMS', 'Count': 1654},\n",
       " {'Label': 'MPS', 'Count': 1639},\n",
       " {'Label': 'XMC', 'Count': 1422},\n",
       " {'Label': 'LTM', 'Count': 1419},\n",
       " {'Label': 'MPM', 'Count': 1390},\n",
       " {'Label': 'LCI', 'Count': 1324},\n",
       " {'Label': 'CFT', 'Count': 1237},\n",
       " {'Label': 'LCZ', 'Count': 1217},\n",
       " {'Label': 'LCD', 'Count': 1036},\n",
       " {'Label': 'LDA', 'Count': 1036},\n",
       " {'Label': 'SCW', 'Count': 1005},\n",
       " {'Label': 'LCY', 'Count': 1000},\n",
       " {'Label': 'LCW', 'Count': 950},\n",
       " {'Label': 'NYF', 'Count': 926},\n",
       " {'Label': 'NYM', 'Count': 827},\n",
       " {'Label': 'LCA', 'Count': 817},\n",
       " {'Label': 'SCI', 'Count': 782},\n",
       " {'Label': 'MGE', 'Count': 724},\n",
       " {'Label': 'NYL', 'Count': 703},\n",
       " {'Label': 'IEG', 'Count': 690},\n",
       " {'Label': 'LCN', 'Count': 636},\n",
       " {'Label': 'IMM', 'Count': 612},\n",
       " {'Label': 'PKN', 'Count': 585},\n",
       " {'Label': 'BBA', 'Count': 577},\n",
       " {'Label': 'LTG', 'Count': 563},\n",
       " {'Label': 'XMG', 'Count': 561},\n",
       " {'Label': 'MPG', 'Count': 554},\n",
       " {'Label': 'NBA', 'Count': 508},\n",
       " {'Label': 'PCQ', 'Count': 417},\n",
       " {'Label': 'CC3', 'Count': 370},\n",
       " {'Label': 'PKC', 'Count': 338},\n",
       " {'Label': 'EUS', 'Count': 199},\n",
       " {'Label': 'PKL', 'Count': 185},\n",
       " {'Label': 'QBB', 'Count': 158},\n",
       " {'Label': 'KBT', 'Count': 152},\n",
       " {'Label': 'BBM', 'Count': 93},\n",
       " {'Label': 'PBT', 'Count': 77},\n",
       " {'Label': 'CMX', 'Count': 40},\n",
       " {'Label': 'NYC', 'Count': 29},\n",
       " {'Label': 'BTQ', 'Count': 23},\n",
       " {'Label': 'MSI', 'Count': 14},\n",
       " {'Label': 'DJI', 'Count': 13},\n",
       " {'Label': 'ICQ', 'Count': 13},\n",
       " {'Label': 'QXE', 'Count': 12},\n",
       " {'Label': 'AOE', 'Count': 9},\n",
       " {'Label': 'NSF', 'Count': 9},\n",
       " {'Label': 'CSC', 'Count': 8},\n",
       " {'Label': 'MKX', 'Count': 4},\n",
       " {'Label': 'WCQ', 'Count': 4},\n",
       " {'Label': 'HAM', 'Count': 3},\n",
       " {'Label': 'BOX', 'Count': 2},\n",
       " {'Label': 'PAO', 'Count': 2},\n",
       " {'Label': 'PHO', 'Count': 2},\n",
       " {'Label': 'PQX', 'Count': 2},\n",
       " {'Label': 'QXP', 'Count': 2},\n",
       " {'Label': 'AOI', 'Count': 1},\n",
       " {'Label': 'ASQ|NYQ', 'Count': 1},\n",
       " {'Label': 'CCE', 'Count': 1},\n",
       " {'Label': 'ELX', 'Count': 1},\n",
       " {'Label': 'ISO', 'Count': 1},\n",
       " {'Label': 'MXP', 'Count': 1},\n",
       " {'Label': 'NML', 'Count': 1},\n",
       " {'Label': 'NYQ|UCQ', 'Count': 1},\n",
       " {'Label': 'QXI', 'Count': 1}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Segmenting the search\n",
    "When we started with the above search to retrieve the list of exchange codes within the United States, we discovered that the result set returned the entire universe of instruments.  If our goal is to capture the entire instrument list, we cannot group and bucket the result set as we did above.  The # of hits returned is over 4 million so we are forced to go through a tedious process of segmenting the requests.\n",
    "\n",
    "One way to do this is to choose some kind of indicator that will allow you to group your individual requests to successfully segment the result set.  However, you need to first ask yourself - do I need the entire data universe?  You may only be interested in a specific asset category thus reducing the universe of results significantly.\n",
    "\n",
    "One possible way to approach this is to first capture the list of asset categories using a navigator on the property: 'RCSAssetCategoryLeaf'.  \n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'Label': 'Equity Future', 'Count': 1459351},\n",
       " {'Label': 'Equity Cash Option', 'Count': 1327063},\n",
       " {'Label': 'Stock Index Future Option', 'Count': 705436},\n",
       " {'Label': 'Ordinary Share', 'Count': 375056},\n",
       " {'Label': 'Stock Index Cash Option', 'Count': 70981},\n",
       " {'Label': 'American Depository Receipt', 'Count': 27709},\n",
       " {'Label': 'Unit', 'Count': 22923},\n",
       " {'Label': 'Equity Future Option', 'Count': 21068},\n",
       " {'Label': 'Preferred Share', 'Count': 18452},\n",
       " {'Label': 'Equity Future Spread', 'Count': 17561},\n",
       " {'Label': 'Stock Index Future', 'Count': 16760},\n",
       " {'Label': 'Preference Share', 'Count': 12795},\n",
       " {'Label': 'Depository Receipt', 'Count': 8018},\n",
       " {'Label': 'Company Warrant', 'Count': 7963},\n",
       " {'Label': 'Depository Share', 'Count': 5949},\n",
       " {'Label': 'Bond', 'Count': 5765},\n",
       " {'Label': 'Equity-Linked Security', 'Count': 5497},\n",
       " {'Label': 'Right', 'Count': 5220},\n",
       " {'Label': 'Fully Paid Ordinary Share', 'Count': 5206},\n",
       " {'Label': 'Index-Linked Security', 'Count': 4436},\n",
       " {'Label': 'Convertible Preference Share', 'Count': 2780},\n",
       " {'Label': 'Currency Cash Option', 'Count': 2048},\n",
       " {'Label': 'Cumulative Preference Share', 'Count': 1592},\n",
       " {'Label': 'Global Depository Receipt', 'Count': 1227},\n",
       " {'Label': 'Investment Certificate', 'Count': 1121},\n",
       " {'Label': 'Equity Market Statistics', 'Count': 988},\n",
       " {'Label': 'Enhanced Trust Preferred Security', 'Count': 917},\n",
       " {'Label': 'Guarantee Certificate', 'Count': 434},\n",
       " {'Label': 'Thai Non-Voting Depository Receipt', 'Count': 341},\n",
       " {'Label': 'ETF Future', 'Count': 284},\n",
       " {'Label': 'Subscription Right', 'Count': 280},\n",
       " {'Label': 'Stapled Security', 'Count': 168},\n",
       " {'Label': 'Currency Future', 'Count': 167},\n",
       " {'Label': 'Non-Cumulative Preference Share', 'Count': 162},\n",
       " {'Label': 'CHESS Depository Interest', 'Count': 134},\n",
       " {'Label': 'Redeemable Preference Share', 'Count': 128},\n",
       " {'Label': 'Traditional Warrant', 'Count': 104},\n",
       " {'Label': 'Future', 'Count': 74},\n",
       " {'Label': 'Commodity ETF', 'Count': 61},\n",
       " {'Label': 'Spread', 'Count': 56},\n",
       " {'Label': 'Equity', 'Count': 54},\n",
       " {'Label': 'Warrant', 'Count': 50},\n",
       " {'Label': 'Interest Rate-Linked Security', 'Count': 41},\n",
       " {'Label': 'ETF Option', 'Count': 37},\n",
       " {'Label': 'Commodity-Linked Security', 'Count': 36},\n",
       " {'Label': 'Savings Share', 'Count': 31},\n",
       " {'Label': 'Swedish Depository Receipt', 'Count': 27},\n",
       " {'Label': 'Participating Preference Share', 'Count': 23},\n",
       " {'Label': 'Mexican Ordinary Participation Certificate', 'Count': 18},\n",
       " {'Label': 'Participation Share', 'Count': 17},\n",
       " {'Label': 'Dutch Certificate', 'Count': 15},\n",
       " {'Label': 'Peruvian Investment Share', 'Count': 11},\n",
       " {'Label': 'Stock Dividend', 'Count': 10},\n",
       " {'Label': 'Brazilian Unit', 'Count': 8},\n",
       " {'Label': 'Deferred Share', 'Count': 8},\n",
       " {'Label': 'Brazilian Depository Receipt', 'Count': 5},\n",
       " {'Label': 'Tracker Certificate', 'Count': 5},\n",
       " {'Label': 'Genussschein', 'Count': 4},\n",
       " {'Label': 'Open-End Fund', 'Count': 4},\n",
       " {'Label': 'Redemption Share', 'Count': 4},\n",
       " {'Label': 'European Depository Receipt', 'Count': 3},\n",
       " {'Label': 'Partly Paid Ordinary Share', 'Count': 3},\n",
       " {'Label': 'Philippine Depository Receipt', 'Count': 3},\n",
       " {'Label': 'Commodity Fundamentals', 'Count': 2},\n",
       " {'Label': 'Company Option', 'Count': 2},\n",
       " {'Label': 'Express Certificate', 'Count': 2},\n",
       " {'Label': 'Ordinary Subscription Receipt', 'Count': 2},\n",
       " {'Label': 'Paid Subscription Share', 'Count': 2},\n",
       " {'Label': 'Commodity Future', 'Count': 1},\n",
       " {'Label': 'Currency Future Option', 'Count': 1},\n",
       " {'Label': 'Income Deposit Security', 'Count': 1},\n",
       " {'Label': 'Index Future Spread', 'Count': 1},\n",
       " {'Label': 'Nil Paid Ordinary Share', 'Count': 1},\n",
       " {'Label': 'Option Right', 'Count': 1}]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response=rdp.Search.search(\n",
    "    view = rdp.SearchViews.EquityQuotes,\n",
    "    filter = \"RCSExchangeCountryLeaf eq 'United States'\",\n",
    "    top = 0,\n",
    "    navigators = \"RCSAssetCategoryLeaf\"\n",
    ")\n",
    "response.data.raw['Navigators']['RCSAssetCategoryLeaf']['Buckets']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of this will not only provide the complete list of categories for you to potentially select the desired ones, but for each, you can see the number of results.  This will further allow you to tune your requests based on these totals.\n",
    "\n",
    "However, the above summary shows many categories that easily exceed the limits of the server.  If you need to further segment, you can possibly use the ***market cap*** to segment a specific asset category.\n",
    "\n",
    "For example, let's choose an asset category where we can get a breakdown of the market cap:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'Label': 'Below 2647630.5',\n",
       "  'Filter': 'MktCapTotal lt 2647630.5',\n",
       "  'Count': 9164},\n",
       " {'Label': 'Between 2647630.5 And 17056311.54',\n",
       "  'Filter': '(MktCapTotal ge 2647630.5 and MktCapTotal lt 17056311.54)',\n",
       "  'Count': 9121},\n",
       " {'Label': 'Between 17056311.54 And 50778929.76',\n",
       "  'Filter': '(MktCapTotal ge 17056311.54 and MktCapTotal lt 50778929.76)',\n",
       "  'Count': 9181},\n",
       " {'Label': 'Between 50778929.76 And 119790272.11',\n",
       "  'Filter': '(MktCapTotal ge 50778929.76 and MktCapTotal lt 119790272.11)',\n",
       "  'Count': 9134},\n",
       " {'Label': 'Between 119790272.11 And 241796034.39',\n",
       "  'Filter': '(MktCapTotal ge 119790272.11 and MktCapTotal lt 241796034.39)',\n",
       "  'Count': 9133},\n",
       " {'Label': 'Between 241796034.39 And 434408606.88',\n",
       "  'Filter': '(MktCapTotal ge 241796034.39 and MktCapTotal lt 434408606.88)',\n",
       "  'Count': 9169},\n",
       " {'Label': 'Between 434408606.88 And 800080346.66',\n",
       "  'Filter': '(MktCapTotal ge 434408606.88 and MktCapTotal lt 800080346.66)',\n",
       "  'Count': 9142},\n",
       " {'Label': 'Between 800080346.66 And 1468717120.04',\n",
       "  'Filter': '(MktCapTotal ge 800080346.66 and MktCapTotal lt 1468717120.04)',\n",
       "  'Count': 9146},\n",
       " {'Label': 'Between 1468717120.04 And 2678502775.78',\n",
       "  'Filter': '(MktCapTotal ge 1468717120.04 and MktCapTotal lt 2678502775.78)',\n",
       "  'Count': 9169},\n",
       " {'Label': 'Between 2678502775.78 And 4996190693.14',\n",
       "  'Filter': '(MktCapTotal ge 2678502775.78 and MktCapTotal lt 4996190693.14)',\n",
       "  'Count': 9146},\n",
       " {'Label': 'Between 4996190693.14 And 13308327910.29',\n",
       "  'Filter': '(MktCapTotal ge 4996190693.14 and MktCapTotal lt 13308327910.29)',\n",
       "  'Count': 9157},\n",
       " {'Label': 'Above 13308327910.29',\n",
       "  'Filter': 'MktCapTotal ge 13308327910.29',\n",
       "  'Count': 9147}]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The following navigator will prepare the buckets of evenly distributed market cap ranges such that they fulfill \n",
    "# the limit requirements.  Below, I chose 12 as this will produce reasonable buckets we can work with.\n",
    "response=rdp.Search.search(\n",
    "    view = rdp.SearchViews.EquityQuotes,\n",
    "    filter = \"RCSExchangeCountryLeaf eq 'United States' and RCSAssetCategoryLeaf xeq 'Ordinary Share'\",\n",
    "    top = 0,\n",
    "    navigators = \"MktCapTotal(type:range, buckets:12)\"\n",
    ")\n",
    "response.data.raw[\"Navigators\"][\"MktCapTotal\"][\"Buckets\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing to note is that the 'Count' values for each bucket are within the valid limit of the server.  Based on this output, we can use the convenient Filter expressions provided to drive our segmented search requests.\n",
    "\n",
    "For demonstration purposes, I will select one to retrieve the list of RICs for the specific asset category with the specified market cap range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"RCSExchangeCountryLeaf eq 'United States' and RCSAssetCategoryLeaf xeq 'Ordinary Share' and (MktCapTotal ge 2647630.5 and MktCapTotal lt 17056311.54)\""
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define our filter\n",
    "range1 = response.data.raw[\"Navigators\"][\"MktCapTotal\"][\"Buckets\"][1][\"Filter\"]\n",
    "filter = f\"RCSExchangeCountryLeaf eq 'United States' and RCSAssetCategoryLeaf xeq 'Ordinary Share' and {range1}\"\n",
    "filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Request resulted in a segement of 9121 documents.'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = rdp.Search.search(\n",
    "    view = rdp.SearchViews.EquityQuotes,\n",
    "    filter = filter,\n",
    "    top = 10000\n",
    ")\n",
    "f'Request resulted in a segement of {response.data.total} documents.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the buckets I defined, I can now safely use a filter to pull out a segment of instruments.  Despite using a combination of navigators and filters to conveniently define how to break up the segments to avoid these limits, the work to do so is still relatively complicated.\n",
    "\n",
    "While it may be possible to pull out excessive amounts of data, you should ask yourself if you need to do this.  In most cases, you may be able to reduce the result set when you set up your search instead of pulling in everything then massage the results once you have them in hand.  Search was designed specifically to allow users to filter out unwanted content prior to returning the results.  If you think this way through your searching patterns, you will undoubtedly avoid situations where you need to create complicated algorithms to unnecessarily pull excessive amounts of data. Whether narrowing the request based on interested categories, or data for a specific region, you will find that you can significantly simplify your logic and avoid issues with limits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}