Web-Scraping-Notebook-Full.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web Scraping Walkthrough - Curating Healthcare Professionals' Information with Python and Selenium\n",
    "___\n",
    "Link to Medium article: https://medium.com/swlh/web-scrapping-healthcare-professionals-information-1372385d639d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Motivation**  \n",
    "Analytics is not possible without data, and web scraping is one of the many tools out there for us to curate data. The concept of web scraping has always fascinated me, and I felt it would be fun to practice my coding chops while exploring the public information of registered healthcare professionals in Singapore.\n",
    "\n",
    "I also notice that most existing web scraping tutorials tend to be overly brief, therefore a more detailed walkthrough would certainly be beneficial for everyone.\n",
    "\n",
    "The Ministry of Health (MOH) Professional Registration System site provides public access to the information of healthcare professionals. As with most search sites, the records are not available for viewing in its entirety, and that makes for a good web scrapping practice.\n",
    "\n",
    "There are different sites for the different categories of healthcare practitioners (e.g. doctors, pharmacists, nurses, dentists etc). Being a pharmacist myself, it was natural for me to first test things out on the Pharmacists dataset. \n",
    "___\n",
    "\n",
    "**Objective**  \n",
    "Use Python and Selenium to curate a list of healthcare professionals registered in Singapore, along with their respective public information.\n",
    "___\n",
    "\n",
    "**What is Web Scraping**  \n",
    "Web scraping refers to the process of extracting content and data from a website. This information is collected and then exported into a format that is more useful for the user. This process can be done manually, but the amount of effort involved can make it impractical to do so (e.g. costly, slow, human error etc). This is where automated web scraping comes into play, delivering efficiency and accuracy in data extraction.  \n",
    "___\n",
    "\n",
    "**DISCLAIMER**  \n",
    "This tutorial is purely for educational purposes only. The information accessed is based on **publicly available content** provided on the MOH site. The contents extracted has not been (and will not be) reproduced, republished, uploaded, posted, transmitted or otherwise distributed in any way. Please refer to the Terms of Service of the MOH website for more details - https://www.healthprofessionals.gov.sg/spc/terms-of-use\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table of Contents\n",
    "- [Section 1 - Setup](#section1) \n",
    "- [Section 2 - Initial Exploration](#section2)  \n",
    "- [Section 3 - Running Main Script](#section3)  \n",
    "- [Section 4 - References](#section4)  \n",
    " ___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"section1\"></a>\n",
    "## Section 1 - Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before starting any web scraping, let's make sure that the essential dependencies and drivers are correctly installed and imported. I will briefly explain the three key common components involved in web scraping."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(1) BeautifulSoup**  \n",
    "\n",
    "BeautifulSoup is a Python library used to extract data out of HTML and XML files. You will notice that many web scraping tutorials do mention about this library, so all scrapers should have an understanding of it. This Wiki page gives a good summary of BeautifulSoup - https://en.wikipedia.org/wiki/Beautiful_Soup_(HTML_parser)\n",
    "\n",
    "We will be utilizing BeautifulSoup4 (BS) just for the initial exploration section, purely to briefly demonstrate how it can be used. Here is the command to get the BS installation going.\n",
    "\n",
    "```\n",
    "pip install beautifulsoup4\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(2) Selenium Setup**  \n",
    "\n",
    "Selenium is a powerful open-sourced web-based automation tool for controlling web browsers and performing browser automation. Essentially what it means is that you can use it to automate repetitive actions throughout the website, such as clicking buttons, filling forms, extracting text, scrolling page etc. This host of Selenium functions allows for step-by-step interactions with a webpage and assessment of the response of a browser to various changes. \n",
    "\n",
    "Selenium can be installed on pip with this command:\n",
    "\n",
    "```\n",
    "pip install selenium\n",
    "```\n",
    "\n",
    "More information on Selenium can be found here: https://selenium-python.readthedocs.io/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(3) Chrome WebDriver**\n",
    "\n",
    "The Selenium API (described above) requires the WebDriver protocol to control a web browser, like Chrome, Firefox or Safari. WebDriver provides a platform and language-neutral wire protocol as a way for out-of-process programs to remotely instruct the behavior of web browsers. For this project, I utilized the Chrome WebDriver since Google Chrome is one of the most popular browsers around.\n",
    "\n",
    "- First check the Chrome version you are currently running, by visiting https://www.whatismybrowser.com/detect/what-version-of-chrome-do-i-have.  \n",
    "\n",
    "\n",
    "- Next, visit https://chromedriver.chromium.org/downloads to download the appropriate WebDriver for Chrome (which comes in an .exe file). For example, my Chrome version is version 86, and so I downloaded ChromeDriver 86.0.4240.22. Do note that the chromedriver.exe file in this repo is that of version 86.0.4240.22 as well.  \n",
    "\n",
    "\n",
    "- Make sure that the chromedriver.exe application is in the same folder as your script or notebook.\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the above installations complete, it is time to import the dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import urllib\n",
    "import re\n",
    "import time\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "from selenium import webdriver\n",
    "from selenium.common.exceptions import NoSuchElementException\n",
    "from selenium.common.exceptions import ElementClickInterceptedException\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "from selenium.webdriver.common.action_chains import ActionChains"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "<a name=\"section2\"></a>\n",
    "## Section 2 - Initial Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although different websites will invariably have different functionalities and design, they still do contain numerous commonalities within the structures since there is always HTML/CSS involved. \n",
    "\n",
    "Thus, the first step (before any coding) is to understand the website structure. That means you should first browse the site and click the various buttons and links to see how they work and where they bring you. It is only upon having a clear understanding of the site flow that you are able to build any form of script to automate the scraping process. For the purpose of this exercise, we want to do the following: \n",
    "\n",
    "**Step 1**: Click the *'Search'* button (*without* keying any input in the two search bar fields above) to load all the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/homepage_searchbar.png\" width=\"70%\" align = \"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 2**: Click the *'View More Details'* link to access the professional's public details, and you will be redirected to the dedicated details page for that individual."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/view_more_details.png\" width=\"70%\" align = \"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 3**: On the details page, we can see the text fields which we can scrape. Once done, click the 'Back to Search Results' link at the bottom to head back to the Search Results page, and repeat this with the next record on that page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/detailed_info.png\" width=\"70%\" align = \"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 4**: Once you are done with the page, click the next number on the pagination row to go to the next page of records. For example, if you are already on page 1, then click '2' to go to the next page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/search_results_next_page.png\" width=\"88%\" align = \"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One important thing to note is that the next set of pages in the pagination section (located at bottom left of the page) will only appear upon clicking a number closer to the last number on the existing row. For example, to go to page 14, you will need to first click '10' of (Page 1 2 3 4 5 6 7 8 9 10 ..), after which the next pagination row will appear, which is (Page .. 5 6 7 8 9 10 11 12 13 14 ..).\n",
    "\n",
    "This is actually a source of quite a bit of pain, which you will witness later on.\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After understanding how the target website works, the next step is to explore the underlying HTML structure of the page. Let's put BeautifulSoup into action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "<html>\n",
       "<head>\n",
       "<title>Professional Registration System</title>\n",
       "<script src=\"https://assets.wogaa.sg/scripts/wogaa.js\"></script>\n",
       "<meta content=\"IE=9\" http-equiv=\"X-UA-Compatible\" id=\"metaCompatible\"/><!-- added for PRS-7074 on Apr23,2014  -->\n",
       "<link href=\"/prs/css/site.css\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<meta content=\"text/html; charset=utf-8\" http-equiv=\"Content-Type\"/>\n",
       "<meta content=\"noindex, nofollow\" name=\"robots\"/>\n",
       "</head>\n",
       "<script language=\"Javascript1.1\" type=\"text/javascript\">\n",
       "    var today = new Date();\n",
       "    var expired = new Date(today.getTime() - 48 * 60 * 60 * 1000); // less 2 days\n",
       "    var bikky = document.cookie;\n",
       "\tvar isInSameFrame = true;\n",
       "\n",
       "    function deleteCookie(attribute) {\n",
       "        document.cookie = attribute + \"=null; path=/; expires=\"\n",
       "                + expired.toGMTString();\n",
       "        bikky = document.cookie;\n",
       "    }\n",
       "\n",
       "    function dode() {\n",
       "        deleteCookie(\"AA_JSessionInfo_Cookie_IP\");\n",
       "        deleteCookie(\"AA_JSessionInfo_Cookie\");\n",
       "    }\n",
       "</script>\n",
       "<style> html{display : none ; } </style>\n",
       "<script> \n",
       "\t\tif( self == top ) { \n",
       "\t\t\tdocument.documentElement.style.display = 'block' ; \n",
       "\t\t} else { \n",
       "\t\t\ttop.location = self.location ;\n",
       "\t\t}\n",
       "\t</script>\n",
       "<frameset border=\"0\" cols=\"*\" frameborder=\"NO\" framespacing=\"0\" id=\"mainFrameSet\" rows=\"131,*,43\">\n",
       "<frame name=\"header\" noresize=\"\" scrolling=\"NO\" src=\"/prs/internet/common/FrameHeaderNoLogin.action\"/>\n",
       "<frameset border=\"0\" cols=\"*,*,822,*,*\" frameborder=\"NO\" framespacing=\"0\" id=\"bottomFrameSet\" rows=\"*\">\n",
       "<frame name=\"blank1\" noresize=\"\" scrolling=\"NO\" src=\"/prs/internet/common/IeBlankForLayout.action\"/>\n",
       "<frame name=\"blank3\" noresize=\"\" scrolling=\"auto\" src=\"/prs/internet/common/IeBlankForLayout.action\"/>\n",
       "<frame name=\"msg_main\" noresize=\"\" scrolling=\"auto\" src=\"/prs/internet/profSearch/showSearchSummaryByName.action\"/>\n",
       "<frame name=\"blank4\" noresize=\"\" scrolling=\"auto\" src=\"/prs/internet/common/IeBlankForLayout.action\"/>\n",
       "<frame name=\"blank2\" noresize=\"\" scrolling=\"NO\" src=\"/prs/internet/common/IeBlankForLayout.action\"/>\n",
       "</frameset>\n",
       "<frame name=\"footer\" noresize=\"\" scrolling=\"NO\" src=\"/prs/internet/common/FrameFooter.action\"/>\n",
       "</frameset>\n",
       "</html>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define target website\n",
    "home_page = \"https://prs.moh.gov.sg/prs/internet/profSearch/main.action?hpe=SPC\"\n",
    "\n",
    "# Use urllib to open home page, and BeautifulSoup to parse HTML\n",
    "home_page_content = urllib.request.urlopen(home_page)\n",
    "home_page_html = BeautifulSoup(home_page_content, 'html.parser')\n",
    "home_page_html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: Having basic HTML knowledge will help you greatly in digesting the chunk of HTML code above.*  \n",
    "\n",
    "Looking at the HTML returned above, it seems odd that we are unable to find the 'Search' button. What I did next was to head back to the page on the browser to inspect the page. To do that, you can click the F12 button (or press Ctrl + Shift + I), and the following screen will appear. The HTML is fully displayed under the Elements tab, and you can click further into the dropdowns to have a detailed look at the entire layout structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/inspect_page.png\" width=\"88%\" align = \"left\">\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is only upon this further inspection that I discovered there is a need to go deeper into this frame named `msg_main`:\n",
    "\n",
    "```\n",
    "<frame name=\"msg_main\" noresize=\"\" scrolling=\"auto\" src=\"/prs/internet/profSearch/showSearchSummaryByName.action\"/>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/deeper_meme.png\" width=\"50%\" align = \"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is because this particular HTML frame contains the 'Search' button element we are looking to activate. With that in mind, it's now time to call upon Selenium and Chrome WebDriver to do a test run to access this deeper frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting up options for WebDriver\n",
    "options = webdriver.ChromeOptions()\n",
    "options.add_argument('--ignore-certificate-errors')\n",
    "\n",
    "# Initiate WebDriver\n",
    "driver = webdriver.Chrome(options=options)\n",
    "\n",
    "# Direct WebDriver to the target home page\n",
    "driver.get(home_page)\n",
    "driver.switch_to.frame(driver.find_element_by_name('msg_main'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code will initiate the WebDriver, and you will see a new Chrome window appearing on your screen. In that window, you are able to see that Chrome recognizes an automation test is about to begin. Remember not to close this window, as WebDriver will be using this window to perform the automated tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/automated_test.png\" width=\"60%\" align = \"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following that, by using the `switch_to.frame()` and `find_element_by_name` methods, we can then access the HTML code within the frame we are targeting. These two methods are part of a whole host of Selenium functions for us to leverage, and you can visit the Selenium documentation page to explore what is available. For example, to find out how to locate elements on a website, you can check out this specific page: https://selenium-python.readthedocs.io/locating-elements.html. I will be showcasing more of these Selenium functions later.\n",
    "\n",
    "After accessing the frame, the next step is to utilize BeautifulSoup to extract the HTML from the current page the WebDriver is at. Remember that our objective is still to find out where the 'Search' button is, so that we can understand how to activate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<html lang=\"en\" style=\"height:100%\" xml:lang=\"en\" xmlns=\"http://www.w3.org/1999/xhtml\"><head>\n",
       "<title>Professionals Search</title>\n",
       "<meta content=\"IE=9\" http-equiv=\"X-UA-Compatible\"/>\n",
       "<meta content=\"text/html; charset=utf-8\" http-equiv=\"Content-Type\"/>\n",
       "<meta content=\"no-cache\" http-equiv=\"Cache-Control\"/>\n",
       "<meta content=\"no-cache\" http-equiv=\"Pragma\"/>\n",
       "<meta content=\"0\" http-equiv=\"Expires\"/>\n",
       "<meta content=\"noindex, nofollow\" name=\"robots\"/>\n",
       "<script async=\"\" src=\"https://assets.wogaa.sg/snowplow/2.14.0/sp.js\"></script><script src=\"https://assets.wogaa.sg/scripts/wogaa.js\"></script><script async=\"\" src=\"https://assets.wogaa.sg/scripts/wogaa.js?url=https%3A%2F%2Fprs.moh.gov.sg%2Fprs%2Finternet%2FprofSearch%2FshowSearchSummaryByName.action\"></script>\n",
       "<link href=\"/prs/css/internet/spc/header.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/public-header.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<!--[if lt IE 8]><link rel=\"stylesheet\" type=\"text/css\" href=\"/prs/css/internet/ie.css\" media=\"screen\"/><![endif]-->\n",
       "<link href=\"/prs/css/internet/footer.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/public.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/spc/main.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/public-main.css?version=20151125\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/psearch.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<script src=\"/prs/scripts/jquery.js\" type=\"text/javascript\"></script><script async=\"\" src=\"//assets.adobedtm.com/launch-ENaf340d988e354d18ba897b99e3538f23.min.js\"></script><script async=\"\" src=\"https://assets.adobedtm.com/extensions/EPbde2f7ca14e540399dcc1f8208860b7b/AppMeasurement.min.js\"></script><script async=\"\" src=\"https://assets.adobedtm.com/extensions/EPbde2f7ca14e540399dcc1f8208860b7b/AppMeasurement_Module_ActivityMap.min.js\"></script><script async=\"\" src=\"https://assets.wogaa.sg/datalayer.min.js\"></script>\n",
       "<script src=\"/prs/scripts/jquery.watermark.min.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/jquery.inputlimiter.js?version=20141111\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/iextend.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/common.js?version=2\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/uiTimer.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/mdetect.js\" type=\"text/javascript\"></script>\n",
       "<style type=\"text/css\">\n",
       "    @media print {\n",
       "        .notPrint {\n",
       "            display:none;\n",
       "        }\n",
       "        .path {\n",
       "            display:none;\n",
       "        }\n",
       "    }\n",
       "    </style>\n",
       "<script type=\"text/javascript\">\n",
       "\t$(function () {\n",
       "\t    $(\"#searchsite\").watermark(\"Enter search keywords here\", \"watermark\");\n",
       "\t});\n",
       "\t</script>\n",
       "</head>\n",
       "<body class=\"page\" style=\"height:100%; background-attachment:fixed;\">\n",
       "<!--<div id=\"header-wrapper\">\n",
       "\t<div id=\"header\">\n",
       "\t\t<div class=\"header-bar\">\n",
       "\t\t\t<div class=\"logo-area\">\n",
       "\t\t\t\t<h1 class=\"logo\"><a href=\"#\">Singapore Pharmacy Council</a></h1>\n",
       "\t\t\t\t<strong class=\"slogan\">Singapore Pharmacy Council</strong>\n",
       "\t\t\t</div>\n",
       "\t\t\t<div id=\"access\" class=\"visuallyhidden\">\n",
       "\t\t\t\t<a href=\"#search-1\" title=\"Search\">Search</a> | <a href=\"#content\" title=\"Skip to content\">Skip to content</a> | <a href=\"#footer\" title=\"Skip to footer site map\">Skip to footer site map</a>\n",
       "\t\t\t</div>\n",
       "\t\t\t<div class=\"header-section\">\n",
       "\t\t\t\t<div class=\"ad-box\">\n",
       "\t\t\t\t\t<a href=\"#\"><img src=\"/prs/images/ad-image1.gif\" alt=\"Singapore Government\" width=\"155\" height=\"27\" /></a>\n",
       "\t\t\t\t</div>\n",
       "\t                  <ul class=\"top-menu\">\n",
       "\t\t\t\t\t<li><a href=\"#\">Contact</a></li>\n",
       "\t\t\t\t\t<li><a href=\"#\">Feedback</a></li>\n",
       "\t\t\t\t\t<li><a href=\"#\">Sitemap</a></li>\n",
       "\t\t\t\t</ul>\n",
       "\t\t\t\t\n",
       "\t\t\t</div>\n",
       "\t\t</div>\n",
       "\t</div>\n",
       "\t<br/><br/>\n",
       "</div>-->\n",
       "<table style=\"width:785px;\">\n",
       "<tbody><tr>\n",
       "<td>\n",
       "<div class=\"main-holder\">\n",
       "<div id=\"content\" style=\"margin:0px;width:100%;\">\n",
       "<div class=\"content-entry\">\n",
       "<div class=\"entry-holder\">\n",
       "<script type=\"text/javascript\">\n",
       "    function resubmit() {\n",
       "    \t$(\"#getSearchSummary\").attr(\"action\", \"getSearchSummaryByName.action\");\n",
       "        $(\"#getSearchSummary\").submit();\n",
       "    }\n",
       "    \n",
       "    function toggleSearchOptions(){\n",
       "    \t$(\"#searchOptions\").toggle();\n",
       "    }\n",
       "    \n",
       "    function DetectDevice(){\n",
       "    \tif (checkMobile()){\n",
       "\t\t\twindow.location = \"mshowSearchSummaryByName.action?hpe=SPC\";\n",
       "\t\t}\n",
       "\t}\n",
       "    \n",
       "    function toggleSpecialtySelect(selectedVal){\n",
       "  \t\tif (selectedVal.value == \"selected\"){\n",
       "  \t\t\t$(\".specialty\").removeAttr(\"disabled\");\n",
       "  \t\t} else {\n",
       "  \t\t\t$(\".specialty\").attr(\"disabled\",true);\n",
       "  \t\t}\n",
       "    }\n",
       "\t\n",
       "    function viewMoreDetails(regNo){\n",
       "    \t$(\"#regNo\").val(regNo);\n",
       "    \t$(\"#getSearchSummary\").attr(\"action\", \"getSearchDetails.action\");\n",
       "    \t$(\"#getSearchSummary\").submit();\n",
       "    }\n",
       "    \n",
       "    function resetSearch(){\n",
       "    \twindow.location = \"resetSearchSummaryByName.action?hpe=SPC\";\n",
       "    }\n",
       "    \n",
       "\tDetectDevice();\n",
       "</script>\n",
       "<form action=\"/prs/internet/profSearch/showSearchSummaryByName.action\" autocomplete=\"off\" id=\"getSearchSummary\" method=\"post\" name=\"getSearchSummary\">\n",
       "<input id=\"hpe\" name=\"hpe\" type=\"hidden\" value=\"SPC\"/>\n",
       "<input id=\"regNo\" name=\"regNo\" type=\"hidden\" value=\"\"/>\n",
       "<input id=\"getSearchSummary_psearchParamVO_language\" name=\"psearchParamVO.language\" type=\"hidden\" value=\"eng\"/>\n",
       "<input id=\"getSearchSummary_psearchParamVO_searchBy\" name=\"psearchParamVO.searchBy\" type=\"hidden\" value=\"N\"/>\n",
       "<div class=\"article\" id=\"searchProf\" style=\"min-height:100%;\">\n",
       "<div class=\"search-prof-head search-prof-head-bg\">\n",
       "<a class=\"search-prof-padding current-link red-color no-decoration\" href=\"/prs/internet/profSearch/showSearchSummaryByName.action?hpe=SPC\">Search by Name</a>\n",
       "<a class=\"search-prof-padding red-color no-decoration\" href=\"/prs/internet/profSearch/showSearchSummaryByRegion.action?hpe=SPC\">Search by Region</a>\n",
       "</div>\n",
       "<div class=\"search-prof-body search-prof-body-bg\">\n",
       "<table width=\"100%\">\n",
       "<tbody><tr>\n",
       "<td class=\"no-border\" width=\"30%\">Name</td>\n",
       "<td class=\"no-border\" width=\"70%\">\n",
       "<input id=\"getSearchSummary_psearchParamVO_name\" name=\"psearchParamVO.name\" size=\"45%\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "<input id=\"getSearchSummary_psearchParamVO_startWith\" name=\"psearchParamVO.startWith\" type=\"checkbox\" value=\"startWith\"/><input id=\"__checkbox_getSearchSummary_psearchParamVO_startWith\" name=\"__checkbox_psearchParamVO.startWith\" type=\"hidden\" value=\"startWith\"/>\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\tNames starting with...\n",
       "\t\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border\">Name of Place of Practice</td>\n",
       "<td class=\"no-border\">\n",
       "<input id=\"getSearchSummary_psearchParamVO_pracPlaceName\" name=\"psearchParamVO.pracPlaceName\" size=\"45%\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"align-right no-border\" colspan=\"2\"><input name=\"btnSearch\" onclick=\"resubmit();\" type=\"button\" value=\"Search\"/></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border font11px\" colspan=\"2\">\n",
       "<a class=\"red-color no-decoration underline\" href=\"#\" onclick=\"toggleSearchOptions();return false;\">\n",
       "\t\t\t\t\t\t\t► More Search Options\n",
       "\t\t\t\t\t\t</a>\n",
       "</td>\n",
       "</tr>\n",
       "</tbody></table>\n",
       "</div>\n",
       "<div class=\"search-prof-body search-prof-body-bg align-left\" id=\"searchOptions\" style=\"display: none;\">\n",
       "<div class=\"align-right\">\n",
       "\t\t\t\tRegistration Number (Optional)\n",
       "\t\t\t\t<input id=\"getSearchSummary_psearchParamVO_regNo\" name=\"psearchParamVO.regNo\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "</div>\n",
       "<hr/>\n",
       "<div \"=\"\" style=\"display:none;\">\n",
       "<input checked=\"checked\" name=\"selectType\" onclick=\"toggleSpecialtySelect(this)\" type=\"radio\" value=\"all\"/> All Specialties<br/>\n",
       "<input name=\"selectType\" onclick=\"toggleSpecialtySelect(this)\" type=\"radio\" value=\"selected\"/> Selected Specialties\n",
       "\t\t\t</div>\n",
       "<table id=\"specialtyTable\" style=\"display:none;\" width=\"100%\">\n",
       "<tbody><tr>\n",
       "<td class=\"no-border\" colspan=\"2\">Specialties</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"2\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"2\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tCardiology Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"6\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"6\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tCritical Care Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"3\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"3\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tGeriatric Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"4\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"4\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tInfectious Diseases Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"1\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"1\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tOncology Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"7\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"7\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tPaediatric Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"5\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"5\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tPsychiatric Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "<td class=\"no-border\" width=\"50%\"> </td>\n",
       "</tr>\n",
       "</tbody></table>\n",
       "</div>\n",
       "<div class=\"search-result-head\" id=\"searchResultHead\">\n",
       "</div>\n",
       "<script type=\"text/javascript\">\n",
       "displayGoogleMap();\n",
       "</script>\n",
       "</div>\n",
       "</form>\n",
       "</div>\n",
       "</div>\n",
       "</div>\n",
       "</div>\n",
       "</td>\n",
       "</tr>\n",
       "</tbody></table>\n",
       "<!--<div id=\"footer\" class=\"footer-holder\">\n",
       "\t<div class=\"footer-bar\">\n",
       "\t\t<div id=\"privacy\">\n",
       "\t\t\t<ul class=\"footer-menu\">\n",
       "\t\t\t\t<li><a href=\"#\">Privacy Statement</a></li>\n",
       "\t\t\t\t<li><a href=\"#\">Terms of Use</a></li>\n",
       "\t\t\t</ul>\n",
       "\t\t\t<span class=\"copyright\">&copy; Singapore Pharmacy Council</span>\n",
       "\t\t</div>\n",
       "\t</div>\n",
       "</div>-->\n",
       "<script language=\"javascript\">\n",
       "var disableButton = true;\n",
       "function disableAllButtons() {\n",
       "    if(disableButton) {\n",
       "        var buttons = document.getElementsByTagName(\"input\");\n",
       "        for(var i = 0; i < buttons.length; i++) {\n",
       "            var button = buttons[i];\n",
       "            var nodeType = button.type.toLowerCase();\n",
       "            if(nodeType == \"button\") {\n",
       "                button.disabled=true;\n",
       "            }\n",
       "        }\n",
       "    } else {\n",
       "        disableButton = true;\n",
       "    }\n",
       "}\n",
       "$(function() {\n",
       "    $(\"input[name='btnSearch']\").bind(\"click\", disableAllButtons);\n",
       "});\n",
       "</script>\n",
       "</body></html>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Parse HTML of current page source\n",
    "frame_html = BeautifulSoup(driver.page_source, 'html.parser')\n",
    "frame_html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lo and behold, after scrolling through the HTML text above, we can finally see the 'Search' button, and it is coded with: \n",
    "\n",
    "```\n",
    "<input type = \"button\" value=\"Search\" name=\"btnSearch\" onclick\"resubmit();\">\n",
    "```\n",
    "\n",
    "Let us do a simple Selenium function to click this 'Search' button we have found (using `find_element_by_xpath`), and then parse the HTML of the resulting page through BeautifulSoup again. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Locate and click the Search button\n",
    "search_button = driver.find_element_by_xpath(\"//input[@name='btnSearch']\")\n",
    "search_button.click()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will notice that WebDriver Chrome window has loaded the first page of search results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<html lang=\"en\" style=\"height:100%\" xml:lang=\"en\" xmlns=\"http://www.w3.org/1999/xhtml\"><head>\n",
       "<title>Professionals Search</title>\n",
       "<meta content=\"IE=9\" http-equiv=\"X-UA-Compatible\"/>\n",
       "<meta content=\"text/html; charset=utf-8\" http-equiv=\"Content-Type\"/>\n",
       "<meta content=\"no-cache\" http-equiv=\"Cache-Control\"/>\n",
       "<meta content=\"no-cache\" http-equiv=\"Pragma\"/>\n",
       "<meta content=\"0\" http-equiv=\"Expires\"/>\n",
       "<meta content=\"noindex, nofollow\" name=\"robots\"/>\n",
       "<script async=\"\" src=\"https://assets.wogaa.sg/snowplow/2.14.0/sp.js\"></script><script src=\"https://assets.wogaa.sg/scripts/wogaa.js\"></script><script async=\"\" src=\"https://assets.wogaa.sg/scripts/wogaa.js?url=https%3A%2F%2Fprs.moh.gov.sg%2Fprs%2Finternet%2FprofSearch%2FgetSearchSummaryByName.action\"></script>\n",
       "<link href=\"/prs/css/internet/spc/header.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/public-header.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<!--[if lt IE 8]><link rel=\"stylesheet\" type=\"text/css\" href=\"/prs/css/internet/ie.css\" media=\"screen\"/><![endif]-->\n",
       "<link href=\"/prs/css/internet/footer.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/public.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/spc/main.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/public-main.css?version=20151125\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/psearch.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<script src=\"/prs/scripts/jquery.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/jquery.watermark.min.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/jquery.inputlimiter.js?version=20141111\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/iextend.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/common.js?version=2\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/uiTimer.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/mdetect.js\" type=\"text/javascript\"></script>\n",
       "<style type=\"text/css\">\n",
       "    @media print {\n",
       "        .notPrint {\n",
       "            display:none;\n",
       "        }\n",
       "        .path {\n",
       "            display:none;\n",
       "        }\n",
       "    }\n",
       "    </style>\n",
       "<script type=\"text/javascript\">\n",
       "\t$(function () {\n",
       "\t    $(\"#searchsite\").watermark(\"Enter search keywords here\", \"watermark\");\n",
       "\t});\n",
       "\t</script>\n",
       "<script async=\"\" src=\"//assets.adobedtm.com/launch-ENaf340d988e354d18ba897b99e3538f23.min.js\"></script><script async=\"\" src=\"https://assets.adobedtm.com/extensions/EPbde2f7ca14e540399dcc1f8208860b7b/AppMeasurement.min.js\"></script><script async=\"\" src=\"https://assets.adobedtm.com/extensions/EPbde2f7ca14e540399dcc1f8208860b7b/AppMeasurement_Module_ActivityMap.min.js\"></script><script async=\"\" src=\"https://assets.wogaa.sg/datalayer.min.js\"></script></head>\n",
       "<body class=\"page\" style=\"height:100%; background-attachment:fixed;\">\n",
       "<!--<div id=\"header-wrapper\">\n",
       "\t<div id=\"header\">\n",
       "\t\t<div class=\"header-bar\">\n",
       "\t\t\t<div class=\"logo-area\">\n",
       "\t\t\t\t<h1 class=\"logo\"><a href=\"#\">Singapore Pharmacy Council</a></h1>\n",
       "\t\t\t\t<strong class=\"slogan\">Singapore Pharmacy Council</strong>\n",
       "\t\t\t</div>\n",
       "\t\t\t<div id=\"access\" class=\"visuallyhidden\">\n",
       "\t\t\t\t<a href=\"#search-1\" title=\"Search\">Search</a> | <a href=\"#content\" title=\"Skip to content\">Skip to content</a> | <a href=\"#footer\" title=\"Skip to footer site map\">Skip to footer site map</a>\n",
       "\t\t\t</div>\n",
       "\t\t\t<div class=\"header-section\">\n",
       "\t\t\t\t<div class=\"ad-box\">\n",
       "\t\t\t\t\t<a href=\"#\"><img src=\"/prs/images/ad-image1.gif\" alt=\"Singapore Government\" width=\"155\" height=\"27\" /></a>\n",
       "\t\t\t\t</div>\n",
       "\t                  <ul class=\"top-menu\">\n",
       "\t\t\t\t\t<li><a href=\"#\">Contact</a></li>\n",
       "\t\t\t\t\t<li><a href=\"#\">Feedback</a></li>\n",
       "\t\t\t\t\t<li><a href=\"#\">Sitemap</a></li>\n",
       "\t\t\t\t</ul>\n",
       "\t\t\t\t\n",
       "\t\t\t</div>\n",
       "\t\t</div>\n",
       "\t</div>\n",
       "\t<br/><br/>\n",
       "</div>-->\n",
       "<table style=\"width:785px;\">\n",
       "<tbody><tr>\n",
       "<td>\n",
       "<div class=\"main-holder\">\n",
       "<div id=\"content\" style=\"margin:0px;width:100%;\">\n",
       "<div class=\"content-entry\">\n",
       "<div class=\"entry-holder\">\n",
       "<script type=\"text/javascript\">\n",
       "    function resubmit() {\n",
       "    \t$(\"#getSearchSummary\").attr(\"action\", \"getSearchSummaryByName.action\");\n",
       "        $(\"#getSearchSummary\").submit();\n",
       "    }\n",
       "    \n",
       "    function toggleSearchOptions(){\n",
       "    \t$(\"#searchOptions\").toggle();\n",
       "    }\n",
       "    \n",
       "    function DetectDevice(){\n",
       "    \tif (checkMobile()){\n",
       "\t\t\twindow.location = \"mshowSearchSummaryByName.action?hpe=SPC\";\n",
       "\t\t}\n",
       "\t}\n",
       "    \n",
       "    function toggleSpecialtySelect(selectedVal){\n",
       "  \t\tif (selectedVal.value == \"selected\"){\n",
       "  \t\t\t$(\".specialty\").removeAttr(\"disabled\");\n",
       "  \t\t} else {\n",
       "  \t\t\t$(\".specialty\").attr(\"disabled\",true);\n",
       "  \t\t}\n",
       "    }\n",
       "\t\n",
       "    function viewMoreDetails(regNo){\n",
       "    \t$(\"#regNo\").val(regNo);\n",
       "    \t$(\"#getSearchSummary\").attr(\"action\", \"getSearchDetails.action\");\n",
       "    \t$(\"#getSearchSummary\").submit();\n",
       "    }\n",
       "    \n",
       "    function resetSearch(){\n",
       "    \twindow.location = \"resetSearchSummaryByName.action?hpe=SPC\";\n",
       "    }\n",
       "    \n",
       "\tDetectDevice();\n",
       "</script>\n",
       "<form action=\"/prs/internet/profSearch/getSearchSummaryByName.action\" autocomplete=\"off\" id=\"getSearchSummary\" method=\"post\" name=\"getSearchSummary\">\n",
       "<input id=\"hpe\" name=\"hpe\" type=\"hidden\" value=\"SPC\"/>\n",
       "<input id=\"regNo\" name=\"regNo\" type=\"hidden\" value=\"\"/>\n",
       "<input id=\"getSearchSummary_psearchParamVO_language\" name=\"psearchParamVO.language\" type=\"hidden\" value=\"eng\"/>\n",
       "<input id=\"getSearchSummary_psearchParamVO_searchBy\" name=\"psearchParamVO.searchBy\" type=\"hidden\" value=\"N\"/>\n",
       "<div class=\"article\" id=\"searchProf\" style=\"min-height:100%;\">\n",
       "<div class=\"search-prof-head search-prof-head-bg\">\n",
       "<a class=\"search-prof-padding current-link red-color no-decoration\" href=\"/prs/internet/profSearch/showSearchSummaryByName.action?hpe=SPC\">Search by Name</a>\n",
       "<a class=\"search-prof-padding red-color no-decoration\" href=\"/prs/internet/profSearch/showSearchSummaryByRegion.action?hpe=SPC\">Search by Region</a>\n",
       "</div>\n",
       "<div class=\"search-prof-body search-prof-body-bg\">\n",
       "<table width=\"100%\">\n",
       "<tbody><tr>\n",
       "<td class=\"no-border\" width=\"30%\">Name</td>\n",
       "<td class=\"no-border\" width=\"70%\">\n",
       "<input id=\"getSearchSummary_psearchParamVO_name\" name=\"psearchParamVO.name\" size=\"45%\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "<input id=\"getSearchSummary_psearchParamVO_startWith\" name=\"psearchParamVO.startWith\" type=\"checkbox\" value=\"startWith\"/><input id=\"__checkbox_getSearchSummary_psearchParamVO_startWith\" name=\"__checkbox_psearchParamVO.startWith\" type=\"hidden\" value=\"startWith\"/>\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\tNames starting with...\n",
       "\t\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border\">Name of Place of Practice</td>\n",
       "<td class=\"no-border\">\n",
       "<input id=\"getSearchSummary_psearchParamVO_pracPlaceName\" name=\"psearchParamVO.pracPlaceName\" size=\"45%\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"align-right no-border\" colspan=\"2\"><input name=\"btnSearch\" onclick=\"resubmit();\" type=\"button\" value=\"Search\"/></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border font11px\" colspan=\"2\">\n",
       "<a class=\"red-color no-decoration underline\" href=\"#\" onclick=\"toggleSearchOptions();return false;\">\n",
       "\t\t\t\t\t\t\t► More Search Options\n",
       "\t\t\t\t\t\t</a>\n",
       "</td>\n",
       "</tr>\n",
       "</tbody></table>\n",
       "</div>\n",
       "<div class=\"search-prof-body search-prof-body-bg align-left\" id=\"searchOptions\" style=\"display: none;\">\n",
       "<div class=\"align-right\">\n",
       "\t\t\t\tRegistration Number (Optional)\n",
       "\t\t\t\t<input id=\"getSearchSummary_psearchParamVO_regNo\" name=\"psearchParamVO.regNo\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "</div>\n",
       "<hr/>\n",
       "<div \"=\"\" style=\"display:none;\">\n",
       "<input checked=\"checked\" name=\"selectType\" onclick=\"toggleSpecialtySelect(this)\" type=\"radio\" value=\"all\"/> All Specialties<br/>\n",
       "<input name=\"selectType\" onclick=\"toggleSpecialtySelect(this)\" type=\"radio\" value=\"selected\"/> Selected Specialties\n",
       "\t\t\t</div>\n",
       "<table id=\"specialtyTable\" style=\"display:none;\" width=\"100%\">\n",
       "<tbody><tr>\n",
       "<td class=\"no-border\" colspan=\"2\">Specialties</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"2\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"2\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tCardiology Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"6\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"6\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tCritical Care Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"3\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"3\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tGeriatric Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"4\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"4\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tInfectious Diseases Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"1\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"1\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tOncology Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"7\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"7\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tPaediatric Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border font12px\" width=\"50%\">\n",
       "<input class=\"specialty\" disabled=\"disabled\" id=\"getSearchSummary_psearchParamVO_cbSpecialty\" name=\"psearchParamVO.cbSpecialty\" type=\"checkbox\" value=\"5\"/><input disabled=\"disabled\" id=\"__checkbox_getSearchSummary_psearchParamVO_cbSpecialty\" name=\"__checkbox_psearchParamVO.cbSpecialty\" type=\"hidden\" value=\"5\"/>\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tPsychiatric Pharmacy\n",
       "\t\t\t\t\t\t\t</td>\n",
       "<td class=\"no-border\" width=\"50%\"> </td>\n",
       "</tr>\n",
       "</tbody></table>\n",
       "</div>\n",
       "<div class=\"search-result-head\" id=\"searchResultHead\">\n",
       "\t\n",
       "\t\tDisplaying 1 - 10 of 3396 records\n",
       "\t\t<hr/>\n",
       "<table width=\"100%\">\n",
       "<script language=\"Javascript1.1\" type=\"text/javascript\"> \n",
       "function gotoPageDEFAULT( page ) {\n",
       "if (setPaginationCookieDEFAULT) setPaginationCookieDEFAULT();\n",
       "  try { if (setSortOrderCookieDEFAULT) setSortOrderCookieDEFAULT(); } catch(error) {} \n",
       "  document.cookie='cookie.tableId=DEFAULT';\n",
       "  document.cookie='cookie.currentPage=DEFAULT.' + page + ';path=/prs;';\n",
       "  resubmit(); // To be provided by individual applications\n",
       "}\n",
       "function showAllDEFAULT( checkBox ) {\n",
       "  try { if (setSortOrderCookieDEFAULT) setSortOrderCookieDEFAULT(); } catch(error) {}\n",
       "  document.cookie='cookie.tableId=DEFAULT';\n",
       "  document.cookie='cookie.showAll=DEFAULT.' + checkBox.checked + ';path=/prs;';\n",
       "  document.cookie='cookie.currentPage=DEFAULT.1;path=/prs;';\n",
       "  resubmit(); // To be provided by individual applications\n",
       "}\n",
       "function setPaginationCookieDEFAULT () {\n",
       "};\n",
       "</script>\n",
       "<tbody><tr>\n",
       "<td class=\"table-head no-border\">\n",
       "<div class=\"font15px\">\t\n",
       "\t\t\t\t\t\t\tAARON CHEW SONG TA (P02392B)\n",
       "\t\t\t\t\t\t</div>\n",
       "<div class=\"bold font11px\">\n",
       "<a href=\"#\" onclick=\"viewMoreDetails('P02392B'); return false;\">View more details</a>\n",
       "</div>\n",
       "<span class=\"bold font11px\">Type of Register</span>   \n",
       "\t\t\t\t\t\t<span class=\"font11px\">Pharmacist</span>\n",
       "<hr/>\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"table-head no-border\">\n",
       "<div class=\"font15px\">\t\n",
       "\t\t\t\t\t\t\tAARON JASON MARTIN (P03376F)\n",
       "\t\t\t\t\t\t</div>\n",
       "<div class=\"bold font11px\">\n",
       "<a href=\"#\" onclick=\"viewMoreDetails('P03376F'); return false;\">View more details</a>\n",
       "</div>\n",
       "<span class=\"bold font11px\">Type of Register</span>   \n",
       "\t\t\t\t\t\t<span class=\"font11px\">Pharmacist</span>\n",
       "<hr/>\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"table-head no-border\">\n",
       "<div class=\"font15px\">\t\n",
       "\t\t\t\t\t\t\tAARON YAP JUN YI (P03995J)\n",
       "\t\t\t\t\t\t</div>\n",
       "<div class=\"bold font11px\">\n",
       "<a href=\"#\" onclick=\"viewMoreDetails('P03995J'); return false;\">View more details</a>\n",
       "</div>\n",
       "<span class=\"bold font11px\">Type of Register</span>   \n",
       "\t\t\t\t\t\t<span class=\"font11px\">Pharmacist</span>\n",
       "<hr/>\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"table-head no-border\">\n",
       "<div class=\"font15px\">\t\n",
       "\t\t\t\t\t\t\tABDUL HAMEED S/O ANWARUDEEN (P03945D)\n",
       "\t\t\t\t\t\t</div>\n",
       "<div class=\"bold font11px\">\n",
       "<a href=\"#\" onclick=\"viewMoreDetails('P03945D'); return false;\">View more details</a>\n",
       "</div>\n",
       "<span class=\"bold font11px\">Type of Register</span>   \n",
       "\t\t\t\t\t\t<span class=\"font11px\">Pharmacist</span>\n",
       "<hr/>\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"table-head no-border\">\n",
       "<div class=\"font15px\">\t\n",
       "\t\t\t\t\t\t\tADRIAN TOH SHU KHING (P04296Z)\n",
       "\t\t\t\t\t\t</div>\n",
       "<div class=\"bold font11px\">\n",
       "<a href=\"#\" onclick=\"viewMoreDetails('P04296Z'); return false;\">View more details</a>\n",
       "</div>\n",
       "<span class=\"bold font11px\">Type of Register</span>   \n",
       "\t\t\t\t\t\t<span class=\"font11px\">Pharmacist</span>\n",
       "<hr/>\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"table-head no-border\">\n",
       "<div class=\"font15px\">\t\n",
       "\t\t\t\t\t\t\tADRIENNE CHEW DAN (P03956Z)\n",
       "\t\t\t\t\t\t</div>\n",
       "<div class=\"bold font11px\">\n",
       "<a href=\"#\" onclick=\"viewMoreDetails('P03956Z'); return false;\">View more details</a>\n",
       "</div>\n",
       "<span class=\"bold font11px\">Type of Register</span>   \n",
       "\t\t\t\t\t\t<span class=\"font11px\">Pharmacist</span>\n",
       "<hr/>\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"table-head no-border\">\n",
       "<div class=\"font15px\">\t\n",
       "\t\t\t\t\t\t\tAFIDAH BINTE A MANAF (P01062F)\n",
       "\t\t\t\t\t\t</div>\n",
       "<div class=\"bold font11px\">\n",
       "<a href=\"#\" onclick=\"viewMoreDetails('P01062F'); return false;\">View more details</a>\n",
       "</div>\n",
       "<span class=\"bold font11px\">Type of Register</span>   \n",
       "\t\t\t\t\t\t<span class=\"font11px\">Pharmacist</span><span class=\"font11px\">, Specialist</span>\n",
       "<hr/>\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"table-head no-border\">\n",
       "<div class=\"font15px\">\t\n",
       "\t\t\t\t\t\t\tAINSLEY CHAN (P04294C)\n",
       "\t\t\t\t\t\t</div>\n",
       "<div class=\"bold font11px\">\n",
       "<a href=\"#\" onclick=\"viewMoreDetails('P04294C'); return false;\">View more details</a>\n",
       "</div>\n",
       "<span class=\"bold font11px\">Type of Register</span>   \n",
       "\t\t\t\t\t\t<span class=\"font11px\">Pharmacist</span>\n",
       "<hr/>\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"table-head no-border\">\n",
       "<div class=\"font15px\">\t\n",
       "\t\t\t\t\t\t\tALCARAZ SARAH GRACE SERRANO (P04307I)\n",
       "\t\t\t\t\t\t</div>\n",
       "<div class=\"bold font11px\">\n",
       "<a href=\"#\" onclick=\"viewMoreDetails('P04307I'); return false;\">View more details</a>\n",
       "</div>\n",
       "<span class=\"bold font11px\">Type of Register</span>   \n",
       "\t\t\t\t\t\t<span class=\"font11px\">Pharmacist</span>\n",
       "<hr/>\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"table-head no-border\">\n",
       "<div class=\"font15px\">\t\n",
       "\t\t\t\t\t\t\tALDO WENEDY (P03890C)\n",
       "\t\t\t\t\t\t</div>\n",
       "<div class=\"bold font11px\">\n",
       "<a href=\"#\" onclick=\"viewMoreDetails('P03890C'); return false;\">View more details</a>\n",
       "</div>\n",
       "<span class=\"bold font11px\">Type of Register</span>   \n",
       "\t\t\t\t\t\t<span class=\"font11px\">Pharmacist</span>\n",
       "<hr/>\n",
       "</td>\n",
       "</tr>\n",
       "</tbody></table>\n",
       "<table class=\"noborder\" width=\"100%\">\n",
       "<tbody><tr>\n",
       "<td width=\"50%\">\n",
       "\t\t\t\t\tPage \n",
       "\t\t\t\t\t<label class=\"pagination_selected_page\">1</label>\n",
       "<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(2)\">2</a>\n",
       "<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(3)\">3</a>\n",
       "<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(4)\">4</a>\n",
       "<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(5)\">5</a>\n",
       "<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(6)\">6</a>\n",
       "<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(7)\">7</a>\n",
       "<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(8)\">8</a>\n",
       "<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(9)\">9</a>\n",
       "<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(10)\">10</a>\n",
       "..\n",
       "\n",
       "\t\t\t\t</td>\n",
       "<td align=\"right\" class=\"text_right\" width=\"50%\">\n",
       "\t\t\t\t\t[First]\n",
       " | \n",
       "[Previous]\n",
       " | \n",
       "[<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(2)\">Next</a>]\n",
       " | \n",
       "[<a class=\"pagination\" href=\"javascript:gotoPageDEFAULT(340)\">Last</a>]\n",
       "\n",
       "\t\t\t\t</td>\n",
       "</tr>\n",
       "</tbody></table>\n",
       "<a class=\"red-color no-decoration underline\" href=\"#\">Back to Top</a> |\n",
       "\t        <a class=\"red-color no-decoration underline\" href=\"#\" onclick=\"resetSearch();return false;\">Reset Search</a>\n",
       "</div>\n",
       "<script type=\"text/javascript\">\n",
       "displayGoogleMap();\n",
       "</script>\n",
       "</div>\n",
       "</form>\n",
       "</div>\n",
       "</div>\n",
       "</div>\n",
       "</div>\n",
       "</td>\n",
       "</tr>\n",
       "</tbody></table>\n",
       "<!--<div id=\"footer\" class=\"footer-holder\">\n",
       "\t<div class=\"footer-bar\">\n",
       "\t\t<div id=\"privacy\">\n",
       "\t\t\t<ul class=\"footer-menu\">\n",
       "\t\t\t\t<li><a href=\"#\">Privacy Statement</a></li>\n",
       "\t\t\t\t<li><a href=\"#\">Terms of Use</a></li>\n",
       "\t\t\t</ul>\n",
       "\t\t\t<span class=\"copyright\">&copy; Singapore Pharmacy Council</span>\n",
       "\t\t</div>\n",
       "\t</div>\n",
       "</div>-->\n",
       "<script language=\"javascript\">\n",
       "var disableButton = true;\n",
       "function disableAllButtons() {\n",
       "    if(disableButton) {\n",
       "        var buttons = document.getElementsByTagName(\"input\");\n",
       "        for(var i = 0; i < buttons.length; i++) {\n",
       "            var button = buttons[i];\n",
       "            var nodeType = button.type.toLowerCase();\n",
       "            if(nodeType == \"button\") {\n",
       "                button.disabled=true;\n",
       "            }\n",
       "        }\n",
       "    } else {\n",
       "        disableButton = true;\n",
       "    }\n",
       "}\n",
       "$(function() {\n",
       "    $(\"input[name='btnSearch']\").bind(\"click\", disableAllButtons);\n",
       "});\n",
       "</script>\n",
       "</body></html>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Parse and display HTML of results page (after Search button is clicked)\n",
    "results_html = BeautifulSoup(driver.page_source, 'html.parser')\n",
    "results_html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upon extracting the HTML, we can see that all the records has been loaded, and we are able to actually see the first 10 results embedded in the HTML we just extracted. More importantly, we are able to identify the next link to click in order to access the details of each record i.e. View More Details:\n",
    "\n",
    "```\n",
    "<a href=\"#\" onclick=\"viewMoreDetails('PXXXXXX'); return false;\">View more details</a>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since every page loads 10 records, we need a way to collate the list of 10 IDs of the professionals listed on the page. This is the first step in creating a for loop so that we can loop over each record later on. To do this, we make use of regular expression (regex) since every ID is of the same format i.e. Letter P, followed by 5 digits, and ending with another letter (A-Z)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['P02392B',\n",
       " 'P03376F',\n",
       " 'P03995J',\n",
       " 'P03945D',\n",
       " 'P04296Z',\n",
       " 'P03956Z',\n",
       " 'P01062F',\n",
       " 'P04294C',\n",
       " 'P04307I',\n",
       " 'P03890C']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get list of PRN IDs on current page\n",
    "results_html_text = results_html.get_text() # Convert HTML to text so that we can run regex\n",
    "\n",
    "id_list = re.findall(\"P[0-9]{5}[A-Z]{1}\", results_html_text)\n",
    "id_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with what we have done earlier, the key thing is about locating the elements to click or extract. The following steps are executed based upon the knowledge of what the elements are named within the HTML structure. For example, I know that the name of the professional is within the `table-head` div class, so I run this code to extract the name:\n",
    "\n",
    "```\n",
    "hcp_name =  driver.find_element_by_xpath(\"//div[@class='table-head']\").text\n",
    "```\n",
    "\n",
    "From the individual details page, we can then view the various fields and text we can extract."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'P02392B'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test one record\n",
    "id_single_list = id_list[0]\n",
    "id_single_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Extract fields and HTML of the record\n",
    "for index, hcp_id in enumerate(id_single_list):\n",
    "    \n",
    "    # Click the 'View More Details' link for the specific professional ID\n",
    "    # This is coded as such because every View More Details anchor link is appended with the HCP ID\n",
    "    # i.e. <a href=\"#\" onclick=\"viewMoreDetails('P12345A'); return false;\">View more details</a>\n",
    "    driver.find_element_by_xpath(f\"//a[contains(@onclick,'{hcp_id}')]\").click()\n",
    "    \n",
    "    hcp_html = BeautifulSoup(driver.page_source, 'html.parser')\n",
    "    hcp_name =  driver.find_element_by_xpath(\"//div[@class='table-head']\").text\n",
    "    all_fields = driver.find_elements_by_xpath(\"//td[@class='no-border table-data']\") # Using find element since there are multiple elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<html lang=\"en\" style=\"height:100%\" xml:lang=\"en\" xmlns=\"http://www.w3.org/1999/xhtml\"><head>\n",
       "<title>Professionals Search</title>\n",
       "<meta content=\"IE=9\" http-equiv=\"X-UA-Compatible\"/>\n",
       "<meta content=\"text/html; charset=utf-8\" http-equiv=\"Content-Type\"/>\n",
       "<meta content=\"no-cache\" http-equiv=\"Cache-Control\"/>\n",
       "<meta content=\"no-cache\" http-equiv=\"Pragma\"/>\n",
       "<meta content=\"0\" http-equiv=\"Expires\"/>\n",
       "<meta content=\"noindex, nofollow\" name=\"robots\"/>\n",
       "<script async=\"\" src=\"https://assets.wogaa.sg/snowplow/2.14.0/sp.js\"></script><script src=\"https://assets.wogaa.sg/scripts/wogaa.js\"></script><script async=\"\" src=\"https://assets.wogaa.sg/scripts/wogaa.js?url=https%3A%2F%2Fprs.moh.gov.sg%2Fprs%2Finternet%2FprofSearch%2FgetSearchDetails.action\"></script>\n",
       "<link href=\"/prs/css/internet/spc/header.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/public-header.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<!--[if lt IE 8]><link rel=\"stylesheet\" type=\"text/css\" href=\"/prs/css/internet/ie.css\" media=\"screen\"/><![endif]-->\n",
       "<link href=\"/prs/css/internet/footer.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/public.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/spc/main.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/public-main.css?version=20151125\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<link href=\"/prs/css/internet/psearch.css\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\"/>\n",
       "<script src=\"/prs/scripts/jquery.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/jquery.watermark.min.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/jquery.inputlimiter.js?version=20141111\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/iextend.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/common.js?version=2\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/uiTimer.js\" type=\"text/javascript\"></script>\n",
       "<script src=\"/prs/scripts/mdetect.js\" type=\"text/javascript\"></script>\n",
       "<style type=\"text/css\">\n",
       "    @media print {\n",
       "        .notPrint {\n",
       "            display:none;\n",
       "        }\n",
       "        .path {\n",
       "            display:none;\n",
       "        }\n",
       "    }\n",
       "    </style>\n",
       "<script type=\"text/javascript\">\n",
       "\t$(function () {\n",
       "\t    $(\"#searchsite\").watermark(\"Enter search keywords here\", \"watermark\");\n",
       "\t});\n",
       "\t</script>\n",
       "<script async=\"\" src=\"//assets.adobedtm.com/launch-ENaf340d988e354d18ba897b99e3538f23.min.js\"></script><script async=\"\" src=\"https://assets.adobedtm.com/extensions/EPbde2f7ca14e540399dcc1f8208860b7b/AppMeasurement.min.js\"></script><script async=\"\" src=\"https://assets.adobedtm.com/extensions/EPbde2f7ca14e540399dcc1f8208860b7b/AppMeasurement_Module_ActivityMap.min.js\"></script><script async=\"\" src=\"https://assets.wogaa.sg/datalayer.min.js\"></script></head>\n",
       "<body class=\"page\" style=\"height:100%; background-attachment:fixed;\">\n",
       "<!--<div id=\"header-wrapper\">\n",
       "\t<div id=\"header\">\n",
       "\t\t<div class=\"header-bar\">\n",
       "\t\t\t<div class=\"logo-area\">\n",
       "\t\t\t\t<h1 class=\"logo\"><a href=\"#\">Singapore Pharmacy Council</a></h1>\n",
       "\t\t\t\t<strong class=\"slogan\">Singapore Pharmacy Council</strong>\n",
       "\t\t\t</div>\n",
       "\t\t\t<div id=\"access\" class=\"visuallyhidden\">\n",
       "\t\t\t\t<a href=\"#search-1\" title=\"Search\">Search</a> | <a href=\"#content\" title=\"Skip to content\">Skip to content</a> | <a href=\"#footer\" title=\"Skip to footer site map\">Skip to footer site map</a>\n",
       "\t\t\t</div>\n",
       "\t\t\t<div class=\"header-section\">\n",
       "\t\t\t\t<div class=\"ad-box\">\n",
       "\t\t\t\t\t<a href=\"#\"><img src=\"/prs/images/ad-image1.gif\" alt=\"Singapore Government\" width=\"155\" height=\"27\" /></a>\n",
       "\t\t\t\t</div>\n",
       "\t                  <ul class=\"top-menu\">\n",
       "\t\t\t\t\t<li><a href=\"#\">Contact</a></li>\n",
       "\t\t\t\t\t<li><a href=\"#\">Feedback</a></li>\n",
       "\t\t\t\t\t<li><a href=\"#\">Sitemap</a></li>\n",
       "\t\t\t\t</ul>\n",
       "\t\t\t\t\n",
       "\t\t\t</div>\n",
       "\t\t</div>\n",
       "\t</div>\n",
       "\t<br/><br/>\n",
       "</div>-->\n",
       "<table style=\"width:785px;\">\n",
       "<tbody><tr>\n",
       "<td>\n",
       "<div class=\"main-holder\">\n",
       "<div id=\"content\" style=\"margin:0px;width:100%;\">\n",
       "<div class=\"content-entry\">\n",
       "<div class=\"entry-holder\">\n",
       "<input id=\"transIdAck\" type=\"hidden\" value=\"a5ae777e5dad4a699bf0ae7495c6ddd7\"/>\n",
       "<input id=\"appId\" name=\"appId\" type=\"hidden\" value=\"26088ee5287fc3de\"/>\n",
       "<script type=\"text/javascript\">\n",
       "\tfunction openGoogleMap(value){\n",
       "\t\tvar newWindow = window.open(value);\n",
       "\t}\n",
       "\t\n",
       "\tfunction openOneMap(value){\n",
       "\t\tvar newWindow = window.open(value);\n",
       "\t}\n",
       "\t\n",
       "\tfunction DetectDevice(){\n",
       "\t    if (checkMobile()){\n",
       "\t\t\twindow.location = \"msearchDetailsSPC.action?regNo=P02392Bhpe=SPC\";\n",
       "\t\t}\n",
       "\t}\n",
       "\t\n",
       "\tfunction backToSearchResults(){\n",
       "\t\tvar searchBy = $(\"#searchBy\").val();\n",
       "\t\t\n",
       "\t\tif(searchBy == \"N\"){\n",
       "\t\t\t$(\"#searchDetails\").attr(\"action\",\"backToSearchByName.action\");\n",
       "\t\t} else if (searchBy == \"L\"){\n",
       "\t\t\t$(\"#searchDetails\").attr(\"action\",\"backToSearchByLocation.action\");\n",
       "\t\t} else if (searchBy == \"R\"){\n",
       "\t\t\t$(\"#searchDetails\").attr(\"action\",\"backToSearchByRegion.action\");\n",
       "\t\t}\n",
       "\t\t\n",
       "\t\t$(\"#searchDetails\").submit();\n",
       "\t}\n",
       "\t\n",
       "\tDetectDevice();\n",
       "</script>\n",
       "<form action=\"/prs/internet/profSearch/getSearchDetails.action\" autocomplete=\"off\" id=\"searchDetails\" method=\"post\" name=\"searchDetails\">\n",
       "<input id=\"hpe\" name=\"hpe\" type=\"hidden\" value=\"SPC\"/>\n",
       "<div id=\"searchParams\" style=\"display: none;\">\n",
       "\tName: <input id=\"searchDetails_psearchParamVO_name\" name=\"psearchParamVO.name\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "<br/>\n",
       "\tStart With: <input id=\"searchDetails_psearchParamVO_startWith\" name=\"psearchParamVO.startWith\" type=\"text\" value=\"false\"/>\n",
       "<!-- Add fieldError -->\n",
       "<br/>\n",
       "\tPrac Place Name: <input id=\"searchDetails_psearchParamVO_pracPlaceName\" name=\"psearchParamVO.pracPlaceName\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "<br/><br/>\n",
       "\t\n",
       "\tPostal Code: <input id=\"searchDetails_psearchParamVO_postalCode\" name=\"psearchParamVO.postalCode\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "<br/>\n",
       "\tRange: <input id=\"searchDetails_psearchParamVO_range\" name=\"psearchParamVO.range\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "<br/><br/>\n",
       "<input id=\"cb1\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"1\"/><input id=\"__checkbox_cb1\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"1\"/>\n",
       "\t\t\n",
       "\t\tTelok Blangah\n",
       "\t\t<br/>\n",
       "<input id=\"cb2\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"2\"/><input id=\"__checkbox_cb2\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"2\"/>\n",
       "\t\t\n",
       "\t\tCommonwealth\n",
       "\t\t<br/>\n",
       "<input id=\"cb3\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"3\"/><input id=\"__checkbox_cb3\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"3\"/>\n",
       "\t\t\n",
       "\t\tBukit Merah\n",
       "\t\t<br/>\n",
       "<input id=\"cb4\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"4\"/><input id=\"__checkbox_cb4\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"4\"/>\n",
       "\t\t\n",
       "\t\tClementi\n",
       "\t\t<br/>\n",
       "<input id=\"cb5\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"5\"/><input id=\"__checkbox_cb5\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"5\"/>\n",
       "\t\t\n",
       "\t\tCentral\n",
       "\t\t<br/>\n",
       "<input id=\"cb6\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"6\"/><input id=\"__checkbox_cb6\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"6\"/>\n",
       "\t\t\n",
       "\t\tSerangoon\n",
       "\t\t<br/>\n",
       "<input id=\"cb7\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"7\"/><input id=\"__checkbox_cb7\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"7\"/>\n",
       "\t\t\n",
       "\t\tOrchard\n",
       "\t\t<br/>\n",
       "<input id=\"cb8\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"8\"/><input id=\"__checkbox_cb8\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"8\"/>\n",
       "\t\t\n",
       "\t\tHolland\n",
       "\t\t<br/>\n",
       "<input id=\"cb9\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"9\"/><input id=\"__checkbox_cb9\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"9\"/>\n",
       "\t\t\n",
       "\t\tToa Payoh\n",
       "\t\t<br/>\n",
       "<input id=\"cb10\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"10\"/><input id=\"__checkbox_cb10\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"10\"/>\n",
       "\t\t\n",
       "\t\tNovena\n",
       "\t\t<br/>\n",
       "<input id=\"cb11\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"11\"/><input id=\"__checkbox_cb11\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"11\"/>\n",
       "\t\t\n",
       "\t\tKallang\n",
       "\t\t<br/>\n",
       "<input id=\"cb12\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"12\"/><input id=\"__checkbox_cb12\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"12\"/>\n",
       "\t\t\n",
       "\t\tGeylang\n",
       "\t\t<br/>\n",
       "<input id=\"cb13\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"13\"/><input id=\"__checkbox_cb13\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"13\"/>\n",
       "\t\t\n",
       "\t\tMarine Parade\n",
       "\t\t<br/>\n",
       "<input id=\"cb14\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"14\"/><input id=\"__checkbox_cb14\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"14\"/>\n",
       "\t\t\n",
       "\t\tBedok\n",
       "\t\t<br/>\n",
       "<input id=\"cb15\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"15\"/><input id=\"__checkbox_cb15\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"15\"/>\n",
       "\t\t\n",
       "\t\tBedok Reservoir\n",
       "\t\t<br/>\n",
       "<input id=\"cb16\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"16\"/><input id=\"__checkbox_cb16\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"16\"/>\n",
       "\t\t\n",
       "\t\tChangi\n",
       "\t\t<br/>\n",
       "<input id=\"cb17\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"17\"/><input id=\"__checkbox_cb17\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"17\"/>\n",
       "\t\t\n",
       "\t\tTampines\n",
       "\t\t<br/>\n",
       "<input id=\"cb18\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"18\"/><input id=\"__checkbox_cb18\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"18\"/>\n",
       "\t\t\n",
       "\t\tPasir Ris\n",
       "\t\t<br/>\n",
       "<input id=\"cb19\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"19\"/><input id=\"__checkbox_cb19\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"19\"/>\n",
       "\t\t\n",
       "\t\tHougang\n",
       "\t\t<br/>\n",
       "<input id=\"cb20\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"20\"/><input id=\"__checkbox_cb20\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"20\"/>\n",
       "\t\t\n",
       "\t\tSengkang/Punggol\n",
       "\t\t<br/>\n",
       "<input id=\"cb21\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"21\"/><input id=\"__checkbox_cb21\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"21\"/>\n",
       "\t\t\n",
       "\t\tAng Mo Kio\n",
       "\t\t<br/>\n",
       "<input id=\"cb22\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"22\"/><input id=\"__checkbox_cb22\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"22\"/>\n",
       "\t\t\n",
       "\t\tBishan\n",
       "\t\t<br/>\n",
       "<input id=\"cb23\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"23\"/><input id=\"__checkbox_cb23\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"23\"/>\n",
       "\t\t\n",
       "\t\tBukit Timah\n",
       "\t\t<br/>\n",
       "<input id=\"cb24\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"24\"/><input id=\"__checkbox_cb24\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"24\"/>\n",
       "\t\t\n",
       "\t\tJurong West\n",
       "\t\t<br/>\n",
       "<input id=\"cb25\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"25\"/><input id=\"__checkbox_cb25\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"25\"/>\n",
       "\t\t\n",
       "\t\tJurong East\n",
       "\t\t<br/>\n",
       "<input id=\"cb26\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"26\"/><input id=\"__checkbox_cb26\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"26\"/>\n",
       "\t\t\n",
       "\t\tChoa Chu Kang\n",
       "\t\t<br/>\n",
       "<input id=\"cb27\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"27\"/><input id=\"__checkbox_cb27\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"27\"/>\n",
       "\t\t\n",
       "\t\tBukit Batok\n",
       "\t\t<br/>\n",
       "<input id=\"cb28\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"28\"/><input id=\"__checkbox_cb28\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"28\"/>\n",
       "\t\t\n",
       "\t\tBukit Panjang\n",
       "\t\t<br/>\n",
       "<input id=\"cb29\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"29\"/><input id=\"__checkbox_cb29\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"29\"/>\n",
       "\t\t\n",
       "\t\tWoodlands\n",
       "\t\t<br/>\n",
       "<input id=\"cb30\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"30\"/><input id=\"__checkbox_cb30\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"30\"/>\n",
       "\t\t\n",
       "\t\tYishun\n",
       "\t\t<br/>\n",
       "<input id=\"cb31\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"31\"/><input id=\"__checkbox_cb31\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"31\"/>\n",
       "\t\t\n",
       "\t\tSembawang\n",
       "\t\t<br/>\n",
       "<input id=\"cb32\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"32\"/><input id=\"__checkbox_cb32\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"32\"/>\n",
       "\t\t\n",
       "\t\tSeletar\n",
       "\t\t<br/>\n",
       "<input id=\"cb33\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"33\"/><input id=\"__checkbox_cb33\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"33\"/>\n",
       "\t\t\n",
       "\t\tLim Chu Kang\n",
       "\t\t<br/>\n",
       "<input id=\"cb34\" name=\"psearchParamVO.region\" type=\"checkbox\" value=\"34\"/><input id=\"__checkbox_cb34\" name=\"__checkbox_psearchParamVO.region\" type=\"hidden\" value=\"34\"/>\n",
       "\t\t\n",
       "\t\tBoon Lay\n",
       "\t\t<br/>\n",
       "<br/>\n",
       "\t\n",
       "\tRbtn Register <input id=\"searchDetails_psearchParamVO_rbtnRegister\" name=\"psearchParamVO.rbtnRegister\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "<br/>\n",
       "\tRegNo <input id=\"searchDetails_psearchParamVO_regNo\" name=\"psearchParamVO.regNo\" type=\"text\" value=\"\"/>\n",
       "<!-- Add fieldError -->\n",
       "<br/>\n",
       "\tLanguage <input id=\"searchDetails_psearchParamVO_language\" name=\"psearchParamVO.language\" type=\"text\" value=\"eng\"/>\n",
       "<!-- Add fieldError -->\n",
       "<br/>\n",
       "\tSearch By <input id=\"searchBy\" name=\"psearchParamVO.searchBy\" type=\"text\" value=\"N\"/>\n",
       "<!-- Add fieldError -->\n",
       "<br/><br/>\n",
       "</div>\n",
       "<div class=\"article\" id=\"profDetails\" style=\"min-height:100%;\">\n",
       "<div class=\"table-head\">\n",
       "\t\tAARON CHEW SONG TA (P02392B)\n",
       "\t</div>\n",
       "<hr class=\"horizontal-line\"/>\n",
       "<table width=\"100%\">\n",
       "<tbody><tr>\n",
       "<td class=\"no-border table-title\" width=\"20%\">Registration Number</td>\n",
       "<td class=\"no-border table-data\" width=\"30%\">P02392B</td>\n",
       "<td class=\"no-border table-title\" width=\"20%\"></td>\n",
       "<td class=\"no-border table-data\" width=\"30%\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border table-title\">Registration Date</td>\n",
       "<td class=\"no-border table-data\">\n",
       "\t\t\t\t26/08/2014\n",
       "\t\t\t</td>\n",
       "<td class=\"no-border table-title\">Registration End Date</td>\n",
       "<td class=\"no-border table-data\">\n",
       "</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border table-title\">Registration Type</td>\n",
       "<td class=\"no-border table-data\">Full Registration</td>\n",
       "<td class=\"no-border table-title\">Practice Status</td>\n",
       "<td class=\"no-border table-data\">\n",
       "\t\t\t\t\n",
       "\t\t\t\tActive\n",
       "\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border table-title\">Practising Certificate Start Date</td>\n",
       "<td class=\"no-border table-data\">\n",
       "\t\t\t\t01/01/2019\n",
       "\t\t\t</td>\n",
       "<td class=\"no-border table-title\">Practising Certificate End Date</td>\n",
       "<td class=\"no-border table-data\">\n",
       "\t\t\t\t31/12/2020\n",
       "\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border table-title\">Qualifications</td>\n",
       "<td class=\"no-border table-data\" colspan=\"3\">\n",
       "\t\t\t\t\n",
       "\t\t\t\t\tB Pharm (Hons),\n",
       "\t\t\t\t\tUniversiti Sains Malaysia,\n",
       "\t\t\t\t\t\n",
       "\t              \tMalaysia,\n",
       "\t              \t2010\n",
       "\t\t\t\t\t<br/>\n",
       "</td>\n",
       "</tr>\n",
       "</tbody></table><br/>\n",
       "<div class=\"table-title align-left\">\n",
       "\t\t\n",
       "\t\t\n",
       "\t\t\tType of Register: Pharmacist\n",
       "\t\t\n",
       "\t</div>\n",
       "<hr class=\"horizontal-line\"/>\n",
       "<div>\n",
       "</div><br/>\n",
       "<div class=\"table-title align-left\">Primary Place of Practice</div>\n",
       "<hr class=\"horizontal-line\"/>\n",
       "<div>\n",
       "<table width=\"100%\">\n",
       "<tbody><tr>\n",
       "<td class=\"no-border table-title\" width=\"20%\">Name of Place of Practice</td>\n",
       "<td class=\"no-border table-data\" width=\"*\">\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\tGuardian Health &amp; Beauty\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border table-title\">Address of Place of Practice</td>\n",
       "<td class=\"no-border table-data\">\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\t21\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tTAMPINES NORTH DRIVE 2          <br/>\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\t#03 -\n",
       "\t\t\t\t\t\t\t\t01\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\t\tSingapore 528765\n",
       "\t\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border table-title\">Tel</td>\n",
       "<td class=\"no-border table-data\">\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t\t\t\n",
       "\t\t\t\t\t\t68918000\n",
       "\t\t\t\t\t\n",
       "\t\t\t\t</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td class=\"no-border table-title\">Map</td>\n",
       "<td class=\"no-border table-data\">\n",
       "<a class=\"no-decoration red-color underline\" href=\"#\" onclick=\"openGoogleMap('http://maps.google.com/maps?q=Singapore+528765&amp;output=embed'); return false;\">\n",
       "<img height=\"12\" src=\"/prs/images/ico-map.gif\" width=\"12\"/> Google Map\n",
       "\t\t\t\t\t\t\t</a>\n",
       "<a class=\"no-decoration red-color underline\" href=\"#\" onclick=\"openOneMap('http://www.onemap.sg/minimap/mm.html?mWidth=450&amp;mHeight=450&amp;searchval=528765'); return false;\">\n",
       "<img height=\"12\" src=\"/prs/images/ico-map.gif\" width=\"12\"/> One Map\n",
       "\t\t\t\t\t\t\t</a>\n",
       "</td>\n",
       "</tr>\n",
       "</tbody></table>\n",
       "</div><br/>\n",
       "<div>\n",
       "<a class=\"red-color no-decoration underline\" href=\"#\">Back to Top</a> |\n",
       "\t\t<a class=\"red-color no-decoration underline\" href=\"#\" onclick=\"backToSearchResults(); return false;\">Back to Search Results</a> |\n",
       "\t\t<a class=\"red-color no-decoration underline\" href=\"/prs/internet/profSearch/showSearchSummaryByName.action?hpe=SPC\">Reset Search</a> |\n",
       "        <a class=\"red-color no-decoration underline\" href=\"javascript:addUsageRatings(document.getElementById('transIdAck').value,document.getElementById('appId').value);\">Rate this service</a>\n",
       "</div>\n",
       "</div>\n",
       "</form>\n",
       "</div>\n",
       "</div>\n",
       "</div>\n",
       "</div>\n",
       "</td>\n",
       "</tr>\n",
       "</tbody></table>\n",
       "<!--<div id=\"footer\" class=\"footer-holder\">\n",
       "\t<div class=\"footer-bar\">\n",
       "\t\t<div id=\"privacy\">\n",
       "\t\t\t<ul class=\"footer-menu\">\n",
       "\t\t\t\t<li><a href=\"#\">Privacy Statement</a></li>\n",
       "\t\t\t\t<li><a href=\"#\">Terms of Use</a></li>\n",
       "\t\t\t</ul>\n",
       "\t\t\t<span class=\"copyright\">&copy; Singapore Pharmacy Council</span>\n",
       "\t\t</div>\n",
       "\t</div>\n",
       "</div>-->\n",
       "<script language=\"javascript\">\n",
       "var disableButton = true;\n",
       "function disableAllButtons() {\n",
       "    if(disableButton) {\n",
       "        var buttons = document.getElementsByTagName(\"input\");\n",
       "        for(var i = 0; i < buttons.length; i++) {\n",
       "            var button = buttons[i];\n",
       "            var nodeType = button.type.toLowerCase();\n",
       "            if(nodeType == \"button\") {\n",
       "                button.disabled=true;\n",
       "            }\n",
       "        }\n",
       "    } else {\n",
       "        disableButton = true;\n",
       "    }\n",
       "}\n",
       "$(function() {\n",
       "    $(\"input[name='btnSearch']\").bind(\"click\", disableAllButtons);\n",
       "});\n",
       "</script>\n",
       "</body></html>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hcp_html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have successfully gone all the way in to expose the HTML codes of the public details of a single healthcare professional. Now that the initial exploration is complete, let's get to the real action of automated web scraping!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"section3\"></a>\n",
    "## Section 3 - Running Main Script\n",
    "\n",
    "With the same dependencies installed and imported earlier, we can proceed to build a script to automate the tedious web scrapping process for us. I previously mentioned that there are various healthcare professional bodies on the MOH webpage, and they are actually represented by different abbreviations. For example, SDC for Singapore Dental Council, SMC for Singapore Medical Council etc. For the Pharmacists group I am looking at, the abbreviation will be SPC.\n",
    "\n",
    "We prepare our WebDriver once again to access the target HTML frame, along with several WebDriver options to make sure our WebDriver runs smoothly. In particular, we are introducing wait times and sleep times so that there can be appropriate pauses during the scraping process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (i) WebDriver Initiation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define healthcare professional (HCP) body \n",
    "hcp_body = 'SPC'\n",
    "\n",
    "# Set wait times\n",
    "waittime = 20\n",
    "sleeptime = 2\n",
    "\n",
    "# Initiate web driver\n",
    "try:\n",
    "    driver.close() # Close any existing WebDrivers\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "# Access professional registration system (PRS) for specific healthcare professional body\n",
    "home_page = f\"https://prs.moh.gov.sg/prs/internet/profSearch/main.action?hpe={hcp_body}\"\n",
    "\n",
    "# Set webdriver options\n",
    "options = webdriver.ChromeOptions()\n",
    "options.add_argument('--no-sandbox')\n",
    "options.add_argument('ignore-certificate-errors')\n",
    "\n",
    "# Initiate webdriver\n",
    "driver = webdriver.Chrome(options=options) \n",
    "\n",
    "# Get driver to retrieve homepage\n",
    "driver.get(home_page)\n",
    "\n",
    "# Switch to frame which contains HTML for Search section\n",
    "driver.switch_to.frame(driver.find_element_by_name('msg_main'))\n",
    "\n",
    "# Click Search button (named btnSearch) to load all results\n",
    "WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, \"//input[@name='btnSearch']\"))).click()\n",
    "\n",
    "# Sleep a short while for page loading to be fully completed\n",
    "time.sleep(sleeptime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (ii) Setting up key functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because there is a series of actions to take in order to navigate through the different records across different Search results pages, it is best to setup functions to better organize all these steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (1) Setup master list CSV to store all the records\n",
    "We first generate a CSV file to store the records which we will be appending later. The column names are based on the earlier exploration of the web page, where we identified the various fields we can extract from each public record."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filename master_list_test.csv already exists\n"
     ]
    }
   ],
   "source": [
    "# Set file name (based on your own preference)\n",
    "file_name = 'master_list.csv'\n",
    "\n",
    "# Check if file already exists\n",
    "if os.path.isfile(f'./{file_name}'):\n",
    "    print(f'Filename {file_name} already exists')\n",
    "else:\n",
    "    # Set names of fields we want to extract\n",
    "    column_names = ['name','reg_number','reg_date','reg_end_date','reg_type','practice_status','cert_start_date',\n",
    "                    'cert_end_date','qualification','practice_place_name','practice_place_address','practice_place_phone']\n",
    "    \n",
    "    # Generate template dataframe\n",
    "    df_template = pd.DataFrame(columns = column_names)\n",
    "    \n",
    "    # Generate csv file from the dataframe template\n",
    "    df_template.to_csv(f'{file_name}', header=True)\n",
    "    \n",
    "    print('Created new master list file')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (2) Get current page number\n",
    "We need a function to track which page of the Search results the WebDriver-controlled browser is currently on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_current_page():\n",
    "    \"\"\"\n",
    "    Get current Search results page number the WebDriver is on\n",
    "    \"\"\"\n",
    "    # Wait until element is located, then click it\n",
    "    current_page_elem = WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, \"//label[@class='pagination_selected_page']\"))).text\n",
    "    \n",
    "    # Get page number as integer\n",
    "    current_page_num = int(current_page_elem)\n",
    "    \n",
    "    return current_page_num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (3) Get absolute last page number\n",
    "It is important to know how many Search results pages we have in total, so that we can create a page loop accordingly. For example, at the time of this exercise, the last page number is 340 (i.e. there are 340 Search results pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_absolute_last_page():\n",
    "    \"\"\"\n",
    "    Get last page number of all Search results records\n",
    "    \"\"\"\n",
    "    # Find all elements with pagination class (since it contains page numbers)\n",
    "    WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, \"//a[@class='pagination']\")))\n",
    "    all_pages = driver.find_elements_by_xpath(\"//a[@class='pagination']\")\n",
    "\n",
    "    # Get final element, which corresponds to 'Last' hyperlink (which will go to last page number)\n",
    "    last_elem = all_pages[-1].get_attribute('href')\n",
    "\n",
    "    # Keep only the number of last page\n",
    "    last_page_num = int(re.sub(\"[^0-9]\", \"\", last_elem))\n",
    "    \n",
    "    return last_page_num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (4) Extract data from detailed information page of a single record\n",
    "While we are on the details page of a particular healthcare professional (after clicking the 'View More Details' link), we need a way to extract all the text from the respective fields available for public viewing. The information for each record is consolidated into a Python dictionary.\n",
    "\n",
    "Do note that we are longer using BeautifulSoup here (since BS was used just for demo purposes earlier). We are now solely depending on Selenium to do the parsing and text extraction for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gen_hcp_dict():\n",
    "    \"\"\"\n",
    "    Extract text from details fields and consolidate into dictionary\n",
    "    \"\"\"\n",
    "    WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, \"//div[@class='table-head']\")))\n",
    "    hcp_name =  driver.find_element_by_xpath(\"//div[@class='table-head']\").text    \n",
    "    all_fields = driver.find_elements_by_xpath(\"//td[@class='no-border table-data']\") # Using find elementS since there are multiple elements      \n",
    "    hcp_data = []\n",
    "    \n",
    "    # Extract test for every field within the table\n",
    "    for field in all_fields:\n",
    "        hcp_data.append(field.text)\n",
    "\n",
    "    hcp_dict = {}\n",
    "    \n",
    "    # Assign extracted data into respective key-value pairs in dictionary\n",
    "    hcp_dict['name'] = hcp_name\n",
    "    hcp_dict['reg_number'] = hcp_data[0]\n",
    "    # hcp_data[1] is just a blank space, so it can be ignored\n",
    "    hcp_dict['reg_date'] = hcp_data[2]\n",
    "    hcp_dict['reg_end_date'] = hcp_data[3]\n",
    "    hcp_dict['reg_type'] = hcp_data[4]\n",
    "    hcp_dict['practice_status'] = hcp_data[5]\n",
    "    hcp_dict['cert_start_date'] = hcp_data[6]\n",
    "    hcp_dict['cert_end_date'] = hcp_data[7]\n",
    "    hcp_dict['qualification'] = hcp_data[8]\n",
    "    hcp_dict['practice_place_name'] = hcp_data[9]\n",
    "    hcp_dict['practice_place_address'] = hcp_data[10]\n",
    "    hcp_dict['practice_place_phone'] = hcp_data[11]\n",
    "    \n",
    "    return hcp_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (5) Get current pagination range\n",
    "There is a pagination range that is available on each page of the Search results, and it is usually limited to around 10 digits. This means that you are only able to directly access a particular page from that specific range visible to you. Thus, we need a function to determine what is the current pagination range that is accessible. Below is the screenshot of the Pagination range I am referring to:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/pagination_range.png\" width=\"60%\" align = \"left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_current_pagination_range():\n",
    "    \"\"\"\n",
    "    Get range of pages visible in pagination range on current page\n",
    "    \"\"\"\n",
    "    WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, \"//a[@class='pagination']\")))\n",
    "    \n",
    "    # Get all elements related to pagination\n",
    "    all_pages = driver.find_elements_by_xpath(\"//a[@class='pagination']\")\n",
    "    \n",
    "    # Implicit wait to allow page to load\n",
    "    driver.implicitly_wait(1)\n",
    "    \n",
    "    # Find numbers of pagination range, and append to list\n",
    "    pagination_range_on_page = []\n",
    "    for elem in all_pages:\n",
    "        \n",
    "        # Only extracting numeric values of pagination range\n",
    "        if elem.text.isnumeric():\n",
    "            pagination_range_on_page.append(int(elem.text))\n",
    "            driver.implicitly_wait(1)\n",
    "        else:\n",
    "            pass\n",
    "    driver.implicitly_wait(1)\n",
    "    return pagination_range_on_page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (6) Click last pagination number on current page\n",
    "With the pagination range known, we need a function to click on the last number displayed in this range. This is for us to proceed further into the pages beyond the given range. For example, to reveal pages beyond page 10, we need to click the number 10 from the range (Page 1 2 3 4 5 6 7 8 9 10 ..)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def click_last_pagination_num(pagination_range):\n",
    "    \"\"\"\n",
    "    Click last page number on current visible pagination range\n",
    "    \"\"\"\n",
    "    last_pagination_num = pagination_range[-1] \n",
    "    \n",
    "    # Introduce implicit wait for page to complete loading\n",
    "    driver.implicitly_wait(1)\n",
    "    \n",
    "    # Click page number once element is located\n",
    "    WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, f'{last_pagination_num}'))).click() \n",
    "    driver.implicitly_wait(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (7) Click first pagination number on current page\n",
    "Since there are hundreds of pages, it does not make sense to always start from page 1. This is because it will require an extremely tedious amount of effort to click on the last visible page number of each pagination range to reach the later half of the Search results. \n",
    "\n",
    "The solution to this is that for page numbers greater than the halfway mark (e.g. pages beyond 100 if there is a total of 200 pages), our program will instead start from the last page, and go back in reverse to reach those latter pages. This strategy of starting from the last page to reach these latter pages is much more efficient compared to always starting from the first page.\n",
    "\n",
    "To make that happen, we need a function to click the first number of the existing pagination range. For example, if our current pagination range is (Page .. 151 152 153 154 155 156 157 158 ..), and we want to go to page 147, the program will be clicking the page number 151 so as to reveal the preceding set of pagination range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def click_first_pagination_num(pagination_range):\n",
    "    \"\"\"\n",
    "    Click first page number on current visible pagination range\n",
    "    \"\"\"\n",
    "    first_pagination_num = pagination_range[0] \n",
    "    \n",
    "    # Introduce implicit wait for page to complete loading\n",
    "    driver.implicitly_wait(1)\n",
    "    \n",
    "    # Click page number once element is located\n",
    "    WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, f'{first_pagination_num}'))).click() \n",
    "    driver.implicitly_wait(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (8) Locate and access target page\n",
    "Let's put the earlier functions into action. We now need to find a way to get our WebDriver to go to each page to scrape all the relevant information for each of the 10 professionals listed.\n",
    "\n",
    "A massive pain comes from the fact that when you click the *'Back to Search Results'* link while on the details page of a particular professional, it brings you all the way back to Page 1 of the Search results instead of the current page. This means that if you were looking at the details page of Mr Chan on Page 10 and you click *'Back to Search Results'*, you will be brought back to Page 1 INSTEAD of the page 10 you were on.\n",
    "\n",
    "As such, we need a function that can repeatedly click the appropriate numbers of the pagination range to return to the target page we were scraping EVERY TIME after the *'Back to Search Results'* link is clicked. The following code sets up the automation to do just that, with the strategic use of conditional statements and loops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def locate_target_page(target_page):\n",
    "    \"\"\"\n",
    "    Navigate to target page to proceed with scraping\n",
    "    \"\"\"\n",
    "    last_page_num = get_absolute_last_page()\n",
    "    \n",
    "    # Find midway point of all search results page\n",
    "    midway_point = last_page_num/2\n",
    "\n",
    "    # If target page is in first half, start clicking from the start\n",
    "    if target_page < midway_point: \n",
    "        current_page_num = get_current_page()\n",
    "    \n",
    "        if current_page_num == target_page:\n",
    "            pass\n",
    "        else:            \n",
    "            pagination_range = get_current_pagination_range()\n",
    "\n",
    "            while target_page not in pagination_range:\n",
    "                driver.implicitly_wait(1)\n",
    "                \n",
    "                # If target page is not in pagination range, keep clicking last pagination number to go further down the list\n",
    "                click_last_pagination_num(pagination_range) \n",
    "                current_page_num = get_current_page()\n",
    "                pagination_range = get_current_pagination_range()\n",
    "                driver.implicitly_wait(1)\n",
    "            else:\n",
    "                WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, f\"{target_page}\"))).click() # Once target page is in pagination page, go to the target page\n",
    "\n",
    "    else: # If target page is in later half of list, go to Last page and move in reverse (This saves alot of time)\n",
    "        WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, 'Last'))).click()  # Go to last page\n",
    "        time.sleep(sleeptime)\n",
    "        current_page_num = get_current_page()\n",
    "\n",
    "        if current_page_num == target_page:\n",
    "            pass\n",
    "        else:           \n",
    "            pagination_range = get_current_pagination_range()\n",
    "\n",
    "            while target_page not in pagination_range:\n",
    "                driver.implicitly_wait(2)\n",
    "                click_first_pagination_num(pagination_range)\n",
    "                current_page_num = get_current_page()\n",
    "                pagination_range = get_current_pagination_range()\n",
    "            else:\n",
    "                WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.LINK_TEXT, f\"{target_page}\"))).click() # Once target page is in pagination page, go to the target page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (9) Create script that automates the navigation through the website portal\n",
    "And now we setup the last function, which will bring everything together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def full_scrape(target_page):\n",
    "    \"\"\"\n",
    "    Perform scraping for all pages in loaded search results\n",
    "    \"\"\"\n",
    "    last_page_num = get_absolute_last_page()\n",
    "    driver.implicitly_wait(1)\n",
    "    \n",
    "    while target_page != last_page_num:\n",
    "        locate_target_page(target_page)\n",
    "        print('Starting with target page ' + str(target_page))\n",
    "        \n",
    "        # Retrieve HTML from search page\n",
    "        target_page_html = driver.find_element_by_xpath(\"//body\").get_attribute('outerHTML')\n",
    "        driver.implicitly_wait(1)\n",
    "        \n",
    "        # Find list of all IDs on that page, and keep the unique IDs\n",
    "        all_ids = re.findall(\"P[0-9]{5}[A-Z]{1}\", target_page_html)\n",
    "        id_list = list(dict.fromkeys(all_ids))\n",
    "\n",
    "        for index, hcp_id in enumerate(id_list): # Tracking the healthcare professional (HCP)'s ID\n",
    "            # Click 'View More Details' link to access details page for that professional with the specific ID\n",
    "            WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, f\"//a[contains(@onclick,'{hcp_id}')]\"))).click()\n",
    "            \n",
    "            # Scrape relevant data from details page into a dictionary\n",
    "            hcp_dict = gen_hcp_dict()\n",
    "            \n",
    "            # Convert dict to pandas dataframe (Need to set an index since we are passing scalar values)\n",
    "            df_hcp_dict = pd.DataFrame(hcp_dict, index=[0])\n",
    "            \n",
    "            # Append df row directly to existing master list csv\n",
    "            df_hcp_dict.to_csv(f'{file_name}', mode='a', header=False)\n",
    "            \n",
    "            # Print row that was just scraped (To track progress)\n",
    "            print(f'Scraped row {index+1} of page {target_page}')\n",
    "                  \n",
    "            # Update (+1) target page after successfully scrapping all 10 records on page\n",
    "            if index == len(id_list):\n",
    "                print(f'Completed scraping for page {target_page}')\n",
    "                target_page += 1\n",
    "                print('Updated target page ' + str(target_page))\n",
    "            else:\n",
    "                pass\n",
    "\n",
    "            # Head back to homepage by clicking 'Back to Search Results' link\n",
    "            WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, 'Back to Search Results'))).click() \n",
    "            \n",
    "            # Go to latest updated target page\n",
    "            locate_target_page(target_page)\n",
    "    \n",
    "    # Define steps when program reaches last page (essentially similar to the steps above)\n",
    "    else:\n",
    "        locate_target_page(target_page)\n",
    "        print('Working on last page')\n",
    "        target_page_html = driver.find_element_by_xpath(\"//body\").get_attribute('outerHTML')\n",
    "        driver.implicitly_wait(1)\n",
    "        all_ids = re.findall(\"P[0-9]{5}[A-Z]{1}\", target_page_html)\n",
    "        id_list = list(dict.fromkeys(all_ids))\n",
    "\n",
    "        for index, hcp_id in enumerate(id_list): \n",
    "            WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.XPATH, f\"//a[contains(@onclick,'{hcp_id}')]\"))).click()           \n",
    "            hcp_dict = gen_hcp_dict()\n",
    "            df_hcp_dict = pd.DataFrame(hcp_dict, index=[0])\n",
    "            df_hcp_dict.to_csv('master_list.csv', mode='a', header=False)\n",
    "            print(f'Scraped row {index+1} of {target_page}')\n",
    "                  \n",
    "            if index == len(id_list)-1:\n",
    "                print(f'Completed scraping for page {target_page}')\n",
    "                print('Mission Complete')\n",
    "            else:\n",
    "                pass\n",
    "            \n",
    "            WebDriverWait(driver, waittime).until(EC.presence_of_element_located((By.LINK_TEXT, 'Back to Search Results'))).click() \n",
    "            locate_target_page(target_page)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (iii) Kickstart automated web scraping\n",
    "It's been a relatively intense process so far, in terms of setting up all the functions correctly. This will require plenty of trial and error, so do prepare to spend some time and effort to get the automation right. Putting things in perspective, this short term hassle will still deliver much much more efficiency compared to doing this scraping manually.\n",
    "\n",
    "Now that we are done with the hardest part of the coding, so it is time to execute all our functions and witness the fruits of our labour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16:02:11\n"
     ]
    }
   ],
   "source": [
    "# Start with selected target page. target_page = 1 by default (to start from page 1)\n",
    "target_page = 1\n",
    "\n",
    "# Display time started\n",
    "print(time.strftime(\"%H:%M:%S\", time.localtime()))\n",
    "\n",
    "# Run main web scraping function\n",
    "full_scrape(target_page)\n",
    "\n",
    "# Display time concluded\n",
    "print(time.strftime(\"%H:%M:%S\", time.localtime()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**: You will notice that there are many wait times peppered across the script. This is because the scraping will fail if a particular page is not completely loaded. Therefore, these wait times provide some buffer time for page loading. More info on waits can be found here: https://selenium-python.readthedocs.io/locating-elements.html\n",
    "\n",
    "Despite that, the webpage loading may still fail from time to time. If that is the case, simply take note which page was last scraped successfully (from the output screen in the above cell). You then update the `target_page` in the above chunk, and rerun the automated scraping. You may do so by re-running the code from the WebDriver initiation section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (iv) Data Cleaning of Master List"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once scraping is completed, we can now tidy up the master list CSV file where our data has been progressively appended to. This is important since there will likely be row duplicates if you encounter scraping failures along the way and need to re-run scraping from break points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Unnamed: 0', 'name', 'reg_number', 'reg_date', 'reg_end_date',\n",
       "       'reg_type', 'practice_status', 'cert_start_date', 'cert_end_date',\n",
       "       'qualification', 'practice_place_name', 'practice_place_address',\n",
       "       'practice_place_phone'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import master list file\n",
    "df_master = pd.read_csv(f'{file_name}')\n",
    "\n",
    "# Display master list columns\n",
    "df_master.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop leftmost index column (Unnecessary column)\n",
    "df_master.drop(columns = [df_master.columns[0]], axis = 1, inplace = True)\n",
    "\n",
    "# Remove duplicates\n",
    "df_master.drop_duplicates(keep='first', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export final curated list\n",
    "df_master.to_excel('master_list_final.xlsx', index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (v) Conclusion\n",
    "\n",
    "And that's it! You have successfully performed web scrapping on a website that is relatively difficult to navigate. The concepts for the other types of websites are also similar, so you are in good stead to try out web scraping for other kinds of projects. If you do encounter specific errors or problems, do always head back to the usual trusty resources like StackOverflow and Selenium documentation pages.\n",
    "\n",
    "Once the data has been curated, you can then proceed to perform fun analysis on the dataset e.g. Finding out proportions of professionals who graduated from various countries.\n",
    "\n",
    "Please feel free to reach out to me at https://www.linkedin.com/in/kennethleungty/ should you have any suggestions or queries.\n",
    "\n",
    "Happy web scraping!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"section4\"></a>\n",
    "## Section 4 - References  \n",
    "\n",
    "- https://selenium-python.readthedocs.io/locating-elements.html\n",
    "- https://stackoverflow.com/questions/49171370/python-selenium-how-can-click-on-onclick-elements\n",
    "- https://stackoverflow.com/questions/23924008/get-the-text-from-multiple-elements-with-the-same-class-in-selenium-for-python\n",
    "- https://selenium-python.readthedocs.io/waits.html\n",
    "- https://stackoverflow.com/questions/28778142/selenium-webdriver-give-nosuchframeexception\n",
    "- https://medium.com/dunder-data/jupyter-to-medium-initial-post-ecd140d339f0"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}