long_sentence_abstrative_summary.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Appraoch 1: current model + summary"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Load relevant packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', 0)\n",
    "import pandas as pd\n",
    "import trafilatura\n",
    "from summa import summarizer\n",
    "import time \n",
    "import nltk\n",
    "# nltk.download('punkt')\n",
    "pd.options.display.float_format = '{:.10f}'.format\n",
    "\n",
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline\n",
    "import torch\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "from tqdm.notebook import tqdm\n",
    "tqdm.pandas()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Loading the current model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we have restricted the length of the text being fed into our models. Bert in particular is restricted to consuming 512 tokens per sample. For many use-cases, this is most likely not a problem - but in some cases it can be. For instance in our case we need context when computing sentiment analysis for a sentence. On these longer pieces of text, the actual sentiment from the author may not be clear from the first 512 tokens. We need to consider the full post. BUt to start, we test with scores from original approach"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_checkpoint = \"nlptown/bert-base-multilingual-uncased-sentiment\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # initialize our model and tokenizer\n",
    "def_tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast = True)\n",
    "def_model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint)\n",
    "def_sentiment_pipeline = pipeline(\"sentiment-analysis\", model=def_model ,tokenizer=def_tokenizer,  top_k=5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Custom functions to increase text readability + compute sentiment "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The cleaning steps include:\n",
    "    # Removing non-alphanumeric characters and specific French characters.\n",
    "    # Removing URLs.\n",
    "    # Removing \"rt\" (retweet) tags.\n",
    "    # Removing excessive spaces and replacing consecutive spaces with a single space.\n",
    "    # Stripping leading and trailing whitespace.\n",
    "    # Replacing newline characters with spaces.\n",
    "    # Removing the possessive form \"'s\" by replacing it with \"s\".\n",
    "    # If any error occurs during the cleaning process, an error message is printed, and the original text is returned without any modifications.\n",
    "\n",
    "def clean_string(text:str) -> str:\n",
    "    try: \n",
    "        # utf8_apostrophe = b'\\xe2\\x80\\x99'.decode(\"utf8\")\n",
    "        DATE_REGEX = r\"\\d+\\/\\d+\\/\\d+\" # Regular expression pattern for matching date format\n",
    "        text = re.sub(r\"([^0-9A-Za-z àâäèéêëîïôœùûüÿçÀÂÄÈÉÊËÎÏÔŒÙÛÜŸÇ\\~\\°\\&\\“\\%\\'\\:\\;\\!\\-\\’\\\"\\.\\,\\?\\€\\$\\t\\n`(\\d+\\/\\d+\\/\\d+)``\\(.*\\)`])|(\\w+:\\/\\/\\S+)|^rt|http.+?\", \"\", text)\n",
    "        # Remove unwanted characters, URLs, and RT tags from the text\n",
    "        text = re.sub(' {2,}', ' ', text) # Remove inconsistent spaces (2 or more spaces become a single space)\n",
    "        text = re.sub(' \\. ', '.', text) # Remove inconsistent spaces around periods\n",
    "        # text = re.sub(utf8_apostrophe, \"'\", text)  # Substitute UTF-8 apostrophe with a regular apostrophe\n",
    "        text = text.strip()  # Strip leading and trailing whitespace\n",
    "        text = re.sub(r\"\\n\", \" \", text) # Replace newline characters with a space\n",
    "        text = re.sub(r\"\\'s\\b\", \"s\", text) # Replace \"'s\" with \"s\" (possessive form)\n",
    "        return text\n",
    "    except Exception as e:\n",
    "        print(f\"Error cleaning text: {e}\")\n",
    "        return text"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3.1: Logic to calculate sentiment, use multipmiers based on entity found"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _calculate_sentiment(html_content:str, title_sentences:str, name:str, model, tokenizer ):\n",
    "\n",
    "    # A quick cleaninig of html_content and title\n",
    "    html_content = clean_string(text = html_content)  #\n",
    "    title_sentences = clean_string(text = title_sentences) #\n",
    "    html_content = re.sub(re.escape(title_sentences), \"\", html_content) # this code eliminates the duplicated title text in html_content\n",
    "    content_sentences = nltk.sent_tokenize(html_content)\n",
    "    content_inputs = tokenizer(content_sentences,  padding=True, truncation=True, return_tensors=\"pt\").to(device)\n",
    "\n",
    "    #  torch.no_grad() = context manager that temporarily disables gradient calculation, which reduces memory usage and \n",
    "    #  speeds up computation when you only need to perform forward passes through the network (e.g., during inference).\n",
    "    with torch.no_grad(): \n",
    "        content_outputs = model(**content_inputs)\n",
    "\n",
    "    # Softmax for probability scaling, i.e. sum of all probabilities = 1    \n",
    "    content_scores = torch.nn.functional.softmax(content_outputs.logits, dim=-1)\n",
    "    # Define the scale and shift it, in this case to --> -1, -0.5, 0, 0.5, 1\n",
    "    weights = torch.linspace(-1, 1, content_scores.shape[1])\n",
    "    normalised_content_scores = torch.matmul(content_scores, weights).tolist()\n",
    "    \n",
    "    # For title content\n",
    "    title_inputs = tokenizer(title_sentences, padding=True, truncation=True, return_tensors=\"pt\").to(device)\n",
    "    with torch.no_grad():       \n",
    "        title_outputs = model(**title_inputs)\n",
    "    title_scores = torch.nn.functional.softmax(title_outputs.logits, dim=-1)\n",
    "    normalised_title_scores = torch.matmul(title_scores, weights).tolist()\n",
    "\n",
    "    mentions = 0\n",
    "    score = 0.0\n",
    "    count = 0\n",
    "    for sentence, sentiment in zip(content_sentences, normalised_content_scores):\n",
    "        multiplier = 1\n",
    "        if name.lower() in sentence.lower():\n",
    "            mentions += sentence.lower().count(name.lower())\n",
    "            multiplier = 5\n",
    "\n",
    "        count += multiplier\n",
    "        score += sentiment * multiplier\n",
    "\n",
    "    for sentence, sentiment in zip(title_sentences, normalised_title_scores):\n",
    "        mentions += sentence.lower().count(name.lower())\n",
    "        multiplier = 5\n",
    "        count += multiplier\n",
    "        score += sentiment * multiplier\n",
    "\n",
    "    average_score = score / count\n",
    "    article_score = average_score * 50 + 50\n",
    "\n",
    "    return round(article_score, 5), mentions"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3.2: Fetch the data from databse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_text = \"What is Volkswagen accused of?\\nIt's been dubbed the \\\"diesel dupe\\\". In September, the Environmental Protection Agency (EPA) found that many VW cars being sold in America had a \\\"defeat device\\\" - or software - in diesel engines that could detect when they were being tested, changing the performance accordingly to improve results. The German car giant has since admitted cheating emissions tests in the US.\\nVW has had a major push to sell diesel cars in the US, backed by a huge marketing campaign trumpeting its cars' low emissions. The EPA's findings cover 482,000 cars in the US only, including the VW-manufactured Audi A3, and the VW models Jetta, Beetle, Golf and Passat. But VW has admitted that about 11 million cars worldwide, including eight million in Europe, are fitted with the so-called \\\"defeat device\\\".\\nThe company has also been accused by the EPA of modifying software on the 3 litre diesel engines fitted to some Porsche and Audi as well as VW models. VW has denied the claims, which affect at least 10,000 vehicles.\\nIn November, VW said it had found \\\"irregularities\\\" in tests to measure carbon dioxide emissions levels that could affect about 800,000 cars in Europe - including petrol vehicles. However, in December it said that following investigations, it had established that this only affected about 36,000 of the cars it produces each year.\\nThis 'defeat device' sounds like a sophisticated piece of kit.\\nFull details of how it worked are sketchy, although the EPA has said that the engines had computer software that could sense test scenarios by monitoring speed, engine operation, air pressure and even the position of the steering wheel.\\nWhen the cars were operating under controlled laboratory conditions - which typically involve putting them on a stationary test rig - the device appears to have put the vehicle into a sort of safety mode in which the engine ran below normal power and performance. Once on the road, the engines switched out of this test mode.\\nThe result? The engines emitted nitrogen oxide pollutants up to 40 times above what is allowed in the US.\\nWhat has been VW's response?\\n\\\"We've totally screwed up,\\\" said VW America boss Michael Horn, while the group's chief executive at the time, Martin Winterkorn, said his company had \\\"broken the trust of our customers and the public\\\". Mr Winterkorn resigned as a direct result of the scandal and was replaced by Matthias Mueller, the former boss of Porsche.\\n\\\"My most urgent task is to win back trust for the Volkswagen Group - by leaving no stone unturned,\\\" Mr Mueller said on taking up his new post.\\nVW has also launched an internal inquiry.\\nBut that's unlikely to be the end of the financial impact. The EPA has the power to fine a company up to $37,500 for each vehicle that breaches standards - a maximum fine of about $18bn.\\nThe costs of possible legal action by car owners and shareholders \\\"cannot be estimated at the current time\\\", VW added.\\nHow widespread are VW's problems?\\nWhat started in the US has spread to a growing number of countries. The UK, Italy, France, South Korea, Canada and, of course, Germany, have opened investigations. Throughout the world, politicians, regulators and environmental groups are questioning the legitimacy of VW's emissions testing.\\nVW will recall 8.5 million cars in Europe, including 2.4 million in Germany and 1.2 million in the UK, and 500,000 in the US as a result of the emissions scandal.\\nNo wonder the carmaker's shares have fallen by about a third since the scandal broke.\\nWill more heads roll?\\nIt's still unclear who knew what and when, although VW must have had a chain of management command that approved fitting cheating devices to its engines, so further departures are likely.\\nChristian Klingler, a management board member and head of sales and marketing is leaving the company, although VW said this was part of long-term planned structural changes and was not related to recent events.\\nIn 2014, in the US, regulators raised concerns about VW emissions levels, but these were dismissed by the company as \\\"technical issues\\\" and \\\"unexpected\\\" real-world conditions. If executives and managers wilfully misled officials (or their own VW superiors) it's difficult to see them surviving.\\nAre other carmakers implicated?\\nThat's for the various regulatory and government inquiries to determine. California's Air Resources Board is now looking into other manufacturers' testing results. Ford, BMW and Renault-Nissan have said they did not use \\\"defeat devices\\\", while other firms have either not commented or simply stated that they comply with the law.\\nThe UK trade body for the car industry, the SMMT, said: \\\"The EU operates a fundamentally different system to the US - with all European tests performed in strict conditions as required by EU law and witnessed by a government-appointed independent approval agency.\\\"\\nBut it added: \\\"The industry acknowledges that the current test method is outdated and is seeking agreement from the European Commission for a new emissions test that embraces new testing technologies and is more representative of on-road conditions.\\\"\\nThat sounds like EU testing rules need tightening, too.\\nEnvironmental campaigners have long argued that emissions rules are being flouted. \\\"Diesel cars in Europe operate with worse technology on average than the US,\\\" said Jos Dings, from the pressure group Transport & Environment. \\\"Our latest report demonstrated that almost 90% of diesel vehicles didn't meet emission limits when they drive on the road. We are talking millions of vehicles.\\\"\\nCar analysts at the financial research firm Bernstein agree that European standards are not as strict as those in the US. However, the analysts said in a report that there was, therefore, \\\"less need to cheat\\\". So, if other European carmakers' results are suspect, Bernstein says the \\\"consequences are likely to be a change in the test cycle rather than legal action and fines\\\".\\nIt's all another blow for the diesel market.\\nCertainly is. Over the past decade and more, carmakers have poured a fortune into the production of diesel vehicles - with the support of many governments - believing that they are better for the environment. Latest scientific evidence suggests that's not the case, and there are even moves to limit diesel cars in some cities.\\nDiesel sales were already slowing, so the VW scandal came at a bad time. \\\"The revelations are likely to lead to a sharp fall in demand for diesel engine cars,\\\" said Richard Gane, automotive expert at consultants Vendigital.\\n\\\"In the US, the diesel car market currently represents around 1% of all new car sales and this is unlikely to increase in the short to medium term.\\n\\\"However, in Europe the impact could be much more significant, leading to a large tranche of the market switching to petrol engine cars virtually overnight.\\\"\"\n",
    "summary = \"In September, the Environmental Protection Agency (EPA) found that many VW cars being sold in America had a \\\"defeat device\\\" - or software - in diesel engines that could detect when they were being tested, changing the performance accordingly to improve results. VW has had a major push to sell diesel cars in the US, backed by a huge marketing campaign trumpeting its cars' low emissions. The company has also been accused by the EPA of modifying software on the 3 litre diesel engines fitted to some Porsche and Audi as well as VW models. In November, VW said it had found \\\"irregularities\\\" in tests to measure carbon dioxide emissions levels that could affect about 800,000 cars in Europe - including petrol vehicles. VW will recall 8.5 million cars in Europe, including 2.4 million in Germany and 1.2 million in the UK, and 500,000 in the US as a result of the emissions scandal.\"\n",
    "title = \"Volkswagen: The scandal explained - BBC News\"\n",
    "name = \"Volkswagen\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "article_score,_ = _calculate_sentiment(html_content = html_text, title_sentences = title, name=name, model= def_model, tokenizer=def_tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38.10496"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ADDITIONAL CODE \n",
    "# Following code scrapes information from news/blog webistes\n",
    "# then you can run summarizer from summa library to have an abstractive summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/'\n",
    "downloaded = trafilatura.fetch_url(url=url)\n",
    "url_data = trafilatura.bare_extraction(downloaded, url=url, favor_precision=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': 'Leader spotlight: Erin Spiceland',\n",
       " 'author': 'Jessica Rudder',\n",
       " 'url': 'https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/',\n",
       " 'hostname': 'github.blog',\n",
       " 'description': 'We’re spending Women’s History Month with women leaders who are making history every day in the tech community.',\n",
       " 'sitename': 'The GitHub Blog',\n",
       " 'date': '2019-03-29',\n",
       " 'categories': ['Community'],\n",
       " 'tags': [],\n",
       " 'fingerprint': None,\n",
       " 'id': None,\n",
       " 'license': None,\n",
       " 'body': None,\n",
       " 'comments': '',\n",
       " 'commentsbody': None,\n",
       " 'raw_text': None,\n",
       " 'text': 'Low-code enables developers and non-developers to build custom applications and solutions with less effort. In this blog, we show you how to automate your low-code deployments using GitHub Actions.\\nEvery March we recognize the women who have shaped history—and now, we’re taking a look forward. From driving software development in large companies to maintaining thriving open source communities, we’re spending Women’s History Month with women leaders who are making history every day in the tech community. Erin Spiceland is a Software Engineer for SpaceX. Born and raised in rural south Georgia, she is a Choctaw and Chickasaw mother of two now living in downtown Los Angeles. Erin didn’t finish college—she’s a predominantly self-taught software engineer. In her spare time, she makes handmade Native American beadwork and regalia and attends powwows.\\nMy career has been a winding road through periods of stimulation and health as well as periods of personal misery. During it all, I’ve learned a variety of programming languages and technologies while working on a diverse array of products and services. I’m a domestic abuse survivor and a Choctaw bisexual polyamorous woman. I’m so proud of myself that I made it this far considering where I came from.\\nIn 2007, I had a three-year-old daughter and I was trying to finish my computer science degree one class at a time, all while keeping my house and family running smoothly. I found the math classes exciting and quickly finished my math minor, leaving only computer science classes. I was looking at about five years before I would graduate. Then, my husband at the time recommended me for an entry software developer position at a telecom and digital communications company.\\nWhen faced with the choice between an expensive computer science degree and getting paid to do what I loved, I dropped out of college and accepted the job. I was hired to work on internal tooling, and eventually, products. I did a lot of development on product front-ends, embedded network devices, and a distributed platform-as-a-service. I learned Java/JSP, Python, JavaScript/CSS, Node.js, as well as MySQL, PostgreSQL, and distributed systems architecture. It was an intense experience that required a lot of self-teaching, asking others for help, and daycare, but it set me up for my later successes.\\n“Leadership is about enabling those below, above, and around you to be at their healthiest and most effective so that all of you can accurately understand your surroundings, make effective plans and goals for the future, and achieve those goals.”\\nI appreciate and admire technical, effective leaders who care for their reports as humans, not as lines on a burndown chart, and forego heavy-handed direction in favor of communication and mutual dialogue. I think it’s as important for a leader to concern herself with her coworkers’ personal well-being as it is for her to direct their performance.\\nLast year I took a pay cut to move from a safe, easy job where I had security to work in a language I hadn’t seen in years and with systems more complicated than anything I’d worked with before. I moved from a place where I had a huge four bedroom house to a studio apartment that was twice the price. I moved away from my children, of who I share custody with my ex-husband. We fly across the U.S. to see each other now. I miss my children every day. However, I get to be a wonderful role model for them.\\n“I get to show my children that a Native woman who grew up in poverty, lost her mother and her culture, and who didn’t finish college can learn, grow, and build whatever career and life she wants.”\\nI can’t wait to wake up every day with my partner who loves me so much. I’m looking forward to showing my children exactly how far they can go. I’m excited to keep exploring Los Angeles.\\n“I expect to learn so much more about software and about life, and I want to experience everything.”\\nWant to learn more about featured leaders for Women’s History Month? Read about:\\n- Laura Frank Tacho, Director of Engineering at CloudBees\\n- Rachel White, Developer Experience Lead at American Express\\n- Kathy Pham, Computer Scientist and Product Leader at Mozilla and Harvard\\n- Heidy Khlaaf, Research Consultant at Adelard LLP\\nCheck back in soon—we’ll be adding new interviews weekly throughout March.',\n",
       " 'language': None,\n",
       " 'image': 'https://github.blog/wp-content/uploads/2019/03/Erin_FB.png?fit=4801%2C2521',\n",
       " 'pagetype': 'article'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(summarizer.summarize(text))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.6 ('bert_cloudrun')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "b1975ba15f5c6f49e9393a9ffcbc1714a9b9c74c65b1617749f5a6fb7923474c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}