apis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "title: \"Finding Undocumented APIs\"\n",
    "pagetitle: \"Finding Undocumented APIs\"\n",
    "description-meta: \"Introduction, case studies, and exercises for finding and using hidden and undocumented APIs.\"\n",
    "description-title: \"Introduction, case studies, and exercises for finding and using hidden and undocumented APIs.\"\n",
    "author: \"Leon Yin\"\n",
    "author-meta: \"Leon Yin\"\n",
    "date: \"02-24-2023\"\n",
    "date-modified: \"06-15-2023\"\n",
    "bibliography: references.bib\n",
    "execute: \n",
    "  enabled: false\n",
    "keywords: data collection, hidden api, undocumented api, web scraping, api\n",
    "twitter-card:\n",
    "  title: Finding Undocumented APIs\n",
    "  description: Introduction, case studies, and exercises for finding and using undocumented APIs hidden in plain sight.\n",
    "  image: assets/inspect-element-logo.jpg\n",
    "open-graph:\n",
    "  title: Finding Undocumented APIs\n",
    "  description: Introduction, case studies, and exercises for finding and using undocumented APIs hidden in plain sight.\n",
    "  locale: us_EN\n",
    "  site-name: Inspect Element\n",
    "  image: assets/inspect-element-logo.jpg\n",
    "href: apis\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "<a type=\"button\" class=\"btn btn-outline-primary btn-sm\" target=\"_blank\" href=\"https://inspectelement.org/apis\">📖 Read online</a>\n",
       "<a type=\"button\" class=\"btn btn-outline-primary btn-sm\" target=\"_blank\" href=\"https://colab.research.google.com/github/yinleon/inspect-element/blob/main/apis.ipynb\">🖥️ Interactive version</a>\n",
       "<a type=\"button\" class=\"btn btn-outline-primary btn-sm\" target=\"_blank\" href=\"https://github.com/yinleon/inspect-element/blob/main/apis.ipynb\">⚙️ GitHub</a>\n",
       "<a type=\"button\" class=\"btn btn-outline-primary btn-sm\" target=\"_self\" href=\"#citation\">🏛 Citation</a>\n",
       "<br>\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#| echo: false\n",
    "from utils import build_buttons\n",
    "build_buttons(link= 'apis', \n",
    "              github= 'https://github.com/yinleon/inspect-element/blob/main/apis.ipynb', \n",
    "              citation= True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most APIs are undocumented and hidden in plain sight. \n",
    "\n",
    "Being able to find these APIs can provide a rich, reliable, and scalable method of building your own datasets.\n",
    "\n",
    "Learn how to find them in the wild, and how they've been used in past investigations.\n",
    "\n",
    "👉[Click here to jump to the tutorial](#tutorial)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro\n",
    "## What is an A-P-I?\n",
    "\n",
    "If you have tried to get a driver's license or a travel visa, you have experienced bureaucracy at its finest-- a series of lines, forms, credential-showing, and waiting.\n",
    "\n",
    "Application Program Interfaces, or APIs, are digitized bureaucracy. You make a request, and then wait in a queue to be served by a server. However, instead of leaving with a driver's license or a custom plate, what you’re waiting for is well-formatted data. As for making mistakes... well, you'll get an automated rejection and zero sympathy.\n",
    "\n",
    "Most APIs are undocumented and hidden in plain sight. There is no set terminology for these APIs, so for simplicity's sake we'll refer to them as \"undocumented APIs\".\n",
    "\n",
    "Some investigations are only possible after finding and reverse-engineering undocumented APIs. Our first case study illustrates this in detail, where my reporting partner Aaron Sankin and I discovered undocumented APIs out of necessity while reporting on YouTube."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A key tool for investigations\n",
    "\n",
    "YouTube is the [largest video-hosting platform](https://www.statista.com/topics/2019/youtube/) in the world, and plays a major role in the creator economy thanks to the YouTube Partner Program that shares ad revenue with eligible creators.\n",
    "\n",
    "Advertising is central to YouTube, and major companies have boycotted the platform in the past in response to their ads appearing alongside extremist content. We wanted to better understand YouTube's advertising system, especially how they treated hateful conspiracy theories, which at the time, seemed to thrive on the platform.\n",
    "    \n",
    "To start investigating this topic, we got acquainted with the Google Ads portal. Anyone can sign-up, and see all the tools that marketers can use to reach users across the Google adverse. YouTube has a special section of the ad portal, where marketers can target their ads based on user demographics and the content of videos.\n",
    "\n",
    "We investigated a specific targeting tool that allows ad-buyers to use keyword searches to find videos and channels to place their ads on. \n",
    "\n",
    "<i>Read the [investigation](https://themarkup.org/google-the-giant/2021/04/08/google-youtube-hate-videos-ad-keywords-blocklist-failures) along with its accompanying [methodology](https://themarkup.org/google-the-giant/2021/04/08/how-we-discovered-googles-hate-blocklist-for-ad-placements-on-youtube)</i>.\n",
    "\n",
    "In an initial test, we found that searching for the racist \"[White genocide](https://www.adl.org/resources/glossary-term/white-genocide)\" conspiracy theory returned no videos, but by simply removing spaces, we were shown relevant results.\n",
    "\n",
    "<figure>\n",
    "<video src=\"https://mrkp-static-production.themarkup.org/graphics/youtube-hate-video/1617820765826/assets/white-genocide--compressed.mp4\" autoplay=true loop=true width=100%></video><figcaption align = \"left\" style=\"font-size:80%;\">Searching the Google Ads portal for YouTube videos related to a conspiracy theory, and circumventing blocking measures. Source: Google.com/The Markup</figcaption>\n",
    "</figure>\n",
    "\n",
    "This anecdotal test suggested that Google was using a keyword blocklist that hid results for certain search terms, but not others. We performed further tests and found that searching for swears and strings of gibberish also surfaced no results. \n",
    "\n",
    "We wanted to verify if Google was using a keyword blocklist, and test how consistent that blocklist was with YouTube's \"advertiser friendly\" [guidelines](https://support.google.com/youtube/answer/6162278?hl=en).  Unfortunately, the portal did not make it possible to discern between a blocked keyword and one that may have been too obscure to return any results.\n",
    "\n",
    "Our colleague Surya Mattu suggested using the web browser's built-in `developer tools` to monitor [network requests](https://doc.arcgis.com/en/appstudio/extend-apps/apinetworkrequest.htm) while we made searches in the portal. This proved to be a breakthrough that allowed us to isolate the API-endpoint being called during this process, reverse-engineer it to return results for any given keyword, and analyze the API's response before its contents were displayed in the portal.\n",
    "\n",
    "By looking closely at the API responses, we were able to identify clear structural differences based on Google's verdict of the keyword. Blocked terms returned an empty JSON string `{}`, whereas obscure terms returned a JSON with labels but no results:\n",
    "\n",
    "```{“videos”: [], “channels\": []}```\n",
    "\n",
    "With the categorization scheme established, we could confirm search terms were being blocked (read about this in detail [here](https://themarkup.org/google-the-giant/2021/04/08/how-we-discovered-googles-hate-blocklist-for-ad-placements-on-youtube#data-collection)). Moreover, with the API at our service, we could test any set of keywords, so we tested well-known [hate terms](https://themarkup.org/google-the-giant/2021/04/08/how-we-discovered-googles-hate-blocklist-for-ad-placements-on-youtube#sourcing-hate-keywords) and phrases related to \"racial justice and representation\" that we asked [independent advocacy groups](https://themarkup.org/google-the-giant/2021/04/09/how-we-discovered-googles-social-justice-blocklist-for-youtube-ad-placements#sourcing-social-justice-keywords) to send us. \n",
    "\n",
    "After testing the two keyword lists, we saw a pattern of Google blocking racial justice terms (like \"Black power\"), while showing advertisers results for well-known hate terms (like \"White power\").\n",
    "\n",
    "<figure>\n",
    "<img src=\"assets/blocked-terms.png\" style=\"width:90%\" aria-label='Video thumbnails of videos suggested for \"white power\", but no thumbnails for the blocked \"black power\" phrase.'>\n",
    "</figure>\n",
    "\n",
    "This was my first time finding and using an undocumented API for an investigation. Doing so revealed essential information that was not visible to everyday users, and it allowed us to perform systematic tests and bring receipts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we dive deeper into how to find undocumented APIs, it's important to note that some APIs are \"official\" and well-documented to the public. These APIs can also be used to great effect.\n",
    "\n",
    "## Documented APIs\n",
    "\n",
    "Many businesses sell their services using APIs.\n",
    "\n",
    "The benefit of documented APIs is self-explanatory, you know what you are going to get, and there are notes and examples to help developers use the tool as intended.\n",
    "\n",
    "Some documented APIs are also free to use, making them a great tool for teaching and research. \n",
    "One such API that journalists frequent is the [Census Data API](https://www.census.gov/data/developers/guidance/api-user-guide.html), which we use to retrieve statistical survey data from across the United States.\n",
    "Unfortunately, free APIs can often disappear or have their access severely limited-- as we've seen with Twitter (no longer free), YouTube (severely restricted), and Facebook (deprecated entirely)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How have Documented APIs been used?\n",
    "\n",
    "- [Gender Shades](https://gendershades.org/) was an audit of three commercially-available facial recognition APIs used to automate gender classification (from Microsoft, IBM, and Face++). The authors created a benchmark image dataset of faces hand-labeled by gender and skin tone, and tested each facial recognition model by sending the benchmark dataset through each respective model's API [@pmlr-v81-buolamwini18a]. The authors found that many models had high error rates for female and Black faces, with the worst performance on Black female faces.\n",
    "\n",
    "- Google's Perspective API was developed to filter out toxic comments for publishers such as [The New York Times](https://www.nytimes.com/2017/06/13/insider/have-a-comment-leave-a-comment.htm). Importantly, Perspective used \"[training data](https://en.wikipedia.org/wiki/Training,_validation,_and_test_data_sets)\" sourced from human-labeled [Wikipedia](https://en.wikipedia.org/wiki/Artificial_intelligence_in_Wikimedia_projects) edits. [An academic study](https://maartensap.com/pdfs/sap2019risk.pdf) found racially biased classifications of Tweets. For example, the use of certain identifiers for minority groups would flag a comment as \"toxic\" [@sap-etal-2019-risk]. Because Google had released the API publicly, researchers could access and audit this technology directly through the API.\n",
    "\n",
    "Now, let's get back to APIs that are undocumented and hidden."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Undocumented APIs\n",
    "\n",
    "These are the unsung heroes making sure websites run, often times executing essential functions behind the scenes.\n",
    "Many of these functions are so mundane, you probably don't even realize that something is happening.\n",
    "\n",
    "If you spend time on social media platforms, you'll find that the good times keep rolling, and you'll never reach the end of the page. That is because \"infinite scroll\" is powered by an API that is called upon as you approach the bottom of the page to load more fun things to eat up your day.\n",
    "\n",
    "Sometimes engineers find these API endpoints and build open source software to access public data programmatically. See [Instaloader](https://instaloader.github.io/) (for Instagram) as an example.\n",
    "\n",
    "Learning how to find and use these publicly available APIs can help you build evidence and test hypotheses that are otherwise unreachable due to lack of access."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Case Studies\n",
    "\n",
    "## *How have undocumented APIs been used?*\n",
    "\n",
    "Journalists and researchers have used undocumented APIs to catalog Amazon Ring's sprawling surveillance network [@calacci-2022; @gizmodo-ring-2019], measure inequities in internet access using Facebook's advertising ecosystem [@garcia-2018], and parse complex government documents listing presidential appointees [@willis-plum].\n",
    "\n",
    "Using undocumented APIs has three key strengths:\n",
    "\n",
    "1. **Richness**: APIs often contain information that is not visible on web pages.<br>\n",
    "2. **Reliability**: APIs execute essential functions, so they don't change often. This can make for a reliable data source over time.<br>\n",
    "3. **Scalability**: You can collect more information in less time using this method compared to [headless browsers](https://en.wikipedia.org/wiki/Headless_browser), such as Selenium, Puppeteer, and Playwright (Not throwing shade-- these tools have their purpose).\n",
    "\n",
    "Next we will cover three case studies, each of which is intended to highlight one of these benefits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case study on richness: Google's blocklist for YouTube advertisers\n",
    "\n",
    "I'm not going to rehash this case study, since we led with it in the [introduction](#a-key-tool-for-investigations), but...\n",
    "\n",
    "Using undocumented APIs can reveal **rich** metadata. This includes hidden fields that are not displayed to everyday users of a website, as well as subtle changes to the structural in how data is returned. \n",
    "\n",
    "Using this metadata produces receipts you can follow by deciphering the meaning of these hidden fields, finding traces left by missing data, and identifying patterns that are otherwise hidden from the surface (front-end) world.\n",
    "\n",
    "Certainly this was the case with the YouTube investigation, and something that we'll brush on again in the hands-on tutorial at the end of this section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case study on reliability: Amazon branded products\n",
    "\n",
    "If you have ever scraped HTML from a website, you've likely found yourself with a broken scraper. \n",
    "\n",
    "This occurs when class names, accessibility labels, text, or something else has changed and confused your scraper. In this sense, HTML scraping can be fragile and fickle, especially if your collecting data over a prolonged period of time.\n",
    "\n",
    "A stark example is Facebook's timeline, where you'll find [elements](https://developer.mozilla.org/en-US/docs/Glossary/Element) of the page are arbitrarily named, oddly-nested, and ever-changing.\n",
    "\n",
    "Using undocumented APIs can often get you the same information with a higher success-rate. This is because these APIs interact with the same backend (fetching information before being rendered, named, and nestled neatly into a webpage), and are often essential to the operation of the website.\n",
    "\n",
    "In the investigation \"[Amazon's Advantage](https://themarkup.org/amazons-advantage/2021/10/14/how-we-analyzed-amazons-treatment-of-its-brands-in-search-results)\", Adrianne Jeffries and I found a **reliable** method of identifying Amazon brands and exclusive products. At the time, these products were not clearly labeled, most Americans we surveyed were unable to identify Amazon's top brands, and no source of truth existed. \n",
    "\n",
    "We developed a approach to identify these products as Amazon private label using a filter found in the user interface of Amazon's website. The \"Our brands\" filter did a lot of heavy lifting in our investigation, and we found that it was powered by a undocumented API that listed all the Amazon branded products for a given search.\n",
    "\n",
    "This method was key to our investigation, which required persistent data collection over a period of several months. To our surprise, the API continued to work after we went to Amazon for comments on our detailed methodology, after we published our investigation, and even after Amazon executives were accused of perjury by members of the U.S. Congress.\n",
    "\n",
    "Usually the party gets shut down once you call the parents, but in this case it didn't.\n",
    "\n",
    "Because the API continued to work, we used it in a browser extension ([_Amazon Brand Detector_](https://themarkup.org/amazons-advantage/2021/11/29/introducing-amazon-brand-detector)) that we (including Ritu Ghiya and Jeff Crouse) built to highlight Amazon brands for shoppers around the globe. About half a year later, Amazon added an orange disclaimer of \"Amazon brand\" to their branded products, but the API and extension still work at the time of writing, more than a year later.\n",
    "\n",
    "This case study emphasizes the reliability of using undocumented APIs, not only for collecting datasets, but for persistent accountability efforts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case study on scalability: collecting Internet Plans\n",
    "\n",
    "In the investigation, \"[Still Loading](https://themarkup.org/show-your-work/2022/10/19/how-we-uncovered-disparities-in-internet-deals)\" my reporting partner Aaron Sankin and I collected and analyzed over 1 million internet service plans across major cities in the United States. \n",
    "\n",
    "We learned a technique from a trio of researchers from Princeton, that used the lookup tools found on the internet service providers' websites to retrieve internet plans for a specific address [@princeton-2020].\n",
    "\n",
    "However, doing this using a browser (as a real person would) is incredibly slow. Even with 10 automated browsers (see below) with unique IP addresses, it would have taken months to collect a representative sample of a single major American city."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<video width=100% controls loop>\n",
    "  <source src=\"assets/att-scraper-selenium.mp4\" type=\"video/mp4\">\n",
    "</video><figcaption align = \"left\" style=\"font-size:80%;\">Automating checking for internet plans from AT&amp;T using Selenium browser automation.</figcaption>\n",
    "</figure>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Browser automation is bulky. Not only do you need to load every asset of a web page, there is also the compute resources necessary to spin up a browser. When you can get away without having to mock user interactions, or use rendered page elements, finding the underlying API(s) can be quicker and more eloquent.\n",
    "\n",
    "Initially, the workflow for getting an internet plan seemed too complex to pull off using an API-- there was user authentication that set a cookie, choosing an address from a list of suggestions, and adding an apartment number when prompted.\n",
    "\n",
    "However, we were able to keep track of cookies using a `session` (read about this advanced topic [here](https://requests.readthedocs.io/en/latest/user/advanced/#session-objects)), and speed things up by bundling the sequence of successive API calls into a function.\n",
    "\n",
    "Not only was this function easier to write, but it was able to be written and executed [asynchronously](https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)). Meaning we could request internet plans from many addresses at the same time.\n",
    "\n",
    "This allowed us to collect AT&T internet plans for a representative sample of 21 cities in two days, rather than two years. \n",
    "\n",
    "Timely data collection is key. Solving this issue allowed us to be ambitious in the scope of our investigation, which [ultimately found](https://themarkup.org/still-loading/2022/10/19/dollars-to-megabits-you-may-be-paying-400-times-as-much-as-your-neighbor-for-internet-service) that Internet pricing disparities were common for lower-income, least-White, and historically redlined areas.\n",
    "\n",
    "When it comes to web scraping, undocumented APIs offer unmatched **scalability** to collect massive amounts of data. This is especially true when you orchestrate them with asynchronous and multi-threaded programming (another topic we plan to cover in a future section).\n",
    "\n",
    "Although the process of finding undocumented APIs is not too complicated (as you'll see in the tutorial), the chances of finding one that is helpful for your investigation or research are still quite low. Don't be deterred, that just makes finding a useful one more special."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## *How to find and use undocumented APIs*\n",
    "\n",
    "In this exercise, you'll learn to sniff out undocumented APIs using the web browser’s `developer tools` (shortened to dev tools), figure out how they work, test different inputs, and analyze API responses.\n",
    "\n",
    "You can do most of this tutorial with zero coding, but it'll hold you back from using APIs to their fullest.\n",
    "\n",
    "::: {.callout-note}\n",
    "Note that if you're in a workshop setting: hitting the example API at the same time will get us all blocked from the website!\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. First open the developer console. \n",
    "\n",
    "See how on [Chrome](https://developer.chrome.com/docs/devtools/open/) or [Firefox](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_are_browser_developer_tools) here.  \n",
    "\n",
    "In this tutorial, we'll see how Amazon.com autocomplete search suggestions work.\n",
    "\n",
    "One way to get to the dev tools it to right-click and “Inspect” an element on the page. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<img src=\"assets/inspect-panel.png\" style=\"width:50%\">\n",
    "<figcaption align = \"left\" style=\"font-size:80%;\"> Example of inspecting an element on a page using a right-click </figcaption>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "This will open the dev tools under the “Elements” tab, which is used to explore the source code of a page. \n",
    "\n",
    "Page source code is useful because it reveals clues that are otherwise unseen by regular users. Often times, clues are in accessibility features known as ARIA elements.\n",
    "\n",
    "However, this tutorial is not about source code... it's about API requests that populate what we see on the page, and the hidden fields that we don't see.\n",
    "\n",
    "Let's try this!\n",
    "\n",
    "With dev tools open, go to Amazon.com, select the search bar on the website, and start typing a query (such as \"spicy\")."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Click the “Network” tab.\n",
    "\n",
    "This section of the dev tools is used to monitor network requests.\n",
    "\n",
    "*Background*\n",
    "\n",
    "Everything on a page is retrieved from some outside source, likely a server. This includes things like images embedded on the page, JavaScript code running in the background, and all the bits of “content” that populate the page before us.\n",
    "\n",
    "Using the `Network` tab, we can find out how this information is requested from a server, and intercept the response before it is rendered on the page.\n",
    "\n",
    "These responses are information-rich, and contain fields that don’t end up in the source code *or* in the user interface that most people encounter when they visit a site.\n",
    "\n",
    "Further, we can reverse-engineer how this request is made, and use it to collect structured data at scale. This is the power of finding undocumented APIs.\n",
    "\n",
    "*Back to the console...*\n",
    "\n",
    "The `Network` tab can look pretty hectic at first. It has many uses, and a lot of information. We'll cover some of the basics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<video width=100% controls loop>\n",
    "  <source src=\"assets/dev-console.mp4\" type=\"video/mp4\">\n",
    "</video>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Filter requests by fetch/XHR\n",
    "\n",
    "This will reveal only API calls made to servers. This includes internal servers that are hosted by the website we’re inspecting, as well as external servers. The latter often includes [third-party trackers](https://themarkup.org/blacklight) used in adtech, and verification services to authenticate user behavior."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<video width=100% controls loop>\n",
    "  <source src=\"assets/filter-network.mp4\" type=\"video/mp4\">\n",
    "</video>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might see quite a few network requests that were loaded onto the page. Look at \"Domain\" and \"File\" to narrow down where requests were sent, and whether the names are telling of the purpose of the request. \n",
    "\n",
    "::: {.callout-tip}\n",
    "#### Pro tip:\n",
    "You can \"Filter URLs\" using different properties (see how to do this for [Chrome](https://developer.chrome.com/docs/devtools/network/reference/#filter-by-property) and [Firefox](https://firefox-source-docs.mozilla.org/devtools-user/network_monitor/request_list/index.html#filtering-requests)).\n",
    ":::\n",
    "\n",
    "In this example, notice that a request was sent to the \"Domain\" `completion.amazon.com`, using an API endpoint (in the \"File\" column) named `suggestions`. This is likely the API being called to populate autocompleted search suggestions on the Amazon marketplace. Reading \"File\" names can help determine each API's function.\n",
    "\n",
    "When clicking the network request, you'll see \"Headers\". Those are the [HTTP headers](https://developer.mozilla.org/en-US/docs/Glossary/Request_header) that were sent along with the network request. This is not useful for us _just yet_, instead we want to see what data gets transferred as a result of the API call.\n",
    "\n",
    "To do this, we'll look at the request's \"Response\" attributes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Analyze the response\n",
    "This might seem intimidating at first, but let me _key_ you in on some tips. Responses are almost always JSON-formatted. JSON is made up of lists and [key-value](https://en.wikipedia.org/wiki/Name%E2%80%93value_pair) pairs. This means the information is stored like a dictionary, with words and their corresponding definitions.\n",
    "\n",
    "Looking at the JSON response, it looks like Amazon's `completion.amazon.com/suggestions` API returns a list of \"suggestions\". Each item in the list of suggestions has a \"value\", in the example above that \"value\" is `spicy ramen`. \n",
    "\n",
    "**Check your work**: confirm this interpretation is correct by cross-referencing the API response with what a user would see on the website.\n",
    "\n",
    "\n",
    "<figure>\n",
    "<img src=\"assets/spicy.png\" width=50%>\n",
    "<figcaption align = \"left\" style=\"font-size:80%;\"> Amazon's suggestions for \"spicy\".</figcaption>\n",
    "</figure>\n",
    "\n",
    "Another check you can perform `CTRL+F` the JSON response for a unique string. This could be a string of text on the page (or something else) that serves as a unique tracer. Verifying its presence will help pinpoint the right API call."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting these steps down, is your one way ticket to spicy town, and you don't need to code at all.\n",
    "\n",
    "However, some rudimentary coding can help you figure out how to use the API for vast inputs to collect your own dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Copy as cURL\n",
    "\n",
    "If you find an HTTP request that returns a response with useful information you can start to reverse-engineer it. To do that, we can isolate it by right-clicking the HTTP request and selecting “copy as cURL”. ([cURL](https://developer.ibm.com/articles/what-is-curl-command/) stands for client URL, and is a tool used to transfer data across networks.)\n",
    "\n",
    "<img src=\"assets/copy-curl.png\" width=90%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Curl to requests\n",
    "We can use a site like [curlconverter.com](https://curlconverter.com/) to convert the cURL we copied into a reusable API call. In this example, we use the default conversion to a Python `requests` script. You can do the same for any language and framework.\n",
    "\n",
    "Here is what the converted cURL looks like after being converted to a Python request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "cookies = {\n",
    "    'aws-ubid-main': '836-8365128-6734270',\n",
    "    'session-id-time': '2082787201l',\n",
    "    'ubid-main': '135-7086948-2591317',\n",
    "    'aws-priv': 'eyJ2IjoxLCJldSI6MCwic3QiOjB9',\n",
    "    'aws-target-static-id': '1593060129944-225088',\n",
    "    'lc-main': 'en_US',\n",
    "    'x-main': 'Oz3Tb5n2p0ic7OhF3cU5dc9B4ZR2gFjhKEsP4zikHHD3Gk2O7NpSmuShBxLFrhpZ',\n",
    "    'at-main': 'Atza|IwEBILB5ARQ_IgTCiBLam_XE2pyT76jXTbAXHOm2AJomLPmDgoJUJIIlUmyFeh_gChLHCycKjNlys-5CqqMabKieAzqSf607ChJsNevw-V06e7VKgcWjvoMaZRWlGiZ-c5wSJ-e4QzIWzAxTS1EI6sRUaRZRv-a0ZpOJQ-sHHB99006ytcrHhubdrXYPJRqEP5Q-_30JtESMpAkASoOs4vETSFp5BDBJfSWWETeotpIVXwA4NoC8E59bZb_5wHTW9cRBSWYGi1XL7CRl2xGbJaO2Gv3unuhGMB1tiq9iwxodSPBBTw',\n",
    "    'sess-at-main': '\"PUq9PW1TbO9CTYhGMo7l1Dz+wedh40Ki8Z9rPC+1TSI=\"',\n",
    "    'sst-main': 'Sst1|PQHsbeSFCMSY0X0_WgvTo5NUCaZkG2J9RPqWWy0fCpyWopJXgu6_drU_LstOdJB2cDmaVCXwkNpsF5yNPrBDj3Wtx-TC-AaYZn6WUdp8vNRPb6iYqxPAjRDnfK3pCnHqt19I0GoG7Bd1wnOxkAvnH0992IUq14kH6Ojm0J8noVPwMez0lltD-jxBwtDQ_EZYUkZG741RDVEojfziawJY9iKc-cLCnKmhi-ca1PPJnsimPV4lXRtMAGFbf9nMkKq4CbpkaRMdVtlPr20vF9eqg_V_-LY_V7S44WlO-_t_bFBnK8Q',\n",
    "    'i18n-prefs': 'USD',\n",
    "    'session-token': 'ptze73uznXExrMCSV9AklvNOKa1ND9F0rlQH2ioSM26Vr6hSheH8O4v4P8Lg3zuv7oDM+HZ+8f2TlyoPXUmPShprMXdvEpAQieXUw7+83PZOJvkkg1jwP0NiG0ZqksIYOr3Zuwt3omMcfCKRReWKxl5rGaDEM6AISpwI5aMDDCnA7fWbVO/QQYNxUZMifc599EZ5Fg3uGjCAhBlb6I7UO8ewRbXJ1bo9',\n",
    "    'session-id': '139-9925917-2023535',\n",
    "    'aws-userInfo-signed': 'eyJ0eXAiOiJKV1MiLCJrZXlSZWdpb24iOiJ1cy1lYXN0LTEiLCJhbGciOiJFUzM4NCIsImtpZCI6ImFhNDFkZjRjLTMxMzgtNGVkOC04YmU5LWYyMzUzYzNkOTEzYiJ9..LWFZOJMDcYdu6od6Nk8TmhAFMGA9O98O4tIOsVlR7w5vAS_JgVixL8j75u6jTgjfWkdddhKqa5kgsXDmGNbjhzLIsD48ch1BUodlzxqeQfn0r8onIwLbUIHEnk6X-AJE',\n",
    "    'skin': 'noskin',\n",
    "}\n",
    "\n",
    "headers = {\n",
    "    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0',\n",
    "    'Accept': 'application/json, text/javascript, */*; q=0.01',\n",
    "    'Accept-Language': 'en-US,en;q=0.5',\n",
    "    # 'Accept-Encoding': 'gzip, deflate, br',\n",
    "    'Origin': 'https://www.amazon.com',\n",
    "    'Connection': 'keep-alive',\n",
    "    'Referer': 'https://www.amazon.com/',\n",
    "   \n",
    "    'Sec-Fetch-Dest': 'empty',\n",
    "    'Sec-Fetch-Mode': 'cors',\n",
    "    'Sec-Fetch-Site': 'same-site',\n",
    "}\n",
    "\n",
    "params = {\n",
    "    'limit': '11',\n",
    "    'prefix': 'spicy',\n",
    "    'suggestion-type': [\n",
    "        'WIDGET',\n",
    "        'KEYWORD',\n",
    "    ],\n",
    "    'page-type': 'Gateway',\n",
    "    'alias': 'aps',\n",
    "    'site-variant': 'desktop',\n",
    "    'version': '3',\n",
    "    'event': 'onKeyPress',\n",
    "    'wc': '',\n",
    "    'lop': 'en_US',\n",
    "    'last-prefix': '\\0',\n",
    "    'avg-ks-time': '2486',\n",
    "    'fb': '1',\n",
    "    'session-id': '139-9925917-2023535',\n",
    "    'request-id': 'SVMTJXRDBQ9T8M7BRGNJ',\n",
    "    'mid': 'ATVPDKIKX0DER',\n",
    "    'plain-mid': '1',\n",
    "    'client-info': 'amazon-search-ui',\n",
    "}\n",
    "\n",
    "response = requests.get('https://completion.amazon.com/api/2017/suggestions', \n",
    "                        params=params, cookies=cookies, headers=headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can run this Python code, as-is, and it should work.\n",
    "\n",
    "Just a reminder that you can run this code interactively on [Google Colab](https://colab.research.google.com/) by [copying this tutorial](https://colab.research.google.com/github/yinleon/inspect-element/blob/main/apis.ipynb) (it's written as a Jupyter Notebook), or running a \"new notebook\" and pasting the code in.\n",
    "\n",
    "Press the little play button on the left to run the code. You should see something that looks similar to what you saw in the inspector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to see the response, run this cell:\n",
    "response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Strip it down\n",
    "\n",
    "You might be overwhelmed with the parameters that go into this API request. Like the response output, the inputs are formatted like a JSON, too. Start removing these parameters one-by-one. \n",
    "\n",
    "Keep parameters for authentication, and also the input parameters that you can change for your own purposes. Notice that the example query of \"spicy\" stored in the `prefix` parameter.\n",
    "\n",
    "::: {.callout-tip}\n",
    "#### Pro tip:\n",
    "Parameter values can expire, so periodically test the request and each parameter to assure you only keep the shelf-stable parts.\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {\n",
    "    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0',\n",
    "    'Accept': 'application/json, text/javascript, */*; q=0.01',\n",
    "    'Accept-Language': 'en-US,en;q=0.5',\n",
    "}\n",
    "\n",
    "params = {\n",
    "    'prefix': 'spicy',\n",
    "    'suggestion-type': [\n",
    "        'WIDGET',\n",
    "        'KEYWORD',\n",
    "    ],\n",
    "    'alias': 'aps',\n",
    "    'plain-mid': '1',\n",
    "}\n",
    "\n",
    "response = requests.get('https://completion.amazon.com/api/2017/suggestions', params=params, headers=headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Recycle and reuse\n",
    "\n",
    "With the stripped down request, try to submit a few— let’s say 10 or 20, requests with new parameters set by you.\n",
    "\n",
    "For convenience, we can write the stripped down API call as a function that takes any `keyword` as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import time\n",
    "\n",
    "def search_suggestions(keyword):\n",
    "    \"\"\"\n",
    "    Get autocompleted search suggestions for a `keyword` search on Amazon.com.\n",
    "    \"\"\"\n",
    "    headers = {\n",
    "        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0',\n",
    "        'Accept': 'application/json, text/javascript, */*; q=0.01',\n",
    "        'Accept-Language': 'en-US,en;q=0.5',\n",
    "    }\n",
    "\n",
    "    params = {\n",
    "        'prefix': keyword,\n",
    "        'suggestion-type': [\n",
    "            'WIDGET',\n",
    "            'KEYWORD',\n",
    "        ],\n",
    "        'alias': 'aps',\n",
    "        'plain-mid': '1',\n",
    "    }\n",
    "\n",
    "    response = requests.get('https://completion.amazon.com/api/2017/suggestions', \n",
    "                            params=params, headers=headers)\n",
    "    return response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step the code gets refactored to make it repeatable and reusable, but it's a bit of a jump if you're not a coder. If that's you, still try to read the code -- you should be able to get a rough idea of what does what, and how it's similar to the code you had in the last step. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can set new input parameters in `keyword`, and make the an API call using each keyword. Try changing some of the code (eg. the keywords) and rerunning it to check your understanding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here are our inputs (what searches we'll get autocompleted)\n",
    "keywords = [\n",
    "    'a', 'b', 'cookie', 'sock', 'zelda', '12'\n",
    "]\n",
    "\n",
    "# Here we'll go through each input, get the suggestions, and then add the `suggestions` to a list.\n",
    "data = []\n",
    "for keyword in keywords:\n",
    "    suggestions = search_suggestions(keyword)\n",
    "    suggestions['search_word'] = keyword # keep track of the seed keyword\n",
    "    time.sleep(1) # best practice to put some time between API calls.\n",
    "    data.extend(suggestions['suggestions'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We saved the API responses in a list called `data`, and put them into a [Pandas](https://pandas.pydata.org/) DataFrame to analyze."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>suggType</th>\n",
       "      <th>type</th>\n",
       "      <th>value</th>\n",
       "      <th>refTag</th>\n",
       "      <th>candidateSources</th>\n",
       "      <th>strategyId</th>\n",
       "      <th>prior</th>\n",
       "      <th>ghost</th>\n",
       "      <th>help</th>\n",
       "      <th>queryUnderstandingFeatures</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>KeywordSuggestion</td>\n",
       "      <td>KEYWORD</td>\n",
       "      <td>asmanex twisthaler 30 inhaler</td>\n",
       "      <td>nb_sb_ss_i_5_1</td>\n",
       "      <td>local</td>\n",
       "      <td>organic</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'source': 'QU_TOOL', 'annotations': []}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>KeywordSuggestion</td>\n",
       "      <td>KEYWORD</td>\n",
       "      <td>bathroom organizer</td>\n",
       "      <td>nb_sb_ss_i_4_1</td>\n",
       "      <td>local</td>\n",
       "      <td>organic</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'source': 'QU_TOOL', 'annotations': []}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>KeywordSuggestion</td>\n",
       "      <td>KEYWORD</td>\n",
       "      <td>baby wipes</td>\n",
       "      <td>nb_sb_ss_i_10_1</td>\n",
       "      <td>local</td>\n",
       "      <td>organic</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'source': 'QU_TOOL', 'annotations': []}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>KeywordSuggestion</td>\n",
       "      <td>KEYWORD</td>\n",
       "      <td>baby registry search</td>\n",
       "      <td>nb_sb_ss_i_3_1</td>\n",
       "      <td>local</td>\n",
       "      <td>organic</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'source': 'QU_TOOL', 'annotations': []}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>KeywordSuggestion</td>\n",
       "      <td>KEYWORD</td>\n",
       "      <td>b013xkha4m b08xzrxczm b07xxphqzk b09rwjblc7</td>\n",
       "      <td>nb_sb_ss_i_7_1</td>\n",
       "      <td>local</td>\n",
       "      <td>organic</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>[{'source': 'QU_TOOL', 'annotations': []}]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             suggType     type                                        value  \\\n",
       "4   KeywordSuggestion  KEYWORD                asmanex twisthaler 30 inhaler   \n",
       "13  KeywordSuggestion  KEYWORD                           bathroom organizer   \n",
       "19  KeywordSuggestion  KEYWORD                                   baby wipes   \n",
       "12  KeywordSuggestion  KEYWORD                         baby registry search   \n",
       "16  KeywordSuggestion  KEYWORD  b013xkha4m b08xzrxczm b07xxphqzk b09rwjblc7   \n",
       "\n",
       "             refTag candidateSources strategyId  prior  ghost   help  \\\n",
       "4    nb_sb_ss_i_5_1            local    organic    0.0  False  False   \n",
       "13   nb_sb_ss_i_4_1            local    organic    0.0  False  False   \n",
       "19  nb_sb_ss_i_10_1            local    organic    0.0  False  False   \n",
       "12   nb_sb_ss_i_3_1            local    organic    0.0  False  False   \n",
       "16   nb_sb_ss_i_7_1            local    organic    0.0  False  False   \n",
       "\n",
       "                    queryUnderstandingFeatures  \n",
       "4   [{'source': 'QU_TOOL', 'annotations': []}]  \n",
       "13  [{'source': 'QU_TOOL', 'annotations': []}]  \n",
       "19  [{'source': 'QU_TOOL', 'annotations': []}]  \n",
       "12  [{'source': 'QU_TOOL', 'annotations': []}]  \n",
       "16  [{'source': 'QU_TOOL', 'annotations': []}]  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(data)\n",
    "\n",
    "# show 5 random auto suggestions\n",
    "df.sample(5, random_state=303)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look at the columns, you might be flooded with more questions:\n",
    "\n",
    "- Some terms may be `blackListed`, what does that mean and what words, if any, are `blackListed = True`?<br>\n",
    "- Are some searches paid for, and not `organic`?<br>\n",
    "- What is `ghost`?<br>\n",
    "\n",
    "This metadata is only visible from the API, and can lead to new story ideas and directions to pursue. \n",
    "\n",
    "Unfortunately, because this API is undocumented, asking these questions and figuring out what everything represents is difficult. Use your curiosity and look at many examples. The feature of the API is being able to make many queries at scale, which should help answer these questions. Reporting this out with sources is also essential in this process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Do it yourself\n",
    "Find an API in the wild, isolate it, strip it down, reverse-engineer it and analyze some of its results.\n",
    "\n",
    "If a website has a search bar or a text box that queries a server or database, there's a good chance that you can find an API.\n",
    "\n",
    "Revisit the steps we outlined above, and apply them to a new website.\n",
    "If you aren't a coder, try to get steps 1-6 (I believe in you!).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are a coder, try some of the advanced usage below.\n",
    "\n",
    "### For advanced usage...\n",
    "- Handle errors for bad requests, rate limiting, and other issues that could arise.<br>\n",
    "- Restructure the API response to better analyze (called \"parsing\" the data).<br>\n",
    "- you can use `session` instead of pure requests. This is helpful if cookies and authentication are involved. Read more about that [here](https://requests.readthedocs.io/en/latest/user/advanced/#session-objects).<br>\n",
    "- you can make a request asynchronous to speed up data collection (without overloading the site's servers, of course).<br>\n",
    "- Implement steps 6-onwards in another programming language.<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predetermined prompts\n",
    "Don't know where to look? Here are some ideas:\n",
    "\n",
    "- YouTube recommendations.<br>\n",
    "- [Blacklight](https://themarkup.org/blacklight)'s API to find third-party trackers on a website.<br>\n",
    "- Amtrak's train statuses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework assignment\n",
    "\n",
    "**Practice**: Keep trying to find APIs in the wild. Think about the websites you frequent, topics that interest you, or stories you're currently working on. You won't always find an API, and that's OK.\n",
    "\n",
    "**Scoping**: Determine how the API could be used to produce data to answer a reporting question or hypothesis. What will be your sample for a quick test, i.e. how many data points are enough to know if you have something? \n",
    "\n",
    "**Reporting**: Determine the meaning and significance of hidden fields that are returned.\n",
    "\n",
    "Ultimately APIs are a tool, and data is useless without a purpose. Hopefully this worksheet helps you in your time of need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Related readings\n",
    "More tutorials on the same subject:\n",
    "\n",
    "- [\"Scraping XHR\"](https://scrapism.lav.io/scraping-xhr/) - Sam Lavigne<br>\n",
    "- [\"Web Scraping 201: finding the API\"](https://www.gregreda.com/2015/02/15/web-scraping-finding-the-api/) - Greg Reda<br>\n",
    "- [\"How to use undocumented web APIs\"](https://jvns.ca/blog/2022/03/10/how-to-use-undocumented-web-apis/) - Julia Evans\n",
    "\n",
    "Topical and timeless:\n",
    "\n",
    "- [\"Computational research in the post-API age\"](https://dfreelon.org/publications/2018_Computational_research_in_the_postAPI_age.pdf) - Deen Freelon\n",
    "\n",
    "Notable investigations and audits using undocumented APIs:\n",
    "\n",
    "- [\"Ring’s Hidden Data Let Us Map Amazon's Sprawling Home Surveillance Network\"](https://gizmodo.com/ring-s-hidden-data-let-us-map-amazons-sprawling-home-su-1840312279) - Dell Cameron and Dhruv Mehrota<br>\n",
    "- \"[Porch piracy: are we overracting to package thefts from doorsteps?](https://www.theguardian.com/us-news/2022/aug/25/porch-piracy-package-thefts-doorstep-delivery)\" - Lam Thuy Vo<br>\n",
    "- \"[The Cop in Your Neighbor's Doorbell](https://site.dcalacci.net/papers/ring-cscw-2021.pdf)\" - Dan Calacci et al.\n",
    "- \"[Analyzing gender inequality through large-scale Facebook advertising data](https://www.pnas.org/doi/full/10.1073/pnas.1717781115)\" - David Garcia et al.<br>\n",
    "- \"[Freeing the Plum Book](https://source.opennews.org/articles/freeing-plum-book/)\" - Derek Willis<br>\n",
    "\n",
    "Please reach out with more examples to add."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Artifacts\n",
    "Slides from workshops can be found here:\n",
    "\n",
    "[2023-02-24 @ Tow Center Columbia](https://docs.google.com/presentation/d/1e1QoSNXv2m90lhhyUMSzUlMxXJD_Ar5DtC43PcpTcWU)<br>\n",
    "[2023-03-04 @ NICAR](https://docs.google.com/presentation/d/1hWMqcBNfs9BbaVywMGJPf_BcR9PpVLlHPCGBsAzz-No/)<br>\n",
    "[2023-06-12 @ FAccT](https://docs.google.com/presentation/d/1-9rODLyxJawasNIn_rp9E_oHUOB1SUGCpAb8TbmyxKU/edit?usp=sharing)<br>\n",
    "[2023-06-14 @ Journocoders](https://paper.dropbox.com/doc/Journocoders-June-2023-1t7nrjYNWoiPhK0sx8rH0)<br>\n",
    "[2023-06-22 @ C+J DATAJ](https://docs.google.com/presentation/d/10_mWNwr_fsrX0r8e6xWhFruhdUZUbLEA9HqLI3HtEFg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Citation\n",
    "\n",
    "To cite this chapter, please use the following BibTex entry:\n",
    "\n",
    "<pre>\n",
    "@incollection{inspect2023browser,\n",
    "  author    = {Yin, Leon},\n",
    "  title     = {Finding Undocumented APIs},\n",
    "  booktitle = {Inspect Element: The practitioner's guide to hypothesis-driven data investigations},\n",
    "  year      = {2023},\n",
    "  editor    = {Yin, Leon and Sapiezynski, Piotr},\n",
    "  note      = {\\url{https://inspectelement.org}}\n",
    "}\n",
    "</pre>\n",
    "\n",
    "## Acknowledgements\n",
    "\n",
    "Thank you Max Harlow for suggestions in making the tutorial easier to understand, and a well-compressed thank you to Simon Fondrie-Teitler for helping optimize this page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}