Code

38 scrape tripadvisor airline reviews#41

Code

Merged

algo7 merged 71 commits intomainalgo7/TripAdvisor-Review-Scraper:mainfrom 38-scrape-tripadvisor-airline-reviewsalgo7/TripAdvisor-Review-Scraper:38-scrape-tripadvisor-airline-reviewsCopy head branch name to clipboardOct 5, 2023

+748-156

Commits

Commits on Oct 4, 2023

• ad airline.js

  algo7

  committed
  055f49b

  Copy full SHA for 055f49b

• add logging level and container_provisioner env var

  algo7

  committed
  06999cc

  Copy full SHA for 06999cc

• add dataSourceAirline var to app.js

  algo7

  committed
  fe5543f

  Copy full SHA for fe5543f

• change data source path in airline scraper init func

  algo7

  committed
  34446a3

  Copy full SHA for 34446a3

• add extractAllReviewPageUrls func template to airline.js

  algo7

  committed
  8332352

  Copy full SHA for 8332352

• rename input vars forairline page url extraction

  algo7

  committed
  3300d18

  Copy full SHA for 3300d18

• change rev per page

  algo7

  committed
  9ad1152

  Copy full SHA for 9ad1152

• fix denominator value for page calculation

  algo7

  committed
  1fb4156

  Copy full SHA for 1fb4156

• change review page url forming logic

  algo7

  committed
  a7bdbad

  Copy full SHA for a7bdbad

• change url var name

  algo7

  committed
  961cb01

  Copy full SHA for 961cb01

• add scrape function template to airline.js

  algo7

  committed
  e00a86c

  Copy full SHA for e00a86c

• comment title block selection logic

  algo7

  committed
  4442acf

  Copy full SHA for 4442acf

• change comment rating extraction logic

  algo7

  committed
  139ea6c

  Copy full SHA for 139ea6c

• remove commented block

  algo7

  committed
  4bab453

  Copy full SHA for 4bab453

• import monthStringToNumber, commentRatingStringToNumber to airlinejs

  algo7

  committed
  bd0769d

  Copy full SHA for bd0769d

• change monthStringToNumber function to account for recent review date

  algo7

  committed
  ff77a89

  Copy full SHA for ff77a89

• add logic to handle diff review date formatting

  algo7

  committed
  ff8aef0

  Copy full SHA for ff8aef0

• handle expandble reviews

  algo7

  committed
  03876c2

  Copy full SHA for 03876c2

• refactor clicking action on expandable review

  algo7

  committed
  3f959a0

  Copy full SHA for 3f959a0

• fix typo

  algo7

  committed
  2bf7b11

  Copy full SHA for 2bf7b11

• format return data from the scrape function

  algo7

  committed
  05591f2

  Copy full SHA for 05591f2

• fix var name, comment desc, add start func to airline.js

  algo7

  committed
  883271e

  Copy full SHA for 883271e

• change func desc for airlineScraperInit

  algo7

  committed
  4350ce9

  Copy full SHA for 4350ce9

• add AIRLINE as a scrape mode to the init func

  algo7

  committed
  c492d55

  Copy full SHA for c492d55

• let user be able to start airline scraper by passing args via env var

  algo7

  committed
  6cebf58

  Copy full SHA for 6cebf58

• fix airline scraper rawData env var

  algo7

  committed
  b0d42d9

  Copy full SHA for b0d42d9

• fix args pass to resto scraper

  algo7

  committed
  8f3cfdc

  Copy full SHA for 8f3cfdc

• fix scraper name

  algo7

  committed
  2f4512b

  Copy full SHA for 2f4512b

• change browser arg type in func desc

  algo7

  committed
  e25cbc7

  Copy full SHA for e25cbc7

• remove lang arg being passed to airline scraper

  algo7

  committed
  8f0f732

  Copy full SHA for 8f0f732

• use optional channing for determing if the review is of the current month or not

  algo7

  committed
  348a6d3

  Copy full SHA for 348a6d3

• fix currentTime out of scope

  algo7

  committed
  02f484c

  Copy full SHA for 02f484c

• fix logging

  algo7

  committed
  85fc680

  Copy full SHA for 85fc680

• add timeout to wait for the review urls to load

  algo7

  committed
  a8af9a4

  Copy full SHA for a8af9a4

• potential fix for concurrency is useless

  algo7

  committed
  d29f5a9

  Copy full SHA for d29f5a9

• fix page handback logic in airline.js

  algo7

  committed
  6a30e84

  Copy full SHA for 6a30e84

• fix page handback logic for both hotel and resto scraper

  algo7

  committed
  3bcb194

  Copy full SHA for 3bcb194

• remove unused console logs

  algo7

  committed
  d1a80ad

  Copy full SHA for d1a80ad

• set browser instance to be headless

  algo7

  committed
  d8e8739

  Copy full SHA for d8e8739

• add missing return to the hotel scraper

  algo7

  committed
  21f849a

  Copy full SHA for 21f849a

• use event driven approach for the waiting logic in getNewPage

  algo7

  committed
  4eea3b2

  Copy full SHA for 4eea3b2

• refactor browser.handBack logic

  algo7

  committed
  7bb1823

  Copy full SHA for 7bb1823

• headless default for browser

  algo7

  committed
  6d9527b

  Copy full SHA for 6d9527b

• fix wrong data source var for airline scraper init func

  algo7

  committed
  6b07ec3

  Copy full SHA for 6b07ec3

Commits on Oct 5, 2023

• add scrape option dropdown to the UI

  algo7

  committed
  976cbe7

  Copy full SHA for 976cbe7

• add form selection parsing logic

  algo7

  committed
  19ba0be

  Copy full SHA for 19ba0be

• add regex pattern for restos and airlines. compile regex and pkg level

  algo7

  committed
  8b8a9b5

  Copy full SHA for 8b8a9b5

• refactor ValidateTripAdvisorURL to account for diff scrape options

  algo7

  committed
  d0cca94

  Copy full SHA for d0cca94

• add backend logic to validate scrape target

  algo7

  committed
  0ffea0f

  Copy full SHA for 0ffea0f

• refactor get scrape target name func

  algo7

  committed
  fba9213

  Copy full SHA for fba9213

• add ContainerConfigGenerator

  algo7

  committed
  1d0ac31

  Copy full SHA for 1d0ac31

• refactor CreateContainer func

  algo7

  committed
  6f953fc

  Copy full SHA for 6f953fc

• refactor Scrape function

  algo7

  committed
  e870fef

  Copy full SHA for e870fef

• refactor ListContainers

  algo7

  committed
  e6e4a86

  Copy full SHA for e6e4a86

• refactor

  algo7

  committed
  b4a6f54

  Copy full SHA for b4a6f54

• refactor tasks.html

  algo7

  committed
  74587ff

  Copy full SHA for 74587ff

• refac restoScraperInit to take in env var

  algo7

  committed
  7d98785

  Copy full SHA for 7d98785

• formatting

  algo7

  committed
  038d400

  Copy full SHA for 038d400

• fix ValidateTripAdvisorURL switch statement

  algo7

  committed
  188d8b5

  Copy full SHA for 188d8b5

• refactor ListContainers function

  algo7

  committed
  94c78b7

  Copy full SHA for 94c78b7

• refactor list container func and remove running tasks struct

  algo7

  committed
  dad1b68

  Copy full SHA for dad1b68

• refactor getRunningTasks func

  algo7

  committed
  2a10e29

  Copy full SHA for 2a10e29

• remove debug logging

  algo7

  committed
  ab8ddb3

  Copy full SHA for ab8ddb3

• refactor logging logic for resto scraper

  algo7

  committed
  06e061a

  Copy full SHA for 06e061a

• add exit logic to hotel and resto scraper init func

  algo7

  committed
  6c0c7a2

  Copy full SHA for 6c0c7a2

• add logging

  algo7

  committed
  c480dbe

  Copy full SHA for c480dbe

• remove CountRunningContainer func

  algo7

  committed
  eb84183

  Copy full SHA for eb84183

• fix resto logging logic

  algo7

  committed
  2bd6a9e

  Copy full SHA for 2bd6a9e

• change invalid url msg

  algo7

  committed
  d994f6c

  Copy full SHA for d994f6c

• change main.html url bar text

  algo7

  committed
  c75af07

  Copy full SHA for c75af07

• add IP ban warning

  algo7

  committed
  5fa4b18

  Copy full SHA for 5fa4b18