Code

Scrapix V2#101

Code

Open

qdequele wants to merge 37 commits intomainmeilisearch/scrapix:mainfrom new-crawlersmeilisearch/scrapix:new-crawlersCopy head branch name to clipboard

+44,265-14,629

Commits

Commits on Sep 22, 2024

• change to cheerio for scraping, keep puppeteer for crawling.

  qdequele

  committed
  e49c556

  Copy full SHA for e49c556

Commits on Oct 12, 2024

• big update

  qdequele

  committed
  5f461e7

  Copy full SHA for 5f461e7

• merge the maximum code of the crawlers together

  qdequele

  committed
  aca8cb1

  Copy full SHA for aca8cb1

Commits on Oct 13, 2024

• another big commit.

  qdequele

  committed
  896e6c1

  Copy full SHA for 896e6c1

• add markdown scraper

  qdequele

  committed
  45a203f

  Copy full SHA for 45a203f

• add custom scraper

  qdequele

  committed
  6195809

  Copy full SHA for 6195809

Commits on Oct 19, 2024

• remove startCrawl; comment playwright

  qdequele

  committed
  60f2416

  Copy full SHA for 60f2416

Commits on Nov 8, 2024

• fix #99

  qdequele

  committed
  b3db1bb

  Copy full SHA for b3db1bb

• update packages

  qdequele

  committed
  550a4ab

  Copy full SHA for 550a4ab

• fix #56: Throw error when redis server is not answering

  qdequele

  committed
  6e17d28

  Copy full SHA for 6e17d28

• fix #48: add the automatic detection of 404 pages to skip with the possibility to add custom selector for 404 pages.

  qdequele

  committed
  a79061b

  Copy full SHA for a79061b

Commits on Nov 9, 2024

• By default use cheerio instead of Puppeteer #113

  qdequele

  committed
  536326a

  Copy full SHA for 536326a

• fix #112: Remove the useless headless option

  qdequele

  committed
  ad970e6

  Copy full SHA for ad970e6

• remove launcher_option and launcher

  qdequele

  committed
  49324fc

  Copy full SHA for 49324fc

• Update Documentation

  qdequele

  committed
  c2ba9bf

  Copy full SHA for c2ba9bf

Commits on Nov 10, 2024

• fix #103: Keep the previous settings

  qdequele

  committed
  a43ee93

  Copy full SHA for a43ee93

• fix #102: Load the sitemap as starter point for crawling.

  qdequele

  committed
  f5e9944

  Copy full SHA for f5e9944

Commits on Nov 15, 2024

• add a new playground

  qdequele

  committed
  8e1adef

  Copy full SHA for 8e1adef

• extract sitemap

  qdequele

  committed
  62b8dce

  Copy full SHA for 62b8dce

• add pdf scraper

  qdequele

  committed
  2105cc7

  Copy full SHA for 2105cc7

• Update testing

  qdequele

  committed
  81be6b4

  Copy full SHA for 81be6b4

Commits on Nov 22, 2024

• add the pdfs on the playground

  qdequele

  committed
  27db8e4

  Copy full SHA for 27db8e4

• add a lot of pages

  qdequele

  committed
  abe215f

  Copy full SHA for abe215f

Commits on Nov 30, 2024

• make docker works for playground, scrapix and meilisearch

  qdequele

  committed
  ec4dd44

  Copy full SHA for ec4dd44

• full working base with zod #35

  qdequele

  committed
  a836dd8

  Copy full SHA for a836dd8

• fix tests

  qdequele

  committed
  7837b39

  Copy full SHA for 7837b39

• update github CI

  qdequele

  committed
  1234229

  Copy full SHA for 1234229

• update test CI

  qdequele

  committed
  713a6a6

  Copy full SHA for 713a6a6

• Update Node.js version in GitHub Actions workflow from 18 to 20

  qdequele

  committed
  55a3d0e

  Copy full SHA for 55a3d0e

• Make wait-for-it.sh executable in GitHub Actions workflow

  qdequele

  committed
  8d73bbb

  Copy full SHA for 8d73bbb

• Remove deprecated configuration files for previous tests

  qdequele

  committed
  36bde9e

  Copy full SHA for 36bde9e

• Refactor BaseTest and ScraperTestHelper to streamline index UID handling

  Show description for 6ec776c
  qdequele

  committed
  6ec776c

  Copy full SHA for 6ec776c

Commits on Jan 20, 2025

• use start_urls as crawling pages

  qdequele

  committed
  f1a394b

  Copy full SHA for f1a394b

• remove unecessary pagination detection leading to avoid scraping intersting pages

  qdequele

  committed
  9c1192f

  Copy full SHA for 9c1192f

• update and simplify scraper

  qdequele

  committed
  d88def5

  Copy full SHA for d88def5

Commits on Feb 4, 2025

• Improve Meilisearch index settings handling and remove debug logging

  qdequele

  committed
  a90004f

  Copy full SHA for a90004f

Commits on Feb 11, 2025

• Fix #115

  Show description for 70024bb
  CaroFG

  committed
  70024bb

  Copy full SHA for 70024bb