Code

Fix/robots.txt parsing#708

Code

Open

flancast90 wants to merge 5 commits intounclecode:mainunclecode/crawl4ai:mainfrom flancast90:fix/robots.txt-parsingflancast90/crawl4ai:fix/robots.txt-parsingCopy head branch name to clipboard

+128-62

Commits

Commits on Feb 17, 2025

• fix: typo in test file name for parsing

  flancast90

  committed
  9a3ae2b

  Copy full SHA for 9a3ae2b

• fix: remove old, improperly named test file

  flancast90

  committed
  16fabac

  Copy full SHA for 16fabac

• feat: move off of RobotFileParser too newer (Google) spec

  flancast90

  committed
  334a7b1

  Copy full SHA for 334a7b1

• feat: Add close() method to fix parallel connection errors specifiacally on Windows

  flancast90

  committed
  f11dfca

  Copy full SHA for f11dfca

• fix: improve SQLite cleanup in robots.txt parser tests

  flancast90

  committed
  109b7b3

  Copy full SHA for 109b7b3