Create custom scraper

[bidoubiwa]

Currently there are 3 strategies

• default
• docssearch
• schema

They all answer to a certain common use case. We suppose these use-cases answer 90% of the website structures.
Nonetheless, other app may not follow that structure or the user might want to only index a specific part of each page and not all of it.

For these cases, we want to create the possibility to provide a custom scraper.

A base to create the custom scraper was made but removed

scrapix/src/scrapers/custom.js

Lines 1 to 45 in 966675f

	import prettier from "prettier";
	import { v4 as uuidv4 } from "uuid";
	export default class CustomScaper {
	constructor(sender, config) {
	console.info("CustomScaper::constructor");
	this.sender = sender;
	this.config = config;
	if (config.custom_settings) {
	this.sender.updateSettings(config.custom_settings);
	}
	}
	async get(url, page) {
	let data = {};
	if (this.custom_crawler.get_title || false) {
	data.title = await page.title();
	}
	data.uid = uuidv4();
	if (this.custom_crawler.get_meta || false) {
	const meta = await page.evaluate(() => {
	const metas = document.getElementsByTagName("meta");
	const meta = {};
	for (let i = 0; i < metas.length; i++) {
	const name = metas[i].getAttribute("name");
	const content = metas[i].getAttribute("content");
	if (name && content) {
	meta[name] = content;
	}
	}
	return meta;
	});
	data.meta = meta;
	}
	if (this.custom_crawler.get_url || false) {
	data.url = url;
	}
	await this.sender.add(data);
	}
	}

Nonetheless, it has been removed until we deem this feature necessary