Feature request: Opt out mechanisms

[slatian]

Please make it easy for your users to implement the following opt out mechanisms:

• robots.txt
• deviantarts noai tag
• the w3m TDM protocol (see Please respect TDM Reservation Protocol #124)

With github actively recommending you and 21K+ stars you have a certain kind of responsibility. This is a matter of respect towards the people whose data your users rely on.

One line mentions in the docs I needed grep to find aren't enough.

[aravindkarnam]

This is a brilliant suggestion @slatian. We have already added robots.txt compliance in our upcoming scraper module ( it traverses and crawls the entire site instead of just a page). We'll keep in mind the support for noai tag and TDM reservation protocol while planning our future roadmap.

[mvl22]

We have already added robots.txt compliance

What is the crawler name?