Add robots.txt

[frvge]

Currently, the robots.txt gives a 404 page.

Request: create a robots.txt

[Nexuist]

I can take this! I'll include a generic policy that enables crawlers unless there is anything you guys want to specifically block out.

[molant]

@Nexuist that sounds perfect, thanks a lot!

[molant]

Hi @Nexuist,

Just in case, you are looking into adding the robots.txt file into https://github.com/webhintio/webhint.io/tree/master/src/webhint-theme/source/static.

npm run build should compress that file and add it into the dist folder, but you might want to double check that.

Thanks!