Skip to content

Commit

Permalink
Updated robots.txt and fixed small issue on image
Browse files Browse the repository at this point in the history
  • Loading branch information
DynTylluan committed Oct 24, 2023
1 parent 561f6eb commit 7f93ea9
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 1 deletion.
Binary file modified nb21/newsletter.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
23 changes: 22 additions & 1 deletion robots.txt
Original file line number Diff line number Diff line change
Expand Up @@ -43,4 +43,25 @@ Disallow: /
# Kitty
# This bot is cool - go wild, queen
User-Agent: digikitty_x86
Allow: /
Allow: /

# This is where I copy Bytemoth
# I am taking it in good faith that Bytemoth knows what he's doing and that
# the following bots are for marketers - so I am blocking them.
# As taken from the following URL: http://bytemoth.nfshost.com/robots.txt
User-agent: adsbot
User-agent: AhrefsBot
User-agent: BLEXBot
User-agent: dotbot
User agent: MTRobot
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: SemrushBot-BA
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: SemrushBot-CT
User-agent: SemrushBot-BM
User-agent: serpstatbot
User-agent: Pandalytics
User-agent: PageThing
Disallow: /

0 comments on commit 7f93ea9

Please sign in to comment.