Skip to content

Commit

Permalink
fix(seo): remove contributors.txt from robots.txt (#11311)
Browse files Browse the repository at this point in the history
We added */contributors.txt to our robots.txt with the intention 
of hiding those pages from Google, but it seems to prevent Google 
from seeing the X-Robots-Tag header, keeping the pages in the index,
although we would like to see them removed from the index.
  • Loading branch information
caugner authored Jun 14, 2024
1 parent 1609a74 commit 1e561bb
Showing 1 changed file with 0 additions and 1 deletion.
1 change: 0 additions & 1 deletion tool/build-robots-txt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ User-agent: *
Sitemap: https://developer.mozilla.org/sitemap.xml
Disallow: /api/
Disallow: /*/contributors.txt
Disallow: /*/files/
Disallow: /media
`;
Expand Down

0 comments on commit 1e561bb

Please sign in to comment.