docs: add robots.txt for sitemap (withastro#2083)

Co-authored-by: Chris Swithinbank <swithinbank@gmail.com>
morinokami · Aug 6, 2024 · 7ad7113 · 7ad7113
1 parent 41506e5
commit 7ad7113
Show file tree

Hide file tree

Showing 2 changed files with 6 additions and 0 deletions.
diff --git a/docs/src/assets/robots.txt b/docs/src/assets/robots.txt
@@ -0,0 +1,4 @@
+User-agent: *
+Allow: /
+
+Sitemap: https://starlight.astro.build/sitemap-index.xml
diff --git a/docs/src/content/docs/guides/customization.mdx b/docs/src/content/docs/guides/customization.mdx
@@ -107,6 +107,8 @@ export default defineConfig({
 });
 ```
 
+Learn how to [add a sitemap link to `robots.txt`](https://docs.astro.build/en/guides/integrations-guide/sitemap/#sitemap-link-in-robotstxt) in the Astro Docs.
+
 ## Page layout
 
 By default, Starlight pages use a layout with a global navigation sidebar and a table of contents that shows the current page headings.
-Original file line number
+Diff line change
@@ Expand Up / @@ -107,6 +107,8 @@ export default defineConfig({ @@
     });
     ```
+    Learn how to [add a sitemap link to `robots.txt`](https://docs.astro.build/en/guides/integrations-guide/sitemap/#sitemap-link-in-robotstxt) in the Astro Docs.
     ## Page layout
     By default, Starlight pages use a layout with a global navigation sidebar and a table of contents that shows the current page headings.
@@ Expand Down @@