Create robots.txt

[odow]

Here are the subdirectories: https://github.com/jump-dev/JuMP.jl/tree/gh-pages

If I understand robots.txt correctly, it seems we really just want the stable builds being indexed?

x-ref: jump-dev/JuMP.jl#2606 (comment)

[mlubin]

We usually use versioned links, right? If links to stable docs don't appear anywhere, then will it be hard to get them to appear in searches? (This is just speculation, not based on any insider knowledge.)

[odow]

This is just speculation, not based on any insider knowledge.

I thought PageRank was old news, but that could be a concern. Although the alternative is to constantly have to build up new links to x.y.z to ensure the most recent version stays popular which is also hard?

It seems better to focus our attention into a single, long-term documentation link like https://jump.dev/JuMP.jl/stable, rather than splitting it into version-specific ones, each with less traffic.

[odow]

Did some digging on the Documenter side.

It seems to be just use robots.txt, or we could use noindex tag to force remove the old documentation:
JuliaDocs/Documenter.jl#1133
JuliaDocs/Documenter.jl#1302

I see there's also this very recent PR which adds a warning to the out-dated docs which would also help:
JuliaDocs/Documenter.jl#1577

[mlubin]

It seems better to focus our attention into a single, long-term documentation link like https://jump.dev/JuMP.jl/stable, rather than splitting it into version-specific ones, each with less traffic.

https://jump.dev/JuMP.jl/stable is fine, but links to subsections are labels within "stable" will just break over time, which is worse than not showing up in search results.

The warning banner doesn't address the issue of search results. We had banners on old JuMP docs for a long time, and that didn't prevent them from ranking high in search results.

I'm not familiar with sitemaps but that seems potentially useful.

[odow]

The warning banner doesn't address the issue of search results.

No. But at least it would reduce the confusion of things like jump-dev/JuMP.jl#2606 (comment)

From https://developers.google.com/search/docs/advanced/robots/intro

So we should continue linking to specific versions.

It's also an argument for jump-dev/JuMP.jl#2522. Then we don't have a problem linking between JuMP and MOI.

I'm not familiar with sitemaps but that seems potentially useful.

JuliaDocs/Documenter.jl#1135

[odow]

Bump. Any more thoughts?