robots: also disallow %5E (urlencoded caret)

[jsha]

Part of #1438
Followup to #2695

https://www.ietf.org/rfc/rfc1738.txt says:

Other characters are unsafe because
gateways and other transport agents are known to sometimes modify
such characters. These characters are "{", "}", "|", "", "^", "~",
"[", "]", and "`".

All unsafe characters must always be encoded within a URL.

https://datatracker.ietf.org/doc/html/rfc3986, which updates it, doesn't mention ^. 🤷🏻

[syphar]

@jsha code is deployed.

Just a thought in retrospect: since ~ is mentioned here too, should we also add the encoded variant for it?

[jsha]

From the search console I can see that the tilde urls are successfully getting blocked. Which suggests Google is not urlencoding the tilde.