Include useragent in requests done for link expander

[xPaw]

Something like Shout/v0.0.0 (https://github.com/erming/shout) would work.

[astorije]

Hey @xPaw, I have a few questions on this:

• What user agent is being used at the moment?
• Apart from respecting standards and specs, what would be the pragmatic benefits of defining and including a user agent?
• Same question about using an existing user agent and a custom user agent? Is there already one that could suit us?

All details welcome :-)

[AlMcKinlay]

I was in the irc when this was initially discussed. Correct me if I'm wrong, @xPaw.

• I believe there isn't one being used, and this was the issue?
• We noticed that cloudflare (maybe others as well) doesn't allow calls to sites without a user agent, so I believe that this means we can't get some site information for the thumbnails
• In general, it's best to use a custom user-agent, as most of the existing ones would be browsers, which we don't really want to pretend to be (actually, maybe we do? Because the request comes from the browser itself, so maybe we should use the user's browser's user agent)

[xPaw]

That's correct.