[feature]: Implement customizable cache control for web fee estimator

[mrfelton]

Problem:

LND's web fee estimator (configured by setting feeurl) uses a hardcoded refresh schedule (between 5-20 minutes) for fetching fee information. This can lead to inaccurate fee estimations due to rapid changes in mempool conditions, creating the potential for priority transactions to miss the next block, and various underpayment or overpayment scenarios.

Desired solution:

Implement customizable cache control for the web fee estimator, allowing users to set higher refresh frequencies for more timely fee updates.

Alternatives considered:

Manually handling fee estimation outside LND: This only works for specific use cases and duplicates effort, not ideal for integrating with LND's built-in features.

Additional context:

https://strike.me/blog/blended-bitcoin-fee-estimations/ (explains how the web fee estimator can be used to incorporate mempool-based fee estimation into lnd to improve estimates for priority transactions. This could be more effective with more frequent rate refreshes from lnd).

[alexbosworth]

Could use cache control headers, this is something that is part of HTTP to handle caching schedules

[mrfelton]

Is there any specific reason why an internal caching system was implemented in the first place rather than honouring standard http cache control headers?

The two such public services that I know of (https://nodes.lightning.computer/fees/v1/btc-fee-estimates.json and https://bitcoinchainfees.strike.me/) do already implement cache control headers.

However, relying on cache control headers alone would put the responsibility of defining caching schedules on the developers/maintainers of web fee url endpoints. And, switching from the current internally defined refresh schedule to one that is derived from cache control headers would result in a potentially damaging change in behaviour for anyone using a fee estimator endpoint that doesn't already implement cache control.

[alexbosworth]

The original system anticipated a data source like Bitcoin Core which doesn't update until a new block is mined so that may help to explain some of the thinking there. I'm not sure why the client doesn't use the HTTP header information to inform its caching

Cache control is not only used by HTTP servers themselves it is also useful for CDNs and other proxies. I don't remember exactly what happened with the implementation but I vaguely remember a reason caching was added was also to prevent server hammering

[saubyk]

closed with #8719