doc: add detail to confmap watcher

[mattsains]

Description

I tried to improve the docs on watchers based on the issue below

Link to tracking issue

Fixes #12115

Documentation

Clarified some code comments that were incorrect (there's no Retrieved.Get method) and included a new code sample for a Provider that uses the watcher func

[mattsains]

I can write a CHANGELOG entry for this, but I don't think it's helpful. Let me know if I should anyway to get the CI to pass

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.84%. Comparing base (b09a65b) to head (cf2d8cf).
Report is 44 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12150      +/-   ##
==========================================
+ Coverage   91.73%   91.84%   +0.11%     
==========================================
  Files         463      465       +2     
  Lines       24819    25325     +506     
==========================================
+ Hits        22767    23261     +494     
- Misses       1670     1675       +5     
- Partials      382      389       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mx-psi]

@bogdandrutu Could you double check this since you designed this in the first place?

[mx-psi]

Thinking about this, I think this introduces the risk of flakes on CI, what about

1. defining a channel ack := make(chan struct{})
2. On watcherFunc, sending a struct{}{} to ack
3. Replacing this sleep with a wait on ack

We can probably replace the second sleep with a similar pattern.

What do you think?

[mattsains]

I replaced the first sleep with a channel (the one waiting for a notification of a new config). However, keeping in mind that this is destined to be documentation and not test case, I couldn't find a natural way to notify when the retriever has unsubscribed the watchFunc. Any way I tried to do it compromised the structure of the Provider in a way that I think would detract from the documentational value. For example:

• I considered returning a channel as the rawConf return, but that doesn't make sense because the config is never going to be a channel in normal uses.
• I considered cancelling the context of the Retrieve() method call, but that doesn't make sense because you usually wouldn't do that in a Provider.
• I also considered creating a channel as a public instance member of the Provider, but again that is not didactically sound because a single Retrieve call has a different duration/lifetime than the entire Provider instance.

I do agree that the sleeps introduce a chance of a race in the tests. I've removed the test for the unsubscribe of the watcher. If you'd still like that tested, I'm just not sure how to do that without making it more like a confusing test and less like documentation

[mx-psi]

Thank you for taking a look at this, and sorry that this slipped on the first review. I think this looks good to merge now, if we get feedback that this is confusing or the 'unsubscribe watcher' would be helpful, we can add it.