feat(diagnostics) add diff endpoints

[rainest]

What this PR does / why we need it:

Adds a diff endpoint to the diagnostic server and sets up infrastructure for DB mode updates to ship their diffs to the diagnostic server. This endpoint is available when sensitive config dumps are enabled, as it includes diffs of sensitive resources. We could add filters to exclude certs/passwords/etc. while retaining other items, but for the initial MVP, easiest to just require an opt in to sensitive diagnostic output.

Which issue this PR fixes:

Fix #5289

Special notes for your reviewer:

This requires Kong/go-database-reconciler#119. My original work in GDR was only sending error events; it should have been sending everything. This currently has a tmp commit against to update the module to the branch. That GDR PR needs to merged and tagged as fix release before merging this.

Per Kong/go-database-reconciler#120 I observed an odd bug with mysterious empty events 👻 Dunno where these are from, but trying to figure that out will take more time than is available. As a workaround, the event handler on the KIC side discards any event with no action, as such events should never be of interest.

Manual testing via running TEST_DATABASE_MODE=postgres KONG_TEST_CLUSTER=kind:kind GOFLAGS= dlv test -- -test.run SomeTestThatModifiesConfig. Almost any test will do, since they almost all modify config. You can either set up a watch curl -sv localhost:10256/debug/config/diff-report to see it over time or debugger pause after the diff send to see an individual diff. Output looks something like example.txt.

Unit testing isn't exhaustive, but covers most of the endpoint functionality. It checks the KIC-side behavior, not content, with the expectation that if content's broken, that's GDR's problem.

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[rainest]

Moving into 3.3 as a stretch. IIRC this should be 50% done, just dropped from 3.2 between time constraints and low priority.

[rainest]

Fix one bug, find another:

This has somehow broken the envtest for DB mode error event generation. That test should normally function by creating a consumer and tossing it at a mock admin API that always returns an error, which should in turn make GDR bubble up an error event through the result channel, which KIC will then turn into a Kubernetes Event.

Looking at the event handler only shows yet more of the wonderful 👻 events described in Kong/go-database-reconciler#120, though unfortunately I can't get the debugger to stop anywhere in the GDR portion, even at the low level of setting a watchpoint on the channel buffer itself. IDK what the heck's going there, since that's not something the envtest mock stuff should skip.

Tried to pin down the problem to either this or the GDR change:

• Reverting to the GDR release prior to fix: send change events always go-database-reconciler#119 in this PR does not clear this issue.
• Running envtests against current main plus GDR v1.14.3 passes.

so whatever causes it appears to be in this PR's new code, but IDK what. The new skip phantom events case isn't on the error handling side and shouldn't affect it, and git bisect isn't giving me good clues ☹️

[codecov]

Codecov Report

Attention: Patch coverage is 81.06509% with 32 lines in your changes missing coverage. Please review.

Project coverage is 77.7%. Comparing base (f343327) to head (2622177).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/diagnostics/server.go	64.3%	22 Missing and 4 partials ⚠️
internal/diagnostics/diff.go	88.6%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##            main   #6131    +/-   ##
======================================
  Coverage   77.7%   77.7%            
======================================
  Files        207     208     +1     
  Lines      24681   24833   +152     
======================================
+ Hits       19181   19314   +133     
- Misses      4523    4537    +14     
- Partials     977     982     +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pmalek]

@randmonkey Made the change to remove the dependency on https://github.com/golang-collections/collections ( which is an 11 year old package) and used containers/list instead.

Also added some UTs.

PTAL b90c664. Feel free to merge this if you feel this looks good now.

[randmonkey]

Code scanning results / CodeQL has been queued for 40+ mins. Let me close and reopen it to see if it fixed.