Add ServiceMonitor for scraping gitops-controller in rancher-monitoring chart

[Tommy12789]

Issue:

The direct issue is here : rancher/fleet#2450

The PR that introduced metrics into fleet: rancher/fleet#2172

Problem

Enabling further additions to monitoring that are related to fleet metrics, for which reasons Prometheus needs to scrape the data of the gitops controller by creating an additional ServiceMonitor which points to the Kubernetes services created by the fleet chart, which in turn point to the gitops-controller metrics.

Solution

An additional ServiceMonitor needs to be created when the rancher-monitoring chart is installed, so that the thereby installed Prometheus instance is automatically configured to scrape the data of the gitops-controllers.

This enables further additions of monitoring capabilities to Rancher using the rancher-monitoring chart, for instance the addition of Prometheus alerts or Grafana dashboards. The latter may be embedded into Rancher, similarly as the Grafana dashboards are already embedded into Rancher and displayed through the Rancher UI when the rancher-monitoring chart is installed.

Testing

• On a cluster with Rancher and a fleet version with the changed introduced by this pr : Add ServiceMonitor for scraping gitops-controller in rancher-monitoring chart #4101, install the rancher-monitoring chart that includes the changes of this PR.
• Open the Prometheus UI, navigate to Targets and check for gitops-controller

Engineering Testing

Manual Testing

Performed as described in Testing, including testing with sharding enabled in fleet.

Automated Testing

N/A

QA Testing Considerations

N/A

Regressions Considerations

The probability of this change introducing regressions is low, as it simply extends already implemented functionality by a rather simple resource, which is part of the rancher-monitoring-crd chart.

Backporting considerations

This change does not need to be backported to other versions. This is a new feature in fleet and no plans exist to backport it.

[github-actions]

Validation steps

• Ensure all container images have repository and tag on the same level to ensure that all container images are included in rancher-images.txt which are used by airgap customers.

  Ex:-
    longhorn-controller:
      repository: rancher/hardened-sriov-cni
      tag: v2.6.3-build20230913
  

• Add a 👍 (thumbs up) reaction to this comment once done. CI won't pass without this reaction to the github-action bot's latest validation comment.
• Approve the PR to run the CI check.

[github-actions]

Validation steps

• Ensure all container images have repository and tag on the same level to ensure that all container images are included in rancher-images.txt which are used by airgap customers.

  Ex:-
    longhorn-controller:
      repository: rancher/hardened-sriov-cni
      tag: v2.6.3-build20230913
  

• Add a 👍 (thumbs up) reaction to this comment once done. CI won't pass without this reaction to the github-action bot's latest validation comment.
• Approve the PR to run the CI check.

[github-actions]

Validation steps

• Ensure all container images have repository and tag on the same level to ensure that all container images are included in rancher-images.txt which are used by airgap customers.

  Ex:-
    longhorn-controller:
      repository: rancher/hardened-sriov-cni
      tag: v2.6.3-build20230913
  

• Add a 👍 (thumbs up) reaction to this comment once done. CI won't pass without this reaction to the github-action bot's latest validation comment.
• Approve the PR to run the CI check.

[p-se]

This PR will enable scraping of metrics from Gitops controller that the performance dashboard will visualize. Fleet controller related metrics (including performance metrics) are already scraped, as a ServiceMonitor for the fleet controller already has been merged.

[github-actions]

Validation steps

• Ensure all container images have repository and tag on the same level to ensure that all container images are included in rancher-images.txt which are used by airgap customers.

  Ex:-
    longhorn-controller:
      repository: rancher/hardened-sriov-cni
      tag: v2.6.3-build20230913
  

• Add a 👍 (thumbs up) reaction to this comment once done. CI won't pass without this reaction to the github-action bot's latest validation comment.
• Approve the PR to run the CI check.

[github-actions]

Validation steps

• Ensure all container images have repository and tag on the same level to ensure that all container images are included in rancher-images.txt which are used by airgap customers.

  Ex:-
    longhorn-controller:
      repository: rancher/hardened-sriov-cni
      tag: v2.6.3-build20230913
  

• Add a 👍 (thumbs up) reaction to this comment once done. CI won't pass without this reaction to the github-action bot's latest validation comment.
• Approve the PR to run the CI check.

[github-actions]

Validation steps

• Ensure all container images have repository and tag on the same level to ensure that all container images are included in rancher-images.txt which are used by airgap customers.

  Ex:-
    longhorn-controller:
      repository: rancher/hardened-sriov-cni
      tag: v2.6.3-build20230913
  

• Add a 👍 (thumbs up) reaction to this comment once done. CI won't pass without this reaction to the github-action bot's latest validation comment.
• Approve the PR to run the CI check.

[thehejik]

Tested on v2.9-dcf32c221741b5b2d595d2c7ec2c44f5dd0c8fd2-head with rancher-monitoring:104.0.0-rc1+up45.31.1 and fleet:104.0.0+up0.10.0-rc.17.

There is new Prometheus target monitoring-gitops-controller present:

Also the target can be selected on Fleet/Controller-Runtime Dashboard in its job dropdown and it's showing graphs:

[p-se]

Relates to rancher/fleet#2460

[github-actions]

Validation steps

• Ensure all container images have repository and tag on the same level to ensure that all container images are included in rancher-images.txt which are used by airgap customers.

  Ex:-
    longhorn-controller:
      repository: rancher/hardened-sriov-cni
      tag: v2.6.3-build20230913
  

• Add a 👍 (thumbs up) reaction to this comment once done. CI won't pass without this reaction to the github-action bot's latest validation comment.
• Approve the PR to run the CI check.