Prometheus grafana & add cirun.io

[Adam-D-Lewis]

Initial integration of prometheus/grafana into qhub. I tested it locally and it seems to be working. The kubernetes tests appear to be broken for an unrelated reason.

[costrouc]

I'd prefer if we could pull in the helm charts to minimize the number of files in qhub (also making updates easier) similar to https://github.com/Quansight/qhub/blob/main/qhub/template/%7B%7B%20cookiecutter.repo_directory%20%7D%7D/infrastructure/modules/kubernetes/services/jupyterhub/main.tf#L6. This still pins the chart to a particular version. Is there a reason that we are copying the entire chart locally?

[Adam-D-Lewis]

@costrouc I was following the example of what was done for the clearml integration (https://github.com/Quansight/qhub/tree/main/qhub/template/%7B%7B%20cookiecutter.repo_directory%20%7D%7D/infrastructure/modules/kubernetes/services/clearml), but I don't think there is anything preventing us from importing the chart. @aktech any thoughts for or against putting the entire helm chart in the repo?

[aktech]

@Adam-D-Lewis I agree with @costrouc we shouldn't put helm charts in the repository. ClearML was a special case, as we had to modify the chart. Also, since the last update on ClearML, I'll see if we can remove the ClearML charts as well.

[Adam-D-Lewis]

It seems like the following files can be removed:

values-monitoring-qhub.yaml
values-original.yaml
Also the diff between the original and overridden files is just following:

❯ diff values-original.yaml values-monitoring-qhub.yaml
628a629,637

grafana.ini:
server:
# The full public facing url you use in browser, used for redirects and emails
domain: github-actions.qhub.dev
root_url: "%(protocol)s://%(domain)s/monitoring"
server_from_sub_path: true
router_logging: true
enable_gzip: true
732c741
<

---

Which means you can set the particular values via terraform's helm set attribute.

I made the requested changes.

[aktech]

Waiting for the tests to pass here: #748 to see if the failed K8s tests in here are relevant or not.

[aktech]

Seems like the tests pass on main, can you check why k8s tests fail? I can rerun again to confirm,

[Adam-D-Lewis]

Cyprus says there is insufficient cpu on the node with the additional pods in this PR. Maybe I can reduce the used cpu or maybe we can use CIrun

[Adam-D-Lewis]

Blocked by #756

[costrouc]

Thanks for all your hard work on this @Adam-D-Lewis. This looks great to me. Looks perfect to me. Also excited to have a bigger CI for testing the Kubernetes cluster. This should allow out tests to be more comprehensive.

[costrouc]

Could you add an entry to the changelog?