Migrate Dashboards into Tenants

[Rotfuks]

Motivation

After we have implemented multi-tenancy with #3695 we need to use it to organize our dashboards in a better way, having a real difference between private and public dashboards for our customers. 

Todo

• Rename Main Org. into Shared or public org TBD
• Investigate all existing dashboards if they are giantswarm or public resources
  • move all internal dashboards into the giantswarm organisation
  • move all shared dashboards into the shared organisation
  • Get rid of the grafana-permissions job
• Document how customers can make their dashboards shared or private

Outcome

• We have a real difference in the accessibility of dashboards being shared or private.

[hervenicol]

I just had a look at it.

How it currently works:

Upstream doc: https://grafana.com/docs/grafana/latest/administration/provisioning/#dashboards

We set some dashboard provisioners via chart values (.grafana.dashboardProviders."dashboardproviders.yaml").
Currently we set 3 provisioners: private, public and customer dashboards, that put dashboards in different folders

How it could work:

Here is how I think we can update it to support orgs:

• each new org adds a dashboardprovisioners file that looks like this:

providers:
  - allowUiUpdates: false
    disableDeletion: true
    editable: false
    folder: Customer1 org dashboards
    folderUid: customer1
    name: private
    options:
      path: /var/lib/grafana/dashboards/org_customer1
    orgId: 2
    type: file
    updateIntervalSeconds: 60

• then we should ensure that the dashboards get the proper path annotation (.metadata.annotations.k8s-sidecar-target-directory: /var/lib/grafana/dashboards/org_customer1)

How I failed at testing it

I tried to set a new dashboardprovider via grafana-user-values. But it gets overridden because it seems observability-operator owns these user-values. Not sure it's good.

Then I tried adding extraConfigs to grafana, and define my new provider in these. But the app gets overridden quite fast as well.

Next steps

I could push this in giantswarm-configs for my tests, but I don't want to do multiple PRs for this test.
So I think I'd rather have a dev version of observability-operator which sets a test dasboard provisionner it grafana-user-values.
I just have to synchronize with the team so we don't break each other's tests.

[hervenicol]

So, at startup grafana does not have orgs, so grafana won't manage to load the org dashboard (and maybe even the dashboard providers) that reference non-existing orgs.
It may even crash at startup (that's what happened with datasources provisioning, and one of the reasons why they are now managed by olly-op).

So, we now have to chose:

• we can do all the provisioning via olly-op
• or if we have persistence in grafana maybe we can have a mix

Persistence will be discussed next week, because it's needed for clickops dashboards management.

Note: grafana currently refuses PRs that improve current provisioning system, because they plan to completely rework it starting next major release (can be expected around may/june as we had 10.0.0 on 2023-06-12 and 11.0.0 on 2024-05-14)

[hervenicol]

We decided persistence is not for now, and we want to keep proper gitops support.

So, the dashboards provisioning for orgs should be done via olly-op.

[hervenicol]

I wanted to do something similar to the datasources logic, but it looks like a bad idea.

The logic for datasources is basically:

• build a list of current datasources in the org
• build a list of datasources we want to load
• from the differences
  • create a list of datasources tu create
  • create a list of datasources to update
  • (no list of datasources to delete yet)

We can't do the same with dashboards because:

• there's no getDashboards function in the API, we're supposed to either search or getByUID.
• the dashboards once loaded into grafana are updated / changed by grafana, so they will nearly always differ from the configmap.
• we can't rely on the UID because not all configmap dashboards will have an UID. It's optional.

So we should rather work with configmap status.
But if we react to a grafana pod reload we should just upload all dashboards.

[hervenicol]

• Feature has been implemented and released in olly-op.
• private dashboards are now deployed in GS tenant as well as shared tenant

Remaining:

• move public dashboards loading from files provisioning to olly-op provisioning
• disable files provisioning for dashboards
• disable permissions maintenance cronjob
• figure out dashboards links with multiple grafana orgs (grafana multi-org and dashboard links #3826)

[QuentinBisson]

I think we're missing some private dashboards as well (I guess because they have no UID set). For instance, on alba, we have 110 private dashboards in the shared org but only 88 in the giantswarm org

[QuentinBisson]

FYI some dashboards are in their apps https://github.com/search?q=org%3Agiantswarm+k8s-sidecar-target-directory&type=code