Have one Ingress controller per network or one only if they are the same

[nootal]

Component:

'network', 'kubernetes', 'ingress'

Why this is needed:

Because we want to expose different services using Ingress objects that belong to different networks. For example, we have the K8S API and the Metalk8s UI (when #1602 will be done) that are hosted on the control plane network, and the Grafana UI (and probably the Solution UIs) that are hosted on the workload plane network.

Right now we already have one Ingress controller on the workload plane network if I'm correct. So we want to have a second one on the control plane network if it's a different network (because it will bind on a different IP address). But if both networks are the same, then we can have only one controller otherwise both will try binding the same IP address and the same port and that can not work.

What should be done:

Some information is available here: https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/

Implementation proposal (strongly recommended):

Each Ingress controller is responsible for an Ingress class that Ingress objects should declare in their annotations so an Ingress controller knows which Ingress objects it is supposed to expose.
As described in the Important note in here https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/#multiple-ingress-nginx-controllers it could be interesting to have the Ingress controller used for the workload plane network to have the default class, so any Ingress object not directly targeting the Ingress class related to the control plane network would end up in the workload plane network.

Test plan:

• Make sure that if networks are different then we have two Ingress controllers
• Make sure that if networks are the same then we have one Ingress controller