kustomize package to setup a namespace for using Kubeflow (alternative to profile controller)

[jlewi]

We need a kustomize package that sets up a namespace for using Kubeflow. This package should basically do everything the profile controller is doing. e.g.

• Create default service accounts
• Create role bindings
• Platform specific overlays (e.g. setup workload identity for GCP)

A declarative approach to namespace setup should better support platform teams looking to setup and control kubeflow deployments on behalf of multiple teams. See the discussion in this thread

Creating a controller to setup namespaces made sense if we were trying to create an abstraction to hide the details of namespace to support programmatic creation e.g. through the UI.

However, as discussed in that thread we are seeing friction around that approach particularly for platform teams.

To support platform-teams I think we want to move in the direction of GitOps. If a team is using GitOps to bootstrap individual namespaces then using a controller makes less sense then just providing a kustomize package for the relevant K8s resources.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/feature	0.90

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[lalithvaka]

Some additional features to think about as following

1. Kustomize Declarative approach for ResourceQuota for each namespace
2. Group based profile / namespace support
3. Multi owner support for profile / namespace
4. Externalized enable/disable profile self serv capability - (Enable in Dev Env but disable in Prod)
5. Default resource quota setup for self serv profiles if they are enabled (Declarative?).

[discordianfish]

In general I like a declarative approach but as described here, it would require some glue I'd prefer we could avoid:

We're using an authenticating (via LDAP) reverse proxy in front of kubeflow that sets trusted user identity headers. That means right now we just need to grant users access to that proxy and when they log in the first time, they can create their namespace. With the proposed change, we would have add glue to poll LDAP and add/delete namespaces from the kustomize package dynamically.

That's definitely doable but I feel like many people would have to solve that somehow.

So the ideal solution to me would be if the profile controller creates a namespace (non-interactively) automatically based on some sort of declarative template.

[jlewi]

@discordianfish

So the ideal solution to me would be if the profile controller creates a namespace (non-interactively) automatically based on some sort of declarative template.

I don't think its necessarily a one size fits all. If you want to to propose and build out the case for doing things programmatically/automatically that sounds good to me.

On GCP what we are seeing is that

1. Platform teams want to own creating and setting up namespaces
2. GitOps is becoming the pattern of choice for managing infrastructure

Automatic creation probably won't work in our case because typically each namespace will require some user defined input that will vary from customer to customer.

A simple case is how customers map namespaces to GCP service accounts.

[discordianfish]

Ok makes sense. I think whatever the outcome of this will be, it should be possible to build whatever automation on top.

[jlewi]

@lalithvaka any interest at taking an initial stab at this issue?

[jlewi]

@lalithvaka and @bmorphism any progress on this?

[connorlwilkes]

@jlewi if no one else has picked this up I don't mind giving it a go? Seems fairly straight forward and helps with my particular use case

[jlewi]

@connorlwilkes That would be fantastic!

[connorlwilkes]

@jlewi brilliant. I may come back with some queries on current implementation and approach.

[tamera-lanham]

@connorlwilkes Have you made progress on this? It would be quite useful for my team

[jlewi]

A quick suggestion for anyone interested in picking this up; create a profile and then just dump all the K8s resources for the namespace.

kubectl get -n ${NAMESPACE} all -o yaml

That would probably provide a good starting point.

I would then turn that into a kpt package using the kpt CLI tool. Something like

kpt pkg cfg init ./
kpt pkg create-setter ...
kpt pkg create-subst ...

kpt provides a real nice way to make it reusable.

At least that's what I plan on doing if I find myself blocked on this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been closed due to inactivity.