feat: enable istio sidecar for kubeflow namespace

[Bobgy]

This PR uses kubeflow/manifest's namespaces package instead.

https://github.com/kubeflow/manifests/blob/master/namespaces/base/namespaces.yaml

Therefore, istio sidecar auto injection is turned on.

[kubeflow-bot]

This change is 

[Bobgy]

/assign @jlewi

[jlewi]

@Bobgy Are you planning on turning this on for 1.1? This seems like a pretty big change which is landing pretty late in the 1.1 release cycle. I'm ok with enabling it on master but on 1.1 seems very risky.

[Bobgy]

@jlewi I understand your concern. Sorry, I should have brought it up earlier, but I'll list a few reasons why I think we should better have it.

• This change has been on kubeflow/manifest master for a while at least for GCP kfdef, it passed e2e tests.
• I have added istio sidecar disable annotation to every deployment, statefulset, it should be a no-op for other services.
• As we discussed before, user dynamic workloads should go into user namespaces. If we stick to the rule, they won't get affected.
• KFP multi user mode needs this security. Without istio mTLS + authorization rules, any workload in cluster can send a message to KFP api server and fake a user by making up the header.

I have tested this setup in my own cluster, I am not observing any behavior changes. Can we give it a try and if something doesn't work, roll back?

[jlewi]

I was confused I thought this was in kubeflow/manifests but its kubeflow/gcp-blueprints.

Are the kubeflow/manifests changes already on the 1.1 branch of kubeflow/manifests?

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Bobgy]

@jlewi Yes, it's there: https://github.com/kubeflow/manifests/blob/v1.1-branch/namespaces/base/namespaces.yaml.
But I think only GCP kfdef was enabling it.