Skip to content

Latest commit

 

History

History
263 lines (210 loc) · 7.56 KB

README.md

File metadata and controls

263 lines (210 loc) · 7.56 KB

flux2-multi-tenancy

This repository serves as a starting point for managing multi-tenant clusters with Git and Flux v2.

Roles

Platform Admin

  • Has cluster admin access to the fleet of clusters
  • Has maintainer access to the fleet Git repository
  • Manages cluster wide resources (CRDs, controllers, cluster roles, etc)
  • Onboards the tenant’s main GitRepository and Kustomization
  • Manages tenants by assigning namespaces, service accounts and role binding to the tenant's apps

Tenant

  • Has admin access to the namespaces assigned to them by the platform admin
  • Has maintainer access to the tenant Git repository and apps repositories
  • Manages app deployments with GitRepositories and Kustomizations
  • Manages app releases with HelmRepositories and HelmReleases

Repository structure

The platform admin repository contains the following top directories:

  • clusters dir contains the Flux configuration per cluster
  • infrastructure dir contains common infra tools such as admission controllers, CRDs and cluster-wide polices
  • tenants dir contains namespaces, service accounts, role bindings and Flux custom resources for registering tenant repositories
├── clusters
│   ├── production
│   └── staging
├── infrastructure
│   ├── kyverno
│   └── kyverno-policies
└── tenants
    ├── base
    ├── production
    └── staging

A tenant repository contains the following top directories:

  • base dir contains HelmRepository and HelmRelease manifests
  • staging dir contains HelmRelease Kustomize patches for deploying pre-releases on the staging cluster
  • production dir contains HelmRelease Kustomize patches for deploying stable releases on the production cluster
├── base
│   ├── kustomization.yaml
│   ├── podinfo-release.yaml
│   └── podinfo-repository.yaml
├── production
│   ├── kustomization.yaml
│   └── podinfo-values.yaml
└── staging
    ├── kustomization.yaml
    └── podinfo-values.yaml

How to define tenants

The Flux CLI offers commands to generate the Kubernetes manifests needed to define tenants.

Assuming a platform admin wants to create a tenant named dev-team with access to the apps namespace.

Create the tenant base directory:

mkdir -p ./tenants/base/dev-team

Generate the namespace, service account and role binding for the dev-team:

flux create tenant dev-team --with-namespace=apps \
    --export > ./tenants/base/dev-team/rbac.yaml

Create the sync manifests for the tenant Git repository:

flux create source git dev-team \
    --namespace=apps \
    --url=https://github.com/<org>/<dev-team> \
    --branch=main \
    --export > ./tenants/base/dev-team/sync.yaml

flux create kustomization dev-team \
    --namespace=apps \
    --service-account=dev-team \
    --source=GitRepository/dev-team \
    --path="./" \
    --export >> ./tenants/base/dev-team/sync.yaml

Create the staging overlay and set the path to the staging dir inside the tenant repository:

cat << EOF | tee ./tenants/staging/dev-team-patch.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: dev-team
  namespace: apps
spec:
  path: ./staging
EOF

cat << EOF | tee ./tenants/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base/dev-team
patchesStrategicMerge:
  - dev-team-patch.yaml
EOF

With the above configuration, the Flux instance running on the staging cluster will clone the dev-team's repository, and it will reconcile the ./staging directory from the tenant's repo using the dev-team service account. Since that service account is restricted to the apps namespace, the dev-team repository must contain Kubernetes objects scoped to the apps namespace only.

Enforce tenant isolation

Inside the clusters dir we define in which order the infrastructure items, and the tenant workloads are going to be reconciled on the staging and production clusters:

./clusters/
├── production
│   ├── infrastructure.yaml
│   └── tenants.yaml
└── staging
    ├── infrastructure.yaml
    └── tenants.yaml

First we setup the reconciliation of custom resource definitions and their controllers. For this example we'll use Kyverno:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: kyverno
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/kyverno
  prune: true
  validation: client
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: kyverno
      namespace: kyverno

Then we setup cluster policies (Kyverno custom resources) to enforce tenant isolation:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: kyverno-policies
  namespace: flux-system
spec:
  dependsOn:
    - name: kyverno
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/kyverno-policies
  prune: true
  validation: client

With dependsOn we tell Flux to install Kyverno before deploying the cluster policies.

And finally we setup the reconciliation for the tenants workloads with:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: tenants
  namespace: flux-system
spec:
  dependsOn:
    - name: kyverno-policies
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./tenants/staging
  prune: true
  validation: client

With the above configuration, we ensure that the Kyverno validation webhook will reject Kustomizations and HelmReleases that don't specify a service account name when deployed in a tenant's namespace.

Bootstrap staging and production

Install the Flux CLI and fork this repository on your personal GitHub account and export your GitHub access token, username and repo name:

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
export GITHUB_REPO=<repository-name>

Verify that your staging cluster satisfies the prerequisites with:

flux check --pre

Set the kubectl context to your staging cluster and bootstrap Flux:

flux bootstrap github \
    --context=staging \
    --owner=${GITHUB_USER} \
    --repository=${GITHUB_REPO} \
    --branch=main \
    --personal \
    --path=clusters/staging

The bootstrap command commits the manifests for the Flux components in clusters/staging/flux-system dir and creates a deploy key with read-only access on GitHub, so it can pull changes inside the cluster.

Wait for the staging cluster reconciliation to finish:

$ watch flux get kustomization
NAME            	READY  	MESSAGE                                                        	
flux-system     	True   	Applied revision: main/616001c38e7bc81b00ef2c65ac8cfd58140155b8	
kyverno         	Unknown	reconciliation in progress
kyverno-policies	False  	dependency 'flux-system/kyverno' is not ready
tenants         	False  	dependency 'flux-system/kyverno-policies' is not ready

Verify that the tenant repository has been cloned inside the cluster:

$ flux -n apps get sources git
NAME    	READY	MESSAGE 
dev-team	True 	Fetched revision: dev-team/ca8ec25405cc03f2f374d2f35f9299d84ced01e4