[BUG] gitRepo resources aren't applying to clusters when changing fleet workspace #1845

slickwarren · 2023-10-05T23:45:03Z

Rancher Server Setup

Rancher version: v2.8-head (fc12e7d)
Installation option (Docker install/Helm Chart): rke2
- If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): v1.27.6+rke2r1
Proxy/Cert Details: valid certs

Information about the Cluster

Kubernetes version: v1.27.6 (rke2, k3s, rke1)
Cluster Type (Local/Downstream): downstream
- If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): aws

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)
- If custom, define the set of permissions: tested with admin

Describe the bug

when changing a cluster's workspace, the git-repo resources for fleet aren't applying to the clusters.

To Reproduce

create a cluster (by default, deploys to fleet-default workspace)
create a new workspace
in the new workspace, create a git-repo
transfer the cluster to the new workspace

Result
resources from fleet's new workspace aren't deploying. The cluster count remains at 0

Expected Result

resources from the gitRepo in the user added workspace should apply to any clusters that are matched.

Screenshots

Additional context

adding a git-repo and switching the cluster back to fleet-default results in the expected behavior
tested by using the 'all clusters in workspace' setting and manually selecting a cluster in the workspace, but neither is working

possibly affected by rancher/rancher#43078 as this was found on the same setup

aiyengar2 · 2023-10-06T16:24:40Z

The root cause of this issue is an error seen in the Job resource created by the GitJob resource created by the GitRepo resource in any user-created Fleet workspace.

Error creating: pods "ctw-test1-44c94-bkdzs" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "gitcloner-initializer", "fleet" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "gitcloner-initializer", "fleet" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "gitcloner-initializer", "fleet" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "gitcloner-initializer", "fleet" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

This is because the base cluster used when filing this issue was a hardened 1.25+ (1.27) cluster, which means that it enforces a restrictive set of Pod Security Standards.

Since the new namespace is not "allow-listed" to deploy privileged pods, the initContainer of the Job tied to the GitJob tied to the GitRepo is not able to progress, which results in the GitRepo creating no new Bundle resources. From there, fleet-agent is responding as expected by not creating any resources.

I suspect this might be a general issue that was not tested in previous releases, so here are some new steps for reproduction:

To Reproduce

create a Rancher instance that is running Kubernetes 1.25+ and follows the relevant hardening guide on the local cluster. Ensure PSA enforcement is happening.
create a new workspace
in the new workspace, create a git-repo
create a cluster in the new workspace

Expected Bad Result
The cluster sees the same symptoms as we see here. The Job in that namespace tied to the GitJob tied to the GitRepo should be prevented from running due to PSA.

@slickwarren can you explicitly test this on an older Rancher version?

cc: @manno @olblak, this is a Fleet issue, so we should ideally transfer this over.

slickwarren · 2023-10-06T18:27:15Z

tested on a released version of rancher, 2.7.8, with a hardened local cluster and am experiencing the same error there:

 7 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "working-dir-initializer", "place-tools", "step-git-source", "fleet" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "working-dir-initializer", "place-tools", "step-git-source", "fleet" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "working-dir-initializer", "place-tools", "step-git-source", "fleet" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "working-dir-initializer", "place-tools", "step-git-source", "fleet" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

I'm actually seeing similar behavior on non-hardened clusters. (on 2.8-head)

Error creating: pods "ctwtest-44c94-nb7c2" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "gitcloner-initializer", "fleet" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "gitcloner-initializer", "fleet" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "gitcloner-initializer", "fleet" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "gitcloner-initializer", "fleet" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

jiaqiluo · 2023-10-06T20:06:11Z

here are my 2 cents:
If it is true that Fleet components must run with those mentioned permissions ( allowPrivilegeEscalation, runAsNonRoot, etc.), we have to either modify the PSS level on that namespace or whitelist that namespace.

kkaempf · 2023-10-09T07:53:26Z

@sbulage - can you please verify this bug ?

sbulage · 2023-10-09T12:17:47Z

~~Hello @slickwarren, I am hitting this issue while installing Rancher from 2.8-head.~~

~~can you please tell me the way to install Rancher on k8s version > 1.27.0-0 ?~~

~~Thanks in advance.~~

aiyengar2 · 2023-10-09T20:41:58Z

here are my 2 cents: If it is true that Fleet components must run with those mentioned permissions ( allowPrivilegeEscalation, runAsNonRoot, etc.), we have to either modify the PSS level on that namespace or whitelist that namespace.

The issue here is that these workspaces are user-created resources, so there may be security implications with any automatic process that would attempt to whitelist such a namespace. cc: @macedogm @pjbgf, this may fall in your wheelhouse

macedogm · 2023-10-10T03:12:32Z

We should not add new namespaces to the PSAC exempt list already provided by Rancher, because such namespaces would be user controlled (outside of Rancher's trust boundaries). Additionally, doing such allow lists dynamically is risky.

The recommendation is to fix Fleet's deployment to make sure that all components run with an unprivileged securityContext, as Jiaqi and Arvind mentioned.

I haven't dig deep, but if I saw correctly, wouldn't be the case of fixing GitJob's deployment to match Fleet's and Fleet-Agent's deployment, respectively?

fleet/charts/fleet/templates/deployment.yaml

Lines 62 to 68 in a1dd929

    
           securityContext: 
        
             allowPrivilegeEscalation: false 
        
             readOnlyRootFilesystem: true 
        
             privileged: false 
        
             capabilities: 
        
               drop: 
        
               - ALL

fleet/charts/fleet-agent/templates/deployment.yaml

Lines 29 to 35 in a1dd929

    
           securityContext: 
        
             allowPrivilegeEscalation: false 
        
             readOnlyRootFilesystem: true 
        
             privileged: false 
        
             capabilities: 
        
               drop: 
        
               - ALL

Note: @aiyengar2 thanks for pinging us on this.

raulcabello · 2023-10-10T15:58:42Z

We should not add new namespaces to the PSAC exempt list already provided by Rancher, because such namespaces would be user controlled (outside of Rancher's trust boundaries). Additionally, doing such allow lists dynamically is risky.

The recommendation is to fix Fleet's deployment to make sure that all components run with an unprivileged securityContext, as Jiaqi and Arvind mentioned.

I haven't dig deep, but if I saw correctly, wouldn't be the case of fixing GitJob's deployment to match Fleet's and Fleet-Agent's deployment, respectively?

fleet/charts/fleet/templates/deployment.yaml

Lines 62 to 68 in a1dd929

securityContext:

allowPrivilegeEscalation: false

readOnlyRootFilesystem: true

privileged: false

capabilities:

drop:

- ALL

fleet/charts/fleet-agent/templates/deployment.yaml

Lines 29 to 35 in a1dd929

securityContext:

allowPrivilegeEscalation: false

readOnlyRootFilesystem: true

privileged: false

capabilities:

drop:

- ALL

Note: @aiyengar2 thanks for pinging us on this.

This is the GitJob deployment, not the k8s Job that is created when a GitRepo is created or modified. I think we should also add the securityContext here and here to fix this issue.

I can't transfer a Cluster to a different workspace, but I don't see anything in the logs.

Rancher version: v2.8.0-alpha2
Installation option (Docker install/Helm Chart): rke2 v1.27.6+rke2r1
Downstream cluster: k3s v1.27.6+k3s1

Steps:

create a cluster (by default, deploys to fleet-default workspace)
create a new workspace
in the new workspace, create a GitRepo
transfer the cluster to the new workspace

Then Cluster stays in Wait Check-In state and it is never moved to the new workspace I created. However I don't see anything in the gitjob or fleet-controller logs. And the k8s job that should run fleet apply is never created. Where should I look for the error?

aiyengar2 · 2023-10-11T10:52:30Z

@raulcabello the relationship between the GitRepo and Job is through the GitJob. When the GitRepo is modified, the related GitJob will also be changed:

fleet/internal/cmd/controller/controllers/git/git.go

Line 433 in a1dd929

&gitjob.GitJob{

That’s why the GitJob spawns a new Job.

So it is indeed tied to Job created when a GitRepo is modified. That is the k8s Job that runs the fleet apply.

aiyengar2 · 2023-10-11T10:53:45Z

Transferring the Fleet cluster is gated by a feature flag in Rancher, once you enable that in the UI (or by modifying the feature resource in the management cluster) you should be able to execute a transfer.

sbulage · 2023-10-11T11:28:54Z

After enabling the feature flag mentioned in this PR: rancher/dashboard#9730 (comment) #1845 (comment).

It is working as expected. As we (me and @raulcabello ) can see the cluster can be moved from fleet-default workspace to newly created workspace (GitRepo is already present.).

Cluster details: It is non-hardened cluster

Rancher version: 2.8-head
Fleet version: 0.9.0-rc3
Upstream cluster RKE2 : v1.27.6+rke2r1
Downstream cluster k3s: v1.26.4+k3s1

Also, tried to create new GitRepo in the newly created workspace and it is also working as expected.

No _error or warning_ traces found in gitjob pods.

K8s job is created and completed with no errors.

We will try with hardened cluster and try to reproduce it.

raulcabello · 2023-10-11T15:18:10Z

I think #1852 and rancher/gitjob#331 should fix this issue. However, we can't test it as we are still not able to reproduce the issue.

@sbulage will try tomorrow to reproduce it in a hardened cluster

kkaempf · 2023-10-11T15:44:04Z

It needs to be

a hardened cluster
k8s 1.25+
PSA enforcement needs to be turned on

Also:

one would not get messages about violates PodSecurity on non-hardened clusters (restricted AdmissionConfiguration
needs to be set at cluster level, see https://kubernetes.io/docs/tutorials/security/cluster-level-pss/ for more context).

raulcabello · 2023-10-13T13:46:12Z

We should not add new namespaces to the PSAC exempt list already provided by Rancher, because such namespaces would be user controlled (outside of Rancher's trust boundaries). Additionally, doing such allow lists dynamically is risky.

The recommendation is to fix Fleet's deployment to make sure that all components run with an unprivileged securityContext, as Jiaqi and Arvind mentioned.

I haven't dig deep, but if I saw correctly, wouldn't be the case of fixing GitJob's deployment to match Fleet's and Fleet-Agent's deployment, respectively?

fleet/charts/fleet/templates/deployment.yaml

Lines 62 to 68 in a1dd929

securityContext:

allowPrivilegeEscalation: false

readOnlyRootFilesystem: true

privileged: false

capabilities:

drop:

- ALL

fleet/charts/fleet-agent/templates/deployment.yaml

Lines 29 to 35 in a1dd929

securityContext:

allowPrivilegeEscalation: false

readOnlyRootFilesystem: true

privileged: false

capabilities:

drop:

- ALL

Note: @aiyengar2 thanks for pinging us on this.

@macedogm is it ok if we don't add the readOnlyRootFilesystem: true to the securityContext? This is causing problems when cloning git repos with go-git. go-git is creating temporary files that are causing the problem. go-git is a third party library.

The following securityContext in both containers in the job created by gitjob is working fine in hardened clusters in our test env.

	SecurityContext: &corev1.SecurityContext{
		AllowPrivilegeEscalation: &[]bool{false}[0],
		Privileged:               &[]bool{false}[0],
		RunAsNonRoot:             &[]bool{true}[0],
		SeccompProfile: &corev1.SeccompProfile{
			Type: corev1.SeccompProfileTypeRuntimeDefault,
		},
		Capabilities: &corev1.Capabilities{Drop: []corev1.Capability{"ALL"}},
	},

Is this securityContext enough? see #1860 and rancher/gitjob#331

pjbgf · 2023-10-13T14:16:16Z

This is causing problems when cloning git repos with go-git. go-git is creating temporary files that are causing the problem. go-git is a third party library.

@raulcabello You can mount an emptyDir to that specific path (/tmp) to bypass this issue.

macedogm · 2023-10-13T14:19:41Z

@raulcabello agree with Paulo's suggestion above (in case its feasible).

sbulage · 2023-10-17T08:48:02Z

I have tested with Raul's fix with different images and found that fleet-agent pods are not re-creating with newer images. Till the time Raul's PR gets merged, he uploaded images and gave it to me for testing.

Throwing below error:
Pods "fleet-agent-79fc9f8d57-8k6kw" is forbidden: violates PodSecurity "restricted:latest": unrestricted capabilities (container "fleet-agent" must set securityContext.capabilities.drop=["ALL"]; container "fleet-agent" must not include "ALL" in securityContext.capabilities.add), runAsNonRoot != true (container "fleet-agent" must not set securityContext.runAsNonRoot=false), seccompProfile (pod or container "fleet-agent" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost"):Updated: 0/1
I closely look at the PSA exempted namespace list and saw that cattle-fleet-local-system namespace which is managed by Fleet-controller is absent. In order test the patch is working, I followed below stesp:

I updated PSA file by adding cattle-fleet-local-system namespace to exempted namespace list.

Restarted rk2 server

Updated the fleet-agents image
I observed that fleet-agent image gets updated without any error.

@pjbgf @macedogm What should I do? should I create new issue for it, if yes, then where? please let me know thanks 😄

Ignore this comment as I see cattle-fleet-local-system is already exempted. (Here)

manno · 2023-10-17T12:36:54Z

/backport v2.8.0 release/v0.9

slickwarren · 2023-10-17T21:45:52Z

@manno or @sbulage it doesn't look like rancher has picked up the new RC. Is this something your team can update so that I can run another round of testing?
rancher/charts#3138

sbulage · 2023-10-18T08:35:34Z

@slickwarren There was some issue with CI in rancher/charts which seems like it is fixed. It will be available once rancher/charts#3138 merged(hopefully in next hour or so. 🤞 )

raulcabello · 2023-10-18T09:17:49Z

It is available now

sbulage · 2023-10-18T10:45:02Z

All below test scenario performed on RKE2 cluster with and without hardening it.

Environment Details:

Rancher version: 2.8-head
Fleet version: 0.9.0-rc.5
Downstream cluster: k3s
K8s version: v1.27.6+rke2r1

QA TEST PLAN

Scenarios

Scenario	Test Case
1	Test GitRepo deploys application in newly created workspace when cluster is moved from fleet-default workspace to newly created workspace in Hardened cluster.
2	Test GitRepo deploys application in newly created workspace when cluster is moved from fleet-default workspace to newly created workspace in non-hardened cluster.
3	Test GitRepo deploys helm application in newly created workspace when cluster is moved from fleet-default workspace to newly created workspace in Hardened cluster.
4	Test private GitRepo deploys application in newly created workspace when cluster is moved from fleet-default workspace to newly created workspace in Hardened cluster.
5	Test private GitRepo deploys application in newly created workspace when cluster is moved from fleet-default workspace to newly created workspace non-hardened cluster.

sbulage · 2023-10-18T12:15:29Z

TEST RESULT

RKE2 hardened cluster is created by following documentation.

Scenarios

Scenario	Test Case	Result
1	Test GitRepo deploys application in newly created workspace when cluster is moved from fleet-default workspace to newly created workspace in Hardened cluster.	✔️
2	Test GitRepo deploys application in newly created workspace when cluster is moved from fleet-default workspace to newly created workspace in non-hardened cluster.	✔️
3	Test GitRepo deploys helm application in newly created workspace when cluster is moved from fleet-default workspace to newly created workspace in Hardened cluster.	✔️
4	Test private GitRepo deploys application in newly created workspace when cluster is moved from fleet-default workspace to newly created workspace in Hardened cluster.	✔️
5	Test private GitRepo deploys application in newly created workspace when cluster is moved from fleet-default workspace to newly created workspace non-hardened cluster.	✔️

REPRO STEPS

RKE2 Hardened cluster used
Scenario 1

Create a new workspace new-workspace1.
Create a GitRepo which deploys nginx.
Go to the Continuous Delivery --> Clusters.
Change workspace of the imported-cluster from fleet-default to new-workspace1.
Verified that nginx app deployed cluster which moved to new-workspace without any issue.

RKE2 Non-hardened cluster used
Scenario 2

Create a new workspace new-workspace2.
Create a GitRepo which deploys nginx.
Go to the Continuous Delivery --> Clusters.
Change workspace of the imported-cluster from fleet-default to new-workspace2.
Verified that nginx app deployed cluster which moved to new-workspace without any issue.

RKE2 Hardened cluster used
Scenario 3

Create a new workspace new-workspace3.
Create a GitRepo which deploys grafana helm application.
Go to the Continuous Delivery --> Clusters.
Change workspace of the imported-cluster from fleet-default to new-3.
Verified that grafana app deployed cluster which moved to new-workspace without any issue.

RKE2 Hardened cluster used
Scenario 4

Create a new workspace new-workspace4.
Create a GitRepo which deploys nginx from private github repository.
Go to the Continuous Delivery --> Clusters.
Change workspace of the imported-cluster from fleet-default to new-workspace4.
Verified that nginx app deployed cluster which moved to new-workspace without any issue.

RKE2 Non-hardened cluster used
Scenario 5

Create a new workspace new-workspace5.
Create a GitRepo which deploys grafana from private github repository.
Go to the Continuous Delivery --> Clusters.
Change workspace of the imported-cluster from fleet-default to new-workspace5.
Verified that grafana app deployed cluster which moved to new-workspace without any issue.

kkaempf · 2023-10-18T12:58:19Z

@slickwarren - please give it a try 😉

slickwarren · 2023-10-19T00:06:16Z

my tests through rancher using rc5 are working well, and I've closed the rancher-side issues. Not sure of fleet's process for closing these, but rancher-qa signs off on this 👍🏼

sbulage · 2023-10-19T04:40:48Z

Thanks @slickwarren I will close this issue now. 👍

slickwarren added status/blocker area/fleet labels Oct 5, 2023

slickwarren added this to the v2.8.0 milestone Oct 5, 2023

slickwarren self-assigned this Oct 5, 2023

slickwarren mentioned this issue Oct 5, 2023

Cannot assign non-RKE1 Clusters to a Fleet Workspace rancher/rancher#36132

Closed

Sahota1225 transferred this issue from rancher/rancher Oct 6, 2023

rancherbot added this to Fleet Oct 6, 2023

github-project-automation bot moved this to 🆕 New in Fleet Oct 6, 2023

kkaempf moved this from 🆕 New to 📋 Backlog in Fleet Oct 9, 2023

raulcabello moved this from 📋 Backlog to 🏗 In progress in Fleet Oct 11, 2023

raulcabello self-assigned this Oct 11, 2023

This was referenced Oct 11, 2023

Add SecurityContext to deployment and init container rancher/gitjob#331

Merged

Add SecurityContext to job container that executes fleet apply #1852

Closed

raulcabello mentioned this issue Oct 13, 2023

Add SecurityContext to job container #1860

Merged

This comment was marked as resolved.

Sign in to view

raulcabello mentioned this issue Oct 17, 2023

Add SecurityContext to job container #1875

Merged

Jono-SUSE-Rancher added the [zube]: Working label Oct 17, 2023

raulcabello moved this from 🏗 In progress to Needs QA review in Fleet Oct 18, 2023

sbulage added [zube]: QA Working and removed [zube]: Working labels Oct 18, 2023

sbulage added [zube]: Done and removed [zube]: QA Working labels Oct 19, 2023

sbulage moved this from Needs QA review to ✅ Done in Fleet Oct 19, 2023

zube bot closed this as completed Oct 19, 2023

zube bot removed the [zube]: Done label Jan 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] gitRepo resources aren't applying to clusters when changing fleet workspace #1845

[BUG] gitRepo resources aren't applying to clusters when changing fleet workspace #1845

slickwarren commented Oct 5, 2023 •

edited

Loading

aiyengar2 commented Oct 6, 2023

slickwarren commented Oct 6, 2023 •

edited

Loading

jiaqiluo commented Oct 6, 2023

kkaempf commented Oct 9, 2023

sbulage commented Oct 9, 2023 •

edited

Loading

aiyengar2 commented Oct 9, 2023

macedogm commented Oct 10, 2023

raulcabello commented Oct 10, 2023

aiyengar2 commented Oct 11, 2023 •

edited

Loading

aiyengar2 commented Oct 11, 2023 •

edited

Loading

sbulage commented Oct 11, 2023 •

edited

Loading

raulcabello commented Oct 11, 2023 •

edited

Loading

kkaempf commented Oct 11, 2023

raulcabello commented Oct 13, 2023 •

edited

Loading

pjbgf commented Oct 13, 2023

macedogm commented Oct 13, 2023

This comment was marked as resolved.

sbulage commented Oct 17, 2023

manno commented Oct 17, 2023

slickwarren commented Oct 17, 2023 •

edited

Loading

sbulage commented Oct 18, 2023

raulcabello commented Oct 18, 2023

sbulage commented Oct 18, 2023 •

edited

Loading

sbulage commented Oct 18, 2023

kkaempf commented Oct 18, 2023

slickwarren commented Oct 19, 2023

sbulage commented Oct 19, 2023

[BUG] gitRepo resources aren't applying to clusters when changing fleet workspace #1845

[BUG] gitRepo resources aren't applying to clusters when changing fleet workspace #1845

Comments

slickwarren commented Oct 5, 2023 • edited Loading

aiyengar2 commented Oct 6, 2023

slickwarren commented Oct 6, 2023 • edited Loading

jiaqiluo commented Oct 6, 2023

kkaempf commented Oct 9, 2023

sbulage commented Oct 9, 2023 • edited Loading

aiyengar2 commented Oct 9, 2023

macedogm commented Oct 10, 2023

raulcabello commented Oct 10, 2023

aiyengar2 commented Oct 11, 2023 • edited Loading

aiyengar2 commented Oct 11, 2023 • edited Loading

sbulage commented Oct 11, 2023 • edited Loading

raulcabello commented Oct 11, 2023 • edited Loading

kkaempf commented Oct 11, 2023

raulcabello commented Oct 13, 2023 • edited Loading

pjbgf commented Oct 13, 2023

macedogm commented Oct 13, 2023

This comment was marked as resolved.

sbulage commented Oct 17, 2023

manno commented Oct 17, 2023

slickwarren commented Oct 17, 2023 • edited Loading

sbulage commented Oct 18, 2023

raulcabello commented Oct 18, 2023

sbulage commented Oct 18, 2023 • edited Loading

sbulage commented Oct 18, 2023

kkaempf commented Oct 18, 2023

slickwarren commented Oct 19, 2023

sbulage commented Oct 19, 2023

slickwarren commented Oct 5, 2023 •

edited

Loading

slickwarren commented Oct 6, 2023 •

edited

Loading

sbulage commented Oct 9, 2023 •

edited

Loading

aiyengar2 commented Oct 11, 2023 •

edited

Loading

aiyengar2 commented Oct 11, 2023 •

edited

Loading

sbulage commented Oct 11, 2023 •

edited

Loading

raulcabello commented Oct 11, 2023 •

edited

Loading

raulcabello commented Oct 13, 2023 •

edited

Loading

slickwarren commented Oct 17, 2023 •

edited

Loading

sbulage commented Oct 18, 2023 •

edited

Loading