[JENKINS-47591] dynamic pvc workspace volume

[runzexia]

Signed-off-by: runzexia runzexia@yunify.com
Add this kind of workspace volume because during use, we found that some IO-intensive tasks have some problems with the performance of emptyDir.
I want to use dynamic pvc to avoid this performance problem.

When I implemented this part, I found that the original WorkspaceVolume abstract class seems to be only suitable for static PVC.
So I want to find some comments about this part of the code, any comments are very grateful

@Vlatombe @jglick

[carlossg]

The interesting places where this gets used is

• kubernetes-plugin/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java

  Line 135 in e8b7cc3

  volumes.put(volumeName, volume.buildVolume(volumeName));

• kubernetes-plugin/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java

  Line 143 in e8b7cc3

  volumes.put(WORKSPACE_VOLUME_NAME, template.getWorkspaceVolume().buildVolume(WORKSPACE_VOLUME_NAME));

I guess you can ignore the volume name and set some random name. Or could add the podName to the arguments, so you can name the volume podName-xxxx

You would need to clean up the PVCs after the builds too, I think it would be possible to just set the owner reference in the PVC and let k8s delete the pvc when the pod is deleted
https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/#owners-and-dependents

There is a Jira at https://issues.jenkins-ci.org/browse/JENKINS-47591

[runzexia]

The interesting places where this gets used is

• kubernetes-plugin/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java

  Line 135 in e8b7cc3

  volumes.put(volumeName, volume.buildVolume(volumeName));

• kubernetes-plugin/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java

  Line 143 in e8b7cc3

  volumes.put(WORKSPACE_VOLUME_NAME, template.getWorkspaceVolume().buildVolume(WORKSPACE_VOLUME_NAME));

I guess you can ignore the volume name and set some random name. Or could add the podName to the arguments, so you can name the volume podName-xxxx

You would need to clean up the PVCs after the builds too, I think it would be possible to just set the owner reference in the PVC and let k8s delete the pvc when the pod is deleted
https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/#owners-and-dependents

There is a Jira at https://issues.jenkins-ci.org/browse/JENKINS-47591

Very useful guide, thanks! ! I will try it recently.

[runzexia]

@carlossg @Vlatombe @jglick ready for review, ci failed with timeout, need permission to create PVC?

[Vlatombe]

@runzexia the test can be guarded similarly to

kubernetes-plugin/src/test/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/KubernetesPipelineTest.java

Lines 375 to 380 in 6e34b99

	try {
	cloud.connect().apps().deployments().withName("cascading-delete").delete();
	} catch (KubernetesClientException x) {
	// Failure executing: DELETE at: https://…/apis/apps/v1/namespaces/kubernetes-plugin-test/deployments/cascading-delete. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. deployments.apps "cascading-delete" is forbidden: User "system:serviceaccount:…:…" cannot delete resource "deployments" in API group "apps" in the namespace "kubernetes-plugin-test".
	assumeNoException("was not permitted to clean up any previous deployment, so presumably cannot run test either", x);
	}

[runzexia]

the test can be guarded similarly to

@Vlatombe
I think this method is a resource for handling conflict names?
The createPvc method is created from the pod name and now includes cascading deletes in the code.
IMHO, the test timeout is because there is no permission to create pvc now, causing the agent to never start.

[Vlatombe]

I added a guard block to skip the new test if the sa doesn't have the right to handle pvcs.

[Vlatombe]

Please avoid force-pushes, it breaks incremental review.

[runzexia]

Ok, I will avoid this in the future.
By the way, I would like to ask if there is a plan to draft a release after this PR is merged?

[Vlatombe]

Yes, I'll cut a release

[Vlatombe]

LGTM.

@jglick any more comments?

[tarioch]

Is there a way to configure RequestsSize and StorageClassName through the UI? I don't seem to be able to choose/enter those values when selecting the "Dynamic Persistent Volume Claim Workspace Volume".

What would be needed to expose those two settings through the UI?

Then for doing a PR for the jenkins helm chart to include those permissions in the RBAC setup, would it just be the resource "persistentvolumeclaims"?

[runzexia]

have created a pull request
#614
@tarioch

[xxxvodnikxxx]

@runzexia
Hi, I can see PR was merged, so new version was released - 1.19
But in the jenkins I am able still see as latest 1.16.7

Can you please double checked, if you did not forgot eg. to change the plugin version or something?

Thank you :)

[Vlatombe]

@pavenova Please update your Jenkins instance, you probably have a core that is too old. We generally bump the required Jenkins core version when bumping the second digit (1.16 -> 1.17)

[xxxvodnikxxx]

@pavenova Please update your Jenkins instance, you probably have a core that is too old. We generally bump the required Jenkins core version when bumping the second digit (1.16 -> 1.17)

Hello Vicent, thanks for the point,
I have ver. 2.164.3 LTS , and new version of plugin did not appeared (of course the plugin list was successfully refreshed), so from the comment I guess jenkins version should be sufficient

[Vlatombe]

@pavenova Please refer to release notes for the required core version. Latest version of the plugin requires core 2.176.1 or later.

[xxxvodnikxxx]

@pavenova Please refer to release notes for the required core version. Latest version of the plugin requires core 2.176.1 or later.

Ahaah, got it, thank you :)

[asreich]

I'm testing this new addition for Dynamic PVCs on 2.190.1, Kubernetes Plugin version 1.20.0, the PV and PVCs are properly being created and disappearing, but I'm seeing this when the container attempts to start:

Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
Oct 11, 2019 11:20:18 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: default-dn816
Oct 11, 2019 11:20:18 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Oct 11, 2019 11:20:18 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3.29
Exception in thread "main" java.io.IOException: The specified working directory should be fully accessible to the remoting executable (RWX): /home/jenkins/agent
        at org.jenkinsci.remoting.engine.WorkDirManager.verifyDirectory(WorkDirManager.java:249)
        at org.jenkinsci.remoting.engine.WorkDirManager.initializeWorkDir(WorkDirManager.java:202)
        at hudson.remoting.Engine.startEngine(Engine.java:251)
        at hudson.remoting.Engine.startEngine(Engine.java:227)
        at hudson.remoting.jnlp.Main.main(Main.java:228)
        at hudson.remoting.jnlp.Main._main(Main.java:223)
        at hudson.remoting.jnlp.Main.main(Main.java:189)

I'm using Ceph RBD with a ReadWriteOnce and the mainline Jenkins Helm chart with JCasC 1.32. Wondering if this is mounting as "root" or something. Let me know if you need more info. Any thoughts?

[runzexia]

@austinReichert
try set

  securityContext:
    runAsUser:0
    fsGroup: 0

in yaml

[Vlatombe]

Or just

securityContext:
  fsGroup: 1000

[tarioch]

Created a PR for the jenkins helm chart helm/charts#17973

[DonAndrey]

Hello, currently I'm trying to set up a new jenkins on kubernetes, but I am facing some problems with the new dynamic pvc provision. You can see the error that it fires in the following image:

Any help would be really helpful!

[Vlatombe]

@DonAndrey It says the size you provided is invalid.

[Vlatombe]

@DonAndrey nvm. Revert to an older version of the plugin until this is fixed.

[DonAndrey]

Thanks @Vlatombe !

[josiahp]

I apologize for replying to this merged PR, but I could not find any method of contact for this repository and it seems like Issues are disabled.

Is it possible to set workspaceVolume when using the yaml based podTemplate? I can't see how to set it using the declarative pipeline, and setting "workspaceVolume" doesn't seem to work. It throws an error when I set it in the kubernetes agent section and it is ignored when set in the yaml. Any advice as to what I am doing wrong?

Neither of the two examples below work.

pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
spec:
  workspaceVolume: 'dynamicPVC'
...

pipeline {
  agent {
    kubernetes {
      workspaceVolume dynamicPVC
      yaml """
...
"""

[Vlatombe]

At the moment, it is only available in the podTemplate step with scripted syntax. Only a matter of implementing it for declarative (example for a different field).

The yaml syntax takes raw yaml that is passed directly to kubernetes, so it will never be supported with this method.

[nfalco79]

As information for peoples interested in this feature without spend hours to looking what's going wrong.
As reported by the short description in the Pipeline Syntax page

Note that this requires the Jenkins master to have additional RBAC permissions than are typically needed for agent provisioning.

this feature requires additionals permissions but does not specify which ones. After a long search I get the answer on OpenShift knowledge base (that you can see only with a suscription). I would suggest to add it in a help_...html file as tips or report into the README.MD

dynamicPvc requires a new cluster role permission:

apiVersion: authorization.openshift.io/v1 
kind: ClusterRole 
metadata: 
  name: pod-finalizers
rules: 
- apiGroups: 
  - ''
  attributeRestrictions: null 
  resources: 
  - pods/finalizers 
  verbs: 
  - update

to bind to the service account of the namespace you are using for agent provisioning and

  - verbs:
      - create
      - get
      - list
      - update
      - watch
    apiGroups:
      - ''
    resources:
      - persistentvolumeclaims

in addition to the "jenkins" (name used in every guide I found) Role binded to the same service account. I hope this will facilitate the use of this feature.

In real I'm looking to use this feature keeping the PVC until the multi pipeline project branch job is removed instead to remove the pvc after the pod agent is cancelled (usually at the end of the build)