Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rbd: omap leak caught in jenkins ci #5175

Open
Rakshith-R opened this issue Feb 25, 2025 · 2 comments
Open

rbd: omap leak caught in jenkins ci #5175

Rakshith-R opened this issue Feb 25, 2025 · 2 comments
Labels
bug Something isn't working component/rbd Issues related to RBD

Comments

@Rakshith-R
Copy link
Contributor 

          > > /retest ci/centos/mini-e2e/k8s-1.32

The logs of the failure do not immediate show what the issue is, I think:

  �[1mSTEP:�[0m create a PVC and check PVC/PV metadata on RBD image �[38;5;243m@ 02/20/25 11:57:08.555�[0m
  I0220 11:57:08.558944 86817 pvc.go:60] Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc  rbd-1827    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[group:test] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},} to be in Bound state
  I0220 11:57:08.558987 86817 pvc.go:63] waiting for PVC rbd-pvc (0 seconds elapsed)
  I0220 11:57:08.563534 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563572 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563799 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563817 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a
  I0220 11:57:08.563825 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563833 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563840 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-ca574bf2-9632-49b9-8125-66951fc004c2
  I0220 11:57:10.559694 86817 pvc.go:63] waiting for PVC rbd-pvc (2 seconds elapsed)
  I0220 11:57:10.563681 86817 pv.go:468] Waiting for PV pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to bind to PVC rbd-pvc
  I0220 11:57:10.563735 86817 pv.go:790] Waiting up to timeout=10m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound
  I0220 11:57:10.567465 86817 pv.go:801] PersistentVolumeClaim rbd-pvc found and phase=Bound (3.716004ms)
  I0220 11:57:10.567495 86817 pv.go:763] Waiting up to 10m0s for PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to have phase Bound
  I0220 11:57:10.569559 86817 pv.go:771] PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb found and phase=Bound (2.052301ms)
  I0220 11:57:10.574121 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.574146 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.574201 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.685176 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool | grep -v default | grep -v csi.volume.group. |  grep -c ^csi.volume.] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.685215 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.685248 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool+%7C+grep+-v+default+%7C+grep+-v+csi.volume.group.+%7C++grep+-c+%5Ecsi.volume.&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.807774 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.807838 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.807873 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.917795 86817 utils.go:251] additional debug info: rados ls command output: rbd_header.13de5595a987
  rbd_directory
  rbd_info
  csi.volume.aec40a24-45ed-4919-97ca-b2c5b00517b5
  rbd_task
  csi.volumes.default
  rbd_trash
  csi.volume.860bd5be-e492-4bd0-b3de-7ce56fa535e1
  rbd_id.csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5
  , stdErr: 
  �[38;5;9m[FAILED]�[0m in [It] - /go/src/github.com/ceph/ceph-csi/e2e/utils.go:254 �[38;5;243m@ 02/20/25 11:57:10.918�[0m

[38;5;9m[FAILED] failed to validate omap count for rados ls --pool=replicapool | grep -v default | grep -v csi.volume.group. | grep -c ^csi.volume.: expected omap object count 1, got 2�[0m

i think there is a omap leak.

Originally posted by @Madhu-1 in #5149 (comment)
@Rakshith-R Rakshith-R added bug Something isn't working component/rbd Issues related to RBD labels Feb 25, 2025
@Rakshith-R
Copy link
Contributor Author

failed in the following step:

  �[1mSTEP:�[0m create a PVC and check PVC/PV metadata on RBD image �[38;5;243m@ 02/20/25 11:57:08.555�[0m
  I0220 11:57:08.558944 86817 pvc.go:60] Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc  rbd-1827    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[group:test] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},} to be in Bound state
  I0220 11:57:08.558987 86817 pvc.go:63] waiting for PVC rbd-pvc (0 seconds elapsed)
  I0220 11:57:08.563534 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563572 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563799 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563817 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a
  I0220 11:57:08.563825 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563833 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563840 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-ca574bf2-9632-49b9-8125-66951fc004c2
  I0220 11:57:10.559694 86817 pvc.go:63] waiting for PVC rbd-pvc (2 seconds elapsed)
  I0220 11:57:10.563681 86817 pv.go:468] Waiting for PV pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to bind to PVC rbd-pvc
  I0220 11:57:10.563735 86817 pv.go:790] Waiting up to timeout=10m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound
  I0220 11:57:10.567465 86817 pv.go:801] PersistentVolumeClaim rbd-pvc found and phase=Bound (3.716004ms)
  I0220 11:57:10.567495 86817 pv.go:763] Waiting up to 10m0s for PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to have phase Bound
  I0220 11:57:10.569559 86817 pv.go:771] PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb found and phase=Bound (2.052301ms)
  I0220 11:57:10.574121 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.574146 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.574201 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.685176 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool | grep -v default | grep -v csi.volume.group. |  grep -c ^csi.volume.] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.685215 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.685248 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool+%7C+grep+-v+default+%7C+grep+-v+csi.volume.group.+%7C++grep+-c+%5Ecsi.volume.&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.807774 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.807838 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.807873 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.917795 86817 utils.go:251] additional debug info: rados ls command output: rbd_header.13de5595a987
  rbd_directory
  rbd_info
  csi.volume.aec40a24-45ed-4919-97ca-b2c5b00517b5
  rbd_task
  csi.volumes.default
  rbd_trash
  csi.volume.860bd5be-e492-4bd0-b3de-7ce56fa535e1
  rbd_id.csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5
  , stdErr: 
  �[38;5;9m[FAILED]�[0m in [It] - /go/src/github.com/ceph/ceph-csi/e2e/utils.go:254 �[38;5;243m@ 02/20/25 11:57:10.918�[0m

csi-rbdplugin logs:

  I0220 11:56:24.300245       1 cephcsi.go:202] Driver version: canary and Git version: 9201f124150e0bc2b3618d94011a6c23bebdb121
  I0220 11:56:24.300365       1 cephcsi.go:234] Starting driver type: rbd with name: rbd.csi.ceph.com
  I0220 11:56:24.300576       1 driver.go:110] Enabling controller service capability: CREATE_DELETE_VOLUME
  I0220 11:56:24.300589       1 driver.go:110] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
  I0220 11:56:24.300593       1 driver.go:110] Enabling controller service capability: CLONE_VOLUME
  I0220 11:56:24.300596       1 driver.go:110] Enabling controller service capability: EXPAND_VOLUME
  I0220 11:56:24.300602       1 driver.go:123] Enabling volume access mode: SINGLE_NODE_WRITER
  I0220 11:56:24.300606       1 driver.go:123] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
  I0220 11:56:24.300609       1 driver.go:123] Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER
  I0220 11:56:24.300613       1 driver.go:123] Enabling volume access mode: SINGLE_NODE_MULTI_WRITER
  I0220 11:56:24.300666       1 driver.go:136] not enabling VolumeGroupSnapshot service capability
  I0220 11:56:24.300981       1 server.go:125] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi-provisioner.sock", Net:"unix"}
  I0220 11:56:24.300995       1 server.go:114] listening for CSI-Addons requests on address: &net.UnixAddr{Name:"/csi/csi-addons.sock", Net:"unix"}
  I0220 11:56:24.383336       1 utils.go:266] ID: 1 GRPC call: /csi.v1.Identity/Probe
  I0220 11:56:24.383359       1 utils.go:267] ID: 1 GRPC request: {}
  I0220 11:56:24.383426       1 utils.go:273] ID: 1 GRPC response: {}
  I0220 11:56:24.383973       1 utils.go:266] ID: 2 GRPC call: /csi.v1.Identity/GetPluginInfo
  I0220 11:56:24.383994       1 utils.go:267] ID: 2 GRPC request: {}
  I0220 11:56:24.384013       1 identityserver-default.go:40] ID: 2 Using default GetPluginInfo
  I0220 11:56:24.384142       1 utils.go:273] ID: 2 GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"canary"}
  I0220 11:56:24.385939       1 utils.go:266] ID: 3 GRPC call: /csi.v1.Identity/GetPluginCapabilities
  I0220 11:56:24.385976       1 utils.go:267] ID: 3 GRPC request: {}
  I0220 11:56:24.386152       1 utils.go:273] ID: 3 GRPC response: {"capabilities":[{"service":{"type":"CONTROLLER_SERVICE"}},{"volume_expansion":{"type":"ONLINE"}},{"service":{"type":"VOLUME_ACCESSIBILITY_CONSTRAINTS"}}]}
  I0220 11:56:24.386942       1 utils.go:266] ID: 4 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
  I0220 11:56:24.386963       1 utils.go:267] ID: 4 GRPC request: {}
  I0220 11:56:24.386977       1 controllerserver-default.go:42] ID: 4 Using default ControllerGetCapabilities
  I0220 11:56:24.387097       1 utils.go:273] ID: 4 GRPC response: {"capabilities":[{"rpc":{"type":"CREATE_DELETE_VOLUME"}},{"rpc":{"type":"CREATE_DELETE_SNAPSHOT"}},{"rpc":{"type":"CLONE_VOLUME"}},{"rpc":{"type":"EXPAND_VOLUME"}}]}
  I0220 11:56:24.549776       1 utils.go:266] ID: 5 GRPC call: /csi.v1.Identity/GetPluginInfo
  I0220 11:56:24.549815       1 utils.go:267] ID: 5 GRPC request: {}
  I0220 11:56:24.549844       1 identityserver-default.go:40] ID: 5 Using default GetPluginInfo
  I0220 11:56:24.549897       1 utils.go:273] ID: 5 GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"canary"}
  I0220 11:56:24.550661       1 utils.go:266] ID: 6 GRPC call: /csi.v1.Identity/Probe
  I0220 11:56:24.550679       1 utils.go:267] ID: 6 GRPC request: {}
  I0220 11:56:24.550691       1 utils.go:273] ID: 6 GRPC response: {}
  I0220 11:56:24.551069       1 utils.go:266] ID: 7 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
  I0220 11:56:24.551087       1 utils.go:267] ID: 7 GRPC request: {}
  I0220 11:56:24.551097       1 controllerserver-default.go:42] ID: 7 Using default ControllerGetCapabilities
  I0220 11:56:24.551139       1 utils.go:273] ID: 7 GRPC response: {"capabilities":[{"rpc":{"type":"CREATE_DELETE_VOLUME"}},{"rpc":{"type":"CREATE_DELETE_SNAPSHOT"}},{"rpc":{"type":"CLONE_VOLUME"}},{"rpc":{"type":"EXPAND_VOLUME"}}]}
  I0220 11:56:24.551547       1 utils.go:266] ID: 8 GRPC call: /csi.v1.GroupController/GroupControllerGetCapabilities
  I0220 11:56:24.551562       1 utils.go:267] ID: 8 GRPC request: {}
  I0220 11:56:24.551570       1 controllerserver-default.go:58] ID: 8 Using default GroupControllerGetCapabilities
  I0220 11:56:24.551604       1 utils.go:273] ID: 8 GRPC response: {}
  I0220 11:56:24.660602       1 utils.go:266] ID: 9 GRPC call: /csi.v1.Identity/Probe
  I0220 11:56:24.660632       1 utils.go:267] ID: 9 GRPC request: {}
  I0220 11:56:24.660644       1 utils.go:273] ID: 9 GRPC response: {}
  I0220 11:56:24.661354       1 utils.go:266] ID: 10 GRPC call: /csi.v1.Identity/GetPluginInfo
  I0220 11:56:24.661373       1 utils.go:267] ID: 10 GRPC request: {}
  I0220 11:56:24.661398       1 identityserver-default.go:40] ID: 10 Using default GetPluginInfo
  I0220 11:56:24.661426       1 utils.go:273] ID: 10 GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"canary"}
  I0220 11:56:24.661977       1 utils.go:266] ID: 11 GRPC call: /csi.v1.Identity/GetPluginCapabilities
  I0220 11:56:24.661997       1 utils.go:267] ID: 11 GRPC request: {}
  I0220 11:56:24.662041       1 utils.go:273] ID: 11 GRPC response: {"capabilities":[{"service":{"type":"CONTROLLER_SERVICE"}},{"volume_expansion":{"type":"ONLINE"}},{"service":{"type":"VOLUME_ACCESSIBILITY_CONSTRAINTS"}}]}
  I0220 11:56:24.662423       1 utils.go:266] ID: 12 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
  I0220 11:56:24.662439       1 utils.go:267] ID: 12 GRPC request: {}
  I0220 11:56:24.662463       1 controllerserver-default.go:42] ID: 12 Using default ControllerGetCapabilities
  I0220 11:56:24.662493       1 utils.go:273] ID: 12 GRPC response: {"capabilities":[{"rpc":{"type":"CREATE_DELETE_VOLUME"}},{"rpc":{"type":"CREATE_DELETE_SNAPSHOT"}},{"rpc":{"type":"CLONE_VOLUME"}},{"rpc":{"type":"EXPAND_VOLUME"}}]}
  I0220 11:56:24.770040       1 utils.go:266] ID: 13 GRPC call: /csi.v1.Identity/Probe
  I0220 11:56:24.770069       1 utils.go:267] ID: 13 GRPC request: {}
  I0220 11:56:24.770088       1 utils.go:273] ID: 13 GRPC response: {}
  I0220 11:56:24.770482       1 utils.go:266] ID: 14 GRPC call: /csi.v1.Identity/GetPluginInfo
  I0220 11:56:24.770504       1 utils.go:267] ID: 14 GRPC request: {}
  I0220 11:56:24.770514       1 identityserver-default.go:40] ID: 14 Using default GetPluginInfo
  I0220 11:56:24.770562       1 utils.go:273] ID: 14 GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"canary"}
  I0220 11:56:24.771112       1 utils.go:266] ID: 15 GRPC call: /csi.v1.Identity/GetPluginCapabilities
  I0220 11:56:24.771137       1 utils.go:267] ID: 15 GRPC request: {}
  I0220 11:56:24.771473       1 utils.go:273] ID: 15 GRPC response: {"capabilities":[{"service":{"type":"CONTROLLER_SERVICE"}},{"volume_expansion":{"type":"ONLINE"}},{"service":{"type":"VOLUME_ACCESSIBILITY_CONSTRAINTS"}}]}
  I0220 11:56:24.774028       1 utils.go:266] ID: 16 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
  I0220 11:56:24.774044       1 utils.go:267] ID: 16 GRPC request: {}
  I0220 11:56:24.774067       1 controllerserver-default.go:42] ID: 16 Using default ControllerGetCapabilities
  I0220 11:56:24.774101       1 utils.go:273] ID: 16 GRPC response: {"capabilities":[{"rpc":{"type":"CREATE_DELETE_VOLUME"}},{"rpc":{"type":"CREATE_DELETE_SNAPSHOT"}},{"rpc":{"type":"CLONE_VOLUME"}},{"rpc":{"type":"EXPAND_VOLUME"}}]}
  I0220 11:56:24.774430       1 utils.go:266] ID: 17 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
  I0220 11:56:24.774443       1 utils.go:267] ID: 17 GRPC request: {}
  I0220 11:56:24.774450       1 controllerserver-default.go:42] ID: 17 Using default ControllerGetCapabilities
  I0220 11:56:24.774479       1 utils.go:273] ID: 17 GRPC response: {"capabilities":[{"rpc":{"type":"CREATE_DELETE_VOLUME"}},{"rpc":{"type":"CREATE_DELETE_SNAPSHOT"}},{"rpc":{"type":"CLONE_VOLUME"}},{"rpc":{"type":"EXPAND_VOLUME"}}]}
  I0220 11:56:24.774884       1 utils.go:266] ID: 18 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
  I0220 11:56:24.774902       1 utils.go:267] ID: 18 GRPC request: {}
  I0220 11:56:24.774910       1 controllerserver-default.go:42] ID: 18 Using default ControllerGetCapabilities
  I0220 11:56:24.774943       1 utils.go:273] ID: 18 GRPC response: {"capabilities":[{"rpc":{"type":"CREATE_DELETE_VOLUME"}},{"rpc":{"type":"CREATE_DELETE_SNAPSHOT"}},{"rpc":{"type":"CLONE_VOLUME"}},{"rpc":{"type":"EXPAND_VOLUME"}}]}
  I0220 11:56:44.983216       1 utils.go:266] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a GRPC call: /csi.v1.Controller/CreateVolume
  I0220 11:56:44.983308       1 utils.go:267] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a","parameters":{"clusterID":"52ae6411-5b70-4fc1-93e3-9b0860bbff79","csi.storage.k8s.io/pv/name":"pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"rbd-1827","imageFeatures":"layering","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4","mount_flags":["discard"]}}]}
  I0220 11:56:44.983560       1 rbd_util.go:1361] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a setting disableInUseChecks: false image features: [layering] mounter: rbd
  E0220 11:56:45.010888       1 omap.go:80] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a omap not found (pool="replicapool", namespace="", name="csi.volumes.default"): rados: ret=-2, No such file or directory
  I0220 11:56:45.025751       1 omap.go:159] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a set omap keys (pool="replicapool", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a:2dbd1702-0f4b-4e8a-83de-b80b4927b70d])
  I0220 11:56:45.028474       1 omap.go:159] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a set omap keys (pool="replicapool", namespace="", name="csi.volume.2dbd1702-0f4b-4e8a-83de-b80b4927b70d"): map[csi.imagename:csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d csi.volname:pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a csi.volume.owner:rbd-1827])
  I0220 11:56:45.028501       1 rbd_journal.go:519] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a generated Volume ID (0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d) and image name (csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d) for request name (pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a)
  I0220 11:56:45.028560       1 rbd_util.go:443] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a rbd: create replicapool/csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d size 1024M (features: [layering]) using mon rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789
  I0220 11:56:45.028584       1 rbd_util.go:1615] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a setting image options on replicapool/csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d
  I0220 11:56:45.043015       1 controllerserver.go:793] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a created image replicapool/csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d backed for request name pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a
  I0220 11:56:45.060056       1 omap.go:159] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a set omap keys (pool="replicapool", namespace="", name="csi.volume.2dbd1702-0f4b-4e8a-83de-b80b4927b70d"): map[csi.imageid:13debe418b63])
  I0220 11:56:45.131275       1 utils.go:273] ID: 19 Req-ID: pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a GRPC response: {"volume":{"capacity_bytes":1073741824,"volume_context":{"clusterID":"52ae6411-5b70-4fc1-93e3-9b0860bbff79","imageFeatures":"layering","imageName":"csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d","journalPool":"replicapool","pool":"replicapool"},"volume_id":"0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d"}}
  I0220 11:56:58.487244       1 utils.go:266] ID: 20 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d GRPC call: /csi.v1.Controller/DeleteVolume
  I0220 11:56:58.487288       1 utils.go:267] ID: 20 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d GRPC request: {"secrets":"***stripped***","volume_id":"0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d"}
  I0220 11:56:58.493101       1 omap.go:89] ID: 20 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d got omap values: (pool="replicapool", namespace="", name="csi.volume.2dbd1702-0f4b-4e8a-83de-b80b4927b70d"): map[csi.imageid:13debe418b63 csi.imagename:csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d csi.volname:pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a csi.volume.owner:rbd-1827]
  I0220 11:56:58.546902       1 rbd_util.go:663] ID: 20 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d rbd: delete csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d-temp using mon rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789, pool replicapool
  I0220 11:56:58.551020       1 controllerserver.go:1083] ID: 20 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d deleting image csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d
  I0220 11:56:58.551039       1 rbd_util.go:663] ID: 20 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d rbd: delete csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d using mon rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789, pool replicapool
  I0220 11:56:58.580987       1 rbd_util.go:709] ID: 20 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d rbd: adding task to remove image "replicapool/csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d" with id "13debe418b63" from trash
  I0220 11:56:58.590142       1 rbd_util.go:737] ID: 20 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d rbd: successfully added task to move image "replicapool/csi-vol-2dbd1702-0f4b-4e8a-83de-b80b4927b70d" with id "13debe418b63" to trash
  I0220 11:56:58.595741       1 omap.go:126] ID: 20 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d removed omap keys (pool="replicapool", namespace="", name="csi.volumes.default"): [csi.volume.pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a]
  I0220 11:56:58.595826       1 utils.go:273] ID: 20 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-2dbd1702-0f4b-4e8a-83de-b80b4927b70d GRPC response: {}
  I0220 11:57:00.489721       1 utils.go:266] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 GRPC call: /csi.v1.Controller/CreateVolume
  I0220 11:57:00.489818       1 utils.go:267] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-ca574bf2-9632-49b9-8125-66951fc004c2","parameters":{"clusterID":"52ae6411-5b70-4fc1-93e3-9b0860bbff79","csi.storage.k8s.io/pv/name":"pvc-ca574bf2-9632-49b9-8125-66951fc004c2","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"rbd-1827","imageFeatures":"layering","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4","mount_flags":["discard"]}}]}
  I0220 11:57:00.489984       1 rbd_util.go:1361] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 setting disableInUseChecks: false image features: [layering] mounter: rbd
  I0220 11:57:00.492422       1 omap.go:89] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 got omap values: (pool="replicapool", namespace="", name="csi.volumes.default"): map[]
  I0220 11:57:00.499023       1 omap.go:159] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 set omap keys (pool="replicapool", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-ca574bf2-9632-49b9-8125-66951fc004c2:903e957e-d065-428a-9c2a-3f8a65dcf986])
  I0220 11:57:00.502158       1 omap.go:159] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 set omap keys (pool="replicapool", namespace="", name="csi.volume.903e957e-d065-428a-9c2a-3f8a65dcf986"): map[csi.imagename:csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986 csi.volname:pvc-ca574bf2-9632-49b9-8125-66951fc004c2 csi.volume.owner:rbd-1827])
  I0220 11:57:00.502185       1 rbd_journal.go:519] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 generated Volume ID (0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986) and image name (csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986) for request name (pvc-ca574bf2-9632-49b9-8125-66951fc004c2)
  I0220 11:57:00.502241       1 rbd_util.go:443] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 rbd: create replicapool/csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986 size 1024M (features: [layering]) using mon rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789
  I0220 11:57:00.502261       1 rbd_util.go:1615] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 setting image options on replicapool/csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986
  I0220 11:57:00.516391       1 controllerserver.go:793] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 created image replicapool/csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986 backed for request name pvc-ca574bf2-9632-49b9-8125-66951fc004c2
  I0220 11:57:00.532369       1 omap.go:159] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 set omap keys (pool="replicapool", namespace="", name="csi.volume.903e957e-d065-428a-9c2a-3f8a65dcf986"): map[csi.imageid:13de59275098])
  I0220 11:57:00.600039       1 utils.go:273] ID: 21 Req-ID: pvc-ca574bf2-9632-49b9-8125-66951fc004c2 GRPC response: {"volume":{"capacity_bytes":1073741824,"volume_context":{"clusterID":"52ae6411-5b70-4fc1-93e3-9b0860bbff79","imageFeatures":"layering","imageName":"csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986","journalPool":"replicapool","pool":"replicapool"},"volume_id":"0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986"}}
  I0220 11:57:06.564989       1 utils.go:266] ID: 22 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986 GRPC call: /csi.v1.Controller/DeleteVolume
  I0220 11:57:06.565037       1 utils.go:267] ID: 22 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986 GRPC request: {"secrets":"***stripped***","volume_id":"0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986"}
  I0220 11:57:06.566155       1 omap.go:89] ID: 22 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986 got omap values: (pool="replicapool", namespace="", name="csi.volume.903e957e-d065-428a-9c2a-3f8a65dcf986"): map[csi.imageid:13de59275098 csi.imagename:csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986 csi.volname:pvc-ca574bf2-9632-49b9-8125-66951fc004c2 csi.volume.owner:rbd-1827]
  I0220 11:57:06.618942       1 rbd_util.go:663] ID: 22 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986 rbd: delete csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986-temp using mon rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789, pool replicapool
  I0220 11:57:06.622427       1 controllerserver.go:1083] ID: 22 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986 deleting image csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986
  I0220 11:57:06.622448       1 rbd_util.go:663] ID: 22 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986 rbd: delete csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986 using mon rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789, pool replicapool
  I0220 11:57:06.651315       1 rbd_util.go:709] ID: 22 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986 rbd: adding task to remove image "replicapool/csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986" with id "13de59275098" from trash
  I0220 11:57:06.657889       1 rbd_util.go:737] ID: 22 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986 rbd: successfully added task to move image "replicapool/csi-vol-903e957e-d065-428a-9c2a-3f8a65dcf986" with id "13de59275098" to trash
  I0220 11:57:06.663192       1 omap.go:126] ID: 22 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986 removed omap keys (pool="replicapool", namespace="", name="csi.volumes.default"): [csi.volume.pvc-ca574bf2-9632-49b9-8125-66951fc004c2]
  I0220 11:57:06.663267       1 utils.go:273] ID: 22 Req-ID: 0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-903e957e-d065-428a-9c2a-3f8a65dcf986 GRPC response: {}
  I0220 11:57:08.565773       1 utils.go:266] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb GRPC call: /csi.v1.Controller/CreateVolume
  I0220 11:57:08.565878       1 utils.go:267] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb","parameters":{"clusterID":"52ae6411-5b70-4fc1-93e3-9b0860bbff79","csi.storage.k8s.io/pv/name":"pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"rbd-1827","imageFeatures":"layering","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4","mount_flags":["discard"]}}]}
  I0220 11:57:08.566067       1 rbd_util.go:1361] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb setting disableInUseChecks: false image features: [layering] mounter: rbd
  I0220 11:57:08.566781       1 omap.go:89] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb got omap values: (pool="replicapool", namespace="", name="csi.volumes.default"): map[]
  I0220 11:57:08.573131       1 omap.go:159] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb set omap keys (pool="replicapool", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb:aec40a24-45ed-4919-97ca-b2c5b00517b5])
  I0220 11:57:08.576000       1 omap.go:159] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb set omap keys (pool="replicapool", namespace="", name="csi.volume.aec40a24-45ed-4919-97ca-b2c5b00517b5"): map[csi.imagename:csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5 csi.volname:pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb csi.volume.owner:rbd-1827])
  I0220 11:57:08.576025       1 rbd_journal.go:519] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb generated Volume ID (0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-aec40a24-45ed-4919-97ca-b2c5b00517b5) and image name (csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5) for request name (pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb)
  I0220 11:57:08.576087       1 rbd_util.go:443] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb rbd: create replicapool/csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5 size 1024M (features: [layering]) using mon rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789
  I0220 11:57:08.576107       1 rbd_util.go:1615] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb setting image options on replicapool/csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5
  I0220 11:57:08.591192       1 controllerserver.go:793] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb created image replicapool/csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5 backed for request name pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb
  I0220 11:57:08.606093       1 omap.go:159] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb set omap keys (pool="replicapool", namespace="", name="csi.volume.aec40a24-45ed-4919-97ca-b2c5b00517b5"): map[csi.imageid:13de5595a987])
  I0220 11:57:08.678644       1 utils.go:273] ID: 23 Req-ID: pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb GRPC response: {"volume":{"capacity_bytes":1073741824,"volume_context":{"clusterID":"52ae6411-5b70-4fc1-93e3-9b0860bbff79","imageFeatures":"layering","imageName":"csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5","journalPool":"replicapool","pool":"replicapool"},"volume_id":"0001-0024-52ae6411-5b70-4fc1-93e3-9b0860bbff79-0000000000000004-aec40a24-45ed-4919-97ca-b2c5b00517b5"}}


  ENDLOG for container cephcsi-e2e-43fda518:csi-rbdplugin-provisioner-6d88977bd8-zst6h:csi-rbdplugin

@Rakshith-R
Copy link
Contributor Author

csi-rbdplugin-provisioner-84c67cdf4-2mwf6

There was another provisioner pod which was scaled down and then up to change cmdline args which may have caused initial request omap leak since the id is sent by external-provisioner sidecar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working component/rbd Issues related to RBD
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Rakshith-R