Kernel crash while taking snapshot [dattobd 0.11.3]

[vishnukirannemani]

While taking the snapshot on the source server, I see the following kernel crash in the system logs:

---

Feb 4 13:14:20 mccd-mlssource3 kernel: Unexpected gfp: 0xff800006 (__GFP_HIGHMEM|GFP_DMA32|0xff800000). Fixing up to gfp: 0x7ffff9 (GFP_USER|GFP_DMA|__GFP_HIGH|__GFP_ATOMIC|__GFP_NOWARN|__GFP_RETRY_MAYFAIL|__GFP_NOFAIL|__GFP_NORETRY|__GFP_COMP|__GFP_ZERO|__GFP_NOMEMALLOC|__GFP_MEMALLOC|__GFP_THISNODE|__GFP_RECLAIMABLE|__GFP_MOVABLE|__GFP_ACCOUNT|__GFP_WRITE). Fix your code!
Feb 4 13:14:20 mccd-mlssource3 kernel: CPU: 3 PID: 1337474 Comm: ***** Kdump: loaded Tainted: G OE --------- - - 4.18.0-147.el8.x86_64 #1
Feb 4 13:14:20 mccd-mlssource3 kernel: Hardware name: VMware, Inc. VMware20,1/440BX Desktop Reference Platform, BIOS VMW201.00V.21805430.B64.2305221830 05/22/2023
Feb 4 13:14:20 mccd-mlssource3 kernel: Call Trace:
Feb 4 13:14:20 mccd-mlssource3 kernel: dump_stack+0x5c/0x80
Feb 4 13:14:20 mccd-mlssource3 kernel: new_slab.cold.96+0x2a/0x33
Feb 4 13:14:20 mccd-mlssource3 kernel: ___slab_alloc+0x36b/0x4e0
Feb 4 13:14:20 mccd-mlssource3 kernel: ? blk_alloc_queue_node+0x27/0x280
Feb 4 13:14:20 mccd-mlssource3 kernel: ? pcpu_next_unpop+0x37/0x50
Feb 4 13:14:20 mccd-mlssource3 kernel: ? cpumask_next+0x17/0x20
Feb 4 13:14:20 mccd-mlssource3 kernel: ? pcpu_alloc+0x321/0x640
Feb 4 13:14:20 mccd-mlssource3 kernel: ? blk_alloc_queue_node+0x27/0x280
Feb 4 13:14:20 mccd-mlssource3 kernel: __slab_alloc+0x1c/0x30
Feb 4 13:14:20 mccd-mlssource3 kernel: kmem_cache_alloc_node+0xa8/0x1f0
Feb 4 13:14:20 mccd-mlssource3 kernel: blk_alloc_queue_node+0x27/0x280
Feb 4 13:14:20 mccd-mlssource3 kernel: __tracer_setup_snap+0x84/0x250 [dattobd]
Feb 4 13:14:20 mccd-mlssource3 kernel: tracer_setup_active_snap+0xd0/0x130 [dattobd]
Feb 4 13:14:20 mccd-mlssource3 kernel: __ioctl_setup+0xbc/0x140 [dattobd]
Feb 4 13:14:20 mccd-mlssource3 kernel: ctrl_ioctl+0x2f4/0x410 [dattobd]
Feb 4 13:14:20 mccd-mlssource3 kernel: do_vfs_ioctl+0xa4/0x630
Feb 4 13:14:20 mccd-mlssource3 kernel: ksys_ioctl+0x60/0x90
Feb 4 13:14:20 mccd-mlssource3 kernel: __x64_sys_ioctl+0x16/0x20
Feb 4 13:14:20 mccd-mlssource3 kernel: do_syscall_64+0x5b/0x1b0
Feb 4 13:14:20 mccd-mlssource3 kernel: entry_SYSCALL_64_after_hwframe+0x65/0xca
Feb 4 13:14:20 mccd-mlssource3 kernel: RIP: 0033:0xa645c7
Feb 4 13:14:20 mccd-mlssource3 kernel: Code: 48 83 c4 08 48 89 d8 5b 5d c3 66 0f 1f 84 00 00 00 00 00 48 89 e8 48 f7 d8 48 39 c3 0f 92 c0 eb 92 66 90 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 0f 83 3d b3 f9 ff c3 66 2e 0f 1f 84 00 00 00 00
Feb 4 13:14:20 mccd-mlssource3 kernel: RSP: 002b:00007fff42191418 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
Feb 4 13:14:20 mccd-mlssource3 kernel: RAX: ffffffffffffffda RBX: 000000c0001bc448 RCX: 0000000000a645c7
Feb 4 13:14:20 mccd-mlssource3 kernel: RDX: 00007fff42191460 RSI: 0000000040289101 RDI: 000000000000000f
Feb 4 13:14:20 mccd-mlssource3 kernel: RBP: 00007fff42191490 R08: 0000000000000000 R09: 0000000000000000
Feb 4 13:14:20 mccd-mlssource3 kernel: R10: 0000000000000070 R11: 0000000000000206 R12: 0000000000000000
Feb 4 13:14:20 mccd-mlssource3 kernel: R13: 00000000013d6940 R14: 000000c000247a00 R15: 00000000000000c8

---

dattobd version : 0.11.3
kernel version : 4.18.0-147.el8.x86_64

os-info

NAME="Red Hat Enterprise Linux"
VERSION="8.10 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.10 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.10
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.10"

---

I suspect that there is a memory allocation issue while taking the snapshot. Not able to understand what needs to be done here. Due to this after replication I am not able to boot up the target server(it goes into shell mode).

[yito24]

@vishnukirannemani

The driver version is old.
You might want to try the latest driver(0.11.10)?
I have previously tested RHEL8.9 and 0.11.3, occur the kernel crash.

[iamandrii]

Hello, @vishnukirannemani.

The problem you describe, was completely fixed in DattoBD 0.11.10. Please update, and reach me back, to ensure you have no more issues.

Thanks

[Yukesh-Baskar]

Hi @iamandrii

I've tested with version 0.11.10, while creating snapshot I don't see any kernel crashes on the system logs.
But while deleting the snapshot I'm getting these errors on the system logs.

Feb 6 10:44:05 source~ kernel: datto: device specified already exists: -16
Feb 6 10:44:05 source~ kernel: datto: verify_minor_available: -16
Feb 6 10:44:05 source~ kernel: datto: error during setup ioctl handler: -16
Feb 6 10:44:29 source~ kernel: datto: closing file that is still unlocked
Feb 6 10:44:29 source~ kernel: datto: closing file that is still unlocked
Feb 6 10:44:29 source~ kernel: VFS: Close: file count is 0

kernel version : 4.18.0-147.el8.x86_64

os-info

NAME="Red Hat Enterprise Linux"
VERSION="8.10 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.10 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.10
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.10"

[yito24]

@Yukesh-Baskar

Why are you using the 8.1 kernel?

[yito24]

@Yukesh-Baskar

I tested 0.10.11 on RHEL8.1.
It works absolutely without any problems.

kernel: 4.18.0-147.el8.x86_64
Red Hat Enterprise Linux release 8.1 (Ootpa)
"version": "0.11.10",

[iamandrii]

Hello, please recheck with newer version 0.12.0. Thanks.