Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

zfs_inode_update() NULL pointer deref #3805

Closed
adlerd opened this issue Sep 19, 2015 · 2 comments
Closed

zfs_inode_update() NULL pointer deref #3805

adlerd opened this issue Sep 19, 2015 · 2 comments
Milestone
0.8.0

Comments

@adlerd
Copy link

adlerd commented Sep 19, 2015

When importing and implicitly mounting my pool under zfs 0.6.5, linux 4.2.0, I get the following dmesg output and the pool refuses to mount any datasets or export. The stderr of zpool import reports nothing more than that there was an i/o error, and then zpool export complains that the pool is busy.
zpool iostat reports some write actions as well, which take a very long time to complete or never complete.
zpool scrub appears to work fine and reports that the pool is clean.

Sep 19 13:37:45 lodoghir kernel: BUG: unable to handle kernel NULL pointer dereference at 000000000000033f
Sep 19 13:37:45 lodoghir kernel: IP: [] zfs_inode_update+0x21/0x180
Sep 19 13:37:45 lodoghir kernel: PGD 404b0c067 PUD 403d5d067 PMD 0
Sep 19 13:37:45 lodoghir kernel: Oops: 0000 [#1] SMP
Sep 19 13:37:45 lodoghir kernel: CPU: 3 PID: 3211 Comm: mount.zfs Not tainted 4.2.0 #13
Sep 19 13:37:45 lodoghir kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.0 04/24/2015
Sep 19 13:37:45 lodoghir kernel: task: ffff88040d16ab00 ti: ffff8804043e8000 task.ti: ffff8804043e8000
Sep 19 13:37:45 lodoghir kernel: RIP: 0010:[] [] zfs_inode_update+0x21/0x180
Sep 19 13:37:45 lodoghir kernel: RSP: 0018:ffff8804043eb9c8 EFLAGS: 00010292
Sep 19 13:37:45 lodoghir kernel: RAX: ffffffffffffffff RBX: ffff880404388000 RCX: 0000000000000001
Sep 19 13:37:45 lodoghir kernel: RDX: 0000000000000001 RSI: ffff880406246428 RDI: ffff880404388218
Sep 19 13:37:45 lodoghir kernel: RBP: ffff8804043eba28 R08: ffff8804043ebba0 R09: 0000000000000008
Sep 19 13:37:45 lodoghir kernel: R10: 0000000000000010 R11: 0000000000000008 R12: ffff880404388218
Sep 19 13:37:45 lodoghir kernel: R13: 0000000000000004 R14: 0000000000000000 R15: ffff880404388218
Sep 19 13:37:45 lodoghir kernel: FS: 00007f028cfc6780(0000) GS:ffff88041fcc0000(0000) knlGS:0000000000000000
Sep 19 13:37:45 lodoghir kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 19 13:37:45 lodoghir kernel: CR2: 000000000000033f CR3: 0000000404370000 CR4: 00000000001406e0
Sep 19 13:37:45 lodoghir kernel: Stack:
Sep 19 13:37:45 lodoghir kernel: ffff88040404d2b0 ffff880403d5e000 ffff880403d5e090 ffff8804043eba60
Sep 19 13:37:45 lodoghir kernel: 0000000000000009 ffff880404388218 ffff8804043eba28 ffffffff811cc258
Sep 19 13:37:45 lodoghir kernel: ffff88040404d000 ffff88040404d000 ffff880404388000 0000000000000004
Sep 19 13:37:45 lodoghir kernel: Call Trace:
Sep 19 13:37:45 lodoghir kernel: [] ? sa_bulk_lookup+0x68/0x80
Sep 19 13:37:45 lodoghir kernel: [] zfs_znode_alloc+0x2e9/0x510
Sep 19 13:37:45 lodoghir kernel: [] zfs_zget+0x280/0x2f0
Sep 19 13:37:45 lodoghir kernel: [] zfs_root+0x65/0xa0
Sep 19 13:37:45 lodoghir kernel: [] zfs_domount+0x251/0x320
Sep 19 13:37:45 lodoghir kernel: [] ? get_anon_bdev+0xd0/0xd0
Sep 19 13:37:45 lodoghir kernel: [] ? zpl_kill_sb+0x20/0x20
Sep 19 13:37:45 lodoghir kernel: [] zpl_fill_super+0x27/0x40
Sep 19 13:37:45 lodoghir kernel: [] mount_nodev+0x48/0xa0
Sep 19 13:37:45 lodoghir kernel: [] zpl_mount+0x51/0x80
Sep 19 13:37:45 lodoghir kernel: [] mount_fs+0x14/0xa0
Sep 19 13:37:45 lodoghir kernel: [] vfs_kern_mount+0x66/0x110
Sep 19 13:37:45 lodoghir kernel: [] do_mount+0x1e5/0xc10
Sep 19 13:37:45 lodoghir kernel: [] ? __get_free_pages+0x12/0x40
Sep 19 13:37:45 lodoghir kernel: [] ? copy_mount_options+0x35/0x150
Sep 19 13:37:45 lodoghir kernel: [] SyS_mount+0x86/0xd0
Sep 19 13:37:45 lodoghir kernel: [] entry_SYSCALL_64_fastpath+0x12/0x6a
Sep 19 13:37:45 lodoghir kernel: Code: 83 c8 04 89 46 0c 5d c3 66 90 55 48 89 e5 41 55 41 54 4c 8d a7 18 02 00 00 53 48 89 fb 48 83 ec 48 48 8b 87 30 02 00 00 4c 89 e7 <4c> 8b a8 40 03 00 00 e8 13 12 fe ff 85 c0 74 0b 48 83 c4 48 5b
Sep 19 13:37:45 lodoghir kernel: RIP [] zfs_inode_update+0x21/0x180
Sep 19 13:37:45 lodoghir kernel: RSP
Sep 19 13:37:45 lodoghir kernel: CR2: 000000000000033f
Sep 19 13:37:45 lodoghir kernel: ---[ end trace 62e1d27c12df8d72 ]---
Sep 19 13:37:45 lodoghir kernel: BUG: unable to handle kernel NULL pointer dereference at 000000000000033f
Sep 19 13:37:45 lodoghir kernel: IP: [] zfs_inode_update+0x21/0x180
Sep 19 13:37:45 lodoghir kernel: PGD 404374067 PUD 404373067 PMD 0
Sep 19 13:37:45 lodoghir kernel: Oops: 0000 [#2] SMP
Sep 19 13:37:45 lodoghir kernel: CPU: 3 PID: 3215 Comm: mount.zfs Tainted: G D 4.2.0 #13
Sep 19 13:37:45 lodoghir kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.0 04/24/2015
Sep 19 13:37:45 lodoghir kernel: task: ffff880407dfa040 ti: ffff880404380000 task.ti: ffff880404380000
Sep 19 13:37:45 lodoghir kernel: RIP: 0010:[] [] zfs_inode_update+0x21/0x180
Sep 19 13:37:45 lodoghir kernel: RSP: 0018:ffff8804043839c8 EFLAGS: 00010292
Sep 19 13:37:45 lodoghir kernel: RAX: ffffffffffffffff RBX: ffff880404388420 RCX: 0000000000000001
Sep 19 13:37:45 lodoghir kernel: RDX: 0000000000000001 RSI: ffff880406245a28 RDI: ffff880404388638
Sep 19 13:37:45 lodoghir kernel: RBP: ffff880404383a28 R08: ffff880404383ba0 R09: 0000000000000008
Sep 19 13:37:45 lodoghir kernel: R10: 0000000000000010 R11: 0000000000000008 R12: ffff880404388638
Sep 19 13:37:45 lodoghir kernel: R13: 0000000000000004 R14: 0000000000000000 R15: ffff880404388638
Sep 19 13:37:45 lodoghir kernel: FS: 00007f06cd745780(0000) GS:ffff88041fcc0000(0000) knlGS:0000000000000000
Sep 19 13:37:45 lodoghir kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 19 13:37:45 lodoghir kernel: CR2: 000000000000033f CR3: 0000000404371000 CR4: 00000000001406e0
Sep 19 13:37:45 lodoghir kernel: Stack:
Sep 19 13:37:45 lodoghir kernel: ffff8804040492b0 ffff880403d5e0f8 ffff880403d5e188 ffff880404383a60
Sep 19 13:37:45 lodoghir kernel: 0000000000000009 ffff880404388638 ffff880404383a28 ffffffff811cc258
Sep 19 13:37:45 lodoghir kernel: ffff880404049000 ffff880404049000 ffff880404388420 0000000000000004
Sep 19 13:37:45 lodoghir kernel: Call Trace:
Sep 19 13:37:45 lodoghir kernel: [] ? sa_bulk_lookup+0x68/0x80
Sep 19 13:37:45 lodoghir kernel: [] zfs_znode_alloc+0x2e9/0x510
Sep 19 13:37:45 lodoghir kernel: [] zfs_zget+0x280/0x2f0
Sep 19 13:37:45 lodoghir kernel: [] zfs_root+0x65/0xa0
Sep 19 13:37:45 lodoghir kernel: [] zfs_domount+0x251/0x320
Sep 19 13:37:45 lodoghir kernel: [] ? get_anon_bdev+0xd0/0xd0
Sep 19 13:37:45 lodoghir kernel: [] ? zpl_kill_sb+0x20/0x20
Sep 19 13:37:45 lodoghir kernel: [] zpl_fill_super+0x27/0x40
Sep 19 13:37:45 lodoghir kernel: [] mount_nodev+0x48/0xa0
Sep 19 13:37:45 lodoghir kernel: [] zpl_mount+0x51/0x80
Sep 19 13:37:45 lodoghir kernel: [] mount_fs+0x14/0xa0
Sep 19 13:37:45 lodoghir kernel: [] vfs_kern_mount+0x66/0x110
Sep 19 13:37:45 lodoghir kernel: [] do_mount+0x1e5/0xc10
Sep 19 13:37:45 lodoghir kernel: [] ? __get_free_pages+0x12/0x40
Sep 19 13:37:45 lodoghir kernel: [] ? copy_mount_options+0x35/0x150
Sep 19 13:37:45 lodoghir kernel: [] SyS_mount+0x86/0xd0
Sep 19 13:37:45 lodoghir kernel: [] entry_SYSCALL_64_fastpath+0x12/0x6a
Sep 19 13:37:45 lodoghir kernel: Code: 83 c8 04 89 46 0c 5d c3 66 90 55 48 89 e5 41 55 41 54 4c 8d a7 18 02 00 00 53 48 89 fb 48 83 ec 48 48 8b 87 30 02 00 00 4c 89 e7 <4c> 8b a8 40 03 00 00 e8 13 12 fe ff 85 c0 74 0b 48 83 c4 48 5b
Sep 19 13:37:45 lodoghir kernel: RIP [] zfs_inode_update+0x21/0x180
Sep 19 13:37:45 lodoghir kernel: RSP
Sep 19 13:37:45 lodoghir kernel: CR2: 000000000000033f
Sep 19 13:37:45 lodoghir kernel: ---[ end trace 62e1d27c12df8d73 ]---
Sep 19 13:37:45 lodoghir kernel: BUG: unable to handle kernel NULL pointer dereference at 000000000000033f
Sep 19 13:37:45 lodoghir kernel: IP: [] zfs_inode_update+0x21/0x180
Sep 19 13:37:45 lodoghir kernel: PGD 404d50067 PUD 4045fc067 PMD 0
Sep 19 13:37:45 lodoghir kernel: Oops: 0000 [#3] SMP
Sep 19 13:37:45 lodoghir kernel: CPU: 0 PID: 3219 Comm: mount.zfs Tainted: G D 4.2.0 #13
Sep 19 13:37:45 lodoghir kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.0 04/24/2015
Sep 19 13:37:45 lodoghir kernel: task: ffff88040cd95600 ti: ffff8804043d8000 task.ti: ffff8804043d8000
Sep 19 13:37:45 lodoghir kernel: RIP: 0010:[] [] zfs_inode_update+0x21/0x180
Sep 19 13:37:45 lodoghir kernel: RSP: 0018:ffff8804043db9c8 EFLAGS: 00010292
Sep 19 13:37:45 lodoghir kernel: RAX: ffffffffffffffff RBX: ffff880403da8000 RCX: 0000000000000001
Sep 19 13:37:45 lodoghir kernel: RDX: 0000000000000001 RSI: ffff88040a25c228 RDI: ffff880403da8218
Sep 19 13:37:45 lodoghir kernel: RBP: ffff8804043dba28 R08: ffff8804043dbba0 R09: 0000000000000008
Sep 19 13:37:45 lodoghir kernel: R10: 0000000000000010 R11: 0000000000000008 R12: ffff880403da8218
Sep 19 13:37:45 lodoghir kernel: R13: 0000000000000004 R14: 0000000000000000 R15: ffff880403da8218
Sep 19 13:37:45 lodoghir kernel: FS: 00007fb3d2a6f780(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000
Sep 19 13:37:45 lodoghir kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 19 13:37:45 lodoghir kernel: CR2: 000000000000033f CR3: 0000000404370000 CR4: 00000000001406f0
Sep 19 13:37:45 lodoghir kernel: Stack:
Sep 19 13:37:45 lodoghir kernel: ffff88040403a2b0 ffff88040406c000 ffff88040406c090 ffff8804043dba60
Sep 19 13:37:45 lodoghir kernel: 0000000000000009 ffff880403da8218 ffff8804043dba28 ffffffff811cc258
Sep 19 13:37:45 lodoghir kernel: ffff88040403a000 ffff88040403a000 ffff880403da8000 0000000000000004
Sep 19 13:37:45 lodoghir kernel: Call Trace:
Sep 19 13:37:45 lodoghir kernel: [] ? sa_bulk_lookup+0x68/0x80
Sep 19 13:37:45 lodoghir kernel: [] zfs_znode_alloc+0x2e9/0x510
Sep 19 13:37:45 lodoghir kernel: [] zfs_zget+0x280/0x2f0
Sep 19 13:37:45 lodoghir kernel: [] zfs_root+0x65/0xa0
Sep 19 13:37:45 lodoghir kernel: [] zfs_domount+0x251/0x320
Sep 19 13:37:45 lodoghir kernel: [] ? get_anon_bdev+0xd0/0xd0
Sep 19 13:37:45 lodoghir kernel: [] ? zpl_kill_sb+0x20/0x20
Sep 19 13:37:45 lodoghir kernel: [] zpl_fill_super+0x27/0x40
Sep 19 13:37:45 lodoghir kernel: [] mount_nodev+0x48/0xa0
Sep 19 13:37:45 lodoghir kernel: [] zpl_mount+0x51/0x80
Sep 19 13:37:45 lodoghir kernel: [] mount_fs+0x14/0xa0
Sep 19 13:37:45 lodoghir kernel: [] vfs_kern_mount+0x66/0x110
Sep 19 13:37:45 lodoghir kernel: [] do_mount+0x1e5/0xc10
Sep 19 13:37:45 lodoghir kernel: [] ? __get_free_pages+0x12/0x40
Sep 19 13:37:45 lodoghir kernel: [] ? copy_mount_options+0x35/0x150
Sep 19 13:37:45 lodoghir kernel: [] SyS_mount+0x86/0xd0
Sep 19 13:37:45 lodoghir kernel: [] entry_SYSCALL_64_fastpath+0x12/0x6a
Sep 19 13:37:45 lodoghir kernel: Code: 83 c8 04 89 46 0c 5d c3 66 90 55 48 89 e5 41 55 41 54 4c 8d a7 18 02 00 00 53 48 89 fb 48 83 ec 48 48 8b 87 30 02 00 00 4c 89 e7 <4c> 8b a8 40 03 00 00 e8 13 12 fe ff 85 c0 74 0b 48 83 c4 48 5b
Sep 19 13:37:45 lodoghir kernel: RIP [] zfs_inode_update+0x21/0x180
Sep 19 13:37:45 lodoghir kernel: RSP
Sep 19 13:37:45 lodoghir kernel: CR2: 000000000000033f
Sep 19 13:37:45 lodoghir kernel: ---[ end trace 62e1d27c12df8d74 ]---
@adlerd
Copy link
Author

adlerd commented Sep 19, 2015

This continued to repro across several reboots. I recovered the pool by mounting it in 0.6.3 (good thing the new features aren't active! I wish I'd created the pool without them...); now after a kernel rebuild the problem doesn't repro in 0.6.5.
It's probably still worth a look though, since (a) it means complete inability to access the data (although I don't think there was data loss) and (b) it didn't repro in an older version, indicating a probable new bug.

@behlendorf
Copy link
Contributor

@adlerd the stack shows a NULL pointer dereference while attempting to create the root inode. I can't really explain how that's possible and it certainly shouldn't have caused any data issues. Unfortunately, there's not a lot to dig in to here but we can certainly leave it open for now in case someone else hits something similar.

@behlendorf behlendorf changed the title Null pointer deref in kernel; unable to mount or export 0.6.5 zfs_inode_update() NULL pointer deref Sep 22, 2015
@behlendorf behlendorf added this to the 0.7.0 milestone Sep 22, 2015
@behlendorf behlendorf modified the milestones: 0.8.0, 0.7.0 Mar 26, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.8.0
Development

No branches or pull requests
2 participants
@behlendorf @adlerd