Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zpool create - internal error: out of memory #13538

Closed
gchmurka123 opened this issue Jun 8, 2022 · 27 comments
Closed

zpool create - internal error: out of memory #13538

gchmurka123 opened this issue Jun 8, 2022 · 27 comments
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@gchmurka123
Copy link

gchmurka123 commented Jun 8, 2022

System information

Debian
x86_64-linux-gnu
zfs-2.1.4-1
zfs-kmod-2.1.4-1
Linux chmura 4.9.316 #5 SMP Tue Jun 7 13:02:00 CEST 2022 x86_64 GNU/Linux

Describe the problem you're observing

I am facing the development of a new zfs for very old systems with
kernel 4.9.316 (new build)
glibc 2.13
gcc version 4.7.2 (Debian 4.7.2-5)

Everything builds correctly, the zfs kernel module works, I can import the old pool created on zfs 0.8.6, update them, the tools work (like zfs create, destroy, snapshot, etc ) except for one command:
zpool create test device (or file)
always return
internal error: out of memory

Describe how to reproduce the problem

cd /build/zfs/zfs-2.1.4
./autogen.sh
./configure --disable-pyzfs --disable-systemd --with-linux=/build/linux-4.9.316 --with-linux-obj=/build/linux-4.9.316
make
make install

after make install (or make deb and install packages) command
zpool create test /dev/sdb
returns
internal error: out of memory

built old versions of zfs 0.8.6 work perfectly fine

@gchmurka123 gchmurka123 added the Type: Defect Incorrect behavior (e.g. crash, hang) label Jun 8, 2022
@mcmilk
Copy link
Contributor

mcmilk commented Jun 10, 2022

How much RAM do you have. Is an OOM (out of memory) ... maybe just a OOM ?
What does free tell you about this?
Can you maybe do some strace zpool create test /dev/sdb ... and check what causes this OOM?

@gchmurka123
Copy link
Author

gchmurka123 commented Jun 10, 2022

The ram is not problem
asd/root# free -m
total used free shared buff/cache available
Mem: 3940 1973 1273 0 693 1908
Swap: 1980 320 1660

and strace in attached file from create pool on 1GB file (real device like /dev/sdb, or other device same error)

zpool-create.log

@mcmilk
Copy link
Contributor

mcmilk commented Jun 10, 2022

I see no failures there :-(
Maybe I can setup a small box with the same parameters.. what distribution do you have?
The kernel seems also self compiled?

@gchmurka123
Copy link
Author

gchmurka123 commented Jun 10, 2022

I change all out of memory errors to my debug messeage, and found place in source where the error is generate
zfs/zfs-2.1.4/lib/libzfs/libzfs_util.c:748 - 755

yes, kerenel is self build, but is vanilla (no extra patches included). I used kernel 4.4 before and on zfs 2.x i had the same problem with zpool ;( , 0.8.6 no problem on both 4.4 and 4.9

@mcmilk
Copy link
Contributor

mcmilk commented Jun 10, 2022

This is the source where the error is reported... can you try to patch you zfs with this diff:

diff --git a/include/libzfs_impl.h b/include/libzfs_impl.h
index 96b11dad1..45cba3e90 100644
--- a/include/libzfs_impl.h
+++ b/include/libzfs_impl.h
@@ -142,7 +142,10 @@ void *zfs_alloc(libzfs_handle_t *, size_t);
 void *zfs_realloc(libzfs_handle_t *, void *, size_t, size_t);
 char *zfs_asprintf(libzfs_handle_t *, const char *, ...);
 char *zfs_strdup(libzfs_handle_t *, const char *);
-int no_memory(libzfs_handle_t *);
+
+#define no_memory(h) no_memory_real(h, __func__, __LINE__)
+int no_memory_real(libzfs_handle_t *, const char *fn, int line);
+

 int zfs_standard_error(libzfs_handle_t *, int, const char *);
 int zfs_standard_error_fmt(libzfs_handle_t *, int, const char *, ...);
diff --git a/lib/libzfs/libzfs_util.c b/lib/libzfs/libzfs_util.c
index 7dd38bb3d..ce4f44276 100644
--- a/lib/libzfs/libzfs_util.c
+++ b/lib/libzfs/libzfs_util.c
@@ -749,9 +749,9 @@ zpool_standard_error_fmt(libzfs_handle_t *hdl, int error, const char *fmt, ...)
  * Display an out of memory error message and abort the current program.
  */
 int
-no_memory(libzfs_handle_t *hdl)
+no_memory_real(libzfs_handle_t *hdl, char *fn, int line)
 {
-       return (zfs_error(hdl, EZFS_NOMEM, "internal error"));
+       return (zfs_error_fmt(hdl, EZFS_NOMEM, "internal error func:%s line:%d", fn, line));
 }

 /*

This will report the real problem function ...

@gchmurka123
Copy link
Author

gchmurka123 commented Jun 11, 2022

First of all, I would like to thank you for your help.
The patch doesn't work, I have the following compile error

Making all in libzfs
make[3]: Entering directory `/root/zfs/zfs-2.1.4/lib/libzfs'
  CC     libzfs_changelist.lo
  CC     libzfs_config.lo
  CC     libzfs_crypto.lo
  CC     libzfs_dataset.lo
  CC     libzfs_diff.lo
  CC     libzfs_import.lo
  CC     libzfs_iter.lo
  CC     libzfs_mount.lo
  CC     libzfs_pool.lo
  CC     libzfs_sendrecv.lo
  CC     libzfs_status.lo
  CC     libzfs_util.lo
libzfs_util.c:752:1: error: conflicting types for ‘no_memory_real’
In file included from libzfs_util.c:59:0:
../../include/libzfs_impl.h:147:5: note: previous declaration of ‘no_memory_real’ was here
libzfs_util.c: In function ‘zfs_alloc’:
libzfs_util.c:766:3: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c: In function ‘zfs_asprintf’:
libzfs_util.c:789:3: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c: In function ‘zfs_realloc’:
libzfs_util.c:805:3: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c: In function ‘zfs_strdup’:
libzfs_util.c:822:3: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c: In function ‘zcmd_read_dst_nvlist’:
libzfs_util.c:1259:3: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c: In function ‘zprop_parse_value’:
libzfs_util.c:1748:4: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c:1753:4: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
make[3]: *** [libzfs_util.lo] Error 1
make[3]: Leaving directory `/root/zfs/zfs-2.1.4/lib/libzfs'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/root/zfs/zfs-2.1.4/lib'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/zfs/zfs-2.1.4'
make: *** [all] Error 2

@mcmilk
Copy link
Contributor

mcmilk commented Jun 11, 2022

replace this:
+no_memory_real(libzfs_handle_t *hdl, char *fn, int line)
with this:
+no_memory_real(libzfs_handle_t *hdl, const char *fn, int line)

@gchmurka123
Copy link
Author

internal error func:zpool_in_use line:316: out of memory

@mcmilk
Copy link
Contributor

mcmilk commented Jun 11, 2022

This shows us, that within the function zpool_in_use() of the file lib/libzfs/libzfs_import.c the called function zpool_read_label() returns with != 0 ... and this is catched as an generic no_memory() error - which is a wrong error description:

if (zpool_read_label(fd, &config, NULL) != 0) {
  (void) no_memory(hdl);
  return (-1);
}

Can you debug the function zpool_read_label() yourself?
Just put some printf() statements in there ... to see what is happening... then you get the real error, and maybe some solution for it.

@gchmurka123
Copy link
Author

gchmurka123 commented Jun 13, 2022

zpool create break after gchmurka123_11 debug printf()

gchmurka123_6
gchmurka123_7
skip 8
gchmurka123_9
skip 10
gchmurka123_11

so i think lio_listio() function is problem

there is my debug gchmurka123 brake points:

        printf("gchmurka123_6\n");
        if (lio_listio(LIO_WAIT, aiocbps, VDEV_LABELS, NULL) != 0) {
                int saved_errno = errno;
                boolean_t do_slow = B_FALSE;
                error = -1;

                printf("gchmurka123_7\n");
                if (errno == EAGAIN || errno == EINTR || errno == EIO) {
                        /*
                         * A portion of the requests may have been submitted.
                         * Clean them up.
                         */
                        printf("gchmurka123_8\n");
                        for (l = 0; l < VDEV_LABELS; l++) {
                                errno = 0;
                                switch (aio_error(&aiocbs[l])) {
                                case EINVAL:
                                        break;
                                case EINPROGRESS:
                                        // This shouldn't be possible to
                                        // encounter, die if we do.
                                        ASSERT(B_FALSE);
                                        fallthrough;
                                case EOPNOTSUPP:
                                case ENOSYS:
                                        do_slow = B_TRUE;
                                        fallthrough;
                                case 0:
                                default:
                                        (void) aio_return(&aiocbs[l]);
                                }
                        }
                }
                printf("gchmurka123_9\n");
                if (do_slow) {
                        /*
                         * At least some IO involved access unsafe-for-AIO
                         * files. Let's try again, without AIO this time.
                         */
                        printf("gchmurka123_10\n");
                        error = zpool_read_label_slow(fd, config, num_labels);
                        saved_errno = errno;
                }
                printf("gchmurka123_11\n");
                free(labels);
                errno = saved_errno;
                return (error);
        }

@mcmilk
Copy link
Contributor

mcmilk commented Jun 13, 2022

Can you extend the printf a bit, to get more details?

Maybe like this: printf("gchmurka123.. errno=%d error=%d saved_errno=%d \n", errno, error, saved_errno);

@gchmurka123
Copy link
Author

gchmurka123 commented Jun 13, 2022

errno=22 error=-1 saved_errno=22

@mcmilk
Copy link
Contributor

mcmilk commented Jun 13, 2022

Did you get these errors from the lio_listio() call?
errno = 22 means EINVAL ... so some argument seems wrong there

can you replace the:

                                case EINVAL:
                                        break;

with this:

                                case EINVAL:
                                        do_slow = B_TRUE;
                                        break;

@gchmurka123
Copy link
Author

gchmurka123 commented Jun 14, 2022

This patch work for me...

--- zfs-2.1.4/lib/libzutil/zutil_import.c       2022-03-23 18:02:05.356762049 +0100
+++ zfs-2.1.4-fix/lib/libzutil/zutil_import.c   2022-06-14 09:30:16.871226978 +0200
@@ -1009,7 +1009,7 @@ zpool_read_label(int fd, nvlist_t **conf
                boolean_t do_slow = B_FALSE;
                error = -1;
 
-               if (errno == EAGAIN || errno == EINTR || errno == EIO) {
+               if (errno == EAGAIN || errno == EINTR || errno == EIO || errno == EINVAL) {
                        /*
                         * A portion of the requests may have been submitted.
                         * Clean them up.
@@ -1018,6 +1018,7 @@ zpool_read_label(int fd, nvlist_t **conf
                                errno = 0;
                                switch (aio_error(&aiocbs[l])) {
                                case EINVAL:
+                                       do_slow = B_TRUE;
                                        break;
                                case EINPROGRESS:
                                        // This shouldn't be possible to

It looks like my problem started to occur with optimization of zpool_read_label with AIO:
e50b521

@mcmilk
Copy link
Contributor

mcmilk commented Jun 14, 2022

@behlendorf - should this change make it into OpenZFS?

@gchmurka123 - can you mark this issue solved and close it?

@gchmurka123
Copy link
Author

gchmurka123 commented Jun 14, 2022

If I understand correctly and calling do_slow run older read labels sequentially method and this method is still safe (but slower), then I will close the issue as solved. Thank you @mcmilk for your help.

@behlendorf
Copy link
Contributor

It'd be nice to understand why zpool_read_label is returning EINVAL. If it were just a matter of it not being supported I've have expected ENOSYS though. That said, there's no harm is falling back to the older slow mode so making these change in OpenZFS would be fine, and preferable to failing with a misleading error.

@gchmurka123
Copy link
Author

gchmurka123 commented Jun 14, 2022

In my opinion kernel 4.9 it will be AOL on Jan. 2023, so I don't know if there is any point in investigating this bug further.

For some reasons, I have to use 4.9 + zfs 2.x it in my environments for a while. If my patch is safe for me, I will close issiue, but you don't have to make changes to the main ZFS branches.
In the next 6 months, kernel 4.9 will be AOL and the bug will be unimportant ;)

@behlendorf
Copy link
Contributor

The proposed change is definitely safe for you to use. Why it's needed is just a bit mysterious and I suppose it must be due to a quirk of your environment. Then I'll close this out.

@dkambersky
Copy link

dkambersky commented Jul 13, 2022

FWIW this bug seems to be happening on more recent kernels too. Same issue.

Error: 'internal error: out of memory' after a zpool create [..] invocation
Distro: Ubuntu 22.04 LTS x86_64
Kernel: 5.15.0-41-generic
OpenZFS version:
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.2-1ubuntu3

The hardware itself is somewhat old - PC from 2013ish, disks potentially even older. In case that helps.

@gchmurka123
Copy link
Author

gchmurka123 commented Jul 14, 2022

First to check and rule out a hardware/disk problem try create pool on file:
like this:

dd if=/dev/zero of=/my-pool-file bs=1M count=1024
zpool create tank /my-pool-file

@michaelfranzl
Copy link
Contributor

I saw the same issue. However, in my case the reason was that there was a regular file with zero size instead of a block device for the underlying disk. So, the error message internal error: out of memory was kind of correct. I wouldn't have called it memory though.

@gchmurka123
Copy link
Author

The minimum size of each device in a pool is 64 MB

@xin3liang
Copy link
Contributor

xin3liang commented Aug 9, 2023

I saw the same issue. However, in my case the reason was that there was a regular file with zero size instead of a block device for the underlying disk. So, the error message internal error: out of memory was kind of correct. I wouldn't have called it memory though.

I met the same issue with you. The zero file is located in /tmp/(tmpfs) directory. I think zfs should be able to create a pool from a memory file for testing purpose.

@SilverBut
Copy link

We encountered a similar issue. A running zpool (zfs-kmod-2.2.1-1) with several NVMe drive reported degraded status. Both zpool online/offline and zpool replace report internal error: out of memory. The server is too critical to reboot. After some research, seems the faulty disk is having wrong state on PCIe state, causing read the raw disk failed, but the /dev/nvme* still there. After remove via /sys/bus/pci/devices/PATH/remove and rescan the bus, zpool online recovered.

Current error message is confusing. Anything we can do to improve it?

behlendorf added a commit to behlendorf/zfs that referenced this issue Jan 8, 2024
Drop the no_memory() call from zpool_in_use() when reading the
label fails and instead return the error to the caller.  This
prevents a misleading "internal error: out of memory" error
when the label can't be read.  This will result in is_spare()
returning B_FALSE instead of aborting, which is already safely
handled.

Furthermore, on Linux it's possible for EREMOTEIO to returned
by an NVMVe device if the device has been low-level formatted
and not rescanned.  In this case we want to fallback to the
legacy scanning method and read any of the labels we can.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#13538
behlendorf added a commit to behlendorf/zfs that referenced this issue Jan 8, 2024
Drop the no_memory() call from zpool_in_use() when reading the
label fails and instead return the error to the caller.  This
prevents a misleading "internal error: out of memory" error
when the label can't be read.  This will result in is_spare()
returning B_FALSE instead of aborting, which is already safely
handled.

Furthermore, on Linux it's possible for EREMOTEIO to returned
by an NVMe device if the device has been low-level formatted
and not rescanned.  In this case we want to fallback to the
legacy scanning method and read any of the labels we can.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#13538
behlendorf added a commit that referenced this issue Jan 12, 2024
Drop the no_memory() call from zpool_in_use() when reading the
label fails and instead return the error to the caller.  This
prevents a misleading "internal error: out of memory" error
when the label can't be read.  This will result in is_spare()
returning B_FALSE instead of aborting, which is already safely
handled.

Furthermore, on Linux it's possible for EREMOTEIO to returned
by an NVMe device if the device has been low-level formatted
and not rescanned.  In this case we want to fallback to the
legacy scanning method and read any of the labels we can.

Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #13538
Closes #15747
behlendorf added a commit to behlendorf/zfs that referenced this issue Jan 29, 2024
Drop the no_memory() call from zpool_in_use() when reading the
label fails and instead return the error to the caller.  This
prevents a misleading "internal error: out of memory" error
when the label can't be read.  This will result in is_spare()
returning B_FALSE instead of aborting, which is already safely
handled.

Furthermore, on Linux it's possible for EREMOTEIO to returned
by an NVMe device if the device has been low-level formatted
and not rescanned.  In this case we want to fallback to the
legacy scanning method and read any of the labels we can.

Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#13538
Closes openzfs#15747
behlendorf added a commit that referenced this issue Jan 29, 2024
Drop the no_memory() call from zpool_in_use() when reading the
label fails and instead return the error to the caller.  This
prevents a misleading "internal error: out of memory" error
when the label can't be read.  This will result in is_spare()
returning B_FALSE instead of aborting, which is already safely
handled.

Furthermore, on Linux it's possible for EREMOTEIO to returned
by an NVMe device if the device has been low-level formatted
and not rescanned.  In this case we want to fallback to the
legacy scanning method and read any of the labels we can.

Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #13538
Closes #15747
lundman pushed a commit to openzfsonwindows/openzfs that referenced this issue Mar 13, 2024
Drop the no_memory() call from zpool_in_use() when reading the
label fails and instead return the error to the caller.  This
prevents a misleading "internal error: out of memory" error
when the label can't be read.  This will result in is_spare()
returning B_FALSE instead of aborting, which is already safely
handled.

Furthermore, on Linux it's possible for EREMOTEIO to returned
by an NVMe device if the device has been low-level formatted
and not rescanned.  In this case we want to fallback to the
legacy scanning method and read any of the labels we can.

Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#13538
Closes openzfs#15747
lundman pushed a commit to openzfsonwindows/openzfs that referenced this issue Mar 13, 2024
Drop the no_memory() call from zpool_in_use() when reading the
label fails and instead return the error to the caller.  This
prevents a misleading "internal error: out of memory" error
when the label can't be read.  This will result in is_spare()
returning B_FALSE instead of aborting, which is already safely
handled.

Furthermore, on Linux it's possible for EREMOTEIO to returned
by an NVMe device if the device has been low-level formatted
and not rescanned.  In this case we want to fallback to the
legacy scanning method and read any of the labels we can.

Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue openzfs#13538
Closes openzfs#15747
@go-while
Copy link

go-while commented Jun 3, 2024

Hello

disk layout
/swap 8G
/boot ext3 2G
/ ext4 50G

I created new partition 4 as "extended" and did:
zpool create tank -o ashift=12 mirror /dev/nvme0n1-part4 /dev/nvme1n1-part4

internal error: out of memory

deleting part4 with fdisk and recreating as primary works.

@fuggla
Copy link

fuggla commented Aug 6, 2024

@go-while Thanks, that fixed it for me as well. I had the exact same scenario.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Development

No branches or pull requests

9 participants