zpool create - internal error: out of memory

[gchmurka123]

System information

Debian
x86_64-linux-gnu
zfs-2.1.4-1
zfs-kmod-2.1.4-1
Linux chmura 4.9.316 #5 SMP Tue Jun 7 13:02:00 CEST 2022 x86_64 GNU/Linux

Describe the problem you're observing

I am facing the development of a new zfs for very old systems with
kernel 4.9.316 (new build)
glibc 2.13
gcc version 4.7.2 (Debian 4.7.2-5)

Everything builds correctly, the zfs kernel module works, I can import the old pool created on zfs 0.8.6, update them, the tools work (like zfs create, destroy, snapshot, etc ) except for one command:
zpool create test device (or file)
always return
internal error: out of memory

Describe how to reproduce the problem

cd /build/zfs/zfs-2.1.4
./autogen.sh
./configure --disable-pyzfs --disable-systemd --with-linux=/build/linux-4.9.316 --with-linux-obj=/build/linux-4.9.316
make
make install

after make install (or make deb and install packages) command
zpool create test /dev/sdb
returns
internal error: out of memory

built old versions of zfs 0.8.6 work perfectly fine

[mcmilk]

How much RAM do you have. Is an OOM (out of memory) ... maybe just a OOM ?
What does free tell you about this?
Can you maybe do some strace zpool create test /dev/sdb ... and check what causes this OOM?

[gchmurka123]

The ram is not problem
asd/root# free -m
total used free shared buff/cache available
Mem: 3940 1973 1273 0 693 1908
Swap: 1980 320 1660

and strace in attached file from create pool on 1GB file (real device like /dev/sdb, or other device same error)

zpool-create.log

[mcmilk]

I see no failures there :-(
Maybe I can setup a small box with the same parameters.. what distribution do you have?
The kernel seems also self compiled?

[gchmurka123]

I change all out of memory errors to my debug messeage, and found place in source where the error is generate
zfs/zfs-2.1.4/lib/libzfs/libzfs_util.c:748 - 755

yes, kerenel is self build, but is vanilla (no extra patches included). I used kernel 4.4 before and on zfs 2.x i had the same problem with zpool ;( , 0.8.6 no problem on both 4.4 and 4.9

[mcmilk]

This is the source where the error is reported... can you try to patch you zfs with this diff:

diff --git a/include/libzfs_impl.h b/include/libzfs_impl.h
index 96b11dad1..45cba3e90 100644
--- a/include/libzfs_impl.h
+++ b/include/libzfs_impl.h
@@ -142,7 +142,10 @@ void *zfs_alloc(libzfs_handle_t *, size_t);
 void *zfs_realloc(libzfs_handle_t *, void *, size_t, size_t);
 char *zfs_asprintf(libzfs_handle_t *, const char *, ...);
 char *zfs_strdup(libzfs_handle_t *, const char *);
-int no_memory(libzfs_handle_t *);
+
+#define no_memory(h) no_memory_real(h, __func__, __LINE__)
+int no_memory_real(libzfs_handle_t *, const char *fn, int line);
+

 int zfs_standard_error(libzfs_handle_t *, int, const char *);
 int zfs_standard_error_fmt(libzfs_handle_t *, int, const char *, ...);
diff --git a/lib/libzfs/libzfs_util.c b/lib/libzfs/libzfs_util.c
index 7dd38bb3d..ce4f44276 100644
--- a/lib/libzfs/libzfs_util.c
+++ b/lib/libzfs/libzfs_util.c
@@ -749,9 +749,9 @@ zpool_standard_error_fmt(libzfs_handle_t *hdl, int error, const char *fmt, ...)
  * Display an out of memory error message and abort the current program.
  */
 int
-no_memory(libzfs_handle_t *hdl)
+no_memory_real(libzfs_handle_t *hdl, char *fn, int line)
 {
-       return (zfs_error(hdl, EZFS_NOMEM, "internal error"));
+       return (zfs_error_fmt(hdl, EZFS_NOMEM, "internal error func:%s line:%d", fn, line));
 }

 /*

This will report the real problem function ...

[gchmurka123]

First of all, I would like to thank you for your help.
The patch doesn't work, I have the following compile error

Making all in libzfs
make[3]: Entering directory `/root/zfs/zfs-2.1.4/lib/libzfs'
  CC     libzfs_changelist.lo
  CC     libzfs_config.lo
  CC     libzfs_crypto.lo
  CC     libzfs_dataset.lo
  CC     libzfs_diff.lo
  CC     libzfs_import.lo
  CC     libzfs_iter.lo
  CC     libzfs_mount.lo
  CC     libzfs_pool.lo
  CC     libzfs_sendrecv.lo
  CC     libzfs_status.lo
  CC     libzfs_util.lo
libzfs_util.c:752:1: error: conflicting types for ‘no_memory_real’
In file included from libzfs_util.c:59:0:
../../include/libzfs_impl.h:147:5: note: previous declaration of ‘no_memory_real’ was here
libzfs_util.c: In function ‘zfs_alloc’:
libzfs_util.c:766:3: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c: In function ‘zfs_asprintf’:
libzfs_util.c:789:3: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c: In function ‘zfs_realloc’:
libzfs_util.c:805:3: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c: In function ‘zfs_strdup’:
libzfs_util.c:822:3: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c: In function ‘zcmd_read_dst_nvlist’:
libzfs_util.c:1259:3: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c: In function ‘zprop_parse_value’:
libzfs_util.c:1748:4: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
libzfs_util.c:1753:4: warning: passing argument 2 of ‘no_memory_real’ discards ‘const’ qualifier from pointer target type [enabled by default]
libzfs_util.c:752:1: note: expected ‘char *’ but argument is of type ‘const char *’
make[3]: *** [libzfs_util.lo] Error 1
make[3]: Leaving directory `/root/zfs/zfs-2.1.4/lib/libzfs'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/root/zfs/zfs-2.1.4/lib'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/zfs/zfs-2.1.4'
make: *** [all] Error 2

[mcmilk]

replace this:
+no_memory_real(libzfs_handle_t *hdl, char *fn, int line)
with this:
+no_memory_real(libzfs_handle_t *hdl, const char *fn, int line)

[gchmurka123]

internal error func:zpool_in_use line:316: out of memory

[mcmilk]

This shows us, that within the function zpool_in_use() of the file lib/libzfs/libzfs_import.c the called function zpool_read_label() returns with != 0 ... and this is catched as an generic no_memory() error - which is a wrong error description:

if (zpool_read_label(fd, &config, NULL) != 0) {
  (void) no_memory(hdl);
  return (-1);
}

Can you debug the function zpool_read_label() yourself?
Just put some printf() statements in there ... to see what is happening... then you get the real error, and maybe some solution for it.

[gchmurka123]

zpool create break after gchmurka123_11 debug printf()

gchmurka123_6
gchmurka123_7
skip 8
gchmurka123_9
skip 10
gchmurka123_11

so i think lio_listio() function is problem

there is my debug gchmurka123 brake points:

        printf("gchmurka123_6\n");
        if (lio_listio(LIO_WAIT, aiocbps, VDEV_LABELS, NULL) != 0) {
                int saved_errno = errno;
                boolean_t do_slow = B_FALSE;
                error = -1;

                printf("gchmurka123_7\n");
                if (errno == EAGAIN || errno == EINTR || errno == EIO) {
                        /*
                         * A portion of the requests may have been submitted.
                         * Clean them up.
                         */
                        printf("gchmurka123_8\n");
                        for (l = 0; l < VDEV_LABELS; l++) {
                                errno = 0;
                                switch (aio_error(&aiocbs[l])) {
                                case EINVAL:
                                        break;
                                case EINPROGRESS:
                                        // This shouldn't be possible to
                                        // encounter, die if we do.
                                        ASSERT(B_FALSE);
                                        fallthrough;
                                case EOPNOTSUPP:
                                case ENOSYS:
                                        do_slow = B_TRUE;
                                        fallthrough;
                                case 0:
                                default:
                                        (void) aio_return(&aiocbs[l]);
                                }
                        }
                }
                printf("gchmurka123_9\n");
                if (do_slow) {
                        /*
                         * At least some IO involved access unsafe-for-AIO
                         * files. Let's try again, without AIO this time.
                         */
                        printf("gchmurka123_10\n");
                        error = zpool_read_label_slow(fd, config, num_labels);
                        saved_errno = errno;
                }
                printf("gchmurka123_11\n");
                free(labels);
                errno = saved_errno;
                return (error);
        }

[mcmilk]

Can you extend the printf a bit, to get more details?

Maybe like this: printf("gchmurka123.. errno=%d error=%d saved_errno=%d \n", errno, error, saved_errno);

[gchmurka123]

errno=22 error=-1 saved_errno=22

[mcmilk]

Did you get these errors from the lio_listio() call?
errno = 22 means EINVAL ... so some argument seems wrong there

can you replace the:

                                case EINVAL:
                                        break;

with this:

                                case EINVAL:
                                        do_slow = B_TRUE;
                                        break;

[gchmurka123]

This patch work for me...

--- zfs-2.1.4/lib/libzutil/zutil_import.c       2022-03-23 18:02:05.356762049 +0100
+++ zfs-2.1.4-fix/lib/libzutil/zutil_import.c   2022-06-14 09:30:16.871226978 +0200
@@ -1009,7 +1009,7 @@ zpool_read_label(int fd, nvlist_t **conf
                boolean_t do_slow = B_FALSE;
                error = -1;
 
-               if (errno == EAGAIN || errno == EINTR || errno == EIO) {
+               if (errno == EAGAIN || errno == EINTR || errno == EIO || errno == EINVAL) {
                        /*
                         * A portion of the requests may have been submitted.
                         * Clean them up.
@@ -1018,6 +1018,7 @@ zpool_read_label(int fd, nvlist_t **conf
                                errno = 0;
                                switch (aio_error(&aiocbs[l])) {
                                case EINVAL:
+                                       do_slow = B_TRUE;
                                        break;
                                case EINPROGRESS:
                                        // This shouldn't be possible to

It looks like my problem started to occur with optimization of zpool_read_label with AIO:
e50b521

[mcmilk]

@behlendorf - should this change make it into OpenZFS?

@gchmurka123 - can you mark this issue solved and close it?

[gchmurka123]

If I understand correctly and calling do_slow run older read labels sequentially method and this method is still safe (but slower), then I will close the issue as solved. Thank you @mcmilk for your help.

[behlendorf]

It'd be nice to understand why zpool_read_label is returning EINVAL. If it were just a matter of it not being supported I've have expected ENOSYS though. That said, there's no harm is falling back to the older slow mode so making these change in OpenZFS would be fine, and preferable to failing with a misleading error.

[gchmurka123]

In my opinion kernel 4.9 it will be AOL on Jan. 2023, so I don't know if there is any point in investigating this bug further.

For some reasons, I have to use 4.9 + zfs 2.x it in my environments for a while. If my patch is safe for me, I will close issiue, but you don't have to make changes to the main ZFS branches.
In the next 6 months, kernel 4.9 will be AOL and the bug will be unimportant ;)

[behlendorf]

The proposed change is definitely safe for you to use. Why it's needed is just a bit mysterious and I suppose it must be due to a quirk of your environment. Then I'll close this out.

[dkambersky]

FWIW this bug seems to be happening on more recent kernels too. Same issue.

Error: 'internal error: out of memory' after a zpool create [..] invocation
Distro: Ubuntu 22.04 LTS x86_64
Kernel: 5.15.0-41-generic
OpenZFS version:
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.2-1ubuntu3

The hardware itself is somewhat old - PC from 2013ish, disks potentially even older. In case that helps.

[gchmurka123]

First to check and rule out a hardware/disk problem try create pool on file:
like this:

dd if=/dev/zero of=/my-pool-file bs=1M count=1024
zpool create tank /my-pool-file

[michaelfranzl]

I saw the same issue. However, in my case the reason was that there was a regular file with zero size instead of a block device for the underlying disk. So, the error message internal error: out of memory was kind of correct. I wouldn't have called it memory though.

[gchmurka123]

The minimum size of each device in a pool is 64 MB

[xin3liang]

I saw the same issue. However, in my case the reason was that there was a regular file with zero size instead of a block device for the underlying disk. So, the error message internal error: out of memory was kind of correct. I wouldn't have called it memory though.

I met the same issue with you. The zero file is located in /tmp/(tmpfs) directory. I think zfs should be able to create a pool from a memory file for testing purpose.

[SilverBut]

We encountered a similar issue. A running zpool (zfs-kmod-2.2.1-1) with several NVMe drive reported degraded status. Both zpool online/offline and zpool replace report internal error: out of memory. The server is too critical to reboot. After some research, seems the faulty disk is having wrong state on PCIe state, causing read the raw disk failed, but the /dev/nvme* still there. After remove via /sys/bus/pci/devices/PATH/remove and rescan the bus, zpool online recovered.

Current error message is confusing. Anything we can do to improve it?

[go-while]

Hello

disk layout
/swap 8G
/boot ext3 2G
/ ext4 50G

I created new partition 4 as "extended" and did:
zpool create tank -o ashift=12 mirror /dev/nvme0n1-part4 /dev/nvme1n1-part4

internal error: out of memory

deleting part4 with fdisk and recreating as primary works.

[fuggla]

@go-while Thanks, that fixed it for me as well. I had the exact same scenario.