Do not are working commands ZFS...

[apaxucok]

Hello! I compiled the kernel with modules SPL and ZFS for my Parallella board, everything has been successful. After that, I loaded the Ubuntu 14.04 with this kernel and decided to test the native ZFS, but faced with such a problem does not work ZFS commands. Here is the log:

linaro-nano:~> dmesg | grep -i zfs

ZFS: Loaded module v0.6.3-99_g83e9986, ZFS pool version 5000, ZFS filesystem version 5

linaro-nano:~> dmesg | grep -i spl

SPL: Loaded module v0.6.3-10_ge302072
SPLAT: Loaded module v0.6.3-10_ge302072

linaro-nano:~> sudo su

root@linaro-nano:/home/linaro# cd /usr/local/sbin/

root@linaro-nano:/usr/local/sbin# ls

spl splat zdb zfs zinject zpios zpool ztest

root@linaro-nano:/usr/local/sbin# zpool

bash: /usr/local/sbin/zpool: No such file or directory

root@linaro-nano:/usr/local/sbin# z
zcat zenity zinject zipnote zpool
zcmp zfgrep zip zipsplit zpool_layout
zdb zforce zipcloak zless zsoelim
zdiff zfs zipdetails zmore ztest
zdump zgrep zipgrep znew
zegrep zic zipinfo zpios
root@linaro-nano:/usr/local/sbin# ./zpool

bash: ./zpool: No such file or directory
root@linaro-nano:/usr/local/sbin#

And now I do not know what to do next.Please help me...

[behlendorf]

@apaxucok did you install them using make install? And are do they have executable permissions?

[apaxucok]

I understand if the ZFS and SPL included in the kernel they need not be further set with 'make install'...
That's what happens when I try this:

linaro-nano:/usr/local/sbin> sudo su
root@linaro-nano:/usr/local/sbin# ls
spl splat zdb zfs zinject zpios zpool ztest
root@linaro-nano:/usr/local/sbin# make install zpool
make: * No rule to make target install'. Stop. root@linaro-nano:/usr/local/sbin# make install zfs make: *** No rule to make targetinstall'. Stop.
root@linaro-nano:/usr/local/sbin# ll
total 1040
drwxr-xr-x 2 root root 4096 Oct 15 10:19 ./
drwxr-xr-x 12 root root 4096 Mar 28 2012 ../
-rwxr-xr-x 1 root root 15550 Mar 28 2012 spl
-rwxr-xr-x 1 root root 44309 Mar 28 2012 splat
-rwxr-xr-x 1 root root 205884 Mar 28 2012 zdb*
-rwxr-xr-x 1 root root 197735 Mar 28 2012 zfs*
-rwxr-xr-x 1 root root 79195 Mar 28 2012 zinject*
-rwxr-xr-x 1 root root 46685 Mar 28 2012 zpios*
-rwxr-xr-x 1 root root 218453 Mar 28 2012 zpool*
-rwxr-xr-x 1 root root 232653 Mar 28 2012 ztest*

[apaxucok]

Maybe this will help as that ...
root@linaro-nano:~# uname -a
Linux linaro-nano 3.14.12-parallella-xilinx-g1095261-dirty #5 SMP PREEMPT Wed Oct 15 17:29:10 UTC 2014 armv7l armv7l armv7l GNU/Linux

[apaxucok]

Maybe there's another way to install ZFS on Parallella BOARD? But I will say at
once that there is no headers.

2014-10-16 17:38 GMT+04:00 Brian Behlendorf notifications@github.com:

@apaxucok https://github.com/apaxucok did you install them using make
install? And are do they have executable permissions?

—
Reply to this email directly or view it on GitHub
#2803 (comment).

[behlendorf]

One reason this can happen is if you're running a 32-bit binary on a 64-bit system (or vise-versa). There's a nice summary of this issue here. I'm not familiar with parallella but I see it's a ARM board. You'll need to make sure the runtime for the resulting binary is installed.

[apaxucok]

Yes on Parallella BOARD a 32 bit processor arm, so the file system is 32-bit.What should I do order to ZFS ran at arm 32 bit?

[behlendorf]

@apaxucok Can you run file /usr/local/bin/zpool to see what kind of binary was built.

[apaxucok]

So if I run file /usr/local/bin/zpool :
root@linaro-nano:/usr/local/sbin# file /usr/local/bin/zpool
/usr/local/bin/zpool: ERROR: cannot open `/usr/local/bin/zpool' (No such file or directory)

[apaxucok]

I will describe how I built the kernel:
1 sudo git clone https://github.com/parallella/parallella-linux.git (kernel sources )
2 cd parallella-linux
3 sudo make mrproper
4 sudo make parallella_defconfig (Build config)
5 sudo make nconfig
6 '-- Cryptographic API ---> ' then '<> Deflate compression algorithm' and '<> Zlib compression algorithm'
7 Then I turned on all ACL
root@linaro-nano:/src/parallella-linux# grep ACL .config
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_FS_POSIX_ACL=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_ACL_SUPPORT=y
8 Dowloaded spl-0.6.3.tar.gz and zfs-0.6.3.tar.gz
9 Unpacked them
10 cd /home/linaro/scripts/spl/
11 sudo ./autogen.sh
12 sudo ./configure --with-linux=/src/parallella-linux --with-linux-obj=/src/parallella-linux --enable-linux-builtin=yes
13 sudo ./copy-builtin /src/parallella-linux
14 Then repeat steps 1,5
15 Has activated SPL '<> Solaris Porting Layer (SPL)'
16 cd /home/linaro/scripts/zfs
17 Then repeat steps 11,12,13,1,5
18 Has activated ZFS '<*> ZFS filesystem support'
19 Make kernel 'make -j 4 LOADADDR=0x8000 uImage'
20 Reboot Parallella Board with new kernel
21 linaro-nano:~> dmesg | grep -i zfs
ZFS: Loaded module v0.6.3-99_g83e9986, ZFS pool version 5000, ZFS filesystem version 5

22 linaro-nano:~> dmesg | grep -i spl
SPL: Loaded module v0.6.3-10_ge302072
SPLAT: Loaded module v0.6.3-10_ge302072

p.s All of these actions were done on Parallella BOARD
What ideas or options?

[apaxucok]

The correct command - 'sudo file /usr/local/sbin/zpool'
linaro-nano:~> sudo file /usr/local/sbin/zpool
/usr/local/sbin/zpool: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, BuildID[sha1]=b29e5aff8968976159afe751a08cefd723300426, not stripped

linaro-nano:~> sudo file /usr/local/sbin/zfs
/usr/local/sbin/zfs: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, BuildID[sha1]=d39d26c8a54234365f8c52f91da9b00526ec5ac9, not stripped

sudo file /usr/local/sbin/spl
/usr/local/sbin/spl: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, BuildID[sha1]=a4c36111b934b39e44ac44f37ac49499fed810a1, not stripped

Then what's the problem?

[behlendorf]

@abyxcos Everything looks like it built correctly and the resulting utilities are 32-bit. Have you tried running the zfs/zpool command under sudo? You seem to have need to use sudo for the file command to work. For example sudo zpool import.

[apaxucok]

This does not help...I tried to do so, but the result is the same
bash: /usr/local/sbin/zpool: No such file or directory

[apaxucok]

I think if the problem is was in this, then the komand 'sudo su' would have helped...

[apaxucok]

@behlendorf What should be included in the kernel for the proper operation of ZFS? For example before installing SPL need to enable '<> Deflate compression algorithm' and '<> Zlib compression algorithm'.

[behlendorf]

@apaxucok Yes, you'll need zlib compression and decompression. Although in your case the kernel side appears to have built fine. You very likely wouldn't have been able to load the module if something went wrong. Your issue is with the user space utilities and looks to be something arm specific.

[mat128]

@apaxucok Try sudo ldd /usr/local/sbin/zpool

A little bit of research pointed me a ServerFault thread, my understanding is your binary is using some libraries not available at runtime, thus giving you a No such file or directory error OR your filesystem is not mounted with the exec flag, but I guess bash's autocomplete would not list that for you if that was the case.

Please run mount and cat /etc/fstab just so we can have a look.

[maci0]

maybe the LD_LIBRARY_PATH is broken
what is the content of /etc/ld.so.conf and /etc/ld.so.conf.d/* ?
at least on some Linux Distributions i know /usr/local/lib is not anywhere in there. So the dynamic library loader might have problems here.

[apaxucok]

For @maci0 :
linaro@linaro-nano:~$ cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf

linaro@linaro-nano:~$ ls /etc/ld.so.conf.d
arm-linux-gnueabihf.conf fakeroot-arm-linux-gnueabihf.conf libc.conf

For @mat128 :
linaro@linaro-nano:~$ sudo ldd /usr/local/sbin/zpool
not a dynamic executable

linaro@linaro-nano:~$ mount
/dev/root on / type ext4 (rw,relatime,data=ordered)
sysfs on /sys type sysfs (rw,relatime)
none on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
none on /proc type proc (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/cgroup type tmpfs (rw,relatime,size=4k,mode=755)
none on /run type tmpfs (rw,nosuid,noexec,relatime,size=99804k,mode=755)
none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
none on /run/shm type tmpfs (rw,nosuid,nodev,relatime)
none on /run/user type tmpfs (rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755)

linaro@linaro-nano:~$ cat /etc/fstab

UNCONFIGURED FSTAB FOR BASE SYSTEM

linaro@linaro-nano:~$ sudo blkid
/dev/mmcblk0p1: UUID="c87b757c-f1ae-4990-8ebd-79bfb01c3af8" TYPE="ext4"

I so understand that it is necessary configure the file /etc/fstab...I did not encounter this before so need help ... Thanks guys @behlendorf,@maci0 and @mat128 !!!

[apaxucok]

Maybe it too, as it will help ...

linaro@linaro-nano:~$ cat /etc/mtab
rootfs / rootfs rw 0 0
/dev/root / ext4 rw,relatime,data=ordered 0 0
sysfs /sys sysfs rw,relatime 0 0
none /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
none /proc proc rw,nosuid,nodev,noexec,relatime 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /run tmpfs rw,nosuid,noexec,relatime,size=99804k,mode=755 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0

[maci0]

@apaxucok something is really wrong here .. this is from your output earlier

linaro-nano:~> sudo file /usr/local/sbin/zpool
/usr/local/sbin/zpool: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, BuildID[sha1]=b29e5aff8968976159afe751a08cefd723300426, not stripped

How can it be that ldd reports /usr/local/sbin/zpool is not a dynamic executable

One possibility here is a mixup with the different arm targets. EABI vs. EABI-HF etc..
Looks like the zfs userland tools were compiled for EABI which seems to be unsupported by your operating system's userland (glibc, binutils, etc....), at least your /etc/ld.so.conf.d/ output makes me think your OS uses "Linaro ARMv7 Linux GNU EABI HF (arm-linux-gnueabihf-*)"

[apaxucok]

Hmm...What can be done with it? @maci0

[maci0]

@apaxucok tbh I would just follow the zfsonlinux guide for generic DEBs and build spl and zfs as kernel modules and not into the kernel directly, also this should create nice packages for the userland tools

[apaxucok]

Yes, I agree that the build deb packages would be better, but I have not got it because of this kernel haven't headers and modules...
linaro@linaro-nano:~$ ls /lib/
arm-linux-gnueabihf init lsb systemd
cpp klibc-JsQhEnahZRs4zAFAYMqmbGmjTlY.so modprobe.d terminfo
ifupdown ld-linux-armhf.so.3 plymouth udev

Does not exist headers for this kernel :
linaro@linaro-nano:~$ uname -r
3.14.12-parallella-xilinx-gc542445

[apaxucok]

Maybe I should try to recompile the kernel using the Linaro Toolchain (arm-linux-gnueabihf-*)?
Or do I just need to adjust the /etc/fstab ?

[maci0]

@apaxucok you should compile the kernel and the zfs userland tools with the same toolchain the rest of your userland is compiled with.
By now I am fairly sure the issue you reported is toolchain related not even architecture specific

[behlendorf]

Closing as stale.