Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance degradation on large files removal #609

Closed
mikhmv opened this issue Mar 19, 2012 · 8 comments
Closed

Performance degradation on large files removal #609

mikhmv opened this issue Mar 19, 2012 · 8 comments
Labels
Type: Performance Performance improvement or performance problem

Comments

@mikhmv
Copy link

mikhmv commented Mar 19, 2012

Hi current version of ZFS (daily build) with most recent kernel on ubuntu 12.04 has issues with deleting of large files.
It is too slow.
It was better before.
@ryao
Copy link
Contributor

ryao commented Mar 19, 2012

Do you have performance figures and software versions that you can cite?

@mikhmv
Copy link
Author

mikhmv commented Mar 19, 2012

Here is a system info:

max@s0:~$ uname -a
Linux s0 3.2.0-19-generic #30-Ubuntu SMP Fri Mar 16 16:27:15 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

max@s0:$ dpkg -s zfs-dkms
Package: zfs-dkms
Status: install ok installed
Priority: extra
Section: kernel
Installed-Size: 9437
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.54-0ubuntu1precise1

max@s0:$ dpkg -s zfsutils
Package: zfsutils
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 696
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.54-0ubuntu1precise1
max@s0:$ dpkg -s libuutil1
Package: libuutil1
Status: install ok installed
Priority: extra
Section: libs
Installed-Size: 147
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.54-0ubuntu1precise1

max@s0:$ dpkg -s libzfs1
Package: libzfs1
Status: install ok installed
Priority: extra
Section: libs
Installed-Size: 307
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.54-0ubuntu1precise1

max@s0:$ dpkg -s libzpool1
Package: libzpool1
Status: install ok installed
Priority: extra
Section: libs
Installed-Size: 1122
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.54-0ubuntu1precise1

max@s0:$ dpkg -s zfs-auto-snapshot
Package: zfs-auto-snapshot
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 67
Maintainer: Darik Horn dajhorn@vanadac.com
Architecture: all
Version: 1.0.8-0ubuntu1precise1

max@s0:~$ sudo zpool list
[sudo] password for max:
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
tank 9.06T 6.37T 2.69T 70% 1.01x ONLINE -

max@s0:~$ sudo zpool status
pool: tank
state: ONLINE
scan: scrub canceled on Thu Mar 15 20:19:58 2012
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        d1      ONLINE       0     0     0
        d2      ONLINE       0     0     0
        d3      ONLINE       0     0     0
        d4      ONLINE       0     0     0
        d5      ONLINE       0     0     0

errors: No known data errors

max@s0:~$ sudo zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 5.11T 2.04T 256K /tank
tank/Irina 310G 2.04T 310G /tank/Irina
tank/OpenNebula 866G 2.04T 863G /tank/OpenNebula
tank/biouml-shared 3.94T 2.04T 3.79T /tank/biouml-shared

max@s0:~$ zfs get all tank/biouml-shared
exportfs: could not open /var/lib/nfs/.etab.lock for locking: errno 13 (Permission denied)
NAME PROPERTY VALUE SOURCE
tank/biouml-shared type filesystem -
tank/biouml-shared creation Sun Feb 5 8:29 2012 -
tank/biouml-shared used 3.94T -
tank/biouml-shared available 2.04T -
tank/biouml-shared referenced 3.79T -
tank/biouml-shared compressratio 1.08x -
tank/biouml-shared mounted yes -
tank/biouml-shared quota none default
tank/biouml-shared reservation none default
tank/biouml-shared recordsize 128K default
tank/biouml-shared mountpoint /tank/biouml-shared default
tank/biouml-shared sharenfs off local
tank/biouml-shared checksum on default
tank/biouml-shared compression gzip local
tank/biouml-shared atime on default
tank/biouml-shared devices on default
tank/biouml-shared exec on default
tank/biouml-shared setuid on default
tank/biouml-shared readonly off default
tank/biouml-shared zoned off default
tank/biouml-shared snapdir hidden default
tank/biouml-shared aclinherit restricted default
tank/biouml-shared canmount on default
tank/biouml-shared xattr on default
tank/biouml-shared copies 1 default
tank/biouml-shared version 5 -
tank/biouml-shared utf8only off -
tank/biouml-shared normalization none -
tank/biouml-shared casesensitivity sensitive -
tank/biouml-shared vscan off default
tank/biouml-shared nbmand off default
tank/biouml-shared sharesmb off default
tank/biouml-shared refquota none default
tank/biouml-shared refreservation none default
tank/biouml-shared primarycache all default
tank/biouml-shared secondarycache all default
tank/biouml-shared usedbysnapshots 154G -
tank/biouml-shared usedbydataset 3.79T -
tank/biouml-shared usedbychildren 0 -
tank/biouml-shared usedbyrefreservation 0 -
tank/biouml-shared logbias latency default
tank/biouml-shared dedup on inherited from tank
tank/biouml-shared mlslabel none default
tank/biouml-shared sync standard default
tank/biouml-shared refcompressratio 1.04x -

Regular hardrive test:
max@s0:~$ time echo test zfs speed > test.txt

real 0m0.016s
user 0m0.000s
sys 0m0.000s

ZFS:
oneadmin@s0:/tank/biouml-shared/tmp-tools$ time echo test zfs speed > test.txt

real 0m2.446s
user 0m0.000s
sys 0m0.000s

oneadmin@s0:/tank/biouml-shared/tmp$ time ls -lahs > test.time.txt

real 0m8.420s
user 0m0.000s
sys 0m0.040s

max@s0:~$ zpool iostat 5
exportfs: could not open /var/lib/nfs/.etab.lock for locking: errno 13 (Permission denied)
capacity operations bandwidth
pool alloc free read write read write

tank 6.37T 2.69T 307 194 30.2M 783K
tank 6.37T 2.69T 353 121 31.8M 700K
tank 6.37T 2.69T 248 209 14.0M 1.22M
tank 6.37T 2.69T 219 226 13.9M 1.38M
tank 6.37T 2.69T 333 97 35.4M 544K
tank 6.37T 2.69T 245 334 11.4M 1.86M
tank 6.37T 2.69T 232 128 15.5M 662K
tank 6.37T 2.69T 317 50 38.2M 110K

I performed these tests when system is a little responsive. It was worse before.

max@s0:~$ sudo lsof | grep tank| wc -l
124

I will provide tests again when system will be under heavy load.

@behlendorf
Copy link
Contributor

Might these files also have xattrs? There still a know issue with removing a large number of files with directory based xattrs, see issue #457

@behlendorf
Copy link
Contributor

Is this still the case with the latest master source? We've improved quite a few things in the last 6 months.

@stephanvaningen
Copy link

I think I can confirm that this is indeed still the case: I clean-installed u12.10asm64 with zfs 0.6.0.80 - deleting a folder with files of 50MB each costs a few seconds per file.
Which tech-info would you like if you want more info from me?

@behlendorf
Copy link
Contributor

@stephanvaningen A few seconds per file, that's horrible. Can you reproduce this issue when destroying a single file by name in that directory? If so can you run strace -T rm <file> to get the timing information to ensure we're stuck in the unlink system call for really a few seconds. Is there anything special about the file, lots of xattrs perhaps?

@behlendorf
Copy link
Contributor

The last reports of problems were from 0.6.0.80. There has been a huge amount of improvement since then. I'm going to close this out as stale. Let's open a new issue if there is still a concern that unlink performance needs to be improved.

@behlendorf behlendorf removed this from the 0.6.7 milestone Oct 6, 2014
@stephanvaningen
Copy link

thanks
Sorry to keep this open from my side,

Stephan van Ingen

http://stephanvaningen.net

Me transmitte sursum, Caledoni!

On 6 October 2014 23:05, Brian Behlendorf notifications@github.com wrote:

Closed #609 #609.

—
Reply to this email directly or view it on GitHub
#609 (comment).

behlendorf pushed a commit to behlendorf/zfs that referenced this issue May 21, 2018
@bprotopopov @behlendorf
Add assert under lock to detect cases of dispach of a preallocated
9243b0f 
taskq work item to more than one queue concurrently. Also, please
see discussion in openzfs#3840.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Boris Protopopov <boris.protopopov@actifio.com>
Closes openzfs#609
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Performance Performance improvement or performance problem
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@behlendorf @stephanvaningen @ryao @mikhmv