ZoL 0.6.4 on Fedora 21 claims memory until system freezes

[HeinzApfel]

My system uses more and more memory until it starts to swap and then freezes.

I already tried to set "options zfs zfs_arc_max=17179869184" in modprobe.d and verified that it's really set (/proc/spl/kstat/zfs/arcstats). Setting arc_max lower allows longer system uptime but leads to the same problem.

The system is a fresh installed Fedora 21 using kernel 3.19.3 with ZoL 0.6.4 rpm for Fedora. It has 32GB ECC memory with a raidz-2 based 6x WDC Red 4TB.
If I let the system go, it's working stable and used memory (htop report) is about 18GB.
When I then start access to the filesystem like a "find /tank -print0" or "rsync -av" over the ZFS drive, the system starts using more and more memory.
Interrupting the "find" stops eating more memory. Memory is not released. If I export/import the pool, all memory is released.

Question: Can someone please point me to some things I could try to find the cause of the problem?

[kernelOfTruth]

Things to consider:

Limit ARC's size to a little less than half of the max amount of your total amount of RAM (e.g. 15 GB), (#3259)

You might want to reduce the size of your l2arc or ZIL devices (if applicable)

Make sure that you have the latest 0.6.4.y ZFS release: https://github.com/zfsonlinux/zfs/commits/zfs-0.6.4-release which includes the following commit: 5074bfe
(see: #3283 )

and

e548a59
which also should prevent several lockup possibilities

If it's an option and you're heavily using rsync - consider using a patched rsync with the fadvise functionality http://insights.oetiker.ch/linux/fadvise/
which adds the

--drop-cache

option

I'll see what your

find /tank -print0

does on my system

Yes, I've observed that behavior, too that memory isn't really released - as long as ARC and the portion of SUnreclaim doesn't exceed the pre-set limit of

zfs_arc_max

it should be expected behavior - from what I understood

Hope that helps somewhat

edit:

the find command ran fine

My setup isn't really comparable ( mirrored 1.8 TB, 32 GB ECC) - max usage of ARC was around 6 GB (out of the max limit of 12)

[HeinzApfel]

Dear kernelOfTruth,

thanks for your quick and helpful reply. I tried that right away and had a more stable system since then.

By enabling the zfs-testing repository, ZFS up to commit de6d197 was installed. But still reaches swapping when doing directory searches (find) or backups (rsync).

It looks like the problem is caused by the larger number of small files and directories I have in my zpool. So far I downgraded to 0.6.4 and set "options zfs zfs_arc_meta_limit=5368709120". No longer "zfs_arc_max". This seems to fix the problem.

[chjohnst]

Just out of curiosity, since this comes up frequently should we maybe in
code adjust the arc as default to only 1/4 memory as a starting point?
Similar issue happened to me in CentOS on a pretty heavily taxed NFS
server.
On Apr 21, 2015 4:05 AM, "HeinzApfel" notifications@github.com wrote:

Dear kernelOfTruth,

thanks for your quick and helpful reply. I tried that right away and had a
more stable system since then.

By enabling the zfs-testing repository, ZFS up to commit de6d197
de6d197
was installed. But still reaches swapping when doing directory searches
(find) or backups (rsync).

It looks like the problem is caused by the larger number of small files
and directories I have in my zpool. So far I downgraded to 0.6.4 and set
"options zfs zfs_arc_meta_limit=5368709120". No longer "zfs_arc_max". This
seems to fix the problem.

—
Reply to this email directly or view it on GitHub
#3320 (comment).

[behlendorf]

This issue was resolved by limiting the number of cached dnodes.