Date: Tue, 08 Feb 2011 11:48:05 +0100 From: Bartosz Stec <admin@kkip.pl> To: Jeremy Chadwick <freebsd@jdc.parodius.com> Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? Message-ID: <4D511F65.2050503@kkip.pl> In-Reply-To: <20110208102727.GA8555@icarus.home.lan> References: <AANLkTi=8fFwiaQ4%2Bm_cWFkXwpa4_W0_DDV2aW8vyNU4E@mail.gmail.com> <op.vqjyb21daevz08@ghost-pc.home.lan> <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
W dniu 2011-02-08 11:27, Jeremy Chadwick pisze: > On Tue, Feb 08, 2011 at 10:24:11AM +0100, Bartosz Stec wrote: >> W dniu 2011-02-07 22:37, Emil Muratov pisze: >>>> For the past few weeks, I noticed that the amount of memory >>>> reported in top >>>> (sum of active, inact, wired, cache buf and free) keeps >>>> decreasing as the >>>> uptime increases. I can't pinpoint to when I first noticed this, >>>> as I have >>>> updated the system a few times just in case this has been fixed. >>> Yes, I have the same issue on my home file storage. My system is >>> 8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives. >>> After updating to stable a couple of days ago I noticed that the >>> system leaks memory very fast. Checking here and there I found >>> that the issue concerns sendfile (yep, again!). >>> >>> How to reproduce: >>> Configure samba with aio and sendfile (mine is version 3.5.6) >>> >>> smb.conf >>> [global] >>> use sendfile=true >>> aio read size = 16384 >>> >>> Download a couple of large samba shared files (8-10 gigs). >>> >>> >>> While downloading files I can see that memory decreazes to nowhere >>> very-very fast, several MBs per second! First it drains free mem, >>> than active and inactive, than comes wired until the whole system >>> commits suicide suffocating itself to the death. >>> The only way to free memory is to reboot the system. I can't >>> unload zfs module like PJD suggested to do, 'cause my root is on >>> zfs :( >>> I'll try to make a bootable flash and move root to the flash to >>> try to unload module and what will happen. >>> >>> Everything was OK in stable before the new year, sendfile used to >>> pump free and wired memory to inactive than slowly reclaiming it >>> back. But it seems something was changed after NY holydays? >> I'm glad someone else finally picked that problem, so there's >> appareantly no memory-eating ghost in my machine ;) >> Here's my thread on stable list about this issue: >> http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html >> >> And in fact, PC reported in thread above is also SAMBA server with >> aio/sendfile enabled and ZFS. >> >> I would be happy testing some patches if necessary, because until >> now I need to monitor memory and reboot this server before it dies. > The source and build date of your kernel will matter greatly here. > > I can't speak about the memory utilisation aspect, but I tend to disable > sendfile everywhere possible when ZFS is in use on a system. The reason > is based on something I and another user experienced back in October > 2010 pertaining to sendfile() on ZFS locking up processes (making them > unkillable). See here[1] for details; this problem has since been > fixed[2] (look for commits around October). You'll also find some > commits that went through in November pertaining to ZFS and sendfile. > This is why I said the date of your kernel/sources matters. :-) I tried rebuild since original thread, hoping that problem is fixed already, so now it's very fresh: 8.2-PRERELEASE #18: Sun Feb 6 03:04:47 CET 2011. Problem is still here: Mem: 37M Active, 78M Inact, 1154M Wired, 64M Cache, 199M Buf, 40M Free About 1373MB instead of 2GB, and it's not even 2 days of uptime. > The issue I referenced in [1] is not related to memory utilisation, but > does indicate use of sendfile with ZFS may be a bad idea (by this I > mean, there may be aspects of its implementation when mixed with ZFS > that have been overlooked). > > Simple test: if you disable use of sendfile (but not AIO) in Samba, does > the problem go away? I've just disabled sendfile in smb.conf and I'll report in about 2 days, after reboot which I will perform tonight. I hope it won't hit samba performance too much ;) -- Bartosz Stec
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D511F65.2050503>