Date: Mon, 4 Apr 2011 18:01:48 -0700 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: Boris Kochergin <spawk@acm.poly.edu> Cc: Kostik Belousov <kostikbel@gmail.com>, FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: Kernel memory leak in 8.2-PRERELEASE? Message-ID: <20110405010148.GA67821@icarus.home.lan> In-Reply-To: <4D9A68AA.6040803@acm.poly.edu> References: <4D972FF7.6010901@acm.poly.edu> <20110402153315.GP78089@deviant.kiev.zoral.com.ua> <4D974393.80606@acm.poly.edu> <4D9A307F.9070408@acm.poly.edu> <20110404224334.GA64297@icarus.home.lan> <4D9A68AA.6040803@acm.poly.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Apr 04, 2011 at 08:56:10PM -0400, Boris Kochergin wrote: > On 04/04/11 18:43, Jeremy Chadwick wrote: > >On Mon, Apr 04, 2011 at 04:56:31PM -0400, Boris Kochergin wrote: > >>On 04/02/11 11:41, Boris Kochergin wrote: > >>>On 04/02/11 11:33, Kostik Belousov wrote: > >>>>On Sat, Apr 02, 2011 at 10:17:27AM -0400, Boris Kochergin wrote: > >>>>>Ahoy. This morning, I awoke to the following on one of my servers: > >>>>> > >>>>>pid 59630 (httpd), uid 80, was killed: out of swap space > >>>>>pid 59341 (find), uid 0, was killed: out of swap space > >>>>>pid 23134 (irssi), uid 1001, was killed: out of swap space > >>>>>pid 49332 (sshd), uid 1001, was killed: out of swap space > >>>>>pid 69074 (httpd), uid 0, was killed: out of swap space > >>>>>pid 11879 (eggdrop-1.6.19), uid 1001, was killed: out of swap space > >>>>>... > >>>>> > >>>>>And so on. > >>>>> > >>>>>The machine is: > >>>>> > >>>>>FreeBSD exodus.poly.edu 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #2: Thu > >>>>>Dec 2 11:39:21 EST 2010 > >>>>>spawk@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 > >>>>> > >>>>>10:13AM up 120 days, 20:06, 2 users, load averages: 0.00, 0.01, 0.00 > >>>>> > >>>>>The memory line from top intrigued me: > >>>>> > >>>>>Mem: 16M Active, 48M Inact, 6996M Wired, 229M Cache, 828M Buf, > >>>>>605M Free > >>>>> > >>>>>The machine has 8 gigs of memory, and I don't know what all that wired > >>>>>memory is being used for. There is a large-ish (6 x 1.5-TB) ZFS RAID-Z2 > >>>>>on it which has had a disk in the UNAVAIL state for a few months: > >>>>> > >>>>># zpool status > >>>>> pool: home > >>>>> state: DEGRADED > >>>>>status: One or more devices could not be used because the label is > >>>>>missing or > >>>>> invalid. Sufficient replicas exist for the pool to continue > >>>>> functioning in a degraded state. > >>>>>action: Replace the device using 'zpool replace'. > >>>>> see: http://www.sun.com/msg/ZFS-8000-4J > >>>>> scrub: none requested > >>>>>config: > >>>>> > >>>>> NAME STATE READ WRITE CKSUM > >>>>> home DEGRADED 0 0 0 > >>>>> raidz2 DEGRADED 0 0 0 > >>>>> ada0 ONLINE 0 0 0 > >>>>> ada1 ONLINE 0 0 0 > >>>>> ada2 ONLINE 0 0 0 > >>>>> ada3 ONLINE 0 0 0 > >>>>> ada4 ONLINE 0 0 0 > >>>>> ada5 UNAVAIL 0 85 11 experienced > >>>>>I/O failures > >>>>> > >>>>>errors: No known data errors > >>>>> > >>>>>"vmstat -m" and "vmstat -z" output: > >>>>> > >>>>>http://acm.poly.edu/~spawk/vmstat-m.txt > >>>>>http://acm.poly.edu/~spawk/vmstat-z.txt > >>>>> > >>>>>Anyone have a clue? I know it's just going to happen again if I reboot > >>>>>the machine. It is still up in case there are diagnostics for > >>>>>me to run. > >>>>Try r218795. Most likely, your issue is not leak. > >>>Thanks. Will update to today's 8-STABLE and report back. > >>> > >>>-Boris > >>The problem persists, I'm afraid, and seems to have crept up a lot > >>more quickly than before: > >> > >># uname -a > >>FreeBSD exodus.poly.edu 8.2-STABLE FreeBSD 8.2-STABLE #3: Sat Apr 2 > >>11:48:43 EDT 2011 > >>spawk@exodus.poly.edu:/usr/obj/usr/src/sys/EXODUS amd64 > >> > >>Mem: 314M Active, 955M Inact, 6356M Wired, 267M Cache, 828M Buf, 18M Free > >> > >>Any ideas for a diagnostic recourse? > >Can you please provide the details I requested here? Thanks. > > > >http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/062147.html > > > > No swap, blank /boot/loader.conf, default /etc/sysctl.conf. I'm > going to try this ARC tuning thing. I vaguely recall several claims > that tuning wasn't necessary anymore on amd64 systems with the > amount of memory mine has, but that's obviously not the case. Given that you don't have swap (again: very, very bad idea), your applications crashing due to there not being any swap space is expected: no place to swap them out to. All you should need to set, in /boot/loader.conf, is: vfs.zfs.arc_max For example, if you want to limit the ARC to only use up to 2GB of RAM: vfs.zfs.arc_max="2048M" This would reserve (on an 8GB machine) approximately ~6GB of RAM for userland applications, the kernel, network buffers/mbufs, etc.. Finally, please note that most of the stuff you'll read online for ZFS tuning on FreeBSD is outdated with 8.2. E.g. you should not need to set vm.kmem_size and you should never need to adjust vm.kmem_size_max. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110405010148.GA67821>