Date: Mon, 27 Dec 2004 13:35:21 +0100 From: <techlists@motrix.dk> To: "'Doug White'" <dwhite@gumbysoft.com>, <freebsd-stable@freebsd.org> Cc: mlaier@freebsd.org Subject: RE: netstat fails with memory allocation error and error in kvm_read Message-ID: <20041227123520.9F3FCC2EF@brugere.aub.dk> In-Reply-To: <20041213101844.S92964@carver.gumbysoft.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> > > You appear to be running out of kernel memory. Since you're > > > capturing the output of vmstat -m, you should check that for any > > > bins that are growing at a high rate of speed. > > > > > > Seems possible that its in pf :) > > > > I've checked the numbers from just before the freeze (it's > within 15 > > secs) with two sets of data: From a fresh boot and five minutes > > minutes before the freeze. > > You might also log 'sysctl vm.kvm_free' and 'sysctl vm.zone'. sysctl vm.zone is identical to vmstat -z (according to man vmstat). I've graphed the output from iostat (idle/user/...), vmstat -i (interrupt rate), vmstat -m (in use), vmstat -z (used), sysctl vm.kvm_free (which is constant) and the number of pfstates. The graphs are at <http://www.aub.dk/~jmp/fw/plots/>. The newest data are from just after a deadlock. Are there something else I should graph? IRQ 20 is the NIC on our internal network (800+ machines), IRQ 18 and IRQ21 are NICs connected to the internet. There are a lot of changes on the vmstat -m graphs just before midnight last night that seems to correspond with the increase in interrupts on IRQ 18. The only graphs I can see changing up to the deadlock are: irq20 (internal NIC), irq21 (primary external NIC), the buckets (vmstat -z) all grow (I suppose this is normal?) the Mbufs seems to grow, but nothing extreme pffrag, pffrent (but not to levels they haven't been at before) Most notably most of the pf graphs doesn't change. Where can I see memory used by pf/altq? If it is pfaltqpl (in vmstat -z), it doesn't change at all. I'm in the process of setting up a serial console in the hope that I can break to the debugger with that. I'm also trying to provoke the deadlock so it will happen more frequently. /Martin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20041227123520.9F3FCC2EF>