Date: Fri, 01 Aug 2008 08:09:59 -0800 From: Royce Williams <royce@alaska.net> To: freebsd-stable@FreeBSD.org Subject: Re: 6.3-RELEASE-p3 recurring panics on multiple SM PDSMi+ Message-ID: <48933557.8010904@alaska.net> In-Reply-To: <4886D1E9.1090905@alaska.net> References: <488638DA.7010005@alaska.net> <20080723053404.GA46617@eos.sc1.parodius.com> <4886D1E9.1090905@alaska.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Royce Williams wrote, on 7/22/2008 10:38 PM: > Jeremy Chadwick wrote, on 7/22/2008 9:34 PM: >> On Tue, Jul 22, 2008 at 11:45:30AM -0800, Royce Williams wrote: >>> We have 10 SuperMicro PDSMi+ 5015M-MTs that are panic'ing every few >>> days. This started shortly after upgrade from 6.2-RELEASE to >>> 6.3-RELEASE with freebsd-update. >> We use the same hardware (board and chassis), and have no such problems >> running both RELENG_6 and RELENG_7. >> >> I don't think your issue is specific to the board or chassis. Kris's >> explanation makes a lot more sense. :-) > > Jeremy/Kris/Clifton - > > Looks like we have consensus. :-) Thanks, all of you, for your > helpful insight. > > I've bumped vm.kmem_size up to 400M on half of the affected boxes, > leaving the other half as a control group. I'll report back once I > have something to report. After having bumped up to 400M, a few boxes panic'd again yesterday. I caught a core, and it is "kmem_map too small", just as Kris suspected: Jul 31 15:38:05 [redacted] savecore: reboot after panic: kmem_malloc(4096): kmem_map too small: 419430400 total allocated The docs state that 400M should be plenty for systems up to 6G, but Kris said earlier in this thread that it's better to say 'increase until the pain stops'. :-) Accordingly, I have some some follow-up questions; hopefully, this will be useful to others. - What is a reasonable increment? (I'm trying 448M next). - What are the practical and hard maximums? - I suspect that it's worth trying to make kmem 'as big as I need, but no bigger', so that non-kernel memory is also maximized? - In a larger sense, if 400M is probably big enough for 6G systems, and these are 4G systems, should I be suspicious that 400M isn't cutting it? In other words, is there a point at which should I be looking for obvious places where the kernel is eating too much memory and reduce them, rather than feeding it more? For example, I recall now that a network guy in my group did some sysctl tuning relating to networking on these systems, and I see from man tuning(7) that a number of these tweaks (obviously) can cause increased kernel consumption. $ egrep -v '^#|^$' /etc/sysctl.conf | sort kern.corefile=/var/cores/%U/%N-%P.core kern.ipc.maxsockbuf=8388608 kern.ipc.maxsockets=32768 kern.ipc.nmbclusters=65535 kern.ipc.somaxconn=4096 kern.maxfiles=262144 kern.maxfilesperproc=65535 net.inet.ip.portrange.first=8192 net.inet.ip.portrange.hifirst=8192 net.inet.ip.portrange.hilast=65535 net.inet.ip.portrange.last=65535 net.inet.ipf.fr_tcpclosed=60 net.inet.ipf.fr_tcpclosewait=120 net.inet.ipf.fr_tcphalfclosed=300 net.inet.ipf.fr_udptimeout=120 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.msl=10000 net.inet.tcp.mssdflt=1460 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65535 vfs.nfs.iodmax=32 vfs.nfs.iodmin=8 My apologies for not including this sooner. I didn't think of it until yesterday, primarily because it had been fine under 6.2. In retrospect, that was bad reasoning. Royce -- Royce D. Williams - http://royce.ws/ Reason is a very light rider, and easily shook off. - Jonathan Swift
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?48933557.8010904>