Date: Fri, 4 May 2001 01:38:28 -0700 From: "Nick's Lists" <mrlist@noid.org> To: <freebsd-stable@freebsd.org>, <alfred@freebsd.org> Subject: Re: panic: pipeinit: cannot allocate pipe -- out of kvm -- code = 3 Message-ID: <030801c0d475$bd1e0fe0$0102010a@noid.org> References: <026b01c0d1b9$161f0320$3ba2640a@int.netzero.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Thought I'd repost this, and provide an update and see if anyone has any suggestions :) I've gone ahead and done a cvs-up to 4.3Stable with no change in stability (So much for stable huh? <grin>). The box is staying up for about 6 hours at a time between crashes with maxusers = 1024. I'm going to go ahead and put a new kernel in that's got maxusers = 16, and set maxfiles = 5000 and NPROC = 2068 in param.c. I'm doing this mainly to follow along with my theory that bumping maxusers up too high is squeezing the system for kvm somehow, and by keeping maxusers down, and artificially bumping maxfiles and NPROC up I can try and keep the system stable. The whole reason this is coming about is that I'm currently evaluating FreeBSD vs Solaris x86 to use in my production mail environment to replace the 30 something Sparc Solaris and 3 or 4 Linux boxes that I'm currently using. Its interesting to note that an identical box running Solaris x86 8 is handling the same load without any problems at all. Its a bit dispointing tho, as I"ve done some side by side comparisons of FreeBSD and Sol x86 running on identical hardware taking an identical load, and the BSD box uses less memory and CPU then the x86. If I could just keep the thing stable, it would make it a much better choice than going with Sol x86. Any help or suggestions would be greatly appreciated! Thanks! - Nick ----- Original Message ----- From: "Nick's Lists" <mrlist@noid.org> To: <freebsd-stable@FreeBSD.ORG> Cc: <alfred@FreeBSD.ORG> Sent: Monday, April 30, 2001 2:03 PM Subject: panic: pipeinit: cannot allocate pipe -- out of kvm -- code = 3 > The box is a dual P3 with 2g ram, running Qmail which is handling inbound > email. I'm running 4.2 Release (though I've had similar problems with 3.4 > Stable a few months back, and never solved them either). > > After a 36 - 48hrs, the box will panic with "panic: pipeinit: cannot > allocate pipe -- out of kvm -- code = 3". It doesn't seem to happen during > periods of peak load, the last crash occurred at 3:45AM and there have been > days when the box has made it through our peak load times with no problems. > > When the box was initially installed, it was running out of files. I built a > new kernel with maxusers = 512. The box seemed to last about 48 hours or so > before crashing with the out of kvm panic. > > After a bit of digging around, I came to the conclusion (erroneously?) that > the problem was caused by bumping maxusers up too high and chewing through > too much memory allocating space for a large number of files and processes. > > As a result, I've tried maxusers at 256, 128, and 64. With maxusers at 64, > I've bumped maxfiles to 5000 via sysctl. That didn't seem to make any > difference, the box seemed to crash every 36 - 48 hours with the same panic. > > It was suggested to me a couple of days ago, to try running maxusers = 1024. > I tried that after the box crashed this last time. With that setting, the > box only stayed up for 6 hours, from 3:45AM to 9:45AM. > > The last few lines of vmstat 2 prior to the crash are: > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ml0 md0 in sy cs us sy > id > 10 8 0 24248 778400 3022 0 0 0 2453 0 171 0 744 10659 1586 7 > 28 65 > 5 5 0 20900 780140 3353 0 0 0 3610 0 295 0 953 8824 2595 7 39 > 54 > 0 5 0 19096 780644 1339 0 0 0 1437 0 95 0 512 3188 886 1 15 > 84 > 1 4 0 17460 781020 940 0 0 0 973 0 83 0 417 3143 711 3 14 > 83 > 1 5 0 17384 780988 1842 0 0 0 1762 0 172 0 646 5964 1271 4 21 > 75 > 5 8 0 19908 779424 1527 0 0 0 1249 0 110 0 541 5080 799 4 17 > 79 > > I was also running vmstat -m every 5 seconds - the last output of that is: > > Mon Apr 30 03:43:54 PDT 2001 > Memory Totals: In Use Free Requests > 62374K 12435K 73429925 > > The only notable event I've noticed is that a few minutes prior to each > crash, the number of established connections (netstat -n | grep -c EST) > seems to spike up pretty dramatically. > > About 15 min prior to the 3:45AM crash, the number of established > connections spiked from the 90 - 100 range to about 250, and then dropped > back to the 90 - 100 range. Another, non BSD box, running at the same weight > behind the load balancer stayed in the 90 - 100 connections range at the > time the FBSD box's connections spiked up. I've seen similar behavior with > all the prior crashes. > > I have savecore enabled, but it doesn't seem to be leaving a core when it > reboots. > > My next step is going to be to cvsup to 4.3-stable and enable ddb. I'm doing > this mainly in an effort to be on the latest in the hopes that this was > fixed between 4.2 and 4.3. > > Does anyone have any suggestions? > > TIA! > > - Nick > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-stable" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?030801c0d475$bd1e0fe0$0102010a>