From owner-freebsd-stable Sun May 6 21:51:48 2001 Delivered-To: freebsd-stable@freebsd.org Received: from netcore.fi (netcore.fi [193.94.160.1]) by hub.freebsd.org (Postfix) with ESMTP id 0892837B422; Sun, 6 May 2001 21:51:43 -0700 (PDT) (envelope-from pekkas@netcore.fi) Received: from localhost (pekkas@localhost) by netcore.fi (8.11.1/8.11.1) with ESMTP id f474pf223640; Mon, 7 May 2001 07:51:41 +0300 Date: Mon, 7 May 2001 07:51:41 +0300 (EEST) From: Pekka Savola To: Cc: Subject: Re: 4.3-S: No buffer space available [SOLVED: dummynet] In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Sat, 5 May 2001, Pekka Savola wrote: Ok. This was caused by the dummynet rule, either directly or indirectly. The symptoms: 1) 'No buffer space available' 2) hard crashes: does not respond to ping, closole goes blank, ethernet link led in the switch goes off etc (about 3-4 times in 48h) 3) soft crashes: responds to ping, traceroute, tcp establishment ok, but the userland is dead (2 times in 48h) The first rule was: --- $fwcmd pipe 1 config delay 0 plr 0 bw 20000Kbit/s queue 100 $fwcmd add pipe 1 ip from any to any --- In addition, net.inet.ip.fw.one_pass=0 and 50-500 ipfw rules (assigned using external signals). I recompiled the kernel with 100% same options, only leaving dummynet out. Everything wortked like a charm. Other things had also been tried, like removing SMP support, no luck. The traffic being shaped to 20Mbit/s ranged from 25-35 Mbit/s (steady), mostly outgoing. Has dummynet been tested in this kind of heavy environment? Is there a better value for 'queue', e.g. 1000Kbytes in this scenario? Regards, Pekka Savola > Running on 4.3-S on Dual P3/866 (self-compiled for SMP, dummynet etc.): > > [root@xxx /root] # snmpwalk xxx yyy > snmpwalk: Failure in sendto (No buffer space available) > > [root@xxx /root] # netstat -m > 8012/8576/65536 mbufs in use (current/peak/max): > 6581 mbufs allocated to data > 1431 mbufs allocated to packet headers > 6300/6682/16384 mbuf clusters in use (current/peak/max) > 15508 Kbytes allocated to network (31% of mb_map in use) > 0 requests for memory denied > 0 requests for memory delayed > 0 calls to protocol drain routines > > [root@xxx /root] # sysctl kern | grep files > kern.maxfiles: 16424 > kern.maxfilesperproc: 16424 > kern.openfiles: 2709 > > --- > last pid: 84147; load averages: 0.52, 0.66, 0.61 up 0+01:38:21 13:12:34 > 713 processes: 12 running, 688 sleeping, 13 stopped > CPU states: 14.1% user, 0.0% nice, 20.7% system, 4.0% interrupt, 61.3% idle > Mem: 516M Active, 316M Inact, 134M Wired, 37M Cache, 112M Buf, 1664K Free > Swap: 1024M Total, 92K Used, 1024M Free > --- > > Pushing through 20 Mbit/s steady as we speak. > > This seems to have been mentioned in the beginning of April with no clear > resolution. > > A few kernel options mentioned in the posts: > --- > cpu I686_CPU > maxusers 512 > options NMBCLUSTERS=16384 > options SMP # Symmetric MultiProcessor Kernel > options APIC_IO # Symmetric (APIC) I/O > --- > This is a system with 1 GB of RAM. Network card is: > > fxp0: port 0xccc0-0xccff mem > 0xf9000000-0xf90fffff,0xf9100000-0xf9100fff irq 10 at device 8.0 on pci1 > > > What's wrong? There definitely should be enough buffers. > > Also: userspace froze earlier today for some odd reason; ping and > traceroute responded, ipfw worked, tcp connection could be established but > the daemon listening to the port never replied. Nothing in the log or the > console. Ideas? > > Please Cc:. > > -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message