Date: Mon, 7 May 2001 09:41:40 +0300 (EEST) From: Pekka Savola <pekkas@netcore.fi> To: Luigi Rizzo <luigi@info.iet.unipi.it> Cc: <freebsd-stable@freebsd.org> Subject: Re: 4.3-S: No buffer space available [SOLVED: dummynet] Message-ID: <Pine.LNX.4.33.0105070933300.24100-100000@netcore.fi> In-Reply-To: <200105070625.IAA93694@info.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 7 May 2001, Luigi Rizzo wrote: > > On Sat, 5 May 2001, Pekka Savola wrote: > > > > Ok. This was caused by the dummynet rule, either directly or indirectly. > > > > The symptoms: > > 1) 'No buffer space available' > > 2) hard crashes: does not respond to ping, closole goes blank, ethernet > > link led in the switch goes off etc (about 3-4 times in 48h) > > 3) soft crashes: responds to ping, traceroute, tcp establishment ok, but > > the userland is dead (2 times in 48h) > > > > The first rule was: > > --- > > $fwcmd pipe 1 config delay 0 plr 0 bw 20000Kbit/s queue 100 > > $fwcmd add pipe 1 ip from any to any > > --- > > In addition, net.inet.ip.fw.one_pass=0 and 50-500 ipfw rules (assigned > > using external signals). > > dummynet consumes 1 mbuf/cluster per queue entry, so if you only > have one pipe as above that should not be a major problem. > I can only think of some leak, e.g. caused by mishandling of > ipfw reconfigurations, but have no specific example in mind -- > in any case, if the problem is there, you should see the > number of mbufs in use go up with time. This should be no problem, I think -- netstat -m does not show significant increase in mbuf usage; both mbuf and mbuf cluster peaks are well below the average (see the original message). Of course, there _could_ be a surge just before the crash for some odd reason. If I re-enable dummynet, I'll put in a cron job to monitor this along with 'ipfw -ta l' and 'ipfw pipe list'. Any other issues to look for? I had also tried disabling the ""dynamic"" ipfw rule creation; it did not help for the crashing. > I have a comment on your configuration though -- it sounds kind of strange > to have a single pipe limiting everyghing (in and out) upfront, > and then do the actual filtering afterwards. > > It would be much more efficient to do the filtering first, > and replace all "accept" rules with a "skipto 60000" where at > rule 60000 you have the pipe limiting outgoing traffic (and > maybe a different one for limiting input traffic). > This way you would also not need the net.inet.ip.fw.one_pass=0 > setting. Are there significant advantages to this? Only makes this more complex IMO. Worth a test anyway, if all else fails, I suppose. Why dummynet is used here, is because we've been "strongly encouraged" to use only 20Mbit/s max; it has no other traffic shaping functionality here. > Also, the 50-500 rules sound strange, wouldn't you be able to > replace a fair amount of them with dynamic (aka stateful) rules ? No; perhaps I didn't state this clearly. These are not "keep-state" kind of rules; more of "automatic IDS control" kind of rules. Regards, Pekka Savola > > I recompiled the kernel with 100% same options, only leaving dummynet out. > > Everything wortked like a charm. Other things had also been tried, like > > removing SMP support, no luck. > > > > The traffic being shaped to 20Mbit/s ranged from 25-35 Mbit/s (steady), > > mostly outgoing. > > > > Has dummynet been tested in this kind of heavy environment? > > > > Is there a better value for 'queue', e.g. 1000Kbytes in this scenario? > > > > Regards, > > Pekka Savola > > > > > Running on 4.3-S on Dual P3/866 (self-compiled for SMP, dummynet etc.): > > > > > > [root@xxx /root] # snmpwalk xxx yyy > > > snmpwalk: Failure in sendto (No buffer space available) > > > > > > [root@xxx /root] # netstat -m > > > 8012/8576/65536 mbufs in use (current/peak/max): > > > 6581 mbufs allocated to data > > > 1431 mbufs allocated to packet headers > > > 6300/6682/16384 mbuf clusters in use (current/peak/max) > > > 15508 Kbytes allocated to network (31% of mb_map in use) > > > 0 requests for memory denied > > > 0 requests for memory delayed > > > 0 calls to protocol drain routines > > > > > > [root@xxx /root] # sysctl kern | grep files > > > kern.maxfiles: 16424 > > > kern.maxfilesperproc: 16424 > > > kern.openfiles: 2709 > > > > > > --- > > > last pid: 84147; load averages: 0.52, 0.66, 0.61 up 0+01:38:21 13:12:34 > > > 713 processes: 12 running, 688 sleeping, 13 stopped > > > CPU states: 14.1% user, 0.0% nice, 20.7% system, 4.0% interrupt, 61.3% idle > > > Mem: 516M Active, 316M Inact, 134M Wired, 37M Cache, 112M Buf, 1664K Free > > > Swap: 1024M Total, 92K Used, 1024M Free > > > --- > > > > > > Pushing through 20 Mbit/s steady as we speak. > > > > > > This seems to have been mentioned in the beginning of April with no clear > > > resolution. > > > > > > A few kernel options mentioned in the posts: > > > --- > > > cpu I686_CPU > > > maxusers 512 > > > options NMBCLUSTERS=16384 > > > options SMP # Symmetric MultiProcessor Kernel > > > options APIC_IO # Symmetric (APIC) I/O > > > --- > > > This is a system with 1 GB of RAM. Network card is: > > > > > > fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0xccc0-0xccff mem > > > 0xf9000000-0xf90fffff,0xf9100000-0xf9100fff irq 10 at device 8.0 on pci1 > > > > > > > > > What's wrong? There definitely should be enough buffers. > > > > > > Also: userspace froze earlier today for some odd reason; ping and > > > traceroute responded, ipfw worked, tcp connection could be established but > > > the daemon listening to the port never replied. Nothing in the log or the > > > console. Ideas? > > > > > > Please Cc:. > > > > > > > > > > -- > > Pekka Savola "Tell me of difficulties surmounted, > > Netcore Oy not those you stumble over and fall" > > Systems. Networks. Security. -- Robert Jordan: A Crown of Swords > > > > > > > -- Pekka Savola "Tell me of difficulties surmounted, Netcore Oy not those you stumble over and fall" Systems. Networks. Security. -- Robert Jordan: A Crown of Swords To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.LNX.4.33.0105070933300.24100-100000>