Date: Sun, 5 May 2002 22:46:17 -0500 From: "Maildrop" <maildrop@qwest.net> To: "Benjamin Krueger" <benjamin@macguire.net>, "Maildrop" <maildrop@qwest.net>, "Bill Moran" <wmoran@potentialtech.com> Cc: questions@freebsd.org Subject: RE: Networking Buffers Message-ID: <NGBBIILBAKIFGHHCHOHPMEILCOAA.maildrop@qwest.net> In-Reply-To: <20020503001955.B86614@rain.macguire.net>
next in thread | previous in thread | raw e-mail | index | archive | help
I do appericate your suggestions, I apoligize if my tone was a little rough
in the last posts, the frustration and lack of sleep had got the better of
me :O
I changed NMBCLUSTERS to 32768, 65535 and also tried using
net.inet.tcp.always_keepalive to both 1 and 0. I tried also
kern.ipc.somaxconn at default (100 something?), 1024, 2048 and 10,000,000.
None of this options have had a positive effect.
I tried a differant network card. No good. I rebuilt then entire system
and the default 4.5-rel out of the box (ie. Generic, no patches) crashes
even more than this and decides to take the console with it. :( I patched
back up to 4.5-Stable and have generic kernel, expect for NMBCLUSTERS (which
is set to 32768).
I tried a differant model network card. No good.
I think it has something to do with coping more than one file at the same
time (for example the last time it crashed it was apox 6 files copies, 2
over smb, 4 over nfs and the average file size was 500 megs. This was take
litteraly 20 seconds before it crashed (expect for all.log, that is 5
minutes after I got the machine back up (did a ifconfig down; ifconfig up)
bud@hydra:/usr/src/sys/i386/conf/ --> netstat -m
133/704/131072 mbufs in use (current/peak/max):
131 mbufs allocated to data
2 mbufs allocated to packet headers
129/288/32768 mbuf clusters in use (current/peak/max)
752 Kbytes allocated to network (0% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
bud@hydra:/usr/src/sys/i386/conf/ --> netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs
Coll
fxp0 1500 <Link#1> 00:03:47:40:2c:93 67561 0 114108 0
0
fxp0 1500 192.168.17 hydra 35325 -
4227 - -
dc0 1500 <Link#2> 00:03:6d:1a:5a:31 331655 0 235408 0
0
dc0 1500 64.231.238.22 64.231.238.225 291355 -
5387 - -
dc0 1500 64.231.238.22 64.231.238.226 3 -
0 - -
dc0 1500 64.231.238.22 64.231.238.227 436 -
0 - -
dc0 1500 64.231.238.22 64.231.238.228 10 -
0 - -
dc0 1500 64.231.238.22 64.231.238.229 9 -
0 - -
lo0 16384 <Link#3> 12675 0 12675 0
0
lo0 16384 your-net localhost 12519 -
2519 - -
bud@hydra:/usr/src/sys/i386/conf/ --> ifconfig -a
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 192.168.17.1 netmask 0xffffff00 broadcast 192.168.17.255
ether 00:03:47:40:2c:93
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
dc0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 64.231.238.225 netmask 0xfffffff8 broadcast 64.231.238.231
inet 64.231.238.226 netmask 0xffffffff broadcast 64.231.238.226
inet 64.231.238.227 netmask 0xffffffff broadcast 64.231.238.227
inet 64.231.238.228 netmask 0xffffffff broadcast 64.231.238.228
inet 64.231.238.229 netmask 0xffffffff broadcast 64.231.238.229
ether 00:03:6d:1a:5a:31
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
inet 127.0.0.1 netmask 0xff000000
bud@hydra:/usr/src/sys/i386/conf/ --> netstat -s | grep con
82 control packets
13 connection requests
64 connection accepts
0 bad connection attempts
76 connections established (including accepts)
75 connections closed (including 37 drops)
13 connections updated cached RTT on close
13 connections updated cached RTT variance on close
6 connections updated cached ssthresh on close
1 embryonic connection dropped
0 connections dropped by rexmit timeout
0 connections dropped by persist timeout
0 connections dropped by keepalive
bud@hydra:/usr/src/sys/i386/conf/ --> tail /var/log/all.log
May 5 21:55:00 hydra /usr/sbin/cron[960]: (bud) CMD (/home/bud/bin/mrtg.sh)
May 5 21:58:18 hydra su: bud to root on /dev/ttyp7
May 5 21:58:18 hydra /kernel: May 5 21:58:18 hydra su: bud to root on
/dev/ttyp7
May 5 22:00:00 hydra /usr/sbin/cron[998]: (root) CMD (newsyslog -v )
May 5 22:00:00 hydra /usr/sbin/cron[999]: (root) CMD (/usr/libexec/atrun)
May 5 22:00:00 hydra /usr/sbin/cron[1000]: (bud) CMD
(/home/bud/bin/mrtg.sh)
May 5 22:00:01 hydra sendmail[1006]: gethostbyaddr(64.231.238.225) failed:
1
May 5 22:00:01 hydra sendmail[1006]: gethostbyaddr(64.231.238.226) failed:
1
May 5 22:00:01 hydra sendmail[1006]: gethostbyaddr(64.231.238.227) failed:
1
May 5 22:00:01 hydra sendmail[1006]: gethostbyaddr(64.231.238.228) failed:
1
May 5 22:00:01 hydra sendmail[1006]: gethostbyaddr(64.231.238.229) failed:
1
May 5 22:00:01 hydra sendmail[1006]: g46301p01006: from=root, size=1180,
class=0, nrcpts=1, msgid=<200205060300.g46301p01006@63.231.238.225>,
relay=root@localhost
May 5 22:00:02 hydra sendmail[1006]: g46301p01006: to=bud, ctladdr=root
(0/0), delay=00:00:01, xdelay=00:00:01, mailer=local, pri=31180,
relay=local, dsn=2.0.0, stat=Sent
May 5 22:03:41 hydra inetd[1013]: connection from jenny, service qpopper
(tcp)
May 5 22:05:00 hydra /usr/sbin/cron[1016]: (root) CMD (/usr/libexec/atrun)
May 5 22:05:00 hydra /usr/sbin/cron[1017]: (bud) CMD
(/home/bud/bin/mrtg.sh)
May 5 22:10:00 hydra /usr/sbin/cron[1027]: (bud) CMD
(/home/bud/bin/mrtg.sh)
May 5 22:10:00 hydra /usr/sbin/cron[1026]: (root) CMD (/usr/libexec/atrun)
May 5 22:13:25 hydra su: bud to root on /dev/ttyp7
May 5 22:13:25 hydra /kernel: May 5 22:13:25 hydra su: bud to root on
/dev/ttyp7
The current settings before the crash where 32768 NBMCLUSTERS, keepalive =
0, maxconn = 1024
Everything looks normal, do you guys see anything?
Also, I look those the documents you guys suggested and I still can't fine a
complete and update info on sysctl and kernel options. The pages you sent
where helpful, but has less than 10% of all the options avaiable and it
scattered around. Maybe I am looking at the wrong thing?
Any insight on what might be going on?
Regards,
Jack
> -----Original Message-----
> From: Benjamin Krueger [mailto:benjamin@macguire.net]
> Sent: Friday, May 03, 2002 2:20 AM
> To: Maildrop
> Subject: Re: Networking Buffers
>
>
> * Maildrop (maildrop@qwest.net) [020502 23:47]:
> >
> > WTF, now this is happening on both interfaces :( sigh
> >
> > ping: socket: No buffer space available
> > telnet: socket: No buffer space available
> >
> > Anyone know why FreeBSD is starting to suck so bad when it comes to
> > networking?
> >
> > J
>
> My mail server is borked right now, so I can't send to the list.
> Basically,
> this is user config error. ;) You need to increase the
> available amount of
> network buffers your server has. When you run out of mbufs, thats
> it. Until
> those network connections time out, you're just going to wait.
>
> Here's the scoop. When you compiled the kernel, you set an option called
> NMBCLUSTERS. The default is likely too small. Run netstat -m and
> you'll likely
> see that all available mbufs are being used. You can adjust the number of
> available nmbclusters using the kern.ipc.nmbclusters sysctl oid.
>
> The number you will want to use will be highly dependant on what
> kind of load
> you're running. See tuning(7) and search for mbuf. It has a short
> and simple
> guide to figuring out a decent number to enter for nmbclusters.
> Once you get
> that squared away and tuned, your FreeBSD server should run
> happily again. You
> might consider investing some time into reading some of the
> advanced tutorials
> in the documentation, as well as the entire tuning(7) page. They
> all make for
> excellent reading.
>
> -ben
>
> > > The issuse I am having on it, is it seems I am "blowing out"
> one of the
> > > nics.
> > > I am sure this FreeBSD related, because this box use to run W2K with a
> > > higher
> > > network load and didn't expeirence any issuses.
> > >
> > > After a high spike it network load (when we backup all data in
> > > the middle of
> > > the night) one
> > > of the ip address will stop responding. I log in locally and
> the network
> > > tables are fine, but
> > > when I try to ping out of that network card it gives the error message
> > > "ping: no buffers avaiable"
> > >
> > > Taking the nic up then back down (`ifconfig fxp0 down;
> ifconfig fxp0 up`)
> > > will fix the issuse, but
> > > is a huge problem because it is very distrubating to network
> operation,
> > > espically during nightly
> > > backups/data merges when no one is onsite. I replaced the nic
> > > with a brand
> > > new one that is the same
> > > model, and also with a new one that is a differant
> brand/model and it is
> > > still having issuses.
> > >
> > > In sysctl I changed (IIRC) "net.inet.tcp.sendspace" and
> > > "net.inet.recvspace"
> > > both to 8192 (thinking it is Kb?),
> > > that still had issuses, so I change it to 8192000 (thinking
> it might be
> > > bytes) and also tried 8192000000.
> > > All these settings still cause FreeBSD to drop the nic. What is
> > > the size of
> > > these variables? (ie. mb, kb, etc ?)
> > >
> > > Also, are these the correct settings? Is network buffers
> configured some
> > > where else? I have yet to find
> > > complete and full documentation on sysctl, anyone know where
> I could get
> > > these docs, freebsd.com has very
> > > incomplete and out of data documentation in regards to sysctl
> > > variables and
> > > kernel tweaking in general...
> > >
> > > Also is there any type of intelligent networking buffering in FreeBSD?
> > > Something that would
> > > say "If network buffers are full, pause network traffic,
> flush buffers,
> > > continue with network
> > > traffic" ? This way, if the buffers are full, it won't take
> out the nic
> > > completly,
> > > but rather hang it, and then continue after the buffers had
> be purged? I
> > > think this would be a more
> > > graceful method of handling this, instead of "Opps, buffers are
> > > full, sorry
> > > but your server is useless now"...
> > > Anyone know how to enable this?
> > >
> > > Also it is very strange that something this critical isn't logged
> > > to syslog,
> > > what do I have to have in
> > > /etc/syslog.conf to get this info, I basically turned on everything to
> > > debug, and didn't see anything to
> > > messages, console or all.log.
> > >
> > > It is kinda funny, this server went down again when I was typing this
> > > email.. I can tell when it goes down
> > > cause, winamp disconnects from server and then I hear the
> solaris admin 4
> > > rooms down swearing at freebsd :)
> > >
> > > I got to get this fixed soon or will be forced to move it
> back to W2K, any
> > > ideas?
> > >
> > > Regards,
> > > Jack
>
> --
> Benjamin Krueger
>
> "Life is far too important a thing ever to talk seriously about."
> - Oscar Wilde (1854 - 1900)
> ----------------------------------------------------------------
> Send mail w/ subject 'send public key' or query for (0x251A4B18)
> Fingerprint = A642 F299 C1C1 C828 F186 A851 CFF0 7711 251A 4B18
>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?NGBBIILBAKIFGHHCHOHPMEILCOAA.maildrop>