Date: Mon, 19 Jul 1999 16:20:55 -0600 From: "Tim Pushor" <timp@orion.ab.ca> To: "Wes Peters" <wes@softweyr.com> Cc: <questions@freebsd.org> Subject: Fw: 'Out of buffer space' problem Message-ID: <00a601bed235$0c0bc470$9828f99f@shl.com>
next in thread | raw e-mail | index | archive | help
I have upped the NMBCLUSTERS to 8K, and I have sendmail and named running
concurrently now without a problem, but we are also past the busiest part of
the day..
output of netstat -m
68 mbufs in use:
24 mbufs allocated to data
37 mbufs allocated to packet headers
5 mbufs allocated to protocol control blocks
2 mbufs allocated to socket names and addresses
20/120 mbuf clusters in use
248 Kbytes allocated to network (19% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
output of netstat -s:
ip:
232198 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
229111 packets for this host
3081 packets for unknown/unsupported protocol
0 packets forwarded
0 packets not forwardable
0 redirects sent
227772 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
3 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
icmp:
508 calls to icmp_error
0 errors not generated 'cuz old message was icmp
Output histogram:
echo reply: 4716
destination unreachable: 508
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
destination unreachable: 3080
source quench: 1
echo: 4716
4716 message responses generated
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
tcp:
80121 packets sent
47392 data packets (30732589 bytes)
142 data packets (29394 bytes) retransmitted
0 resends initiated by MTU discovery
16163 ack-only packets (5152 delayed)
0 URG only packets
4 window probe packets
10933 window update packets
5487 control packets
83089 packets received
42473 acks (for 30724312 bytes)
2931 duplicate acks
0 acks for unsent data
48834 packets (30291382 bytes) received in-sequence
476 completely duplicate packets (340324 bytes)
0 old duplicate packets
7 packets with some dup. data (3185 bytes duped)
6296 out-of-order packets (4317486 bytes)
3 packets (4380 bytes) of data after window
0 window probes
175 window update packets
5 packets received after close
3 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
2610 connection requests
1228 connection accepts
31 bad connection attempts
0 listen queue overflows
2887 connections established (including accepts)
3785 connections closed (including 45 drops)
116 connections updated cached RTT on close
116 connections updated cached RTT variance on close
80 connections updated cached ssthresh on close
585 embryonic connections dropped
36207 segments updated rtt (of 37327 attempts)
1175 retransmit timeouts
3 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
1 keepalive timeout
0 keepalive probes sent
1 connection dropped by keepalive
8795 correct ACK header predictions
31461 correct data packet header predictions
udp:
141306 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
508 dropped due to no socket
1305 broadcast/multicast datagrams dropped due to no socket
0 dropped due to full socket buffers
0 not for hashed pcb
139493 delivered
142362 datagrams output
The counts are low because I just upped the NMBCLUSTERS to 8K and rebooted a
while ago..
----- Original Message -----
From: Wes Peters <wes@softweyr.com>
To: Tim Pushor <timp@orion.ab.ca>
Cc: <questions@freebsd.org>; <net@freebsd.org>
Sent: Monday, July 19, 1999 4:11 PM
Subject: Re: 'Out of buffer space' problem
> Tim Pushor wrote:
> > Wes Peters ranted:
> > > Tim Pushor wrote:
> > > >
> > > > I work for a rather large organization and have convinced management
to
> > > > replace our aging AIX SMTP relays/DNS servers with Compaq Servers
running
> > > > FreeBSD. Saturday I had officially turned off our old AIX boxes and
were
> > > > running on pure FreeBSD boxes.
> > > >
> > > > On the first full day of production, half way through (today) one of
the
> > > > boxes that the companies primary DNS was running on stopped
responding. The
> > > > error I was getting was 'out of buffer space'.
> > > >
> > > > I am now in Panic mode, as I am the one responsible for reccomending
this
> > > > solution.
> > > >
> > > > Can anyone out there help me track the problem down? I know there
can be
> > > > several factors involved, but I *really* have to get this one licked
rather
> > > > quickly.
> > >
> > > Sure, but you'll need to provide some useful information about your
systems.
> > > What version of FreeBSD and BIND are you running? What is the
configuration
> > > of your machine -- CPU(s), memory, and network cards certainly. Have
you
> > > compiled a custom kernel for the machine; if so include the kernel
config
> > > file you're using. If not, the problem is simple to diagnose: you
need
> > > more network buffer space to handle the load. The simplest way to do
this
> > > is to increase the "maxusers" figure in the configuration until the
problem
> > > stops. If you're running the 3.2-RELEASE generic kernel, raise
maxusers to
> > > 64 and try again.
> > >
> > > Please, post more information so we can help you. We don't want you
to (and
> > > us by reference) to get embarrased on this. ;^)
> >
> > I'm sorry for not providing enough information, I am in panic mode here.
> > Thanks for listening to me ;-) I *really* hope someone can help..
>
> OK, now we're getting somewhere. A quick point, though: until you've
> gotten a solution, keep mailing to the lists as well. I'm not necessarily
> the configuration expert, so we'll want to get lots of eyeballs on this.
> I've forwarded your reply and my observations to the original lists,
> which were probably a good starting place.
>
> > Hardware:
> >
> > Compaq Proliant 1600 PII-450 W/512M RAM, Integrated NCR SCSI, one 4G
Wide
> > SCSI disk, Integrated ThunderLAN 10/100 NIC
>
> That should be sufficient for a DNS server.
>
> > Software:
> >
> > FreeBSD 2.2.8-RELEASE
> > Stock named (4.9.4?)
> > Maxusers 256 (so NMBclusters should be 4608)
>
> A reasonable starting point.
>
> > Config file:
> > ** BTW this is not really GENERIC
>
> So change the comments, like:
>
>
> > #
> # SHLRELAY, created dd/mm/yyyy from:
> > # GENERIC -- Generic machine with WD/AHx/NCR/BTx family disks
> > #
> > # For more information read the handbook part System Administration ->
> > # Configuring the FreeBSD Kernel -> The Configuration File.
> > # The handbook is available in /usr/share/doc/handbook or online as
> > # latest version from the FreeBSD World Wide Web server
> > # <URL:http://www.FreeBSD.ORG/>;
> > #
> > # An exhaustive list of options and more detailed explanations of the
> > # device lines is present in the ./LINT configuration file. If you are
> > # in doubt as to the purpose or necessity of a line, check first in
LINT.
> > #
> > # $Id: GENERIC,v 1.77.2.28 1998/09/26 17:36:14 wpaul Exp $
> > machine "i386"
> > cpu "I586_CPU"
> > cpu "I686_CPU"
> > ident SHLRELAY
> > maxusers 256
> > options INET #InterNETworking
> > options FFS #Berkeley Fast Filesystem
> > options MSDOSFS #MSDOS Filesystem
> > options "CD9660" #ISO 9660 Filesystem
> > options PROCFS #Process filesystem
> > options "COMPAT_43" #Compatible with BSD 4.3 [KEEP
> > THIS!]
> > options SCSI_DELAY=15 #Be pessimistic about Joe SCSI
> > device
> > options FAILSAFE #Be conservative
> > options IPFIREWALL
> > options "MAXMEM=(512*1024)"
> > options INCLUDE_CONFIG_FILE
> > config kernel root on wd0
> > controller isa0
> > #controller eisa0
> > controller pci0
> > controller fdc0 at isa? port "IO_FD1" bio irq 6 drq 2 vector
fdintr
> > disk fd0 at fdc0 drive 0
> > disk fd1 at fdc0 drive 1
> > options "CMD640" # work around CMD640 chip deficiency
> > controller wdc0 at isa? port "IO_WD1" bio irq 14 vector wdintr
> > controller wdc1 at isa? port "IO_WD2" bio irq 15 vector wdintr
> > options ATAPI #Enable ATAPI support for IDE bus
> > options ATAPI_STATIC #Don't do it as an LKM
> > device wcd0 #IDE CD-ROM
> > # A single entry for any of these controllers (ncr, ahb, ahc, amd) is
> > # sufficient for any number of installed devices.
> > controller ncr0
> > controller scbus0
> > device sd0
> > device st0
> > device cd0 #Only need one of these, the code dynamically
grows
> > # syscons is the default console driver, resembling an SCO console
> > device sc0 at isa? port "IO_KBD" tty irq 1 vector scintr
> > # Mandatory, don't remove
> > device npx0 at isa? port "IO_NPX" flags 0x1 irq 13 vector
> > npxintr
> > device sio0 at isa? port "IO_COM1" tty irq 4 vector siointr
> > device sio1 at isa? port "IO_COM2" tty irq 3 vector siointr
> > device lpt0 at isa? port? tty irq 7 vector lptintr
> > device tl0
> > pseudo-device loop
> > pseudo-device ether
> > pseudo-device log
> > pseudo-device vn 4
> > pseudo-device tun 4
> > pseudo-device pty 32
> > pseudo-device gzip # Exec gzipped a.out's
> > pseudo-device bpfilter 4
> > # This provides support for System V shared memory.
> > #
> > options SYSVSHM
> > options SYSVMSG
> > options SYSVSEM
>
> I don't see anything obviously wrong here, either. Send the output of
both
> netstat -m and netstat -s, so we can see what's going on in the network
stack.
>
> --
> "Where am I, and what am I doing in this handbasket?"
>
> Wes Peters
Softweyr LLC
> http://softweyr.com/
wes@softweyr.com
>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?00a601bed235$0c0bc470$9828f99f>