Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 28 Jul 2000 13:17:46 -0400 (EDT)
From:      gallatin@freebsd.org
To:        freebsd-stable@freebsd.org
Cc:        dillon@freebsd.org
Subject:   NFS server running out of bufs & locking up
Message-ID:  <14721.48065.766815.376959@grasshopper.cs.duke.edu>

next in thread | raw e-mail | index | archive | help

I have an NFS server which I updated to RELENG_4 (as of Jul 13th) from
4.0-current (as of Jul 13th 1999).  Since the upgrade, it has locked
up 3 times; it had been up for 180+ days prior to the upgrade.

The machine serves a large (64GB) volume stripped across 4 ATA drives
with CCD mounted with soft updates.  When it locks up, it is getting
beaten upon by a compute farm of 50+ Solaris boxes running NFS over
TCP (via 100Mb ethernet).

When it locks, most procs are waiting in biofre, and the nfsd's are
wating on inode.  I've been unable to get a dump, the most I have
is ps from ddb. (appended below).  Its somewhat interesting that 
3 of the nfsds are waiting on the same inode

Stopped at      siointr1+0xb1:  jmp     siointr1+0x1a0
db> ps
  pid   proc     addr    uid  ppid  pgrp  flag stat wmesg   wchan   cmd
  494 d28fb2a0 d292a000    0   120   120 002004  3  biofre c02aa6d8 amd
  493 d0ad9c20 d2869000    0   373   492 004006  3  biofre c02aa6d8 grep
  473 d0ad7380 d28df000 1597   320   473 004006  3  biofre c02aa6d8 netdump_server
  460 d28fbe00 d28fc000 1597   458   460 004086  3   ttyin c1729630 tcsh
  458 d0ad76c0 d28c8000    0   205   205 000084  3  select c02be9ec sshd1
  394 d28fb440 d290f000    1     1   394 000104  3  biofre c02aa6d8 portmap
  373 d0ad6ea0 d28eb000    0   285   373 2004086  3  opause d28eb108 tcsh
  320 d0ad7040 d28e4000 1597   317   320 2004086  3  opause d28e4108 tcsh
  317 d0ad71e0 d28e9000    0   205   205 000084  3  select c02be9ec sshd1
  285 d0ad7520 d28db000 1387   283   285 2004086  3  opause d28db108 tcsh
  283 d0ad7860 d28c4000    0   205   205 000084  3  select c02be9ec sshd1
  260 d0ad7a00 d28bf000 1387   259   260 004106  3  biofre c02aa6d8 systat
  259 d0ad83c0 d2899000 1387   233   259 004186  3  select c02be9ec xterm
  233 d0ada440 d284e000 1387   230   233 004006  3   inode c16f8000 tcsh
  230 d0ad8220 d289d000    0   205   205 000004  3  biofre c02aa6d8 sshd1
  223 d0ada5e0 d284b000    0     1   223 004006  3  biofre c02aa6d8 getty
  218 d0ad7ba0 d28bd000    0     1   218 000084  3  sbwait d0668acc zhm
  205 d0ad7d40 d28b1000    0     1   205 000084  3  select c02be9ec sshd1
  147 d0ad8080 d28a1000    0     1   147 2000184  3   pause d28a1108 sendmail
  144 d0ad7ee0 d28a4000    0     1   144 000084  3  nanslp c02aa580 cron
  142 d0ad9a80 d286c000    0     1   142 000084  3  select c02be9ec inetd
  120 d0ad8be0 d2889000    0     1   120 000084  3  select c02be9ec amd
  115 d0ad8560 d2895000    0     1   110 000084  3  nfsidl c02c0d4c nfsiod
  114 d0ad8700 d2892000    0     1   110 000084  3  nfsidl c02c0d48 nfsiod
  113 d0ad88a0 d288f000    0     1   110 000084  3  nfsidl c02c0d44 nfsiod
  112 d0ad8a40 d288c000    0     1   110 000084  3  nfsidl c02c0d40 nfsiod
  108 d0ad8d80 d2886000    0     1   108 000084  3  select c02be9ec rpc.statd
  105 d0ad8f20 d2882000    0   100   100 000004  3   inode c16c3400 nfsd
  104 d0ad90c0 d287f000    0   100   100 000004  3   inode c16c3400 nfsd
  103 d0ad9260 d287c000    0   100   100 000004  3   inode c16c3400 nfsd
  102 d0ad9400 d2878000    0   100   100 000004  3  biofre c02aa6d8 nfsd
  100 d0ad95a0 d2875000    0     1   100 000084  3  accept d06663f6 nfsd
   98 d0ad9740 d2872000    0     1    98 000084  3  select c02be9ec mountd
   92 d0ad98e0 d286f000    0     1    92 000084  3  select c02be9ec ypbind
   87 d0ad9dc0 d2866000    0     1    87 000084  3  select c02be9ec ntpd
   80 d0ad9f60 d285c000    0     1    80 000084  3  select c02be9ec syslogd
   33 d0ada100 d2858000    0     1    33 2000084  3   pause d2858108 adjkerntz
   25 d0ada2a0 d2855000    0     1    25 000084  3  mfsidl d0ad3d00 mount_mfs
    5 d0ada780 d0ae7000    0     0     0 000204  3  biofre c02aa6d8 syncer
    4 d0ada920 d0ae5000    0     0     0 100204  3  psleep c02aa6a8 bufdaemon
    3 d0adaac0 d0ae3000    0     0     0 000204  3  psleep c02b5fa0 vmdaemon
    2 d0adac60 d0ae1000    0     0     0 100204  3  psleep c029c8b8 pagedaemon
    1 d0adae00 d0adf000    0     0     1 004284  3    wait d0adae00 init
    0 c02bdd80 c0322000    0     0     0 000204  3   sched c02bdd80 swapper


About 30 seconds before this lockup, I was looking at how much buf
space is available:

#sysctl -a | grep buf
kern.ipc.maxsockbuf: 262144
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.mbuf_wait: 32
kern.ipc.nmbufs: 10240
vfs.nfs.bufpackets: 0
vfs.numdirtybuffers: 18
vfs.hidirtybuffers: 796
vfs.numfreebuffers: 3083
vfs.lofreebuffers: 177
vfs.hifreebuffers: 354
vfs.runningbufspace: 32768
vfs.maxbufspace: 50872320
vfs.hibufspace: 50216960
vfs.lobufspace: 50151424
vfs.bufspace: 50151424
vfs.maxmallocbufspace: 2510848
vfs.bufmallocspace: 4096
vfs.getnewbufcalls: 512584
vfs.getnewbufrestarts: 0
vfs.bufdefragcnt: 0
vfs.buffreekvacnt: 0
vfs.bufreusecnt: 3061
vfs.reassignbufcalls: 426411
vfs.reassignbufloops: 0
vfs.reassignbufsortgood: 144583
vfs.reassignbufsortbad: 4776
vfs.reassignbufmethod: 1
vfs.aio.max_buf_aio: 16
vfs.aio.num_buf_aio: 0
debug.bpf_bufsize: 4096
debug.bpf_maxbufsize: 524288
machdep.msgbuf: 
machdep.msgbuf_clear: 0


I have appended my config file & boot messages.

Thanks for any help you can give,

Drew

------------------------------------------------------------------------------
Andrew Gallatin, Sr Systems Programmer	http://www.cs.duke.edu/~gallatin
Duke University				Email: gallatin@cs.duke.edu
Department of Computer Science		Phone: (919) 660-6590


Copyright (c) 1992-2000 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 4.0-STABLE #0: Thu Jul 13 12:11:33 EDT 2000
    gallatin@grits.cs.duke.edu:/usr/src/sys/compile/NFSSERVER
Timecounter "i8254"  frequency 1193182 Hz
Timecounter "TSC"  frequency 451024727 Hz
CPU: Pentium II/Pentium II Xeon/Celeron (451.02-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x652  Stepping = 2
  Features=0x183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR>
real memory  = 402640896 (393204K bytes)
avail memory = 388145152 (379048K bytes)
Preloaded elf kernel "kernel" at 0xc030f000.
ccd0-3: Concatenated disk drivers
Pentium Pro MTRR support enabled
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Intel 82443BX (440 BX) host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
pcib1: <Intel 82443BX (440 BX) PCI-PCI (AGP) bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
isab0: <Intel 82371AB PCI to ISA bridge> at device 4.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 ATA33 controller> port 0xd800-0xd80f at device 4.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
pci0: <Intel 82371AB/EB (PIIX4) USB controller> at 4.2
chip1: <Intel 82371AB Power management controller> port 0xe800-0xe80f at device 4.3 on pci0
atapci1: <Promise ATA33 controller> port 0xa800-0xa81f,0xb004-0xb007,0xb400-0xb407,0xb804-0xb807,0xd000-0xd007 irq 12 at device 9.0 on pci0
ata2: at 0xd000 on atapci1
ata3: at 0xb400 on atapci1
fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0xa400-0xa41f mem 0xe2000000-0xe20fffff,0xe3000000-0xe3000fff irq 10 at device 10.0 on pci0
fxp0: Ethernet address 00:a0:c9:e7:95:bb
atapci2: <Promise ATA33 controller> port 0x8800-0x881f,0x9004-0x9007,0x9400-0x9407,0x9804-0x9807,0xa000-0xa007 irq 11 at device 12.0 on pci0
ata4: at 0xa000 on atapci2
ata5: at 0x9400 on atapci2
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppi0: <Parallel I/O> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
plip0: <PLIP network interface> on ppbus0
ad0: 4892MB <QUANTUM FIREBALL EX5.1A> [10602/15/63] at ata0-master using UDMA33
ad1: 16479MB <Maxtor 91728D8> [33483/16/63] at ata2-master using UDMA33
ad2: 16479MB <Maxtor 91728D8> [33483/16/63] at ata3-master using UDMA33
ad3: 16479MB <Maxtor 91728D8> [33483/16/63] at ata4-master using UDMA33
ad4: 16479MB <Maxtor 91728D8> [33483/16/63] at ata5-master using UDMA33
Mounting root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted


## NFSSERVER
machine		i386
cpu		I686_CPU
ident		NFSSERVER
maxusers	128

#makeoptions	DEBUG=-g		#Build kernel with gdb(1) debug symbols

options 	INET			#InterNETworking
options 	FFS			#Berkeley Fast Filesystem
options 	FFS_ROOT		#FFS usable as root device [keep this!]
options 	SOFTUPDATES		#Enable FFS soft updates support
options 	MFS			#Memory Filesystem
options 	NFS			#Network Filesystem
options 	MSDOSFS			#MSDOS Filesystem
options 	CD9660			#ISO 9660 Filesystem
options 	CD9660_ROOT		#CD-ROM usable as root, CD9660 required
options 	PROCFS			#Process filesystem
options 	COMPAT_43		#Compatible with BSD 4.3 [KEEP THIS!]
options 	SCSI_DELAY=1500	#Delay (in ms) before probing SCSI
options 	UCONSOLE		#Allow users to grab the console
options 	USERCONFIG		#boot -c editor
options 	VISUAL_USERCONFIG	#visual boot -c editor
options 	KTRACE			#ktrace(1) support
options 	SYSVSHM			#SYSV-style shared memory
options 	SYSVMSG			#SYSV-style message queues
options 	SYSVSEM			#SYSV-style semaphores
options 	P1003_1B		#Posix P1003_1B real-time extensions
options 	_KPOSIX_PRIORITY_SCHEDULING
options		ICMP_BANDLIM		#Rate limit bad replies
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev

# To make an SMP kernel, the next two are needed
#options 	SMP			# Symmetric MultiProcessor Kernel
#options 	APIC_IO			# Symmetric (APIC) I/O
# Optionally these may need tweaked, (defaults shown):
#options 	NCPU=2			# number of CPUs
#options 	NBUS=4			# number of busses
#options 	NAPIC=1			# number of IO APICs
#options 	NINTR=24		# number of INTs

device		isa
device		pci

# Floppy drives
device		fdc0	at isa? port IO_FD1 irq 6 drq 2
device		fd0	at fdc0 drive 0

# ATA and ATAPI devices
device		ata0	at isa? port IO_WD1 irq 14
device		ata1	at isa? port IO_WD2 irq 15
device		ata
device		atadisk			# ATA disk drives
device		atapicd			# ATAPI CDROM drives


# SCSI Controllers
#device		ahb		# EISA AHA1742 family
device		ahc		# AHA2940 and onboard AIC7xxx devices
#device		amd		# AMD 53C974 (Teckram DC-390(T))
#device		dpt		# DPT Smartcache - See LINT for options!
#device		isp		# Qlogic family
#device		ncr		# NCR/Symbios Logic
device		sym		# NCR/Symbios Logic (newer chipsets)
options		SYM_SETUP_LP_PROBE_MAP=0x40
				# Allow ncr to attach legacy NCR devices when 
				# both sym and ncr are configured

#device		adv0	at isa?
#device		adw
#device		bt0	at isa?
#device		aha0	at isa?
#device		aic0	at isa?

# SCSI peripherals
device		scbus		# SCSI bus (required)
device		da		# Direct Access (disks)
device		sa		# Sequential Access (tape etc)
device		cd		# CD
device		pass		# Passthrough device (direct SCSI access)

# atkbdc0 controls both the keyboard and the PS/2 mouse
device		atkbdc0	at isa? port IO_KBD
device		atkbd0	at atkbdc? irq 1 flags 0x1
device		psm0	at atkbdc? irq 12

device		vga0	at isa?

# splash screen/screen saver
pseudo-device	splash

# syscons is the default console driver, resembling an SCO console
device		sc0	at isa? flags 0x100

# Floating point support - do not disable.
device		npx0	at nexus? port IO_NPX irq 13

# Serial (COM) ports
device		sio0	at isa? port IO_COM1 flags 0x10 irq 4
device		sio1	at isa? port IO_COM2 irq 3

# Parallel port
device		ppc0	at isa? irq 7
device		ppbus		# Parallel port bus (required)
device		lpt		# Printer
device		plip		# TCP/IP over parallel
device		ppi		# Parallel port interface device
device		vpo		# Requires scbus and da


# PCI Ethernet NICs.
device		de		# DEC/Intel DC21x4x (``Tulip'')
device		fxp		# Intel EtherExpress PRO/100B (82557, 82558)

# Pseudo devices - the number indicates how many units to allocated.
pseudo-device	loop		# Network loopback
pseudo-device	ether		# Ethernet support
pseudo-device	pty		# Pseudo-ttys (telnet etc)

# The `bpf' pseudo-device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
pseudo-device	bpf		#Berkeley packet filter

pseudo-device	ccd	4	#Concatenated disk driver

# Size of the kernel message buffer.  Should be N * pagesize.
options 	MSGBUF_SIZE=40960

#
# Enable the kernel debugger.
#
options 	DDB

#
# Don't drop into DDB for a panic. Intended for unattended operation
# where you may want to drop to DDB from the console, but still want
# the machine to recover from a panic
#
options 	DDB_UNATTENDED

# Options for serial drivers that support consoles (only for sio now):
options 	BREAK_TO_DEBUGGER	#a BREAK on a comconsole goes to
					#DDB, if available.


To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?14721.48065.766815.376959>