Date: Wed, 17 Sep 2008 16:50:05 GMT From: Christian Peron <csjp@freebsd.org> To: freebsd-pf@FreeBSD.org Subject: Re: kern/127439: deadlock in pf Message-ID: <200809171650.m8HGo5gv096228@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/127439; it has been noted by GNATS. From: Christian Peron <csjp@freebsd.org> To: Geoffrey Mainland <mainland@apeiron.net> Cc: FreeBSD-gnats-submit@freebsd.org Subject: Re: kern/127439: deadlock in pf Date: Wed, 17 Sep 2008 11:16:01 -0500 Can you provide a copy of your pf ruleset? On Wed, Sep 17, 2008 at 08:33:23AM -0400, Geoffrey Mainland wrote: > > >Number: 127439 > >Category: kern > >Synopsis: deadlock in pf > >Confidential: no > >Severity: critical > >Priority: high > >Responsible: freebsd-bugs > >State: open > >Quarter: > >Keywords: > >Date-Required: > >Class: sw-bug > >Submitter-Id: current-users > >Arrival-Date: Wed Sep 17 12:50:01 UTC 2008 > >Closed-Date: > >Last-Modified: > >Originator: Geoffrey Mainland > >Release: FreeBSD 7.1-PRERELEASE i386 > >Organization: > >Environment: > System: FreeBSD zeno.apeiron.net 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #7: Tue Sep 16 09:28:16 EDT 2008 toor@zeno.apeiron.net:/usr/obj/usr/src/sys/ZENO i386 > > > >Description: > > This happens reliably every night. I'm not sure what's running that triggers it. > > ifconfig: > > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> > ether 00:0e:0c:5f:c1:f8 > inet6 fe80::20e:cff:fe5f:c1f8%em0 prefixlen 64 scopeid 0x1 > inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255 > inet 192.168.0.1 netmask 0xffffffff broadcast 192.168.0.1 > inet 192.168.0.2 netmask 0xffffffff broadcast 192.168.0.2 > media: Ethernet autoselect (100baseTX <full-duplex>) > status: active > fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=8<VLAN_MTU> > ether 00:90:27:62:87:4d > inet6 fe80::290:27ff:fe62:874d%fxp0 prefixlen 64 scopeid 0x2 > inet 68.164.219.98 netmask 0xfffffff8 broadcast 68.164.219.103 > inet 68.164.219.99 netmask 0xffffffff broadcast 68.164.219.99 > inet 68.164.219.100 netmask 0xffffffff broadcast 68.164.219.100 > inet 68.164.219.101 netmask 0xffffffff broadcast 68.164.219.101 > media: Ethernet autoselect (100baseTX <full-duplex>) > status: active > vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=2808<VLAN_MTU,WOL_UCAST,WOL_MAGIC> > ether 00:15:f2:43:48:7b > inet6 fe80::215:f2ff:fe43:487b%vr0 prefixlen 64 scopeid 0x3 > inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 > inet 192.168.1.2 netmask 0xffffffff broadcast 192.168.1.2 > media: Ethernet autoselect (none) > status: no carrier > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 > inet 127.0.0.1 netmask 0xff000000 > pfsync0: flags=0<> metric 0 mtu 1460 > syncpeer: 224.0.0.240 maxupd: 128 > pflog0: flags=0<> metric 0 mtu 33204 > gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280 > tunnel inet 68.164.219.98 --> 66.55.128.25 > inet6 fe80::20e:cff:fe5f:c1f8%gif0 prefixlen 64 scopeid 0x7 > inet6 2001:4830:1200:10b::2 --> 2001:4830:1200:10b::1 prefixlen 128 > tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 > inet6 fe80::20e:cff:fe5f:c1f8%tun0 prefixlen 64 scopeid 0x8 > inet 192.168.2.1 --> 192.168.2.2 netmask 0xffffffff > Opened by PID 1454 > > Kernel config: > > cpu I686_CPU > ident ZENO > options SCHED_ULE > options SMP > options PREEMPTION > options DEVICE_POLLING > options HZ=2000 > options _KPOSIX_PRIORITY_SCHEDULING > options P1003_1B_MQUEUE > options KDB > options KDB_TRACE > options DDB > options WITNESS > options INVARIANTS > options INVARIANT_SUPPORT > makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols > options COMPAT_FREEBSD4 > options COMPAT_FREEBSD5 > options COMPAT_FREEBSD6 > options SYSVSHM > options SYSVSEM > options SYSVMSG > options STACK > options INET #Internet communications protocols > options INET6 #IPv6 communications protocols > options IPSEC #IP security (requires device crypto) > options NETATALK #Appletalk communications protocols > options NETSMB #SMB/CIFS requester > options LIBMCHAIN > options SCTP > options NETGRAPH # netgraph(4) system > device ether #Generic Ethernet > device loop #Network loopback device > device bpf #Berkeley packet filter > device tap #Virtual Ethernet driver > device tun #Tunnel driver (ppp(8), nos-tun(8)) > device gre #IP over IP tunneling > device pf #PF OpenBSD packet-filter firewall > device pflog #logging support interface for PF > device pfsync #synchronization interface for PF > device gif #IPv6 and IPv4 tunneling > device faith #for IPv6 and IPv4 translation > device stf #6to4 IPv6 over IPv4 encapsulation > options FFS #Fast filesystem > options NFSCLIENT #Network File System client > options CD9660 #ISO 9660 filesystem > options MSDOSFS #MS DOS File System (FAT, FAT32) > options NFSSERVER #Network File System server > options NFSLOCKD #Network Lock Manager > options NTFS #NT File System > options PROCFS #Process filesystem (requires PSEUDOFS) > options PSEUDOFS #Pseudo-filesystem framework > options SMBFS #SMB/CIFS filesystem > options UDF #Universal Disk Format > options NFS_ROOT #NFS usable as root device > options SOFTUPDATES > options UFS_ACL > options UFS_DIRHASH > device random > device mem > options AUDIT > device scbus #base SCSI code > device da #SCSI direct access devices (aka disks) > device cd #SCSI CD-ROMs > device pt #SCSI processor > device pass #CAM passthrough driver > device pty #Pseudo ttys > device md #Memory/malloc disk > options LIBICONV > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > device splash # Splash screen and screen saver support > device sc > options SC_DISABLE_KDBKEY # disable `debug' key > device ata > device atadisk # ATA disk drives > device ataraid # ATA RAID drives > device atapicd # ATAPI CDROM drives > device atapifd # ATAPI floppy drives > device atapicam # emulate ATAPI devices as SCSI ditto via CAM > options ATA_STATIC_ID > device fdc > device sound > device ppc > device ppbus > device lpt > device ppi > device uhci > device ehci > device usb > device crypto # core crypto support > device cryptodev # /dev/crypto for access to h/w > device apic # I/O apic > device nvram # Access to rtc cmos via /dev/nvram > device sio > device eisa > device pci > options VESA > device psm > device atkbdc > device atkbd > device vga > options COMPAT_LINUX > options COMPAT_AOUT > options LINPROCFS > options LINSYSFS > > > > > > dmesg output (after crash): > > Copyright (c) 1992-2008 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 7.1-PRERELEASE #7: Tue Sep 16 09:28:16 EDT 2008 > toor@zeno.apeiron.net:/usr/obj/usr/src/sys/ZENO > WARNING: WITNESS option enabled, expect reduced performance. > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: AMD Sempron(tm) Processor 3100+ (1800.09-MHz 686-class CPU) > Origin = "AuthenticAMD" Id = 0x10fc0 Stepping = 0 > Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2> > AMD Features=0xc2500800<SYSCALL,NX,MMX+,FFXSR,3DNow!+,3DNow!> > AMD Features2=0x1<LAHF> > real memory = 1073414144 (1023 MB) > avail memory = 1040887808 (992 MB) > WITNESS: spin lock cpuset not in order list > WITNESS: spin lock intrcnt not in order list > netsmb_dev: loaded > cryptosoft0: <software crypto> on motherboard > acpi0: <A M I OEMRSDT> on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > acpi0: reservation of 100000, 3fef0000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 > pci0: <ACPI PCI bus> on pcib0 > pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 > pci1: <ACPI PCI bus> on pcib1 > vgapci0: <VGA-compatible display> mem > 0xfb000000-0xfbffffff,0xf0000000-0xf7ffffff irq 11 at device 0.0 on pci1 > em0: <Intel(R) PRO/1000 Network Connection 6.9.5> port 0xe800-0xe83f mem > 0xfae00000-0xfae1ffff,0xfad00000-0xfad1ffff irq 11 at device 11.0 on pci0 > em0: [FILTER] > em0: Ethernet address: 00:0e:0c:5f:c1:f8 > fxp0: <Intel 82559 Pro/100 Ethernet> port 0xe400-0xe43f mem > 0xfab00000-0xfab00fff,0xfaa00000-0xfaafffff irq 10 at device 12.0 on pci0 > miibus0: <MII bus> on fxp0 > inphy0: <i82555 10/100 media interface> PHY 1 on miibus0 > inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > fxp0: Ethernet address: 00:90:27:62:87:4d > fxp0: [ITHREAD] > atapci0: <VIA 6420 SATA150 controller> port > 0xe000-0xe007,0xd800-0xd803,0xd400-0xd407,0xd000-0xd003,0xc800-0xc80f,0xc400-0xc4ff > irq 10 at device 15.0 on pci0 > atapci0: [ITHREAD] > ata2: <ATA channel 0> on atapci0 > ata2: [ITHREAD] > ata3: <ATA channel 1> on atapci0 > ata3: [ITHREAD] > atapci1: <VIA 8237 UDMA133 controller> port > 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 15.1 on pci0 > ata0: <ATA channel 0> on atapci1 > ata0: [ITHREAD] > ata1: <ATA channel 1> on atapci1 > ata1: [ITHREAD] > uhci0: <VIA 83C572 USB controller> port 0xb000-0xb01f irq 11 at device 16.0 on > pci0 > uhci0: [GIANT-LOCKED] > uhci0: [ITHREAD] > usb0: <VIA 83C572 USB controller> on uhci0 > usb0: USB revision 1.0 > uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0 > uhub0: 2 ports with 2 removable, self powered > uhci1: <VIA 83C572 USB controller> port 0xb400-0xb41f irq 11 at device 16.1 on > pci0 > uhci1: [GIANT-LOCKED] > uhci1: [ITHREAD] > usb1: <VIA 83C572 USB controller> on uhci1 > usb1: USB revision 1.0 > uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1 > uhub1: 2 ports with 2 removable, self powered > uhci2: <VIA 83C572 USB controller> port 0xb800-0xb81f irq 10 at device 16.2 on > pci0 > uhci2: [GIANT-LOCKED] > uhci2: [ITHREAD] > usb2: <VIA 83C572 USB controller> on uhci2 > usb2: USB revision 1.0 > uhub2: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2 > uhub2: 2 ports with 2 removable, self powered > uhci3: <VIA 83C572 USB controller> port 0xc000-0xc01f irq 10 at device 16.3 on > pci0 > uhci3: [GIANT-LOCKED] > uhci3: [ITHREAD] > usb3: <VIA 83C572 USB controller> on uhci3 > usb3: USB revision 1.0 > uhub3: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3 > uhub3: 2 ports with 2 removable, self powered > ehci0: <VIA VT6202 USB 2.0 controller> mem 0xfa700000-0xfa7000ff irq 5 at device > 16.4 on pci0 > ehci0: [GIANT-LOCKED] > ehci0: [ITHREAD] > usb4: EHCI version 1.0 > usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 > usb4: <VIA VT6202 USB 2.0 controller> on ehci0 > usb4: USB revision 2.0 > uhub4: <VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4 > uhub4: 8 ports with 8 removable, self powered > isab0: <PCI-ISA bridge> at device 17.0 on pci0 > isa0: <ISA bus> on isab0 > pci0: <multimedia, audio> at device 17.5 (no driver attached) > vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xa400-0xa4ff mem > 0xfa600000-0xfa6000ff irq 11 at device 18.0 on pci0 > vr0: Quirks: 0x0 > vr0: Revision: 0x78 > miibus1: <MII bus> on vr0 > rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus1 > rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > vr0: Ethernet address: 00:15:f2:43:48:7b > vr0: [ITHREAD] > cpu0: <ACPI CPU> on acpi0 > acpi_button0: <Power Button> on acpi0 > acpi_button1: <Sleep Button> on acpi0 > atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 > atkbd0: <AT Keyboard> irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on > acpi0 > fdc0: [FILTER] > fd0: <1440-KB 3.5" drive> on fdc0 drive 0 > sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > sio0: type 16550A > sio0: [FILTER] > orm0: <ISA Option ROMs> at iomem 0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xd3fff > pnpid ORM0000 on isa0 > sc0: <System console> at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppbus0: <Parallel port bus> on ppc0 > ppbus0: [ITHREAD] > lpt0: <Printer> on ppbus0 > lpt0: Interrupt-driven port > ppi0: <Parallel I/O> on ppbus0 > ppc0: [GIANT-LOCKED] > ppc0: [ITHREAD] > sio1: configured irq 3 not in bitmap of probed irqs 0 > sio1: port may not be enabled > Timecounter "TSC" frequency 1800086355 Hz quality 800 > Timecounters tick every 1.000 msec > IPsec: Initialized Security Association Processing. > ad0: 194481MB <Maxtor 6B200P0 BAH41BM0> at ata0-master UDMA133 > acd0: DVDR <NEC DVD RW ND-3550A/1.05> at ata1-master UDMA33 > ad4: 239372MB <Maxtor 7L250S0 BANC1G10> at ata2-master SATA150 > cd0 at ata1 bus 0 target 0 lun 0 > cd0: <_NEC DVD_RW ND-3550A 1.05> Removable CD-ROM SCSI-0 device > cd0: 33.000MB/s transfers > cd0: Attempt to query device size failed: NOT READY, Medium not present > WARNING: WITNESS option enabled, expect reduced performance. > Trying to mount root from ufs:/dev/ad4s1a > WARNING: / was not properly dismounted > lock order reversal: > 1st 0xc0907fcc pf task mtx (pf task mtx) @ > /usr/src/sys/contrib/pf/net/pf_ioctl.c:1394 > 2nd 0xc0973488 ifnet (ifnet) @ /usr/src/sys/net/if.c:1558 > KDB: stack backtrace: > db_trace_self_wrapper(c088cf61,e658ba3c,c05eb7b6,c088f4ad,c0973488,...) at > db_trace_self_wrapper+0x26 > kdb_backtrace(c088f4ad,c0973488,c0896cfd,c0896cfd,c0896b56,...) at > kdb_backtrace+0x29 > witness_checkorder(c0973488,9,c0896b56,616,0,...) at witness_checkorder+0x6d6 > _mtx_lock_flags(c0973488,0,c0896b56,616,c3f37a70,...) at _mtx_lock_flags+0xbc > ifunit(c3f37a70,0,c08711f2,572,c05e958e,...) at ifunit+0x2f > pfioctl(c3d2d800,c0104414,c3f37a70,3,c3f48690,...) at pfioctl+0x23b5 > devfs_ioctl_f(c3f49c2c,c0104414,c3f37a70,c3b2c000,c3f48690,...) at > devfs_ioctl_f+0xe5 > kern_ioctl(c3f48690,3,c0104414,c3f37a70,1000000,...) at kern_ioctl+0x243 > ioctl(c3f48690,e658bcfc,c,c08bade8,c08d3630,...) at ioctl+0x134 > syscall(e658bd38) at syscall+0x274 > Xint0x80_syscall() at Xint0x80_syscall+0x20 > --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x281aac4b, esp = 0xbfbfde5c, ebp > = 0xbfbfde88 --- > lock order reversal: > 1st 0xc097830c tcp (tcp) @ /usr/src/sys/netinet/tcp_input.c:400 > 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @ > /usr/src/sys/net/pfil.c:73 > KDB: stack backtrace: > db_trace_self_wrapper(c088cf61,e42579ac,c05eb7b6,c088f4ad,c09775d8,...) at > db_trace_self_wrapper+0x26 > kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at > kdb_backtrace+0x29 > witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at > witness_checkorder+0x6d6 > _rw_rlock(c09775d8,c0897d93,49,e4257a6c,0,...) at _rw_rlock+0x8e > pfil_run_hooks(c09775c0,e4257a8c,c3c31c00,2,0,...) at pfil_run_hooks+0x35 > ip_output(c3c46100,0,e4257a50,0,0,0,c08e7c90,0,0,0,c067c807,c08e7c94,c08e7c9c,c8) > at ip_output+0x90f > tcp_respond(0,c3c87020,c3c87034,c3c46100,2da9088c,...) at tcp_respond+0x3e7 > tcp_dropwithreset(1,3,c089c953,353,1900,...) at tcp_dropwithreset+0x152 > tcp_input(c3c46100,14,c3c31c00,1,0,...) at tcp_input+0xe45 > ip_input(c3c46100,c3c46100,800,c3c31c00,800,...) at ip_input+0x686 > netisr_dispatch(2,c3c46100,10,3,0,...) at netisr_dispatch+0x72 > ether_demux(c3c31c00,c3c46100,3,0,3,...) at ether_demux+0x2e5 > ether_input(c3c31c00,c3c46100,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f > fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4 > fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0 > ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5 > fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 --- > lock order reversal: > 1st 0xc4013d44 udpinp (udpinp) @ /usr/src/sys/netinet/udp_usrreq.c:878 > 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @ > /usr/src/sys/net/pfil.c:73 > KDB: stack backtrace: > db_trace_self_wrapper(c088cf61,e658ba14,c05eb7b6,c088f4ad,c09775d8,...) at > db_trace_self_wrapper+0x26 > kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at > kdb_backtrace+0x29 > witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at > witness_checkorder+0x6d6 > _rw_rlock(c09775d8,c0897d93,49,e658bad4,c4013ca8,...) at _rw_rlock+0x8e > pfil_run_hooks(c09775c0,e658baf4,c3d44000,2,c4013ca8,...) at pfil_run_hooks+0x35 > ip_output(c3ef6100,0,e658bab8,0,0,...) at ip_output+0x90f > udp_send(c42454e0,0,c3ef6100,0,0,...) at udp_send+0x8cd > sosend_dgram(c42454e0,0,e658bbec,c3ef6100,0,...) at sosend_dgram+0x351 > sosend(c42454e0,0,e658bbec,0,0,...) at sosend+0x54 > kern_sendit(c3f48690,4,e658bc68,0,0,...) at kern_sendit+0xdb > sendit(0,8143023,0,0,0,...) at sendit+0xb1 > sendto(c3f48690,e658bcfc,18,c08a5d78,c08d3d98,...) at sendto+0x48 > syscall(e658bd38) at syscall+0x274 > Xint0x80_syscall() at Xint0x80_syscall+0x20 > --- syscall (133, FreeBSD ELF32, sendto), eip = 0x2816bc83, esp = 0xbfbfd73c, > ebp = 0xbfbfd768 --- > lock order reversal: > 1st 0xc423f150 tcpinp (tcpinp) @ /usr/src/sys/netinet/tcp_usrreq.c:472 > 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @ > /usr/src/sys/net/pfil.c:73 > KDB: stack backtrace: > db_trace_self_wrapper(c088cf61,e65a3a30,c05eb7b6,c088f4ad,c09775d8,...) at > db_trace_self_wrapper+0x26 > kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at > kdb_backtrace+0x29 > witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at > witness_checkorder+0x6d6 > _rw_rlock(c09775d8,c0897d93,49,e65a3af0,c423f0b4,...) at _rw_rlock+0x8e > pfil_run_hooks(c09775c0,e65a3b10,c3d44000,2,c423f0b4,...) at pfil_run_hooks+0x35 > ip_output(c3c94e00,0,e65a3ad4,0,0,...) at ip_output+0x90f > tcp_output(c42421d0,c3d2bc50,1d8,c423f150,c4259000,...) at tcp_output+0x140c > tcp_usr_connect(c4259000,c3d2bc50,c3d2f8c0,25,e65a3c64,...) at > tcp_usr_connect+0x11c > soconnect(c4259000,c3d2bc50,c3d2f8c0,10,16,...) at soconnect+0x52 > kern_connect(c3d2f8c0,9,c3d2bc50,c3d2bc50,0,...) at kern_connect+0x59 > connect(c3d2f8c0,e65a3cfc,c,c088ff65,c08d3a50,...) at connect+0x46 > syscall(e65a3d38) at syscall+0x274 > Xint0x80_syscall() at Xint0x80_syscall+0x20 > --- syscall (98, FreeBSD ELF32, connect), eip = 0x28161e9b, esp = 0xbfbfe71c, > ebp = 0xbfbfe868 --- > lock order reversal: > 1st 0xc3eda524 tcp_sc_head (tcp_sc_head) @ > /usr/src/sys/netinet/tcp_syncache.c:494 > 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @ > /usr/src/sys/net/pfil.c:73 > KDB: stack backtrace: > db_trace_self_wrapper(c088cf61,e4257854,c05eb7b6,c088f4ad,c09775d8,...) at > db_trace_self_wrapper+0x26 > kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at > kdb_backtrace+0x29 > witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at > witness_checkorder+0x6d6 > _rw_rlock(c09775d8,c0897d93,49,e4257914,0,...) at _rw_rlock+0x8e > pfil_run_hooks(c09775c0,e4257934,c3c31c00,2,0,...) at pfil_run_hooks+0x35 > ip_output(c3ef7a00,0,e42578f8,0,0,...) at ip_output+0x90f > syncache_respond(c426ad70,c40c0834,0,0,c40c0834,...) at syncache_respond+0x3a2 > _syncache_add(c42400b4,e4257ba8,c40b3700,0,0,...) at _syncache_add+0x2b0 > syncache_add(e4257b68,e4257b90,c40c0834,c42400b4,e4257ba8,...) at > syncache_add+0x38 > tcp_input(c40b3700,14,c3c31c00,1,0,...) at tcp_input+0xd6b > ip_input(c40b3700,c40b3700,800,c3c31c00,800,...) at ip_input+0x686 > netisr_dispatch(2,c40b3700,10,3,0,...) at netisr_dispatch+0x72 > ether_demux(c3c31c00,c40b3700,3,0,3,...) at ether_demux+0x2e5 > ether_input(c3c31c00,c40b3700,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f > fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4 > fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0 > ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5 > fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 --- > lock order reversal: > 1st 0xc09786cc udp (udp) @ /usr/src/sys/netinet/udp_usrreq.c:395 > 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @ > /usr/src/sys/net/pfil.c:73 > KDB: stack backtrace: > db_trace_self_wrapper(c088cf61,e42579b8,c05eb7b6,c088f4ad,c09775d8,...) at > db_trace_self_wrapper+0x26 > kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at > kdb_backtrace+0x29 > witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at > witness_checkorder+0x6d6 > _rw_rlock(c09775d8,c0897d93,49,e4257a78,0,...) at _rw_rlock+0x8e > pfil_run_hooks(c09775c0,e4257a98,c3c31c00,2,0,...) at pfil_run_hooks+0x35 > ip_output(c3efae00,0,e4257a5c,0,0,...) at ip_output+0x90f > icmp_reflect(c40c6020,c3efaec8,14,c3efaf00,c40c6020,...) at icmp_reflect+0x3df > icmp_error(c40b4d00,3,3,0,0,...) at icmp_error+0x3bd > udp_input(c40b4d00,14,c3c31c00,1,0,...) at udp_input+0x5ea > ip_input(c40b4d00,c40b4d00,800,c3c31c00,800,...) at ip_input+0x686 > netisr_dispatch(2,c40b4d00,10,3,0,...) at netisr_dispatch+0x72 > ether_demux(c3c31c00,c40b4d00,3,0,3,...) at ether_demux+0x2e5 > ether_input(c3c31c00,c40b4d00,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f > fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4 > fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0 > ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5 > fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 --- > > > > > > kernel backtrace: > > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-marcel-freebsd"... > > Unread portion of the kernel message buffer: > panic: _rw_rlock (tcp): wlock already held @ > /usr/src/sys/contrib/pf/net/pf.c:3016 > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper(c088cf61,e6846220,c05ae7df,c08b659d,0,...) at > db_trace_self_wrapper+0x26 > kdb_backtrace(c08b659d,0,c0889c7e,e684622c,0,...) at kdb_backtrace+0x29 > panic(c0889c7e,c085a754,c088f55e,c087092d,bc8,...) at panic+0x10f > _rw_rlock(c097830c,c087092d,bc8,c08d9624,c087092d,...) at _rw_rlock+0x73 > pf_socket_lookup(2,e68463dc,0,cc4,3,...) at pf_socket_lookup+0x208 > pf_test_tcp(e6846444,e6846440,2,c3efee00,c3c8e900,...) at pf_test_tcp+0x142 > pf_test6(2,c3d44000,e68464a0,0,0,...) at pf_test6+0x8a0 > pf_check6_out(0,e68464a0,c3d44000,2,0,...) at pf_check6_out+0x47 > pfil_run_hooks(c097ad00,e6846638,c3d44000,2,0,...) at pfil_run_hooks+0x88 > ip6_output(c3c8e900,0,e6846618,0,0,...) at ip6_output+0x122e > pf_send_tcp(c4fcfe00,c41259b4,1c,c4fcfe5c,c4fcfe4c,...) at pf_send_tcp+0x6dd > pf_test_tcp(e68468e8,e68468e4,2,c3f20900,c4fcfe00,...) at pf_test_tcp+0xcef > pf_test6(2,c3f06400,e6846944,0,c446b7bc,...) at pf_test6+0x8a0 > pf_check6_out(0,e6846944,c3f06400,2,c446b7bc,...) at pf_check6_out+0x47 > pfil_run_hooks(c097ad00,e6846adc,c3f06400,2,c446b7bc,...) at pfil_run_hooks+0x88 > ip6_output(c4fcfe00,0,e6846abc,0,0,...) at ip6_output+0x122e > tcp_output(c45553a0,c447e7c0,201,c446b858,c45553a0,...) at tcp_output+0x137e > tcp6_usr_connect(c50cd340,c447e7c0,c4eed690,25,e6846c64,...) at > tcp6_usr_connect+0x171 > soconnect(c50cd340,c447e7c0,c4eed690,1c,16,...) at soconnect+0x52 > kern_connect(c4eed690,3,c447e7c0,c447e7c0,0,...) at kern_connect+0x59 > connect(c4eed690,e6846cfc,c,c08a288e,c08d3a50,...) at connect+0x46 > syscall(e6846d38) at syscall+0x274 > Xint0x80_syscall() at Xint0x80_syscall+0x20 > --- syscall (98, FreeBSD ELF32, connect), eip = 0x282e6e9b, esp = 0xbfbfe7ec, > ebp = 0xbfbfe848 --- > KDB: enter: panic > shared rw PFil hook read/write mutex r = 1 (0xc097ad18) locked @ > /usr/src/sys/net/pfil.c:73 > exclusive rw tcpinp r = 0 (0xc446b858) locked @ > /usr/src/sys/netinet/tcp_usrreq.c:513 > exclusive rw tcp r = 0 (0xc097830c) locked @ > /usr/src/sys/netinet/tcp_usrreq.c:510 > exclusive sx so_rcv_sx r = 0 (0xc452fbec) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc483cbec) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc4e89bec) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc4e8970c) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc483c22c) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc480d70c) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc4e8a08c) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc4e8a56c) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc41a456c) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc41c156c) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc41c18ac) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc41c1bec) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > exclusive sx so_rcv_sx r = 0 (0xc41f108c) locked @ > /usr/src/sys/kern/uipc_sockbuf.c:148 > shared rw udpinp r = 0 (0xc400f63c) locked @ > /usr/src/sys/netinet/udp_usrreq.c:878 > Uptime: 16h23m36s > Physical memory: 1015 MB > Dumping 166 MB: 151 135 119 103 87 71 55 39 23 7 > > Reading symbols from /boot/kernel/if_em.ko...Reading symbols from > /boot/kernel/if_em.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/if_em.ko > Reading symbols from /boot/kernel/if_fxp.ko...Reading symbols from > /boot/kernel/if_fxp.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/if_fxp.ko > Reading symbols from /boot/kernel/miibus.ko...Reading symbols from > /boot/kernel/miibus.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/miibus.ko > Reading symbols from /boot/kernel/if_vr.ko...Reading symbols from > /boot/kernel/if_vr.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/if_vr.ko > Reading symbols from /boot/kernel/ulpt.ko...Reading symbols from > /boot/kernel/ulpt.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/ulpt.ko > Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from > /boot/kernel/accf_http.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/accf_http.ko > Reading symbols from /boot/kernel/acpi.ko...Reading symbols from > /boot/kernel/acpi.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/acpi.ko > #0 doadump () at pcpu.h:196 > 196 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) bt > #0 doadump () at pcpu.h:196 > #1 0xc05ae54c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 > #2 0xc05ae816 in panic (fmt=Variable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:572 > #3 0xc05acf63 in _rw_rlock (rw=0xc097830c, file=0xc087092d > "/usr/src/sys/contrib/pf/net/pf.c", line=3016) > at /usr/src/sys/kern/kern_rwlock.c:253 > #4 0xc0473e58 in pf_socket_lookup (direction=2, pd=0xe68463dc, inp_arg=0x0) at > /usr/src/sys/contrib/pf/net/pf.c:3016 > #5 0xc047dd62 in pf_test_tcp (rm=0xe6846444, sm=0xe6846440, direction=2, > kif=0xc3efee00, m=0xc3c8e900, off=40, > h=0xc3c8e944, pd=0xe68463dc, am=0xe6846448, rsm=0xe684643c, ifq=0x0, > inp=0x0) > at /usr/src/sys/contrib/pf/net/pf.c:3270 > #6 0xc04816c0 in pf_test6 (dir=2, ifp=0xc3d44000, m0=0xe68464a0, eh=0x0, > inp=0x0) > at /usr/src/sys/contrib/pf/net/pf.c:7368 > #7 0xc0484e37 in pf_check6_out (arg=0x0, m=0xe68464a0, ifp=0xc3d44000, dir=2, > inp=0x0) > at /usr/src/sys/contrib/pf/net/pf_ioctl.c:3739 > #8 0xc0657618 in pfil_run_hooks (ph=0xc097ad00, mp=0xe6846638, ifp=0xc3d44000, > dir=2, inp=0x0) > at /usr/src/sys/net/pfil.c:78 > #9 0xc07034fe in ip6_output (m0=0xc3c8e900, opt=0x0, ro=0xe6846618, > flags=Variable "flags" is not available. > ) at /usr/src/sys/netinet6/ip6_output.c:853 > #10 0xc0477dad in pf_send_tcp (replyto=0xc4fcfe00, r=0xc41259b4, af=28 '\034', > saddr=0xc4fcfe5c, daddr=0xc4fcfe4c, > sport=20480, dport=46591, seq=0, ack=1170313007, flags=20 '\024', win=0, > mss=0, ttl=0 '\0', tag=1, rtag=0, eh=0x0, > ifp=0xc3f06400) at /usr/src/sys/contrib/pf/net/pf.c:1978 > #11 0xc047e90f in pf_test_tcp (rm=0xe68468e8, sm=0xe68468e4, direction=2, > kif=0xc3f20900, m=0xc4fcfe00, off=40, > h=0xc4fcfe44, pd=0xe6846880, am=0xe68468ec, rsm=0xe68468e0, ifq=0x0, > inp=0xc446b7bc) > at /usr/src/sys/contrib/pf/net/pf.c:3424 > #12 0xc04816c0 in pf_test6 (dir=2, ifp=0xc3f06400, m0=0xe6846944, eh=0x0, > inp=0xc446b7bc) > at /usr/src/sys/contrib/pf/net/pf.c:7368 > #13 0xc0484e37 in pf_check6_out (arg=0x0, m=0xe6846944, ifp=0xc3f06400, dir=2, > inp=0xc446b7bc) > at /usr/src/sys/contrib/pf/net/pf_ioctl.c:3739 > #14 0xc0657618 in pfil_run_hooks (ph=0xc097ad00, mp=0xe6846adc, ifp=0xc3f06400, > dir=2, inp=0xc446b7bc) > at /usr/src/sys/net/pfil.c:78 > #15 0xc07034fe in ip6_output (m0=0xc4fcfe00, opt=0x0, ro=0xe6846abc, > flags=Variable "flags" is not available. > ) at /usr/src/sys/netinet6/ip6_output.c:853 > #16 0xc06debbe in tcp_output (tp=0xc45553a0) at > /usr/src/sys/netinet/tcp_output.c:1114 > #17 0xc06ea5d1 in tcp6_usr_connect (so=0xc50cd340, nam=0xc447e7c0, > td=0xc4eed690) at tcp_offload.h:257 > #18 0xc060b002 in soconnect (so=0xc50cd340, nam=0xc447e7c0, td=0xc4eed690) at > /usr/src/sys/kern/uipc_socket.c:771 > #19 0xc06129e9 in kern_connect (td=0xc4eed690, fd=3, sa=0xc447e7c0) at > /usr/src/sys/kern/uipc_syscalls.c:570 > #20 0xc0612b56 in connect (td=0xc4eed690, uap=0xe6846cfc) at > /usr/src/sys/kern/uipc_syscalls.c:534 > #21 0xc083a2d4 in syscall (frame=0xe6846d38) at > /usr/src/sys/i386/i386/trap.c:1090 > #22 0xc0821220 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 > #23 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) > > >How-To-Repeat: > > >Fix: > > > > >Release-Note: > >Audit-Trail: > >Unformatted: > _______________________________________________ > freebsd-bugs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-bugs > To unsubscribe, send any mail to "freebsd-bugs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200809171650.m8HGo5gv096228>