From owner-freebsd-bugs@FreeBSD.ORG Wed Sep 17 12:50:01 2008 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9E451065673 for ; Wed, 17 Sep 2008 12:50:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C391F8FC1D for ; Wed, 17 Sep 2008 12:50:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m8HCo1ks076839 for ; Wed, 17 Sep 2008 12:50:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m8HCo1Tt076838; Wed, 17 Sep 2008 12:50:01 GMT (envelope-from gnats) Resent-Date: Wed, 17 Sep 2008 12:50:01 GMT Resent-Message-Id: <200809171250.m8HCo1Tt076838@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Geoffrey Mainland Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 66200106564A for ; Wed, 17 Sep 2008 12:44:45 +0000 (UTC) (envelope-from mainland@apeiron.net) Received: from zeno.apeiron.net (h-68-164-219-98.cmbrmaor.covad.net [68.164.219.98]) by mx1.freebsd.org (Postfix) with ESMTP id 490048FC13 for ; Wed, 17 Sep 2008 12:44:44 +0000 (UTC) (envelope-from mainland@apeiron.net) Received: from zeno.apeiron.net (localhost [127.0.0.1]) by zeno.apeiron.net (Postfix) with ESMTP id 1CED422CCC for ; Wed, 17 Sep 2008 08:33:24 -0400 (EDT) Received: by zeno.apeiron.net (Postfix, from userid 4001) id EFF8F22CCB; Wed, 17 Sep 2008 08:33:23 -0400 (EDT) Message-Id: <20080917123323.EFF8F22CCB@zeno.apeiron.net> Date: Wed, 17 Sep 2008 08:33:23 -0400 (EDT) From: Geoffrey Mainland To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: Subject: kern/127439: deadlock in pf X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Geoffrey Mainland List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Sep 2008 12:50:02 -0000 >Number: 127439 >Category: kern >Synopsis: deadlock in pf >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Sep 17 12:50:01 UTC 2008 >Closed-Date: >Last-Modified: >Originator: Geoffrey Mainland >Release: FreeBSD 7.1-PRERELEASE i386 >Organization: >Environment: System: FreeBSD zeno.apeiron.net 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #7: Tue Sep 16 09:28:16 EDT 2008 toor@zeno.apeiron.net:/usr/obj/usr/src/sys/ZENO i386 >Description: This happens reliably every night. I'm not sure what's running that triggers it. ifconfig: em0: flags=8843 metric 0 mtu 1500 options=9b ether 00:0e:0c:5f:c1:f8 inet6 fe80::20e:cff:fe5f:c1f8%em0 prefixlen 64 scopeid 0x1 inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255 inet 192.168.0.1 netmask 0xffffffff broadcast 192.168.0.1 inet 192.168.0.2 netmask 0xffffffff broadcast 192.168.0.2 media: Ethernet autoselect (100baseTX ) status: active fxp0: flags=8843 metric 0 mtu 1500 options=8 ether 00:90:27:62:87:4d inet6 fe80::290:27ff:fe62:874d%fxp0 prefixlen 64 scopeid 0x2 inet 68.164.219.98 netmask 0xfffffff8 broadcast 68.164.219.103 inet 68.164.219.99 netmask 0xffffffff broadcast 68.164.219.99 inet 68.164.219.100 netmask 0xffffffff broadcast 68.164.219.100 inet 68.164.219.101 netmask 0xffffffff broadcast 68.164.219.101 media: Ethernet autoselect (100baseTX ) status: active vr0: flags=8843 metric 0 mtu 1500 options=2808 ether 00:15:f2:43:48:7b inet6 fe80::215:f2ff:fe43:487b%vr0 prefixlen 64 scopeid 0x3 inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 inet 192.168.1.2 netmask 0xffffffff broadcast 192.168.1.2 media: Ethernet autoselect (none) status: no carrier lo0: flags=8049 metric 0 mtu 16384 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet 127.0.0.1 netmask 0xff000000 pfsync0: flags=0<> metric 0 mtu 1460 syncpeer: 224.0.0.240 maxupd: 128 pflog0: flags=0<> metric 0 mtu 33204 gif0: flags=8051 metric 0 mtu 1280 tunnel inet 68.164.219.98 --> 66.55.128.25 inet6 fe80::20e:cff:fe5f:c1f8%gif0 prefixlen 64 scopeid 0x7 inet6 2001:4830:1200:10b::2 --> 2001:4830:1200:10b::1 prefixlen 128 tun0: flags=8051 metric 0 mtu 1500 inet6 fe80::20e:cff:fe5f:c1f8%tun0 prefixlen 64 scopeid 0x8 inet 192.168.2.1 --> 192.168.2.2 netmask 0xffffffff Opened by PID 1454 Kernel config: cpu I686_CPU ident ZENO options SCHED_ULE options SMP options PREEMPTION options DEVICE_POLLING options HZ=2000 options _KPOSIX_PRIORITY_SCHEDULING options P1003_1B_MQUEUE options KDB options KDB_TRACE options DDB options WITNESS options INVARIANTS options INVARIANT_SUPPORT makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols options COMPAT_FREEBSD4 options COMPAT_FREEBSD5 options COMPAT_FREEBSD6 options SYSVSHM options SYSVSEM options SYSVMSG options STACK options INET #Internet communications protocols options INET6 #IPv6 communications protocols options IPSEC #IP security (requires device crypto) options NETATALK #Appletalk communications protocols options NETSMB #SMB/CIFS requester options LIBMCHAIN options SCTP options NETGRAPH # netgraph(4) system device ether #Generic Ethernet device loop #Network loopback device device bpf #Berkeley packet filter device tap #Virtual Ethernet driver device tun #Tunnel driver (ppp(8), nos-tun(8)) device gre #IP over IP tunneling device pf #PF OpenBSD packet-filter firewall device pflog #logging support interface for PF device pfsync #synchronization interface for PF device gif #IPv6 and IPv4 tunneling device faith #for IPv6 and IPv4 translation device stf #6to4 IPv6 over IPv4 encapsulation options FFS #Fast filesystem options NFSCLIENT #Network File System client options CD9660 #ISO 9660 filesystem options MSDOSFS #MS DOS File System (FAT, FAT32) options NFSSERVER #Network File System server options NFSLOCKD #Network Lock Manager options NTFS #NT File System options PROCFS #Process filesystem (requires PSEUDOFS) options PSEUDOFS #Pseudo-filesystem framework options SMBFS #SMB/CIFS filesystem options UDF #Universal Disk Format options NFS_ROOT #NFS usable as root device options SOFTUPDATES options UFS_ACL options UFS_DIRHASH device random device mem options AUDIT device scbus #base SCSI code device da #SCSI direct access devices (aka disks) device cd #SCSI CD-ROMs device pt #SCSI processor device pass #CAM passthrough driver device pty #Pseudo ttys device md #Memory/malloc disk options LIBICONV options KBD_INSTALL_CDEV # install a CDEV entry in /dev device splash # Splash screen and screen saver support device sc options SC_DISABLE_KDBKEY # disable `debug' key device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives device atapifd # ATAPI floppy drives device atapicam # emulate ATAPI devices as SCSI ditto via CAM options ATA_STATIC_ID device fdc device sound device ppc device ppbus device lpt device ppi device uhci device ehci device usb device crypto # core crypto support device cryptodev # /dev/crypto for access to h/w device apic # I/O apic device nvram # Access to rtc cmos via /dev/nvram device sio device eisa device pci options VESA device psm device atkbdc device atkbd device vga options COMPAT_LINUX options COMPAT_AOUT options LINPROCFS options LINSYSFS dmesg output (after crash): Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-PRERELEASE #7: Tue Sep 16 09:28:16 EDT 2008 toor@zeno.apeiron.net:/usr/obj/usr/src/sys/ZENO WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Sempron(tm) Processor 3100+ (1800.09-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x10fc0 Stepping = 0 Features=0x78bfbff AMD Features=0xc2500800 AMD Features2=0x1 real memory = 1073414144 (1023 MB) avail memory = 1040887808 (992 MB) WITNESS: spin lock cpuset not in order list WITNESS: spin lock intrcnt not in order list netsmb_dev: loaded cryptosoft0: on motherboard acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 3fef0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: mem 0xfb000000-0xfbffffff,0xf0000000-0xf7ffffff irq 11 at device 0.0 on pci1 em0: port 0xe800-0xe83f mem 0xfae00000-0xfae1ffff,0xfad00000-0xfad1ffff irq 11 at device 11.0 on pci0 em0: [FILTER] em0: Ethernet address: 00:0e:0c:5f:c1:f8 fxp0: port 0xe400-0xe43f mem 0xfab00000-0xfab00fff,0xfaa00000-0xfaafffff irq 10 at device 12.0 on pci0 miibus0: on fxp0 inphy0: PHY 1 on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:90:27:62:87:4d fxp0: [ITHREAD] atapci0: port 0xe000-0xe007,0xd800-0xd803,0xd400-0xd407,0xd000-0xd003,0xc800-0xc80f,0xc400-0xc4ff irq 10 at device 15.0 on pci0 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 15.1 on pci0 ata0: on atapci1 ata0: [ITHREAD] ata1: on atapci1 ata1: [ITHREAD] uhci0: port 0xb000-0xb01f irq 11 at device 16.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xb400-0xb41f irq 11 at device 16.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0xb800-0xb81f irq 10 at device 16.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: port 0xc000-0xc01f irq 10 at device 16.3 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb3: on uhci3 usb3: USB revision 1.0 uhub3: on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xfa700000-0xfa7000ff irq 5 at device 16.4 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: on usb4 uhub4: 8 ports with 8 removable, self powered isab0: at device 17.0 on pci0 isa0: on isab0 pci0: at device 17.5 (no driver attached) vr0: port 0xa400-0xa4ff mem 0xfa600000-0xfa6000ff irq 11 at device 18.0 on pci0 vr0: Quirks: 0x0 vr0: Revision: 0x78 miibus1: on vr0 rlphy0: PHY 1 on miibus1 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto vr0: Ethernet address: 00:15:f2:43:48:7b vr0: [ITHREAD] cpu0: on acpi0 acpi_button0: on acpi0 acpi_button1: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] orm0: at iomem 0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xd3fff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppbus0: on ppc0 ppbus0: [ITHREAD] lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled Timecounter "TSC" frequency 1800086355 Hz quality 800 Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. ad0: 194481MB at ata0-master UDMA133 acd0: DVDR at ata1-master UDMA33 ad4: 239372MB at ata2-master SATA150 cd0 at ata1 bus 0 target 0 lun 0 cd0: <_NEC DVD_RW ND-3550A 1.05> Removable CD-ROM SCSI-0 device cd0: 33.000MB/s transfers cd0: Attempt to query device size failed: NOT READY, Medium not present WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/ad4s1a WARNING: / was not properly dismounted lock order reversal: 1st 0xc0907fcc pf task mtx (pf task mtx) @ /usr/src/sys/contrib/pf/net/pf_ioctl.c:1394 2nd 0xc0973488 ifnet (ifnet) @ /usr/src/sys/net/if.c:1558 KDB: stack backtrace: db_trace_self_wrapper(c088cf61,e658ba3c,c05eb7b6,c088f4ad,c0973488,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c088f4ad,c0973488,c0896cfd,c0896cfd,c0896b56,...) at kdb_backtrace+0x29 witness_checkorder(c0973488,9,c0896b56,616,0,...) at witness_checkorder+0x6d6 _mtx_lock_flags(c0973488,0,c0896b56,616,c3f37a70,...) at _mtx_lock_flags+0xbc ifunit(c3f37a70,0,c08711f2,572,c05e958e,...) at ifunit+0x2f pfioctl(c3d2d800,c0104414,c3f37a70,3,c3f48690,...) at pfioctl+0x23b5 devfs_ioctl_f(c3f49c2c,c0104414,c3f37a70,c3b2c000,c3f48690,...) at devfs_ioctl_f+0xe5 kern_ioctl(c3f48690,3,c0104414,c3f37a70,1000000,...) at kern_ioctl+0x243 ioctl(c3f48690,e658bcfc,c,c08bade8,c08d3630,...) at ioctl+0x134 syscall(e658bd38) at syscall+0x274 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x281aac4b, esp = 0xbfbfde5c, ebp = 0xbfbfde88 --- lock order reversal: 1st 0xc097830c tcp (tcp) @ /usr/src/sys/netinet/tcp_input.c:400 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 KDB: stack backtrace: db_trace_self_wrapper(c088cf61,e42579ac,c05eb7b6,c088f4ad,c09775d8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at kdb_backtrace+0x29 witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at witness_checkorder+0x6d6 _rw_rlock(c09775d8,c0897d93,49,e4257a6c,0,...) at _rw_rlock+0x8e pfil_run_hooks(c09775c0,e4257a8c,c3c31c00,2,0,...) at pfil_run_hooks+0x35 ip_output(c3c46100,0,e4257a50,0,0,0,c08e7c90,0,0,0,c067c807,c08e7c94,c08e7c9c,c8) at ip_output+0x90f tcp_respond(0,c3c87020,c3c87034,c3c46100,2da9088c,...) at tcp_respond+0x3e7 tcp_dropwithreset(1,3,c089c953,353,1900,...) at tcp_dropwithreset+0x152 tcp_input(c3c46100,14,c3c31c00,1,0,...) at tcp_input+0xe45 ip_input(c3c46100,c3c46100,800,c3c31c00,800,...) at ip_input+0x686 netisr_dispatch(2,c3c46100,10,3,0,...) at netisr_dispatch+0x72 ether_demux(c3c31c00,c3c46100,3,0,3,...) at ether_demux+0x2e5 ether_input(c3c31c00,c3c46100,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4 fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0 ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5 fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 --- lock order reversal: 1st 0xc4013d44 udpinp (udpinp) @ /usr/src/sys/netinet/udp_usrreq.c:878 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 KDB: stack backtrace: db_trace_self_wrapper(c088cf61,e658ba14,c05eb7b6,c088f4ad,c09775d8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at kdb_backtrace+0x29 witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at witness_checkorder+0x6d6 _rw_rlock(c09775d8,c0897d93,49,e658bad4,c4013ca8,...) at _rw_rlock+0x8e pfil_run_hooks(c09775c0,e658baf4,c3d44000,2,c4013ca8,...) at pfil_run_hooks+0x35 ip_output(c3ef6100,0,e658bab8,0,0,...) at ip_output+0x90f udp_send(c42454e0,0,c3ef6100,0,0,...) at udp_send+0x8cd sosend_dgram(c42454e0,0,e658bbec,c3ef6100,0,...) at sosend_dgram+0x351 sosend(c42454e0,0,e658bbec,0,0,...) at sosend+0x54 kern_sendit(c3f48690,4,e658bc68,0,0,...) at kern_sendit+0xdb sendit(0,8143023,0,0,0,...) at sendit+0xb1 sendto(c3f48690,e658bcfc,18,c08a5d78,c08d3d98,...) at sendto+0x48 syscall(e658bd38) at syscall+0x274 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (133, FreeBSD ELF32, sendto), eip = 0x2816bc83, esp = 0xbfbfd73c, ebp = 0xbfbfd768 --- lock order reversal: 1st 0xc423f150 tcpinp (tcpinp) @ /usr/src/sys/netinet/tcp_usrreq.c:472 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 KDB: stack backtrace: db_trace_self_wrapper(c088cf61,e65a3a30,c05eb7b6,c088f4ad,c09775d8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at kdb_backtrace+0x29 witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at witness_checkorder+0x6d6 _rw_rlock(c09775d8,c0897d93,49,e65a3af0,c423f0b4,...) at _rw_rlock+0x8e pfil_run_hooks(c09775c0,e65a3b10,c3d44000,2,c423f0b4,...) at pfil_run_hooks+0x35 ip_output(c3c94e00,0,e65a3ad4,0,0,...) at ip_output+0x90f tcp_output(c42421d0,c3d2bc50,1d8,c423f150,c4259000,...) at tcp_output+0x140c tcp_usr_connect(c4259000,c3d2bc50,c3d2f8c0,25,e65a3c64,...) at tcp_usr_connect+0x11c soconnect(c4259000,c3d2bc50,c3d2f8c0,10,16,...) at soconnect+0x52 kern_connect(c3d2f8c0,9,c3d2bc50,c3d2bc50,0,...) at kern_connect+0x59 connect(c3d2f8c0,e65a3cfc,c,c088ff65,c08d3a50,...) at connect+0x46 syscall(e65a3d38) at syscall+0x274 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (98, FreeBSD ELF32, connect), eip = 0x28161e9b, esp = 0xbfbfe71c, ebp = 0xbfbfe868 --- lock order reversal: 1st 0xc3eda524 tcp_sc_head (tcp_sc_head) @ /usr/src/sys/netinet/tcp_syncache.c:494 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 KDB: stack backtrace: db_trace_self_wrapper(c088cf61,e4257854,c05eb7b6,c088f4ad,c09775d8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at kdb_backtrace+0x29 witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at witness_checkorder+0x6d6 _rw_rlock(c09775d8,c0897d93,49,e4257914,0,...) at _rw_rlock+0x8e pfil_run_hooks(c09775c0,e4257934,c3c31c00,2,0,...) at pfil_run_hooks+0x35 ip_output(c3ef7a00,0,e42578f8,0,0,...) at ip_output+0x90f syncache_respond(c426ad70,c40c0834,0,0,c40c0834,...) at syncache_respond+0x3a2 _syncache_add(c42400b4,e4257ba8,c40b3700,0,0,...) at _syncache_add+0x2b0 syncache_add(e4257b68,e4257b90,c40c0834,c42400b4,e4257ba8,...) at syncache_add+0x38 tcp_input(c40b3700,14,c3c31c00,1,0,...) at tcp_input+0xd6b ip_input(c40b3700,c40b3700,800,c3c31c00,800,...) at ip_input+0x686 netisr_dispatch(2,c40b3700,10,3,0,...) at netisr_dispatch+0x72 ether_demux(c3c31c00,c40b3700,3,0,3,...) at ether_demux+0x2e5 ether_input(c3c31c00,c40b3700,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4 fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0 ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5 fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 --- lock order reversal: 1st 0xc09786cc udp (udp) @ /usr/src/sys/netinet/udp_usrreq.c:395 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 KDB: stack backtrace: db_trace_self_wrapper(c088cf61,e42579b8,c05eb7b6,c088f4ad,c09775d8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at kdb_backtrace+0x29 witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at witness_checkorder+0x6d6 _rw_rlock(c09775d8,c0897d93,49,e4257a78,0,...) at _rw_rlock+0x8e pfil_run_hooks(c09775c0,e4257a98,c3c31c00,2,0,...) at pfil_run_hooks+0x35 ip_output(c3efae00,0,e4257a5c,0,0,...) at ip_output+0x90f icmp_reflect(c40c6020,c3efaec8,14,c3efaf00,c40c6020,...) at icmp_reflect+0x3df icmp_error(c40b4d00,3,3,0,0,...) at icmp_error+0x3bd udp_input(c40b4d00,14,c3c31c00,1,0,...) at udp_input+0x5ea ip_input(c40b4d00,c40b4d00,800,c3c31c00,800,...) at ip_input+0x686 netisr_dispatch(2,c40b4d00,10,3,0,...) at netisr_dispatch+0x72 ether_demux(c3c31c00,c40b4d00,3,0,3,...) at ether_demux+0x2e5 ether_input(c3c31c00,c40b4d00,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4 fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0 ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5 fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 --- kernel backtrace: GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: panic: _rw_rlock (tcp): wlock already held @ /usr/src/sys/contrib/pf/net/pf.c:3016 cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper(c088cf61,e6846220,c05ae7df,c08b659d,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08b659d,0,c0889c7e,e684622c,0,...) at kdb_backtrace+0x29 panic(c0889c7e,c085a754,c088f55e,c087092d,bc8,...) at panic+0x10f _rw_rlock(c097830c,c087092d,bc8,c08d9624,c087092d,...) at _rw_rlock+0x73 pf_socket_lookup(2,e68463dc,0,cc4,3,...) at pf_socket_lookup+0x208 pf_test_tcp(e6846444,e6846440,2,c3efee00,c3c8e900,...) at pf_test_tcp+0x142 pf_test6(2,c3d44000,e68464a0,0,0,...) at pf_test6+0x8a0 pf_check6_out(0,e68464a0,c3d44000,2,0,...) at pf_check6_out+0x47 pfil_run_hooks(c097ad00,e6846638,c3d44000,2,0,...) at pfil_run_hooks+0x88 ip6_output(c3c8e900,0,e6846618,0,0,...) at ip6_output+0x122e pf_send_tcp(c4fcfe00,c41259b4,1c,c4fcfe5c,c4fcfe4c,...) at pf_send_tcp+0x6dd pf_test_tcp(e68468e8,e68468e4,2,c3f20900,c4fcfe00,...) at pf_test_tcp+0xcef pf_test6(2,c3f06400,e6846944,0,c446b7bc,...) at pf_test6+0x8a0 pf_check6_out(0,e6846944,c3f06400,2,c446b7bc,...) at pf_check6_out+0x47 pfil_run_hooks(c097ad00,e6846adc,c3f06400,2,c446b7bc,...) at pfil_run_hooks+0x88 ip6_output(c4fcfe00,0,e6846abc,0,0,...) at ip6_output+0x122e tcp_output(c45553a0,c447e7c0,201,c446b858,c45553a0,...) at tcp_output+0x137e tcp6_usr_connect(c50cd340,c447e7c0,c4eed690,25,e6846c64,...) at tcp6_usr_connect+0x171 soconnect(c50cd340,c447e7c0,c4eed690,1c,16,...) at soconnect+0x52 kern_connect(c4eed690,3,c447e7c0,c447e7c0,0,...) at kern_connect+0x59 connect(c4eed690,e6846cfc,c,c08a288e,c08d3a50,...) at connect+0x46 syscall(e6846d38) at syscall+0x274 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (98, FreeBSD ELF32, connect), eip = 0x282e6e9b, esp = 0xbfbfe7ec, ebp = 0xbfbfe848 --- KDB: enter: panic shared rw PFil hook read/write mutex r = 1 (0xc097ad18) locked @ /usr/src/sys/net/pfil.c:73 exclusive rw tcpinp r = 0 (0xc446b858) locked @ /usr/src/sys/netinet/tcp_usrreq.c:513 exclusive rw tcp r = 0 (0xc097830c) locked @ /usr/src/sys/netinet/tcp_usrreq.c:510 exclusive sx so_rcv_sx r = 0 (0xc452fbec) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc483cbec) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc4e89bec) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc4e8970c) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc483c22c) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc480d70c) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc4e8a08c) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc4e8a56c) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc41a456c) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc41c156c) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc41c18ac) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc41c1bec) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx r = 0 (0xc41f108c) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 shared rw udpinp r = 0 (0xc400f63c) locked @ /usr/src/sys/netinet/udp_usrreq.c:878 Uptime: 16h23m36s Physical memory: 1015 MB Dumping 166 MB: 151 135 119 103 87 71 55 39 23 7 Reading symbols from /boot/kernel/if_em.ko...Reading symbols from /boot/kernel/if_em.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_em.ko Reading symbols from /boot/kernel/if_fxp.ko...Reading symbols from /boot/kernel/if_fxp.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_fxp.ko Reading symbols from /boot/kernel/miibus.ko...Reading symbols from /boot/kernel/miibus.ko.symbols...done. done. Loaded symbols for /boot/kernel/miibus.ko Reading symbols from /boot/kernel/if_vr.ko...Reading symbols from /boot/kernel/if_vr.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_vr.ko Reading symbols from /boot/kernel/ulpt.ko...Reading symbols from /boot/kernel/ulpt.ko.symbols...done. done. Loaded symbols for /boot/kernel/ulpt.ko Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done. done. Loaded symbols for /boot/kernel/accf_http.ko Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done. done. Loaded symbols for /boot/kernel/acpi.ko #0 doadump () at pcpu.h:196 196 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:196 #1 0xc05ae54c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xc05ae816 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:572 #3 0xc05acf63 in _rw_rlock (rw=0xc097830c, file=0xc087092d "/usr/src/sys/contrib/pf/net/pf.c", line=3016) at /usr/src/sys/kern/kern_rwlock.c:253 #4 0xc0473e58 in pf_socket_lookup (direction=2, pd=0xe68463dc, inp_arg=0x0) at /usr/src/sys/contrib/pf/net/pf.c:3016 #5 0xc047dd62 in pf_test_tcp (rm=0xe6846444, sm=0xe6846440, direction=2, kif=0xc3efee00, m=0xc3c8e900, off=40, h=0xc3c8e944, pd=0xe68463dc, am=0xe6846448, rsm=0xe684643c, ifq=0x0, inp=0x0) at /usr/src/sys/contrib/pf/net/pf.c:3270 #6 0xc04816c0 in pf_test6 (dir=2, ifp=0xc3d44000, m0=0xe68464a0, eh=0x0, inp=0x0) at /usr/src/sys/contrib/pf/net/pf.c:7368 #7 0xc0484e37 in pf_check6_out (arg=0x0, m=0xe68464a0, ifp=0xc3d44000, dir=2, inp=0x0) at /usr/src/sys/contrib/pf/net/pf_ioctl.c:3739 #8 0xc0657618 in pfil_run_hooks (ph=0xc097ad00, mp=0xe6846638, ifp=0xc3d44000, dir=2, inp=0x0) at /usr/src/sys/net/pfil.c:78 #9 0xc07034fe in ip6_output (m0=0xc3c8e900, opt=0x0, ro=0xe6846618, flags=Variable "flags" is not available. ) at /usr/src/sys/netinet6/ip6_output.c:853 #10 0xc0477dad in pf_send_tcp (replyto=0xc4fcfe00, r=0xc41259b4, af=28 '\034', saddr=0xc4fcfe5c, daddr=0xc4fcfe4c, sport=20480, dport=46591, seq=0, ack=1170313007, flags=20 '\024', win=0, mss=0, ttl=0 '\0', tag=1, rtag=0, eh=0x0, ifp=0xc3f06400) at /usr/src/sys/contrib/pf/net/pf.c:1978 #11 0xc047e90f in pf_test_tcp (rm=0xe68468e8, sm=0xe68468e4, direction=2, kif=0xc3f20900, m=0xc4fcfe00, off=40, h=0xc4fcfe44, pd=0xe6846880, am=0xe68468ec, rsm=0xe68468e0, ifq=0x0, inp=0xc446b7bc) at /usr/src/sys/contrib/pf/net/pf.c:3424 #12 0xc04816c0 in pf_test6 (dir=2, ifp=0xc3f06400, m0=0xe6846944, eh=0x0, inp=0xc446b7bc) at /usr/src/sys/contrib/pf/net/pf.c:7368 #13 0xc0484e37 in pf_check6_out (arg=0x0, m=0xe6846944, ifp=0xc3f06400, dir=2, inp=0xc446b7bc) at /usr/src/sys/contrib/pf/net/pf_ioctl.c:3739 #14 0xc0657618 in pfil_run_hooks (ph=0xc097ad00, mp=0xe6846adc, ifp=0xc3f06400, dir=2, inp=0xc446b7bc) at /usr/src/sys/net/pfil.c:78 #15 0xc07034fe in ip6_output (m0=0xc4fcfe00, opt=0x0, ro=0xe6846abc, flags=Variable "flags" is not available. ) at /usr/src/sys/netinet6/ip6_output.c:853 #16 0xc06debbe in tcp_output (tp=0xc45553a0) at /usr/src/sys/netinet/tcp_output.c:1114 #17 0xc06ea5d1 in tcp6_usr_connect (so=0xc50cd340, nam=0xc447e7c0, td=0xc4eed690) at tcp_offload.h:257 #18 0xc060b002 in soconnect (so=0xc50cd340, nam=0xc447e7c0, td=0xc4eed690) at /usr/src/sys/kern/uipc_socket.c:771 #19 0xc06129e9 in kern_connect (td=0xc4eed690, fd=3, sa=0xc447e7c0) at /usr/src/sys/kern/uipc_syscalls.c:570 #20 0xc0612b56 in connect (td=0xc4eed690, uap=0xe6846cfc) at /usr/src/sys/kern/uipc_syscalls.c:534 #21 0xc083a2d4 in syscall (frame=0xe6846d38) at /usr/src/sys/i386/i386/trap.c:1090 #22 0xc0821220 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #23 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted: