From owner-freebsd-current@FreeBSD.ORG Tue Nov 25 12:52:35 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 39E0C16A4CE for ; Tue, 25 Nov 2003 12:52:35 -0800 (PST) Received: from mx.dsl.isometry.net (cpc3-oxfd2-6-0-cust207.oxfd.cable.ntl.com [81.103.193.207]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5304F43FCB for ; Tue, 25 Nov 2003 12:52:31 -0800 (PST) (envelope-from robin@isometry.net) Received: from isometry.net (ishadow.dsl.isometry.net [195.137.51.150]) by mx.dsl.isometry.net (Postfix) with ESMTP id 26A21AA for ; Tue, 25 Nov 2003 20:52:27 +0000 (UTC) Message-ID: <3FC3C109.8030101@isometry.net> Date: Tue, 25 Nov 2003 20:52:25 +0000 From: Robin Breathe User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6b) Gecko/20031119 Thunderbird/0.4a X-Accept-Language: en-us, en MIME-Version: 1.0 To: current@freebsd.org Content-Type: multipart/mixed; boundary="------------050406030803040301080709" Subject: netgraph/ng_eiface double panic & turnstile/sio lock order reversal in 5.2-BETA X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Nov 2003 20:52:35 -0000 This is a multi-part message in MIME format. --------------050406030803040301080709 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit I've been experiencing a repeatable panic using ng_eiface(4) on -CURRENT of the last few days. Environment: FreeBSD twiddle.local 5.2-BETA FreeBSD 5.2-BETA #0: Tue Nov 25 19:28:22 UTC 2003 root@twiddle.local:/home/data/work/usr/src/sys/TWIDDLE i386 Description: Shutting down an ng_eiface node which has a non-zero lladdr causes a panic. How-To-Repeat: Configure an ng_eiface(4) node, set its lladdr with ifconfig(8), shut it down: ################ # cat >/tmp/ngctl mkpeer . eiface hook ether name .:hook vif0 rmhook . hook # ngctl -f /tmp/ngctl # ifconfig ngeth0 link '00:bd:03:11:25:01' # ngctl shutdown vif0: lock order reversal 1st 0xc0765d8c turnstile chain (turnstile chain) @ /usr/src/sys/kern/subr_turnstile.c:370 2nd 0xc079a500 sio (sio) @ /usr/src/sys/dev/sio/sio.c:3203 Stack backtrace: backtrace(c070794e,c079a500,c07502c0,c07502c0,c0719757) at backtrace+0x17 witness_lock(c079a500,8,c0719757,c83,3f8) at witness_lock+0x672 _mtx_lock_spin_flags(c079a500,0,c0719757,c83,1) at _mtx_lock_spin_flags+0xda siocnputc(c0750440,6b,5,ddcb08b4,6b) at siocnputc+0x81 cnputc(6b,c076e780,1,c4a40500,c071d675) at cnputc+0x7a putchar(6b,ddcb08b4,c053a72d,c076e800,1) at putchar+0x6c kvprintf(c071d674,c0564b80,ddcb08b4,a,ddcb08d4) at kvprintf+0x8d printf(c071d674,c,c056ced2,c078ac40,38) at printf+0x57 trap(c0760018,c0760010,c0760010,0,c4a40500) at trap+0xd7 calltrap() at calltrap+0x5 --- trap 0xc, eip = 0xc056a146, esp = 0xddcb0958, ebp = 0xddcb0978 --- turnstile_wait(0,c479b9c8,1103bd00,1cc,c479b9c8) at turnstile_wait+0x86 _mtx_lock_sleep(c479b9c8,0,c070c7ca,250,c49efe7c) at _mtx_lock_sleep+0x115 _mtx_lock_flags(c479b9c8,0,c070c7ca,250,c0538e1c) at _mtx_lock_flags+0x97 if_detach(c479b808,c4d64300,ddcb0a58,c4dbfa51,c479b808) at if_detach+0x394 ether_ifdetach(c479b808,c070d32d,820,c4d64300,c4d64300) at ether_ifdetach+0x30 ng_eiface_rmnode(c4d64300,0,0,c4d64300,c4d64300) at ng_eiface_rmnode+0x61 ng_rmnode(c4d64300,0,0,0,0) at ng_rmnode+0xc7 ng_generic_msg(c4d64300,c48b7780,0,c0768598,c07acfd4) at ng_generic_msg+0x11f ng_apply_item(c4d64300,c48b7780,c070d32d,7d6,c48b7780) at ng_apply_item+0x365 ng_snd_item(c48b7780,0,c47a4360,0,0) at ng_snd_item+0x7b6 ngc_send(c4ab4780,0,c1d12800,c47a4820,0) at ngc_send+0x146 sosend(c4ab4780,c47a4820,ddcb0c4c,c1d12800,0) at sosend+0x4cd kern_sendit(c4a40500,3,ddcb0cc4,0,0) at kern_sendit+0x17c sendit(c4a40500,3,ddcb0cc4,0,804f034) at sendit+0x16e sendto(c4a40500,ddcb0d14,c071d6f6,3ee,6) at sendto+0x5b syscall(2f,2f,2f,bfbfe9c0,bfbfe9c2) at syscall+0x2c0 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (133), eip = 0x280c58cf, esp = 0xbfbfe96c, ebp = 0xbfbfebe8 --- kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address = 0x1103bd00 fault code = supervisor read, page not present instruction pointer = 0x8:0xc056a146 stack pointer = 0x10:0xddcb0958 frame pointer = 0x10:0xddcb0978 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 582 (ngctl) kernel: type 12 trap, code=0 Stopped at turnstile_wait+0x86: movl 0(%edx),%eax db> panic panic: from debugger Debugger("panic") Fatal trap 3: breakpoint instruction fault while in kernel mode instruction pointer = 0x8:0xc06a9254 stack pointer = 0x10:0xddcb0720 frame pointer = 0x10:0xddcb072c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = IOPL = 0 current process = 582 (ngctl) Stopped at turnstile_wait+0x86: movl 0(%edx),%eax db> where turnstile_wait(0,c479b9c8,1103bd00,1cc,c479b9c8) at turnstile_wait+0x86 _mtx_lock_sleep(c479b9c8,0,c070c7ca,250,c49efe7c) at _mtx_lock_sleep+0x115 _mtx_lock_flags(c479b9c8,0,c070c7ca,250,c0538e1c) at _mtx_lock_flags+0x97 if_detach(c479b808,c4d64300,ddcb0a58,c4dbfa51,c479b808) at if_detach+0x394 ether_ifdetach(c479b808,c070d32d,820,c4d64300,c4d64300) at ether_ifdetach+0x30 ng_eiface_rmnode(c4d64300,0,0,c4d64300,c4d64300) at ng_eiface_rmnode+0x61 ng_rmnode(c4d64300,0,0,0,0) at ng_rmnode+0xc7 ng_generic_msg(c4d64300,c48b7780,0,c0768598,c07acfd4) at ng_generic_msg+0x11f ng_apply_item(c4d64300,c48b7780,c070d32d,7d6,c48b7780) at ng_apply_item+0x365 ng_snd_item(c48b7780,0,c47a4360,0,0) at ng_snd_item+0x7b6 ngc_send(c4ab4780,0,c1d12800,c47a4820,0) at ngc_send+0x146 sosend(c4ab4780,c47a4820,ddcb0c4c,c1d12800,0) at sosend+0x4cd kern_sendit(c4a40500,3,ddcb0cc4,0,0) at kern_sendit+0x17c sendit(c4a40500,3,ddcb0cc4,0,804f034) at sendit+0x16e sendto(c4a40500,ddcb0d14,c071d6f6,3ee,6) at sendto+0x5b syscall(2f,2f,2f,bfbfe9c0,bfbfe9c2) at syscall+0x2c0 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (133, FreeBSD ELF32, sendto), eip = 0x280c58cf, esp = 0xbfbfe96c, ebp = 0xbfbfebe8 --- db> panic panic: from debugger Uptime: 2m3s panic: mi_switch: switch in a critical section Uptime: 2m3s panic: msleep Uptime: 2m3s # repeated a few times Uptime: 2m3s panic: msleep Uptime: 2m3s panic: msleep Fatal double fault: eip = 0xc053a580 esp = 0xddcaf000 ebp = 0xddcaf030 panic: double fault Uptime: 2m3s panic: msleep Uptime: 2m3s # repeated a few more times panic: msleep Uptime: 2m3s panic: msleep spin lock sched lock held by Console: serial port ################ Notes: With kernel debugging options disabled, but still 'options DDB; makeoptions DEBUG=-g', the LOR doesn't show up, but the netgraph panic is still present. The above panic is from -CURRENT as of Tue Nov 25 18:00:00 UTC 2003. This, at least superficially, looks similar to kern/55784 filed against 4.8-STABLE. Find attached a full dmesg, with example script, panic and kernel config. The system is failing to dump core. I'll happily help debug this as much as is necessary before 5.2-REL. Hopeful, - Robin -- Robin Breathe robin@isometry.net +441865741800 --------------050406030803040301080709 Content-Type: text/plain; name="ng_eiface_panic.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="ng_eiface_panic.txt" Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.2-BETA #0: Tue Nov 25 19:28:22 UTC 2003 root@twiddle.local:/home/data/work/usr/src/sys/TWIDDLE Preloaded elf kernel "/boot/kernel/kernel" at 0xc0848000. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) III CPU - S 1400MHz (1407.05-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6b4 Stepping = 4 Features=0x383fbff real memory = 536784896 (511 MB) avail memory = 511860736 (488 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard Pentium Pro MTRR support enabled VESA: v3.0, 32768k memory, flags:0x1, mode table:0xc07af442 (1000022) VESA: NVidia acpi0: on motherboard acpi0: Overriding SCI Interrupt from IRQ 9 to IRQ 20 pcibios: BIOS version 2.10 Using $PIR table, 11 entries at 0xc00f13a0 acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0xe408-0xe40b on acpi0 acpi_cpu0: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 agp0: mem 0xf8000000-0xfbffffff at device 0.0 on pci0 pcib1: at device 1.0 on pci0 pci1: on pcib1 pci1: at device 0.0 (no driver attached) pcib2: at device 30.0 on pci0 pci2: on pcib2 ahc0: port 0xd800-0xd8ff mem 0xed800000-0xed800fff irq 21 at device 9.0 on pci2 aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs fxp0: port 0xd400-0xd43f mem 0xec800000-0xec81ffff,0xed000000-0xed000fff irq 22 at device 10.0 on pci2 fxp0: Ethernet address 00:02:b3:a7:de:2b miibus0: on fxp0 inphy0: on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto pcib3: at device 11.0 on pci2 pci3: on pcib3 pcib3: slot 8 INTA is routed to irq 23 pcib3: slot 8 INTB is routed to irq 20 pcib3: slot 8 INTC is routed to irq 21 pcib3: slot 11 INTA is routed to irq 22 ohci0: mem 0xec000000-0xec000fff irq 23 at device 8.0 on pci3 usb0: OHCI version 1.0 usb0: on ohci0 usb0: USB revision 1.0 uhub0: NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 3 ports with 3 removable, self powered ohci1: mem 0xeb800000-0xeb800fff irq 20 at device 8.1 on pci3 usb1: OHCI version 1.0 usb1: on ohci1 usb1: USB revision 1.0 uhub1: NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered pci3: at device 8.2 (no driver attached) fwohci0: mem 0xea000000-0xea003fff,0xea800000-0xea8007ff irq 22 at device 11.0 on pci3 fwohci0: OHCI version 1.0 (ROM=1) fwohci0: No. of Isochronous channel is 4. fwohci0: EUI64 00:50:42:b5:00:01:11:36 fwohci0: Phy 1394a available S400, 3 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 sbp0: on firewire0 fwohci0: Initiate bus reset fwohci0: BUS reset fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me) firewire0: bus manager 0 (me) pci2: at device 12.0 (no driver attached) pci2: at device 12.1 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0xa800-0xa80f at device 31.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata0: [MPSAFE] ata1: at 0x170 irq 15 on atapci0 ata1: [MPSAFE] uhci0: port 0xa400-0xa41f irq 19 at device 31.2 on pci0 usb2: on uhci0 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered ichsmb0: port 0xe800-0xe80f irq 17 at device 31.3 on pci0 smbus0: on ichsmb0 smb0: on smbus0 uhci1: port 0xa000-0xa01f irq 23 at device 31.4 on pci0 usb3: on uhci1 usb3: USB revision 1.0 uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub3: 2 ports with 2 removable, self powered sio0 port 0x3f8-0x3ff irq 4 on acpi0 sio0: type 16550A, console sio1 port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A atkbdc0: port 0x64,0x60 irq 1 on acpi0 npx0: [FAST] stray irq13 npx0: on motherboard npx0: INT 16 interface orm0: