Date: Sun, 30 Oct 2016 03:49:30 +0000 From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837) Message-ID: <bug-213903-8@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D213903 Bug ID: 213903 Summary: Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837) Product: Base System Version: CURRENT Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: woodsb02@freebsd.org I am currently experiencing semi-regular kernel crashes on my FreeBSD 12-current machine. I am new to kernel debugging, and hoping someone can ha= ve a look at the debugging output below to point me in the direction of what the problem might be. My machine is a FreeNAS-mini from iXsystems which I have formatted and installed stock FreeBSD onto. My kernel is the default generic-nodebug with= the VIMAGE options added. $ uname -a FreeBSD freenas.woods.am 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r305311M: Sat= Sep 3 12:29:01 AWST 2016=20=20=20=20 woodsb02@freenas.woods.am:/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE amd64 $ cat /usr/src/sys/amd64/conf/GENERIC-NODEBUG-VIMAGE # SPARTICUS -- WITNESS and INVARIANTS free kernel configuration file # for FreeBSD/amd64 include GENERIC-NODEBUG ident GENERIC-NODEBUG-VIMAGE #nooptions SCTP # Stream Control Transmission Protocol options VIMAGE # VNET/Vimage support Output from kernel crash dump debug with kgdb below: /usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE)# kgdb kernel.debug /var/crash/vmcore.last GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid =3D 2; apic id =3D 04 fault virtual address =3D 0x30 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80b4d91c stack pointer =3D 0x28:0xfffffe046813a440 frame pointer =3D 0x28:0xfffffe046813a470 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D resume, IOPL =3D 0 current process =3D 33487 (sh) Uptime: 15m16s Dumping 1664 out of 16338 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..= 91% Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko.debug... done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko...Read= ing symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko...Rea= ding symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko.debug...don= e. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko...Read= ing symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko...Rea= ding symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko.debug...don= e. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko...Rea= ding symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko.debug...don= e. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko...Read= ing symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko.debug...do= ne. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko.debug...do= ne. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko...Read= ing symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko...Rea= ding symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko.debug...don= e. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko.debug...= done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko...Readi= ng symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko...Readi= ng symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko...Read= ing symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko #0 doadump (textdump=3D1) at pcpu.h:221 221 __asm("movq %%gs:%1,%0" : "=3Dr" (td) (kgdb) list *0xffffffff80b4d91c 0xffffffff80b4d91c is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837). 832 833 /* 834 * Transfer the blocked list to the pending list. 835 */ 836 mtx_lock_spin(&td_contested_lock); 837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); 838 mtx_unlock_spin(&td_contested_lock); 839 840 /* 841 * Give a turnstile to each thread. The last thread gets Current language: auto; currently minimal (kgdb) backtrace #0 doadump (textdump=3D1) at pcpu.h:221 #1 0xffffffff80aea40e in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:366 #2 0xffffffff80aea9db in vpanic (fmt=3D<value optimized out>, ap=3D<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80aea813 in panic (fmt=3D0x0) at /usr/src/sys/kern/kern_shutdown.c:690 #4 0xffffffff8039e197 in db_panic (addr=3D<value optimized out>, have_addr=3Dfalse, count=3D0, modif=3D0x0) at /usr/src/sys/ddb/db_command.c= :486 #5 0xffffffff8039d689 in db_command (cmd_table=3D<value optimized out>) at /usr/src/sys/ddb/db_command.c:453 #6 0xffffffff8039d3e4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:506 #7 0xffffffff803a053b in db_trap (type=3D<value optimized out>, code=3D<va= lue optimized out>) at /usr/src/sys/ddb/db_main.c:251 #8 0xffffffff80b36b33 in kdb_trap (type=3D<value optimized out>, code=3D<v= alue optimized out>, tf=3D<value optimized out>) at /usr/src/sys/kern/subr_kdb.c= :654 #9 0xffffffff80fdd441 in trap_fatal (frame=3D0xfffffe046813a390, eva=3D48)= at /usr/src/sys/amd64/amd64/trap.c:836 #10 0xffffffff80fdd673 in trap_pfault (frame=3D0xfffffe046813a390, usermode= =3D0) at /usr/src/sys/amd64/amd64/trap.c:691 #11 0xffffffff80fdcbfc in trap (frame=3D0xfffffe046813a390) at /usr/src/sys/amd64/amd64/trap.c:442 #12 0xffffffff80fbf491 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 #13 0xffffffff80b4d91c in turnstile_broadcast (ts=3D0x0, queue=3D1) at /usr/src/sys/kern/subr_turnstile.c:837 #14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=3D0xfffff803f886d960, tid=3D= <value optimized out>, file=3D<value optimized out>, line=3D<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1027 #15 0xffffffff80e525dc in vm_map_delete (map=3D<value optimized out>, start=3D<value optimized out>, end=3D<value optimized out>) at /usr/src/sys/vm/vm_map.c:2960 #16 0xffffffff80e54477 in vm_map_remove (map=3D0xfffff8035540f000, start=3D140737488355328, end=3D1) at /usr/src/sys/vm/vm_map.c:3077 #17 0xffffffff80a9863f in exec_new_vmspace (imgp=3D0xfffffe046813a860, sv=3D0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096 #18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=3D<value optimized out>) = at /usr/src/sys/kern/imgact_elf.c:896 #19 0xffffffff80a9670d in kern_execve (td=3D<value optimized out>, args=3D<= value optimized out>, mac_p=3D0x0) at /usr/src/sys/kern/kern_exec.c:603 #20 0xffffffff80a95b9c in sys_execve (td=3D0xfffff8032893aa00, uap=3D0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219 #21 0xffffffff80fddde8 in amd64_syscall (td=3D<value optimized out>, traced= =3D0) at subr_syscall.c:135 #22 0xffffffff80fbf77b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 #23 0x0000000800b468ea in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) up 11 #11 0xffffffff80fdcbfc in trap (frame=3D0xfffffe046813a390) at /usr/src/sys/amd64/amd64/trap.c:442 442 (void) trap_pfault(frame, FALSE); (kgdb) list 437 438 KASSERT(cold || td->td_ucred !=3D NULL, 439 ("kernel trap doesn't have ucred")); 440 switch (type) { 441 case T_PAGEFLT: /* page fault */ 442 (void) trap_pfault(frame, FALSE); 443 goto out; 444 445 case T_DNA: 446 if (PCB_USER_FPU(td->td_pcb)) (kgdb) print td $1 =3D (struct thread *) 0xfffff8032893aa00 (kgdb) print td->td_ucred $2 =3D (struct ucred *) 0xfffff8004005ec00 (kgdb) print type $3 =3D 12 (kgdb) up #12 0xffffffff80fbf491 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 236 call trap_check Current language: auto; currently asm (kgdb) up #13 0xffffffff80b4d91c in turnstile_broadcast (ts=3D0x0, queue=3D1) at /usr/src/sys/kern/subr_turnstile.c:837 837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); Current language: auto; currently minimal (kgdb) up #14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=3D0xfffff803f886d960, tid=3D= <value optimized out>, file=3D<value optimized out>, line=3D<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1027 1027 turnstile_broadcast(ts, queue); (kgdb) up #15 0xffffffff80e525dc in vm_map_delete (map=3D<value optimized out>, start=3D<value optimized out>, end=3D<value optimized out>) at /usr/src/sys/vm/vm_map.c:2960 2960 VM_OBJECT_WUNLOCK(object); (kgdb) up #16 0xffffffff80e54477 in vm_map_remove (map=3D0xfffff8035540f000, start=3D140737488355328, end=3D1) at /usr/src/sys/vm/vm_map.c:3077 3077 result =3D vm_map_delete(map, start, end); (kgdb) up #17 0xffffffff80a9863f in exec_new_vmspace (imgp=3D0xfffffe046813a860, sv=3D0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096 1096 vm_map_remove(map, vm_map_min(map), vm_map_max(map)= ); (kgdb) up #18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=3D<value optimized out>) = at /usr/src/sys/kern/imgact_elf.c:896 896 error =3D exec_new_vmspace(imgp, sv); (kgdb) up #19 0xffffffff80a9670d in kern_execve (td=3D<value optimized out>, args=3D<= value optimized out>, mac_p=3D0x0) at /usr/src/sys/kern/kern_exec.c:603 603 error =3D (*execsw[i]->ex_imgact)(imgp); (kgdb) up #20 0xffffffff80a95b9c in sys_execve (td=3D0xfffff8032893aa00, uap=3D0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219 219 error =3D kern_execve(td, &args, NULL); (kgdb) up #21 0xffffffff80fddde8 in amd64_syscall (td=3D<value optimized out>, traced= =3D0) at subr_syscall.c:135 135 error =3D (sa->callp->sy_call)(td, sa->args); (kgdb) up #22 0xffffffff80fbf77b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 396 call amd64_syscall Current language: auto; currently asm (kgdb) up #23 0x0000000800b468ea in ?? () (kgdb) up Initial frame selected; you cannot go up. (kgdb) quit After being requested for more info from Mateusz Guzik <mjguzik@gmail.com>,= I also performed the following kgdb command: (kgdb) f 14 #14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=3D0xfffff803f886d960, tid=3D= <value optimized out>, file=3D<value optimized out>, line=3D<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1027 1027 turnstile_broadcast(ts, queue); Current language: auto; currently minimal (kgdb) x/xg c 0xfffff803f886d960: 0xfffff8032893aa00 --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-213903-8>