From owner-freebsd-net@freebsd.org Tue Sep 22 16:49:47 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9275A074D7 for ; Tue, 22 Sep 2015 16:49:46 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from mail.pingpong.net (mail.pingpong.net [79.136.116.202]) by mx1.freebsd.org (Postfix) with ESMTP id 4A42E1C02; Tue, 22 Sep 2015 16:49:45 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from mail.pingpong.net (localhost [127.0.0.1]) by mail.pingpong.net (Postfix) with ESMTP id 4ED0AD13C; Tue, 22 Sep 2015 18:49:45 +0200 (CEST) X-Virus-Scanned: by amavisd-new at pingpong.net Received: from mail.pingpong.net ([127.0.0.1]) by mail.pingpong.net (mail.pingpong.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id CSKo1GG-g3nl; Tue, 22 Sep 2015 18:49:44 +0200 (CEST) Received: from [10.0.0.143] (citron2.pingpong.net [195.178.173.68]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.pingpong.net (Postfix) with ESMTPSA id CEE1BD139; Tue, 22 Sep 2015 18:49:41 +0200 (CEST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: Kernel panics in tcp_twclose From: Palle Girgensohn In-Reply-To: <73856F2B-3E70-483C-9988-C84E798CEB44@FreeBSD.org> Date: Tue, 22 Sep 2015 18:49:41 +0200 Cc: Konstantin Belousov , freebsd-net@freebsd.org, Hans Petter Selasky Content-Transfer-Encoding: quoted-printable Message-Id: <44EBAC98-4761-4E47-8E47-5032430A1C8A@FreeBSD.org> References: <26B0FF93-8AE3-4514-BDA1-B966230AAB65@FreeBSD.org> <55FC1809.3070903@freebsd.org> <20150918160605.GN67105@kib.kiev.ua> <55FFBE01.6060706@freebsd.org> <3721F099-F45D-4DCD-8AB3-84D1ABC44145@FreeBSD.org> <73856F2B-3E70-483C-9988-C84E798CEB44@FreeBSD.org> To: Julien Charbon X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Sep 2015 16:49:47 -0000 > 22 sep 2015 kl. 18:46 skrev Palle Girgensohn : >=20 > Hi all, >=20 >=20 >> 21 sep 2015 kl. 15:53 skrev Palle Girgensohn : >>=20 >>>=20 >>> 21 sep 2015 kl. 10:21 skrev Julien Charbon : >>>=20 >>>=20 >>> Hi Konstantin, Hi Palle, >>>=20 >>> On 18/09/15 18:06, Konstantin Belousov wrote: >>>> On Fri, Sep 18, 2015 at 03:56:25PM +0200, Julien Charbon wrote: >>>>> Hi Palle, >>>>>=20 >>>>> On 18/09/15 11:12, Palle Girgensohn wrote: >>>>>> We see daily panics on our production systems (web server, apache >>>>>> running MPM event, openjdk8. Kernel with VIMAGE. Jails using = netgraph >>>>>> interfaces [not epair]). >>>>>>=20 >>>>>> The problem started after the summer. Normal port upgrades seems = to >>>>>> be the only difference. The problem occurs with 10.2-p2 kernel as >>>>>> well as 10.1-p4 and 10.1-p15. >>>>>>=20 >>>>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203175 >>>>>>=20 >>>>>> Any ideas? >>>>>=20 >>>>> Thanks for you detailed report. I am not aware of any = tcp_twclose() >>>>> related issues (without VIMAGE) since FreeBSD 10.0 (does not mean = there >>>>> are none). Few interesting facts (at least for me): >>>>>=20 >>>>> - Your crash happens when unlocking a inp exclusive lock with = INP_WUNLOCK() >>>>>=20 >>>>> - Something is already wrong before calling turnstile_broadcast() = as it >>>>> is called with ts =3D NULL: >>>> In the kernel without witness this is a 99%-sure indication of = attempt to >>>> unlock not owned lock. >>>=20 >>> Thanks, this is useful. So far I did not find any path where >>> tcp_twclose() can call INP_WUNLOCK without having the exclusive lock >>> held, that makes this issue interesting. >>>=20 >>>>> I won't go to far here as I am not expert enough in VIMAGE, but = one >>>>> question anyway: >>>>>=20 >>>>> - Can you correlate this kernel panic to a particular event? Like = for >>>>> example a VIMAGE/VNET jail destruction. >>>>>=20 >>>>> I will test that on my side on a 10.2 machine. >>>=20 >>> I did not find any issues while testing 10.2 + VIMAGE on my side. = Thus >>> Palle what I would suggest: >>>=20 >>> - First, test with stable/10 to see if by chance this issue has = already >>> been fixed in stable branch. >>>=20 >>> - Second, if issue is still in stable/10, compile 10.2 kernel with >>> these options: >>>=20 >>> options DDB >>> options DEADLKRES >>> options INVARIANTS >>> options INVARIANT_SUPPORT >>> options WITNESS >>> options WITNESS_SKIPSPIN >>>=20 >>> To see where the original fault is coming from. >>=20 >> Hi, >>=20 >> We just had two crashes within 15 minutes using 10.2 with these two = added: >>=20 >> https://svnweb.freebsd.org/changeset/base/287261 >>=20 >> https://svnweb.freebsd.org/changeset/base/287780=20 >>=20 >> We don't always get a core dump, but the second time, we did. >>=20 >> very similar stack trace, but not identical: >>=20 >> (kgdb) #0 doadump (textdump=3D) at pcpu.h:219 >> #1 0xffffffff80949a82 in kern_reboot (howto=3D260) >> at /usr/src/sys/kern/kern_shutdown.c:451 >> #2 0xffffffff80949e65 in vpanic (fmt=3D, >> ap=3D) at = /usr/src/sys/kern/kern_shutdown.c:758 >> #3 0xffffffff80949cf3 in panic (fmt=3D0x0) >> at /usr/src/sys/kern/kern_shutdown.c:687 >> #4 0xffffffff80d5d0bb in trap_fatal (frame=3D, >> eva=3D) at /usr/src/sys/amd64/amd64/trap.c:851 >> #5 0xffffffff80d5d3bd in trap_pfault (frame=3D0xfffffe1760bc1840, >> usermode=3D) at = /usr/src/sys/amd64/amd64/trap.c:674 >> #6 0xffffffff80d5ca5a in trap (frame=3D0xfffffe1760bc1840) >> at /usr/src/sys/amd64/amd64/trap.c:440 >> #7 0xffffffff80d42dd2 in calltrap () >> at /usr/src/sys/amd64/amd64/exception.S:236 >> #8 0xffffffff8099861c in turnstile_broadcast (ts=3D0x0, queue=3D1) >> at /usr/src/sys/kern/subr_turnstile.c:838 >> #9 0xffffffff80948100 in __rw_wunlock_hard (c=3D0xfffff811c43487a0, = tid=3D1, >> file=3D0x1
, line=3D1) >> at /usr/src/sys/kern/kern_rwlock.c:988 >> #10 0xffffffff80b067c4 in tcp_twclose (tw=3D, >> reuse=3D) at = /usr/src/sys/netinet/tcp_timewait.c:540 >> #11 0xffffffff80b06e0b in tcp_tw_2msl_scan (reuse=3D0) >> at /usr/src/sys/netinet/tcp_timewait.c:748 >> #12 0xffffffff80b04b0e in tcp_slowtimo () >> at /usr/src/sys/netinet/tcp_timer.c:198 >> #13 0xffffffff809b7a04 in pfslowtimo (arg=3D0x0) >> at /usr/src/sys/kern/uipc_domain.c:508 >> #14 0xffffffff8095f91b in softclock_call_cc (c=3D0xffffffff81620bf0, >> cc=3D0xffffffff8169dc00, direct=3D0) at = /usr/src/sys/kern/kern_timeout.c:685 >> #15 0xffffffff8095fd44 in softclock (arg=3D0xffffffff8169dc00) >> at /usr/src/sys/kern/kern_timeout.c:814 >> #16 0xffffffff8091592b in intr_event_execute_handlers ( >> p=3D, ie=3D0xfffff801102e0d00) >> at /usr/src/sys/kern/kern_intr.c:1264 >> #17 0xffffffff80915d76 in ithread_loop (arg=3D0xfffff801102adee0) >> at /usr/src/sys/kern/kern_intr.c:1277 >> #18 0xffffffff8091347a in fork_exit ( >> callout=3D0xffffffff80915ce0 , = arg=3D0xfffff801102adee0, >> frame=3D0xfffffe1760bc1c00) at /usr/src/sys/kern/kern_fork.c:1018 >> #19 0xffffffff80d4330e in fork_trampoline () >> at /usr/src/sys/amd64/amd64/exception.S:611 >> #20 0x0000000000000000 in ?? () >>=20 >>=20 >>=20 >> I'll try stable/10 now. Would you suggest a "clean" stable/10, or = could 287621 and 287780 help? >>=20 >> I'll add the debugging suggested options right away. >>=20 >> Palle >=20 >=20 > I have a new core dump from ^/stable/10 with: >=20 >=20 > options DDB > options DEADLKRES > options INVARIANTS > options INVARIANT_SUPPORT > options WITNESS > options WITNESS_SKIPSPIN >=20 >=20 > What can I do with the core dump? "corrupt stack"... >=20 > (kgdb) #0 doadump (textdump=3D1) at pcpu.h:219 > #1 0xffffffff8094b337 in kern_reboot (howto=3D260) > at /usr/src/sys/kern/kern_shutdown.c:451 > #2 0xffffffff8094b845 in vpanic (fmt=3D, > ap=3D) at = /usr/src/sys/kern/kern_shutdown.c:758 > #3 0xffffffff8094b6d9 in kassert_panic (fmt=3D) > at /usr/src/sys/kern/kern_shutdown.c:646 > #4 0xffffffff80b1ee59 in tcp_usr_detach (so=3D) > at /usr/src/sys/netinet/tcp_usrreq.c:202 > #5 0xffffffff809cd291 in sofree (so=3D0xfffff801dd302000) > at /usr/src/sys/kern/uipc_socket.c:747 > #6 0xffffffff809cdb00 in soclose (so=3D) > at /usr/src/sys/kern/uipc_socket.c:849 > #7 0xffffffff808fe659 in _fdrop (fp=3D0xfffff802a593db40, td=3D0x0) = at file.h:343 > #8 0xffffffff80901092 in closef (fp=3D0xfffff802a593db40, > td=3D0xfffff80eebc894a0) at /usr/src/sys/kern/kern_descrip.c:2338 > #9 0xffffffff808feb5d in closefp (fdp=3D0xfffff80b20cce000, > fd=3D, fp=3D0xfffff802a593db40, = td=3D0xfffff80eebc894a0, > holdleaders=3D) > at /usr/src/sys/kern/kern_descrip.c:1194 > #10 0xffffffff80d7bc3a in amd64_syscall (td=3D0xfffff80eebc894a0, = traced=3D0) > at subr_syscall.c:134 > #11 0xffffffff80d5f1db in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:396 > #12 0x0000000801c8d94a in ?? () > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) >=20 >=20 > Thanks, > Palle >=20 # kgdb kernel /var/crash/vmcore.2 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you = are welcome to change it and/or distribute copies of it under certain = conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for = details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: panic: tcp_detach: INP_TIMEWAIT && INP_DROPPED && tp !=3D NULL cpuid =3D 16 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame = 0xfffffe183d9e97e0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe183d9e9890 vpanic() at vpanic+0x126/frame 0xfffffe183d9e98d0 kassert_panic() at kassert_panic+0x139/frame 0xfffffe183d9e9940 tcp_usr_detach() at tcp_usr_detach+0xf9/frame 0xfffffe183d9e9970 sofree() at sofree+0x1f1/frame 0xfffffe183d9e99a0 soclose() at soclose+0x3a0/frame 0xfffffe183d9e99f0 _fdrop() at _fdrop+0x29/frame 0xfffffe183d9e9a10 closef() at closef+0x1e2/frame 0xfffffe183d9e9aa0 closefp() at closefp+0x9d/frame 0xfffffe183d9e9ae0 amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe183d9e9bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe183d9e9bf0 --- syscall (6, FreeBSD ELF64, sys_close), rip =3D 0x801c8d94a, rsp =3D = 0x7ffff91c8668, rbp =3D 0x7ffff91c8680 --- KDB: enter: panic Uptime: 18h57m59s Dumping 23085 out of 98263 = MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/nullfs.ko.symbols...done. Loaded symbols for /boot/kernel/nullfs.ko.symbols Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols Reading symbols from /boot/kernel/ng_bridge.ko.symbols...done. Loaded symbols for /boot/kernel/ng_bridge.ko.symbols Reading symbols from /boot/kernel/netgraph.ko.symbols...done. Loaded symbols for /boot/kernel/netgraph.ko.symbols Reading symbols from /boot/kernel/ng_eiface.ko.symbols...done. Loaded symbols for /boot/kernel/ng_eiface.ko.symbols Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. Loaded symbols for /boot/kernel/ng_ether.ko.symbols Reading symbols from /boot/kernel/accf_data.ko.symbols...done. Loaded symbols for /boot/kernel/accf_data.ko.symbols Reading symbols from /boot/kernel/accf_http.ko.symbols...done. Loaded symbols for /boot/kernel/accf_http.ko.symbols Reading symbols from /boot/kernel/ums.ko.symbols...done. Loaded symbols for /boot/kernel/ums.ko.symbols Reading symbols from /boot/kernel/ng_socket.ko.symbols...done. Loaded symbols for /boot/kernel/ng_socket.ko.symbols Reading symbols from /boot/kernel/fdescfs.ko.symbols...done. Loaded symbols for /boot/kernel/fdescfs.ko.symbols #0 doadump (textdump=3D1) at pcpu.h:219 219 __asm("movq %%gs:%1,%0" : "=3Dr" (td) (kgdb) bt #0 doadump (textdump=3D1) at pcpu.h:219 #1 0xffffffff8094b337 in kern_reboot (howto=3D260) at = /usr/src/sys/kern/kern_shutdown.c:451 #2 0xffffffff8094b845 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:758 #3 0xffffffff8094b6d9 in kassert_panic (fmt=3D) at = /usr/src/sys/kern/kern_shutdown.c:646 #4 0xffffffff80b1ee59 in tcp_usr_detach (so=3D) at = /usr/src/sys/netinet/tcp_usrreq.c:202 #5 0xffffffff809cd291 in sofree (so=3D0xfffff801dd302000) at = /usr/src/sys/kern/uipc_socket.c:747 #6 0xffffffff809cdb00 in soclose (so=3D) at = /usr/src/sys/kern/uipc_socket.c:849 #7 0xffffffff808fe659 in _fdrop (fp=3D0xfffff802a593db40, td=3D0x0) at = file.h:343 #8 0xffffffff80901092 in closef (fp=3D0xfffff802a593db40, = td=3D0xfffff80eebc894a0) at /usr/src/sys/kern/kern_descrip.c:2338 #9 0xffffffff808feb5d in closefp (fdp=3D0xfffff80b20cce000, fd=3D, fp=3D0xfffff802a593db40,=20 td=3D0xfffff80eebc894a0, holdleaders=3D) at = /usr/src/sys/kern/kern_descrip.c:1194 #10 0xffffffff80d7bc3a in amd64_syscall (td=3D0xfffff80eebc894a0, = traced=3D0) at subr_syscall.c:134 #11 0xffffffff80d5f1db in Xfast_syscall () at = /usr/src/sys/amd64/amd64/exception.S:396 #12 0x0000000801c8d94a in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) f 8 #8 0xffffffff80901092 in closef (fp=3D0xfffff802a593db40, = td=3D0xfffff80eebc894a0) at /usr/src/sys/kern/kern_descrip.c:2338 2338 return (fdrop(fp, td)); (kgdb) help=20 List of classes of commands: aliases -- Aliases of other commands breakpoints -- Making program stop at certain points data -- Examining data files -- Specifying and examining files internals -- Maintenance commands obscure -- Obscure features running -- Running the program stack -- Examining the stack status -- Status inquiries support -- Support facilities tracepoints -- Tracing of program execution without stopping the program user-defined -- User-defined commands Type "help" followed by a class name for a list of commands in that = class. Type "help" followed by command name for full documentation. Command name abbreviations are allowed if unambiguous. (kgdb) disassemble Dump of assembler code for function closef: 0xffffffff80900eb0 : push %rbp 0xffffffff80900eb1 : mov %rsp,%rbp 0xffffffff80900eb4 : push %r15 0xffffffff80900eb6 : push %r14 0xffffffff80900eb8 : push %r13 0xffffffff80900eba : push %r12 0xffffffff80900ebc : push %rbx 0xffffffff80900ebd : sub $0x58,%rsp 0xffffffff80900ec1 : mov %rsi,%r12 0xffffffff80900ec4 : mov %rdi,%r14 0xffffffff80900ec7 : cmpw $0x1,0x20(%r14) 0xffffffff80900ecd : jne 0xffffffff80901077 0xffffffff80900ed3 : test %r12,%r12 0xffffffff80900ed6 : je 0xffffffff80901077 0xffffffff80900edc : mov 0x8(%r12),%rax 0xffffffff80900ee1 : mov 0x428(%rax),%rcx 0xffffffff80900ee8 : testb $0x1,0xb0(%rcx) 0xffffffff80900eef : je 0xffffffff80900f50 0xffffffff80900ef1 : mov 0x18(%r14),%rcx 0xffffffff80900ef5 : movw $0x0,-0x62(%rbp) 0xffffffff80900efb : movq $0x0,-0x78(%rbp) 0xffffffff80900f03 : movq $0x0,-0x70(%rbp) 0xffffffff80900f0b : movw $0x2,-0x64(%rbp) 0xffffffff80900f11 : mov 0x428(%rax),%rax 0xffffffff80900f18 : movq = $0xffffffff81557f68,-0x58(%rbp) 0xffffffff80900f20 : mov %rcx,-0x50(%rbp) 0xffffffff80900f24 : mov %rax,-0x48(%rbp) 0xffffffff80900f28 : movl $0x2,-0x40(%rbp) 0xffffffff80900f2f : lea -0x78(%rbp),%rax 0xffffffff80900f33 : mov %rax,-0x38(%rbp) 0xffffffff80900f37 : movl $0x40,-0x30(%rbp) 0xffffffff80900f3e : mov 0x8(%rcx),%rdi 0xffffffff80900f42 : lea -0x58(%rbp),%rsi 0xffffffff80900f46 : callq 0xffffffff80ea8870 = 0xffffffff80900f4b : mov 0x8(%r12),%rax 0xffffffff80900f50 : mov 0x50(%rax),%rbx 0xffffffff80900f54 : test %rbx,%rbx 0xffffffff80900f57 : je 0xffffffff80901077 = 0xffffffff80900f5d : mov 0x48(%rax),%r15 0xffffffff80900f61 : add $0x40,%r15 0xffffffff80900f65 : xor %esi,%esi 0xffffffff80900f67 : mov $0xffffffff810042e9,%rdx 0xffffffff80900f6e : mov $0x906,%ecx 0xffffffff80900f73 : mov %r15,%rdi 0xffffffff80900f76 : callq 0xffffffff80952ba0 = <_sx_xlock> 0xffffffff80900f7b : mov 0x20(%rbx),%rbx 0xffffffff80900f7f : mov 0x8(%r12),%rax 0xffffffff80900f84 : cmp 0x50(%rax),%rbx ---Type to continue, or q to quit--- 0xffffffff80900f88 : je 0xffffffff80901063 = 0xffffffff80900f8e : lea -0x58(%rbp),%r13 0xffffffff80900f92 : nopw %cs:0x0(%rax,%rax,1) 0xffffffff80900fa0 : mov 0x10(%rbx),%rax 0xffffffff80900fa4 : testb $0x1,0xb0(%rax) 0xffffffff80900fab : je 0xffffffff80901050 = 0xffffffff80900fb1 : incl 0x4(%rbx) 0xffffffff80900fb4 : mov $0xffffffff810042e9,%rsi 0xffffffff80900fbb : mov $0x90e,%edx 0xffffffff80900fc0 : mov %r15,%rdi 0xffffffff80900fc3 : callq 0xffffffff80952f90 = <_sx_xunlock> 0xffffffff80900fc8 : movw $0x0,-0x62(%rbp) 0xffffffff80900fce : movq $0x0,-0x78(%rbp) 0xffffffff80900fd6 : movq $0x0,-0x70(%rbp) 0xffffffff80900fde : movw $0x2,-0x64(%rbp) 0xffffffff80900fe4 : mov 0x18(%r14),%rax 0xffffffff80900fe8 : mov 0x10(%rbx),%rcx 0xffffffff80900fec : movq = $0xffffffff81557f68,-0x58(%rbp) 0xffffffff80900ff4 : mov %rax,-0x50(%rbp) 0xffffffff80900ff8 : mov %rcx,-0x48(%rbp) 0xffffffff80900ffc : movl $0x2,-0x40(%rbp) 0xffffffff80901003 : lea -0x78(%rbp),%rcx 0xffffffff80901007 : mov %rcx,-0x38(%rbp) 0xffffffff8090100b : movl $0x40,-0x30(%rbp) 0xffffffff80901012 : mov 0x8(%rax),%rdi 0xffffffff80901016 : mov %r13,%rsi 0xffffffff80901019 : callq 0xffffffff80ea8870 = 0xffffffff8090101e : xor %esi,%esi 0xffffffff80901020 : mov $0xffffffff810042e9,%rdx 0xffffffff80901027 : mov $0x917,%ecx 0xffffffff8090102c : mov %r15,%rdi 0xffffffff8090102f : callq 0xffffffff80952ba0 = <_sx_xlock> 0xffffffff80901034 : decl 0x4(%rbx) 0xffffffff80901037 : jne 0xffffffff80901050 = 0xffffffff80901039 : cmpl $0x0,0x8(%rbx) 0xffffffff8090103d : je 0xffffffff80901050 = 0xffffffff8090103f : movl $0x0,0x8(%rbx) 0xffffffff80901046 : mov %rbx,%rdi 0xffffffff80901049 : callq 0xffffffff80954a40 = 0xffffffff8090104e : xchg %ax,%ax 0xffffffff80901050 : mov 0x20(%rbx),%rbx 0xffffffff80901054 : mov 0x8(%r12),%rax 0xffffffff80901059 : cmp 0x50(%rax),%rbx 0xffffffff8090105d : jne 0xffffffff80900fa0 = 0xffffffff80901063 : mov $0xffffffff810042e9,%rsi 0xffffffff8090106a : mov $0x91f,%edx 0xffffffff8090106f : mov %r15,%rdi 0xffffffff80901072 : callq 0xffffffff80952f90 = <_sx_xunlock> 0xffffffff80901077 : mov $0xffffffff,%eax ---Type to continue, or q to quit--- 0xffffffff8090107c : lock xadd %eax,0x28(%r14) 0xffffffff80901082 : cmp $0x1,%eax 0xffffffff80901085 : jne 0xffffffff809010a5 = 0xffffffff80901087 : mov %r14,%rdi 0xffffffff8090108a : mov %r12,%rsi 0xffffffff8090108d : callq 0xffffffff808fe630 = <_fdrop> 0xffffffff80901092 : mov %eax,%ebx 0xffffffff80901094 : mov %ebx,%eax 0xffffffff80901096 : add $0x58,%rsp 0xffffffff8090109a : pop %rbx 0xffffffff8090109b : pop %r12 0xffffffff8090109d : pop %r13 0xffffffff8090109f : pop %r14 0xffffffff809010a1 : pop %r15 0xffffffff809010a3 : pop %rbp 0xffffffff809010a4 : retq =20 0xffffffff809010a5 : xor %ebx,%ebx 0xffffffff809010a7 : test %eax,%eax 0xffffffff809010a9 : jne 0xffffffff80901094 = 0xffffffff809010ab : add $0x28,%r14 0xffffffff809010af : xor %ebx,%ebx 0xffffffff809010b1 : mov $0xffffffff80ebcddb,%rdi 0xffffffff809010b8 : xor %eax,%eax 0xffffffff809010ba : mov %r14,%rsi 0xffffffff809010bd : callq 0xffffffff8094b5a0 = 0xffffffff809010c2 : jmp 0xffffffff80901094 = End of assembler dump.