From owner-freebsd-current@FreeBSD.ORG Sat Oct 22 10:46:18 2005 Return-Path: X-Original-To: current@freebsd.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 00F0C16A420 for ; Sat, 22 Oct 2005 10:46:18 +0000 (GMT) (envelope-from shoesoft@gmx.net) Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by mx1.FreeBSD.org (Postfix) with SMTP id ED50343D45 for ; Sat, 22 Oct 2005 10:46:16 +0000 (GMT) (envelope-from shoesoft@gmx.net) Received: (qmail invoked by alias); 22 Oct 2005 10:46:14 -0000 Received: from h081217094062.dyn.cm.kabsi.at (EHLO h081217094062.dyn.cm.kabsi.at) [81.217.94.62] by mail.gmx.net (mp013) with SMTP; 22 Oct 2005 12:46:14 +0200 X-Authenticated: #16703784 From: Stefan Ehmann To: Kris Kennaway In-Reply-To: <20051021072448.GA15130@xor.obsecurity.org> References: <1129879049.771.6.camel@localhost> <20051021072448.GA15130@xor.obsecurity.org> Content-Type: multipart/mixed; boundary="=-iiNGWelgi3oh58Bxq97i" Date: Sat, 22 Oct 2005 12:46:14 +0200 Message-Id: <1129977974.987.19.camel@taxman.pepperland> Mime-Version: 1.0 X-Mailer: Evolution 2.2.3 FreeBSD GNOME Team Port X-Y-GMX-Trusted: 0 Cc: current@freebsd.org Subject: Re: PREEMPTION still unusable with 6.0-RC1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Oct 2005 10:46:18 -0000 --=-iiNGWelgi3oh58Bxq97i Content-Type: text/plain Content-Transfer-Encoding: 7bit On Fri, 2005-10-21 at 03:24 -0400, Kris Kennaway wrote: > On Fri, Oct 21, 2005 at 09:17:29AM +0200, Stefan Ehmann wrote: > > I recently upgraded my 5.4-RELEASE machine to 6.0-RC1. Runs fine so far > > if I disable PREEMPTION. > > > > With PREEMPTION enabled, I still get the same issues as described here: > > http://lists.freebsd.org/pipermail/freebsd-current/2004-September/037949.html > > > > kernel config: > > http://stud4.tuwien.ac.at/~e0125637/fbsd/kernconf-6.0-RC1 > > > > /var/log/messages output: > > http://stud4.tuwien.ac.at/~e0125637/fbsd/messages-6.0-RC1 > > > > Any one else still experiencing these problems or has any idea what > > causes this? > > Turn on WITNESS, INVARIANTS and see if they report anything. Break to > DDB and use the various debugging commands (ps, the various show > commands, ...) to trace things. I should have thought of that myself... Anyway, I might have found the LOR that causes the problem [1]. show allocks output can be seen in [2] It's similar to http://sources.zabbadoz.net/freebsd/lor/016.html (as it also happens in check_uidgid(). There's a nice explaination by rwatson in the linked thread. So I removed the corresponding ipfw rule (I use it together with dummynet for traffic shaping for a spedific user). It seems to run fine so far. If I re-add the rule the system hangs within minutes. I still have to test if the system runs stable over a longer period. I also got another LOR [3] but it hasn't caused any problems so far. --=-iiNGWelgi3oh58Bxq97i Content-Disposition: attachment; filename=lor.txt Content-Type: text/plain; name=lor.txt; charset=ISO8859-1 Content-Transfer-Encoding: 7bit [1] lock order reversal 1st 0xc1cef090 inp (divinp) @ /usr/src/sys/netinet/ip_divert.c:327 2nd 0xc079794c tcp (tcp) @ /usr/src/sys/netinet/ip_fw2.c:1952 KDB: stack backtrace: kdb_backtrace(c06ef8e3,c079794c,c06ef3b7,c06ef3b7,c06f7f50) at kdb_backtrace+0x2e witness_checkorder(c079794c,9,c06f7f50,7a0,c04f7ce4) at witness_checkorder+0x6c3 _mtx_lock_flags(c079794c,0,c06f7f50,7a0,c0548199) at _mtx_lock_flags+0x8a check_uidgid(c1c59ac4,6,c1a6fc00,1809bccd,1446) at check_uidgid+0x111 ipfw_chk(dadf89ec,41ec0d7e,0,0,c1b78000) at ipfw_chk+0xd86 ipfw_check_out(0,dadf8ae4,c1a6fc00,2,0) at ipfw_check_out+0x10c pfil_run_hooks(c07974e0,dadf8b58,c1a6fc00,2,0) at pfil_run_hooks+0x101 ip_output(c1b78000,0,dadf8b24,22,0) at ip_output+0x6f0 div_output(c1c66de8,c1b78000,c1bb7720,0,dadf8c00) at div_output+0x1d3 div_send(c1c66de8,0,c1b78000,c1bb7720,0) at div_send+0x5d sosend(c1c66de8,c1bb7720,dadf8c34,c1b78000,0) at sosend+0x6e1 kern_sendit(c1bd6180,3,dadf8cb4,0,0) at kern_sendit+0x12f sendit(c1bd6180,3,dadf8cb4,0,bfbdebc8) at sendit+0x1ab sendto(c1bd6180,dadf8d04,18,418,6) at sendto+0x5b syscall(3b,3b,3b,bfbdeba0,2) at syscall+0x2a2 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (133, FreeBSD ELF32, sendto), eip = 0x280d0c3f, esp = 0xbfbdeb0c, ebp = 0xbfbeebb8 --- [2] Process 962 (fetchmail) thread 0xc1bd7480 (100094) exclusive sleep mutex inp (tcpinp) r = 0 (0xc20aed38) locked @ /usr/src/sys/netinet/tcp_usrreq.c:646 Process 300 (natd) thread 0xc1bd6180 (100074) exclusive sleep mutex tcp r = 0 (0xc079794c) locked @ /usr/src/sys/netinet/ip_fw2.c:1952 exclusive sleep mutex inp (divinp) r = 0 (0xc1cef090) locked @ /usr/src/sys/netinet/ip_divert.c:327 exclusive sleep mutex div r = 0 (0xc079652c) locked @ /usr/src/sys/netinet/ip_divert.c:325 Process 12 (irq1: atkbd0) thread 0xc198a900 (100004) exclusive sleep mutex Giant r = 0 (0xc0749420) locked @ /usr/src/sys/kern/kern_intr.c:546 [3] lock order reversal 1st 0xc1d10090 inp (divinp) @ /usr/src/sys/netinet/ip_divert.c:327 2nd 0xc0796440 in_multi_mtx (in_multi_mtx) @ /usr/src/sys/netinet/ip_output.c:295 KDB: stack backtrace: kdb_backtrace(c06ef8e3,c0796440,c06ef386,c06ef386,c06f8d28) at kdb_backtrace+0x2e witness_checkorder(c0796440,9,c06f8d28,127,c06f6927) at witness_checkorder+0x6c3 _mtx_lock_flags(c0796440,0,c06f8d28,127,c052d396) at _mtx_lock_flags+0x8a ip_output(c1ca3c00,0,dae0ab24,22,0) at ip_output+0x46c div_output(c1c8bde8,c1ca3c00,c2e4c530,0,dae0ac00) at div_output+0x1d3 div_send(c1c8bde8,0,c1ca3c00,c2e4c530,0) at div_send+0x5d sosend(c1c8bde8,c2e4c530,dae0ac34,c1ca3c00,0) at sosend+0x6e1 kern_sendit(c1bd6a80,3,dae0acb4,0,0) at kern_sendit+0x12f sendit(c1bd6a80,3,dae0acb4,0,bfbdebc0) at sendit+0x1ab sendto(c1bd6a80,dae0ad04,18,418,6) at sendto+0x5b syscall(3b,3b,3b,bfbdeba0,2) at syscall+0x2a2 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (133, FreeBSD ELF32, sendto), eip = 0x280d0c3f, esp = 0xbfbdeb0c, ebp = 0xbfbeebb8 --- --=-iiNGWelgi3oh58Bxq97i--