Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 22 Oct 2005 12:46:14 +0200
From:      Stefan Ehmann <shoesoft@gmx.net>
To:        Kris Kennaway <kris@obsecurity.org>
Cc:        current@freebsd.org
Subject:   Re: PREEMPTION still unusable with 6.0-RC1
Message-ID:  <1129977974.987.19.camel@taxman.pepperland>
In-Reply-To: <20051021072448.GA15130@xor.obsecurity.org>
References:  <1129879049.771.6.camel@localhost> <20051021072448.GA15130@xor.obsecurity.org>

next in thread | previous in thread | raw e-mail | index | archive | help

--=-iiNGWelgi3oh58Bxq97i
Content-Type: text/plain
Content-Transfer-Encoding: 7bit

On Fri, 2005-10-21 at 03:24 -0400, Kris Kennaway wrote:
> On Fri, Oct 21, 2005 at 09:17:29AM +0200, Stefan Ehmann wrote:
> > I recently upgraded my 5.4-RELEASE machine to 6.0-RC1. Runs fine so far
> > if I disable PREEMPTION.
> > 
> > With PREEMPTION enabled, I still get the same issues as described here:
> > http://lists.freebsd.org/pipermail/freebsd-current/2004-September/037949.html
> > 
> > kernel config:
> > http://stud4.tuwien.ac.at/~e0125637/fbsd/kernconf-6.0-RC1
> > 
> > /var/log/messages output:
> > http://stud4.tuwien.ac.at/~e0125637/fbsd/messages-6.0-RC1
> > 
> > Any one else still experiencing these problems or has any idea what
> > causes this?
> 
> Turn on WITNESS, INVARIANTS and see if they report anything.  Break to
> DDB and use the various debugging commands (ps, the various show
> commands, ...) to trace things.

I should have thought of that myself...

Anyway, I might have found the LOR that causes the problem [1]. show
allocks output can be seen in [2]

It's similar to http://sources.zabbadoz.net/freebsd/lor/016.html (as it
also happens in check_uidgid(). There's a nice explaination by rwatson
in the linked thread.

So I removed the corresponding ipfw rule (I use it together with
dummynet for traffic shaping for a spedific user). It seems to run fine
so far. If I re-add the rule the system hangs within minutes. I still
have to test if the system runs stable over a longer period.

I also got another LOR [3] but it hasn't caused any problems so far.

--=-iiNGWelgi3oh58Bxq97i
Content-Disposition: attachment; filename=lor.txt
Content-Type: text/plain; name=lor.txt; charset=ISO8859-1
Content-Transfer-Encoding: 7bit

[1]
lock order reversal
 1st 0xc1cef090 inp (divinp) @ /usr/src/sys/netinet/ip_divert.c:327
 2nd 0xc079794c tcp (tcp) @ /usr/src/sys/netinet/ip_fw2.c:1952
KDB: stack backtrace:
kdb_backtrace(c06ef8e3,c079794c,c06ef3b7,c06ef3b7,c06f7f50) at kdb_backtrace+0x2e
witness_checkorder(c079794c,9,c06f7f50,7a0,c04f7ce4) at witness_checkorder+0x6c3
_mtx_lock_flags(c079794c,0,c06f7f50,7a0,c0548199) at _mtx_lock_flags+0x8a
check_uidgid(c1c59ac4,6,c1a6fc00,1809bccd,1446) at check_uidgid+0x111
ipfw_chk(dadf89ec,41ec0d7e,0,0,c1b78000) at ipfw_chk+0xd86
ipfw_check_out(0,dadf8ae4,c1a6fc00,2,0) at ipfw_check_out+0x10c
pfil_run_hooks(c07974e0,dadf8b58,c1a6fc00,2,0) at pfil_run_hooks+0x101
ip_output(c1b78000,0,dadf8b24,22,0) at ip_output+0x6f0
div_output(c1c66de8,c1b78000,c1bb7720,0,dadf8c00) at div_output+0x1d3
div_send(c1c66de8,0,c1b78000,c1bb7720,0) at div_send+0x5d
sosend(c1c66de8,c1bb7720,dadf8c34,c1b78000,0) at sosend+0x6e1
kern_sendit(c1bd6180,3,dadf8cb4,0,0) at kern_sendit+0x12f
sendit(c1bd6180,3,dadf8cb4,0,bfbdebc8) at sendit+0x1ab
sendto(c1bd6180,dadf8d04,18,418,6) at sendto+0x5b
syscall(3b,3b,3b,bfbdeba0,2) at syscall+0x2a2
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (133, FreeBSD ELF32, sendto), eip = 0x280d0c3f, esp = 0xbfbdeb0c, ebp = 0xbfbeebb8 ---

[2]
Process 962 (fetchmail) thread 0xc1bd7480 (100094)
exclusive sleep mutex inp (tcpinp) r = 0 (0xc20aed38) locked @ /usr/src/sys/netinet/tcp_usrreq.c:646
Process 300 (natd) thread 0xc1bd6180 (100074)
exclusive sleep mutex tcp r = 0 (0xc079794c) locked @ /usr/src/sys/netinet/ip_fw2.c:1952
exclusive sleep mutex inp (divinp) r = 0 (0xc1cef090) locked @ /usr/src/sys/netinet/ip_divert.c:327
exclusive sleep mutex div r = 0 (0xc079652c) locked @ /usr/src/sys/netinet/ip_divert.c:325
Process 12 (irq1: atkbd0) thread 0xc198a900 (100004)
exclusive sleep mutex Giant r = 0 (0xc0749420) locked @ /usr/src/sys/kern/kern_intr.c:546

[3]
lock order reversal
 1st 0xc1d10090 inp (divinp) @ /usr/src/sys/netinet/ip_divert.c:327
 2nd 0xc0796440 in_multi_mtx (in_multi_mtx) @ /usr/src/sys/netinet/ip_output.c:295
KDB: stack backtrace:
kdb_backtrace(c06ef8e3,c0796440,c06ef386,c06ef386,c06f8d28) at kdb_backtrace+0x2e
witness_checkorder(c0796440,9,c06f8d28,127,c06f6927) at witness_checkorder+0x6c3
_mtx_lock_flags(c0796440,0,c06f8d28,127,c052d396) at _mtx_lock_flags+0x8a
ip_output(c1ca3c00,0,dae0ab24,22,0) at ip_output+0x46c
div_output(c1c8bde8,c1ca3c00,c2e4c530,0,dae0ac00) at div_output+0x1d3
div_send(c1c8bde8,0,c1ca3c00,c2e4c530,0) at div_send+0x5d
sosend(c1c8bde8,c2e4c530,dae0ac34,c1ca3c00,0) at sosend+0x6e1
kern_sendit(c1bd6a80,3,dae0acb4,0,0) at kern_sendit+0x12f
sendit(c1bd6a80,3,dae0acb4,0,bfbdebc0) at sendit+0x1ab
sendto(c1bd6a80,dae0ad04,18,418,6) at sendto+0x5b
syscall(3b,3b,3b,bfbdeba0,2) at syscall+0x2a2
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (133, FreeBSD ELF32, sendto), eip = 0x280d0c3f, esp = 0xbfbdeb0c, ebp = 0xbfbeebb8 ---

--=-iiNGWelgi3oh58Bxq97i--




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1129977974.987.19.camel>