Date: Sat, 27 Aug 2005 18:20:24 +0100 (BST) From: Robert Watson <rwatson@FreeBSD.org> To: "M. Warner Losh" <imp@bsdimp.com> Cc: bzeeb-lists@lists.zabbadoz.net, freebsd-current@freebsd.org, dandee@volny.cz Subject: Re: LOR route vr0 Message-ID: <20050827181827.O24510@fledge.watson.org> In-Reply-To: <20050827.104631.10908351.imp@bsdimp.com> References: <20050826115503.D4B0C4E704@pipa.profix.cz> <Pine.BSF.4.53.0508270912550.969@e0-0.zab2.int.zabbadoz.net> <20050827.104631.10908351.imp@bsdimp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 27 Aug 2005, M. Warner Losh wrote: > In message: <Pine.BSF.4.53.0508270912550.969@e0-0.zab2.int.zabbadoz.net> > "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> writes: > : > lock order reversal > : > 1st 0xc17621ec rtentry (rtentry) @ /usr/src/sys/net/route.c:1269 > : > 2nd 0xc15ec938 vr0 (network driver) @ /usr/src/sys/pci/if_vr.c:1391 > : > : added with ID 140: http://sources.zabbadoz.net/freebsd/lor.html#140 > > I've noticed a *HUGE* number of LORs that look like this: > > ock order reversal > 1st 0xc17490e4 rtentry (rtentry) @ sys/netinet/if_ether.c:445 > 2nd 0xc15c94b0 rl1 (network driver) @ sys/pci/if_rl.c:1451 Generally speaking, network interface device driver locks follow network stack locks in the lock order. However, I've not really looked much at the route table locking so can't speak to whether that is the case specifically for routing locks. If it is, the below traces reflect the correct order, and you might want to add a hard-coded entry to witness in order to catch the reverse order. Lock order reversals between the network stack and device drivers tend to occur as a result of the device driver calling into the network stack while holding the device driver mutex. Someone (tm) should work out if the right order is route locks -> device driver locks, as it's likely a common calss of bugs across many drivers. Robert N M Watson > KDB: stack backtrace: > kdb_backtrace(c07dcab1,c15c94b0,c160a1b0,c07c54d7,c07f401e) at kdb_backtrace+0x2e > witness_checkorder(c15c94b0,9,c07f401e,5ab,c07e32bd) at witness_checkorder+0x6c3 > _mtx_lock_flags(c15c94b0,0,c07f401e,5ab,c152cc00) at _mtx_lock_flags+0x8a > rl_start(c152cc00,1,c07e2eae,836) at rl_start+0x37 > if_start(c152cc00,0,c07e32bd,195,202) at if_start+0x99 > ether_output_frame(c152cc00,c169c100,6,c0562c28,c169c100) > at ether_output_frame+0x218 > ether_output(c152cc00,c169c100,cbfe79f0,0,2,c1740001,2302,c07e66e8,1bd,519) > at ether_output+0x47e > arprequest(c152cc00,c16cfcc8,cbfe7ae4,c15fa6ab,c05998a6) at arprequest+0x109 > arpresolve(c152cc00,c1749084,c169a400,cbfe7ae0,cbfe7a64) at arpresolve+0x32d > ether_output(c152cc00,c169a400,cbfe7ae0,c1749084,0) at ether_output+0x7b > ip_output(c169a400,0,cbfe7adc,0,0) at ip_output+0xb7a > > and am seeing the following in my newly locked ed driver: > > lock order reversal > 1st 0xc1cb3588 rtentry (rtentry) @ net/route.c:1269 > 2nd 0xc1fd3420 ed1 (network driver) @ /dell/imp/p4/newcard/src/sys/modules/ed/../../dev/ed/if_ed.c:697 > KDB: stack backtrace: > kdb_backtrace(0,ffffffff,c0680950,c067f5a0,c064bd44) at kdb_backtrace+0x29 > witness_checkorder(c1fd3420,9,c201ff8b,2b9) at witness_checkorder+0x52c > _mtx_lock_flags(c1fd3420,0,c201ff8b,2b9,c1a86c00) at _mtx_lock_flags+0x5b > ed_start(c1a86c00) at ed_start+0x1f > if_start(c1a86c00) at if_start+0x7b > ether_output_frame(c1a86c00,c1bbeb00,c04c0920,ffffffff,0) at ether_output_frame+0x1dc > ether_output(c1a86c00,c1bbeb00,e5832a38,0,2) at ether_output+0x3e4 > arprequest(c1a86c00,c1d77ac8,e5832b08,c20236ab) at arprequest+0xd8 > arpresolve(c1a86c00,c1cb3528,c1bbed00,e5832b04,e5832aa8) at arpresolve+0x30b > ether_output(c1a86c00,c1bbed00,e5832b04,c1cb3528,c1d77a00) at ether_output+0x6b > ip_output(c1bbed00,0,e5832b00,0,0) at ip_output+0x78c > udp_output(c1cb1168,c1bbed00,0,0,c1a8d600) at udp_output+0x4a7 > udp_send(c1d59c84,0,c1bbed00,0,0) at udp_send+0x1a > sosend(c1d59c84,0,e5832c3c,c1bbed00,0) at sosend+0x5e3 > kern_sendit(c1a8d600,4,e5832cbc,0,0) at kern_sendit+0x104 > sendit(c1a8d600,4,e5832cbc,0,807a023) at sendit+0x163 > sendto(c1a8d600,e5832d04,6,0,206) at sendto+0x4d > syscall(3b,3b,3b,805a000,28219fa4) at syscall+0x22f > > and was wondering if there's a common cause. > > Warner > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050827181827.O24510>