From owner-freebsd-current@FreeBSD.ORG Sat Aug 16 14:25:17 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4922F37B401 for ; Sat, 16 Aug 2003 14:25:17 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 051F243F93 for ; Sat, 16 Aug 2003 14:25:13 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.9/8.12.9) with ESMTP id h7GLOkM7024098; Sat, 16 Aug 2003 14:24:49 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <200308162124.h7GLOkM7024098@gw.catspoiler.org> Date: Sat, 16 Aug 2003 14:24:46 -0700 (PDT) From: Don Lewis To: tilman@arved.de In-Reply-To: <20030816132109.GA80431@huckfinn.arved.de> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii cc: freebsd-current@FreeBSD.org Subject: Re: LOR tcp_input.c vs. tcp_usrreq.c (was: Re: 2 LORs on my NFS server.) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Aug 2003 21:25:17 -0000 On 16 Aug, Tilman Linneweh wrote: > * Tilman Linneweh [Fr, 15 Aug 2003 at 16:17 GMT]: >> >> My CURRENT is already a bit old: >> >> # uname -a >> FreeBSD polly.arved.de 5.1-CURRENT FreeBSD 5.1-CURRENT #1: Sun Jul 20 >> 01:00:14 CEST 2003 >> tilman@sauna.arved.de:/usr/obj/usr/src/CURRENT/sys/POLLY i386 > > I updated my CURRENT to > > polly# uname -a > FreeBSD polly.arved.de 5.1-CURRENT FreeBSD 5.1-CURRENT #1: Sat Aug 16 > 10:11:52 CEST 2003 > tilman@sauna.arved.de:/usr/obj/usr/source/CURRENT/sys/POLLY i386 > > and this LOR is reproducable. > >> This happend while the machine was NFS-serving around 3 clients with >> normal udp NFS and a fourth. client tried to mount something via >> mount_nfs -T -a 2 > > The problem is the client with TCP mounts. I tried this time with a single > NetBSD client that does a TCP mount and cd'd to the mounted directory. > > lock order reversal > 1st 0xc1a17278 inp (inp) @ /usr/source/CURRENT/sys/netinet/tcp_input.c:654 > 2nd 0xc046bd6c tcp (tcp) @ /usr/source/CURRENT/sys/netinet/tcp_usrreq.c:621 > Stack backtrace: > backtrace(1,0,ffffffff,c0445068,c04451d0) at backtrace+0x12 > witness_lock(c046bd6c,8,c03c334c,26d,0) at witness_lock+0x55e > _mtx_lock_flags(c046bd6c,0,c03c334c,26d) at _mtx_lock_flags+0x7d > tcp_usr_rcvd(c1ce8800,80) at tcp_usr_rcvd+0x1b > soreceive(c1ce8800,c891ab1c,c891ab28,c891ab20,0) at soreceive+0x815 > nfsrv_rcv(c1ce8800,c1a70780,4) at nfsrv_rcv+0x75 > sowakeup(c1ce8800,c1ce884c) at sowakeup+0x7f > tcp_input(c0b9ac00,14) at tcp_input+0x11f6 > ip_input(c0b9ac00) at ip_input+0x7c8 > swi_net(0) at swi_net+0xe6 > ithread_loop(c0b87180,c891ad48,c0b87180,c0221660,0) at ithread_loop+0x11c > fork_exit(c0221660,c0b87180,c891ad48) at fork_exit+0xab > fork_trampoline() at fork_trampoline+0x8 > --- trap 0x1, eip = 0, esp = 0xc891ad7c, ebp = 0 --- > Debugger("witness_lock") > Stopped at Debugger+0x45: xchgl %ebx,in_Debugger.0 > This is a known issue. ------ Forwarded message ------ From: Don Lewis Subject: Re: LOR in NFS server Date: Thu, 24 Apr 2003 21:20:56 -0700 (PDT) To: gordont@gnf.org Cc: current@freebsd.org On 24 Apr, Gordon Tetlow wrote: > I generated it while running nessus against my local machine. > > lock order reversal > 1st 0xc9384c44 inp (inp) @ /local/usr.src/sys/netinet/tcp_input.c:649 > 2nd 0xc05aa84c tcp (tcp) @ /local/usr.src/sys/netinet/tcp_usrreq.c:621 > Stack backtrace: > backtrace(c04e9f03,c05aa84c,c04f0770,c04f0770,c04f1ae4) at backtrace+0x17 > witness_lock(c05aa84c,8,c04f1ae4,26d,0) at witness_lock+0x692 > _mtx_lock_flags(c05aa84c,0,c04f1ae4,26d,0) at _mtx_lock_flags+0xb2 > tcp_usr_rcvd(c8a63800,80,c04ea514,df0e9a9c,3b9aca00) at tcp_usr_rcvd+0x30 > soreceive(c8a63800,df0e9ad8,df0e9ae4,df0e9adc,0) at soreceive+0x86a > nfsrv_rcv(c8a63800,c6d4fb00,4,34,10430) at nfsrv_rcv+0x8a > sowakeup(c8a63800,c8a6384c,c04f11d5,434,108) at sowakeup+0x97 > tcp_input(c21f5400,14,c0304f91,df0e9c5c,c02f60ba) at tcp_input+0x1341 > ip_input(c21f5400,0,c04efede,e9,c21bd280) at ip_input+0x7b0 > swi_net(0,0,c04e4eed,217,c21c73c0) at swi_net+0x111 > ithread_loop(c21c6100,df0e9d48,c04e4d5d,314,c21c8d10) at ithread_loop+0x16c > fork_exit(c02ec2d0,c21c6100,df0e9d48) at fork_exit+0xc0 > fork_trampoline() at fork_trampoline+0x1a > --- trap 0x1, eip = 0, esp = 0xdf0e9d7c, ebp = 0 --- Hmn ... does NFS over TCP even work with a -current box as the server? It looks like tcp_input() has grabbed the locks in tcbinfo and inp, and then tcp_usr_rcvd() attempts to grab the same locks. I can think of three possible ways of fixing this problem. 1) Drop the locks in tcp_input() before calling sorwakeup() and grab them again if necessary. One has to be careful not to break anything by doing this. This also adds overhead for non-NFS traffic. 2) Never call soreceive() from nfsrv_rcv(), always wake nfsd instead. This has the advantage of minimizing the amount of time that the locks are held, but increases overhead under lightly loaded conditions. 3) Somehow tell tcp_usr_rcvd() not to attempt to grab the locks in this specific case. ------ End forwarded message ------ ------ Forwarded message ------ From: Jeffrey Hsu Subject: Re: LOR in NFS server Date: Fri, 25 Apr 2003 01:02:56 -0700 To: gordont@gnf.org Cc: current@freebsd.org > 1st 0xc9384c44 inp (inp) @ /local/usr.src/sys/netinet/tcp_input.c:649 > 2nd 0xc05aa84c tcp (tcp) @ /local/usr.src/sys/netinet/tcp_usrreq.c:621 This old nag warning has been there since last year and was first reported by Lars Eggert . I made up a fix for him at the time which you can find at http://www.freebsd.org/~hsu/hammer.diff. Lars has verified that this eliminates the nag warnings. But, I'm hoping to have a more unified solution to the general sowakeup problem, so have not committed this patch. Jeffrey ------ End forwarded message ------