From owner-freebsd-current@FreeBSD.ORG Fri Sep 10 02:50:50 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA3EE1065672; Fri, 10 Sep 2010 02:50:50 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 5F4C08FC17; Fri, 10 Sep 2010 02:50:50 +0000 (UTC) Received: from bigwig.baldwin.cx (unknown [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 1511646B17; Thu, 9 Sep 2010 14:33:56 -0400 (EDT) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 8CED18A03C; Thu, 9 Sep 2010 14:32:52 -0400 (EDT) From: John Baldwin To: Weongyo Jeong Date: Thu, 9 Sep 2010 14:32:52 -0400 User-Agent: KMail/1.13.5 (FreeBSD/7.3-CBSD-20100819; KDE/4.4.5; amd64; ; ) References: <20100908201419.GF1328@weongyo> <201009090936.19412.jhb@freebsd.org> <20100909174146.GG1328@weongyo> In-Reply-To: <20100909174146.GG1328@weongyo> MIME-Version: 1.0 Message-Id: <201009091432.52066.jhb@freebsd.org> Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Thu, 09 Sep 2010 14:32:52 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=4.2 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: freebsd-current@freebsd.org Subject: Re: about in_multi_mtx @ netinet/in_mcast.c:1095 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 02:50:50 -0000 On Thursday, September 09, 2010 1:41:46 pm Weongyo Jeong wrote: > On Thu, Sep 09, 2010 at 09:36:19AM -0400, John Baldwin wrote: > > On Wednesday, September 08, 2010 4:14:19 pm Weongyo Jeong wrote: > > > Hello, > > > > > > I have a question about IN_MULTI_LOCK() because it uses MTX_DEF flag > > > when it's initialized so I always encounters the following LOR > > > > > > lock order reversal: (sleepable after non-sleepable) > > > 1st 0xffffffff80d0b560 in_multi_mtx (in_multi_mtx) @ > > netinet/in_mcast.c:1095 > > > 2nd 0xffffff00014e3850 USB device SX lock (USB device SX lock) @ > > /usr/home/freebsd_usb/sys/modules/usb/usb/../../../dev/usb/usb_request.c:441 > > > KDB: stack backtrace: > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > > _witness_debugger() at _witness_debugger+0x2e > > > witness_checkorder() at witness_checkorder+0x807 > > > _sx_xlock() at _sx_xlock+0x55 > > > usbd_do_request_flags() at usbd_do_request_flags+0xe5 > > > axe_cmd() at axe_cmd+0xc7 > > > axe_setmulti_locked() at axe_setmulti_locked+0x70 > > > axe_setmulti() at axe_setmulti+0x3e > > > axe_ioctl() at axe_ioctl+0x132 > > > if_addmulti() at if_addmulti+0x19b > > > in_joingroup_locked() at in_joingroup_locked+0x1bc > > > in_joingroup() at in_joingroup+0x52 > > > in_control() at in_control+0x1144 > > > ifioctl() at ifioctl+0x1118 > > > kern_ioctl() at kern_ioctl+0xbe > > > ioctl() at ioctl+0xfd > > > syscallenter() at syscallenter+0x1aa > > > syscall() at syscall+0x4c > > > Xfast_syscall() at Xfast_syscall+0xe2 > > > > > > when I uses the following code at driver's ioctl routine: > > > > > > case SIOCADDMULTI: > > > case SIOCDELMULTI: > > > axe_setmulti(sc, 0); > > > break; > > > > > > It means that USB driver always should defer SIOCADDMULTI / > > > SIOCDELMULTI handling to the other process context to avoid LOR. > > > > > > My question is that is it safe if the multicasting operations for USB > > > device happens without IN_MULTI_LOCK? Or is there any race cases if the > > > task is deferred? > > > > Why is USB using an sx lock instead of a mutex? > > Frankly speaking I also don't know why hps@ uses sx lock. That is one > of things I'd like to change it. > > Just looking the comment at usb_request.c@441: > > /* > * Grab the default sx-lock so that serialisation > * is achieved when multiple threads are involved: > */ > sx_xlock(&udev->ctrl_sx); > > I think he might want to hold the lock even if the thread is going into > sleep. It might be for serialization. > > However even if we succeed to change the lock from sx to mutex, it's > hard to avoid the requests going into the sleep. It means USB stack > should call like below: > > mtx_sleep(chan, IN_MULTI_LOCK, ...); > > to avoid the kernel's complain (would be `sleep with holding > non-sleepable lock'). > > What I'd like to say is that the sleeping is big problem if mutex is > used that it'd be worse when multiple mutex locks are used. > > So I'm looking for a fundamental solution to solve this problem. > Welcomes any ideas. It is probably fine to just schedule a task to do the actual work of axe_setmulti(). I think you do not need to lock IN_MULTI_LOCK yourself in your task handler as long as your handler holds the appropriate lock (if_maddr_rlock() IIRC) when walking the interface's multicast address list. -- John Baldwin