From owner-freebsd-smp@FreeBSD.ORG Mon Nov 27 16:56:33 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BFB7116A60B for ; Mon, 27 Nov 2006 16:56:33 +0000 (UTC) (envelope-from marsdeni@paramounttube.com) Received: from paramounttube.com (d01m-213-44-214-53.d4.club-internet.fr [213.44.214.53]) by mx1.FreeBSD.org (Postfix) with SMTP id 2D66843D60 for ; Mon, 27 Nov 2006 16:55:30 +0000 (GMT) (envelope-from marsdeni@paramounttube.com) Message-ID: <000001c71243$bfa4dcf0$c951a8c0@qbkhct> From: "Dave Winget" To: freebsd-smp@freebsd.org Date: Mon, 27 Nov 2006 08:47:39 -0800 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: nystagmu X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Dave Winget List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2006 16:56:33 -0000 Hi, =20 VjAGRA_yl_$1,78 CjALiS_qt_$3,00 LEVjTRA_sp_$3,33 =20 www [dot] rx44 [dot] info _____ =20 now. From owner-freebsd-smp@FreeBSD.ORG Thu Nov 30 23:20:20 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9A68D16A51F for ; Thu, 30 Nov 2006 23:20:20 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5172843CA2 for ; Thu, 30 Nov 2006 23:20:09 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.6/8.13.6) with ESMTP id kAUNKBcM005454; Thu, 30 Nov 2006 18:20:13 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-smp@freebsd.org Date: Thu, 30 Nov 2006 16:58:23 -0500 User-Agent: KMail/1.9.1 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200611301658.23898.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Thu, 30 Nov 2006 18:20:13 -0500 (EST) X-Virus-Scanned: ClamAV 0.88.3/2264/Thu Nov 30 14:13:30 2006 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Ashok TM Subject: Re: Setting CPU affinity to process( Freebsd smp kernel) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Nov 2006 23:20:20 -0000 On Monday 20 November 2006 06:24, Ashok TM wrote: > Hello, > > Is there a way to set a process's CPU affinity with freebsd smp kernel. > By doing so a process is bound to and will only run on the processor (where > the affinity was set ) even though the other processors are free. (similar > to sched_setaffinity available on Linux kernel) . > > Does freebsd smp scheduler supports setting affinity? Yes, but currently only in the kernel. -- John Baldwin From owner-freebsd-smp@FreeBSD.ORG Fri Dec 1 19:17:32 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 08E2416A407; Fri, 1 Dec 2006 19:17:32 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 31D8343CA6; Fri, 1 Dec 2006 19:17:16 +0000 (GMT) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id C650B1A4D9E; Fri, 1 Dec 2006 11:17:31 -0800 (PST) Date: Fri, 1 Dec 2006 11:17:31 -0800 From: Alfred Perlstein To: freebsd-usb@freebsd.org, freebsd-smp@freebsd.org Message-ID: <20061201191731.GR38808@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i Cc: Subject: RFC: Fixing USB ethernet for FreeBSD 7.0. X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2006 19:17:32 -0000 [I am not subscribed to these lists, please do not trim me off the cc list.] This email is has a short war story and a request for comments. I recently had the displeasure of trying to use an USB etherdongle under FreeBSD. Result: panic when the interface was started. I fixed it using a stopgap: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/usb/if_aue.c?rev=1.101&content-type=text/x-cvsweb-markup There are still some major issues: 1) requires Giant. 2) several error paths will still panic the kernel. I would like to fix them, however that does not seem easy given the existing infrastructure. I'm going to walk you through my thought process on the whole system and I would like feedback please. Basically, consider this a one-way conversation that you may interrupt at any time to correct me or to make suggestions. If there is a major flaw in anything I've said, then feel free to discard the following points and we will start up from the flaw. NOTE: I am interested in minimizing the impact of handling a "slow bus" on the rest of the kernel, but at the same time I am not interested in saving nanoseconds off IO time at the expense of having a programmatically dismal API. Meaning, I will be happy to sacrifice a modicum of performance in order to provide a programming model that allows us to have a stable device driver base. Onto the RFC: ********************************************************************** RFC: Fixing USB ethernet API: Statement #1: USB is a slow bus. While doing IO to the usb bus one should be able to yeild the CPU to another thread. Hence an interrupt can not use DELAY() to wait for data, instead it should arrainge for a callback upon completion of IO, and userspace should use tsleep to wait for data. Statement #2: Using callbacks to do all IO during an interrupt is programatically complex and painful. For instance, take the case of the following code pulled from aue_stop() (which can be called from interrupt context): AUE_LOCK(sc); ... aue_csr_write_1(sc, AUE_CTL0, 0); aue_csr_write_1(sc, AUE_CTL1, 0); aue_reset(sc); ... /* Stop transfers. */ if (sc->aue_ep[AUE_ENDPT_RX] != NULL) { err = usbd_abort_pipe(sc->aue_ep[AUE_ENDPT_RX]); if (err) { printf("aue%d: abort rx pipe failed: %s\n", sc->aue_unit, usbd_errstr(err)); } err = usbd_close_pipe(sc->aue_ep[AUE_ENDPT_RX]); There are probably about a dozen IOs here, splitting this into callbacks would be terribly inconvenient. Note the aue_reset() call which does several syncronous IOs that should not be interrupted! Statement #3: We need to provide the same atomic guarantees that we give other device drivers, specifically the ability to do FOO_LOCK/FOO_UNLOCK and not have to worry about user contexts sneaking into interrupt contexts. (we do not have these problems on "fast bus" devices because of per-driver locks (FXP_LOCK/FXP_UNLOCK)) Statement #4: To do all of this in a manner that provides a programmitically safe way we need to run some drivers under a full kthread process context even under interrupts. Statement #5: We have only a minor mechanism to do so at the current time, only the usb_taskqs exist. Using just the usb_taskqs would serialize IO too much and slow down USB IO, additionally if any device or driver wedges, the whole stack will stop working. Proposal: Each USB device that needs this (I envision most devices moving to this model) will require the following: A process context. A process style recursive lock (lockmgr). I am leaning towards 1 thread per device instance, the reason being that if a device (or its driver) goes out to lunch, it should not bring down the whole stack. If anyone has the "thousands of usb devices" then they can invent some sort of "usb taskq pool" to make life easier. What I will provide is: API for creating a per-device kthread. API for deleteing per-device kthread. API for recursive process locks. (simple layer over lockmgr) What I would like from FreeBSD is a discussion about if there could be a "better way" that _IS PROGRAMMITICALLY FEASABLE_. Sounds good? Let me know, now is the time to discuss this with me before I waste a lot of time writing something that someone may have to rewrite in a few months because they didn't speak up now. I'm available for phone calls if that will help. thank you, -- - Alfred Perlstein, RED Incorporated Consulting. - coder / sysadmin / FreeBSD Hacker / All that jazz - From owner-freebsd-smp@FreeBSD.ORG Fri Dec 1 19:28:48 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 155C616A403; Fri, 1 Dec 2006 19:28:48 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (vc4-2-0-87.dsl.netrack.net [199.45.160.85]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6128043CBB; Fri, 1 Dec 2006 19:28:30 +0000 (GMT) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.4/8.13.4) with ESMTP id kB1JNiBr060040; Fri, 1 Dec 2006 12:23:45 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Fri, 01 Dec 2006 12:24:33 -0700 (MST) Message-Id: <20061201.122433.-749245269.imp@bsdimp.com> To: alfred@freebsd.org From: "M. Warner Losh" In-Reply-To: <20061201191731.GR38808@elvis.mu.org> References: <20061201191731.GR38808@elvis.mu.org> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Fri, 01 Dec 2006 12:23:45 -0700 (MST) Cc: freebsd-smp@freebsd.org, freebsd-usb@freebsd.org Subject: Re: RFC: Fixing USB ethernet for FreeBSD 7.0. X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2006 19:28:48 -0000 Have you looked at the usb work that Hans Petter Selasky at http://www.turbocat.net/~hselasky/usb4bsd yet? Warner From owner-freebsd-smp@FreeBSD.ORG Fri Dec 1 19:43:24 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 139E316A50E; Fri, 1 Dec 2006 19:43:24 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 13D8543CA7; Fri, 1 Dec 2006 19:43:08 +0000 (GMT) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id ADF181A4D9B; Fri, 1 Dec 2006 11:43:23 -0800 (PST) Date: Fri, 1 Dec 2006 11:43:23 -0800 From: Alfred Perlstein To: "M. Warner Losh" Message-ID: <20061201194323.GT38808@elvis.mu.org> References: <20061201191731.GR38808@elvis.mu.org> <20061201.122433.-749245269.imp@bsdimp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20061201.122433.-749245269.imp@bsdimp.com> User-Agent: Mutt/1.4.2.2i Cc: freebsd-smp@freebsd.org, freebsd-usb@freebsd.org Subject: Re: RFC: Fixing USB ethernet for FreeBSD 7.0. X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2006 19:43:24 -0000 * M. Warner Losh [061201 11:30] wrote: > Have you looked at the usb work that Hans Petter Selasky > at http://www.turbocat.net/~hselasky/usb4bsd yet? I just did, while it solves a lock order problem, this doesn't appear to solve the programmitic issues, namely multiple usb IOs requiring callbacks and how to issue a series of complex usb IOs from interrupt context. Basically, I need to be able to do USB IO as if I was doing normal BUS IO, usb does not offer this in interrupt context except as a series of callbacks that appear to be programmatically impossible to implement. Have a look at if_aue.c, then look at any of the error cases that might be called from interrupt context, they wind up doing sync IO to the device which is illegal (sleeping while holding a driver lock). Hans Petter Selasky's work is nice, however it doesn't solve these issues, the lock would still have to be held. Do you understand that I'm trying to give usb ethernet the same ease of programming that other devices on a "fast bus" have? -- - Alfred Perlstein, RED Incorporated Consulting. - coder / sysadmin / FreeBSD Hacker / All that jazz - From owner-freebsd-smp@FreeBSD.ORG Fri Dec 1 22:32:35 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D4AC616A47B; Fri, 1 Dec 2006 22:32:35 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe06.swip.net [212.247.154.161]) by mx1.FreeBSD.org (Postfix) with ESMTP id 302D643CA7; Fri, 1 Dec 2006 22:32:17 +0000 (GMT) (envelope-from hselasky@c2i.net) X-T2-Posting-ID: NtQinKAjKMU7NZuf7ppucg== X-Cloudmark-Score: 0.000000 [] Received: from [193.216.121.55] (account mc467741@c2i.net HELO [10.0.0.249]) by mailfe06.swip.net (CommuniGate Pro SMTP 5.0.12) with ESMTPA id 344676319; Fri, 01 Dec 2006 23:32:31 +0100 From: Hans Petter Selasky To: freebsd-usb@freebsd.org Date: Fri, 1 Dec 2006 23:32:10 +0100 User-Agent: KMail/1.7 References: <20061201191731.GR38808@elvis.mu.org> <20061201.122433.-749245269.imp@bsdimp.com> <20061201194323.GT38808@elvis.mu.org> In-Reply-To: <20061201194323.GT38808@elvis.mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200612012332.12796.hselasky@c2i.net> Cc: Alfred Perlstein , freebsd-smp@freebsd.org, "M. Warner Losh" Subject: Re: RFC: Fixing USB ethernet for FreeBSD 7.0. X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2006 22:32:35 -0000 On Friday 01 December 2006 20:43, Alfred Perlstein wrote: > * M. Warner Losh [061201 11:30] wrote: > > Have you looked at the usb work that Hans Petter Selasky > > at http://www.turbocat.net/~hselasky/usb4bsd yet? > > I just did, while it solves a lock order problem, this doesn't > appear to solve the programmitic issues, namely multiple usb IOs > requiring callbacks and how to issue a series of complex usb > IOs from interrupt context. > > Basically, I need to be able to do USB IO as if I was doing normal > BUS IO, usb does not offer this in interrupt context except as a > series of callbacks that appear to be programmatically impossible > to implement. Yes, yes, this is supported. See the USBD_USE_POLLING flag. > > Have a look at if_aue.c, then look at any of the error cases that > might be called from interrupt context, they wind up doing sync IO > to the device which is illegal (sleeping while holding a driver > lock). > > Hans Petter Selasky's work is nice, however it doesn't solve > these issues, the lock would still have to be held. You don't want to do things without holding a lock. > Do you understand that I'm trying to give usb ethernet the > same ease of programming that other devices on a "fast bus" > have? Yes, but that comes at an expense. Higher CPU usage, more delay. --HPS