From owner-freebsd-arch@FreeBSD.ORG Sun Mar 2 05:21:51 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B190106566B for ; Sun, 2 Mar 2008 05:21:51 +0000 (UTC) (envelope-from Juergen.Dankoweit@T-Online.de) Received: from mailout01.sul.t-online.com (mailout01.sul.t-online.de [194.25.134.80]) by mx1.freebsd.org (Postfix) with ESMTP id 169448FC18 for ; Sun, 2 Mar 2008 05:21:50 +0000 (UTC) (envelope-from Juergen.Dankoweit@T-Online.de) Received: from fwd33.aul.t-online.de by mailout01.sul.t-online.com with smtp id 1JVgP2-0007yO-03; Sun, 02 Mar 2008 06:06:24 +0100 Received: from mail.juergendankoweit.net (V8PROiZr8hEP+QUG6iWgcVc-5asN7gW72DKNZDqmLuG+YsYASsuJKkA-zL2kihRgP0@[87.174.208.201]) by fwd33.aul.t-online.de with esmtp id 1JVgOz-0js8qe0; Sun, 2 Mar 2008 06:06:21 +0100 Received: from localhost (localhost.juergendankoweit.net [127.0.0.1]) by mail.juergendankoweit.net (Postfix) with ESMTP id 07A67114C2 for ; Sun, 2 Mar 2008 06:06:21 +0100 (CET) X-Virus-Scanned: amavisd-new at juergendankoweit.net Received: from mail.juergendankoweit.net ([127.0.0.1]) by localhost (mail.juergendankoweit.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZW+M+rmPe+wE for ; Sun, 2 Mar 2008 06:06:16 +0100 (CET) Received: from primergy470.juergendankoweit.net (primergy470.juergendankoweit.net [192.168.1.1]) by mail.juergendankoweit.net (Postfix) with ESMTP id 97A73121EA for ; Sun, 2 Mar 2008 06:06:16 +0100 (CET) From: Juergen Dankoweit To: freebsd-arch@freebsd.org Content-Type: text/plain Date: Sun, 02 Mar 2008 06:06:14 +0100 Message-Id: <1204434374.3416.2.camel@primergy470.juergendankoweit.net> Mime-Version: 1.0 X-Mailer: Evolution 2.4.2.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-ID: V8PROiZr8hEP+QUG6iWgcVc-5asN7gW72DKNZDqmLuG+YsYASsuJKkA-zL2kihRgP0 X-TOI-MSGID: 479f106b-5e48-4993-9caf-30e114324b73 Subject: PR 114597 and bug solving X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Juergen.Dankoweit@T-Online.de List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Mar 2008 05:21:51 -0000 Hello to the List, in the next days I want to some debugging on the sym driver and the communication with the cam layer. Where do I find some documentation how the driver communicates with the cam layer, which parameters are needed for the function calls and which return codes are generated? Thank you very much for your help. Best regards J. Dankoweit From owner-freebsd-arch@FreeBSD.ORG Mon Mar 3 11:07:04 2008 Return-Path: Delivered-To: freebsd-arch@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF55210656D0 for ; Mon, 3 Mar 2008 11:07:04 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 482D28FC2C for ; Mon, 3 Mar 2008 11:07:04 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m23B74Hi021985 for ; Mon, 3 Mar 2008 11:07:04 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m23B73YO021981 for freebsd-arch@FreeBSD.org; Mon, 3 Mar 2008 11:07:03 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 3 Mar 2008 11:07:03 GMT Message-Id: <200803031107.m23B73YO021981@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-arch@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-arch@FreeBSD.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Mar 2008 11:07:05 -0000 Current FreeBSD problem reports Critical problems Serious problems Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/120749 arch [request] Suggest upping the default kern.ps_arg_cache 1 problem total. From owner-freebsd-arch@FreeBSD.ORG Tue Mar 4 23:30:17 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 761AB1065673 for ; Tue, 4 Mar 2008 23:30:17 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.nuxi.org (trang.nuxi.org [74.95.12.85]) by mx1.freebsd.org (Postfix) with ESMTP id 5D03F8FC15 for ; Tue, 4 Mar 2008 23:30:17 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.nuxi.org (obrien@localhost [127.0.0.1]) by dragon.nuxi.org (8.14.1/8.14.1) with ESMTP id m24NUGFe002589 for ; Tue, 4 Mar 2008 15:30:17 -0800 (PST) (envelope-from obrien@dragon.nuxi.org) Received: (from obrien@localhost) by dragon.nuxi.org (8.14.2/8.14.1/Submit) id m24NUGBc002588 for freebsd-arch@freebsd.org; Tue, 4 Mar 2008 15:30:16 -0800 (PST) (envelope-from obrien) Date: Tue, 4 Mar 2008 15:30:16 -0800 From: "David O'Brien" To: freebsd-arch@freebsd.org Message-ID: <20080304233016.GA2334@dragon.NUXI.org> Mail-Followup-To: obrien@freebsd.org, freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Operating-System: FreeBSD 8.0-CURRENT User-Agent: Mutt/1.5.16 (2007-06-09) Subject: Tweaking the "Remaking Makefiles" feature X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: obrien@freebsd.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Mar 2008 23:30:17 -0000 Last year Max Khon implemented something called the "Remaking Makefiles" feature to make(1): After reading Makefile and all the files that are included using .include or .sinclude directives (source Makefiles) make considers each source Makefile as a target and tries to rebuild it. Both explicit and implicit rules are checked and all source Makefiles are updated if necessary. If any of the source Makefiles were rebuilt, make restarts from clean state. .. When remaking a source Makefile options -t (touch target), -q (query mode), and -n (no exec) do not take effect, unless source Makefile is specified explicitly as a target in make command line. Additionally, system makefiles and .depend are not considered as a Makefiles that can be rebuilt. While I find the feature neat, I find this feature (enabled by default) way too dangerous. It caused a major problem with the CVS 1.11.22 import I did, and I feel it could easily catch others off guard. So I think it should be turned off by default. (How many of you didn't realize 'make -n' could write to files?) I propose a new directive called ".MAKEFILEDEPS" which enables it. (TODO: man page update) Index: globals.h =================================================================== RCS file: /home/ncvs/src/usr.bin/make/globals.h,v retrieving revision 1.11 diff -u -p -r1.11 globals.h --- globals.h 8 Mar 2007 09:16:10 -0000 1.11 +++ globals.h 4 Mar 2008 22:14:27 -0000 @@ -78,6 +78,7 @@ extern Boolean beVerbose; /* True if sho extern Boolean noExecute; /* True if should execute nothing */ extern Boolean allPrecious; /* True if every target is precious */ extern Boolean is_posix; /* .POSIX target seen */ +extern Boolean mfAutoDeps; /* .MAKEFILEDEPS target seen */ /* True if should continue on unaffected portions of the graph * when have an error in one portion */ Index: hash_tables.c =================================================================== RCS file: /home/ncvs/src/usr.bin/make/hash_tables.c,v retrieving revision 1.3 diff -u -p -r1.3 hash_tables.c --- hash_tables.c 26 Sep 2005 20:31:00 -0000 1.3 +++ hash_tables.c 4 Mar 2008 23:13:34 -0000 @@ -1,7 +1,7 @@ /* * DO NOT EDIT - * $FreeBSD: src/usr.bin/make/hash_tables.c,v 1.3 2005/09/26 20:31:00 phk Exp $ - * auto-generated from FreeBSD: src/usr.bin/make/parse.c,v 1.108 2005/05/24 15:30:03 harti Exp + * $FreeBSD$ + * auto-generated from FreeBSD: src/usr.bin/make/parse.c,v 1.113 2007/04/12 18:14:00 ru Exp * DO NOT EDIT */ #include @@ -66,12 +66,12 @@ directive_hash(const u_char *key, size_t } /* * d=2 - * n=72 - * m=34 + * n=74 + * m=35 * c=2.09 * maxlen=1 * minklen=4 - * maxklen=12 + * maxklen=13 * minchar=46 * maxchar=95 * loop=0 @@ -80,30 +80,30 @@ directive_hash(const u_char *key, size_t */ static const signed char keyword_g[] = { - 8, 15, -1, 25, 22, 20, -1, 33, 16, -1, - 21, 31, 0, 0, 0, 29, 30, 8, -1, 0, - -1, 21, -1, 0, -1, -1, -1, -1, -1, 4, - -1, -1, 25, 28, -1, 27, 11, 23, 0, 0, - 24, -1, -1, 0, 3, 0, -1, 24, 0, 0, - -1, 28, 12, -1, 20, 13, -1, 5, -1, 1, - 0, 0, -1, 0, 10, 19, 13, 9, -1, 2, - -1, -1, + 12, 18, 7, 25, 30, 5, -1, -1, -1, 7, + -1, 0, 33, 0, 4, -1, -1, 13, 29, 0, + -1, 28, -1, 28, -1, 0, -1, 27, 4, 34, + -1, -1, -1, 30, 13, 10, -1, -1, 0, 10, + 24, -1, -1, -1, 0, 6, 0, 0, -1, 23, + -1, -1, -1, 0, -1, 23, -1, -1, 19, 4, + -1, 31, 12, 16, -1, 20, 22, 9, 0, -1, + -1, 9, 4, 0, }; static const u_char keyword_T0[] = { - 32, 10, 54, 61, 2, 35, 62, 50, 52, 53, - 70, 7, 62, 18, 24, 30, 31, 66, 10, 61, - 52, 71, 56, 56, 28, 6, 33, 67, 12, 41, - 39, 45, 51, 21, 34, 53, 56, 40, 47, 52, - 21, 61, 60, 12, 7, 28, 42, 38, 38, 52, + 34, 28, 50, 61, 14, 57, 48, 60, 20, 67, + 60, 63, 0, 24, 28, 2, 49, 64, 18, 23, + 36, 33, 40, 14, 38, 42, 71, 49, 2, 53, + 53, 37, 7, 29, 24, 21, 12, 50, 59, 10, + 43, 23, 0, 44, 47, 6, 46, 22, 48, 64, }; static const u_char keyword_T1[] = { - 0, 39, 65, 48, 13, 62, 46, 42, 5, 50, - 69, 69, 69, 43, 2, 46, 12, 6, 11, 9, - 24, 10, 25, 64, 68, 13, 57, 55, 17, 33, - 1, 18, 0, 67, 10, 14, 57, 56, 0, 6, - 50, 13, 3, 47, 56, 22, 37, 13, 28, 48, + 18, 67, 39, 60, 7, 70, 2, 26, 31, 18, + 73, 47, 61, 17, 38, 50, 22, 52, 13, 55, + 56, 32, 63, 4, 64, 55, 49, 21, 47, 67, + 33, 66, 60, 73, 30, 68, 69, 32, 72, 4, + 28, 49, 51, 15, 66, 68, 43, 67, 46, 56, }; @@ -113,7 +113,7 @@ keyword_hash(const u_char *key, size_t l unsigned f0, f1; const u_char *kp = key; - if (len < 4 || len > 12) + if (len < 4 || len > 13) return -1; for (f0=f1=0; *kp; ++kp) { @@ -123,8 +123,8 @@ keyword_hash(const u_char *key, size_t l f1 += keyword_T1[-46 + *kp]; } - f0 %= 72; - f1 %= 72; + f0 %= 74; + f1 %= 74; - return (keyword_g[f0] + keyword_g[f1]) % 34; + return (keyword_g[f0] + keyword_g[f1]) % 35; } Index: main.c =================================================================== RCS file: /home/ncvs/src/usr.bin/make/main.c,v retrieving revision 1.165 diff -u -p -r1.165 main.c --- main.c 4 Mar 2008 06:08:59 -0000 1.165 +++ main.c 4 Mar 2008 22:14:20 -0000 @@ -123,6 +123,7 @@ Lst create = Lst_Initializer(create); Boolean allPrecious; /* .PRECIOUS given on line by itself */ Boolean is_posix; /* .POSIX target seen */ +Boolean mfAutoDeps; /* .MAKEFILEDEPS target seen */ Boolean beSilent; /* -s flag */ Boolean beVerbose; /* -v flag */ Boolean compatMake; /* -B argument */ @@ -1250,7 +1251,7 @@ main(int argc, char **argv) */ Lst targs = Lst_Initializer(targs); - if (!is_posix) { + if (!is_posix && mfAutoDeps) { /* * Check if any of the makefiles are out-of-date. */ Index: parse.c =================================================================== RCS file: /home/ncvs/src/usr.bin/make/parse.c,v retrieving revision 1.113 diff -u -p -r1.113 parse.c --- parse.c 12 Apr 2007 18:14:00 -0000 1.113 +++ parse.c 4 Mar 2008 22:14:47 -0000 @@ -168,6 +168,7 @@ typedef enum { ExPath, /* .PATH */ Phony, /* .PHONY */ Posix, /* .POSIX */ + MakefileDeps, /* .MAKEFILEDEPS */ Precious, /* .PRECIOUS */ ExShell, /* .SHELL */ Silent, /* .SILENT */ @@ -213,6 +214,7 @@ static const struct keyword { { ".LIBS", Libs, 0 }, { ".MAIN", Main, 0 }, { ".MAKE", Attribute, OP_MAKE }, + { ".MAKEFILEDEPS", MakefileDeps, 0 }, { ".MAKEFLAGS", MFlags, 0 }, { ".MFLAGS", MFlags, 0 }, { ".NOTMAIN", Attribute, OP_NOTMAIN }, @@ -1069,6 +1071,9 @@ ParseDoDependency(char *line) LST_FOREACH(ln, &paths) Path_Clear(Lst_Datum(ln)); break; + case MakefileDeps: + mfAutoDeps = TRUE; + break; case Posix: is_posix = TRUE; Var_Set("%POSIX", "1003.2", VAR_GLOBAL); From owner-freebsd-arch@FreeBSD.ORG Fri Mar 7 12:13:59 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A528B1065672; Fri, 7 Mar 2008 12:13:59 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 6D1008FC1E; Fri, 7 Mar 2008 12:13:59 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.107] (cpe-24-94-75-93.hawaii.res.rr.com [24.94.75.93]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m27CDpcK074502; Fri, 7 Mar 2008 07:13:55 -0500 (EST) (envelope-from jroberson@chesapeake.net) Date: Fri, 7 Mar 2008 02:16:30 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: arch@freebsd.org Message-ID: <20080307020626.G920@desktop> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Getting rid of the static msleep priority boost X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 12:13:59 -0000 Hello, I've been studying some problems with recent scheduler improvements that help a lot on some workloads and hurt on others. I've tracked the problem down to static priority boosts handed out by msleep/cv_broadcastpri. The basic problem is that a user thread will be woken with a kernel priority thus allowing it to preempt a thread running on any processor with a lesser priority. The lesser priority thread may in fact hold some resource that the higher priority thread requires. Thus we context switch several times and perhaps go through priority propagation as well. I have verified that disabling these static priority boosts entirely fixes the performance problem I've run into on at least one workload. There are probably others that it helps and hopefully we can discover that. I'd like to know if anyone has a strong preference to keep this feature. It is likely that it helps in some interactive situations. I'm not sure how much however. I propose that we make a sysctl that disables it and turn it off by default. If we see complaints on current@ we can suggest that they toggle the sysctl to see if it alleviates problems. Based on feedback from that experiment and some testing we can then choose a few options: 1) Disable the static boosts entirely. Leave kernel priorities for kernel threads and priority propagation. Most other kernels do this. Would make my life in ULE much easier as well. 2) Leave the support for static boosts but remove it from all but a few key locations. Leaving it in the api would give some flexibility but might confuse developers. 3) Leave things as they are. undesirable. I'm leaning towards #2 based on the information I have presently. This is almost a significant change to historic BSD behavior so we might want to tread lightly. Thanks, Jeff From owner-freebsd-arch@FreeBSD.ORG Fri Mar 7 14:36:09 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 06F441065671 for ; Fri, 7 Mar 2008 14:36:09 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id E8E228FC21 for ; Fri, 7 Mar 2008 14:36:08 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from zion.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by elvis.mu.org (Postfix) with ESMTP id 74DCE1A4D8F; Fri, 7 Mar 2008 06:18:41 -0800 (PST) From: John Baldwin To: Jeff Roberson Date: Fri, 7 Mar 2008 08:42:37 -0500 User-Agent: KMail/1.9.7 References: <20080307020626.G920@desktop> In-Reply-To: <20080307020626.G920@desktop> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803070842.37248.jhb@freebsd.org> Cc: arch@freebsd.org Subject: Re: Getting rid of the static msleep priority boost X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 14:36:09 -0000 On Friday 07 March 2008 07:16:30 am Jeff Roberson wrote: > Hello, > > I've been studying some problems with recent scheduler improvements that > help a lot on some workloads and hurt on others. I've tracked the problem > down to static priority boosts handed out by msleep/cv_broadcastpri. The > basic problem is that a user thread will be woken with a kernel priority > thus allowing it to preempt a thread running on any processor with a > lesser priority. The lesser priority thread may in fact hold some > resource that the higher priority thread requires. Thus we context switch > several times and perhaps go through priority propagation as well. > > I have verified that disabling these static priority boosts entirely fixes > the performance problem I've run into on at least one workload. There are > probably others that it helps and hopefully we can discover that. > > I'd like to know if anyone has a strong preference to keep this feature. > It is likely that it helps in some interactive situations. I'm not sure > how much however. I propose that we make a sysctl that disables it and > turn it off by default. If we see complaints on current@ we can suggest > that they toggle the sysctl to see if it alleviates problems. > > Based on feedback from that experiment and some testing we can then choose > a few options: > > 1) Disable the static boosts entirely. Leave kernel priorities for > kernel threads and priority propagation. Most other kernels do this. > Would make my life in ULE much easier as well. > > 2) Leave the support for static boosts but remove it from all but a few > key locations. Leaving it in the api would give some flexibility but > might confuse developers. > > 3) Leave things as they are. undesirable. > > I'm leaning towards #2 based on the information I have presently. This is > almost a significant change to historic BSD behavior so we might want to > tread lightly. One thing to note is that we actually depend on the priority boost (evilly) to pick processes to swap out. (I think we check for <= PSOCK and don't swap those out). One thing that I've wanted to happen for a while is that the sleep priority for msleep() just be a parameter available to the scheduler that the scheduler can use to calculate the real internal priority rather than just being a set. That is, I imagine having: void sched_set_sleep_prio(struct thread *td, u_char pri); u_char sched_get_sleep_prio(struct thread *td); (The swap check would use the get call). The 4BSD scheduler's implementation of sched_set_sleep_prio would look like this: void sched_set_sleep_prio(struct thread *td, u_char pri) { td->td_sched->sleep_pri = pri; sched_prio(td, pri); } void sched_userret(..) { ... td->td_sched->sleep_pri = 0; /* not in the kernel anymore */ } but other schedulers may just save it and recalculate the priority where the priority calculation just considers the sleep priority as one among many factors. If nothing else, this allows it to be a scheduler decision to ignore it (so 4BSD could continue to do what it does now, but ULE may ignore it, or ignore certain levels, etc.) -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Fri Mar 7 15:36:08 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD9D2106566C for ; Fri, 7 Mar 2008 15:36:08 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 9EDFE8FC28 for ; Fri, 7 Mar 2008 15:36:08 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from zion.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by elvis.mu.org (Postfix) with ESMTP id 4742F1A4D7C; Fri, 7 Mar 2008 07:16:24 -0800 (PST) From: John Baldwin To: freebsd-arch@freebsd.org Date: Fri, 7 Mar 2008 10:16:46 -0500 User-Agent: KMail/1.9.7 References: <20080307020626.G920@desktop> <200803070842.37248.jhb@freebsd.org> In-Reply-To: <200803070842.37248.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803071016.46944.jhb@freebsd.org> Cc: Subject: Re: Getting rid of the static msleep priority boost X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 15:36:09 -0000 On Friday 07 March 2008 08:42:37 am John Baldwin wrote: > On Friday 07 March 2008 07:16:30 am Jeff Roberson wrote: > > Hello, > > > > I've been studying some problems with recent scheduler improvements that > > help a lot on some workloads and hurt on others. I've tracked the > > problem down to static priority boosts handed out by > > msleep/cv_broadcastpri. The basic problem is that a user thread will be > > woken with a kernel priority thus allowing it to preempt a thread running > > on any processor with a lesser priority. The lesser priority thread may > > in fact hold some resource that the higher priority thread requires. > > Thus we context switch several times and perhaps go through priority > > propagation as well. > > > > I have verified that disabling these static priority boosts entirely > > fixes the performance problem I've run into on at least one workload. > > There are probably others that it helps and hopefully we can discover > > that. > > > > I'd like to know if anyone has a strong preference to keep this feature. > > It is likely that it helps in some interactive situations. I'm not sure > > how much however. I propose that we make a sysctl that disables it and > > turn it off by default. If we see complaints on current@ we can suggest > > that they toggle the sysctl to see if it alleviates problems. > > > > Based on feedback from that experiment and some testing we can then > > choose a few options: > > > > 1) Disable the static boosts entirely. Leave kernel priorities for > > kernel threads and priority propagation. Most other kernels do this. > > Would make my life in ULE much easier as well. > > > > 2) Leave the support for static boosts but remove it from all but a few > > key locations. Leaving it in the api would give some flexibility but > > might confuse developers. > > > > 3) Leave things as they are. undesirable. > > > > I'm leaning towards #2 based on the information I have presently. This > > is almost a significant change to historic BSD behavior so we might want > > to tread lightly. > > One thing to note is that we actually depend on the priority boost (evilly) > to pick processes to swap out. (I think we check for <= PSOCK and don't > swap those out). One thing that I've wanted to happen for a while is that > the sleep priority for msleep() just be a parameter available to the > scheduler that the scheduler can use to calculate the real internal > priority rather than just being a set. That is, I imagine having: > > void sched_set_sleep_prio(struct thread *td, u_char pri); > u_char sched_get_sleep_prio(struct thread *td); > > (The swap check would use the get call). The 4BSD scheduler's > implementation of sched_set_sleep_prio would look like this: > > void > sched_set_sleep_prio(struct thread *td, u_char pri) > { > > td->td_sched->sleep_pri = pri; > sched_prio(td, pri); > } > > void > sched_userret(..) > { > > ... > td->td_sched->sleep_pri = 0; /* not in the kernel anymore */ > } > > but other schedulers may just save it and recalculate the priority where > the priority calculation just considers the sleep priority as one among > many factors. If nothing else, this allows it to be a scheduler decision > to ignore it (so 4BSD could continue to do what it does now, but ULE may > ignore it, or ignore certain levels, etc.) One thing to clarify: I'm not opposed to replacing the PSOCK check with something more suitable in the swap code, (in fact, that would be desirable), but it might take a good bit of work to do that and is probably easier to work on that as a separate change. I also think there can be some merit in having code paths hint to the scheduler the relative interactivity/priority of a sleep. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Fri Mar 7 16:01:29 2008 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F4C01065674 for ; Fri, 7 Mar 2008 16:01:29 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id CEC968FC19 for ; Fri, 7 Mar 2008 16:01:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id CED5946B8F for ; Fri, 7 Mar 2008 11:01:26 -0500 (EST) Date: Fri, 7 Mar 2008 16:01:26 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: arch@FreeBSD.org In-Reply-To: <20071224103322.C40176@fledge.watson.org> Message-ID: <20080307153236.P25342@fledge.watson.org> References: <20071224103322.C40176@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Re: 8.0 network stack MPsafety goals X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 16:01:29 -0000 On Mon, 24 Dec 2007, Robert Watson wrote: > Date Goals > ---- ----- > 26 Dec 2007 Post proposed schedule for flag and infrastructure removal > Post affected driver list > > 26 Jan 2008 Repost proposed schedule for flag and infrastructure removal > Post updated affected driver list > > 26 Feb 2008 Adjust boot-time printf for affect drivers to generate a loud > warning. > Post updated affected driver list Dear all, Per the in-progress plan to remove IFF_NEEDSGIANT support, I have increased the verbosity of the boot-time warning for IFF_NEEDSGIANT-dependent network interface drivers. 8-CURRENT users who are seeing this more verbose warning in their dmesg might want to watch out for the next two scheduled steps in May and June respectively. I've attached the remainder of the schedule and related details below. Thanks, Robert N M Watson Computer Laboratory University of Cambridge > 26 May 2008 Post HEADS UP of impending driver disabling > Post updated affected driver list > > 26 Jun 2008 Disable build of all drivers requiring IFF_NEEDSGIANT > Post updated affected driver list > > 26 Sep 2008 Post HEADS up of impending driver removal > Post updated affected driver list > > 26 Oct 2008 Delete source of all drivers requiring IFF_NEEDSGIANT > Remove flag and infrastructure > > Here is a list of potentially affected drivers: > > Name Bus Man page description > --- --- -------------------- > ar ISA/PCI synchronous Digi/Arnet device driver > arl ISA Aironet Arlan 655 wireless network adapter driver > awi PCCARD AMD PCnetMobile IEEE 802.11 PCMCIA wireless network > driver > axe USB ASIX Electronics AX88172 USB Ethernet driver > cdce USB USB Communication Device Class Ethernet driver > cnw PCCARD Netwave AirSurfer wireless network driver > cs ISA/PCCARD Ethernet device driver > cue USB CATC USB-EL1210A USB Ethernet driver > ex ISA/PCCARD Ethernet device driver for the Intel EtherExpress > Pro/10 and Pro/10+ > fe CBUS/ISA/PCCARD Fujitsu MB86960A/MB86965A based Ethernet adapters > ic I2C I2C bus system > ie ISA Ethernet device driver > kue USB Kawasaki LSI KL5KUSB101B USB Ethernet driver > oltr ISA/PCI Olicom Token Ring device driver > plip PPBUS printer port Internet Protocol driver > ppp TTY point to point protocol network interface > ray PCCARD Raytheon Raylink/Webgear Aviator PCCard driver > rue USB RealTek RTL8150 USB to Fast Ethernet controller > driver > rum USB Ralink Technology USB IEEE 802.11a/b/g wireless > network device > sbni ISA/PCI Granch SBNI12 leased line modem driver > sbsh PCI Granch SBNI16 SHDSL modem device driver > sl TTY slip network interface > snc ISA/PCCARD National Semiconductor DP8393X SONIC Ethernet adapter > driver > sr ISA/PCI synchronous RISCom/N2 / WANic 400/405 device driver > udav USB Davicom DM9601 USB Ethernet driver > ural USB Ralink Technology RT2500USB IEEE 802.11 driver > xe PCCARD Xircom PCMCIA Ethernet device driver > zyd USB ZyDAS ZD1211/ZD1211B USB IEEE 802.11b/g wireless > network device > > In some cases, the requirement for Giant is a property of a subsystem the > driver depends on as the driver itself; for example, the tty subsystem for > SLIP and PPP, and the USB subsystem for a number of USB ethernet and wireless > drivers. With most of a year before to go on the proposed schedule, my hope > is that we will have lots of time to address these issues, but wanted to get > a roadmap out from a network protocol stack architecture perspective so that > device driver and subsystem authors could have a schedule in mind. > > FYI, the following drivers also reference IFF_NEEDSGIANT, but only in order > to provide their own conditional MPSAFEty, which can be removed without > affecting device driver functionality (I believe): > > Name Bus Man page description > --- --- -------------------- > ce PCI driver for synchronous Cronyx Tau-PCI/32 WAN adapters > cp PCI driver for synchronous Cronyx Tau-PCI WAN adapters > ctau ISA driver for synchronous Cronyx Tau WAN adapters > cx ISA driver for synchronous/asynchronous Cronyx Sigma WAN > adapters > > Developers and users of the above drivers are heavily encouraged to update > the drivers to remove dependence on Giant, and/or make other contingency > plans. > > Robert N M Watson > Computer Laboratory > University of Cambridge > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Fri Mar 7 16:13:12 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACEFE106568D; Fri, 7 Mar 2008 16:13:12 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [64.81.247.49]) by mx1.freebsd.org (Postfix) with ESMTP id 896288FC1C; Fri, 7 Mar 2008 16:13:12 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost.mckusick.com [127.0.0.1]) by chez.mckusick.com (8.13.8/8.13.6) with ESMTP id m27FeSU6096030; Fri, 7 Mar 2008 07:40:36 -0800 (PST) (envelope-from mckusick@chez.mckusick.com) Message-Id: <200803071540.m27FeSU6096030@chez.mckusick.com> To: John Baldwin Date: Fri, 07 Mar 2008 07:40:28 -0800 From: Kirk McKusick Cc: arch@freebsd.org Subject: Re: Getting rid of the static msleep priority boost X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 16:13:12 -0000 > From: John Baldwin > To: Jeff Roberson > Date: Fri, 7 Mar 2008 08:42:37 -0500 > Cc: arch@freebsd.org > Subject: Re: Getting rid of the static msleep priority boost > > On Friday 07 March 2008 07:16:30 am Jeff Roberson wrote: > > Hello, > > > > I've been studying some problems with recent scheduler improvements that > > help a lot on some workloads and hurt on others. I've tracked the problem > > down to static priority boosts handed out by msleep/cv_broadcastpri. > > ... > > ... > > This change allows the decision on priority boost to be a scheduler > decision to ignore it (so 4BSD could continue to do what it does now, > but ULE may ignore it, or ignore certain levels, etc.) > > -- > John Baldwin I strongly agree with John's suggestion. The 4BSD scheduler will continue to have its historic behavior (which was `tuned' by careful selection of priority boosts) while more sophisticated schedulers like ULE will be able to use/ignore the priority boosts based on their better knowledge of system behavior. Kirk McKusick From owner-freebsd-arch@FreeBSD.ORG Fri Mar 7 16:20:01 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 520351065673 for ; Fri, 7 Mar 2008 16:20:01 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id E64C58FC12 for ; Fri, 7 Mar 2008 16:20:00 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.2/8.14.2/NETPLEX) with ESMTP id m27GJpKI021292; Fri, 7 Mar 2008 11:19:51 -0500 (EST) X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.0 (mail.netplex.net [204.213.176.10]); Fri, 07 Mar 2008 11:19:51 -0500 (EST) Date: Fri, 7 Mar 2008 11:19:51 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Jeff Roberson In-Reply-To: <20080307020626.G920@desktop> Message-ID: References: <20080307020626.G920@desktop> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: Getting rid of the static msleep priority boost X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 16:20:01 -0000 On Fri, 7 Mar 2008, Jeff Roberson wrote: > Hello, > > I've been studying some problems with recent scheduler improvements that help > a lot on some workloads and hurt on others. I've tracked the problem down to > static priority boosts handed out by msleep/cv_broadcastpri. The basic > problem is that a user thread will be woken with a kernel priority thus > allowing it to preempt a thread running on any processor with a lesser > priority. The lesser priority thread may in fact hold some resource that the > higher priority thread requires. Thus we context switch several times and > perhaps go through priority propagation as well. > > I have verified that disabling these static priority boosts entirely fixes > the performance problem I've run into on at least one workload. There are > probably others that it helps and hopefully we can discover that. > > I'd like to know if anyone has a strong preference to keep this feature. It > is likely that it helps in some interactive situations. I'm not sure how > much however. I propose that we make a sysctl that disables it and turn it > off by default. If we see complaints on current@ we can suggest that they > toggle the sysctl to see if it alleviates problems. > > Based on feedback from that experiment and some testing we can then choose a > few options: > > 1) Disable the static boosts entirely. Leave kernel priorities for kernel > threads and priority propagation. Most other kernels do this. Would make my > life in ULE much easier as well. > > 2) Leave the support for static boosts but remove it from all but a few key > locations. Leaving it in the api would give some flexibility but might > confuse developers. > > 3) Leave things as they are. undesirable. > > I'm leaning towards #2 based on the information I have presently. This is > almost a significant change to historic BSD behavior so we might want to > tread lightly. I'm not sure if any of the above remove the priority from the API, but it would be nice to get rid of msleep totally and replace it with an equivalent cv_wait(). I read jhb's comments, so perhaps some form of msleep() or equivalent that gives hints for the priority could be kept, but removed from all other places that don't need it. -- DE From owner-freebsd-arch@FreeBSD.ORG Fri Mar 7 20:50:08 2008 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9261A1065675 for ; Fri, 7 Mar 2008 20:50:08 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from speedfactory.net (mail.speedfactory.net [66.23.216.219]) by mx1.freebsd.org (Postfix) with ESMTP id 1B0A68FC16 for ; Fri, 7 Mar 2008 20:50:07 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from server.baldwin.cx (unverified [66.23.211.162]) by speedfactory.net (SurgeMail 3.8s) with ESMTP id 234663694-1834499 for ; Fri, 07 Mar 2008 15:51:27 -0500 Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m27KnsaE051834 for ; Fri, 7 Mar 2008 15:49:55 -0500 (EST) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: arch@FreeBSD.org Date: Fri, 7 Mar 2008 15:49:46 -0500 User-Agent: KMail/1.9.7 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803071549.46730.jhb@FreeBSD.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Fri, 07 Mar 2008 15:49:55 -0500 (EST) X-Virus-Scanned: ClamAV 0.91.2/6166/Fri Mar 7 11:36:07 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Subject: Change the x86 nexus to support different drivers for different platforms X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 20:50:08 -0000 Right now the two x86 platforms (i386 and amd64) support two different platforms (ACPI vs. legacy) via some rather hackish code (legacy checks for an acpi0 device in its probe routine, etc.). This makes it harder for 3rd parties to add support for proprietary platforms (such as for some embedded x86 appliances that run FreeBSD). This patch attempts to address this by allowing different platforms to provide their own nexus driver. The platform will do its probe in the nexus probe routine and then add suitable platform devices under its nexus device. One side effect of this change is that on x86 any devices that attach to directly to the nexus must be platform independent (this is already true) if they use identify routines. So the changes involved are to make the default nexus driver return BUS_PROBE_GENERIC and for it to explicitly add a legacy0 device in its attach routine (legacy no longer uses a normal identify routine). The ACPI platform now uses a custom nexus driver (which inherits from the default) in acpi_machdep.c which checks for ACPI in its probe routine and explicitly adds acpi0 in its attach routine. The ACPI bus driver no longer uses a standard acpi_identify routine, but instead exports a public 'acpi_identify()' function which serves as a probe routine for the MD nexus drivers to use. It might be nice, btw to possibly make other x86 platforms (such as xbox perhaps?) use their own nexus driver, but that can happen as a followup change later. Patch is at http://www.FreeBSD.org/~jhb/patches/acpi_nexus_cvs.patch -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Fri Mar 7 22:41:54 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1AAA6106566B; Fri, 7 Mar 2008 22:41:54 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id C948A8FC1A; Fri, 7 Mar 2008 22:41:53 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.107] (cpe-24-94-75-93.hawaii.res.rr.com [24.94.75.93]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m27MfliC040566; Fri, 7 Mar 2008 17:41:50 -0500 (EST) (envelope-from jroberson@chesapeake.net) Date: Fri, 7 Mar 2008 12:44:28 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: John Baldwin In-Reply-To: <200803071016.46944.jhb@freebsd.org> Message-ID: <20080307124038.I920@desktop> References: <20080307020626.G920@desktop> <200803070842.37248.jhb@freebsd.org> <200803071016.46944.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-arch@freebsd.org Subject: Re: Getting rid of the static msleep priority boost X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 22:41:54 -0000 On Fri, 7 Mar 2008, John Baldwin wrote: > On Friday 07 March 2008 08:42:37 am John Baldwin wrote: >> On Friday 07 March 2008 07:16:30 am Jeff Roberson wrote: >>> Hello, >>> >>> I've been studying some problems with recent scheduler improvements that >>> help a lot on some workloads and hurt on others. I've tracked the >>> problem down to static priority boosts handed out by >>> msleep/cv_broadcastpri. The basic problem is that a user thread will be >>> woken with a kernel priority thus allowing it to preempt a thread running >>> on any processor with a lesser priority. The lesser priority thread may >>> in fact hold some resource that the higher priority thread requires. >>> Thus we context switch several times and perhaps go through priority >>> propagation as well. >>> >>> I have verified that disabling these static priority boosts entirely >>> fixes the performance problem I've run into on at least one workload. >>> There are probably others that it helps and hopefully we can discover >>> that. >>> >>> I'd like to know if anyone has a strong preference to keep this feature. >>> It is likely that it helps in some interactive situations. I'm not sure >>> how much however. I propose that we make a sysctl that disables it and >>> turn it off by default. If we see complaints on current@ we can suggest >>> that they toggle the sysctl to see if it alleviates problems. >>> >>> Based on feedback from that experiment and some testing we can then >>> choose a few options: >>> >>> 1) Disable the static boosts entirely. Leave kernel priorities for >>> kernel threads and priority propagation. Most other kernels do this. >>> Would make my life in ULE much easier as well. >>> >>> 2) Leave the support for static boosts but remove it from all but a few >>> key locations. Leaving it in the api would give some flexibility but >>> might confuse developers. >>> >>> 3) Leave things as they are. undesirable. >>> >>> I'm leaning towards #2 based on the information I have presently. This >>> is almost a significant change to historic BSD behavior so we might want >>> to tread lightly. >> >> One thing to note is that we actually depend on the priority boost (evilly) >> to pick processes to swap out. (I think we check for <= PSOCK and don't >> swap those out). One thing that I've wanted to happen for a while is that >> the sleep priority for msleep() just be a parameter available to the >> scheduler that the scheduler can use to calculate the real internal >> priority rather than just being a set. That is, I imagine having: >> >> void sched_set_sleep_prio(struct thread *td, u_char pri); >> u_char sched_get_sleep_prio(struct thread *td); >> >> (The swap check would use the get call). The 4BSD scheduler's >> implementation of sched_set_sleep_prio would look like this: >> >> void >> sched_set_sleep_prio(struct thread *td, u_char pri) >> { >> >> td->td_sched->sleep_pri = pri; >> sched_prio(td, pri); >> } >> >> void >> sched_userret(..) >> { >> >> ... >> td->td_sched->sleep_pri = 0; /* not in the kernel anymore */ >> } >> >> but other schedulers may just save it and recalculate the priority where >> the priority calculation just considers the sleep priority as one among >> many factors. If nothing else, this allows it to be a scheduler decision >> to ignore it (so 4BSD could continue to do what it does now, but ULE may >> ignore it, or ignore certain levels, etc.) > > One thing to clarify: I'm not opposed to replacing the PSOCK check with > something more suitable in the swap code, (in fact, that would be desirable), > but it might take a good bit of work to do that and is probably easier to > work on that as a separate change. I also think there can be some merit in > having code paths hint to the scheduler the relative interactivity/priority > of a sleep. Couple of notes.. The priority argument to sleep is a reasonable way for the code to hint at the relative priority/interactivity. So that argues for leaving these arguments in place and making them more advisory. I don't think we have to change the api to take advantage of that. I'll look more closely for places like the swap that care about the absolute priority of a process and see what I can come up with. Thanks for raising that concern. I'd like to avoid apis that require the sched lock in seperate steps like msleep does now to elevate the priority. So far all sched* apis require the thread lock on enter and I'd hate to deviate from that norm. But another option may be just to make a globally visible td_sleep_pri that doesn't require the lock for write but does for read. The other option is to bubble the argument down through the sleepq code and into sched_sleep() and sched_wakeup(). I like that the best but it's the most api churn. Thanks, Jeff > > -- > John Baldwin > From owner-freebsd-arch@FreeBSD.ORG Fri Mar 7 22:50:55 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F483106566C; Fri, 7 Mar 2008 22:50:55 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 33CE58FC1D; Fri, 7 Mar 2008 22:50:55 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.107] (cpe-24-94-75-93.hawaii.res.rr.com [24.94.75.93]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m27MojhD042624; Fri, 7 Mar 2008 17:50:53 -0500 (EST) (envelope-from jroberson@chesapeake.net) Date: Fri, 7 Mar 2008 12:53:26 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Kirk McKusick In-Reply-To: <200803071540.m27FeSU6096030@chez.mckusick.com> Message-ID: <20080307124455.Y920@desktop> References: <200803071540.m27FeSU6096030@chez.mckusick.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: Getting rid of the static msleep priority boost X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 22:50:55 -0000 On Fri, 7 Mar 2008, Kirk McKusick wrote: >> From: John Baldwin >> To: Jeff Roberson >> Date: Fri, 7 Mar 2008 08:42:37 -0500 >> Cc: arch@freebsd.org >> Subject: Re: Getting rid of the static msleep priority boost >> >> On Friday 07 March 2008 07:16:30 am Jeff Roberson wrote: >>> Hello, >>> >>> I've been studying some problems with recent scheduler improvements that >>> help a lot on some workloads and hurt on others. I've tracked the problem >>> down to static priority boosts handed out by msleep/cv_broadcastpri. >>> ... >> >> ... >> >> This change allows the decision on priority boost to be a scheduler >> decision to ignore it (so 4BSD could continue to do what it does now, >> but ULE may ignore it, or ignore certain levels, etc.) >> >> -- >> John Baldwin > > I strongly agree with John's suggestion. The 4BSD scheduler will continue > to have its historic behavior (which was `tuned' by careful selection of > priority boosts) while more sophisticated schedulers like ULE will be able > to use/ignore the priority boosts based on their better knowledge of system > behavior. Yes, this is a good point. 4BSD is starting to fall behind featurewise on SMP but we should try to keep it in good working order as it serves as an excellent point for comparison. Eventually as systems with more than 4 cores become standard we're going to have to make some decisions regarding its long term fate. One option would be for someone to keep the 4bsd priority scheduling code and couple it with the ULE per-cpu load balancing code to keep it going. However the global run-queue is seeing the end of its days. Thanks, Jeff > > Kirk McKusick > From owner-freebsd-arch@FreeBSD.ORG Sat Mar 8 09:46:01 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C2374106566B; Sat, 8 Mar 2008 09:46:01 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 7B81B8FC12; Sat, 8 Mar 2008 09:46:01 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.107] (cpe-24-94-75-93.hawaii.res.rr.com [24.94.75.93]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m289juW5025373; Sat, 8 Mar 2008 04:45:57 -0500 (EST) (envelope-from jroberson@chesapeake.net) Date: Fri, 7 Mar 2008 23:46:32 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: John Baldwin In-Reply-To: <20080307124038.I920@desktop> Message-ID: <20080307234452.U1091@desktop> References: <20080307020626.G920@desktop> <200803070842.37248.jhb@freebsd.org> <200803071016.46944.jhb@freebsd.org> <20080307124038.I920@desktop> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-arch@freebsd.org Subject: Re: Getting rid of the static msleep priority boost X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Mar 2008 09:46:01 -0000 On Fri, 7 Mar 2008, Jeff Roberson wrote: > On Fri, 7 Mar 2008, John Baldwin wrote: > >> On Friday 07 March 2008 08:42:37 am John Baldwin wrote: >>> On Friday 07 March 2008 07:16:30 am Jeff Roberson wrote: >>>> Hello, >>>> >>>> I've been studying some problems with recent scheduler improvements that >>>> help a lot on some workloads and hurt on others. I've tracked the >>>> problem down to static priority boosts handed out by >>>> msleep/cv_broadcastpri. The basic problem is that a user thread will be >>>> woken with a kernel priority thus allowing it to preempt a thread running >>>> on any processor with a lesser priority. The lesser priority thread may >>>> in fact hold some resource that the higher priority thread requires. >>>> Thus we context switch several times and perhaps go through priority >>>> propagation as well. >>>> >>>> I have verified that disabling these static priority boosts entirely >>>> fixes the performance problem I've run into on at least one workload. >>>> There are probably others that it helps and hopefully we can discover >>>> that. >>>> >>>> I'd like to know if anyone has a strong preference to keep this feature. >>>> It is likely that it helps in some interactive situations. I'm not sure >>>> how much however. I propose that we make a sysctl that disables it and >>>> turn it off by default. If we see complaints on current@ we can suggest >>>> that they toggle the sysctl to see if it alleviates problems. >>>> >>>> Based on feedback from that experiment and some testing we can then >>>> choose a few options: >>>> >>>> 1) Disable the static boosts entirely. Leave kernel priorities for >>>> kernel threads and priority propagation. Most other kernels do this. >>>> Would make my life in ULE much easier as well. >>>> >>>> 2) Leave the support for static boosts but remove it from all but a few >>>> key locations. Leaving it in the api would give some flexibility but >>>> might confuse developers. >>>> >>>> 3) Leave things as they are. undesirable. >>>> >>>> I'm leaning towards #2 based on the information I have presently. This >>>> is almost a significant change to historic BSD behavior so we might want >>>> to tread lightly. >>> >>> One thing to note is that we actually depend on the priority boost >>> (evilly) >>> to pick processes to swap out. (I think we check for <= PSOCK and don't >>> swap those out). One thing that I've wanted to happen for a while is that >>> the sleep priority for msleep() just be a parameter available to the >>> scheduler that the scheduler can use to calculate the real internal >>> priority rather than just being a set. That is, I imagine having: >>> >>> void sched_set_sleep_prio(struct thread *td, u_char pri); >>> u_char sched_get_sleep_prio(struct thread *td); >>> >>> (The swap check would use the get call). The 4BSD scheduler's >>> implementation of sched_set_sleep_prio would look like this: >>> >>> void >>> sched_set_sleep_prio(struct thread *td, u_char pri) >>> { >>> >>> td->td_sched->sleep_pri = pri; >>> sched_prio(td, pri); >>> } >>> >>> void >>> sched_userret(..) >>> { >>> >>> ... >>> td->td_sched->sleep_pri = 0; /* not in the kernel anymore */ >>> } >>> >>> but other schedulers may just save it and recalculate the priority where >>> the priority calculation just considers the sleep priority as one among >>> many factors. If nothing else, this allows it to be a scheduler decision >>> to ignore it (so 4BSD could continue to do what it does now, but ULE may >>> ignore it, or ignore certain levels, etc.) >> >> One thing to clarify: I'm not opposed to replacing the PSOCK check with >> something more suitable in the swap code, (in fact, that would be >> desirable), >> but it might take a good bit of work to do that and is probably easier to >> work on that as a separate change. I also think there can be some merit in >> having code paths hint to the scheduler the relative interactivity/priority >> of a sleep. > > Couple of notes.. > > The priority argument to sleep is a reasonable way for the code to hint at > the relative priority/interactivity. So that argues for leaving these > arguments in place and making them more advisory. I don't think we have to > change the api to take advantage of that. > > I'll look more closely for places like the swap that care about the absolute > priority of a process and see what I can come up with. Thanks for raising > that concern. > > I'd like to avoid apis that require the sched lock in seperate steps like > msleep does now to elevate the priority. So far all sched* apis require the > thread lock on enter and I'd hate to deviate from that norm. But another > option may be just to make a globally visible td_sleep_pri that doesn't > require the lock for write but does for read. The other option is to bubble > the argument down through the sleepq code and into sched_sleep() and > sched_wakeup(). I like that the best but it's the most api churn. http://people.freebsd.org/~jeff/sleeppri.diff What do you think of this? I added another parameter to sleepq_add() and sched_sleep(). So the scheduler is responsible for adjusting the priority. We could do the same thing for wakeup time adjustments like sleepq_broadcastpri() but we'd have to pass it through setrunnable() as well. I'd like to normalize the other pri arguments in sleepq to use the same 0 is not set vs -1 that msleep did. I realize that 0 is a valid priority but for practical purposes this makes things consistent and does not really restrict the api. > > Thanks, > Jeff > >> >> -- >> John Baldwin >> > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" >