From owner-freebsd-current@FreeBSD.ORG Sun Oct 29 04:00:09 2006 Return-Path: X-Original-To: current@freebsd.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7413216A4C8; Sun, 29 Oct 2006 04:00:09 +0000 (UTC) (envelope-from prvs=julian=4503dd1c6@elischer.org) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 249D643D5A; Sun, 29 Oct 2006 04:00:04 +0000 (GMT) (envelope-from prvs=julian=4503dd1c6@elischer.org) Received: from unknown (HELO [192.168.2.5]) ([10.251.60.35]) by a50.ironport.com with ESMTP; 28 Oct 2006 21:00:04 -0700 Message-ID: <45442744.6080103@elischer.org> Date: Sat, 28 Oct 2006 21:00:04 -0700 From: Julian Elischer User-Agent: Thunderbird 1.5.0.7 (Macintosh/20060909) MIME-Version: 1.0 To: "Alexandre \"Sunny\" Kovalenko" References: <45425D92.8060205@elischer.org> <20061027201838.GH30707@riyal.ugcs.caltech.edu> <1161998104.872.18.camel@RabbitsDen.RabbitsLawn.verizon.net> <4542B171.8050601@elischer.org> <1161999387.872.29.camel@RabbitsDen.RabbitsLawn.verizon.net> <4542D3A8.1040500@elischer.org> <1162059531.872.45.camel@RabbitsDen.RabbitsLawn.verizon.net> In-Reply-To: <1162059531.872.45.camel@RabbitsDen.RabbitsLawn.verizon.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Daniel Eischen , Paul Allen , current@freebsd.org Subject: Re: Comments on the KSE option X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Oct 2006 04:00:09 -0000 Alexandre "Sunny" Kovalenko wrote: > On Fri, 2006-10-27 at 20:51 -0700, Julian Elischer wrote: >> Alexandre "Sunny" Kovalenko wrote: >>> On Fri, 2006-10-27 at 18:25 -0700, Julian Elischer wrote: >>>> Alexandre "Sunny" Kovalenko wrote: >>>>> On Fri, 2006-10-27 at 16:41 -0400, Daniel Eischen wrote: >>>>>> On Fri, 27 Oct 2006, Paul Allen wrote: >>>>>> >>>>>>>> From Julian Elischer , Fri, Oct 27, 2006 at 12:27:14PM -0700: >>>>>>>> The aim of the fair scheduling code is to ensure that if you, as a user, >>>>>>>> make a process that starts 1000 threads, and I as a user, make an >>>>>>>> unthreaded process, then I can still get to the CPU at somewhat similar >>>>>>>> rates to you. A naive scheduler would give you 1000 cpu slots and me 1. >>>>>>> Ah. Let me be one of the first to take a crack at attacking this idea as >>>>>>> a mistake. >>>>>> No, it is POSIX. You, the application, can write a program with >>>>>> system scope or process scope threads and get whatever you behavior >>>>>> you want, within rlimits of course. >>>>>> >>>>>> If you want unfair scheduling, then create your threads with >>>>>> system scope contention, otherwise use process scope. The >>>>>> kernel should be designed to allow both, and have adjustable >>>>>> limits in place for (at least) system scope threads. >>>>>> >>>>>> Noone is saying that you can't have as many system scope threads >>>>>> as you want (and as allowed by limits), just that you must also >>>>>> be able to have process scope threads (with probably higher limits >>>>>> or possibly no limits). >>>>>> >>>>> I might be missing something here, but OP was separating M:N (which is >>>>> what you are referring to above), and "fairness" (not giving process >>>>> with 1000 *system scope* threads 1000 CPU scheduling slots). As far as I >>>>> know the first one is POSIX and the second one is not. >>>>> >>>>> FWIW: as an application programmer who spent considerable amount of time >>>>> lately trying to make heavily multithreaded application run most >>>>> efficiently on 32-way machine, I would rather not have to deal with >>>>> "fairness" -- M:N is bad enough. >>>>> >>>> no, fairness is making sure that 1000 process scope threads >>>> do not negatively impact other processes. >>>> 1000 system scope threads are controlled by your ulimit settings >>>> (Each one counts as a process.) >>>> >>>> >>> I apologize for misinterpreting your words. But then, if I have M:N set >>> to 10:1, I would expect application with 1000 process scope threads to >>> have as many CPU slots as 100 processes, or, if I have 10 system scope >>> threads and 990 process scope threads, I would expect application to >>> have as many CPU slots as 109 processes. Is this what you refer to as >>> "fairness"? >>> >> M:N is not a ratio, but rather the notation to say that M user threads >> are enacted using N kernel schedulable entities (kernel threads). >> usually N is limited to something like NCPU kernel schedulable entities >> running at a time. (not including sleeping threads waiting for IO) >> (NCPU is the number of CPUs). >> >> so in fact M:N is usually M user threads over over some number like 4 or >> 8 kernel threads (depending on #cpus) plus the number of threads waiting >> for IO. >> >> Julian > In the environment I am most familiar with -- IBM AIX -- M:N is the > ratio and it is settable either systemwide or for the specific user > using environment variable, e.g.: > > export AIXTHREAD_MNRATIO=8:1 > > with the minimum kernel threads allocated according to another setting: > > export AIXTHREAD_MINKTHREADS=4 > > Neither one depends on the physical CPU count in the box (which could > change in the middle of application execution anyway). > > Both settings have known default values (8:1 and 8 respectively). > > Between the two I can always tell how many kernel threads given amount > of the process scope threads will use. This gives me both flexibility > and predictability. > > Am I understanding correctly that what you have implemented fixes number > of kernel threads at boot time and changes the M:N ratio throughout the > run time of the application? no. the "number of available slots" for concurency is a number that is tunable on the fly. I think we may even be able to tune it in a program but I would have to look to see if that ever got committed. >