From owner-freebsd-questions@FreeBSD.ORG Thu Apr 6 08:58:50 2006 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D3E9B16A400 for ; Thu, 6 Apr 2006 08:58:50 +0000 (UTC) (envelope-from erikt@owl.midgard.homeip.net) Received: from pne-smtpout2-sn2.hy.skanova.net (pne-smtpout2-sn2.hy.skanova.net [81.228.8.164]) by mx1.FreeBSD.org (Postfix) with ESMTP id 56C7043D49 for ; Thu, 6 Apr 2006 08:58:50 +0000 (GMT) (envelope-from erikt@owl.midgard.homeip.net) Received: from falcon.midgard.homeip.net (83.253.29.241) by pne-smtpout2-sn2.hy.skanova.net (7.2.070) id 4430FC02000E3F01 for freebsd-questions@freebsd.org; Thu, 6 Apr 2006 10:58:49 +0200 Received: (qmail 60164 invoked from network); 6 Apr 2006 10:58:48 +0200 Received: from owl.midgard.homeip.net (10.1.5.7) by falcon.midgard.homeip.net with SMTP; 6 Apr 2006 10:58:48 +0200 Received: (qmail 16429 invoked by uid 1001); 6 Apr 2006 10:58:48 +0200 Date: Thu, 6 Apr 2006 10:58:48 +0200 From: Erik Trulsson To: Karl Ma Message-ID: <20060406085848.GA16393@owl.midgard.homeip.net> Mail-Followup-To: Karl Ma , freebsd-questions@freebsd.org References: <42a5f58c0604060143g5e68e806r870761a1beeefbb5@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <42a5f58c0604060143g5e68e806r870761a1beeefbb5@mail.gmail.com> User-Agent: Mutt/1.5.11 Cc: freebsd-questions@freebsd.org Subject: Re: Giving more CPU time to a swapping process? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Apr 2006 08:58:50 -0000 On Thu, Apr 06, 2006 at 04:43:42PM +0800, Karl Ma wrote: > Hi, > > I have a python program in freebsd, doing a heavey indexing job involving a > mega size array. > > The process is so memory-hungry that it starts swap after the physical RAM > max out. (To be exact, I've lowered the per-process limitation to make this > possible). > > However, when I use top to monitor the status, the STATE of the process > started to stay as "swread" for most of the time (instead of RUN before > using swap) and its priority has dropped to -20; and the corresponding WCPU > drops to around 1% only. And the CPU consumption time in total (for the > whole job) would only increase a minute or two even the process has been > running for more than a few hours. "swread" means that the process is busy waiting for information to be read from (or possibly written to) the swap space. The process cannot use any actual CPU time until it has gotten that information from swap. It sounds like the process has entered that condition generally known as "thrashing" where it spends most of the time swapping instead of doing useful work. > In Windows XP, which has less per-task resource restriction (I guess?), I > did successfully complete the task on the same hardware machine; although it > takes more than 30 mins. Apparently the algorithms Windows XP uses to manage swap are a better fit for this particular programs memory usage pattern than those of FreeBSD (or you might run fewer other programs in parallell of XP, leaving more free memory for this particular program.) It is quite likely that for some other programs you would see the reverse situation with XP being much slower. > > How can I push up the priority of the whole paging task? How can I > allocate more CPU attention to this process? I've tried using "nice" > but it does not help. That won't help. You need to add more RAM (or decrease the memory usage of the program, or remove any other memory-hungry programs running at the same time, or change the memory access pattern of the program so that it has greater reference locality so it does not need to swap as often.) > > Please refer to the below top's snapshots at different times. (A) is earlier > than (B), and so on. > > Thanks for your help in advance. > > > = (A) = > last pid: 766; load averages: 0.66, 0.21, 0.11 up 0+00:19:54 > 23:23:04 > 63 processes: 2 running, 61 sleeping > CPU states: 96.6% user, 0.0% nice, 3.4% system, 0.0% interrupt, 0.0%idle > Mem: 341M Active, 35M Inact, 98M Wired, 704K Cache, 91M Buf, 522M Free > Swap: 998M Total, 998M Free > > PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND > 751 root 1 118 0 245M 244M RUN 0:59 96.57% python > > = (B) = > last pid: 792; load averages: 1.02, 0.70, 0.36 up 0+00:24:20 > 23:27:30 > 62 processes: 2 running, 60 sleeping > CPU states: 92.5% user, 0.0% nice, 6.0% system, 1.5% interrupt, 0.0%idle > Mem: 766M Active, 67M Inact, 115M Wired, 45M Cache, 109M Buf, 3636K Free > Swap: 998M Total, 37M Used, 962M Free, 3% Inuse, 17M Out > > PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND > 751 root 1 128 0 744M 743M RUN 5:14 94.63% python > > = (C) = > last pid: 792; load averages: 1.06, 0.81, 0.43 up 0+00:25:54 > 23:29:04 > 62 processes: 2 running, 60 sleeping > CPU states: 95.5% user, 0.0% nice, 4.1% system, 0.4% interrupt, 0.0%idle > Mem: 849M Active, 2868K Inact, 115M Wired, 28M Cache, 109M Buf, 1656K Free > Swap: 998M Total, 95M Used, 903M Free, 9% Inuse, 236K Out > > PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND > 751 root 1 128 0 848M 841M RUN 6:42 91.46% python > > = (D) = > last pid: 792; load averages: 1.15, 0.87, 0.47 up 0+00:26:36 > 23:29:46 > 62 processes: 1 running, 61 sleeping > CPU states: 28.6% user, 0.0% nice, 18.8% system, 1.5% interrupt, 51.1%idle > Mem: 843M Active, 5380K Inact, 116M Wired, 31M Cache, 109M Buf, 1656K Free > Swap: 998M Total, 142M Used, 856M Free, 14% Inuse, 932K In, 56M Out > > PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND > 751 root 1 -20 0 878M 841M swread 7:14 79.05% python > > = (E) = > last pid: 817; load averages: 0.06, 0.36, 0.36 up 0+00:32:31 > 23:35:41 > 62 processes: 1 running, 61 sleeping > CPU states: 4.9% user, 0.0% nice, 7.1% system, 0.0% interrupt, 88.0%idle > Mem: 803M Active, 76M Inact, 88M Wired, 28M Cache, 109M Buf, 1656K Free > Swap: 998M Total, 234M Used, 765M Free, 23% Inuse, 3148K In, 2284K Out > > PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND > 751 root 1 -20 0 915M 756M swread 8:01 1.03% python -- Erik Trulsson ertr1013@student.uu.se