From owner-freebsd-questions@FreeBSD.ORG Tue Mar 22 15:37:55 2011 Return-Path: Delivered-To: questions@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 1233) id E10D11065670; Tue, 22 Mar 2011 15:37:55 +0000 (UTC) Date: Tue, 22 Mar 2011 15:37:55 +0000 From: Alexander Best To: Laszlo Nagy Message-ID: <20110322153755.GA40457@freebsd.org> References: <4D870DCD.9000404@shopzeus.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D870DCD.9000404@shopzeus.com> Cc: questions@freebsd.org, Daniel Fekete Subject: Re: fstat bug? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Mar 2011 15:37:56 -0000 On Mon Mar 21 11, Laszlo Nagy wrote: > > Hi All, > > I have a Python program that goes up to 100% CPU. Just like this (top): you might want to re-post this message to freebsd-hackers@. in my experience freebsd-questions@ is suited for user-related questions and not that much for developers who seek answers to very techie questions. cheers. alex > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 80212 user1 2 44 0 70520K 16212K select 1 0:30 100.00% > /usr/local/bin/python process_updates_ss_od.py -l 10 > > I have added extra logs and it turns out that there are two threads. One > thread is calling "time.sleep()" and the other is calling "os.stat" > call. (Actually it is calling os.path.isfile, but I hunted down the last > link in the chain.) The most interesting thing is that the process is in > "SELECT" state. As far as I know, CPU load should be 0% because "select" > state should block program execution until the I/O completes. > > I must also tell you that the os.stat call is taking long because this > system has about 7 million files on a slow disk under heavy load. It > would be normal for an os.stat call to return after 10 seconds. I have > no problem with that. But I think that the 100% CPU is not acceptable. I > guess that the code is running a system call in kernel mode. I think > this because I can send a KILL signal to it and the state changes to the > following: > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 80212 user1 2 44 0 70520K 15256K STOP 5 1:27 100.00% > /usr/local/bin/python process_updates_ss_od.py -l 10 > > So the state of the process changes to "STOP", but the program does not > stop until the os.stat call returns back. Sometimes for a minute. > > Could it be a problem with the operation system? Is it possible that an > os.stat call requires 100% CPU power from the OS? (I believe that > os.stat ends in fstat()). Or is it a problem with the Python implementation? > > (Unfortunately I cannot give you an example program. Giving an example > would require giving you a slow I/O device with millions of files on it.) > > OS version: FreeBSD 8.1-STABLE amd64 > Python version: 2.6.6 > > Thanks, > > Laszlo > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > -- a13x