Date: Tue, 22 Mar 2011 15:37:55 +0000 From: Alexander Best <arundel@freebsd.org> To: Laszlo Nagy <gandalf@shopzeus.com> Cc: questions@freebsd.org, Daniel Fekete <dani@shopzeus.com> Subject: Re: fstat bug? Message-ID: <20110322153755.GA40457@freebsd.org> In-Reply-To: <4D870DCD.9000404@shopzeus.com> References: <4D870DCD.9000404@shopzeus.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon Mar 21 11, Laszlo Nagy wrote: > > Hi All, > > I have a Python program that goes up to 100% CPU. Just like this (top): you might want to re-post this message to freebsd-hackers@. in my experience freebsd-questions@ is suited for user-related questions and not that much for developers who seek answers to very techie questions. cheers. alex > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 80212 user1 2 44 0 70520K 16212K select 1 0:30 100.00% > /usr/local/bin/python process_updates_ss_od.py -l 10 > > I have added extra logs and it turns out that there are two threads. One > thread is calling "time.sleep()" and the other is calling "os.stat" > call. (Actually it is calling os.path.isfile, but I hunted down the last > link in the chain.) The most interesting thing is that the process is in > "SELECT" state. As far as I know, CPU load should be 0% because "select" > state should block program execution until the I/O completes. > > I must also tell you that the os.stat call is taking long because this > system has about 7 million files on a slow disk under heavy load. It > would be normal for an os.stat call to return after 10 seconds. I have > no problem with that. But I think that the 100% CPU is not acceptable. I > guess that the code is running a system call in kernel mode. I think > this because I can send a KILL signal to it and the state changes to the > following: > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 80212 user1 2 44 0 70520K 15256K STOP 5 1:27 100.00% > /usr/local/bin/python process_updates_ss_od.py -l 10 > > So the state of the process changes to "STOP", but the program does not > stop until the os.stat call returns back. Sometimes for a minute. > > Could it be a problem with the operation system? Is it possible that an > os.stat call requires 100% CPU power from the OS? (I believe that > os.stat ends in fstat()). Or is it a problem with the Python implementation? > > (Unfortunately I cannot give you an example program. Giving an example > would require giving you a slow I/O device with millions of files on it.) > > OS version: FreeBSD 8.1-STABLE amd64 > Python version: 2.6.6 > > Thanks, > > Laszlo > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > -- a13x
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110322153755.GA40457>