Date: Tue, 9 Dec 2003 14:11:07 +0100 From: "Fabian Thylmann" <fthylmann@stats.net> To: <freebsd-questions@freebsd.org> Subject: inode state Message-ID: <01b001c3be55$e932bd20$2102a8c0@NoteFabi>
next in thread | raw e-mail | index | archive | help
Hi, I have a heavily used threaded server program running on one of my Dell = Poweredge 1750 servers. Its a statistical analysis package for websites. = Currently it analyses over 60 million requests a day, which (because of = many different reasons) causes it to handle around 120 million http = requests a day. At peaks around 1500 requests a second. The system stores most many statistics in memory which is flushed to = disk in circles by a worker thread. Another big part is stored in an on-disk database which is mmap()'d into = memory. Because we do not have enough memory to keep everything in = memory at one time the mmap() system of course pages data in and out. When I look in systat -v I see that dirtybuf climbs to about 1700 and = then they get flushed to disk, causing high disk usage of around 300-400 = tps whcih renders the disks useless for anything else. When those flushes occure, my apps state as displayed by top(1) gets = into inode state, PRI is set to -14 and cpu usage rapidly drops. The = program and ALL of its threads are stalled at that time. Those inode = states take around 2 oe 3 seconds and happen every 30 seconds or so. In those 3 seconds we lose around 1500 hits at peak times for processing = because the app can not handle them fast enough. This results in around = 2 million or so hits lost over the day for processing. I am now wondering if anyone can explain to me why ALL threads and not = just the threads that actually do I/O work get blocked when dirty = buffers are flushed and what to do to fix this problem. I would be very happy if someone could reply and point me into the right = direction! Fabian Thylmann
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?01b001c3be55$e932bd20$2102a8c0>