Date: Tue, 9 Dec 2003 13:38:34 -0500 (EST) From: Robert Watson <rwatson@freebsd.org> To: Fabian Thylmann <fthylmann@stats.net> Cc: freebsd-questions@freebsd.org Subject: Re: inode state Message-ID: <Pine.NEB.3.96L.1031209133446.21344H-100000@fledge.watson.org> In-Reply-To: <01b001c3be55$e932bd20$2102a8c0@NoteFabi>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 9 Dec 2003, Fabian Thylmann wrote: > I have a heavily used threaded server program running on one of my Dell > Poweredge 1750 servers. Its a statistical analysis package for websites. > Currently it analyses over 60 million requests a day, which (because of > many different reasons) causes it to handle around 120 million http > requests a day. At peaks around 1500 requests a second. > > The system stores most many statistics in memory which is flushed to > disk in circles by a worker thread. > > Another big part is stored in an on-disk database which is mmap()'d into > memory. Because we do not have enough memory to keep everything in > memory at one time the mmap() system of course pages data in and out. > > When I look in systat -v I see that dirtybuf climbs to about 1700 and > then they get flushed to disk, causing high disk usage of around 300-400 > tps whcih renders the disks useless for anything else. > > When those flushes occure, my apps state as displayed by top(1) gets > into inode state, PRI is set to -14 and cpu usage rapidly drops. The > program and ALL of its threads are stalled at that time. Those inode > states take around 2 oe 3 seconds and happen every 30 seconds or so. > > In those 3 seconds we lose around 1500 hits at peak times for processing > because the app can not handle them fast enough. This results in around > 2 million or so hits lost over the day for processing. > > I am now wondering if anyone can explain to me why ALL threads and not > just the threads that actually do I/O work get blocked when dirty > buffers are flushed and what to do to fix this problem. > > I would be very happy if someone could reply and point me into the right > direction! You don't mention which version of FreeBSD you're running -- if 4.x, you probably want to relink your application against the "linuxthreads" port. This is because libc_r implements threads inside a single process without the support of the kernel, which means that if the process is blocked in kernel, all threads will be blocked in kernel. The linuxthreads package uses a model similar to Linux's threading implementation (hence the name) to allow the threads to be scheduled using lightweight versions of processes (shared file descriptors, etc). This isn't quite POSIX-compliant, but it works quite well for disk-bound applications such as databases. If you're running on 5.x, especially recent 5.1 or 5.2 prereleases, you probably want to give libkse a try. It's the new m:n threading implementation that will become the default in 5.3, and also permits parallelism (only in a more POSIX-compliant way, and in theory offering much greater scalability for large numbers of threads). I stick the following lines in my /etc/libmap.conf on 5.x boxes to force all applications linked against libc_r to use libkse instead: libc_r.so.5 libkse.so.1 libc_r.so libkse.so One particularly nice thing about the m:n thread support is that you can run-time plug the thread library between several options (libc_r, libthr, libkse) to pick the one that performs best for your application. Another benefit of running with a non-libc_r threads package is that if you have an SMP box, you'll see real parallelism. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.NEB.3.96L.1031209133446.21344H-100000>