Date: Mon, 10 Feb 2003 14:39:04 -0800 From: Alfred Perlstein <bright@mu.org> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: Morten Rodal <morten@rodal.no>, phk@phk.freebsd.dk, David Schultz <dschultz@uclink.Berkeley.EDU>, arch@FreeBSD.ORG Subject: Re: Our lemming-syncer caught in the act. Message-ID: <20030210223904.GG88781@elvis.mu.org> In-Reply-To: <200302102225.h1AMPkTE023700@apollo.backplane.com> References: <20030210091317.GD5165@HAL9000.homeunix.com> <37473.1044868995@critter.freebsd.dk> <20030210204523.GF12240@slurp.rodal.no> <20030210205443.GA88781@elvis.mu.org> <200302102225.h1AMPkTE023700@apollo.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
* Matthew Dillon <dillon@apollo.backplane.com> [030210 14:25] wrote:
>
> Try reproducing the problem with kernel profiling turned on. The
> syncer is not likely to be the cause of the large file removal issue,
> since that involves only bitmaps. The cause is more likely to be
> softupdate's bitmap dependancies. Try turning off softupdates and see
> if that improves performance. Softupdates creates some rather nasty
> buffer cache issues due to the number of buffers it removes from
> management. Also, dependancies prevent the buffer cache from being
> able to flush a buffer. The buffer cache was not really designed to
> deal with flushes which don't undirty a buffer or large numbers of
> buffers being temporarily removed from management.
>
> The syncer is more likely to be the cause of performance issues
> during large file data operations. Because the syncer holds onto the
> vnode lock you can get into a situation where lookup()s cause a directory
> locking chain to reach the root directory which has the effect of stalling
> just about everything. Either Net or OpenBSD solved this problem by
> doing partial fsyncs instead of full fsyncs to avoid holding onto the
> vnode lock for insanely long periods of time. Another solution would
> be to use the namei cache to lock path names from create/remove/rename
> ops so the directory vnodes do not have to be locked for the duration
> of a path lookup. If the directory vnode is not exclusively locked then
> a race-to-root will not happen due to some random file vnode being locked
> for a long period of time.
There are several good points in your mail, especially about performing
partial fsyncs, however this doesn't really explain why 4.x is "ok" versus
5.x having issues. The only explanation I can think of is that the syncer
runs with giant for a long time but under 4.x only needs splbio() for short
periods.
I don't have a 5.x box handy here, but if someone could run this patch
and let me know what it does I would appreciate it.
cd /usr/src/sys/kern && patch < path_to_this_file
cvs server: Diffing .
Index: vfs_subr.c
===================================================================
RCS file: /home/ncvs/src/sys/kern/vfs_subr.c,v
retrieving revision 1.427
diff -u -r1.427 vfs_subr.c
--- vfs_subr.c 21 Jan 2003 08:55:55 -0000 1.427
+++ vfs_subr.c 10 Feb 2003 22:38:31 -0000
@@ -1725,6 +1725,11 @@
VOP_UNLOCK(vp, 0, td);
vn_finished_write(mp);
}
+#ifdef other_premption_did_not_work
+ /* XXX: pre-emption point to avoid Giant hogging. */
+ mtx_unlock(&Giant);
+ mtx_lock(&Giant);
+#endif
s = splbio();
mtx_lock(&sync_mtx);
if (LIST_FIRST(slp) == vp) {
@@ -1775,6 +1780,9 @@
if (rushjob > 0) {
rushjob -= 1;
mtx_unlock(&sync_mtx);
+ /* XXX: pre-emption point to avoid Giant hogging. */
+ mtx_unlock(&Giant);
+ mtx_lock(&Giant);
continue;
}
mtx_unlock(&sync_mtx);
--
-Alfred Perlstein [alfred@freebsd.org]
'Instead of asking why a piece of software is using "1970s technology,"
start asking why software is ignoring 30 years of accumulated wisdom.'
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030210223904.GG88781>