From owner-freebsd-questions@FreeBSD.ORG Thu Mar 31 02:07:26 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF4AF1065672 for ; Thu, 31 Mar 2011 02:07:25 +0000 (UTC) (envelope-from LukeD@pobox.com) Received: from sasl.smtp.pobox.com (a-pb-sasl-sd.pobox.com [64.74.157.62]) by mx1.freebsd.org (Postfix) with ESMTP id B13258FC14 for ; Thu, 31 Mar 2011 02:07:25 +0000 (UTC) Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTP id 889E3478F; Wed, 30 Mar 2011 22:09:12 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=date:from :reply-to:to:cc:subject:in-reply-to:message-id:references :mime-version:content-type; s=sasl; bh=AX4hbYMtUmB6n+ICHbk5dDr87 dc=; b=EbrdBoj/ndPN0B2apnNfVWtMtC/UudSNzMn03nfLspEnSexnw9jLDmXS1 WOCPMgTyi7+3d/9F1gsNs/zVyQZJNdPe6px17jTs5htHi5FCIVUECyPSbVhWkdpU 6k7oAt/4KMDjpODZSp3lQEOpOPbZce81yQeDi1V3iMedXP9/K4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=date:from :reply-to:to:cc:subject:in-reply-to:message-id:references :mime-version:content-type; q=dns; s=sasl; b=LuW/YCVBNNRDvwTYpOx H9LySyJ9loqB5CwtnHrZ11W684UwLPhgrGlDyt2T1RmWIJ6P8GbU/jIDvwQ/qWSY DkF6L2jgiIzKKHTe/shG8RuwtELrP21t4hbtoRHvVy34Kgu3OkhsDJ6m+scLXPRL oVt8QGCQsqhHCgGIiMlD+oCU= Received: from a-pb-sasl-sd.pobox.com (unknown [127.0.0.1]) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTP id 765F8478B; Wed, 30 Mar 2011 22:09:10 -0400 (EDT) Received: from [192.168.10.100] (unknown [50.35.181.63]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTPSA id BCFD44789; Wed, 30 Mar 2011 22:09:08 -0400 (EDT) Date: Wed, 30 Mar 2011 19:07:15 -0700 (PDT) From: Luke Dean X-X-Sender: lukas@tower.lukas.is-a-geek.org To: Luke Dean In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Pobox-Relay-ID: DDD378F8-5B3B-11E0-AA09-E8AB60295C12-96347044!a-pb-sasl-sd.pobox.com Cc: freebsd-questions@freebsd.org Subject: Re: kern.geom.journal.stats.low_mem refers to what? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Luke Dean List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2011 02:07:26 -0000 > I've recently set up gjournal on top of gmirror on FreeBSD 8.2. I understand > that this setup has a lot of redundant writing. It is working, but I'm not > sure I've set it up as efficiently as I should. > > During prolonged writes, such as copying large files to the file system > across the network or producing very large logfiles, the "low_mem" and > "skipped_bytes" statisics rise rapidly and the system becomes less > responsive. "top" always reports free memory, so I don't think that's the > issue. "journal_full" and "wait_for_copy" have never occurred. Here's a > sample of what happens after a couple hours of intense writing... > > kern.geom.journal.stats.low_mem: 5379 > kern.geom.journal.stats.journal_full: 0 > kern.geom.journal.stats.wait_for_copy: 0 > kern.geom.journal.stats.switches: 7543 > kern.geom.journal.stats.combined_ios: 265318 > kern.geom.journal.stats.skipped_bytes: 935712768 > > "low_mem" sounds like a bad thing. What could I do to remedy that? Did I > make the journal too small? The stats say that "journal_full" has never > happened, so maybe not? Is there a setting I should tweak? > > The Handbook says 1GB is good enough most of the time, but it also says that > 3x the amount of physical memory is a good size as well. I compromised > between the two and made an 8GB journal for this system that has ~4GB of > memory. Replying to myself for posterity... I think I figured it out. I believe "low_mem" means its running out of gjournal cache, defined by kern.geom.journal.cache.limit, which is initially set by kern.geom.journal.cache.divisor, which is initially 2, which means half of the kernel memory, which is vm.kmem_size, which is 330MB on my i386 8.2 system by default. I don't know where this default comes from, but it seems common. This gives me about 167MB of cache for journalling. During big file transfers, I can watch the "journal_data" statistic in "vmstat -m". It climbs close to the 167MB cache size. That's when I see kern.geom.journal.stats.low_mem tick over and the stutter happens. So I think I understand what I'm seeing now. I could try increasing vm.kmem_size and possibly the KMEM_SIZE option that's compiled into my kernel as described in the ZFS tuning documents, since it's a similar issue. This would allow the kernel to have more memory and thus allow gjournal to use more memory for caching. If I'm consistently filling the cache faster than I can empty it, making the cache bigger probably won't help. This journal is on top of a mirror, so I'm always expecting it to read faster than it writes... Copying large files locally might always max out the buffer? No harm in trying, I guess. Or I could just live with it. It's doing what I told it to do.