Date: Wed, 30 Mar 2011 19:07:15 -0700 (PDT) From: Luke Dean <LukeD@pobox.com> To: Luke Dean <LukeD@pobox.com> Cc: freebsd-questions@freebsd.org Subject: Re: kern.geom.journal.stats.low_mem refers to what? Message-ID: <alpine.BSF.2.00.1103301830050.36948@tower.lukas.is-a-geek.org> In-Reply-To: <alpine.BSF.2.00.1103271901080.16486@tower.lukas.is-a-geek.org> References: <alpine.BSF.2.00.1103271901080.16486@tower.lukas.is-a-geek.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> I've recently set up gjournal on top of gmirror on FreeBSD 8.2. I understand > that this setup has a lot of redundant writing. It is working, but I'm not > sure I've set it up as efficiently as I should. > > During prolonged writes, such as copying large files to the file system > across the network or producing very large logfiles, the "low_mem" and > "skipped_bytes" statisics rise rapidly and the system becomes less > responsive. "top" always reports free memory, so I don't think that's the > issue. "journal_full" and "wait_for_copy" have never occurred. Here's a > sample of what happens after a couple hours of intense writing... > > kern.geom.journal.stats.low_mem: 5379 > kern.geom.journal.stats.journal_full: 0 > kern.geom.journal.stats.wait_for_copy: 0 > kern.geom.journal.stats.switches: 7543 > kern.geom.journal.stats.combined_ios: 265318 > kern.geom.journal.stats.skipped_bytes: 935712768 > > "low_mem" sounds like a bad thing. What could I do to remedy that? Did I > make the journal too small? The stats say that "journal_full" has never > happened, so maybe not? Is there a setting I should tweak? > > The Handbook says 1GB is good enough most of the time, but it also says that > 3x the amount of physical memory is a good size as well. I compromised > between the two and made an 8GB journal for this system that has ~4GB of > memory. Replying to myself for posterity... I think I figured it out. I believe "low_mem" means its running out of gjournal cache, defined by kern.geom.journal.cache.limit, which is initially set by kern.geom.journal.cache.divisor, which is initially 2, which means half of the kernel memory, which is vm.kmem_size, which is 330MB on my i386 8.2 system by default. I don't know where this default comes from, but it seems common. This gives me about 167MB of cache for journalling. During big file transfers, I can watch the "journal_data" statistic in "vmstat -m". It climbs close to the 167MB cache size. That's when I see kern.geom.journal.stats.low_mem tick over and the stutter happens. So I think I understand what I'm seeing now. I could try increasing vm.kmem_size and possibly the KMEM_SIZE option that's compiled into my kernel as described in the ZFS tuning documents, since it's a similar issue. This would allow the kernel to have more memory and thus allow gjournal to use more memory for caching. If I'm consistently filling the cache faster than I can empty it, making the cache bigger probably won't help. This journal is on top of a mirror, so I'm always expecting it to read faster than it writes... Copying large files locally might always max out the buffer? No harm in trying, I guess. Or I could just live with it. It's doing what I told it to do.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.1103301830050.36948>