From owner-freebsd-stable@FreeBSD.ORG Wed Jun 29 22:33:13 2005 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B7EFD16A41C for ; Wed, 29 Jun 2005 22:33:13 +0000 (GMT) (envelope-from sven@dmv.com) Received: from smtp-gw-cl-c.dmv.com (smtp-gw-cl-c.dmv.com [216.240.97.41]) by mx1.FreeBSD.org (Postfix) with ESMTP id 69E6843D48 for ; Wed, 29 Jun 2005 22:33:13 +0000 (GMT) (envelope-from sven@dmv.com) Received: from lanshark.dmv.com (lanshark.dmv.com [216.240.97.46]) by smtp-gw-cl-c.dmv.com (8.12.10/8.12.10) with ESMTP id j5TMX8gr065785; Wed, 29 Jun 2005 18:33:08 -0400 (EDT) (envelope-from sven@dmv.com) From: Sven Willenberger To: Charles Swiger In-Reply-To: <1120078715.19598.79.camel@lanshark.dmv.com> References: <1120050088.19603.7.camel@lanshark.dmv.com> <58067976-00C6-4380-90DF-F448D9008C81@khera.org> <1120075946.19603.68.camel@lanshark.dmv.com> <51C9ABA2-20B8-4DE8-B647-A26A168A78FC@mac.com> <1120078715.19598.79.camel@lanshark.dmv.com> Content-Type: text/plain Date: Wed, 29 Jun 2005 18:34:01 -0400 Message-Id: <1120084441.19614.87.camel@lanshark.dmv.com> Mime-Version: 1.0 X-Mailer: Evolution 2.2.1.1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.39 Cc: Vivek Khera , stable@freebsd.org, postgres general Subject: Re: [GENERAL] PostgreSQL's vacuumdb fails to allocate memory for X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jun 2005 22:33:13 -0000 On Wed, 2005-06-29 at 16:58 -0400, Sven Willenberger wrote: > On Wed, 2005-06-29 at 16:40 -0400, Charles Swiger wrote: > > On Jun 29, 2005, at 4:12 PM, Sven Willenberger wrote: > > [ ... ] > > > Something I have noticed, > > > when the memory error occurs during the psql session (after a failed > > > vacuumdb attempt) the memory stays at 600+MB in top (under size) until > > > the psql session is closed -- that may just be the way top reports it > > > though. > > > > Double-check your system limits via "ulimit -a" or "ulimit -aH". By > > default, FreeBSD will probably restrict the maximum data size of the > > process to 512MB, which may be what you are running into. You can > > rebuild the kernel to permit a larger data size, or else tweak /boot/ > > loader.conf: > > > > echo 'kern.maxdsiz="1024M"' >> /boot/loader.conf > > > > :>ulimit -a > cpu time (seconds, -t) unlimited > file size (512-blocks, -f) unlimited > data seg size (kbytes, -d) 1048576 > stack size (kbytes, -s) 65536 > core file size (512-blocks, -c) unlimited > max memory size (kbytes, -m) unlimited > locked memory (kbytes, -l) unlimited > max user processes (-u) 5547 > open files (-n) 11095 > virtual mem size (kbytes, -v) unlimited > sbsize (bytes, -b) unlimited > :> cat /boot/loader.conf > kern.maxdsiz="1073741824" > kern.dfldsiz="891289600" > > and if I don't run vacuumdb at all, but rather connect to the backend > via psql and run vacuum, it works ok with full memory allocation. Still > testing RAM to see if the issue is physical. > > Sven > > I have found the answer/problem. On a hunch I increased maxdsiz to 1.5G in the loader.conf file and rebooted. I ran vacuumdb and watched top as the process proceeded. What I saw was SIZE sitting at 603MB (which was 512MB plus another 91MB which corresponded nicely to the value of RES for the process. A bit into the process I saw SIZE jump to 1115 -- i.e. another 512 MB of RAM was requested and this time allocated. At one point SIZE dropped back to 603 and then back up to 1115. I suspect the same type of issue was occuring in regular vacuum from the psql client connecting to the backend, for some reason not as frequently. I am gathering that maintenance work mem is either not being recognized as having already been allocated and another malloc is made or the process is thinking the memory was released and tried to grab a chunk of memory again. This would correspond to the situation where I was size stuck at 603MB after a failed memory allocation (when maxdsiz was only 1G). Now I am not sure if I will run into the situation where yet another 512MB request would be made (when already 1115 appears in SIZE) but if so, then the same problem will arise. I will keep an eye on it ... Sven