Date: Wed, 6 Feb 2013 17:15:09 -0600 From: Dustin Wenz <dustinwenz@ebureau.com> To: freebsd-stable@freebsd.org Subject: I/O hanging while hosting Postgres database Message-ID: <B0A2F9DE-EC90-441B-AF82-85545BD341EF@ebureau.com>
next in thread | raw e-mail | index | archive | help
I'm seeing a condition on FreeBSD 9.1 (built October 24th) where I/O = seems to hang on any local zpools after several hours of hosting a = large-ish Postgres database. The database occupies about 14TB of a 38TB = zpool with a single SSD ZIL. The OS is on a ZFS boot disk. The system = also has 24GB of physical memory. Smartmon tools reports no errors on = any disks attached to the system, and IPMI reports all temperatures, CPU = voltages, and fan speeds are normal. The database has been gradually increasing in size since it was first = deployed on FreeBSD 9.1 this fall. There were no problems until last = night, when the database became unresponsive. Attempts to interact with = the shell would block (specifically, any interaction with the disk), and = no error messages were logged to the console. I restarted the system at = that time, and brought the database back up. Everything seemed normal = until this morning, where the database had become unresponsive again. = Fortunately, I was able to grab some system statistics before the shell = and console went AWOL. The only finding that I thought was suspicious were the kmem_map = numbers: vm.kmem_map_free: 655360 vm.kmem_map_size: 17141383168 It's something like 0.004% free. I haven't been able to find much = documentation on what to expect here, but I don't see anything like that = for other databases that I've monitored. It is possible that kmem_map = can become exhausted without generating a kernel panic? Could it be = indicative of severe memory fragmentation? - .Dustin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B0A2F9DE-EC90-441B-AF82-85545BD341EF>