From owner-freebsd-stable@FreeBSD.ORG Tue Dec 10 10:46:27 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04DA01A4 for ; Tue, 10 Dec 2013 10:46:27 +0000 (UTC) Received: from gw01-1.biotronik.org (gw01-1.biotronik.org [91.204.10.44]) by mx1.freebsd.org (Postfix) with ESMTP id 9490B1EA6 for ; Tue, 10 Dec 2013 10:46:25 +0000 (UTC) X-Disclaimed: 10422 To: freebsd-stable@freebsd.org MIME-Version: 1.0 Subject: Swap on ZFS - bad idea? (swap_pager: indefinite wait buffer) X-KeepSent: B3F5370B:79D00F5F-C1257C3D:0038AB15; type=4; name=$KeepSent X-Mailer: Lotus Notes Release 8.5.3FP3 November 15, 2012 From: Benjamin Lutz Message-ID: Date: Tue, 10 Dec 2013 11:41:16 +0100 X-MIMETrack: Serialize by Router on GW01-1/SRV/MSC(Release 9.0.1 HF34|November 26, 2013) at 10.12.2013 11:46:21, Serialize complete at 10.12.2013 11:46:21 Content-Type: text/plain; charset="US-ASCII" X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Dec 2013 10:46:27 -0000 We have storage server that I've found unresponsive this morning. The console was filled with these messages: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 17344, size: 61440 The lines were all exactly the same, no variation in the numbers. Ctrl-Alt-Delete had no effect. Now, the only thing related to swap that's special about this machine, is that we use swap on ZFS, because we figured, silent disk corruption is just as bad when it happens to swapped memory as it is when it happens to stored data. However, might this be the problem? One thought I was having was: ZFS hits its memory bounds, a process makes the kernel want to allocate a swap buffer which causes more ZFS operations which never complete because ZFS would need to do some housework to free memory, but the kernel is waiting for swap to become available, some sort of deadlock? I'm just speculating here, not knowing the internals of this subsystem. The machine is a quad core system with 12GiB RAM and a RAIDZ3 consisting of 12x2TB disks. It's used as a storage system, with accesses mainly through netatalk (an AFP server, what OS X uses) and a bit of iSCSI. I've set the following sysctls: # http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29 vfs.zfs.vdev.max_pending=20 # limit kernel size to 10GB to leave some RAM for processes vm.kmem_size_max="10737418240" # increase number of mbuf clusters, see igb(4). The default is 25600 kern.ipc.nmbclusters="102400" This sort of hang seems to happen every couple of months. I'd really like some pointers on how to increase the stability of this system. Maybe there are some more tunables I need to adjust? Or is swap on ZFS maybe just not a great idea? Cheers, Benjamin -- Benjamin Lutz | Software Engineer | BIOLAB Technology AG Dufourstr. 80 | CH-8008 Zurich | www.biolab.ch | benjamin.lutz@biolab.ch PHONE +41 44 295 97 13 | MOBILE +41 79 558 57 13 | FAX +41 44 295 97 19 This e-mail and the information it contains including attachments are confidential and meant only for use by the intended recipient(s); disclosure or copying is strictly prohibited. If you are not addressed, but in the possession of this e-mail, please notify the sender immediately.