From owner-freebsd-fs@freebsd.org Sun Aug 2 10:50:08 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 540269B0D71 for ; Sun, 2 Aug 2015 10:50:08 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3ABB3958 for ; Sun, 2 Aug 2015 10:50:08 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 375449B0D70; Sun, 2 Aug 2015 10:50:08 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36E4A9B0D6F for ; Sun, 2 Aug 2015 10:50:08 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F1079955; Sun, 2 Aug 2015 10:50:07 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id BFC1115340A; Sun, 2 Aug 2015 12:50:03 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GdmWGJot4vtt; Sun, 2 Aug 2015 12:49:54 +0200 (CEST) Received: from [IPv6:2001:4cb8:3:1:d42b:9fbf:b53b:a3c0] (unknown [IPv6:2001:4cb8:3:1:d42b:9fbf:b53b:a3c0]) by smtp.digiware.nl (Postfix) with ESMTPA id B38D4153431; Sun, 2 Aug 2015 12:49:54 +0200 (CEST) Subject: Re: Multiple entries in ZFS "sharenfs" property? To: fs@freebsd.org, Lev Serebryakov References: <130767529.20150801150343@serebryakov.spb.ru> From: Willem Jan Withagen Message-ID: <55BDF5D2.3090306@digiware.nl> Date: Sun, 2 Aug 2015 12:49:54 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <130767529.20150801150343@serebryakov.spb.ru> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2015 10:50:08 -0000 On 01/08/2015 14:03, Lev Serebryakov wrote: > Hello FreeBSD, > > Is it possible to put multiple entries (for multiple networks) into > "sharenfs" property for ZFS filesystem? > Something like? -alldirs -maproot=0 -network 192.168.10.0 -mask 255.255.252.0 I think I just did it by putting the text between '' 's: ' Which then ends up in /etc/zfs/exports: /home/wjw -alldirs -maproot=0 -network 192.168.10.0 -mask 255.255.252.0 --WjW From owner-freebsd-fs@freebsd.org Sun Aug 2 18:08:27 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64B7C9B1465 for ; Sun, 2 Aug 2015 18:08:27 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4FF5C1AAF for ; Sun, 2 Aug 2015 18:08:27 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 4D30A9B1464; Sun, 2 Aug 2015 18:08:27 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4BCE99B1463 for ; Sun, 2 Aug 2015 18:08:27 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id 095A91AAE for ; Sun, 2 Aug 2015 18:08:27 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:2924:7e01:7d9c:bbfe]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPSA id 737E9213E; Sun, 2 Aug 2015 21:08:17 +0300 (MSK) Date: Sun, 2 Aug 2015 21:08:08 +0300 From: Lev Serebryakov Reply-To: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <1941826477.20150802210808@serebryakov.spb.ru> To: Willem Jan Withagen CC: fs@freebsd.org Subject: Re: Multiple entries in ZFS "sharenfs" property? In-Reply-To: <55BDF5D2.3090306@digiware.nl> References: <130767529.20150801150343@serebryakov.spb.ru> <55BDF5D2.3090306@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2015 18:08:27 -0000 Hello Willem, Sunday, August 2, 2015, 1:49:54 PM, you wrote: >> Is it possible to put multiple entries (for multiple networks) into >> "sharenfs" property for ZFS filesystem? > Something like? > -alldirs -maproot=0 -network 192.168.10.0 -mask 255.255.252.0 > I think I just did it by putting the text between '' 's: > ' > Which then ends up in /etc/zfs/exports: > /home/wjw -alldirs -maproot=0 -network 192.168.10.0 -mask 255.255.252.0 I need 4 such lines per FS (with different -network / -mask arguments). One line works. Four lines don't. -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-fs@freebsd.org Sun Aug 2 20:56:49 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB9909B0AC6 for ; Sun, 2 Aug 2015 20:56:49 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A19C2B24 for ; Sun, 2 Aug 2015 20:56:49 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 9E8199B0AC5; Sun, 2 Aug 2015 20:56:49 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9D1FA9B0AC4 for ; Sun, 2 Aug 2015 20:56:49 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 181DCB22; Sun, 2 Aug 2015 20:56:48 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 03532153431; Sun, 2 Aug 2015 22:56:39 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LoASFUSLQeem; Sun, 2 Aug 2015 22:56:28 +0200 (CEST) Received: from [IPv6:2001:4cb8:3:1:9d45:b378:a938:3c8] (unknown [IPv6:2001:4cb8:3:1:9d45:b378:a938:3c8]) by smtp.digiware.nl (Postfix) with ESMTPA id E1481153430; Sun, 2 Aug 2015 22:56:28 +0200 (CEST) Subject: Re: Multiple entries in ZFS "sharenfs" property? To: Lev Serebryakov References: <130767529.20150801150343@serebryakov.spb.ru> <55BDF5D2.3090306@digiware.nl> <1941826477.20150802210808@serebryakov.spb.ru> Cc: fs@freebsd.org From: Willem Jan Withagen Message-ID: <55BE83FA.1060208@digiware.nl> Date: Sun, 2 Aug 2015 22:56:26 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <1941826477.20150802210808@serebryakov.spb.ru> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2015 20:56:49 -0000 On 2-8-2015 20:08, Lev Serebryakov wrote: > Hello Willem, > > Sunday, August 2, 2015, 1:49:54 PM, you wrote: >>> Is it possible to put multiple entries (for multiple networks) into >>> "sharenfs" property for ZFS filesystem? > >> Something like? >> >> I think I just did it by putting the text between '' 's: >> ' > >> Which then ends up in /etc/zfs/exports: >> /home/wjw -alldirs -maproot=0 -network 192.168.10.0 -mask 255.255.252.0 > I need 4 such lines per FS (with different -network / -mask arguments). One line works. Four lines don't. Hi Lev, Nope, that doesn't work, as far as I can tell. You'd have to revert to editting /etc/exports. Or hack on 'zfs set sharenfs' to generate multiple lines, in some sort of format. Like making ';' a line separator, and then prefix each part with the volume we are modifying. Place to give it a go are in: cddl/compat/opensolaris/misc/fsshare.c:213 if (share) { fprintf(newfd, "%s\t%s\n", mountpoint, translate_opts(shareopts)); } And there split the shareopts on the split char (eg. ';') in several shareopts and then loop over them. Disadvantage is that the max length op the options is: MAXPATHLEN. So you can easily run out of space if you have many exports to do. -WjW From owner-freebsd-fs@freebsd.org Sun Aug 2 21:00:51 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 081F99B0D92 for ; Sun, 2 Aug 2015 21:00:51 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D62B1E9B for ; Sun, 2 Aug 2015 21:00:50 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t72L0oHf084254 for ; Sun, 2 Aug 2015 21:00:50 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201508022100.t72L0oHf084254@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 02 Aug 2015 21:00:50 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2015 21:00:51 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 3 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Aug 3 02:10:51 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 891979B2494 for ; Mon, 3 Aug 2015 02:10:51 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: from mail-qg0-f53.google.com (mail-qg0-f53.google.com [209.85.192.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C9D31364 for ; Mon, 3 Aug 2015 02:10:50 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: by qgeu79 with SMTP id u79so80382636qge.1 for ; Sun, 02 Aug 2015 19:10:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:mime-version:content-type:from :in-reply-to:date:content-transfer-encoding:message-id:references:to; bh=jNDIhZ5qWZFeYa+gdLI6yIp4JhyZd8NdH0iXFFpq4ao=; b=RinSQCpu5UyPL2emhKEcA8MLDq///KoskOBKFZvOsggrQmPwJYj1lX+j3pIdh6ZNsc qSuv/z+yhz3sgFmFGNvOQMtkUVBsBWVs49azHnL7l0lFi4RcI9DUdOSjYo8br7/QWDxX /BSke67fR6/KxVe9wQLinmoSRsGAoYhcnBop70RDh0zouCSlK+d/MWA0LVE9nd99fGFq 2QKQ7gtW/h7zKPyTO/r/JWTx/g1cXb6NLx2gGcp6vfdkmINlsKsxSbeJK6qrvxNrauHV HYxO8wG/8GOTtTkF5MCYjiF9cWSieSqT30ukeQAAAEagmcnNOfkR/QbjHccFTDkHgcOD 7q0Q== X-Gm-Message-State: ALoCoQm22iaAJbY0QIeiNKr2qp57z78NO4ky2dwONkcJUflAFc4A75B1JvV5DgQOzWyFsyWavKHk X-Received: by 10.140.217.207 with SMTP id n198mr22126755qhb.92.1438567844268; Sun, 02 Aug 2015 19:10:44 -0700 (PDT) Received: from [192.168.2.137] (pool-100-4-179-8.albyny.fios.verizon.net. [100.4.179.8]) by smtp.gmail.com with ESMTPSA id f106sm6189412qgd.30.2015.08.02.19.10.43 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Aug 2015 19:10:43 -0700 (PDT) Subject: Re: NFS & ZFS: how to export whole FS hierarhy to mount it with one command on client? Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Content-Type: text/plain; charset=windows-1252 From: Paul Kraus X-Priority: 3 (Normal) In-Reply-To: <795246861.20150801140429@serebryakov.spb.ru> Date: Sun, 2 Aug 2015 22:10:41 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: <795246861.20150801140429@serebryakov.spb.ru> To: FreeBSD FS X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 02:10:51 -0000 On Aug 1, 2015, at 7:04, Lev Serebryakov wrote: > I had "/usr/home" UFS exported to several hosts (all of them are = FreeBSD), > and it worked as intended: remote host mounted "server:/usr/home" and = got > all user home dirs. I hope you mean NFS exported... > Now I converted "/usr/home" to ZFS and created one FS per user (so, = here is > FSes "zhome/lev", "zhome/sveta", etc., on pool "zhome=94). Why are you using one zfs dataset per user ? That was a recommendation = very early on in the days of zfs before user level quotas. Other than = the ability to snapshot individual user=92s home directories, what is = that configuration getting you ? -- Paul Kraus paul@kraus-haus.org From owner-freebsd-fs@freebsd.org Mon Aug 3 02:27:20 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 174339B267E for ; Mon, 3 Aug 2015 02:27:20 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: from mail-qg0-f44.google.com (mail-qg0-f44.google.com [209.85.192.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D5D961989 for ; Mon, 3 Aug 2015 02:27:19 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: by qgeu79 with SMTP id u79so80540004qge.1 for ; Sun, 02 Aug 2015 19:27:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:content-transfer-encoding:message-id:references:to; bh=zR2E4ChgESQCRHIJ1gHsWdFW4j686tE5jifccmnJFl8=; b=MBi6/4BGwqmLFKO9QcGMh8BiH9WzJqVemHyqZDkrQPINXhk5ewu3di8uYlnid56A64 5E3/FXvjsgq7wYythc7TZdtzn9L40GngH3PzdO+rpwN7nPU3m1iCuzeRdt9eHUW4YC6K 6ODS+TTollvcY6eLwqNBXKSJR/jlGZHg0RXKgpgxcG0k/5s2A8Ay9NRMV8ku2D4d+0UN Dd2kuA1ZVqDUm79qx45avcKYxycIR89/pZMl0iQi+dyNQiLs46jqkZx2h0Z8A5yu42ns SxsjdlGB+u6gVbOCbCx8y8vPcnvNjayqOaNvdHMmtuAroeU5EUKF+qwWC5OqOfEEvwmd TdcA== X-Gm-Message-State: ALoCoQncPF8qiwajGhwPE0TAfVFLRHEKQpolmOPgG4SjJ4y5bUFAzgLLq4uHq+IjtvHgpZSWDPtS X-Received: by 10.140.236.22 with SMTP id h22mr5414468qhc.20.1438567522589; Sun, 02 Aug 2015 19:05:22 -0700 (PDT) Received: from [192.168.2.137] (pool-100-4-179-8.albyny.fios.verizon.net. [100.4.179.8]) by smtp.gmail.com with ESMTPSA id h49sm6186357qgd.24.2015.08.02.19.05.20 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Aug 2015 19:05:20 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: ZFS: Disabling ARC? From: Paul Kraus In-Reply-To: <55BC14B7.9010009@sneakertech.com> Date: Sun, 2 Aug 2015 22:05:18 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <9DBE58C6-8C42-498B-AB66-7D9BBDFAA90F@kraus-haus.org> References: <55BC14B7.9010009@sneakertech.com> To: FreeBSD FS X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 02:27:20 -0000 On Jul 31, 2015, at 20:37, Quartz wrote: > Can someone help clear up a few ZFS basics for me? >=20 > A few recent threads about ARC issues and memory-induced panics have = made me realize I'm not 100% sure I understand ARC as well as I thought = I did. > If I understand ARC correctly this would be a worst case scenario, = right? Besides hogging ram, would ARC cause any problems here? Would = disabling ARC and devoting the ram to other things be a wise idea? Is = disabling ARC ever a wise idea? The ZFS ARC is both powerful and often misunderstood. In order to do = anything intelligently with the ARC (and it=92s tunings) you need to = first know what it is doing=85 get a copy of arcstat.pl from here = https://github.com/mharsch/arcstat/blob/master/arcstat.pl =85 I prefer = it to the zfs-stats port, but any tool that shows us the realtime ARC = usage is what you need. The ARC is _supposed_ to use whatever RAM is otherwise unused _and_ if = other processes need RAM, the ARC is supposed to give it back. I know = there have been a number of issues with the ARC not freeing RAM fast = enough, but I have not seen that issue under 10.1 (I have seen it under = 9.x). On my main 9.x VM host I leave an 'arcstat.pl 60' running all the = time so I can quickly see if it is freeing as it should. Also note that the ARC does not cache files but blocks (if, as Alex = mentioned) it is cacheing data at all (or just metadata). If you are really worried about the ARC hogging RAM, then set a cap. The = kernel tunables here are: [ppk@FreeBSD2 ~]$ sysctl -a | grep zfs | grep arc vfs.zfs.arc_max: 12884901888 vfs.zfs.arc_min: 1919138304 vfs.zfs.arc_meta_used: 3221144040 vfs.zfs.arc_meta_limit: 3221225472 vfs.zfs.l2arc_write_max: 8388608 vfs.zfs.l2arc_write_boost: 8388608 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_noprefetch: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_norw: 1 =85 followed by _lots_ of arcstats, which tell you what the ARC is = doing.=20 The first one is what you really want to tune if you are RAM limited. It = sets a hard cap on how much RAM the system will use for the entire ARC. = On my home server I have the following tunings in /boot/loader.conf [ppk@FreeBSD2 /boot]$ cat loader.conf=20 zfs_load=3D"YES" # set ARC max to 12 GB with 16 GB of RAM vfs.zfs.arc_max=3D"12884901888" So I set the ARC maximum to 12 GB to leave me 4 GB for OS and = applications. I have production servers with between 24 and 80 GB of RAM and do not = generally limit the ARC but let it =93float=94 and give and take as RAM = is available. Please note that I have never seen a _panic_ due to ARC RAM issues, I = have had systems starved for RAM for periods and processes (VMs) get = very angry, but the system as a whole usually recovers. I then restart = the processes that got angry.=20 -- Paul Kraus paul@kraus-haus.org From owner-freebsd-fs@freebsd.org Mon Aug 3 03:56:34 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C0A69B2811 for ; Mon, 3 Aug 2015 03:56:34 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4546611C8 for ; Mon, 3 Aug 2015 03:56:34 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-239-102.lns20.per1.internode.on.net [121.45.239.102]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t733uTbM087159 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Sun, 2 Aug 2015 20:56:33 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: NFS & ZFS: how to export whole FS hierarhy to mount it with one command on client? To: freebsd-fs@freebsd.org References: <795246861.20150801140429@serebryakov.spb.ru> <1363497421.7238055.1438428070047.JavaMail.zimbra@uoguelph.ca> <1593307781.20150801143052@serebryakov.spb.ru> From: Julian Elischer Message-ID: <55BEE668.3080303@freebsd.org> Date: Mon, 3 Aug 2015 11:56:24 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <1593307781.20150801143052@serebryakov.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 03:56:34 -0000 On 8/1/15 7:30 PM, Lev Serebryakov wrote: > Hello Rick, > > Saturday, August 1, 2015, 2:21:10 PM, you wrote: > >> To mount multiple file systems as one mount, you'll need to use NFSv4. I believe >> you will have to have a separate export entry in the server for each of the file >> systems. > So, /etc/exports needs to have BOTH v3-style exports & V4: root of tree line? OR you can have a non-standard patch that pjd wrote to do recursive mounts of sub-filesystems. it is not supposed to happen according to the standard but we have found it useful. Unfortnately it is written agains the old NFS Server. Rick, if I gave you the original pjd patch for the old server, could you integrate it into the new server as an option? > From owner-freebsd-fs@freebsd.org Mon Aug 3 07:04:24 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B0599B20BA for ; Mon, 3 Aug 2015 07:04:24 +0000 (UTC) (envelope-from quartz@sneakertech.com) Received: from douhisi.pair.com (douhisi.pair.com [209.68.5.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 59B4F160C for ; Mon, 3 Aug 2015 07:04:24 +0000 (UTC) (envelope-from quartz@sneakertech.com) Received: from [10.2.2.1] (pool-173-48-121-235.bstnma.fios.verizon.net [173.48.121.235]) by douhisi.pair.com (Postfix) with ESMTPSA id 35B383F7C4 for ; Mon, 3 Aug 2015 03:04:17 -0400 (EDT) Message-ID: <55BF1270.10003@sneakertech.com> Date: Mon, 03 Aug 2015 03:04:16 -0400 From: Quartz MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: ZFS: Disabling ARC? References: <55BC14B7.9010009@sneakertech.com> <9DBE58C6-8C42-498B-AB66-7D9BBDFAA90F@kraus-haus.org> In-Reply-To: <9DBE58C6-8C42-498B-AB66-7D9BBDFAA90F@kraus-haus.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 07:04:24 -0000 > If you are really worried about the ARC hogging RAM, then set a cap. > The kernel tunables here are: I'm not worried about it hogging ram per se, but rather I'm a little confused about where and when it helps, where it's useless or detrimental (if ever), and consequently I don't really know when I should tune it or what to tune it *to*. Basically, my question is the subject line of this thread: is there ever a reason to attempt to disable ARC, and what would that situation probably look like? > Please note that I have never seen a _panic_ due to ARC RAM issues, I > have had systems starved for RAM for periods and processes (VMs) get > very angry, but the system as a whole usually recovers. I then > restart the processes that got angry. I didn't mean panics specifically due to ARC, but that in the process of reading various threads about memory related panics and ARC memory issues (separately) I realized that I wasn't really clear about memory management on FreeBSD and how ARC interacted there. From owner-freebsd-fs@freebsd.org Mon Aug 3 09:32:59 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D53659B1482 for ; Mon, 3 Aug 2015 09:32:59 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BAE4FF49 for ; Mon, 3 Aug 2015 09:32:59 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id BA0A39B1481; Mon, 3 Aug 2015 09:32:59 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9EF189B1480 for ; Mon, 3 Aug 2015 09:32:59 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2AFFDF47; Mon, 3 Aug 2015 09:32:58 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 60B491534C4; Mon, 3 Aug 2015 11:32:54 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dbbDqpUqv1CZ; Mon, 3 Aug 2015 11:32:53 +0200 (CEST) Received: from [IPv6:2001:4cb8:3:1:bd55:9a1f:88:880] (unknown [IPv6:2001:4cb8:3:1:bd55:9a1f:88:880]) by smtp.digiware.nl (Postfix) with ESMTP id 1D4771534C0; Mon, 3 Aug 2015 11:32:53 +0200 (CEST) Message-ID: <55BF3542.2070206@digiware.nl> Date: Mon, 03 Aug 2015 11:32:50 +0200 From: Willem Jan Withagen Organization: Digiware Management b.v. User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Lev Serebryakov CC: fs@freebsd.org Subject: Re: Multiple entries in ZFS "sharenfs" property? References: <130767529.20150801150343@serebryakov.spb.ru> <55BDF5D2.3090306@digiware.nl> <1941826477.20150802210808@serebryakov.spb.ru> <55BE83FA.1060208@digiware.nl> In-Reply-To: <55BE83FA.1060208@digiware.nl> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 09:33:00 -0000 On 2-8-2015 22:56, Willem Jan Withagen wrote: > On 2-8-2015 20:08, Lev Serebryakov wrote: >> Hello Willem, >> >> Sunday, August 2, 2015, 1:49:54 PM, you wrote: >>>> Is it possible to put multiple entries (for multiple networks) into >>>> "sharenfs" property for ZFS filesystem? >> >>> Something like? >>> >>> I think I just did it by putting the text between '' 's: >>> ' >> >>> Which then ends up in /etc/zfs/exports: >>> /home/wjw -alldirs -maproot=0 -network 192.168.10.0 -mask 255.255.252.0 >> I need 4 such lines per FS (with different -network / -mask arguments). One line works. Four lines don't. > > Hi Lev, > > Nope, that doesn't work, as far as I can tell. > You'd have to revert to editting /etc/exports. > > Or hack on 'zfs set sharenfs' to generate multiple lines, in some sort > of format. Like making ';' a line separator, and then prefix each part > with the volume we are modifying. > > Place to give it a go are in: > cddl/compat/opensolaris/misc/fsshare.c:213 > if (share) { > fprintf(newfd, "%s\t%s\n", mountpoint, > translate_opts(shareopts)); > } > And there split the shareopts on the split char (eg. ';') in several > shareopts and then loop over them. Disadvantage is that the max length > op the options is: MAXPATHLEN. So you can easily run out of space if you > have many exports to do. Could not resist... --WjW Perhaps something like: Index: compat/opensolaris/misc/fsshare.c =================================================================== --- compat/opensolaris/misc/fsshare.c (revision 286222) +++ compat/opensolaris/misc/fsshare.c (working copy) @@ -151,6 +151,7 @@ int share) { char tmpfile[PATH_MAX]; + char tmpshareopts[OPTSSIZE], *sharep, *sepp; char *line; FILE *newfd, *oldfd; int fd, error; @@ -211,8 +212,25 @@ goto out; } if (share) { - fprintf(newfd, "%s\t%s\n", mountpoint, - translate_opts(shareopts)); + /* Feed shareopts in piecces to translate_opts */ + if (strlcpy(tmpshareopts, shareopts, sizeof(tmpshareopts)) + >= sizeof(tmpshareopts)) + return (ENAMETOOLONG); + sharep = tmpshareopts; + do { + sepp = strchr(sharep, ';'); + if ( sepp != sharep) { + if (sepp != NULL) + *sepp = '\0'; + fprintf(newfd, "%s\t%s\n", mountpoint, + translate_opts(sharep)); + sharep= sepp+1; + } + /* + * exit if no seperator was found + * then we just did the last iteration + */ + } while (sepp != NULL); } out: From owner-freebsd-fs@freebsd.org Mon Aug 3 12:06:13 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57AFE9B17E1 for ; Mon, 3 Aug 2015 12:06:13 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 421581D9C for ; Mon, 3 Aug 2015 12:06:13 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 417CC9B17E0; Mon, 3 Aug 2015 12:06:13 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 411119B17DF for ; Mon, 3 Aug 2015 12:06:13 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 0B0DC1D9B for ; Mon, 3 Aug 2015 12:06:13 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [127.0.0.1] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPSA id 9F8AF2264; Mon, 3 Aug 2015 15:06:11 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: Multiple entries in ZFS "sharenfs" property? References: <130767529.20150801150343@serebryakov.spb.ru> <55BDF5D2.3090306@digiware.nl> <1941826477.20150802210808@serebryakov.spb.ru> <55BE83FA.1060208@digiware.nl> To: Willem Jan Withagen Cc: fs@freebsd.org From: Lev Serebryakov Organization: FreeBSD Message-ID: <55BF5932.2090107@FreeBSD.org> Date: Mon, 3 Aug 2015 15:06:10 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55BE83FA.1060208@digiware.nl> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 12:06:13 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 02.08.2015 23:56, Willem Jan Withagen wrote: > Nope, that doesn't work, as far as I can tell. You'd have to revert > to editting /etc/exports. Yep, I know. > Or hack on 'zfs set sharenfs' to generate multiple lines, in some > sort of format. Like making ';' a line separator, and then prefix > each part with the volume we are modifying. > > Place to give it a go are in: > cddl/compat/opensolaris/misc/fsshare.c:213 if (share) { > fprintf(newfd, "%s\t%s\n", mountpoint, translate_opts(shareopts)); > } And there split the shareopts on the split char (eg. ';') in > several shareopts and then loop over them. Disadvantage is that the > max length op the options is: MAXPATHLEN. So you can easily run out > of space if you have many exports to do. Here is PR with patch about this, from 2012 :( - -- // Lev Serebryakov -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJVv1kyXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EeP9d0P/229Jje1vw8QB9LuWkNH81xE SIeYOnXRs4OoJDBmxW1xi5kaKqT+j6A9m7lWEXjm/nWGsULNcjDtP4uqQehVZDl3 7d/eGZDQgrZa5zUxvxMfb0hoWxCZNYFf6N1sPB+98ZwPVH2ADD+7h4ljusoI19gD ZVjTEDeWKBln2YWWHXDDJwjFeF/AqwZzf6o0Nn6Wg2k49pAW4hjvRfVFBXYVo8CA sWsD9ilgYteD6LEPdHbPy2pgprlsmIwLV84zjGb2Lk0Txs7k4CP5RnRo276wkmKY lFdvlyDOFna2UsJqTbAdYD5XeTC0Zx08sqd0dPvgmugYICpgFou1rClTd6HYLDTC +n1kmY8CrGn6omOwLeuckzkroPVFV/RgzRfKFdCabTfAoRe1a60Y7MTFCKUVmZnc OOz2F1Ph3PJAyl5aiIZAuDikj+PEkeqbYJPQxeh2neKwOLTLj4G9oGx+CFo8t6Fk Y793GqkrmIkh4O7o8LrLqhT7/II/MybC2a31RjZRfdukofIgC+CEw56IcP7tgb8U L2f+QxprnQDbwFwkjJAvU/x6dPvy4cztF5vA3qtY5HAW21u+ROrAHBorANq83nZT Y7Ab4vrcThoXTJnNgzN0usOz0OUdtHtVz+5OOoLNJsPVbFFrdIa2I67jW4giDiVv +Vo4KfE2/wL2MpiviQQ7 =GB/t -----END PGP SIGNATURE----- From owner-freebsd-fs@freebsd.org Mon Aug 3 12:07:28 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5ABF69B183E for ; Mon, 3 Aug 2015 12:07:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 20BA71E21 for ; Mon, 3 Aug 2015 12:07:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [127.0.0.1] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPSA id 0C8F52266; Mon, 3 Aug 2015 15:07:26 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: NFS & ZFS: how to export whole FS hierarhy to mount it with one command on client? References: <795246861.20150801140429@serebryakov.spb.ru> To: Paul Kraus , FreeBSD FS From: Lev Serebryakov Organization: FreeBSD Message-ID: <55BF597E.9020903@FreeBSD.org> Date: Mon, 3 Aug 2015 15:07:26 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 12:07:28 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 03.08.2015 05:10, Paul Kraus wrote: > Why are you using one zfs dataset per user ? That was a > recommendation very early on in the days of zfs before user level > quotas. Other than the ability to snapshot individual user’s home > directories, what is that configuration getting you ? Different block sizes ("recordsize"), different compression, different snapshot policy. I don't need quotes at all, but I need all of these. - -- // Lev Serebryakov -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJVv1l+XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EePVcsQANWANHjZbthaBhR49JQy0jwh YV25mWn3f6YAN56TiIo+i8tQQhjf/Gr1513HdRnpFsuEN8bWCebbFTAR/buxmerB WCi5sw69Bi88TJcm2gnf9DaXiItaeqwV4MkTIUrAjAlHzyq3DlxoITQBJf0JD5BP LhJD+T3sK5k0Y1sP2MMvOKzCsnDSWimWKzKEFhc7holtXhvDL1XaamMTNf0xaDQY RpRcJxhv6r10i/WhRM9QboeJim6SXG7iRQYw+LEgOMvmvpeZQa0nLW7if0NrUnP7 g8t3yaW39JPJJWD4J5s+M1LpaPtnhj9FXTLBZnSCl17KoMLV/mJ+n/Fh4E8ucTws 2wTaY9kr8mJkVWKzv4Qibe6CiG593GJQvOzZZkPsXDoipoKyvzu/Kv84FmR9rPBR Dg07qI+1G63X6iGPgAbI6O78Bw1yHY93XoQ2oSBWyhIetW7SpYoNZnVaZ/0KcM6L F0iuvULMg+FvN0Wt4IB0M0j8z8LmSw5ULUB+juS4gDNH22ghwlcBIMxx08iG41b3 F3BnuJwb8GRUdqmR2PO3YPvUpQGyS2WSRqpYrE4KH5PUE/RthiswEAW65eXPoZpq uj/ZKbDeN6hJTzMm1hM4Tky36P96c8DNmZxY1f8h6W3zulletHQSoUs3f8zPi7jS SUAsCkGdsTqpd7uNHoQg =WMLm -----END PGP SIGNATURE----- From owner-freebsd-fs@freebsd.org Mon Aug 3 12:22:07 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DAF509B1C6E for ; Mon, 3 Aug 2015 12:22:07 +0000 (UTC) (envelope-from info@pk1048.com) Received: from cpanel61.fastdnsservers.com (server61.fastdnsservers.com [216.51.232.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BA48FC0D for ; Mon, 3 Aug 2015 12:22:07 +0000 (UTC) (envelope-from info@pk1048.com) Received: from pool-100-4-179-8.albyny.fios.verizon.net ([100.4.179.8]:49644 helo=[192.168.2.133]) by cpanel61.fastdnsservers.com with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.85) (envelope-from ) id 1ZMEkn-001FWt-NT; Mon, 03 Aug 2015 07:22:05 -0500 References: <795246861.20150801140429@serebryakov.spb.ru> <55BF597E.9020903@FreeBSD.org> In-Reply-To: <55BF597E.9020903@FreeBSD.org> Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Message-Id: Cc: Paul Kraus X-Mailer: iPad Mail (12H143) From: PK1048 Subject: Re: NFS & ZFS: how to export whole FS hierarhy to mount it with one command on client? Date: Mon, 3 Aug 2015 08:22:04 -0400 To: FreeBSD Filesystems X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel61.fastdnsservers.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - pk1048.com X-Get-Message-Sender-Via: cpanel61.fastdnsservers.com: authenticated_id: info@pk1048.com X-Source: X-Source-Args: X-Source-Dir: X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 12:22:07 -0000 Sent from my portable device > On Aug 3, 2015, at 08:07, Lev Serebryakov wrote: >=20 > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 >=20 >> On 03.08.2015 05:10, Paul Kraus wrote: >>=20 >> Why are you using one zfs dataset per user ? That was a >> recommendation very early on in the days of zfs before user level >> quotas. Other than the ability to snapshot individual user=E2=80=99s home= >> directories, what is that configuration getting you ? > Different block sizes ("recordsize"), different compression, > different snapshot policy. I don't need quotes at all, but I need all > of these. I thought these were home directories, what are users putting here that need= different turnings ?=20 Remember that "recordsize" is not block size, but a maximum suggested record= size. ZFS will use what it considers the best recordsize for each write at h= and. From owner-freebsd-fs@freebsd.org Mon Aug 3 12:23:02 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 764849B1CF1 for ; Mon, 3 Aug 2015 12:23:02 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5C778CB1 for ; Mon, 3 Aug 2015 12:23:02 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 59D769B1CEF; Mon, 3 Aug 2015 12:23:02 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5976B9B1CEE for ; Mon, 3 Aug 2015 12:23:02 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0DFFCCAF; Mon, 3 Aug 2015 12:23:01 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 0215615344D; Mon, 3 Aug 2015 14:22:58 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c-WQdtP2j6GN; Mon, 3 Aug 2015 14:22:46 +0200 (CEST) Received: from [192.168.101.176] (vpn.ecoracks.nl [31.223.170.173]) by smtp.digiware.nl (Postfix) with ESMTPA id 526E1153431; Mon, 3 Aug 2015 14:22:46 +0200 (CEST) Subject: Re: Multiple entries in ZFS "sharenfs" property? To: lev@FreeBSD.org References: <130767529.20150801150343@serebryakov.spb.ru> <55BDF5D2.3090306@digiware.nl> <1941826477.20150802210808@serebryakov.spb.ru> <55BE83FA.1060208@digiware.nl> <55BF5932.2090107@FreeBSD.org> Cc: fs@freebsd.org From: Willem Jan Withagen Message-ID: <55BF5D16.9060705@digiware.nl> Date: Mon, 3 Aug 2015 14:22:46 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55BF5932.2090107@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 12:23:02 -0000 On 3-8-2015 14:06, Lev Serebryakov wrote: > On 02.08.2015 23:56, Willem Jan Withagen wrote: > >> Nope, that doesn't work, as far as I can tell. You'd have to revert >> to editting /etc/exports. > Yep, I know. > >> Or hack on 'zfs set sharenfs' to generate multiple lines, in some >> sort of format. Like making ';' a line separator, and then prefix >> each part with the volume we are modifying. > >> Place to give it a go are in: >> cddl/compat/opensolaris/misc/fsshare.c:213 if (share) { >> fprintf(newfd, "%s\t%s\n", mountpoint, translate_opts(shareopts)); >> } And there split the shareopts on the split char (eg. ';') in >> several shareopts and then loop over them. Disadvantage is that the >> max length op the options is: MAXPATHLEN. So you can easily run out >> of space if you have many exports to do. > Here is PR with patch about this, from 2012 :( 'mmmm PR URL is missing, I looked but did not find anything. Just send you some code this morning.... Don't you have commit bits??? So you could checkin a fix if you wanted? Usually if I have a patch, I just ask (and reask) a previous committer of that file if he/she wants to do the honnors. Mostly things go fast from there. --WjW From owner-freebsd-fs@freebsd.org Mon Aug 3 12:25:18 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBE5D9B1E0B for ; Mon, 3 Aug 2015 12:25:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 5B9C0D54; Mon, 3 Aug 2015 12:25:17 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AGAwAgXb9V/61jaINbGQEBAYNSaQaDHbktCYFdHQqFL0oCgWAUAQEBAQEBAYEKhCQBAQEDAQEBICsgCxACAQgYAgINGQICJwEJJgIECAcEARwEiA0Ns3+VYQEBAQEBAQQBAQEBAR2BIoothC8HAQEFFzQHgmmBQwWUeYR7gmKCEYQlRoZukEsCJoIOHBWBWiIxB34JFyOBBAEBAQ X-IronPort-AV: E=Sophos;i="5.15,601,1432612800"; d="scan'208";a="228782695" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 03 Aug 2015 08:25:14 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id C35E615F542; Mon, 3 Aug 2015 08:25:14 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id iWkxCeA3Qb-e; Mon, 3 Aug 2015 08:25:14 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id F233415F55D; Mon, 3 Aug 2015 08:25:13 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id mnlUEnp0QrPa; Mon, 3 Aug 2015 08:25:13 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id D49B615F542; Mon, 3 Aug 2015 08:25:13 -0400 (EDT) Date: Mon, 3 Aug 2015 08:25:13 -0400 (EDT) From: Rick Macklem To: Julian Elischer Cc: freebsd-fs@freebsd.org Message-ID: <67101638.8226696.1438604713620.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <55BEE668.3080303@freebsd.org> References: <795246861.20150801140429@serebryakov.spb.ru> <1363497421.7238055.1438428070047.JavaMail.zimbra@uoguelph.ca> <1593307781.20150801143052@serebryakov.spb.ru> <55BEE668.3080303@freebsd.org> Subject: Re: NFS & ZFS: how to export whole FS hierarhy to mount it with one command on client? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: NFS & ZFS: how to export whole FS hierarhy to mount it with one command on client? Thread-Index: 2RZ+wToGPDqqBYcRYZz0ktt3TrYzSg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 12:25:18 -0000 Julian Elischer wrote: > On 8/1/15 7:30 PM, Lev Serebryakov wrote: > > Hello Rick, > > > > Saturday, August 1, 2015, 2:21:10 PM, you wrote: > > > >> To mount multiple file systems as one mount, you'll need to use NFSv4. I > >> believe > >> you will have to have a separate export entry in the server for each of > >> the file > >> systems. > > So, /etc/exports needs to have BOTH v3-style exports & V4: root of tree > > line? > > OR you can have a non-standard patch that pjd wrote to do recursive > mounts of sub-filesystems. > it is not supposed to happen according to the standard but we have > found it useful. > Unfortnately it is written agains the old NFS Server. > > Rick, if I gave you the original pjd patch for the old server, could > you integrate it into the new server as an option? > A patch like this basically inserts the file system volume identifier in the high order bits of the fileid# (inode# if you prefer), so that duplicate fileid#s don't show up in a "consolidated file system" (for want of a better term). It also replies with the same "fake" fsid for all volumes involved. I see certain issues w.r.t. this: 1 - What happens when the exported volumes are disjoint and don't form one tree? (I think any just option should be restricted to volumes that form a tree, but I don't know an easy way to enforce that restriction?) 2 - It would be fine at this point to use the high order bits of the fileid#, since NFSv3 defines it as 64bits and FreeBSD's ino_t is 32bits. However, I believe FreeBSD is going to have to increase ino_t to 64bits soon. (I hope such a patch will be in FreeBSD11.) Once ino_t is 64bits, this option would have to assume that some # of the high order bits of the fileid# are always 0. Something like "the high order 24bits are always 0" would work ok for a while, then someone would build a file system large enough to overflow the 40bit (I know that's a lot, but some are already exceeding 32bits for # of fileids) field and cause trouble. 3 - You could get weird behaviour when the tree includes exports with different export options. This discussion includes just that and NFSv3 clients don't expect things to change within a mount. (An example would be having part of this consolidated tree require Kerberos authentication. Another might be having parts of the consolidated tree use different uid mapping for AUTH_SYS.) 4 - Some file systems (msdosfs ie. FAT) have limited capabilities w.r.t. what the NFS server can do to the file system. If one of these was imbedded in the consolidated tree, then it could cause confusion similar to #3. All in all, the "hack" is relatively easy to do, if: You use one kind of file system (for example ZFS) and make everything you are exporting one directory tree which is all exported in a compatible way. You also "know" that all the fileid#s in the underlying file systems will fit in the low order K bits of the 64bit fileid#. My biggest concern is #2, once ino_t becomes 64bits. If the collective thinks this is a good idea despite the issues above and can propose a good way to do it. (Maybe an export flag for all the volumes that will participate in the "consolidated file system"? The exports(5) man page could then try to clearly explain the limitations of its use, etc. Even with that, I suspect some would misuse the option and cause themselves grief.) Personally, since NFSv4 does this correctly, I don't see a need to "hack it" for NFSv3, but I'll leave it up to the collective. rick ps: Julian, you might want to repost this under a separate subject line, so people not interested in how ZFS can export multiple volumes without separate entries will read it. > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Mon Aug 3 12:30:02 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0B679B1EE2 for ; Mon, 3 Aug 2015 12:30:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id AC38EE2C for ; Mon, 3 Aug 2015 12:30:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.ysv.freebsd.org (Postfix) id A98269B1EE0; Mon, 3 Aug 2015 12:30:02 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90E0F9B1EDF for ; Mon, 3 Aug 2015 12:30:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 2D985E2A; Mon, 3 Aug 2015 12:30:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AGAwBxXr9V/61jaINbGQEBAYNSaQaDHbktCYFbHwqFL0oCgWAUAQEBAQEBAYEKhCMBAQEBAgEBAQEgKyALEAIBCBgCAg0ZAgInAQkmAgwHBAEcBIgFCA2zcJVhAQEBAQYBAQEBHoEiii2EGhwBARw0BxeCUoFDBZR5hHuEc4RrlzkCJoINHYFvIjEHgQc6gQQBAQE X-IronPort-AV: E=Sophos;i="5.15,601,1432612800"; d="scan'208";a="228783220" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 03 Aug 2015 08:30:01 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 0608815F542; Mon, 3 Aug 2015 08:30:01 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id aF4IWEvBz4eG; Mon, 3 Aug 2015 08:30:00 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 6CD7C15F55D; Mon, 3 Aug 2015 08:30:00 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ADzxQR-S6wb5; Mon, 3 Aug 2015 08:30:00 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 4BB4515F542; Mon, 3 Aug 2015 08:30:00 -0400 (EDT) Date: Mon, 3 Aug 2015 08:30:00 -0400 (EDT) From: Rick Macklem To: lev@FreeBSD.org Cc: Willem Jan Withagen , fs@freebsd.org Message-ID: <598623926.8227323.1438605000273.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <55BF5932.2090107@FreeBSD.org> References: <130767529.20150801150343@serebryakov.spb.ru> <55BDF5D2.3090306@digiware.nl> <1941826477.20150802210808@serebryakov.spb.ru> <55BE83FA.1060208@digiware.nl> <55BF5932.2090107@FreeBSD.org> Subject: Re: Multiple entries in ZFS "sharenfs" property? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: Multiple entries in ZFS "sharenfs" property? Thread-Index: W0/B+5QgqAVEvCw0Rgh/t/U1CBFcbA== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 12:30:02 -0000 Lev wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 02.08.2015 23:56, Willem Jan Withagen wrote: > > > Nope, that doesn't work, as far as I can tell. You'd have to revert > > to editting /etc/exports. > Yep, I know. > Btw, even if the "hack" Julian Elischer requested was in the system, you would still have to export all the volumes, so this discussion doesn't really change. The only difference would be that the clients could use an NFSv3 mount and not an NFSv4 one. Just wanted to make this clear, rick > > Or hack on 'zfs set sharenfs' to generate multiple lines, in some > > sort of format. Like making ';' a line separator, and then prefix > > each part with the volume we are modifying. > > > > Place to give it a go are in: > > cddl/compat/opensolaris/misc/fsshare.c:213 if (share) { > > fprintf(newfd, "%s\t%s\n", mountpoint, translate_opts(shareopts)); > > } And there split the shareopts on the split char (eg. ';') in > > several shareopts and then loop over them. Disadvantage is that the > > max length op the options is: MAXPATHLEN. So you can easily run out > > of space if you have many exports to do. > Here is PR with patch about this, from 2012 :( > > - -- > // Lev Serebryakov > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iQJ8BAEBCgBmBQJVv1kyXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w > ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF > QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EeP9d0P/229Jje1vw8QB9LuWkNH81xE > SIeYOnXRs4OoJDBmxW1xi5kaKqT+j6A9m7lWEXjm/nWGsULNcjDtP4uqQehVZDl3 > 7d/eGZDQgrZa5zUxvxMfb0hoWxCZNYFf6N1sPB+98ZwPVH2ADD+7h4ljusoI19gD > ZVjTEDeWKBln2YWWHXDDJwjFeF/AqwZzf6o0Nn6Wg2k49pAW4hjvRfVFBXYVo8CA > sWsD9ilgYteD6LEPdHbPy2pgprlsmIwLV84zjGb2Lk0Txs7k4CP5RnRo276wkmKY > lFdvlyDOFna2UsJqTbAdYD5XeTC0Zx08sqd0dPvgmugYICpgFou1rClTd6HYLDTC > +n1kmY8CrGn6omOwLeuckzkroPVFV/RgzRfKFdCabTfAoRe1a60Y7MTFCKUVmZnc > OOz2F1Ph3PJAyl5aiIZAuDikj+PEkeqbYJPQxeh2neKwOLTLj4G9oGx+CFo8t6Fk > Y793GqkrmIkh4O7o8LrLqhT7/II/MybC2a31RjZRfdukofIgC+CEw56IcP7tgb8U > L2f+QxprnQDbwFwkjJAvU/x6dPvy4cztF5vA3qtY5HAW21u+ROrAHBorANq83nZT > Y7Ab4vrcThoXTJnNgzN0usOz0OUdtHtVz+5OOoLNJsPVbFFrdIa2I67jW4giDiVv > +Vo4KfE2/wL2MpiviQQ7 > =GB/t > -----END PGP SIGNATURE----- > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Mon Aug 3 12:42:32 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E04F69B2176 for ; Mon, 3 Aug 2015 12:42:32 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CA4C01518 for ; Mon, 3 Aug 2015 12:42:32 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id C95F49B2175; Mon, 3 Aug 2015 12:42:32 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8F729B2174 for ; Mon, 3 Aug 2015 12:42:32 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 925AC1517 for ; Mon, 3 Aug 2015 12:42:32 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [127.0.0.1] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPSA id BA4482273; Mon, 3 Aug 2015 15:42:29 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: Multiple entries in ZFS "sharenfs" property? References: <130767529.20150801150343@serebryakov.spb.ru> <55BDF5D2.3090306@digiware.nl> <1941826477.20150802210808@serebryakov.spb.ru> <55BE83FA.1060208@digiware.nl> <55BF5932.2090107@FreeBSD.org> <55BF5D16.9060705@digiware.nl> To: Willem Jan Withagen Cc: fs@freebsd.org From: Lev Serebryakov Organization: FreeBSD Message-ID: <55BF61B5.1040404@FreeBSD.org> Date: Mon, 3 Aug 2015 15:42:29 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55BF5D16.9060705@digiware.nl> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 12:42:33 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 03.08.2015 15:22, Willem Jan Withagen wrote: > 'mmmm PR URL is missing, I looked but did not find anything. It was in beginning of thread: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=147881 > Just send you some code this morning.... Don't you have commit > bits??? So you could checkin a fix if you wanted? I'm almost sure, that such commit to ZFS from me will be reverted ASAP :) there are some objections against such change in PR147881. - -- // Lev Serebryakov -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJVv2G1XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EeP8WkQANPOQwtBCJ+5X3plNvrHKFYH iH+F8KZY1XCkIgahSt9Y01xcpsGaYaX+bowj++/3AWpe+y0aJjTy3X1Aag/ELM2+ c9fvWPKpRU4eUtmkCGs2s4wCoiVdVP+v6GI5JymQ4hkp3//6+VbkG+klgkI1ZQax XX/QlYrLsHaerlpC8x543d5t+PD7pa2IFY4HC9pLpFVBOI/DD0epUF2d/mZTgMcz n4hsFJDsHHAWX7LEpoaSONV3wTKCqCIKsaa/pzwb89YPx4LxbNK8dcFQ7t+lrfJh kV/BrQ9dIQhfzfD3BA301VNE73goYOTbaC7DsH+kaPOZN+tllrF03Gmb17u33ajd rw3oMOgehURViUg42YJMo5fvWjIIxnO6WVRCy/08W+QDWcrynOJE4ueTEab5RN+1 yT30xpvQ7jcsaWNekrKQ08Pf6DnavU4YMc0VL/B12v1cpSg/gB5oH22xhvnqWhiP KptSyQV3QppkeZ10d2bLGco6T9W70z5aKYcw4wVB9nnCnD+E4rmWd9gBpYQ3Q9Bq ZBlO13oJ+frFuLBYmK5bS5XSHhDXwE+hW3Kt5RDiIi8XDy005Dasri8nDlufU6Ri /FdWSGtR+QhMqWjDVb9vn1dWZ+cnRMC7/f0/GKiSN+xAO85AUgdC8wobxba4puwb jCfOZxtv+LOGLV/9+VUP =lCar -----END PGP SIGNATURE----- From owner-freebsd-fs@freebsd.org Mon Aug 3 13:43:26 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E62359B2F30 for ; Mon, 3 Aug 2015 13:43:26 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B17371B34 for ; Mon, 3 Aug 2015 13:43:26 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id t73DVkAQ016078; Mon, 3 Aug 2015 08:31:47 -0500 (CDT) Date: Mon, 3 Aug 2015 08:31:46 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Quartz cc: freebsd-fs@freebsd.org Subject: Re: ZFS: Disabling ARC? In-Reply-To: <55BF1270.10003@sneakertech.com> Message-ID: References: <55BC14B7.9010009@sneakertech.com> <9DBE58C6-8C42-498B-AB66-7D9BBDFAA90F@kraus-haus.org> <55BF1270.10003@sneakertech.com> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Mon, 03 Aug 2015 08:31:47 -0500 (CDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 13:43:27 -0000 On Mon, 3 Aug 2015, Quartz wrote: >> If you are really worried about the ARC hogging RAM, then set a cap. >> The kernel tunables here are: > > I'm not worried about it hogging ram per se, but rather I'm a little confused > about where and when it helps, where it's useless or detrimental (if ever), > and consequently I don't really know when I should tune it or what to tune it > *to*. > > Basically, my question is the subject line of this thread: is there ever a > reason to attempt to disable ARC, and what would that situation probably look > like? ARC is intrinsic to the design of zfs and so it can not be entirely disabled. Without caching, zfs would perform terribly, and some caching is needed in order for transaction groups to work. The types of data (data/metadata) cached in the ARC are tunable on a filesystem basis. If you do not encounter a problem, then you should not worry about it. The whole point of the ARC algorithm is that it self-tunes itself to meet available resources and requirements. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@freebsd.org Mon Aug 3 13:49:01 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0300E9B2FF3 for ; Mon, 3 Aug 2015 13:49:01 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DCADA1C30 for ; Mon, 3 Aug 2015 13:49:00 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id D94229B2FF2; Mon, 3 Aug 2015 13:49:00 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8D529B2FF1 for ; Mon, 3 Aug 2015 13:49:00 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B87D1C2F; Mon, 3 Aug 2015 13:49:00 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 02DE6153466; Mon, 3 Aug 2015 15:48:58 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QdkD0jl-sliS; Mon, 3 Aug 2015 15:48:48 +0200 (CEST) Received: from [192.168.101.176] (vpn.ecoracks.nl [31.223.170.173]) by smtp.digiware.nl (Postfix) with ESMTPA id 615C9153431; Mon, 3 Aug 2015 15:48:48 +0200 (CEST) Subject: Re: Multiple entries in ZFS "sharenfs" property? To: lev@FreeBSD.org References: <130767529.20150801150343@serebryakov.spb.ru> <55BDF5D2.3090306@digiware.nl> <1941826477.20150802210808@serebryakov.spb.ru> <55BE83FA.1060208@digiware.nl> <55BF5932.2090107@FreeBSD.org> <55BF5D16.9060705@digiware.nl> <55BF61B5.1040404@FreeBSD.org> Cc: fs@freebsd.org From: Willem Jan Withagen Message-ID: <55BF7140.3090804@digiware.nl> Date: Mon, 3 Aug 2015 15:48:48 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55BF61B5.1040404@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 13:49:01 -0000 On 3-8-2015 14:42, Lev Serebryakov wrote: > On 03.08.2015 15:22, Willem Jan Withagen wrote: > >> 'mmmm PR URL is missing, I looked but did not find anything. > It was in beginning of thread: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=147881 Yup, found it. >> Just send you some code this morning.... Don't you have commit >> bits??? So you could checkin a fix if you wanted? > I'm almost sure, that such commit to ZFS from me will be reverted > ASAP :) there are some objections against such change in PR147881. Which is sad.... The only objections from bugs.. (although not unimportant) were from Martin Matuska. And they are more about the incompatibility between the other versions for ZFS running on Solaris-variants... He then closes with: Have you considered using /etc/exports. Which is of course a dead giveaway, with that as argument why even bother to export sharenfs.... Just block access to it, and have people always use /etc/exports...., rip out the fsshare stuff, and there are no more code incompatibilities. And as far as I can tell fsshare.c is already a FreeBSDism? So the mismatch with other OSes is there and will stay there. It is not like you'd want to merge your pools to a different OS and automagically upon import get NFS exports for that pool? As Sysadmin I'd rather not have that happen. The code in 147881 is quite a lot of lines, but does the trick. The patch I made is a lot less and no comments allowed, does about the same. The status of the report is still "In progress". So nobody saw fit to close it. And thus it is still waiting for a fix?? --WjW From owner-freebsd-fs@freebsd.org Mon Aug 3 13:49:07 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB78E9B2007 for ; Mon, 3 Aug 2015 13:49:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D847B1C3B for ; Mon, 3 Aug 2015 13:49:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t73Dn78q011135 for ; Mon, 3 Aug 2015 13:49:07 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 146941] [zfs] [panic] Kernel Double Fault - Happens constantly Date: Mon, 03 Aug 2015 13:49:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: smh@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 13:49:08 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=146941 Steven Hartland changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |Feedback Timeout CC| |smh@FreeBSD.org Status|In Progress |Closed -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Mon Aug 3 13:50:05 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D53659B20C1 for ; Mon, 3 Aug 2015 13:50:05 +0000 (UTC) (envelope-from info@pk1048.com) Received: from cpanel61.fastdnsservers.com (server61.fastdnsservers.com [216.51.232.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B5FEE1FCB for ; Mon, 3 Aug 2015 13:50:05 +0000 (UTC) (envelope-from info@pk1048.com) Received: from pool-100-4-179-8.albyny.fios.verizon.net ([100.4.179.8]:49640 helo=[192.168.2.133]) by cpanel61.fastdnsservers.com with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.85) (envelope-from ) id 1ZMEgr-001EqK-Pk; Mon, 03 Aug 2015 07:18:01 -0500 References: <55BC14B7.9010009@sneakertech.com> <9DBE58C6-8C42-498B-AB66-7D9BBDFAA90F@kraus-haus.org> <55BF1270.10003@sneakertech.com> In-Reply-To: <55BF1270.10003@sneakertech.com> Mime-Version: 1.0 (1.0) Content-Type: text/plain; charset=us-ascii Message-Id: <12AA569B-829F-418F-B7B4-897F34B92067@pk1048.com> Content-Transfer-Encoding: quoted-printable X-Mailer: iPad Mail (12H143) From: PK1048 Subject: Re: ZFS: Disabling ARC? Date: Mon, 3 Aug 2015 08:18:00 -0400 To: FreeBSD Filesystems X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel61.fastdnsservers.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - pk1048.com X-Get-Message-Sender-Via: cpanel61.fastdnsservers.com: authenticated_id: info@pk1048.com X-Source: X-Source-Args: X-Source-Dir: X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 13:50:05 -0000 Sent from my portable device On Aug 3, 2015, at 03:04, Quartz wrote: >> If you are really worried about the ARC hogging RAM, then set a cap. >> The kernel tunables here are: >=20 > I'm not worried about it hogging ram per se, but rather I'm a little confu= sed about where and when it helps, where it's useless or detrimental (if eve= r), and consequently I don't really know when I should tune it or what to tu= ne it *to*. > Basically, my question is the subject line of this thread: is there ever a= reason to attempt to disable ARC, and what would that situation probably lo= ok like? I expect that ZFS would be functionally useless (from a performance standpoi= nt) if you completely disabled the ARC, I'm not even sure you can. Take a look at http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tunin= g_Guide before doing any ZFS tuning. It was written in the early (Solaris on= ly) days of ZFS, but is still for the most part applicable today. Brendan Gregg has a very good blog entry on the ARC here http://dtrace.org/b= logs/brendan/2012/01/09/activity-of-the-zfs-arc/ I suspect that will answer m= ost of your ARC questions in detail. From owner-freebsd-fs@freebsd.org Mon Aug 3 15:10:11 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15A0A9AE41C for ; Mon, 3 Aug 2015 15:10:11 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-outbound.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.userve.net", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B609DFC0 for ; Mon, 3 Aug 2015 15:10:10 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) by smtp-outbound.userve.net (8.14.7/8.14.7) with ESMTP id t73F0854046209 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=FAIL); Mon, 3 Aug 2015 16:00:08 +0100 (BST) (envelope-from matt.churchyard@userve.net) Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Mon, 3 Aug 2015 16:00:07 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Mon, 3 Aug 2015 16:00:07 +0100 From: Matt Churchyard To: Alexander Leidinger , Quartz CC: FreeBSD FS Subject: RE: ZFS: Disabling ARC? Thread-Topic: ZFS: Disabling ARC? Thread-Index: AQHQy/JC06cuMEFgTU69kuki7wDyb5328/yAgANps/A= Date: Mon, 3 Aug 2015 15:00:07 +0000 Message-ID: <45e74b89dd754991a366a46aa5101822@SERVER.ad.usd-group.com> References: <55BC14B7.9010009@sneakertech.com> <20150801133635.00002ecc@Leidinger.net> In-Reply-To: <20150801133635.00002ecc@Leidinger.net> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 15:10:11 -0000 >On Fri, 31 Jul 2015 20:37:11 -0400 >Quartz wrote: > Can someone help clear up a few ZFS basics for me? >=20 > A few recent threads about ARC issues and memory-induced panics have=20 > made me realize I'm not 100% sure I understand ARC as well as I=20 > thought I did. >=20 > Say you have a ZFS file server that houses very large single files=20 > which are very infrequently accessed. For the sake of argument, let's=20 > say you're using ZFS on a home server for your family, and it holds=20 > exclusively a whole bunch of multi-gig bluray rips or whatever=20 > (nothing else). When someone wants to watch something, they copy the=20 > file to their desktop and watch it there. Although the family will=20 > watch several videos each day, any given file will only be accessed=20 > maybe once every couple months. (I know streaming would make more=20 > sense in real life, and that this example is kinda silly in general,=20 > but ignore that for now). >No matter if you stream or copy, it's the same operation, read once in a w= hile. > If I understand ARC correctly this would be a worst case scenario,=20 > right? Besides hogging ram, would ARC cause any problems here? Would=20 > disabling ARC and devoting the ram to other things be a wise idea? Is=20 > disabling ARC ever a wise idea? >You can tune how the ARC is used: ># zfs get all space/export/Movies | grep cache >space/export/Movies primarycache metadata local >space/export/Movies secondarycache none local >"primarycache" is the ARC in RAM, "secondarycache" is a cache device / L2A= RC (SSD). >"metadata" is directory listing, file sizes, access permissions and the li= ke. >So the above example means that metadata is allowed to go to the ARC in RA= M, and nothing of the real data in this dataset shall be cached >anywhere a= t all (neither in a cache device nor in RAM). >Bye, >Alexander. >-- >http://www.Leidinger.net Alexander@Leidinger.net: PGP 0xC773696B3BAC17DC >http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0xC773696B3BAC17DC I don't know if it's changed, but even changing ARC to metadata only used t= o cause massive performance issues when reading large files in small chunks= . Reading a 128k ZFS record in 4k chunks would cause ZFS to read the same 1= 28k record from disk 32 times. There's a forum thread about it here - https://forums.freebsd.org/threads/z= fs-primarycache-all-versus-metadata.45555/ Generally I've found ARC to be one of the most important parts of ZFS. Ther= e's no case I know of where it will actually adversely affect performance (= not that I've really looked). Its only downside is that it doesn't seem to = manage memory as well as it should, and so a lot of people (including me) h= ave resorted to limiting it. While it seems easy to "tune" using the primarycache option, it's rarely a = good idea except for very specific situations. If ARC isn't getting many hits (I use sysutils/zfs-stats to view stats), an= d you really don't want it using up all your memory, just reduce arc_max. Matt From owner-freebsd-fs@freebsd.org Mon Aug 3 16:36:19 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2DB29B2A15 for ; Mon, 3 Aug 2015 16:36:19 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9D881B39 for ; Mon, 3 Aug 2015 16:36:19 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id t73GaHma016868; Mon, 3 Aug 2015 11:36:17 -0500 (CDT) Date: Mon, 3 Aug 2015 11:36:17 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Matt Churchyard cc: FreeBSD FS Subject: RE: ZFS: Disabling ARC? In-Reply-To: <45e74b89dd754991a366a46aa5101822@SERVER.ad.usd-group.com> Message-ID: References: <55BC14B7.9010009@sneakertech.com> <20150801133635.00002ecc@Leidinger.net> <45e74b89dd754991a366a46aa5101822@SERVER.ad.usd-group.com> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Mon, 03 Aug 2015 11:36:17 -0500 (CDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 16:36:19 -0000 On Mon, 3 Aug 2015, Matt Churchyard wrote: > > I don't know if it's changed, but even changing ARC to metadata only used to cause massive performance issues when reading large files in small chunks. Reading a 128k ZFS record in 4k chunks would cause ZFS to read the same 128k record from disk 32 times. > There's a forum thread about it here - https://forums.freebsd.org/threads/zfs-primarycache-all-versus-metadata.45555/ > > Generally I've found ARC to be one of the most important parts of > ZFS. There's no case I know of where it will actually adversely > affect performance (not that I've really looked). Its only downside > is that it doesn't seem to manage memory as well as it should, and > so a lot of people (including me) have resorted to limiting it. The main reason to disable caching is if it is known in advance that the data will only be read once, and that the block will be read in a single operation. This may be useful for block-oriented databases which do their own caching, and video servers which carefully read full blocks. The reason to disable the caching is to avoid wasting memory. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@freebsd.org Mon Aug 3 19:48:39 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86A429B2F3A for ; Mon, 3 Aug 2015 19:48:39 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 68DAA1599 for ; Mon, 3 Aug 2015 19:48:39 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (50-196-156-133-static.hfc.comcastbusiness.net [50.196.156.133]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t73JmXXY090776 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 3 Aug 2015 12:48:37 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: changes to export whole FS hierarhy to mount it with one command on client? [changed subject] To: Rick Macklem References: <795246861.20150801140429@serebryakov.spb.ru> <1363497421.7238055.1438428070047.JavaMail.zimbra@uoguelph.ca> <1593307781.20150801143052@serebryakov.spb.ru> <55BEE668.3080303@freebsd.org> <67101638.8226696.1438604713620.JavaMail.zimbra@uoguelph.ca> Cc: freebsd-fs@freebsd.org From: Julian Elischer Message-ID: <55BFC58C.6030802@freebsd.org> Date: Tue, 4 Aug 2015 03:48:28 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <67101638.8226696.1438604713620.JavaMail.zimbra@uoguelph.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 19:48:39 -0000 On 8/3/15 8:25 PM, Rick Macklem wrote: > Julian Elischer wrote: >> On 8/1/15 7:30 PM, Lev Serebryakov wrote: >>> Hello Rick, >>> >>> Saturday, August 1, 2015, 2:21:10 PM, you wrote: >>> >>>> To mount multiple file systems as one mount, you'll need to use NFSv4. I >>>> believe >>>> you will have to have a separate export entry in the server for each of >>>> the file >>>> systems. >>> So, /etc/exports needs to have BOTH v3-style exports & V4: root of tree >>> line? >> OR you can have a non-standard patch that pjd wrote to do recursive >> mounts of sub-filesystems. >> it is not supposed to happen according to the standard but we have >> found it useful. >> Unfortnately it is written agains the old NFS Server. >> >> Rick, if I gave you the original pjd patch for the old server, could >> you integrate it into the new server as an option? >> > A patch like this basically inserts the file system volume identifier > in the high order bits of the fileid# (inode# if you prefer), so that > duplicate fileid#s don't show up in a "consolidated file system" (for > want of a better term). It also replies with the same "fake" fsid for > all volumes involved. > > I see certain issues w.r.t. this: > 1 - What happens when the exported volumes are disjoint and don't form > one tree? (I think any just option should be restricted to volumes > that form a tree, but I don't know an easy way to enforce that restriction?) > 2 - It would be fine at this point to use the high order bits of the fileid#, > since NFSv3 defines it as 64bits and FreeBSD's ino_t is 32bits. However, > I believe FreeBSD is going to have to increase ino_t to 64bits soon. > (I hope such a patch will be in FreeBSD11.) > Once ino_t is 64bits, this option would have to assume that some # of > the high order bits of the fileid# are always 0. Something like > "the high order 24bits are always 0" would work ok for a while, then > someone would build a file system large enough to overflow the 40bit > (I know that's a lot, but some are already exceeding 32bits for # of > fileids) field and cause trouble. > 3 - You could get weird behaviour when the tree includes exports with different > export options. This discussion includes just that and NFSv3 clients > don't expect things to change within a mount. (An example would be having > part of this consolidated tree require Kerberos authentication. Another > might be having parts of the consolidated tree use different uid mapping > for AUTH_SYS.) > 4 - Some file systems (msdosfs ie. FAT) have limited capabilities w.r.t. what > the NFS server can do to the file system. If one of these was imbedded in > the consolidated tree, then it could cause confusion similar to #3. > > All in all, the "hack" is relatively easy to do, if: > You use one kind of file system (for example ZFS) and make everything you are > exporting one directory tree which is all exported in a compatible way. > You also "know" that all the fileid#s in the underlying file systems will fit > in the low order K bits of the 64bit fileid#. > > My biggest concern is #2, once ino_t becomes 64bits. > > If the collective thinks this is a good idea despite the issues above and can > propose a good way to do it. (Maybe an export flag for all the volumes that > will participate in the "consolidated file system"? The exports(5) man page > could then try to clearly explain the limitations of its use, etc. Even with > that, I suspect some would misuse the option and cause themselves grief.) > > Personally, since NFSv4 does this correctly, I don't see a need to "hack it" > for NFSv3, but I'll leave it up to the collective. > > rick > ps: Julian, you might want to repost this under a separate subject line, so > people not interested in how ZFS can export multiple volumes without > separate entries will read it. > In our environment we need to export V3 (and maybe even V2) in a single hierarchy, even though it's multiple ZFS filesystems. It's not dissimilar to having a separate ZFS for each user, except in this case it's a separate ZFS for each site. The "modified ZFS" filesystems have very special characteristics. We are only having our very first nibbles (questions) about NFSv4. Until now it's all NFS3. Possibly we'd only have to support it for NFSv3 if V4 can use its native mechanisms. From owner-freebsd-fs@freebsd.org Mon Aug 3 19:53:12 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB6DB9B20E7 for ; Mon, 3 Aug 2015 19:53:12 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 900321A8F for ; Mon, 3 Aug 2015 19:53:12 +0000 (UTC) (envelope-from julian@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 8D4759B20E6; Mon, 3 Aug 2015 19:53:12 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 730379B20E4 for ; Mon, 3 Aug 2015 19:53:12 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4289B1A8E; Mon, 3 Aug 2015 19:53:11 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (50-196-156-133-static.hfc.comcastbusiness.net [50.196.156.133]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t73Jr6ax090788 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 3 Aug 2015 12:53:09 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: Multiple entries in ZFS "sharenfs" property? To: Rick Macklem , lev@FreeBSD.org References: <130767529.20150801150343@serebryakov.spb.ru> <55BDF5D2.3090306@digiware.nl> <1941826477.20150802210808@serebryakov.spb.ru> <55BE83FA.1060208@digiware.nl> <55BF5932.2090107@FreeBSD.org> <598623926.8227323.1438605000273.JavaMail.zimbra@uoguelph.ca> Cc: fs@freebsd.org From: Julian Elischer Message-ID: <55BFC69C.4000700@freebsd.org> Date: Tue, 4 Aug 2015 03:53:00 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <598623926.8227323.1438605000273.JavaMail.zimbra@uoguelph.ca> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 19:53:12 -0000 On 8/3/15 8:30 PM, Rick Macklem wrote: > Lev wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> On 02.08.2015 23:56, Willem Jan Withagen wrote: >> >>> Nope, that doesn't work, as far as I can tell. You'd have to revert >>> to editting /etc/exports. >> Yep, I know. >> > Btw, even if the "hack" Julian Elischer requested was in the system, > you would still have to export all the volumes, so this discussion > doesn't really change. > The only difference would be that the clients could use an NFSv3 mount > and not an NFSv4 one. I don't believe we use a separate export entry for each sub-fs but I'd have to go look again. > Just wanted to make this clear, rick > >>> Or hack on 'zfs set sharenfs' to generate multiple lines, in some >>> sort of format. Like making ';' a line separator, and then prefix >>> each part with the volume we are modifying. >>> >>> Place to give it a go are in: >>> cddl/compat/opensolaris/misc/fsshare.c:213 if (share) { >>> fprintf(newfd, "%s\t%s\n", mountpoint, translate_opts(shareopts)); >>> } And there split the shareopts on the split char (eg. ';') in >>> several shareopts and then loop over them. Disadvantage is that the >>> max length op the options is: MAXPATHLEN. So you can easily run out >>> of space if you have many exports to do. >> Here is PR with patch about this, from 2012 :( >> >> - -- >> // Lev Serebryakov >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v2 >> >> iQJ8BAEBCgBmBQJVv1kyXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w >> ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF >> QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EeP9d0P/229Jje1vw8QB9LuWkNH81xE >> SIeYOnXRs4OoJDBmxW1xi5kaKqT+j6A9m7lWEXjm/nWGsULNcjDtP4uqQehVZDl3 >> 7d/eGZDQgrZa5zUxvxMfb0hoWxCZNYFf6N1sPB+98ZwPVH2ADD+7h4ljusoI19gD >> ZVjTEDeWKBln2YWWHXDDJwjFeF/AqwZzf6o0Nn6Wg2k49pAW4hjvRfVFBXYVo8CA >> sWsD9ilgYteD6LEPdHbPy2pgprlsmIwLV84zjGb2Lk0Txs7k4CP5RnRo276wkmKY >> lFdvlyDOFna2UsJqTbAdYD5XeTC0Zx08sqd0dPvgmugYICpgFou1rClTd6HYLDTC >> +n1kmY8CrGn6omOwLeuckzkroPVFV/RgzRfKFdCabTfAoRe1a60Y7MTFCKUVmZnc >> OOz2F1Ph3PJAyl5aiIZAuDikj+PEkeqbYJPQxeh2neKwOLTLj4G9oGx+CFo8t6Fk >> Y793GqkrmIkh4O7o8LrLqhT7/II/MybC2a31RjZRfdukofIgC+CEw56IcP7tgb8U >> L2f+QxprnQDbwFwkjJAvU/x6dPvy4cztF5vA3qtY5HAW21u+ROrAHBorANq83nZT >> Y7Ab4vrcThoXTJnNgzN0usOz0OUdtHtVz+5OOoLNJsPVbFFrdIa2I67jW4giDiVv >> +Vo4KfE2/wL2MpiviQQ7 >> =GB/t >> -----END PGP SIGNATURE----- >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Mon Aug 3 21:14:36 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F1B309B27E2 for ; Mon, 3 Aug 2015 21:14:36 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 659561A82; Mon, 3 Aug 2015 21:14:35 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DTBAAq2b9V/61jaINbGQEBAYRBgx25OYUnglYCgXATAQEBAQEBAYEKhCQBAQQjVhACAQgYAgINGQICVwIEiEG1JpYSAQEBAQEFAQEBAQEdgSKKLYQvDhc0B4JpgUMFlHmHXYY2hCCDFIxpg2ICJoIOHBWBWiKBNkOBBAEBAQ X-IronPort-AV: E=Sophos;i="5.15,603,1432612800"; d="scan'208";a="230479091" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 03 Aug 2015 17:14:28 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 08A5D15F542; Mon, 3 Aug 2015 17:14:28 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 5wn-b4RrGQDA; Mon, 3 Aug 2015 17:14:27 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 4A61115F55D; Mon, 3 Aug 2015 17:14:27 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ktnsV9UpvpHD; Mon, 3 Aug 2015 17:14:27 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 2BD9815F542; Mon, 3 Aug 2015 17:14:27 -0400 (EDT) Date: Mon, 3 Aug 2015 17:14:27 -0400 (EDT) From: Rick Macklem To: Julian Elischer Cc: freebsd-fs@freebsd.org Message-ID: <987522757.8576059.1438636467059.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <55BFC58C.6030802@freebsd.org> References: <795246861.20150801140429@serebryakov.spb.ru> <1363497421.7238055.1438428070047.JavaMail.zimbra@uoguelph.ca> <1593307781.20150801143052@serebryakov.spb.ru> <55BEE668.3080303@freebsd.org> <67101638.8226696.1438604713620.JavaMail.zimbra@uoguelph.ca> <55BFC58C.6030802@freebsd.org> Subject: Re: changes to export whole FS hierarhy to mount it with one command on client? [changed subject] MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: changes to export whole FS hierarhy to mount it with one command on client? [changed subject] Thread-Index: tIoeJjffiBssP4LN/c2rASRkwqw/JA== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 21:14:37 -0000 Julian Elischer wrote: > On 8/3/15 8:25 PM, Rick Macklem wrote: > > Julian Elischer wrote: > >> On 8/1/15 7:30 PM, Lev Serebryakov wrote: > >>> Hello Rick, > >>> > >>> Saturday, August 1, 2015, 2:21:10 PM, you wrote: > >>> > >>>> To mount multiple file systems as one mount, you'll need to use NFSv4. I > >>>> believe > >>>> you will have to have a separate export entry in the server for each of > >>>> the file > >>>> systems. > >>> So, /etc/exports needs to have BOTH v3-style exports & V4: root of > >>> tree > >>> line? > >> OR you can have a non-standard patch that pjd wrote to do recursive > >> mounts of sub-filesystems. > >> it is not supposed to happen according to the standard but we have > >> found it useful. > >> Unfortnately it is written agains the old NFS Server. > >> > >> Rick, if I gave you the original pjd patch for the old server, could > >> you integrate it into the new server as an option? > >> > > A patch like this basically inserts the file system volume identifier > > in the high order bits of the fileid# (inode# if you prefer), so that > > duplicate fileid#s don't show up in a "consolidated file system" (for > > want of a better term). It also replies with the same "fake" fsid for > > all volumes involved. > > > > I see certain issues w.r.t. this: > > 1 - What happens when the exported volumes are disjoint and don't form > > one tree? (I think any just option should be restricted to volumes > > that form a tree, but I don't know an easy way to enforce that > > restriction?) > > 2 - It would be fine at this point to use the high order bits of the > > fileid#, > > since NFSv3 defines it as 64bits and FreeBSD's ino_t is 32bits. > > However, > > I believe FreeBSD is going to have to increase ino_t to 64bits soon. > > (I hope such a patch will be in FreeBSD11.) > > Once ino_t is 64bits, this option would have to assume that some # of > > the high order bits of the fileid# are always 0. Something like > > "the high order 24bits are always 0" would work ok for a while, then > > someone would build a file system large enough to overflow the 40bit > > (I know that's a lot, but some are already exceeding 32bits for # of > > fileids) field and cause trouble. > > 3 - You could get weird behaviour when the tree includes exports with > > different > > export options. This discussion includes just that and NFSv3 clients > > don't expect things to change within a mount. (An example would be > > having > > part of this consolidated tree require Kerberos authentication. > > Another > > might be having parts of the consolidated tree use different uid > > mapping > > for AUTH_SYS.) > > 4 - Some file systems (msdosfs ie. FAT) have limited capabilities w.r.t. > > what > > the NFS server can do to the file system. If one of these was imbedded > > in > > the consolidated tree, then it could cause confusion similar to #3. > > > > All in all, the "hack" is relatively easy to do, if: > > You use one kind of file system (for example ZFS) and make everything you > > are > > exporting one directory tree which is all exported in a compatible way. > > You also "know" that all the fileid#s in the underlying file systems will > > fit > > in the low order K bits of the 64bit fileid#. > > > > My biggest concern is #2, once ino_t becomes 64bits. > > > > If the collective thinks this is a good idea despite the issues above and > > can > > propose a good way to do it. (Maybe an export flag for all the volumes that > > will participate in the "consolidated file system"? The exports(5) man page > > could then try to clearly explain the limitations of its use, etc. Even > > with > > that, I suspect some would misuse the option and cause themselves grief.) > > > > Personally, since NFSv4 does this correctly, I don't see a need to "hack > > it" > > for NFSv3, but I'll leave it up to the collective. > > > > rick > > ps: Julian, you might want to repost this under a separate subject line, so > > people not interested in how ZFS can export multiple volumes without > > separate entries will read it. > > > In our environment we need to export V3 (and maybe even V2) in a > single hierarchy, even though it's multiple ZFS filesystems. > It's not dissimilar to having a separate ZFS for each user, except in > this case it's a separate ZFS for each site. > The "modified ZFS" filesystems have very special characteristics. We > are only having our very first nibbles (questions) about NFSv4. Until > now it's all NFS3. Possibly we'd only have to support it for NFSv3 > if V4 can use its native mechanisms. > > Sure. You have a particular environment where it is useful and you understand how to use it in that situation. I could do it here in about 10minutes and would do so if I needed it myself. The trick is I understand what is going on and the limitations w.r.t. doing it. If you know your file systems are all in one directory hierarchy (tree), all are ZFS and none of them even generate fileid#s that don't fit in 32bits and you are exporting them all in the same way, it's pretty easy to do. Unfortunately, that isn't what generic NFS server support for FreeBSD does. (If this is done, I think it has to be somehow restricted to the above or at least documented that it only works for the above cases.) Since an NFSv2 fileid# is 32bits, I don't believe this is practical for NFSv2 and I don't think anyone would care. Since NFSv4 does this "out of the box", I think the question is whether or not it should be done for NFSv3? The challenge would be to put it in FreeBSD in a way that people who don't necessarily understand what is "behind the curtain" can use it effectively and not run into problems. (An example being the previous thread where the different file systems are being created with different characteristics for different users. That could be confusing if the sysadmin thought it was "one volume".) I'll leave whether or not to do this up to the collective. (This is yet another one of these "easy to code" but hard to tell if it the correct thing to do situations.) If done I'd suggest: - Restricted to one file system type (ZFS or UFS or ...). The code would probably have file system specifics in it. The correct way to do this will be different for ZFS than UFS, I think? - A check for any fileid# that has high order bits set that would syslog an error. - Enabled by an export option, so it doesn't automatically apply to all file systems on the server. This also provides a place for it to be documented, including limitations. Anyhow, if anyone has an opinion on whether ir not this should be in FreeBSD, please post, rick From owner-freebsd-fs@freebsd.org Mon Aug 3 21:16:51 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 257329B2836 for ; Mon, 3 Aug 2015 21:16:51 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 001951B29 for ; Mon, 3 Aug 2015 21:16:51 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.ysv.freebsd.org (Postfix) id F17ED9B2835; Mon, 3 Aug 2015 21:16:50 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F11069B2834 for ; Mon, 3 Aug 2015 21:16:50 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 7CCAF1B28; Mon, 3 Aug 2015 21:16:50 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AGAwCn2b9V/61jaINbGQEBAYNSaQaDHbkwCYFdHQqFL0oCgW8UAQEBAQEBAYEKhCMBAQEBAgEBAQEgKyALEAIBCBgCAg0ZAgInAQkmAgQIBwQBHASIBQgNtRmWEgEBAQEBAQQBAQEBAR2BIoothBocAQEcNAeCaYFDBZR5hHuEc4QlRpNXg2ICJoINHYFvIjEHgQc6gQQBAQE X-IronPort-AV: E=Sophos;i="5.15,603,1432612800"; d="scan'208";a="228846539" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 03 Aug 2015 17:16:48 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id B126815F542; Mon, 3 Aug 2015 17:16:48 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id wW0HJPsT2_oi; Mon, 3 Aug 2015 17:16:48 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 0347D15F55D; Mon, 3 Aug 2015 17:16:48 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id iHs7qi_PPmg1; Mon, 3 Aug 2015 17:16:47 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id D825115F542; Mon, 3 Aug 2015 17:16:47 -0400 (EDT) Date: Mon, 3 Aug 2015 17:16:47 -0400 (EDT) From: Rick Macklem To: Julian Elischer Cc: lev@FreeBSD.org, fs@freebsd.org Message-ID: <958669562.8576376.1438636607852.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <55BFC69C.4000700@freebsd.org> References: <130767529.20150801150343@serebryakov.spb.ru> <55BDF5D2.3090306@digiware.nl> <1941826477.20150802210808@serebryakov.spb.ru> <55BE83FA.1060208@digiware.nl> <55BF5932.2090107@FreeBSD.org> <598623926.8227323.1438605000273.JavaMail.zimbra@uoguelph.ca> <55BFC69C.4000700@freebsd.org> Subject: Re: Multiple entries in ZFS "sharenfs" property? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: Multiple entries in ZFS "sharenfs" property? Thread-Index: 8Tcd/jmEsizO008NOqn60RbB9D4bsw== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 21:16:51 -0000 Julian Elischer wrote: > On 8/3/15 8:30 PM, Rick Macklem wrote: > > Lev wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA512 > >> > >> On 02.08.2015 23:56, Willem Jan Withagen wrote: > >> > >>> Nope, that doesn't work, as far as I can tell. You'd have to revert > >>> to editting /etc/exports. > >> Yep, I know. > >> > > Btw, even if the "hack" Julian Elischer requested was in the system, > > you would still have to export all the volumes, so this discussion > > doesn't really change. > > The only difference would be that the clients could use an NFSv3 mount > > and not an NFSv4 one. > I don't believe we use a separate export entry for each sub-fs > but I'd have to go look again. > Maybe you use the "inherited" feature of ZFS's sharenfs? (I'm pretty sure the exports will be stored/checked on a per-file system basis and will need to exist for all file systems in the hierarchy. Maybe your patch changed that, but it wasn't in what you emailed me months ago.) rick > > > Just wanted to make this clear, rick > > > >>> Or hack on 'zfs set sharenfs' to generate multiple lines, in some > >>> sort of format. Like making ';' a line separator, and then prefix > >>> each part with the volume we are modifying. > >>> > >>> Place to give it a go are in: > >>> cddl/compat/opensolaris/misc/fsshare.c:213 if (share) { > >>> fprintf(newfd, "%s\t%s\n", mountpoint, translate_opts(shareopts)); > >>> } And there split the shareopts on the split char (eg. ';') in > >>> several shareopts and then loop over them. Disadvantage is that the > >>> max length op the options is: MAXPATHLEN. So you can easily run out > >>> of space if you have many exports to do. > >> Here is PR with patch about this, from 2012 :( > >> > >> - -- > >> // Lev Serebryakov > >> -----BEGIN PGP SIGNATURE----- > >> Version: GnuPG v2 > >> > >> iQJ8BAEBCgBmBQJVv1kyXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w > >> ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF > >> QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EeP9d0P/229Jje1vw8QB9LuWkNH81xE > >> SIeYOnXRs4OoJDBmxW1xi5kaKqT+j6A9m7lWEXjm/nWGsULNcjDtP4uqQehVZDl3 > >> 7d/eGZDQgrZa5zUxvxMfb0hoWxCZNYFf6N1sPB+98ZwPVH2ADD+7h4ljusoI19gD > >> ZVjTEDeWKBln2YWWHXDDJwjFeF/AqwZzf6o0Nn6Wg2k49pAW4hjvRfVFBXYVo8CA > >> sWsD9ilgYteD6LEPdHbPy2pgprlsmIwLV84zjGb2Lk0Txs7k4CP5RnRo276wkmKY > >> lFdvlyDOFna2UsJqTbAdYD5XeTC0Zx08sqd0dPvgmugYICpgFou1rClTd6HYLDTC > >> +n1kmY8CrGn6omOwLeuckzkroPVFV/RgzRfKFdCabTfAoRe1a60Y7MTFCKUVmZnc > >> OOz2F1Ph3PJAyl5aiIZAuDikj+PEkeqbYJPQxeh2neKwOLTLj4G9oGx+CFo8t6Fk > >> Y793GqkrmIkh4O7o8LrLqhT7/II/MybC2a31RjZRfdukofIgC+CEw56IcP7tgb8U > >> L2f+QxprnQDbwFwkjJAvU/x6dPvy4cztF5vA3qtY5HAW21u+ROrAHBorANq83nZT > >> Y7Ab4vrcThoXTJnNgzN0usOz0OUdtHtVz+5OOoLNJsPVbFFrdIa2I67jW4giDiVv > >> +Vo4KfE2/wL2MpiviQQ7 > >> =GB/t > >> -----END PGP SIGNATURE----- > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > >> > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > > From owner-freebsd-fs@freebsd.org Mon Aug 3 21:21:25 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34FC39B29E0 for ; Mon, 3 Aug 2015 21:21:25 +0000 (UTC) (envelope-from email.ahmedkamal@googlemail.com) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CD1A31F6D for ; Mon, 3 Aug 2015 21:21:24 +0000 (UTC) (envelope-from email.ahmedkamal@googlemail.com) Received: by wicgj17 with SMTP id gj17so123124816wic.1 for ; Mon, 03 Aug 2015 14:21:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=QSFlJGKeT6YwoMzKY0wym5buTTpq/SukEbGs/wfL8ZA=; b=s+Bm7nreuqoR5Y/sW4qXQTokcwF19M3vZTLZtGqNM1ORgS7jnFhDdoSzCObctuRuDu BiRgVo1m8CmB+3xz7loDylGoUf79aWm2iVk6qusp3xnbWVmZraiMB1B/mj2KV+fd0ZzB mXgO8j9Gxh9yGkLtnCc4m5G3OkB4yGn4TxCSLSZkS3qj4BpWw2+GWAUXH6ENMQlmFTYi AVgOEzIc0hqTLIPiXH6Wj6Ekv6KLnSApXCjbOaSRMHTK+5vEGlPpRtuCRB1BA+jTZOts hNcbVQMz9bt2yRXafFZG8OAqUOUIEX67mJnyhr26x1ci5MWUbI9g8qC6A8s5cpksfhUt Y9yg== X-Received: by 10.181.12.20 with SMTP id em20mr38662367wid.28.1438636883287; Mon, 03 Aug 2015 14:21:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.6.143 with HTTP; Mon, 3 Aug 2015 14:21:03 -0700 (PDT) From: Ahmed Kamal Date: Mon, 3 Aug 2015 23:21:03 +0200 Message-ID: Subject: NFS automated testing To: freebsd-fs Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 21:21:25 -0000 Hi BSD'ers, I've hit more bugs in FreeBSD's NFS 4/4.1 code paths in the past few months than I would have liked. I started thinking perhaps the NFS kernel stack is not getting the testing it deserves. * Is there automated QA processes that tests NFS stack like continuous integration ? * Does this test against Linux clients and other OSs ? * Can one contribute more tests ? How ? * If all of that doesn't exist .. Is this something needed? I might spend some cycles on it if so Wishing you all rock solid file systems! Thanks! From owner-freebsd-fs@freebsd.org Tue Aug 4 02:56:30 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F234F9B2125 for ; Tue, 4 Aug 2015 02:56:29 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BDCC711A1 for ; Tue, 4 Aug 2015 02:56:29 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: by ioii16 with SMTP id i16so4295621ioi.0 for ; Mon, 03 Aug 2015 19:56:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=1FiRPCq8/1mbarkXL7ATV/0af4EftYJMPqvTPz4QJx0=; b=ZEsv15epDFO3yA/tmNiUi01ByLJOhYANkGKTgm0enOKb9QZ5qVSnYqHWFggZLTwvcV OIVOvCQ3XYDC1XmsuGtjAh9rYC61qXbPtTKt0eSw0CAKsGEMBD6Vom/UkQRpiuyLirQR ccOKNeBljAOMeNgA/ZUIzxiL69zVGM24DcBjGUd9JYW5ukABdg5ubePPtDCoC4LFzu6w kWvCdgaNWHKLPYt7htBCHFuy9mrOXl0iKe2QdBeDypHspQMKOWyrdVv5xag9ZAetOPq+ kigrbh3EGeBWBga30aA/S6c1PIm3HN6+mquJ8yss/8zay79jW9XixlNimtwa5IpCOXxw 6Z9w== MIME-Version: 1.0 X-Received: by 10.107.15.210 with SMTP id 79mr1151058iop.192.1438656989226; Mon, 03 Aug 2015 19:56:29 -0700 (PDT) Received: by 10.65.15.33 with HTTP; Mon, 3 Aug 2015 19:56:29 -0700 (PDT) In-Reply-To: References: Date: Mon, 3 Aug 2015 19:56:29 -0700 Message-ID: Subject: Re: NFS automated testing From: Mehmet Erol Sanliturk To: Ahmed Kamal Cc: freebsd-fs Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2015 02:56:30 -0000 On Mon, Aug 3, 2015 at 2:21 PM, Ahmed Kamal via freebsd-fs < freebsd-fs@freebsd.org> wrote: > Hi BSD'ers, > > I've hit more bugs in FreeBSD's NFS 4/4.1 code paths in the past few months > than I would have liked. I started thinking perhaps the NFS kernel stack is > not getting the testing it deserves. > * Is there automated QA processes that tests NFS stack like continuous > integration ? > * Does this test against Linux clients and other OSs ? > * Can one contribute more tests ? How ? > * If all of that doesn't exist .. Is this something needed? I might spend > some cycles on it if so > > Wishing you all rock solid file systems! > Thanks! > _______________________________________________ > > In Google , please search : nfs test suite nfs testing tools nfs test cases Mehmet Erol Sanliturk From owner-freebsd-fs@freebsd.org Tue Aug 4 03:56:18 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02A4E9B2FFA for ; Tue, 4 Aug 2015 03:56:18 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D235D199A for ; Tue, 4 Aug 2015 03:56:17 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-239-102.lns20.per1.internode.on.net [121.45.239.102]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t743u6E3092311 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 3 Aug 2015 20:56:09 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: changes to export whole FS hierarhy to mount it with one command on client? [changed subject] To: Rick Macklem References: <795246861.20150801140429@serebryakov.spb.ru> <1363497421.7238055.1438428070047.JavaMail.zimbra@uoguelph.ca> <1593307781.20150801143052@serebryakov.spb.ru> <55BEE668.3080303@freebsd.org> <67101638.8226696.1438604713620.JavaMail.zimbra@uoguelph.ca> <55BFC58C.6030802@freebsd.org> <987522757.8576059.1438636467059.JavaMail.zimbra@uoguelph.ca> Cc: freebsd-fs@freebsd.org From: Julian Elischer Message-ID: <55C037D0.1000606@freebsd.org> Date: Tue, 4 Aug 2015 11:56:00 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <987522757.8576059.1438636467059.JavaMail.zimbra@uoguelph.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2015 03:56:18 -0000 On 8/4/15 5:14 AM, Rick Macklem wrote: > Julian Elischer wrote: >> On 8/3/15 8:25 PM, Rick Macklem wrote: >>> Julian Elischer wrote: >>>> On 8/1/15 7:30 PM, Lev Serebryakov wrote: >>>>> Hello Rick, >>>>> >>>>> Saturday, August 1, 2015, 2:21:10 PM, you wrote: >>>>> >>>>>> To mount multiple file systems as one mount, you'll need to use NFSv4. I >>>>>> believe >>>>>> you will have to have a separate export entry in the server for each of >>>>>> the file >>>>>> systems. >>>>> So, /etc/exports needs to have BOTH v3-style exports & V4: root of >>>>> tree >>>>> line? >>>> OR you can have a non-standard patch that pjd wrote to do recursive >>>> mounts of sub-filesystems. >>>> it is not supposed to happen according to the standard but we have >>>> found it useful. >>>> Unfortnately it is written agains the old NFS Server. >>>> >>>> Rick, if I gave you the original pjd patch for the old server, could >>>> you integrate it into the new server as an option? >>>> >>> A patch like this basically inserts the file system volume identifier >>> in the high order bits of the fileid# (inode# if you prefer), so that >>> duplicate fileid#s don't show up in a "consolidated file system" (for >>> want of a better term). It also replies with the same "fake" fsid for >>> all volumes involved. >>> >>> I see certain issues w.r.t. this: >>> 1 - What happens when the exported volumes are disjoint and don't form >>> one tree? (I think any just option should be restricted to volumes >>> that form a tree, but I don't know an easy way to enforce that >>> restriction?) >>> 2 - It would be fine at this point to use the high order bits of the >>> fileid#, >>> since NFSv3 defines it as 64bits and FreeBSD's ino_t is 32bits. >>> However, >>> I believe FreeBSD is going to have to increase ino_t to 64bits soon. >>> (I hope such a patch will be in FreeBSD11.) >>> Once ino_t is 64bits, this option would have to assume that some # of >>> the high order bits of the fileid# are always 0. Something like >>> "the high order 24bits are always 0" would work ok for a while, then >>> someone would build a file system large enough to overflow the 40bit >>> (I know that's a lot, but some are already exceeding 32bits for # of >>> fileids) field and cause trouble. >>> 3 - You could get weird behaviour when the tree includes exports with >>> different >>> export options. This discussion includes just that and NFSv3 clients >>> don't expect things to change within a mount. (An example would be >>> having >>> part of this consolidated tree require Kerberos authentication. >>> Another >>> might be having parts of the consolidated tree use different uid >>> mapping >>> for AUTH_SYS.) >>> 4 - Some file systems (msdosfs ie. FAT) have limited capabilities w.r.t. >>> what >>> the NFS server can do to the file system. If one of these was imbedded >>> in >>> the consolidated tree, then it could cause confusion similar to #3. >>> >>> All in all, the "hack" is relatively easy to do, if: >>> You use one kind of file system (for example ZFS) and make everything you >>> are >>> exporting one directory tree which is all exported in a compatible way. >>> You also "know" that all the fileid#s in the underlying file systems will >>> fit >>> in the low order K bits of the 64bit fileid#. >>> >>> My biggest concern is #2, once ino_t becomes 64bits. >>> >>> If the collective thinks this is a good idea despite the issues above and >>> can >>> propose a good way to do it. (Maybe an export flag for all the volumes that >>> will participate in the "consolidated file system"? The exports(5) man page >>> could then try to clearly explain the limitations of its use, etc. Even >>> with >>> that, I suspect some would misuse the option and cause themselves grief.) >>> >>> Personally, since NFSv4 does this correctly, I don't see a need to "hack >>> it" >>> for NFSv3, but I'll leave it up to the collective. >>> >>> rick >>> ps: Julian, you might want to repost this under a separate subject line, so >>> people not interested in how ZFS can export multiple volumes without >>> separate entries will read it. >>> >> In our environment we need to export V3 (and maybe even V2) in a >> single hierarchy, even though it's multiple ZFS filesystems. >> It's not dissimilar to having a separate ZFS for each user, except in >> this case it's a separate ZFS for each site. >> The "modified ZFS" filesystems have very special characteristics. We >> are only having our very first nibbles (questions) about NFSv4. Until >> now it's all NFS3. Possibly we'd only have to support it for NFSv3 >> if V4 can use its native mechanisms. >> >> > Sure. You have a particular environment where it is useful and you understand > how to use it in that situation. I could do it here in about 10minutes and would > do so if I needed it myself. The trick is I understand what is going on and the > limitations w.r.t. doing it. > > If you know your file systems are all in one directory hierarchy (tree), all are ZFS > and none of them even generate fileid#s that don't fit in 32bits and you are exporting > them all in the same way, it's pretty easy to do. > Unfortunately, that isn't what generic NFS server support for FreeBSD does. > (If this is done, I think it has to be somehow restricted to the above or at least > documented that it only works for the above cases.) > > Since an NFSv2 fileid# is 32bits, I don't believe this is practical for NFSv2 > and I don't think anyone would care. Since NFSv4 does this "out of the box", > I think the question is whether or not it should be done for NFSv3? > > The challenge would be to put it in FreeBSD in a way that people who don't > necessarily understand what is "behind the curtain" can use it effectively > and not run into problems. (An example being the previous thread where the > different file systems are being created with different characteristics for > different users. That could be confusing if the sysadmin thought it was > "one volume".) > I'll leave whether or not to do this up to the collective. (This is yet another one > of these "easy to code" but hard to tell if it the correct thing to do situations.) > If done I'd suggest: > - Restricted to one file system type (ZFS or UFS or ...). The code would probably > have file system specifics in it. The correct way to do this will be different for > ZFS than UFS, I think? > - A check for any fileid# that has high order bits set that would syslog an error. > - Enabled by an export option, so it doesn't automatically apply to all file systems > on the server. This also provides a place for it to be documented, including limitations. Well obviously I would like it, becasue we need it and I don't want to have to maintain patches going forward. If it is in the tree YOU work on then it would automatically get updated as needed. The mount option is nice, but at the moment we just have it wired on, and only export a single (PZFS) hierarchy. (PZFS is our own heavily modified version of ZFS that uses Amazon storage(*) as a block backend in parallel with the local drives (which are more a cache.. the cloud is authoritative). (*) a gross simplification. Different parts of the hierarchy are actually be different cloud 'buckets',, (e.g. theoretically some could be amazon and some could be google cloud storage). These sub-filesystems are unified as a hierachy of ZFS filesystems into a single storage hierarchy via PZFS and exported to the user via NFS and CIFS/Samba. If I need to maintain a separate set of changes for the option then that's life. but it's of course preferable to me to have it upstreamed. p.s. to any Filesystem types.. yes we are hiring FreeBSD filesystem people.. http://panzura.com/company/careers-panzura/senior-software-engineer/.. resumes via me for fast track :-) .. > > Anyhow, if anyone has an opinion on whether ir not this should be in FreeBSD, please post, rick > From owner-freebsd-fs@freebsd.org Tue Aug 4 07:28:56 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 370129B3AF7 for ; Tue, 4 Aug 2015 07:28:56 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DB243130D for ; Tue, 4 Aug 2015 07:28:55 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (p57BB94CE.dip0.t-ipconnect.de [87.187.148.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id 2508483E402; Tue, 4 Aug 2015 09:28:21 +0200 (CEST) Received: from webmail.leidinger.net (webmail.Leidinger.net [IPv6:fd73:10c7:2053:1::3:102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id 12BE110AF; Tue, 4 Aug 2015 09:28:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=leidinger.net; s=outgoing-alex; t=1438673298; bh=FANboUP9iEkytN6JfWJ9cSAhkJbFJ0/lwLOJ7qWL+Rw=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=1JOYAYCVNk9y0gij4AMuX6GDAQEcPUZV9fGJ7X2L6tUn82t/ivAJxmVlShK68jqL9 YQ+YpO6PYlb/pMWytEtZrSjpyYVw+9zia3Ku526gEewXM3HsQwmdMdyUAiAJDRyiU+ 078TFlEUQCF+sofun6RcTkgEIwGiK4+eg0t4hca8ju06SpunSrDRriGNZ4QKtFpcW+ K6urzTcI5j7CX4GqdXZMeJHzlEodEporUxSpbMdMN5eA0GmQnOfxGHtBb/OyK9HLv6 rXv2AQOepgLOMcQA+ElVXHoYWYrx442T9v6/C8tt/AYQFz7TRsYxzeHtmZkoEshb+g vKlQQp1LyOSJA== Received: (from www@localhost) by webmail.leidinger.net (8.14.9/8.14.4/Submit) id t747SHS9055842; Tue, 4 Aug 2015 09:28:17 +0200 (CEST) (envelope-from Alexander@leidinger.net) X-Authentication-Warning: webmail.leidinger.net: www set sender to Alexander@leidinger.net using -f Received: from 217.197.101.97 ([217.197.101.97]) by webmail.leidinger.net (Horde Framework) with HTTP; Tue, 04 Aug 2015 09:28:17 +0200 Date: Tue, 04 Aug 2015 09:28:17 +0200 Message-ID: <20150804092817.Horde.aha5Ph2llansySR3xJXVMN2@webmail.leidinger.net> From: Alexander Leidinger To: Matt Churchyard Cc: Quartz , FreeBSD FS Subject: Re: ZFS: Disabling ARC? References: <55BC14B7.9010009@sneakertech.com> <20150801133635.00002ecc@Leidinger.net> <45e74b89dd754991a366a46aa5101822@SERVER.ad.usd-group.com> In-Reply-To: <45e74b89dd754991a366a46aa5101822@SERVER.ad.usd-group.com> User-Agent: Horde Application Framework 5 Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes MIME-Version: 1.0 Content-Disposition: inline X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: 2508483E402.AFFBD X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-0.1, required 6, autolearn=disabled, DKIM_SIGNED 0.10, DKIM_VALID -0.10, DKIM_VALID_AU -0.10) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1439278101.55458@adi5cCj9+tZ6rdPkUh3VEg X-EBL-Spam-Status: No X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2015 07:28:56 -0000 Quoting Matt Churchyard (from Mon, 3 Aug 2015 15:00:07 +0000): > I don't know if it's changed, but even changing ARC to metadata only > used to cause massive performance issues when reading large files in > small chunks. Reading a 128k ZFS record in 4k chunks would cause ZFS > to read the same 128k record from disk 32 times. > There's a forum thread about it here - > https://forums.freebsd.org/threads/zfs-primarycache-all-versus-metadata.45555/ When using "metadata" as the setting, you tell that you are OK to trade in some (data access, not metadata-access) performance for memory. Determining if the resulting performance is acceptable is up to you. If you blindly set the value without validating that it is OK for you, you shall not complain (= "measure, change, measure" and not "measure, change, hope"). I agree that not for all changes the resulting effects are 100% obvious and explicitely stating the edge-cases is appreciated. Feel free to add some words about it to https://wiki.freebsd.org/ZFSTuningGuide Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0xC773696B3BAC17DC http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0xC773696B3BAC17DC From owner-freebsd-fs@freebsd.org Wed Aug 5 02:11:54 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8943F9B34A2 for ; Wed, 5 Aug 2015 02:11:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5A18D11D4 for ; Wed, 5 Aug 2015 02:11:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t752BsKp066985 for ; Wed, 5 Aug 2015 02:11:54 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 201912] panic in smbfs during mount Date: Wed, 05 Aug 2015 02:11:53 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mi@ALDAN.algebra.com X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Aug 2015 02:11:54 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201912 Mikhail T. changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mi@ALDAN.algebra.com --- Comment #4 from Mikhail T. --- Created attachment 159559 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=159559&action=edit Output of "bt full" from kgdb I think, I'm seeing the same problem on 9.3 stable from June 23. See attachment for "bt full". Here is the ident output: /boot/kernel/smbfs.ko: $FreeBSD: stable/9/sys/kern/md4c.c 139804 2005-01-06 23:35:40Z imp $ $FreeBSD: stable/9/sys/netsmb/smb_conn.c 249132 2013-04-05 08:22:11Z mav $ $FreeBSD: stable/9/sys/netsmb/smb_dev.c 206361 2010-04-07 16:50:38Z joel $ $FreeBSD: stable/9/sys/netsmb/smb_trantcp.c 264425 2014-04-13 22:00:50Z dteske $ $FreeBSD: stable/9/sys/netsmb/smb_smb.c 230196 2012-01-16 05:15:13Z kevlo $ $FreeBSD: stable/9/sys/netsmb/smb_subr.c 249132 2013-04-05 08:22:11Z mav $ $FreeBSD: stable/9/sys/netsmb/smb_rq.c 249132 2013-04-05 08:22:11Z mav $ $FreeBSD: stable/9/sys/netsmb/smb_usr.c 206361 2010-04-07 16:50:38Z joel $ $FreeBSD: stable/9/sys/netsmb/smb_crypt.c 161523 2006-08-22 03:05:51Z marcel $ $FreeBSD: stable/9/sys/netsmb/smb_iod.c 265246 2014-05-02 21:54:36Z ae $ $FreeBSD: stable/8/sys/crypto/des/des_ecb.c 130443 2004-06-14 00:38:54Z obrien $ $FreeBSD: stable/8/sys/crypto/des/des_setkey.c 130443 2004-06-14 00:38:54Z obrien $ -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Thu Aug 6 00:22:21 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7F779B1018 for ; Thu, 6 Aug 2015 00:22:21 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-io0-x22d.google.com (mail-io0-x22d.google.com [IPv6:2607:f8b0:4001:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 797E7F88 for ; Thu, 6 Aug 2015 00:22:21 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: by ioii16 with SMTP id i16so66135497ioi.0 for ; Wed, 05 Aug 2015 17:22:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=MLBreon3c1tBUqqKDRY/8jZMBZ7lP8pbcZFdEmD5VE0=; b=ZhBKxh4pJAobZfgs7lUc9qOYUfpnCWvRXWvNIG/imBK1ZRlLs7hay0KMjpUI2JaAdL zf2hcnsID4RiraXjzEtazQ5wCOY1k8U/N5JStISuhp9BK6cEUBcUk12ZtsoxlJqce7UW ttQmwZduZnj1j12Fjb02YyvyDwgb3wl4iFxdU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to:cc :content-type; bh=MLBreon3c1tBUqqKDRY/8jZMBZ7lP8pbcZFdEmD5VE0=; b=XeS4gxM4AWXuTAUz+R0252T47kun1jjPeWxbXAxws7LhS+VKU7vXfecKleL9lVAFt3 LMSqCVlMoLdfgdhMO7vND7isi2lIzZbCl1ei9dDE+mo6m6C48fpmJPp4oHXbEuO36H2j xKEaSEcyK7buHjk3pjTLyF7foTbKFG99XBgYdVBWLnd2d01ybRkoconbk7EQWUXSxm6s 0Zr0hCcr11SHNAif4FxkgRj/u97MVNnajuTxvM+F+mn70Jb2tTzBIOmqVxp+/Uq6rxi7 LCT4IH1ytwih0dgI+WSliuRweOTBazAHBPSafXZ/cHYCdyt88UZ9I5qVF9QKfaa+4/64 5DxA== X-Gm-Message-State: ALoCoQmyUV6ouqt5OPrhmAds82Q1zOpIecKaNp2D03WHLh8oSX5puFEU8ABnRlZFS2ZLKbwjRdI8 MIME-Version: 1.0 X-Received: by 10.107.131.199 with SMTP id n68mr13514188ioi.63.1438820540648; Wed, 05 Aug 2015 17:22:20 -0700 (PDT) Received: by 10.36.85.197 with HTTP; Wed, 5 Aug 2015 17:22:20 -0700 (PDT) Date: Wed, 5 Aug 2015 17:22:20 -0700 Message-ID: Subject: OpenZFS Developer Summit 2015 - talk proposals From: Matthew Ahrens To: developer Cc: illumos-zfs , zfs-discuss , freebsd-fs Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Aug 2015 00:22:21 -0000 The third annual OpenZFS Developer Summit will be held in San Francisco, October 19 - 20, 2015. I'd like to especially remind folks that the deadline for talk proposals is August 31st. *If you want to give a talk, submit a proposal via email to admin@open-zfs.org , including a 1-2 paragraph abstract.* We would love to hear about a new feature you're developing, a proposal for a new procedure or feature, or a big performance win. We're also interested in especially compelling stories of how and why you are using OpenZFS to run your business. I (Matt) will review all of the submissions and select talks that will be most interesting for the audience. The registration fee will be waived for speakers. See http://www.open-zfs.org/wiki/OpenZFS_Developer_Summit for all of the details, including slides and videos from previous years' conferences. --matt From owner-freebsd-fs@freebsd.org Thu Aug 6 04:33:55 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24DF59B4D81 for ; Thu, 6 Aug 2015 04:33:55 +0000 (UTC) (envelope-from phillip.nordwall@gmail.com) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E83BE916; Thu, 6 Aug 2015 04:33:54 +0000 (UTC) (envelope-from phillip.nordwall@gmail.com) Received: by ioeg141 with SMTP id g141so70050926ioe.3; Wed, 05 Aug 2015 21:33:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=M9bECEqwg5UU0gWcS8hUwunuqO5mlgeRghDFi34d91o=; b=a6R2jPTi5S5utuNc5v7/FZaqQnCg5H8q5wi0UR5nS/3XlUZ08adj3eGsR/OoU3Jcly DW2o6ixqQtkSWFBt/ptOTcyRtt0FoLBRSl9oXeuYF9PaudNJZE+IYCiQi7W997Kr+M+c xU0+6sORBVhYn0Ko4YLqEOk38G1JWqeTd6/Bc1G2H77PrM/FtdvwP2HLAgAyQN4CBaNC 0P7EsfuLGtIP6Rya165PRnxLZrxKsZk5iLF0Ms4VZStsoXV79TCeVASh7aayeWcsoZ/h iZo6r0hfOsxDFC+mChiPfIVkFfbZhbT3LsTZzFh81i9qkM9ibuQyISWICfvVgH1IL+8R Ptjg== MIME-Version: 1.0 X-Received: by 10.107.18.31 with SMTP id a31mr16366853ioj.22.1438835634311; Wed, 05 Aug 2015 21:33:54 -0700 (PDT) Received: by 10.107.37.149 with HTTP; Wed, 5 Aug 2015 21:33:54 -0700 (PDT) In-Reply-To: <130767529.20150801150343@serebryakov.spb.ru> References: <130767529.20150801150343@serebryakov.spb.ru> Date: Wed, 5 Aug 2015 21:33:54 -0700 Message-ID: Subject: Re: Multiple entries in ZFS "sharenfs" property? From: Phillip Nordwall To: lev@freebsd.org Cc: FreeBSD FS Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Aug 2015 04:33:55 -0000 Last time I played with this I was able to place a newline character in the sharenfs property, followed by a full entry as it would appear in /etc/exports. If the user setting this was root this would work. If the user setting this just had sharenfs modification permissions the new share would not take effect until the system rebooted, or root made a different export change causing nfs to be restarted. One of my tests shared / out to everyone, without squashing root, with only the permission to change sharenfs, upon reboot of the server. Phillip Nordwall From owner-freebsd-fs@freebsd.org Fri Aug 7 20:32:42 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F1A609B67FF for ; Fri, 7 Aug 2015 20:32:42 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CF9E6D5F for ; Fri, 7 Aug 2015 20:32:42 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (75-48-78-19.lightspeed.cncrca.sbcglobal.net [75.48.78.19]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C5B79B9C8; Fri, 7 Aug 2015 16:32:41 -0400 (EDT) From: John Baldwin To: freebsd-fs@freebsd.org Subject: Re: BTX halted Date: Fri, 07 Aug 2015 12:29:07 -0700 Message-ID: <2241493.vMCzZXQH3f@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-PRERELEASE; KDE/4.14.3; amd64; ; ) In-Reply-To: <1429520064.12915.56.camel@data-b104.adm.slu.se> References: <1429520064.12915.56.camel@data-b104.adm.slu.se> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="iso-8859-1" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 07 Aug 2015 16:32:42 -0400 (EDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Aug 2015 20:32:43 -0000 On Monday, April 20, 2015 08:54:24 AM Karli Sj=F6berg wrote: > Hey all! >=20 > I was just upgrading one of our storage's to 10-STABLE as per Bug > 191348[*] when something strange happened after writing new boot code= > and rebooting: >=20 > BTX loader 1.00 BTX version is 1.02 > Consoles: internal video/keyboard > BIOS drive C: is disk8 > BIOS drive D: is disk9 >=20 > int=3D00000000 err=3D00000000 efl=3D00010246 eip=3D00037d34 > eax=3D00000001 ebx=3D00000000 ecx=3D00000000 edx=3D00000000 > esi=3D00000000 edi=3D00000000 ebp=3D0008f600 esp=3D0008f598 > cs=3D002b ds=3D0033 es=3D0033 fs=3D0033 gs=3D0033 ss=3D0033 > cs:eip=3Df7 35 d0 f4 03 00 85 f6-74 05 89 3e 89 5e 04 89 > c2 e9 cc 00 00 00 66 c7-45 ea 00 00 89 d8 c1 e8 > ss:esp=3D00 00 20 00 00 00 00 00-00 00 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 > BTX halted >=20 > Never happened to me before. I then booted with a 10.1-RELEASE CD and= > wrote bootcode from that, same problem. 9.3; same there... >=20 > Any pointers? Hmm, looks like a divide-by-zero fault: 00000000 F735D0F40300 div dword [dword 0x3f4d0] 00000006 85F6 test esi,esi 00000008 7405 jz 0xf 0000000A 893E mov [esi],edi 0000000C 895E04 mov [esi+0x4],ebx 0000000F 89C2 mov edx,eax 00000011 E9CC000000 jmp dword 0xe2 00000016 66C745EA0000 mov word [ebp-0x16],0x0 0000001C 89D8 mov eax,ebx Are you using ZFS or UFS? Also, do you have the world build from when = you did this? If so we can use gdb on the 'loader.sym' file to map the fau= lting instruction (eip above) to a source line. --=20 John Baldwin From owner-freebsd-fs@freebsd.org Fri Aug 7 21:45:03 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EB169B6420 for ; Fri, 7 Aug 2015 21:45:03 +0000 (UTC) (envelope-from wfc@mintsol.com) Received: from scully.mintsol.com (scully.mintsol.com [199.182.77.206]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27469F06; Fri, 7 Aug 2015 21:45:02 +0000 (UTC) (envelope-from wfc@mintsol.com) Received: from mintsol.com (adsl-99-96-152-170.dsl.sfldmi.sbcglobal.net [99.96.152.170]) by scully.mintsol.com with esmtp; Fri, 07 Aug 2015 17:39:53 -0400 id 0008FC7A.0000000055C525A9.000161EC Received: from localhost (localhost [127.0.0.1]) (IDENT: uid 1002) by mintsol.com with esmtp; Fri, 07 Aug 2015 17:39:53 -0400 id 00000232.55C525A9.00003DB9 Date: Fri, 7 Aug 2015 17:39:53 -0400 (EDT) From: Walter Cramer To: John Baldwin cc: freebsd-fs@freebsd.org Subject: Re: BTX halted In-Reply-To: <2241493.vMCzZXQH3f@ralph.baldwin.cx> Message-ID: <20150807172126.H12135@mulder.mintsol.com> References: <1429520064.12915.56.camel@data-b104.adm.slu.se> <2241493.vMCzZXQH3f@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Aug 2015 21:45:03 -0000 FWIW, I've gotten very similar errors with SuperMicro H8SMi-2 motherboards= =20 (NOT on their FreeBSD Compatible list) with a 10.1-Release & RootOnZFS=20 test HD. Switching the BIOS settings from "optimal" to "failsafe" fixes=20 the problem; setting ACPI to v3.0 brings it back. I have an idle H2SMi-2 test system, but not much free time in the next=20 week. Walter Cramer On Fri, 7 Aug 2015, John Baldwin wrote: > On Monday, April 20, 2015 08:54:24 AM Karli Sj=F6berg wrote: >> Hey all! >> >> I was just upgrading one of our storage's to 10-STABLE as per Bug >> 191348[*] when something strange happened after writing new boot code >> and rebooting: >> >> BTX loader 1.00 BTX version is 1.02 >> Consoles: internal video/keyboard >> BIOS drive C: is disk8 >> BIOS drive D: is disk9 >> >> int=3D00000000 err=3D00000000 efl=3D00010246 eip=3D00037d34 >> eax=3D00000001 ebx=3D00000000 ecx=3D00000000 edx=3D00000000 >> esi=3D00000000 edi=3D00000000 ebp=3D0008f600 esp=3D0008f598 >> cs=3D002b ds=3D0033 es=3D0033 fs=3D0033 gs=3D0033 ss=3D0033 >> cs:eip=3Df7 35 d0 f4 03 00 85 f6-74 05 89 3e 89 5e 04 89 >> c2 e9 cc 00 00 00 66 c7-45 ea 00 00 89 d8 c1 e8 >> ss:esp=3D00 00 20 00 00 00 00 00-00 00 00 00 00 00 00 00 >> 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 >> BTX halted >> >> Never happened to me before. I then booted with a 10.1-RELEASE CD and >> wrote bootcode from that, same problem. 9.3; same there... >> >> Any pointers? > > Hmm, looks like a divide-by-zero fault: > > 00000000 F735D0F40300 div dword [dword 0x3f4d0] > 00000006 85F6 test esi,esi > 00000008 7405 jz 0xf > 0000000A 893E mov [esi],edi > 0000000C 895E04 mov [esi+0x4],ebx > 0000000F 89C2 mov edx,eax > 00000011 E9CC000000 jmp dword 0xe2 > 00000016 66C745EA0000 mov word [ebp-0x16],0x0 > 0000001C 89D8 mov eax,ebx > > Are you using ZFS or UFS? Also, do you have the world build from when yo= u > did this? If so we can use gdb on the 'loader.sym' file to map the fault= ing > instruction (eip above) to a source line. > > --=20 > John Baldwin > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Sat Aug 8 00:29:32 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 598DC9B6F74 for ; Sat, 8 Aug 2015 00:29:32 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id DCB3219AC; Sat, 8 Aug 2015 00:29:31 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DtBQCvTMVV/61jaINbg29qBYMevBCCQ4M2AoFzEQEBAQEBAQGBCoQjAQEBAwEjBFIFCwIBCBgCAg0ZAgJXAgQsiA0IDbdPlgMBAQEBAQUBAQEBAR2BIoothDAOAhU0B4JpgUMFhx2Na4UAgmeGPEaDXYMXjQ+DYwImgg4cFYFaIoE2AR8jgQQBAQE X-IronPort-AV: E=Sophos;i="5.15,632,1432612800"; d="scan'208";a="229709396" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 07 Aug 2015 20:28:22 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 8F78515F5D2; Fri, 7 Aug 2015 20:28:22 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id Ka-yWqHx18pl; Fri, 7 Aug 2015 20:28:21 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 35AA915F5DA; Fri, 7 Aug 2015 20:28:21 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id bKKwLjTcja73; Fri, 7 Aug 2015 20:28:21 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 17A6315F5D2; Fri, 7 Aug 2015 20:28:21 -0400 (EDT) Date: Fri, 7 Aug 2015 20:28:20 -0400 (EDT) From: Rick Macklem To: Julian Elischer Cc: freebsd-fs@freebsd.org Message-ID: <1970606860.13124005.1438993700838.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <55C037D0.1000606@freebsd.org> References: <795246861.20150801140429@serebryakov.spb.ru> <1363497421.7238055.1438428070047.JavaMail.zimbra@uoguelph.ca> <1593307781.20150801143052@serebryakov.spb.ru> <55BEE668.3080303@freebsd.org> <67101638.8226696.1438604713620.JavaMail.zimbra@uoguelph.ca> <55BFC58C.6030802@freebsd.org> <987522757.8576059.1438636467059.JavaMail.zimbra@uoguelph.ca> <55C037D0.1000606@freebsd.org> Subject: Re: changes to export whole FS hierarhy to mount it with one command on client? [changed subject] MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: changes to export whole FS hierarhy to mount it with one command on client? [changed subject] Thread-Index: +P5YmT8KfOaGOxZCs0iB7xzPOEUAhw== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 00:29:32 -0000 Julian Elischer wrote: > On 8/4/15 5:14 AM, Rick Macklem wrote: > > Julian Elischer wrote: > >> On 8/3/15 8:25 PM, Rick Macklem wrote: > >>> Julian Elischer wrote: > >>>> On 8/1/15 7:30 PM, Lev Serebryakov wrote: > >>>>> Hello Rick, > >>>>> > >>>>> Saturday, August 1, 2015, 2:21:10 PM, you wrote: > >>>>> > >>>>>> To mount multiple file systems as one mount, you'll need to use NFSv4. > >>>>>> I > >>>>>> believe > >>>>>> you will have to have a separate export entry in the server for each > >>>>>> of > >>>>>> the file > >>>>>> systems. > >>>>> So, /etc/exports needs to have BOTH v3-style exports & V4: root of > >>>>> tree > >>>>> line? > >>>> OR you can have a non-standard patch that pjd wrote to do recursive > >>>> mounts of sub-filesystems. > >>>> it is not supposed to happen according to the standard but we have > >>>> found it useful. > >>>> Unfortnately it is written agains the old NFS Server. > >>>> > >>>> Rick, if I gave you the original pjd patch for the old server, could > >>>> you integrate it into the new server as an option? > >>>> > >>> A patch like this basically inserts the file system volume identifier > >>> in the high order bits of the fileid# (inode# if you prefer), so that > >>> duplicate fileid#s don't show up in a "consolidated file system" (for > >>> want of a better term). It also replies with the same "fake" fsid for > >>> all volumes involved. > >>> > >>> I see certain issues w.r.t. this: > >>> 1 - What happens when the exported volumes are disjoint and don't form > >>> one tree? (I think any just option should be restricted to volumes > >>> that form a tree, but I don't know an easy way to enforce that > >>> restriction?) > >>> 2 - It would be fine at this point to use the high order bits of the > >>> fileid#, > >>> since NFSv3 defines it as 64bits and FreeBSD's ino_t is 32bits. > >>> However, > >>> I believe FreeBSD is going to have to increase ino_t to 64bits > >>> soon. > >>> (I hope such a patch will be in FreeBSD11.) > >>> Once ino_t is 64bits, this option would have to assume that some # > >>> of > >>> the high order bits of the fileid# are always 0. Something like > >>> "the high order 24bits are always 0" would work ok for a while, > >>> then > >>> someone would build a file system large enough to overflow the > >>> 40bit > >>> (I know that's a lot, but some are already exceeding 32bits for # > >>> of > >>> fileids) field and cause trouble. > >>> 3 - You could get weird behaviour when the tree includes exports with > >>> different > >>> export options. This discussion includes just that and NFSv3 > >>> clients > >>> don't expect things to change within a mount. (An example would be > >>> having > >>> part of this consolidated tree require Kerberos authentication. > >>> Another > >>> might be having parts of the consolidated tree use different uid > >>> mapping > >>> for AUTH_SYS.) > >>> 4 - Some file systems (msdosfs ie. FAT) have limited capabilities w.r.t. > >>> what > >>> the NFS server can do to the file system. If one of these was > >>> imbedded > >>> in > >>> the consolidated tree, then it could cause confusion similar to #3. > >>> > >>> All in all, the "hack" is relatively easy to do, if: > >>> You use one kind of file system (for example ZFS) and make everything you > >>> are > >>> exporting one directory tree which is all exported in a compatible way. > >>> You also "know" that all the fileid#s in the underlying file systems will > >>> fit > >>> in the low order K bits of the 64bit fileid#. > >>> > >>> My biggest concern is #2, once ino_t becomes 64bits. > >>> > >>> If the collective thinks this is a good idea despite the issues above and > >>> can > >>> propose a good way to do it. (Maybe an export flag for all the volumes > >>> that > >>> will participate in the "consolidated file system"? The exports(5) man > >>> page > >>> could then try to clearly explain the limitations of its use, etc. Even > >>> with > >>> that, I suspect some would misuse the option and cause themselves grief.) > >>> > >>> Personally, since NFSv4 does this correctly, I don't see a need to "hack > >>> it" > >>> for NFSv3, but I'll leave it up to the collective. > >>> > >>> rick > >>> ps: Julian, you might want to repost this under a separate subject line, > >>> so > >>> people not interested in how ZFS can export multiple volumes > >>> without > >>> separate entries will read it. > >>> > >> In our environment we need to export V3 (and maybe even V2) in a > >> single hierarchy, even though it's multiple ZFS filesystems. > >> It's not dissimilar to having a separate ZFS for each user, except in > >> this case it's a separate ZFS for each site. > >> The "modified ZFS" filesystems have very special characteristics. We > >> are only having our very first nibbles (questions) about NFSv4. Until > >> now it's all NFS3. Possibly we'd only have to support it for NFSv3 > >> if V4 can use its native mechanisms. > >> > >> > > Sure. You have a particular environment where it is useful and you > > understand > > how to use it in that situation. I could do it here in about 10minutes and > > would > > do so if I needed it myself. The trick is I understand what is going on and > > the > > limitations w.r.t. doing it. > > > > If you know your file systems are all in one directory hierarchy (tree), > > all are ZFS > > and none of them even generate fileid#s that don't fit in 32bits and you > > are exporting > > them all in the same way, it's pretty easy to do. > > Unfortunately, that isn't what generic NFS server support for FreeBSD does. > > (If this is done, I think it has to be somehow restricted to the above or > > at least > > documented that it only works for the above cases.) > > > > Since an NFSv2 fileid# is 32bits, I don't believe this is practical for > > NFSv2 > > and I don't think anyone would care. Since NFSv4 does this "out of the > > box", > > I think the question is whether or not it should be done for NFSv3? > > > > The challenge would be to put it in FreeBSD in a way that people who don't > > necessarily understand what is "behind the curtain" can use it effectively > > and not run into problems. (An example being the previous thread where the > > different file systems are being created with different characteristics for > > different users. That could be confusing if the sysadmin thought it was > > "one volume".) > > I'll leave whether or not to do this up to the collective. (This is yet > > another one > > of these "easy to code" but hard to tell if it the correct thing to do > > situations.) > > If done I'd suggest: > > - Restricted to one file system type (ZFS or UFS or ...). The code would > > probably > > have file system specifics in it. The correct way to do this will be > > different for > > ZFS than UFS, I think? > > - A check for any fileid# that has high order bits set that would syslog an > > error. > > - Enabled by an export option, so it doesn't automatically apply to all > > file systems > > on the server. This also provides a place for it to be documented, > > including limitations. > Oh, I remembered another issue related to this: - Until FreeBSD changes ino_t to 64bits, any "virtual consolidated file system" from an NFSv3 server needs to provide fileid#s that are unique in the low order 32bits so that it won't break the FreeBSD client. --> I think this implies that it shouldn't go into head until after ino_t becomes 64bits, at the earliest. Even then this option would break older FreeBSD clients. (The documentation for the export option would need to mention this.) Without 64bit ino_t, the fileid the server puts on the wire would need to be something like 24bits from the file system on the server + 8bits for which file system it is (to uniquify the 32bit value). Expecting the fileid# to fit in 24bits on the underlying file system seems too restrictive to me. rick > Well obviously I would like it, becasue we need it and I don't want to > have to maintain patches going forward. > If it is in the tree YOU work on then it would automatically get > updated as needed. The mount option is nice, but at the moment we just > have it wired on, and only export a single (PZFS) hierarchy. (PZFS is > our own heavily modified version of ZFS that uses Amazon storage(*) as > a block backend in parallel with the local drives (which are more a > cache.. the cloud is authoritative). > > (*) a gross simplification. > > Different parts of the hierarchy are actually be different cloud > 'buckets',, (e.g. theoretically some could be amazon and some could be > google cloud storage). These sub-filesystems are unified as a > hierachy of ZFS filesystems into a single storage hierarchy via PZFS > and exported to the user via NFS and CIFS/Samba. > > If I need to maintain a separate set of changes for the option then > that's life. but it's of course preferable to me to have it upstreamed. > > p.s. to any Filesystem types.. yes we are hiring FreeBSD filesystem > people.. > http://panzura.com/company/careers-panzura/senior-software-engineer/.. > resumes via me for fast track :-) .. > > > > > > > > > Anyhow, if anyone has an opinion on whether ir not this should be in > > FreeBSD, please post, rick > > > > From owner-freebsd-fs@freebsd.org Sat Aug 8 10:06:41 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 712E09B57B7 for ; Sat, 8 Aug 2015 10:06:41 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5A1E23B1 for ; Sat, 8 Aug 2015 10:06:41 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 573239B57B6; Sat, 8 Aug 2015 10:06:41 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56CD59B57B5 for ; Sat, 8 Aug 2015 10:06:41 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1BCD33B0 for ; Sat, 8 Aug 2015 10:06:40 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 3D7BC1534C6 for ; Sat, 8 Aug 2015 12:06:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QQyUEkQMZaBB; Sat, 8 Aug 2015 12:06:06 +0200 (CEST) Received: from [192.168.10.10] (asus [192.168.10.10]) by smtp.digiware.nl (Postfix) with ESMTPA id D55981534C7 for ; Sat, 8 Aug 2015 12:06:06 +0200 (CEST) To: fs@freebsd.org From: Willem Jan Withagen Subject: Using SSDs as swap Message-ID: <55C5D48E.6010605@digiware.nl> Date: Sat, 8 Aug 2015 12:06:06 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 10:06:41 -0000 one of the following commits just passed with this in the log, and it triggered again a question I've been having for some time again already. ---- Log: Enable BIO_DELETE passthru in GELI, so TRIM/UNMAP can work as expected when GELI is used on a SSD or inside virtual machine, so that guest can tell host that it is no longer using some of the storage. ----- In ZFS I slice my SSD's into log and caches, but on a a server with little memory (which can't be grown) I use a partion on each ssd as swap as well. So swappinging does not have to seek, and has faster loading time. To allocate a few GB on aan SSD to swap is not really all that painfull, given current sizes, but the speed difference with regular spindels is impressive. But the questions are: 1) Does the swap driver understand that backing-store needs a TRIM? 1a) if not would it be useful, and what would it take to implement? 2) Does the SSD "suffer" unneeded from swapping on it? Thanx, --WjW From owner-freebsd-fs@freebsd.org Sat Aug 8 10:29:13 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 741079B5D8B for ; Sat, 8 Aug 2015 10:29:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5967BE9F for ; Sat, 8 Aug 2015 10:29:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 565319B5D8A; Sat, 8 Aug 2015 10:29:13 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55EA69B5D89 for ; Sat, 8 Aug 2015 10:29:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C76E3E9E for ; Sat, 8 Aug 2015 10:29:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id t78AT27l005719 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 8 Aug 2015 13:29:02 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua t78AT27l005719 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id t78AT0Nn005718; Sat, 8 Aug 2015 13:29:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 8 Aug 2015 13:29:00 +0300 From: Konstantin Belousov To: Willem Jan Withagen Cc: fs@freebsd.org Subject: Re: Using SSDs as swap Message-ID: <20150808102900.GA2072@kib.kiev.ua> References: <55C5D48E.6010605@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55C5D48E.6010605@digiware.nl> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 10:29:13 -0000 On Sat, Aug 08, 2015 at 12:06:06PM +0200, Willem Jan Withagen wrote: > one of the following commits just passed with this in the log, and it > triggered again a question I've been having for some time again already. > > ---- > Log: > Enable BIO_DELETE passthru in GELI, so TRIM/UNMAP can work as expected > when > GELI is used on a SSD or inside virtual machine, so that guest can tell > host that it is no longer using some of the storage. > ----- > > In ZFS I slice my SSD's into log and caches, but on a a server with > little memory (which can't be grown) I use a partion on each ssd as swap > as well. So swappinging does not have to seek, and has faster loading > time. To allocate a few GB on aan SSD to swap is not really all that > painfull, given current sizes, but the speed difference with regular > spindels is impressive. > > But the questions are: > 1) Does the swap driver understand that backing-store needs a TRIM? No. > 1a) if not would it be useful, and what would it take to implement? One good thing is that it is simply the question of coding: the VM already has a place where it informs the swap pager that the page copy in swap is no longer needed. this is the vm_pager_page_unswapped() call and swap pager method swap_pager_unswapped(). swp_pager_meta_ctl() would need to issue BIO_DELETE to the backing storage. On the other hand, note that this would increase the amount of work performed, even for the swap volumes located on the rotating media, which is more typical and reasonable setup. I think an implementation and a knob to turn it off, or configure per swap partition, would be reasonable. > 2) Does the SSD "suffer" unneeded from swapping on it? Depends on your swapping activity, but yes, I believe that intense swapping would wear SSD in the short time. From owner-freebsd-fs@freebsd.org Sat Aug 8 10:38:16 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D0AC9B503C for ; Sat, 8 Aug 2015 10:38:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 21B871365 for ; Sat, 8 Aug 2015 10:38:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1ED859B503B; Sat, 8 Aug 2015 10:38:16 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1E6999B503A for ; Sat, 8 Aug 2015 10:38:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A7C321364 for ; Sat, 8 Aug 2015 10:38:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id t78AcAbf008072 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 8 Aug 2015 13:38:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua t78AcAbf008072 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id t78AcAjT008071; Sat, 8 Aug 2015 13:38:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 8 Aug 2015 13:38:10 +0300 From: Konstantin Belousov To: Willem Jan Withagen Cc: fs@freebsd.org Subject: Re: Using SSDs as swap Message-ID: <20150808103810.GB2072@kib.kiev.ua> References: <55C5D48E.6010605@digiware.nl> <20150808102900.GA2072@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150808102900.GA2072@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 10:38:16 -0000 On Sat, Aug 08, 2015 at 01:29:00PM +0300, Konstantin Belousov wrote: > On Sat, Aug 08, 2015 at 12:06:06PM +0200, Willem Jan Withagen wrote: > > one of the following commits just passed with this in the log, and it > > triggered again a question I've been having for some time again already. > > > > ---- > > Log: > > Enable BIO_DELETE passthru in GELI, so TRIM/UNMAP can work as expected > > when > > GELI is used on a SSD or inside virtual machine, so that guest can tell > > host that it is no longer using some of the storage. > > ----- > > > > In ZFS I slice my SSD's into log and caches, but on a a server with > > little memory (which can't be grown) I use a partion on each ssd as swap > > as well. So swappinging does not have to seek, and has faster loading > > time. To allocate a few GB on aan SSD to swap is not really all that > > painfull, given current sizes, but the speed difference with regular > > spindels is impressive. > > > > But the questions are: > > 1) Does the swap driver understand that backing-store needs a TRIM? > No. > > > 1a) if not would it be useful, and what would it take to implement? > One good thing is that it is simply the question of coding: the VM > already has a place where it informs the swap pager that the page copy > in swap is no longer needed. this is the vm_pager_page_unswapped() call > and swap pager method swap_pager_unswapped(). swp_pager_meta_ctl() would > need to issue BIO_DELETE to the backing storage. > > On the other hand, note that this would increase the amount of work > performed, even for the swap volumes located on the rotating media, > which is more typical and reasonable setup. > > I think an implementation and a knob to turn it off, or configure per > swap partition, would be reasonable. One additional thing: while BIO_DELETE is in progress, the swap block cannot be marked free, since otherwise we could write other page and get it obliterated with the TRIM. This can be done async, but the consequence is that swap space would be released and usable some time after the page-in. This will affect loads which are close to OOM. > > > 2) Does the SSD "suffer" unneeded from swapping on it? > Depends on your swapping activity, but yes, I believe that intense swapping > would wear SSD in the short time. From owner-freebsd-fs@freebsd.org Sat Aug 8 11:09:13 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A3F19B57CE for ; Sat, 8 Aug 2015 11:09:13 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 41B7BC for ; Sat, 8 Aug 2015 11:09:13 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 3E9289B57CD; Sat, 8 Aug 2015 11:09:13 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D2F79B57CC for ; Sat, 8 Aug 2015 11:09:13 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF407B for ; Sat, 8 Aug 2015 11:09:12 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 92C8B15344D; Sat, 8 Aug 2015 13:09:09 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YPgb8gqH2v5L; Sat, 8 Aug 2015 13:08:59 +0200 (CEST) Received: from [192.168.10.10] (asus [192.168.10.10]) by smtp.digiware.nl (Postfix) with ESMTPA id 523421534C0; Sat, 8 Aug 2015 13:08:59 +0200 (CEST) Subject: Re: Using SSDs as swap To: Konstantin Belousov References: <55C5D48E.6010605@digiware.nl> <20150808102900.GA2072@kib.kiev.ua> Cc: fs@freebsd.org From: Willem Jan Withagen Message-ID: <55C5E34B.9010905@digiware.nl> Date: Sat, 8 Aug 2015 13:08:59 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20150808102900.GA2072@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 11:09:13 -0000 On 8-8-2015 12:29, Konstantin Belousov wrote: > On Sat, Aug 08, 2015 at 12:06:06PM +0200, Willem Jan Withagen wrote: >> 2) Does the SSD "suffer" unneeded from swapping on it? > Depends on your swapping activity, but yes, I believe that intense swapping > would wear SSD in the short time. Would it be more intense than being beaten to death as a ARC cache?? If I look at most of my systems I try to prevent them even using really swap since that usually kills performance big time... So things get swapped out under pressure, but rarely get swapped back in, because the LRU properties of that part of the application minimise that chance of things getting paged back in. So the number of times that swap actually gets used is rather seldom. In writting swap, how are the allocations made where things are swapped. Do things always end up in a lineair order on swap writting things as close to the start as slots in swap allow it. SSD would profit from it being sort of a circular buffer... (Guess I have to try and understand the swap code.... ) --WjW From owner-freebsd-fs@freebsd.org Sat Aug 8 11:23:34 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3EAE39B5EFC for ; Sat, 8 Aug 2015 11:23:34 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 237E8DA3 for ; Sat, 8 Aug 2015 11:23:34 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 22B259B5EFB; Sat, 8 Aug 2015 11:23:34 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 087289B5EFA for ; Sat, 8 Aug 2015 11:23:34 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8260ADA2 for ; Sat, 8 Aug 2015 11:23:33 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 4D1401534C8; Sat, 8 Aug 2015 13:23:31 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6sRRi-b6HbSJ; Sat, 8 Aug 2015 13:23:03 +0200 (CEST) Received: from [192.168.10.10] (asus [192.168.10.10]) by smtp.digiware.nl (Postfix) with ESMTPA id 79E991534C0; Sat, 8 Aug 2015 13:23:03 +0200 (CEST) Subject: Re: Using SSDs as swap To: Konstantin Belousov References: <55C5D48E.6010605@digiware.nl> <20150808102900.GA2072@kib.kiev.ua> <20150808103810.GB2072@kib.kiev.ua> Cc: fs@freebsd.org From: Willem Jan Withagen Message-ID: <55C5E697.4080102@digiware.nl> Date: Sat, 8 Aug 2015 13:23:03 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20150808103810.GB2072@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 11:23:34 -0000 On 8-8-2015 12:38, Konstantin Belousov wrote: > On Sat, Aug 08, 2015 at 01:29:00PM +0300, Konstantin Belousov wrote: >> On Sat, Aug 08, 2015 at 12:06:06PM +0200, Willem Jan Withagen wrote: >>> one of the following commits just passed with this in the log, and it >>> triggered again a question I've been having for some time again already. >>> >>> ---- >>> Log: >>> Enable BIO_DELETE passthru in GELI, so TRIM/UNMAP can work as expected >>> when >>> GELI is used on a SSD or inside virtual machine, so that guest can tell >>> host that it is no longer using some of the storage. >>> ----- >>> >>> In ZFS I slice my SSD's into log and caches, but on a a server with >>> little memory (which can't be grown) I use a partion on each ssd as swap >>> as well. So swappinging does not have to seek, and has faster loading >>> time. To allocate a few GB on aan SSD to swap is not really all that >>> painfull, given current sizes, but the speed difference with regular >>> spindels is impressive. >>> >>> But the questions are: >>> 1) Does the swap driver understand that backing-store needs a TRIM? >> No. >> >>> 1a) if not would it be useful, and what would it take to implement? >> One good thing is that it is simply the question of coding: the VM >> already has a place where it informs the swap pager that the page copy >> in swap is no longer needed. this is the vm_pager_page_unswapped() call >> and swap pager method swap_pager_unswapped(). swp_pager_meta_ctl() would >> need to issue BIO_DELETE to the backing storage. >> >> On the other hand, note that this would increase the amount of work >> performed, even for the swap volumes located on the rotating media, >> which is more typical and reasonable setup. >> >> I think an implementation and a knob to turn it off, or configure per >> swap partition, would be reasonable. > > One additional thing: while BIO_DELETE is in progress, the swap block > cannot be marked free, since otherwise we could write other page and > get it obliterated with the TRIM. This can be done async, but the > consequence is that swap space would be released and usable some time > after the page-in. This will affect loads which are close to OOM. Sort of makes sense to me... I take it that BIO_DELETE fires and returns before TRIM is completed? But then the SSD accepts writes to a TRIMmed block, but then mixes this up? Possibly deleting a write to a to be trimmed block? This sort of strikes me as odd, but then I do not know the full intricate details of TRIM on SSD Would it be possible to be notified that a TRIM has completed, only then to actually free the swap sectors? And then perhaps the swap bookkeeping does not yet accommodate for a possible extra state? Speaking about blocks.... Does Swap take into account that disks could be of a sectorsize other than 512 bytes. I would guess so, since we could have a 4K disk as swap disk, and doing read-modify-write for swap is sure going to kill performance. --WjW From owner-freebsd-fs@freebsd.org Sat Aug 8 11:37:58 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C0B49B61DB for ; Sat, 8 Aug 2015 11:37:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 30881135E for ; Sat, 8 Aug 2015 11:37:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 2DB009B61DA; Sat, 8 Aug 2015 11:37:58 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 147B49B61D9 for ; Sat, 8 Aug 2015 11:37:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AE697135B for ; Sat, 8 Aug 2015 11:37:57 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id t78Bbon1022106 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 8 Aug 2015 14:37:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua t78Bbon1022106 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id t78Bbo95022105; Sat, 8 Aug 2015 14:37:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 8 Aug 2015 14:37:50 +0300 From: Konstantin Belousov To: Willem Jan Withagen Cc: fs@freebsd.org Subject: Re: Using SSDs as swap Message-ID: <20150808113750.GC2072@kib.kiev.ua> References: <55C5D48E.6010605@digiware.nl> <20150808102900.GA2072@kib.kiev.ua> <55C5E34B.9010905@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55C5E34B.9010905@digiware.nl> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 11:37:58 -0000 On Sat, Aug 08, 2015 at 01:08:59PM +0200, Willem Jan Withagen wrote: > On 8-8-2015 12:29, Konstantin Belousov wrote: > > On Sat, Aug 08, 2015 at 12:06:06PM +0200, Willem Jan Withagen wrote: > > >> 2) Does the SSD "suffer" unneeded from swapping on it? > > Depends on your swapping activity, but yes, I believe that intense swapping > > would wear SSD in the short time. > > Would it be more intense than being beaten to death as a ARC cache?? No idea. Ask somebody who understands how ARC works. Swap is used for pageouts, and typical pageout involves the page and some amount of pages surrounding it in the vm object' queue. Defaults are 32 pages as the maximum clustered pageout size, but I believe that it is typically cannot be achieved, I would expect that often the very minimalistic i/o requests are blasted. It might be better if the dirty mappings are promoted to superpages, but I think that the promotion is not likely to happen if your machine already started swapping. Anyway, I do not posses a statistic for the pageout patters on the real workloads. With the current state of hardware, it is much more reasonable to buy enough RAM, which would give you two orders of magnitude of the speedup, than to target swap subsystem optimization to provide you e.g. 10-20% improvements comparing with the current state of the code. > > If I look at most of my systems I try to prevent them even using really > swap since that usually kills performance big time... > So things get swapped out under pressure, but rarely get swapped back > in, because the LRU properties of that part of the application minimise > that chance of things getting paged back in. > > So the number of times that swap actually gets used is rather seldom. > > In writting swap, how are the allocations made where things are swapped. > Do things always end up in a lineair order on swap writting things as > close to the start as slots in swap allow it. SSD would profit from it > being sort of a circular buffer... > (Guess I have to try and understand the swap code.... ) The swap block allocator is in kern/subr_blist.c. It is a radix tree search algorithm which looks for the contig range of free swap blocks of the given size. There are no any provisions for providing streaming write behaviour (and it would be unnatural for the pageouts to have this behaviour), I do not think that it would exhibit such behaviour accidentally. From owner-freebsd-fs@freebsd.org Sat Aug 8 11:41:14 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 288099B6310 for ; Sat, 8 Aug 2015 11:41:14 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0C4FA14EB for ; Sat, 8 Aug 2015 11:41:14 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 0B8969B630F; Sat, 8 Aug 2015 11:41:14 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A2A69B630E for ; Sat, 8 Aug 2015 11:41:14 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8738A14E9 for ; Sat, 8 Aug 2015 11:41:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id t78Bf7Lw023254 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 8 Aug 2015 14:41:08 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua t78Bf7Lw023254 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id t78Bf7jB023253; Sat, 8 Aug 2015 14:41:07 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 8 Aug 2015 14:41:07 +0300 From: Konstantin Belousov To: Willem Jan Withagen Cc: fs@freebsd.org Subject: Re: Using SSDs as swap Message-ID: <20150808114107.GD2072@kib.kiev.ua> References: <55C5D48E.6010605@digiware.nl> <20150808102900.GA2072@kib.kiev.ua> <20150808103810.GB2072@kib.kiev.ua> <55C5E697.4080102@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55C5E697.4080102@digiware.nl> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 11:41:14 -0000 On Sat, Aug 08, 2015 at 01:23:03PM +0200, Willem Jan Withagen wrote: > On 8-8-2015 12:38, Konstantin Belousov wrote: > > On Sat, Aug 08, 2015 at 01:29:00PM +0300, Konstantin Belousov wrote: > >> On Sat, Aug 08, 2015 at 12:06:06PM +0200, Willem Jan Withagen wrote: > >>> one of the following commits just passed with this in the log, and it > >>> triggered again a question I've been having for some time again already. > >>> > >>> ---- > >>> Log: > >>> Enable BIO_DELETE passthru in GELI, so TRIM/UNMAP can work as expected > >>> when > >>> GELI is used on a SSD or inside virtual machine, so that guest can tell > >>> host that it is no longer using some of the storage. > >>> ----- > >>> > >>> In ZFS I slice my SSD's into log and caches, but on a a server with > >>> little memory (which can't be grown) I use a partion on each ssd as swap > >>> as well. So swappinging does not have to seek, and has faster loading > >>> time. To allocate a few GB on aan SSD to swap is not really all that > >>> painfull, given current sizes, but the speed difference with regular > >>> spindels is impressive. > >>> > >>> But the questions are: > >>> 1) Does the swap driver understand that backing-store needs a TRIM? > >> No. > >> > >>> 1a) if not would it be useful, and what would it take to implement? > >> One good thing is that it is simply the question of coding: the VM > >> already has a place where it informs the swap pager that the page copy > >> in swap is no longer needed. this is the vm_pager_page_unswapped() call > >> and swap pager method swap_pager_unswapped(). swp_pager_meta_ctl() would > >> need to issue BIO_DELETE to the backing storage. > >> > >> On the other hand, note that this would increase the amount of work > >> performed, even for the swap volumes located on the rotating media, > >> which is more typical and reasonable setup. > >> > >> I think an implementation and a knob to turn it off, or configure per > >> swap partition, would be reasonable. > > > > One additional thing: while BIO_DELETE is in progress, the swap block > > cannot be marked free, since otherwise we could write other page and > > get it obliterated with the TRIM. This can be done async, but the > > consequence is that swap space would be released and usable some time > > after the page-in. This will affect loads which are close to OOM. > > Sort of makes sense to me... > > I take it that BIO_DELETE fires and returns before TRIM is completed? > But then the SSD accepts writes to a TRIMmed block, but then mixes this > up? Possibly deleting a write to a to be trimmed block? This sort of > strikes me as odd, but then I do not know the full intricate details of > TRIM on SSD > > Would it be possible to be notified that a TRIM has completed, only then > to actually free the swap sectors? This is exactly what I wrote above. > And then perhaps the swap bookkeeping does not yet accommodate for a > possible extra state? It does not need to. The in-flight BIO_DELETE remembers the intermediate state, the swap block should be freed only after the storage reported the BIO_DELETE as finished. It is exactly the same as UFS handles trimming of the free blocks, the bitmap of the used/freed blocks is only updated after the BIO_DELETE is finished, not when the inode drops reference to the block. > > Speaking about blocks.... Does Swap take into account that disks could > be of a sectorsize other than 512 bytes. I would guess so, since we > could have a 4K disk as swap disk, and doing read-modify-write for swap > is sure going to kill performance. swap performs i/o in the page-sized chunks at least, which are min 4k on all supported platforms (even on arms, where we do not support smaller pages AFAIK). From owner-freebsd-fs@freebsd.org Sat Aug 8 13:29:51 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A3079B5B85 for ; Sat, 8 Aug 2015 13:29:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4F2D17FD for ; Sat, 8 Aug 2015 13:29:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 4C4799B5B84; Sat, 8 Aug 2015 13:29:51 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 310B99B5B83 for ; Sat, 8 Aug 2015 13:29:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E3D3A7FB for ; Sat, 8 Aug 2015 13:29:50 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id D316315340A; Sat, 8 Aug 2015 15:29:46 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EqIlyNTI4-bw; Sat, 8 Aug 2015 15:29:37 +0200 (CEST) Received: from [192.168.10.10] (asus [192.168.10.10]) by smtp.digiware.nl (Postfix) with ESMTPA id CB444153401; Sat, 8 Aug 2015 15:29:37 +0200 (CEST) Subject: Re: Using SSDs as swap To: Konstantin Belousov References: <55C5D48E.6010605@digiware.nl> <20150808102900.GA2072@kib.kiev.ua> <55C5E34B.9010905@digiware.nl> <20150808113750.GC2072@kib.kiev.ua> Cc: fs@freebsd.org From: Willem Jan Withagen Message-ID: <55C60441.7040906@digiware.nl> Date: Sat, 8 Aug 2015 15:29:37 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20150808113750.GC2072@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 13:29:51 -0000 On 8-8-2015 13:37, Konstantin Belousov wrote: > On Sat, Aug 08, 2015 at 01:08:59PM +0200, Willem Jan Withagen wrote: >> Would it be more intense than being beaten to death as a ARC cache?? > No idea. Ask somebody who understands how ARC works. > > Swap is used for pageouts, and typical pageout involves the page and > some amount of pages surrounding it in the vm object' queue. Defaults > are 32 pages as the maximum clustered pageout size, but I believe that > it is typically cannot be achieved, I would expect that often the > very minimalistic i/o requests are blasted. It might be better if the > dirty mappings are promoted to superpages, but I think that the promotion > is not likely to happen if your machine already started swapping. > > Anyway, I do not posses a statistic for the pageout patters on the > real workloads. With the current state of hardware, it is much more > reasonable to buy enough RAM, which would give you two orders of > magnitude of the speedup, than to target swap subsystem optimization to > provide you e.g. 10-20% improvements comparing with the current state of > the code. I agree fully, but if you are running with hardware that does not allow this, that is not really an option. :( There are quite a lot of motherboard/processors combinations about that little RAM to either 4G of 8G. Upgrading that so something that allows you to do 32G, sets you back about 500-700 euros. You also get more power out of the CPU, but like in (small) fileservers, CPU power is rarely the problem. And what I see on some of the servers is that there very little moments in time that there is some swapping going on.... Like during nightly chores: building world/kernel, building locate.db, etc.... It starts to push a bit of stuff into swap. Its not even that bad, about 300-800Mb, 2 Gb if I really push it very hard. BTW, as far as pricing I can get 8Gb ECC RAM for about 55 euro. Looking at a Samsung 256Gb SSD (evo) prices as about 100 euro in the last purchase. So it is: 1Gb Ram for 7 euro versus 1Gb SSD for 0,40 Euro. So the price difference is a factor of ~20. So on systems that are not continously swapping, I would consider this a fair tradeoff. So perhaps the nicest thing to do for the SSDs is TRIM swap at startup??? So the the SSD controller van do its garbage collection and then keep the remainder of the stuff as it is. I did have cache and swap traces when I was working at the University, but that was late 80's. So caches were like 2K (no l1,l2,l3 stuff) and RAM ended somewhere at 64Mbyte. Life has changed a lot since then. :) I still have the DAT tape with the traces as a memorabilia ... The HP drive has long since died, and the tape probably won't read anyways. --WjW From owner-freebsd-fs@freebsd.org Sat Aug 8 18:26:43 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4B889B5D77 for ; Sat, 8 Aug 2015 18:26:43 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BAA2DBB7 for ; Sat, 8 Aug 2015 18:26:43 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: by mailman.ysv.freebsd.org (Postfix) id B7B589B5D76; Sat, 8 Aug 2015 18:26:43 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B74C79B5D75 for ; Sat, 8 Aug 2015 18:26:43 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "wonkity.com", Issuer "wonkity.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7952BBB6 for ; Sat, 8 Aug 2015 18:26:43 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.15.2/8.15.2) with ESMTPS id t78IQdGH042727 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 8 Aug 2015 12:26:39 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.15.2/8.15.2/Submit) with ESMTP id t78IQcJ5042720; Sat, 8 Aug 2015 12:26:39 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Sat, 8 Aug 2015 12:26:38 -0600 (MDT) From: Warren Block To: Willem Jan Withagen cc: Konstantin Belousov , fs@freebsd.org Subject: Re: Using SSDs as swap In-Reply-To: <55C60441.7040906@digiware.nl> Message-ID: References: <55C5D48E.6010605@digiware.nl> <20150808102900.GA2072@kib.kiev.ua> <55C5E34B.9010905@digiware.nl> <20150808113750.GC2072@kib.kiev.ua> <55C60441.7040906@digiware.nl> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Sat, 08 Aug 2015 12:26:39 -0600 (MDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 18:26:43 -0000 On Sat, 8 Aug 2015, Willem Jan Withagen wrote: > > So perhaps the nicest thing to do for the SSDs is TRIM swap at > startup??? So the the SSD controller van do its garbage collection and > then keep the remainder of the stuff as it is. This can be done now by using a swap file on a UFS partition with trim enabled. The catch is that the swap file has to be deleted and recreated to trigger the trim. The delete is quick, but the create depends on the size of the file and the speed of the hardware. (And no, sparse files do not work as swap files.) Maybe rotate swap files like log files, so they could be created when the system is idle. From owner-freebsd-fs@freebsd.org Sat Aug 8 18:32:43 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 324D29B5EF3 for ; Sat, 8 Aug 2015 18:32:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 15287F90 for ; Sat, 8 Aug 2015 18:32:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 141AC9B5EF2; Sat, 8 Aug 2015 18:32:43 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13B059B5EF1 for ; Sat, 8 Aug 2015 18:32:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 804ECF8F for ; Sat, 8 Aug 2015 18:32:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id t78IWYaX045990 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 8 Aug 2015 21:32:34 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua t78IWYaX045990 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id t78IWYBI045989; Sat, 8 Aug 2015 21:32:34 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 8 Aug 2015 21:32:34 +0300 From: Konstantin Belousov To: Warren Block Cc: Willem Jan Withagen , fs@freebsd.org Subject: Re: Using SSDs as swap Message-ID: <20150808183234.GG2072@kib.kiev.ua> References: <55C5D48E.6010605@digiware.nl> <20150808102900.GA2072@kib.kiev.ua> <55C5E34B.9010905@digiware.nl> <20150808113750.GC2072@kib.kiev.ua> <55C60441.7040906@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 18:32:43 -0000 On Sat, Aug 08, 2015 at 12:26:38PM -0600, Warren Block wrote: > On Sat, 8 Aug 2015, Willem Jan Withagen wrote: > > > > So perhaps the nicest thing to do for the SSDs is TRIM swap at > > startup??? So the the SSD controller van do its garbage collection and > > then keep the remainder of the stuff as it is. > > This can be done now by using a swap file on a UFS partition with trim > enabled. The catch is that the swap file has to be deleted and > recreated to trigger the trim. The delete is quick, but the create > depends on the size of the file and the speed of the hardware. (And no, > sparse files do not work as swap files.) This could work, in the sense that swap would indeed work as a swap, and not as a deadlock generator. But it adds very significant (up to 100% in the CPU time, I think) overhead. Note that you cannot swap to file directly, you must create md(4) over the file and swap to it. But doing such layer over layer to get the TRIM is somewhat silly. > > Maybe rotate swap files like log files, so they could be created when > the system is idle. From owner-freebsd-fs@freebsd.org Sat Aug 8 18:51:44 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BCB269B6284 for ; Sat, 8 Aug 2015 18:51:44 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A17331950 for ; Sat, 8 Aug 2015 18:51:44 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.ysv.freebsd.org (Postfix) id A0AED9B6283; Sat, 8 Aug 2015 18:51:44 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 868759B6282 for ; Sat, 8 Aug 2015 18:51:44 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B90F194F for ; Sat, 8 Aug 2015 18:51:44 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 043CC153413; Sat, 8 Aug 2015 20:51:40 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NCEwvqnFOBKn; Sat, 8 Aug 2015 20:51:29 +0200 (CEST) Received: from [192.168.10.10] (asus [192.168.10.10]) by smtp.digiware.nl (Postfix) with ESMTPA id 4803215340A; Sat, 8 Aug 2015 20:51:29 +0200 (CEST) Subject: Re: Using SSDs as swap To: Konstantin Belousov , Warren Block References: <55C5D48E.6010605@digiware.nl> <20150808102900.GA2072@kib.kiev.ua> <55C5E34B.9010905@digiware.nl> <20150808113750.GC2072@kib.kiev.ua> <55C60441.7040906@digiware.nl> <20150808183234.GG2072@kib.kiev.ua> Cc: fs@freebsd.org From: Willem Jan Withagen Message-ID: <55C64FB1.9070402@digiware.nl> Date: Sat, 8 Aug 2015 20:51:29 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20150808183234.GG2072@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 18:51:44 -0000 On 8-8-2015 20:32, Konstantin Belousov wrote: > On Sat, Aug 08, 2015 at 12:26:38PM -0600, Warren Block wrote: >> On Sat, 8 Aug 2015, Willem Jan Withagen wrote: >>> >>> So perhaps the nicest thing to do for the SSDs is TRIM swap at >>> startup??? So the the SSD controller van do its garbage collection and >>> then keep the remainder of the stuff as it is. >> >> This can be done now by using a swap file on a UFS partition with trim >> enabled. The catch is that the swap file has to be deleted and >> recreated to trigger the trim. The delete is quick, but the create >> depends on the size of the file and the speed of the hardware. (And no, >> sparse files do not work as swap files.) > This could work, in the sense that swap would indeed work as a swap, > and not as a deadlock generator. But it adds very significant (up to > 100% in the CPU time, I think) overhead. > > Note that you cannot swap to file directly, you must create md(4) over > the file and swap to it. > > But doing such layer over layer to get the TRIM is somewhat silly. Right, added layering is not really something you'd be looking for here. We talking about incidental swapping to SSD, so then the system is already under pressure. >> Maybe rotate swap files like log files, so they could be created when >> the system is idle. 'mmm, once swap is in use, I do not think you can unswap? So even if the sytem is idle, was was not swapped back in because it is needed stays out on swap. Look at my curent swap on a ZFS file server, I added 2 SSDs partitions about 1,2 ago. But even the old swap was not released.... { swap on SSD's are on ad{2,3}p2 } Device 1K-blocks Used Avail Capacity /dev/ada2p2 8388608 122M 7.9G 1% /dev/ada3p2 8388608 123M 7.9G 2% /dev/gpt/swap0 8388608 4.6M 8.0G 0% /dev/gpt/swap1 8388608 5.0M 8.0G 0% Total 33554432 255M 32G 1% (Yes, I realise that 32Gb swap for a 8Gb system is somewhat silly) Different idea if we do not want to TRIM from the kernel when issueing swapon: Would it be possible to use camcontrol to issue a "trim" on a (GPT) swap partition? Or does gpart anything that could be conceived as a TRIM. Because then the initscript would first check for saved cores, and once done that trim the swappartition with a commandline tool? --WjW From owner-freebsd-fs@freebsd.org Sat Aug 8 21:28:41 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8138F9B6D40 for ; Sat, 8 Aug 2015 21:28:41 +0000 (UTC) (envelope-from rpokala@panasas.com) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0095.outbound.protection.outlook.com [65.55.169.95]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E72E710A0; Sat, 8 Aug 2015 21:28:40 +0000 (UTC) (envelope-from rpokala@panasas.com) Received: from CY1PR08MB1803.namprd08.prod.outlook.com (10.162.218.25) by CY1PR08MB1802.namprd08.prod.outlook.com (10.162.218.24) with Microsoft SMTP Server (TLS) id 15.1.225.19; Sat, 8 Aug 2015 21:28:31 +0000 Received: from CY1PR08MB1803.namprd08.prod.outlook.com ([10.162.218.25]) by CY1PR08MB1803.namprd08.prod.outlook.com ([10.162.218.25]) with mapi id 15.01.0225.018; Sat, 8 Aug 2015 21:28:30 +0000 From: "Pokala, Ravi" To: "freebsd-fs@freebsd.org" , "kostikbel@gmail.com" , "wjw@digiware.nl" , "cem@freebsd.org" Subject: Re: freebsd-fs Digest, Vol 630, Issue 5 Thread-Topic: freebsd-fs Digest, Vol 630, Issue 5 Thread-Index: AQHQ0dHRmJ09FWlQake+sJdwGAG7Up4CKX2A Date: Sat, 8 Aug 2015 21:28:29 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.5.3.150624 authentication-results: spf=none (sender IP is ) smtp.mailfrom=rpokala@panasas.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [24.6.178.251] x-microsoft-exchange-diagnostics: 1; CY1PR08MB1802; 5:xVCHkO8SmUXJKMaz6ejfKdD9tOKH+wT+xsFYxgBLmKx+tgLV6HdQmbLfXsH+Y3+0EyI2kdBaw7LWZbqeG6sPNKHXeqVsjcI32/ZDjz1H9mWDY/3Af1w0yrwfq6BqbaCArtUqhAqDz/Enywyxy6id6A==; 24:ARzMeyUWJ1cYRq+0xbx3fKL+ifiGJgi7+NbSNSej83++rrS+a27eO8S09W+xqlsSDLMq3JHW7oRlkFyIDVIRA1hTZ3VLrAYxjaka4I6P7r0=; 20:uLRdW0OIFJxdpBkGyML8/78RAFz33Js5E4jnDVoDuJMfFIJHsJ0OHz/ljM72bjDDtCS1/AH0UmX26tFPjY+fVA== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY1PR08MB1802; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(5005006)(3002001); SRVR:CY1PR08MB1802; BCL:0; PCL:0; RULEID:; SRVR:CY1PR08MB1802; x-forefront-prvs: 06628F7CA4 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(18543002)(189002)(5403001)(199003)(10533003)(2501003)(106356001)(99286002)(19580395003)(106116001)(92566002)(76176999)(46102003)(105586002)(2201001)(101416001)(2900100001)(50986999)(10400500002)(54356999)(4001540100001)(77096005)(2950100001)(68736005)(2656002)(64706001)(87936001)(19580405001)(66066001)(97736004)(102836002)(36756003)(4001350100001)(189998001)(83506001)(5001830100001)(77156002)(86362001)(5001960100002)(5002640100001)(122556002)(40100003)(5001860100001)(107886002)(62966003)(5001770100001)(81156007); DIR:OUT; SFP:1101; SCL:1; SRVR:CY1PR08MB1802; H:CY1PR08MB1803.namprd08.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: panasas.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: <96C3C6D50A992044ACF41F89917CE4FD@namprd08.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: panasas.com X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Aug 2015 21:28:29.7814 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR08MB1802 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2015 21:28:41 -0000 >Date: Sat, 8 Aug 2015 14:41:07 +0300 >From: Konstantin Belousov >To: Willem Jan Withagen >Cc: fs@freebsd.org >Subject: Re: Using SSDs as swap >Message-ID: <20150808114107.GD2072@kib.kiev.ua> >Content-Type: text/plain; charset=3Dus-ascii > >>=20 >> Speaking about blocks.... Does Swap take into account that disks could >>be of a sectorsize other than 512 bytes. I would guess so, since we >>could have a 4K disk as swap disk, and doing read-modify-write for swap >>is sure going to kill performance. >swap performs i/o in the page-sized chunks at least, which are min 4k on >all supported platforms (even on arms, where we do not support smaller >pages AFAIK). I can confirm this first-hand - I have systems which use AF-4Kn drives for swap, without any problems. HOWEVER, there is a different - but related - snag. Frequently, the same device is used for both swap and vmcores. While swapping (aka paging) is done in 4KB chunks, some structures used in dumping core are always 512B. This does not work well on AF-4Kn drives. [PR 194279] discusses the problem. cem@ did some work on the dump path "recently" (yikes - according to the bug, it was close to a year ago!) -Ravi