From owner-freebsd-fs@freebsd.org Mon Nov 27 15:29:22 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5B1ABE54EA9; Mon, 27 Nov 2017 15:29:22 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2766977653; Mon, 27 Nov 2017 15:29:21 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 9910B209F9; Mon, 27 Nov 2017 10:29:14 -0500 (EST) Received: from frontend2 ([10.202.2.161]) by compute6.internal (MEProxy); Mon, 27 Nov 2017 10:29:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsco.org; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=MHh2Qhy/trxoyPVOFzbQh5tKlSTVu Pvo190/Bdb/M5w=; b=DpDiqHRqwco+jcWBg0YjR06Th+9defM8k+vSKzMdzLT4i MVUEsVLNvKgxeqhql3PemZvGgH5wuHpOGiU/XHp/ZecDHTTY6COOZ5i361tI3Ccg dK1XGPfx1Ijawa1oiWS034WNGrhcfZEGobc1b1qmwrVQ7DVRVjaCjgPDjWlB7vy3 uUSTIypyJ9O5OT3kg6uFwoq/PVuv1RQ7WprvajNFQs9pmbvj7Og/9bxi+K7D5jgl RaPGzIoRbmIFsHMg6flns2T8Qo/NR79XbNozX1rmkPShigFSCt7ukS1pIP8w+3Cr f8vTJ75PzX/lHi7y6tuBIVpExOCNzgHWpv7oMy3WQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=MHh2Qh y/trxoyPVOFzbQh5tKlSTVuPvo190/Bdb/M5w=; b=A2uJyB7oFCSO0UNCw6TB/V I2kBiWs9brQFLhQfD6q+RLghECWtdg323jvRLzAmmlNm7XpvqWsZ467/4FgNCN7G N7cd5ci1pjLryyl8P8IiLM3aKM73VKY0Iv7OIG3/vxBPZR4Pp7ZJfvfikI6L3qj2 sppsejgEmaEpkwmeingIvxgQAj1/lhWYJrZ+vSEX9PnETEvrIBkYrK5Sr1jVvrAb fRRG724bc30i4Y0iIgRZNH+0i6/BXbWGxJ2+ZuW0HrxDUgQ/76NuagebA0WnD5b5 DqyIIjQy3P12rADec16JZsLV7Vzhta3O4oF4H/0EAD+uPn+icWd9pDI0+6IocBgA == X-ME-Sender: Received: from [192.168.0.119] (unknown [161.97.249.191]) by mail.messagingengine.com (Postfix) with ESMTPA id ED66F24009; Mon, 27 Nov 2017 10:29:13 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom From: Scott Long In-Reply-To: <33101e6c-0c74-34b7-ee92-f9c4a11685d5@FreeBSD.org> Date: Mon, 27 Nov 2017 08:29:11 -0700 Cc: Warner Losh , FreeBSD FS , freebsd-geom@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <783CA790-C0D3-442D-A5A2-4CB424FCBD62@samsco.org> References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> <33101e6c-0c74-34b7-ee92-f9c4a11685d5@FreeBSD.org> To: Andriy Gapon X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 15:29:22 -0000 > On Nov 25, 2017, at 10:36 AM, Andriy Gapon wrote: >=20 > On 25/11/2017 12:54, Scott Long wrote: >>=20 >>=20 >>> On Nov 24, 2017, at 10:17 AM, Andriy Gapon wrote: >>>=20 >>>=20 >>>>> IMO, this is not optimal. I'd rather pass BIO_NORETRY to the = first read, get >>>>> the error back sooner and try the other disk sooner. Only if I = know that there >>>>> are no other copies to try, then I would use the normal read with = all the retrying. >>>>>=20 >>>>=20 >>>> I agree with Warner that what you are proposing is not correct. It = weakens the >>>> contract between the disk layer and the upper layers, making it = less clear who is >>>> responsible for retries and less clear what =E2=80=9CEIO=E2=80=9D = means. That contract is already >>>> weak due to poor design decisions in VFS-BIO and GEOM, and Warner = and I >>>> are working on a plan to fix that. >>>=20 >>> Well... I do realize now that there is some problem in this area, = both you and >>> Warner mentioned it. But knowing that it exists is not the same as = knowing what >>> it is :-) >>> I understand that it could be rather complex and not easy to = describe in a short >>> email=E2=80=A6 >>>=20 >>=20 >> There are too many questions to ask, I will do my best to keep the = conversation >> logical. First, how do you propose to distinguish between EIO due to = a lengthy >> set of timeouts, vs EIO due to an immediate error returned by the = disk hardware? >=20 > At what layer / component? > If I am the issuer of the request then I know how I issued that = request and what > kind of request it was. If I am an intermediate layer, then what do I = care. >=20 I don=E2=80=99t understand your response, I think we are talking about 2 = different things. >> CAM has an extensive table-driven error recovery protocol who=E2=80=99s= purpose is to >> decide whether or not to do retries based on hardware state = information that is >> not made available to the upper layers. Do you have a test case that = demonstrates >> the problem that you=E2=80=99re trying to solve? Maybe the error = recovery table is wrong >> and you=E2=80=99re encountering a case that should not be retried. = If that=E2=80=99s what=E2=80=99s going on, >> we should fix CAM instead of inventing a new work-around. >=20 > Let's assume that I am talking about the case of not being able to = read an HDD > sector that is gone bad. > Here is a real world example: >=20 > Jun 16 10:40:18 trant kernel: ahcich0: NCQ error, slot =3D 20, port =3D = -1 > Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. = ACB: 60 > 00 00 58 62 40 2c 00 00 08 00 00 > Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): CAM status: ATA = Status Error > Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): ATA status: 41 = (DRDY ERR), > error: 40 (UNC ) > Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): RES: 41 40 68 58 = 62 40 2c 00 > 00 00 00 > Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): Retrying command > Jun 16 10:40:20 trant kernel: ahcich0: NCQ error, slot =3D 22, port =3D = -1 ... > I do not see anything wrong in what CAM / ahci /ata_da did here. > They did what I would expect them to do. They tried very hard to get = data that > I told them I need. Two things I see here. The first is that the drive is trying for 2 = seconds to get good data off of the media. The second is that it=E2=80=99s failing and = reporting the error as uncorrectable. I think that retries at the OS/driver layer are therefore useless; the drive is already doing a bunch of its = own retries and is failing, and is telling you that it=E2=80=99s failing. In the past, = the conventional wisdom has been to do retries, because 30 years ago drives had minimal firmware and = didn=E2=80=99t do a good job at managing data integrity. Now they do an extensive amount = of analysis-driven error correction and retries, so I think it=E2=80=99s = time to change the=20 conventional wisdom. I=E2=80=99d propose that for direct-attach SSDs = and HDDs we treat this error as non-retriable. Normally this would be a one-line change, but I = think it needs to be nuanced to distinguish between optical drives, simple flash media = drives, and regular HDDs and SSDs. An interim solution would be to just set the kern.cam.ada.retry_count to = 0. >=20 > Timestamp of the first error is Jun 16 10:40:18. > Timestamp of the last error is Jun 16 10:40:27. > So, it took additional 9 seconds to finally produce EIO. > That disk is a part of a ZFS mirror. If the request was failed after = the first > attempt, then ZFS would be able to get the data from a good disk much = sooner. It=E2=80=99s the fault of ZFS for not reading all members in parallel. = I ask people to consider and possibly accept that ZFS is not perfect. >=20 > And don't take me wrong, I do NOT want CAM or GEOM to make that = decision by > itself. I want ZFS to be able to tell the lower layers when they = should try as > hard as they normally do and when they should report an I/O error as = soon as it > happens without any retries. >=20 Again, I=E2=80=99m not sure I understand your answer. It is the job of = the disk drivers to reliably complete storage requests, not to just try once and then assume = that an upper layer will figure things out. As I said in my previous email, the = upper layers, and in this case I mean everything above the drivers and CAM, do not = have the error information to know why something failed, and thus have only = limited knowledge of what to do to remediate a failure. The error recovery = knowledge is in CAM and the drivers. It sounds like you=E2=80=99re proposing to make these simple, dumb = layers and move the error recovery up to ZFS. I understand what you=E2=80=99re saying about = wanting this being opt-in and not affecting other filesystems, but I think that you are missing the point of what = CAM does and provides. By definition the =E2=80=98da=E2=80=99 and =E2=80=98ada=E2=80=99= drivers provide reliable I/O transactions to the drive. That means that they handle error recovery, retries, etc. = If you do not want reliable I/O transactions, then you should not use the da/ada = drivers, and instead should either write your own periph driver (let=E2=80=99s call = it =E2=80=9Czfsda=E2=80=9D), or you should have ZFS communicate with the disks via the pass driver. That is = what userland tools like cdparanoia do when they want to bypass the = traditional operational characteristics and constraints of the standard periphs. Think of it this way, would you propose a flag to a TCP socket that = tells TCP to not do retransmits? No! If you wanted that semantic then you=E2=80=99d = use UDP and invent your own retry/reliability layer. I think it=E2=80=99s a big = job to re-invent error recovery into ZFS, while it would be relatively easy to adapt the CAM retry policy to the behavior of modern disks, i.e. believe = them when they say that there=E2=80=99s an uncorrectable read error and = don=E2=80=99t blindly attempt a retry. >> Second, what about disk subsystems that do retries internally, out of = the control >> of the FreeBSD driver? This would include most hardware RAID = controllers. >> Should what you are proposing only work for a subset of the kinds of = storage >> systems that are available and in common use? >=20 > Yes. I do not worry about things that are beyond my control. > Those subsystems would behave as they do now. So, nothing would get = worse. It gets worse because it builds inconsistency into the system and makes = expectations for behavior unclear. That turns into more hacky and incorrect code = over time. >=20 >> Third, let=E2=80=99s say that you run out of alternate copies to try, = and as you stated >> originally, that will force you to retry the copies that had returned = EIO. How >> will you know when you can retry? How will you know how many times = you >> will retry? How will you know that a retry is even possible? >=20 > I am continuing to use ZFS as an example. It already has all the = logic built > in. If all vdev zio-s (requests to mirror members as an example) = fail, then > their parent zio (a logical read from the mirror) will be retried (by = ZFS) and > when ZFS retries it sets a special flag (ZIO_FLAG_IO_RETRY) on that = zio and on > its future child zio-s. >=20 > Essentially, my answer is you have to program it correctly, there is = no magic. Again, I think that you=E2=80=99re missing the point of what I=E2=80=99m = asking. All that ZFS can see is =E2=80=9CEIO=E2=80=9D. Maybe the EIO was generated because = of an uncorrectable media error, and all attempts to retry will be futile. Maybe it was = generated because of a correctable error, and a retry might be useful. There are = many, many, many error cases that are tested for an handled in CAM. You are = collapsing them down to 1 case, and I don=E2=80=99t agree with that. >=20 >> Should the retries >> be able to be canceled? >=20 > I think that this is an orthogonal question. > I do not have any answer and I am not ready to discuss this at all. >=20 Please be ready to discuss this. >> Why is overloading EIO so bad? brelse() will call bdirty() when a = BIO_WRITE >> command has failed with EIO. Calling bdirty() has the effect of = retrying the I/O. >> This disregards the fact that disk drivers only return EIO when = they=E2=80=99ve decided >> that the I/O cannot be retried. It has no termination condition for = the retries, and >> will endlessly retry I/O in vain; I=E2=80=99ve seen this quite = frequently. It also disregards >> the fact that I/O marked as B_PAGING can=E2=80=99t be retried in this = fashion, and will >> trigger a panic. Because we pretend that EIO can be retried, we are = left with >> a system that is very fragile when I/O actually does fail. Instead = of adding >> more special cases and blurred lines, I want to go back to enforcing = strict >> contracts between the layers and force the core parts of the system = to respect >> those contracts and handle errors properly, instead of just retrying = and >> hoping for the best. >=20 > So, I suggest that the buffer layer (all the b* functions) does not = use the > proposed flag. Any problems that exist in it should be resolved = first. > ZFS does not use that layer. >=20 I think it=E2=80=99s foolish to propose a change in the disk API and = then say =E2=80=9Conly ZFS should use this=E2=80=9D >>> But then, this flag is optional, it's off by default and no one is = forced to >>> used it. If it's used only by ZFS, then it would not be horrible. >>> Unless it makes things very hard for the infrastructure. >>> But I am circling back to not knowing what problem(s) you and Warner = are >>> planning to fix. >>>=20 >>=20 >> Saying that a feature is optional means nothing; while consumers of = the API >> might be able to ignore it, the producers of the API cannot ignore = it. It is >> these producers who are sick right now and should be fixed, instead = of >> creating new ways to get even more sick. >=20 > I completely agree. > But which producers of the API do you mean specifically? > So far, you mentioned only the consumer level problems with the = b-layer. >=20 > Having said all of the above, I must admit one thing. > When I proposed BIO_NORETRY I had only the simplest GEOM topology in = mind: > ZFS -> [partition] -> disk. > Now that I start to think about more complex topologies I am not = really sure how > the flag should be handled by geom-s with complex internal behavior. = If that > can be implemented reasonably and clearly, if the flag will create a = big mess. > E.g., things like gmirrors on top of gmirrors and so on. > Maybe the flag, if it ever accepted, should never be propagated = automatically. > Only geom-s that are aware of it should propagate or request it. That = should be > safer. >=20 Yes, it gets chaotic and unreliable once you have multiple layers each = thinking that they know better than the previous layer. That is the situation = that FreeBSD is currently in with CAM, GEOM, and VFS-BIO. I do not propose to make = it even more chaotic. Through just basic rules of abstraction, basic computer = science principles, it follows that the layers of software closest to the = operation (in this case, the hardware) have the least abstraction and the most amount of information on how to act. Saying that ZFS knows best how the hardware = works, when it has no access to the actual control channel information of the = hardware, is confusing and illogical to me. I think that there=E2=80=99s a legitimate problem with CAM error = recovery trying too hard to do retries when the disk is clearly telling it that it=E2=80=99s = futile. I don=E2=80=99t have a quick patch for that because I would like to think how to make it dependent on = the media type. However, it=E2=80=99s something that I=E2=80=99m willing to = discuss further and work on. Scott From owner-freebsd-fs@freebsd.org Tue Nov 28 10:41:41 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9EB3E51B11; Tue, 28 Nov 2017 10:41:41 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.blih.net", Issuer "mail.blih.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E8C027F85E; Tue, 28 Nov 2017 10:41:40 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) by mail.blih.net (OpenSMTPD) with ESMTP id 6c33a164; Tue, 28 Nov 2017 11:41:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:mime-version:content-type :content-transfer-encoding; s=mail; bh=wQ41Mb8MapTzKaga6r19wBGQb 2I=; b=Q64oZ0OeZhJrR3iiReQafsrg1z3CtUfv/zddobO1QmtDw/pa6FMJvG7jE ixDWyqIg9wsrSOaTp42OnmeaJEMfR+aM6F/wNAeBBGh+OuFm8Zf8Kjgt6pDPiTJ+ eGFH38ZfmvFs10cTWFTWhF7XpQu5rD9te48vkLr0bT/2tXKEe0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:mime-version:content-type :content-transfer-encoding; q=dns; s=mail; b=mIzuZvxiEQuD1WS5r/l KNHKuu6Zyzb6vN4u9y0/XrHggu87l54F0sSU4yro6DZIaqxfA1mKBNALehTitnGC 9q3bi5wxe9qrvn3OatJViY7lEKeveO8Bh9opuDMkL0YqbbcSHVU9ZzMFCJ76W8+9 7WJ0h/qdEx8Rx929EC5oE8WY= Received: from arcadia (evadot.gandi.net [217.70.181.36]) by mail.blih.net (OpenSMTPD) with ESMTPSA id 741b79b9 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO; Tue, 28 Nov 2017 11:41:37 +0100 (CET) Date: Tue, 28 Nov 2017 11:41:36 +0100 From: Emmanuel Vadot To: FreeBSD Current , freebsd-fs@freebsd.org Cc: Rick Macklem Subject: Switch vfs.nfsd.issue_delegations to TUNABLE ? Message-Id: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> X-Mailer: Sylpheed 3.6.0 (GTK+ 2.24.31; amd64-portbld-freebsd12.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 10:41:41 -0000 Hello, I would like to switch the vfs.nfsd.issue_delegations sysctl to a tunable. The reason behind it is recent problem at work on some on our filer related to NFS. We use NFSv4 without delegation as we never been able to have good performance with FreeBSD server and Linux client (we need to do test again but that's for later). We recently see all of our filers with NFS enabled perform pourly, resulting in really bad performance on the service. After doing some analyze with pmcstat we've seen that we spend ~50% of our time in lock delay, called by nfsrv_checkgetattr (See [1]). This function is only usefull when using delegation, as it search for some write delegations issued to client, but it's called anyway when there so no delegation and cause a lot of problem when having a lot of activities. We've patched the kernel with the included diff and now everything is fine (tm) (See [2]). The problem for upstreaming this patch is that since issue_delegations is a sysctl we cannot know if the checkgetattr called is needed, hence the question to switch it to a TUNABLE. Also maybe some other code path could benefit from it, I haven't read all the source of nfsserver yet. Note that I won't MFC the changes as it would break POLA. Cheers, [1] https://people.freebsd.org/~manu/m8.svg [2] https://people.freebsd.org/~manu/m8-new.svg >From 0cba277f406d3ccf3c9e943a3d4e17b529e31c89 Mon Sep 17 00:00:00 2001 From: Emmanuel Vadot Date: Fri, 24 Nov 2017 11:17:18 +0100 Subject: [PATCH 2/4] Do not call nfsrv_checkgetttr if delegation isn't enable. Signed-off-by: Emmanuel Vadot --- sys/fs/nfsserver/nfs_nfsdserv.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c b/sys/fs/nfsserver/nfs_nfsdserv.c index 8102c5810a9..8daf0142360 100644 --- a/sys/fs/nfsserver/nfs_nfsdserv.c +++ b/sys/fs/nfsserver/nfs_nfsdserv.c @@ -54,6 +54,7 @@ extern struct timeval nfsboottime; extern int nfs_rootfhset; extern int nfsrv_enable_crossmntpt; extern int nfsrv_statehashsize; +extern int nfsrv_issuedelegs; #endif /* !APPLEKEXT */ static int nfs_async = 0; @@ -240,7 +241,7 @@ nfsrvd_getattr(struct nfsrv_descript *nd, int isdgram, if (nd->nd_flag & ND_NFSV4) { if (NFSISSET_ATTRBIT(&attrbits, NFSATTRBIT_FILEHANDLE)) nd->nd_repstat = nfsvno_getfh(vp, &fh, p); - if (!nd->nd_repstat) + if (nd->nd_repstat == 0 && nfsrv_issuedelegs == 1) nd->nd_repstat = nfsrv_checkgetattr(nd, vp, &nva, &attrbits, nd->nd_cred, p); if (nd->nd_repstat == 0) { -- 2.14.2 -- Emmanuel Vadot From owner-freebsd-fs@freebsd.org Tue Nov 28 11:04:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55383E52418; Tue, 28 Nov 2017 11:04:35 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CCBCB80350; Tue, 28 Nov 2017 11:04:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id vASB4Sga020537 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 28 Nov 2017 13:04:28 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua vASB4Sga020537 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id vASB4Srw020536; Tue, 28 Nov 2017 13:04:28 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 28 Nov 2017 13:04:28 +0200 From: Konstantin Belousov To: Emmanuel Vadot Cc: FreeBSD Current , freebsd-fs@freebsd.org, Rick Macklem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Message-ID: <20171128110428.GN2272@kib.kiev.ua> References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> User-Agent: Mutt/1.9.1 (2017-09-22) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 11:04:35 -0000 On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote: > > Hello, > > I would like to switch the vfs.nfsd.issue_delegations sysctl to a > tunable. > The reason behind it is recent problem at work on some on our filer > related to NFS. > We use NFSv4 without delegation as we never been able to have good > performance with FreeBSD server and Linux client (we need to do test > again but that's for later). We recently see all of our filers with NFS > enabled perform pourly, resulting in really bad performance on the > service. > After doing some analyze with pmcstat we've seen that we spend ~50% > of our time in lock delay, called by nfsrv_checkgetattr (See [1]). > This function is only usefull when using delegation, as it search for > some write delegations issued to client, but it's called anyway when > there so no delegation and cause a lot of problem when having a lot of > activities. > We've patched the kernel with the included diff and now everything is > fine (tm) (See [2]). > The problem for upstreaming this patch is that since issue_delegations > is a sysctl we cannot know if the checkgetattr called is needed, hence > the question to switch it to a TUNABLE. Also maybe some other code path > could benefit from it, I haven't read all the source of nfsserver yet. > > Note that I won't MFC the changes as it would break POLA. Perhaps make nodeleg per-mount flag ? The you can check it safely by dereferencing vp->v_mount->mnt_data without acquiring any locks, while the vnode lock is owned and the vnode is not reclaimed. > > Cheers, > > [1] https://people.freebsd.org/~manu/m8.svg > [2] https://people.freebsd.org/~manu/m8-new.svg > > >From 0cba277f406d3ccf3c9e943a3d4e17b529e31c89 Mon Sep 17 00:00:00 2001 > From: Emmanuel Vadot > Date: Fri, 24 Nov 2017 11:17:18 +0100 > Subject: [PATCH 2/4] Do not call nfsrv_checkgetttr if delegation isn't > enable. > > Signed-off-by: Emmanuel Vadot > --- > sys/fs/nfsserver/nfs_nfsdserv.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c > b/sys/fs/nfsserver/nfs_nfsdserv.c index 8102c5810a9..8daf0142360 100644 > --- a/sys/fs/nfsserver/nfs_nfsdserv.c > +++ b/sys/fs/nfsserver/nfs_nfsdserv.c > @@ -54,6 +54,7 @@ extern struct timeval nfsboottime; > extern int nfs_rootfhset; > extern int nfsrv_enable_crossmntpt; > extern int nfsrv_statehashsize; > +extern int nfsrv_issuedelegs; > #endif /* !APPLEKEXT */ > > static int nfs_async = 0; > @@ -240,7 +241,7 @@ nfsrvd_getattr(struct nfsrv_descript *nd, int > isdgram, if (nd->nd_flag & ND_NFSV4) { > if (NFSISSET_ATTRBIT(&attrbits, > NFSATTRBIT_FILEHANDLE)) nd->nd_repstat = nfsvno_getfh(vp, &fh, p); > - if (!nd->nd_repstat) > + if (nd->nd_repstat == 0 && nfsrv_issuedelegs > == 1) nd->nd_repstat = nfsrv_checkgetattr(nd, vp, > &nva, &attrbits, nd->nd_cred, p); > if (nd->nd_repstat == 0) { > -- > 2.14.2 > > > -- > Emmanuel Vadot > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Nov 28 13:26:21 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFC71E56953; Tue, 28 Nov 2017 13:26:21 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.blih.net", Issuer "mail.blih.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id EC8A764AA8; Tue, 28 Nov 2017 13:26:20 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) by mail.blih.net (OpenSMTPD) with ESMTP id 3b2b6b5e; Tue, 28 Nov 2017 14:26:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=mail; bh=uQh2t/8+hXQh6T6SHB5qna9PjpM=; b=DqQFFgEOH8S3DsSWOhMvq5PZVa/E 6vOnCovRWEypmRoczT94wmLuYPVJMhoeqqXEBzaDEKCaQysU/Kx/naJZahEN6w6h LSmLH60J6G1qDydc8/oX8rpdWe9qwZYqmDGF+qTFOruP4/AZSOSjnrKVq0YiTiMG i9kNoIq3Z/X/kIk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; q=dns; s= mail; b=SYdzurtAJVfP4R0oSW52QDpyUzqFkn6FEcmK4xFCY1+YGz2CBaTOpjIb zoRVrhXiSEw1spgxdMdfIKEu4oKvLRIH5sIUyBQMdRjHpaT87Ys6CDM9XQ8hnn3r AG3hyZCF/kQZ804miXHWlB36mw5Z03eEboqagk5Tt4Ry8Jgjyks= Received: from arcadia (evadot.gandi.net [217.70.181.36]) by mail.blih.net (OpenSMTPD) with ESMTPSA id 1130363c TLS version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO; Tue, 28 Nov 2017 14:26:11 +0100 (CET) Date: Tue, 28 Nov 2017 14:26:10 +0100 From: Emmanuel Vadot To: Konstantin Belousov Cc: FreeBSD Current , freebsd-fs@freebsd.org, Rick Macklem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Message-Id: <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> In-Reply-To: <20171128110428.GN2272@kib.kiev.ua> References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> X-Mailer: Sylpheed 3.6.0 (GTK+ 2.24.31; amd64-portbld-freebsd12.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 13:26:21 -0000 On Tue, 28 Nov 2017 13:04:28 +0200 Konstantin Belousov wrote: > On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote: > > > > Hello, > > > > I would like to switch the vfs.nfsd.issue_delegations sysctl to a > > tunable. > > The reason behind it is recent problem at work on some on our filer > > related to NFS. > > We use NFSv4 without delegation as we never been able to have good > > performance with FreeBSD server and Linux client (we need to do test > > again but that's for later). We recently see all of our filers with NFS > > enabled perform pourly, resulting in really bad performance on the > > service. > > After doing some analyze with pmcstat we've seen that we spend ~50% > > of our time in lock delay, called by nfsrv_checkgetattr (See [1]). > > This function is only usefull when using delegation, as it search for > > some write delegations issued to client, but it's called anyway when > > there so no delegation and cause a lot of problem when having a lot of > > activities. > > We've patched the kernel with the included diff and now everything is > > fine (tm) (See [2]). > > The problem for upstreaming this patch is that since issue_delegations > > is a sysctl we cannot know if the checkgetattr called is needed, hence > > the question to switch it to a TUNABLE. Also maybe some other code path > > could benefit from it, I haven't read all the source of nfsserver yet. > > > > Note that I won't MFC the changes as it would break POLA. > Perhaps make nodeleg per-mount flag ? > The you can check it safely by dereferencing vp->v_mount->mnt_data without > acquiring any locks, while the vnode lock is owned and the vnode is not > reclaimed. That won't work with current code. Currently if you have delegation enabled and connect one client to a mountpoint, then disable delegation, the current client will still have delegation while new ones will not. I have not tested restarting nfsd in this situation but I suspect that client will behave badly. This is a another +1 for making it a tunable I think. > > > > Cheers, > > > > [1] https://people.freebsd.org/~manu/m8.svg > > [2] https://people.freebsd.org/~manu/m8-new.svg > > > > >From 0cba277f406d3ccf3c9e943a3d4e17b529e31c89 Mon Sep 17 00:00:00 2001 > > From: Emmanuel Vadot > > Date: Fri, 24 Nov 2017 11:17:18 +0100 > > Subject: [PATCH 2/4] Do not call nfsrv_checkgetttr if delegation isn't > > enable. > > > > Signed-off-by: Emmanuel Vadot > > --- > > sys/fs/nfsserver/nfs_nfsdserv.c | 3 ++- > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c > > b/sys/fs/nfsserver/nfs_nfsdserv.c index 8102c5810a9..8daf0142360 100644 > > --- a/sys/fs/nfsserver/nfs_nfsdserv.c > > +++ b/sys/fs/nfsserver/nfs_nfsdserv.c > > @@ -54,6 +54,7 @@ extern struct timeval nfsboottime; > > extern int nfs_rootfhset; > > extern int nfsrv_enable_crossmntpt; > > extern int nfsrv_statehashsize; > > +extern int nfsrv_issuedelegs; > > #endif /* !APPLEKEXT */ > > > > static int nfs_async = 0; > > @@ -240,7 +241,7 @@ nfsrvd_getattr(struct nfsrv_descript *nd, int > > isdgram, if (nd->nd_flag & ND_NFSV4) { > > if (NFSISSET_ATTRBIT(&attrbits, > > NFSATTRBIT_FILEHANDLE)) nd->nd_repstat = nfsvno_getfh(vp, &fh, p); > > - if (!nd->nd_repstat) > > + if (nd->nd_repstat == 0 && nfsrv_issuedelegs > > == 1) nd->nd_repstat = nfsrv_checkgetattr(nd, vp, > > &nva, &attrbits, nd->nd_cred, p); > > if (nd->nd_repstat == 0) { > > -- > > 2.14.2 > > > > > > -- > > Emmanuel Vadot > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" -- Emmanuel Vadot From owner-freebsd-fs@freebsd.org Tue Nov 28 13:40:20 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1BDC7E56EA1; Tue, 28 Nov 2017 13:40:20 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9F098651DF; Tue, 28 Nov 2017 13:40:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id vASDe9Gu056890 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 28 Nov 2017 15:40:09 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua vASDe9Gu056890 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id vASDe9sj056888; Tue, 28 Nov 2017 15:40:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 28 Nov 2017 15:40:09 +0200 From: Konstantin Belousov To: Emmanuel Vadot Cc: FreeBSD Current , freebsd-fs@freebsd.org, Rick Macklem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Message-ID: <20171128134009.GQ2272@kib.kiev.ua> References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> User-Agent: Mutt/1.9.1 (2017-09-22) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 13:40:20 -0000 On Tue, Nov 28, 2017 at 02:26:10PM +0100, Emmanuel Vadot wrote: > On Tue, 28 Nov 2017 13:04:28 +0200 > Konstantin Belousov wrote: > > > On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote: > > > > > > Hello, > > > > > > I would like to switch the vfs.nfsd.issue_delegations sysctl to a > > > tunable. > > > The reason behind it is recent problem at work on some on our filer > > > related to NFS. > > > We use NFSv4 without delegation as we never been able to have good > > > performance with FreeBSD server and Linux client (we need to do test > > > again but that's for later). We recently see all of our filers with NFS > > > enabled perform pourly, resulting in really bad performance on the > > > service. > > > After doing some analyze with pmcstat we've seen that we spend ~50% > > > of our time in lock delay, called by nfsrv_checkgetattr (See [1]). > > > This function is only usefull when using delegation, as it search for > > > some write delegations issued to client, but it's called anyway when > > > there so no delegation and cause a lot of problem when having a lot of > > > activities. > > > We've patched the kernel with the included diff and now everything is > > > fine (tm) (See [2]). > > > The problem for upstreaming this patch is that since issue_delegations > > > is a sysctl we cannot know if the checkgetattr called is needed, hence > > > the question to switch it to a TUNABLE. Also maybe some other code path > > > could benefit from it, I haven't read all the source of nfsserver yet. > > > > > > Note that I won't MFC the changes as it would break POLA. > > Perhaps make nodeleg per-mount flag ? > > The you can check it safely by dereferencing vp->v_mount->mnt_data without > > acquiring any locks, while the vnode lock is owned and the vnode is not > > reclaimed. > > That won't work with current code. Why ? > Currently if you have delegation enabled and connect one client to a > mountpoint, then disable delegation, the current client will still have > delegation while new ones will not. I have not tested restarting nfsd in > this situation but I suspect that client will behave badly. This is a > another +1 for making it a tunable I think. It is up to the filesystem to handle remount, in particular, fs can disable changing mount options on mount upgrade if the operation is not supported. In other words, you would do mount -o nodeleg ... /mnt and mount -u -o nonodeleg ... /mnt needs to return EINVAL. > > > > > > > Cheers, > > > > > > [1] https://people.freebsd.org/~manu/m8.svg > > > [2] https://people.freebsd.org/~manu/m8-new.svg > > > > > > >From 0cba277f406d3ccf3c9e943a3d4e17b529e31c89 Mon Sep 17 00:00:00 2001 > > > From: Emmanuel Vadot > > > Date: Fri, 24 Nov 2017 11:17:18 +0100 > > > Subject: [PATCH 2/4] Do not call nfsrv_checkgetttr if delegation isn't > > > enable. > > > > > > Signed-off-by: Emmanuel Vadot > > > --- > > > sys/fs/nfsserver/nfs_nfsdserv.c | 3 ++- > > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > > > diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c > > > b/sys/fs/nfsserver/nfs_nfsdserv.c index 8102c5810a9..8daf0142360 100644 > > > --- a/sys/fs/nfsserver/nfs_nfsdserv.c > > > +++ b/sys/fs/nfsserver/nfs_nfsdserv.c > > > @@ -54,6 +54,7 @@ extern struct timeval nfsboottime; > > > extern int nfs_rootfhset; > > > extern int nfsrv_enable_crossmntpt; > > > extern int nfsrv_statehashsize; > > > +extern int nfsrv_issuedelegs; > > > #endif /* !APPLEKEXT */ > > > > > > static int nfs_async = 0; > > > @@ -240,7 +241,7 @@ nfsrvd_getattr(struct nfsrv_descript *nd, int > > > isdgram, if (nd->nd_flag & ND_NFSV4) { > > > if (NFSISSET_ATTRBIT(&attrbits, > > > NFSATTRBIT_FILEHANDLE)) nd->nd_repstat = nfsvno_getfh(vp, &fh, p); > > > - if (!nd->nd_repstat) > > > + if (nd->nd_repstat == 0 && nfsrv_issuedelegs > > > == 1) nd->nd_repstat = nfsrv_checkgetattr(nd, vp, > > > &nva, &attrbits, nd->nd_cred, p); > > > if (nd->nd_repstat == 0) { > > > -- > > > 2.14.2 > > > > > > > > > -- > > > Emmanuel Vadot > > > _______________________________________________ > > > freebsd-fs@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > > > -- > Emmanuel Vadot From owner-freebsd-fs@freebsd.org Tue Nov 28 14:12:40 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E039AE57BED; Tue, 28 Nov 2017 14:12:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-co1nam03on0078.outbound.protection.outlook.com [104.47.40.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D16E662CF; Tue, 28 Nov 2017 14:12:39 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM (52.132.46.161) by YTOPR0101MB2171.CANPRD01.PROD.OUTLOOK.COM (52.132.46.160) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.260.4; Tue, 28 Nov 2017 14:12:38 +0000 Received: from YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM ([fe80::f072:85d3:769:e6f8]) by YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM ([fe80::f072:85d3:769:e6f8%13]) with mapi id 15.20.0260.007; Tue, 28 Nov 2017 14:12:38 +0000 From: Rick Macklem To: Konstantin Belousov , Emmanuel Vadot CC: FreeBSD Current , "freebsd-fs@freebsd.org" , Rick Macklem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Topic: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Index: AQHTaEyAfy6q2cCLHkW9s1ZyeaxyRaMpzDeAgAAEfyo= Date: Tue, 28 Nov 2017 14:12:38 +0000 Message-ID: References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com>, <20171128134009.GQ2272@kib.kiev.ua> In-Reply-To: <20171128134009.GQ2272@kib.kiev.ua> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB2171; 6:ix74uNI1SocEsUxRJU+8XuYUKYqxzilCzO3DXT4Si2RPKRvF5szOjimqYCmXzh+F3tNvXg+bd4bSma2EX5xaZFux6EEjvIrhhjXNbBJREY9f0FvUYIifJhB3/9EOP63pMnzEywSjSYzPzdtVjGgCCGJcKgGjObJcS3+ZnaDl9369XhO8i9bCPRMO1jB6GgRUy4WGE1OjWFC/bg8UmAErUchj1u/6zq/h/cALgUxDHKxsa2KvimBcOMAANq84XMPik3TBfGFI05Kj91pBcjvho4M1MHikY/3GACi3bFGhebRDvkv8rEwlGWA6yvr2HIURXUR+oeyi1u2Oc6jMLv2vIIHKCE+jEJXFwHKMjJ/OiL0=; 5:iERIKJKB16qwB2o3QhLxcAPrvcsdMeEXN8qsIEb52Fr+k7rPKuCxyg1Z9wLpAS5ji1pYnrBf0f8yhRy3WgjCWKlHvP8OzruOgltEjhhZWql3cu/FB0NRSLkkKQb+u/vJytD0VLAsSoQYjur3OaeQgCPrOp4MKwmRMUmJ8KZYLVU=; 24:R5LNBBN8LPw/vsNeo0pcl0+J/E0jJcxWluXtXrjVTYj0Jyjqs7PbW1EKIL5AzrUEJ355vn4CdyxK/RcqCmLzSxbElXqwDwf9OKDRo4x+gUs=; 7:vd8n8oukPMNBUqJ0hg+oEcW9F7YVzZvGVrGSzTzMuW9bg047GbxbtDKaOZRD8hvwJniQRT7/h7J47jWmln4gIa9y+cr928CPLhkc8i4CIT88KT0kkQXEMWp9BdHmaejU8MZ8341TiVWQhtjYpBlk9lu5sihJQ/c3smeTvY1WF+JLi7YRj2mQwoYUA7q7j+5IwtGr8fELfWqmfORS57Gdy1RqbERPHhbjCOsdxoXU6+xLqxffdYaR+mUZmAjRd8zO x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 5caea99f-8e08-4fdd-4d13-08d5366a14c3 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(4534020)(4602075)(4627115)(8989060)(201703031133081)(201702281549075)(8990040)(5600026)(4604075)(2017052603258); SRVR:YTOPR0101MB2171; x-ms-traffictypediagnostic: YTOPR0101MB2171: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(5005006)(8121501046)(3231022)(10201501046)(93006095)(93001095)(3002001)(6041248)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123558100)(20161123555025)(20161123562025)(20161123560025)(6072148)(201708071742011); SRVR:YTOPR0101MB2171; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:YTOPR0101MB2171; x-forefront-prvs: 0505147DDB x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(376002)(346002)(366004)(24454002)(189002)(199003)(316002)(25786009)(786003)(5250100002)(478600001)(110136005)(86362001)(55016002)(54906003)(68736007)(97736004)(7696005)(74316002)(4326008)(8676002)(3280700002)(189998001)(305945005)(81156014)(9686003)(81166006)(5660300001)(3660700001)(2950100002)(93886005)(105586002)(2900100001)(14454004)(2906002)(39060400002)(8936002)(33656002)(76176999)(102836003)(229853002)(74482002)(54356999)(99286004)(6246003)(53936002)(106356001)(50986999)(6436002)(6506006)(101416001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB2171; H:YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 5caea99f-8e08-4fdd-4d13-08d5366a14c3 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Nov 2017 14:12:38.4632 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB2171 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 14:12:41 -0000 Konstantin Belousov wrote: >On Tue, Nov 28, 2017 at 02:26:10PM +0100, Emmanuel Vadot wrote: >> On Tue, 28 Nov 2017 13:04:28 +0200 >> Konstantin Belousov wrote: >> >> > On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote: >> > > >> > > Hello, >> > > >> > > I would like to switch the vfs.nfsd.issue_delegations sysctl to a >> > > tunable. Since it defaults to "disabled", I don't see why a tunable would be necessa= ry? (Just do nothing and delegations don't happen. If you want the server to issue delegations, then use the sysctl to turn them on. If you want to = turn them off again at the server, reboot the server without setting the sysctl= .) > > > The reason behind it is recent problem at work on some on our filer > > > related to NFS. > > > We use NFSv4 without delegation as we never been able to have good > > > performance with FreeBSD server and Linux client (we need to do test > > > again but that's for later). Delegations are almost never useful, especially with Linux clients. [stuff snipped] > > Perhaps make nodeleg per-mount flag ? The sysctl vfs.nfsd.issue_delegations only affects the server. If you have a FreeBSD client that is mounting a delegations enabled server = and does not want delegations, just don't run the "nfscbd" daemon on the client= . The only time you want the "nfscbd" daemon running is for delegations and pNFS layouts. (I suppose there is the case of a client using NFSv4.1/pNFS a= gainst a server with delegations enabled, but since delegations aren't very useful= , I'd just disable delegations on the server for this case.) Having a per-mount version of this would be overkill, I think. It would hav= e to disable callbacks on the mount point, since there is no way for a client to= say "don't give me delegations" except disabling callbacks, which the server requires for delegation issue. [stuff snipped] The case where there has never been delegations issued will result in an empty delegation queue. Maybe this case can be handled without acquiring the "global NFSv4 state lock", which is what sounds like is causing the performance issue. Maybe a global counter of how many delegations are issued that is handled by atomic ops. --> If it is 0, nfsrv_checkgetattr() can just return without acquiring the = lock. I'll take a look at this, rick= From owner-freebsd-fs@freebsd.org Tue Nov 28 15:38:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 07C68DB9DFB; Tue, 28 Nov 2017 15:38:38 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.blih.net", Issuer "mail.blih.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 45A7669371; Tue, 28 Nov 2017 15:38:36 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) by mail.blih.net (OpenSMTPD) with ESMTP id 4219a0a8; Tue, 28 Nov 2017 16:38:34 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=mail; bh=DFMtGTwnA+GslL9YwFlfrZ1cOxg=; b=KuyGXYZ9yKjp/v7xZDLap7a5C6MO Jre68EvRCLcnRm++XLoND05858y0C0WgPscV53ddZ0FNvbP8bn7I/iHz0VIlSwBE 3cCD69wQk7Chg13VAS9AfAo63N+TBz3cpPuQJuHcfcx2CSh3ObCOKgm7pJeCTuXv 2rL/9xsLWf8OWns= DomainKey-Signature: a=rsa-sha1; c=nofws; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; q=dns; s= mail; b=KNnFRi0IqesS0s/Xq33+dQ5mr34o8+t83lNNyaWx6gLdebYhs1h3TENZ +6hiVq5/TmLSK3DCFyadmflBFBCLqXdyNq/xk45Ji2MPvd+1S4/TEKxvqlbPfMJ1 gAHJlH1IKv4qNTg/htEBvxHw33b/Lc1YIaTVVB8slCezMDd8nGk= Received: from arcadia (evadot.gandi.net [217.70.181.36]) by mail.blih.net (OpenSMTPD) with ESMTPSA id 836f99d3 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO; Tue, 28 Nov 2017 16:38:34 +0100 (CET) Date: Tue, 28 Nov 2017 16:38:33 +0100 From: Emmanuel Vadot To: Konstantin Belousov Cc: FreeBSD Current , freebsd-fs@freebsd.org, Rick Macklem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Message-Id: <20171128163833.696c04c8dfc7dd9fc183d2df@bidouilliste.com> In-Reply-To: <20171128134009.GQ2272@kib.kiev.ua> References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> X-Mailer: Sylpheed 3.6.0 (GTK+ 2.24.31; amd64-portbld-freebsd12.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 15:38:38 -0000 On Tue, 28 Nov 2017 15:40:09 +0200 Konstantin Belousov wrote: > On Tue, Nov 28, 2017 at 02:26:10PM +0100, Emmanuel Vadot wrote: > > On Tue, 28 Nov 2017 13:04:28 +0200 > > Konstantin Belousov wrote: > > > > > On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote: > > > > > > > > Hello, > > > > > > > > I would like to switch the vfs.nfsd.issue_delegations sysctl to a > > > > tunable. > > > > The reason behind it is recent problem at work on some on our filer > > > > related to NFS. > > > > We use NFSv4 without delegation as we never been able to have good > > > > performance with FreeBSD server and Linux client (we need to do test > > > > again but that's for later). We recently see all of our filers with NFS > > > > enabled perform pourly, resulting in really bad performance on the > > > > service. > > > > After doing some analyze with pmcstat we've seen that we spend ~50% > > > > of our time in lock delay, called by nfsrv_checkgetattr (See [1]). > > > > This function is only usefull when using delegation, as it search for > > > > some write delegations issued to client, but it's called anyway when > > > > there so no delegation and cause a lot of problem when having a lot of > > > > activities. > > > > We've patched the kernel with the included diff and now everything is > > > > fine (tm) (See [2]). > > > > The problem for upstreaming this patch is that since issue_delegations > > > > is a sysctl we cannot know if the checkgetattr called is needed, hence > > > > the question to switch it to a TUNABLE. Also maybe some other code path > > > > could benefit from it, I haven't read all the source of nfsserver yet. > > > > > > > > Note that I won't MFC the changes as it would break POLA. > > > Perhaps make nodeleg per-mount flag ? > > > The you can check it safely by dereferencing vp->v_mount->mnt_data without > > > acquiring any locks, while the vnode lock is owned and the vnode is not > > > reclaimed. > > > > That won't work with current code. > Why ? > > > Currently if you have delegation enabled and connect one client to a > > mountpoint, then disable delegation, the current client will still have > > delegation while new ones will not. I have not tested restarting nfsd in > > this situation but I suspect that client will behave badly. This is a > > another +1 for making it a tunable I think. > It is up to the filesystem to handle remount, in particular, fs can disable > changing mount options on mount upgrade if the operation is not supported. > In other words, you would do > mount -o nodeleg ... /mnt > and > mount -u -o nonodeleg ... /mnt > needs to return EINVAL. You are talking about client here while I'm talking about server. > > > > > > > > > > Cheers, > > > > > > > > [1] https://people.freebsd.org/~manu/m8.svg > > > > [2] https://people.freebsd.org/~manu/m8-new.svg > > > > > > > > >From 0cba277f406d3ccf3c9e943a3d4e17b529e31c89 Mon Sep 17 00:00:00 2001 > > > > From: Emmanuel Vadot > > > > Date: Fri, 24 Nov 2017 11:17:18 +0100 > > > > Subject: [PATCH 2/4] Do not call nfsrv_checkgetttr if delegation isn't > > > > enable. > > > > > > > > Signed-off-by: Emmanuel Vadot > > > > --- > > > > sys/fs/nfsserver/nfs_nfsdserv.c | 3 ++- > > > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c > > > > b/sys/fs/nfsserver/nfs_nfsdserv.c index 8102c5810a9..8daf0142360 100644 > > > > --- a/sys/fs/nfsserver/nfs_nfsdserv.c > > > > +++ b/sys/fs/nfsserver/nfs_nfsdserv.c > > > > @@ -54,6 +54,7 @@ extern struct timeval nfsboottime; > > > > extern int nfs_rootfhset; > > > > extern int nfsrv_enable_crossmntpt; > > > > extern int nfsrv_statehashsize; > > > > +extern int nfsrv_issuedelegs; > > > > #endif /* !APPLEKEXT */ > > > > > > > > static int nfs_async = 0; > > > > @@ -240,7 +241,7 @@ nfsrvd_getattr(struct nfsrv_descript *nd, int > > > > isdgram, if (nd->nd_flag & ND_NFSV4) { > > > > if (NFSISSET_ATTRBIT(&attrbits, > > > > NFSATTRBIT_FILEHANDLE)) nd->nd_repstat = nfsvno_getfh(vp, &fh, p); > > > > - if (!nd->nd_repstat) > > > > + if (nd->nd_repstat == 0 && nfsrv_issuedelegs > > > > == 1) nd->nd_repstat = nfsrv_checkgetattr(nd, vp, > > > > &nva, &attrbits, nd->nd_cred, p); > > > > if (nd->nd_repstat == 0) { > > > > -- > > > > 2.14.2 > > > > > > > > > > > > -- > > > > Emmanuel Vadot > > > > _______________________________________________ > > > > freebsd-fs@freebsd.org mailing list > > > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > _______________________________________________ > > > freebsd-current@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > > > > > > -- > > Emmanuel Vadot > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" -- Emmanuel Vadot From owner-freebsd-fs@freebsd.org Tue Nov 28 15:41:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4378CDBA2DC; Tue, 28 Nov 2017 15:41:38 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.blih.net", Issuer "mail.blih.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 330BD69788; Tue, 28 Nov 2017 15:41:34 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) by mail.blih.net (OpenSMTPD) with ESMTP id e27b0ab0; Tue, 28 Nov 2017 16:41:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=mail; bh=QTIfWcGWeCrF/MOyhkEvMjwA8/s=; b=bt9yCPrzgk0ZXmDXGfYRblqyrEpc Ra0hUg8knck229h8oeyfawdSliPMsXujvPanJzZUbjbY1TnV2wTnElX/JRVdgMiW aViELFAxi8BYEo//t5jev7ygLb3pevVRthWS11zstnYfILy7Nh4LrmzYX8GQYACq xi2VbVzQcm3WylQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; q=dns; s= mail; b=gStvohJrRckZG+Qfa3F5ux/8362dj2I53ZbsZGnDR3xcs3Ky7hr5f6hu 5uD0gMBxosZY6jU8nNmluCatVBBxmeVvH8eSnPbcKAqNdf5+Dif3XnrWed6yKP/p LCKu1JbIWeieIQALg2aSLulq2V1UiuV167g/kRf5BLtdQY3VlrQ= Received: from arcadia (evadot.gandi.net [217.70.181.36]) by mail.blih.net (OpenSMTPD) with ESMTPSA id 602eac22 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO; Tue, 28 Nov 2017 16:41:32 +0100 (CET) Date: Tue, 28 Nov 2017 16:41:32 +0100 From: Emmanuel Vadot To: Rick Macklem Cc: Konstantin Belousov , FreeBSD Current , "freebsd-fs@freebsd.org" , Rick Macklem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Message-Id: <20171128164132.290bf07218b16164db613242@bidouilliste.com> In-Reply-To: References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> X-Mailer: Sylpheed 3.6.0 (GTK+ 2.24.31; amd64-portbld-freebsd12.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 15:41:38 -0000 On Tue, 28 Nov 2017 14:12:38 +0000 Rick Macklem wrote: > Konstantin Belousov wrote: > >On Tue, Nov 28, 2017 at 02:26:10PM +0100, Emmanuel Vadot wrote: > >> On Tue, 28 Nov 2017 13:04:28 +0200 > >> Konstantin Belousov wrote: > >> > >> > On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote: > >> > > > >> > > Hello, > >> > > > >> > > I would like to switch the vfs.nfsd.issue_delegations sysctl to a > >> > > tunable. > Since it defaults to "disabled", I don't see why a tunable would be necessary? > (Just do nothing and delegations don't happen. If you want the server > to issue delegations, then use the sysctl to turn them on. If you want to turn > them off again at the server, reboot the server without setting the sysctl.) If you need to reboot to make things working again without delegation this shouldn't be a sysctl. > > > > The reason behind it is recent problem at work on some on our filer > > > > related to NFS. > > > > We use NFSv4 without delegation as we never been able to have good > > > > performance with FreeBSD server and Linux client (we need to do test > > > > again but that's for later). > Delegations are almost never useful, especially with Linux clients. Can you elaborate ? Reading what delegation are I see that this is exactly what I'm looking for to have better performance with NFS for my workload (I only have one client per mount point). > [stuff snipped] > > > Perhaps make nodeleg per-mount flag ? > The sysctl vfs.nfsd.issue_delegations only affects the server. > If you have a FreeBSD client that is mounting a delegations enabled server and > does not want delegations, just don't run the "nfscbd" daemon on the client. > The only time you want the "nfscbd" daemon running is for delegations and > pNFS layouts. (I suppose there is the case of a client using NFSv4.1/pNFS against > a server with delegations enabled, but since delegations aren't very useful, > I'd just disable delegations on the server for this case.) > > Having a per-mount version of this would be overkill, I think. It would have to > disable callbacks on the mount point, since there is no way for a client to say > "don't give me delegations" except disabling callbacks, which the server > requires for delegation issue. > [stuff snipped] > The case where there has never been delegations issued will result in an > empty delegation queue. Maybe this case can be handled without acquiring > the "global NFSv4 state lock", which is what sounds like is causing the > performance issue. Maybe a global counter of how many delegations are > issued that is handled by atomic ops. > --> If it is 0, nfsrv_checkgetattr() can just return without acquiring the lock. > > I'll take a look at this, rick > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" -- Emmanuel Vadot From owner-freebsd-fs@freebsd.org Tue Nov 28 19:09:54 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 211A0DEBB16; Tue, 28 Nov 2017 19:09:54 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-sn1nam01on0086.outbound.protection.outlook.com [104.47.32.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B749373F58; Tue, 28 Nov 2017 19:09:53 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM (52.132.46.161) by YTOPR0101MB2170.CANPRD01.PROD.OUTLOOK.COM (52.132.46.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.260.4; Tue, 28 Nov 2017 19:09:51 +0000 Received: from YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM ([fe80::f072:85d3:769:e6f8]) by YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM ([fe80::f072:85d3:769:e6f8%13]) with mapi id 15.20.0260.007; Tue, 28 Nov 2017 19:09:51 +0000 From: Rick Macklem To: Emmanuel Vadot CC: Konstantin Belousov , FreeBSD Current , "freebsd-fs@freebsd.org" , Rick Macklem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Topic: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Index: AQHTaEyAfy6q2cCLHkW9s1ZyeaxyRaMpzDeAgAAEfyqAAB1rAIAANPob Date: Tue, 28 Nov 2017 19:09:51 +0000 Message-ID: References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> , <20171128164132.290bf07218b16164db613242@bidouilliste.com> In-Reply-To: <20171128164132.290bf07218b16164db613242@bidouilliste.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB2170; 6:6P5l40wMw0G7+elFo+SE/zuOwWEr1RYUlHsn1FwowPdxXx8AEcIlVVJQh/P9na1oM5GbrL4+XZL+wQKIpHuLgUnB8UuKEFcgG1SwfSglSc4vG6RUoUR2g3NHiHYY4TY5mbqvT4PrKh6hH45pWOsHDptOgAOX5usJyjO7+RKjx6arDjszX587tmLYwIX+kVsEKs1t5OVGr3X1VIjCluhxFDM/do5emIRWp+ckFKvHRdQb6+7TDz5fBifu1sN1ew8dxF5JQEtI0H1QZpbmB8RGBtrcCa88oDCLknoP7/6IIz862DWf+e5KX06kb7PDIF3HAxJIVKV4WPk6AhmjKgpBA5H+NXUQSPVpaGUL5wkBXFQ=; 5:nT1mQmh6CCcOrdFzC2ZhxI5jcf2gk+IprlrhzgoSzp70cZW0gS2TShWZq6MwI7I075Vucho+gZ9feZUi6skVWlzn1pOx8fRFjluvx9/55aHH6r7AZvSAcuw4IoKC/8O4BZ0mr9qFPSF5kBAohiZrFpj03AY2mAxRB1cB+dh8S3U=; 24:KHuhjr61DDGwuZuJXFIztjb9noqX0oVJWdkYAvYuFhif+W1YTVCFmX/1P+GNRKVEKTQOmwUe9MkKy7FI1Huc7YZKo7IYmfbZz6fBnHM7hGs=; 7:BzNKDNFmpXOqBj3rqwTivgEZZ2/6Va8VKrryNTEeMUkN06Gw+1yVRnXKpVozWuWMpWhYvD3Atb8O9zKZhwRvvrPXBsit9aHmwatsAdtoviojLCD5pCrkr1nXudc+AK2PYBVjWEgyAF9Ol18QGbYfVuQZxJ5PlOj/HrlYdMsmVhJHWHUWFS/0YVLAYkrjCJBwz6bAMw1tYjwDtfbKuJZpa6bYr6QNB64URvOGg1CsauWWfGGGzYtvY0L2N4HTob4P x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 8ffe4927-c342-4077-954c-08d536939a38 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(4534020)(4602075)(4627115)(8989060)(201703031133081)(201702281549075)(8990040)(5600026)(4604075)(2017052603258); SRVR:YTOPR0101MB2170; x-ms-traffictypediagnostic: YTOPR0101MB2170: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(5005006)(8121501046)(3231022)(93006095)(93001095)(3002001)(10201501046)(6041248)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123558100)(20161123555025)(20161123564025)(20161123560025)(6072148)(201708071742011); SRVR:YTOPR0101MB2170; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:YTOPR0101MB2170; x-forefront-prvs: 0505147DDB x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(366004)(346002)(376002)(189002)(24454002)(199003)(68736007)(105586002)(54906003)(2906002)(229853002)(81166006)(53936002)(81156014)(33656002)(8676002)(305945005)(5250100002)(55016002)(478600001)(97736004)(4326008)(316002)(8936002)(76176999)(101416001)(9686003)(2900100001)(6506006)(102836003)(786003)(3660700001)(3280700002)(54356999)(6436002)(50986999)(6916009)(2950100002)(86362001)(14454004)(106356001)(7696005)(99286004)(39060400002)(74482002)(6246003)(74316002)(189998001)(5660300001)(93886005)(25786009); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB2170; H:YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 8ffe4927-c342-4077-954c-08d536939a38 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Nov 2017 19:09:51.7660 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB2170 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 19:09:54 -0000 Emmanuel Vadot wrote: >I wrote: >> Since it defaults to "disabled", I don't see why a tunable would be nece= ssary? >> (Just do nothing and delegations don't happen. If you want the server >> to issue delegations, then use the sysctl to turn them on. If you want = to turn >> them off again at the server, reboot the server without setting the sys= ctl.) > >If you need to reboot to make things working again without delegation >this shouldn't be a sysctl. Turning them off without rebooting doesn't break anything. I just wrote it that way since you seemed to want to use a tunable, which implied rebooting was your preference. > > > > The reason behind it is recent problem at work on some on our file= r > > > > related to NFS. > > > > We use NFSv4 without delegation as we never been able to have good > > > > performance with FreeBSD server and Linux client (we need to do tes= t > > > > again but that's for later). > Delegations are almost never useful, especially with Linux clients. Emmanuel Vadot wrote: >Can you elaborate ? Reading what delegation are I see that this is >exactly what I'm looking for to have better performance with NFS for my >workload (I only have one client per mount point). Delegations allow the client to do Opens locally without contacting the server. Unless, the delegation is a write delegation, this only applied to read-only delegations. Also, since most implementors couldn`t agree on how to check permissions via the delegation, most client implementations still do an Access check at open, which is almost as much overhead as the Open RPC. (For example, Solaris servers always specified an empty ACE in th= e delegation, forcing the client to do an Access. I have no idea what the current Linux server=E9client does. If you capture packets when a Linux client is mounted with delegations enabled, you could look for RPCs like Ac= cess when are Opened multiple times. If you see them, then delegations aren`t saving = RPCs. Also, they are `per file`, so are only useful if the client is Opening the same file multiple times. Further, if another client Opens the same file and the first client got a W= rite delegation, then the write delegation must be recalled, which is a lot of overhead and one of the few cases where the FreeBSD server must exclusively lock the state lists, forcing all other RPCs related to Open, Close to wait= . They sounded good in theory and might have worked well if the implementors had agreed to do them, but that didn=E8t happen. (Companies pay for servers= , but the clients don=E8t get funded so delegation support in the clients are lacking= . I tried to make them useful in the FreeBSD client, but I=E8m not sure I succeeded.) > [stuff snipped] If I understood your original post, you have a performance problem caused by lock contention, where the server grabs the state lock to check for dele= gations for every Getattr from a client. As below, I think the fix is to add code to check for no delegations issued= that does not require acquisition of the state lock. Btw, large numbers of Getattrs will not perform well with delegations. (Again, the client should be able to do Getattr operations locally in the client when it has a delegation for the file, but if the client is not doi= ng that...) I wrote: > > Having a per-mount version of this would be overkill, I think. It would h= ave to > disable callbacks on the mount point, since there is no way for a client = to say > "don't give me delegations" except disabling callbacks, which the server > requires for delegation issue. > [stuff snipped] > The case where there has never been delegations issued will result in an > empty delegation queue. Maybe this case can be handled without acquiring > the "global NFSv4 state lock", which is what sounds like is causing the > performance issue. Maybe a global counter of how many delegations are > issued that is handled by atomic ops. > --> If it is 0, nfsrv_checkgetattr() can just return without acquiring th= e lock. > > I'll take a look at this, rick >=20 rick= From owner-freebsd-fs@freebsd.org Tue Nov 28 22:27:30 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76614DF0AC5 for ; Tue, 28 Nov 2017 22:27:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 623AE7B4DB for ; Tue, 28 Nov 2017 22:27:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vASMRT84090484 for ; Tue, 28 Nov 2017 22:27:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 209571] ZFS and NVMe performing poorly. TRIM requests stall I/O activity Date: Tue, 28 Nov 2017 22:27:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: jim@ks.uiuc.edu X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 22:27:30 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209571 Jim Phillips changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jim@ks.uiuc.edu --- Comment #3 from Jim Phillips --- I'm going to post this as a separate issue, but will add it here since it is related. We were seeing non-NVMe SSDs on ZFS timing out on TRIM operations= and have been able to greatly reduce the incidence by setting (for all drives) sysctl kern.cam.da.0.delete_max=3D536870912 based on the advice here: https://lists.freebsd.org/pipermail/freebsd-scsi/2015-July/006777.html --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Nov 28 22:31:44 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 908C0DF0D24 for ; Tue, 28 Nov 2017 22:31:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7EA697B7B7 for ; Tue, 28 Nov 2017 22:31:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vASMVixK001392 for ; Tue, 28 Nov 2017 22:31:44 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 209571] ZFS and NVMe performing poorly. TRIM requests stall I/O activity Date: Tue, 28 Nov 2017 22:31:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: imp@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 22:31:44 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209571 Warner Losh changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |imp@FreeBSD.org --- Comment #4 from Warner Losh --- nvd and nda have the opposite problem that da/ada have. ada/da collapse as = many BIO_DELETE commands down into a single TRIM to the device. That's why limit= ing helps since huge trims take a long time. nvd/nda, however, has the opposite problem. They do no trim collapsing at a= ll, so flood the device with TRIM requests that starve read/write requests. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Nov 28 23:17:16 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8C58DF1C03; Tue, 28 Nov 2017 23:17:16 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0048.outbound.protection.outlook.com [104.47.38.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 68AF77CF6B; Tue, 28 Nov 2017 23:17:15 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM (52.132.46.161) by YTOPR0101MB2171.CANPRD01.PROD.OUTLOOK.COM (52.132.46.160) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.260.4; Tue, 28 Nov 2017 23:17:13 +0000 Received: from YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM ([fe80::f072:85d3:769:e6f8]) by YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM ([fe80::f072:85d3:769:e6f8%13]) with mapi id 15.20.0260.007; Tue, 28 Nov 2017 23:17:13 +0000 From: Rick Macklem To: Emmanuel Vadot CC: Konstantin Belousov , FreeBSD Current , "freebsd-fs@freebsd.org" , Rick Macklem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Topic: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Index: AQHTaEyAfy6q2cCLHkW9s1ZyeaxyRaMpzDeAgAAEfyqAAB1rAIAANPobgABJSsw= Date: Tue, 28 Nov 2017 23:17:13 +0000 Message-ID: References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> , <20171128164132.290bf07218b16164db613242@bidouilliste.com>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB2171; 6:M1ZckBEATmcErrjwqZv99OTOb+074e4lV++P1f5Rar2Mg+IDfMb9l2yW435gJ6tfeVymd+kX67Mv0NSGiFqcB2CfXhQTJBp+7DaCt5wA8sT0iPuzx+6HuT+zQBQLQyfpBlI/lCL3IqXp/d0bWz8ktEXkfpUz3agz4cwZ4x9N6C7EAp2kfDMffMGBqInlPgf8DNhIDw5Rp6OCPlerWIC5AYy1Bmdtpy2itdNBtprEagmLEf2KSVdJm9m254i3zloixuOo4njyj6HrSE8FnFyS+bbGrFAsPqU5YOdesM9LiS7NWD/2AnsbzI9ip6YeRaLCAG45q9azOJioH/bkU94y1rUXHam5iYUNCHtguGyrYGI=; 5:Y0E6MhBF4B2ebz6+I0YlLx4SdzWz5hOUY0YtAI+bLpVtYifWlG56ttoqkst4yVhP/PrAxFlnwMmfH+xM6d17QqAEkf9YT0hwjfAMLN5nXh6tIeF/U/e2yPJ2GqMgZi/WXG/7jq6PSQzit0tfDkkHis18B+T3C/yChiwhhnSvEgw=; 24:yHrycrDN0C/yibGarjiS7wna39fVmVHnqa4yHyy+r1h1ivspyVgURUEVlvtwB0TkYaKtuUv/xGv9Sg/GXXQ+LPhEyut4vI92De6vm92bpbk=; 7:AjZ0KRU4FvAwpe2ZaIFnWV8jwxWvvIA+0PkgHFL15QUoUMm02KL1ratazp1IJhfGr98bhUOsHnwv3nC+71k7XBAzAnBr15o9MP4hPVJxCZLJdlp2Tv22Ai6N38Hw3ywg6UpNXN2a2UsPT3hNqStgB50EDZFBLZxJ36V5EPvBtZF137ti0L0ChyI0NjoSrnnc30CLzfK3OZt7U4IftRDaSZUcuUPo8Fav79/0+4aeksX7Kq/RhsxmcT9g2H/wXhoX x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 412b0a5f-9dd5-4b57-e7d9-08d536b628c7 x-microsoft-antispam: UriScan:(18154293887054); BCL:0; PCL:0; RULEID:(4534020)(4602075)(4627115)(8989060)(201703031133081)(201702281549075)(8990040)(5600026)(4604075)(2017052603261); SRVR:YTOPR0101MB2171; x-ms-traffictypediagnostic: YTOPR0101MB2171: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(18154293887054); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(5005006)(8121501046)(3231022)(10201501046)(3002001)(93006095)(93001095)(6041248)(20161123560025)(20161123564025)(20161123555025)(20161123558100)(20161123562025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:YTOPR0101MB2171; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:YTOPR0101MB2171; x-forefront-prvs: 0505147DDB x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(376002)(346002)(366004)(24454002)(189002)(199003)(5890100001)(25786009)(316002)(786003)(76176999)(5250100002)(189998001)(478600001)(86362001)(55016002)(54906003)(53546010)(68736007)(2940100002)(4326008)(3280700002)(8676002)(97736004)(7696005)(74316002)(305945005)(81156014)(81166006)(9686003)(3660700001)(5660300001)(2950100002)(14454004)(6916009)(105586002)(2900100001)(93886005)(39060400002)(2906002)(33656002)(8936002)(106356001)(229853002)(74482002)(102836003)(54356999)(99286004)(6246003)(53936002)(6436002)(101416001)(50986999)(6506006); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB2171; H:YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 412b0a5f-9dd5-4b57-e7d9-08d536b628c7 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Nov 2017 23:17:13.7368 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB2171 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 23:17:17 -0000 I think the attached patch should fix your performance problem. It simply checks for nfsrv_delegatecnt !=3D 0 before doing all the locking stuff in nfsrv_checkgetattr(). If this fixes your performance problem (with delegations disabled), I'll probably add a separate counter for write delegations and use atomics to increment/decrement it so that it is SMP safe without acquiring any lock. If you can test this, please let me know how it goes? rick ________________________________________ From: Rick Macklem Sent: Tuesday, November 28, 2017 2:09:51 PM To: Emmanuel Vadot Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Mack= lem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Emmanuel Vadot wrote: >I wrote: >> Since it defaults to "disabled", I don't see why a tunable would be nece= ssary? >> (Just do nothing and delegations don't happen. If you want the server >> to issue delegations, then use the sysctl to turn them on. If you want = to turn >> them off again at the server, reboot the server without setting the sys= ctl.) > >If you need to reboot to make things working again without delegation >this shouldn't be a sysctl. Turning them off without rebooting doesn't break anything. I just wrote it that way since you seemed to want to use a tunable, which implied rebooting was your preference. > > > > The reason behind it is recent problem at work on some on our file= r > > > > related to NFS. > > > > We use NFSv4 without delegation as we never been able to have good > > > > performance with FreeBSD server and Linux client (we need to do tes= t > > > > again but that's for later). > Delegations are almost never useful, especially with Linux clients. Emmanuel Vadot wrote: >Can you elaborate ? Reading what delegation are I see that this is >exactly what I'm looking for to have better performance with NFS for my >workload (I only have one client per mount point). Delegations allow the client to do Opens locally without contacting the server. Unless, the delegation is a write delegation, this only applied to read-only delegations. Also, since most implementors couldn`t agree on how to check permissions via the delegation, most client implementations still do an Access check at open, which is almost as much overhead as the Open RPC. (For example, Solaris servers always specified an empty ACE in th= e delegation, forcing the client to do an Access. I have no idea what the current Linux server=E9client does. If you capture packets when a Linux client is mounted with delegations enabled, you could look for RPCs like Ac= cess when are Opened multiple times. If you see them, then delegations aren`t saving = RPCs. Also, they are `per file`, so are only useful if the client is Opening the same file multiple times. Further, if another client Opens the same file and the first client got a W= rite delegation, then the write delegation must be recalled, which is a lot of overhead and one of the few cases where the FreeBSD server must exclusively lock the state lists, forcing all other RPCs related to Open, Close to wait= . They sounded good in theory and might have worked well if the implementors had agreed to do them, but that didn=E8t happen. (Companies pay for servers= , but the clients don=E8t get funded so delegation support in the clients are lacking= . I tried to make them useful in the FreeBSD client, but I=E8m not sure I succeeded.) > [stuff snipped] If I understood your original post, you have a performance problem caused by lock contention, where the server grabs the state lock to check for dele= gations for every Getattr from a client. As below, I think the fix is to add code to check for no delegations issued= that does not require acquisition of the state lock. Btw, large numbers of Getattrs will not perform well with delegations. (Again, the client should be able to do Getattr operations locally in the client when it has a delegation for the file, but if the client is not doi= ng that...) I wrote: > > Having a per-mount version of this would be overkill, I think. It would h= ave to > disable callbacks on the mount point, since there is no way for a client = to say > "don't give me delegations" except disabling callbacks, which the server > requires for delegation issue. > [stuff snipped] > The case where there has never been delegations issued will result in an > empty delegation queue. Maybe this case can be handled without acquiring > the "global NFSv4 state lock", which is what sounds like is causing the > performance issue. Maybe a global counter of how many delegations are > issued that is handled by atomic ops. > --> If it is 0, nfsrv_checkgetattr() can just return without acquiring th= e lock. > > I'll take a look at this, rick > rick From owner-freebsd-fs@freebsd.org Tue Nov 28 23:19:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7830DF1DDE; Tue, 28 Nov 2017 23:19:01 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-sn1nam01on0042.outbound.protection.outlook.com [104.47.32.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6B3927D29F; Tue, 28 Nov 2017 23:19:00 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM (52.132.46.161) by YTOPR0101MB2171.CANPRD01.PROD.OUTLOOK.COM (52.132.46.160) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.260.4; Tue, 28 Nov 2017 23:18:59 +0000 Received: from YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM ([fe80::f072:85d3:769:e6f8]) by YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM ([fe80::f072:85d3:769:e6f8%13]) with mapi id 15.20.0260.007; Tue, 28 Nov 2017 23:18:57 +0000 From: Rick Macklem To: Emmanuel Vadot CC: Konstantin Belousov , FreeBSD Current , "freebsd-fs@freebsd.org" , Rick Macklem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Topic: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Index: AQHTaEyAfy6q2cCLHkW9s1ZyeaxyRaMpzDeAgAAEfyqAAB1rAIAANPobgABJSsyAAAEe3g== Date: Tue, 28 Nov 2017 23:18:57 +0000 Message-ID: References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> , <20171128164132.290bf07218b16164db613242@bidouilliste.com>, , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB2171; 6:kqoz5f9CHPgWAMFJyzcl5A+3GStgDFV9rEe/BdspUneWbRWr0anaEe0jbBs+x5q42dfA+8ozUockrgnf0aY9t0CV4+fC5vDwuRCdOedY13mPjOlVOWiFQma/g2dgHKuWoUjiFtbbB6uFtn7RVaQsuDAGjIH3Lv19PB5wVnksA0BHYCS5/VMFDrRZQuVteBQvmndLhHXil0kRclf0xQlly7VQ6CyrJY7xvMEuhoXXTmACYEpRldwiw/7rMNe2mtiZ72Ii6+vqZLuV7Cx9+knmQx7FJaXGegLvFjHjubqhAO1y166VfzPVUktdvfCwGiZX2dOaO0CtRNwtFScuqryw/v3TQeATAf01iCipdVYk5Ro=; 5:5bhncfadejoejFRnow/FMLCpNU1kwpQOObVCN5YW5WTuiQHRL9q7Kf58GjVCdmRGrsAI/ytbCChN+EK4qvA35pJJY0hO1FofVwsrKnPfqRe5nBT2K2UJUAQMCheMyaP1fs2BkrzlUnTjkH/XThll25qLpKbxdD/tssPv27zDy2I=; 24:Y87QWcn6vYx6tq9E22giMehLFRK0CAhEjaZ+JUIdgeMqqcxDZ6NVVPuXxuR9p1ml6/iLNHJN11dTnXBGuCXOv5vXRAI+ePB9NTEmUYf97IQ=; 7:VjQ2K4+zpyTuKWX3frij7Xd0ZGy7j/S84j78cDweaWoaByddLojaWPlKJx8WMcsaMBSenQAwfVRZHHKflyEYXX6XH3tFLJg9bKa0UBuwx89Bkt1J4xhLeNj5PMZha2QRRAa5M8M7O+QK16ZR5R3nuSpspfllk7QWj2HBr0hiISuwWEtowszf3q0uFe+P2+1wMuXkPHtAQwBMzOPQi96scH71E6FR0I9V83odKzT8onGLIv8RDRlnk/8909IcqOdf x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 59445177-2c8c-4f87-b52a-08d536b666bb x-microsoft-antispam: UriScan:(18154293887054); BCL:0; PCL:0; RULEID:(4534020)(4602075)(4627115)(8989060)(201703031133081)(201702281549075)(8990040)(5600026)(4604075)(2017052603261)(49563074); SRVR:YTOPR0101MB2171; x-ms-traffictypediagnostic: YTOPR0101MB2171: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(18154293887054); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(102415395)(6040450)(2401047)(5005006)(8121501046)(3231022)(10201501046)(3002001)(93006095)(93001095)(6041248)(20161123560025)(20161123564025)(20161123555025)(20161123558100)(20161123562025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:YTOPR0101MB2171; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:YTOPR0101MB2171; x-forefront-prvs: 0505147DDB x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(376002)(346002)(366004)(24454002)(189002)(199003)(5890100001)(25786009)(316002)(786003)(76176999)(5250100002)(189998001)(478600001)(86362001)(55016002)(54906003)(53546010)(68736007)(2940100002)(4326008)(3280700002)(8676002)(97736004)(7696005)(74316002)(305945005)(81156014)(81166006)(9686003)(3660700001)(5660300001)(2950100002)(14454004)(6916009)(105586002)(2900100001)(93886005)(39060400002)(2906002)(33656002)(8936002)(106356001)(229853002)(74482002)(102836003)(54356999)(99936001)(99286004)(6246003)(53936002)(6436002)(101416001)(50986999)(6506006); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB2171; H:YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/mixed; boundary="_002_YTOPR0101MB21721E82EF605249D38064A1DD3A0YTOPR0101MB2172_" MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 59445177-2c8c-4f87-b52a-08d536b666bb X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Nov 2017 23:18:57.7537 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB2171 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 23:19:02 -0000 --_002_YTOPR0101MB21721E82EF605249D38064A1DD3A0YTOPR0101MB2172_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Did my usual and forgot to attach it. Here's the patch, rick ________________________________________ From: Rick Macklem Sent: Tuesday, November 28, 2017 6:17:13 PM To: Emmanuel Vadot Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Mack= lem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? I think the attached patch should fix your performance problem. It simply checks for nfsrv_delegatecnt !=3D 0 before doing all the locking stuff in nfsrv_checkgetattr(). If this fixes your performance problem (with delegations disabled), I'll probably add a separate counter for write delegations and use atomics to increment/decrement it so that it is SMP safe without acquiring any lock. If you can test this, please let me know how it goes? rick ________________________________________ From: Rick Macklem Sent: Tuesday, November 28, 2017 2:09:51 PM To: Emmanuel Vadot Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Mack= lem Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Emmanuel Vadot wrote: >I wrote: >> Since it defaults to "disabled", I don't see why a tunable would be nece= ssary? >> (Just do nothing and delegations don't happen. If you want the server >> to issue delegations, then use the sysctl to turn them on. If you want = to turn >> them off again at the server, reboot the server without setting the sys= ctl.) > >If you need to reboot to make things working again without delegation >this shouldn't be a sysctl. Turning them off without rebooting doesn't break anything. I just wrote it that way since you seemed to want to use a tunable, which implied rebooting was your preference. > > > > The reason behind it is recent problem at work on some on our file= r > > > > related to NFS. > > > > We use NFSv4 without delegation as we never been able to have good > > > > performance with FreeBSD server and Linux client (we need to do tes= t > > > > again but that's for later). > Delegations are almost never useful, especially with Linux clients. Emmanuel Vadot wrote: >Can you elaborate ? Reading what delegation are I see that this is >exactly what I'm looking for to have better performance with NFS for my >workload (I only have one client per mount point). Delegations allow the client to do Opens locally without contacting the server. Unless, the delegation is a write delegation, this only applied to read-only delegations. Also, since most implementors couldn`t agree on how to check permissions via the delegation, most client implementations still do an Access check at open, which is almost as much overhead as the Open RPC. (For example, Solaris servers always specified an empty ACE in th= e delegation, forcing the client to do an Access. I have no idea what the current Linux server=E9client does. If you capture packets when a Linux client is mounted with delegations enabled, you could look for RPCs like Ac= cess when are Opened multiple times. If you see them, then delegations aren`t saving = RPCs. Also, they are `per file`, so are only useful if the client is Opening the same file multiple times. Further, if another client Opens the same file and the first client got a W= rite delegation, then the write delegation must be recalled, which is a lot of overhead and one of the few cases where the FreeBSD server must exclusively lock the state lists, forcing all other RPCs related to Open, Close to wait= . They sounded good in theory and might have worked well if the implementors had agreed to do them, but that didn=E8t happen. (Companies pay for servers= , but the clients don=E8t get funded so delegation support in the clients are lacking= . I tried to make them useful in the FreeBSD client, but I=E8m not sure I succeeded.) > [stuff snipped] If I understood your original post, you have a performance problem caused by lock contention, where the server grabs the state lock to check for dele= gations for every Getattr from a client. As below, I think the fix is to add code to check for no delegations issued= that does not require acquisition of the state lock. Btw, large numbers of Getattrs will not perform well with delegations. (Again, the client should be able to do Getattr operations locally in the client when it has a delegation for the file, but if the client is not doi= ng that...) I wrote: > > Having a per-mount version of this would be overkill, I think. It would h= ave to > disable callbacks on the mount point, since there is no way for a client = to say > "don't give me delegations" except disabling callbacks, which the server > requires for delegation issue. > [stuff snipped] > The case where there has never been delegations issued will result in an > empty delegation queue. Maybe this case can be handled without acquiring > the "global NFSv4 state lock", which is what sounds like is causing the > performance issue. Maybe a global counter of how many delegations are > issued that is handled by atomic ops. > --> If it is 0, nfsrv_checkgetattr() can just return without acquiring th= e lock. > > I'll take a look at this, rick > rick --_002_YTOPR0101MB21721E82EF605249D38064A1DD3A0YTOPR0101MB2172_ Content-Type: application/octet-stream; name="checkgetattr.patch" Content-Description: checkgetattr.patch Content-Disposition: attachment; filename="checkgetattr.patch"; size=393; creation-date="Tue, 28 Nov 2017 23:18:53 GMT"; modification-date="Tue, 28 Nov 2017 23:18:53 GMT" Content-Transfer-Encoding: base64 LS0tIGZzL25mc3NlcnZlci9uZnNfbmZzZHN0YXRlLmMuc2F2CTIwMTctMTEtMjggMDk6NTY6Mjcu NzA4NTk5MDAwIC0wNTAwCisrKyBmcy9uZnNzZXJ2ZXIvbmZzX25mc2RzdGF0ZS5jCTIwMTctMTEt MjggMDk6NTc6MzEuMTEzMTYzMDAwIC0wNTAwCkBAIC01Mjk4LDcgKzUyOTgsNyBAQCBuZnNydl9j aGVja2dldGF0dHIoc3RydWN0IG5mc3J2X2Rlc2NyaXB0CiAJdV9xdWFkX3QgZGVsZWdmaWxlcmV2 OwogCiAJTkZTQ0JHRVRBVFRSX0FUVFJCSVQoYXR0cmJpdHAsICZjYmJpdHMpOwotCWlmICghTkZT Tk9OWkVST19BVFRSQklUKCZjYmJpdHMpKQorCWlmICghTkZTTk9OWkVST19BVFRSQklUKCZjYmJp dHMpIHx8IG5mc3J2X2RlbGVnYXRlY250ID09IDApCiAJCWdvdG8gb3V0OwogCiAJLyoK --_002_YTOPR0101MB21721E82EF605249D38064A1DD3A0YTOPR0101MB2172_-- From owner-freebsd-fs@freebsd.org Wed Nov 29 13:32:46 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0B1AE542F8 for ; Wed, 29 Nov 2017 13:32:46 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.com) Received: from asp.reflexion.net (outbound-mail-210-119.reflexion.net [208.70.210.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6EA617492C for ; Wed, 29 Nov 2017 13:32:45 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.com) Received: (qmail 9761 invoked from network); 29 Nov 2017 13:32:38 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 29 Nov 2017 13:32:38 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v8.40.3) with SMTP; Wed, 29 Nov 2017 08:32:38 -0500 (EST) Received: (qmail 3055 invoked from network); 29 Nov 2017 13:32:38 -0000 Received: from unknown (HELO mail.quorum.net) (64.74.133.216) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 29 Nov 2017 13:32:38 -0000 Received: from QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e]) by QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e%14]) with mapi id 14.03.0351.000; Wed, 29 Nov 2017 05:32:36 -0800 From: Shiva Bhanujan To: Youzhong Yang , Andriy Gapon CC: "cem@freebsd.org" , "freebsd-fs@freebsd.org" Subject: RE: zio_done panic in 10.3 Thread-Topic: zio_done panic in 10.3 Thread-Index: AdNiPrZNqbG0j29/RtqtjwQQWT08KgAsk0+A//+0h/eAAMUCgP//oow5gACblACAASwOAIAAC+cAgAARrQCACkl8Kw== Date: Wed, 29 Nov 2017 13:32:36 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C3680A9C@QLEXC01.Quorum.local> References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D636@QLEXC01.Quorum.local> <41e2465d-e1b5-33ce-57b5-49bea6087d9a@FreeBSD.org> <78d712d9-dda3-0411-262e-bb64f9ab46eb@FreeBSD.org>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [24.6.174.236] Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Nov 2017 13:32:46 -0000 Hi Andriy, Could you please let me know when could a fix for this be available? Regards, Shiva From: Youzhong Yang =5Byouzhong=40gmail.com=5D Sent: Wednesday, November 22, 2017 8:26 AM To: Andriy Gapon Cc: Shiva Bhanujan; cem=40freebsd.org; freebsd-fs=40freebsd.org Subject: Re: zio_done panic in 10.3 Thanks Andriy. Two bug reports filed: https://www.illumos.org/issues/8857 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223803 On Wed, Nov 22, 2017 at 10:22 AM, Andriy Gapon=20 wrote: On 22/11/2017 16:40, Youzhong Yang wrote: > Hi Andriy, > > This is nice=21 I am 100% sure it's exactly the same issue I experienced = and then > reported to illumos mailing list. In all the crash dumps zio->io_done =3D > l2arc_read_done, so I thought the crash must be related to L2ARC. Once I = set > secondarycache=3Dmetadata, the frequency of crash went from one per 2 = days down to > one per week. I've been puzzled by what could have caused a zio being = destroyed > while there's still child zio. Your explanation definitely makes sense=21 Oh, I now recall seeing your report: https://illumos.topicbox.com/groups/zfs/Tccd8b4463865899e I remember that it raised my interest, but then I forgot about it and didn't correlate it with the latest reports. > By the way, is there a FreeBSD bug report or an illumos bug number = tracking this > issue? I would be more than happy to create one if needed, and also test = your > potential fix here in our environment. I am not aware of any existing bug report. It would be great if you could open one =5B or two :-) =5D If you open an illumos issue, please also add George Wilson as a watcher. I think that George is also interested in fixing this issue and he knows the relevant code better than me. Thank you=21 > On Tue, Nov 21, 2017 at 3:46 PM, Andriy Gapon > wrote: > > > > > On 21/11/2017 21:30, Shiva Bhanujan wrote: > > it did get compressed to 0.5G - still too big to send via email. = I did send some more debug information by running kgdb on the core file to = Andriy, and I'm waiting for any analysis that he might provide. > > Yes, kgdb-over-email turned out to be a far more efficient = compression :-) > I already have an analysis based on the information provided by = Shiva and by > another user who has the same problem and contacted me privately. > I am discussing possible ways to fix the problem with George Wilson = who was very > kind to double-check the analysis, complete it and suggest possible = fixes. > > A short version is that dbuf_prefetch and = dbuf_prefetch_indirect_done functions > chain new zio-s under the same parent zio (a completion of one child = zio may > create another child zio). They do it using arc_read which can = create either a > logical zio in most cases or a vdev zio for a read from a cache = device (2arc). > zio_done() has a check for the completion of a parent zio's children = but that > check is not completely safe and can be broken by the pattern that = dbuf_prefetch > can create. So, under some specific circumstances the parent zio = may complete > and get destroyed while there is a child zio. > > I believe this problem to be rather rare, but there could be = configurations and > workloads where it's triggered more often. > The problem does not happen if there are no cache devices. > > > From: Conrad Meyer =5Bcem=40freebsd.org = =5D > > > > Sent: Tuesday, November 21, 2017 9:04 AM > > > > To: Shiva Bhanujan > > > > Cc: Andriy Gapon;=20 freebsd-fs=40freebsd.org > > > > Subject: Re: zio_done panic in 10.3 > > > > > > > > > > > > > > > > Have you tried compressing it with e.g. xz or zstd? > > > > -- > Andriy Gapon > _______________________________________________ > freebsd-fs=40freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to = =22freebsd-fs-unsubscribe=40freebsd.org > =22 > > -- Andriy Gapon From owner-freebsd-fs@freebsd.org Wed Nov 29 13:40:51 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32D55E5453B; Wed, 29 Nov 2017 13:40:51 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.blih.net", Issuer "mail.blih.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D9CB74AD0; Wed, 29 Nov 2017 13:40:50 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) by mail.blih.net (OpenSMTPD) with ESMTP id 579ada15; Wed, 29 Nov 2017 14:40:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=mail; bh=8XLQth92F4LHfNI9F26nkLXTVBs=; b=qi7wstgRGgxferEn32iV0FRs9YhR sTj/3n9wvtdRvpmEDVL9SZbFeDIAkvEIbFwGUr6O0TSPP77FnCUcPQ22lS8hnNEf kp5GckE0zSyWUmTiAP0Hkw1NAW/B1mstbKpdRtgAZd0IeZQTbIDX1cq5dqYDjQrl uPFEOZQOnKZAKWY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; q=dns; s= mail; b=ncUxWbBPUsLkXbkIds6zwfKGdBJ6aKtKfLFk71u3wXkKARVrSoSjNo7C Ou5py8cV06zElfLrkTKnVvtXlfaiEL6qyz3SmYK+NEgdeRAHtV/jm7GYRcJtDIzo lqxnTGUVhjTon5S2zLl5S6jmOfmk/U8rUfKF0hG2ledeZDbWnGs= Received: from arcadia (evadot.gandi.net [217.70.181.36]) by mail.blih.net (OpenSMTPD) with ESMTPSA id 4f704a56 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO; Wed, 29 Nov 2017 14:40:41 +0100 (CET) Date: Wed, 29 Nov 2017 14:40:40 +0100 From: Emmanuel Vadot To: Rick Macklem Cc: Konstantin Belousov , FreeBSD Current , "freebsd-fs@freebsd.org" Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Message-Id: <20171129144040.29a10234aa5fb1cfa2611bfc@bidouilliste.com> In-Reply-To: References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> <20171128164132.290bf07218b16164db613242@bidouilliste.com> X-Mailer: Sylpheed 3.6.0 (GTK+ 2.24.31; amd64-portbld-freebsd12.0) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Nov 2017 13:40:51 -0000 Hello Rick, On Tue, 28 Nov 2017 23:18:57 +0000 Rick Macklem wrote: > Did my usual and forgot to attach it. Here's the patch, rick >=20 > ________________________________________ > From: Rick Macklem > Sent: Tuesday, November 28, 2017 6:17:13 PM > To: Emmanuel Vadot > Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Ma= cklem > Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? >=20 > I think the attached patch should fix your performance problem. > It simply checks for nfsrv_delegatecnt !=3D 0 before doing all the > locking stuff in nfsrv_checkgetattr(). >=20 > If this fixes your performance problem (with delegations disabled), > I'll probably add a separate counter for write delegations and > use atomics to increment/decrement it so that it is SMP safe without > acquiring any lock. >=20 > If you can test this, please let me know how it goes? rick I haven't test by I can say that it will work, I actually wondered at first doing that. The problem with this patch is what I tried to describe in my first and following mails, since you can turn on and off delegation you can still have delegation (so nfsrv_delegatecnt > 0) even if the sysctl is =3D=3D 0. That's why I went to the tunable way, seems cleaner to me as disabling delegation doesn't really disable them for current client. > ________________________________________ > From: Rick Macklem > Sent: Tuesday, November 28, 2017 2:09:51 PM > To: Emmanuel Vadot > Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Ma= cklem > Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? >=20 > Emmanuel Vadot wrote: > >I wrote: > >> Since it defaults to "disabled", I don't see why a tunable would be ne= cessary? > >> (Just do nothing and delegations don't happen. If you want the server > >> to issue delegations, then use the sysctl to turn them on. If you wan= t to turn > >> them off again at the server, reboot the server without setting the s= ysctl.) > > > >If you need to reboot to make things working again without delegation > >this shouldn't be a sysctl. > Turning them off without rebooting doesn't break anything. > I just wrote it that way since you seemed to want to use a tunable, which > implied rebooting was your preference. > > > > > The reason behind it is recent problem at work on some on our fi= ler > > > > > related to NFS. > > > > > We use NFSv4 without delegation as we never been able to have go= od > > > > > performance with FreeBSD server and Linux client (we need to do t= est > > > > > again but that's for later). > > Delegations are almost never useful, especially with Linux clients. > Emmanuel Vadot wrote: > >Can you elaborate ? Reading what delegation are I see that this is > >exactly what I'm looking for to have better performance with NFS for my > >workload (I only have one client per mount point). > Delegations allow the client to do Opens locally without contacting the > server. Unless, the delegation is a write delegation, this only applied > to read-only delegations. Also, since most implementors couldn`t agree > on how to check permissions via the delegation, most client implementatio= ns > still do an Access check at open, which is almost as much overhead as the > Open RPC. (For example, Solaris servers always specified an empty ACE in = the > delegation, forcing the client to do an Access. I have no idea what the > current Linux server=E9client does. If you capture packets when a Linux > client is mounted with delegations enabled, you could look for RPCs like = Access when > are Opened multiple times. If you see them, then delegations aren`t savin= g RPCs. > Also, they are `per file`, so are only useful if the client is Opening the > same file multiple times. > Further, if another client Opens the same file and the first client got a= Write > delegation, then the write delegation must be recalled, which is a lot of > overhead and one of the few cases where the FreeBSD server must exclusive= ly > lock the state lists, forcing all other RPCs related to Open, Close to wa= it. >=20 > They sounded good in theory and might have worked well if the implementors > had agreed to do them, but that didn=E8t happen. (Companies pay for serve= rs, but the > clients don=E8t get funded so delegation support in the clients are lacki= ng. I tried > to make them useful in the FreeBSD client, but I=E8m not sure I succeeded= .) >=20 > > [stuff snipped] > If I understood your original post, you have a performance problem caused > by lock contention, where the server grabs the state lock to check for de= legations > for every Getattr from a client. >=20 > As below, I think the fix is to add code to check for no delegations issu= ed that > does not require acquisition of the state lock. >=20 > Btw, large numbers of Getattrs will not perform well with delegations. > (Again, the client should be able to do Getattr operations locally in the > client when it has a delegation for the file, but if the client is not d= oing that...) >=20 > I wrote: > > > > Having a per-mount version of this would be overkill, I think. It would= have to > > disable callbacks on the mount point, since there is no way for a clien= t to say > > "don't give me delegations" except disabling callbacks, which the server > > requires for delegation issue. > > [stuff snipped] > > The case where there has never been delegations issued will result in an > > empty delegation queue. Maybe this case can be handled without acquiring > > the "global NFSv4 state lock", which is what sounds like is causing the > > performance issue. Maybe a global counter of how many delegations are > > issued that is handled by atomic ops. > > --> If it is 0, nfsrv_checkgetattr() can just return without acquiring = the lock. > > > > I'll take a look at this, rick > > > rick --=20 Emmanuel Vadot From owner-freebsd-fs@freebsd.org Wed Nov 29 17:28:06 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A76FE5E968; Wed, 29 Nov 2017 17:28:06 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0078.outbound.protection.outlook.com [104.47.42.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 366C07D2AB; Wed, 29 Nov 2017 17:28:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR0101MB2174.CANPRD01.PROD.OUTLOOK.COM (52.132.40.149) by YTXPR0101MB2174.CANPRD01.PROD.OUTLOOK.COM (52.132.40.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.260.4; Wed, 29 Nov 2017 17:28:05 +0000 Received: from YTXPR0101MB2174.CANPRD01.PROD.OUTLOOK.COM ([fe80::937:f79d:d224:f68c]) by YTXPR0101MB2174.CANPRD01.PROD.OUTLOOK.COM ([fe80::937:f79d:d224:f68c%13]) with mapi id 15.20.0260.007; Wed, 29 Nov 2017 17:28:05 +0000 From: Rick Macklem To: Emmanuel Vadot CC: Konstantin Belousov , FreeBSD Current , "freebsd-fs@freebsd.org" Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Topic: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Index: AQHTaEyAfy6q2cCLHkW9s1ZyeaxyRaMpzDeAgAAEfyqAAB1rAIAANPobgABJSsyAAAEe3oAA8S4AgAA7/Xs= Date: Wed, 29 Nov 2017 17:28:05 +0000 Message-ID: References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> <20171128164132.290bf07218b16164db613242@bidouilliste.com> , <20171129144040.29a10234aa5fb1cfa2611bfc@bidouilliste.com> In-Reply-To: <20171129144040.29a10234aa5fb1cfa2611bfc@bidouilliste.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR0101MB2174; 6:X+KElzgTxw2koNR9Tk/TXXxcHJLyz5TanBBE9POSJHsb8MZ1Ed+BpHyfE1YEnENL6TbcDIdGq6a+g2idHtEBrMkdHLGnLeqOneChJv0UxLAYp/1GjJ8N+VPXKS3ftuDj2VrKwQd8wcirL0LVXyBBPI1cNkeNfj1G/UsmVLSEJ4o+wCkEW9SZc60hlQngZW99a8V2lUF5iCODYfMs8naLrNJsbM4ljWwHOEwZxiGRmAhuEevmGsbR9BBCJNY8h2T8+NphV3wrvvfzrycJWS/MB3/sQ60lvsqvFSVHI03ImwNKWDUVH01JRT4Oh8/F0LQx2TnoIh0MwKeG/BXpAbhcYLuaOxGbo+htgI46drNuEH4=; 5:l+fhTLQf4g4HHGlwuZX5dq6iI/oFugNMfoAOADhWnKAekh0A1JiVvXqabpMTi//mxss0H+jdfoKw+DvuwgZIYap/pWrRLufKK2WMRFKxAALqYD8RA8uyJSBJsQCpz71uDq4XEzjgfZSQXNa5W7RyduC5h5habxfA+zlKOmUCKOQ=; 24:2aBVIMwPeqdRJ3SqwwdI8iUoh5XT1Clfs3kcBJqehG7b5M/LzxRecXMa9zv3bIswrf+dvPM3o462r4x1Yigs0MQdq4oNwgkPakp7OWiPEbY=; 7:22dDdENp5S7eDkO4pAA8rYawG2m8F4jgNa1e9DQeTm2oM+sK7dngQ02IwCu+qH8YyLLQH8cfF+mt7kyBAIZhsLPy7ak1hCG4LDoNwZNUQjYAWiQVeFFO0ejSkRhddQu16W5pm9gTKrE/lYfj8OUSSz/Ad3g3hcJeXRfvsMILBuDYgkqJjUGzrmZ/I5sOLIbxDTfBQvx2R7Mv5uctvK3+7rch5qF98ZLgdozH3RHW8FajqjS+0D3rNq/l3IB6+lpx x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: fcb76366-f35a-4eb2-b302-08d5374e8cc7 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(4534020)(4602075)(4627115)(8989060)(201703031133081)(201702281549075)(8990040)(5600026)(4604075)(2017052603273); SRVR:YTXPR0101MB2174; x-ms-traffictypediagnostic: YTXPR0101MB2174: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(8121501046)(5005006)(93006095)(93001095)(3231022)(10201501046)(3002001)(6041248)(20161123564025)(20161123560025)(20161123558100)(20161123555025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(6072148)(201708071742011); SRVR:YTXPR0101MB2174; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:YTXPR0101MB2174; x-forefront-prvs: 05066DEDBB x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(376002)(39830400002)(366004)(346002)(199003)(24454002)(189002)(9686003)(2900100001)(86362001)(2950100002)(5250100002)(33656002)(5660300001)(39060400002)(93886005)(6916009)(189998001)(105586002)(101416001)(106356001)(478600001)(102836003)(4326008)(7696005)(8936002)(2906002)(305945005)(14454004)(8676002)(81166006)(53936002)(74482002)(99286004)(97736004)(81156014)(55016002)(3280700002)(3660700001)(6246003)(786003)(54906003)(229853002)(74316002)(68736007)(6506006)(25786009)(6436002)(316002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR0101MB2174; H:YTXPR0101MB2174.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: fcb76366-f35a-4eb2-b302-08d5374e8cc7 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Nov 2017 17:28:05.0417 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR0101MB2174 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Nov 2017 17:28:06 -0000 Emmanuel Vadot wrote: [stuff snipped] > I haven't test by I can say that it will work, I actually wondered at >first doing that. The problem with this patch is what I tried to >describe in my first and following mails, since you can turn on and off >delegation you can still have delegation (so nfsrv_delegatecnt > 0) >even if the sysctl is =3D=3D 0. That's why I went to the tunable way, seem= s >cleaner to me as disabling delegation doesn't really disable them for >current client. Yes, if you have delegations enabled and then disable them, there will be delegations issued for a "while". During that time, the code in nfsrv_checkgetattr() does need to check for them. Since no new delegations will be issued, the outstanding ones will go away when the client chooses to return them. (You can force this on the client by doing a dismount/mount, at least for the FreeBSD client.) Alternately, rebooting the server forces the clients to "recover" delegatio= ns and, if they are disabled, that fails. (Actually the recover succeeds, but = with a flag set that tells the client it must return them asap.) All the tunable does is make it impossible to disable them without rebooting, but otherwise the effect is the same. I have a patch that keeps a separate counter for write delegations (which are the ones you care about) using atomics to maintain the value. (That will be similar but somewhat better than what this patch does.= ) rick [lots more snipped]= From owner-freebsd-fs@freebsd.org Fri Dec 1 11:37:53 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E7EDDFD3B8 for ; Fri, 1 Dec 2017 11:37:53 +0000 (UTC) (envelope-from marek.salwerowicz@misal.pl) Received: from mail3.misal.pl (mail3.misal.pl [84.10.51.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 89DA67DF1E for ; Fri, 1 Dec 2017 11:37:52 +0000 (UTC) (envelope-from marek.salwerowicz@misal.pl) Received: from localhost (mail3.misal.pl [127.0.0.1]) by mail3.misal.pl (Postfix) with ESMTP id 6B2F61FE7 for ; Fri, 1 Dec 2017 12:30:36 +0100 (CET) X-Virus-Scanned: amavisd X-Spam-Flag: NO X-Spam-Score: -3 X-Spam-Level: X-Spam-Status: No, score=-3 tagged_above=-9999 required=8 tests=[ALL_TRUSTED=-1, BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1] autolearn=ham autolearn_force=no Authentication-Results: mail3.misal.pl (amavisd-new); dkim=pass (1024-bit key) header.d=misal.pl Received: from mail3.misal.pl ([127.0.0.1]) by localhost (mail3.misal.pl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id uQEU0QQ2tWJu for ; Fri, 1 Dec 2017 12:30:35 +0100 (CET) DKIM-Filter: OpenDKIM Filter v2.11.0 mail3.misal.pl 1A809BD DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=misal.pl; s=misal.pl; t=1512127835; bh=rj0HSJh6VMzRO3EY0S3ONuUuLEpsaTfyk+w6MViEbCM=; h=From:Subject:To:Date:From; b=wQUvmkAp7V15aNmMSLCv1juyJtMbBFok986dlZObaYUT4a2a1tPpmJkIG9oe0mUBZ L1VKqpF3I2ofcpIrbCmXIORaID1RH22G+iobzy0IV82lidssE1SVyZ2Ab07ug4SW+0 cgiEcXHk+GAvqBupdnhohMpAu6agUBwLhAjSRfQ8= From: Marek Salwerowicz Subject: Reducing the ZVOL refreservation size? To: "freebsd-fs@freebsd.org" Message-ID: <811b1fb1-58ba-a8f6-238f-e52b0d392c0e@misal.pl> Date: Fri, 1 Dec 2017 12:30:34 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 11:37:53 -0000 Hi list, My box is running FreeBSD 10.3-RELEASE-p11: FreeBSD storage2 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 More than 3 years ago I created a 20 TB ZVOL, to be used as an iSCSI drive for Win2k8 Server. Currently the Server uses around 7 TB of the storage space. However, the total space consumed by the ZVOL is around 27TB (instead of desired 20TB). What I have noticed is that the 'refreservation' size value does not decrease while the 'referenced' size value increases. As I can predict, after filling the ZVOL with data, it will consume around 40 TB in total. Is it safe to reduce the 'refreservation' from 20 TB to eg. 15 TB?   So that there will be around 5TB of storage space freed? Please find below all details for the ZVOL: #zfs get all tank1/PROD/WIN-Drive NAME                   PROPERTY VALUE                  SOURCE tank1/PROD/WIN-Drive  type                  volume - tank1/PROD/WIN-Drive  creation              Wed May 21 10:53 2014 - tank1/PROD/WIN-Drive  used                  27.6T - tank1/PROD/WIN-Drive  available             26.9T - tank1/PROD/WIN-Drive  referenced            6.96T - tank1/PROD/WIN-Drive  compressratio         1.00x - tank1/PROD/WIN-Drive  reservation           none default tank1/PROD/WIN-Drive  volsize               20T local tank1/PROD/WIN-Drive  volblocksize          8K - tank1/PROD/WIN-Drive  checksum              on default tank1/PROD/WIN-Drive  compression           off default tank1/PROD/WIN-Drive  readonly              off default tank1/PROD/WIN-Drive  copies                1 default tank1/PROD/WIN-Drive  refreservation        20.6T local tank1/PROD/WIN-Drive  primarycache          all default tank1/PROD/WIN-Drive  secondarycache        all default tank1/PROD/WIN-Drive  usedbysnapshots       48.8G - tank1/PROD/WIN-Drive  usedbydataset         6.96T - tank1/PROD/WIN-Drive  usedbychildren        0 - tank1/PROD/WIN-Drive  usedbyrefreservation  20.6T - tank1/PROD/WIN-Drive  logbias               latency default tank1/PROD/WIN-Drive  dedup                 off default tank1/PROD/WIN-Drive  mlslabel - tank1/PROD/WIN-Drive  sync                  standard default tank1/PROD/WIN-Drive  refcompressratio      1.00x - tank1/PROD/WIN-Drive  written               18.0G - tank1/PROD/WIN-Drive  logicalused           6.96T - tank1/PROD/WIN-Drive  logicalreferenced     6.91T - tank1/PROD/WIN-Drive  volmode               default default tank1/PROD/WIN-Drive  snapshot_limit        none default tank1/PROD/WIN-Drive  snapshot_count        none default tank1/PROD/WIN-Drive  redundant_metadata    all default Thanks in advance for your help Cheers Marek -- Marek Salwerowicz From owner-freebsd-fs@freebsd.org Fri Dec 1 19:48:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D95D7E68E3D for ; Fri, 1 Dec 2017 19:48:03 +0000 (UTC) (envelope-from karli@inparadise.se) Received: from mail.inparadise.se (h-246-50.A444.priv.bahnhof.se [155.4.246.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B4C96E552 for ; Fri, 1 Dec 2017 19:48:02 +0000 (UTC) (envelope-from karli@inparadise.se) Received: from localhost (localhost [127.0.0.1]) by mail.inparadise.se (Postfix) with ESMTP id 283B34216B; Fri, 1 Dec 2017 20:39:44 +0100 (CET) Received: from mail.inparadise.se ([127.0.0.1]) by localhost (mail.inparadise.se [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id HJ37-LYRAU9z; Fri, 1 Dec 2017 20:39:43 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by mail.inparadise.se (Postfix) with ESMTP id EBFC54216E; Fri, 1 Dec 2017 20:39:42 +0100 (CET) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.inparadise.se EBFC54216E X-Virus-Scanned: amavisd-new at inparadise.se Received: from mail.inparadise.se ([127.0.0.1]) by localhost (mail.inparadise.se [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id zcKlSa8OwUKr; Fri, 1 Dec 2017 20:39:42 +0100 (CET) Received: from mail.inparadise.se (localhost [127.0.0.1]) by mail.inparadise.se (Postfix) with ESMTP id A27384216B; Fri, 1 Dec 2017 20:39:42 +0100 (CET) Date: Fri, 1 Dec 2017 20:39:42 +0100 (CET) Subject: Re: Reducing the ZVOL refreservation size? Message-ID: <576ca8c6-6cc4-4a2b-bf04-5fb2a1433195@email.android.com> X-Android-Message-ID: <576ca8c6-6cc4-4a2b-bf04-5fb2a1433195@email.android.com> To: Marek Salwerowicz Cc: freebsd-fs@freebsd.org Importance: Normal X-Priority: 3 X-MSMail-Priority: Normal From: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= X-Originating-IP: [172.16.1.122, 127.0.0.1] X-Mailer: Zimbra 8.7.11_GA_1854 (Android-Mail/7.11.5.176568039.release(...883836) devip=172.16.1.122 ZPZB/66) Thread-Index: zWGEPjkQ3lOLWjQTFtEalWCD6hINSQ== Thread-Topic: Reducing the ZVOL refreservation size? MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 19:48:03 -0000 From owner-freebsd-fs@freebsd.org Fri Dec 1 22:33:28 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0EE23DB9135; Fri, 1 Dec 2017 22:33:28 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0069.outbound.protection.outlook.com [104.47.38.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 92E8C74742; Fri, 1 Dec 2017 22:33:27 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM (52.132.46.161) by YTOPR0101MB2169.CANPRD01.PROD.OUTLOOK.COM (52.132.46.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.282.5; Fri, 1 Dec 2017 22:33:25 +0000 Received: from YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM ([fe80::f072:85d3:769:e6f8]) by YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM ([fe80::f072:85d3:769:e6f8%13]) with mapi id 15.20.0282.007; Fri, 1 Dec 2017 22:33:25 +0000 From: Rick Macklem To: Emmanuel Vadot CC: Konstantin Belousov , FreeBSD Current , "freebsd-fs@freebsd.org" , John Baldwin Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Topic: Switch vfs.nfsd.issue_delegations to TUNABLE ? Thread-Index: AQHTaEyAfy6q2cCLHkW9s1ZyeaxyRaMpzDeAgAAEfyqAAB1rAIAANPobgABJSsyAAAEe3oAA8S4AgAA7/XuAA3xyQg== Date: Fri, 1 Dec 2017 22:33:25 +0000 Message-ID: References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> <20171128164132.290bf07218b16164db613242@bidouilliste.com> , <20171129144040.29a10234aa5fb1cfa2611bfc@bidouilliste.com>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB2169; 6:lWyj7vkBUH3fXtWJ2TTA2A0tMPfqDRnVXsfaCVLmLmanW8Po/nuQNmLbKqkwa4rUIfF5/rYA2UoIa/cyEEWuhIvmfU2xtBKJuYYK4vIoZhVovuvsnHsSHY5AhITwixcrlWVmDY6Kqi/+DjcdtMXcTFrLN0zVbzwTJWBgxgi9jLZa8p/jldD2wBhi17TBrotJizNbTfYUk9lxsE3BVNbbJpe6e1/3mOy+z7xdSE0vazsjWuVm/ixxvLiSCI5b/ONau6X8lVRl71f6iEXqIa4bFtndnYIvZbMtIgNAftOWwudMCWfvlJt+aQCae30HFmCOU+Vk60YRkkuOpQ4zcViDjEtdWhCCJmjAOjBU9CrCZnE=; 5:F+x51kXHcY9xV3glzJiRiR6KSteVKuhN0rUaz5d90j6/qgYVgKHjEIrQA6pJSzD4aBBzdtftbebxoSY+KN+Wt3d8Oh5hk9NAMXvEDBjjw8mAup7hPCq+oxxmoyllF15cfDUGqO0c/c2bJSRtF1rKWbLhNJ9GV0dcRRlo+0nQDFA=; 24:+QpjuEx4ikiHzJRyWDIDrrbyjCP8+nMc/vN2qkFHf/iIqhF/xxDQCvfgdQtOXGa9R6LfVPcs9EIwGx1o7oiuyIEV2/+hpiwTQ3rwZg1mK/I=; 7:i5hurOlsHVjf2Uc539srOChuTnYNE33YZQcxN6HoItnjcQw17E7SEqDA0nne+RJbvWwzjQ5ZlzYfnFAaO2gX5mGoZwWC4f3GdoT+4bs1BhqdOVGp0K4pXr8EJxmFZQX9Hj1Vh61+JAQIMgkGW0Jjiwbi2X6y20ih9sOgZJ2oDDaoQQTzvSNCDVlvhZwjMhqHVKc3zksAqFByY1yZfADs8J7GU2Wc/jDfQutFK4QeWSZG8Ooyk719ok7OYgu9encd x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 85be4969-18e4-4eb5-c965-08d5390b8933 x-microsoft-antispam: UriScan:(18154293887054); BCL:0; PCL:0; RULEID:(4534020)(4602075)(4627115)(8989060)(201703031133081)(201702281549075)(8990040)(5600026)(4604075)(2017052603286); SRVR:YTOPR0101MB2169; x-ms-traffictypediagnostic: YTOPR0101MB2169: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(75325880899374)(18154293887054); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(8121501046)(5005006)(3231022)(3002001)(10201501046)(93006095)(93001095)(6041248)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123562025)(20161123555025)(20161123560025)(20161123564025)(6072148)(201708071742011); SRVR:YTOPR0101MB2169; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:YTOPR0101MB2169; x-forefront-prvs: 05087F0C24 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(366004)(346002)(376002)(24454002)(189002)(199003)(102836003)(2906002)(81156014)(39060400002)(99286004)(14454004)(86362001)(101416001)(53546010)(4326008)(478600001)(229853002)(6916009)(8676002)(68736007)(33656002)(106356001)(2950100002)(54356011)(3280700002)(97736004)(81166006)(7696005)(25786009)(105586002)(53936002)(9686003)(6246003)(55016002)(54906003)(316002)(74316002)(786003)(3660700001)(6506006)(5250100002)(74482002)(2900100001)(189998001)(93886005)(305945005)(5660300001)(8936002)(6306002)(76176011)(6436002)(966005); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB2169; H:YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 85be4969-18e4-4eb5-c965-08d5390b8933 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Dec 2017 22:33:25.0839 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB2169 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 22:33:28 -0000 I have created D13327 on reviews.freebsd.org with a patch that keeps track of # of issued write delegations and allows nfsrv_checkgetattr() to return without acquiring the lock when it is 0. Hopefully kib@, jhb@ or someone who is familiar with the use of the atomic ops in the kernel can review it. (Where the counter code goes should be fine, but I am not sure I got the use of the atomic ops correct.) Hopefully Emmanuel can test the patch to see if it fixes his performance problem. rick ________________________________________ From: owner-freebsd-current@freebsd.org = on behalf of Rick Macklem Sent: Wednesday, November 29, 2017 12:28:05 PM To: Emmanuel Vadot Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Emmanuel Vadot wrote: [stuff snipped] > I haven't test by I can say that it will work, I actually wondered at >first doing that. The problem with this patch is what I tried to >describe in my first and following mails, since you can turn on and off >delegation you can still have delegation (so nfsrv_delegatecnt > 0) >even if the sysctl is =3D=3D 0. That's why I went to the tunable way, seem= s >cleaner to me as disabling delegation doesn't really disable them for >current client. Yes, if you have delegations enabled and then disable them, there will be delegations issued for a "while". During that time, the code in nfsrv_checkgetattr() does need to check for them. Since no new delegations will be issued, the outstanding ones will go away when the client chooses to return them. (You can force this on the client by doing a dismount/mount, at least for the FreeBSD client.) Alternately, rebooting the server forces the clients to "recover" delegatio= ns and, if they are disabled, that fails. (Actually the recover succeeds, but = with a flag set that tells the client it must return them asap.) All the tunable does is make it impossible to disable them without rebooting, but otherwise the effect is the same. I have a patch that keeps a separate counter for write delegations (which are the ones you care about) using atomics to maintain the value. (That will be similar but somewhat better than what this patch does.= ) rick [lots more snipped] _______________________________________________ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Sat Dec 2 16:30:39 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9DF33E5E9A0 for ; Sat, 2 Dec 2017 16:30:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8BED0758F5 for ; Sat, 2 Dec 2017 16:30:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vB2GUd3e066248 for ; Sat, 2 Dec 2017 16:30:39 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 224037] Kernel crashes when trying to mount certain USB keys reported as WriteProtected Date: Sat, 02 Dec 2017 16:30:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 16:30:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D224037 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org CC| |cem@freebsd.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Dec 2 16:32:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D2AAE5EC20 for ; Sat, 2 Dec 2017 16:32:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B65C75BF2 for ; Sat, 2 Dec 2017 16:32:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vB2GW5JG073318 for ; Sat, 2 Dec 2017 16:32:05 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 224037] Kernel crashes when trying to mount certain USB keys reported as WriteProtected Date: Sat, 02 Dec 2017 16:32:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 16:32:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D224037 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-fs@FreeBSD.org |freebsd-scsi@FreeBSD.org --- Comment #2 from Conrad Meyer --- Hm, this is probably below FS -- CAM. --=20 You are receiving this mail because: You are the assignee for the bug.=