From owner-freebsd-stable@freebsd.org Fri Aug 5 13:22:42 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C0A74BAF0F8 for ; Fri, 5 Aug 2016 13:22:42 +0000 (UTC) (envelope-from daniel.genis@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1EEE516CF for ; Fri, 5 Aug 2016 13:22:41 +0000 (UTC) (envelope-from daniel.genis@gmx.de) Received: from [192.168.101.17] ([37.74.194.90]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0M3NEK-1bDzjz3EyO-00qzOe for ; Fri, 05 Aug 2016 15:22:39 +0200 From: Daniel Genis Subject: zfs recv causes nfs server to throw NFSERR_IO i/o errors To: freebsd-stable@freebsd.org Message-ID: <1df33129-0c3e-dfc3-6867-46ab0473ae57@gmx.de> Date: Fri, 5 Aug 2016 15:22:39 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:d3BSGLWYlr0cJuEd6G4qE0qNUK/Y98UBc6ljkW1ah2i+NS1zrh5 IlTBHpV2h71CYU7OQNeCk6cak7ELFurPtwxnQcruFTdrlH3rPFrnl9UwNnV8Lr0EdrwsEnt oJIv6dcvcs7wxS6slyY/q9MiKMj9x6XKW1p1j+m6OjruISxzZsuGtaf898PD6Hg8jdm8e5g Gl9ZxaZDuU7gNNDoYf2AA== X-UI-Out-Filterresults: notjunk:1;V01:K0:RS4hpvTkmeA=:RFVD9Sl9gkheh5qemnNqEH 1LcLL8Zw7zRVLZfEiGZC8JlfGVRL5P84XK4zR4DJQLQrPJMWrNRMEu0Jn9N4836H93s21yQwj TJ22q7AD9GmXpRpJVoRjWJWGDctSSrFZauQt5fXTEkQTUjCBEFxVZFu0vzzNkiu9ZCxUam5f7 Wkjn8mhLL6VvZ6EMkpf5syNnSo5XdytHLe18/Kd0zu+HyvPJKY04yi9yFnkIojfZXi33lHPgH qU73q81ZbrlBRLc9j0qSJo3MuKS5lYegLflabYUfXGLBdJix+XKdA6PYrFJF+3IXfBYEH9Kl8 JS/Ufu4W3lWWX1UumFnF4PaMDqB0lXlS4+uTiwMr5X+G+SpOdLehIHM78iQ4bEg+BsMgKuFkT WIQpKIeLmiz9+Q+ZdYNcS/LnnHB3lpGV11rWpBQScSDI4Ny+vFo+wJQ861F2cQicdDLTws8SB 6A3+AAQlrxfl0jFXmiQaqKtBKwCv9n3NlmqG92zhTz9bD+/UUQhx2FDG8q57hxB5an1vYQ/iw SSjsWq/XZM4SfJ3zlpSJ/cvtwsEMhpPJH1MZm1GUqa91OpQ1ETMfJ9iYkliIjz0dzqHUjMSbH 1uGNKOoXsBZo8FxO6rZF3iBZ83WOBgINY5PoQw5ff2EKYsBR4vMP8QjnifRnN2ugt1qU+OvXh EJc/cvbicWLeqMj81n7NQs83qF1LvRVWwtm6xYtaLhHuKVHXEDCPxuTJKjnTTCIfVGzF44ZTz KMvt3epZmz/SnjWr/lmVN2Wl9ZBVpEuf9tiK5bIFg9zhmWw3O/md/Af2E39YD9jfwfLNTZQe0 4tyL/yA X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Aug 2016 13:22:42 -0000 Hi everyone, we've been tracing an issue where snapshot replication is causing interruptions for the NFS serivce. The problem is as follows: Every time a zfs recv finishes, there is a chance for the NFS server to return an NFSERR_IO for a GETATTR call. This shows up as input/output errors on the nfs clients. Here the tcpdump showing the NFS conversation: https://nopaste.me/view/95d1a79d NFS 202 V3 GETATTR Call (Reply In 6043), FH: 0x8c711a60 NFS 98 V3 GETATTR Reply (Call In 6042) Error: NFS3ERR_IO NFS 222 V3 LOOKUP Call (Reply In 6046), DH: 0x6694634f/example.file.txt NFS 102 V3 LOOKUP Reply (Call In 6045) Error: NFS3ERR_ACCES We've been able to verify that there is a _direct_ correlation between the zfs recv command and these NFS errors. For every input/output error we can find a log entry of a replication just finishing (zfs recv exiting). The receiving server is running 10.3-RELEASE I've read about a VFS/ZFS deadlock issue which is to be included/fixed in Freebsd 11.0-BETA4. Could our issue be related? Otherwise does anyone have any suggestions how to tackle this issue? For the record, say we have two volumes: tank/volumeA and tank/volumeB If there is a zfs recv busy for tank/volumeA then tank/volumeB can get these NFS "io" errors, it does not have to be the same volume. Has anyone else seen/experience this as well? Any insights are appreciated! With kind regards, Daniel