From owner-freebsd-current@freebsd.org Thu Nov 24 05:17:27 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 78299C52220 for ; Thu, 24 Nov 2016 05:17:27 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-qt0-x22e.google.com (mail-qt0-x22e.google.com [IPv6:2607:f8b0:400d:c0d::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3391AE7B for ; Thu, 24 Nov 2016 05:17:27 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-qt0-x22e.google.com with SMTP id c47so31082198qtc.2 for ; Wed, 23 Nov 2016 21:17:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to; bh=gCd5Xzm28093aMea/K/rXFPY0FWvsniSS0KOj2gtDsc=; b=elXVQp1Qa2mnACI9+wQ2HLM6D96WpynYURLaKNl+G7ekmbsfd7CVMioUsXCVgc/P4v GSC4Ca0KZzfB/FbXSrv2FCG1j3S1DA/Rqa8rhGm55VimGl0knk8BeqXvSrloCoLnIqj3 00+PU1iRYBQpqN6/dAdbQOgjlY2mARhsmtnnJTw87WYbGYcvqsEwv2MpXRQU2WYGTQrx FUZEfD/ZUAMpt0kk9hIeMttq/rgAc50vYbduHfjBFF6Af2yZamlmyeHwUYURAWl2PPip 2x7bTIMj2ZO8rgNVtJfkFlu6FHv1R4W4xVDVTOcMySmeA3EhvQVJfxGC7QPzfXXgz/8J bVUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=gCd5Xzm28093aMea/K/rXFPY0FWvsniSS0KOj2gtDsc=; b=Ztr78UB0KpEAiO2FMrKew9zhBY65eABjwgyELaUIcklSmVRNL3GdywWd9gI4z35MRR 8YmcBpg0jlUW+8Fhyun5UYRK0kPlXnFzy9EehfFYZWgAdyaQS0SLxVrFzYd3Tx8CZ5JM lnmxgQULdCLFy8C+KPsCE0YtBMTUJwyAt2O2aEBKVmOoLfKMrWD07RsvQRUjjXVyfFFI U86uHD55oWtmnXRVNnEDPwzSEu0OGnj9yEBUySntUToFqVts7dia2AK+hkBbyaJMPdWK QZYu+OzBFCw9kndKb8kVWP4PfY++yyXXkWI9oLWq5RGDznJdbZP/lC6vrwgmcciHlXBQ 9v9w== X-Gm-Message-State: AKaTC03mYgeMxFZk+I6QNgRFdDFGnYdd2WWbUA7sbOsZSUq49g141cQ5v0YaboZQMVmvb7ietz/IEeUoIJcQZg== X-Received: by 10.200.48.28 with SMTP id f28mr428224qte.247.1479964646258; Wed, 23 Nov 2016 21:17:26 -0800 (PST) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.12.166.129 with HTTP; Wed, 23 Nov 2016 21:17:25 -0800 (PST) From: Alan Somers Date: Wed, 23 Nov 2016 22:17:25 -0700 X-Google-Sender-Auth: dMyK9cP2ZidTlDMhvn-_UqVa7lA Message-ID: Subject: NFSv4 performance degradation with 12.0-CURRENT client To: FreeBSD CURRENT Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2016 05:17:27 -0000 I have a FreeBSD 10.3-RELEASE-p12 server exporting its home directories over both NFSv3 and NFSv4. I have a TrueOS client (based on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October) mounting the home directories over NFSv4. At first, everything is fine and performance is good. But if the client does a buildworld using sources on NFS and locally stored objects, performance slowly degrades. The degradation is most noticeable with metadata-heavy operations. For example, "ls -l" in a directory with 153 files takes less than 0.1 seconds right after booting. But the longer the buildworld goes on, the slower it gets. Eventually that same "ls -l" takes 19 seconds. When the home directories are mounted over NFSv3 instead, I see no degradation. top shows negligible CPU consumption on the server, and very high consumption on the client when using NFSv4 (nearly 100%). The NFS-using process is spending almost all of its time in system mode, and dtrace shows that almost all of its time is spent in ncl_getpages(). I have delegations disabled on the server. On the client, the home directories are nullfs mounted to two additional locations, and the buildworld was actually using one of those nullfs mounts, not the NFS mount directly. Any ideas? -Alan