From owner-freebsd-bugs@FreeBSD.ORG Tue Nov 25 13:30:02 2008 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 632C81065678 for ; Tue, 25 Nov 2008 13:30:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 4308D8FC1F for ; Tue, 25 Nov 2008 13:30:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id mAPDU2wi058345 for ; Tue, 25 Nov 2008 13:30:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id mAPDU2C5058344; Tue, 25 Nov 2008 13:30:02 GMT (envelope-from gnats) Resent-Date: Tue, 25 Nov 2008 13:30:02 GMT Resent-Message-Id: <200811251330.mAPDU2C5058344@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Weldon Godfrey Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4934D106564A for ; Tue, 25 Nov 2008 13:23:10 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 3B3688FC1F for ; Tue, 25 Nov 2008 13:23:10 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id mAPDN93i013418 for ; Tue, 25 Nov 2008 13:23:09 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id mAPDN9me013412; Tue, 25 Nov 2008 13:23:09 GMT (envelope-from nobody) Message-Id: <200811251323.mAPDN9me013412@www.freebsd.org> Date: Tue, 25 Nov 2008 13:23:09 GMT From: Weldon Godfrey To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: misc/129174: [nfs][zfs][panic] NFS v3 Panic when under high load exporting ZFS file system X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Nov 2008 13:30:02 -0000 >Number: 129174 >Category: misc >Synopsis: [nfs][zfs][panic] NFS v3 Panic when under high load exporting ZFS file system >Confidential: no >Severity: critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Nov 25 13:30:01 UTC 2008 >Closed-Date: >Last-Modified: >Originator: Weldon Godfrey >Release: FreeBSD 7.1-PREPREASE (also happens under 8-CURRENT and 7.x) >Organization: ENA >Environment: FreeBSD store1.mail.ena.net 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #2: Thu Nov 20 10:41:36 CST 2008 root@store1.mail.ena.net:/usr/obj/usr/src/sys/GENERIC amd64 >Description: This issue was origionally reported under another ticket that turned out to be unrelated. I am using 2 IBM EXP 3000 SAS Chasis with a total of 24 300GB 25K SAS drives using a 3ware 9690SE-8 controller on a Dell Poweredge 2950-iii with 2x4core Intel 2.5GHz CPU and 16GB of RAM. I am testing by running 9 instances of Postmark from Centos 5.2 x86_64 (was 3.x i386) NFS clients. I can run 1 on 9 machines or all on 1 machine, after about 2-4 hrs of running the benchmark, the system panics and reboots. From help with the freebsd-fs group, it was determined that the vnode is becoming invalid. I have tried several things including using FreeBSD 8-CURRENT and the ZFS v11 patch but nothing helps except if I force to use NFS v2, so it appears this is a bug with the v3 code. This is my loader.conf, the commented out section are values I have tried but do not have an effect: vm.kmem_size_max="16106127360" vm.kmem_size="1073741824" kern.maxvnodes="800000" #vfs.zfs.debug="1" #vfs.zfs.zil_disable="1" #vfs.zfs.prefetch_disable="1" #vfs.zfs.vdev.cache.bshift="13" the raid is configured RAID 10, each chasis is one side of the mirror (the 1st 11 drives of each chasis, the final one of each is marked as spare). >How-To-Repeat: run multiple instances of postmark benchmark using NFS v3 from NFS clients for several hours. >Fix: workaround: don't use NFS v3, use -2 option with mountd. >Release-Note: >Audit-Trail: >Unformatted: