From owner-freebsd-current@FreeBSD.ORG Wed Nov 12 23:59:04 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E6511065673 for ; Wed, 12 Nov 2008 23:59:04 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.208.78.105]) by mx1.freebsd.org (Postfix) with ESMTP id E31F08FC1C for ; Wed, 12 Nov 2008 23:59:03 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.3/8.14.3) with ESMTP id mACNx31Y019962 for ; Wed, 12 Nov 2008 15:59:03 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.3/8.14.3/Submit) id mACNx3Va019961 for freebsd-current@freebsd.org; Wed, 12 Nov 2008 15:59:03 -0800 (PST) (envelope-from sgk) Date: Wed, 12 Nov 2008 15:59:03 -0800 From: Steve Kargl To: freebsd-current@freebsd.org Message-ID: <20081112235903.GA19865@troutmask.apl.washington.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Subject: GEOM hangover disables NFS X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Nov 2008 23:59:04 -0000 On three nodes in my cluster (nodes n17, n18, and n19), I had GEOM use /dev/ad4s1e for tests with gmirror and ggated/ggatec. I found that GEOM was insufficient for my needs and decided to return the 3 partitions to NFS-exported partitions. It seems that once GEOM touches a partition, the partition can no longer be used by NFS. I'll illustrute the problem with n17:/dev/ad4s1e. In what follows, n10 is the master node. Both n10 and n17 have brand new worlds and kernels from about 45 minutes ago. n10:kargl[203] ssh n17 n17:kargl[201] df Filesystem 1M-blocks Used Avail Capacity Mounted on /dev/ad4s1a 247 104 123 46% / devfs 0 0 0 100% /dev /dev/ad4s1e 222780 0 204958 0% /data /dev/ad4s1d 3962 182 3463 5% /usr n10:/home 193947 92855 85576 52% /home n10:/usr/local 19832 10494 7750 58% /usr/local n17:kargl[202] tail -1 /etc/exports /data -alldirs node10 node21 The above is after a 'newfs -U /dev/ad4s1e' and a reboot. n10:root[244] ls / | grep -E ^n n11/ n12/ n13/ n14/ n15/ n16/ n17/ n18/ n19/ n20/ n21/ n10:root[245] mount_nfs -o tcp n17:/data /n17 n10:root[246] mount -v | grep n17 n17:/data on /n17 (nfs, fsid 0eff000303000000) n10:root[247] ls /n17 ls: /n17: Input/output error n10:root[248] ls / | grep -E ^n ls: n17: Input/output error n11/ n12/ n13/ n14/ n15/ n16/ n18/ n19/ n20/ n21/ n10:root[251] umount /n17 n10:root[252] ls / | grep -E ^n n11/ n12/ n13/ n14/ n15/ n16/ n17/ n18/ n19/ n20/ n21/ So, how does one exorcise GEOM from /dev/ad4s1e? -- Steve