From owner-freebsd-fs@freebsd.org Sun Oct 25 02:54:06 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8D6EA18674 for ; Sun, 25 Oct 2015 02:54:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A51D312BA for ; Sun, 25 Oct 2015 02:54:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9P2s6Kj025338 for ; Sun, 25 Oct 2015 02:54:06 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203185] mountd is exposing parent directories [on zfs mount?] that it should not be Date: Sun, 25 Oct 2015 02:54:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Oct 2015 02:54:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203185 Garrett Cooper,425-314-3911 changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |Works As Intended Status|In Progress |Closed --- Comment #3 from Garrett Cooper,425-314-3911 --- I've been using FreeBSD for over a decade, but missed this nuance.. I'm not sure that it's the right behavior, but it's documented at least. Closing. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Sun Oct 25 10:18:31 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CFE9B859E for ; Sun, 25 Oct 2015 10:18:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B71D51594 for ; Sun, 25 Oct 2015 10:18:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9PAIVVE063188 for ; Sun, 25 Oct 2015 10:18:31 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203330] ZFS free space bug or HDD driver bug Date: Sun, 25 Oct 2015 10:18:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pekka.jarvinen@gmail.com X-Bugzilla-Status: Closed X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Oct 2015 10:18:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203330 Pekka J=C3=A4rvinen changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|New |Closed --- Comment #3 from Pekka J=C3=A4rvinen --- I can't try current / head kernel + world for now. It seems to be fixed somewhere between r287260M and r287929 AFAIK. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Oct 25 21:00:19 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0B4178779 for ; Sun, 25 Oct 2015 21:00:19 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F1E731B5D for ; Sun, 25 Oct 2015 21:00:18 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9PL0IGt055638 for ; Sun, 25 Oct 2015 21:00:18 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201510252100.t9PL0IGt055638@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 25 Oct 2015 21:00:18 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Oct 2015 21:00:19 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 4 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Sun Oct 25 22:12:57 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 00C588970 for ; Sun, 25 Oct 2015 22:12:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E13E51F5D for ; Sun, 25 Oct 2015 22:12:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9PMCuRf066233 for ; Sun, 25 Oct 2015 22:12:56 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 131341] makefs: error "Bad file descriptor" on the mount point of md-presentation makefs image Date: Sun, 25 Oct 2015 22:12:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 7.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Oct 2015 22:12:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=131341 Garrett Cooper,425-314-3911 changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-fs@FreeBSD.org |ngie@FreeBSD.org --- Comment #4 from Garrett Cooper,425-314-3911 --- Bulk taking makefs bugs. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Mon Oct 26 15:18:36 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13FB6A1D2A5 for ; Mon, 26 Oct 2015 15:18:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F03111CC9 for ; Mon, 26 Oct 2015 15:18:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9QFIZBQ090396 for ; Mon, 26 Oct 2015 15:18:35 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204037] Poudriere panics zfs with 'solaris assert: zrl->zr_refcount == 0 (0x6 == 0x0)' Date: Mon, 26 Oct 2015 15:18:36 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: Andrew@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Oct 2015 15:18:36 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204037 Andrew Turner changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Tue Oct 27 20:03:25 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CA3FFA1FFA9 for ; Tue, 27 Oct 2015 20:03:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B64AD1F96 for ; Tue, 27 Oct 2015 20:03:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9RK3P1I030679 for ; Tue, 27 Oct 2015 20:03:25 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 177971] [nfs] FreeBSD 9.1 nfs client dirlist problem w/ nfsv3,rsize=4096,wsize=4096 Date: Tue, 27 Oct 2015 20:03:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: krichy@cflinux.hu X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Oct 2015 20:03:25 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=177971 krichy@cflinux.hu changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |krichy@cflinux.hu --- Comment #2 from krichy@cflinux.hu --- This still exists in releng/10.2. Any solutions? -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Wed Oct 28 02:10:52 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B581AA1F4CD for ; Wed, 28 Oct 2015 02:10:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A247312E7 for ; Wed, 28 Oct 2015 02:10:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9S2AqbS051388 for ; Wed, 28 Oct 2015 02:10:52 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204055] [patch] zfs(8) should not yet list sha512, skein and edonr as supported hash algorithms Date: Wed, 28 Oct 2015 02:10:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Oct 2015 02:10:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204055 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Wed Oct 28 12:20:04 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A4C6A209C1 for ; Wed, 28 Oct 2015 12:20:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 26D531225 for ; Wed, 28 Oct 2015 12:20:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9SCK4rt038472 for ; Wed, 28 Oct 2015 12:20:04 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 177971] [nfs] FreeBSD 9.1 nfs client dirlist problem w/ nfsv3,rsize=4096,wsize=4096 Date: Wed, 28 Oct 2015 12:20:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: krichy@cflinux.hu X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Oct 2015 12:20:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=177971 --- Comment #3 from krichy@cflinux.hu --- I've noticed difference between mounting with wsize and without wsize: # mount -o rsize=2048 10.0.0.2:/home/kh /home/kh/ root@freebsd9:~ # ls -1 /home/kh/fenykepek/20110429 | sort | uniq | wc -l 309 # umount /home/kh/ # mount -o rsize=2048,wsize=2048 10.0.0.2:/home/kh /home/kh/ # ls -1 /home/kh/fenykepek/20110429 | sort | uniq | wc -l 225 Dont know if this helps. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Wed Oct 28 13:46:54 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6697A1F510 for ; Wed, 28 Oct 2015 13:46:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 93C721E0E for ; Wed, 28 Oct 2015 13:46:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9SDkspY082905 for ; Wed, 28 Oct 2015 13:46:54 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203906] ZFS lockup, spa_namespace_lock Date: Wed, 28 Oct 2015 13:46:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: fk@fabiankeil.de X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Oct 2015 13:46:54 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203906 fk@fabiankeil.de changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fk@fabiankeil.de --- Comment #1 from fk@fabiankeil.de --- Can you please attach the complete "procstat -kk -a" output? I suspect that this is the same issue as: https://lists.freebsd.org/pipermail/freebsd-current/2014-September/052018.html Analysis continues at: https://lists.freebsd.org/pipermail/freebsd-current/2015-October/057982.html If that's the case and if your applications can handle it, changing the pool's failmode from "wait" to "continue" may be worth a try. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Wed Oct 28 15:44:30 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A09AA20FEE for ; Wed, 28 Oct 2015 15:44:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8657C18CE for ; Wed, 28 Oct 2015 15:44:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9SFiU80062239 for ; Wed, 28 Oct 2015 15:44:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204093] [zfs] Overflowing the max path for ZFS snapshot automount leaves the zfs in an busy state Date: Wed, 28 Oct 2015 15:44:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Oct 2015 15:44:30 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204093 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Wed Oct 28 17:34:04 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0842CA1EDA6 for ; Wed, 28 Oct 2015 17:34:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E872D1967 for ; Wed, 28 Oct 2015 17:34:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9SHY37J089729 for ; Wed, 28 Oct 2015 17:34:03 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203906] ZFS lockup, spa_namespace_lock Date: Wed, 28 Oct 2015 17:34:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: uwyo.apalmer@gmail.com X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Oct 2015 17:34:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203906 --- Comment #2 from Adrian Palmer --- $ procstat -kk -a PID TID COMM TDNAME KSTACK 1252 100078 csh - mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_sigsuspend+0xf4 sys_sigsuspend+0x31 amd64_syscall+0x351 Xfast_syscall+0xfb 38986 100168 zpool - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d spa_all_configs+0x6e zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb 41360 101772 csh - mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_sigsuspend+0xf4 sys_sigsuspend+0x31 amd64_syscall+0x351 Xfast_syscall+0xfb 41661 100159 gdb - mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_wait6+0x5f4 sys_wait4+0x72 amd64_syscall+0x351 Xfast_syscall+0xfb 41662 101773 zpool - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d spa_all_configs+0x6e zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb 42143 100174 zpool - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d spa_all_configs+0x6e zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb 42242 101757 zfs - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d spa_all_configs+0x6e zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb 45485 101761 csh - mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_sigsuspend+0xf4 sys_sigsuspend+0x31 amd64_syscall+0x351 Xfast_syscall+0xfb 45553 100079 procstat - 51117 101760 ls - mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d dbuf_read+0x45b dmu_buf_hold+0x3d zap_lockdir+0x46 zap_cursor_retrieve+0x131 zfs_freebsd_readdir+0x3e1 VOP_READDIR_APV+0xa7 kern_getdirentries+0x21c sys_getdirentries+0x28 amd64_syscall+0x351 Xfast_syscall+0xfb 65149 101762 csh - mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a tty_wait+0x1c ttydisc_read+0x2d4 ttydev_read+0x86 devfs_read_f+0xeb dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x351 Xfast_syscall+0xfb 65496 101767 ls - mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d dbuf_read+0x45b dmu_buf_hold+0x3d zap_lockdir+0x46 zap_cursor_retrieve+0x131 zfs_freebsd_readdir+0x3e1 VOP_READDIR_APV+0xa7 kern_getdirentries+0x21c sys_getdirentries+0x28 amd64_syscall+0x351 Xfast_syscall+0xfb 65624 101755 zfs - mi_switch+0xe1 sleepq_wait+0x3a _sx_xlock_hard+0x48a _sx_xlock+0x5d spa_all_configs+0x6e zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb There isn't much running on the machine at this point. I'm waiting for ZFS to be a bit more stable. I'll reconfigure the zpool failmode when I reboot. It may take a while for the problem to replicate. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Thu Oct 29 00:32:14 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9A18A1F6FB for ; Thu, 29 Oct 2015 00:32:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ADADD1235 for ; Thu, 29 Oct 2015 00:32:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9T0WEYN042306 for ; Thu, 29 Oct 2015 00:32:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 177971] [nfs] FreeBSD 9.1 nfs client dirlist problem w/ nfsv3,rsize=4096,wsize=4096 Date: Thu, 29 Oct 2015 00:32:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: rmacklem@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 00:32:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=177971 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rmacklem@FreeBSD.org Assignee|freebsd-fs@FreeBSD.org |rmacklem@FreeBSD.org --- Comment #4 from Rick Macklem --- Created attachment 162553 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=162553&action=edit patch to fix this problem I think this patch will fix the problem. When f_iosize which sets bo_bsize is less than the size of a buffer cache buffer problems occur. NFS_DIRBLKSIZ (8K) is the size of the buffer cache blocks used for directories and without this patch, rsize=4096 would set f_iosize->bo_bsize to 4096. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Thu Oct 29 02:20:14 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD4C0A20D92 for ; Thu, 29 Oct 2015 02:20:14 +0000 (UTC) (envelope-from mwlucas@mail.michaelwlucas.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 9340D1DAD for ; Thu, 29 Oct 2015 02:20:14 +0000 (UTC) (envelope-from mwlucas@mail.michaelwlucas.com) Received: by mailman.ysv.freebsd.org (Postfix) id 90537A20D90; Thu, 29 Oct 2015 02:20:14 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 774A4A20D8E for ; Thu, 29 Oct 2015 02:20:14 +0000 (UTC) (envelope-from mwlucas@mail.michaelwlucas.com) Received: from mail.michaelwlucas.com (mail.michaelwlucas.com [104.236.197.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 05B391DAA for ; Thu, 29 Oct 2015 02:20:13 +0000 (UTC) (envelope-from mwlucas@mail.michaelwlucas.com) Received: from mail.michaelwlucas.com (localhost [127.0.0.1]) by mail.michaelwlucas.com (8.14.9/8.14.7) with ESMTP id t9T1vMik095128 for ; Wed, 28 Oct 2015 21:57:22 -0400 (EDT) (envelope-from mwlucas@mail.michaelwlucas.com) Received: (from mwlucas@localhost) by mail.michaelwlucas.com (8.14.9/8.14.7/Submit) id t9T1vMqU095127 for fs@freebsd.org; Wed, 28 Oct 2015 21:57:22 -0400 (EDT) (envelope-from mwlucas) Date: Wed, 28 Oct 2015 21:57:21 -0400 From: "Michael W. Lucas" To: fs@freebsd.org Subject: iSCSI/ZFS strangeness Message-ID: <20151029015721.GA95057@mail.michaelwlucas.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail.michaelwlucas.com X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mail.michaelwlucas.com [127.0.0.1]); Wed, 28 Oct 2015 21:57:27 -0400 (EDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 02:20:14 -0000 Hi, I'm experimenting with iSCSI HA with FreeBSD 10.2 amd64. I know people do this sort of thing, but I can't figure out what I'm missing. (Most of the tutorials cover HAST instead). I suspect the real problem is "Lucas doesn't know the right search terms." The goal is to make an iSCSI-based ZFS pool that's available to two separate hosts, and remains available even if one of the iSCSI servers fails. Instead, the pool hangs when either of the iSCSI servers goes down. My test environment has two iSCSI servers, iscsi1 and iscsi2. They each export three drives as a single target. There's two iSCSI initiators, zfs1 and zfs2. Both of them have active connections to the iSCSI targets. On another host I've created a ZFS pool of striped mirrors. Each mirror has one drive from each iSCSI server. The initiators can both access the iSCSI-based pool--not simultaneously, of course. But CARP, devd, and some shell scripting should get me a highly available pool that can withstand the demise of any one iSCSI server and any one initiator. The hope is that the pool would continue to work even if an iSCSI host shuts down. When the downed iSCSI host returns, the initiators should log back in and the pool auto-resilver. Some ten minutes ago, I killed iscsi2. The pool is live on zfs1. And the drives really have disappeared. # iscsictl Target name Target portal State iqn.2013-11.io.mwl:target0 iscsi2.blackhelicopters.org Operation timed out iqn.2013-11.io.mwl:target0 iscsi1.blackhelicopters.org Connected: da2 da3 da4 I would expect to see the pool appear degraded. But instead, I have: # zpool status iscsi pool: iscsi state: ONLINE scan: resilvered 1.16G in 0h3m with 0 errors on Wed Oct 28 14:13:08 2015 config: NAME STATE READ WRITE CKSUM iscsi ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/iscsi1-0 ONLINE 0 0 0 gpt/iscsi2-0 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/iscsi1-1 ONLINE 0 0 0 gpt/iscsi2-1 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 gpt/iscsi1-2 ONLINE 0 0 0 gpt/iscsi2-2 ONLINE 0 0 0 errors: No known data errors To try to make ZFS realize the pool is degraded, I write to the iSCSI pool. (tar -xvpf ports.tar.gz) Each time, the extract gets to a certain point and hangs. Can't ^C or ^Z out of it. This latest time, the extract reaches: x ports/www/firefox-esr/files/patch-media-mtransport-third_party-nICEr-src-util-mbslen.c I can still SSH into the machine, but if I try to look in any directories under /iscsi/ports/* my terminal hangs. So I restart the downed iSCSI server. The initiators log back into the target. And the hung tar extract picks up where it left off. So, I haven't achieved HA. The pool stays up, but it's not exactly usable. Any hints on what I'm missing? Thanks, ==ml -- Michael W. Lucas - mwlucas@michaelwlucas.com, Twitter @mwlauthor http://www.MichaelWLucas.com/, http://blather.MichaelWLucas.com/ From owner-freebsd-fs@freebsd.org Thu Oct 29 02:33:21 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 530578236 for ; Thu, 29 Oct 2015 02:33:21 +0000 (UTC) (envelope-from pmurray@nevada.net.nz) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3D0881860 for ; Thu, 29 Oct 2015 02:33:21 +0000 (UTC) (envelope-from pmurray@nevada.net.nz) Received: by mailman.ysv.freebsd.org (Postfix) id 3C69D8235; Thu, 29 Oct 2015 02:33:21 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C07C8232 for ; Thu, 29 Oct 2015 02:33:21 +0000 (UTC) (envelope-from pmurray@nevada.net.nz) Received: from bellagio.open2view.net (bellagio.open2view.net [210.48.79.75]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B43B185E for ; Thu, 29 Oct 2015 02:33:20 +0000 (UTC) (envelope-from pmurray@nevada.net.nz) Received: from dhcp100.akl.open2view.lan (unknown [114.23.246.97]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: pmurray@nevada.net.nz) by bellagio.open2view.net (Postfix) with ESMTPSA id 89DBC12AA584; Thu, 29 Oct 2015 15:26:57 +1300 (NZDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) Subject: Re: iSCSI/ZFS strangeness From: Philip Murray In-Reply-To: <20151029015721.GA95057@mail.michaelwlucas.com> Date: Thu, 29 Oct 2015 15:26:57 +1300 Cc: fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20151029015721.GA95057@mail.michaelwlucas.com> To: "Michael W. Lucas" X-Mailer: Apple Mail (2.3096.5) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 02:33:21 -0000 > On 29/10/2015, at 2:57 PM, Michael W. Lucas = wrote: >=20 > Hi, >=20 > I'm experimenting with iSCSI HA with FreeBSD 10.2 amd64. I know people > do this sort of thing, but I can't figure out what I'm missing. (Most > of the tutorials cover HAST instead). I suspect the real problem is > "Lucas doesn't know the right search terms." >=20 > The goal is to make an iSCSI-based ZFS pool that's available to two > separate hosts, and remains available even if one of the iSCSI servers > fails. Instead, the pool hangs when either of the iSCSI servers goes > down. I=E2=80=99m no expert and have never used iSCSI and FreeBSD before, but = I think you might=20 want to look at the kern.iscsi.fail_on_disconnection sysctl. man 4 iscsi That means the devices will fail instead of hang, and ZFS might decide = to mark those=20 devices as faulted instead of waiting for them to respond. Cheers Phil= From owner-freebsd-fs@freebsd.org Thu Oct 29 06:43:01 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3BE9A201A5 for ; Thu, 29 Oct 2015 06:43:01 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 9205314D9 for ; Thu, 29 Oct 2015 06:43:01 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 9109DA201A4; Thu, 29 Oct 2015 06:43:01 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77948A201A3 for ; Thu, 29 Oct 2015 06:43:01 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x229.google.com (mail-wm0-x229.google.com [IPv6:2a00:1450:400c:c09::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1068714D6 for ; Thu, 29 Oct 2015 06:43:01 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by wmff134 with SMTP id f134so18419022wmf.1 for ; Wed, 28 Oct 2015 23:42:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=ICmmBL1qQWwhPHOXAOA/QM0hjWM6hHcHcwWYN8gobkE=; b=KV+1FZfykAb1ERgIPntARuIK34DNiS3lJu11x9mmc8PAWFM331dVUro4JKh1EouTbq KYzH2/r6raofbn8Xxw+EEIPSX+Y5pqSlfYCSKMHWjMzwl14JvS/l8kSpI9eeH5ne1F36 9DDPntejL9fi1HfdVKjaRRajRnDPbI/y80qI0x5Wv2hYwFCbu0F1qkWK/SzrX2cIAMth UTFejlGfdZH0AhsG1Oa2XgkDIv9Ssnk3WFlFKtmyyGBs73T33h1Mrpqj1pxlhjRoUjGd EgyYvbf4PsI6rLAMAwr3HFhZpTDTT0t8bXROK/Ld82cqrMc5mHH/ki6O8I0Ff4iNDBV9 2iew== X-Received: by 10.28.138.140 with SMTP id m134mr1561717wmd.74.1446100979119; Wed, 28 Oct 2015 23:42:59 -0700 (PDT) Received: from [192.168.0.1] (cag06-2-82-237-68-117.fbx.proxad.net. [82.237.68.117]) by smtp.gmail.com with ESMTPSA id om1sm141287wjc.2.2015.10.28.23.42.58 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 28 Oct 2015 23:42:58 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: iSCSI/ZFS strangeness From: Ben RUBSON In-Reply-To: Date: Thu, 29 Oct 2015 07:42:57 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20151029015721.GA95057@mail.michaelwlucas.com> To: fs@freebsd.org X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 06:43:01 -0000 > Le 29 oct. 2015 =C3=A0 03:26, Philip Murray a = =C3=A9crit : >=20 >=20 >> On 29/10/2015, at 2:57 PM, Michael W. Lucas = wrote: >>=20 >> Hi, >>=20 >> I'm experimenting with iSCSI HA with FreeBSD 10.2 amd64. I know = people >> do this sort of thing, but I can't figure out what I'm missing. (Most >> of the tutorials cover HAST instead). I suspect the real problem is >> "Lucas doesn't know the right search terms." >>=20 >> The goal is to make an iSCSI-based ZFS pool that's available to two >> separate hosts, and remains available even if one of the iSCSI = servers >> fails. Instead, the pool hangs when either of the iSCSI servers goes >> down. >=20 > I=E2=80=99m no expert and have never used iSCSI and FreeBSD before, = but I think you might=20 > want to look at the kern.iscsi.fail_on_disconnection sysctl. >=20 > man 4 iscsi >=20 > That means the devices will fail instead of hang, and ZFS might decide = to mark those=20 > devices as faulted instead of waiting for them to respond. Hi, Yes, kern.iscsi.fail_on_disconnection=3D1 in sysctl.conf will do the = trick. In addition, you can tune the following to minimize the impact / freeze = when iSCSI fails : kern.iscsi.ping_timeout=3D10 kern.iscsi.iscsid_timeout=3D10 kern.iscsi.login_timeout=3D10 In addition, in ctl.conf, perhaps you should use one target instruction = per disk. I began with all my disks/luns into a unique target configuration, but = when a disk fails and you replace it, you need to refresh the iSCSI = configuration using iscsictl -M. At this moment you will loose the = connection to all disks, leading into many ZFS devices failed... So I think a target per disks is better, then you only refresh the dead = target : iscsictl -M -i And last thing, thank you very much for your quality books, a must-have = :) Best regards, Ben= From owner-freebsd-fs@freebsd.org Thu Oct 29 09:17:42 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C853A1E17E for ; Thu, 29 Oct 2015 09:17:42 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x234.google.com (mail-wm0-x234.google.com [IPv6:2a00:1450:400c:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8DD21F7D for ; Thu, 29 Oct 2015 09:17:41 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wmeg8 with SMTP id g8so20647425wme.0 for ; Thu, 29 Oct 2015 02:17:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay_co_uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=7yeObzkoazgeuz58B9Heww4m4Ec7+Nsx+OGn7L5aK5k=; b=P5bIF9uAzeB8+xBfnY8Pl1GB3kMVKOyZ5POhfoUkj57arxDCwcV/w028POeyZXVxxC txj1YuDcPTu1Xp3Hdh0llxbCefBo8MCrgbyTC54zjyMknhf9YTmSe3eU60nhIuAaU8nJ 6N7OmJyNRocdkRzRBcncs3Bdej9VEQlAHiUntUBtKFIf3utDepw7ZgDNND9/KC4VNYd4 ZNkgdv/92/AvwAarwpOWnrA5Y+BGoxjfYtuX3ek596aTVY7EoM5TepCGURZm4HQDCuKo fxHzj0gZ1/fzekuyDbpLeguh7fNueewCRfsLQCJrwaXUKKSLOBMhdkQkqP69d3LUq191 T5Kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=7yeObzkoazgeuz58B9Heww4m4Ec7+Nsx+OGn7L5aK5k=; b=dD0GNCe2u16tcVCBK+8sPuEAIjBsgxFoOOE3FarS7Qx4P9dxchgKYYn5jIdmILqRCv zSCtWz4jSuXFkETijzv67MDKgHiFPupyvCcDe3cAGG7gxGT34jZX1Zs1IvUSkNjjwwFD sZMn3Et/QZzbSyHkfEGII8qrWEjWHZI72rjOfc0w++VrYbEJKotzT5vn4aFG/LFZPy6A tH8pHBXSMJaDVtD6JDTC9IA3Iw8kcDZGbL83cNMrdCl6C5V4sQI9IlfHgRFtj0nCaVrd 1WQ1dV0rr6pyFHjZ8+eOGdDihbB+gNBp3Rt2G1dxxwiFjX2fuNtxF2W0DLu2jUTXKHRn 3irw== X-Gm-Message-State: ALoCoQnPVWyCtH9oxmxeRcXVjtjzOvww+P4VkxKoAWMsIsb+1Llyz98YbDoq4fBAR+gw7p0qvcY0 X-Received: by 10.28.135.206 with SMTP id j197mr2541981wmd.14.1446110259971; Thu, 29 Oct 2015 02:17:39 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id cr5sm746873wjb.16.2015.10.29.02.17.38 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Oct 2015 02:17:38 -0700 (PDT) Subject: Re: iSCSI/ZFS strangeness To: freebsd-fs@freebsd.org References: <20151029015721.GA95057@mail.michaelwlucas.com> From: Steven Hartland Message-ID: <5631E43A.4050201@multiplay.co.uk> Date: Thu, 29 Oct 2015 09:17:46 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 09:17:42 -0000 On 29/10/2015 06:42, Ben RUBSON wrote: >> Le 29 oct. 2015 à 03:26, Philip Murray a écrit : >> >> >>> On 29/10/2015, at 2:57 PM, Michael W. Lucas wrote: >>> >>> Hi, >>> >>> I'm experimenting with iSCSI HA with FreeBSD 10.2 amd64. I know people >>> do this sort of thing, but I can't figure out what I'm missing. (Most >>> of the tutorials cover HAST instead). I suspect the real problem is >>> "Lucas doesn't know the right search terms." >>> >>> The goal is to make an iSCSI-based ZFS pool that's available to two >>> separate hosts, and remains available even if one of the iSCSI servers >>> fails. Instead, the pool hangs when either of the iSCSI servers goes >>> down. >> I’m no expert and have never used iSCSI and FreeBSD before, but I think you might >> want to look at the kern.iscsi.fail_on_disconnection sysctl. >> >> man 4 iscsi >> >> That means the devices will fail instead of hang, and ZFS might decide to mark those >> devices as faulted instead of waiting for them to respond. > Hi, > > Yes, kern.iscsi.fail_on_disconnection=1 in sysctl.conf will do the trick. > In addition, you can tune the following to minimize the impact / freeze when iSCSI fails : > kern.iscsi.ping_timeout=10 > kern.iscsi.iscsid_timeout=10 > kern.iscsi.login_timeout=10 > > In addition, in ctl.conf, perhaps you should use one target instruction per disk. > I began with all my disks/luns into a unique target configuration, but when a disk fails and you replace it, you need to refresh the iSCSI configuration using iscsictl -M. At this moment you will loose the connection to all disks, leading into many ZFS devices failed... > So I think a target per disks is better, then you only refresh the dead target : iscsictl -M -i > > And last thing, thank you very much for your quality books, a must-have :) > While not strictly a pure ZFS solution, if your initiators can't both be active at the same time a better solution might be to use geom multipath with ZFS on top of that? Regards Steve From owner-freebsd-fs@freebsd.org Thu Oct 29 11:36:29 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C7D9A20B6D for ; Thu, 29 Oct 2015 11:36:29 +0000 (UTC) (envelope-from mwlucas@mail.michaelwlucas.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1F8611FD3 for ; Thu, 29 Oct 2015 11:36:29 +0000 (UTC) (envelope-from mwlucas@mail.michaelwlucas.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1CBD9A20B66; Thu, 29 Oct 2015 11:36:29 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C623A20B64 for ; Thu, 29 Oct 2015 11:36:29 +0000 (UTC) (envelope-from mwlucas@mail.michaelwlucas.com) Received: from mail.michaelwlucas.com (mail.michaelwlucas.com [104.236.197.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CFB4A1FD2 for ; Thu, 29 Oct 2015 11:36:28 +0000 (UTC) (envelope-from mwlucas@mail.michaelwlucas.com) Received: from mail.michaelwlucas.com (localhost [127.0.0.1]) by mail.michaelwlucas.com (8.14.9/8.14.7) with ESMTP id t9TBZ94t020517; Thu, 29 Oct 2015 07:35:10 -0400 (EDT) (envelope-from mwlucas@mail.michaelwlucas.com) Received: (from mwlucas@localhost) by mail.michaelwlucas.com (8.14.9/8.14.7/Submit) id t9TBZ9A3020516; Thu, 29 Oct 2015 07:35:09 -0400 (EDT) (envelope-from mwlucas) Date: Thu, 29 Oct 2015 07:35:08 -0400 From: "Michael W. Lucas" To: Philip Murray Cc: "Michael W. Lucas" , fs@freebsd.org Subject: Re: iSCSI/ZFS strangeness Message-ID: <20151029113508.GA20477@mail.michaelwlucas.com> References: <20151029015721.GA95057@mail.michaelwlucas.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail.michaelwlucas.com X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mail.michaelwlucas.com [127.0.0.1]); Thu, 29 Oct 2015 07:35:11 -0400 (EDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 11:36:29 -0000 On Thu, Oct 29, 2015 at 03:26:57PM +1300, Philip Murray wrote: > > man 4 iscsi > Thank you. I've been studying section 8 and 5 of the manual, because I'm no programmer... ==ml -- Michael W. Lucas - mwlucas@michaelwlucas.com, Twitter @mwlauthor http://www.MichaelWLucas.com/, http://blather.MichaelWLucas.com/ From owner-freebsd-fs@freebsd.org Thu Oct 29 18:17:44 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8654BA212DF for ; Thu, 29 Oct 2015 18:17:44 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from smtp.rlwinm.de (smtp.rlwinm.de [IPv6:2a01:4f8:201:31ef::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DF3A17C7 for ; Thu, 29 Oct 2015 18:17:44 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from crest.local (unknown [87.253.189.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.rlwinm.de (Postfix) with ESMTPSA id 4553F17D79 for ; Thu, 29 Oct 2015 19:17:41 +0100 (CET) Subject: Re: iSCSI/ZFS strangeness To: freebsd-fs@freebsd.org References: <20151029015721.GA95057@mail.michaelwlucas.com> From: Jan Bramkamp Message-ID: <563262C4.1040706@rlwinm.de> Date: Thu, 29 Oct 2015 19:17:40 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151029015721.GA95057@mail.michaelwlucas.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 18:17:44 -0000 On 29/10/15 02:57, Michael W. Lucas wrote: > The initiators can both access the iSCSI-based pool--not > simultaneously, of course. But CARP, devd, and some shell scripting > should get me a highly available pool that can withstand the demise of > any one iSCSI server and any one initiator. > > The hope is that the pool would continue to work even if an iSCSI host > shuts down. When the downed iSCSI host returns, the initiators should > log back in and the pool auto-resilver. I would recommend against using CARP for this because CARP is prone to split-brain situations and in this case they could destroy your whole storage pool. If the current head node fails the replacement has to `zpool import -f` the pool and and in the case of a split-brain situation both head nodes would continue writing to the iSCSI targets. I would move the leader election to an external service like consul, etcd or zookeeper. This is one case where the added complexity is worth it. If you can't run an external service for this e.g. it would exceed the scope of the chapter you're writing please simplify the setup with more reliable hardware, good monitoring and manual failover for maintenance. CARP isn't designed to implement reliable (enough) master election for your storage cluster. Adding iSCSI to your storage stack adds complexity and overhead. For setups which still fit inside a single rack SAS (with geom_multipath) is normally faster and cheaper. On the other hand you can't spread out SAS storage far enough to implement disaster tolerance should you really need it and it certainly is an interesting setup. From owner-freebsd-fs@freebsd.org Fri Oct 30 07:18:26 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D161A21557 for ; Fri, 30 Oct 2015 07:18:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2861A1349 for ; Fri, 30 Oct 2015 07:18:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9U7IQaV077239 for ; Fri, 30 Oct 2015 07:18:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204140] ZFS's dsl_pool.c and spa_misc.c leak FreeBSD-specific kernel routines into userland libraries Date: Fri, 30 Oct 2015 07:18:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 07:18:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204140 NGie Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|dsl_pool.c and spa_misc.c |ZFS's dsl_pool.c and |leak FreeBSD-specific |spa_misc.c leak |kernel routines into |FreeBSD-specific kernel |userland libraries |routines into userland | |libraries Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org CC| |avg@FreeBSD.org, | |delphij@FreeBSD.org, | |will@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Fri Oct 30 07:18:52 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94404A215B3 for ; Fri, 30 Oct 2015 07:18:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7FE651474 for ; Fri, 30 Oct 2015 07:18:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9U7IqCs077669 for ; Fri, 30 Oct 2015 07:18:52 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 204140] ZFS's dsl_pool.c and spa_misc.c leak FreeBSD-specific kernel routines into userland libraries Date: Fri, 30 Oct 2015 07:18:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 07:18:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204140 --- Comment #1 from NGie Cooper --- (In reply to NGie Cooper from comment #0) Also, there's the obvious "missing #include " in the source files. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Fri Oct 30 10:38:42 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E0AECA20F52 for ; Fri, 30 Oct 2015 10:38:42 +0000 (UTC) (envelope-from amdmi3@amdmi3.ru) Received: from vps.amdmi3.ru (vps.amdmi3.ru [109.234.38.216]) by mx1.freebsd.org (Postfix) with ESMTP id AB9B2132A for ; Fri, 30 Oct 2015 10:38:42 +0000 (UTC) (envelope-from amdmi3@amdmi3.ru) Received: from hive.panopticon (unknown [78.153.152.119]) by vps.amdmi3.ru (Postfix) with ESMTPS id 2B4E9B0616 for ; Fri, 30 Oct 2015 13:38:35 +0300 (MSK) Received: from hades.panopticon (hades.panopticon [192.168.0.32]) by hive.panopticon (Postfix) with ESMTP id D6A3CB11 for ; Fri, 30 Oct 2015 13:34:29 +0300 (MSK) Received: by hades.panopticon (Postfix, from userid 1000) id DA8966E; Fri, 30 Oct 2015 13:36:14 +0300 (MSK) Date: Fri, 30 Oct 2015 13:36:14 +0300 From: Dmitry Marakasov To: freebsd-fs@freebsd.org Subject: ZFS resilver from disk with bad sectors constantly restarts Message-ID: <20151030103614.GL57666@hades.panopticon> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 10:38:43 -0000 Hi! I've just got a case where resilvering a new replacement disk in raidz2 never finished. The problem: one disk in raidz is failing by having a large number of unreadable sectors. It's replaced with a spare. Resilver though is constantly restarted with log full of read error from bad disk. It looks like this: --- pool: spool state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Wed Oct 28 05:26:28 2015 369G scanned out of 9,87T at 123M/s, 22h29m to go 41,4G resilvered, 3,65% done config: NAME STATE READ WRITE CKSUM spool ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 spare-2 ONLINE 0 0 733 ada11 ONLINE 0 0 0 ada2 ONLINE 0 0 0 (resilvering) raidz1-1 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada4 ONLINE 0 0 0 ada5 ONLINE 0 0 0 raidz1-2 ONLINE 0 0 0 ada6 ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada10 ONLINE 0 0 0 spares 588540573008830286 INUSE was /dev/ada2 errors: No known data errors --- `resilver in progress since' date is constantly reset, so resilved progress cannot pass beyond 5% or so. My guess is that it happens on read errors on ada11. I think I've seen (resilvering) on ada11 line couple of times. In the end I've had to offline ada11 and after that resilver completed in under 16 hours. However the situation doesn't seem normal, as I'd prefer to not lose redundancy with offlining dying disk and still be able to use it for resilvering (imagine there were bad sectors on ada0/1 as well, but not intersecting with bad sectors on ada11), or at least some more verbose indication of why the resilver is constantly restarted. I should also note that's outdated FreeBSD 9.1, so maybe that problem was fixed already. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amdmi3@amdmi3.ru ..: jabber: amdmi3@jabber.ru http://amdmi3.ru From owner-freebsd-fs@freebsd.org Fri Oct 30 11:45:15 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6A8EA20E62 for ; Fri, 30 Oct 2015 11:45:15 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [IPv6:2a00:1450:400c:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 888D51A45 for ; Fri, 30 Oct 2015 11:45:15 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wicll6 with SMTP id ll6so8808802wic.1 for ; Fri, 30 Oct 2015 04:45:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay_co_uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=vCVPGOhIl8qgELu/bEN49Ps4f9Y3nwYdaQj7DP1Jyag=; b=kweNKyBGAskKExqtnaRGn/BiaW9jrMSsPxX1wpRUFgT9OT07Mokd6RXU/W2ljJHagC pqrDYBGTVGvtBElOR40RG08N3XVDT5ViKKhhuzfr/q/eYucMUMzEnixA7HsPA1fIMgWR cFLSO7PoVNOLWwo93cE5gFCAE1lM1TLE79k/sVvORdrYfi+D923B7Cvwoy3vJnMg7Z9r r04ZNwxE8QnqHiYZgokAMxq4GgCuKuwMW6/g/JtE21/wCmAE6U3E8d+ytLdA1Xrbh/yJ vEajL+W5GSOMX60zxPix8MLS5JxXPBX/WAlLNMXdCqvohY0YIdwUJ4ekcXdpflILId+J sOZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=vCVPGOhIl8qgELu/bEN49Ps4f9Y3nwYdaQj7DP1Jyag=; b=CuwX4XBf3BgZTb9N9r5Zge9mebD2xYajD88Am0uiOzhPjKH8tD4rqsOfIhOkwhXGhz NF/kh+fmAk3mEvpOKoONVOPhOm+MHLaNn+sgfRe0qieobAYXO2FtedB4h++uuuDdMoCo 02i9BRevklhHBwZW1VgrbalrPv535NHd46p8vbDqYtHD25fZts9e+o21W2pVvkzhjo5I nITSqioZPGCv3GgKReDd5sBX1s+YYv7o5IWFYIoccDfqbvPX/yjTWRHoXIpuATsLHkI1 Ra4qOC2o4sDG95A9wzLLrx2vyfWYMX3za0ypfl/bj9hxvp3ZJLtdlFuTOQRA+kSVEDIT Awnw== X-Gm-Message-State: ALoCoQm1qS+qdZoD+aaoywGDkjsP5S7ET5UaA2br7CSEYtIp+O4fygx2emEFKPlQow9WIVcLfipS X-Received: by 10.194.58.142 with SMTP id r14mr9446092wjq.37.1446205513896; Fri, 30 Oct 2015 04:45:13 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id 77sm2496462wml.20.2015.10.30.04.45.12 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 30 Oct 2015 04:45:12 -0700 (PDT) Subject: Re: ZFS resilver from disk with bad sectors constantly restarts To: freebsd-fs@freebsd.org References: <20151030103614.GL57666@hades.panopticon> From: Steven Hartland Message-ID: <56335851.5070802@multiplay.co.uk> Date: Fri, 30 Oct 2015 11:45:21 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151030103614.GL57666@hades.panopticon> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 11:45:16 -0000 What version? On 30/10/2015 10:36, Dmitry Marakasov wrote: > Hi! > > I've just got a case where resilvering a new replacement disk in raidz2 > never finished. > > The problem: one disk in raidz is failing by having a large number of > unreadable sectors. It's replaced with a spare. Resilver though is > constantly restarted with log full of read error from bad disk. > > It looks like this: > > --- > pool: spool > state: ONLINE > status: One or more devices is currently being resilvered. The pool will > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scan: resilver in progress since Wed Oct 28 05:26:28 2015 > 369G scanned out of 9,87T at 123M/s, 22h29m to go > 41,4G resilvered, 3,65% done > config: > > NAME STATE READ WRITE CKSUM > spool ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > ada0 ONLINE 0 0 0 > ada1 ONLINE 0 0 0 > spare-2 ONLINE 0 0 733 > ada11 ONLINE 0 0 0 > ada2 ONLINE 0 0 0 (resilvering) > raidz1-1 ONLINE 0 0 0 > ada3 ONLINE 0 0 0 > ada4 ONLINE 0 0 0 > ada5 ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > ada6 ONLINE 0 0 0 > ada7 ONLINE 0 0 0 > ada10 ONLINE 0 0 0 > spares > 588540573008830286 INUSE was /dev/ada2 > > errors: No known data errors > --- > > `resilver in progress since' date is constantly reset, so resilved > progress cannot pass beyond 5% or so. My guess is that it happens on > read errors on ada11. I think I've seen (resilvering) on ada11 line > couple of times. > > In the end I've had to offline ada11 and after that resilver completed > in under 16 hours. However the situation doesn't seem normal, as I'd > prefer to not lose redundancy with offlining dying disk and still be > able to use it for resilvering (imagine there were bad sectors on ada0/1 > as well, but not intersecting with bad sectors on ada11), or at least > some more verbose indication of why the resilver is constantly restarted. > > I should also note that's outdated FreeBSD 9.1, so maybe that problem > was fixed already. > From owner-freebsd-fs@freebsd.org Fri Oct 30 11:46:37 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 91352A20EED for ; Fri, 30 Oct 2015 11:46:37 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x234.google.com (mail-wm0-x234.google.com [IPv6:2a00:1450:400c:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 216E11AE9 for ; Fri, 30 Oct 2015 11:46:37 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wmeg8 with SMTP id g8so9900325wme.1 for ; Fri, 30 Oct 2015 04:46:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay_co_uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=+LPDwiZ6EMZK7qeXDH6/7YHInsuVZaSu3AP7fbacmZg=; b=ZnxHcwLnyCTwy8JLQFdRnEl4oMpZX/IBSHeyzTHN8fjwdL19mdDBi+J4VuDSL5FhFS yVKYkZG60pPC4k4s+9q6fFjvsj9dsr5Hjru7nVGa9qdlF3jyvKoUyK2rK5hZd9a5dEVK 2Vj0LRsxYCraJqungoCzF3Z3lHD3G56A33oMVioTPLs5h/ZqYTC8Xc56C9wPNZNwP51I 8B6aWurIAWnv415gVTBFdA88cTRvLHlCv+sGUXk18TfIDNKpB+IhAE+E3RRrJw40WybN LSH/2LtvjcyeNuwctw958VRYG2d/a1IGXFSkUzr6hn7K2ZqHnVZjU8hNXNcSG9d5/49E 6T6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=+LPDwiZ6EMZK7qeXDH6/7YHInsuVZaSu3AP7fbacmZg=; b=Xl9jeeUZDF9r4ZjkJOfL58dKgbJQjsQLnuuZo/UbuIY+YNv5qqXbl7loh9ysdkmWWH +PnNqWoz5eZRHXdWxoB0DjClcj59Px4GJG1UF17UaPWvOorOAsyLQNGlQQZ8Gse/HkZH nd/v2VphiBRd4mnHFv0skeDHfG5/+8a1DFVXlGRsmxI7ThPMUUfMdn6cGmRnXPq1nyrF 3cVIxgMkEP3szJa63TBI7+paTCP8QWfXv1rTKaftkQ5Wdn16Q2rpEagulYETbnhWmBe5 Y9/NiPK8P6IyNFV7UhXE2XVG+zD9w3jzl3ghbcaG1r4bG6yvMDRSlJjRa+CsZ+4g5keb r//Q== X-Gm-Message-State: ALoCoQlvbT02KORTq30+eMf+8mxXXpzEfARH63M3dYoMnPqdSecfI3rbXR8/RRkj3RrJRZTxKEoK X-Received: by 10.28.211.70 with SMTP id k67mr2761833wmg.38.1446205595324; Fri, 30 Oct 2015 04:46:35 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id 186sm2512994wmp.10.2015.10.30.04.46.34 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 30 Oct 2015 04:46:34 -0700 (PDT) Subject: Re: ZFS resilver from disk with bad sectors constantly restarts To: freebsd-fs@freebsd.org References: <20151030103614.GL57666@hades.panopticon> From: Steven Hartland Message-ID: <563358A3.3030209@multiplay.co.uk> Date: Fri, 30 Oct 2015 11:46:43 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151030103614.GL57666@hades.panopticon> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 11:46:37 -0000 Ignore my last post just noticed you 9.1 at the end, which is very old, so it could well be this is already fixed; only way to confirm is upgrade an check ;-) Regards Steve On 30/10/2015 10:36, Dmitry Marakasov wrote: > Hi! > > I've just got a case where resilvering a new replacement disk in raidz2 > never finished. > > The problem: one disk in raidz is failing by having a large number of > unreadable sectors. It's replaced with a spare. Resilver though is > constantly restarted with log full of read error from bad disk. > > It looks like this: > > --- > pool: spool > state: ONLINE > status: One or more devices is currently being resilvered. The pool will > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scan: resilver in progress since Wed Oct 28 05:26:28 2015 > 369G scanned out of 9,87T at 123M/s, 22h29m to go > 41,4G resilvered, 3,65% done > config: > > NAME STATE READ WRITE CKSUM > spool ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > ada0 ONLINE 0 0 0 > ada1 ONLINE 0 0 0 > spare-2 ONLINE 0 0 733 > ada11 ONLINE 0 0 0 > ada2 ONLINE 0 0 0 (resilvering) > raidz1-1 ONLINE 0 0 0 > ada3 ONLINE 0 0 0 > ada4 ONLINE 0 0 0 > ada5 ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > ada6 ONLINE 0 0 0 > ada7 ONLINE 0 0 0 > ada10 ONLINE 0 0 0 > spares > 588540573008830286 INUSE was /dev/ada2 > > errors: No known data errors > --- > > `resilver in progress since' date is constantly reset, so resilved > progress cannot pass beyond 5% or so. My guess is that it happens on > read errors on ada11. I think I've seen (resilvering) on ada11 line > couple of times. > > In the end I've had to offline ada11 and after that resilver completed > in under 16 hours. However the situation doesn't seem normal, as I'd > prefer to not lose redundancy with offlining dying disk and still be > able to use it for resilvering (imagine there were bad sectors on ada0/1 > as well, but not intersecting with bad sectors on ada11), or at least > some more verbose indication of why the resilver is constantly restarted. > > I should also note that's outdated FreeBSD 9.1, so maybe that problem > was fixed already. > From owner-freebsd-fs@freebsd.org Fri Oct 30 12:40:38 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE05BA2129B for ; Fri, 30 Oct 2015 12:40:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A2DB01A2E for ; Fri, 30 Oct 2015 12:40:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9UCecK5079685 for ; Fri, 30 Oct 2015 12:40:38 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 190462] request: don't active feature flags when upgrading zfs pool Date: Fri, 30 Oct 2015 12:40:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: fk@fabiankeil.de X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 12:40:38 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=190462 Fabian Keil changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fk@fabiankeil.de --- Comment #1 from Fabian Keil --- Individual features can be enabled with zpool set. Example: fk@r500 ~ $zpool get all test NAME PROPERTY VALUE SOURCE [...] test feature@async_destroy disabled local test feature@empty_bpobj disabled local test feature@lz4_compress disabled local test feature@multi_vdev_crash_dump disabled local test feature@spacemap_histogram disabled local test feature@enabled_txg disabled local test feature@hole_birth disabled local test feature@extensible_dataset disabled local test feature@embedded_data disabled local test feature@bookmarks disabled local test feature@filesystem_limits disabled local test feature@large_blocks disabled local fk@r500 ~ $sudo zpool set feature@empty_bpobj=enabled test fk@r500 ~ $sudo zpool set feature@embedded_data=enabled test fk@r500 ~ $zpool get all test NAME PROPERTY VALUE SOURCE [...] test feature@async_destroy disabled local test feature@empty_bpobj enabled local test feature@lz4_compress disabled local test feature@multi_vdev_crash_dump disabled local test feature@spacemap_histogram disabled local test feature@enabled_txg disabled local test feature@hole_birth disabled local test feature@extensible_dataset disabled local test feature@embedded_data active local test feature@bookmarks disabled local test feature@filesystem_limits disabled local test feature@large_blocks disabled local While my system is based on 11-CURRENT, this is documented to work on 10.0-RELEASE, too: https://www.freebsd.org/cgi/man.cgi?query=zpool&manpath=FreeBSD+10.0-RELEASE Given the length of the man page, it probably wouldn't hurt to mention this in the "zpool upgrade" section as well. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Fri Oct 30 12:54:55 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67BA9A21649 for ; Fri, 30 Oct 2015 12:54:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 543091162 for ; Fri, 30 Oct 2015 12:54:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9UCstuZ003710 for ; Fri, 30 Oct 2015 12:54:55 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203201] on zfs extattr behaviour broken after unlink Date: Fri, 30 Oct 2015 12:54:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 12:54:55 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203201 --- Comment #8 from Andriy Gapon --- [ping] Are you willing to test the patch? -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Fri Oct 30 13:07:16 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74170A218BA for ; Fri, 30 Oct 2015 13:07:16 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3204017A2 for ; Fri, 30 Oct 2015 13:07:15 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id D5CE72091D for ; Fri, 30 Oct 2015 09:07:13 -0400 (EDT) Received: from frontend1 ([10.202.2.160]) by compute3.internal (MEProxy); Fri, 30 Oct 2015 09:07:13 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=hU0ozuBrxUA/6EK ThIvvKBOUH1U=; b=UI8PRQqxscBxyC5h2nbPcOStsB+5hak91oW3eWz9N0pR3HN btjFpBYgtf/u9IeAZeag1w2IZQWXsTtG9yuz9coVelLC0Vr/VTYxN8CnvwT7WLZO j2a6ODFtnPrt0OrnwcYQnQbcrD002Uo/ENWbOgHGfdh8YVUezefzhsCJ9TYM= X-Sasl-enc: PkYsoCFZFLZ1JMXzyryVS14+7LgPNQIcQNEsQAeBVArP 1446210433 Received: from [192.168.8.142] (184-158-23-49.dyn.centurytel.net [184.158.23.49]) by mail.messagingengine.com (Postfix) with ESMTPA id 60200C00092; Fri, 30 Oct 2015 09:07:13 -0400 (EDT) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable From: Josh Paetzel Mime-Version: 1.0 (1.0) Subject: Re: iSCSI/ZFS strangeness Date: Fri, 30 Oct 2015 08:06:55 -0500 Message-Id: <9D4FE448-28EC-45F6-B525-E660E3AF57B0@tcbug.org> References: <20151029015721.GA95057@mail.michaelwlucas.com> <563262C4.1040706@rlwinm.de> Cc: freebsd-fs@freebsd.org In-Reply-To: <563262C4.1040706@rlwinm.de> To: Jan Bramkamp X-Mailer: iPhone Mail (13B143) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 13:07:16 -0000 > On Oct 29, 2015, at 1:17 PM, Jan Bramkamp wrote: >=20 >> On 29/10/15 02:57, Michael W. Lucas wrote: >> The initiators can both access the iSCSI-based pool--not >> simultaneously, of course. But CARP, devd, and some shell scripting >> should get me a highly available pool that can withstand the demise of >> any one iSCSI server and any one initiator. >>=20 >> The hope is that the pool would continue to work even if an iSCSI host >> shuts down. When the downed iSCSI host returns, the initiators should >> log back in and the pool auto-resilver. >=20 > I would recommend against using CARP for this because CARP is prone to spl= it-brain situations and in this case they could destroy your whole storage p= ool. If the current head node fails the replacement has to `zpool import -f`= the pool and and in the case of a split-brain situation both head nodes wou= ld continue writing to the iSCSI targets. >=20 > I would move the leader election to an external service like consul, etcd o= r zookeeper. This is one case where the added complexity is worth it. If you= can't run an external service for this e.g. it would exceed the scope of th= e chapter you're writing please simplify the setup with more reliable hardwa= re, good monitoring and manual failover for maintenance. CARP isn't designed= to implement reliable (enough) master election for your storage cluster. >=20 > Adding iSCSI to your storage stack adds complexity and overhead. For setup= s which still fit inside a single rack SAS (with geom_multipath) is normally= faster and cheaper. On the other hand you can't spread out SAS storage far e= nough to implement disaster tolerance should you really need it and it certa= inly is an setup. I'll impart some wisdom here. 1) HA with two nodes is impossible to do right. You need a third system to a= chieve quorum. 2) You can do SAS over optical these days. Perfect for having mirrored JBODs= in different fire suppression zones of a datacenter. 3) I've seen a LOT of "cobbled together with shell script" HA rigs. They mo= stly get disabled eventually as it's realized that they go split brain in th= e edge cases and destroy the storage. What we did was go passive/passive an= d then address those cases as "how could we have avoided going passive/passi= ve". It took two years. 4) Leverage mav@'s ALUA support. For block access this will make your life m= uch easier. 5) Give me a call. I type slow and tend to leave things out, but would happi= ly do one or more brain dump sessions.= From owner-freebsd-fs@freebsd.org Fri Oct 30 13:20:48 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE6C5A21A94 for ; Fri, 30 Oct 2015 13:20:48 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7EDDC1B81 for ; Fri, 30 Oct 2015 13:20:48 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 80EED201E6 for ; Fri, 30 Oct 2015 09:20:47 -0400 (EDT) Received: from frontend1 ([10.202.2.160]) by compute1.internal (MEProxy); Fri, 30 Oct 2015 09:20:47 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-sasl-enc :x-sasl-enc; s=smtpout; bh=3LdfgPveWNC5JypssCAlBigFfNE=; b=Hp36Y KQRXud+Q5EVIxsoqV2YxCR4BU+rjNJu7TxrjLczgmIEXmnOew922liUM1ojzsO4Z BmwvBH4kzbhSRGMvOEWI4LZwmcB9ipftYg230tFWrZWbeGf1IIWAtOoPY6d5H4+R onkneNkVWuwNlhrZEo2VWWXY9ru/OLmnM0+lIs= X-Sasl-enc: rdv7hKUboHIfZeeGn6YIVIBVnIA0L8/q5xdF7YZXHoB4 1446211247 Received: from [192.168.8.142] (184-158-23-49.dyn.centurytel.net [184.158.23.49]) by mail.messagingengine.com (Postfix) with ESMTPA id 3105BC00091 for ; Fri, 30 Oct 2015 09:20:47 -0400 (EDT) From: Josh Paetzel Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Subject: NFS --manage-gids Message-Id: <29F53220-75AB-4898-B1E4-C1BF0231E25B@tcbug.org> Date: Fri, 30 Oct 2015 08:20:46 -0500 To: freebsd-fs@freebsd.org X-Mailer: iPhone Mail (13B143) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 13:20:48 -0000 As anyone who is familiar with NFS is aware, auth_sys has a limit of members= hip in 16 aux groups. In today's "everything is in AD universe" it's incredibly common for an acco= unt to be in more than 16 groups. There are various solutions to this. Right now the only reasonable one on Fre= eBSD is to use Kerberos. auth_krb does not have the 16 group membership limi= t. Of course that solution is great if you already use Kerberos, but if you d= on't suggesting that as a solution does't always go over so well. The Linux crowd extended their NFS server years ago with a --manage-gids opt= ion that lets it ignore the group member ship sent over the wire by the clie= nt and look up group membership locally. Does anyone have any objections to that option getting ported to FreeBSD? Thanks, Josh Paetzel= From owner-freebsd-fs@freebsd.org Fri Oct 30 13:50:10 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE50DA1E278 for ; Fri, 30 Oct 2015 13:50:10 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7292A184B for ; Fri, 30 Oct 2015 13:50:10 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wmff134 with SMTP id f134so12308867wmf.1 for ; Fri, 30 Oct 2015 06:50:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay_co_uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=GYVW5riajIKF8/f1ahQzQBbM358hy7sq95x3qc8n6BQ=; b=pX90HsE3Y0kNtSpSTGHaYZjv85LHmzRp4z+NCFTmeyV88CApcu4Iriad/bddHiB0Cm 5CMD8eIsEG7iCj2DR8cgLxwzuiHZSAtrkZRdCPA/85VR7q5TcIvj33JhcgmcR1ijg6bo sCNyxtx5y6FZP+HCE0m2L1DYajJcKQEeVygqaVYMH7j0v3iMiv2X4Ej2ddc4wnALw78o k7mCXwPNcZM1necaHWXOnRv3omthQeumG+eG+NBjLfBqROwFtSjyhMcA2nyg7rltbat9 f2Y8P3GooqsqyR6zN/EWgKdz0Hfq/kOqgVY/J37Y/of79DPFD5qf1oNV10I+vmU9Yk3L i9Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=GYVW5riajIKF8/f1ahQzQBbM358hy7sq95x3qc8n6BQ=; b=QldpJh+Z45Q/rVBRUYjFQGeq81LG8vkVfq9qaalMBKvuuZg+Q1soG9cyMlGJB2GCNP 7Tg3On+jOfW7wp9JItDH45a41Q0HVa7nsyr8NB5r4mzP0e6m9o+cL9nigxCrte5aGX5C g7r92psjwfbvx5ZNjDknb21GGlpso0GfihtXM5oX4M7kNOboHujfUJqWXvq5xton7j1N ahs9/VNHMBi6K7ve4LLz+jWA7G8anPf9SKkg5+xokp78dXzJR34jNptn7IxiSCMvxo1R 1wtwOId1azccCSQkUcsY7WHrVBgz/STniVTs2VAa8Ci2h05OROjW6aa2EYrDKlqNB6EM 8ayw== X-Gm-Message-State: ALoCoQlF+8nRCu35sOjxeJirKzNf1EVRx+rf9nJEKCSR28dg4WT4kEluvUHCv2YNhdhpnt5Eb/s0 X-Received: by 10.28.13.75 with SMTP id 72mr3477908wmn.20.1446213008780; Fri, 30 Oct 2015 06:50:08 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id j4sm3007953wmg.18.2015.10.30.06.50.07 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 30 Oct 2015 06:50:07 -0700 (PDT) Subject: Re: iSCSI/ZFS strangeness To: freebsd-fs@freebsd.org References: <20151029015721.GA95057@mail.michaelwlucas.com> <563262C4.1040706@rlwinm.de> <9D4FE448-28EC-45F6-B525-E660E3AF57B0@tcbug.org> From: Steven Hartland Message-ID: <56337598.2010704@multiplay.co.uk> Date: Fri, 30 Oct 2015 13:50:16 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <9D4FE448-28EC-45F6-B525-E660E3AF57B0@tcbug.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 13:50:11 -0000 On 30/10/2015 13:06, Josh Paetzel wrote: >> On Oct 29, 2015, at 1:17 PM, Jan Bramkamp wrote: >> >>> On 29/10/15 02:57, Michael W. Lucas wrote: >>> The initiators can both access the iSCSI-based pool--not >>> simultaneously, of course. But CARP, devd, and some shell scripting >>> should get me a highly available pool that can withstand the demise of >>> any one iSCSI server and any one initiator. >>> >>> The hope is that the pool would continue to work even if an iSCSI host >>> shuts down. When the downed iSCSI host returns, the initiators should >>> log back in and the pool auto-resilver. >> I would recommend against using CARP for this because CARP is prone to split-brain situations and in this case they could destroy your whole storage pool. If the current head node fails the replacement has to `zpool import -f` the pool and and in the case of a split-brain situation both head nodes would continue writing to the iSCSI targets. >> >> I would move the leader election to an external service like consul, etcd or zookeeper. This is one case where the added complexity is worth it. If you can't run an external service for this e.g. it would exceed the scope of the chapter you're writing please simplify the setup with more reliable hardware, good monitoring and manual failover for maintenance. CARP isn't designed to implement reliable (enough) master election for your storage cluster. >> >> Adding iSCSI to your storage stack adds complexity and overhead. For setups which still fit inside a single rack SAS (with geom_multipath) is normally faster and cheaper. On the other hand you can't spread out SAS storage far enough to implement disaster tolerance should you really need it and it certainly is an setup. > > I'll impart some wisdom here. > > 1) HA with two nodes is impossible to do right. You need a third system to achieve quorum. > > 2) You can do SAS over optical these days. Perfect for having mirrored JBODs in different fire suppression zones of a datacenter. > > 3) I've seen a LOT of "cobbled together with shell script" HA rigs. They mostly get disabled eventually as it's realized that they go split brain in the edge cases and destroy the storage. What we did was go passive/passive and then address those cases as "how could we have avoided going passive/passive". It took two years. > > 4) Leverage mav@'s ALUA support. For block access this will make your life much easier. > > 5) Give me a call. I type slow and tend to leave things out, but would happily do one or more brain dump sessions. Does the use of gmultipath not solve this problem?