From owner-freebsd-fs@freebsd.org Sun Oct 1 14:59:30 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7475E26AD3 for ; Sun, 1 Oct 2017 14:59:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 962DF383A for ; Sun, 1 Oct 2017 14:59:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v91ExUja080444 for ; Sun, 1 Oct 2017 14:59:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 222377] ZFS ABD wasteful... Date: Sun, 01 Oct 2017 14:59:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Oct 2017 14:59:30 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222377 --- Comment #4 from commit-hook@freebsd.org --- A commit references this bug: Author: avg Date: Sun Oct 1 14:58:44 UTC 2017 New revision: 324160 URL: https://svnweb.freebsd.org/changeset/base/324160 Log: MFC r323797: add vfs_zfs.abd_chunk_size tunable It is reported that the default value of 4KB results in a substantial memory use overhead (at least, on some configurations). Using 1KB seems to reduce the overhead significantly. PR: 222377 Changes: _U stable/11/ stable/11/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/abd.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Oct 1 15:04:37 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A8AEDE26EE9 for ; Sun, 1 Oct 2017 15:04:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 97B903E8C for ; Sun, 1 Oct 2017 15:04:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v91F4b9t002888 for ; Sun, 1 Oct 2017 15:04:37 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 222288] g_bio leak after zfs ABD commit Date: Sun, 01 Oct 2017 15:04:37 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Oct 2017 15:04:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222288 --- Comment #10 from commit-hook@freebsd.org --- A commit references this bug: Author: avg Date: Sun Oct 1 15:03:44 UTC 2017 New revision: 324161 URL: https://svnweb.freebsd.org/changeset/base/324161 Log: MFV r323796: fix memory leak in g_bio zone introduced in r320452 I overlooked the fact that that ZIO_IOCTL_PIPELINE does not include ZIO_STAGE_VDEV_IO_DONE stage. We do allocate a struct bio for an ioctl zio (a disk cache flush), but we never freed it. This change splits bio handling into two groups, one for normal read/write i/o that passes data around and, thus, needs the abd data tranform; the other group is for "data-less" i/o such as trim and cache flush. PR: 222288 Changes: _U stable/11/ stable/11/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Sun Oct 1 15:09:59 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2DECE2702B for ; Sun, 1 Oct 2017 15:09:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B20C33F79 for ; Sun, 1 Oct 2017 15:09:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v91F9xhG052152 for ; Sun, 1 Oct 2017 15:09:59 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 222288] g_bio leak after zfs ABD commit Date: Sun, 01 Oct 2017 15:09:59 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Oct 2017 15:09:59 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222288 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|In Progress |Closed --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Mon Oct 2 18:12:08 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59399E235DC for ; Mon, 2 Oct 2017 18:12:08 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x236.google.com (mail-wr0-x236.google.com [IPv6:2a00:1450:400c:c0c::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E0ECC71673 for ; Mon, 2 Oct 2017 18:12:07 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x236.google.com with SMTP id y95so4641941wrb.4 for ; Mon, 02 Oct 2017 11:12:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:subject:message-id:date:to :mime-version; bh=ns3h1BQ68YfnF/86oc+59wdNHLvzcML83/JjDFghP8c=; b=L0kPFGkIUVM3mU2TwnXhXZMVaqZ8hGUYsmz1Wb3qNCmvRRH/S8CtXiFOqUGFwcChm/ Aowy2hBzBmX/lknPTisSrLwt0ldZTLY+R3DXqOwb8Dx09np0hvZrWAgA1BFzFYqXUCy/ NLWxtRDKx3wbd14kCpAiQLrYdNTjMwvbVIOJCg4nub9LKTjflHUS7Yc2Az7J7jKZJPpj w8DztxPwMW72u5uenxf6f/Z5ToXlxpgO6tJChzYO0xevJVSOit1WZhWWCUd4Q8EsK4Oy YPy4OV1+tkk5IZzYD85PdhWaVXiFw3vl2rS8j2d7TH3UuBvTmTIbxucpdgA+mqSSb31f GeXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:subject :message-id:date:to:mime-version; bh=ns3h1BQ68YfnF/86oc+59wdNHLvzcML83/JjDFghP8c=; b=rv7LzmnbM0wfyyJepVcVdvD6H9hWYl4kedrrZ0O9gqc+aFA+bXhgbhu8I0XVMc9QWr deySv6QIYQOC0UcJOWAHonbU5ntnFQmXRE6p5muEiD1i6RtKf9ixtxCbc244H4/pbeNJ KkVl5x6c/Z9kdy8V+oKEQTw/ypUzIfY2IvzkAlH3RrDgTIztcqX1mp5Q0ihKJjmnPy/H oTtgBSjth7NCVXrJl7GUFNmZzj3pQaKfpqfHgmaY1IKUb78xFfqcwJ+mZoBvLEQWO2sJ XoRYKaX/p3qZzbSsTXYXMW1XYu+FgVXR8sFpdJDuY4a14j7ZmqT6pNBxnoc0isVq4Vuo ecEg== X-Gm-Message-State: AHPjjUh8p3b/lyICs8FfsmgEIBTJ+7/pn666XTSmMW8zKFpElpI+UD2h 39s7k1zFViFlPUz+NmY+huAEl7by X-Google-Smtp-Source: AOwi7QBTKZmhO802XM1+0WR0CPg035YLPTUEwArTjdswzLyzufubzVH46+6d4hBvl625oLt3LU/NLw== X-Received: by 10.223.187.74 with SMTP id x10mr13985113wrg.66.1506967926264; Mon, 02 Oct 2017 11:12:06 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id i50sm11664820wrf.84.2017.10.02.11.12.04 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 11:12:05 -0700 (PDT) From: Ben RUBSON Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: ZFS stalled after some mirror disks were lost Message-Id: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> Date: Mon, 2 Oct 2017 20:12:03 +0200 To: Freebsd fs Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 18:12:08 -0000 Hi, On a FreeBSD 11 server, the following online/healthy zpool : home mirror-0 label/local1 label/local2 label/iscsi1 label/iscsi2 mirror-1 label/local3 label/local4 label/iscsi3 label/iscsi4 cache label/local5 label/local6 A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk according to "zpool iostat", nothing on local disks (strange but I noticed that IOs always prefer iscsi disks to local disks). No write IOs. Let's disconnect all iSCSI disks : iscsictl -Ra Expected behavior : IO activity flawlessly continue on local disks. What happened : All IOs stalled, server only answers to IOs are made to its zroot pool. All commands related to the iSCSI disks (iscsictl), or to ZFS (zfs/zpool), don't return. Questions : Why this behavior ? How to know what happens ? (/var/log/messages says almost nothing) I already disconnected the iSCSI disks without any issue in the past, several times, but there were almost no IOs running. Thank you for your help ! Ben From owner-freebsd-fs@freebsd.org Mon Oct 2 18:15:33 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D476E23727 for ; Mon, 2 Oct 2017 18:15:33 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 25841717B0 for ; Mon, 2 Oct 2017 18:15:32 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x232.google.com with SMTP id m72so8367958wmc.0 for ; Mon, 02 Oct 2017 11:15:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=UeFmkcbursTrR2EQNB5Gbf5DgYjFo8neTqL9gshDn68=; b=UGBWoeI5oXAQxZeN6/Z+Uc1gk10MnDowQncaBl8iwiGdJlA1YOBl37REeXWjOfaiBd WD99BtORPPL1fr0Cci4SkWiFZ4UqwUliL4aeoePLTfOBaCtKAoIfyA0ox9JCs3iWUInw e3KBHB6dH0gvKa8ICP+Bm20gZ8/kVXJyh5Ahv9PyNq0CnwNzgkHSuC7DEOi8oAaXrXd/ tq4FY9kvCknUo2jUILhU56KzyscFB0BlFr9VFserAZ1gHNG5jm8IvNaTKg1h0I78JQVu dYxCfF6SMGDNmJ+HtBQrqd2X/oG20eKn4di9cXjpWqa8DoV8tZ5A07l70d+n0+dj0bMa Z9fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=UeFmkcbursTrR2EQNB5Gbf5DgYjFo8neTqL9gshDn68=; b=Y0lfDd7zl87ERrho2njRfhlXr5ImaMnu8n9q7E1tHdGVh1r4l+U6pkD02GFwlKMXWJ yytAY8C5rTXb8GaxIor0Bh5l0gi6Ea0MnEHSv4gNs2oS11RRA4ex8D1cBXTX2xSVNplC zK/L6CPHDONQjjYXF3AHjLjUMCEMxim8eF1X4WNV+h9k8PIgOAuTthm3ozatkLnPzVFa ylbSd3DrAokCDcBd6xDn0gFgkAbtYN2HvFtlArZF63vRY7OgHCsPEIzBcXyWQMP3Vmql xRRUsOVILQnkusC8YqSrhOPGtAR/LPcuyxBUCqpU+dwMAJnmZQw6A1ufuga5V/+ljmUZ Pecw== X-Gm-Message-State: AHPjjUj8RyRx7O8gXAf3htFTMSTipkz2GVMBgsWqF3DBCcKfYUnm+Bfn 1YWQPPN0o/pjahhsTWAKZka6YDcGS70= X-Google-Smtp-Source: AOwi7QCS/D3eiDKjqVnjGgHfXyFZeYZNOQw/CU/JtL6xJ+B7l5S4GSi/zSicWiCvEAt9QqpJ2tABZQ== X-Received: by 10.28.148.203 with SMTP id w194mr10636276wmd.91.1506968130721; Mon, 02 Oct 2017 11:15:30 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.1]) by smtp.gmail.com with ESMTPSA id n14sm6925840wrg.38.2017.10.02.11.15.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Oct 2017 11:15:29 -0700 (PDT) Subject: Re: ZFS stalled after some mirror disks were lost To: freebsd-fs@freebsd.org References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> From: Steven Hartland Message-ID: <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> Date: Mon, 2 Oct 2017 19:15:30 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 18:15:33 -0000 What does zpool status report when you have disconnected the iscsi targets? On 02/10/2017 19:12, Ben RUBSON wrote: > Hi, > > On a FreeBSD 11 server, the following online/healthy zpool : > > home > mirror-0 > label/local1 > label/local2 > label/iscsi1 > label/iscsi2 > mirror-1 > label/local3 > label/local4 > label/iscsi3 > label/iscsi4 > cache > label/local5 > label/local6 > > A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk > according to "zpool iostat", nothing on local disks (strange but I > noticed that IOs always prefer iscsi disks to local disks). > No write IOs. > > Let's disconnect all iSCSI disks : > iscsictl -Ra > > Expected behavior : > IO activity flawlessly continue on local disks. > > What happened : > All IOs stalled, server only answers to IOs are made to its zroot pool. > All commands related to the iSCSI disks (iscsictl), or to ZFS (zfs/zpool), > don't return. > > Questions : > Why this behavior ? > How to know what happens ? (/var/log/messages says almost nothing) > > I already disconnected the iSCSI disks without any issue in the past, > several times, but there were almost no IOs running. > > Thank you for your help ! > > Ben > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Oct 2 18:17:17 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16457E23918 for ; Mon, 2 Oct 2017 18:17:17 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 99C02719E2 for ; Mon, 2 Oct 2017 18:17:16 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22d.google.com with SMTP id u5so4475767wrc.5 for ; Mon, 02 Oct 2017 11:17:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=sFAbywXHJcT7ghnFkfVL6YLo3VuG3VRFWAPSrjd1T0s=; b=blyg8UR2vB5VTE+SPjIlzdjQfCeyh42LwxJtXKL0gY90tF6TxOeiJZm8VFuxTNmxbG vS0qQ1ugPid8ci1Z098QGPoX1BaikDuYYamSkxkX1zhTZnQdSWuO/emrMk3QJbgGArsH 9vSbKewUgncdRAM2dEEW+FKtwliW7wiYEAbbOBD5kERQ2MIhuxWAhXVWSobF9jV6z1Iz HjQSbZzFydzNhwOwUKXuok0QIa3jG4xIDash7iDBYLXSMinjIXu1eN7+lCx1tb8ITV32 wIkXcyRGJ41HKIAGuh7ZtxrZXZtZzG19CLQ1JLGB6pzlJxsi27ZrhM243Pg6Q4rdRyhi TBMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=sFAbywXHJcT7ghnFkfVL6YLo3VuG3VRFWAPSrjd1T0s=; b=n+7pAE+ZWUg5PYaYlTPYpyyyA+KOMc15/CX6fbm9KqmkJIQx9IGzr9FKr/cwRDiOTZ 5KmqrOyeTju22RW24DAjbqxp1xr9fGO0zb85mFXPalxtdzgJ9k46IZ38WLX8nyXlSija uJfwuLYPCq69cwqoiFTqUQY3+wY/tr7DiMnreZcHrSi7Vm427RW9W3Kl/qmiHsEgyMXG zZ3Jh97KEpbZLV4R+uL+Ipog7Yh70dvI3Bgoyo7isgzLxIbWcH+5Wfo0IRtN4rkV3xyg xn+QHkTGuKdKA5og4MEdhtkbtZzmtJ2ECR0suYxO4gG9tAUZHdNRHiKZPxSm7HSKO1Tb FQhw== X-Gm-Message-State: AMCzsaVY/z0jlUBx1thcnGbf3nCzzGl0Reg4IFSwD5GLkprUndDCBuc/ Rb5ftmee66H7sa1Q1gJ+87G7NIed X-Google-Smtp-Source: AOwi7QCi2Hc6SyUZVB0QSIkuq2ryOa3UDYUQ65WoclpgWl/8FiQU6vggQwcB+a3PAQ1BlqF/Ms+cvw== X-Received: by 10.223.184.246 with SMTP id c51mr4140523wrg.250.1506968234692; Mon, 02 Oct 2017 11:17:14 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id 55sm22398478wrw.60.2017.10.02.11.17.13 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 11:17:14 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> Date: Mon, 2 Oct 2017 20:17:13 +0200 Content-Transfer-Encoding: 7bit Message-Id: <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 18:17:17 -0000 Unfortunately the command stalls / does not return :/ > On 02 Oct 2017, at 20:15, Steven Hartland wrote: > > What does zpool status report when you have disconnected the iscsi targets? > > On 02/10/2017 19:12, Ben RUBSON wrote: >> Hi, >> >> On a FreeBSD 11 server, the following online/healthy zpool : >> >> home >> mirror-0 >> label/local1 >> label/local2 >> label/iscsi1 >> label/iscsi2 >> mirror-1 >> label/local3 >> label/local4 >> label/iscsi3 >> label/iscsi4 >> cache >> label/local5 >> label/local6 >> >> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >> according to "zpool iostat", nothing on local disks (strange but I >> noticed that IOs always prefer iscsi disks to local disks). >> No write IOs. >> >> Let's disconnect all iSCSI disks : >> iscsictl -Ra >> >> Expected behavior : >> IO activity flawlessly continue on local disks. >> >> What happened : >> All IOs stalled, server only answers to IOs are made to its zroot pool. >> All commands related to the iSCSI disks (iscsictl), or to ZFS (zfs/zpool), >> don't return. >> >> Questions : >> Why this behavior ? >> How to know what happens ? (/var/log/messages says almost nothing) >> >> I already disconnected the iSCSI disks without any issue in the past, >> several times, but there were almost no IOs running. >> >> Thank you for your help ! >> >> Ben >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Oct 2 18:28:57 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52D0BE23FA0 for ; Mon, 2 Oct 2017 18:28:57 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22b.google.com (mail-wr0-x22b.google.com [IPv6:2a00:1450:400c:c0c::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D536472767 for ; Mon, 2 Oct 2017 18:28:56 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22b.google.com with SMTP id u5so4498942wrc.5 for ; Mon, 02 Oct 2017 11:28:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=ZgBE7ErBDyHq8SNUozkeQYJ3wj/p/I7L9BwBooWZZiw=; b=TF8rSykoy3H8RCgGrDpwiJ9IRVubn94W5bFB2uAKIxT0GQ+2O7zxFCTvHTjTFw0YuE BJi5TMF5ulhNEJKAgk8NIW5ZONCKZARTHudooc3iI/bRlwMYmbWBlW30jT7G67yQDVPA meTdq72TmiR9nvRvKoQ6aoLBEmQIZ0f8xa8nHQBznS6lo1oCsTBqTc9U+PnFHySMMeTi YEfssQHKp88O/LVK6wEhs9TJiysfn493zIWdjSNoGocETg9ZxhSqo3cQGXS+w/g0cHju NdscIe/iS5Xx964/nXgtbWOsc51lVfAX17ZSzwaPApunWJootvN77X3Z+5fl3QwRQ0cg iDmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=ZgBE7ErBDyHq8SNUozkeQYJ3wj/p/I7L9BwBooWZZiw=; b=mqhqUwsH3NzvwIxIPz226QftN7BSpRXV2oeQKOhu3KC3M0LNRyFtbv1uSFamAqBvp6 C+nfShijMAfQtrZdxzo2yITsTOc0aFkAKOoVfEzRRizC/GVjIltr5k024b+r78dL4TVL 6Kns9z1duLTMGq5YQRTabIhAoE+IDQwfKQUyPgkb1gs7hEv56by9zNCWyogLZo7XUxKt ENByaRoJkmilBaRlvuLUh1M7V+8E6avdq/ob68Iod3TcGJW8oKART6gK++e8QFa5H0L2 8LK7e3TN1MgO/om+2+YFP9WT3jcwNmE4baBe5VnYcTrlu7awrKw2JBnPwnxJVAjDez9z sU4Q== X-Gm-Message-State: AHPjjUgnT6AJhFiIjLkyB5d7HndjiY3KNXhzWGOqRFCeXfrn136wi2wG eZGf5Jxd/LsI0gSyng8F1kxqkrkS X-Google-Smtp-Source: AOwi7QBk52w43x215Auh6L8mzlP6konZIPPNitfYBqaawqrmXw/4NDjyJeAKReR/ISb+JCBKz9yS1Q== X-Received: by 10.223.187.148 with SMTP id q20mr14595275wrg.34.1506968934981; Mon, 02 Oct 2017 11:28:54 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id w126sm10774349wme.25.2017.10.02.11.28.53 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 11:28:53 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> Date: Mon, 2 Oct 2017 20:28:52 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <953DD379-C03A-4737-BAD8-14BB2DB4AB05@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 18:28:57 -0000 Before disconnecting the targets the pool was online without any issue. > On 02 Oct 2017, at 20:17, Ben RUBSON wrote: >=20 > Unfortunately the command stalls / does not return :/ >=20 >> On 02 Oct 2017, at 20:15, Steven Hartland = wrote: >>=20 >> What does zpool status report when you have disconnected the iscsi = targets? >>=20 >> On 02/10/2017 19:12, Ben RUBSON wrote: >>> Hi, >>>=20 >>> On a FreeBSD 11 server, the following online/healthy zpool : >>>=20 >>> home >>> mirror-0 >>> label/local1 >>> label/local2 >>> label/iscsi1 >>> label/iscsi2 >>> mirror-1 >>> label/local3 >>> label/local4 >>> label/iscsi3 >>> label/iscsi4 >>> cache >>> label/local5 >>> label/local6 >>>=20 >>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >>> according to "zpool iostat", nothing on local disks (strange but I >>> noticed that IOs always prefer iscsi disks to local disks). >>> No write IOs. >>>=20 >>> Let's disconnect all iSCSI disks : >>> iscsictl -Ra >>>=20 >>> Expected behavior : >>> IO activity flawlessly continue on local disks. >>>=20 >>> What happened : >>> All IOs stalled, server only answers to IOs are made to its zroot = pool. >>> All commands related to the iSCSI disks (iscsictl), or to ZFS = (zfs/zpool), >>> don't return. >>>=20 >>> Questions : >>> Why this behavior ? >>> How to know what happens ? (/var/log/messages says almost nothing) >>>=20 >>> I already disconnected the iSCSI disks without any issue in the = past, >>> several times, but there were almost no IOs running. >>>=20 >>> Thank you for your help ! >>>=20 >>> Ben >>>=20 >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>=20 >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 From owner-freebsd-fs@freebsd.org Mon Oct 2 18:41:59 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B930E244CB for ; Mon, 2 Oct 2017 18:41:59 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x232.google.com (mail-wr0-x232.google.com [IPv6:2a00:1450:400c:c0c::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A606F72EFA for ; Mon, 2 Oct 2017 18:41:58 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x232.google.com with SMTP id t76so4488651wrc.3 for ; Mon, 02 Oct 2017 11:41:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=O4GgAzQkqKJLVOcRiQ1Q0nO8AZLgq1TnDT9kiulgbNE=; b=VzS3DMUkxI2YWT2KtzLMH5PQuicJcfr2gDXnU+V8jDNVwJY5yacI0ETrRhPnKVoElk mpQxZFFROePDx/7klEmZjwqel5NbR7tdVUKw4ZCbwKyDsgXRtkdcfH3J8XM540IA/hNg v+VTKRUQs7holfXLmtyxfE0qIFuy3BzEDgeGzpJejeLQzVvTT83h1PAmG4FAgjTOvKhS pOXhlqgg81GEI0avvPMLHZQ/x1ya1fYEGCgBXzOGVNeg4aV0uIuWjI0rxfxJst2WFt+C PYsexULGzCnIHZy+q8d1cYaOf3VdccZvCKy/Bim7HRHZxg3ybRiVMmXFL62xVqWmqtV0 aR/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=O4GgAzQkqKJLVOcRiQ1Q0nO8AZLgq1TnDT9kiulgbNE=; b=iAsBD5B5YF+snMu4yoVV1QZcLVyHYYCp0yxcwHvBp/b4ZQLXOOaks23HEPB+INXj2b 8d5wT1FggykKjleM2yRoJUTPpEi0/WksaHUIlSXFqgfi4757r5VdxCtvSIPF7ciEKj79 q+EGwdWTZnJjLgLKWidzabqHPbqlCq1LYCWYGuIYLvv9pBRstbYiVys2eOYSdRdvKdNa XbP58dK85XIwG64SQoLtiheZBisGrExs3nPnojlZZoZs9VR2UZ7MmID3tfDoRMHwdmkh /rPIhFBOX27tJD6mC2olmj1xiVO/Y0E4iIFLhTKT68d7UT5Eh04YOu/ywGUmsVZepoJG pRbQ== X-Gm-Message-State: AHPjjUj1ieHkzjRG/H4DxyXSCDk7iu5c9obkBaCoQgxFlUP4Xc76lDgE 82JSN1FrUoZY7hbeUlxg3hERAxHCduM= X-Google-Smtp-Source: AOwi7QCNPWjHCK5FlybMVv4vevauYYDbqD+GbLDLFDPDQxoM9mxasB07RmvuHznlaYg9FDiJYSXQgg== X-Received: by 10.223.195.110 with SMTP id e43mr13548671wrg.219.1506969715984; Mon, 02 Oct 2017 11:41:55 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.1]) by smtp.gmail.com with ESMTPSA id n57sm16710146wrn.29.2017.10.02.11.41.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Oct 2017 11:41:54 -0700 (PDT) Subject: Re: ZFS stalled after some mirror disks were lost To: freebsd-fs@freebsd.org References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> <953DD379-C03A-4737-BAD8-14BB2DB4AB05@gmail.com> From: Steven Hartland Message-ID: <4f725113-bac3-64bb-9858-690811e73153@multiplay.co.uk> Date: Mon, 2 Oct 2017 19:41:56 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <953DD379-C03A-4737-BAD8-14BB2DB4AB05@gmail.com> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 18:41:59 -0000 I'm guessing that the devices haven't disconnected cleanly so are just stalling all requests to them and hence the pool. I'm not that familiar with iscsi, does it still show under under camcontrol or geom? Does iscsid have any options on how to treat failed devices? On 02/10/2017 19:28, Ben RUBSON wrote: > Before disconnecting the targets the pool was online without any issue. > >> On 02 Oct 2017, at 20:17, Ben RUBSON wrote: >> >> Unfortunately the command stalls / does not return :/ >> >>> On 02 Oct 2017, at 20:15, Steven Hartland wrote: >>> >>> What does zpool status report when you have disconnected the iscsi targets? >>> >>> On 02/10/2017 19:12, Ben RUBSON wrote: >>>> Hi, >>>> >>>> On a FreeBSD 11 server, the following online/healthy zpool : >>>> >>>> home >>>> mirror-0 >>>> label/local1 >>>> label/local2 >>>> label/iscsi1 >>>> label/iscsi2 >>>> mirror-1 >>>> label/local3 >>>> label/local4 >>>> label/iscsi3 >>>> label/iscsi4 >>>> cache >>>> label/local5 >>>> label/local6 >>>> >>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >>>> according to "zpool iostat", nothing on local disks (strange but I >>>> noticed that IOs always prefer iscsi disks to local disks). >>>> No write IOs. >>>> >>>> Let's disconnect all iSCSI disks : >>>> iscsictl -Ra >>>> >>>> Expected behavior : >>>> IO activity flawlessly continue on local disks. >>>> >>>> What happened : >>>> All IOs stalled, server only answers to IOs are made to its zroot pool. >>>> All commands related to the iSCSI disks (iscsictl), or to ZFS (zfs/zpool), >>>> don't return. >>>> >>>> Questions : >>>> Why this behavior ? >>>> How to know what happens ? (/var/log/messages says almost nothing) >>>> >>>> I already disconnected the iSCSI disks without any issue in the past, >>>> several times, but there were almost no IOs running. >>>> >>>> Thank you for your help ! >>>> >>>> Ben >>>> >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Oct 2 18:44:50 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CB31E246B1 for ; Mon, 2 Oct 2017 18:44:50 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-io0-x231.google.com (mail-io0-x231.google.com [IPv6:2607:f8b0:4001:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 242F77322E for ; Mon, 2 Oct 2017 18:44:50 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: by mail-io0-x231.google.com with SMTP id w94so5561887ioi.7 for ; Mon, 02 Oct 2017 11:44:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=1+daKYhRxeGmlbIrfXype4Y3Oj1rf/okbs8Eq5jC6Uw=; b=g+SJhgKphswz8iafLpUAHddNg2hg9FUJEj8q3lDSvWrUaAF6BPmAjVu/8ehYKgqKQv dwNyME+dwpQa1TfAPs70Bhg9XDx0z7efuF0vobvKN2XMgPXM4WiERe0mkqZFnsIcHwau q/TfF6yV7SrN58Lh3w0Ge/IIhBI+mVUzVrZPiHMW2E7kL2YvBHfFm6Kg6OxDm0o9w8Mf 9z52w2q3+YJpEWsr4l1W+kjJZLMPVO3zhhy6SZ3mpquCoQ4fE/BloZVOiv08Z/Ov8LR1 wC2BqSHQRlrZDTcWuGWJerz6MRp7iK2pWDHf0IgHbco1nWH5YzGZnZh0CawdONscCgQW l2UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=1+daKYhRxeGmlbIrfXype4Y3Oj1rf/okbs8Eq5jC6Uw=; b=s/yXzVxoOLgx79X7daK3aFBKWtaBCWbgyAPEStwNXbWXbAU106ZTLY1TMF0fBBc4m6 7WErD0DKWdARA4uj7gdGPhg6IXqGXYQtL3PUgC272rXt/HSvv0c1mNgttoZTIzXWc/AT 4pRaO9XTS1mrtPcvK7PNG5fP8flzLKm5SnO/VxsdSzgQQNWLPE4b9qB3819ZL4jCEdC6 tJ+lx1fTgTZMncc7ZFJMc/ApJlQ8ls550UrvD6oVjOae5rm5PgbDL5Ktg15asf8rHVrZ j4QcK2liY2qk09nvh1HoWkk+hCj2oywDW/OLcgnvWVQYRBWBllsEMCCuIQSC9gQ92oVb rOGA== X-Gm-Message-State: AMCzsaVVpOImtq5lt9KHr7RLkv7BV5Uw0JOw99vVmRZlIJT6LlG3QuuX H8SrEgi5OlfvJjdcElOni+7/6/PHOcAnYhrBLP1+/w== X-Google-Smtp-Source: AOwi7QBNR4qOnfih2m2y/YoONGkzOj11uPgOWFeXRt7W+u19//bPXqO1S7YwV+LIHAf/F4nbQnzTExUayZGvYeqIA9o= X-Received: by 10.107.69.2 with SMTP id s2mr79265ioa.16.1506969889455; Mon, 02 Oct 2017 11:44:49 -0700 (PDT) MIME-Version: 1.0 Received: by 10.2.145.141 with HTTP; Mon, 2 Oct 2017 11:44:48 -0700 (PDT) In-Reply-To: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> From: Adam Vande More Date: Mon, 2 Oct 2017 13:44:48 -0500 Message-ID: Subject: Re: ZFS stalled after some mirror disks were lost To: Ben RUBSON Cc: Freebsd fs Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 18:44:50 -0000 On Mon, Oct 2, 2017 at 1:12 PM, Ben RUBSON wrote: > Hi, > > On a FreeBSD 11 server, the following online/healthy zpool : > > home > mirror-0 > label/local1 > label/local2 > label/iscsi1 > label/iscsi2 > mirror-1 > label/local3 > label/local4 > label/iscsi3 > label/iscsi4 > cache > label/local5 > label/local6 > > A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk > according to "zpool iostat", nothing on local disks (strange but I > noticed that IOs always prefer iscsi disks to local disks). > No write IOs. > > Let's disconnect all iSCSI disks : > iscsictl -Ra > > Expected behavior : > IO activity flawlessly continue on local disks. > Perhaps I'm misunderstanding your setup, but my expected behavior would be exactly what you see. I think you'd need something more along the lines of: home mirror label/local1 label/iscsi1 mirror label/local2 label/iscsi2 etc... -- Adam From owner-freebsd-fs@freebsd.org Mon Oct 2 18:46:55 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3347FE2476D for ; Mon, 2 Oct 2017 18:46:55 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 5E5AD7332F for ; Mon, 2 Oct 2017 18:46:53 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id VAA10321; Mon, 02 Oct 2017 21:46:52 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dz5jw-000Jaz-5O; Mon, 02 Oct 2017 21:46:52 +0300 Subject: Re: ZFS stalled after some mirror disks were lost To: Ben RUBSON , Freebsd fs References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> From: Andriy Gapon Message-ID: <5d3e1f0d-c618-afa4-7e52-819c9edf30c9@FreeBSD.org> Date: Mon, 2 Oct 2017 21:45:51 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 18:46:55 -0000 On 02/10/2017 21:17, Ben RUBSON wrote: > Unfortunately the command stalls / does not return :/ Try to take procstat -kk -a. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Mon Oct 2 19:10:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51AC9E24FFD for ; Mon, 2 Oct 2017 19:10:07 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D63E673EA7 for ; Mon, 2 Oct 2017 19:10:06 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id t69so12454508wmt.2 for ; Mon, 02 Oct 2017 12:10:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=I7Xjhiv7W1j5BOq6pc4eyND9LqaWGzTBa1+ylHQw2PY=; b=DCsn9VBPf+HKJGjpB0QGc+Wh1Zk3fbZMdZ+kGh5PMxqickSlQISQdAI9mL43U4lexH RBUMcaQJDDoZxBY0c1aeiG15JNSuMMg5/7EZP4sLjQuhtGy69T+fEI4gOoBuWeHvL/G8 8I+VohDFdEIt88L1HHUfM741QFGy8pd0Lw5Y2TlUeZXE09RjjThlv8Hc2mz+2RwBEFAs K9wvfcrgq3YIV42yK7H39+LhpzpNzf5KB5Cc8bFZMbTYO1pKMV0yF1D9SlPLSbNBUHRX UgKvly51A3eaZWzDvLFH1ulvzCj1lbSz7aRXuyd1lVG4nuMSEdVKqca8i14XcGQkGUOb CLDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=I7Xjhiv7W1j5BOq6pc4eyND9LqaWGzTBa1+ylHQw2PY=; b=fgPEAY/SQGr74hfEAk+8ehHG6fzdChh1XuFSQi9tuM5bjlF2yJcJTsujRcQHn4AT+g e9XOSbXTaCOJbLV0SF3iSgJpWb/KjQI4QQEg3NjoSLqucBVtaVzIWjSuh8+x6LZ35EiD cp4RrPp4x9pK++vEwZfLAEIDyC7dPixrjKQ6v35WGCWqyPeLZYkrtk454Cx8RYy7YeRV D+0x5THKfqyY+PgcgfNsMZ1M+EfAt0H0/9KtDlJmBPU4eE3DT7WwyHVHWaE8HfKQ2Hnz OyoRpuwYPL/n3RraE7kgRwllK2i4cVzcvCA67+mvwHQyzoTUKXXrSWrJSnvitPBrFhnj mLPA== X-Gm-Message-State: AMCzsaXyuCICUmGi1exlw+iLND6a3Rxmm3uFoNJkrYcKCq+XghjzH7ow WZazy/NmTYl6aemo76hy5qPLwbLR X-Google-Smtp-Source: AOwi7QC6604K+GZ7s34JRiIxfL8x7rxlTacFHGg2HEp1d9BX9xaBmZuGhXahRrym4V8qZOEvwpsgvQ== X-Received: by 10.28.17.207 with SMTP id 198mr10871399wmr.38.1506971405067; Mon, 02 Oct 2017 12:10:05 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id q81sm9074721wmd.27.2017.10.02.12.10.04 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 12:10:04 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: <4f725113-bac3-64bb-9858-690811e73153@multiplay.co.uk> Date: Mon, 2 Oct 2017 21:10:03 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <54AD0000-AF0B-4682-9047-6E6C1B82506C@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> <953DD379-C03A-4737-BAD8-14BB2DB4AB05@gmail.com> <4f725113-bac3-64bb-9858-690811e73153@multiplay.co.uk> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 19:10:07 -0000 > On 02 Oct 2017, at 20:41, Steven Hartland = wrote: >=20 > I'm guessing that the devices haven't disconnected cleanly so are just = stalling all requests to them and hence the pool. I even tried to ifconfig down the network interface serving the iscsi = targets, it did not help. > I'm not that familiar with iscsi, does it still show under under = camcontrol or geom? # geom disk list (...) Geom name: da13 Providers: 1. Name: da13 Mediasize: 3999688294912 (3.6T) Sectorsize: 512 Mode: r1w1e2 wither: (null) Geom name: da15 Providers: 1. Name: da15 Mediasize: 3999688294912 (3.6T) Sectorsize: 512 Mode: r1w1e2 wither: (null) Geom name: da16 Providers: 1. Name: da16 Mediasize: 3999688294912 (3.6T) Sectorsize: 512 Mode: r1w1e2 wither: (null) Geom name: da19 Providers: 1. Name: da19 Mediasize: 3999688294912 (3.6T) Sectorsize: 512 Mode: r1w1e2 wither: (null) # camcontrol devlist // does not show the above disks > Does iscsid have any options on how to treat failed devices? iSCSI has some tuning regarding how to treat failing devices, and I did = it : kern.iscsi.ping_timeout=3D5 kern.iscsi.iscsid_timeout=3D5 kern.iscsi.login_timeout=3D85 kern.iscsi.fail_on_disconnection=3D1 However, as I disconnected the targets from the server hosting the = zpool, they should not have been needed. From owner-freebsd-fs@freebsd.org Mon Oct 2 19:13:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47F1BE25262 for ; Mon, 2 Oct 2017 19:13:38 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22a.google.com (mail-wr0-x22a.google.com [IPv6:2a00:1450:400c:c0c::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CE84A74293 for ; Mon, 2 Oct 2017 19:13:37 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22a.google.com with SMTP id p10so2946280wrc.6 for ; Mon, 02 Oct 2017 12:13:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=+MBkVTVtmQJJmqLkAEpQT3I27GHboByz4jquk8zkoCg=; b=Dd/J9K3V9XzqcpNbzsMD5NrydIlrz955Hoqg5sxmJxGfrKdywm0lJ8G+QOhlgiJvQt RfCeRcPohb9KXi8M5p4xumF9n0g5Ugik4lTS+NvsOakQR0zagWFfReYYTBjfmg9GkXr5 1o3wSwHZHovJ0ccB/S3DIc9XHxIfyO9bdgD4oDac41BPIF+ODWgkyO9WXn9uvwPmzYQD BKwSntrgn4o5OpWXTV06NK3dOibC4ePClVb21sMvoybM1Cxqx45oW8+TXsQk1CKcr1aC vQZlG1yvfbC/lR6V0IWAViPTOx8gPyCGNIO371mceB0laXjLxu9E6wEClEh4INevA9Ak tKSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=+MBkVTVtmQJJmqLkAEpQT3I27GHboByz4jquk8zkoCg=; b=cNtafRpTUtAWws6XB8YNW6N6lkUq5iVf9RUUrO9KQnb9K17i24q+W4I/dgFJ4YBSgf FLdfCWspfVUFMIunGjLhuNNxnbEqIsmhqwZARILJlHk2TSzt3x2G/Z2Xw/W3P7dapOyU agQVuLBWgheoJxG2CIVTQiWJdXVvHuXZOcoOX92xmgb/ycPM6GF5KyhdzuzSoxsWJvob MHbA8j3RHOhBfQU628PO099dwrsjsfhf4ArINFt2pfFu7s1OgjmHqn6El/dy4qA0srAD ERkLRN8t4ZIsQQ2z7gCv36SiklX6dBJqYfV0nr1vr6H+bhx5T4WYtcnWXugFaLRx+4XQ kp8Q== X-Gm-Message-State: AHPjjUgPdl40+jIlJK1EaJuAmXYUcQ0e3JIg3Bo/Ya9DmNbe71eU1OVj 9r4sYD6UzKVywtrYQp4rh5sDv8RY X-Google-Smtp-Source: AOwi7QCVbp3dCDBL+wn6WC4YfacNn2pUbflvzprOMsmyfWsMj7UBvPU7iC8caqJC8jwScCbvsvdnaw== X-Received: by 10.223.133.99 with SMTP id 90mr17680869wrh.63.1506971615757; Mon, 02 Oct 2017 12:13:35 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id p6sm6612538wrd.10.2017.10.02.12.13.33 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 12:13:34 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: <5d3e1f0d-c618-afa4-7e52-819c9edf30c9@FreeBSD.org> Date: Mon, 2 Oct 2017 21:13:33 +0200 Content-Transfer-Encoding: 7bit Message-Id: <48D23270-1811-4E09-8AF2-5C0FEC2F9176@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> <5d3e1f0d-c618-afa4-7e52-819c9edf30c9@FreeBSD.org> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 19:13:38 -0000 > On 02 Oct 2017, at 20:45, Andriy Gapon wrote: > > On 02/10/2017 21:17, Ben RUBSON wrote: >> Unfortunately the command stalls / does not return :/ > > Try to take procstat -kk -a. Thank you Andriy for your answer. Here is the procstat output : https://benrubson.github.io/zfs/procstat01.log Ben From owner-freebsd-fs@freebsd.org Mon Oct 2 19:16:13 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6EA2E25349 for ; Mon, 2 Oct 2017 19:16:13 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 58CB374355 for ; Mon, 2 Oct 2017 19:16:13 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22d.google.com with SMTP id b21so4767757wrg.7 for ; Mon, 02 Oct 2017 12:16:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=3UwtB7WqZlN1pNLFVvcoI3qM0qz8MSfx16i5RsyGC9A=; b=dDPvmJ49isQHSea7TnVKHScHUS6I+LY81Eu4cDm/sO9djAC6A8+PSyHA216dEV7cuH LxzW9CGrCnCjK7OPy7Bd1uSeiAn29jN2K2qhZ82IUcabNsknrNxWpb8a8jQkhGAGWPQu wgYoCVJgLB8dvkHLWBBO6bJNLGtKj/fd6eG3pNz7Fhkfr9dKkoeuvBINZRwTp7e61/bO H+0nVOM64Jy1f1kULs9jj3M0ug7vggvekAnlsdfD/gnuGAwL+GGyDCkBX2codiCH1ibd Uy2+jpTJnkXuFsxE0gLdHi5nQqfYwM99CRHMZ0+kgJ9shC1c+68N5CUVL+OCPSEOOt+k e49Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=3UwtB7WqZlN1pNLFVvcoI3qM0qz8MSfx16i5RsyGC9A=; b=rkEigBC6bMkVj5BtesmA7qkMphMntXnSbjeW/LWFhYTwmKri/nVo6PT9S99Xi2GLhw WVnQsHh0UK1mRgIvSARO03Gxc/yx+PRc6dxf2+mn7EsIA+zgiHEaaojuYJ0JT4KUht9+ vjT2ST88JtQnqr1KMBZVWdiR77p71CLZl0WPntHjB6+MS85YfdBCBNpTJRe3nIC1Scda Nd/enp16m2Zc2TuWIlYnMJ9dYtcQAMDP/zFA2/dZA1SXjUM0r8iDQsYvRIxJc5/U+fbA iMX0DPVtSWH3a8LOQ4DeOXi7/IrkzXqK6IsbdCmrrgWdWVHSMelN+VX/yA4q1B9m/8Ag aVUw== X-Gm-Message-State: AMCzsaU7s9W++dEMOp/vxKyYpRzzYPXJxHQodzf4gTeCJBR4AmnRwatT CXo97RwvAUq80G/y+u+Q6MJI9dtg X-Google-Smtp-Source: AOwi7QCx0qxmJQ2z3icbzYCWRlLkOq7TuYK4On73GfYUDTIBlwWkVcztQCzAA+uUaWCovhpHLiTGhA== X-Received: by 10.223.192.138 with SMTP id d10mr5005143wrf.6.1506971771746; Mon, 02 Oct 2017 12:16:11 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id w18sm8919756wra.61.2017.10.02.12.16.10 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 12:16:11 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: Date: Mon, 2 Oct 2017 21:16:10 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 19:16:13 -0000 > On 02 Oct 2017, at 20:44, Adam Vande More = wrote: >=20 >> On Mon, Oct 2, 2017 at 1:12 PM, Ben RUBSON = wrote: >> Hi, >>=20 >> On a FreeBSD 11 server, the following online/healthy zpool : >>=20 >> home >> mirror-0 >> label/local1 >> label/local2 >> label/iscsi1 >> label/iscsi2 >> mirror-1 >> label/local3 >> label/local4 >> label/iscsi3 >> label/iscsi4 >> cache >> label/local5 >> label/local6 >>=20 >> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >> according to "zpool iostat", nothing on local disks (strange but I >> noticed that IOs always prefer iscsi disks to local disks). >> No write IOs. >>=20 >> Let's disconnect all iSCSI disks : >> iscsictl -Ra >>=20 >> Expected behavior : >> IO activity flawlessly continue on local disks. >=20 > Perhaps I'm misunderstanding your setup, but my expected behavior = would be exactly what you see.=20 Unfortunately, what I see is the following quoted : >> What happened : >> All IOs stalled, server only answers to IOs are made to its zroot = pool. >> All commands related to the iSCSI disks (iscsictl), or to ZFS = (zfs/zpool), >> don't return. (and I would have expected the IO activity to flawlessly continue) From owner-freebsd-fs@freebsd.org Mon Oct 2 19:30:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DCFFE25B5D for ; Mon, 2 Oct 2017 19:30:07 +0000 (UTC) (envelope-from SRS0=HbBg=BB=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C4FC574CC5 for ; Mon, 2 Oct 2017 19:30:06 +0000 (UTC) (envelope-from SRS0=HbBg=BB=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id D2AFA28417; Mon, 2 Oct 2017 21:29:57 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 2363428411; Mon, 2 Oct 2017 21:29:57 +0200 (CEST) Subject: Re: ZFS stalled after some mirror disks were lost To: Adam Vande More , Ben RUBSON Cc: Freebsd fs References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <59D293B4.2020702@quip.cz> Date: Mon, 2 Oct 2017 21:29:56 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:42.0) Gecko/20100101 Firefox/42.0 SeaMonkey/2.39 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 19:30:07 -0000 Adam Vande More wrote on 10/02/2017 20:44: > On Mon, Oct 2, 2017 at 1:12 PM, Ben RUBSON wrote: > >> Hi, >> >> On a FreeBSD 11 server, the following online/healthy zpool : >> >> home >> mirror-0 >> label/local1 >> label/local2 >> label/iscsi1 >> label/iscsi2 >> mirror-1 >> label/local3 >> label/local4 >> label/iscsi3 >> label/iscsi4 >> cache >> label/local5 >> label/local6 >> >> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >> according to "zpool iostat", nothing on local disks (strange but I >> noticed that IOs always prefer iscsi disks to local disks). >> No write IOs. >> >> Let's disconnect all iSCSI disks : >> iscsictl -Ra >> >> Expected behavior : >> IO activity flawlessly continue on local disks. >> > > Perhaps I'm misunderstanding your setup, but my expected behavior would be > exactly what you see. I think you'd need something more along the lines of: > > home > mirror > label/local1 > label/iscsi1 > mirror > label/local2 > label/iscsi2 > etc... The OP has four way mirror. It is supposed to work even if 3 devices are missing. Just 1 device should be enough. Miroslav Lachman From owner-freebsd-fs@freebsd.org Mon Oct 2 19:36:24 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD855E25D96 for ; Mon, 2 Oct 2017 19:36:24 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-vk0-x22f.google.com (mail-vk0-x22f.google.com [IPv6:2607:f8b0:400c:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 88CA874FDB for ; Mon, 2 Oct 2017 19:36:24 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: by mail-vk0-x22f.google.com with SMTP id k191so1481752vke.7 for ; Mon, 02 Oct 2017 12:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=Wf6u6QnyUs7jwONWAx1HGwDu1xFyCmMFl7GPrlBfMoM=; b=pp6tkicUukxkZnm6lBiji0rM6/dxLqjjJ3w+4MRoViYCVMZLbD2dc1rZn1IOX5M51u aKZ/+hRslZZu/J+OqSH0wr3qz1uE24j8CRa11rmB3CVzRlZe5Nl+bdDUBSbRptrTeqkG HYsFdB5H7tboOuem/lUg+g72AIZIyPftuFEG3hCcwomsTN68InHs/hcPpSSzyQv/ddfb ohxSFMbogt9a9RHJuSn4624RLZEKFbQn/06qEGCQZKvemOvRfmMqC6ge4YgQ4Wl4Hza+ 5voC68SarT95V0ewVL4UweilbyFWkSAjXLt9YOuNMnjRSSa9NgKqxWjvwW2+GiO4OPu+ 0LmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Wf6u6QnyUs7jwONWAx1HGwDu1xFyCmMFl7GPrlBfMoM=; b=t6EawYk3IAhAJI49rX1TiHMeJbQtkoNnQYJld2T8/0MaYzxcfqhQB1DfPXBqgw1eNB 7GqAFvAiaDBfczrcPkU6zrJtBEYgD6kt8iB6L3iaSkMi0ETrdZ8fA4Nz6HxeHMYZJmPl zUkkwARBEWkI1S2/90Z5ijzvZQ6d8rMC/aLRRlz9VAOXhw1zBMt3zg/PHlqP+p0BWpBN JC4RGqXjlxkOu/qyJ+9++3WcE+vRpk1vrwII1Qw/tsEPMeqc6VEK+EEv7etZuo+hERfl lCAz1cpLzizIjiG0A8/RIn2paY6gjRvKdgIF7hNMIboZ0ryNIc6mOeSftz20ZyPo+n8n oYyw== X-Gm-Message-State: AHPjjUiZAWHMyW58t/m32Lx9L0AjjG6bJgqCydKuz5kEzQPlFohTA1ko JQoTlTDQ9khZnVWTH5ovmg8B36/snpbckXcijirLKQ== X-Google-Smtp-Source: AOwi7QCDQIwqHlbQ3J+0S1Dhb9xoOtwOXi5TomvwwD8HUBYDFpZsfIR2Sy1tynQx0hMymxdLhAkmAIMjNoEPsBLWRAM= X-Received: by 10.31.160.14 with SMTP id j14mr9146973vke.172.1506972982964; Mon, 02 Oct 2017 12:36:22 -0700 (PDT) MIME-Version: 1.0 Received: by 10.159.50.129 with HTTP; Mon, 2 Oct 2017 12:35:42 -0700 (PDT) From: grarpamp Date: Mon, 2 Oct 2017 15:35:42 -0400 Message-ID: Subject: dd: vm_fault: pager read error To: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 19:36:24 -0000 11.0 release amd64 r306420 kern.geom.debugflags=0 (unmodified) dd if=/dev/zero of=/dev/ada0s1 seek=2048 count=1 bs=1m 1+0 records in 1+0 records out 1048576 bytes transferred reboot: Device not configured, and for any other uncached access to the filesystem in ada0s1a, note / is on s1a, dd is past that. vnode_pager_generic_getpages_done: I/O read error 5 vm_fault: pager read error, pid 1 (init) HW reset and all layout, filesystems, and data are fine. Repeatable. Also, echo '' | dd of=/dev/ada0s1 seek=2048 count=1 bs=1m conv=sync does get written and is readable upon reboot, dd if=/dev/ada0s1 skip=2048 count=1 bs=1m ' wherein that read does not trigger the fault. All the offsets and sizes add up sequentially, no overlap, the relavant portions are below, disk is <~= 250G, gpart, fdisk, boot0cfg, bsdlabel all concur without error. What am I overlooking, or is this kernel behaviour a bug? => 63 x ada0 MBR (XG) 63 1 - free - (512B) 64 8388608 ada0s1 freebsd [active] (4.0G) 8388672 109051904 ada0s2 freebsd (52G) => 0 8388608 ada0s1 BSD (4.0G) 0 4194304 ada0s1a freebsd-ufs (2.0G) / 4194304 4194304 - free - (2.0G) From owner-freebsd-fs@freebsd.org Mon Oct 2 19:47:12 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15041E25FEE for ; Mon, 2 Oct 2017 19:47:12 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22c.google.com (mail-wm0-x22c.google.com [IPv6:2a00:1450:400c:c09::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B39457538E for ; Mon, 2 Oct 2017 19:47:11 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22c.google.com with SMTP id m72so12668856wmc.1 for ; Mon, 02 Oct 2017 12:47:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=y4w5vFIrV/gh5rGVfnRA+yk973x633mqcScwQpwVqlc=; b=BiFKBpa8l0Q5bCjMefCv3ECBpdtEG8m7EZiNS8D0T+77I0PUR4L9OfkxwgsK3KkcVt 3tBfgU6DnK/j3UJNO2qQO4mI2Rpm2b8R31YpkGILTXM18iLCwjawT0YRTPIonulaf6ab lfRphq2s0m+BjEbhA3p4x3dv3/czEzgzE34VTeN6X9Jb6vPreSgVDIAojEVnKPzGmuSH T8va6SpnL1W3J50Bs2jYJnMcrAvf1aQT+ifD0c7HCI/K8vvbQTxti8QrM11AuU/ipgj2 3x0G60/ts07eOest9ytDpPb1fOCJoTenep9G4GU9QA2nT5P8ThmVbEQS1oNgWkYJqOtr GIxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=y4w5vFIrV/gh5rGVfnRA+yk973x633mqcScwQpwVqlc=; b=AN2IJPP5/PPZ2yzfYNt+7zeaefg86t2WOHiZXGefQTLOV4leQmU4Iy4rKn6vix6u2E GPrLeD+z5ibeIof5dw95KznvJhBMmzoK5jDYXZUPhwiMgm6RLD18Z3bXhgkWb/GqlbJi jHXXiQwpVClGx7FikDfwKjjFM5iW4vaae/6bYlh7Odwe/hU99oMRiPrAaJjhA3wZER0g Ob3nAbX0dgB6oF3ZYqgMNuAML8+4U4l8w2wRN1Dr9fEumr/0wOXvrkqV3yAIx914Rtey R3FLKiUm17MAHcvK1LFi0JD3ktfuQcu171Og6Rmbrdj9fvIbtHddNcSAPcNeT6G3dPV4 /pgA== X-Gm-Message-State: AMCzsaVzD6KAAfrz5TMA7BBWmy/8m+gAPBvefhwEd1/KBZYSY5izNYp7 eo/hIJkd9iMiW1tB5OxdkxO22IEv3SQ= X-Google-Smtp-Source: AOwi7QCzee8K14g8fr9E91e8CK0vrMP05qHWU/IVDtNxh8iS/9TanqGMPxjzmVrQS16da/MB33vcUQ== X-Received: by 10.28.7.79 with SMTP id 76mr9159699wmh.45.1506973629342; Mon, 02 Oct 2017 12:47:09 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.1]) by smtp.gmail.com with ESMTPSA id a19sm12933744wra.64.2017.10.02.12.47.07 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Oct 2017 12:47:07 -0700 (PDT) Subject: Re: ZFS stalled after some mirror disks were lost To: freebsd-fs@freebsd.org References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> <953DD379-C03A-4737-BAD8-14BB2DB4AB05@gmail.com> <4f725113-bac3-64bb-9858-690811e73153@multiplay.co.uk> <54AD0000-AF0B-4682-9047-6E6C1B82506C@gmail.com> From: Steven Hartland Message-ID: <7fb4c99b-f3a0-1dda-691c-35f25769ed5c@multiplay.co.uk> Date: Mon, 2 Oct 2017 20:47:09 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <54AD0000-AF0B-4682-9047-6E6C1B82506C@gmail.com> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 19:47:12 -0000 On 02/10/2017 20:10, Ben RUBSON wrote: >> On 02 Oct 2017, at 20:41, Steven Hartland wrote: >> >> I'm guessing that the devices haven't disconnected cleanly so are just stalling all requests to them and hence the pool. > I even tried to ifconfig down the network interface serving the iscsi targets, it did not help. > >> I'm not that familiar with iscsi, does it still show under under camcontrol or geom? > # geom disk list > (...) > Geom name: da13 > Providers: > 1. Name: da13 > Mediasize: 3999688294912 (3.6T) > Sectorsize: 512 > Mode: r1w1e2 > wither: (null) > > Geom name: da15 > Providers: > 1. Name: da15 > Mediasize: 3999688294912 (3.6T) > Sectorsize: 512 > Mode: r1w1e2 > wither: (null) > > Geom name: da16 > Providers: > 1. Name: da16 > Mediasize: 3999688294912 (3.6T) > Sectorsize: 512 > Mode: r1w1e2 > wither: (null) > > Geom name: da19 > Providers: > 1. Name: da19 > Mediasize: 3999688294912 (3.6T) > Sectorsize: 512 > Mode: r1w1e2 > wither: (null) > > # camcontrol devlist > // does not show the above disks So these daXX devices represent your iscsi devices? If so looks like your problem is at the iscsi layer, as its not disconnected properly, so as far ZFS is concerned its still waiting for them. > >> Does iscsid have any options on how to treat failed devices? > iSCSI has some tuning regarding how to treat failing devices, and I did it : > kern.iscsi.ping_timeout=5 > kern.iscsi.iscsid_timeout=5 > kern.iscsi.login_timeout=85 > kern.iscsi.fail_on_disconnection=1 > > However, as I disconnected the targets from the server hosting the zpool, > they should not have been needed.     Regards     Steve From owner-freebsd-fs@freebsd.org Mon Oct 2 19:59:52 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79142E26461 for ; Mon, 2 Oct 2017 19:59:52 +0000 (UTC) (envelope-from lobo@bsd.com.br) Received: from mail-qt0-x242.google.com (mail-qt0-x242.google.com [IPv6:2607:f8b0:400d:c0d::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A03D759E9 for ; Mon, 2 Oct 2017 19:59:51 +0000 (UTC) (envelope-from lobo@bsd.com.br) Received: by mail-qt0-x242.google.com with SMTP id e19so268900qta.2 for ; Mon, 02 Oct 2017 12:59:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsd.com.br; s=capeta; h=mime-version:in-reply-to:references:from:date:message-id:subject:cc; bh=grv+AiG+uwyDRqwa3FyQUYqCMG9R2MiDTUvPkiKvUTY=; b=WdyHvHyxH7VBrobO/q14Q4OQiOoFI3YwT/yj0WK8pDUOzk04Thx14NWW3SnQQshyzd zZXqx8WE0u3DkHGRR0pVhWahB2dzdyoDVqVikob1EbOkeuf2wTlkeJhfDJYED8MZhLlk AHYjaha1dc0B0OSy2wO2GYKIntEcNZuW0RPF8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:cc; bh=grv+AiG+uwyDRqwa3FyQUYqCMG9R2MiDTUvPkiKvUTY=; b=fTOk1W6h3T5xgpdGSgXgVJYaRdRk9dfm0lRP0f0tAqNGyFTkqsZxHsITOU32g1TOBi ljRmWNfa3QV0MXfwPM7/VZgeVXVOG5x9986PgTYskvL1EYnq6fh4idHsPxU6wnhdJA/t S5xEgn3efaHAsRbKk2gXKajZ8G7/PqmCV+B8lZ+coMlq3FEgoMmR/ey0VemFSHt34Qo+ uwLZgXFabP6/mLIQDXu2GZRaKCT0zlJrf0DiDiLWuk2AbMIP42BNqDgw1EM0+8yRWav4 Tcen0CS2hvMrD0shS4hZR7EwnJhw966P+2z/3JRaVZn+ry6Y+OPfOK29htuUv9b4GLDn LgtQ== X-Gm-Message-State: AHPjjUjOQTihF3zWJPI7IBfNkyAL2JN4ar/i2cxYSgf5bkpX/aHzHbUZ 45v+y5QYUKJOUO74tzuSkEnoP/+U2SphQR+ZRyfRVA== X-Received: by 10.129.57.3 with SMTP id g3mt13576002ywa.433.1506974390689; Mon, 02 Oct 2017 12:59:50 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.179.130 with HTTP; Mon, 2 Oct 2017 12:59:50 -0700 (PDT) In-Reply-To: <20170928184101.55c8a0ec@Papi.lobos> References: <20170927100635.7b56f8fd@Papi.lobos> <20170928184101.55c8a0ec@Papi.lobos> From: Mario Lobo Date: Mon, 2 Oct 2017 16:59:50 -0300 Message-ID: Subject: Re: mount_smbfs question (re-post) Cc: freebsd-fs@freebsd.org, FreeBSD questions Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 19:59:52 -0000 2017-09-28 18:41 GMT-03:00 Mario Lobo : > On Thu, 28 Sep 2017 17:48:23 -0300 > Mario Lobo wrote: > > > 2017-09-28 17:20 GMT-03:00 Ronald Klop : > > > > > On Thu, 28 Sep 2017 19:08:04 +0200, Erwan Legrand < > > > freebsd@erwanlegrand.com> wrote: > > > > > > On Thu, Sep 28, 2017 at 4:16 PM, Mario Lobo > > > wrote: > > >> > > >>> 2017-09-27 11:20 GMT-03:00 Erwan Legrand > > >>> : > > >>>> On Wed, Sep 27, 2017 at 3:06 PM, Mario Lobo > > >>>> wrote: > > >>>> > Since my environment is tottaly surrounded with shares that no > > >>>> > longer accept SMBv1 (Windows, Linux AND FreeBSD servers), so > > >>>> > basically in the end, what I'm really looking for is a > > >>>> > confirmation that I'll just have to dump all Freebsd samba > > >>>> > clients because the OS can't deal with SMBv2 or above. > > >>>> > > >>>> Perhaps have a look at implementations of SMB on top of FUSE? > > >>>> > > >>>> http://portsmon.freebsd.org/portoverview.py?category=sysutil > > >>>> s&portname=fusefs-smbnetfs > > >>>> > > >>> > > >>> I did. Same problem. > > >>> smbnetfs only works with SMBv1 > > >>> > > >>> > > >> It is based on libsmbclient, thus it should support SMB2 if > > >> smb.conf allows it. According to the following thread, the client > > >> protocol is resticted to SMB1 by default: > > >> > > >> https://lists.samba.org/archive/samba-technical/2016-Novembe > > >> r/thread.html#116999 > > >> > > >> This might be fixed by setting "client max protocol = SMB2" in > > >> smb.conf. ($HOME/.smb/smb.conf in this case?) > > >> > > > > > > I'd suggest setting "client min protocol". > > > ^^^ > > > > > > Regards, > > > Ronald. > > > > > > > > > _______________________________________________ > > >> freebsd-fs@freebsd.org mailing list > > >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > >> To unsubscribe, send any mail to > > >> "freebsd-fs-unsubscribe@freebsd.org" > > > > > I just tested it! > > > > 2 shares. 1 with SMBv1 and 1 with client min protocol=SMBv2 > > They are both SAMBA with FREEBSD. > > > [snip] > > Unless I'm missing some tuning option in smbnetfs for SMBv2 and above, > > It doesen't work! > > > > Thanks, > > > > One more thing. > > If I gear down the second server to SMBv1, I can access it just fine. > > -- > Well, after wearing my eyes off everywhere I could looking for a solution to this issue, I came to the conclusion that accessing an SMBv2+ share with FreeBSD is on a limbo that I can't reach. And it is dormant. I am a stubborn person so I started looking at the OS source code but so far it is way beyond me. I can't even make out how it currently works, and much less implement two new protocols on top of it. But I'll keep going forward ... I also can't find anything other than mount_smbfs and smbnetfs that can access smb shares, and much less integrate with the OS as they do. As much as it breaks my heart to drop a reliable, fast and stable OS because of this ONE issue, on my current environment its seems to be my only choice left. Thanks to all that tried to help. -- Mario Lobo http://www.mallavoodoo.com.br FreeBSD since version 2.2.8 [not Pro-Audio.... YET!!] From owner-freebsd-fs@freebsd.org Mon Oct 2 20:02:26 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9CEC4E26737 for ; Mon, 2 Oct 2017 20:02:26 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2BA0C75DB9 for ; Mon, 2 Oct 2017 20:02:26 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22e.google.com with SMTP id i82so10557159wmd.3 for ; Mon, 02 Oct 2017 13:02:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=GXcZUNcGOb6KAdNz3zdhd8nrcUQEqjcxSRhRrNQAlaM=; b=oNHMNDzWoMlqDEGvM/xT5P22oPD6KtEZXAGZ9YRqnd1zhun69eMLxGbnWJ9j5gNJng HwDHH5xaVsUjWPBxbwa9hoEEkS6KfTUPb5YB09f6dcFocvjNf/ELT63+oSCf3vfe7OO0 1OIkV3dc57AN9gcKRBjwfSx99PpTOL1EACTwQ4DppIM9PoUwFrg/h+dWX7k3QkzXp6gc b1UTaTzkAFdNVHAdRBhICX5gQcYLfLmahFymOYsXIlnZo2cYugBGdcIqfShMRe/Rqfc+ /kJzGdCBL9JIWBQausFF34jGAfA8OEEoluyVvyygVmbv6YXHbHPeDQsfS+Rcey4HcoKI JMdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=GXcZUNcGOb6KAdNz3zdhd8nrcUQEqjcxSRhRrNQAlaM=; b=lAxI0ancHszhdQs14wXmx6WqHk1eDQgYahPLWbMMECcny7qx51aO1O5UE8xnDxAKLw +CBj0Ozh77LuxYL/kkVgll9k53eG3XhV51mwN941qwhrdqydTKPASQPIZkzimOuAMO0T JWcW3lXhXR9p4+iTs5x2iYE8OQ7uKWCgZVhaB97v+aEKezAoLhhkw4BynauI/GnIzC7c 9R+ALxzyyaBePcOEKSw1VKnUfXjemrTr3vxUFdw6XCZorod7H+fasrj30BLgmyaANpOh YJTgKwChJSgXEfSxSsP/xf5L728KhkgCSZVT88h1WhiXzOUEsVldhpBpMPCCBgqhbOzS l/eA== X-Gm-Message-State: AHPjjUgUv5cDNd+kz2h3DVbNFHRtxCFUDWmivN/YU+/N2YS0ZHdRGrAv HBsT5C+l68VXZdzj5NDdFFgjgXgH X-Google-Smtp-Source: AOwi7QBSudFg/4XWXUxzxPjEmpCzhwJg/OX8gt47l2ovEd4wf+woA971LHoamH+8Z2lXUAOHQWAc8g== X-Received: by 10.28.174.67 with SMTP id x64mr12514286wme.82.1506974544460; Mon, 02 Oct 2017 13:02:24 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id x75sm14796881wme.3.2017.10.02.13.02.23 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 13:02:23 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: <7fb4c99b-f3a0-1dda-691c-35f25769ed5c@multiplay.co.uk> Date: Mon, 2 Oct 2017 22:02:23 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> <953DD379-C03A-4737-BAD8-14BB2DB4AB05@gmail.com> <4f725113-bac3-64bb-9858-690811e73153@multiplay.co.uk> <54AD0000-AF0B-4682-9047-6E6C1B82506C@gmail.com> <7fb4c99b-f3a0-1dda-691c-35f25769ed5c@multiplay.co.uk> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 20:02:26 -0000 > On 02 Oct 2017, at 21:47, Steven Hartland = wrote: >=20 > On 02/10/2017 20:10, Ben RUBSON wrote: >>> On 02 Oct 2017, at 20:41, Steven Hartland = wrote: >>>=20 >>> I'm guessing that the devices haven't disconnected cleanly so are = just stalling all requests to them and hence the pool. >> I even tried to ifconfig down the network interface serving the iscsi = targets, it did not help. >>=20 >>> I'm not that familiar with iscsi, does it still show under under = camcontrol or geom? >> # geom disk list >> (...) >> Geom name: da13 >> Providers: >> 1. Name: da13 >> Mediasize: 3999688294912 (3.6T) >> Sectorsize: 512 >> Mode: r1w1e2 >> wither: (null) >>=20 >> Geom name: da15 >> Providers: >> 1. Name: da15 >> Mediasize: 3999688294912 (3.6T) >> Sectorsize: 512 >> Mode: r1w1e2 >> wither: (null) >>=20 >> Geom name: da16 >> Providers: >> 1. Name: da16 >> Mediasize: 3999688294912 (3.6T) >> Sectorsize: 512 >> Mode: r1w1e2 >> wither: (null) >>=20 >> Geom name: da19 >> Providers: >> 1. Name: da19 >> Mediasize: 3999688294912 (3.6T) >> Sectorsize: 512 >> Mode: r1w1e2 >> wither: (null) >>=20 >> # camcontrol devlist >> // does not show the above disks > So these daXX devices represent your iscsi devices? Yes, and only one is still visible under /dev/, with its label under /dev/label/. So I may have one problematic drive among 4. > If so looks like your problem is at the iscsi layer, as its not = disconnected properly, so as far ZFS is concerned its still waiting for = them. Certainly procstat will talk ! I have switched production to another server, so feel free if any other trace is needed. Thank you again, Ben From owner-freebsd-fs@freebsd.org Mon Oct 2 20:56:23 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06D39E27927 for ; Mon, 2 Oct 2017 20:56:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 3410E7776C for ; Mon, 2 Oct 2017 20:56:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id XAA10583; Mon, 02 Oct 2017 23:56:14 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dz7l7-000JfZ-SK; Mon, 02 Oct 2017 23:56:13 +0300 Subject: Re: ZFS stalled after some mirror disks were lost To: Ben RUBSON , Freebsd fs References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> <5d3e1f0d-c618-afa4-7e52-819c9edf30c9@FreeBSD.org> <48D23270-1811-4E09-8AF2-5C0FEC2F9176@gmail.com> From: Andriy Gapon Message-ID: <9ff8ef2c-b445-dad3-d726-b84793c173ee@FreeBSD.org> Date: Mon, 2 Oct 2017 23:55:38 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <48D23270-1811-4E09-8AF2-5C0FEC2F9176@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 20:56:23 -0000 On 02/10/2017 22:13, Ben RUBSON wrote: >> On 02 Oct 2017, at 20:45, Andriy Gapon wrote: >> >> On 02/10/2017 21:17, Ben RUBSON wrote: >>> Unfortunately the command stalls / does not return :/ >> >> Try to take procstat -kk -a. > > Thank you Andriy for your answer. > > Here is the procstat output : > https://benrubson.github.io/zfs/procstat01.log First, it seems that there are some iscsi threads stuck on a lock like: 0 100291 kernel iscsimt mi_switch+0xd2 sleepq_wait+0x3a _sx_xlock_hard+0x592 iscsi_maintenance_thread+0x316 fork_exit+0x85 fork_trampoline+0xe or like 8580 102077 iscsictl - mi_switch+0xd2 sleepq_wait+0x3a _sx_slock_hard+0x325 iscsi_ioctl+0x7ea devfs_ioctl_f+0x13f kern_ioctl+0x2d4 sys_ioctl+0x171 amd64_syscall+0x4ce Xfast_syscall+0xfb Also, there is a thread in cam_sim_free(): 0 100986 kernel iscsimt mi_switch+0xd2 sleepq_wait+0x3a _sleep+0x2a1 cam_sim_free+0x48 iscsi_session_cleanup+0x1bd iscsi_maintenance_thread+0x388 fork_exit+0x85 fork_trampoline+0xe So, it looks like there could be a problem is the iscsi teardown path. Maybe that caused a domino effect in ZFS code. I see a lot of threads waiting either for spa_namespace_lock or a spa config lock (a highly specialized ZFS lock). But it is hard to untangle their inter-dependencies. Some of ZFS I/O threads are also affected, for example: 0 101538 kernel zio_write_issue_ mi_switch+0xd2 sleepq_wait+0x3a _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 zio_execute+0x236 taskqueue_run_locked+0x14a taskqueue_thread_loop+0xe8 fork_exit+0x85 fork_trampoline+0xe 8716 101319 sshd - mi_switch+0xd2 sleepq_wait+0x3a _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 zio_execute+0x236 zio_nowait+0x49 arc_read+0x8e4 dbuf_read+0x6c2 dmu_buf_hold_array_by_dnode+0x1d3 dmu_read_uio_dnode+0x41 dmu_read_uio_dbuf+0x3b zfs_freebsd_read+0x5fc VOP_READ_APV+0x89 vn_read+0x157 vn_io_fault1+0x1c2 vn_io_fault+0x197 dofileread+0x98 71181 101141 encfs - mi_switch+0xd2 sleepq_wait+0x3a _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 zio_execute+0x236 zio_nowait+0x49 arc_read+0x8e4 dbuf_read+0x6c2 dmu_buf_hold+0x3d zap_lockdir+0x43 zap_cursor_retrieve+0x171 zfs_freebsd_readdir+0x3f3 VOP_READDIR_APV+0x8f kern_getdirentries+0x21b sys_getdirentries+0x28 amd64_syscall+0x4ce Xfast_syscall+0xfb 71181 101190 encfs - mi_switch+0xd2 sleepq_wait+0x3a _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 zio_execute+0x236 zio_nowait+0x49 arc_read+0x8e4 dbuf_prefetch_indirect_done+0xcc arc_read+0x425 dbuf_prefetch+0x4f7 dmu_zfetch+0x418 dmu_buf_hold_array_by_dnode+0x34d dmu_read_uio_dnode+0x41 dmu_read_uio_dbuf+0x3b zfs_freebsd_read+0x5fc VOP_READ_APV+0x89 vn_read+0x157 Note that the first of these threads executes a write zio. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Mon Oct 2 21:07:23 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8210E27C24 for ; Mon, 2 Oct 2017 21:07:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 0DDE077B5D for ; Mon, 2 Oct 2017 21:07:22 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id AAA10608; Tue, 03 Oct 2017 00:07:20 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dz7vs-000Jg9-Jz; Tue, 03 Oct 2017 00:07:20 +0300 Subject: Re: ZFS stalled after some mirror disks were lost From: Andriy Gapon To: Ben RUBSON , Freebsd fs References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> <5d3e1f0d-c618-afa4-7e52-819c9edf30c9@FreeBSD.org> <48D23270-1811-4E09-8AF2-5C0FEC2F9176@gmail.com> <9ff8ef2c-b445-dad3-d726-b84793c173ee@FreeBSD.org> Message-ID: <84f5608e-d312-437c-3c6b-d8e5847de8bc@FreeBSD.org> Date: Tue, 3 Oct 2017 00:06:25 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <9ff8ef2c-b445-dad3-d726-b84793c173ee@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 21:07:23 -0000 On 02/10/2017 23:55, Andriy Gapon wrote: > Maybe that caused a domino effect in ZFS code. I see a lot of threads waiting > either for spa_namespace_lock or a spa config lock (a highly specialized ZFS > lock). But it is hard to untangle their inter-dependencies. Forgot to add. It would be nice to determine an owner of spa_namespace_lock. If you have debug symbols then it can be easily done in kgdb on the live system: (kgdb) p spa_namespace_lock -- Andriy Gapon From owner-freebsd-fs@freebsd.org Tue Oct 3 05:40:47 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C059E315A0 for ; Tue, 3 Oct 2017 05:40:47 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x234.google.com (mail-wm0-x234.google.com [IPv6:2a00:1450:400c:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DFE1365396 for ; Tue, 3 Oct 2017 05:40:46 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x234.google.com with SMTP id b189so10367855wmd.4 for ; Mon, 02 Oct 2017 22:40:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=Crdp3T7pSEJ5OumuAiLUq/oHQpdO9ySZrADfi5coiXo=; b=CgHslnVRZf81stqw247PF/F7+Q50dHvsYluxkT6cef75tyonVXh3fd/g+GtDSo3h4L xhjWkH5Q7c21Z0mKWDIhSS7hQIGj4kmtfnKE2BtNg5uLiWuMyIiM6zCBz27ukqwGzvrh IYjlIOyz9lsdgHf+faXmrAbuELFoL0SdVjI/zf/4kyxY0nvh8PHiOONG+uIkQgQ1OR9J Ylkq2Ke2N3KcSLSxehpFZdHGfYU2DWGYyoVJbxi3deCzmFeJAnp3JU+jJUSDuUrYZ5aw SwXpLCM0D6aCVua0/kGpgqoVNBycsuyz0TbNxEfCQW/sOj0olGTZ5ihR3hkxC0Ulo/Ob udUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=Crdp3T7pSEJ5OumuAiLUq/oHQpdO9ySZrADfi5coiXo=; b=jvtk/lsIRf3qjOUQdZYCyvetxidEeRLfWcnLj3deZw0t23EQokwAuxe2dOutljwmjw prJT1RxNKGnppKeMs5f5sdArFGmMmuYVd/A+uRKQaN0e4/adxfUv9HM2/RGiYvdefiV5 FcmGTTctFQeurqjLAgesYQTGGQiRdoAsmvjeYf+0tABEvFQV1kBIqjWGhFVyofnnAZGc N/GqQqPOa/O1jq/91Rxj8P9L2b+Xushbn/EoqDUfghyNOR6A/DqTp2qLwgz3HE2VWN+1 n/f9cZMD2MI7luBp345Ztgtt7R/aEsKzoxcRikzr+0rdIy4aTvdxrfhFGVbPWSUNWT5r luJA== X-Gm-Message-State: AMCzsaWfrU0QGmH3dmK6HSerO3n7QPkuCM+TxZ0/7+3qHQOiljP7KpkF ZdSMhs3H8KURIQag621Hn97Me7ro X-Google-Smtp-Source: AOwi7QDD3ZAKNf7KAQl2UvxQPxPguTz5whGldLNLn4clEYan6+fzKcauQWrIAOQtBlQk2KzztrX4kg== X-Received: by 10.28.125.139 with SMTP id y133mr3497051wmc.25.1507009245230; Mon, 02 Oct 2017 22:40:45 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id p200sm11700213wmg.48.2017.10.02.22.40.44 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 22:40:44 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: <84f5608e-d312-437c-3c6b-d8e5847de8bc@FreeBSD.org> Date: Tue, 3 Oct 2017 07:40:46 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> <5d3e1f0d-c618-afa4-7e52-819c9edf30c9@FreeBSD.org> <48D23270-1811-4E09-8AF2-5C0FEC2F9176@gmail.com> <9ff8ef2c-b445-dad3-d726-b84793c173ee@FreeBSD.org> <84f5608e-d312-437c-3c6b-d8e5847de8bc@FreeBSD.org> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 05:40:47 -0000 > On 02 Oct 2017, at 23:06, Andriy Gapon wrote: >=20 > Forgot to add. It would be nice to determine an owner of = spa_namespace_lock. > If you have debug symbols then it can be easily done in kgdb on the = live system: > (kgdb) p spa_namespace_lock Thank you very much Andriy for your deep analysis, much appreciated ! Unfortunately, I lost access to the server and had to recycle it :| ... I have some everyday maintenance windows on this production infra, so for sure plan is to try to reproduce the issue. I will then let you know. Thank you all again, Ben From owner-freebsd-fs@freebsd.org Tue Oct 3 06:14:16 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 516BEE31EB6 for ; Tue, 3 Oct 2017 06:14:16 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 518B466135; Tue, 3 Oct 2017 06:14:13 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA11854; Tue, 03 Oct 2017 09:14:11 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1dzGT4-000K4r-SF; Tue, 03 Oct 2017 09:14:10 +0300 Subject: Re: ZFS stalled after some mirror disks were lost To: Ben RUBSON , Steven Hartland References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> Cc: Freebsd fs From: Andriy Gapon Message-ID: Date: Tue, 3 Oct 2017 09:12:49 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 06:14:16 -0000 On 02/10/2017 21:12, Ben RUBSON wrote: > A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk > according to "zpool iostat", nothing on local disks (strange but I > noticed that IOs always prefer iscsi disks to local disks). Are your local disks SSD or HDD? Could it be that iSCSI disks appear to be faster than the local disks to the smart ZFS mirror code? Steve, what do you think? -- Andriy Gapon From owner-freebsd-fs@freebsd.org Tue Oct 3 06:19:08 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F143E31FC7; Tue, 3 Oct 2017 06:19:08 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x242.google.com (mail-wr0-x242.google.com [IPv6:2a00:1450:400c:c0c::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 22C4A6635B; Tue, 3 Oct 2017 06:19:08 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x242.google.com with SMTP id y44so171951wry.2; Mon, 02 Oct 2017 23:19:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=srKBKlqL0bm2QIKPSKpGYiCm8msLDkbMXNIiZt3xkus=; b=SI3hpRxnRCyREQkUT9I+JfvlDqAa1VAeiCefJkz4U3eIdsgLT+LvaDHUv4VdtsObGi hdKwajyoRODQNKjCYcffYAu/0YrlQ55gn+qIxUmM4T9Msz4kbmI3ZdC2zOqDhpWdrtL6 qGw5caG8B0nI/H0GEnf0og6Tx64uUB3hS7US4horNQTuydh10N69nUrT802fjToOlbEo Mi472MJLGyqoqlFJEgQwQnnVA5xHrCVjwixwr6yim+T348OLkW4xmF+3mohXd0ED7Jtw 9X2b2F6kgLGZqJzotBH+LgglgnT2MnojL8Lbgv64eUHXp6VTnFh4iGSikjqa6pd3f5+r 9q6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=srKBKlqL0bm2QIKPSKpGYiCm8msLDkbMXNIiZt3xkus=; b=B1CgA+DE3UABMyiRT7MY0l1k0QG4yQmBW99n3cH31B97kh9RvbAWVXBOtuTM71AAfQ Ocff8oZP8YjnzuzZBKb7o7Y9LABBWH62RJGpxapx3HOoLuLFtQkIKh/tjD2VWpleqHp/ 5q736Z/jrvyalRsthASW5oAssAkCu0ym2opdoPorfvcNLDTJsKEcOSGsfabZTg+2laSP 48q/sBPmR57tDI3zE7icCHbijJrRtvmSFUvEsra7ZRGwRXkR58AXhTteeP9mm0MB1YXo uNKYTcjuE5n885iYmBzj6LJ6Ed93J/oC2x1MwOsQ3vNVG+yUorFzOjJqIUEC1d8v19TG FC6g== X-Gm-Message-State: AHPjjUizU8/T0VgGVi7Ea7OIHdC5NScN1gZiCvFaNcvGDy1ayABs+mGR xsM87m5P4iTGSO5B9rBsiIZtrBFh X-Google-Smtp-Source: AOwi7QDp5dVRcwq413co87jY/pCsZXU2soA4q0HWK9xlxejqtdPydqL4CUSW5Ak9R8BapLy1oYcRew== X-Received: by 10.223.198.15 with SMTP id n15mr10905748wrg.200.1507011546433; Mon, 02 Oct 2017 23:19:06 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id 4sm12162741wmg.20.2017.10.02.23.19.04 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 23:19:05 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> Date: Tue, 3 Oct 2017 08:19:04 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> To: Freebsd fs , FreeBSD-scsi X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 06:19:08 -0000 Hi, Putting scsi list as it could be related. > On 02 Oct 2017, at 20:12, Ben RUBSON wrote: >=20 > Hi, >=20 > On a FreeBSD 11 server, the following online/healthy zpool : >=20 > home > mirror-0 > label/local1 > label/local2 > label/iscsi1 > label/iscsi2 > mirror-1 > label/local3 > label/local4 > label/iscsi3 > label/iscsi4 > cache > label/local5 > label/local6 >=20 > A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk > according to "zpool iostat", nothing on local disks (strange but I > noticed that IOs always prefer iscsi disks to local disks). > No write IOs. >=20 > Let's disconnect all iSCSI disks : > iscsictl -Ra >=20 > Expected behavior : > IO activity flawlessly continue on local disks. >=20 > What happened : > All IOs stalled, server only answers to IOs made to its zroot pool. > All commands related to the iSCSI disks (iscsictl), or to ZFS = (zfs/zpool), > don't return. >=20 > Questions : > Why this behavior ? > How to know what happens ? (/var/log/messages says almost nothing) >=20 > I already disconnected the iSCSI disks without any issue in the past, > several times, but there were almost no IOs running. >=20 > Thank you for your help ! >=20 > Ben > On 02 Oct 2017, at 22:55, Andriy Gapon wrote: >=20 >> On 02/10/2017 22:13, Ben RUBSON wrote: >>=20 >>> On 02 Oct 2017, at 20:45, Andriy Gapon wrote: >>>=20 >>>> On 02/10/2017 21:17, Ben RUBSON wrote: >>>>=20 >>>> Unfortunately the zpool command stalls / does not return :/ >>>=20 >>> Try to take procstat -kk -a. >>=20 >> Here is the procstat output : >> https://benrubson.github.io/zfs/procstat01.log >=20 > First, it seems that there are some iscsi threads stuck on a lock = like: > 0 100291 kernel iscsimt mi_switch+0xd2 = sleepq_wait+0x3a > _sx_xlock_hard+0x592 iscsi_maintenance_thread+0x316 fork_exit+0x85 > fork_trampoline+0xe >=20 > or like >=20 > 8580 102077 iscsictl - mi_switch+0xd2 = sleepq_wait+0x3a > _sx_slock_hard+0x325 iscsi_ioctl+0x7ea devfs_ioctl_f+0x13f = kern_ioctl+0x2d4 > sys_ioctl+0x171 amd64_syscall+0x4ce Xfast_syscall+0xfb >=20 > Also, there is a thread in cam_sim_free(): > 0 100986 kernel iscsimt mi_switch+0xd2 = sleepq_wait+0x3a > _sleep+0x2a1 cam_sim_free+0x48 iscsi_session_cleanup+0x1bd > iscsi_maintenance_thread+0x388 fork_exit+0x85 fork_trampoline+0xe >=20 > So, it looks like there could be a problem is the iscsi teardown path. >=20 > Maybe that caused a domino effect in ZFS code. I see a lot of threads = waiting > either for spa_namespace_lock or a spa config lock (a highly = specialized ZFS > lock). But it is hard to untangle their inter-dependencies. >=20 > Some of ZFS I/O threads are also affected, for example: > 0 101538 kernel zio_write_issue_ mi_switch+0xd2 = sleepq_wait+0x3a > _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 = zio_execute+0x236 > taskqueue_run_locked+0x14a taskqueue_thread_loop+0xe8 fork_exit+0x85 > fork_trampoline+0xe > 8716 101319 sshd - mi_switch+0xd2 = sleepq_wait+0x3a > _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 = zio_execute+0x236 > zio_nowait+0x49 arc_read+0x8e4 dbuf_read+0x6c2 = dmu_buf_hold_array_by_dnode+0x1d3 > dmu_read_uio_dnode+0x41 dmu_read_uio_dbuf+0x3b zfs_freebsd_read+0x5fc > VOP_READ_APV+0x89 vn_read+0x157 vn_io_fault1+0x1c2 vn_io_fault+0x197 > dofileread+0x98 > 71181 101141 encfs - mi_switch+0xd2 = sleepq_wait+0x3a > _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 = zio_execute+0x236 > zio_nowait+0x49 arc_read+0x8e4 dbuf_read+0x6c2 dmu_buf_hold+0x3d > zap_lockdir+0x43 zap_cursor_retrieve+0x171 zfs_freebsd_readdir+0x3f3 > VOP_READDIR_APV+0x8f kern_getdirentries+0x21b sys_getdirentries+0x28 > amd64_syscall+0x4ce Xfast_syscall+0xfb > 71181 101190 encfs - mi_switch+0xd2 = sleepq_wait+0x3a > _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 = zio_execute+0x236 > zio_nowait+0x49 arc_read+0x8e4 dbuf_prefetch_indirect_done+0xcc = arc_read+0x425 > dbuf_prefetch+0x4f7 dmu_zfetch+0x418 dmu_buf_hold_array_by_dnode+0x34d > dmu_read_uio_dnode+0x41 dmu_read_uio_dbuf+0x3b zfs_freebsd_read+0x5fc > VOP_READ_APV+0x89 vn_read+0x157 >=20 > Note that the first of these threads executes a write zio. >=20 > It would be nice to determine an owner of spa_namespace_lock. > If you have debug symbols then it can be easily done in kgdb on the = live system: > (kgdb) p spa_namespace_lock So as said a few minutes ago I lost access to the server and had to = recycle it. Thankfully I managed to reproduce the issue, re-playing exactly the same = steps. Curious line in /var/log/messages : kernel: g_access(918): provider da18 has error (da18 is the remaining iSCSI target device which did not disconnect = properly) procstat -kk -a : https://benrubson.github.io/zfs/procstat02.log (kgdb) p spa_namespace_lock $1 =3D -2110867066 Thank you ! Ben From owner-freebsd-fs@freebsd.org Tue Oct 3 06:22:10 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CAEF5E3231B; Tue, 3 Oct 2017 06:22:10 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x231.google.com (mail-wm0-x231.google.com [IPv6:2a00:1450:400c:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5BE7266661; Tue, 3 Oct 2017 06:22:10 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x231.google.com with SMTP id m72so9768187wmc.0; Mon, 02 Oct 2017 23:22:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=YZkiMsy6BmmyyYQlVQZD5mwcE7j2oPo3X0avPs5Z+II=; b=Zxezx4WKo7BbmdBfWso+T2JwMrRUxz/uewl+kvbwbn/TPxh8Jz1hzzwDayUCb7QcF2 voS5oQfL6fSab9AHatgzkYnhjM2DnjEM2NeVdAxC3zCACf7uXmGDklUzUdvSYOfTutDT wVZ5K2GTWXcu654GveBgFdrE9AnJkVM4Y7jQolPA38uOc17/OIWO0ozJG1E/vkORxBTL iF9+03P0k6C4fG7ROWfPjC9Mpgcg+m5mpvajetY4yQLnjd04/mAGvlhaxv4XJ37v949A 6oqb7vOJSWCnDhDAZ/bHwwmOL2vUbZrCyTrAkwQ/dY4vOlVjianilE7ua6KJsdupj5Ls A3MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=YZkiMsy6BmmyyYQlVQZD5mwcE7j2oPo3X0avPs5Z+II=; b=VOsy/oUfx18DAu0LEbZzRYWrH6GdykLzcHJv0J3UOpKC8j3O7o06uvCkMlC/+EY9UI cZ1JsosDMYCnkSRMkp574KLncsUcuzTz4GSruoK6rgvz8qh7louLjqC/xgthhK4WJBnA 7lcRAi9vyxXpvASNYfM9WlTJkiMjV9kE93L1SScGGpDckAxfC22M5bTxosXwbrN1Vnht Ks61n7VZGXLlUlZGDH0CPQ8RVmKUqxfhssjrC1IPB6EVLFW/6TyuYW3Ah7qorXuSt97a FelZATKnKZ4KCa3LdAcnhajBQG3gpno5Ei1H6iVsFOMQ5CaHWYBaPr6qhRZwhuVFT+Y+ YtbA== X-Gm-Message-State: AMCzsaVz9vz+kzvCeymefWE3sanKr2s/WXCwt7bAN9RTlu4Wv8tQ0bZ6 9eym3oT0E0UZxN37fzmtVCcR4fXE X-Google-Smtp-Source: AOwi7QCfSACr0peMLWDBJx3eIygoktqSBax6ahQva+Ju5Fs96305YMSympnjorUdpywd0Y+rYp1pRw== X-Received: by 10.28.211.69 with SMTP id k66mr12100999wmg.1.1507011728568; Mon, 02 Oct 2017 23:22:08 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id v2sm7550275wmf.40.2017.10.02.23.22.07 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 23:22:07 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: Date: Tue, 3 Oct 2017 08:22:06 +0200 Cc: Steven Hartland , Freebsd fs , FreeBSD-scsi Content-Transfer-Encoding: quoted-printable Message-Id: References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> To: Andriy Gapon X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 06:22:10 -0000 > On 03 Oct 2017, at 08:12, Andriy Gapon wrote: >=20 > On 02/10/2017 21:12, Ben RUBSON wrote: >> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >> according to "zpool iostat", nothing on local disks (strange but I >> noticed that IOs always prefer iscsi disks to local disks). >=20 > Are your local disks SSD or HDD? HDD. > Could it be that iSCSI disks appear to be faster than the local disks = to the > smart ZFS mirror code? Or because their /dev/da are greater then the local ones ? (as they are attached after the local disks) (my 2 cents...) For sure we could have expected the local disks to be preferred, or at least the load to be spread among all (local & iscsi) disks. > Steve, what do you think? From owner-freebsd-fs@freebsd.org Tue Oct 3 07:25:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 949ADE33582 for ; Tue, 3 Oct 2017 07:25:35 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2D265683AA for ; Tue, 3 Oct 2017 07:25:34 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-wr0-x22d.google.com with SMTP id t76so5418607wrc.3 for ; Tue, 03 Oct 2017 00:25:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=from:subject:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=vzQ7cqW5WJnSjZqzK8x0JWjMZpLEFsb2v9BmtqZxRsw=; b=laW3PEn68R74ibCuUpOptWME7wCjoeHWjgAw4/wmkfaoUxXJvuBgy7vrTNbQ2DwoBo lYseNsr2lHTYjjvgCP0TtPafYaOTFOJKk85kfLZKRIkPCmEpzdqI/u31/IRaobxNK3sV sXcnpM6p9be5H04ems6TacKtA4hj5dSOMH2bEudNZKCGUkGCX+FxEVqGsre5RfPjjZbQ ScDjACkUn4J+qr9I3kcSkedRS+8bLKmafLHDi5QRyEAs+AKJoWVeS6dRTbSN0i3sv2Ty DlOB/nfnYUIzx8FMpTN+9mL+3WB9qyZFeNvvG6bdEf9NpaFTx2qXP8xDOY+Xh5bPtnp/ w2qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=vzQ7cqW5WJnSjZqzK8x0JWjMZpLEFsb2v9BmtqZxRsw=; b=MGlzNJI0hd0q+48qddiez2RDeDQaJo5pfloDeX8f4ybsxRSlmqN5MhKxMvtZLM+Afa 3Znqz/8AfCQU50Vv5HCaZnUPc5VT5dCxlaPKQ68NkVAaoravzLnQ+Dji7F4sfLMRZwmh Vl3OJdF6H+5L4UlF2hiCKfR/nF3OROFYYKNWXT2iotSBUDy0OzmXvn2Uz6vGjbj6DQ1r jbNrq65soUxyYjiqrFv87EVgFfY6cpCGhDvKMbxjmLEwjAnu3ndqCG/5DJ2VHb6sk61u eUwwonRa/20fqJdmXgVXHfsRnY4IL5z2/3kFtrO35GlraF0huYLCv9GfUfBxXFv8SpFT zPpw== X-Gm-Message-State: AMCzsaV7/Oxo4BC/o9/JNGvz9heLYwnvvjs4ae2ej2eiTPDZSLxdv6JX ppWeKXt1UMf+IV6exi/s4OJTajtwziA= X-Google-Smtp-Source: AOwi7QD0RWuQH4aeAS9YDpUoE0W5E1iZnprxYy7Ofc05LPRkGylehReBH22AxuQLhYYidihQOYhvLg== X-Received: by 10.223.187.201 with SMTP id z9mr10835472wrg.195.1507015533111; Tue, 03 Oct 2017 00:25:33 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.1]) by smtp.gmail.com with ESMTPSA id 69sm13766172wmm.22.2017.10.03.00.25.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Oct 2017 00:25:32 -0700 (PDT) From: Steven Hartland X-Google-Original-From: Steven Hartland Subject: Re: ZFS stalled after some mirror disks were lost To: Andriy Gapon , Ben RUBSON Cc: Freebsd fs References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> Message-ID: <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> Date: Tue, 3 Oct 2017 08:25:34 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 07:25:35 -0000 On 03/10/2017 07:12, Andriy Gapon wrote: > On 02/10/2017 21:12, Ben RUBSON wrote: >> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >> according to "zpool iostat", nothing on local disks (strange but I >> noticed that IOs always prefer iscsi disks to local disks). > Are your local disks SSD or HDD? > Could it be that iSCSI disks appear to be faster than the local disks to the > smart ZFS mirror code? > > Steve, what do you think? Yes that quite possible, the mirror balancing uses the queue depth + rotating bias to determine the load of the disk so if your iSCSI host is processing well and / or is reporting non-rotating vs rotating for the local disks it could well be the mirror is preferring reads from the the less loaded iSCSI devices.     Regards     Steve From owner-freebsd-fs@freebsd.org Tue Oct 3 07:31:53 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E19E9E33743; Tue, 3 Oct 2017 07:31:53 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7245F68665; Tue, 3 Oct 2017 07:31:53 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22d.google.com with SMTP id l39so5638168wrl.12; Tue, 03 Oct 2017 00:31:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=UgM3Amcn+bco2urIcwSNTrv7H6g7cAH4FC9jCmWj2iQ=; b=Hcpc0NPqR9cJf4a7Tozx0Gyd3JCDuU2zWrMyytXgc4gjUJibL0K67ko8kaX0sWTqI7 wnxwz+lRvWb5L7414g6vwtI/IJjd7E3ClqTyXwkBVBPMuoJHxh5z2gVwQUiv7hs6V6jM Auyu7r5Xf/kkMFK4MJRtkUpDT9j3DSfNPdJe/tBVKFPnd8H3bxEqXiTjgc9RNSsvCkT/ 7eeVpZKbK6ha9o6OrVG2uOWnP4cWPNpLJBYCPbTqJvUhqvfWwvduXcH98pgjMCUkg6Ht gpefHIy+K0uKCWKcNTBM7h85vCb4kYyE3Sw1tVoLAMpbe/3LtOA/jWj/Rp+fQAABgYEF KKSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=UgM3Amcn+bco2urIcwSNTrv7H6g7cAH4FC9jCmWj2iQ=; b=bgsHn1ezV+DGjycls9hErTBwaoOm9P4odfZEF51/oG7PLSKRBM3+QvrGe5Jo1LA9l3 1XwBiQ1ESWWoiu631RMRXq3lVbPwgBJfndCa9mHscemA+iic0DHEtK0vrdmbUvsmXwmt AMuimk1/ZDSjltZxAp+Wc5CDKxJoRJTxCrf6/iroW1CKl8/VmR3riRN18iSHiGK1KDOt eNhLAiapcT04L2153fCNSFORmA0MZYQUulAQli3e4axtPw6Wgyq/sZJCyR9El99GPm5b CUdsNzB+nmpOHY0H3EhGUnJIzMZaQgI4YPLMvxnyHO1WL7aSe/9B2P8JC2i6ipkolLYc 56Pw== X-Gm-Message-State: AMCzsaW+1lzJJz1i2FTeL+hMCNtsKpzegOs+07RNMR99V54idlVxMEao ePF5Sn5oVRi3WVw+CODWecpoIa5n X-Google-Smtp-Source: AOwi7QCTm0qGWPIDdfXs7QTgC5IN6WbC6rNxM2Natyy7Msx1YA8M1dP7b3RIL1jv7NUZwxSfRwMmVg== X-Received: by 10.223.178.144 with SMTP id g16mr11264078wrd.76.1507015911918; Tue, 03 Oct 2017 00:31:51 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id l37sm12954776wrl.47.2017.10.03.00.31.51 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Oct 2017 00:31:51 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> Date: Tue, 3 Oct 2017 09:31:50 +0200 Cc: Freebsd fs , FreeBSD-scsi Content-Transfer-Encoding: quoted-printable Message-Id: <1990B359-FC8D-4D6A-992B-7F77A07D83A6@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> To: Steven Hartland , Andriy Gapon X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 07:31:54 -0000 > On 03 Oct 2017, at 09:25, Steven Hartland = wrote: >=20 > On 03/10/2017 07:12, Andriy Gapon wrote: >> On 02/10/2017 21:12, Ben RUBSON wrote: >>=20 >>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >>> according to "zpool iostat", nothing on local disks (strange but I >>> noticed that IOs always prefer iscsi disks to local disks). >>>=20 >> Are your local disks SSD or HDD? >> Could it be that iSCSI disks appear to be faster than the local disks = to the >> smart ZFS mirror code? >>=20 >> Steve, what do you think? >>=20 > Yes that quite possible, the mirror balancing uses the queue depth + = rotating bias to determine the load of the disk so if your iSCSI host is = processing well and / or is reporting non-rotating vs rotating for the = local disks it could well be the mirror is preferring reads from the the = less loaded iSCSI devices. Note that local & iscsi disks are _exactly_ the same (same model number, = same SAS adapter...). So iSCSI ones should be a little bit slower due to network latency (even = if it's very low in my case). Once production back, after having analysed the main issue of this = thread, I should then try to find whether or not iSCSI disks are seen as rotating disks. Thanks for the hint ! Ben From owner-freebsd-fs@freebsd.org Tue Oct 3 07:39:37 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 602D2E338CA for ; Tue, 3 Oct 2017 07:39:37 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E1F4B68823 for ; Tue, 3 Oct 2017 07:39:36 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-wm0-x22e.google.com with SMTP id q124so14781509wmb.0 for ; Tue, 03 Oct 2017 00:39:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=GrdDugNggHtWHMaNQfG4aSNyt9oH2g9RwDNn/ftgLmU=; b=zZz6wklD5H1NP7KE7l9LGGs5UxEBQlB10BWB9S7z2eQ6hO+aVWxEkMMwa+GtjshS8Y Cxz80LYM10mwFk952Xdu5MXzeUnKBQyYUvvZRHCicz/TCheQXglNww/jqWLBHLmneqiL nK4IclmVMoCZgEh8vvnhMr6iydaXuC0jtJoUCre5tLZIbXJAsVeNQBPAVTbMOlamk4ll BJ76b4zy8WeP7FPmEZPjGt2xuJnkWYsxC1bJ2JuqfXs3rJQvVHEB4wIw1CUNnvZo9HtW Lh1uqp2wd0HtrHu5GSW/sigE/wqu/nBtaeQdpjGYcCsLl1+PE2mOBTrqdRRT8XlbtifE dhtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=GrdDugNggHtWHMaNQfG4aSNyt9oH2g9RwDNn/ftgLmU=; b=mfuCcU/GnS+NrJlGpmz3XSWX/gIZvJI/0+YMVao/W9W2d73HDCLmNDKcFbcpE8uXwy 8D3ja4DUD6Yy4oHin/4VkqjsR3VTiffLrgbwX8lr9Vp8vmttrtueQeJmIeASiNM1hwdX BKK/XxmHqT1/Q335P+vL3CN6RaJbXW+Ir7249LPI6VmK5GSagv09ddUDm0OypUQMZ19s kLHRCQpFZ9E5YCnd1n9frZFdjlCbVfHXyDbAiGmthYfR+PUVtSNtWlqpqFGu6BrGPF49 PT1918uy4N5tPViQHKbxG9IxZ+vZDlXpw0zG0Y8fHwOtgB12d8Axmv4N/Gq1dTJZ1OM8 sszQ== X-Gm-Message-State: AHPjjUg7TPw9DU9CLwfXtCZew1g24tKDRMKPlMR/9SphOufibLLAY+h4 oaH3f696CvdIqTnzZQI/TbE09g== X-Google-Smtp-Source: AOwi7QABtvOBGfjQMvW7El+S9dNwANMh3rBpWLRCo55zipq8qtcFGcIWNwwI7ug+klEOpwEH9VtYSg== X-Received: by 10.80.183.231 with SMTP id i36mr22667458ede.262.1507016375361; Tue, 03 Oct 2017 00:39:35 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.1]) by smtp.gmail.com with ESMTPSA id f20sm9116958edm.46.2017.10.03.00.39.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Oct 2017 00:39:34 -0700 (PDT) Subject: Re: ZFS stalled after some mirror disks were lost To: Ben RUBSON , Andriy Gapon Cc: Freebsd fs , FreeBSD-scsi References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> <1990B359-FC8D-4D6A-992B-7F77A07D83A6@gmail.com> From: Steven Hartland Message-ID: <9bce89eb-4d6f-aec1-df44-ebf794a3123b@multiplay.co.uk> Date: Tue, 3 Oct 2017 08:39:36 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <1990B359-FC8D-4D6A-992B-7F77A07D83A6@gmail.com> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 07:39:37 -0000 On 03/10/2017 08:31, Ben RUBSON wrote: >> On 03 Oct 2017, at 09:25, Steven Hartland wrote: >> >> On 03/10/2017 07:12, Andriy Gapon wrote: >>> On 02/10/2017 21:12, Ben RUBSON wrote: >>> >>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >>>> according to "zpool iostat", nothing on local disks (strange but I >>>> noticed that IOs always prefer iscsi disks to local disks). >>>> >>> Are your local disks SSD or HDD? >>> Could it be that iSCSI disks appear to be faster than the local disks to the >>> smart ZFS mirror code? >>> >>> Steve, what do you think? >>> >> Yes that quite possible, the mirror balancing uses the queue depth + rotating bias to determine the load of the disk so if your iSCSI host is processing well and / or is reporting non-rotating vs rotating for the local disks it could well be the mirror is preferring reads from the the less loaded iSCSI devices. > Note that local & iscsi disks are _exactly_ the same (same model number, same SAS adapter...). > So iSCSI ones should be a little bit slower due to network latency (even if it's very low in my case). > Once production back, after having analysed the main issue of this thread, I should then > try to find whether or not iSCSI disks are seen as rotating disks. > > Thanks for the hint ! Hmm, the output from gstat -dp on a loaded machine would be interesting to see too.     Regards     Steve From owner-freebsd-fs@freebsd.org Tue Oct 3 11:43:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 105F0E39AF6; Tue, 3 Oct 2017 11:43:14 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 96AE7708F3; Tue, 3 Oct 2017 11:43:13 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id i82so13926471wmd.3; Tue, 03 Oct 2017 04:43:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=glpyQV2+u2c6fJfkqWnk31JnmpiHOFmGo21o3TTPem4=; b=EjQirs0ZEDtRqHhE44IybPy9mbeC3ANouwMhfvyQ7sdaWhwYxhDAP/kzPkNIfZOhH/ kQUpJ7j2MfQhH1ararAVdP6yVfraT26wio8luOfDU0+1kR9O/av79KX1NtK0pJCW33W/ SMCs7Kx6A44xwfDq9Z6HNtBJ1Tv2KpeLnuQlHnZV+gi97ofWExVrRuyuvM6eKCWjoaQ1 2/Q2iat3xIEVILrDmLWthQJf2DzUFfITkQ5j+IVUJOLk330XAwSH9DuylEj5PM0z91n6 Z9eOiLTuqImS7A51V1qxMM5K0aanr/8EmQpTcFGVpLzl4ql2457HWKQ5CV/LLg8txdRg WZng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=glpyQV2+u2c6fJfkqWnk31JnmpiHOFmGo21o3TTPem4=; b=pzaMbzYhABe18vxY1u4sTvxoeWwAC2QZEspNYkF1uX50JKz4vNnG6TSCroCq2cUQs3 7xkGHpFcMpD9gIwPtsZ8C1+qknGOiQUDmlb9cK2hSuHZheDMkDHNAFH0gDXq+MtixldW d1e/i/pL0VZtOzHH4UmtboeIwKkx+3Bdo1hqFx+6Z3ukwHGaMObWlZqx4o5ui1ZiVuq6 +bXm/BIM2C9UYKO9ZNR9OUkE8+uL37oIhBvPxymKUbN0s0rQDisfL6abQxbZajxX6rgY iiG67dTqWixIB+oNsz2rU7k1fs/VjDvcm8y52kRsu93QZO1Si4bZEFt/UP+3r6okisvh wNFQ== X-Gm-Message-State: AMCzsaUggGDEBWxsqWb3xhP7P5iaWyXV8El7mdteV3ugFA6crocVIzsm pBsbgwzfmA5GiFaiu5IlAfPXwEdd X-Google-Smtp-Source: AOwi7QACemBy8ozmmJZ4ikHMBjJdplLgUvShqzfjOlMdfYFKgVgHLWkTfw5czeA0p08z3XkqXsHZVg== X-Received: by 10.28.136.83 with SMTP id k80mr12670178wmd.159.1507030991716; Tue, 03 Oct 2017 04:43:11 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id k37sm6553666wre.96.2017.10.03.04.43.10 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Oct 2017 04:43:11 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: Date: Tue, 3 Oct 2017 13:43:09 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <63B239EB-47F0-4DDA-982A-794E5B5FC56F@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> To: Freebsd fs , FreeBSD-scsi X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 11:43:14 -0000 > On 03 Oct 2017, at 08:19, Ben RUBSON wrote: >=20 > Hi, >=20 > Putting scsi list as it could be related. >=20 >> On 02 Oct 2017, at 20:12, Ben RUBSON wrote: >>=20 >> Hi, >>=20 >> On a FreeBSD 11 server, the following online/healthy zpool : >>=20 >> home >> mirror-0 >> label/local1 >> label/local2 >> label/iscsi1 >> label/iscsi2 >> mirror-1 >> label/local3 >> label/local4 >> label/iscsi3 >> label/iscsi4 >> cache >> label/local5 >> label/local6 >>=20 >> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >> according to "zpool iostat", nothing on local disks (strange but I >> noticed that IOs always prefer iscsi disks to local disks). >> No write IOs. >>=20 >> Let's disconnect all iSCSI disks : >> iscsictl -Ra >>=20 >> Expected behavior : >> IO activity flawlessly continue on local disks. >>=20 >> What happened : >> All IOs stalled, server only answers to IOs made to its zroot pool. >> All commands related to the iSCSI disks (iscsictl), or to ZFS = (zfs/zpool), >> don't return. >>=20 >> Questions : >> Why this behavior ? >> How to know what happens ? (/var/log/messages says almost nothing) >>=20 >> I already disconnected the iSCSI disks without any issue in the past, >> several times, but there were almost no IOs running. >>=20 >> Thank you for your help ! >>=20 >> Ben >=20 >> On 02 Oct 2017, at 22:55, Andriy Gapon wrote: >>=20 >>> On 02/10/2017 22:13, Ben RUBSON wrote: >>>=20 >>>> On 02 Oct 2017, at 20:45, Andriy Gapon wrote: >>>>=20 >>>>> On 02/10/2017 21:17, Ben RUBSON wrote: >>>>>=20 >>>>> Unfortunately the zpool command stalls / does not return :/ >>>>=20 >>>> Try to take procstat -kk -a. >>>=20 >>> Here is the procstat output : >>> https://benrubson.github.io/zfs/procstat01.log >>=20 >> First, it seems that there are some iscsi threads stuck on a lock = like: >> 0 100291 kernel iscsimt mi_switch+0xd2 = sleepq_wait+0x3a >> _sx_xlock_hard+0x592 iscsi_maintenance_thread+0x316 fork_exit+0x85 >> fork_trampoline+0xe >>=20 >> or like >>=20 >> 8580 102077 iscsictl - mi_switch+0xd2 = sleepq_wait+0x3a >> _sx_slock_hard+0x325 iscsi_ioctl+0x7ea devfs_ioctl_f+0x13f = kern_ioctl+0x2d4 >> sys_ioctl+0x171 amd64_syscall+0x4ce Xfast_syscall+0xfb >>=20 >> Also, there is a thread in cam_sim_free(): >> 0 100986 kernel iscsimt mi_switch+0xd2 = sleepq_wait+0x3a >> _sleep+0x2a1 cam_sim_free+0x48 iscsi_session_cleanup+0x1bd >> iscsi_maintenance_thread+0x388 fork_exit+0x85 fork_trampoline+0xe >>=20 >> So, it looks like there could be a problem is the iscsi teardown = path. >>=20 >> Maybe that caused a domino effect in ZFS code. I see a lot of = threads waiting >> either for spa_namespace_lock or a spa config lock (a highly = specialized ZFS >> lock). But it is hard to untangle their inter-dependencies. >>=20 >> Some of ZFS I/O threads are also affected, for example: >> 0 101538 kernel zio_write_issue_ mi_switch+0xd2 = sleepq_wait+0x3a >> _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 = zio_execute+0x236 >> taskqueue_run_locked+0x14a taskqueue_thread_loop+0xe8 fork_exit+0x85 >> fork_trampoline+0xe >> 8716 101319 sshd - mi_switch+0xd2 = sleepq_wait+0x3a >> _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 = zio_execute+0x236 >> zio_nowait+0x49 arc_read+0x8e4 dbuf_read+0x6c2 = dmu_buf_hold_array_by_dnode+0x1d3 >> dmu_read_uio_dnode+0x41 dmu_read_uio_dbuf+0x3b zfs_freebsd_read+0x5fc >> VOP_READ_APV+0x89 vn_read+0x157 vn_io_fault1+0x1c2 vn_io_fault+0x197 >> dofileread+0x98 >> 71181 101141 encfs - mi_switch+0xd2 = sleepq_wait+0x3a >> _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 = zio_execute+0x236 >> zio_nowait+0x49 arc_read+0x8e4 dbuf_read+0x6c2 dmu_buf_hold+0x3d >> zap_lockdir+0x43 zap_cursor_retrieve+0x171 zfs_freebsd_readdir+0x3f3 >> VOP_READDIR_APV+0x8f kern_getdirentries+0x21b sys_getdirentries+0x28 >> amd64_syscall+0x4ce Xfast_syscall+0xfb >> 71181 101190 encfs - mi_switch+0xd2 = sleepq_wait+0x3a >> _cv_wait+0x194 spa_config_enter+0x9b zio_vdev_io_start+0x1c2 = zio_execute+0x236 >> zio_nowait+0x49 arc_read+0x8e4 dbuf_prefetch_indirect_done+0xcc = arc_read+0x425 >> dbuf_prefetch+0x4f7 dmu_zfetch+0x418 = dmu_buf_hold_array_by_dnode+0x34d >> dmu_read_uio_dnode+0x41 dmu_read_uio_dbuf+0x3b zfs_freebsd_read+0x5fc >> VOP_READ_APV+0x89 vn_read+0x157 >>=20 >> Note that the first of these threads executes a write zio. >>=20 >> It would be nice to determine an owner of spa_namespace_lock. >> If you have debug symbols then it can be easily done in kgdb on the = live system: >> (kgdb) p spa_namespace_lock >=20 > So as said a few minutes ago I lost access to the server and had to = recycle it. > Thankfully I managed to reproduce the issue, re-playing exactly the = same steps. >=20 > Curious line in /var/log/messages : > kernel: g_access(918): provider da18 has error > (da18 is the remaining iSCSI target device which did not disconnect = properly) >=20 > procstat -kk -a : > https://benrubson.github.io/zfs/procstat02.log >=20 > (kgdb) p spa_namespace_lock > $1 =3D -2110867066 This time with debug symbols. procstat -kk -a : https://benrubson.github.io/zfs/procstat03.log (kgdb) p spa_namespace_lock $1 =3D { lock_object =3D { lo_name =3D 0xffffffff822eb986 "spa_namespace_lock",=20 lo_flags =3D 40960000,=20 lo_data =3D 0,=20 lo_witness =3D 0x0 },=20 sx_lock =3D 18446735285324580100 } Easily reproductible. No issue however is there is no IO load. As soon as there is IO load, I can reproduce the issue. Ben From owner-freebsd-fs@freebsd.org Tue Oct 3 14:40:21 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B07C0E3ED48; Tue, 3 Oct 2017 14:40:21 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22c.google.com (mail-wr0-x22c.google.com [IPv6:2a00:1450:400c:c0c::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 493DD77004; Tue, 3 Oct 2017 14:40:21 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22c.google.com with SMTP id u5so6323484wrc.5; Tue, 03 Oct 2017 07:40:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=453TtCA1ft5VW+g9Wy9s0Ho4p5EcUtomwJSbflpk6qY=; b=Um2NeeZyImQIjVmbPcMmkvcEwdmShbpBahrTRuULwQrCVmyDXw9FwQRzUOppwpWGdA Y9R4Or9u62aLxdB9IYAFb5mC8JNXo3F6EN5LnA4lQW7rM3qweIKxdOcJhyrD9AwSV+yi RCeZVZMCHFLSrVPWE34tP3b0pBUjn4LCmwPZ/ypmCY9zY7nT0Z4Y/1NOHAG4pu552o6M PnzfEYX8t/r9lf8dDl2ogfrf3n6Ngu54jtBNyIxftjSM2pKUHejPNVrUEp4mo/JfvnZL eqf3gDP2q8gEw3Frk19E4phnhpN552wRITTxHLAU91H81Sfs0gFYLLwNxN+GbXAn/J7I k6RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=453TtCA1ft5VW+g9Wy9s0Ho4p5EcUtomwJSbflpk6qY=; b=l4QYKyKiw6SiRwveaWFb66uhBn5sw9DtUEVueYDAEBZIMQznrocdraN3fVpqzHRmsH KEEoY/STt3P2TqJ+OjU7M3pBaoocJ5oLFq9C1IFp9gaUztn1tH3SvzyrJav/ciYRGb3c dwWRpOql9M3nJPGSgaC7AJYk8TWUAhqO8U6nhB2gC4trXfrfybH/+3K2cQrqhE8Cs6YX 2IzY/vrXPnGysjwc6PJ0lY0bHWtqQUitl48+OKlX3snL9Mp3DDwtICnuYIoY03KbxU/f 7/CQWE4QNRb3rFOTgiKDMl3Y5/Zd6B4pfNmNMSsCcaePEFK5V8gvRuh7WoVQ8fOWRDfg BTYw== X-Gm-Message-State: AMCzsaVk8zt5rRlCBX0o7uHZHsVTmje/rwPdC9yUCw7PtF2syjfZZdGP WKw3Ae3t/oAVIwlJBX6Lahc6OKUr X-Google-Smtp-Source: AOwi7QAks5RkA2w/emplIiLA0la1EH3lSiJrqnKQ4Tz6cJPJ2t0zpkwg2BaSYYcVoCMvOHXdnCeuUw== X-Received: by 10.223.151.210 with SMTP id t18mr245916wrb.261.1507041619416; Tue, 03 Oct 2017 07:40:19 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id v8sm29638wrg.80.2017.10.03.07.40.18 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Oct 2017 07:40:18 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: ZFS prefers iSCSI disks over local ones ? From: Ben RUBSON In-Reply-To: <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> Date: Tue, 3 Oct 2017 16:40:17 +0200 Cc: Andriy Gapon Content-Transfer-Encoding: quoted-printable Message-Id: <9342D2A7-CE29-445B-9C40-7B6A9C960D59@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> To: Freebsd fs , FreeBSD-scsi , Steven Hartland X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 14:40:21 -0000 Hi, I start a new thread to avoid confusion in the main one. (ZFS stalled after some mirror disks were lost) > On 03 Oct 2017, at 09:39, Steven Hartland wrote: >=20 >> On 03/10/2017 08:31, Ben RUBSON wrote: >>=20 >>> On 03 Oct 2017, at 09:25, Steven Hartland wrote: >>>=20 >>>> On 03/10/2017 07:12, Andriy Gapon wrote: >>>>=20 >>>>> On 02/10/2017 21:12, Ben RUBSON wrote: >>>>>=20 >>>>> Hi, >>>>>=20 >>>>> On a FreeBSD 11 server, the following online/healthy zpool : >>>>>=20 >>>>> home >>>>> mirror-0 >>>>> label/local1 >>>>> label/local2 >>>>> label/iscsi1 >>>>> label/iscsi2 >>>>> mirror-1 >>>>> label/local3 >>>>> label/local4 >>>>> label/iscsi3 >>>>> label/iscsi4 >>>>> cache >>>>> label/local5 >>>>> label/local6 >>>>>=20 >>>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi = disk >>>>> according to "zpool iostat", nothing on local disks (strange but I >>>>> noticed that IOs always prefer iscsi disks to local disks). >>>>=20 >>>> Are your local disks SSD or HDD? >>>> Could it be that iSCSI disks appear to be faster than the local = disks >>>> to the smart ZFS mirror code? >>>>=20 >>>> Steve, what do you think? >>>=20 >>> Yes that quite possible, the mirror balancing uses the queue depth + >>> rotating bias to determine the load of the disk so if your iSCSI = host >>> is processing well and / or is reporting non-rotating vs rotating = for >>> the local disks it could well be the mirror is preferring reads from >>> the the less loaded iSCSI devices. >>=20 >> Note that local & iscsi disks are _exactly_ the same HDD (same model = number, >> same SAS adapter...). So iSCSI ones should be a little bit slower due = to >> network latency (even if it's very low in my case). >=20 > The output from gstat -dp on a loaded machine would be interesting to = see too. So here is the gstat -dp : L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da5 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da7 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da8 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da10 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da11 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da12 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da13 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da14 1 370 370 47326 0.7 0 0 0.0 0 0 0.0 23.2| da15 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da16 0 357 357 45698 1.4 0 0 0.0 0 0 0.0 39.3| da17 0 348 348 44572 0.7 0 0 0.0 0 0 0.0 22.5| da18 0 432 432 55339 0.7 0 0 0.0 0 0 0.0 27.5| da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da21 The 4 active drives are the iSCSI targets of the above quoted pool. A local disk : Geom name: da7 Providers: 1. Name: da7 Mediasize: 4000787030016 (3.6T) Sectorsize: 512 Mode: r0w0e0 descr: HGSTxxx lunid: 5000xxx ident: NHGDxxx rotationrate: 7200 fwsectors: 63 fwheads: 255 A iSCSI disk : Geom name: da19 Providers: 1. Name: da19 Mediasize: 3999688294912 (3.6T) Sectorsize: 512 Mode: r1w1e2 descr: FREEBSD CTLDISK lunname: FREEBSD MYDEVID 12 lunid: FREEBSD MYDEVID 12 ident: iscsi4 rotationrate: 0 fwsectors: 63 fwheads: 255 Sounds like then the faulty thing is the rotationrate set to 0 ? Thx, Ben From owner-freebsd-fs@freebsd.org Tue Oct 3 14:58:24 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57EFCE3F699 for ; Tue, 3 Oct 2017 14:58:24 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D2FE877C88 for ; Tue, 3 Oct 2017 14:58:23 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-wm0-x235.google.com with SMTP id i82so15173186wmd.3 for ; Tue, 03 Oct 2017 07:58:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=2l8y+WmCvuvkO6MuO4FpU34eu/iNFPJi0uNP5KiYwPs=; b=Xbec0zqdBk8hIoxaX/K3OTrJt3fiZMYfjNxXQHWbGSizY1wUTrFjnuy7gOTvTUiKl3 kKO0Hk0EOIkSRt/Ee45+Y7S5UM+0PVLnzoFJGz2CtnHddtu60oZlF4UiN12T+pDN53bI mF2bIf/AehZyQc25DYA+o1ng6YKRPQxrnKKdQjiFrIr5JTa5ZG7HJJsejOMBjSdjNRGu f4p20oFUc6vwc8ngYb79f2Gli5KavSJtRsrDuE6SVdzPacb/SYkrsa5Gn0bS48WWblOY NpzhCfoV/3WsBM7WahmTCfGRIij4obr7vIrwzHMBZmvC5VWlhAEvluss/1ed16gRlfwm hzPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=2l8y+WmCvuvkO6MuO4FpU34eu/iNFPJi0uNP5KiYwPs=; b=DJAsgZ/c92lF8c35k9/mELqMhsMMVZeG2tU2I7zROK6HHpZ2bUT+WP8878cWhWPZKn EAVwrpQJI0rh7M+lWyGPQcCaFyAb6aRTm+UNAc8ur/RfIasf9Rlc/z5H+jrF+ZdHDu/v XazFEEK3p3KIMMRnAcgAeShGf3uWr0yqIIyAwXFhQdZjTOHNp9ArrJ/izjZgFBojrLst hwel3kDLyc0ZRITTpkcXby6JaumndmePS9tgczj0aGP1HqBCuYZlO4GwlVKHpfF+MAzh NJlq64JIedPL8dqNeR9Z3KnjY+vPCywaoXfhjwRZAlhyXGKz0W+KLs9sH/QAfJL8VAuc sqAA== X-Gm-Message-State: AHPjjUjMtUxM9tTK7kpHRGw+HLAYZl67wcSsdU8K4l+AQEUpPQSIXipS 2O6v/bk64SsZY1+zwnC7Mb/aZg== X-Google-Smtp-Source: AOwi7QCGgjl4OXFqHcAb50LuUNJlUhSTB8G8mhp+HEmCsw/lRYlkoblbF5eeQlZDK6ImLfdyUDSmfg== X-Received: by 10.28.109.77 with SMTP id i74mr14805242wmc.67.1507042701050; Tue, 03 Oct 2017 07:58:21 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.1]) by smtp.gmail.com with ESMTPSA id m138sm9043048wmd.29.2017.10.03.07.58.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Oct 2017 07:58:19 -0700 (PDT) Subject: Re: ZFS prefers iSCSI disks over local ones ? To: Ben RUBSON , Freebsd fs , FreeBSD-scsi Cc: Andriy Gapon References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> <9342D2A7-CE29-445B-9C40-7B6A9C960D59@gmail.com> From: Steven Hartland Message-ID: Date: Tue, 3 Oct 2017 15:58:22 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <9342D2A7-CE29-445B-9C40-7B6A9C960D59@gmail.com> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 14:58:24 -0000 On 03/10/2017 15:40, Ben RUBSON wrote: > Hi, > > I start a new thread to avoid confusion in the main one. > (ZFS stalled after some mirror disks were lost) > >> On 03 Oct 2017, at 09:39, Steven Hartland wrote: >> >>> On 03/10/2017 08:31, Ben RUBSON wrote: >>> >>>> On 03 Oct 2017, at 09:25, Steven Hartland wrote: >>>> >>>>> On 03/10/2017 07:12, Andriy Gapon wrote: >>>>> >>>>>> On 02/10/2017 21:12, Ben RUBSON wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> On a FreeBSD 11 server, the following online/healthy zpool : >>>>>> >>>>>> home >>>>>> mirror-0 >>>>>> label/local1 >>>>>> label/local2 >>>>>> label/iscsi1 >>>>>> label/iscsi2 >>>>>> mirror-1 >>>>>> label/local3 >>>>>> label/local4 >>>>>> label/iscsi3 >>>>>> label/iscsi4 >>>>>> cache >>>>>> label/local5 >>>>>> label/local6 >>>>>> >>>>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >>>>>> according to "zpool iostat", nothing on local disks (strange but I >>>>>> noticed that IOs always prefer iscsi disks to local disks). >>>>> Are your local disks SSD or HDD? >>>>> Could it be that iSCSI disks appear to be faster than the local disks >>>>> to the smart ZFS mirror code? >>>>> >>>>> Steve, what do you think? >>>> Yes that quite possible, the mirror balancing uses the queue depth + >>>> rotating bias to determine the load of the disk so if your iSCSI host >>>> is processing well and / or is reporting non-rotating vs rotating for >>>> the local disks it could well be the mirror is preferring reads from >>>> the the less loaded iSCSI devices. >>> Note that local & iscsi disks are _exactly_ the same HDD (same model number, >>> same SAS adapter...). So iSCSI ones should be a little bit slower due to >>> network latency (even if it's very low in my case). >> The output from gstat -dp on a loaded machine would be interesting to see too. > So here is the gstat -dp : > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da0 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da1 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da2 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da3 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da4 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da5 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da6 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da7 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da8 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da9 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da10 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da11 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da12 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da13 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da14 > 1 370 370 47326 0.7 0 0 0.0 0 0 0.0 23.2| da15 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da16 > 0 357 357 45698 1.4 0 0 0.0 0 0 0.0 39.3| da17 > 0 348 348 44572 0.7 0 0 0.0 0 0 0.0 22.5| da18 > 0 432 432 55339 0.7 0 0 0.0 0 0 0.0 27.5| da19 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da20 > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da21 > > The 4 active drives are the iSCSI targets of the above quoted pool. > > A local disk : > > Geom name: da7 > Providers: > 1. Name: da7 > Mediasize: 4000787030016 (3.6T) > Sectorsize: 512 > Mode: r0w0e0 > descr: HGSTxxx > lunid: 5000xxx > ident: NHGDxxx > rotationrate: 7200 > fwsectors: 63 > fwheads: 255 > > A iSCSI disk : > > Geom name: da19 > Providers: > 1. Name: da19 > Mediasize: 3999688294912 (3.6T) > Sectorsize: 512 > Mode: r1w1e2 > descr: FREEBSD CTLDISK > lunname: FREEBSD MYDEVID 12 > lunid: FREEBSD MYDEVID 12 > ident: iscsi4 > rotationrate: 0 > fwsectors: 63 > fwheads: 255 > > Sounds like then the faulty thing is the rotationrate set to 0 ? > > Absolutely and from the looks you're not stressing the iSCSI disks so they get high queuing depths hence the preference. As load increased I would expect the local disks to start seeing activity.     Regards     Steve From owner-freebsd-fs@freebsd.org Tue Oct 3 15:03:22 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 243E8E3F999; Tue, 3 Oct 2017 15:03:22 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x233.google.com (mail-wm0-x233.google.com [IPv6:2a00:1450:400c:c09::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A98907C4AB; Tue, 3 Oct 2017 15:03:21 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x233.google.com with SMTP id i82so15205575wmd.3; Tue, 03 Oct 2017 08:03:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=hCrCM1uH3lHhMowCEUEfzBE9AYQxWOTzX0bceRvaYzA=; b=TKoTJTM/ji+z9gD0WsHrPQ7DA4hc4HUxAVpQuQVXKVUELu5R9BQnAvaKN2eVK52cSb yRr858lJ5muzWHDWEIqrd47JQ4lOUNpbTEqGhNqlbhItyxKL0Yf/F/AbvUMorNWylIF4 fTM+lxh0BKsYQICw89FSw/6tnlhDUGOBWvtApvPL0k2ogN3MfY9LxlfDGeXnEyzXbOEP 092H6ZwCIjDqrycWOTqtjyrZFHHe3SACeDI+AKl033dVuZGqBKPPyDzwoPBX6mCcb6if cSQCaV/SLF2qvFx8FBZEIy9Pz4IkGrcJ6RZkFTQVZohRXY+Kvj00JlZpMwj4h1qth2nA 3F3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=hCrCM1uH3lHhMowCEUEfzBE9AYQxWOTzX0bceRvaYzA=; b=bnoDmMOtLGd7AY5ANfqmGWDBpTnkJMUwnvAjuhyUV0NE2UfyWP4D304Q2UjIQ+M9sa vkeqhPAIB8xwb9Az494FprBpN+ffoi997Rii7IkUfXwgIg0u3f3hqPtuyerYdsTRvr24 Duqcj9/tXZ5cynUJSPA1k7wQx3EZHqXWlelAu7uaw5uceGp3Hg2z2c8ICqD5Rpl+RtOQ H7pB+QLgZ7eWX5skdLiEU0xwAO8+OfFzLRP63u8zkhaUjoAx4lDxYmcjDHn96KlMU/aT h+m6US2r/d2fRklFzEGKeqic5uEfhh9i7VTNxSr/pOeN4hO65gQ3F36MxsSyypNgK47D FY8Q== X-Gm-Message-State: AMCzsaXX+z98ElmHrxF7WdZZjvrBTKrWLN2KuoIcuVZ486k3a6hj4gSy sgobxpR5WoP0LkgBsFquJFuRcmWv X-Google-Smtp-Source: AOwi7QC4rO1x6Z979fymPYKCC/QWMczBWFNllPnUrHDkiirjL7UqkllZobQZ8VRMTtgA8hKwMzrDDA== X-Received: by 10.28.232.138 with SMTP id f10mr2683080wmi.130.1507043000017; Tue, 03 Oct 2017 08:03:20 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id p78sm23655244wma.11.2017.10.03.08.03.19 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Oct 2017 08:03:19 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS prefers iSCSI disks over local ones ? From: Ben RUBSON In-Reply-To: Date: Tue, 3 Oct 2017 17:03:18 +0200 Cc: Andriy Gapon Content-Transfer-Encoding: quoted-printable Message-Id: References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> <9342D2A7-CE29-445B-9C40-7B6A9C960D59@gmail.com> To: Steven Hartland , FreeBSD-scsi , Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 15:03:22 -0000 > On 03 Oct 2017, at 16:58, Steven Hartland = wrote: >=20 > On 03/10/2017 15:40, Ben RUBSON wrote: >> Hi, >>=20 >> I start a new thread to avoid confusion in the main one. >> (ZFS stalled after some mirror disks were lost) >>=20 >>=20 >>> On 03 Oct 2017, at 09:39, Steven Hartland wrote: >>>=20 >>>=20 >>>> On 03/10/2017 08:31, Ben RUBSON wrote: >>>>=20 >>>>=20 >>>>> On 03 Oct 2017, at 09:25, Steven Hartland wrote: >>>>>=20 >>>>>=20 >>>>>> On 03/10/2017 07:12, Andriy Gapon wrote: >>>>>>=20 >>>>>>=20 >>>>>>> On 02/10/2017 21:12, Ben RUBSON wrote: >>>>>>>=20 >>>>>>> Hi, >>>>>>>=20 >>>>>>> On a FreeBSD 11 server, the following online/healthy zpool : >>>>>>>=20 >>>>>>> home >>>>>>> mirror-0 >>>>>>> label/local1 >>>>>>> label/local2 >>>>>>> label/iscsi1 >>>>>>> label/iscsi2 >>>>>>> mirror-1 >>>>>>> label/local3 >>>>>>> label/local4 >>>>>>> label/iscsi3 >>>>>>> label/iscsi4 >>>>>>> cache >>>>>>> label/local5 >>>>>>> label/local6 >>>>>>>=20 >>>>>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi = disk >>>>>>> according to "zpool iostat", nothing on local disks (strange but = I >>>>>>> noticed that IOs always prefer iscsi disks to local disks). >>>>>>>=20 >>>>>> Are your local disks SSD or HDD? >>>>>> Could it be that iSCSI disks appear to be faster than the local = disks >>>>>> to the smart ZFS mirror code? >>>>>>=20 >>>>>> Steve, what do you think? >>>>>>=20 >>>>> Yes that quite possible, the mirror balancing uses the queue depth = + >>>>> rotating bias to determine the load of the disk so if your iSCSI = host >>>>> is processing well and / or is reporting non-rotating vs rotating = for >>>>> the local disks it could well be the mirror is preferring reads = from >>>>> the the less loaded iSCSI devices. >>>>>=20 >>>> Note that local & iscsi disks are _exactly_ the same HDD (same = model number, >>>> same SAS adapter...). So iSCSI ones should be a little bit slower = due to >>>> network latency (even if it's very low in my case). >>>>=20 >>> The output from gstat -dp on a loaded machine would be interesting = to see too. >>>=20 >> So here is the gstat -dp : >>=20 >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da0 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da1 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da2 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da3 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da4 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da5 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da6 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da7 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da8 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da9 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da10 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da11 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da12 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da13 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da14 >> 1 370 370 47326 0.7 0 0 0.0 0 0 0.0 23.2| da15 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da16 >> 0 357 357 45698 1.4 0 0 0.0 0 0 0.0 39.3| da17 >> 0 348 348 44572 0.7 0 0 0.0 0 0 0.0 22.5| da18 >> 0 432 432 55339 0.7 0 0 0.0 0 0 0.0 27.5| da19 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da20 >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da21 >>=20 >> The 4 active drives are the iSCSI targets of the above quoted pool. >>=20 >> A local disk : >>=20 >> Geom name: da7 >> Providers: >> 1. Name: da7 >> Mediasize: 4000787030016 (3.6T) >> Sectorsize: 512 >> Mode: r0w0e0 >> descr: HGSTxxx >> lunid: 5000xxx >> ident: NHGDxxx >> rotationrate: 7200 >> fwsectors: 63 >> fwheads: 255 >>=20 >> A iSCSI disk : >>=20 >> Geom name: da19 >> Providers: >> 1. Name: da19 >> Mediasize: 3999688294912 (3.6T) >> Sectorsize: 512 >> Mode: r1w1e2 >> descr: FREEBSD CTLDISK >> lunname: FREEBSD MYDEVID 12 >> lunid: FREEBSD MYDEVID 12 >> ident: iscsi4 >> rotationrate: 0 >> fwsectors: 63 >> fwheads: 255 >>=20 >> Sounds like then the faulty thing is the rotationrate set to 0 ? >=20 > Absolutely Good catch then, thank you ! > and from the looks you're not stressing the iSCSI disks so they get = high queuing depths hence the preference. > As load increased I would expect the local disks to start seeing = activity. Yes this is also what I see. Any way however to set rotationrate to 7200 (or to a slightly greater = value) as well for iSCSI drives ? I looked through ctl.conf(5) and iscsi.conf(5) but did not found = anything related. Many thanks ! Ben From owner-freebsd-fs@freebsd.org Tue Oct 3 15:07:37 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 176EFE3FB48; Tue, 3 Oct 2017 15:07:37 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9B2BD7C6DF; Tue, 3 Oct 2017 15:07:36 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22e.google.com with SMTP id b189so13483935wmd.4; Tue, 03 Oct 2017 08:07:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=H+N/UGMzAC7gjaMlOP9gGicMWAUPQPwxkCg1OWuqB8Y=; b=bKMP5ilKNmkLzj+iTrpm26KJ6hFJNbLFB4Hd9EW2PMgJOD5CO2hqT+EWZzH+ZMOWu8 G2PvlDden0jHlYIwWW1w8WyjCaex5btTaZKvx/ib22VTwAyQJgtHqOt6+D0ruRKIKM8S SiCN+PAsYcZWST1TOaz1QduJmnaIPY5O337DQ+lkyLrojxoF7idMi9MgZ/wisRz/hfvj MvHhB8ObC5wbdkS/h1Wr3n1jRuCd6rDnYi7bBiuPKOPzp3jf3KKQWWPdsj2NwIxbhXJ0 W0ojeuzqNxw67E15s0l0r2UxVD4j0C8ONazSnOXm5SfIu374WZimAF8pX8McWfKf8atp 4wkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=H+N/UGMzAC7gjaMlOP9gGicMWAUPQPwxkCg1OWuqB8Y=; b=oUyNu9ZWoSIfNqJDmssjOcwHCJYfD21C8a0SSW1yRPsUN01k4J+Ju75EuTo6cuJPLZ XxTRMbeu0vwi6tvz89Rjb34NoFsbkKdYIkGRhws6ju8i6SuWMp6aDcFeVpCLJdoD2wsl ykeycw3btO0dGDXuJ2AiDZ6AFaLG0CrBVCTkojlqPRjIL8Ognxmhy+292eqOaWVJs3BI zOaWv3bbMTWnUXTHxHHX07hhkFyOqgs0RqxS2DqU/qI6QzWbC0LNXnoLko7ecRn8Bo0y kl5iN+J7+LxxsxznSkQTx/HnQ1lhMMpfPmyyceW86F4AsBA3a6c3kbveMXzgilh0X29v uF8Q== X-Gm-Message-State: AMCzsaWMpAq9M/c5GrYzdyJ9pZ+fHHthOa+iECHgf/Ue0TT8DuOxFi85 BQbj4YnomTzHpI0hd1g/pNVp9k2w X-Google-Smtp-Source: AOwi7QC8FzAIEnk5HznczseaUCUU/NeoDBVCSmWkcb4pqLXDyP3LvFdzB0fzvRp//rTyQWEZMGR5TA== X-Received: by 10.28.209.2 with SMTP id i2mr4235886wmg.153.1507043254956; Tue, 03 Oct 2017 08:07:34 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id m8sm2724283wrg.55.2017.10.03.08.07.34 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Oct 2017 08:07:34 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS prefers iSCSI disks over local ones ? From: Ben RUBSON In-Reply-To: Date: Tue, 3 Oct 2017 17:07:33 +0200 Cc: Andriy Gapon Content-Transfer-Encoding: quoted-printable Message-Id: <49ADB654-E68B-4B88-AE8E-49F755092848@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> <9342D2A7-CE29-445B-9C40-7B6A9C960D59@gmail.com> To: Steven Hartland , FreeBSD-scsi , Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 15:07:37 -0000 > On 03 Oct 2017, at 17:03, Ben RUBSON wrote: >=20 >> On 03 Oct 2017, at 16:58, Steven Hartland = wrote: >>=20 >> On 03/10/2017 15:40, Ben RUBSON wrote: >>> Hi, >>>=20 >>> I start a new thread to avoid confusion in the main one. >>> (ZFS stalled after some mirror disks were lost) >>>=20 >>>=20 >>>> On 03 Oct 2017, at 09:39, Steven Hartland wrote: >>>>=20 >>>>=20 >>>>> On 03/10/2017 08:31, Ben RUBSON wrote: >>>>>=20 >>>>>=20 >>>>>> On 03 Oct 2017, at 09:25, Steven Hartland wrote: >>>>>>=20 >>>>>>=20 >>>>>>> On 03/10/2017 07:12, Andriy Gapon wrote: >>>>>>>=20 >>>>>>>=20 >>>>>>>> On 02/10/2017 21:12, Ben RUBSON wrote: >>>>>>>>=20 >>>>>>>> Hi, >>>>>>>>=20 >>>>>>>> On a FreeBSD 11 server, the following online/healthy zpool : >>>>>>>>=20 >>>>>>>> home >>>>>>>> mirror-0 >>>>>>>> label/local1 >>>>>>>> label/local2 >>>>>>>> label/iscsi1 >>>>>>>> label/iscsi2 >>>>>>>> mirror-1 >>>>>>>> label/local3 >>>>>>>> label/local4 >>>>>>>> label/iscsi3 >>>>>>>> label/iscsi4 >>>>>>>> cache >>>>>>>> label/local5 >>>>>>>> label/local6 >>>>>>>>=20 >>>>>>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi = disk >>>>>>>> according to "zpool iostat", nothing on local disks (strange = but I >>>>>>>> noticed that IOs always prefer iscsi disks to local disks). >>>>>>>>=20 >>>>>>> Are your local disks SSD or HDD? >>>>>>> Could it be that iSCSI disks appear to be faster than the local = disks >>>>>>> to the smart ZFS mirror code? >>>>>>>=20 >>>>>>> Steve, what do you think? >>>>>>>=20 >>>>>> Yes that quite possible, the mirror balancing uses the queue = depth + >>>>>> rotating bias to determine the load of the disk so if your iSCSI = host >>>>>> is processing well and / or is reporting non-rotating vs rotating = for >>>>>> the local disks it could well be the mirror is preferring reads = from >>>>>> the the less loaded iSCSI devices. >>>>>>=20 >>>>> Note that local & iscsi disks are _exactly_ the same HDD (same = model number, >>>>> same SAS adapter...). So iSCSI ones should be a little bit slower = due to >>>>> network latency (even if it's very low in my case). >>>>>=20 >>>> The output from gstat -dp on a loaded machine would be interesting = to see too. >>>>=20 >>> So here is the gstat -dp : >>>=20 >>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da0 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da1 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da2 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da3 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da4 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da5 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da6 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da7 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da8 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da9 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da10 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da11 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da12 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da13 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da14 >>> 1 370 370 47326 0.7 0 0 0.0 0 0 0.0 23.2| da15 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da16 >>> 0 357 357 45698 1.4 0 0 0.0 0 0 0.0 39.3| da17 >>> 0 348 348 44572 0.7 0 0 0.0 0 0 0.0 22.5| da18 >>> 0 432 432 55339 0.7 0 0 0.0 0 0 0.0 27.5| da19 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da20 >>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da21 >>>=20 >>> The 4 active drives are the iSCSI targets of the above quoted pool. >>>=20 >>> A local disk : >>>=20 >>> Geom name: da7 >>> Providers: >>> 1. Name: da7 >>> Mediasize: 4000787030016 (3.6T) >>> Sectorsize: 512 >>> Mode: r0w0e0 >>> descr: HGSTxxx >>> lunid: 5000xxx >>> ident: NHGDxxx >>> rotationrate: 7200 >>> fwsectors: 63 >>> fwheads: 255 >>>=20 >>> A iSCSI disk : >>>=20 >>> Geom name: da19 >>> Providers: >>> 1. Name: da19 >>> Mediasize: 3999688294912 (3.6T) >>> Sectorsize: 512 >>> Mode: r1w1e2 >>> descr: FREEBSD CTLDISK >>> lunname: FREEBSD MYDEVID 12 >>> lunid: FREEBSD MYDEVID 12 >>> ident: iscsi4 >>> rotationrate: 0 >>> fwsectors: 63 >>> fwheads: 255 >>>=20 >>> Sounds like then the faulty thing is the rotationrate set to 0 ? >>=20 >> Absolutely >=20 > Good catch then, thank you ! >=20 >> and from the looks you're not stressing the iSCSI disks so they get = high queuing depths hence the preference. >> As load increased I would expect the local disks to start seeing = activity. >=20 > Yes this is also what I see. >=20 > Any way however to set rotationrate to 7200 (or to a slightly greater = value (*)) as well for iSCSI drives ? > I looked through ctl.conf(5) and iscsi.conf(5) but did not found = anything related. Sorry, (*) or to a slightly lower value (of course...). I forgot to mention that as the initiator, target is a FreeBSD 11.0 = server. Ben From owner-freebsd-fs@freebsd.org Tue Oct 3 15:18:53 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F172E3FEB4; Tue, 3 Oct 2017 15:18:53 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 68DA97CC16; Tue, 3 Oct 2017 15:18:53 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.89 (FreeBSD)) (envelope-from ) id 1dzOyB-0003Rn-1F; Tue, 03 Oct 2017 16:18:51 +0100 Date: Tue, 3 Oct 2017 16:18:50 +0100 From: Gary Palmer To: Ben RUBSON Cc: Steven Hartland , FreeBSD-scsi , Freebsd fs , Andriy Gapon Subject: Re: ZFS prefers iSCSI disks over local ones ? Message-ID: <20171003151850.GA65538@in-addr.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> <9342D2A7-CE29-445B-9C40-7B6A9C960D59@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 15:18:53 -0000 On Tue, Oct 03, 2017 at 05:03:18PM +0200, Ben RUBSON wrote: > > On 03 Oct 2017, at 16:58, Steven Hartland wrote: > > > > On 03/10/2017 15:40, Ben RUBSON wrote: > >> Hi, > >> > >> I start a new thread to avoid confusion in the main one. > >> (ZFS stalled after some mirror disks were lost) > >> > >> > >>> On 03 Oct 2017, at 09:39, Steven Hartland wrote: > >>> > >>> > >>>> On 03/10/2017 08:31, Ben RUBSON wrote: > >>>> > >>>> > >>>>> On 03 Oct 2017, at 09:25, Steven Hartland wrote: > >>>>> > >>>>> > >>>>>> On 03/10/2017 07:12, Andriy Gapon wrote: > >>>>>> > >>>>>> > >>>>>>> On 02/10/2017 21:12, Ben RUBSON wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> On a FreeBSD 11 server, the following online/healthy zpool : > >>>>>>> > >>>>>>> home > >>>>>>> mirror-0 > >>>>>>> label/local1 > >>>>>>> label/local2 > >>>>>>> label/iscsi1 > >>>>>>> label/iscsi2 > >>>>>>> mirror-1 > >>>>>>> label/local3 > >>>>>>> label/local4 > >>>>>>> label/iscsi3 > >>>>>>> label/iscsi4 > >>>>>>> cache > >>>>>>> label/local5 > >>>>>>> label/local6 > >>>>>>> > >>>>>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk > >>>>>>> according to "zpool iostat", nothing on local disks (strange but I > >>>>>>> noticed that IOs always prefer iscsi disks to local disks). > >>>>>>> > >>>>>> Are your local disks SSD or HDD? > >>>>>> Could it be that iSCSI disks appear to be faster than the local disks > >>>>>> to the smart ZFS mirror code? > >>>>>> > >>>>>> Steve, what do you think? > >>>>>> > >>>>> Yes that quite possible, the mirror balancing uses the queue depth + > >>>>> rotating bias to determine the load of the disk so if your iSCSI host > >>>>> is processing well and / or is reporting non-rotating vs rotating for > >>>>> the local disks it could well be the mirror is preferring reads from > >>>>> the the less loaded iSCSI devices. > >>>>> > >>>> Note that local & iscsi disks are _exactly_ the same HDD (same model number, > >>>> same SAS adapter...). So iSCSI ones should be a little bit slower due to > >>>> network latency (even if it's very low in my case). > >>>> > >>> The output from gstat -dp on a loaded machine would be interesting to see too. > >>> > >> So here is the gstat -dp : > >> > >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da0 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da1 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da2 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da3 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da4 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da5 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da6 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da7 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da8 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da9 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da10 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da11 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da12 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da13 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da14 > >> 1 370 370 47326 0.7 0 0 0.0 0 0 0.0 23.2| da15 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da16 > >> 0 357 357 45698 1.4 0 0 0.0 0 0 0.0 39.3| da17 > >> 0 348 348 44572 0.7 0 0 0.0 0 0 0.0 22.5| da18 > >> 0 432 432 55339 0.7 0 0 0.0 0 0 0.0 27.5| da19 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da20 > >> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da21 > >> > >> The 4 active drives are the iSCSI targets of the above quoted pool. > >> > >> A local disk : > >> > >> Geom name: da7 > >> Providers: > >> 1. Name: da7 > >> Mediasize: 4000787030016 (3.6T) > >> Sectorsize: 512 > >> Mode: r0w0e0 > >> descr: HGSTxxx > >> lunid: 5000xxx > >> ident: NHGDxxx > >> rotationrate: 7200 > >> fwsectors: 63 > >> fwheads: 255 > >> > >> A iSCSI disk : > >> > >> Geom name: da19 > >> Providers: > >> 1. Name: da19 > >> Mediasize: 3999688294912 (3.6T) > >> Sectorsize: 512 > >> Mode: r1w1e2 > >> descr: FREEBSD CTLDISK > >> lunname: FREEBSD MYDEVID 12 > >> lunid: FREEBSD MYDEVID 12 > >> ident: iscsi4 > >> rotationrate: 0 > >> fwsectors: 63 > >> fwheads: 255 > >> > >> Sounds like then the faulty thing is the rotationrate set to 0 ? > > > > Absolutely > > Good catch then, thank you ! > > > and from the looks you're not stressing the iSCSI disks so they get high queuing depths hence the preference. > > As load increased I would expect the local disks to start seeing activity. > > Yes this is also what I see. > > Any way however to set rotationrate to 7200 (or to a slightly greater value) as well for iSCSI drives ? > I looked through ctl.conf(5) and iscsi.conf(5) but did not found anything related. > > Many thanks ! Use the "option" setting in ctl.conf to change the rpm value (documented in the OPTIONS section of ctladm(8)). Regards, Gary From owner-freebsd-fs@freebsd.org Tue Oct 3 15:37:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1BB29E407E9; Tue, 3 Oct 2017 15:37:35 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x233.google.com (mail-wr0-x233.google.com [IPv6:2a00:1450:400c:c0c::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8FF477E0A4; Tue, 3 Oct 2017 15:37:34 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x233.google.com with SMTP id r79so1290727wrb.13; Tue, 03 Oct 2017 08:37:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=kf3mYkpEpeo5BfnWkpMJiMD5eg9W97WYhG8Brs50U0w=; b=aBsoKuGdj9e1wmVyB32IGlKFEnm4hwtklujvk8EiOeOcjb0utxKwoODnjutnFTBFG4 8rOCQwfH3KCrB8lMMTiY7x/TPKxZzSUSbCjpxacy4/3pD4Nq6PzVkfEAY3Vx47yUPLW+ sq2iZ6Vi/+3l8hEsZwzSZTzxuiawHzodOrpEc+NAeQV0hEQlrOFV9MEr8bk8Mcqoxc26 WgsoGtlqOPjI+ddk0M3ax1xslTZz8KsnCj2f2srBTpIqM13Dkb6eDM/mTFmBpKKQaNnw SEnVpOMiqpSeumLmdDwe+W5jv0Vk23VDC7RadkcQQsgCMWD+NT2/iqOPZaYCo0avt8Jn 3J1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=kf3mYkpEpeo5BfnWkpMJiMD5eg9W97WYhG8Brs50U0w=; b=cEaeo853jRwsEA1KJyFBawp5xUI31KrQVC0vC5jMVIDjHlgGQgOT2RtoMxQ2sLHygI 7YQd23Jcdjjv7ESjidAI6eVyjVNd/yEHl/s/0pdHa3oRcGKf3O1+re7tey95DHDXrCUI 6B89p3XWgv2HSRBoc6D7vDliC0oHCILg8zK4Hx9hJRjcE5zG3o61rw3blwn6E/FgAL0e gxdP4FYe6O7p0wx6GHeiZn/17v61inS7ObPjqHb0bBoFv0aI8nyI+dibppPoEa21TnDM UNmD471TOC05C2QWJu1f4p/9k1P8R5e9JI3LPnxHFKka2cbr3OH4QHcS0xFZf5GYWw63 u9aQ== X-Gm-Message-State: AHPjjUgRmHQGKi+5gYOFLm3Mjq3zOcXWTcABfWnQQI2ar997ikXW7LUt f5Bu8KYnnA5UMLQvceBHzcA= X-Google-Smtp-Source: AOwi7QCBkPbOPxQSsNRGVn8jb0rk/dd/6RhGemrHMigk48Ya5jJ+W+kJP+GAHJo2Y7UjsrIlUGIiag== X-Received: by 10.223.155.203 with SMTP id e11mr13482670wrc.218.1507045053090; Tue, 03 Oct 2017 08:37:33 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id n57sm19561773wrn.29.2017.10.03.08.37.32 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Oct 2017 08:37:32 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS prefers iSCSI disks over local ones ? From: Ben RUBSON In-Reply-To: <20171003172857.2497b931@mwoffice.virtualtec.office> Date: Tue, 3 Oct 2017 17:37:30 +0200 Cc: Andriy Gapon , Steven Hartland Content-Transfer-Encoding: quoted-printable Message-Id: <919C4A38-5192-4AED-BC6A-FBED8EFD6B31@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> <9342D2A7-CE29-445B-9C40-7B6A9C960D59@gmail.com> <49ADB654-E68B-4B88-AE8E-49F755092848@gmail.com> <20171003172857.2497b931@mwoffice.virtualtec.office> To: Markus Wild , FreeBSD-scsi , Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 15:37:35 -0000 > On 03 Oct 2017, at 17:28, Markus Wild wrote: >=20 >>> Any way however to set rotationrate to 7200 (or to a slightly = greater value (*)) as well for iSCSI drives ? >>> I looked through ctl.conf(5) and iscsi.conf(5) but did not found = anything related. =20 >>=20 >> Sorry, (*) or to a slightly lower value (of course...). >> I forgot to mention that as the initiator, target is a FreeBSD 11.0 = server. >=20 > We use this in our ctl.conf to ensure vmware doesn't consider the = iscsi volumes to be ssd drives: >=20 > [...] > lun 1 { path /dev/zvol/data/volumes/zvol1 ; option rpm 10000 } > [...] Markus, thank you very much for the tip ! I'll test this as soon as my production will be fully online. Perfect ! :) Best, Ben= From owner-freebsd-fs@freebsd.org Tue Oct 3 15:40:25 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 183B8E40A14; Tue, 3 Oct 2017 15:40:25 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x233.google.com (mail-wr0-x233.google.com [IPv6:2a00:1450:400c:c0c::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B396A7E6C6; Tue, 3 Oct 2017 15:40:24 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x233.google.com with SMTP id 54so6649055wrz.10; Tue, 03 Oct 2017 08:40:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=FSE6usK7Y8BeHJ1vB2YCw29sv4py6eOBtrt4Z1muMPs=; b=tjQLJ7HEyqC3Ne8lBwZ2X/zmKkev5ktDsQ40Kwl3ZNfeTjxQi2SU7NJfDa+klfKiv0 OIwDz24K0+ETcObhfbvUezspResLUnEhCxUQ0Nl8c0Y5lpzqEofkw54jxb1CYEQqP5K1 uREb58aCoKkdWuWh0IgigAU3TR0zecjqMOHU5YEpWpEBvBYxngHWXHqWd+r3sxhvJuVB XsTbajoojVSg6hjj9SGxIQTOCK74NSkTq4xZ48HDIzK07NzCokTP6rwOmKLPqjSaMnv9 QUyyjELdG+jOeB8qnPwuRLqhlQlu6W58moiNMnSozx3KIA8BU93JnXyqYggAXzXMVmJp Ujpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=FSE6usK7Y8BeHJ1vB2YCw29sv4py6eOBtrt4Z1muMPs=; b=dKM7qifoWMImcXBSTTRxFdnXjCiIXUJQe2sZq5cFrZ+9ifB1JnfRL3D8BAW/zbMpNb vlhUVERHtSJ4dYUFb8ebPH5G8C4nJYSqu9+uDxSTY4fHlHG56CN0Hw92eoHI63ua9LuN 5rfzeK31dJPJbx0Zp/JdrbiC2FgVrCrAR65VhxPYPpJ48jFFdDV5RNfxIqi7ZS4qtWz2 0Hkm9vyMySbvWpwBmEbn5G2z8JZPtvsMpvWxn3sfcRhCCZMOHdHcTSoSp9m3NzKsJK3v PeiyiG6ZXj3Abk/fiFBZs/LbCKRRo+gMWAfP45jvO2RDCTlcY3OrusOMzLdVEPL2qFtI Og4g== X-Gm-Message-State: AMCzsaW0653F+xyassHVQ2C+pnrmd/DWcs/p3eOtx4QOgtEBueNrBkKV eF5VmHXoBLyvtP7FzxacaaaxY4gnh/g= X-Google-Smtp-Source: AOwi7QBu1cG2mB/ax3xgwxZYYa3fd6Cy6w9FPD5NqP+PSqnqx0tuHYrM4BOnL49KSqFFr1UujujjWQ== X-Received: by 10.223.171.73 with SMTP id r9mr3244715wrc.118.1507045222910; Tue, 03 Oct 2017 08:40:22 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id d18sm7277435wra.89.2017.10.03.08.40.22 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Oct 2017 08:40:22 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS prefers iSCSI disks over local ones ? From: Ben RUBSON In-Reply-To: <20171003151850.GA65538@in-addr.com> Date: Tue, 3 Oct 2017 17:40:21 +0200 Cc: Steven Hartland , FreeBSD-scsi , Freebsd fs , Andriy Gapon Content-Transfer-Encoding: quoted-printable Message-Id: References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> <9342D2A7-CE29-445B-9C40-7B6A9C960D59@gmail.com> <20171003151850.GA65538@in-addr.com> To: Gary Palmer X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 15:40:25 -0000 > On 03 Oct 2017, at 17:18, Gary Palmer wrote: >=20 > On Tue, Oct 03, 2017 at 05:03:18PM +0200, Ben RUBSON wrote: >>> On 03 Oct 2017, at 16:58, Steven Hartland = wrote: >>>=20 >>> On 03/10/2017 15:40, Ben RUBSON wrote: >>>> Hi, >>>>=20 >>>> I start a new thread to avoid confusion in the main one. >>>> (ZFS stalled after some mirror disks were lost) >>>>=20 >>>>=20 >>>>> On 03 Oct 2017, at 09:39, Steven Hartland wrote: >>>>>=20 >>>>>=20 >>>>>> On 03/10/2017 08:31, Ben RUBSON wrote: >>>>>>=20 >>>>>>=20 >>>>>>> On 03 Oct 2017, at 09:25, Steven Hartland wrote: >>>>>>>=20 >>>>>>>=20 >>>>>>>> On 03/10/2017 07:12, Andriy Gapon wrote: >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>> On 02/10/2017 21:12, Ben RUBSON wrote: >>>>>>>>>=20 >>>>>>>>> Hi, >>>>>>>>>=20 >>>>>>>>> On a FreeBSD 11 server, the following online/healthy zpool : >>>>>>>>>=20 >>>>>>>>> home >>>>>>>>> mirror-0 >>>>>>>>> label/local1 >>>>>>>>> label/local2 >>>>>>>>> label/iscsi1 >>>>>>>>> label/iscsi2 >>>>>>>>> mirror-1 >>>>>>>>> label/local3 >>>>>>>>> label/local4 >>>>>>>>> label/iscsi3 >>>>>>>>> label/iscsi4 >>>>>>>>> cache >>>>>>>>> label/local5 >>>>>>>>> label/local6 >>>>>>>>>=20 >>>>>>>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi = disk >>>>>>>>> according to "zpool iostat", nothing on local disks (strange = but I >>>>>>>>> noticed that IOs always prefer iscsi disks to local disks). >>>>>>>>>=20 >>>>>>>> Are your local disks SSD or HDD? >>>>>>>> Could it be that iSCSI disks appear to be faster than the local = disks >>>>>>>> to the smart ZFS mirror code? >>>>>>>>=20 >>>>>>>> Steve, what do you think? >>>>>>>>=20 >>>>>>> Yes that quite possible, the mirror balancing uses the queue = depth + >>>>>>> rotating bias to determine the load of the disk so if your iSCSI = host >>>>>>> is processing well and / or is reporting non-rotating vs = rotating for >>>>>>> the local disks it could well be the mirror is preferring reads = from >>>>>>> the the less loaded iSCSI devices. >>>>>>>=20 >>>>>> Note that local & iscsi disks are _exactly_ the same HDD (same = model number, >>>>>> same SAS adapter...). So iSCSI ones should be a little bit slower = due to >>>>>> network latency (even if it's very low in my case). >>>>>>=20 >>>>> The output from gstat -dp on a loaded machine would be interesting = to see too. >>>>>=20 >>>> So here is the gstat -dp : >>>>=20 >>>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da0 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da1 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da2 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da3 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da4 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da5 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da6 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da7 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da8 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da9 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da10 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da11 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da12 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da13 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da14 >>>> 1 370 370 47326 0.7 0 0 0.0 0 0 0.0 23.2| da15 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da16 >>>> 0 357 357 45698 1.4 0 0 0.0 0 0 0.0 39.3| da17 >>>> 0 348 348 44572 0.7 0 0 0.0 0 0 0.0 22.5| da18 >>>> 0 432 432 55339 0.7 0 0 0.0 0 0 0.0 27.5| da19 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da20 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da21 >>>>=20 >>>> The 4 active drives are the iSCSI targets of the above quoted pool. >>>>=20 >>>> A local disk : >>>>=20 >>>> Geom name: da7 >>>> Providers: >>>> 1. Name: da7 >>>> Mediasize: 4000787030016 (3.6T) >>>> Sectorsize: 512 >>>> Mode: r0w0e0 >>>> descr: HGSTxxx >>>> lunid: 5000xxx >>>> ident: NHGDxxx >>>> rotationrate: 7200 >>>> fwsectors: 63 >>>> fwheads: 255 >>>>=20 >>>> A iSCSI disk : >>>>=20 >>>> Geom name: da19 >>>> Providers: >>>> 1. Name: da19 >>>> Mediasize: 3999688294912 (3.6T) >>>> Sectorsize: 512 >>>> Mode: r1w1e2 >>>> descr: FREEBSD CTLDISK >>>> lunname: FREEBSD MYDEVID 12 >>>> lunid: FREEBSD MYDEVID 12 >>>> ident: iscsi4 >>>> rotationrate: 0 >>>> fwsectors: 63 >>>> fwheads: 255 >>>>=20 >>>> Sounds like then the faulty thing is the rotationrate set to 0 ? >>>=20 >>> Absolutely >>=20 >> Good catch then, thank you ! >>=20 >>> and from the looks you're not stressing the iSCSI disks so they get = high queuing depths hence the preference. >>> As load increased I would expect the local disks to start seeing = activity. >>=20 >> Yes this is also what I see. >>=20 >> Any way however to set rotationrate to 7200 (or to a slightly greater = value) as well for iSCSI drives ? >> I looked through ctl.conf(5) and iscsi.conf(5) but did not found = anything related. >>=20 >> Many thanks ! >=20 > Use the "option" setting in ctl.conf to change the rpm value = (documented > in the OPTIONS section of ctladm(8)). Thank you also Gary, and sorry as your mail went to spam :/ Ben From owner-freebsd-fs@freebsd.org Tue Oct 3 22:30:17 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9F5EE26D3B for ; Tue, 3 Oct 2017 22:30:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C892066F9E for ; Tue, 3 Oct 2017 22:30:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v93MUH9M002417 for ; Tue, 3 Oct 2017 22:30:17 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 222734] 11.1-RELEASE kernel panics while importing ZFS pool Date: Tue, 03 Oct 2017 22:30:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 22:30:18 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222734 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Oct 3 22:47:44 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D94CE275A9 for ; Tue, 3 Oct 2017 22:47:44 +0000 (UTC) (envelope-from daveb@spectralogic.com) Received: from mail1.bemta8.messagelabs.com (mail1.bemta8.messagelabs.com [216.82.243.203]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail1.bemta8.messagelabs.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3E6CB67A0A for ; Tue, 3 Oct 2017 22:47:43 +0000 (UTC) (envelope-from daveb@spectralogic.com) Received: from [216.82.242.179] by server-11.bemta-8.messagelabs.com id 6A/EE-06254-CF114D95; Tue, 03 Oct 2017 22:41:00 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrAIsWRWlGSWpSXmKPExsVyQG6fiO5PwSu RBue261sce/yTzWLLnjtsDkweMz7NZ/FYtvIqcwBTFGtmXlJ+RQJrxvN9V1gKzktW/Ls8i6WB cYlkFyMnB5uAlkTPksMsXYwcHCIC6RJ3J3qDhIUFrCV+Na5gBrFFBBwkFn98wAhRoiexuSsAJ MwioCIxY9oBdhCbV8BZom3OPjCbUUBM4vupNUwgNrOAuMStJ/PBbAkBAYkle84zQ9iiEi8f/2 OFsHUkzl5/wghhG0hsXboP7BpmAU2J9bv0IcbYSzS8+cECYStKTOl+CLVWUOLkzCcsExgFZyH ZNguhexaS7llIumch6V7AyLqKUb04tagstUjXTC+pKDM9oyQ3MTNH19DAQi83tbg4MT01JzGp WC85P3cTIzDcGYBgB+OnfudDjJIcTEqivLc4r0QK8SXlp1RmJBZnxBeV5qQWH2KU4eBQkuA1A saPkGBRanpqRVpmDjDyYNISHDxKIrwcAkBp3uKCxNzizHSI1ClGY44ZN+/+YeJ4cm3eXyYhlr z8vFQpcd55IKUCIKUZpXlwg2AJ4RKjrJQwLyPQaUI8BalFuZklqPKvGMU5GJWEedNBpvBk5pX A7XsFdAoT0Clzui6AnFKSiJCSamDsY3J25YrZ3HVsZuWEKylfaz/P+byij6uZPTG8iLUrhs97 uQ7793ezOre7HarYLBYafVDgyaIJbw7NL+hLjDvGltJyVitRcTn7pK23eva++Xw6baqDihrj/ jMrWtijvRl6Hp9kU9py8blYz+tzczilvKPOFFV1zWGRs9RJLJjB/78iT2vikf1KLMUZiYZazE XFiQAO1K4VAwMAAA== X-Env-Sender: daveb@spectralogic.com X-Msg-Ref: server-8.tower-86.messagelabs.com!1507070457!154286217!1 X-Originating-IP: [192.30.190.20] X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 1891 invoked from network); 3 Oct 2017 22:40:57 -0000 Received: from outmx2.spectralogic.com (HELO mail.spectralogic.com) (192.30.190.20) by server-8.tower-86.messagelabs.com with AES256-SHA encrypted SMTP; 3 Oct 2017 22:40:57 -0000 From: Dave Baukus To: "freebsd-fs@freebsd.org" , "zfs@lists.illumos.org" Subject: Ephemeral fguid crash in zfs_log_create() question Thread-Topic: Ephemeral fguid crash in zfs_log_create() question Thread-Index: AQHTPJiioyvQmaEWOEigtWGNKbpGzg== Date: Tue, 3 Oct 2017 22:40:38 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted Content-Type: text/plain; charset="utf-8" Content-ID: Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 22:47:44 -0000 SSBoYXZlIGEgRnJlZUJTRCAoc3RhYmxlIDExKSBaRlMgc3lzdGVtIGNyYXNoaW5nIGluIHpmc19s b2dfY3JlYXRlKCkgYmVjYXVzZQ0KdGhlIHpmc19mdWlkX2luZm9fdCAqZnVpZHAgcGFzc2VkIGlu IGZyb206DQoNCnpmc19sb2dfY3JlYXRlKHppbG9nLCB0eCwgdHh0eXBlLCBkenAsIHpwLCBuYW1l LA0KICAgICAgICAgdnNlY3AsIGFjbF9pZHMuel9mdWlkcCwgdmFwKTt6ZnNfY3JlYXRlKCkNCmlz IE5VTEwuDQoNClRoZSB6ZnNfYWNsX2lkc190IGJ1aWx0IHZpYSB6ZnNfYWNsX2lkc19jcmVhdGUo KSBmb3IgemZzX2NyZWF0ZSgpIGlzDQphcyBmb2xsb3dzOg0KDQpwL3ggKiRhY2xfaWRzDQokNzQg PSB7DQogICB6X2Z1aWQgPSAweDIxMjZkMSwNCiAgIHpfZmdpZCA9IDB4MzAwMDAwMjAxLA0KICAg el9tb2RlID0gMHg4MWI0LA0KICAgel9hY2xwID0gMHhmZmZmZjgwODg2OTAxYjAwLA0KICAgel9m dWlkcCA9IDB4MA0KfQ0KDQpUaGUgaXNzdWUsIGFzIEkndmUgYmVlbiBhYmxlIHRvIHBpZWNlIHRv Z2V0aGVyLCBjb3VsZCBiZSB0aGlzIHNuaXBwZXQgb2YNCmNvZGUgaW4gemZzX2FjbF9pZHNfY3Jl YXRlKCk6DQoNCiAgICAgfSBlbHNlIHsNCiAgICAgICAgICBhY2xfaWRzLT56X2ZnaWQgPSB6ZnNf ZnVpZF9jcmVhdGVfY3JlZCh6ZnN2ZnMsDQogICAgICAgICAgICAgIFpGU19HUk9VUCwgY3IsICZh Y2xfaWRzLT56X2Z1aWRwKTsNCiNpZmRlZiBfX0ZyZWVCU0Rfa2VybmVsX18NCiAgICAgICAgICBn aWQgPSBhY2xfaWRzLT56X2ZnaWQgPSBkenAtPnpfZ2lkOw0KI2Vsc2UNCiAgICAgICAgICBnaWQg PSBjcmdldGdpZChjcik7DQojZW5kaWYNCiAgICAgfQ0KDQp6ZnNfZnVpZF9jcmVhdGVfY3JlZCgp IHdvdWxkIGhhdmUgcmV0dXJuZWQgYSBub24tRVBIRU1FUkFMIHpfZmdpZCBmcm9tIHRoZSBjcmVk Og0KICBwL3ggJGNyZWQtPmNyX2dyb3Vwc1swXQ0KJDcwID0gMHgxZTg2ODENCg0KQnV0IHRoZW4g dGhlIEZyZWVCU0Rfa2VybmVsIGNvZGUgc2V0IGl0IHRvIGFuIEVQSEVNRVJBTCB6X2dpZCBmcm9t IHRoZSBwYXJlbnQgem5vZGU6DQpwL3ggJGR6cC0+el9naWQNCiQ3MyA9IDB4MzAwMDAwMjAxDQoN Ck5vdyB0aGUgcHJvYmxlbSBmb3IgemZzX2xvZ19jcmVhdGUoKSBpcyB0aGF0IHdlIGhhdmUgYW4g RVBIRU1FUkFMIHpfZ2lkIGJ1dCB3ZSBkbyBub3QgaGF2ZQ0KYSBmdWlkcCBhbmQgd2UgY3Jhc2gg aGVyZToNCiAgICAgICAgIGlmICghSVNfRVBIRU1FUkFMKHpwLT56X2dpZCkpIHsNCiAgICAgICAg ICAgICAgICAgbHItPmxyX2dpZCA9ICh1aW50NjRfdCl6cC0+el9naWQ7DQogICAgICAgICB9IGVs c2Ugew0KICAgICAgICAgICAgICAgICBsci0+bHJfZ2lkID0gZnVpZHAtPnpfZnVpZF9ncm91cDsN CiAgICAgICAgIH0NCg0KDQpGaW5hbGx5IHRvIGEgcXVlc3Rpb246DQpXaHkgZG9lc24ndCB0aGUg c25pcHBldCBvZiBjb2RlIChhYm92ZSkgZnJvbSB6ZnNfYWNsX2lkc19jcmVhdGUoKSwgYWxzbyBp bmNsdWRlDQp0aGUgZnVuY3Rpb25hbGl0eSB0byBhZGQgYSBmdWlkICBub2RlIGZvciBlcGhlbWVy YWwgR0lEcyAoaS5lLiB0aGUgc2FtZSBjb2RlIHRoYXQNCmlzIGluIHRoZSBpZiAoZHpwLT56X21v ZGUgJiBTX0lTR0lEKSBibG9jaykgPw0KDQpUaGF0IGlzIHdoeSBub3Qgc29tZXRoaW5nIGxpa2U6 DQoNCiAgICAgaWYgKGR6cC0+el9tb2RlICYgU19JU0dJRCkgew0KICAgICAgICAgLi4uLg0KICAg ICB9IGVsc2Ugew0KICAgICAgICAgYWNsX2lkcy0+el9mZ2lkID0gemZzX2Z1aWRfY3JlYXRlX2Ny ZWQoemZzdmZzLA0KICAgICAgICAgICAgIFpGU19HUk9VUCwgY3IsICZhY2xfaWRzLT56X2Z1aWRw KTsNCg0KI2lmZGVmIF9fRnJlZUJTRF9rZXJuZWxfXw0KICAgICAgICAgZ2lkID0gYWNsX2lkcy0+ el9mZ2lkID0gZHpwLT56X2dpZDsNCg0KICAgICAgICAgaWYgKHpmc3Zmcy0+el91c2VfZnVpZHMg JiYNCiAgICAgICAgICAgICAgSVNfRVBIRU1FUkFMKGFjbF9pZHMtPnpfZmdpZCkpIHsNCg0KICAg ICAgICAgICAgICBkb21haW4gPSB6ZnNfZnVpZF9pZHhfZG9tYWluKA0KICAgICAgICAgICAgICAg ICAgJnpmc3Zmcy0+el9mdWlkX2lkeCwNCiAgICAgICAgICAgICAgICAgIEZVSURfSU5ERVgoYWNs X2lkcy0+el9mZ2lkKSk7DQoNCiAgICAgICAgICAgICAgcmlkID0gRlVJRF9SSUQoYWNsX2lkcy0+ el9mZ2lkKTsNCiAgICAgICAgICAgICAgemZzX2Z1aWRfbm9kZV9hZGQoJmFjbF9pZHMtPnpfZnVp ZHAsDQogICAgICAgICAgICAgICAgICBkb21haW4sIHJpZCwNCiAgICAgICAgICAgICAgICAgIEZV SURfSU5ERVgoYWNsX2lkcy0+el9mZ2lkKSwNCiAgICAgICAgICAgICAgICAgIGFjbF9pZHMtPnpf ZmdpZCwgWkZTX0dST1VQKTsNCiAgICAgICAgIH0NCiNlbmRpZg0KDQoNClRoYW5rcyBmb3IgYW55 IGluc2lnaHRzDQoNCi0tIA0KRGF2ZSBCYXVrdXMNCg== From owner-freebsd-fs@freebsd.org Tue Oct 3 23:03:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 00C9CE27BFF for ; Tue, 3 Oct 2017 23:03:04 +0000 (UTC) (envelope-from daveb@spectralogic.com) Received: from mail1.bemta8.messagelabs.com (mail1.bemta8.messagelabs.com [216.82.243.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail1.bemta8.messagelabs.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B43B468708 for ; Tue, 3 Oct 2017 23:03:03 +0000 (UTC) (envelope-from daveb@spectralogic.com) Received: from [216.82.242.33] by server-14.bemta-8.messagelabs.com id B0/66-01779-02714D95; Tue, 03 Oct 2017 23:02:56 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupkleJIrShJLcpLzFFi42I5ILdPRFde/Eq kQf98Y4tjj3+yWWzZc4fNgcljxqf5LB7LVl5lDmCKYs3MS8qvSGDN+LZxB1vBYpmKS7deMzUw fpHuYuTkYBPQkuhZcpili5GDQ0QgXeLuRG+QsLCAvcSkD2uYQWwRAQeJxR8fMELYRhK9DU+YQ GwWARWJ79O/soHYvALOEveOLGQFsYWA7P1/doDFOQVcJN7evAVWzyggJvH91Bowm1lAXOLWk/ lgtoSAgMSSPeeZIWxRiZeP/7FC2DoSZ68/YYSwDSS2Lt0HdiazgKbE+l36EGPsJV4/PcwIYSt KTOl+yA5xjqDEyZlPWCYwCs9Csm0WQvcsJN2zkHTPQtK9gJF1FaN6cWpRWWqRrqVeUlFmekZJ bmJmjq6hgYVebmpxcWJ6ak5iUrFecn7uJkZghDAAwQ7GdVOdDzFKcjApifK6CF+JFOJLyk+pz EgszogvKs1JLT7EKMPBoSTBu0AUKCdYlJqeWpGWmQOMVZi0BAePkgjvIRGgNG9xQWJucWY6RO oUozHHjJt3/zBxPLk27y+TEEtefl6qlDhvIcgkAZDSjNI8uEGwFHKJUVZKmJcR6DQhnoLUotz MElT5V4ziHIxKwrxTQKbwZOaVwO17BXQKE9Apc7ougJxSkoiQkmpg3Nns6aKpF5/xYkE6493q 57o2Ql1XnSQDhVecMJ/coqz257nQEoWXNQoHPpxYNaW39NuvX/v/ZPYtXTl9Zvksn9rVW7R8t RI6lxvd/TdXMG/hw6o0rcocHuEdJ7lyg9XNV2vfflZgcb2MOSt8m8ZzhZOhP/8xR3YvzNbnXa fVoDjDbnKW2oIIJZbijERDLeai4kQAKnUGzBwDAAA= X-Env-Sender: daveb@spectralogic.com X-Msg-Ref: server-7.tower-55.messagelabs.com!1507071775!135053803!1 X-Originating-IP: [192.30.190.20] X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 46903 invoked from network); 3 Oct 2017 23:02:55 -0000 Received: from outmx2.spectralogic.com (HELO mail.spectralogic.com) (192.30.190.20) by server-7.tower-55.messagelabs.com with AES256-SHA encrypted SMTP; 3 Oct 2017 23:02:55 -0000 From: Dave Baukus To: "freebsd-fs@freebsd.org" , "zfs@lists.illumos.org" Subject: Re: Ephemeral fguid crash in zfs_log_create() question Thread-Topic: Ephemeral fguid crash in zfs_log_create() question Thread-Index: AQHTPJiioyvQmaEWOEigtWGNKbpGzqLTItSA Date: Tue, 3 Oct 2017 23:02:42 +0000 Message-ID: <0acb0d0b-bcf7-cbe3-dfd8-1f7c7ab7cee9@spectralogic.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted Content-Type: text/plain; charset="utf-8" Content-ID: Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 23:03:04 -0000 U21hbGwgZm9ybWF0dGluZyBjb3JyZWN0aW9uIGluY2x1ZGVkIGJlbG93Og0KDQpPbiAxMC8wMy8y MDE3IDA0OjQwIFBNLCBEYXZlIEJhdWt1cyB3cm90ZToNCj4gSSBoYXZlIGEgRnJlZUJTRCAoc3Rh YmxlIDExKSBaRlMgc3lzdGVtIGNyYXNoaW5nIGluIHpmc19sb2dfY3JlYXRlKCkgYmVjYXVzZQ0K PiB0aGUgemZzX2Z1aWRfaW5mb190ICpmdWlkcCBwYXNzZWQgaW4gZnJvbSB6ZnNfY3JlYXRlKCk6 DQo+DQo+IHpmc19sb2dfY3JlYXRlKHppbG9nLCB0eCwgdHh0eXBlLCBkenAsIHpwLCBuYW1lLA0K PiAgICAgICAgICAgdnNlY3AsIGFjbF9pZHMuel9mdWlkcCwgdmFwKTsNCj4gaXMgTlVMTC4NCj4N Cj4gVGhlIHpmc19hY2xfaWRzX3QgYnVpbHQgdmlhIHpmc19hY2xfaWRzX2NyZWF0ZSgpIGZvciB6 ZnNfY3JlYXRlKCkgaXMNCj4gYXMgZm9sbG93czoNCj4NCj4gcC94ICokYWNsX2lkcw0KPiAkNzQg PSB7DQo+ICAgICB6X2Z1aWQgPSAweDIxMjZkMSwNCj4gICAgIHpfZmdpZCA9IDB4MzAwMDAwMjAx LA0KPiAgICAgel9tb2RlID0gMHg4MWI0LA0KPiAgICAgel9hY2xwID0gMHhmZmZmZjgwODg2OTAx YjAwLA0KPiAgICAgel9mdWlkcCA9IDB4MA0KPiB9DQo+DQo+IFRoZSBpc3N1ZSwgYXMgSSd2ZSBi ZWVuIGFibGUgdG8gcGllY2UgdG9nZXRoZXIsIGNvdWxkIGJlIHRoaXMgc25pcHBldCBvZg0KPiBj b2RlIGluIHpmc19hY2xfaWRzX2NyZWF0ZSgpOg0KPg0KPiAgICAgICB9IGVsc2Ugew0KPiAgICAg ICAgICAgIGFjbF9pZHMtPnpfZmdpZCA9IHpmc19mdWlkX2NyZWF0ZV9jcmVkKHpmc3ZmcywNCj4g ICAgICAgICAgICAgICAgWkZTX0dST1VQLCBjciwgJmFjbF9pZHMtPnpfZnVpZHApOw0KPiAjaWZk ZWYgX19GcmVlQlNEX2tlcm5lbF9fDQo+ICAgICAgICAgICAgZ2lkID0gYWNsX2lkcy0+el9mZ2lk ID0gZHpwLT56X2dpZDsNCj4gI2Vsc2UNCj4gICAgICAgICAgICBnaWQgPSBjcmdldGdpZChjcik7 DQo+ICNlbmRpZg0KPiAgICAgICB9DQo+DQo+IHpmc19mdWlkX2NyZWF0ZV9jcmVkKCkgd291bGQg aGF2ZSByZXR1cm5lZCBhIG5vbi1FUEhFTUVSQUwgel9mZ2lkIGZyb20gdGhlIGNyZWQ6DQo+IHAv eCAkY3JlZC0+Y3JfZ3JvdXBzWzBdDQo+ICQ3MCA9IDB4MWU4NjgxDQo+DQo+IEJ1dCB0aGVuIHRo ZSBGcmVlQlNEX2tlcm5lbCBjb2RlIHNldCBpdCB0byBhbiBFUEhFTUVSQUwgel9naWQgZnJvbSB0 aGUgcGFyZW50IHpub2RlOg0KPiBwL3ggJGR6cC0+el9naWQNCj4gJDczID0gMHgzMDAwMDAyMDEN Cj4NCj4gTm93IHRoZSBwcm9ibGVtIGZvciB6ZnNfbG9nX2NyZWF0ZSgpIGlzIHRoYXQgd2UgaGF2 ZSBhbiBFUEhFTUVSQUwgel9naWQgYnV0IHdlIGRvIG5vdCBoYXZlDQo+IGEgZnVpZHAgYW5kIHdl IGNyYXNoIGhlcmU6DQo+ICAgICAgICAgICBpZiAoIUlTX0VQSEVNRVJBTCh6cC0+el9naWQpKSB7 DQo+ICAgICAgICAgICAgICAgICAgIGxyLT5scl9naWQgPSAodWludDY0X3QpenAtPnpfZ2lkOw0K PiAgICAgICAgICAgfSBlbHNlIHsNCj4gICAgICAgICAgICAgICAgICAgbHItPmxyX2dpZCA9IGZ1 aWRwLT56X2Z1aWRfZ3JvdXA7DQo+ICAgICAgICAgICB9DQo+DQo+DQo+IEZpbmFsbHkgdG8gYSBx dWVzdGlvbjoNCj4gV2h5IGRvZXNuJ3QgdGhlIHNuaXBwZXQgb2YgY29kZSAoYWJvdmUpIGZyb20g emZzX2FjbF9pZHNfY3JlYXRlKCksIGFsc28gaW5jbHVkZQ0KPiB0aGUgZnVuY3Rpb25hbGl0eSB0 byBhZGQgYSBmdWlkICBub2RlIGZvciBlcGhlbWVyYWwgR0lEcyAoaS5lLiB0aGUgc2FtZSBjb2Rl IHRoYXQNCj4gaXMgaW4gdGhlIGlmIChkenAtPnpfbW9kZSAmIFNfSVNHSUQpIGJsb2NrKSA/DQo+ DQo+IFRoYXQgaXMgd2h5IG5vdCBzb21ldGhpbmcgbGlrZToNCj4NCj4gICAgICAgaWYgKGR6cC0+ el9tb2RlICYgU19JU0dJRCkgew0KPiAgICAgICAgICAgLi4uLg0KPiAgICAgICB9IGVsc2Ugew0K PiAgICAgICAgICAgYWNsX2lkcy0+el9mZ2lkID0gemZzX2Z1aWRfY3JlYXRlX2NyZWQoemZzdmZz LA0KPiAgICAgICAgICAgICAgIFpGU19HUk9VUCwgY3IsICZhY2xfaWRzLT56X2Z1aWRwKTsNCj4N Cj4gI2lmZGVmIF9fRnJlZUJTRF9rZXJuZWxfXw0KPiAgICAgICAgICAgZ2lkID0gYWNsX2lkcy0+ el9mZ2lkID0gZHpwLT56X2dpZDsNCj4NCj4gICAgICAgICAgIGlmICh6ZnN2ZnMtPnpfdXNlX2Z1 aWRzICYmDQo+ICAgICAgICAgICAgICAgIElTX0VQSEVNRVJBTChhY2xfaWRzLT56X2ZnaWQpKSB7 DQo+DQo+ICAgICAgICAgICAgICAgIGRvbWFpbiA9IHpmc19mdWlkX2lkeF9kb21haW4oDQo+ICAg ICAgICAgICAgICAgICAgICAmemZzdmZzLT56X2Z1aWRfaWR4LA0KPiAgICAgICAgICAgICAgICAg ICAgRlVJRF9JTkRFWChhY2xfaWRzLT56X2ZnaWQpKTsNCj4NCj4gICAgICAgICAgICAgICAgcmlk ID0gRlVJRF9SSUQoYWNsX2lkcy0+el9mZ2lkKTsNCj4gICAgICAgICAgICAgICAgemZzX2Z1aWRf bm9kZV9hZGQoJmFjbF9pZHMtPnpfZnVpZHAsDQo+ICAgICAgICAgICAgICAgICAgICBkb21haW4s IHJpZCwNCj4gICAgICAgICAgICAgICAgICAgIEZVSURfSU5ERVgoYWNsX2lkcy0+el9mZ2lkKSwN Cj4gICAgICAgICAgICAgICAgICAgIGFjbF9pZHMtPnpfZmdpZCwgWkZTX0dST1VQKTsNCj4gICAg ICAgICAgIH0NCj4gI2VuZGlmDQo+DQo+DQo+IFRoYW5rcyBmb3IgYW55IGluc2lnaHRzDQo+DQo= From owner-freebsd-fs@freebsd.org Wed Oct 4 16:15:30 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 112A2E3CF76 for ; Wed, 4 Oct 2017 16:15:30 +0000 (UTC) (envelope-from javocado@gmail.com) Received: from mail-ua0-x236.google.com (mail-ua0-x236.google.com [IPv6:2607:f8b0:400c:c08::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BE29A68156 for ; Wed, 4 Oct 2017 16:15:29 +0000 (UTC) (envelope-from javocado@gmail.com) Received: by mail-ua0-x236.google.com with SMTP id 47so7169049uas.8 for ; Wed, 04 Oct 2017 09:15:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=VoR66OkqVQzR1dkDl5fIUJwKp7m+sxA4QL4wSYgBvM0=; b=dObOOk5fD14ov7FG8zlvOzsBxBkX2kwDxlnCwSTuNnuz3U1bkd3fj+LsEvaVaWU7i7 wzM2nUFiJLtCQlwqbCeUMtq3SNh2c68eHljzEYLG9fpavKpeZtFtxKN9caVSR66F5CuP AmswVFJtu/PoQq+R6mtk9JIRIqYRilWSvOwElfjnIt2eVZU/4AsCVmrGvRPcIVUj7hJ+ BA2sbUd79dfasw2uUszD00JFAhHkhXasu0iOLE2Yb4kkekVwITbluPCk4s9MePCngO80 5DtkcP7bINSaCA46+fa0Zsf0YI3UmOPDcWnW0icONUUNmd9fyjfXlMi2Ao4ejFqhZ3pY EZjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=VoR66OkqVQzR1dkDl5fIUJwKp7m+sxA4QL4wSYgBvM0=; b=E8rxNMhHWBkyLi2/HEZ1s4RvM1pcdHsOUHLADx9ki3z/wdTKkTBTAVqYTHvcCmeAm1 jp0Q9+mh2YmUTo9eOzFf7lUwxL66pdzd/neaDjyTnnu+x2Ec+rq46VHfgCEOeZX3lwm4 zAJLKHl+zL8XtgmltpQxtXaUnf6MpXjNPH22HtQhLyynSRLLthC68YDzW/SaTpbMBkIk 5WrPFl+3fSVeqfotwT8dWU8fH4bOwiMuTVRd4jBLExPskTqy/5DT939exCmVvX5oOuD9 fiAnnDaWIL0UOBgKWrOz1kaqUaIq5/JLqCpQHwLt3KkNtDC0GB/6a4u8L6oZk9neftPW 1OnA== X-Gm-Message-State: AHPjjUi04sxpG2un+QTxRgETWqkMTGg+vlWs9z8ZRWjlR8peA/Wjb39t 031zKGufP7mMNRMZlL7YlvdwzXcqHCeEKuZfgYC7mA== X-Google-Smtp-Source: AOwi7QBkhSedf6YzySREy/pdCERdPYWe0nCWa4C92eaX/qollyGz6dKqa0v1TedoEmvvwrRQuWK4epNlCL7AD6eljgM= X-Received: by 10.159.36.74 with SMTP id 68mr10888560uaq.67.1507133728327; Wed, 04 Oct 2017 09:15:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.159.51.90 with HTTP; Wed, 4 Oct 2017 09:15:27 -0700 (PDT) From: javocado Date: Wed, 4 Oct 2017 09:15:27 -0700 Message-ID: Subject: lockup during zfs destroy To: FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Oct 2017 16:15:30 -0000 I am trying to destroy a dense, large filesystem and it's not going well. Details: - zpool is a raidz3 with 3 x 12 drive vdevs. - target filesystem to be destroyed is ~2T with ~63M inodes. - OS: FreeBSD 10.3amd with 192 GB of RAM. - 120 GB of swap (90GB recently added as swap-on-disk) What happened initially is the system locked after a few hours up and I had to reboot. Upon rebooting and starting zfs, I see sustained disk activity in gstat *and* that the sustained activity is usually just 6 disks reading. Two raidz3 vdevs are involved in this filesystem I am deleting so there are 6 parity disks ... not sure if that is correlated or not. At about the 1h40m mark of uptime I see things start to happen in top: a sudden spike in load, and drop in the amount of "Free" memory as reported in top: ([CODE]Mem: 23M Active, 32M Inact, 28G Wired, 24M Buf, 159G Free[/CODE]) It drops down under a GB and then fluctuates up and down till eventually it reaches some small amount (41 MB). As this drop starts, I see gstat activity on zpool drives cease, and there's some light activity on the swap devices, but not much. Also, the amount of swap used is reported as very little, maybe less than a MB to 24 MB. swapinfo shows nothing used. After the memory usage settles the system eventually ends up in a locked state where: - nothing is going on in gstat; the only non-zero number is the queue length for the swap device which is stuck at 4 - load drops to nothing, and occasionally I see the zfskern and zpool procs stuck in vmwait state*. - shell is unresponsive, but carriage returns register - there are NO kernel/messages of any kind on console indicating a problem or resource exhaustion Finally, I cannot do this: # zdb -dddd pool/filesystem | grep DELETE_QUEUE zdb: can't open 'pool/filesystem': Device busy (presumably because it is pending destroy ...) I had set: vm.kmem_size="384G" (and nothing else in loader) but even removing that and setting more realistic figures like: vm.kmem_size=200862670848 vm.kmem_size_max=200862670848 vfs.zfs.arc_max=187904819200 have not resulted in a different outcome, *though I don't see the processes in vmwait any longer, the state is just "-" I've just lowered these to: vm.kmem_size=198642237440 vm.kmem_size_max=198642237440 vfs.zfs.arc_max=190052302848 to see if that will make a difference. No matter how many times I reboot, so far about 6, I never make it past the 1h40m mark and this memory dip. I don't know if I'm making any progress or just running into the same wall. My questions: - is this what it appears to be, a memory exhaustion? - if so, why isn't swap utilized? - how would I configure my way past this hurdle? - a filesystem has a DELETE_QUEUE ... does the zpool itself have a destroy queue of some kind? I am trying to see if I can see the zpool working and how far along it is, but I do not know what to query with zdb Thanks! From owner-freebsd-fs@freebsd.org Wed Oct 4 16:27:41 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C23C4E3D2BA for ; Wed, 4 Oct 2017 16:27:41 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-oi0-x234.google.com (mail-oi0-x234.google.com [IPv6:2607:f8b0:4003:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8539C686C9 for ; Wed, 4 Oct 2017 16:27:41 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-oi0-x234.google.com with SMTP id n82so13182668oig.3 for ; Wed, 04 Oct 2017 09:27:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=zLkQ/p0/ux8AEvfG6ARTblnUBo0za0iPtwxN2dDRaTc=; b=tJ4k4VEiuFt/ovN33qYnY3Ml54gaTjEzFaZEru6zqPd8/T/cVcIw0rqkSEdYSGXKtQ /lc7hvYmiRThLGd206M+t84xWjnZEkW7gPh89Aml6kXjGFSNREIA3g0W1s/RLA3+2M62 gVgJC9LbWPeAGO8l//zC2kDzgZqskaz9g0Sn6RCAgMgZigA+HcJGnNxsgRfPEKR5iqDP fLXGyw/vmulVFBKqDZOWL7eNDkxi6bAB2hTaTX9roGe4W2C95aEDLoIjhHTSJeq4c59e JWPzNdkUQ6x1sjnci+u65GlQPfnbit4N/VdtpFSwFq+xM0MA5XLiL+kDDtehMM0+HyVt NEQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=zLkQ/p0/ux8AEvfG6ARTblnUBo0za0iPtwxN2dDRaTc=; b=jEMycCvLwMPLT/aHsSrXi96LINRMWJu9ReJuBeVqECMYl9ueIKEU6aiMPa22cGnKj2 gM3WyYqUhvEO38i6gXGzFTPtWpan+0xL4dqxFnRk+bhqSF3oyFITHnmo3D8w2BghP/RE M5HflDyM5P7X1DMvM2+XSW5texrXL00tLhiX5vEZ0soQtiC7T2M0qad1+E+teXlbeP4S tBYOq06BDny4hSSJC3aX8aVnDNPacNmw6q7QCli5lYNyHt1t5gZAmIbXx4S22bimLBWC +0joR34iLNUBP2Nj8gd5+y0T5a3Hvx5pvI8TV9yNZGRXaKYqkhT0UGPrAMWOiDZLnv/U 17yw== X-Gm-Message-State: AMCzsaVdJkiAFV4H8zYwudchdt0+G0bG05t76YkobJKWhYhYOHHhGgO7 BvmxNwfFmHYsVMBiihyoY83x36f6CRnuLCYAUlg= X-Google-Smtp-Source: AOwi7QBYhS4gJ4BSYG9+BUygceZzS9tn12NCTl90vK8zR75fZP+faMdwS3BgX70yBTB9JfBSnRV6rug9H2EMpfFzYSE= X-Received: by 10.157.85.80 with SMTP id h16mr3584962oti.12.1507134460700; Wed, 04 Oct 2017 09:27:40 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.62.245 with HTTP; Wed, 4 Oct 2017 09:27:40 -0700 (PDT) In-Reply-To: References: From: Freddie Cash Date: Wed, 4 Oct 2017 09:27:40 -0700 Message-ID: Subject: Re: lockup during zfs destroy To: javocado Cc: FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Oct 2017 16:27:41 -0000 On Wed, Oct 4, 2017 at 9:15 AM, javocado wrote: > I am trying to destroy a dense, large filesystem and it's not going well. > > Details: > - zpool is a raidz3 with 3 x 12 drive vdevs. > - target filesystem to be destroyed is ~2T with ~63M inodes. > - OS: FreeBSD 10.3amd with 192 GB of RAM. > - 120 GB of swap (90GB recently added as swap-on-disk) > =E2=80=8BDo you have dedupe enabled on any filesystems in the pool? Or was= it enabled at any point in the past? This is a common occurrence when destroying large filesystems or lots of filesystems/snapshots on pools that have/had dedupe enabled and there's not enough RAM/L2ARC to contain the DDT. The system runs out of usable wired memory=E2=80=8B and locks up. Adding more RAM and/or being patient with th= e boot-wait-lockup-repeat cycle will (usually) eventually allow it to finish the destroy. There was a loader.conf tunable (or sysctl) added in the 10.x series that mitigates this by limiting the number of delete operations that occur in a transaction group, but I forget the details on it. Not sure if this affects pools that never had dedupe enabled or not. (We used to suffer through this at least once a year until we enabled a delete-oldest-snapshot-before-running-backups process to limit the number of snapshots.)=E2=80=8B --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@freebsd.org Wed Oct 4 16:43:40 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04AE2E3D940 for ; Wed, 4 Oct 2017 16:43:40 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BD2466965B for ; Wed, 4 Oct 2017 16:43:39 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.89 (FreeBSD)) (envelope-from ) id 1dzmlm-000NhJ-5P; Wed, 04 Oct 2017 17:43:38 +0100 Date: Wed, 4 Oct 2017 17:43:37 +0100 From: Gary Palmer To: javocado Cc: FreeBSD Filesystems Subject: Re: lockup during zfs destroy Message-ID: <20171004164337.GB65538@in-addr.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Oct 2017 16:43:40 -0000 On Wed, Oct 04, 2017 at 09:15:27AM -0700, javocado wrote: > My questions: > > - is this what it appears to be, a memory exhaustion? > - if so, why isn't swap utilized? Kernel memory generally isn't pushed to swap as it could lead to deadlock situations way too easily. > - how would I configure my way past this hurdle? > - a filesystem has a DELETE_QUEUE ... does the zpool itself have a destroy > queue of some kind? I am trying to see if I can see the zpool working > and how far along it is, but I do not know what to query with zdb Yes, it does, I believe behind the feature@async_destroy flag on the pool. "zpool get feature@async_destroy" to see the enabled status. Not sure if you can query the queue to see how it is progressing. I haven't destroyed any pools, but with snapshots you can check the free space on the pool using "zpool list" and it gradully increases in the background. Regards, Gary From owner-freebsd-fs@freebsd.org Wed Oct 4 17:58:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B53ECE3EFE9 for ; Wed, 4 Oct 2017 17:58:03 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-oi0-x233.google.com (mail-oi0-x233.google.com [IPv6:2607:f8b0:4003:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 781BA6BA7D for ; Wed, 4 Oct 2017 17:58:03 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-oi0-x233.google.com with SMTP id j126so20845131oia.10 for ; Wed, 04 Oct 2017 10:58:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=JpW0FGvBGdQZvJJcwxcl2xOPgWkWTP1q2CObIFeAJiQ=; b=DzrSHk7bcVYdf0y0WXPThCdEA+BiVQG94AqO5h2CL0KjV9MAr/nnm8JBFYoypxV9Ip BK9nZ0j1gf0nqVMMlFgkq2rXxCt8xWxmviDmAModo/RXd37jYTvyKJ1ZOLQkTFq2fQWm HwS9Cek/mYC5SpVdDlGBqzmM00QuahYzCJLkatagBMLKlNlApVeR9o7dDgmKBYRxKzwa cGKFCSOi6t6aj0Zgi5EdqaZgkkAqpPkZtpY1hJNJqM31NZH5eKEinamehWAbdeBgYpA/ mLagTHPQ3++Kzu9lW+rKNveDYetyXif7wmbOq2eRudR+as6cx3T6esnXarq1Mh1i0gfA 4AyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=JpW0FGvBGdQZvJJcwxcl2xOPgWkWTP1q2CObIFeAJiQ=; b=uYKELqXjuXNxkXe6Tlz6YDinmwmepPUSLjCvebrtPb2UK0QzTw4JuQoT17TrwBH1pK Zp2OpGe2KQHawd6XNqDlVeBoDs8RXv9TU21UvCNzbZoz8U+OIr5TYoxONR38MtKc1vzD BAONmTElR7n6S3Eo2PjQSDg982bDsWiSPJc6lbJrOmMU/rXuoQDy7VDBn9K+bA49dguD lUN/dKmbeMqGqxJiOA0C2vpLSjoRYsXuI32EwdWVmGKH5YKm8lIWRSDzb+ZZOtOFoEmC rpsMJ6anmG9nmstM8D/SlU60IwD3yFHHmUKaHfTckJJ9IM2artitWYXvo6OhBflXWgfo /9yA== X-Gm-Message-State: AMCzsaUnZtXU36dGd68U6l/ps3dBzganBKi1xN3HgYHB/WucwsMhnRMW F39JTA/03HY3DPppgzznbkEWjKk/yFxB+Bjjo/g= X-Google-Smtp-Source: AOwi7QD0W/VcWYWg6ScMHBkXN5JumtjTtMM5VYSDJ+PjTlQ+LSH5mXI3mO548q41KMiwyTBz/yUkjRpVsgPMKLdEV0o= X-Received: by 10.157.9.195 with SMTP id 3mr12779378otz.431.1507139882693; Wed, 04 Oct 2017 10:58:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.62.245 with HTTP; Wed, 4 Oct 2017 10:58:01 -0700 (PDT) In-Reply-To: References: From: Freddie Cash Date: Wed, 4 Oct 2017 10:58:01 -0700 Message-ID: Subject: Re: lockup during zfs destroy To: javocado Cc: FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Oct 2017 17:58:03 -0000 On Wed, Oct 4, 2017 at 9:27 AM, Freddie Cash wrote: > On Wed, Oct 4, 2017 at 9:15 AM, javocado wrote: > >> I am trying to destroy a dense, large filesystem and it's not going well= . >> >> Details: >> - zpool is a raidz3 with 3 x 12 drive vdevs. >> - target filesystem to be destroyed is ~2T with ~63M inodes. >> - OS: FreeBSD 10.3amd with 192 GB of RAM. >> - 120 GB of swap (90GB recently added as swap-on-disk) >> > > =E2=80=8BDo you have dedupe enabled on any filesystems in the pool? Or w= as it > enabled at any point in the past? > > This is a common occurrence when destroying large filesystems or lots of > filesystems/snapshots on pools that have/had dedupe enabled and there's n= ot > enough RAM/L2ARC to contain the DDT. The system runs out of usable wired > memory=E2=80=8B and locks up. Adding more RAM and/or being patient with = the > boot-wait-lockup-repeat cycle will (usually) eventually allow it to finis= h > the destroy. > > There was a loader.conf tunable (or sysctl) added in the 10.x series that > mitigates this by limiting the number of delete operations that occur in = a > transaction group, but I forget the details on it. > > Not sure if this affects pools that never had dedupe enabled or not. > > (We used to suffer through this at least once a year until we enabled a > delete-oldest-snapshot-before-running-backups process to limit the number > of snapshots.)=E2=80=8B > =E2=80=8BFound it. You can set vfs.zfs.free_max_blocks in /etc/sysctl.conf= . That will limit the number to-be-freed blocks in a single transaction group. You can play with that number until you find a value that won't run the system out of kernel memory trying to free all those blocks in a single transaction. On our problem server, running dedupe with only 64 GB of RAM for a 53 TB pool, we set it to 200,000 blocks: =E2=80=8Bvfs.zfs.free_max_blocks=3D200000 --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@freebsd.org Wed Oct 4 22:11:13 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81033E43DF0 for ; Wed, 4 Oct 2017 22:11:13 +0000 (UTC) (envelope-from javocado@gmail.com) Received: from mail-ua0-x236.google.com (mail-ua0-x236.google.com [IPv6:2607:f8b0:400c:c08::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3918F740F8 for ; Wed, 4 Oct 2017 22:11:13 +0000 (UTC) (envelope-from javocado@gmail.com) Received: by mail-ua0-x236.google.com with SMTP id 47so7751855uas.8 for ; Wed, 04 Oct 2017 15:11:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=zZZ/5A88MUta6RoyqCNdNdtWw3SSwqgk3KPC3ck4DgU=; b=BHXq/u3FUsG5ThvZ3UfIoBQ/xt2U9JX1juOK9QvZLvbGLO/Zspqg3+Kb1OSdwKYg4X XhKUcZ8j7kFGOQNdOnUGiL/stehcwcwa+TckA2Gotti0KLyE8qSGFVMSDLAV0/U+nbbR hPMX6q4jSCaJrj9sn2OjCC+ENMP6gy4LnZlaOy5kNCRObNMZtLOBSrX9PONRNAVKPeLd Cb1n53+AQvyJ7Ly4BzVfB/OKxYKezNI7CXntMuKnmv2NHZj3EY4pBvT2zbomSzMqw2sU 5DPCZB/jcZvSxFFdyssMpxmvx9c+MaZ7jJL4QpS89WJyL0/v/chPizp/GkSaU1lmU180 uyVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=zZZ/5A88MUta6RoyqCNdNdtWw3SSwqgk3KPC3ck4DgU=; b=Kn+95pNb7yctZrcW5j0n0/YIn7mcbQcwqCQ16uM0DqJsTSsnuLee3sNMJq+lBqm1B+ NOhivFLcN3Jts1LoyRTo16zfRel9IpMB7CW9lT9QKN5ba30Lo05DNLtQKmUR8TDEAGQk FYtSK30VQwpqWF6xgLP7kRQfIJ2C2LgPuG3FCIVnBeb0gkRThjl6no4xaDRNtA/HVUH6 wjSdslsN9qQIWzrm5Cdxsya5ZPOsxj6jZADr7fZ70v3mi0/oYCXLxMG986oNaxBRf/ec n5Y2PT4TvLnHLC5E/eKA0oSn4YGmTtbByoT3m8v7YJ+Ycf5ChHnEUFjJWbriELzE93oB 5kbQ== X-Gm-Message-State: AMCzsaVimbW7tTxD89TeUGfDHrXlddU7sVywCIn4Bk6zrm04NAUW5hMH KtjFAh4BHAKT9P0n1vX9QH8u56jZVi+mlIir1QArRg== X-Google-Smtp-Source: AOwi7QCVEOUgqDp4tjMqPL4cgyCHqCjtVQcIdJWycaFfKZsygeVzKFR0/MlNUW2M+thOYJgxuYkB/IbA1HdkgMvP4V8= X-Received: by 10.176.92.74 with SMTP id a10mr11261446uag.165.1507155071983; Wed, 04 Oct 2017 15:11:11 -0700 (PDT) MIME-Version: 1.0 Received: by 10.159.51.90 with HTTP; Wed, 4 Oct 2017 15:11:11 -0700 (PDT) From: javocado Date: Wed, 4 Oct 2017 15:11:11 -0700 Message-ID: Subject: getting job/task info from a booting zpool To: FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Oct 2017 22:11:13 -0000 I have an issue I've detailed in a prior post: https://forums.freebsd.org/threads/62718/ However I'd like to specifically ask the community: When I run this as the zpool is importing: [CODE] # zdb -dddd pool/filesystem | grep DELETE_QUEUE zdb: can't open 'pool/filesystem': Device busy [/CODE] I assume dataset cannot tell me anything because it is pending destroy ... Specifically, I want to see the process of a pending filesystem destroy that is underway - I can't get that from the filesystem itself because it is being destroyed, but is there anything that zdb can tell me, from the zpool, about how the destruction is progressing? From owner-freebsd-fs@freebsd.org Thu Oct 5 05:13:30 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6B3DE2CDA4 for ; Thu, 5 Oct 2017 05:13:30 +0000 (UTC) (envelope-from javocado@gmail.com) Received: from mail-vk0-x22b.google.com (mail-vk0-x22b.google.com [IPv6:2607:f8b0:400c:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 91869FD for ; Thu, 5 Oct 2017 05:13:30 +0000 (UTC) (envelope-from javocado@gmail.com) Received: by mail-vk0-x22b.google.com with SMTP id u128so7178201vkg.10 for ; Wed, 04 Oct 2017 22:13:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=lWaQaMeKsxO2FinrniLFfEwn2C9EkG8rZxBy5wOJv3E=; b=nVVt8GcksbRB0cDReZ6eg5GzL2uXGKnjZWmMjJCf9a3u/FsuLM+vNj08a2ocCapPMk ffao5ZZEs7KHcJPzpGwAJwoPLEikXq5+4YDQYZ2YbIu8xU1QvIcTrsJlNBrV299DPA2U O6DkEbuM1D9sux4HdZ/VBrJ/tSpTjZLxCRsAlZO1aNEyR1RqVzpeOma8PqL++pOl110v lueZ3i+Jx9CzZBJA4V/TnV+JkwzmQBL9rI5SYUHcWv+7xk1d6XqxCiKTaoi2hwk2JSE8 /ue5QQDaMbYMYlb7QfK0J3+CxYA43Lg+IPhT8/Faq1QrQYnw0hKSQhrQwFXJ4TojZqFu nHBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=lWaQaMeKsxO2FinrniLFfEwn2C9EkG8rZxBy5wOJv3E=; b=oytegbsqOW6wvS3MFadFlxnn3Z2BkXW0HuC+ByNNMGw+K2QbXGq9mbiTIyZoeX0Ugq SRtB5BGJ5jpZWO+ip9SlyrhPQQjiFWFgHHznda7v898ihO//1CHuFQCLbEuo3vtuUD7I IqX0SYXqIkAV+y+MjkZfMMZhUr4G8pSl3nXq3RUdqGRt3VRb5OTYrLDxwKygOaZ8xCkX WFfg4uq/jvwWgCHWTRcmkuUwH2fYEC1lUxGvlo8ZNMKpkMwiVzon2Wi7HyRWkGujznOe /Bkdex7vxYtI6bO2P5qwW7k+uuH35P/LuTIH7C3ubzl+TSdXe8GW2NsLSzwAzyypZKTb t+YA== X-Gm-Message-State: AMCzsaW+RYAy+KvxfjDXX5PBxcSn1KqVkBJHNRSJlVMaMdKamgPXopLF pH4y9cNWP2QxTqiIYgNldU+oB1xRkuzavAZeHLo= X-Google-Smtp-Source: AOwi7QBN8M1h7+J9rOiyZqsjPZtOVtlrdMmk4F8A0cShcf2Sx0FIZDy5DHBaMDUgRKgXe/5m1aw729EE4GeqaO2I0JY= X-Received: by 10.31.171.146 with SMTP id u140mr1170419vke.44.1507180409389; Wed, 04 Oct 2017 22:13:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.159.51.90 with HTTP; Wed, 4 Oct 2017 22:13:28 -0700 (PDT) In-Reply-To: References: From: javocado Date: Wed, 4 Oct 2017 22:13:28 -0700 Message-ID: Subject: Re: lockup during zfs destroy To: Freddie Cash Cc: FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Oct 2017 05:13:31 -0000 Setting vfs.zfs.free_max_blocks to 20k has not helped unfortunately. I was able to get a small amount of debug out though. Any thoughts on how I can: - get more detailed debug on the progress of this operation, or whether progress is being made at all each time I reboot and start over after a freeze - configure my way out of this issue? # dtrace -q -n 'zfs-dbgmsg{printf("%s\n", stringof(arg0))}' txg 34628587 open pool version 28; software version 5000/5; uts host 10.3-RELEASE 1003000 amd64 txg 34628587 destroy begin tank/temp (id 3680) txg 34628588 destroy tank/temp (id 3680) On Wed, Oct 4, 2017 at 10:58 AM, Freddie Cash wrote: > On Wed, Oct 4, 2017 at 9:27 AM, Freddie Cash wrote: > >> On Wed, Oct 4, 2017 at 9:15 AM, javocado wrote: >> >>> I am trying to destroy a dense, large filesystem and it's not going wel= l. >>> >>> Details: >>> - zpool is a raidz3 with 3 x 12 drive vdevs. >>> - target filesystem to be destroyed is ~2T with ~63M inodes. >>> - OS: FreeBSD 10.3amd with 192 GB of RAM. >>> - 120 GB of swap (90GB recently added as swap-on-disk) >>> >> >> =E2=80=8BDo you have dedupe enabled on any filesystems in the pool? Or = was it >> enabled at any point in the past? >> >> This is a common occurrence when destroying large filesystems or lots of >> filesystems/snapshots on pools that have/had dedupe enabled and there's = not >> enough RAM/L2ARC to contain the DDT. The system runs out of usable wire= d >> memory=E2=80=8B and locks up. Adding more RAM and/or being patient with= the >> boot-wait-lockup-repeat cycle will (usually) eventually allow it to fini= sh >> the destroy. >> >> There was a loader.conf tunable (or sysctl) added in the 10.x series tha= t >> mitigates this by limiting the number of delete operations that occur in= a >> transaction group, but I forget the details on it. >> >> Not sure if this affects pools that never had dedupe enabled or not. >> >> (We used to suffer through this at least once a year until we enabled a >> delete-oldest-snapshot-before-running-backups process to limit the >> number of snapshots.)=E2=80=8B >> > > =E2=80=8BFound it. You can set vfs.zfs.free_max_blocks in /etc/sysctl.co= nf. That > will limit the number to-be-freed blocks in a single transaction group. > You can play with that number until you find a value that won't run the > system out of kernel memory trying to free all those blocks in a single > transaction. > > On our problem server, running dedupe with only 64 GB of RAM for a 53 TB > pool, we set it to 200,000 blocks: > > =E2=80=8Bvfs.zfs.free_max_blocks=3D200000 > > -- > Freddie Cash > fjwcash@gmail.com > From owner-freebsd-fs@freebsd.org Thu Oct 5 05:28:29 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6ECEDE2D050 for ; Thu, 5 Oct 2017 05:28:29 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-oi0-x236.google.com (mail-oi0-x236.google.com [IPv6:2607:f8b0:4003:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E288772 for ; Thu, 5 Oct 2017 05:28:29 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-oi0-x236.google.com with SMTP id m198so10590940oig.5 for ; Wed, 04 Oct 2017 22:28:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=kUPdpCOvjQduzMZ3nhzx5yg/fz9DMvC2L/IIH/O8n/0=; b=Uvi3I8J5oKWWZ8+4oVA9t26GMuFkqdxqStL+gW8M6cVRyNpg3yLt/dxUGCKHzgl/TR BbIljYqTUd3ZlAOT03EvZYIcNHFsHpx/OHKjSpD7MbOaWLHswAaDlRzVg2a/HB8lrI4V cfmBVfU1hJiZVEHcT99D/ac8PWayQrErrXcWdc4m0qCHBA9K9/EKyPJyC5wss7yhtaUt a8glMQU81ZIZdmmxEBcOwSqJ63xaAU2ZkjQvulR5z0W+caGLHgVOVldJfqauQc6pkCSt n1F3NtRFus8HPowhlwy4098FGOS5bw2Xx0JE8Qg9e6IENpueSZD5iZEBFzmL9gDFw8Hb DTYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=kUPdpCOvjQduzMZ3nhzx5yg/fz9DMvC2L/IIH/O8n/0=; b=SqrBdXetnttf3P4Kqdd4PcL652JNLRUSw6Uby937ynbI7U9UeRlUQm2nPB53C90cgm DIdIw30hptjMdBnGblQvDUb+PmMr/iMVgX/RVyeT3iIn+l+RV4LdrIT1q1ZO6oGvUuXe CR2DfBsJ6BiZ8GQXAKL7ygc5GYGme0mGh5kHfXbm5vejHB26kH1F56mweWpkw6muXnZb YFFse83XSoc40hcD9odAPF9rgjxoWdfrkg74cjCQfGfHjrrEz6wEkh2sLv9tKKb5aG6l 2+DdyCLFiMNeGzzwT56cuPo+obHDg7yugG+wILwLNr2f66IqXorVbb7YVhXrHtK/FgOB UeTw== X-Gm-Message-State: AMCzsaWbsfUR/0NobeVhcBQGfkD4RT2zqa7XfYFdr6wryVVg2KWKjIrb 17x6UxOye/OXfSLPjBXCdZq7mcne34VtxahjSnE= X-Google-Smtp-Source: AOwi7QCNat2R1CCwbxc7SoPOHZcpGBP9RKDrjARe6f1dBToxEsH2DTMvvdCGBf7kj/iGwIzzruI4BzpOEoSFfEFQmr4= X-Received: by 10.157.45.107 with SMTP id v98mr9773483ota.133.1507181308371; Wed, 04 Oct 2017 22:28:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.62.245 with HTTP; Wed, 4 Oct 2017 22:28:27 -0700 (PDT) Received: by 10.157.62.245 with HTTP; Wed, 4 Oct 2017 22:28:27 -0700 (PDT) In-Reply-To: References: From: Freddie Cash Date: Wed, 4 Oct 2017 22:28:27 -0700 Message-ID: Subject: Re: lockup during zfs destroy To: javocado Cc: FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Oct 2017 05:28:29 -0000 On Oct 4, 2017 10:13 PM, "javocado" wrote: Setting vfs.zfs.free_max_blocks to 20k has not helped unfortunately. No, that won't help with this issue as the destroy operation is already in progress and part of a transaction group. But it will mitigate or (hopefully) prevent this issue in the future. Cheers, Freddie From owner-freebsd-fs@freebsd.org Thu Oct 5 16:34:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 91AF0E3C8E3 for ; Thu, 5 Oct 2017 16:34:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8054C7C401 for ; Thu, 5 Oct 2017 16:34:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v95GYEcL058213 for ; Thu, 5 Oct 2017 16:34:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 218626] [PATCH] cuse: new error code CUSE_ERR_NO_DEVICE (ENODEV) Date: Thu, 05 Oct 2017 16:34:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: hselasky@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Oct 2017 16:34:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D218626 --- Comment #6 from Hans Petter Selasky --- I have been a bit busy. Will try to get this patch committed. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Oct 5 16:42:15 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A82CE3CCE2 for ; Thu, 5 Oct 2017 16:42:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 18F297CB8B for ; Thu, 5 Oct 2017 16:42:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v95GgEtM077392 for ; Thu, 5 Oct 2017 16:42:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 218626] [PATCH] cuse: new error code CUSE_ERR_NO_DEVICE (ENODEV) Date: Thu, 05 Oct 2017 16:42:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Oct 2017 16:42:15 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D218626 --- Comment #7 from commit-hook@freebsd.org --- A commit references this bug: Author: hselasky Date: Thu Oct 5 16:42:02 UTC 2017 New revision: 324320 URL: https://svnweb.freebsd.org/changeset/base/324320 Log: Add support for new cuse(3) error code, CUSE_ERR_NO_DEVICE. This error code is useful when emulating Linux input event devices from userspace. PR: 218626 Submitted by: jan.kokemueller@gmail.com MFC after: 1 week Sponsored by: Mellanox Technologies Changes: head/lib/libcuse/cuse.3 head/sys/fs/cuse/cuse.c head/sys/fs/cuse/cuse_defs.h --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Oct 5 16:43:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA865E3CDAA for ; Thu, 5 Oct 2017 16:43:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9898D7CD0D for ; Thu, 5 Oct 2017 16:43:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v95Gh3Xx079160 for ; Thu, 5 Oct 2017 16:43:03 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 218626] [PATCH] cuse: new error code CUSE_ERR_NO_DEVICE (ENODEV) Date: Thu, 05 Oct 2017 16:43:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: hselasky@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Oct 2017 16:43:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D218626 Hans Petter Selasky changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Progress |Closed Resolution|--- |FIXED --- Comment #8 from Hans Petter Selasky --- Thank you for being patient. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Oct 6 03:15:47 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27441E2529A for ; Fri, 6 Oct 2017 03:15:47 +0000 (UTC) (envelope-from rpp@ci.com.au) Received: from mippet.ci.com.au (mippet.ci.com.au [192.65.182.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mippet.ci.com.au", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B5C126C2F8 for ; Fri, 6 Oct 2017 03:15:46 +0000 (UTC) (envelope-from rpp@ci.com.au) Received: from mippet-2.ci.com.au (mippet-2.ci.com.au [192.168.1.254]) by mippet-dkim.ci.com.au (8.15.2/8.15.2/CE050417) with ESMTPS id v96323V7021187 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=OK); Fri, 6 Oct 2017 14:02:04 +1100 (AEDT) (envelope-from rpp@ci.com.au) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ci.com.au; s=jun2016; t=1507258924; bh=mSLse4npXbLjvfwxd7C4deHiJy97kUfAL/TPGRkTxXQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=MQHRXoi8kQBpt2mnRxZjfOXJJr2h29g+SbROjIeml/0lMZ0kwiHH+rdwsLyYiyaeW 74ysN3LrYIWtTpIi89LzZJm1jhMrFdKO2+dZ0uVTKtgLSmUT64w2z2uq4ytXAzs0+n DZSVx3+l+lmM20vpej3YyiAM3B6hKdiHQHym4R8kSVkJKoEN2YVE5JXY3U0Y7Vc6ZP wHffC9AW2X4s6FrlpYA7jFMYJQrnFbQ+Wau9DjV5VZWu1Ph1/mhOwUdDH0twpZjM1u PmW5rxWI/+RVwe/BlEX5dWIr1E96uCwcr9egOesHXrqHbYXAdG2aQSjAd2jqbLxV+k k0U7qAw4mfvlw== Received: from jodi.ci.com.au (jodi.ci.com.au [192.168.1.21]) by mippet.ci.com.au (8.15.2/8.15.2/CE120917) with ESMTPS id v96323s2021184 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Fri, 6 Oct 2017 14:02:03 +1100 (AEDT) (envelope-from rpp@ci.com.au) Received: from jodi.ci.com.au (jodi.ci.com.au [192.168.1.21]) by jodi.ci.com.au (8.15.2/8.15.2) with SMTP id v96323ta030825; Fri, 6 Oct 2017 14:02:03 +1100 (AEDT) (envelope-from rpp@ci.com.au) Date: Fri, 6 Oct 2017 14:02:03 +1100 From: Richard Perini To: FreeBSD Filesystems Cc: javocado Subject: Re: lockup during zfs destroy Message-ID: <20171006030203.GA30590@jodi.ci.com.au> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Oct 2017 03:15:47 -0000 On Wed, Oct 04, 2017 at 10:28:27PM -0700, Freddie Cash wrote: > On Oct 4, 2017 10:13 PM, "javocado" wrote: > > Setting vfs.zfs.free_max_blocks to 20k has not helped unfortunately. > > > No, that won't help with this issue as the destroy operation is already in > progress and part of a transaction group. But it will mitigate or > (hopefully) prevent this issue in the future. A bit of a long shot as there's no mention of the FreeBSD version involoved, but we had somewhat similar symptoms caused by bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222288 Cheers, -- Richard Perini Ramico Australia Pty Ltd Sydney, Australia rpp@ci.com.au +61 2 9552 5500 ----------------------------------------------------------------------------- "The difference between theory and practice is that in theory there is no difference, but in practice there is" From owner-freebsd-fs@freebsd.org Fri Oct 6 10:09:00 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05F44E32C69; Fri, 6 Oct 2017 10:09:00 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x244.google.com (mail-wm0-x244.google.com [IPv6:2a00:1450:400c:c09::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9652C7C264; Fri, 6 Oct 2017 10:08:59 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x244.google.com with SMTP id q132so6924175wmd.2; Fri, 06 Oct 2017 03:08:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=UpHVYrJAyRFA/okHlePHWZbTVK8Y5iq4PgSPAvQf7wM=; b=UUODtDqWhtKIt7hgjHIh2lgw6PDjBGRRdbp9v6rQTAyXIIyTfT835M7t+/hnIEgtaN +6V0Ar/APAR/lFMDJyg9obpXsd1YnG/6ziy3DDqbpWvrfzB6/Urpt4eWX8A76zuagaKG Cuxd4YbUMOLtQLjCX2fJ4akMUL3nJgVxFMVTcbFlh0bdK71UFhu4N2f/rWfNfhccw2kw kKN9RvNeZjckqztmAIKn3e5OBC94EL3/qzRuGT2u70STsuXw3qAtYM3fauo3phN9dFSy op5aF7UIXxL7ZrgkqpFW6sntmmvzjZj9VLIPnZDmjtdoBv8wF54ySoRzGTspjhkMQZ2e XjIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=UpHVYrJAyRFA/okHlePHWZbTVK8Y5iq4PgSPAvQf7wM=; b=R9LfkjmUmIEO/WjPJMGIe4Wk7B+iej5fKJIVnGp83tEZpdrANBWbf8tWOOcA4zPY9B hkUoAzLJta+NaVLRws7Iv9AHjIDrz3vic2tkyNHTqqJnJesR6J7aIYxMlo7dGjrftbO8 AYxumqSmXgf9lghskqPIxDeihmAgBL8U5qkI/xrmsuFx5rpPuDb7nLEdVQmBqGMjU0Ck LOtSKpdrOFf7mrZfS9a46eU9SWrKq0QBmt7QyMbwWQsDpry27l0GJA98k6vTIa26yklu oVT0xAAyLfEJszNCsQFNnCawsSYa8AiKK5j/y2G893o+WB1TAFdYZBC3db+MBkEMIrCZ 5OiQ== X-Gm-Message-State: AMCzsaVImHJhkq59FyGwsq/6rlndaqM9Sw6srdAdvyvXOBnVphNOwwAv K2YeOHctD4CxfO7obCBZnLhNolMq X-Google-Smtp-Source: AOwi7QDrbUNE/rsqNrs4O7gWR+HaVkzwREPc1nFG0/NchrPl3oW79X6VUfI4ae6tT+deecQzd4TLvA== X-Received: by 10.28.153.85 with SMTP id b82mr1125513wme.121.1507284537762; Fri, 06 Oct 2017 03:08:57 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id d17sm985661wrc.13.2017.10.06.03.08.56 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 06 Oct 2017 03:08:57 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: Date: Fri, 6 Oct 2017 12:08:55 +0200 Cc: Freebsd fs Content-Transfer-Encoding: quoted-printable Message-Id: <82632887-E9D4-42D0-AC05-3764ABAC6B86@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> To: FreeBSD-scsi , =?utf-8?Q?Edward_Tomasz_Napiera=C5=82a?= X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Oct 2017 10:09:00 -0000 > On 02 Oct 2017, at 20:12, Ben RUBSON wrote: >=20 > Hi, >=20 > On a FreeBSD 11 server, the following online/healthy zpool : >=20 > home > mirror-0 > label/local1 > label/local2 > label/iscsi1 > label/iscsi2 > mirror-1 > label/local3 > label/local4 > label/iscsi3 > label/iscsi4 > cache > label/local5 > label/local6 >=20 > A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk > according to "zpool iostat", nothing on local disks. > No write IOs. >=20 > Let's disconnect all iSCSI disks : > iscsictl -Ra >=20 > Expected behavior : > IO activity flawlessly continue on local disks. >=20 > What happened : > All IOs stalled, server only answers to IOs made to its zroot pool. > All commands related to the iSCSI disks (iscsictl), or to ZFS = (zfs/zpool), > don't return. >=20 > Questions : > Why this behavior ? > How to know what happens ? (/var/log/messages says almost nothing) >=20 > I already disconnected the iSCSI disks without any issue in the past, > several times, but there were almost no IOs running. >=20 > Thank you for your help ! >=20 > Ben Hello, So first, many thanks again to Andriy, we spent almost 3 hours debugging = the stalled server to find the root cause of the issue. Sounds like I would need help from iSCSI dev team (Edward perhaps ?), as = issue seems to be on this side. Here is Andriy conclusion after the debug session, I quote him : > So, it seems that the root cause of all evil is this outstanding zio = (it might > be not the only one). > In other words, it looks like iscsi stack bailed out without = completing all > outstanding i/o requests that it had. > It should either return success or error for every request, it can not = simply > drop a request. > And that appears to be what happened here. > It looks like ZFS is fragile in the face of this type of errors. > Essentially, each logical i/o request obtains a configuration lock of = type 'zio' > in shared mode to prevent certain configuration changes from happening = while > there are any outsanding zio-s. > If a zio is lost, then this lock is leaked. > Then, the code that deals with vdev failures tries to take this lock = in > exclusive mode while holding a few other configuration locks also in = exclsuive > mode so, any other thread needing those locks would block. > And there are code paths where a configuration lock is taken while > spa_namespace_lock is held. > And when spa_namespace_lock is never dropped then the system is close = to toast, > because all pool lookups would get stuck. > I don't see how this can be fixed in ZFS. > It seems that when the initiator is being removed it doesn't properly = terminate > in-glight requests. > It would be interesting to see what happens if you test other = scenarios. So I tested the following other scenarios : 1 - drop all iSCSI traffic using ipfw on the target 2 - ifdown the iSCSI NIC on the target 3 - ifdown the iSCSI NIC on the initiator 4 - stop ctld (on the target of course) I tested all of them several times, 5 or 6 times each ? I managed to kernel panic (!) 2 times. First time in case 2. Second time in case 4. Not sure I would not have been able to panic in other test cases though. Stack traces : https://s1.postimg.org/2hfdpsvban/panic_case2.png https://s1.postimg.org/2ac5ud9t0f/panic_case4.png (kgdb) list *g_io_request+0x4a7 0xffffffff80a14dc7 is in g_io_request (/usr/src/sys/geom/geom_io.c:638). 633 g_bioq_unlock(&g_bio_run_down); 634 /* Pass it on down. */ 635 if (first) 636 wakeup(&g_wait_down); 637 } 638 } 639=09 640 void 641 g_io_deliver(struct bio *bp, int error) 642 { I had some kernel panics on the same servers a few months ago, loosing iSCSI targets which were used in a gmirror with local disks. gmirror should have continued to work flawlessly (as ZFS) using local disks but the server crashed. Stack traces : https://s1.postimg.org/14v4sabhv3/panic_g_destroy1.png https://s1.postimg.org/437evsk6rz/panic_g_destroy2.png https://s1.postimg.org/8pt1whiy5b/panic_g_destroy3.png (kgdb) list *g_destroy_consumer+0x53 0xffffffff80a18563 is in g_destroy_consumer (geom.h:369). 364 KASSERT(g_valid_obj(ptr) =3D=3D 0, 365 ("g_free(%p) of live object, type %d", ptr, 366 g_valid_obj(ptr))); 367 } 368 #endif 369 free(ptr, M_GEOM); 370 } 371=09 372 #define g_topology_lock() = \ 373 do { = \ > I think that all problems that you have seen are different sides of = the same > underlying issue. It looks like iscsi does not properly depart from = geom and > leaves behind some dangling pointers... >=20 > The panics you got today most likely occurred here: > bp->bio_to->geom->start(bp); >=20 > And the most likely reason is that bio_to points to a destroyed geom = provider. >=20 > I wonder if you'd be able to get into direct contact with a developer > responsible for iscsi in FreeBSD. I think that it is a relatively = recent > addition and it was under a FreeBSD Foundation project. So, I'd = expect that the > developer should be responsive. Feel free then to contact me if you need, so that we can go further on = this ! Thank you very much for your help, Ben From owner-freebsd-fs@freebsd.org Sat Oct 7 13:13:16 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5427AE37FAA for ; Sat, 7 Oct 2017 13:13:16 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay08.ispgateway.de (smtprelay08.ispgateway.de [134.119.228.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F15D47580E; Sat, 7 Oct 2017 13:13:15 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.164.83] (helo=fabiankeil.de) by smtprelay08.ispgateway.de with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1e0otw-0002jl-9a; Sat, 07 Oct 2017 15:12:20 +0200 Date: Sat, 7 Oct 2017 15:08:48 +0200 From: Fabian Keil To: Ben RUBSON Cc: Freebsd fs , Edward Tomasz =?UTF-8?B?TmFwaWVy?= =?UTF-8?B?YcWCYQ==?= Subject: Re: ZFS stalled after some mirror disks were lost Message-ID: <20171007150848.7d50cad4@fabiankeil.de> In-Reply-To: <82632887-E9D4-42D0-AC05-3764ABAC6B86@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <82632887-E9D4-42D0-AC05-3764ABAC6B86@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/1PQyDAVhgdU79DS=P/mu2nf"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Oct 2017 13:13:16 -0000 --Sig_/1PQyDAVhgdU79DS=P/mu2nf Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Ben RUBSON wrote: > So first, many thanks again to Andriy, we spent almost 3 hours debugging > the stalled server to find the root cause of the issue. >=20 > Sounds like I would need help from iSCSI dev team (Edward perhaps ?), as > issue seems to be on this side. Maybe. > Here is Andriy conclusion after the debug session, I quote him : >=20 > > So, it seems that the root cause of all evil is this outstanding zio > > (it might be not the only one). > > In other words, it looks like iscsi stack bailed out without > > completing all outstanding i/o requests that it had. > > It should either return success or error for every request, it can not > > simply drop a request. > > And that appears to be what happened here. =20 >=20 > > It looks like ZFS is fragile in the face of this type of errors. Indeed. In the face of other types of errors as well, though. > > Essentially, each logical i/o request obtains a configuration lock of > > type 'zio' in shared mode to prevent certain configuration changes > > from happening while there are any outsanding zio-s. > > If a zio is lost, then this lock is leaked. > > Then, the code that deals with vdev failures tries to take this lock in > > exclusive mode while holding a few other configuration locks also in > > exclsuive mode so, any other thread needing those locks would block. > > And there are code paths where a configuration lock is taken while > > spa_namespace_lock is held. > > And when spa_namespace_lock is never dropped then the system is close > > to toast, because all pool lookups would get stuck. > > I don't see how this can be fixed in ZFS. =20 While I haven't used iSCSI for a while now, over the years I've seen lots of similar issues with ZFS pools located on external USB disks and ggate devices (backed by systems with patches for the known data corruption issues). At least in my opinion, many of the various known spa_namespace_lock issues are plain ZFS issues and could be fixed in ZFS if someone was motivated enough to spent the time to actually do it (and then jump through the various "upstreaming" hoops). In many cases tolerable workarounds exist, though, and sometimes they work around some of the issues well enough. Here's an example workaround that I've been using for a while now: https://www.fabiankeil.de/sourcecode/electrobsd/ElectroBSD-r312620-6cfa243f= 1516/0222-ZFS-Optionally-let-spa_sync-wait-until-at-least-one-v.diff According to the commit message the issue was previously mentioned on freebsd-current@ in 2014 but I no longer remember all the details and didn't look them up. I'm not claiming that the patch or other workarounds I'm aware of would actually help with your ZFS stalls at all, but it's not obvious to me that your problems can actually be blamed on the iSCSI code either. Did you try to reproduce the problem without iSCSI? BTW, here's another (unrelated but somewhat hilarious) example of a known OpenZFS issue next to nobody seems to care about: https://lists.freebsd.org/pipermail/freebsd-fs/2017-August/025110.html I no longer care about this issue either (and thus really can't complain), but I was a bit surprised by the fact that issues like this one survive for so many years in an "enterprise" file system like ZFS. Anyway, good luck with your ZFS-on-iscsi issue(s). Fabian --Sig_/1PQyDAVhgdU79DS=P/mu2nf Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQTKUNd6H/m3+ByGULIFiohV/3dUnQUCWdjR4QAKCRAFiohV/3dU nR9oAJ0SFKK9AusN1+7tAZJZ+HMZPPeWUwCeLZNzFvzFh7KS/1pcIV+BJxD3xOA= =hvLO -----END PGP SIGNATURE----- --Sig_/1PQyDAVhgdU79DS=P/mu2nf-- From owner-freebsd-fs@freebsd.org Sat Oct 7 13:57:37 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18971E38BB5 for ; Sat, 7 Oct 2017 13:57:37 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78A72825EE; Sat, 7 Oct 2017 13:57:36 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22e.google.com with SMTP id b189so12705065wmd.4; Sat, 07 Oct 2017 06:57:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=8yihce3jASILHlvWfn4dKqrEgVw8cx0NVaP9XsVMWsU=; b=q6mIJZOPmSs+wIrHKEpMDfkLHpzAJMdh1g3YiQvrARKzYmMjkmtq07Gx/3vGHVYmyV wqjByH5ySslBzByQX6o/O3Bn4n+hjtanpo95r+Y7lTtpJbRF0aAWNJlt16evPMUqLbqe FF8ae6T2C8G/ljFN3DcOjLMowGqPyt5uXN7fqrS8sBAohq2pMJysEUa4yihJkEtw8EeX CflffOfbdyqJaDKURQV5FRg0kiuuwmVELCgYjSHvCivDIK2/CdmgaQrHT/3Vf8I+54Yg WC//rCx37mRmwBnbo/GdpSaYcH26YT+d1FH+RUtpgbu98v9ElN/4JfpZ6KkLDspthVYU qOlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=8yihce3jASILHlvWfn4dKqrEgVw8cx0NVaP9XsVMWsU=; b=r8+2vhWn+lqGJZYQGnEoLIOEaHrHHs37wybpPv/s8oENiWor81GM27EMjM4URRUOYI bUmfLC1JhS6AP4PajEBYhCRW1RcLJirbHiftxMg3CESq1l/bj/gaXOoNiGizkUiQw1Lw fUvtOU5pPw7G9pbp/s47/gZecEazZdFN+ve5dpPuXg6KQtDHXqTXvmCcK94zj2b+3jhx J4wWNXTcjUMmXU+VdOv/GZ7mOcizsk1UfvK/UjGx1yomqYTzM1ZOeWrYZlp/P0seMVy8 yJz5z/qpYqJfNO9G0kD2r5uk+tSvUtjw7nWY05tJfruHiJxeUS3Ca5FPrm7lejeDvb0V ZtuQ== X-Gm-Message-State: AMCzsaVWLshmmrdvkJ2thYQYZNdCb8UiYukIjwquoyAU+T9tRHQiA9I8 n021HZETHuiCuB9hNK/X1KEyn09dn7E= X-Google-Smtp-Source: AOwi7QDD0IynrCRXTY3n6m6/23Vz21Wto31S9j14LQ1K21qpQmetjHGQiImITYvGuJ9ElAgmxl75cg== X-Received: by 10.28.48.143 with SMTP id w137mr3618534wmw.3.1507384653048; Sat, 07 Oct 2017 06:57:33 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id p95sm8673178wrc.53.2017.10.07.06.57.31 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 07 Oct 2017 06:57:32 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: <20171007150848.7d50cad4@fabiankeil.de> Date: Sat, 7 Oct 2017 15:57:30 +0200 Cc: =?utf-8?Q?Edward_Tomasz_Napiera=C5=82a?= , Fabian Keil , mav@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <82632887-E9D4-42D0-AC05-3764ABAC6B86@gmail.com> <20171007150848.7d50cad4@fabiankeil.de> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Oct 2017 13:57:37 -0000 > On 07 Oct 2017, at 15:08, Fabian Keil = wrote: >=20 > Ben RUBSON wrote: >=20 >> So first, many thanks again to Andriy, we spent almost 3 hours = debugging >> the stalled server to find the root cause of the issue. >>=20 >> Sounds like I would need help from iSCSI dev team (Edward perhaps ?), = as >> issue seems to be on this side. >=20 > Maybe. >=20 >> Here is Andriy conclusion after the debug session, I quote him : >>=20 >>> So, it seems that the root cause of all evil is this outstanding zio >>> (it might be not the only one). >>> In other words, it looks like iscsi stack bailed out without >>> completing all outstanding i/o requests that it had. >>> It should either return success or error for every request, it can = not >>> simply drop a request. >>> And that appears to be what happened here. =20 >>=20 >>> It looks like ZFS is fragile in the face of this type of errors. >=20 > Indeed. In the face of other types of errors as well, though. >=20 >>> Essentially, each logical i/o request obtains a configuration lock = of >>> type 'zio' in shared mode to prevent certain configuration changes >>> from happening while there are any outsanding zio-s. >>> If a zio is lost, then this lock is leaked. >>> Then, the code that deals with vdev failures tries to take this lock = in >>> exclusive mode while holding a few other configuration locks also in >>> exclsuive mode so, any other thread needing those locks would block. >>> And there are code paths where a configuration lock is taken while >>> spa_namespace_lock is held. >>> And when spa_namespace_lock is never dropped then the system is = close >>> to toast, because all pool lookups would get stuck. >>> I don't see how this can be fixed in ZFS. =20 >=20 > While I haven't used iSCSI for a while now, over the years I've seen > lots of similar issues with ZFS pools located on external USB disks > and ggate devices (backed by systems with patches for the known data > corruption issues). >=20 > At least in my opinion, many of the various known spa_namespace_lock > issues are plain ZFS issues and could be fixed in ZFS if someone was > motivated enough to spent the time to actually do it (and then jump > through the various "upstreaming" hoops). >=20 > In many cases tolerable workarounds exist, though, and sometimes they > work around some of the issues well enough. Here's an example = workaround > that I've been using for a while now: > = https://www.fabiankeil.de/sourcecode/electrobsd/ElectroBSD-r312620-6cfa243= f1516/0222-ZFS-Optionally-let-spa_sync-wait-until-at-least-one-v.diff >=20 > According to the commit message the issue was previously mentioned on > freebsd-current@ in 2014 but I no longer remember all the details and > didn't look them up. There's no mention to code revision in this thread. It finishes with a message from Alexander Motin : "(...) I've got to conclusion that ZFS in many places written in a way that simply does not expect errors. In such cases it just stucks, waiting for disk to reappear and I/O to complete. (...)" > I'm not claiming that the patch or other workarounds I'm aware of > would actually help with your ZFS stalls at all, but it's not obvious > to me that your problems can actually be blamed on the iSCSI code > either. >=20 > Did you try to reproduce the problem without iSCSI? No, I would have to pull out disks from their slots (well...), or = shut-down the SAS2008-IT adapter, or put disks offline (not sure how-to for these = two). I will test in the next few hours without GPT labels and GEOM labels, as I use them and Andriy suspects they could be the culprit. > Anyway, good luck with your ZFS-on-iscsi issue(s). Thank you very much Fabian for your help and contribution, I really hope we'll find the root cause of this issue, as it's quite annoying in a HA-expected production environment :/ Ben