From owner-freebsd-fs@freebsd.org Sun Oct 28 20:01:26 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB22F10DEE59 for ; Sun, 28 Oct 2018 20:01:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 787FE6DB9F for ; Sun, 28 Oct 2018 20:01:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 3D79D10DEE57; Sun, 28 Oct 2018 20:01:26 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2C77610DEE56 for ; Sun, 28 Oct 2018 20:01:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C253B6DB99 for ; Sun, 28 Oct 2018 20:01:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 069791F29E for ; Sun, 28 Oct 2018 20:01:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w9SK1Ose030463 for ; Sun, 28 Oct 2018 20:01:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w9SK1OlZ030458 for fs@FreeBSD.org; Sun, 28 Oct 2018 20:01:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 229614] ZFS lockup in zil_commit_impl Date: Sun, 28 Oct 2018 20:01:19 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: andreas.sommer87@googlemail.com X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: allanjude@FreeBSD.org X-Bugzilla-Flags: mfc-stable11? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Oct 2018 20:01:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229614 --- Comment #32 from Andreas Sommer --- (In reply to Allan Jude from comment #31) I have `kern.maxvnodes: 350367` now. Tried to find the racing process but h= ang the system using procstat =E2=80=93 will try your suggestion next time arou= nd. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Sun Oct 28 21:00:30 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CEF610E07A3 for ; Sun, 28 Oct 2018 21:00:30 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id F20436FBA4 for ; Sun, 28 Oct 2018 21:00:29 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id B6CBB10E079E; Sun, 28 Oct 2018 21:00:29 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9534510E0793 for ; Sun, 28 Oct 2018 21:00:29 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3744B6FB9B for ; Sun, 28 Oct 2018 21:00:29 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 6F8ED1F9F9 for ; Sun, 28 Oct 2018 21:00:28 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w9SL0So2064137 for ; Sun, 28 Oct 2018 21:00:28 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w9SL0Sv2064123 for fs@FreeBSD.org; Sun, 28 Oct 2018 21:00:28 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201810282100.w9SL0Sv2064123@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 28 Oct 2018 21:00:28 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Oct 2018 21:00:30 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off New | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 6 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Oct 29 03:14:09 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4825C10EDF20; Mon, 29 Oct 2018 03:14:09 +0000 (UTC) (envelope-from avv314@gmail.com) Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A59C57B45F; Mon, 29 Oct 2018 03:14:08 +0000 (UTC) (envelope-from avv314@gmail.com) Received: by mail-lf1-x136.google.com with SMTP id c24-v6so4867904lfi.12; Sun, 28 Oct 2018 20:14:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=ggJzXfVQYDNIJENuH0D/T6zuBu68RltBSOiZnJETWm8=; b=Iy+aGVQfIi6TqC92jmsn79DhIHCWXeRYcjdSO1nx3Pp0kfNQ0IG6mp40+JuTyRMAhk SEGL6hh8WZ8WY9KK8CgjCq5B2XsSTs7Pre48GQBkzSjePY9guQGF++nkiZIyUuwi/G8U LzckYaxwekKYEtoUIq0zf5vLBi69Kqli4ZeMBouV+/hx0grH41f24KusbLEojV1jC3St XG8CUAmgZTPnNFHYLUNs5uyIW0Jb7b+ouQQDqpcUEBwURLL5DnVv9wE/vKuTuZ8TEhmT o/R4QpHGWsfQM9I7AM+gw/1wFQrRPvz7h4KCqwI+76Pzcb1o8Pdg3FMGOt/uMxWELpRT RD8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=ggJzXfVQYDNIJENuH0D/T6zuBu68RltBSOiZnJETWm8=; b=ksam3zqF8dYVG0Mf0Lg2FvIITXR9jVenfQJdbbxEz9VnD7jO0viNQ8HlV2emhE27y6 Jzlu7WGM6rrebytG3h5PAwn4ScQHffHBNFU1ylbzVAgimOXYZE5XMlbfvnvBKVitGqm7 5jlJVrT6Q+vlvNkpfu3Us8iU3oJR/usZ5jYYoz8IyIK5/rSSKte6BsDkRVtOb+hKaWaT i4k7TAisimYyddXTL9Zx8q/RY5rd1yeBN7TRb8tPmlZdQ8OIYlJkcxAlywE73UMFie6I K3vi4mnR+kodWjX8w7GSB0UGvikn5gyaeQA/kZ+Vp3q6gCWL/xPHdRZsk9HouFQ+bOLi UUAw== X-Gm-Message-State: AGRZ1gIX1svJBh+U0P9wyrnq8bfbWDyzsEP1Rp6a01F/8mGCU3a2FO7E IN6fxKjOYraCIdUYkXC2Q8ZoVotcsxr/n+W56XLd7ZlZ X-Google-Smtp-Source: AJdET5d6t3DptH6g1eM5Z0kMzKQh2b8B5O0H3UHDoAhIEdxIQPmkKQajYQYvj3ztcV5F7xPTg/tYDWeqJbKdtxoCDY4= X-Received: by 2002:a19:d824:: with SMTP id p36-v6mr6836298lfg.23.1540782846693; Sun, 28 Oct 2018 20:14:06 -0700 (PDT) MIME-Version: 1.0 From: Andrew Vylegzhanin Date: Mon, 29 Oct 2018 06:13:55 +0300 Message-ID: Subject: NFS + Infiniband problem To: freebsd-infiniband@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 03:14:09 -0000 Hello everyone, I have a several FreeBSD machines connected via Infiniband netwok ( FDR switch Mellanox SW3036 + ConnectX-3 VPI cards ). One of them is a NAS-server with multiply ZFS pools. All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried) on server) are with infiniband connected mode (option IPOIB_CM, option SDM) and world with OFED stack support. (WITH_OFED='yes'). File transfers via FTP or SSH between server and clients works almost flawless ( ~ 12 Gbit/s ). But when I try to copy in/out some significant data via NFS share mounted on clients, NFS i/o hangs at all or got extremely slow (couple kB/s) transfer speed after uncertain amount of copied data. For example, on the one node I can copy 1GB file, and after NFS hang on file with size 30 kb. Some details: [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 /mnt [root@node4 ~]# dd if=/dev/zero of=/mnt/N1 bs=1m count=1024 Ctrl-T for "hang" dd load: 0.01 cmd: dd 1061 [bo_wwait] 70.95r 0.00u 0.00s 0% 2112k load: 0.01 cmd: dd 1061 [bo_wwait] 72.89r 0.00u 0.00s 0% 2112k for "slow" dd load: 0.00 cmd: dd 2254 [nfsaio] 224.18r 0.00u 0.13s 0% 3132k load: 0.00 cmd: dd 2254 [nfsaio] 225.94r 0.00u 0.13s 0% 3132k I've tried mount with different wsize option with same result. Any help would be greatly appreciated. -- Andrew From owner-freebsd-fs@freebsd.org Mon Oct 29 05:16:15 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D2CE10F086A; Mon, 29 Oct 2018 05:16:15 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670073.outbound.protection.outlook.com [40.107.67.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 107867E8D8; Mon, 29 Oct 2018 05:16:14 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB1899.CANPRD01.PROD.OUTLOOK.COM (52.132.49.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1273.25; Mon, 29 Oct 2018 05:16:13 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::a086:adbc:2b38:1018]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::a086:adbc:2b38:1018%2]) with mapi id 15.20.1273.027; Mon, 29 Oct 2018 05:16:13 +0000 From: Rick Macklem To: Andrew Vylegzhanin , "freebsd-infiniband@freebsd.org" , "freebsd-fs@freebsd.org" Subject: Re: NFS + Infiniband problem Thread-Topic: NFS + Infiniband problem Thread-Index: AQHUbzV7uxRrUqx4EESM4oETGDUinKU1rGlW Date: Mon, 29 Oct 2018 05:16:13 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB1899; 6:jUEUM4mhLPavH9axN7rp7y95BvXxalJNNngx8oNOa02CADyfeFvW8R8TpPHB58sntXIv633B6tzgUcbwBTnnATcieDgi+jwt9r19/EZ8aMwM/YCHgTMpwciVg3Ne83vGoCQRPbFV5PDcBrjMzcvGBkOkNyH7B1aSFUMPtBxttxXxsRiqHr8x9V/J8fu4E+naZ2GnlvFByUR+BfvMfsWjyDEwlzcu9xEURm0OcfulGB5SlJe0mb0eLDU44DbWHMqbZul1v5WlNcVkuG3WmrPWPk4zWkv2pEJHP51/QXgoP584RNxK30bX6mtW2Vyd5Y785+WImKY6Y3Wzak1UpGfgbENYmwcSbwg01nhwynSHAI6433+GGxJ2X/0XdAJ9QwTAdiiitkHrKPUZ04LFN0mg8sLLTRZwNiQLL15I/XofVeDThJmRCq2MKk9JYIWAB47R0naulhLCYi5OdDON5gOAcw==; 5:lbQ8OfRBk9eSLoyx5p/6hOFLxO23Klzh3PfI8vthOAlRKaFaQd0gGTkbmHJ4a7mdF5HDO0PXoOD01jE//tcNPvBOTeQnYLEaUwpbK6Fi468A3/2O566zy+V8AHxhnStzxcq37pujXZx8bqbpJ2iLNCCQ6yqEMKxFgDxqzLeutUI=; 7:ofrX/rnslXHFmTANd/N16XFNGgv2bJLonTqs5Y4oWTOVloJPBCaC9c9lMy9Ecn2SnF6IOKcaqio+ZTjo4xez9s9It8740A47YtRSpar9M8gZkW1NsYHWFIuSahLD3IIqLDyKEBAMwdV0XdStpOpXHQ== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 6bb000ad-79ed-4330-ddce-08d63d5da556 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB1899; x-ms-traffictypediagnostic: YTOPR0101MB1899: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(3002001)(10201501046)(93006095)(93001095)(3231382)(944501410)(52105095)(148016)(149066)(150057)(6041310)(20161123562045)(20161123564045)(20161123560045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB1899; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB1899; x-forefront-prvs: 084080FC15 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(346002)(376002)(136003)(39860400002)(366004)(189003)(199004)(39060400002)(74482002)(186003)(316002)(105586002)(33656002)(256004)(5250100002)(2501003)(102836004)(55016002)(25786009)(786003)(7696005)(76176011)(14454004)(110136005)(106356001)(6506007)(81166006)(6436002)(68736007)(2906002)(81156014)(2900100001)(53936002)(9686003)(99286004)(486006)(6246003)(97736004)(71190400001)(74316002)(86362001)(46003)(2201001)(8676002)(476003)(229853002)(5660300001)(11346002)(478600001)(446003)(8936002)(305945005)(71200400001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB1899; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: 8ip4rNKt1HbRXthqs8r9tA9MoSfoZoNnIfZ100IiAD/7WNeG+f3nLAXB+DSD5ypoGkqPWt8LqMu3pESV3eqKZTQG44PsGTKwQcAPlrzbRIis0ajWNGjqvlQy1l2riHWC3bCx5srQ9sp7H21xfBZr8PukXmljRTJLcjbi54CMkfPR6kHrbKjN3c6m3LgVvsuETxMwy9A6jUifeAwuLzABvcOhjnnKi2OfpABi+gfEq0NEvAj0KR0QIprg8Y4ufk2M1E++WRq84zCaxz/rpwsstUm2oBrrSfgBAvHDVjlJDHKdLcYqiiWoVO/NSZFPTsEa8dj0oa6wsUFgm7Ni3i86+skpCtx+YDuk9wCA1wAico4= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 6bb000ad-79ed-4330-ddce-08d63d5da556 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Oct 2018 05:16:13.4065 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1899 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 05:16:15 -0000 [stuff related to slow performance snipped] Try disabling TSO, that is the most common cause of NFS RPC transport issues. # sysctl net.inet.tcp.tso=3D0 will do it, if you can't do it for the interface. (Also, try disabling LSO, LRO if the interface will let you do so.) You can also try mounting with "rsize=3D8192,wsize=3D8192" and if that work= s fairly well, then just keep doubling it until it no longer works well. I know nothing about InfiniBand, so if the above doesn't help, hopefully someone familiar with InfiniBand can help. Good luck with it, rick From owner-freebsd-fs@freebsd.org Mon Oct 29 15:06:38 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1308D10DCBC0; Mon, 29 Oct 2018 15:06:38 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5E4F474B2F; Mon, 29 Oct 2018 15:06:37 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w9TF6Yve057203; Mon, 29 Oct 2018 08:06:34 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w9TF6YAP057202; Mon, 29 Oct 2018 08:06:34 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> Subject: Re: NFS + Infiniband problem In-Reply-To: To: Andrew Vylegzhanin Date: Mon, 29 Oct 2018 08:06:34 -0700 (PDT) CC: freebsd-infiniband@freebsd.org, freebsd-fs@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 15:06:38 -0000 > Hello everyone, > > I have a several FreeBSD machines connected via Infiniband netwok ( FDR > switch Mellanox SW3036 + ConnectX-3 VPI cards ). > One of them is a NAS-server with multiply ZFS pools. > > All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried) on > server) are with infiniband connected mode (option IPOIB_CM, option SDM) > and world with OFED stack support. (WITH_OFED='yes'). > > File transfers via FTP or SSH between server and clients works almost > flawless ( ~ 12 Gbit/s ). > > But when I try to copy in/out some significant data via NFS share mounted > on clients, NFS i/o hangs at all or got extremely slow (couple kB/s) > transfer speed after uncertain amount of copied data. For example, on the > one node I can copy 1GB file, and after NFS hang on file with size 30 kb. > > Some details: > [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 /mnt ^^^^^^^^^^^^ I am not sure what the interaction between page sizes, TSO needs, buffer needs and all that are but I always use a power of 2 wsize and rsize. You might try that. And as Rick suggested, turn of TSO, if you can. Is infiniband using RDMA to do this, if so then the page size stuff is probably very important, use multiples of 4096 only. > [root@node4 ~]# dd if=/dev/zero of=/mnt/N1 bs=1m count=1024 > > Ctrl-T for "hang" dd > load: 0.01 cmd: dd 1061 [bo_wwait] 70.95r 0.00u 0.00s 0% 2112k > load: 0.01 cmd: dd 1061 [bo_wwait] 72.89r 0.00u 0.00s 0% 2112k > > for "slow" dd > load: 0.00 cmd: dd 2254 [nfsaio] 224.18r 0.00u 0.13s 0% 3132k > > load: 0.00 cmd: dd 2254 [nfsaio] 225.94r 0.00u 0.13s 0% 3132k > > I've tried mount with different wsize option with same result. > > Any help would be greatly appreciated. > > -- > Andrew > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org" > -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-fs@freebsd.org Mon Oct 29 15:25:10 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C3D610DD41C; Mon, 29 Oct 2018 15:25:10 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670069.outbound.protection.outlook.com [40.107.67.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1F0B17562D; Mon, 29 Oct 2018 15:25:09 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB1292.CANPRD01.PROD.OUTLOOK.COM (52.132.45.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1273.19; Mon, 29 Oct 2018 15:25:08 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::a086:adbc:2b38:1018]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::a086:adbc:2b38:1018%2]) with mapi id 15.20.1273.027; Mon, 29 Oct 2018 15:25:08 +0000 From: Rick Macklem To: "Rodney W. Grimes" , Andrew Vylegzhanin CC: "freebsd-fs@freebsd.org" , "freebsd-infiniband@freebsd.org" Subject: Re: NFS + Infiniband problem Thread-Topic: NFS + Infiniband problem Thread-Index: AQHUbzV7uxRrUqx4EESM4oETGDUinKU2U4sAgAACL+E= Date: Mon, 29 Oct 2018 15:25:07 +0000 Message-ID: References: , <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> In-Reply-To: <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB1292; 6:RxpnYDQZaTkULafKJeSfxynRKRi18R4LdrtG6M8KbTGwcjwznVKtgLA6geazT7TB9GwHfNwisr64fF/rUv6Sp85988vOP4jF3OTbXAUrgjRMeHjpjVZX9j+F9w5kfahR+uRR6SpLdSa1hQv0KVFq5UooZgvsXA6seMebxqypW5l/eJVGDGayk0pOZpCjJHUixq7npEPs3bpivcAECd3MxkEimfflMYIAd4gOdjXPXRw0oFYaUq7zJZzf1JWA3jpljE9pA1YgHm4li56xA1q4+FKyWjsCRGXFgG33mGZXsoy4XfTi+mbtHxBbcWHPpFIrvg8gvcT6QHzuCqQW8npV9t8oK//AFQ774/ao/pzRdbKwhO0nlszG7kUyJ/fdI9F5PPvfMu6Hr/oM2jufZZZ+GZsL9glkGnOVEOG6l/g5Ah6hDXAFDN+CDuddiIOU5n9LIv+c0/unvrtqO1n34TTrVA==; 5:cKuvU5xfYmVdoWHNiVJIsrPj+cNk1mXdUb/qveOOQqr55yePm/CPr44yFvr098JHNqcdT21FugAdn4bNn8FAE23pEFMesxu96TdS2G4HnAK2/0XXyNLyQLQyej/mmAdZAXqsjC3vZUMv50hy4KoG6J7rT68wAEWo4mf7zyIahXg=; 7:oroQrwDBAm7lt+ay6FChmlcoLMH3+mcWf/4dQeWCEURpIigvZJ9X64fq8GNBBEUgVSSCw37RH4mopv2QqQOwdXSV4zslBrYhr29mmvqycR6nejpyxWQ0Nq1Owbk5rv0LThor+rNmj/hDdA2JbRbPIg== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: d17498b0-73db-4aea-b1e3-08d63db2b5a5 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(5600074)(711020)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB1292; x-ms-traffictypediagnostic: YTOPR0101MB1292: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(10201501046)(3002001)(93006095)(93001095)(3231382)(944501410)(52105095)(148016)(149066)(150057)(6041310)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123564045)(20161123560045)(20161123558120)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB1292; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB1292; x-forefront-prvs: 084080FC15 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(39860400002)(396003)(136003)(346002)(376002)(199004)(53754006)(189003)(6436002)(256004)(9686003)(25786009)(2906002)(55016002)(186003)(33656002)(4326008)(76176011)(5250100002)(39060400002)(786003)(316002)(86362001)(99286004)(7696005)(14444005)(478600001)(102836004)(105586002)(6506007)(74482002)(14454004)(106356001)(110136005)(54906003)(446003)(74316002)(6246003)(11346002)(486006)(476003)(5660300001)(71200400001)(71190400001)(8676002)(81156014)(53936002)(46003)(81166006)(8936002)(68736007)(2900100001)(229853002)(97736004)(305945005); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB1292; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: 9kllnFSps1+i/1WMcU5oJUHuXtIycfoHCJ2JUL1i8kIG4SaBywHJj3jlcKldarfC4It5UK0LT6fLPcoUiYmjKXDb9BXCjl2H6mcNpLpcqyZ11aaGZ0nVDmRuRLsgWhQOWE7Xbg6OT0DVtkuMzkVp0mEC1W+FkdbP6UjtAlLbbaI+ksojBBb2gKfIWI4XY7dIVT1Pc2QUwEmpJBtRuluZv0hX+AB05enHgoj+PfRDjYKhlgUE56wI868rsjzjBPrZMKrHaPyMm8DDOAw7pd8VeDyS7Ys25IbBl0D5oVdhGdN8IppN3i9ngmyT7wyttpunBHW7Uh9PJZdeptJEV70dyK1lfd1r+HNpighJvifF/vc= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: d17498b0-73db-4aea-b1e3-08d63db2b5a5 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Oct 2018 15:25:07.9886 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1292 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 15:25:10 -0000 Rodney W. Grimes wrote: Andrew Vylegzhanin wrote: >> Hello everyone, >> >> I have a several FreeBSD machines connected via Infiniband netwok ( FDR >> switch Mellanox SW3036 + ConnectX-3 VPI cards ). >> One of them is a NAS-server with multiply ZFS pools. >> >> All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried) on >> server) are with infiniband connected mode (option IPOIB_CM, option SDM) >> and world with OFED stack support. (WITH_OFED=3D'yes'). >> >> File transfers via FTP or SSH between server and clients works almost >> flawless ( ~ 12 Gbit/s ). >> >> But when I try to copy in/out some significant data via NFS share mounte= d >> on clients, NFS i/o hangs at all or got extremely slow (couple kB/s) >> transfer speed after uncertain amount of copied data. For example, on th= e >> one node I can copy 1GB file, and after NFS hang on file with size 30 kb= . >> >> Some details: >> [root@node4 ~]# mount_nfs -o wsize=3D30000 -o proto=3Dtcp 10.0.2.1:/zdat= a2 /mnt > ^^^^^^^^^^^^ >I am not sure what the interaction between page sizes, TSO needs, >buffer needs and all that are but I always use a power of 2 wsize >and rsize. They should always be a power of 2. I think the code clips the value, but i= t might only clip to a multiple of 512. If it didn't clip this down to 16384, then = that would definitely be a problem. Also, normally the same size for rsize and w= size is used. If you don't do that, you end up with weird sided blocks in the bu= ffer cache. I think it still works when this is done, but could cause performance hits. Probably doesn't matter for a simple performance test. (You can find out what options it is actually using by typing "nfsstat -m" = after doing the mount.) > You might try that. And as Rick suggested, turn of >TSO, if you can. Is infiniband using RDMA to do this, if so then >the page size stuff is probably very important, use multiples of >4096 only. RDMA is not supported by the FreeBSD NFS client. There is a way to use RDMA on a separate connection with NFSv4.1 or later, but I've never written code for that. (Not practical to try to implement without access to hardware tha= t does it.) rick [performance stuff snipped] From owner-freebsd-fs@freebsd.org Mon Oct 29 15:48:50 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7381E10DDAAA; Mon, 29 Oct 2018 15:48:50 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AEBE275FFA; Mon, 29 Oct 2018 15:48:49 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w9TFmltT057418; Mon, 29 Oct 2018 08:48:47 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w9TFmjaD057417; Mon, 29 Oct 2018 08:48:45 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201810291548.w9TFmjaD057417@pdx.rh.CN85.dnsmgr.net> Subject: Re: NFS + Infiniband problem In-Reply-To: To: Rick Macklem Date: Mon, 29 Oct 2018 08:48:45 -0700 (PDT) CC: Andrew Vylegzhanin , "freebsd-fs@freebsd.org" , "freebsd-infiniband@freebsd.org" X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 15:48:50 -0000 > Rodney W. Grimes wrote: > Andrew Vylegzhanin wrote: > >> Hello everyone, > >> > >> I have a several FreeBSD machines connected via Infiniband netwok ( FDR > >> switch Mellanox SW3036 + ConnectX-3 VPI cards ). > >> One of them is a NAS-server with multiply ZFS pools. > >> > >> All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried) on > >> server) are with infiniband connected mode (option IPOIB_CM, option SDM) > >> and world with OFED stack support. (WITH_OFED='yes'). > >> > >> File transfers via FTP or SSH between server and clients works almost > >> flawless ( ~ 12 Gbit/s ). > >> > >> But when I try to copy in/out some significant data via NFS share mounted > >> on clients, NFS i/o hangs at all or got extremely slow (couple kB/s) > >> transfer speed after uncertain amount of copied data. For example, on the > >> one node I can copy 1GB file, and after NFS hang on file with size 30 kb. > >> > >> Some details: > >> [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 /mnt > > ^^^^^^^^^^^^ > >I am not sure what the interaction between page sizes, TSO needs, > >buffer needs and all that are but I always use a power of 2 wsize > >and rsize. > They should always be a power of 2. I think the code clips the value, but it might > only clip to a multiple of 512. If it didn't clip this down to 16384, then that > would definitely be a problem. Also, normally the same size for rsize and wsize > is used. If you don't do that, you end up with weird sided blocks in the buffer cache. > I think it still works when this is done, but could cause performance hits. > Probably doesn't matter for a simple performance test. > (You can find out what options it is actually using by typing "nfsstat -m" after doing the mount.) > > > You might try that. And as Rick suggested, turn of > >TSO, if you can. Is infiniband using RDMA to do this, if so then > >the page size stuff is probably very important, use multiples of > >4096 only. > RDMA is not supported by the FreeBSD NFS client. There is a way to use RDMA > on a separate connection with NFSv4.1 or later, but I've never written code > for that. (Not practical to try to implement without access to hardware that > does it.) It would be very easy to arrange for a pair of PCIE 10G IB cards and a cable to go betweem them if that would be of use to you in some day doing some of this work, or for that mater for even playing with NFS over IB. > rick -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-fs@freebsd.org Mon Oct 29 15:56:08 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE15710DE2FD for ; Mon, 29 Oct 2018 15:56:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670050.outbound.protection.outlook.com [40.107.67.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7B778765BE for ; Mon, 29 Oct 2018 15:56:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB1145.CANPRD01.PROD.OUTLOOK.COM (52.132.50.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1273.21; Mon, 29 Oct 2018 15:56:02 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::a086:adbc:2b38:1018]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::a086:adbc:2b38:1018%2]) with mapi id 15.20.1273.027; Mon, 29 Oct 2018 15:56:02 +0000 From: Rick Macklem To: FreeBSD Filesystems Subject: How to fill in the fsid for file systems? Thread-Topic: How to fill in the fsid for file systems? Thread-Index: AQHUb557F1RNqdJl0kuwS+F5J/DjsA== Date: Mon, 29 Oct 2018 15:56:02 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB1145; 6:q3QBZ/oeSR+lC3J3OF5m8mtUD6UR/egl61Fz1AAXjlHAi1mEox81Xgn3FYSFmtxeAQvGpQ/Z0dDGhvZRSglof9uKeDv76f2KaF+urVTS/Bxyc2OWZfeM4VpwLnYd06Dp5gP2BHfv+GeNGeogzo3qvyUHSu7ObIfPZ8LAO/ocaKLcyc42tWAUVVQgH4HPUpuNAeJHJ0JRV6vh7zNHkEXgWLhfEAXa5Y2S/VBo5tItlgWb/702jrpDaw9pHaU05H7GeBvkSX24/DPxzsodb3UP9nN+rAwWFceX37BQ1BBH11dlwDFDGfeVd4DgBFFS3vtOBrE0+b9fVS1YCmkonLb/OFWzKf87c9G9xLVQytlmDXaEs09jqSBQX6oXrA3cjj6dV6YCYfjtJ5e88QozMIXMnrb77dJ/T9vCjXm23anX8HPkm5PVw1egQ1NrW7E0L+vgllunBPHbYS3sy5BxVPzueQ==; 5:C3NyiX8qxt3khF1mIC8jPVmE2wWvhzntzFZ794czp8DNJarsZfqS8eraz+5eJfpxutYkxohl9rpvr9rom9x6z7ZLv5TpVe2xE1nz2scRZyI3agxm//4AmJK9q2I/NYbFgvKa6Zjbo4nSm5ma8W6QCyR5JS28MulRSOkIAxO0BfQ=; 7:MItGkTrjmrb3gBkl2R3UFrho3QPCERe710FCW6Pb4yJ8KMmSA13jEELFa3/qkPFFHQSaDFaivJLbGAP+RB0KleowUGvAs/4vcZhHksxveSSbiYBzJ1/sG5+eOvRClgXeN6MHCbLtiQhfKM2qDKEfPA== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 33f7ef66-eb3e-4ef0-eecd-08d63db706ec x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB1145; x-ms-traffictypediagnostic: YTOPR0101MB1145: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(10201501046)(3002001)(93006095)(93001095)(3231382)(944501410)(52105095)(148016)(149066)(150057)(6041310)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123564045)(20161123560045)(20161123562045)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB1145; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB1145; x-forefront-prvs: 084080FC15 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(136003)(376002)(366004)(39860400002)(396003)(199004)(189003)(6916009)(71200400001)(71190400001)(105586002)(6506007)(5660300001)(186003)(74316002)(486006)(86362001)(97736004)(786003)(2906002)(14454004)(316002)(305945005)(14444005)(106356001)(25786009)(81166006)(81156014)(476003)(53936002)(102836004)(256004)(5250100002)(2900100001)(6436002)(7696005)(74482002)(55016002)(8676002)(9686003)(68736007)(33656002)(8936002)(478600001)(46003)(99286004); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB1145; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: NciE+ta5uilk++/COCUClpwd+hNlc+o65vjjTvZp8CDrJYqw3FPm0PPL/Nn+J5b37WLEHWJ8m4HxRlAfqUCxburfSB10wYhsLWEYLLyZ5AwxCQg76JMv2bO6qHa8vz8QeUOu9TDXwO3+fzWCdXvXvtL1obv4gwtfMRV0RFI/XLFT3DsUdU1KLPSCsTwdYbXraoYMg8xMnfPnIvvz1EnAb2KSeFKMknALRa05ZCyfebR4AwmfuNx6x8ZHTnueHD43wuoj3zwdyz/g4Lht1U3HZ8ow1yUaQ9PwRiAZoAlAzHiab6CoPrUkbZBqyKaqUB653ei9rh2fFheDkh1252QIDKlTXEUD0XJdJlcoHKLmbes= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 33f7ef66-eb3e-4ef0-eecd-08d63db706ec X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Oct 2018 15:56:02.3379 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1145 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 15:56:08 -0000 Hi, I have been working with Josh Paetzel on a patch to add support for the "fs= id=3DN" option (like what Linux has) to /etc/exports. (It is used to ensure that th= e fsid for a file system doesn't change when migrated to a different machine, so t= hat file handles don't change.) We have code that seems to work, but it is not obvious what should be fille= d in to f_fsid.val[1]? - cd9660 and msdosfs just set it to vfc_typenum - ZFS sets the low order 8bits to vfc_typenum and the high order 24 bits to= the high order bits of its "56bit objset unique ID" - UFS uses a value fs_id[1] in the superblock that appears to be filled in = with a random value at fs creation time by newnfs(8). It seems "fsid=3DN" does need to set f_fsid.val[1]. I can think of two poss= ibilities: 1 - Do what ZFS does and set the low order 8bits to vfc_typenum and the hig= h order 24bits from bits 32->55 of "N". or 2 - Just fill the 32bits in with the high order (32->63) bits of "N" and fo= rget about vfc_typenum. The only reason I can see for using vfc_typenum is to avoid collisions (sam= e fsid value) with fsids for mounts of other file system types. What do others think? rick From owner-freebsd-fs@freebsd.org Tue Oct 30 01:22:52 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0288510F174D for ; Tue, 30 Oct 2018 01:22:51 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5DEAA8F63B for ; Tue, 30 Oct 2018 01:22:51 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w9U1MeaO015879 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 30 Oct 2018 03:22:43 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w9U1MeaO015879 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w9U1MeP7015877; Tue, 30 Oct 2018 03:22:40 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 30 Oct 2018 03:22:40 +0200 From: Konstantin Belousov To: Rick Macklem Cc: FreeBSD Filesystems Subject: Re: How to fill in the fsid for file systems? Message-ID: <20181030012240.GM5335@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Oct 2018 01:22:52 -0000 On Mon, Oct 29, 2018 at 03:56:02PM +0000, Rick Macklem wrote: > Hi, > > I have been working with Josh Paetzel on a patch to add support for > the "fsid=N" option (like what Linux has) to /etc/exports. (It is used > to ensure that the fsid for a file system doesn't change when migrated > to a different machine, so that file handles don't change.) Why do you consider this an option for exports file and not for nmount(2) and fstab ? Do you intend to mangle fsid for mount protocol only ? From owner-freebsd-fs@freebsd.org Tue Oct 30 04:57:35 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3EF7010D4199; Tue, 30 Oct 2018 04:57:35 +0000 (UTC) (envelope-from avv314@gmail.com) Received: from mail-lj1-x233.google.com (mail-lj1-x233.google.com [IPv6:2a00:1450:4864:20::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 992F1723D6; Tue, 30 Oct 2018 04:57:34 +0000 (UTC) (envelope-from avv314@gmail.com) Received: by mail-lj1-x233.google.com with SMTP id z21-v6so10067513ljz.0; Mon, 29 Oct 2018 21:57:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=cdbpjA3mf+ego3EpdU6NFImyQyexxEKRIDIWRjFwTBY=; b=u7lGBwR80m5mLYZcJdoklg96XCV3mXABRD0VxpNcnnRxiglPdDblXA5T9i5FpOILLr SdI07AaUNQ7AIs9H+nD0QHu1N+OJfYybPg9kahiMB+HgudUPplv0sxdKF9Uj08bZpoj8 NKTmMDp3WGL0SSKbuMANXNMlkzTIkBFMo9t+F1P4i/wPpXPCjy9f7QwHe8PGW1UsPW6E PlykQpTK8ZD/NiDnF7wasK+qCCcKcaYI1CrcA6Q2IX5qje1bDdcueUWoLNjAbpihu0ql hvnGnPhzdMRKuNWbsVNjCqE2NAxs1GrcdEpm3AnVfkTpz1tKUI0JMRxxD2AmxhsVIv4d YP1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=cdbpjA3mf+ego3EpdU6NFImyQyexxEKRIDIWRjFwTBY=; b=LQOBsRC2fLOs0PSAdhJhEVOsCAAaM7BJMbNuzqaqH5QwNNz6M4tItlb/RWfQ61Jm4/ oD7IigRcRky21dA1NblNz3mPInDZ+F59772Hk+Ia0BVDT+xqGOaEaY4HXqfjwvs6nDFR xou8kKKGz4iwcPN+QMYXvi6XQnsFUqdM1wd5vZUX24GA0TPwSPPrTMPW0lFFUj3Ca3nG jEI6Af4p5hJ41V3fxi6FD6v0BjoVksmbHsau2wJX3snfwgZU7VFpV0seoGygHKzESE/o +8RFE7LXZuDUMp0uhtvAz9ppJRj3g8RVi771hCL5Kh2vFP8BcAaHF8XbYToqbYID1YyM Zgtg== X-Gm-Message-State: AGRZ1gIza7IjnxHbov+gHk1Td1RpVF+LkVGvnbdtrxqqZQYZQiVSt+T3 kSWnZqj+JZWc3d5ommluKHRptjVcYLAVPA83gfE0ea2V X-Google-Smtp-Source: AJdET5dtuDJuv0TEkMpcPkfAcnCjawYCi0lJiOK1UccPXTEVkMHbAUu0pKxvp+HpcS6Tgi7DKqj42LjY/l1cQXvvSBY= X-Received: by 2002:a2e:478f:: with SMTP id u137-v6mr395322lja.142.1540875452870; Mon, 29 Oct 2018 21:57:32 -0700 (PDT) MIME-Version: 1.0 References: <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> In-Reply-To: From: Andrew Vylegzhanin Date: Tue, 30 Oct 2018 07:57:21 +0300 Message-ID: Subject: Re: NFS + Infiniband problem To: rmacklem@uoguelph.ca Cc: "Rodney W. Grimes" , freebsd-fs@freebsd.org, freebsd-infiniband@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Oct 2018 04:57:35 -0000 > >> Some details: > >> [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 /mnt > > ^^^^^^^^^^^^ > >I am not sure what the interaction between page sizes, TSO needs, > >buffer needs and all that are but I always use a power of 2 wsize > >and rsize. Again after some tests. I've tried 4096,8192,16384 wsize/rsize. Only 4096 value give some measurable result and it's extremely slow ~ 10-16MB/s for writing (depend on number of threads), 10-12 MB/s for reading. With other values NFS hangs (or almost hangs - couple kB/s in average) Changing sysctl net.inet.tcp.tso=0 (w/o reboot) on both sides had no effect. AFAIK, infiniband interface has no option for TSO,LRO: ib0: flags=8043 metric 0 mtu 65520 options=80018 lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.50.df.51 inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255 nd6 options=29 BTW, hers is my sysctl.conf file with optimisation for congestion control and tcp buffers on 10/40 Gbit/s links (the server had 40 Gbit/s Intel ixl ethernet also): kern.ipc.maxsockbuf=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_inc=524288 net.inet.tcp.cc.algorithm=htcp > > You might try that. And as Rick suggested, turn of > >TSO, if you can. Is infiniband using RDMA to do this, if so then > >the page size stuff is probably very important, use multiples of > >4096 only. > RDMA is not supported by the FreeBSD NFS client. There is a way to use RDMA > on a separate connection with NFSv4.1 or later, but I've never written code > for that. (Not practical to try to implement without access to hardware that > does it.) > Hope I could help with testing. -- Andrew > rick > [performance stuff snipped] From owner-freebsd-fs@freebsd.org Tue Oct 30 08:18:11 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E579F10DA642 for ; Tue, 30 Oct 2018 08:18:10 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f171.google.com (mail-lj1-f171.google.com [209.85.208.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DEA8772BB for ; Tue, 30 Oct 2018 08:18:10 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f171.google.com with SMTP id k19-v6so1925071lji.11 for ; Tue, 30 Oct 2018 01:18:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=2UNdu+RrxvRP1bG1YCrKwH21abA6jf8GLMqAw4mg+q0=; b=ttUOCFVu6gB6KNHKi+9f28j09a/gVuqjS3XyxGV8kIdMX8prSVfla6/uaPVTrCZYQn 7/JrfDAVLQOl+NxIO5VmlOggpOpnm3nyiFkFjXJQir9mYq4+uIdCWm4kwAufoK6eXXQ+ bDj587LJ7rwKKzcxU7i2UyC/HztbXULY7cBdmuy7SjiBUBuMNojG586aKH0ccjWqfTKB O1cs7uwuD4rYVwF32BrmZ0uL8ztIDXvhzVC9gFWIj1MWcoV9IySFtpUMO1q+fTaaF9Rw 0pEM1ovw/I6+9ID6FiGpN1KfAYQLWShVtE7bDGJh89mGGd0ONHSS+tT4gb2rG3RtCQAK oZbA== X-Gm-Message-State: AGRZ1gK5NFjhWSWcZbdHl4YU7YjOJgOpFiHn6ntmhw6CPUWli4Za38Sn yVNZeKojxsOIp32smDniNcGWOfuZ X-Google-Smtp-Source: AJdET5d4sXUtgvFBRrYGfRJceRlZUYUQ0t8KV4emSWFquMp4mGabEF+WSLHcuLV+BRWixUU+w1lhNw== X-Received: by 2002:a2e:9c59:: with SMTP id t25-v6mr11405933ljj.107.1540887482814; Tue, 30 Oct 2018 01:18:02 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id h12-v6sm155089ljc.90.2018.10.30.01.18.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 30 Oct 2018 01:18:02 -0700 (PDT) Subject: Re: How to fill in the fsid for file systems? To: Rick Macklem , FreeBSD Filesystems References: From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Tue, 30 Oct 2018 10:18:00 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Oct 2018 08:18:11 -0000 On 29/10/2018 17:56, Rick Macklem wrote: > It seems "fsid=N" does need to set f_fsid.val[1]. I can think of two possibilities: > 1 - Do what ZFS does and set the low order 8bits to vfc_typenum and the high > order 24bits from bits 32->55 of "N". > or > 2 - Just fill the 32bits in with the high order (32->63) bits of "N" and forget about > vfc_typenum. > The only reason I can see for using vfc_typenum is to avoid collisions (same fsid value) > with fsids for mounts of other file system types. I have come up with an option #3 :-) Fill f_fsid.val[1] with some constant magic value that signals that val[0] is manually set. Some possible magic values: 0x0, 0xffffffff. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Wed Oct 31 02:23:32 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 28098107418E for ; Wed, 31 Oct 2018 02:23:32 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670063.outbound.protection.outlook.com [40.107.67.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BCEA4860BE; Wed, 31 Oct 2018 02:23:31 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB1004.CANPRD01.PROD.OUTLOOK.COM (52.132.47.154) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1273.25; Wed, 31 Oct 2018 02:23:30 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b%3]) with mapi id 15.20.1294.021; Wed, 31 Oct 2018 02:23:29 +0000 From: Rick Macklem To: Konstantin Belousov CC: FreeBSD Filesystems , Josh Paetzel , Andriy Gapon Subject: Re: How to fill in the fsid for file systems? Thread-Topic: How to fill in the fsid for file systems? Thread-Index: AQHUb557F1RNqdJl0kuwS+F5J/DjsKU2/twAgAGdwkI= Date: Wed, 31 Oct 2018 02:23:29 +0000 Message-ID: References: , <20181030012240.GM5335@kib.kiev.ua> In-Reply-To: <20181030012240.GM5335@kib.kiev.ua> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB1004; 6:uyE/UoXnm8xAHgdMUXqOdgmmFgj31cvP5zXEEEOszKUuSRO0ADRmwTnkOBxCqIrygmmtiePBlbe1gxvou/WupIqf0wsQcGdEdpHZz3m8kePeBs6SHXzE8goGMZz+I2UFsiMQ0nWCaLA8BmeV0Ays8HbJfmV6hdJK6hXWWcZZuH7TNs5L8q7o4HUFu6WmF264/ST9XR55xtI83IHIDeN5k7efTRMWNmPkOCdEOHb8fYSZeWZhoB7C7Nn+fpUAcn2Ri2zCpiyHXzb8/1nXhtK4dxyhWN+4xIFY3rPGWGiIz1ZnmDXUgFGLX4ptaNd5QjnFbDeRp+3wvM+TwE0qpEvX0a/ZeK6iZGiqG+kUihcYcCXV8IbnTjWo23bGp3pULoL1fXoEF72w5OagjPCGvRFdsJSdKX/Fxl4Xi5OTyf9+TqAZJYxyNkUEeUV+I9qIzm2RmT5xKEfcet+T+zF0klxn8g==; 5:tWd5rXZpVL1JgPGDw4AkRBhUgKh4raYTaaemHUezAlV5YBuq/y9/7UloOt5S53nwnIChjbkrZMM3jyv2eb+22cAu13y6u4oWjgJ3qPEyIG5QkR6KBOENo6Rzd2KG3u6wYLsWoegZyt5eNSJJvlROqeqjTDqQdrMA4fNIJDmwdoA=; 7:/+O7GgLKEzWergTfjpHLd4IYIuAXnlFzdw2WzIwMkMI4gnt0JqmrEy5LMTzB1M8qjL9CD83QY/6E8Y6RsdLblB+14Z2q/a0VIYGbujq/Bndcr6y3oixKSITPo2618WKLKe54Bu1X+Io/AHkBjlJ3WA== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: ea4fd4d8-d3f8-490e-f472-08d63ed7d90b x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB1004; x-ms-traffictypediagnostic: YTOPR0101MB1004: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(3002001)(10201501046)(93006095)(93001095)(3231382)(944501410)(52105095)(148016)(149066)(150057)(6041310)(20161123560045)(20161123564045)(20161123562045)(20161123558120)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB1004; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB1004; x-forefront-prvs: 084285FC5C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(346002)(396003)(136003)(376002)(39860400002)(199004)(189003)(305945005)(74316002)(81156014)(81166006)(8676002)(39060400002)(25786009)(486006)(2900100001)(11346002)(4326008)(476003)(106356001)(5660300001)(446003)(14444005)(256004)(229853002)(54906003)(316002)(86362001)(786003)(68736007)(6916009)(478600001)(33656002)(71200400001)(53936002)(76176011)(74482002)(9686003)(5250100002)(14454004)(2906002)(6246003)(6506007)(71190400001)(55016002)(99286004)(1411001)(46003)(8936002)(105586002)(7696005)(102836004)(6436002)(97736004)(186003); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB1004; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: 73Pgk4cLY9E+M5vXxfkQfMoxGFx6dyf9PpN80PiF/+cDWUW7lEuYx15wXcyIb+ZTsSK2KmJKJeQabIi8iSED1QTSOp4vyMsuK3FSep+jpI/uG8Rj1dT3tPzUV91K6J56qugZnVZRHyFtDoBJuHbODntNaMOLfps4N/oF2LyR4y/7QTt0vM5GSGP/Eg039Q81PshOcClstK0RshpdfNHWH+xHAKD6VJYUchWNwBBK21x6rcQTNI1BCRSSXrFUcOiYiluIswzc8K2xB4wLGMXdLCQroXLNnrDjOzZeQBHsyjEUovWdexeaZH5cecyBUoZMbUMwU74BoHDRqOMsmyQ/9jy5gfIvejnoBQT6VtKqV5I= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: ea4fd4d8-d3f8-490e-f472-08d63ed7d90b X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Oct 2018 02:23:29.8740 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1004 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2018 02:23:32 -0000 Konstantin Belousov wrote: >On Mon, Oct 29, 2018 at 03:56:02PM +0000, Rick Macklem wrote: >> Hi, >> >> I have been working with Josh Paetzel on a patch to add support for >> the "fsid=3DN" option (like what Linux has) to /etc/exports. (It is used >> to ensure that the fsid for a file system doesn't change when migrated >> to a different machine, so that file handles don't change.) >Why do you consider this an option for exports file and not for nmount(2) >and fstab ? Do you intend to mangle fsid for mount protocol only ? Well, Josh Paetzel proposed this. I believe it was because "that's how Linu= x does it" (but I can't speak for Josh). The kernel code currently sets this via an nmount(2) option called "export.= fsid". Although Josh's patch has this done in mountd.c, it could simply be used as a mount option in /etc/fstab or manual mounting. (The code currently proces= ses this option in vfs_domount_update(), so it would need to be moved to where options are processed during mounting, but that wouldn't be hard.) This would actually avoid patching mountd.c. It would also avoid the fact t= hat the "export fsid option" applies to all clients and is not per-host or per-= net. (Doing it only for "exports" requires a moderate amount of code change, suc= h as a new function to be used instead of vfs_busyfs() to busy the file syste= m after looking up the fsid in the export stuff. I haven't coded this, so I'm= not sure how much work that would be?) I actually think making it a generic nmount(2) option and not an export spe= cific one sounds like a good idea. What do others think of making "fsid=3DN" a generic mount option for overriding the selection of fsid done by the file system? rick ps: Although I don't try to be "Linux incompatible", I don't see why we sho= uld be compatible when doing it a slightly different way seems more logic= al. From owner-freebsd-fs@freebsd.org Wed Oct 31 02:53:51 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 680F31074DAA; Wed, 31 Oct 2018 02:53:51 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670065.outbound.protection.outlook.com [40.107.67.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 07CAD8714D; Wed, 31 Oct 2018 02:53:50 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB2092.CANPRD01.PROD.OUTLOOK.COM (52.132.46.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1294.20; Wed, 31 Oct 2018 02:53:49 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b%3]) with mapi id 15.20.1294.021; Wed, 31 Oct 2018 02:53:49 +0000 From: Rick Macklem To: Andrew Vylegzhanin CC: "Rodney W. Grimes" , "freebsd-fs@freebsd.org" , "freebsd-infiniband@freebsd.org" Subject: Re: NFS + Infiniband problem Thread-Topic: NFS + Infiniband problem Thread-Index: AQHUbzV7uxRrUqx4EESM4oETGDUinKU2U4sAgAACL+GAAOXvgIABagv/ Date: Wed, 31 Oct 2018 02:53:49 +0000 Message-ID: References: <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB2092; 6:yeTLYMXkIIjIk/LnVtMCbg0A9ibdqPAbmt55FH9lhCiSZBNU2RVko/odXF9gzdjNTtZNjNlRg/HlxjRZoJivF89eiSqUhsQNShSYc5M0ti/4FoPnKwvK6+lEWFBT3yzFmWKmLGbCaeustMQ88IvR+Ahqxe6aHsi3B9N1RCSTQ+iyFLFyRDtYlT/eg4YT3m0rhISozv8fGbA492FAmfoPQfyGz07/VDIKNb4ebm5o8GrSvxwqKt9tpL2T2SvPAtNANUfSCAvg95/o8r9UejDvk+yf8+7izSl/rIyjWbnUVEZ8yn3g4MYMwIBcVrCk9ZIKSjt2vOoovhLaZyAFZP8gyKK7vEi/pU47dfFQc0eip75mLA+iGt1nLx+ExKHko5qdiobun8G6zL/TUqr6RxMr1YGxTqofnpUqiHzc9zDz0whGCp6e0G1bl7xFohgo65g/qcRnoSc6W/XdsWOQNf/IDQ==; 5:qdUaEUvVU2NEtkt9O6Sgu4+J13lf79Ofbkw8y27OrTPwR0X5n5k4Su9s2cVdqRgnX2imTlD0Zy3oz+q0vxQl8xMVMlMlm95OyLpxeBRJwKNQxN0LVx9yxiLvNP/DN/UEz4vvbTSTzlO26w6augt8Ql08/pzPM1tcvRKFZvooBq4=; 7:TgzUUP7tBG3vurSey9tk1gV7pOKVcIb9cjJ2H+CgZW6dk3azzGqTwxh71/JYmDrBzj8ZwHf/JC28od66HJqadymWksNR/ofPoKU3hyfwy9gSTotH4k/1Oz4rSnagXV3aXmnM5X/fuF4Mfxeo5aKSrQ== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 9cc5da94-a157-46ef-f0e9-08d63edc1595 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB2092; x-ms-traffictypediagnostic: YTOPR0101MB2092: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(3231382)(944501410)(52105095)(3002001)(10201501046)(93006095)(93001095)(148016)(149066)(150057)(6041310)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123560045)(20161123564045)(20161123558120)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB2092; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB2092; x-forefront-prvs: 084285FC5C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(136003)(396003)(366004)(376002)(346002)(189003)(199004)(11346002)(2906002)(316002)(71190400001)(71200400001)(6916009)(54906003)(786003)(68736007)(1411001)(6506007)(76176011)(33656002)(102836004)(93886005)(14454004)(81166006)(81156014)(105586002)(8936002)(7696005)(25786009)(8676002)(6436002)(53936002)(6246003)(229853002)(106356001)(2900100001)(55016002)(5660300001)(99286004)(256004)(9686003)(478600001)(74482002)(97736004)(476003)(46003)(86362001)(74316002)(305945005)(446003)(186003)(4326008)(486006)(39060400002)(5250100002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB2092; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: 6mNL7yOraiA6rHG7oMhNhIaTgdm0cK3BjuBIJQgt9EQmyqcEE1Kg1p8TPcu8+FJz/GE7Cdx85c0psADAnwAL0yhR1eXJ/SYQQWYfX+HfD/DGM7Zgn4xYELQDdPZwOoJqq2+mUUEf2EpVpAwE1jtR27MbmMtQbdGjcxx3SNnVQOZDSVabAzG3HG9zYBvNZQe5DTO2MxOf/mT2TiuH5+cbwQLGWT9xxIR4hrAel38SLrMGM5BjiV6vjwXUTApuhHemV+sAVpmIeONPb+FtwiQ+GfvHjHHyR60VLFHb16vIaAiQFiqOkddvNA+ExnR+Nt5Ct3sRHzJQiMZVyJEsjzpZtQ2PPhbcTU1b9/nzWxBzA8I= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 9cc5da94-a157-46ef-f0e9-08d63edc1595 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Oct 2018 02:53:49.5066 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB2092 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2018 02:53:51 -0000 Andrew Vylegzhanin wrote: >> >> Some details: >> >> [root@node4 ~]# mount_nfs -o wsize=3D30000 -o proto=3Dtcp 10.0.2.1:/z= data2 /mnt >> > ^^^^^^^^^^^^ > > >Again after some tests. >I've tried 4096,8192,16384 wsize/rsize. Only 4096 value give some measurab= le >result and it's extremely slow ~ 10-16MB/s for writing (depend on numbe= r of >threads), 10-12 MB/s for reading. With other values NFS hangs (or alm= ost hangs - >couple kB/s in average) > >Changing sysctl net.inet.tcp.tso=3D0 (w/o reboot) on both sides had no ef= fect. > >AFAIK, infiniband interface has no option for TSO,LRO: >ib0: flags=3D8043 metric 0 mtu 65520 > options=3D80018 > lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.50.df.51 > inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255 > nd6 options=3D29 > > >BTW, hers is my sysctl.conf file with optimisation for congestion control = and tcp >buffers on 10/40 Gbit/s links (the server had 40 Gbit/s Intel ixl = ethernet also): > >kern.ipc.maxsockbuf=3D16777216 > >net.inet.tcp.sendbuf_max=3D16777216 > >net.inet.tcp.recvbuf_max=3D16777216 > >net.inet.tcp.sendbuf_auto=3D1 > >net.inet.tcp.recvbuf_auto=3D1 > >net.inet.tcp.sendbuf_inc=3D16384 > >net.inet.tcp.recvbuf_inc=3D524288 > >net.inet.tcp.cc.algorithm=3Dhtcp Well, I'm not familiar with the current TCP stack (and, as noted before, I = know nothing about InfiniBand). All I can suggest is testing with the default co= ngestion control algorithm. (I always test with the default, which appears to be new= reno.) NFS traffic looks very different than a typical use of TCP. Lots of small T= CP segments in both directions interspersed with some larger ones (the write requests or read replies). rick From owner-freebsd-fs@freebsd.org Wed Oct 31 08:02:10 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE4BE107CEC3 for ; Wed, 31 Oct 2018 08:02:09 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 582496A4AE for ; Wed, 31 Oct 2018 08:02:09 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f196.google.com with SMTP id a28-v6so10911788ljd.6 for ; Wed, 31 Oct 2018 01:02:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=m1rK9cn8575AbZgzGe8P3MTm6WkUbt53k4uQFfXnoDs=; b=hpKK3Tzz1p6Btegxyxyz1PnKHGrQK4mbCYk9YgGvVXO6jybVEV76XEfKsAZdQACJoY eYZ0mdcm5h6wreC0Mm/NUvQUMVoc6du/erXTZ0uOS6BxR1qFGg6qpYhNq/YTtWC1+pqn kr6UCL7IhCVz7C11+Pr3z50digt4GJPmhyUpSXX4W870QA8uvIm2U+bdJOImKSd8B/DJ Vsw6wU71Ylj6LdKLKKiO1QbA68Y6RLe12cR/f0NbOoHfSzqahQCoquGKCyH/iXWEVAMr sRFyYINAZtnVevCQx7+/ZFailUbKa/N1DvfXO4pF1n0b7NCN1ouoHoUbVfAiT8Lc+rdp v4IA== X-Gm-Message-State: AGRZ1gJoV333wwlopCb8QODRCuZiYzwZqVT4wwvaNEcDO4qZDcfZpkIO f1qvQEsKtLjdja9+U0ISpwM= X-Google-Smtp-Source: AJdET5cBZZaVMQekq5kBZdGqoqfKtX+FwsVHECx5J+nYpDCPZFXWTxAodCr0uHCQmZq1rkzzm3h7XQ== X-Received: by 2002:a2e:55d3:: with SMTP id g80-v6mr1442355lje.78.1540972927510; Wed, 31 Oct 2018 01:02:07 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id y9-v6sm3933605ljk.35.2018.10.31.01.02.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 Oct 2018 01:02:06 -0700 (PDT) Subject: Re: How to fill in the fsid for file systems? To: Rick Macklem , Konstantin Belousov Cc: FreeBSD Filesystems , Josh Paetzel References: <20181030012240.GM5335@kib.kiev.ua> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Wed, 31 Oct 2018 10:02:05 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2018 08:02:10 -0000 On 31/10/2018 04:23, Rick Macklem wrote: > Konstantin Belousov wrote: >> Why do you consider this an option for exports file and not for nmount(2) >> and fstab ? Do you intend to mangle fsid for mount protocol only ? > Well, Josh Paetzel proposed this. I believe it was because "that's how Linux > does it" (but I can't speak for Josh). > > The kernel code currently sets this via an nmount(2) option called "export.fsid". > Although Josh's patch has this done in mountd.c, it could simply be used as > a mount option in /etc/fstab or manual mounting. (The code currently processes > this option in vfs_domount_update(), so it would need to be moved to where > options are processed during mounting, but that wouldn't be hard.) > > This would actually avoid patching mountd.c. It would also avoid the fact that > the "export fsid option" applies to all clients and is not per-host or per-net. > (Doing it only for "exports" requires a moderate amount of code change, such > as a new function to be used instead of vfs_busyfs() to busy the file system > after looking up the fsid in the export stuff. I haven't coded this, so I'm not > sure how much work that would be?) > > I actually think making it a generic nmount(2) option and not an export specific > one sounds like a good idea. > > What do others think of making "fsid=N" a generic mount option for > overriding the selection of fsid done by the file system? > > rick > ps: Although I don't try to be "Linux incompatible", I don't see why we should > be compatible when doing it a slightly different way seems more logical. I see two issues here. One is practical. How do we provide fsid to ZFS filesystems? I mean I would hate to resort to mounting ZFS via fstab just to provide fsid whereas today ZFS filesystems are mounted auto-magically. We could add a FreeBSD specific fsid ZFS property, but that's also some extra code. The other issue is a potential design issue. Right now there is the "one true" fsid that's embedded into struct mount (mnt_stat.f_fsid) and NFS uses it to match a handle to a specific filesystem. Obviously, to support the fsid override we have to modify the one true fsid. My thinking is why can't NFS have its own fsid (equal to the one true fsid by default) that it would use internally to match the handle to an exported filesystem... In that case we would not need to alter the one true fsid (in struct mount), we would just set the NFS export fsid. In that case, I think, it would be natural that the fsid override is provided as an export option. This, of course, would require a new implementation for nfsvno_getfh() to use the NFS fsid instead of vp->v_mount->mnt_stat.f_fsid. Also, nfsd_fhtovp() would not be able to use vfs_busyfs(), instead it would do an internal look-up. Same for nfsvno_getvp(). [I think that nfsrvd_compound() would keep using the one true fsid]. But maybe this is a too radical idea? -- Andriy Gapon From owner-freebsd-fs@freebsd.org Wed Oct 31 15:50:34 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A36E310DD944 for ; Wed, 31 Oct 2018 15:50:34 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670063.outbound.protection.outlook.com [40.107.67.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 470997BD3C; Wed, 31 Oct 2018 15:50:34 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB0842.CANPRD01.PROD.OUTLOOK.COM (52.132.48.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1273.24; Wed, 31 Oct 2018 15:50:32 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b%3]) with mapi id 15.20.1294.021; Wed, 31 Oct 2018 15:50:32 +0000 From: Rick Macklem To: Andriy Gapon , Konstantin Belousov CC: FreeBSD Filesystems , Josh Paetzel Subject: Re: How to fill in the fsid for file systems? Thread-Topic: How to fill in the fsid for file systems? Thread-Index: AQHUb557F1RNqdJl0kuwS+F5J/DjsKU2/twAgAGdwkKAAGQrgIAAehmG Date: Wed, 31 Oct 2018 15:50:32 +0000 Message-ID: References: <20181030012240.GM5335@kib.kiev.ua> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB0842; 6:MI43b6h+wddc4RfVd5w7w/TAFq/Y21AA3TFOg+4XO41HC/FZlq+HBC4f1OlCQjUgqn1w6o4uMP4/5Rn1NXzlDD9rMKsuFFH9J5qa2npELD8CwCRN6w2Td3e+HwyZhLCwVkCwkvhFvfH2fn6UYTAOC8FuZYwW/SsncDkiR65QssD/h1yI8sVlF8tIlDZQHXm9qvWsUbDI5+cCnLKa4LaHyAZ5ja8Ce93u5Yc5yJ3UYnrC46Ns2dcKnAf4E6Jp0ldCMdcavCzFBKmtIxZiWPo7AFdZtl9xGikDdxOaVBJuxdj18P+9O/4zwoMLzU6g3OPVnr9lRtoW76kA/xDQF928ptLBulum9MIc8cIxaRM1JSlR2gMP3DFHIw7UwbvOLAPFROXIFiHOxTTyKIqMFT+n+jvIKHeoJVg4LGTMNAxB7oRtS0TXNOL0y1GptWMmnwZf7Ge3Y/xY6XiW3J72mWI17A==; 5:fddT1os1ok3MCyr+XUTj8+HgYiOrLGCYe8BlItY4hi8V/gas+aVsrbHvwvg4r6Be7tAwsH08QSbt2pOA0dGiVihodDg+ENSJSI9lsDwTQ8di+ka9PHZ9MTT8YNR7CliqVCMbQmn2xAdvpuD9DWqzaO+XIOflrUcbtikM71yq9nw=; 7:+NDaM1gEY2iNnGumrQbheYd3DEY+NzkLM2tE+Smrce+FkXHyqUPvgIVmhz+zwpVyGPAMYgf7NnnnyOdnLye9ayJZLx2iRThYu351JiJDi5/YyTrf4H0+fvza8rE1kWUVA9GrR4c4bUY6N8UMvmqV3w== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: bcb208e9-1fba-43c8-50bd-08d63f4896f9 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB0842; x-ms-traffictypediagnostic: YTOPR0101MB0842: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(10201501046)(3002001)(93006095)(93001095)(3231382)(944501410)(52105095)(148016)(149066)(150057)(6041310)(20161123564045)(20161123562045)(20161123558120)(20161123560045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB0842; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB0842; x-forefront-prvs: 084285FC5C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(366004)(396003)(39860400002)(346002)(376002)(51444003)(189003)(199004)(6436002)(74482002)(14444005)(55016002)(74316002)(229853002)(305945005)(5250100002)(5660300001)(256004)(110136005)(316002)(54906003)(786003)(2900100001)(93886005)(186003)(478600001)(97736004)(476003)(106356001)(446003)(11346002)(105586002)(14454004)(486006)(46003)(76176011)(6506007)(102836004)(4326008)(71200400001)(68736007)(86362001)(99286004)(33656002)(6246003)(8676002)(81156014)(81166006)(39060400002)(25786009)(8936002)(9686003)(2906002)(71190400001)(53936002)(7696005); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB0842; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: xCb/vuvN8YM9TWdn1IEBy5rVE8Zyl/12QqL/rETUpwTgsarVeINTm5KgfILvr8Rw9+7rJqDVqgIITLxn6lX4Xo7jLccRn3ba9mDMiSLO0AMUCAZgIho65fpZsr7R9bNATuGsuO79VyoRprzapJ/CrcjSJHu/KAQ8DL7vhT39gBNyMiHZI+GVTLHKfRTQRngjSJPGbjeCYbAVnngsZhKWvnv9HRazBZ5/86lelRoJgcOW0iP9pPQfapLcPJoiwI6YPsK6mAzeXRGkmXkNWgeWYJ+7V8Hyhjd+h3kO0i2pj0eBrw3sj5BPcwsBfKj9TzSPXUQkjOu4WdVi6veReb1xi2eRW0su5GpCZTfvHgRJAVI= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: bcb208e9-1fba-43c8-50bd-08d63f4896f9 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Oct 2018 15:50:32.1839 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB0842 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2018 15:50:35 -0000 Andriy Gapon wrote: >On 31/10/2018 04:23, Rick Macklem wrote: >> Konstantin Belousov wrote: >>> Why do you consider this an option for exports file and not for nmount(= 2) >>> and fstab ? Do you intend to mangle fsid for mount protocol only ? >> Well, Josh Paetzel proposed this. I believe it was because "that's how L= inux >> does it" (but I can't speak for Josh). >> >> The kernel code currently sets this via an nmount(2) option called "expo= rt.fsid". >> Although Josh's patch has this done in mountd.c, it could simply be used= as >> a mount option in /etc/fstab or manual mounting. (The code currently pro= cesses >> this option in vfs_domount_update(), so it would need to be moved to whe= re >> options are processed during mounting, but that wouldn't be hard.) >> >> This would actually avoid patching mountd.c. It would also avoid the fac= t that >> the "export fsid option" applies to all clients and is not per-host or p= er-net. >> (Doing it only for "exports" requires a moderate amount of code change, = such >> as a new function to be used instead of vfs_busyfs() to busy the file sy= stem >> after looking up the fsid in the export stuff. I haven't coded this, so = I'm not >> sure how much work that would be?) >> >> I actually think making it a generic nmount(2) option and not an export = specific >> one sounds like a good idea. >> >> What do others think of making "fsid=3DN" a generic mount option for >> overriding the selection of fsid done by the file system? >> >> rick >> ps: Although I don't try to be "Linux incompatible", I don't see why we = should >> be compatible when doing it a slightly different way seems more lo= gical. > >I see two issues here. > >One is practical. How do we provide fsid to ZFS filesystems? >I mean I would hate to resort to mounting ZFS via fstab just to provide fs= id >whereas today ZFS filesystems are mounted auto-magically. >We could add a FreeBSD specific fsid ZFS property, but that's also some ex= tra code. Good point. I'm not a ZFS guy, so I wouldn't have thought of this. >The other issue is a potential design issue. >Right now there is the "one true" fsid that's embedded into struct mount >(mnt_stat.f_fsid) and NFS uses it to match a handle to a specific filesyst= em. >Obviously, to support the fsid override we have to modify the one true fsi= d. >My thinking is why can't NFS have its own fsid (equal to the one true fsid= by >default) that it would use internally to match the handle to an exported >filesystem... >In that case we would not need to alter the one true fsid (in struct mount= ), we >would just set the NFS export fsid. In that case, I think, it would be na= tural >that the fsid override is provided as an export option. >This, of course, would require a new implementation for nfsvno_getfh() to = use >the NFS fsid instead of vp->v_mount->mnt_stat.f_fsid. Also, nfsd_fhtovp()= would >not be able to use vfs_busyfs(), instead it would do an internal look-up. = Same >for nfsvno_getvp(). >[I think that nfsrvd_compound() would keep using the one true fsid]. If NFS starts to use a different fsid, it would need to change everywhere. >But maybe this is a too radical idea? I alluded to this option in my last post. I think both fsids will need to b= e in the mount structure, since finding the correct mount point via the fsid is the = first step in translating a file handle to a vnode. (After that VFS_FHTOVP() does= the rest.) And I think it will get messy. A couple of examples. There are some syscalls that use file handles. fhopen(2), fhstat(2), fhstat= fs(2), getfh(2), lgetfh(2) I had assumed that the "file handles" used by these should be the same file handles that NFS uses (ie. file handles are a generic VFS component and not= NFS specific) but I can see the argument either way. I actually don't know what apps/utilities use fhopen/fhstat/fhstatfs, but getfh(2) is used by mountd. Since mountd uses getfh(2) to get a file handle for NFS mounts, it needs to return the NFS fsid to keep the old binaries working, even if you add a new getnfsfh(2) function for mountd to use. - Now you have some file handle system calls using file handles with the NF= S fsid in them and some using file handles with the "true" fsid in them. (Sounds confusing to me.) Since the first step in turning a file handle into a vnode is looking up th= e fsid in the mount list, if you had two fsids, I think they both would end up in = the mount structure so that lookup could be done easily. This lookup is normally done by vfs_busyfs() { that appears to be the only = use of vfs_busyfs() in sys/kern. I haven't looked in the various file systems }= . With two fsids, you need two functions and need to be careful which one you= use. rick From owner-freebsd-fs@freebsd.org Wed Oct 31 20:02:39 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A88B10E848B; Wed, 31 Oct 2018 20:02:39 +0000 (UTC) (envelope-from avv314@gmail.com) Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DCE5E86C15; Wed, 31 Oct 2018 20:02:38 +0000 (UTC) (envelope-from avv314@gmail.com) Received: by mail-lf1-x133.google.com with SMTP id p17so6164919lfh.4; Wed, 31 Oct 2018 13:02:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=m8Rgzg4293v3MVlEqea2Z8/+X11CqprUEMKGGxpMf7M=; b=rXmPAuR4J2WinO4QN0stdQXBuu2dw6s30eyTwy1rfW/s0AslJk3T/9PJWzjltTV/us eD21hKF6ddUsYfQJxPc4e2PjOoIuzzewSZKe1GRToUrtj5jqNKzE2/6NNvOJxfxTppsN 1J6QPbKPxuLMiT2r8moT20qfCsiwGx9OjhRaGsomhrpFWqdXcIXYbEuKylyPiT/8pkQh eE5CNS0DuXRKZ2b43bXr5y+GjYtnCo7HtndfTGeIUlxcyqhRePmp/z5AS+H/FKKz/lGD Uyi7MzMPZqfdlzJVbYuR03Szvo2aikkbQuuEvQE25s89pilOVSe2SpzKK34GgIyn6fjV qVaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=m8Rgzg4293v3MVlEqea2Z8/+X11CqprUEMKGGxpMf7M=; b=PZHjdVA1FRZDKFyZS+GU9Cbco3DwHK7VvEgcB18HDcwANbVionT85CadCm5yusc92n i7ZYB52VusgPSYEi46jMbHL1+k5zobc8zMENUEPOAPjDCJA8iy9G2R31GU4iOxKDMzn4 00BBR+VW1CD53GOZk5nFbF8XclQPopxEKCOHQyU5Fr3wJLYC1qF/KOJrOQYTtMT3bMum 7vnDuGn1rP4Zlm/DZmy4Gy8uJv9epAHmcg/0ddWss6y/RcyC164uNu9KQ7JXtg835sGK 8ybjw8la3rpikFJDwTcV0xvzE1yJGpC43KXipw7UTEV+bI/ZHWaADM9yUpYgBlXClFOx Wlfg== X-Gm-Message-State: AGRZ1gJPg2ZoqQmv2ck2qTnQk4VenGZMEc7SWuK4ky5giHenNjAtve3p Z3KG1tM7WtfAaAb+REoOiq7/CfVwrzvcogm/8Ro= X-Google-Smtp-Source: AJdET5fK5/hXnhtK1kk37hm04GiVsSI8EQhhTYspjvmtsLFwkUBnpiq4vkNWHfgvcZVmtVINJJGcoNNBxt4QlosWLas= X-Received: by 2002:a19:6a13:: with SMTP id u19mr2700626lfu.46.1541016157376; Wed, 31 Oct 2018 13:02:37 -0700 (PDT) MIME-Version: 1.0 References: <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> In-Reply-To: From: Andrew Vylegzhanin Date: Wed, 31 Oct 2018 23:02:25 +0300 Message-ID: Subject: Re: NFS + Infiniband problem To: rmacklem@uoguelph.ca Cc: "Rodney W. Grimes" , freebsd-fs@freebsd.org, freebsd-infiniband@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2018 20:02:39 -0000 =D1=81=D1=80, 31 =D0=BE=D0=BA=D1=82. 2018 =D0=B3. =D0=B2 5:53, Rick Macklem= : > > > > >net.inet.tcp.cc.algorithm=3Dhtcp > Well, I'm not familiar with the current TCP stack (and, as noted before, I know > nothing about InfiniBand). All I can suggest is testing with the default congestion > control algorithm. (I always test with the default, which appears to be newreno.) > NFS traffic looks very different than a typical use of TCP. Lots of small TCP > segments in both directions interspersed with some larger ones (the write > requests or read replies). With this TCP settings same server serve NFS requests via 40G Ethernet on multiply clients with speed via 1G Eth ~ 105/110 MB/sec write/read. Of course I'll try to change congestion algorithm, but I don't think that will help. Also need to test setup with infniband set from connected mode to datagram mode. -- Andrew > > rick From owner-freebsd-fs@freebsd.org Wed Oct 31 21:14:55 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 570BD10E9EF9 for ; Wed, 31 Oct 2018 21:14:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id ADCAD6C3E7 for ; Wed, 31 Oct 2018 21:14:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6FA5710E9EF8; Wed, 31 Oct 2018 21:14:54 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0218610E9EF7 for ; Wed, 31 Oct 2018 21:14:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7414E6C3E5 for ; Wed, 31 Oct 2018 21:14:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B35B356CE for ; Wed, 31 Oct 2018 21:14:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w9VLEqmT092911 for ; Wed, 31 Oct 2018 21:14:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w9VLEqsu092901 for fs@FreeBSD.org; Wed, 31 Oct 2018 21:14:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 229614] ZFS lockup in zil_commit_impl Date: Wed, 31 Oct 2018 21:14:52 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: andreas.sommer87@googlemail.com X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: allanjude@FreeBSD.org X-Bugzilla-Flags: mfc-stable11? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2018 21:14:55 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229614 --- Comment #33 from Andreas Sommer --- Seems I can test this every few days =E2=80=93=C2=A0that's how often it sti= ll happens. Did the kernel dump this time. Does any of this help? (kgdb) info threads * 414 Thread 100499 (PID=3D95019: sysctl) 0xffffffff80af691b in doadump () 413 Thread 101525 (PID=3D94981: csh) 0xffffffff80b25edd in sched_switch = () 412 Thread 100778 (PID=3D94980: su) 0xffffffff80b25edd in sched_switch () 411 Thread 100917 (PID=3D94979: sudo) 0xffffffff80b25edd in sched_switch= () 410 Thread 100677 (PID=3D94926: bash) 0xffffffff80b25edd in sched_switch= () 409 Thread 100630 (PID=3D94925: sshd) 0xffffffff80b25edd in sched_switch= () 408 Thread 100613 (PID=3D94923: sshd) 0xffffffff80b25edd in sched_switch= () 407 Thread 101554 (PID=3D94497: vim) 0xffffffff80b25edd in sched_switch = () 406 Thread 101348 (PID=3D94494: csh) 0xffffffff80b25edd in sched_switch = () 405 Thread 101335 (PID=3D94493: su) 0xffffffff80b25edd in sched_switch () 404 Thread 100497 (PID=3D94492: sudo) 0xffffffff80b25edd in sched_switch= () 403 Thread 101262 (PID=3D89931: pkg) 0xffffffff80b25edd in sched_switch = () 402 Thread 101526 (PID=3D89930: wc) 0xffffffff80b25edd in sched_switch () 401 Thread 101524 (PID=3D89929: tee) 0xffffffff80b25edd in sched_switch = () 400 Thread 101426 (PID=3D89928: sed) 0xffffffff80b25edd in sched_switch = () 399 Thread 101573 (PID=3D89927: pkg) 0xffffffff80b25edd in sched_switch = () 398 Thread 100628 (PID=3D89926: sh) 0xffffffff80b25edd in sched_switch () 397 Thread 101123 (PID=3D89923: sh) 0xffffffff80b25edd in sched_switch () 396 Thread 101403 (PID=3D89773: mail) 0xffffffff80b25edd in sched_switch= () 395 Thread 101513 (PID=3D89772: sh) 0xffffffff80b25edd in sched_switch () 394 Thread 101520 (PID=3D89759: sh) 0xffffffff80b25edd in sched_switch () 393 Thread 100680 (PID=3D89757: lockf) 0xffffffff80b25edd in sched_switc= h () 392 Thread 101014 (PID=3D89754: sh) 0xffffffff80b25edd in sched_switch () 391 Thread 100400 (PID=3D89753: sh) 0xffffffff80b25edd in sched_switch () 390 Thread 100620 (PID=3D89461: mail) 0xffffffff80b25edd in sched_switch= () 389 Thread 100521 (PID=3D89460: sh) 0xffffffff80b25edd in sched_switch () 388 Thread 101469 (PID=3D89442: sh) 0xffffffff80b25edd in sched_switch () ---Type to continue, or q to quit--- 387 Thread 100643 (PID=3D89441: lockf) 0xffffffff80b25edd in sched_switc= h () 386 Thread 100426 (PID=3D89435: sh) 0xffffffff80b25edd in sched_switch () 385 Thread 100700 (PID=3D89432: cron) 0xffffffff80b25edd in sched_switch= () 384 Thread 100525 (PID=3D87040: python3.6) 0xffffffff810f10b8 in cpustop_handler () 383 Thread 100512 (PID=3D84285: gpg-agent) 0xffffffff80b25edd in sched_s= witch () 382 Thread 100358 (PID=3D1257: getty) 0xffffffff80b25edd in sched_switch= () 381 Thread 100359 (PID=3D1256: getty) 0xffffffff80b25edd in sched_switch= () 380 Thread 100381 (PID=3D1255: getty) 0xffffffff80b25edd in sched_switch= () 379 Thread 100389 (PID=3D1254: getty) 0xffffffff80b25edd in sched_switch= () 378 Thread 100394 (PID=3D1253: getty) 0xffffffff80b25edd in sched_switch= () 377 Thread 100390 (PID=3D1252: getty) 0xffffffff80b25edd in sched_switch= () 376 Thread 100409 (PID=3D1251: getty) 0xffffffff80b25edd in sched_switch= () 375 Thread 100396 (PID=3D1250: getty) 0xffffffff80b25edd in sched_switch= () 374 Thread 100362 (PID=3D1249: getty) 0xffffffff80b25edd in sched_switch= () 373 Thread 100410 (PID=3D1205: cron) 0xffffffff80b25edd in sched_switch = () 372 Thread 100403 (PID=3D1191: sendmail) 0xffffffff80b25edd in sched_swi= tch () 371 Thread 100388 (PID=3D1188: sendmail) 0xffffffff80b25edd in sched_swi= tch () 370 Thread 100402 (PID=3D1132: cron) 0xffffffff80b25edd in sched_switch = () 369 Thread 100374 (PID=3D1128: sendmail) 0xffffffff80b25edd in sched_swi= tch () 368 Thread 100397 (PID=3D1125: sendmail) 0xffffffff80b25edd in sched_swi= tch () 367 Thread 100408 (PID=3D1122: multilog) 0xffffffff80b25edd in sched_swi= tch () 366 Thread 100387 (PID=3D1121: python3.6) 0xffffffff80b25edd in sched_sw= itch () 365 Thread 100417 (PID=3D1121: python3.6) 0xffffffff80b25edd in sched_sw= itch () 364 Thread 100418 (PID=3D1121: python3.6) 0xffffffff80b25edd in sched_sw= itch () 363 Thread 101413 (PID=3D1121: python3.6) 0xffffffff80b25edd in sched_sw= itch () 362 Thread 100407 (PID=3D1117: supervise) 0xffffffff80b25edd in sched_sw= itch () 361 Thread 100406 (PID=3D1116: supervise) 0xffffffff80b25edd in sched_sw= itch () ---Type to continue, or q to quit--- 360 Thread 100405 (PID=3D1113: readproctitle) 0xffffffff80b25edd in sched_switch () 359 Thread 100384 (PID=3D1112: svscan) 0xffffffff80b25edd in sched_switc= h () 358 Thread 100398 (PID=3D1059: syslogd) 0xffffffff80b25edd in sched_swit= ch () 357 Thread 100364 (PID=3D965: cron) 0xffffffff80b25edd in sched_switch () 356 Thread 100361 (PID=3D961: sendmail) 0xffffffff80b25edd in sched_swit= ch () 355 Thread 100395 (PID=3D958: sendmail) 0xffffffff80b25edd in sched_swit= ch () 354 Thread 100365 (PID=3D957: python3.6) 0xffffffff80b25edd in sched_swi= tch () 353 Thread 100370 (PID=3D950: multilog) 0xffffffff80b25edd in sched_swit= ch () 352 Thread 100382 (PID=3D949: sudo) 0xffffffff80b25edd in sched_switch () 351 Thread 100385 (PID=3D944: supervise) 0xffffffff80b25edd in sched_swi= tch () 350 Thread 100391 (PID=3D943: supervise) 0xffffffff80b25edd in sched_swi= tch () 349 Thread 100360 (PID=3D940: readproctitle) 0xffffffff80b25edd in sched_switch () 348 Thread 100393 (PID=3D939: svscan) 0xffffffff80b25edd in sched_switch= () 347 Thread 100392 (PID=3D891: syslogd) 0xffffffff80b25edd in sched_switc= h () 346 Thread 100386 (PID=3D787: sshd) 0xffffffff80b25edd in sched_switch () 345 Thread 100366 (PID=3D768: nginx) 0xffffffff80b25edd in sched_switch = () 344 Thread 100356 (PID=3D767: nginx) 0xffffffff80b25edd in sched_switch = () 343 Thread 100379 (PID=3D587: syslogd) 0xffffffff80b25edd in sched_switc= h () 342 Thread 100377 (PID=3D512: pf purge) 0xffffffff80b25edd in sched_swit= ch () 341 Thread 100375 (PID=3D504: devd) 0xffffffff80b25edd in sched_switch () 340 Thread 100372 (PID=3D498: dhclient) 0xffffffff80b25edd in sched_swit= ch () 339 Thread 100369 (PID=3D452: dhclient) 0xffffffff80b25edd in sched_swit= ch () 338 Thread 100092 (PID=3D25: syncer) 0xffffffff80b25edd in sched_switch = () 337 Thread 100091 (PID=3D24: vnlru) 0xffffffff80b25edd in sched_switch () 336 Thread 100090 (PID=3D23: bufdaemon) 0xffffffff80b25edd in sched_swit= ch () 335 Thread 100089 (PID=3D22: bufspacedaemon) 0xffffffff80b25edd in sched_switch () 334 Thread 100087 (PID=3D21: pagezero) 0xffffffff80b25edd in sched_switc= h () ---Type to continue, or q to quit--- 333 Thread 100085 (PID=3D20: vmdaemon) 0xffffffff80b25edd in sched_switc= h () 332 Thread 100084 (PID=3D19: pagedaemon/dom0) 0xffffffff80b25edd in sched_switch () 331 Thread 100086 (PID=3D19: pagedaemon/laundry: dom0) 0xffffffff80b25ed= d in sched_switch () 330 Thread 100088 (PID=3D19: pagedaemon/uma) 0xffffffff80b25edd in sched_switch () 329 Thread 100080 (PID=3D18: balloon) 0xffffffff80b25edd in sched_switch= () 328 Thread 100074 (PID=3D17: rand_harvestq) 0xffffffff80b25edd in sched_= switch () 327 Thread 100073 (PID=3D16: sctp_iterator) 0xffffffff80b25edd in sched_= switch () 326 Thread 100052 (PID=3D9: zfskern/arc_reclaim_thread) 0xffffffff80b25e= dd in sched_switch () 325 Thread 100053 (PID=3D9: zfskern/arc_dnlc_evicts_thr) 0xffffffff80b25= edd in sched_switch () 324 Thread 100055 (PID=3D9: zfskern/dbuf_evict_thread) 0xffffffff80b25ed= d in sched_switch () 323 Thread 100072 (PID=3D9: zfskern/l2arc_feed_thread) 0xffffffff80b25ed= d in sched_switch () 322 Thread 100323 (PID=3D9: zfskern/trim zroot) 0xffffffff80b25edd in sched_switch () 321 Thread 100348 (PID=3D9: zfskern/txg_thread_enter) 0xffffffff80b25edd= in sched_switch () 320 Thread 100349 (PID=3D9: zfskern/txg_thread_enter) 0xffffffff80b25edd= in sched_switch () 319 Thread 100350 (PID=3D9: zfskern/solthread 0xfffffff) 0xffffffff80b25= edd in sched_switch () 318 Thread 100351 (PID=3D9: zfskern/solthread 0xfffffff) 0xffffffff80b25= edd in sched_switch () 317 Thread 100051 (PID=3D8: soaiod4) 0xffffffff80b25edd in sched_switch = () 316 Thread 100050 (PID=3D7: soaiod3) 0xffffffff80b25edd in sched_switch = () 315 Thread 100049 (PID=3D6: soaiod2) 0xffffffff80b25edd in sched_switch = () 314 Thread 100048 (PID=3D5: soaiod1) 0xffffffff80b25edd in sched_switch = () 313 Thread 100042 (PID=3D15: xenstore_rcv) 0xffffffff80b25edd in sched_s= witch () 312 Thread 100041 (PID=3D14: xenwatch) 0xffffffff80b25edd in sched_switc= h () 311 Thread 100034 (PID=3D4: cam/doneq0) 0xffffffff80b25edd in sched_swit= ch () 310 Thread 100079 (PID=3D4: cam/scanner) 0xffffffff80b25edd in sched_swi= tch () 309 Thread 100033 (PID=3D3: crypto returns) 0xffffffff80b25edd in sched_= switch () 308 Thread 100032 (PID=3D2: crypto) 0xffffffff80b25edd in sched_switch () 307 Thread 100028 (PID=3D13: geom/g_event) 0xffffffff80b25edd in sched_s= witch () ---Type to continue, or q to quit--- 306 Thread 100029 (PID=3D13: geom/g_up) 0xffffffff80b25edd in sched_swit= ch () 305 Thread 100030 (PID=3D13: geom/g_down) 0xffffffff80b25edd in sched_sw= itch () 304 Thread 100008 (PID=3D12: intr/swi5: fast taskq) 0xffffffff80f58ed0 in fork_trampoline () 303 Thread 100010 (PID=3D12: intr/swi6: task queue) 0xffffffff80b25edd in sched_switch () 302 Thread 100012 (PID=3D12: intr/swi6: Giant taskq) 0xffffffff80b25edd = in sched_switch () 301 Thread 100022 (PID=3D12: intr/swi1: netisr 0) 0xffffffff80b25edd in sched_switch () 300 Thread 100023 (PID=3D12: intr/swi4: clock (0)) 0xffffffff80b25edd in sched_switch () 299 Thread 100024 (PID=3D12: intr/swi4: clock (1)) 0xffffffff80f58ed0 in fork_trampoline () 298 Thread 100025 (PID=3D12: intr/swi4: clock (2)) 0xffffffff80f58ed0 in fork_trampoline () 297 Thread 100026 (PID=3D12: intr/swi4: clock (3)) 0xffffffff80f58ed0 in fork_trampoline () 296 Thread 100027 (PID=3D12: intr/swi3: vm) 0xffffffff80f58ed0 in fork_trampoline () 295 Thread 100035 (PID=3D12: intr/irq14: ata0) 0xffffffff80f58ed0 in fork_trampoline () 294 Thread 100036 (PID=3D12: intr/irq15: ata1) 0xffffffff80f58ed0 in fork_trampoline () 293 Thread 100037 (PID=3D12: intr/irq1: atkbd0) 0xffffffff80b25edd in sched_switch () 292 Thread 100038 (PID=3D12: intr/irq12: psm0) 0xffffffff80f58ed0 in fork_trampoline () 291 Thread 100039 (PID=3D12: intr/swi0: uart) 0xffffffff80b25edd in sched_switch () 290 Thread 100040 (PID=3D12: intr/irq808: xenstore0) 0xffffffff80b25edd = in sched_switch () 289 Thread 100081 (PID=3D12: intr/irq813: xbd0) 0xffffffff80b25edd in sched_switch () 288 Thread 100083 (PID=3D12: intr/irq814: xn0) 0xffffffff80b25edd in sched_switch () 287 Thread 100378 (PID=3D12: intr/swi1: pf send) 0xffffffff80f58ed0 in fork_trampoline () 286 Thread 101269 (PID=3D12: intr/irq815: xbd5) 0xffffffff80b25edd in sched_switch () 285 Thread 100003 (PID=3D11: idle/idle: cpu0) 0xffffffff80b25edd in sched_switch () 284 Thread 100004 (PID=3D11: idle/idle: cpu1) 0xffffffff810f10b8 in cpustop_handler () 283 Thread 100005 (PID=3D11: idle/idle: cpu2) 0xffffffff80b25edd in sched_switch () 282 Thread 100006 (PID=3D11: idle/idle: cpu3) 0xffffffff810f10b8 in cpustop_handler () 281 Thread 100002 (PID=3D1: init) 0xffffffff80b25edd in sched_switch () 280 Thread 100001 (PID=3D10: audit) 0xffffffff80b25edd in sched_switch () ---Type to continue, or q to quit--- 279 Thread 100000 (PID=3D0: kernel/swapper) 0xffffffff80b25edd in sched_= switch () 278 Thread 100007 (PID=3D0: kernel/thread taskq) 0xffffffff80b25edd in sched_switch () 277 Thread 100009 (PID=3D0: kernel/kqueue_ctx taskq) 0xffffffff80b25edd = in sched_switch () 276 Thread 100011 (PID=3D0: kernel/aiod_kick taskq) 0xffffffff80b25edd in sched_switch () 275 Thread 100013 (PID=3D0: kernel/if_io_tqg_0) 0xffffffff80b25edd in sched_switch () 274 Thread 100014 (PID=3D0: kernel/if_io_tqg_1) 0xffffffff80b25edd in sched_switch () 273 Thread 100015 (PID=3D0: kernel/if_io_tqg_2) 0xffffffff80b25edd in sched_switch () 272 Thread 100016 (PID=3D0: kernel/if_io_tqg_3) 0xffffffff80b25edd in sched_switch () 271 Thread 100017 (PID=3D0: kernel/softirq_0) 0xffffffff80b25edd in sched_switch () 270 Thread 100018 (PID=3D0: kernel/softirq_1) 0xffffffff80b25edd in sched_switch () 269 Thread 100019 (PID=3D0: kernel/softirq_2) 0xffffffff80b25edd in sched_switch () 268 Thread 100020 (PID=3D0: kernel/softirq_3) 0xffffffff80b25edd in sched_switch () 267 Thread 100021 (PID=3D0: kernel/if_config_tqg_0) 0xffffffff80b25edd in sched_switch () 266 Thread 100031 (PID=3D0: kernel/firmware taskq) 0xffffffff80b25edd in sched_switch () 265 Thread 100043 (PID=3D0: kernel/mca taskq) 0xffffffff80b25edd in sched_switch () 264 Thread 100044 (PID=3D0: kernel/system_taskq_0) 0xffffffff80b25edd in sched_switch () 263 Thread 100045 (PID=3D0: kernel/system_taskq_1) 0xffffffff80b25edd in sched_switch () 262 Thread 100046 (PID=3D0: kernel/system_taskq_2) 0xffffffff80b25edd in sched_switch () 261 Thread 100047 (PID=3D0: kernel/system_taskq_3) 0xffffffff80b25edd in sched_switch () 260 Thread 100054 (PID=3D0: kernel/dbu_evict) 0xffffffff80b25edd in sched_switch () 259 Thread 100056 (PID=3D0: kernel/z_vdev_file_0) 0xffffffff80b25edd in sched_switch () 258 Thread 100057 (PID=3D0: kernel/z_vdev_file_1) 0xffffffff80b25edd in sched_switch () 257 Thread 100058 (PID=3D0: kernel/z_vdev_file_2) 0xffffffff80b25edd in sched_switch () 256 Thread 100059 (PID=3D0: kernel/z_vdev_file_3) 0xffffffff80b25edd in sched_switch () 255 Thread 100060 (PID=3D0: kernel/z_vdev_file_4) 0xffffffff80b25edd in sched_switch () 254 Thread 100061 (PID=3D0: kernel/z_vdev_file_5) 0xffffffff80b25edd in sched_switch () 253 Thread 100062 (PID=3D0: kernel/z_vdev_file_6) 0xffffffff80b25edd in sched_switch () ---Type to continue, or q to quit--- 252 Thread 100063 (PID=3D0: kernel/z_vdev_file_7) 0xffffffff80b25edd in sched_switch () 251 Thread 100064 (PID=3D0: kernel/z_vdev_file_8) 0xffffffff80b25edd in sched_switch () 250 Thread 100065 (PID=3D0: kernel/z_vdev_file_9) 0xffffffff80b25edd in sched_switch () 249 Thread 100066 (PID=3D0: kernel/z_vdev_file_10) 0xffffffff80b25edd in sched_switch () 248 Thread 100067 (PID=3D0: kernel/z_vdev_file_11) 0xffffffff80b25edd in sched_switch () 247 Thread 100068 (PID=3D0: kernel/z_vdev_file_12) 0xffffffff80b25edd in sched_switch () 246 Thread 100069 (PID=3D0: kernel/z_vdev_file_13) 0xffffffff80b25edd in sched_switch () 245 Thread 100070 (PID=3D0: kernel/z_vdev_file_14) 0xffffffff80b25edd in sched_switch () 244 Thread 100071 (PID=3D0: kernel/z_vdev_file_15) 0xffffffff80b25edd in sched_switch () 243 Thread 100075 (PID=3D0: kernel/acpi_task_0) 0xffffffff80b25edd in sched_switch () 242 Thread 100076 (PID=3D0: kernel/acpi_task_1) 0xffffffff80b25edd in sched_switch () 241 Thread 100077 (PID=3D0: kernel/acpi_task_2) 0xffffffff80b25edd in sched_switch () 240 Thread 100078 (PID=3D0: kernel/CAM taskq) 0xffffffff80b25edd in sched_switch () 239 Thread 100095 (PID=3D0: kernel/zio_null_issue) 0xffffffff80b25edd in sched_switch () 238 Thread 100096 (PID=3D0: kernel/zio_null_intr) 0xffffffff80b25edd in sched_switch () 237 Thread 100097 (PID=3D0: kernel/zio_read_issue_0) 0xffffffff80b25edd = in sched_switch () 236 Thread 100098 (PID=3D0: kernel/zio_read_issue_1) 0xffffffff80b25edd = in sched_switch () 235 Thread 100099 (PID=3D0: kernel/zio_read_issue_2) 0xffffffff80b25edd = in sched_switch () 234 Thread 100100 (PID=3D0: kernel/zio_read_issue_3) 0xffffffff80b25edd = in sched_switch () 233 Thread 100101 (PID=3D0: kernel/zio_read_issue_4) 0xffffffff80b25edd = in sched_switch () 232 Thread 100102 (PID=3D0: kernel/zio_read_issue_5) 0xffffffff80b25edd = in sched_switch () 231 Thread 100103 (PID=3D0: kernel/zio_read_issue_6) 0xffffffff80b25edd = in sched_switch () 230 Thread 100104 (PID=3D0: kernel/zio_read_issue_7) 0xffffffff80b25edd = in sched_switch () 229 Thread 100105 (PID=3D0: kernel/zio_read_intr_0_0) 0xffffffff80b25edd= in sched_switch () 228 Thread 100106 (PID=3D0: kernel/zio_read_intr_0_1) 0xffffffff80b25edd= in sched_switch () 227 Thread 100107 (PID=3D0: kernel/zio_read_intr_0_2) 0xffffffff80b25edd= in sched_switch () 226 Thread 100108 (PID=3D0: kernel/zio_read_intr_0_3) 0xffffffff80b25edd= in sched_switch () ---Type to continue, or q to quit--- 225 Thread 100109 (PID=3D0: kernel/zio_read_intr_0_4) 0xffffffff80b25edd= in sched_switch () 224 Thread 100110 (PID=3D0: kernel/zio_read_intr_0_5) 0xffffffff80b25edd= in sched_switch () 223 Thread 100111 (PID=3D0: kernel/zio_read_intr_0_6) 0xffffffff80b25edd= in sched_switch () 222 Thread 100112 (PID=3D0: kernel/zio_read_intr_0_7) 0xffffffff80b25edd= in sched_switch () 221 Thread 100113 (PID=3D0: kernel/zio_read_intr_0_8) 0xffffffff80b25edd= in sched_switch () 220 Thread 100114 (PID=3D0: kernel/zio_read_intr_0_9) 0xffffffff80b25edd= in sched_switch () 219 Thread 100115 (PID=3D0: kernel/zio_read_intr_0_10) 0xffffffff80b25ed= d in sched_switch () 218 Thread 100116 (PID=3D0: kernel/zio_read_intr_0_11) 0xffffffff80b25ed= d in sched_switch () 217 Thread 100117 (PID=3D0: kernel/zio_read_intr_1_0) 0xffffffff80b25edd= in sched_switch () 216 Thread 100118 (PID=3D0: kernel/zio_read_intr_1_1) 0xffffffff80b25edd= in sched_switch () 215 Thread 100119 (PID=3D0: kernel/zio_read_intr_1_2) 0xffffffff80b25edd= in sched_switch () 214 Thread 100120 (PID=3D0: kernel/zio_read_intr_1_3) 0xffffffff80b25edd= in sched_switch () 213 Thread 100121 (PID=3D0: kernel/zio_read_intr_1_4) 0xffffffff80b25edd= in sched_switch () 212 Thread 100122 (PID=3D0: kernel/zio_read_intr_1_5) 0xffffffff80b25edd= in sched_switch () 211 Thread 100123 (PID=3D0: kernel/zio_read_intr_1_6) 0xffffffff80b25edd= in sched_switch () 210 Thread 100124 (PID=3D0: kernel/zio_read_intr_1_7) 0xffffffff80b25edd= in sched_switch () 209 Thread 100125 (PID=3D0: kernel/zio_read_intr_1_8) 0xffffffff80b25edd= in sched_switch () 208 Thread 100126 (PID=3D0: kernel/zio_read_intr_1_9) 0xffffffff80b25edd= in sched_switch () 207 Thread 100127 (PID=3D0: kernel/zio_read_intr_1_10) 0xffffffff80b25ed= d in sched_switch () 206 Thread 100128 (PID=3D0: kernel/zio_read_intr_1_11) 0xffffffff80b25ed= d in sched_switch () 205 Thread 100129 (PID=3D0: kernel/zio_read_intr_2_0) 0xffffffff80b25edd= in sched_switch () 204 Thread 100130 (PID=3D0: kernel/zio_read_intr_2_1) 0xffffffff80b25edd= in sched_switch () 203 Thread 100131 (PID=3D0: kernel/zio_read_intr_2_2) 0xffffffff80b25edd= in sched_switch () 202 Thread 100132 (PID=3D0: kernel/zio_read_intr_2_3) 0xffffffff80b25edd= in sched_switch () 201 Thread 100133 (PID=3D0: kernel/zio_read_intr_2_4) 0xffffffff80b25edd= in sched_switch () 200 Thread 100134 (PID=3D0: kernel/zio_read_intr_2_5) 0xffffffff80b25edd= in sched_switch () 199 Thread 100135 (PID=3D0: kernel/zio_read_intr_2_6) 0xffffffff80b25edd= in sched_switch () ---Type to continue, or q to quit--- 198 Thread 100136 (PID=3D0: kernel/zio_read_intr_2_7) 0xffffffff80b25edd= in sched_switch () 197 Thread 100137 (PID=3D0: kernel/zio_read_intr_2_8) 0xffffffff80b25edd= in sched_switch () 196 Thread 100138 (PID=3D0: kernel/zio_read_intr_2_9) 0xffffffff80b25edd= in sched_switch () 195 Thread 100139 (PID=3D0: kernel/zio_read_intr_2_10) 0xffffffff80b25ed= d in sched_switch () 194 Thread 100140 (PID=3D0: kernel/zio_read_intr_2_11) 0xffffffff80b25ed= d in sched_switch () 193 Thread 100141 (PID=3D0: kernel/zio_read_intr_3_0) 0xffffffff80b25edd= in sched_switch () 192 Thread 100142 (PID=3D0: kernel/zio_read_intr_3_1) 0xffffffff80b25edd= in sched_switch () 191 Thread 100143 (PID=3D0: kernel/zio_read_intr_3_2) 0xffffffff80b25edd= in sched_switch () 190 Thread 100144 (PID=3D0: kernel/zio_read_intr_3_3) 0xffffffff80b25edd= in sched_switch () 189 Thread 100145 (PID=3D0: kernel/zio_read_intr_3_4) 0xffffffff80b25edd= in sched_switch () 188 Thread 100146 (PID=3D0: kernel/zio_read_intr_3_5) 0xffffffff80b25edd= in sched_switch () 187 Thread 100147 (PID=3D0: kernel/zio_read_intr_3_6) 0xffffffff80b25edd= in sched_switch () 186 Thread 100148 (PID=3D0: kernel/zio_read_intr_3_7) 0xffffffff80b25edd= in sched_switch () 185 Thread 100149 (PID=3D0: kernel/zio_read_intr_3_8) 0xffffffff80b25edd= in sched_switch () 184 Thread 100150 (PID=3D0: kernel/zio_read_intr_3_9) 0xffffffff80b25edd= in sched_switch () 183 Thread 100151 (PID=3D0: kernel/zio_read_intr_3_10) 0xffffffff80b25ed= d in sched_switch () 182 Thread 100152 (PID=3D0: kernel/zio_read_intr_3_11) 0xffffffff80b25ed= d in sched_switch () 181 Thread 100153 (PID=3D0: kernel/zio_read_intr_4_0) 0xffffffff80b25edd= in sched_switch () 180 Thread 100154 (PID=3D0: kernel/zio_read_intr_4_1) 0xffffffff80b25edd= in sched_switch () 179 Thread 100155 (PID=3D0: kernel/zio_read_intr_4_2) 0xffffffff80b25edd= in sched_switch () 178 Thread 100156 (PID=3D0: kernel/zio_read_intr_4_3) 0xffffffff80b25edd= in sched_switch () 177 Thread 100157 (PID=3D0: kernel/zio_read_intr_4_4) 0xffffffff80b25edd= in sched_switch () 176 Thread 100158 (PID=3D0: kernel/zio_read_intr_4_5) 0xffffffff80b25edd= in sched_switch () 175 Thread 100159 (PID=3D0: kernel/zio_read_intr_4_6) 0xffffffff80b25edd= in sched_switch () 174 Thread 100160 (PID=3D0: kernel/zio_read_intr_4_7) 0xffffffff80b25edd= in sched_switch () 173 Thread 100161 (PID=3D0: kernel/zio_read_intr_4_8) 0xffffffff80b25edd= in sched_switch () 172 Thread 100162 (PID=3D0: kernel/zio_read_intr_4_9) 0xffffffff80b25edd= in sched_switch () ---Type to continue, or q to quit--- 171 Thread 100163 (PID=3D0: kernel/zio_read_intr_4_10) 0xffffffff80b25ed= d in sched_switch () 170 Thread 100164 (PID=3D0: kernel/zio_read_intr_4_11) 0xffffffff80b25ed= d in sched_switch () 169 Thread 100165 (PID=3D0: kernel/zio_read_intr_5_0) 0xffffffff80b25edd= in sched_switch () 168 Thread 100166 (PID=3D0: kernel/zio_read_intr_5_1) 0xffffffff80b25edd= in sched_switch () 167 Thread 100167 (PID=3D0: kernel/zio_read_intr_5_2) 0xffffffff80b25edd= in sched_switch () 166 Thread 100168 (PID=3D0: kernel/zio_read_intr_5_3) 0xffffffff80b25edd= in sched_switch () 165 Thread 100169 (PID=3D0: kernel/zio_read_intr_5_4) 0xffffffff80b25edd= in sched_switch () 164 Thread 100170 (PID=3D0: kernel/zio_read_intr_5_5) 0xffffffff80b25edd= in sched_switch () 163 Thread 100171 (PID=3D0: kernel/zio_read_intr_5_6) 0xffffffff80b25edd= in sched_switch () 162 Thread 100172 (PID=3D0: kernel/zio_read_intr_5_7) 0xffffffff80b25edd= in sched_switch () 161 Thread 100173 (PID=3D0: kernel/zio_read_intr_5_8) 0xffffffff80b25edd= in sched_switch () 160 Thread 100174 (PID=3D0: kernel/zio_read_intr_5_9) 0xffffffff80b25edd= in sched_switch () 159 Thread 100175 (PID=3D0: kernel/zio_read_intr_5_10) 0xffffffff80b25ed= d in sched_switch () 158 Thread 100176 (PID=3D0: kernel/zio_read_intr_5_11) 0xffffffff80b25ed= d in sched_switch () 157 Thread 100177 (PID=3D0: kernel/zio_read_intr_6_0) 0xffffffff80b25edd= in sched_switch () 156 Thread 100178 (PID=3D0: kernel/zio_read_intr_6_1) 0xffffffff80b25edd= in sched_switch () 155 Thread 100179 (PID=3D0: kernel/zio_read_intr_6_2) 0xffffffff80b25edd= in sched_switch () 154 Thread 100180 (PID=3D0: kernel/zio_read_intr_6_3) 0xffffffff80b25edd= in sched_switch () 153 Thread 100181 (PID=3D0: kernel/zio_read_intr_6_4) 0xffffffff80b25edd= in sched_switch () 152 Thread 100182 (PID=3D0: kernel/zio_read_intr_6_5) 0xffffffff80b25edd= in sched_switch () 151 Thread 100183 (PID=3D0: kernel/zio_read_intr_6_6) 0xffffffff80b25edd= in sched_switch () 150 Thread 100184 (PID=3D0: kernel/zio_read_intr_6_7) 0xffffffff80b25edd= in sched_switch () 149 Thread 100185 (PID=3D0: kernel/zio_read_intr_6_8) 0xffffffff80b25edd= in sched_switch () 148 Thread 100186 (PID=3D0: kernel/zio_read_intr_6_9) 0xffffffff80b25edd= in sched_switch () 147 Thread 100187 (PID=3D0: kernel/zio_read_intr_6_10) 0xffffffff80b25ed= d in sched_switch () 146 Thread 100188 (PID=3D0: kernel/zio_read_intr_6_11) 0xffffffff80b25ed= d in sched_switch () 145 Thread 100189 (PID=3D0: kernel/zio_read_intr_7_0) 0xffffffff80b25edd= in sched_switch () ---Type to continue, or q to quit--- 144 Thread 100190 (PID=3D0: kernel/zio_read_intr_7_1) 0xffffffff80b25edd= in sched_switch () 143 Thread 100191 (PID=3D0: kernel/zio_read_intr_7_2) 0xffffffff80b25edd= in sched_switch () 142 Thread 100192 (PID=3D0: kernel/zio_read_intr_7_3) 0xffffffff80b25edd= in sched_switch () 141 Thread 100193 (PID=3D0: kernel/zio_read_intr_7_4) 0xffffffff80b25edd= in sched_switch () 140 Thread 100194 (PID=3D0: kernel/zio_read_intr_7_5) 0xffffffff80b25edd= in sched_switch () 139 Thread 100195 (PID=3D0: kernel/zio_read_intr_7_6) 0xffffffff80b25edd= in sched_switch () 138 Thread 100196 (PID=3D0: kernel/zio_read_intr_7_7) 0xffffffff80b25edd= in sched_switch () 137 Thread 100197 (PID=3D0: kernel/zio_read_intr_7_8) 0xffffffff80b25edd= in sched_switch () 136 Thread 100198 (PID=3D0: kernel/zio_read_intr_7_9) 0xffffffff80b25edd= in sched_switch () 135 Thread 100199 (PID=3D0: kernel/zio_read_intr_7_10) 0xffffffff80b25ed= d in sched_switch () 134 Thread 100200 (PID=3D0: kernel/zio_read_intr_7_11) 0xffffffff80b25ed= d in sched_switch () 133 Thread 100201 (PID=3D0: kernel/zio_write_issue_0) 0xffffffff80b25edd= in sched_switch () 132 Thread 100202 (PID=3D0: kernel/zio_write_issue_1) 0xffffffff80b25edd= in sched_switch () 131 Thread 100203 (PID=3D0: kernel/zio_write_issue_2) 0xffffffff80b25edd= in sched_switch () 130 Thread 100204 (PID=3D0: kernel/zio_write_issue_hig) 0xffffffff80b25e= dd in sched_switch () 129 Thread 100205 (PID=3D0: kernel/zio_write_issue_hig) 0xffffffff80b25e= dd in sched_switch () 128 Thread 100206 (PID=3D0: kernel/zio_write_issue_hig) 0xffffffff80b25e= dd in sched_switch () 127 Thread 100207 (PID=3D0: kernel/zio_write_issue_hig) 0xffffffff80b25e= dd in sched_switch () 126 Thread 100208 (PID=3D0: kernel/zio_write_issue_hig) 0xffffffff80b25e= dd in sched_switch () 125 Thread 100209 (PID=3D0: kernel/zio_write_intr_0) 0xffffffff80b25edd = in sched_switch () 124 Thread 100210 (PID=3D0: kernel/zio_write_intr_1) 0xffffffff80b25edd = in sched_switch () 123 Thread 100211 (PID=3D0: kernel/zio_write_intr_2) 0xffffffff80b25edd = in sched_switch () 122 Thread 100212 (PID=3D0: kernel/zio_write_intr_3) 0xffffffff80b25edd = in sched_switch () 121 Thread 100213 (PID=3D0: kernel/zio_write_intr_4) 0xffffffff80b25edd = in sched_switch () 120 Thread 100214 (PID=3D0: kernel/zio_write_intr_5) 0xffffffff80b25edd = in sched_switch () 119 Thread 100215 (PID=3D0: kernel/zio_write_intr_6) 0xffffffff80b25edd = in sched_switch () 118 Thread 100216 (PID=3D0: kernel/zio_write_intr_7) 0xffffffff80b25edd = in sched_switch () ---Type to continue, or q to quit--- 117 Thread 100217 (PID=3D0: kernel/zio_write_intr_high) 0xffffffff80b25e= dd in sched_switch () 116 Thread 100218 (PID=3D0: kernel/zio_write_intr_high) 0xffffffff80b25e= dd in sched_switch () 115 Thread 100219 (PID=3D0: kernel/zio_write_intr_high) 0xffffffff80b25e= dd in sched_switch () 114 Thread 100220 (PID=3D0: kernel/zio_write_intr_high) 0xffffffff80b25e= dd in sched_switch () 113 Thread 100221 (PID=3D0: kernel/zio_write_intr_high) 0xffffffff80b25e= dd in sched_switch () 112 Thread 100222 (PID=3D0: kernel/zio_free_issue_0_0) 0xffffffff80b25ed= d in sched_switch () 111 Thread 100223 (PID=3D0: kernel/zio_free_issue_0_1) 0xffffffff80b25ed= d in sched_switch () 110 Thread 100224 (PID=3D0: kernel/zio_free_issue_0_2) 0xffffffff80b25ed= d in sched_switch () 109 Thread 100225 (PID=3D0: kernel/zio_free_issue_0_3) 0xffffffff80b25ed= d in sched_switch () 108 Thread 100226 (PID=3D0: kernel/zio_free_issue_0_4) 0xffffffff80b25ed= d in sched_switch () 107 Thread 100227 (PID=3D0: kernel/zio_free_issue_0_5) 0xffffffff80b25ed= d in sched_switch () 106 Thread 100228 (PID=3D0: kernel/zio_free_issue_0_6) 0xffffffff80b25ed= d in sched_switch () 105 Thread 100229 (PID=3D0: kernel/zio_free_issue_0_7) 0xffffffff80b25ed= d in sched_switch () 104 Thread 100230 (PID=3D0: kernel/zio_free_issue_0_8) 0xffffffff80b25ed= d in sched_switch () 103 Thread 100231 (PID=3D0: kernel/zio_free_issue_0_9) 0xffffffff80b25ed= d in sched_switch () 102 Thread 100232 (PID=3D0: kernel/zio_free_issue_0_10) 0xffffffff80b25e= dd in sched_switch () 101 Thread 100233 (PID=3D0: kernel/zio_free_issue_0_11) 0xffffffff80b25e= dd in sched_switch () 100 Thread 100234 (PID=3D0: kernel/zio_free_issue_1_0) 0xffffffff80b25ed= d in sched_switch () 99 Thread 100235 (PID=3D0: kernel/zio_free_issue_1_1) 0xffffffff80b25edd= in sched_switch () 98 Thread 100236 (PID=3D0: kernel/zio_free_issue_1_2) 0xffffffff80b25edd= in sched_switch () 97 Thread 100237 (PID=3D0: kernel/zio_free_issue_1_3) 0xffffffff80b25edd= in sched_switch () 96 Thread 100238 (PID=3D0: kernel/zio_free_issue_1_4) 0xffffffff80b25edd= in sched_switch () 95 Thread 100239 (PID=3D0: kernel/zio_free_issue_1_5) 0xffffffff80b25edd= in sched_switch () 94 Thread 100240 (PID=3D0: kernel/zio_free_issue_1_6) 0xffffffff80b25edd= in sched_switch () 93 Thread 100241 (PID=3D0: kernel/zio_free_issue_1_7) 0xffffffff80b25edd= in sched_switch () 92 Thread 100242 (PID=3D0: kernel/zio_free_issue_1_8) 0xffffffff80b25edd= in sched_switch () 91 Thread 100243 (PID=3D0: kernel/zio_free_issue_1_9) 0xffffffff80b25edd= in sched_switch () ---Type to continue, or q to quit--- 90 Thread 100244 (PID=3D0: kernel/zio_free_issue_1_10) 0xffffffff80b25ed= d in sched_switch () 89 Thread 100245 (PID=3D0: kernel/zio_free_issue_1_11) 0xffffffff80b25ed= d in sched_switch () 88 Thread 100246 (PID=3D0: kernel/zio_free_issue_2_0) 0xffffffff80b25edd= in sched_switch () 87 Thread 100247 (PID=3D0: kernel/zio_free_issue_2_1) 0xffffffff80b25edd= in sched_switch () 86 Thread 100248 (PID=3D0: kernel/zio_free_issue_2_2) 0xffffffff80b25edd= in sched_switch () 85 Thread 100249 (PID=3D0: kernel/zio_free_issue_2_3) 0xffffffff80b25edd= in sched_switch () 84 Thread 100250 (PID=3D0: kernel/zio_free_issue_2_4) 0xffffffff80b25edd= in sched_switch () 83 Thread 100251 (PID=3D0: kernel/zio_free_issue_2_5) 0xffffffff80b25edd= in sched_switch () 82 Thread 100252 (PID=3D0: kernel/zio_free_issue_2_6) 0xffffffff80b25edd= in sched_switch () 81 Thread 100253 (PID=3D0: kernel/zio_free_issue_2_7) 0xffffffff80b25edd= in sched_switch () 80 Thread 100254 (PID=3D0: kernel/zio_free_issue_2_8) 0xffffffff80b25edd= in sched_switch () 79 Thread 100255 (PID=3D0: kernel/zio_free_issue_2_9) 0xffffffff80b25edd= in sched_switch () 78 Thread 100256 (PID=3D0: kernel/zio_free_issue_2_10) 0xffffffff80b25ed= d in sched_switch () 77 Thread 100257 (PID=3D0: kernel/zio_free_issue_2_11) 0xffffffff80b25ed= d in sched_switch () 76 Thread 100258 (PID=3D0: kernel/zio_free_issue_3_0) 0xffffffff80b25edd= in sched_switch () 75 Thread 100259 (PID=3D0: kernel/zio_free_issue_3_1) 0xffffffff80b25edd= in sched_switch () 74 Thread 100260 (PID=3D0: kernel/zio_free_issue_3_2) 0xffffffff80b25edd= in sched_switch () 73 Thread 100261 (PID=3D0: kernel/zio_free_issue_3_3) 0xffffffff80b25edd= in sched_switch () 72 Thread 100262 (PID=3D0: kernel/zio_free_issue_3_4) 0xffffffff80b25edd= in sched_switch () 71 Thread 100263 (PID=3D0: kernel/zio_free_issue_3_5) 0xffffffff80b25edd= in sched_switch () 70 Thread 100264 (PID=3D0: kernel/zio_free_issue_3_6) 0xffffffff80b25edd= in sched_switch () 69 Thread 100265 (PID=3D0: kernel/zio_free_issue_3_7) 0xffffffff80b25edd= in sched_switch () 68 Thread 100266 (PID=3D0: kernel/zio_free_issue_3_8) 0xffffffff80b25edd= in sched_switch () 67 Thread 100267 (PID=3D0: kernel/zio_free_issue_3_9) 0xffffffff80b25edd= in sched_switch () 66 Thread 100268 (PID=3D0: kernel/zio_free_issue_3_10) 0xffffffff80b25ed= d in sched_switch () 65 Thread 100269 (PID=3D0: kernel/zio_free_issue_3_11) 0xffffffff80b25ed= d in sched_switch () 64 Thread 100270 (PID=3D0: kernel/zio_free_issue_4_0) 0xffffffff80b25edd= in sched_switch () ---Type to continue, or q to quit--- 63 Thread 100271 (PID=3D0: kernel/zio_free_issue_4_1) 0xffffffff80b25edd= in sched_switch () 62 Thread 100272 (PID=3D0: kernel/zio_free_issue_4_2) 0xffffffff80b25edd= in sched_switch () 61 Thread 100273 (PID=3D0: kernel/zio_free_issue_4_3) 0xffffffff80b25edd= in sched_switch () 60 Thread 100274 (PID=3D0: kernel/zio_free_issue_4_4) 0xffffffff80b25edd= in sched_switch () 59 Thread 100275 (PID=3D0: kernel/zio_free_issue_4_5) 0xffffffff80b25edd= in sched_switch () 58 Thread 100276 (PID=3D0: kernel/zio_free_issue_4_6) 0xffffffff80b25edd= in sched_switch () 57 Thread 100277 (PID=3D0: kernel/zio_free_issue_4_7) 0xffffffff80b25edd= in sched_switch () 56 Thread 100278 (PID=3D0: kernel/zio_free_issue_4_8) 0xffffffff80b25edd= in sched_switch () 55 Thread 100279 (PID=3D0: kernel/zio_free_issue_4_9) 0xffffffff80b25edd= in sched_switch () 54 Thread 100280 (PID=3D0: kernel/zio_free_issue_4_10) 0xffffffff80b25ed= d in sched_switch () 53 Thread 100281 (PID=3D0: kernel/zio_free_issue_4_11) 0xffffffff80b25ed= d in sched_switch () 52 Thread 100282 (PID=3D0: kernel/zio_free_issue_5_0) 0xffffffff80b25edd= in sched_switch () 51 Thread 100283 (PID=3D0: kernel/zio_free_issue_5_1) 0xffffffff80b25edd= in sched_switch () 50 Thread 100284 (PID=3D0: kernel/zio_free_issue_5_2) 0xffffffff80b25edd= in sched_switch () 49 Thread 100285 (PID=3D0: kernel/zio_free_issue_5_3) 0xffffffff80b25edd= in sched_switch () 48 Thread 100286 (PID=3D0: kernel/zio_free_issue_5_4) 0xffffffff80b25edd= in sched_switch () 47 Thread 100287 (PID=3D0: kernel/zio_free_issue_5_5) 0xffffffff80b25edd= in sched_switch () 46 Thread 100288 (PID=3D0: kernel/zio_free_issue_5_6) 0xffffffff80b25edd= in sched_switch () 45 Thread 100289 (PID=3D0: kernel/zio_free_issue_5_7) 0xffffffff80b25edd= in sched_switch () 44 Thread 100290 (PID=3D0: kernel/zio_free_issue_5_8) 0xffffffff80b25edd= in sched_switch () 43 Thread 100291 (PID=3D0: kernel/zio_free_issue_5_9) 0xffffffff80b25edd= in sched_switch () 42 Thread 100292 (PID=3D0: kernel/zio_free_issue_5_10) 0xffffffff80b25ed= d in sched_switch () 41 Thread 100293 (PID=3D0: kernel/zio_free_issue_5_11) 0xffffffff80b25ed= d in sched_switch () 40 Thread 100294 (PID=3D0: kernel/zio_free_issue_6_0) 0xffffffff80b25edd= in sched_switch () 39 Thread 100295 (PID=3D0: kernel/zio_free_issue_6_1) 0xffffffff80b25edd= in sched_switch () 38 Thread 100296 (PID=3D0: kernel/zio_free_issue_6_2) 0xffffffff80b25edd= in sched_switch () 37 Thread 100297 (PID=3D0: kernel/zio_free_issue_6_3) 0xffffffff80b25edd= in sched_switch () ---Type to continue, or q to quit--- 36 Thread 100298 (PID=3D0: kernel/zio_free_issue_6_4) 0xffffffff80b25edd= in sched_switch () 35 Thread 100299 (PID=3D0: kernel/zio_free_issue_6_5) 0xffffffff80b25edd= in sched_switch () 34 Thread 100300 (PID=3D0: kernel/zio_free_issue_6_6) 0xffffffff80b25edd= in sched_switch () 33 Thread 100301 (PID=3D0: kernel/zio_free_issue_6_7) 0xffffffff80b25edd= in sched_switch () 32 Thread 100302 (PID=3D0: kernel/zio_free_issue_6_8) 0xffffffff80b25edd= in sched_switch () 31 Thread 100303 (PID=3D0: kernel/zio_free_issue_6_9) 0xffffffff80b25edd= in sched_switch () 30 Thread 100304 (PID=3D0: kernel/zio_free_issue_6_10) 0xffffffff80b25ed= d in sched_switch () 29 Thread 100305 (PID=3D0: kernel/zio_free_issue_6_11) 0xffffffff80b25ed= d in sched_switch () 28 Thread 100306 (PID=3D0: kernel/zio_free_issue_7_0) 0xffffffff80b25edd= in sched_switch () 27 Thread 100307 (PID=3D0: kernel/zio_free_issue_7_1) 0xffffffff80b25edd= in sched_switch () 26 Thread 100308 (PID=3D0: kernel/zio_free_issue_7_2) 0xffffffff80b25edd= in sched_switch () 25 Thread 100309 (PID=3D0: kernel/zio_free_issue_7_3) 0xffffffff80b25edd= in sched_switch () 24 Thread 100310 (PID=3D0: kernel/zio_free_issue_7_4) 0xffffffff80b25edd= in sched_switch () 23 Thread 100311 (PID=3D0: kernel/zio_free_issue_7_5) 0xffffffff80b25edd= in sched_switch () 22 Thread 100312 (PID=3D0: kernel/zio_free_issue_7_6) 0xffffffff80b25edd= in sched_switch () 21 Thread 100313 (PID=3D0: kernel/zio_free_issue_7_7) 0xffffffff80b25edd= in sched_switch () 20 Thread 100314 (PID=3D0: kernel/zio_free_issue_7_8) 0xffffffff80b25edd= in sched_switch () 19 Thread 100315 (PID=3D0: kernel/zio_free_issue_7_9) 0xffffffff80b25edd= in sched_switch () 18 Thread 100316 (PID=3D0: kernel/zio_free_issue_7_10) 0xffffffff80b25ed= d in sched_switch () 17 Thread 100317 (PID=3D0: kernel/zio_free_issue_7_11) 0xffffffff80b25ed= d in sched_switch () 16 Thread 100318 (PID=3D0: kernel/zio_free_intr) 0xffffffff80b25edd in sched_switch () 15 Thread 100319 (PID=3D0: kernel/zio_claim_issue) 0xffffffff80b25edd in sched_switch () 14 Thread 100320 (PID=3D0: kernel/zio_claim_intr) 0xffffffff80b25edd in sched_switch () 13 Thread 100321 (PID=3D0: kernel/zio_ioctl_issue) 0xffffffff80b25edd in sched_switch () 12 Thread 100322 (PID=3D0: kernel/zio_ioctl_intr) 0xffffffff80b25edd in sched_switch () 11 Thread 100326 (PID=3D0: kernel/dp_sync_taskq_0) 0xffffffff80b25edd in sched_switch () 10 Thread 100327 (PID=3D0: kernel/dp_sync_taskq_1) 0xffffffff80b25edd in sched_switch () ---Type to continue, or q to quit--- 9 Thread 100328 (PID=3D0: kernel/dp_sync_taskq_2) 0xffffffff80b25edd in sched_switch () 8 Thread 100329 (PID=3D0: kernel/dp_zil_clean_taskq_) 0xffffffff80b25edd= in sched_switch () 7 Thread 100330 (PID=3D0: kernel/dp_zil_clean_taskq_) 0xffffffff80b25edd= in sched_switch () 6 Thread 100331 (PID=3D0: kernel/dp_zil_clean_taskq_) 0xffffffff80b25edd= in sched_switch () 5 Thread 100332 (PID=3D0: kernel/dp_zil_clean_taskq_) 0xffffffff80b25edd= in sched_switch () 4 Thread 100333 (PID=3D0: kernel/zfs_vn_rele_taskq) 0xffffffff80b25edd in sched_switch () 3 Thread 100334 (PID=3D0: kernel/metaslab_group_task) 0xffffffff80b25edd= in sched_switch () 2 Thread 100335 (PID=3D0: kernel/metaslab_group_task) 0xffffffff80b25edd= in sched_switch () 1 Thread 100368 (PID=3D0: kernel/xn0 txq 0) 0xffffffff80b25edd in sched_= switch () --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Thu Nov 1 00:27:19 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3636910D373A; Thu, 1 Nov 2018 00:27:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660062.outbound.protection.outlook.com [40.107.66.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C1641735BC; Thu, 1 Nov 2018 00:27:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB1546.CANPRD01.PROD.OUTLOOK.COM (52.132.49.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1273.26; Thu, 1 Nov 2018 00:27:17 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b%3]) with mapi id 15.20.1294.021; Thu, 1 Nov 2018 00:27:17 +0000 From: Rick Macklem To: Andrew Vylegzhanin CC: "Rodney W. Grimes" , "freebsd-fs@freebsd.org" , "freebsd-infiniband@freebsd.org" Subject: Re: NFS + Infiniband problem Thread-Topic: NFS + Infiniband problem Thread-Index: AQHUbzV7uxRrUqx4EESM4oETGDUinKU2U4sAgAACL+GAAOXvgIABagv/gAElKYCAAEaZ4w== Date: Thu, 1 Nov 2018 00:27:16 +0000 Message-ID: References: <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB1546; 6:pegFwrocxH1W7ICIxfAFgOryQEvmi8NKbCWsQYdXUI+zKbmcULytGT25PLf7T4Ne2tdfzCHW5PTxkdReHDZGQQ6I0pPXiXO27vjvYKpSfZHZOvU1pT+i4UW0EpdGZhkr1bDkNqfidBzecZfP1KCLB4nvm63wOniKKXmza5IaCoVbup2gA78XiGBqXb/3e8w0Lyp3mQ8T9WMXYIMJX8fI4con4mtsuVGWxrcHphpeUk/1ONIssCw03vdg7NrUHDkcPCBzFmIss6PdBmEPsMOH5blt5wTr8+wyabeXRr9mV4aGPnE83w/QVZuyC63JfBywObmpkRnOfmge0gRRHtnLDVzL7i30KujV7RIQnJj3HRLbTOIhKdfhFkfCzP1PlTxwC4u5A8cXlo+lbI4qwI4/LAN4SNvTWyQxc8TMoxi7TBXsJh7agNR7mE0tsgkloXP8B0iKkoMMNvVfWt5QplCd/Q==; 5:6kzEtACuDzgk+evXMg0j0QwPiDYKONy3JCx6vA7W3Kl1+GOF3ukIN/YOxUpgHnUj1qRkCLLH2DhuU423QIjSHZzEsFfWwUiAkBJS3peVBTGopUeLMfdSocHBHxMHTvM8sDeN1t/99tSVuo+2exrVe6sY7/hcpKRZXcPJYQ2klyw=; 7:x5XABEz59c51KCXN/3J+F8PGaZtDkueGF6l+VotXWM0zP8Q5x5hyLcvr3MmxmBF85lWIeaPaL1g0KS13ZOC38VhkgTLR6BBNO6Ds/9qFb7efv91jffOdnhzRqV+1sCfIwRkGSq9yAt258wUTREwuug== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 863b1fdc-90eb-446b-38fe-08d63f90c740 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB1546; x-ms-traffictypediagnostic: YTOPR0101MB1546: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(3231382)(944501410)(52105095)(93006095)(93001095)(3002001)(10201501046)(148016)(149066)(150057)(6041310)(20161123560045)(20161123564045)(20161123558120)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB1546; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB1546; x-forefront-prvs: 0843C17679 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(346002)(396003)(39860400002)(376002)(366004)(189003)(199004)(71190400001)(6436002)(105586002)(33656002)(6506007)(5660300001)(76176011)(106356001)(74316002)(99286004)(68736007)(478600001)(6916009)(7696005)(2906002)(1411001)(71200400001)(305945005)(97736004)(229853002)(476003)(446003)(46003)(93886005)(6246003)(4326008)(8676002)(54906003)(81166006)(11346002)(14454004)(81156014)(39060400002)(9686003)(25786009)(102836004)(5250100002)(53936002)(256004)(55016002)(2900100001)(8936002)(786003)(316002)(86362001)(186003)(486006)(74482002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB1546; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: QJNnb3pcZUvn+43cV1LCDqwHT4QZWQ5o9abUCvhEosfA8SbhxWqgfY9yRJAh5u7L21c7eGC+ALBI4QMU5GAVPFyi71yjCI9kCMLIkzgsFsEpG2np0NJRcM1jHNygIExtJgBC4MCdfHH0VOhLFLfrY9mG/Hv89a8i4wA4i4BZsglwmKEP4qKx+jG2MUBN2oieClG6kdCYFGVbr5jtnyTQ7S6Z10mk9OqKARI0Zg+2VKWYsnKG0ycdoCY8pJc+HcQqppaUvZxK0WtzglMWmtWbgmBoLG58AYNkjoq3CmLHmMGmel2jVttbjXENSr4IvQvqjFGobfVPfXytRS/CBN3AfrGd+9wc1Lv2Nz4o2EiifaA= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 863b1fdc-90eb-446b-38fe-08d63f90c740 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Nov 2018 00:27:16.9281 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1546 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 00:27:19 -0000 Andrew Vylegzhanin wrote: [stuff snipped] >With this TCP settings same server serve NFS requests via 40G Ethernet on = multiply >clients with speed via 1G Eth ~ 105/110 MB/sec write/read. >Of course I'll try to change congestion algorithm, but I don't think that = will help. Yes, I doubt changing the congestion algorithm will make much difference. >Also need to test setup with infniband set from connected mode to datagram= mode. Are you using IPv6 by any chance? Why I ask is that there was a problem with IPv6 fragmentation re-assembly. = If InfiniBand is using a larger MTU than the ethernet, the switch would probab= ly fragment the IP datagram. If any fragment is lost (or fragmentation re-asse= mbly is broken), it isn't going to work well. "netstat -s | fgrep "fragments dropped after" will show you the count of fragments dropped after timeout. If this value i= s increasing, then fragmentation re-assembly is an issue. (Check on the recei= ving end. The server for writes.) You might want to post on freebsd-net@, since someone there might know more about InfiniBand (and someone over there will definitely know about the IPv= 6 fragmentation problem. rick= From owner-freebsd-fs@freebsd.org Thu Nov 1 01:46:42 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6489910DC6D5; Thu, 1 Nov 2018 01:46:42 +0000 (UTC) (envelope-from avv314@gmail.com) Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [IPv6:2a00:1450:4864:20::12b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CCDFD7723A; Thu, 1 Nov 2018 01:46:41 +0000 (UTC) (envelope-from avv314@gmail.com) Received: by mail-lf1-x12b.google.com with SMTP id n3-v6so13115951lfe.7; Wed, 31 Oct 2018 18:46:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8Jhwbm7kdrIBbWgP9UbP7r5BTcHsIlc7DP09ghEcGZA=; b=WFB6mqm1o1ujrgswAIfS4S0qnA8pZP3fCVXBN0KTvI2ZssoOAPuIkD/oBwpy6XZEMY kRCI+rSHANx2jTFnDWkNmqjcvTnmLr8zZ3ujCD72h4/yoktsXQgl+Wdxb1DytaS3nlFN aNxINAeTNQperi5JK2RvUrQMq+CRshyMeCH4h1eGWuPqi2lM/V4pGtQsPUAtBfWQ/Aye +8YKFoZxeqmKB914aci7NTrmI8ndjeSOvvG1UHypTqp2g+Ftqm3yjlsXDrfPXI5BUbWI POz3gpGkPH6tqvDFgw5rRJXODAZXnc/9bXZG5MdLyuxu6/K1tZH9uozP4fmV26gbX3hG GPAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8Jhwbm7kdrIBbWgP9UbP7r5BTcHsIlc7DP09ghEcGZA=; b=ph1dCUjUzTyKvxBgsxjy5qwLFbtzDM6v+avoK4cyQ7julBbSFUuv+7hlkex7wL8Lov jT0TSl+iW0jhFI7v+XlwA8TAy9qigtxQ1V7veybVJfdiakX276hYSncn7RbtbKxae+V9 eHOvkAtpuUE4spKqeIbatqhmWM/5wyNNz4aDUUIkr4BIbWNRcdm9+DxWmo3jt9sEA0fS QiqS8oMnIza9LHgvMTX6zSYZJl3ZL+c4z1bdv03qYtJ1zc7eLoYevJqU0gJ796qSIb+s 6pwBjrTv5ojJg7dnmpSWS9bOP/8vIHtQwDucvlwYBS7KYg6hEiwVDS8/YD1AnyAj+qbb 5kIA== X-Gm-Message-State: AGRZ1gK03gnPzI9/FBQsfIemxhFhJg8t5tb5AmCO05jXHT5iui0qMc2X CoqWrTy2Nh8RFzgQFxN/1D3T7qMzBga6GKvD26M= X-Google-Smtp-Source: AJdET5cqwOqSaY9mGe2n8T+BGKqrJX69m24iYN6LYk3eK1XvOSXkzerQmTfvERCIp3Da0uy4an5Xd+6HubtnUdMEs0o= X-Received: by 2002:a19:a84e:: with SMTP id r75mr3063086lfe.18.1541036800224; Wed, 31 Oct 2018 18:46:40 -0700 (PDT) MIME-Version: 1.0 References: <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> In-Reply-To: From: Andrew Vylegzhanin Date: Thu, 1 Nov 2018 04:46:28 +0300 Message-ID: Subject: Re: NFS + Infiniband problem To: rmacklem@uoguelph.ca Cc: "Rodney W. Grimes" , freebsd-fs@freebsd.org, freebsd-infiniband@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 01:46:42 -0000 =D1=87=D1=82, 1 =D0=BD=D0=BE=D1=8F=D0=B1. 2018 =D0=B3. =D0=B2 3:27, Rick Ma= cklem : > >Also need to test setup with infniband set from connected mode to datagram mode. > Are you using IPv6 by any chance? No, I don't. > Why I ask is that there was a problem with IPv6 fragmentation re-assembly. If > InfiniBand is using a larger MTU than the ethernet, the switch would probably > fragment the IP datagram. If any fragment is lost (or fragmentation re-assembly > is broken), it isn't going to work well. > "netstat -s | fgrep "fragments dropped after" [root@nas1 ~]# netstat -s | fgrep "fragments dropped after" 0 fragments dropped after timeout 0 fragments dropped after timeout > will show you the count of fragments dropped after timeout. If this value is > increasing, then fragmentation re-assembly is an issue. (Check on the receiving > end. The server for writes.) > > You might want to post on freebsd-net@, since someone there might know more > about InfiniBand (and someone over there will definitely know about the IPv6 > fragmentation problem. I'm cc'ed to freebsd-infiniband@ also, but silence. > > rick -- Andrew From owner-freebsd-fs@freebsd.org Thu Nov 1 08:11:15 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B09F510F078D for ; Thu, 1 Nov 2018 08:11:15 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 240286F724 for ; Thu, 1 Nov 2018 08:11:14 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-wm1-f41.google.com with SMTP id l2-v6so594339wmh.3 for ; Thu, 01 Nov 2018 01:11:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=u1K8EpMYr76ZMTGNvpJAqYleHiLEe2UiXgQaHmr0nRQ=; b=ftxG/m3MaPXqLYE4tU9X2cePxSly8Mb5f/hF3R93OjlYY4UIgyAlWm+b9/MX+xf0r/ BS38gejtqwPV9ZNzvxW5/DJL/7tUe7JzoNLIbMF3AB9cOJvtHz7+HeNvROgKtBMIDPDG 5URQ5XonuXtfwASTZstQNS5heUHGCewOoS9DR03vhUfgA7ILXqJ1hw/PiIgW4XKtDHJ/ MR9aPXO7OmMp/r54iO22YaVnYDEJJi8QbCWn9c6V7BJw57lvrMo5mU16ib2/7bSZxPnu IMiMswNbaexjTSKCypJemFFJgkggBS/BiYRji1V8pbscPshrQ4Wr35wwQ4dFMC4qihrU 2//A== X-Gm-Message-State: AGRZ1gJ0gmEnD+X4tjIlyr8Blfnvw99TY2dYXg+rzgKfBEEhN/Y1Yw9L PFuBGIiEgBzdPRy4S58PYt75Juvx X-Google-Smtp-Source: AJdET5cIGFUqnvKwuxsxNGS4cWikujVmQUFtUq2yUF32Gu8UnE1DM58KVo4rq1v+aUtjTmDf7mPHyQ== X-Received: by 2002:a1c:c016:: with SMTP id q22-v6mr4718459wmf.151.1541056635427; Thu, 01 Nov 2018 00:17:15 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id q17-v6sm1930001wrw.17.2018.11.01.00.17.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Nov 2018 00:17:14 -0700 (PDT) Subject: Re: How to fill in the fsid for file systems? To: Rick Macklem , Konstantin Belousov Cc: FreeBSD Filesystems , Josh Paetzel References: <20181030012240.GM5335@kib.kiev.ua> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Thu, 1 Nov 2018 09:17:13 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 08:11:16 -0000 On 31/10/2018 17:50, Rick Macklem wrote: > I alluded to this option in my last post. I think both fsids will need to be in the > mount structure, since finding the correct mount point via the fsid is the first > step in translating a file handle to a vnode. (After that VFS_FHTOVP() does the rest.) > > And I think it will get messy. A couple of examples. > There are some syscalls that use file handles. fhopen(2), fhstat(2), fhstatfs(2), > getfh(2), lgetfh(2) > I had assumed that the "file handles" used by these should be the same file > handles that NFS uses (ie. file handles are a generic VFS component and not NFS > specific) but I can see the argument either way. I actually don't know what > apps/utilities use fhopen/fhstat/fhstatfs, but getfh(2) is used by mountd. > Since mountd uses getfh(2) to get a file handle for NFS mounts, it needs to > return the NFS fsid to keep the old binaries working, even if you add a > new getnfsfh(2) function for mountd to use. > - Now you have some file handle system calls using file handles with the NFS > fsid in them and some using file handles with the "true" fsid in them. > (Sounds confusing to me.) > > Since the first step in turning a file handle into a vnode is looking up the fsid > in the mount list, if you had two fsids, I think they both would end up in the > mount structure so that lookup could be done easily. > This lookup is normally done by vfs_busyfs() { that appears to be the only use > of vfs_busyfs() in sys/kern. I haven't looked in the various file systems }. > With two fsids, you need two functions and need to be careful which one you use. I originally thought about having a separate filesystem list for NFS that would contain only exported filesystems. But I suspect that managing it could be problematic. An alternative idea is to use osd(9) framework to attach NFS specific data to struct mount without modifying the structure and without exposing the NFS data to other consumers. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Thu Nov 1 11:08:53 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD4C810E9ABF for ; Thu, 1 Nov 2018 11:08:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 680796FE0D for ; Thu, 1 Nov 2018 11:08:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 2C3B010E9AAA; Thu, 1 Nov 2018 11:08:53 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B24F10E9AA8 for ; Thu, 1 Nov 2018 11:08:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B18996FE0B for ; Thu, 1 Nov 2018 11:08:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id E827CC987 for ; Thu, 1 Nov 2018 11:08:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wA1B8pr2012676 for ; Thu, 1 Nov 2018 11:08:51 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wA1B8pA6012675 for fs@FreeBSD.org; Thu, 1 Nov 2018 11:08:51 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 217686] zfs_enable=YES causing vm_fault: pager read error, pid 1 (init) Date: Thu, 01 Nov 2018 11:08:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: bblister@gmail.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 11:08:54 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217686 BB Lister changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bblister@gmail.com --- Comment #1 from BB Lister --- I upgraded from 11.1 to=20 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018=20= =20=20=20 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 and =CE=99 have the same problem. As a workarround, I now have zfs_enable=3D"NO" on /etc/rc.conf and on crontab: @reboot ( sleep 100; /etc/rc.d/zfs onestart ) > /den/null 2>&1 Now my system is back stable. Of course, I would like to have zfs_enable=3D"YES" but this causes an infin= ite number of=20 vm_fault: pager read error, pid 1 (init) If you would like more info please let me know how can I help. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Nov 1 12:17:47 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56AC910F0D23 for ; Thu, 1 Nov 2018 12:17:47 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id D1CA67361C for ; Thu, 1 Nov 2018 12:17:46 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id E04ED144 for ; Thu, 1 Nov 2018 15:17:45 +0300 (MSK) To: FreeBSD Filesystems Reply-To: lev@FreeBSD.org From: Lev Serebryakov Subject: boot2 / loader can not check superblock checksum and refuse to boot, but fsck_ffs doesn't see problems Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Thu, 1 Nov 2018 15:17:38 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="1cwqErtmxte5NtSHi5LKjm1CoBVrOAqI7" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 12:17:47 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --1cwqErtmxte5NtSHi5LKjm1CoBVrOAqI7 Content-Type: multipart/mixed; boundary="q2AtePnIb2eUryp8qRnFQWJ7GSQbVWBYL"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: FreeBSD Filesystems Message-ID: Subject: boot2 / loader can not check superblock checksum and refuse to boot, but fsck_ffs doesn't see problems --q2AtePnIb2eUryp8qRnFQWJ7GSQbVWBYL Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable I have NanoBSD image built from head/r339933, which passes all checks on other system: "fsck_ffs" passes, it could be mounted without problems, everything could be read. But when I try to boot from it, boot2 (or loader?) complains about failed superblock checksum several times (for each copy?), like this: Superblock check-hash failed: recorded check-hash 0xcfa0ae67 !=3D compute= d check-hash 0x21993fb ALL these messages contains SAME "recorded" and "compuded" hashes, so it doesn't look like corrupted flash. Looks like boot2/loader wrongly calculate hash... --=20 // Lev Serebryakov --q2AtePnIb2eUryp8qRnFQWJ7GSQbVWBYL-- --1cwqErtmxte5NtSHi5LKjm1CoBVrOAqI7 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlva7uJfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R4+y7Q/+JpD32Cr2tce70/3SVpwAdmnMXR8ipGKVRjZU0szfSKywbQ11ejm5sHQ9 OfVqKAk6LcTsoF6HcywttQP4PHZnc3sVLH8eKhff7Lk8fh/DVs06hQTKVKNcabwR EqbKvlhJ7JzJ7aBGc/c7CX7+tQFU9HdS9SKftmlMJmPvozvP1p8zwwQn8E16EvUL dgGJ/qdFeb2q1CGILKXOEm1+tSTRznLxxZEdeDSkZtxzNNfmtiqqeq4u0NB7ktW1 KZvp8fC+H/CNjPDiMio/+p8X98oEoCIh1vwpYcnCffIXABxNZ5DRbwK1HTgr0e0p 1oXPyQC+lhpBASIHTKLGawlF+lITXMpW6FSP+kHRDGmTEGEKarymUh42n+7MNTht sXyFn77a8etOXkyAeLPAWUgEsUfDSxlxfYbXKXk9wKsMJr8loCX9QGxu6yemXfs/ bsB60NYhjQemLpubf5XrC9X790esCzPLN67gIokpeQQXNEzLkMRdDkI1AjY3K/6f lRkDnxQ24CD61xxTwTHB2gyu/OoHNqDFDryrx2XDhJgN/XA5Ul/uONE1FgMoqh/e Nx4tJkpkpLs4CivT9pmYkezB0tYVptD+0oNo/+vORn7VuGbPAmW5hwHiCxm2Je9d SikF65yAoCyCvcsEcF2bg+4flbPeUBDir8mRhzv+1IcOw/nWxHk= =VefH -----END PGP SIGNATURE----- --1cwqErtmxte5NtSHi5LKjm1CoBVrOAqI7-- From owner-freebsd-fs@freebsd.org Thu Nov 1 14:31:19 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB55F10F4465 for ; Thu, 1 Nov 2018 14:31:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4866978238 for ; Thu, 1 Nov 2018 14:31:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 0AB0110F4464; Thu, 1 Nov 2018 14:31:19 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EBD9410F4463 for ; Thu, 1 Nov 2018 14:31:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8436478235 for ; Thu, 1 Nov 2018 14:31:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id CC943E544 for ; Thu, 1 Nov 2018 14:31:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wA1EVHuL094170 for ; Thu, 1 Nov 2018 14:31:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wA1EVHj8094169 for fs@FreeBSD.org; Thu, 1 Nov 2018 14:31:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 217686] zfs_enable=YES causing vm_fault: pager read error, pid 1 (init) Date: Thu, 01 Nov 2018 14:31:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 14:31:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217686 --- Comment #2 from Andriy Gapon --- (In reply to BB Lister from comment #1) Check mountpoint properties of all your ZFS filesystems to see if any of th= em gets mounted over /. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Nov 1 15:54:33 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98F1D10F6E86 for ; Thu, 1 Nov 2018 15:54:33 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A6347CF2C; Thu, 1 Nov 2018 15:54:32 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id wA1FsU7R018656; Thu, 1 Nov 2018 16:54:30 +0100 (CET) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 6C888F4B; Thu, 1 Nov 2018 16:54:30 +0100 (CET) Subject: Re: boot2 / loader can not check superblock checksum and refuse to boot, but fsck_ffs doesn't see problems To: lev@freebsd.org, FreeBSD Filesystems References: From: Harry Schmalzbauer Organization: OmniLAN Message-ID: Date: Thu, 1 Nov 2018 16:54:29 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Thu, 01 Nov 2018 16:54:30 +0100 (CET) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 15:54:33 -0000 Am 01.11.2018 um 13:17 schrieb Lev Serebryakov: > I have NanoBSD image built from head/r339933, which passes all checks > on other system: "fsck_ffs" passes, it could be mounted without > problems, everything could be read. > > But when I try to boot from it, boot2 (or loader?) complains about > failed superblock checksum several times (for each copy?), like this: > > Superblock check-hash failed: recorded check-hash 0xcfa0ae67 != computed > check-hash 0x21993fb > > ALL these messages contains SAME "recorded" and "compuded" hashes, so > it doesn't look like corrupted flash. > > Looks like boot2/loader wrongly calculate hash... I think I've had the same / a similar problem some dozend moons ago, which I could track down to a 'newfs' induced problem.  'newfs'ing the same partition [/md(4) deivce  without partitions in my case] multiple times (4 times) triggered that problem, as far as I remember. Can't find any notices :-( And haven't investigated further :-( Maybe it's worth checking if your problem is also related to the number of newfs runs – unfortunately I can't remember if it really was boot2 where my – at least very similar – error came from. Other symptoms match – no problem by fsck_ffs(8) found. -harry From owner-freebsd-fs@freebsd.org Thu Nov 1 16:10:06 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19BE910F74EB for ; Thu, 1 Nov 2018 16:10:06 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id 975AB7D897 for ; Thu, 1 Nov 2018 16:10:05 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 383621C8; Thu, 1 Nov 2018 19:10:04 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: boot2 / loader can not check superblock checksum and refuse to boot, but fsck_ffs doesn't see problems To: Harry Schmalzbauer , FreeBSD Filesystems References: From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Thu, 1 Nov 2018 19:10:03 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="w7ENF4mcIfAdQseAw3S3ZeM93d4X15YBP" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 16:10:06 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --w7ENF4mcIfAdQseAw3S3ZeM93d4X15YBP Content-Type: multipart/mixed; boundary="m5Izw3w6yiHx3v7ciQHs1ZI2hR7Va4tgY"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Harry Schmalzbauer , FreeBSD Filesystems Message-ID: Subject: Re: boot2 / loader can not check superblock checksum and refuse to boot, but fsck_ffs doesn't see problems References: In-Reply-To: --m5Izw3w6yiHx3v7ciQHs1ZI2hR7Va4tgY Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 01.11.2018 18:54, Harry Schmalzbauer wrote: > I think I've had the same / a similar problem some dozend moons ago, > which I could track down to a 'newfs' induced problem.=C2=A0 'newfs'ing= the > same partition [/md(4) deivce=C2=A0 without partitions in my case] mult= iple > times (4 times) triggered that problem, as far as I remember. > Can't find any notices :-( And haven't investigated further :-( > Maybe it's worth checking if your problem is also related to the number= > of newfs runs =E2=80=93 unfortunately I can't remember if it really was= boot2 > where my =E2=80=93 at least very similar =E2=80=93 error came from. Oth= er symptoms match > =E2=80=93 no problem by fsck_ffs(8) found. I'm using same md(4)-bakcing file for many-many times, yes (much more than 4), but it is first time I got this problem. Ok, I'll re-create backing file on image builder and try to create image again. But it looks like very strange bug. --=20 // Lev Serebryakov --m5Izw3w6yiHx3v7ciQHs1ZI2hR7Va4tgY-- --w7ENF4mcIfAdQseAw3S3ZeM93d4X15YBP Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKSBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlvbJVtfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R49yhA/43idsfFnANnn3Zi1Tsu5CWIKsCFOXWQulJdHVdw6LS9wfgiVZMAmRmuIA dHGkRwR2AjhfRLbOgWn7ukZo6w3WFDOyZZi9rdjy+fuzZQ39gwFSH+vKw0qK9RZx Gf+r9R58sB9ZOxQb6MB+QIdh+28moZm7lPMptJ+RgwPbiAVtiq1HsyIY0Z8qK+Y6 95lDLj++szJJidIHya6vCZcXXmtEIpcGc3bxWulu8TuQSe4w/9tJwtKy+d/d9BU2 n5Y/O5pxYX4PfKii1lG3zD0PMfI8KnC3jCBBi62NeXnZBixV4dWO/meK4kVXzax9 dJ8mOPt4xlBZ1EWgiclxYSYLcqAUa3pvNE/LHIiDcK86UcFbxXQXFmBMxujb2Ezn ns1JaqWiyfwDI1rbbt9fRhZNO/Cerx5sPAV1pIhaWk4qCLh0fdTSXyNVU1KuBRO3 NfKSa2WjvL2KkNX/xMSO6jOMwC9g0VI2Z2PhfPFsDcDEmhCExrIKzszsXeY27fat M1m7eQZLCRDbf1Cl76I+Ja711KTrwT39K55+s86ehC/XRC91WGCyVsWC89EHXF2L EYMbEmWoZMXzZ2QauqiVSWIgC/hGXpBzVrM7JzDEPrOvU30lXjVjzm3GNr0lPiKJ eCmcRwyxjiayNj2Kv9cQJG4GpqBu9JPhEWZ/ZfqvTQDu3p8oTw== =H4SG -----END PGP SIGNATURE----- --w7ENF4mcIfAdQseAw3S3ZeM93d4X15YBP-- From owner-freebsd-fs@freebsd.org Fri Nov 2 02:24:21 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2F0F10E1BBC for ; Fri, 2 Nov 2018 02:24:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660067.outbound.protection.outlook.com [40.107.66.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B87B71FA3; Fri, 2 Nov 2018 02:24:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB1212.CANPRD01.PROD.OUTLOOK.COM (52.132.43.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1294.21; Fri, 2 Nov 2018 02:24:19 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b%3]) with mapi id 15.20.1294.024; Fri, 2 Nov 2018 02:24:18 +0000 From: Rick Macklem To: Andriy Gapon , Konstantin Belousov CC: FreeBSD Filesystems , Josh Paetzel Subject: Re: How to fill in the fsid for file systems? Thread-Topic: How to fill in the fsid for file systems? Thread-Index: AQHUb557F1RNqdJl0kuwS+F5J/DjsKU2/twAgAGdwkKAAGQrgIAAehmGgAELs4CAATs6uA== Date: Fri, 2 Nov 2018 02:24:18 +0000 Message-ID: References: <20181030012240.GM5335@kib.kiev.ua> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB1212; 6:DNe8KQiJmtlpbSOoUitwB48nGn6Z+Uq3zgOj0OcKgD4PEXHEHxEioFunMD0QiqZol0My2bHxvs76dt+JykUrdQYJtEQFwXDBPtPuAyqsa6AVCLhmSLoGGJBmOR3JiiiFI0oYpUhh/dqTirBmOK7q+7hl5seUyNZy4FTdZN2PPMMukBuio8Z4mF4yIkhlQl7wqw+lkMUziptXmMfBHXZrXPvJ+2zuJJyJo2PcMkHF4jI36KhikUqPmjuEql7+jP53bakNTLAvu9iD6eXGE40JAIB6kwfESjoy5FGAEYHSKyxP0aMXNbPp9GOhDxOdlkyhSjXknkwJKuc2WPB3MBr32FuG1eF+O9Y/tZbQWzCgQ7Abm2bQZDSvNjw7E0G2dqip8ar4RgucCMBv8xCwuwSdLhimi3HpxMgaTTjbgEbkcskP3BVcSAq39neOsvDiHrSg2NRc7lTk9KRm50FZJ2cptw==; 5:k4uAtptzvZ3m0HPADIy1cLus+z62IV8geegQmjoX3DJQNgRGfbaYkvzGV3STl94wqet7FqmHTUG0mMNL1PpWhBbHaFLJqck+H6617LyWYQTxj9A1vx3SxRrNhLPK0Mi1+L3rSljekLHn6W3esoikXX9/ihyaE7ePCTfqfQud/AQ=; 7:WnnOD2dT50HdCI/xEI6eTqWFO3fVMuMUUueXDyWd8os9IMQjfa3qhCHScapHWpFroYvYoAIA8mlS0AIoofLByvEM2ri1y+HcHwSozNmOL9JpyxiqQ2YL658+k9fn5Ag+LgDVkafuTpVcY+rnUjaf5w== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 150e811f-1c6c-4b32-5c05-08d6406a4b13 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB1212; x-ms-traffictypediagnostic: YTOPR0101MB1212: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3002001)(3231382)(944501410)(52105095)(148016)(149066)(150057)(6041310)(20161123560045)(20161123564045)(20161123562045)(20161123558120)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB1212; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB1212; x-forefront-prvs: 08444C7C87 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(136003)(39860400002)(366004)(376002)(346002)(199004)(189003)(86362001)(106356001)(229853002)(305945005)(256004)(186003)(33656002)(105586002)(5024004)(14444005)(46003)(74316002)(6246003)(9686003)(74482002)(55016002)(39060400002)(97736004)(2906002)(5660300001)(316002)(81156014)(81166006)(7696005)(8676002)(93886005)(6436002)(8936002)(786003)(6506007)(68736007)(102836004)(25786009)(53936002)(4326008)(2900100001)(446003)(11346002)(71200400001)(14454004)(486006)(71190400001)(476003)(110136005)(478600001)(54906003)(76176011)(99286004); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB1212; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: UM2/FFORNIkY5XgNCv1nRcQvxG92Nm6A5Q602wC8/Yx4Fu/erIkf8aqrSokRZ95CCiiLKKcT0Sltht7O/+I03GKpltL1A38Akt346ShG5WmTJFDw9I4uypUh5bQHojvdZZ57ma0jZF1rRIWc975VoBqe5mr2MMBdqbfjkEXrtVwOYtwwfhXrzau5kGUsaqYcwDhw44PN36b2Ib57xLj5FDQFoijBEgUuwjnNszvHGPUNijBpVG48HkTRosxMhO9hs5gwC7ihPS9Ps3JOBBkRtKXRywvbhYE5paZEVYFj3oi0gyC7DvGc6vnfC0VRQaJH+31/LRMOThqdxI3mRc/EQjOr7fpASd+U5xIm+SOMDeQ= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 150e811f-1c6c-4b32-5c05-08d6406a4b13 X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Nov 2018 02:24:18.9462 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1212 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 02:24:21 -0000 Andriy Gapon wrote >On 31/10/2018 17:50, Rick Macklem wrote: >> I alluded to this option in my last post. I think both fsids will need t= o be in the >> mount structure, since finding the correct mount point via the fsid is t= he first >> step in translating a file handle to a vnode. (After that VFS_FHTOVP() d= oes the rest.) >> >> And I think it will get messy. A couple of examples. >> There are some syscalls that use file handles. fhopen(2), fhstat(2), fhs= tatfs(2), >> getfh(2), lgetfh(2) >> I had assumed that the "file handles" used by these should be the same f= ile >> handles that NFS uses (ie. file handles are a generic VFS component and = not NFS >> specific) but I can see the argument either way. I actually don't know w= hat >> apps/utilities use fhopen/fhstat/fhstatfs, but getfh(2) is used by mount= d. >> Since mountd uses getfh(2) to get a file handle for NFS mounts, it needs= to >> return the NFS fsid to keep the old binaries working, even if you add a >> new getnfsfh(2) function for mountd to use. >> - Now you have some file handle system calls using file handles with the= NFS >> fsid in them and some using file handles with the "true" fsid in them. >> (Sounds confusing to me.) >> >> Since the first step in turning a file handle into a vnode is looking up= the fsid >> in the mount list, if you had two fsids, I think they both would end up = in the >> mount structure so that lookup could be done easily. >> This lookup is normally done by vfs_busyfs() { that appears to be the on= ly use >> of vfs_busyfs() in sys/kern. I haven't looked in the various file system= s }. >> With two fsids, you need two functions and need to be careful which one = you use. > >I originally thought about having a separate filesystem list for NFS that = would >contain only exported filesystems. But I suspect that managing it could b= e >problematic. > >An alternative idea is to use osd(9) framework to attach NFS specific data= to >struct mount without modifying the structure and without exposing the NFS = data >to other consumers. I have two comments. The first is related to the above and the second is a = big picture question related to doing this. Specifically w.r.t. the above. I probably rambled and didn't make what I wa= s trying to say clear, so I'll try again... - getfh(2) returns a file handle that is used for NFS, but is also used for= system calls (fhopen(2), fhstat(2) and fhstatfs(2)) that are not related to NFS. --> A file handle isn't really NFS specific, although NFS is the big user= of it. If the above is correct, then it seems that there should only be one kind o= f file handle. --> Since the fsid is a key part of a file handle, one kind of file handle = implies one kind of fsid. I just think trying to create two kinds of fsid and two kinds of file handl= e would make the code confusing and unnecessarily complex. On the big picture..the more I look at this, the less obvious the usage of = setting an fsid other than what the file system sets seems to become. I thought the= usage of this was to ensure that when a file system was moved (just moving the st= orage hardware or a block by block cloning) to a different system, the same fsid = would be used so that file handles wouldn't change. Note that a file handle also = has a fileid number (think I-node#) in it, so the file system "move" can't chan= ge the file system's structure. - UFS keeps the fsid in the superblock, so moving the file system should re= tain the same fsid value. (There might be an obscure case of another file syst= em having the same fsid, so it has to change. However, since newfs(8) uses t= he creation time and a random# to create it, a conflict seems highly unlikel= y.) - This leaves ZFS. I know nothing about ZFS, but a glance at the code seems to indicate it normally comes out of the "physical dataset" field ds_fsi= d_guid. --> Does this mean it changes when "moved" or stays the same, like UFS? If the answer is "stays the same", I don't see that being able to manually = set the fsid is useful? (There are file systems like cd9660 and msdosfs where the fsid will change, but I don't see those used a storage for NFS servers where they might be moved/cloned/=85) Maybe someone can explain when it would be useful for FFS (or not)? rick From owner-freebsd-fs@freebsd.org Fri Nov 2 06:11:41 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6DAB010EA1C5 for ; Fri, 2 Nov 2018 06:11:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0984777CF1 for ; Fri, 2 Nov 2018 06:11:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id BF46110EA1C4; Fri, 2 Nov 2018 06:11:40 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9D59710EA1C3 for ; Fri, 2 Nov 2018 06:11:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2419E77CEE for ; Fri, 2 Nov 2018 06:11:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5BC35167C0 for ; Fri, 2 Nov 2018 06:11:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wA26Bdq3080887 for ; Fri, 2 Nov 2018 06:11:39 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wA26BdVV080882 for fs@FreeBSD.org; Fri, 2 Nov 2018 06:11:39 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 217686] zfs_enable=YES causing vm_fault: pager read error, pid 1 (init) Date: Fri, 02 Nov 2018 06:11:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: bblister@gmail.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 06:11:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217686 --- Comment #3 from BB Lister --- My /etc/fstab is # Device Mountpoint FStype Options=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 Dump Pass# /dev/ada0s1b none swap sw=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 0 0 /dev/ada0s1a / ufs ro 1= =20=20=20=20=20=20 1 /dev/ufs/b5home /home ufs rw,nosuid,noexec=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20 2 2 /dev/ufs/b5usr /usr ufs ro=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 2 2 /dev/ufs/b5var /var ufs rw,nosuid=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 2 2 md /tmp mfs rw,noatime,nosuid,noexec,-s96m=20= =20=20=20=20=20=20=20=20 0 0 md /var/run mfs rw,nosuid,noexec,-s4m,-p=3D777=20= =20=20=20=20=20=20=20=20=20=20 0 0 /dev/acd0 /cdrom cd9660 ro,noauto=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 0 0 tmpfs /var/db/ramdisk tmpfs rw,nosuid,noexec,mode=3D777 0= =20=20=20=20=20=20 0 My ZFS filesystems are all mounted at tank/=20 mount | grep zfs tank on /tank (zfs, local, noatime, nfsv4acls) tank/_Programs on /tank/_Programs (zfs, local, noatime, nfsv4acls) tank/jails on /tank/jails (zfs, local, noatime, nfsv4acls) tank/tmp on /tank/tmp (zfs, local, noatime, nosuid, nfsv4acls) tank/virtualbox on /tank/virtualbox (zfs, local, noatime, nfsv4acls) tank/.backupdir on /tank/.backupdir (zfs, local, noatime, nfsv4acls) I can assume that this seems like a race error, where first the ufs directo= ries should be mounted (after fsck) and then the ZFS. Now I assume (perhaps I am wrong) that zfs tries to get mounted before any UFS filesystem is mounted a= nd this causes that problem. If the UFS directories are mounted first then I can start zfs whithout prob= lem, as I have shown in my previous post. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 2 06:45:19 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF73410EAC2B for ; Fri, 2 Nov 2018 06:45:18 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C51E78DCD for ; Fri, 2 Nov 2018 06:45:18 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f178.google.com with SMTP id a28-v6so789234ljd.6 for ; Thu, 01 Nov 2018 23:45:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=JyES6HYeeV66zm68LL7Xz3BzjispXlLQYQdPucyP7+U=; b=XyDAKotol+dln5t9qYX+PrULVkenxlK+xplx3EKOXPCAyXqy1DrETvLtXP0bNdICrS KGMDXlRX4k9lh61vtJTmkGVf3367RNxgzkSFv7rQlQ2QuMfegwXH9NwDYhcc/hs+NAEd lAt3qPY8S1JCcdNC3zFJxiy6bvLU08W/1Yfo5z5RxODYxl6iN9f/+zDeaPdb6rnK3F+H OdiYWUAc1y0fmbN1G/CtOJypZRohiztSjPne9FQ/TSBrlqCjgGl185JjpAuZFSs3Nr39 KPTdxobgPQtW4oeuRh+9ew6fI1hiGLSaQ9Q+IrmgijceBz7TGAL+w02RxHeQ1MRFnBRW khew== X-Gm-Message-State: AGRZ1gJ29uY3vWFHvoR+fGCEBlvgP8zc5ZtJoza1cWhiAWW7qkA46m0+ YBcFu3pxgf77/ciNa1kjej4= X-Google-Smtp-Source: AJdET5cRgGZdfgYypmS1XGsBQvqbRpx0sSHkbWyTX0B/AKRsAuxjrjfBz0ZAXb4lsP2LdxhfX58toA== X-Received: by 2002:a2e:5dd9:: with SMTP id v86-v6mr7145205lje.86.1541141111205; Thu, 01 Nov 2018 23:45:11 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id q2-v6sm4950982lfa.37.2018.11.01.23.45.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Nov 2018 23:45:10 -0700 (PDT) Subject: Re: How to fill in the fsid for file systems? To: Rick Macklem , Konstantin Belousov Cc: FreeBSD Filesystems , Josh Paetzel References: <20181030012240.GM5335@kib.kiev.ua> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <4269ff9a-8e11-f441-fbb5-b23a6d8e253b@FreeBSD.org> Date: Fri, 2 Nov 2018 08:45:09 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 06:45:19 -0000 On 02/11/2018 04:24, Rick Macklem wrote: > I have two comments. The first is related to the above and the second is a big > picture question related to doing this. > > Specifically w.r.t. the above. I probably rambled and didn't make what I was > trying to say clear, so I'll try again... > - getfh(2) returns a file handle that is used for NFS, but is also used for system > calls (fhopen(2), fhstat(2) and fhstatfs(2)) that are not related to NFS. > --> A file handle isn't really NFS specific, although NFS is the big user of it. > If the above is correct, then it seems that there should only be one kind of file handle. > --> Since the fsid is a key part of a file handle, one kind of file handle implies one > kind of fsid. > I just think trying to create two kinds of fsid and two kinds of file handle would > make the code confusing and unnecessarily complex. As as far as I understand, VFS calls like, say, VOP_VPTOFH fill only the file ID portion of the file handle, fh_fid. fh_fsid is left to a caller, so potentially we could already have a discrepancy there (but we don't). Also, I do not think that NFS uses getfh(2), but I could be wrong. I think that NFS, being in kernel, directly uses VFS interfaces like the mentioned VOP_VPTOFH. I am not sure if getfh(2) has any requirement that its result should be compatible with anything NFS. My impression is that it should only be usable for fhopen(2) and the like. But I can be wrong again. So, if any of my assumptions is wrong, then you are right, of course, and I should withdraw my argument. Thanks! -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Nov 2 07:01:27 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C18C510EB2A2 for ; Fri, 2 Nov 2018 07:01:26 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf1-f66.google.com (mail-lf1-f66.google.com [209.85.167.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 276DB794CA for ; Fri, 2 Nov 2018 07:01:26 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf1-f66.google.com with SMTP id p17so628831lfh.4 for ; Fri, 02 Nov 2018 00:01:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=pnITwGwFcjlKW3gM3tOX8tmcii6TUVmOtYbaUXcATFI=; b=HWZU/XPw9LMJDyN7eoyBm8eWObUSrKdG/aPr/5OrbmCJPfAygmdo/4g71G5rqD8V/K B3LOaBNBqF4IglZSNWNc2pEDY81CeuzitKZKq57TQHBH3NMromMAgiW1uqc2enltvetu Mg4OOijAwo4oLKFVgaiYbQONeZSRSL1/i+W8Q7xDtReOcIFQSz1OPmboCBjJaYYMDjQE M6mZYenVfmc9ZC9S+GRdw20Eg2pyvtXneLMlwbc/D2GdwyWazw6JfAEoJ39RdaGNJ2+7 TreM7UStUt7zq69cu0scofb94K3cPi4XpBwiUYilwBGmWSkEv+SxYpejn2MmLVgLsOL8 v6ow== X-Gm-Message-State: AGRZ1gIxzDGaycjcudWXguH0R2RdJHA8NOhKnclsP0ul82rdqXEYgoOs sLbLaPSMS/IcGuGRtOQIEzs= X-Google-Smtp-Source: AJdET5fnUxZ6LUENOQoh9FnIwJspRfN+khBg3QXDw0foEDUVtua76dW94a+37FU27rxVz/KDRC189w== X-Received: by 2002:a19:d5:: with SMTP id 204mr5709521lfa.116.1541141603905; Thu, 01 Nov 2018 23:53:23 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id u21-v6sm179214lju.46.2018.11.01.23.53.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Nov 2018 23:53:22 -0700 (PDT) Subject: Re: How to fill in the fsid for file systems? From: Andriy Gapon To: Rick Macklem , Konstantin Belousov Cc: FreeBSD Filesystems , Josh Paetzel References: <20181030012240.GM5335@kib.kiev.ua> <4269ff9a-8e11-f441-fbb5-b23a6d8e253b@FreeBSD.org> Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Fri, 2 Nov 2018 08:53:22 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <4269ff9a-8e11-f441-fbb5-b23a6d8e253b@FreeBSD.org> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 07:01:27 -0000 On 02/11/2018 08:45, Andriy Gapon wrote: > On 02/11/2018 04:24, Rick Macklem wrote: >> I have two comments. The first is related to the above and the second is a big >> picture question related to doing this. >> >> Specifically w.r.t. the above. I probably rambled and didn't make what I was >> trying to say clear, so I'll try again... >> - getfh(2) returns a file handle that is used for NFS, but is also used for system >> calls (fhopen(2), fhstat(2) and fhstatfs(2)) that are not related to NFS. >> --> A file handle isn't really NFS specific, although NFS is the big user of it. >> If the above is correct, then it seems that there should only be one kind of file handle. >> --> Since the fsid is a key part of a file handle, one kind of file handle implies one >> kind of fsid. >> I just think trying to create two kinds of fsid and two kinds of file handle would >> make the code confusing and unnecessarily complex. > > As as far as I understand, VFS calls like, say, VOP_VPTOFH fill only the file ID > portion of the file handle, fh_fid. fh_fsid is left to a caller, so potentially > we could already have a discrepancy there (but we don't). > > Also, I do not think that NFS uses getfh(2), but I could be wrong. I think that > NFS, being in kernel, directly uses VFS interfaces like the mentioned VOP_VPTOFH. > > I am not sure if getfh(2) has any requirement that its result should be > compatible with anything NFS. My impression is that it should only be usable > for fhopen(2) and the like. But I can be wrong again. > > So, if any of my assumptions is wrong, then you are right, of course, and I > should withdraw my argument. > Thanks! > Looking through the actual code, it appears that rpc.lockd (and only it?) could be mixing NFS and local file handles... I suspect, but not sure, that lockd could pass a file handle received "from NFS" to fhopen(2). -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Nov 2 10:28:06 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A122210F150E for ; Fri, 2 Nov 2018 10:28:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3E636818C4 for ; Fri, 2 Nov 2018 10:28:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 0258310F150C; Fri, 2 Nov 2018 10:28:06 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E55E110F150B for ; Fri, 2 Nov 2018 10:28:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 86D4E818C1 for ; Fri, 2 Nov 2018 10:28:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B5A00189EF for ; Fri, 2 Nov 2018 10:28:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wA2AS4OI047999 for ; Fri, 2 Nov 2018 10:28:04 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wA2AS4Ir047998 for fs@FreeBSD.org; Fri, 2 Nov 2018 10:28:04 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 217686] zfs_enable=YES causing vm_fault: pager read error, pid 1 (init) Date: Fri, 02 Nov 2018 10:28:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 10:28:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217686 --- Comment #4 from Andriy Gapon --- So, can you check mountpoint properties of all your ZFS filesystems? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 2 11:36:22 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BE1010F309B for ; Fri, 2 Nov 2018 11:36:22 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B4D7D841D3; Fri, 2 Nov 2018 11:36:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id wA2BaAnM056958 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 2 Nov 2018 13:36:13 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua wA2BaAnM056958 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id wA2Ba9E5056957; Fri, 2 Nov 2018 13:36:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 2 Nov 2018 13:36:09 +0200 From: Konstantin Belousov To: Andriy Gapon Cc: Rick Macklem , FreeBSD Filesystems , Josh Paetzel Subject: Re: How to fill in the fsid for file systems? Message-ID: <20181102113609.GL5335@kib.kiev.ua> References: <20181030012240.GM5335@kib.kiev.ua> <4269ff9a-8e11-f441-fbb5-b23a6d8e253b@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 11:36:22 -0000 On Fri, Nov 02, 2018 at 08:53:22AM +0200, Andriy Gapon wrote: > On 02/11/2018 08:45, Andriy Gapon wrote: > > On 02/11/2018 04:24, Rick Macklem wrote: > >> I have two comments. The first is related to the above and the second is a big > >> picture question related to doing this. > >> > >> Specifically w.r.t. the above. I probably rambled and didn't make what I was > >> trying to say clear, so I'll try again... > >> - getfh(2) returns a file handle that is used for NFS, but is also used for system > >> calls (fhopen(2), fhstat(2) and fhstatfs(2)) that are not related to NFS. > >> --> A file handle isn't really NFS specific, although NFS is the big user of it. > >> If the above is correct, then it seems that there should only be one kind of file handle. > >> --> Since the fsid is a key part of a file handle, one kind of file handle implies one > >> kind of fsid. > >> I just think trying to create two kinds of fsid and two kinds of file handle would > >> make the code confusing and unnecessarily complex. > > > > As as far as I understand, VFS calls like, say, VOP_VPTOFH fill only the file ID > > portion of the file handle, fh_fid. fh_fsid is left to a caller, so potentially > > we could already have a discrepancy there (but we don't). > > > > Also, I do not think that NFS uses getfh(2), but I could be wrong. I think that > > NFS, being in kernel, directly uses VFS interfaces like the mentioned VOP_VPTOFH. > > > > I am not sure if getfh(2) has any requirement that its result should be > > compatible with anything NFS. My impression is that it should only be usable > > for fhopen(2) and the like. But I can be wrong again. > > > > So, if any of my assumptions is wrong, then you are right, of course, and I > > should withdraw my argument. > > Thanks! > > > > Looking through the actual code, it appears that rpc.lockd (and only it?) could > be mixing NFS and local file handles... I suspect, but not sure, that lockd > could pass a file handle received "from NFS" to fhopen(2). I believe userspace nfs server implementations exist, and they have to use fhopen(2). From owner-freebsd-fs@freebsd.org Fri Nov 2 12:08:00 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBC1910F45BD for ; Fri, 2 Nov 2018 12:08:00 +0000 (UTC) (envelope-from andyf@andyit.com.au) Received: from alpine.spintel.net.au (alpine.spintel.net.au [IPv6:2407:e400:1::b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1268A8545A for ; Fri, 2 Nov 2018 12:07:59 +0000 (UTC) (envelope-from andyf@andyit.com.au) Received: from drunkfish.andyit.com.au (210-1-210-40-cpe.spintel.net.au [210.1.210.40]) by alpine.spintel.net.au (Postfix) with ESMTPS id AF2BA4C2703 for ; Fri, 2 Nov 2018 23:07:19 +1100 (AEDT) Received: from snuggles.andyit.com.au (snuggles.andyit.com.au [172.22.2.2]) by drunkfish.andyit.com.au (8.15.2/8.15.2) with ESMTP id wA2C7Hqr050989 for ; Fri, 2 Nov 2018 22:07:19 +1000 (AEST) (envelope-from andyf@andyit.com.au) Message-ID: <5BDC3DF5.9020501@andyit.com.au> Date: Fri, 02 Nov 2018 22:07:17 +1000 From: Andy Farkas User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120614 Thunderbird/13.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: zpool scrub 9TB finishes in ~40 mins Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 12:08:01 -0000 [TL;DR - scrub finishes in ~40 min whereas previously it took ~9 hrs] My home server/gateway is a Dell PowerEdge T110 server with 12GB RAM running: # uname -mv FreeBSD 11.1-STABLE #0 r331113: Sun Mar 18 19:57:45 AEST 2018 root@drunkfish.andyit.com.au:/usr/obj/usr/src/sys/DRUNKFISH amd64 It currently has the following HDDs in it: # camcontrol devlist at scbus0 target 3 lun 0 (pass0,da0) at scbus0 target 4 lun 0 (pass1,da1) at scbus0 target 5 lun 0 (pass2,da2) at scbus0 target 6 lun 0 (pass3,da3) at scbus1 target 0 lun 0 (ada0,pass4) at scbus3 target 0 lun 0 (ada1,pass5) at scbus7 target 0 lun 0 (ses0,pass6) The WDs are a ZFS mirror for booting, and the Seagates are a ZFS RAIDZ1 data store. My issue is with the Seagates for the data store. The WDs are attached to the mainboard SATA ports, and the Seagates to a: # grep mps /var/run/dmesg.boot | head -3 mps0: port 0xfc00-0xfcff mem 0xdeab0000-0xdeabffff,0xdeac0000-0xdeafffff irq 16 at device 0.0 on pci2 mps0: Firmware: 07.15.08.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 185c Originally there were 4 x ST4000DM000-1F21 but about 6 months ago 2 of them started showing Offline_Uncorrectable errors so I needed to replace them. Not having immediate cash available I only got around to it a few weeks ago. The pool held up until recently. The controller started "invalidating disk pack" due to too many read errors. At first it only dropped 1 disk (so the pool stayed up) but shortly after, the controller started dropping the second disk. It kept doing this every time it was rebooted when a resilver started. No more zpool :( So I physically moved the Seagates to another box and attached them to its mainboard. I booted a TrueOS DVD and using the "Emergency shell" imported the pool which initiated a resilver. Lots of read/write errors on the 2 failing disks reported in /var/log/messages but at least they weren't being "invalidated". I left it attempting to resilver for a few days but it was obvious it was not going to finish until next decade due to read/write timeouts and retries. I then decided to take each bad disk and clone them to the new disks. The first one I did using 'dd if= of= conv=noerror,sync' and the second one via 'recoverdisk ' - I only found out about recoverdisk(1) after asking a question regarding what does dd(1) do when it gets a read error with bs=1m; does it fill the buffer with good data, then pad the read errors, and continue filling the buffer, or does it pad out the rest of the buffer and start reading the next 1m... another question for another time. So the bad disks were cloned to the good disks sans read errors. I moved the 4 good disks back into the Dell, fired it up, and went "yay!" my pool is back. It finished resilvering, I hadn't really payed attention but it seemed the resilver happened fairly quickly (I was expecting many hours but when I came back an hour later it was finished). Interestingly, zpool status showed: scan: resilvered 143G in 190h41m with 59191 errors on Fri Nov 2 15:54:56 2018 even though the server had only been running for an hour. I presume this was because of having the disks in the other box and the various attempts at resilvering with lots of retry/timeouts on the bad disks. Then I issued a zpool scrub expecting it to take all night. I checked it a few times and saw: # zpool status z pool: z state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub in progress since Fri Nov 2 16:59:24 2018 365G scanned out of 11.9T at 228M/s, 14h47m to go 2.47M repaired, 2.99% done ...about what I was expecting. Another check ~10 mins later: # zpool status z pool: z state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub in progress since Fri Nov 2 16:59:24 2018 560G scanned out of 11.9T at 238M/s, 13h54m to go 3.07M repaired, 4.59% done ...looks good. Dinner time. Came back a bit later to check again: # zpool status z pool: z state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 3.26M in 0h41m with 59191 errors on Fri Nov 2 17:40:58 2018 Its finished? In only 41 minutes? Very strange. So I issued another scrub. Here is the complete output: # zpool status z ; date pool: z state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 0 in 0h39m with 59191 errors on Fri Nov 2 19:23:26 2018 config: NAME STATE READ WRITE CKSUM z ONLINE 0 0 139K raidz1-0 ONLINE 0 0 278K diskid/DISK-WFN02KHJ ONLINE 0 0 11 block size: 512B configured, 4096B native da2 ONLINE 0 0 1.66K block size: 512B configured, 4096B native diskid/DISK-Z300CYLN ONLINE 0 0 0 block size: 512B configured, 4096B native diskid/DISK-Z300C2CK ONLINE 0 0 0 block size: 512B configured, 4096B native errors: 59191 data errors, use '-v' for a list Fri Nov 2 19:33:28 AEST 2018 # Finished again in less than 40 minutes what used to take > 9hrs. Has anyone an explanation? Also why did da2 not get its proper label back? I very much doubt the pool has been scrubbed properly. Next I'm going to clear the (expected) errors and see what happens.... Thanks for reading, -andyf PS. Yes, I did a backup of the pool to another box (via rsync) a few months ago when the errors started showing up. Trying to scrounge together another 10TB of storage as a hobbyist is fairly difficult. But I did actually do it. Unfortunately when I went to restore the files that ZFS -v showed from the backup, 1 of the 6 2TB disks crashed and wiped out the (non-redundant) pool. # zfs list z NAME USED AVAIL REFER MOUNTPOINT z 8.92T 1.63T 8.92T /z From owner-freebsd-fs@freebsd.org Fri Nov 2 13:21:39 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D896D10F638E for ; Fri, 2 Nov 2018 13:21:39 +0000 (UTC) (envelope-from SRS0=GaWu=NN=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7700169E51 for ; Fri, 2 Nov 2018 13:21:39 +0000 (UTC) (envelope-from SRS0=GaWu=NN=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id B60CD2840C for ; Fri, 2 Nov 2018 14:21:37 +0100 (CET) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 00BD528422 for ; Fri, 2 Nov 2018 14:21:28 +0100 (CET) Subject: Re: zpool scrub 9TB finishes in ~40 mins To: freebsd-fs@freebsd.org References: <5BDC3DF5.9020501@andyit.com.au> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: Date: Fri, 2 Nov 2018 14:21:28 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: <5BDC3DF5.9020501@andyit.com.au> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 13:21:40 -0000 Andy Farkas wrote on 2018/11/02 13:07: > # zpool status z >   pool: z >  state: ONLINE > status: One or more devices has experienced an error resulting in data >     corruption.  Applications may be affected. > action: Restore the file in question if possible.  Otherwise restore the >     entire pool from backup. >    see: http://illumos.org/msg/ZFS-8000-8A >   scan: scrub in progress since Fri Nov  2 16:59:24 2018 >     365G scanned out of 11.9T at 228M/s, 14h47m to go >         2.47M repaired, 2.99% done I definitely need this speed of scrub! 238M/s is awesome. I have RAIDZ from 4x ST4000VN000-1H4168 SC43 and the speed of scrub is about 20MB/s. Scrub takes more than week to finish: pool: tank0 state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0 in 262h56m with 0 errors on Sun Sep 16 02:04:25 2018 config: NAME STATE READ WRITE CKSUM tank0 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gpt/disk0tank0 ONLINE 0 0 0 gpt/disk1tank0 ONLINE 0 0 0 gpt/disk2tank0 ONLINE 0 0 0 gpt/disk3tank0 ONLINE 0 0 0 This is on HP ProLiant ML 110 G5 (very old machine) with only 5GB of RAM. Miroslav Lachman From owner-freebsd-fs@freebsd.org Fri Nov 2 13:34:53 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFD8910F70C3 for ; Fri, 2 Nov 2018 13:34:53 +0000 (UTC) (envelope-from oshogbo.vx@gmail.com) Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 11F866ABCB for ; Fri, 2 Nov 2018 13:34:53 +0000 (UTC) (envelope-from oshogbo.vx@gmail.com) Received: by mail-lj1-x22d.google.com with SMTP id z21-v6so1792897ljz.0 for ; Fri, 02 Nov 2018 06:34:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=k3qRDFiTRKRtLPbAUsVWj8PB6KUZEOFVHclmav+iqlc=; b=N3NrnMGG5s46mcjOIP8nuty8Z05WiwsrBMsmygUWIASoqxSNWWt0INMBgRDLVTSitz 3bLMmX2GCIRdUTDOnxGUczjsx9fvrk0pSUgEK/72wbFRA43T/cFBf8SVHrFAHB+g4mNS LvFJcPktzUPVv+bLjRCpWRQstj4SBNCmRiqw8Dkmh7/fhgT83R4aLc+yl/kCPkBL0dQ6 jZOr/cv3qJoAU3OKOyK37LrBkhy6h6axElio0watsGCv7bdVHXQd0w1LA6LVYq0O91dK 6cowlGkOKs7Fa9OAM/V7xXGrDdvh7oLjkqTuHuhmMBr6oW7WVzfy+MX3N7huiSZa8+ml uFJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=k3qRDFiTRKRtLPbAUsVWj8PB6KUZEOFVHclmav+iqlc=; b=Ngwa+9aMyCZMxJu6PVsgs+Q2YMlcBpgsd+JP0LjQA5rHmzN9/KfT+sD6OtAPmJqowB o9Cs1+qZYmus/ny1oFhSOlfkuparMcxQdqctxf4sp/kMe+UdL4GTRCfI8+Fxr9PA4OOV QC6itNWyfJ3PJj7ueNPgFfskVDsvNy4hfb3A9sYjZVl7zg5i5+v7LrOOnV9BIu58aW0H WwHlLyqWzirm1mU/U6+K8VlyG1snaX54U6JLaMpKF3NdNo75LHBhhSdJHZ2zZPQkP3kF TPcV8ge1iSSgmpdurxtSqSCCFtqtyKecPD6EY6Zx3H7JQTupwiYEIQ+9hP4E/AYMK9Dt DgWw== X-Gm-Message-State: AGRZ1gIIJ6GPvdhEnsMrZOi54uvwOVO6MhZ0YOF+1LLk3CU5UmOCx2Z4 WsqGOInGDw0t7KGVqt95WJtgCsXG X-Google-Smtp-Source: AJdET5csF/zbiv/uG27RZd5KobthwS4jLqBta7LljHSSHXILXvQDxndR8bVLqPtPznb1jx86Kkl+eg== X-Received: by 2002:a2e:d11:: with SMTP id 17-v6mr8330857ljn.18.1541165691618; Fri, 02 Nov 2018 06:34:51 -0700 (PDT) Received: from jarvis ([77.79.224.226]) by smtp.gmail.com with ESMTPSA id m13-v6sm1999941ljg.56.2018.11.02.06.34.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Nov 2018 06:34:50 -0700 (PDT) Sender: Mariusz Zaborski Date: Fri, 2 Nov 2018 14:32:54 +0100 From: Mariusz Zaborski To: Miroslav Lachman <000.fbsd@quip.cz> Cc: freebsd-fs@freebsd.org Subject: Re: zpool scrub 9TB finishes in ~40 mins Message-ID: <20181102133254.GA35599@jarvis> References: <5BDC3DF5.9020501@andyit.com.au> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="fUYQa+Pmc3FrFX/N" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 13:34:54 -0000 --fUYQa+Pmc3FrFX/N Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Probably because: https://github.com/zfsonlinux/zfs/pull/6256/commits/1c6275b1fcdacd734bb4eef= d02a123b6b610ca48 FreeBSD commit - r334844. On Fri, Nov 02, 2018 at 02:21:28PM +0100, Miroslav Lachman wrote: > Andy Farkas wrote on 2018/11/02 13:07: >=20 > > # zpool status z > > =A0 pool: z > > =A0state: ONLINE > > status: One or more devices has experienced an error resulting in data > > =A0=A0=A0 corruption.=A0 Applications may be affected. > > action: Restore the file in question if possible.=A0 Otherwise restore = the > > =A0=A0=A0 entire pool from backup. > > =A0=A0 see: http://illumos.org/msg/ZFS-8000-8A > > =A0 scan: scrub in progress since Fri Nov=A0 2 16:59:24 2018 > > =A0=A0=A0 365G scanned out of 11.9T at 228M/s, 14h47m to go > > =A0=A0=A0=A0=A0=A0=A0 2.47M repaired, 2.99% done >=20 > I definitely need this speed of scrub! 238M/s is awesome. > I have RAIDZ from 4x ST4000VN000-1H4168 SC43 and the speed of scrub is=20 > about 20MB/s. >=20 > Scrub takes more than week to finish: >=20 > pool: tank0 > state: ONLINE > status: Some supported features are not enabled on the pool. The pool can > still be used, but some features are unavailable. > action: Enable all features using 'zpool upgrade'. Once this is done, > the pool may no longer be accessible by software that does not= =20 > support > the features. See zpool-features(7) for details. > scan: scrub repaired 0 in 262h56m with 0 errors on Sun Sep 16=20 > 02:04:25 2018 > config: >=20 > NAME STATE READ WRITE CKSUM > tank0 ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > gpt/disk0tank0 ONLINE 0 0 0 > gpt/disk1tank0 ONLINE 0 0 0 > gpt/disk2tank0 ONLINE 0 0 0 > gpt/disk3tank0 ONLINE 0 0 0 >=20 > This is on HP ProLiant ML 110 G5 (very old machine) with only 5GB of RAM. >=20 > Miroslav Lachman > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 Mariusz Zaborski oshogbo//vx | http://oshogbo.vexillium.org FreeBSD committer | https://freebsd.org Software developer | http://wheelsystems.com If it's not broken, let's fix it till it is!!1 --fUYQa+Pmc3FrFX/N Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEkD1x0xkJXVVY1Gwf38KEGuLGxWQFAlvcUgMACgkQ38KEGuLG xWT5cA//QAHscyZocB/f5MfJxT1BTT1UoSQ8AMASREkABPD9WEIbCsDYgN4wx1VX hAL6R9WqXqxlx5brCQHl9EAktOgS9WCa2+A8usumXpm6xUWpFWSrhv4lBFqOLzpe WWSKLpyoqHpCB1Z9lzG5j5o6WeeB/S/cHAUoCNtOxBQ+1k7uWEY0he7m9AFv1JTr pprVSIp4PQpPKUEd77nnzrzIMT0srKa13eFglceFED/lIkXDcaXfCz2YSqekWFDV z3akIHY/dIqE2qMNfiw91AI94w0Q43dtPuCtR3Yh0s9i/OHBlNxI2bIpP0J3FrEA svw/Dau5BP8lTgho/wEErjHFvInI4H/s7mIxMqNGrvMWe//RwfTKMj1GaAsa0XoI tuW30rJFKH1DsEouvhhsjmPOxy+o0jPAidZDEPtjRpEzC23HbFrsONwi19GmmlzX SsV/0CLMzjAFjRjcm8KprlEzS8VeclsP/phBEdOIyVVc8RqGQMUrreUljnS6GWDa nDAp1LpMcMl86RgAS8+Tg0fMi4CouHFXBgXx5XhkrcdvpcEem/PM9ptB8Hi8ARYg 4jZDLOn3Bwary8cJeobdYHbtkaGyY2z+UzCo3yyZs9//ZV08tj9hmmJGeakUROI1 nzshpSnrTm28BnuqzfG42j1vMZWiqICi2bZPtdvZ6YtxUHPR1q4= =sHHl -----END PGP SIGNATURE----- --fUYQa+Pmc3FrFX/N-- From owner-freebsd-fs@freebsd.org Fri Nov 2 13:59:14 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 083A710F7B72 for ; Fri, 2 Nov 2018 13:59:14 +0000 (UTC) (envelope-from andrnils@gmail.com) Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C4DC6BDBF for ; Fri, 2 Nov 2018 13:59:13 +0000 (UTC) (envelope-from andrnils@gmail.com) Received: by mail-lj1-x229.google.com with SMTP id k19-v6so1818199lji.11 for ; Fri, 02 Nov 2018 06:59:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=e2x4csgEa6zGA1bes6hP3pxYu+SsFtK+6YIMuVeUy7Y=; b=lPYhMH0r17EO6eN6WK21DwcO6Qgro5AiMl6nR/V8EcjK6455xYMvKCVxIpd71Y7Jzl z9Y/1FuvFCM2n1tRonpiYL+Miv+rbBX6IEM/t3Ry1/dVQ55WztCzgfzSFjtHLg1tHODJ xT7296a0OeE5gzC6FehJjoOCTYWnVqIQcaqOkOOB6zKb1v69ac9aCyBq/VbftAFqEINg FWu78zlIJeqllsQJ1J/6Jt4sttxIHR8f1r8TNWkDpOf2RX16DVTju6WDqr3NeEo81LXt NmdRK5X1Mo96TOX4296ECzK45VUIazzy7DufsI2HXooETfL3SGSSmpgG3hfLlMMkc+3d tg6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=e2x4csgEa6zGA1bes6hP3pxYu+SsFtK+6YIMuVeUy7Y=; b=uNY2ZXwsbogH9NevwNa2nH2Itwd0vSanJtwTle1PowxeUWckVLw6ryvFa/AUrCR2Kr KTiWPXDQLg9xjXcTwQ7F4qVE9i/F5ekiuq/uWmsHc8juacAbvtDW84LUiRguDJDfex2Z T5E6ASJOw0gcYruwkcdT8kX3/gROqon1nmiFBWWiMTZ1Nc/7SSlSn1misRzgmdPR/66H DoAOdaS1Nyc+BennePvAORREhBT+VQKLueqNI6GOUEXBaBNucremRZjCUGZmFR/r+jNX OclFgooCtrn98Pe2Eo/LtBHNSEOoyWEgHJ3h3ovhA7VF3C74oUvcygc6eIhBR+L+pFo7 irCQ== X-Gm-Message-State: AGRZ1gJJd/MTutzqwtRLoMxK1NcgZiaRO+kkAfv55IvbE3bbhtAYPvPZ HVUU/EoekPay66JnU6bO7kRF74hD7g7un2eVFBXZbM36 X-Google-Smtp-Source: AJdET5e1p+ZIH4lwi3r2jEgxVyfvRFkhfRdCZflYWQwZ++mJhI/0JFhQpWkXf5Djkg5ctYbrjHxsi1pbsE4QvRZyKL8= X-Received: by 2002:a2e:59ca:: with SMTP id g71-v6mr7810081ljf.79.1541167151815; Fri, 02 Nov 2018 06:59:11 -0700 (PDT) MIME-Version: 1.0 References: <5BDC3DF5.9020501@andyit.com.au> In-Reply-To: From: Andreas Nilsson Date: Fri, 2 Nov 2018 14:54:56 +0100 Message-ID: Subject: Re: zpool scrub 9TB finishes in ~40 mins To: Miroslav Lachman <000.fbsd@quip.cz> Cc: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 13:59:14 -0000 On Fri, Nov 2, 2018 at 2:22 PM Miroslav Lachman <000.fbsd@quip.cz> wrote: > Andy Farkas wrote on 2018/11/02 13:07: > > > # zpool status z > > pool: z > > state: ONLINE > > status: One or more devices has experienced an error resulting in data > > corruption. Applications may be affected. > > action: Restore the file in question if possible. Otherwise restore the > > entire pool from backup. > > see: http://illumos.org/msg/ZFS-8000-8A > > scan: scrub in progress since Fri Nov 2 16:59:24 2018 > > 365G scanned out of 11.9T at 228M/s, 14h47m to go > > 2.47M repaired, 2.99% done > > I definitely need this speed of scrub! 238M/s is awesome. > I have RAIDZ from 4x ST4000VN000-1H4168 SC43 and the speed of scrub is > about 20MB/s. > > Scrub takes more than week to finish: > > pool: tank0 > state: ONLINE > status: Some supported features are not enabled on the pool. The pool can > still be used, but some features are unavailable. > action: Enable all features using 'zpool upgrade'. Once this is done, > the pool may no longer be accessible by software that does not > support > the features. See zpool-features(7) for details. > scan: scrub repaired 0 in 262h56m with 0 errors on Sun Sep 16 > 02:04:25 2018 > config: > > NAME STATE READ WRITE CKSUM > tank0 ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > gpt/disk0tank0 ONLINE 0 0 0 > gpt/disk1tank0 ONLINE 0 0 0 > gpt/disk2tank0 ONLINE 0 0 0 > gpt/disk3tank0 ONLINE 0 0 0 > > This is on HP ProLiant ML 110 G5 (very old machine) with only 5GB of RAM. > > Miroslav Lachman > As a data point I can add: $ zpool status storage-0 pool: storage-0 state: ONLINE scan: scrub in progress since Fri Nov 2 14:39:55 2018 293G scanned out of 2.85T at 424M/s, 1h45m to go 0 repaired, 10.06% done config: NAME STATE READ WRITE CKSUM storage-0 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gpt/store-00 ONLINE 0 0 0 gpt/store-01 ONLINE 0 0 0 gpt/store-02 ONLINE 0 0 0 gpt/store-03 ONLINE 0 0 0 Speed is till increasing. This is on an old supermicro with CPU: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz and 8 gb of ram with and ST2000LM015-2E8174 SDM1 2.5'' spinning rust connected to the ahci0: Best regards Andreas From owner-freebsd-fs@freebsd.org Fri Nov 2 14:03:51 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 111D410F7E62 for ; Fri, 2 Nov 2018 14:03:51 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it1-x133.google.com (mail-it1-x133.google.com [IPv6:2607:f8b0:4864:20::133]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 80FED6C1AD for ; Fri, 2 Nov 2018 14:03:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it1-x133.google.com with SMTP id k206-v6so3293269ite.0 for ; Fri, 02 Nov 2018 07:03:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=swkaXV4lgrehlwlObbJM1qCX/VPkAxUu0azitj2QHrQ=; b=OyOPYYuM0L0L8R9vIiRd8t/HpDH3ZM1c6GZd9e8aIwD7RFvZKt/DJ1ZUrm1LMjWqEJ Ixw5vPCYdTHJJPuWo595VUR5fWzjpA6Lit2i73T1e/nufXe08KyRkAzuLLsGyrrEbt6C Rg0FOhyx5f3ZGF6To+SHHBGr6x2yWIz8WKDQ7IvJBMDJn1Z1dPpBSsNUqI3SRXXXws3A 8aYXaJkeIHH+dpo2bl/utKaGvvoeWmzm6K7bcOY/BRgphN5wFDFR/6MMJ7+ozv1DZW1t wgq3QpIms8vNWlUOuedeMsiftEV4rjFK5EMI9GWKxNs/BPNeXP5aXFzRg2Z9DkL5Nst4 zeBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=swkaXV4lgrehlwlObbJM1qCX/VPkAxUu0azitj2QHrQ=; b=pmdd9hsys3PbaIgZ2x8JHVRcqdAlqbZLDCb/yHauVWQc4ziaSGGDjq903js3u+gt3g OAdVLHBZ9L9HujOidMZECEFPpEL/GLhBGT3z1vn5aiGgKVVttqdytlzQhNTybkeu6tdR iZvu/yPu+AdY7KBSEhNaEa08UCCpzZwzRB5bUmySMxELAE02uu032VetrkPv3ts31Rc4 BAk5dkJ5lurJAt1b0uwAm2dFUD1kugvnuycz/zbEG07COrO5WhDvZEEv83zfs6uMCtx8 86wRHUdv20i5NHJX5X+f4fA/0NIyVgqUCDAOS/gmobG76EH8NT71qcTAGn2BngWLmrAr MoOw== X-Gm-Message-State: AGRZ1gIphkm1eaOjwjjsh8hqpnvD8Y8+iPw0x+nstPcdluw7ckPv9aUT p2scOyhDtfxWhFrUw55aD+g9v5gUI06a7FmFKpsuCQ== X-Google-Smtp-Source: AJdET5drVa0DGsJ15e6RWhghEd5wZaEAx5TfEIvuqZ7rPNy5aFHlxNX+gWNhV1o9eCSfoqPY1kxTcwJSzKg0v/XYxqA= X-Received: by 2002:a24:eb0b:: with SMTP id h11-v6mr85733itj.47.1541167429655; Fri, 02 Nov 2018 07:03:49 -0700 (PDT) MIME-Version: 1.0 References: <20181030012240.GM5335@kib.kiev.ua> <4269ff9a-8e11-f441-fbb5-b23a6d8e253b@FreeBSD.org> <20181102113609.GL5335@kib.kiev.ua> In-Reply-To: <20181102113609.GL5335@kib.kiev.ua> From: Warner Losh Date: Fri, 2 Nov 2018 08:03:38 -0600 Message-ID: Subject: Re: How to fill in the fsid for file systems? To: Konstantin Belousov Cc: Andriy Gapon , FreeBSD FS Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 14:03:51 -0000 On Fri, Nov 2, 2018 at 5:37 AM Konstantin Belousov wrote: > On Fri, Nov 02, 2018 at 08:53:22AM +0200, Andriy Gapon wrote: > > On 02/11/2018 08:45, Andriy Gapon wrote: > > > On 02/11/2018 04:24, Rick Macklem wrote: > > >> I have two comments. The first is related to the above and the second > is a big > > >> picture question related to doing this. > > >> > > >> Specifically w.r.t. the above. I probably rambled and didn't make > what I was > > >> trying to say clear, so I'll try again... > > >> - getfh(2) returns a file handle that is used for NFS, but is also > used for system > > >> calls (fhopen(2), fhstat(2) and fhstatfs(2)) that are not related > to NFS. > > >> --> A file handle isn't really NFS specific, although NFS is the > big user of it. > > >> If the above is correct, then it seems that there should only be one > kind of file handle. > > >> --> Since the fsid is a key part of a file handle, one kind of file > handle implies one > > >> kind of fsid. > > >> I just think trying to create two kinds of fsid and two kinds of file > handle would > > >> make the code confusing and unnecessarily complex. > > > > > > As as far as I understand, VFS calls like, say, VOP_VPTOFH fill only > the file ID > > > portion of the file handle, fh_fid. fh_fsid is left to a caller, so > potentially > > > we could already have a discrepancy there (but we don't). > > > > > > Also, I do not think that NFS uses getfh(2), but I could be wrong. I > think that > > > NFS, being in kernel, directly uses VFS interfaces like the mentioned > VOP_VPTOFH. > > > > > > I am not sure if getfh(2) has any requirement that its result should be > > > compatible with anything NFS. My impression is that it should only be > usable > > > for fhopen(2) and the like. But I can be wrong again. > > > > > > So, if any of my assumptions is wrong, then you are right, of course, > and I > > > should withdraw my argument. > > > Thanks! > > > > > > > Looking through the actual code, it appears that rpc.lockd (and only > it?) could > > be mixing NFS and local file handles... I suspect, but not sure, that > lockd > > could pass a file handle received "from NFS" to fhopen(2). > > I believe userspace nfs server implementations exist, and they have to use > fhopen(2). > You are correct. NFS is the reason these interfaces exist because it has historically (or at least initially) been implemented in userspace. Warner From owner-freebsd-fs@freebsd.org Fri Nov 2 14:17:02 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0B5A810F86EB for ; Fri, 2 Nov 2018 14:17:02 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-lj1-x242.google.com (mail-lj1-x242.google.com [IPv6:2a00:1450:4864:20::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6EBE66D011; Fri, 2 Nov 2018 14:17:01 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: by mail-lj1-x242.google.com with SMTP id g26-v6so1878013lja.10; Fri, 02 Nov 2018 07:17:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dDBPNwSJCZBAdv9vMGOP84Td3YSj1CImZSF/Nz3Y5cs=; b=KkR+om4asSG6swoRKl35H33wxN+PaHIYwbUaDd27yavhwO94FnEZV7ctdnFPdTYcRh qGvpV6go/jp0KPF6Z8LZixf6+itPyL11lG2eWWcdxyexKBz6pb/Rxi/snNDNAzA0YNSd ArY1HmOIu4GdJPMULmJkr4+HhDJdD6qxY5zzUQ/kgrWAVuNm9pTWniKRIck3WrkgBQTM zO2BHCPI0OhfL7mdEMVMuOrSJ220PWKdpo2SjNvAJ/2aH3ncy6XzD6/k27Sxh8evE9zt ZiOnW2WBd+RsGq6UwLjLpaaPoTs0taaI0PRsQ+R0u7sUhIFt5btrFixl5qym6lW6WIuv lVVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dDBPNwSJCZBAdv9vMGOP84Td3YSj1CImZSF/Nz3Y5cs=; b=mdGE27vgQou4AksTb2ouO8jLXyNxBU9m5P5EuEGKYLP25qvHQB2KBauWCO+fC/S+e+ FblbeDKIZx3v/tsO/zWugeiFqF+VL87R4tkbz2ZDsYx7/hIMF6vrvW+usnj7HOpcSmFd pO3edj60PYwkWCHgeg3Y1gFupRdS3eA9SbDgcJKP8a2+VYNpcx/XmjPRrplD/b05GlXC z1g7QxxI3G/ZEbkvDFNVohTkJ8K55kzX598CaTPWv3O453WnVHUFMav33qbSWO0yl4/2 IGeqZyTahZTmoEq+LDRa1R8D62Y4mgzeBGK+8/UHtj++vqB5iQUgayVEGWZWEjC3I9xI wJrQ== X-Gm-Message-State: AGRZ1gI7h9Wsr/IKCrnyCHrrhXpRPEGs479lJl/ji05vQwxwc81HP4uk hr9f4v8KTNp/Jt3nY3zCL6VQqPSLEYDqeLdbNCbTvFFg X-Google-Smtp-Source: AJdET5c2HZRM4h5UpchV2LK6d7qKKYe6Qs7FKS0S/cTY+k49CpcUn5IwGHRpUK26/I3TYl3wVZXGaguF8+8CyfLkdCk= X-Received: by 2002:a2e:920c:: with SMTP id k12-v6mr8398489ljg.178.1541168219322; Fri, 02 Nov 2018 07:16:59 -0700 (PDT) MIME-Version: 1.0 References: <5BDC3DF5.9020501@andyit.com.au> <20181102133254.GA35599@jarvis> In-Reply-To: <20181102133254.GA35599@jarvis> From: Rich Date: Fri, 2 Nov 2018 10:16:51 -0400 Message-ID: Subject: Re: zpool scrub 9TB finishes in ~40 mins To: oshogbo@freebsd.org Cc: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 14:17:02 -0000 Unless FBSD merged it incompletely, that's very unlikely, b/c zpool status output during resilvers/scrubs changes post-sequential resilver work: from: (void) printf(gettext("\t%s scanned out of %s at %s/s") to: (void) printf(gettext("\t%s scanned at %s/s, "%s issued at %s/s, %s total\n"), So those status messages are older than that work, which is also validated by the version running predating the work merging (r331113 versus r334844). I would _guess_ that it encountered so many metadata errors it couldn't get to a lot of the pool, so it vacuously completed faster. Unless large swathes of the disks were unreadable, that's far more errors than I'd expect from a couple of unreadable blocks each. You and ddrescue might want to be friends, though that might make things worse since the pool has continued being some value of "in use" since the failing disks were last in it. - Rich On Fri, Nov 2, 2018 at 9:36 AM Mariusz Zaborski wrote: > > Probably because: > https://github.com/zfsonlinux/zfs/pull/6256/commits/1c6275b1fcdacd734bb4eefd02a123b6b610ca48 > > FreeBSD commit - r334844. > > > On Fri, Nov 02, 2018 at 02:21:28PM +0100, Miroslav Lachman wrote: > > Andy Farkas wrote on 2018/11/02 13:07: > > > > > # zpool status z > > > pool: z > > > state: ONLINE > > > status: One or more devices has experienced an error resulting in data > > > corruption. Applications may be affected. > > > action: Restore the file in question if possible. Otherwise restore the > > > entire pool from backup. > > > see: http://illumos.org/msg/ZFS-8000-8A > > > scan: scrub in progress since Fri Nov 2 16:59:24 2018 > > > 365G scanned out of 11.9T at 228M/s, 14h47m to go > > > 2.47M repaired, 2.99% done > > > > I definitely need this speed of scrub! 238M/s is awesome. > > I have RAIDZ from 4x ST4000VN000-1H4168 SC43 and the speed of scrub is > > about 20MB/s. > > > > Scrub takes more than week to finish: > > > > pool: tank0 > > state: ONLINE > > status: Some supported features are not enabled on the pool. The pool can > > still be used, but some features are unavailable. > > action: Enable all features using 'zpool upgrade'. Once this is done, > > the pool may no longer be accessible by software that does not > > support > > the features. See zpool-features(7) for details. > > scan: scrub repaired 0 in 262h56m with 0 errors on Sun Sep 16 > > 02:04:25 2018 > > config: > > > > NAME STATE READ WRITE CKSUM > > tank0 ONLINE 0 0 0 > > raidz1-0 ONLINE 0 0 0 > > gpt/disk0tank0 ONLINE 0 0 0 > > gpt/disk1tank0 ONLINE 0 0 0 > > gpt/disk2tank0 ONLINE 0 0 0 > > gpt/disk3tank0 ONLINE 0 0 0 > > > > This is on HP ProLiant ML 110 G5 (very old machine) with only 5GB of RAM. > > > > Miroslav Lachman > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > -- > Mariusz Zaborski > oshogbo//vx | http://oshogbo.vexillium.org > FreeBSD committer | https://freebsd.org > Software developer | http://wheelsystems.com > If it's not broken, let's fix it till it is!!1 From owner-freebsd-fs@freebsd.org Fri Nov 2 14:35:30 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 789F210F923E for ; Fri, 2 Nov 2018 14:35:30 +0000 (UTC) (envelope-from SRS0=GaWu=NN=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14AD66E0E8; Fri, 2 Nov 2018 14:35:29 +0000 (UTC) (envelope-from SRS0=GaWu=NN=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 3B30F2840C; Fri, 2 Nov 2018 15:35:28 +0100 (CET) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 4CD2E2842B; Fri, 2 Nov 2018 15:35:27 +0100 (CET) Subject: Re: zpool scrub 9TB finishes in ~40 mins To: Mariusz Zaborski Cc: freebsd-fs@freebsd.org References: <5BDC3DF5.9020501@andyit.com.au> <20181102133254.GA35599@jarvis> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: Date: Fri, 2 Nov 2018 15:35:22 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: <20181102133254.GA35599@jarvis> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 14:35:30 -0000 Mariusz Zaborski wrote on 2018/11/02 14:32: > Probably because: > https://github.com/zfsonlinux/zfs/pull/6256/commits/1c6275b1fcdacd734bb4eefd02a123b6b610ca48 > > FreeBSD commit - r334844. Was it MFCed to 11.2? My data was from 10.4 but the machine was upgraded to 11.2 week ago so next scrub will be on 11.2. Kind regards Miroslav Lachman From owner-freebsd-fs@freebsd.org Fri Nov 2 14:54:53 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A523110F9FEE for ; Fri, 2 Nov 2018 14:54:53 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from smtp.simplesystems.org (smtp.simplesystems.org [65.66.246.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D21C6F190 for ; Fri, 2 Nov 2018 14:54:53 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from scrappy.simplesystems.org (scrappy.simplesystems.org [65.66.246.73]) by smtp.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id wA2EoQAa024714; Fri, 2 Nov 2018 09:50:26 -0500 (CDT) Date: Fri, 2 Nov 2018 09:50:26 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@scrappy.simplesystems.org To: Miroslav Lachman <000.fbsd@quip.cz> cc: freebsd-fs@freebsd.org Subject: Re: zpool scrub 9TB finishes in ~40 mins In-Reply-To: Message-ID: References: <5BDC3DF5.9020501@andyit.com.au> User-Agent: Alpine 2.20 (GSO 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (smtp.simplesystems.org [65.66.246.90]); Fri, 02 Nov 2018 09:50:26 -0500 (CDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 14:54:53 -0000 On Fri, 2 Nov 2018, Miroslav Lachman wrote: > > I definitely need this speed of scrub! 238M/s is awesome. > I have RAIDZ from 4x ST4000VN000-1H4168 SC43 and the speed of scrub is about > 20MB/s. I appear to see an average rate of 335 MB/s here using an Illumos-based system with 16 2GB nearline-SAS drives. The data transfer rates reported during scrub is perhaps 850 MB/s but there are many snapshots in this pool. Rates with an all-SSD pool can be far higher. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@freebsd.org Fri Nov 2 14:59:06 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3799710FA1F2 for ; Fri, 2 Nov 2018 14:59:06 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C00496F44E for ; Fri, 2 Nov 2018 14:59:05 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io1-xd36.google.com with SMTP id p83-v6so1545322iod.12 for ; Fri, 02 Nov 2018 07:59:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=7pDSaH5VPvSvL3e0LL9tb872zdiGNpXBn62iW3GGLd0=; b=lbkiAgAuIoFHZ62jppcRBHP2TL3BsmBK2OJzxiEsZSd7haNFb2cOL+K+cJ8/wwiJc3 CP+sgZAXEPw3KxcAW4nuZgJV6zoEH7h7/ik5qum08gcDIwCCRtTCyjKL0asArjh8yZt4 IeQk2DyegoUYtUzkLkSIKRihGPDymIfOgrv4FH8SIMpc58iGR6aO2DeRL2kw1CkmSGFg REujdW4p7uoTZCbdKZyO64VoUZXd8+Ke0mRDT4ddhaKg6S7xqwpUC04M/gRKJ3xmCpRo xa63fiioTEIFZs5uowqqh/M2iH68roe43kCV1VcKVt8LGLHaRoREuphLQrmyQO4XpGMV 0wvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=7pDSaH5VPvSvL3e0LL9tb872zdiGNpXBn62iW3GGLd0=; b=XwUS+1zB0cBW2GS/SdRrIZn3vWImGYFlSZwSuQ6aTUOucdEt1+PCKUmW7UDoLPRd+G NmDQU2uXsqtENOw4myM9CgUTeiDHFpK/z64djSsupwCDblq6PZ9Z4fqjNfNE9Nm2lb0M ShemUAT3bAptwW8NRkHwjlVrM8FZK2QtBzYhcFbi/sxIjYILv/FedGiSHhM1aavibvC8 CsHXrArFdXbFWUKZMYfSI/62trBVqL4c0jLeEkQkad7c5iqpvhOjG6OOfC7ww5VbqDVa +ZyshWkDf6FSvMTrQjcI5fwRsac2TQxaO+8rcqrQWEkoYbSXwGMZ6YgyxMmzRlG/aVqt wxBw== X-Gm-Message-State: AGRZ1gLdIiH6zC2+8vOn/WV6W7Nv2YKKbfj9rj3lI5oU1vynez4iJbJt WED8PwXSnUclmKKN3PfngJWiftTZfW15oWEeRqphqbu050k= X-Google-Smtp-Source: AJdET5d5FU6CKzTklQ4QzsHeLkM0Ysu1dwP1u5S474Esz24aohXcoAlnRGvjpt0+BN/n9/vaV5OCRReRI2Y5CEOurlc= X-Received: by 2002:a6b:7809:: with SMTP id j9-v6mr2438761iom.299.1541170744925; Fri, 02 Nov 2018 07:59:04 -0700 (PDT) MIME-Version: 1.0 References: <5BDC3DF5.9020501@andyit.com.au> In-Reply-To: From: Warner Losh Date: Fri, 2 Nov 2018 08:58:53 -0600 Message-ID: Subject: Re: zpool scrub 9TB finishes in ~40 mins To: bfriesen@simple.dallas.tx.us Cc: Miroslav Lachman <000.fbsd@quip.cz>, FreeBSD FS Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 14:59:06 -0000 On Fri, Nov 2, 2018 at 8:55 AM Bob Friesenhahn wrote: > Rates with an all-SSD pool can be far higher. > scan: scrub in progress since Fri Nov 2 08:54:11 2018 406G scanned at 2.49G/s, 1.48M issued at 54K/s, 2.67T total 0 repaired, 0.00% done, no estimated completion time Just a smidge, eh? This is a zpool of 4 Micron M600 drives. gstat says they are doing 425-450MB/s each. Warner From owner-freebsd-fs@freebsd.org Fri Nov 2 15:25:31 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E45F10FAEEB for ; Fri, 2 Nov 2018 15:25:31 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D58B970C32 for ; Fri, 2 Nov 2018 15:25:30 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f173.google.com with SMTP id z21-v6so2119085ljz.0 for ; Fri, 02 Nov 2018 08:25:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=wWMCtikGODl4XwV6hlgWjpgFB3coqbKg4IXGd6E+6L0=; b=tNAS3LIK22bOFQFxv8nvMVMMwfgZfrtbx1Ymw9HY4J7g+4xcRKu8mazrJd19nfn6mF JW2DoVcIryP4BAXUzZd4fDfNv2xsXWzhpRQn2dBRNYR6Sw6PDApKNU/gnrd3L3Q55KUC 3O7NL2Eu7RALeIFByy9A8/jdL399FjwXfksn8pq8bQbauDiKJ/gPv9KYjyPkmiYujvew O1u6fnhmm+JYl4apZQtQgiNimc6jaj0dfP29EKbtI5UI6+NKNPcCwhZCs0T0e0doXxSd i4CSrAE1EoK1rnioTxUpH4jPlcktKQuSNChdfemhutyWCkDlNvjExKlsEd6g3vlcqP3i Zycg== X-Gm-Message-State: AGRZ1gKYsocMpy0bM5xEVFDKkm7x0bnATxsNUGThNh0c/0ZsA2WgB/xI hwoZkFIuGynXkUppdSmXaEY= X-Google-Smtp-Source: AJdET5fVNS44VkOeD9IBgqxDZPNR7WN0a/m3FOUsK59u6fq67awWqcNmzbXExzy0raC69dqg+CuK5w== X-Received: by 2002:a2e:330d:: with SMTP id d13-v6mr8734638ljc.0.1541172322518; Fri, 02 Nov 2018 08:25:22 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id t77-v6sm5789265lfi.63.2018.11.02.08.25.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Nov 2018 08:25:21 -0700 (PDT) Subject: Re: How to fill in the fsid for file systems? To: Rick Macklem , Konstantin Belousov Cc: FreeBSD Filesystems , Josh Paetzel References: <20181030012240.GM5335@kib.kiev.ua> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Fri, 2 Nov 2018 17:25:20 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 15:25:31 -0000 On 31/10/2018 17:50, Rick Macklem wrote: > Andriy Gapon wrote: >> One is practical. How do we provide fsid to ZFS filesystems? >> I mean I would hate to resort to mounting ZFS via fstab just to provide fsid >> whereas today ZFS filesystems are mounted auto-magically. >> We could add a FreeBSD specific fsid ZFS property, but that's also some extra code. > Good point. I'm not a ZFS guy, so I wouldn't have thought of this. A counter-point to my own point. If we implement the fsid override in the common code, like vfs_do[n]mount, then we would not need to worry about any filesystem specifics. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Nov 2 15:51:32 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCFAE10FB720 for ; Fri, 2 Nov 2018 15:51:31 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670078.outbound.protection.outlook.com [40.107.67.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 801E571B62; Fri, 2 Nov 2018 15:51:31 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB0716.CANPRD01.PROD.OUTLOOK.COM (52.132.43.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1294.26; Fri, 2 Nov 2018 15:51:30 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b%3]) with mapi id 15.20.1294.024; Fri, 2 Nov 2018 15:51:30 +0000 From: Rick Macklem To: Andriy Gapon , Konstantin Belousov CC: FreeBSD Filesystems , Josh Paetzel Subject: Re: How to fill in the fsid for file systems? Thread-Topic: How to fill in the fsid for file systems? Thread-Index: AQHUb557F1RNqdJl0kuwS+F5J/DjsKU2/twAgAGdwkKAAGQrgIAAehmGgAELs4CAATs6uIAATiWAgAACTACAAJQ+WA== Date: Fri, 2 Nov 2018 15:51:29 +0000 Message-ID: References: <20181030012240.GM5335@kib.kiev.ua> <4269ff9a-8e11-f441-fbb5-b23a6d8e253b@FreeBSD.org>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB0716; 6:WuuzOgnjgFuvQTMZwoM/dWljZojh9shNBbL2AqjUtoap4s1IWeTSoMfIB1xaAbhUZeSMtKZUWyaY/6cLu6bSUY9jCv3oEH2Bu6GfyxN5mBEH6zRzTiIs7TQyxZtzKi4LtJUSAQTRFlktygytkFKsr3geMsW8/apYNHIZLnycf82ECCo2303Cu41w9icABvAd7Q1Z49v+qh5PQWXGKfI9cwBGNZjTsspzUyRBelLx5eo28XfYFyU/pQY+SQk3dQBKp+07+z8OyQzLoDQ3CQfGrYn64cU6j39Sgs5ChJ1G1CSv48lcV7HxUsGF4vkxORFypg0Hlfp/RDFARWhF0kMWq17muUc+yWSLzqlfOHc/8eUDFgvRN9YQPpfeTr54im+HNDAFpGEVZeEVvkb6mS/JCHMI8GvLpxJl3IcN51tqiWqJp1bi64Wb6q0Cnmu39dVZ/aD80v6uEkOOTd+FKqfJJw==; 5:SB9jlotCp/z7pfHVUTMB02KvzzoCCqyw0X+2mF93cK68YdhLCjxV8Q7pC0KvIzeotpZol1FHaXjrIwi0Ad0NpXfps2aNphyMhxms2eT/+1IsTnHpGX/UFaSrAofrunL3GipnL7iJGvrat+NesUbFw76axE/uFjMl1KLCZFd4ajI=; 7:QomrWlOvJlmLzprlVQYGsdaCDTUil9GKbYOvd08Gr5aQAIyQc0FYdu00hRKVwJmuhin4Doj7APV7JIeNvGc61lUiWdm8mB/vaVTWR7RZ9gBUOEMaPNNtRS7G0SGljOYW3pcc+MOCoDulCOZuVAwU6Q== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 473c86b0-ee3c-4046-fb2a-08d640db0e45 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(5600074)(711020)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB0716; x-ms-traffictypediagnostic: YTOPR0101MB0716: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(3231382)(944501410)(52105095)(3002001)(10201501046)(93006095)(93001095)(148016)(149066)(150057)(6041310)(20161123558120)(20161123562045)(20161123564045)(20161123560045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB0716; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB0716; x-forefront-prvs: 08444C7C87 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(39860400002)(396003)(376002)(136003)(346002)(189003)(199004)(51444003)(446003)(81166006)(39060400002)(305945005)(99286004)(76176011)(7696005)(71200400001)(71190400001)(2900100001)(46003)(74316002)(186003)(6506007)(14454004)(4326008)(74482002)(8936002)(5660300001)(102836004)(6436002)(68736007)(786003)(316002)(476003)(53936002)(93886005)(229853002)(2906002)(486006)(86362001)(53546011)(9686003)(97736004)(6246003)(105586002)(256004)(25786009)(33656002)(54906003)(106356001)(110136005)(11346002)(8676002)(81156014)(55016002)(478600001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB0716; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: a1PkB6mUWikzu0QYEnXbB0J9SBxhK/ogZvr6jT28S6j8L7OW0DIltPd6ukzTXmVlC7CLaaMepdrSrn/9SPJvWKZtRNHCRA278jjRI7rAXdICNSd8HANKZqtbl5k88vISZKOziLC7rdOcMjDsP+H5hm6lkb1/+EvWLr44uF3NtAJQ3WnyZ1Dqax1FG+mq+y+2x3NdViYA3E3nYf8+91P3c7P4Xhj2Gnop7xo0VwxVycsPR1Gg8SlbJ8OobgkUW5sfsbAITh5r37cpw5nuU4y1ZENifwNwn/1z/0S9PvWHp9ZYMz2oiB/tKUPw9bK5AThmdl3QFjiDwl72JG/PUcWsVN9xFaNUSZr6xeN0AQm3pNw= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 473c86b0-ee3c-4046-fb2a-08d640db0e45 X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Nov 2018 15:51:30.0048 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB0716 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 15:51:32 -0000 Andriy Gapon wrote: >On 02/11/2018 08:45, Andriy Gapon wrote: >> On 02/11/2018 04:24, Rick Macklem wrote: >>> I have two comments. The first is related to the above and the second i= s a big >>> picture question related to doing this. >>> >>> Specifically w.r.t. the above. I probably rambled and didn't make what = I was >>> trying to say clear, so I'll try again... >>> - getfh(2) returns a file handle that is used for NFS, but is also used= for system >>> calls (fhopen(2), fhstat(2) and fhstatfs(2)) that are not related to = NFS. >>> --> A file handle isn't really NFS specific, although NFS is the big = user of it. >>> If the above is correct, then it seems that there should only be one ki= nd of file handle. >>> --> Since the fsid is a key part of a file handle, one kind of file han= dle implies one >>> kind of fsid. >>> I just think trying to create two kinds of fsid and two kinds of file h= andle would >>> make the code confusing and unnecessarily complex. >> >> As as far as I understand, VFS calls like, say, VOP_VPTOFH fill only the= file ID >> portion of the file handle, fh_fid. fh_fsid is left to a caller, so pot= entially >> we could already have a discrepancy there (but we don't). >> >> Also, I do not think that NFS uses getfh(2), but I could be wrong. I th= ink that >> NFS, being in kernel, directly uses VFS interfaces like the mentioned VO= P_VPTOFH. >> >> I am not sure if getfh(2) has any requirement that its result should be >> compatible with anything NFS. My impression is that it should only be u= sable >> for fhopen(2) and the like. But I can be wrong again. mountd uses getfh(2) to acquire a file handle at mount time for a client. S= o, for NFSv3 it does use getfh(2) to acquire the "root" file handle for the mount. (Technically, it's the Mount protocol, but since that is required for NFSv3= mounting, you might as well call it NFS and this one needs to be an NFS usable file = handle.) >> >> So, if any of my assumptions is wrong, then you are right, of course, an= d I >> should withdraw my argument. >> Thanks! >> > >Looking through the actual code, it appears that rpc.lockd (and only it?) = could >be mixing NFS and local file handles... I suspect, but not sure, that loc= kd >could pass a file handle received "from NFS" to fhopen(2). Good point. SInce the NLM isn't NFS I tend to forget about it. rick From owner-freebsd-fs@freebsd.org Fri Nov 2 19:05:10 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BEF0510D7668 for ; Fri, 2 Nov 2018 19:05:10 +0000 (UTC) (envelope-from byhkaan@gmail.com) Received: from mail-lj1-x22e.google.com (mail-lj1-x22e.google.com [IPv6:2a00:1450:4864:20::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23BC379E82 for ; Fri, 2 Nov 2018 19:05:10 +0000 (UTC) (envelope-from byhkaan@gmail.com) Received: by mail-lj1-x22e.google.com with SMTP id x85-v6so2662662ljb.2 for ; Fri, 02 Nov 2018 12:05:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=ILZ53mJE2cB8Q9Monohu6FnceW9OZ0+xdLH9ZrB+wzk=; b=sFcFWPoFPKB573juiUBKLm+R/4zuKPwVPdfrgfideaqxu1WzuApidaoKJshexlUMIy 6Dp/v86ygvZ9i/HKys1TS4RO4jns7jz2M6ym/cl9j/kLQ4LXd6uQE3xqiCLLwJZodJff be4dpkGO/kN7mxmDhJwGQ1FOU9MdN66s3o0ceEvcVkCdSAEDbK1EhF6HfSZnR2X0xbFP tbRgMKWZKZF3VJYNNgFnw1rEbYaqr+VWvh6g/Ht2idmz/oNU/RajXn2LUk2IP+OTIwb9 w8xKs7G8TL4TR3VqFrhMkb4qlv1/3iRzOgvSj3JIDwBarRjJBdmo59b1FkP7xemMZeCP 7mBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=ILZ53mJE2cB8Q9Monohu6FnceW9OZ0+xdLH9ZrB+wzk=; b=NkyEc/o/s/pujVOZGTr/f20aOtg4SPNvWQiE6wvjWkEjgODB8r0vtiiOSlYKNPQx8+ JRDxFMbp5A61d63GG3edHkANn0yyhG2r/QVwifc5eIQRvnQrxLNwdukJ/FSkcwq02OIF 0JwIauFAXmeK0CcT76l4vc9cAtgE53+ChfsFUG7iyfpqEhutW3nQlNsTXyUeehocRJ4S ZFVNaywoaFF5olsSBtlu4dEj17u/4SE/ojesrPmuzMGegSkQo3fwS7PUI8KzD5dwKD08 YoZUZL1boIq9Dmuic+wQLPbnQC7Fcy3f25t25RsgxLtuKnGpztXMs4TeDu+3xLylLdI5 5CmA== X-Gm-Message-State: AGRZ1gLnrPo8poGNtu7CmxTXjQJ+SRiqcZKUl/oY3gXijOnIwW8ldsjy 6ei6h86uVuQN/XtP+imLHFJABQpXID415mMKSNZcRrUD X-Google-Smtp-Source: AJdET5foz5Ki0oPT4uIazMBguIB3hlv99D6Y+aXCTaMyMcFl0+A8MKiPy4lOFuk7/k8tS9wFLc8pBrcDgSXQTHbD57Y= X-Received: by 2002:a2e:2281:: with SMTP id i123-v6mr7365104lji.154.1541185508234; Fri, 02 Nov 2018 12:05:08 -0700 (PDT) MIME-Version: 1.0 From: No Name Date: Fri, 2 Nov 2018 22:05:03 +0300 Message-ID: Subject: Re: NFS + Infiniband problem To: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 19:05:10 -0000 Hi, Change MTU to 1500, and than try max 9000. >ib0: flags=8043 metric 0 mtu 65520 From owner-freebsd-fs@freebsd.org Fri Nov 2 20:59:22 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FB0310DB3B8 for ; Fri, 2 Nov 2018 20:59:22 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660085.outbound.protection.outlook.com [40.107.66.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 107CB7E156; Fri, 2 Nov 2018 20:59:21 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB2026.CANPRD01.PROD.OUTLOOK.COM (52.132.49.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1294.21; Fri, 2 Nov 2018 20:59:17 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b%3]) with mapi id 15.20.1294.024; Fri, 2 Nov 2018 20:59:17 +0000 From: Rick Macklem To: Andriy Gapon , Konstantin Belousov CC: FreeBSD Filesystems , Josh Paetzel Subject: Re: How to fill in the fsid for file systems? Thread-Topic: How to fill in the fsid for file systems? Thread-Index: AQHUb557F1RNqdJl0kuwS+F5J/DjsKU2/twAgAGdwkKAAGQrgIAAehmGgAMmaACAAFwYtg== Date: Fri, 2 Nov 2018 20:59:17 +0000 Message-ID: References: <20181030012240.GM5335@kib.kiev.ua> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB2026; 6:W1WbuPqosvhX4CisreGr6lZjn9iz0Pze+6HIleBm+bzCLiyBiZDXqx48yrGx6FncKoicsVgb2qf6HSZPT1iYeO0NnkhX6EvQJBB6I+/oR20U5/KJ6rSydo0FV4jk8Lc7ap7l/sEY9XwZu+gg7rnJCqkHr+K7dGReMo8QOMhrwCDizpUlPXOf7ekry/v8Rv2iavIWNdwR0YxC1YZoiXew4RjxmnJTqROJI+fLCZg0VR9CTaGKh8St0253xRqbZQEwzCM2VVnnym394sl+XyGbM7QeVlPNGJSu3P3Y3C5qLRHXVpgp4BqUGQt1XHGSl3i5vdWgatNRLLpVDNoJl3s4kkTxQNoaVuSntRgwCUlWX34kP6R8H9gShmDBJH7O9fpkOHBu5KVwMeC0J4sKk/QqFPyDxyhx3RhXYsYo4h1bOoFbG7BwEh64VySsbX7AWP/HukO/yiagDqhcJPrbIcrskA==; 5:bDFu/oGdqtzxMDfeSG+7N2UjpHkBCxPDFWOpnJP1iOwa4ejnsMjjwsk+yvWn4RKRwOR6VExGpY+QDKCjGBvNjI1npBJpflP6N8lrBOHY8e/GY609IMdUQ431MV/d6kLz0H4JHg3qw2agnG9PfgXnyu1MRRf6/S/9REzlYrQ+iyw=; 7:t/enFopTH9MIUgWnr8BnS+ZLduOQnWedVsUC7VLusUESESNm/0Y4kQpAOiC89DoUwsPAbjebioZRJTPjKCeZi5bt+nHG9xBjdt5Y25AG2TRzUjHLPkp2bJgXBfkw/yIjD0A4VYQwR82UKoY52LnOxQ== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 2e07f2a7-8da3-4221-837d-08d641060dcf x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(5600074)(711020)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB2026; x-ms-traffictypediagnostic: YTOPR0101MB2026: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(3231382)(944501410)(52105095)(3002001)(10201501046)(93006095)(93001095)(148016)(149066)(150057)(6041310)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123562045)(20161123558120)(20161123564045)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB2026; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB2026; x-forefront-prvs: 08444C7C87 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(136003)(346002)(396003)(366004)(39860400002)(189003)(199004)(74482002)(4326008)(2900100001)(33656002)(25786009)(14444005)(102836004)(256004)(6506007)(71190400001)(39060400002)(99286004)(71200400001)(305945005)(2906002)(105586002)(5660300001)(76176011)(74316002)(54906003)(7696005)(786003)(316002)(110136005)(68736007)(106356001)(476003)(11346002)(446003)(46003)(93886005)(486006)(14454004)(186003)(229853002)(6436002)(8936002)(8676002)(81156014)(53936002)(97736004)(6246003)(9686003)(55016002)(478600001)(81166006)(86362001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB2026; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: qVz3eX4/ZIptQV877SX1Qtmx15yVDBBy6+aB63IDSHP7sU5REi6uVwroSETUWI9LJl5p1dxthhVTD+uyAOxn24S9b8ClUeDk6ZcEhm47RpDwBG+5EYTvYlpZq+HArSj24U3pQhLkSP+V1zbNSPWkzVAs3BGqdjdD+Orr7Xgkt2vS1hjumiNSu7fZNFww2EKvoY9PVesLn+5YUx50v9SigHchbDAddSnZs9FEx3oYM/30squamLv6ufbEa3pp8T3Re8gXm+eFJwqPwPa1EhxXtyu4uQ7LF9B85eB5V8Tu/xt/0Nb2AgcaDSAx8H3jyoTa7qqQCbKOJB2Gyac3I1Vg6NDrPha6gEopSUs2GZZgKuk= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 2e07f2a7-8da3-4221-837d-08d641060dcf X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Nov 2018 20:59:17.6035 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB2026 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 20:59:22 -0000 Andriy Gapon wrote: >On 31/10/2018 17:50, Rick Macklem wrote: >> Andriy Gapon wrote: >>> One is practical. How do we provide fsid to ZFS filesystems? >>> I mean I would hate to resort to mounting ZFS via fstab just to provide= fsid >>> whereas today ZFS filesystems are mounted auto-magically. >>> We could add a FreeBSD specific fsid ZFS property, but that's also some= extra code. >> Good point. I'm not a ZFS guy, so I wouldn't have thought of this. > >A counter-point to my own point. If we implement the fsid override in the >common code, like vfs_do[n]mount, then we would not need to worry about an= y >filesystem specifics. Yes, and it would make the code easily MFC'able, since it avoids any change= s to struct export_args and mountd.c. There probably is still an argument for rev'ing struct export_args (making ex_flags 64bits so that the high order bits of mnt_flag don't get truncated= ) and allowing the anonymous user to have more than 16 gids, but there would be no rush nor a need to MFC this, I think? I'll code up a generic "fsid=3DN" option and see what others think of the p= atch, rick From owner-freebsd-fs@freebsd.org Fri Nov 2 21:23:04 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F256A10DBD4B for ; Fri, 2 Nov 2018 21:23:03 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670044.outbound.protection.outlook.com [40.107.67.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 95BE47F600; Fri, 2 Nov 2018 21:23:03 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB1866.CANPRD01.PROD.OUTLOOK.COM (52.132.48.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1294.21; Fri, 2 Nov 2018 21:23:02 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b%3]) with mapi id 15.20.1294.024; Fri, 2 Nov 2018 21:23:02 +0000 From: Rick Macklem To: Warner Losh , Konstantin Belousov CC: FreeBSD FS , Andriy Gapon Subject: Re: How to fill in the fsid for file systems? Thread-Topic: How to fill in the fsid for file systems? Thread-Index: AQHUb557F1RNqdJl0kuwS+F5J/DjsKU2/twAgAGdwkKAAGQrgIAAehmGgAELs4CAATs6uIAATiWAgAACTACAAE8CgIAAKTUAgAB1nRk= Date: Fri, 2 Nov 2018 21:23:02 +0000 Message-ID: References: <20181030012240.GM5335@kib.kiev.ua> <4269ff9a-8e11-f441-fbb5-b23a6d8e253b@FreeBSD.org> <20181102113609.GL5335@kib.kiev.ua>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB1866; 6:nQ39sTzQkeKDyrJGFAnjbi7HsYLopgaWrw3IjZW/cdVRwP2lXuweGZT4gfld4DT3Q0QRC5OvvhxpJ2SBJlF6Bw0/tl+CbLzdE9KsWm6yUNvhRi6spJW8E86TIe7LUH5rq3z5iJ/XqHHjECDkrWk48fk1C/6kDxYtOgnB8AUKOf0oFUFUv2rsIUONVnQrLzluEnNM4wnvWqeIZ/jhKSrF540bhLGRzDoJe7EDCECvU42ISPqU3DbOQBIk5Ka3ReA1rin0syjPC4q/fzevaPQTqSHpWE+L8RDYcLEiQbKgEmUTq8eNgfLW6zpi8UCckoYxWiwG3GJweOG/jzmfxcYY26VhEONccm8mlYD2RDg+ACjFxHEapFzXsdmSgRm7YHfipL10/k5U5fyGwPkj7WV4QLXS8w0JEwchcTF6jdg41DOmcHbDRnYEsg+KaKRF1hL1LLlisnZSoVD7IT5p6xEuAQ==; 5:yEGXQ777KSX63FP6qeABaw/gX31Mub/2rhxP3kcJdGnYYcLNNHS8+vX3iJhu4GQJctD650GJgKxwdEhylV0FVcAIP2A5VK4PxC7yPOOGQt4Errx7Qvt6FoiqUWi6QdE4sx3BobZ3+S+3FafrtWawonn6lq9K1D1eviddHc8zEys=; 7:SK2rxFjlWXz0HOiCssWccvbQRGGVlCUGEIL+Emhj8m2eZBmrF0VitAdPqG3NIPv+d3XvyRGHNOF8C7NLsRwOiuL+Xhw06JSA5FZyzlBf5bdHMhfWOaffq1FvqMRqAcLoHx4nxGxVtZzBds3ziSUOlw== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 87760efa-5c52-4e67-6658-08d641095f22 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB1866; x-ms-traffictypediagnostic: YTOPR0101MB1866: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(3231382)(944501410)(52105095)(3002001)(10201501046)(93006095)(93001095)(148016)(149066)(150057)(6041310)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123560045)(20161123562045)(20161123558120)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB1866; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB1866; x-forefront-prvs: 08444C7C87 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(136003)(366004)(39860400002)(376002)(396003)(51444003)(199004)(189003)(102836004)(316002)(81166006)(106356001)(74316002)(8936002)(6436002)(9686003)(7696005)(81156014)(105586002)(68736007)(4326008)(6246003)(6506007)(76176011)(55016002)(8676002)(33656002)(53936002)(71200400001)(99286004)(305945005)(786003)(93886005)(25786009)(229853002)(97736004)(446003)(476003)(2900100001)(11346002)(486006)(39060400002)(71190400001)(74482002)(186003)(86362001)(46003)(54906003)(478600001)(110136005)(5660300001)(14454004)(256004)(2906002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB1866; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: S0Jijj5LDTGswEdM0CzzwcAGtOD4LX8VX2Gri7NTxSnnebCsZLGu9mz1Wb7UzD4Vm35OvvgNqPPu7P7GRPk53EZVqysBqPdp6qv5cj0mnvbdpM5z1bh0/HVsv+JYfhZKBpqL3YRHZH746aEBKOrVyTfUvZjCVDlOWs31LUoGqxT9iQkBA3vDHYyCxmaxEnObqVN7zkRQQD1yS+BqxsQK/ouwDbk2BkhLste4tr3/l/jI22t2gqWmZChAG9ZSv9RIrYoGjRzzO8qvkIW1Ny6UPO4hVTUY4N3X5eKcZk9DxqHOjslKNCxHk4gy6U3JTNQfZ6kS35IQx2lUGZtdKhyucnYHsvXER0XmN908zjlTFj4= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 87760efa-5c52-4e67-6658-08d641095f22 X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Nov 2018 21:23:02.5636 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1866 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 21:23:04 -0000 Warner Losh wrote: >Konstantin Belousov wrote: [stuff snipped] >> >> I believe userspace nfs server implementations exist, and they have to u= se >> fhopen(2). >> Yes, I had not thought of these, but it makes sense that they would use fho= pen(2). NFS-Ganesha is quite widely used on Linux and is being integrated with Glus= terFS, which could result in a largely scalable NFS server (which supports NFSv4.1= and pNFS). My experience with GlusterFS performance wasn't good, but I think that was because I was using fuse and the kernel based NFS server (lots of userspace= /kernel boundary crossings, context switches, etc). Unfortunately NFS-Ganesha dropped their FreeBSD port when they went to version 2 (because they started using very Linux specific thread primitives= , if I recall correctly). However, maybe someone will step up to get it ported aga= in? Also, Isilon has some kind of proprietary NFS/SMB server that runs in users= pace, I think. No idea if it uses fhopen(2)? >You are correct. NFS is the reason these interfaces exist because it has >historically (or at least initially) been implemented in userspace. Well, I guess my grey hair is showing... I believe the first open source (although it predated that term) NFS server= implementation was the code I did that was first released in 4.3BSD Reno. = (Admittedly it didn't get "open sourced" until the Networking-2 release cam= e out.) It was always kernel based and helped a few small startups like Netapp to g= et going. (Just to be clear, I'm not suggesting that the current Ontap has any= of that code in it, but I think they used it in the early days.) Most early NFS servers for unix-like systems were kernel based, using the proprietary Sun Reference port. The first NFS server for Linux was userspace, from what I recall. rick= From owner-freebsd-fs@freebsd.org Sat Nov 3 02:49:07 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C41710EA64E for ; Sat, 3 Nov 2018 02:49:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1A0646DA84 for ; Sat, 3 Nov 2018 02:49:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id D2EFF10EA64B; Sat, 3 Nov 2018 02:49:06 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1D2910EA64A for ; Sat, 3 Nov 2018 02:49:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5E2EB6DA81 for ; Sat, 3 Nov 2018 02:49:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 8E0D8FD9 for ; Sat, 3 Nov 2018 02:49:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wA32n5Nj084313 for ; Sat, 3 Nov 2018 02:49:05 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wA32n5eY084312 for fs@FreeBSD.org; Sat, 3 Nov 2018 02:49:05 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 232738] ZFS panics on device removal (solaris assert: cvd->vdev_ashift == spa->spa_max_ashift) Date: Sat, 03 Nov 2018 02:49:05 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: koobs@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc keywords cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Nov 2018 02:49:07 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D232738 Kubilay Kocak changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|13-current zfs device |ZFS panics on device |removal assfail |removal (solaris assert: | |cvd->vdev_ashift =3D=3D | |spa->spa_max_ashift) Keywords| |crash, needs-qa CC| |fs@FreeBSD.org Status|New |Open --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Sat Nov 3 03:10:48 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 512C110EB4C0 for ; Sat, 3 Nov 2018 03:10:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E1CB06EC28 for ; Sat, 3 Nov 2018 03:10:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id A394110EB4BF; Sat, 3 Nov 2018 03:10:47 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8FB4F10EB4BE for ; Sat, 3 Nov 2018 03:10:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 319B76EC21 for ; Sat, 3 Nov 2018 03:10:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 7087D12ED for ; Sat, 3 Nov 2018 03:10:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wA33Aksn065777 for ; Sat, 3 Nov 2018 03:10:46 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wA33AkEj065752 for fs@FreeBSD.org; Sat, 3 Nov 2018 03:10:46 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 198457] zfs acl lost after zfs send-receive. Kernel panic Date: Sat, 03 Nov 2018 03:10:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Nov 2018 03:10:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D198457 --- Comment #8 from commit-hook@freebsd.org --- A commit references this bug: Author: mav Date: Sat Nov 3 03:10:06 UTC 2018 New revision: 340096 URL: https://svnweb.freebsd.org/changeset/base/340096 Log: 9952 Block size change during zfs receive drops spill block Replication code in receive_object() falsely assumes that if received object block size is different from local, then it must be a new object and calls dmu_object_reclaim() to wipe it out. In most cases it is not a problem, since all dnode, bonus buffer and data block(s) are immediately rewritten any way, but the problem is that spill block (if used) is not. This means loss of ACLs, extended attributes, etc. This issue can be triggered in very simple way: 1. create 4KB file with 10+ ACL entries; 2. take snapshot and send it to different dataset; 3. append another 4KB to the file; 4. take another snapshot and send incrementally; 5. witness ACL loss on receive side. PR: 198457 Discussed with: mahrens MFC after: 2 weeks Sponsored by: iXsystems, Inc. Changes: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Nov 3 03:14:23 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E44F10EB6EA for ; Sat, 3 Nov 2018 03:14:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C45086EEAE for ; Sat, 3 Nov 2018 03:14:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 895FC10EB6E9; Sat, 3 Nov 2018 03:14:22 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7824D10EB6E8 for ; Sat, 3 Nov 2018 03:14:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 15FA56EEAC for ; Sat, 3 Nov 2018 03:14:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 11C4D1456 for ; Sat, 3 Nov 2018 03:14:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wA33EK3g084991 for ; Sat, 3 Nov 2018 03:14:20 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wA33EKG9084990 for fs@FreeBSD.org; Sat, 3 Nov 2018 03:14:20 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 198457] zfs acl lost after zfs send-receive. Kernel panic Date: Sat, 03 Nov 2018 03:14:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mav@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: mav@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Nov 2018 03:14:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D198457 Alexander Motin changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |In Progress Assignee|fs@FreeBSD.org |mav@FreeBSD.org CC| |mav@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.=