From owner-freebsd-fs@freebsd.org Sun Feb 26 00:02:58 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9CE3FCED6FB for ; Sun, 26 Feb 2017 00:02:58 +0000 (UTC) (envelope-from gldisater@gmail.com) Received: from mail-it0-x22f.google.com (mail-it0-x22f.google.com [IPv6:2607:f8b0:4001:c0b::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 631A1D7C for ; Sun, 26 Feb 2017 00:02:58 +0000 (UTC) (envelope-from gldisater@gmail.com) Received: by mail-it0-x22f.google.com with SMTP id 203so47258594ith.0 for ; Sat, 25 Feb 2017 16:02:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=iXgiW1oVqU1hzaJO6Uodx/GirdhZ6x6sXcgBn4udzA8=; b=mwtKrMrZskNt9v3qoOd/a2pHStrgH2v85H4yEPIh988AsTntcPiJY8/fHKkgmH1Y8N 0hV2X3Ch2nkKzUY9HhUBDWj3c/iKuw6dqei6N7Vs1hFcaPtrWuCqiOtY03qvQJlTLBWD ZLZu3IRjkcRLOt76ErmNuUJl2/B+MJHN2IqfvI50A45LPuNSXIh+EB+yfEDw5oAg8tLm f9lb0vWLvR+/6aD48dl/YrZste/BASUUI5eoCjoWNd6sQ9YMr9gP75OMXqlLeK66Zt17 w2MG9xgGqJlR8pb+5rMaX7NBW69N9hTF5SH/5NEBpzuzVmFe2zLCudke7G9MuXjVBtMq 85pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=iXgiW1oVqU1hzaJO6Uodx/GirdhZ6x6sXcgBn4udzA8=; b=pP5vdj2rAVRLX4bBqcdEi0JzU9F1zrSzbcVR9vMPLmwAdcq2/3j4vhs9fX5LIb9RmM ZxlkFY9j0L3ABJYUDDr4EYcvd9jzzmwWqzenl1OlxH6/2gF9zIpDmL30S5iX7Q1s3gu8 EQ2F3ZKO+bbGH6uxdC/GI8eduh6xRXxKigBuKJy0Bevi33S9QvaYvuG+TFUeHCBpuTE3 kas+HmiQPfzD/nDnEOGp3yHWqREWYyr0ERkG4FBKtnkkc6jxR6GeUYH8q/T4Y/SpuUKi gs9Yp0lIHRF5If9cprnTXx0I9RpUJ/s8hjsLThoGStHT+5AUP/11vqOVyocku/mXLorx 2jwg== X-Gm-Message-State: AMke39lEq2BjlgKt2kocmmkUWA9X7ud+TwxyrQBikU6/NohY4MYZiAKlcItsZtwwdex1rg== X-Received: by 10.36.70.145 with SMTP id j139mr8966613itb.111.1488067377773; Sat, 25 Feb 2017 16:02:57 -0800 (PST) Received: from [192.168.1.152] (dhcp-108-168-15-83.cable.user.start.ca. [108.168.15.83]) by smtp.gmail.com with ESMTPSA id g200sm2716620ioe.28.2017.02.25.16.02.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 25 Feb 2017 16:02:56 -0800 (PST) Subject: Re: FreeBSD restartable send/receive over WAN To: freebsd-fs@freebsd.org References: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB0FD40@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB11492@QLEXC01.Quorum.local> From: Jeremy Faulkner Message-ID: <67d44270-b644-612b-38be-30a038f45746@gmail.com> Date: Sat, 25 Feb 2017 19:03:07 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB11492@QLEXC01.Quorum.local> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2017 00:02:58 -0000 Pipe it through a compressor On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: > Hi, > > I just tried restartable send/receive in 10.3 and it works like a charm. I was wondering if compressed send has made its way into FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the -c/--compressed option. Any pointers? > > Regards, > Shiva > > > ________________________________________ > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on behalf of Adam Nowacki [nowakpl@platinum.linux.pl] > Sent: Thursday, February 16, 2017 10:41 AM > To: freebsd-fs@freebsd.org > Subject: Re: FreeBSD restartable send/receive over WAN > > On 2017-02-16 19:22, Shiva Bhanujan wrote: >> Hello, >> >> I was wondering if restartable send/receive is available in FreeBSD? We're running 10.2 and have a requirement of sending and receiving ZFS snapshots over a WAN link. The snapshots could be more than a few terabytes. >> >> Can somebody please give me pointers, and if this feature is or isn't available in FreeBSD? > > FreeBSD 10.3 and later. > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Sun Feb 26 11:02:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52DF1CED884 for ; Sun, 26 Feb 2017 11:02:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 42122300 for ; Sun, 26 Feb 2017 11:02:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v1QB2iDA039580 for ; Sun, 26 Feb 2017 11:02:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 217062] for file systems mounted with -o noexec, exec=off property does not work for mmap Date: Sun, 26 Feb 2017 11:02:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? mfc-stable11? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2017 11:02:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217062 --- Comment #11 from commit-hook@freebsd.org --- A commit references this bug: Author: kib Date: Sun Feb 26 11:02:14 UTC 2017 New revision: 314298 URL: https://svnweb.freebsd.org/changeset/base/314298 Log: MFC r313967: Apply noexec mount option for mmap(PROT_EXEC). PR: 217062 Changes: _U stable/11/ stable/11/sys/fs/devfs/devfs_vnops.c stable/11/sys/kern/vfs_vnops.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Feb 26 14:09:02 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 993DBCEEA39 for ; Sun, 26 Feb 2017 14:09:02 +0000 (UTC) (envelope-from Shiva.Bhanujan@Quorum.net) Received: from mail.quorumlabs.com (mail.quorum.net [64.74.133.216]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "mail.quorumlabs.com", Issuer "mail.quorumlabs.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7AF8BCA1 for ; Sun, 26 Feb 2017 14:09:01 +0000 (UTC) (envelope-from Shiva.Bhanujan@Quorum.net) Received: from QLEXC01.Quorum.local ([fe80::79eb:fde8:ed8e:5648]) by QLEXC01.Quorum.local ([fe80::79eb:fde8:ed8e:5648%14]) with mapi id 14.02.0318.001; Sun, 26 Feb 2017 06:09:00 -0800 From: Shiva Bhanujan To: Jeremy Faulkner , "freebsd-fs@freebsd.org" Subject: RE: FreeBSD restartable send/receive over WAN Thread-Topic: FreeBSD restartable send/receive over WAN Thread-Index: AdKIga96XkuYo0TvTyGgnDeYS4b6VAARaL2AAbS/t0UAGxyIgAAMgRdh Date: Sun, 26 Feb 2017 14:08:59 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB114C7@QLEXC01.Quorum.local> References: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB0FD40@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB11492@QLEXC01.Quorum.local>, <67d44270-b644-612b-38be-30a038f45746@gmail.com> In-Reply-To: <67d44270-b644-612b-38be-30a038f45746@gmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [24.6.174.236] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2017 14:09:02 -0000 The compression that we use on our ZFS filesystems is lz4. So, if I have t= o pipe it through a compression algorithm, that'd be uncompressing and comp= ressing it 4 times.=0A= =0A= disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network) -> unco= mpress (gzip) -> zfs recv (compress) -> disk (lz4)=0A= =0A= isn't this quite expensive? We have to transfer multi terabyte files on a = WAN link. I'm also of the understanding that gzip by itself is single-thre= aded, so that'd peg one of the CPUs to 100%. there might be other compress= ion algorithms that can be used, but sending the ZFS as it is compressed on= the filesystem is something that would be optimal, and would reduce the ov= erhead of the additional [de]compressions that are taking place?=0A= =0A= =0A= ________________________________________=0A= From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on behalf= of Jeremy Faulkner [gldisater@gmail.com]=0A= Sent: Saturday, February 25, 2017 4:03 PM=0A= To: freebsd-fs@freebsd.org=0A= Subject: Re: FreeBSD restartable send/receive over WAN=0A= =0A= Pipe it through a compressor=0A= =0A= On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:=0A= > Hi,=0A= >=0A= > I just tried restartable send/receive in 10.3 and it works like a charm. = I was wondering if compressed send has made its way into FreeBSD? I check= ed 10.3 and 11.0-RELEASE, and I don't see the -c/--compressed option. Any = pointers?=0A= >=0A= > Regards,=0A= > Shiva=0A= >=0A= >=0A= > ________________________________________=0A= > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on beha= lf of Adam Nowacki [nowakpl@platinum.linux.pl]=0A= > Sent: Thursday, February 16, 2017 10:41 AM=0A= > To: freebsd-fs@freebsd.org=0A= > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >=0A= > On 2017-02-16 19:22, Shiva Bhanujan wrote:=0A= >> Hello,=0A= >>=0A= >> I was wondering if restartable send/receive is available in FreeBSD? We= 're running 10.2 and have a requirement of sending and receiving ZFS snapsh= ots over a WAN link. The snapshots could be more than a few terabytes.=0A= >>=0A= >> Can somebody please give me pointers, and if this feature is or isn't av= ailable in FreeBSD?=0A= >=0A= > FreeBSD 10.3 and later.=0A= >=0A= > _______________________________________________=0A= > freebsd-fs@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= > _______________________________________________=0A= > freebsd-fs@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= >=0A= _______________________________________________=0A= freebsd-fs@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= From owner-freebsd-fs@freebsd.org Sun Feb 26 15:16:31 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3AE5ACEED30 for ; Sun, 26 Feb 2017 15:16:31 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EEDABC1F for ; Sun, 26 Feb 2017 15:16:30 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.88 (FreeBSD)) (envelope-from ) id 1ci0Ym-0001NF-F3; Sun, 26 Feb 2017 15:16:28 +0000 Date: Sun, 26 Feb 2017 15:16:28 +0000 From: Gary Palmer To: Shiva Bhanujan Cc: Jeremy Faulkner , "freebsd-fs@freebsd.org" Subject: Re: FreeBSD restartable send/receive over WAN Message-ID: <20170226151628.GB51196@in-addr.com> References: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB0FD40@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB11492@QLEXC01.Quorum.local> <67d44270-b644-612b-38be-30a038f45746@gmail.com> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB114C7@QLEXC01.Quorum.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB114C7@QLEXC01.Quorum.local> X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2017 15:16:31 -0000 On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote: > The compression that we use on our ZFS filesystems is lz4. So, if I have to pipe it through a compression algorithm, that'd be uncompressing and compressing it 4 times. > > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network) -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4) > > isn't this quite expensive? We have to transfer multi terabyte files on a WAN link. I'm also of the understanding that gzip by itself is single-threaded, so that'd peg one of the CPUs to 100%. there might be other compression algorithms that can be used, but sending the ZFS as it is compressed on the filesystem is something that would be optimal, and would reduce the overhead of the additional [de]compressions that are taking place? Without going into the efficiency part of your message: archivers/pigz: Parallel GZIP archivers/pbzip2: Parallel BZIP2 archivers/pixz: Parallel, indexing version of XZ archivers/pxz: Parallel LZMA compressor using liblzma Regards, Gary > > ________________________________________ > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on behalf of Jeremy Faulkner [gldisater@gmail.com] > Sent: Saturday, February 25, 2017 4:03 PM > To: freebsd-fs@freebsd.org > Subject: Re: FreeBSD restartable send/receive over WAN > > Pipe it through a compressor > > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: > > Hi, > > > > I just tried restartable send/receive in 10.3 and it works like a charm. I was wondering if compressed send has made its way into FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the -c/--compressed option. Any pointers? > > > > Regards, > > Shiva > > > > > > ________________________________________ > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on behalf of Adam Nowacki [nowakpl@platinum.linux.pl] > > Sent: Thursday, February 16, 2017 10:41 AM > > To: freebsd-fs@freebsd.org > > Subject: Re: FreeBSD restartable send/receive over WAN > > > > On 2017-02-16 19:22, Shiva Bhanujan wrote: > >> Hello, > >> > >> I was wondering if restartable send/receive is available in FreeBSD? We're running 10.2 and have a requirement of sending and receiving ZFS snapshots over a WAN link. The snapshots could be more than a few terabytes. > >> > >> Can somebody please give me pointers, and if this feature is or isn't available in FreeBSD? > > > > FreeBSD 10.3 and later. > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Sun Feb 26 15:50:57 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9F1CCEE4E4 for ; Sun, 26 Feb 2017 15:50:57 +0000 (UTC) (envelope-from nowakpl@platinum.linux.pl) Received: from platinum.edu.pl (platinum.edu.pl [137.74.1.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7DC54E4B for ; Sun, 26 Feb 2017 15:50:57 +0000 (UTC) (envelope-from nowakpl@platinum.linux.pl) Received: from platinum.edu.pl (localhost [127.0.0.1]) by platinum.edu.pl (Postfix) with ESMTP id E5CE578022F for ; Sun, 26 Feb 2017 16:41:52 +0100 (CET) Received: by platinum.edu.pl (Postfix, from userid 87) id D5303780228; Sun, 26 Feb 2017 16:41:52 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on platinum.edu.pl X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.1 Received: from [10.255.1.11] (unknown [109.231.16.141]) by platinum.edu.pl (Postfix) with ESMTPA id 4251B780228 for ; Sun, 26 Feb 2017 16:41:51 +0100 (CET) Subject: Re: FreeBSD restartable send/receive over WAN To: freebsd-fs@freebsd.org References: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB0FD40@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB11492@QLEXC01.Quorum.local> <67d44270-b644-612b-38be-30a038f45746@gmail.com> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB114C7@QLEXC01.Quorum.local> From: Adam Nowacki Message-ID: <504fda7c-260e-74dd-1de8-cbdb542cdfbb@platinum.linux.pl> Date: Sun, 26 Feb 2017 16:41:54 +0100 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB114C7@QLEXC01.Quorum.local> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2017 15:50:57 -0000 On 2017-02-26 15:08, Shiva Bhanujan wrote: > The compression that we use on our ZFS filesystems is lz4. So, if I have to pipe it through a compression algorithm, that'd be uncompressing and compressing it 4 times. > > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network) -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4) > > isn't this quite expensive? We have to transfer multi terabyte files on a WAN link. I'm also of the understanding that gzip by itself is single-threaded, so that'd peg one of the CPUs to 100%. there might be other compression algorithms that can be used, but sending the ZFS as it is compressed on the filesystem is something that would be optimal, and would reduce the overhead of the additional [de]compressions that are taking place? Well, yes, there is more cpu work but also better compression ratio. It all depends on how fast you can actually transfer the data over that WAN link. xz -0 should consume 5-10MB/s of uncompressed data while compressing 2x better than LZ4. > > > ________________________________________ > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on behalf of Jeremy Faulkner [gldisater@gmail.com] > Sent: Saturday, February 25, 2017 4:03 PM > To: freebsd-fs@freebsd.org > Subject: Re: FreeBSD restartable send/receive over WAN > > Pipe it through a compressor > > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: >> Hi, >> >> I just tried restartable send/receive in 10.3 and it works like a charm. I was wondering if compressed send has made its way into FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the -c/--compressed option. Any pointers? >> >> Regards, >> Shiva From owner-freebsd-fs@freebsd.org Sun Feb 26 16:56:18 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD1CDCEE993 for ; Sun, 26 Feb 2017 16:56:18 +0000 (UTC) (envelope-from karli.sjoberg@slu.se) Received: from Exch2-2.slu.se (pop.slu.se [77.235.224.122]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "webmail.slu.se", Issuer "TERENA SSL CA 3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 35017755; Sun, 26 Feb 2017 16:56:17 +0000 (UTC) (envelope-from karli.sjoberg@slu.se) Received: from exch2-4.slu.se (77.235.224.124) by Exch2-2.slu.se (77.235.224.122) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Sun, 26 Feb 2017 17:41:05 +0100 Received: from exch2-4.slu.se ([fe80::e006:857e:1307:5005]) by exch2-4.slu.se ([fe80::e006:857e:1307:5005%22]) with mapi id 15.00.1236.000; Sun, 26 Feb 2017 17:41:05 +0100 From: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= To: Gary Palmer CC: Shiva Bhanujan , Jeremy Faulkner , "freebsd-fs@freebsd.org" Subject: Re: FreeBSD restartable send/receive over WAN Thread-Topic: FreeBSD restartable send/receive over WAN Thread-Index: AQHSkE8fXkuYo0TvTyGgnDeYS4b6VA== Date: Sun, 26 Feb 2017 16:41:04 +0000 Message-ID: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> Accept-Language: sv-SE, en-US Content-Language: sv-SE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2017 16:56:19 -0000 DQpEZW4gMjYgZmViLiAyMDE3IDQ6MTYgZW0gc2tyZXYgR2FyeSBQYWxtZXIgPGdwYWxtZXJAZnJl ZWJzZC5vcmc+Og0KPg0KPiBPbiBTdW4sIEZlYiAyNiwgMjAxNyBhdCAwMjowODo1OVBNICswMDAw LCBTaGl2YSBCaGFudWphbiB3cm90ZToNCj4gPiBUaGUgY29tcHJlc3Npb24gdGhhdCB3ZSB1c2Ug b24gb3VyIFpGUyBmaWxlc3lzdGVtcyBpcyBsejQuICBTbywgaWYgSSBoYXZlIHRvIHBpcGUgaXQg dGhyb3VnaCBhIGNvbXByZXNzaW9uIGFsZ29yaXRobSwgdGhhdCdkIGJlIHVuY29tcHJlc3Npbmcg YW5kIGNvbXByZXNzaW5nIGl0IDQgdGltZXMuDQo+ID4NCj4gPiBkaXNrIChsejQpIC0+IHpmcyBz ZW5kICh1bmNvbXByZXNzKSAtPiBjb21wcmVzcyAoZ3ppcCkgLT4gKG5ldHdvcmspIC0+IHVuY29t cHJlc3MgKGd6aXApIC0+IHpmcyByZWN2IChjb21wcmVzcykgLT4gZGlzayAobHo0KQ0KPiA+DQo+ ID4gaXNuJ3QgdGhpcyBxdWl0ZSBleHBlbnNpdmU/ICBXZSBoYXZlIHRvIHRyYW5zZmVyIG11bHRp IHRlcmFieXRlIGZpbGVzIG9uIGEgV0FOIGxpbmsuICBJJ20gYWxzbyBvZiB0aGUgdW5kZXJzdGFu ZGluZyB0aGF0IGd6aXAgYnkgaXRzZWxmIGlzIHNpbmdsZS10aHJlYWRlZCwgc28gdGhhdCdkIHBl ZyBvbmUgb2YgdGhlIENQVXMgdG8gMTAwJS4gIHRoZXJlIG1pZ2h0IGJlIG90aGVyIGNvbXByZXNz aW9uIGFsZ29yaXRobXMgdGhhdCBjYW4gYmUgdXNlZCwgYnV0IHNlbmRpbmcgdGhlIFpGUyBhcyBp dCBpcyBjb21wcmVzc2VkIG9uIHRoZSBmaWxlc3lzdGVtIGlzIHNvbWV0aGluZyB0aGF0IHdvdWxk IGJlIG9wdGltYWwsIGFuZCB3b3VsZCByZWR1Y2UgdGhlIG92ZXJoZWFkIG9mIHRoZSBhZGRpdGlv bmFsIFtkZV1jb21wcmVzc2lvbnMgdGhhdCBhcmUgdGFraW5nIHBsYWNlPw0KPg0KPiBXaXRob3V0 IGdvaW5nIGludG8gdGhlIGVmZmljaWVuY3kgcGFydCBvZiB5b3VyIG1lc3NhZ2U6DQo+DQo+IGFy Y2hpdmVycy9waWd6OiBQYXJhbGxlbCBHWklQDQo+IGFyY2hpdmVycy9wYnppcDI6IFBhcmFsbGVs IEJaSVAyDQo+IGFyY2hpdmVycy9waXh6OiBQYXJhbGxlbCwgaW5kZXhpbmcgdmVyc2lvbiBvZiBY Wg0KPiBhcmNoaXZlcnMvcHh6OiBQYXJhbGxlbCBMWk1BIGNvbXByZXNzb3IgdXNpbmcgbGlibHpt YQ0KDQpBbHNvIHdvcnRoIG1lbnRpb25pbmcgaXMsIG9idmlvdXNseToNCmFyY2hpdmVycy9sejQN Cg0KOikNCg0KL0sNCg0KPg0KPiBSZWdhcmRzLA0KPg0KPiBHYXJ5DQo+DQo+ID4NCj4gPiBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ID4gRnJvbTogb3duZXItZnJl ZWJzZC1mc0BmcmVlYnNkLm9yZyBbb3duZXItZnJlZWJzZC1mc0BmcmVlYnNkLm9yZ10gb24gYmVo YWxmIG9mIEplcmVteSBGYXVsa25lciBbZ2xkaXNhdGVyQGdtYWlsLmNvbV0NCj4gPiBTZW50OiBT YXR1cmRheSwgRmVicnVhcnkgMjUsIDIwMTcgNDowMyBQTQ0KPiA+IFRvOiBmcmVlYnNkLWZzQGZy ZWVic2Qub3JnDQo+ID4gU3ViamVjdDogUmU6IEZyZWVCU0QgcmVzdGFydGFibGUgc2VuZC9yZWNl aXZlIG92ZXIgV0FODQo+ID4NCj4gPiBQaXBlIGl0IHRocm91Z2ggYSBjb21wcmVzc29yDQo+ID4N Cj4gPiBPbiAyMDE3LTAyLTI1IDI6MDkgUE0sIFNoaXZhIEJoYW51amFuIHdyb3RlOg0KPiA+ID4g SGksDQo+ID4gPg0KPiA+ID4gSSBqdXN0IHRyaWVkIHJlc3RhcnRhYmxlIHNlbmQvcmVjZWl2ZSBp biAxMC4zIGFuZCBpdCB3b3JrcyBsaWtlIGEgY2hhcm0uICBJIHdhcyB3b25kZXJpbmcgaWYgY29t cHJlc3NlZCBzZW5kIGhhcyBtYWRlIGl0cyB3YXkgaW50byBGcmVlQlNEPyAgSSBjaGVja2VkIDEw LjMgYW5kIDExLjAtUkVMRUFTRSwgYW5kIEkgZG9uJ3Qgc2VlIHRoZSAtYy8tLWNvbXByZXNzZWQg b3B0aW9uLiAgQW55IHBvaW50ZXJzPw0KPiA+ID4NCj4gPiA+IFJlZ2FyZHMsDQo+ID4gPiBTaGl2 YQ0KPiA+ID4NCj4gPiA+DQo+ID4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fDQo+ID4gPiBGcm9tOiBvd25lci1mcmVlYnNkLWZzQGZyZWVic2Qub3JnIFtvd25lci1m cmVlYnNkLWZzQGZyZWVic2Qub3JnXSBvbiBiZWhhbGYgb2YgQWRhbSBOb3dhY2tpIFtub3dha3Bs QHBsYXRpbnVtLmxpbnV4LnBsXQ0KPiA+ID4gU2VudDogVGh1cnNkYXksIEZlYnJ1YXJ5IDE2LCAy MDE3IDEwOjQxIEFNDQo+ID4gPiBUbzogZnJlZWJzZC1mc0BmcmVlYnNkLm9yZw0KPiA+ID4gU3Vi amVjdDogUmU6IEZyZWVCU0QgcmVzdGFydGFibGUgc2VuZC9yZWNlaXZlIG92ZXIgV0FODQo+ID4g Pg0KPiA+ID4gT24gMjAxNy0wMi0xNiAxOToyMiwgU2hpdmEgQmhhbnVqYW4gd3JvdGU6DQo+ID4g Pj4gSGVsbG8sDQo+ID4gPj4NCj4gPiA+PiBJIHdhcyB3b25kZXJpbmcgaWYgcmVzdGFydGFibGUg c2VuZC9yZWNlaXZlIGlzIGF2YWlsYWJsZSBpbiBGcmVlQlNEPyAgV2UncmUgcnVubmluZyAxMC4y IGFuZCBoYXZlIGEgcmVxdWlyZW1lbnQgb2Ygc2VuZGluZyBhbmQgcmVjZWl2aW5nIFpGUyBzbmFw c2hvdHMgb3ZlciBhIFdBTiBsaW5rLiAgVGhlIHNuYXBzaG90cyBjb3VsZCBiZSBtb3JlIHRoYW4g YSBmZXcgdGVyYWJ5dGVzLg0KPiA+ID4+DQo+ID4gPj4gQ2FuIHNvbWVib2R5IHBsZWFzZSBnaXZl IG1lIHBvaW50ZXJzLCBhbmQgaWYgdGhpcyBmZWF0dXJlIGlzIG9yIGlzbid0IGF2YWlsYWJsZSBp biBGcmVlQlNEPw0KPiA+ID4NCj4gPiA+IEZyZWVCU0QgMTAuMyBhbmQgbGF0ZXIuDQo+ID4gPg0K PiA+ID4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4g PiA+IGZyZWVic2QtZnNAZnJlZWJzZC5vcmcgbWFpbGluZyBsaXN0DQo+ID4gPiBodHRwczovL2xp c3RzLmZyZWVic2Qub3JnL21haWxtYW4vbGlzdGluZm8vZnJlZWJzZC1mcw0KPiA+ID4gVG8gdW5z dWJzY3JpYmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVic2QtZnMtdW5zdWJzY3JpYmVAZnJlZWJz ZC5vcmciDQo+ID4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXw0KPiA+ID4gZnJlZWJzZC1mc0BmcmVlYnNkLm9yZyBtYWlsaW5nIGxpc3QNCj4gPiA+IGh0 dHBzOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNkLWZzDQo+ID4g PiBUbyB1bnN1YnNjcmliZSwgc2VuZCBhbnkgbWFpbCB0byAiZnJlZWJzZC1mcy11bnN1YnNjcmli ZUBmcmVlYnNkLm9yZyINCj4gPiA+DQo+ID4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18NCj4gPiBmcmVlYnNkLWZzQGZyZWVic2Qub3JnIG1haWxpbmcgbGlz dA0KPiA+IGh0dHBzOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNk LWZzDQo+ID4gVG8gdW5zdWJzY3JpYmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVic2QtZnMtdW5z dWJzY3JpYmVAZnJlZWJzZC5vcmciDQo+ID4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18NCj4gPiBmcmVlYnNkLWZzQGZyZWVic2Qub3JnIG1haWxpbmcgbGlz dA0KPiA+IGh0dHBzOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNk LWZzDQo+ID4gVG8gdW5zdWJzY3JpYmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVic2QtZnMtdW5z dWJzY3JpYmVAZnJlZWJzZC5vcmciDQo+ID4NCj4gX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX18NCj4gZnJlZWJzZC1mc0BmcmVlYnNkLm9yZyBtYWlsaW5nIGxp c3QNCj4gaHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ZyZWVic2Qt ZnMNCj4gVG8gdW5zdWJzY3JpYmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVic2QtZnMtdW5zdWJz Y3JpYmVAZnJlZWJzZC5vcmciDQo= From owner-freebsd-fs@freebsd.org Sun Feb 26 18:59:19 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9844ECEEE4B for ; Sun, 26 Feb 2017 18:59:19 +0000 (UTC) (envelope-from sem@semmy.ru) Received: from sunner.semmy.ru (sunner.semmy.ru [IPv6:2a00:14d0:0:20::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 59B9889F for ; Sun, 26 Feb 2017 18:59:19 +0000 (UTC) (envelope-from sem@semmy.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=semmy.ru; s=dkim; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Date: Message-ID:Subject:From:To:Sender:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=3CtGurKsqiUns6GO8hGgE9PdyHDpS0z71PI3x6NVi7U=; b=f1Pl8Tjmu2MkmR0WnSzw+PF58w Bipo15yTfgyMuj05Pe8H04lA2hbKgtiAqpnFPn6Fw2tgtjPo+6dV0W4lyEaL2GCT8eO4Rc3coiNNz 2d4NN1oPvW4EwdqtK2CfNBQtu33yDcZWuPWxJ88RgxY1dyyyIQ53vK8vaLpCaUVtchk4=; Received: from [37.204.164.247] (helo=homeipv6.semmy.ru) by sunner.semmy.ru with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.88 (FreeBSD)) (envelope-from ) id 1ci42N-0000R8-3F for freebsd-fs@freebsd.org; Sun, 26 Feb 2017 21:59:15 +0300 To: freebsd-fs@freebsd.org From: Sergey Matveychuk Subject: ufs2+su+journal-snapshots Message-ID: Date: Sun, 26 Feb 2017 21:59:15 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2017 18:59:19 -0000 Hi. I know, it's an annoying question but is there a plan to fix snapshots with journaling? May be any project to try and to test? Journal enabled in default install now and we lost snapshots (and backups). It does ufs2 almost useless now. Sure, I can disable journaling and back to 2000s but it's sadly. Many fellows has moved to ZFS and I did the same on a few servers but often ZFS is overkill (and slow) for me. From owner-freebsd-fs@freebsd.org Sun Feb 26 21:01:08 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB3DBCECC7D for ; Sun, 26 Feb 2017 21:01:08 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DD526CC2 for ; Sun, 26 Feb 2017 21:01:08 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v1QL01eY061275 for ; Sun, 26 Feb 2017 21:01:08 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201702262101.v1QL01eY061275@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Sun, 26 Feb 2017 21:01:08 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Feb 2017 21:01:09 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 203419 | solaris assert: (dn->dn_phys->dn_nlevels == 0 && Open | 211491 | System hangs after "Uptime" on reboot with ZFS 7 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Feb 27 09:08:58 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03748CEFD2A for ; Mon, 27 Feb 2017 09:08:58 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from smtp.rlwinm.de (smtp.rlwinm.de [IPv6:2a01:4f8:201:31ef::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C559F62D for ; Mon, 27 Feb 2017 09:08:57 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from crest.lan.bultmann.eu (unknown [87.253.189.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.rlwinm.de (Postfix) with ESMTPSA id E73497544; Mon, 27 Feb 2017 10:08:54 +0100 (CET) Subject: Re: ufs2+su+journal-snapshots To: Sergey Matveychuk , freebsd-fs@freebsd.org References: From: Jan Bramkamp Message-ID: <83019546-b7a8-52cf-7fed-768874b50d42@rlwinm.de> Date: Mon, 27 Feb 2017 10:08:54 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Feb 2017 09:08:58 -0000 On 26/02/2017 19:59, Sergey Matveychuk wrote: > Hi. > > I know, it's an annoying question but is there a plan to fix snapshots > with journaling? May be any project to try and to test? > > Journal enabled in default install now and we lost snapshots (and > backups). It does ufs2 almost useless now. Sure, I can disable > journaling and back to 2000s but it's sadly. > > Many fellows has moved to ZFS and I did the same on a few servers but > often ZFS is overkill (and slow) for me. I'm just reporting hearsay from the last EuroBSDcon, but afaik there is nobody working on UFS with SU+J and Snapshots. The main problem is the interaction of the SU+J code and Snapshots across reboots. Apparently it would be a lot simpler to implement Snapshots that are lost at shutdown. This would be enough for dump -L and other backup tools. Disabling just the journaling part of the soft-updates isn't that bad for small file systems. It should even improve performance in most cases, but it requires a *background* fsck after an unclean shutdown. On small file systems this isn't a huge problem. It becomes a problem if either the ratio of bandwidth to capacity or the ratio of physical RAM to capacity become too large. In either case ZFS and (a bit) more RAM is the best solution available in FreeBSD. From owner-freebsd-fs@freebsd.org Mon Feb 27 17:01:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3B8ECF018C for ; Mon, 27 Feb 2017 17:01:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 93A74C57 for ; Mon, 27 Feb 2017 17:01:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v1RH146t046156 for ; Mon, 27 Feb 2017 17:01:04 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 216964] boot1.efi: can't boot from ZFS on 4kn HDD Date: Mon, 27 Feb 2017 17:01:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kozlov.sergey.404@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Feb 2017 17:01:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D216964 --- Comment #4 from Sergey Kozlov --- (In reply to Steven Hartland from comment #3) Tried Allan's patch on head, had to do some minor adjustments so the patch applies cleanly. The result is exactly the same. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Feb 28 15:04:24 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E27C4CF0536 for ; Tue, 28 Feb 2017 15:04:24 +0000 (UTC) (envelope-from Shiva.Bhanujan@Quorum.net) Received: from mail.quorumlabs.com (mail.quorum.net [64.74.133.216]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "mail.quorumlabs.com", Issuer "mail.quorumlabs.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CF3B6FB0; Tue, 28 Feb 2017 15:04:23 +0000 (UTC) (envelope-from Shiva.Bhanujan@Quorum.net) Received: from QLEXC01.Quorum.local ([fe80::79eb:fde8:ed8e:5648]) by QLEXC01.Quorum.local ([fe80::79eb:fde8:ed8e:5648%14]) with mapi id 14.02.0318.001; Tue, 28 Feb 2017 07:04:17 -0800 From: Shiva Bhanujan To: =?iso-8859-1?Q?Karli_Sj=F6berg?= , Gary Palmer CC: Jeremy Faulkner , "freebsd-fs@freebsd.org" Subject: RE: FreeBSD restartable send/receive over WAN Thread-Topic: FreeBSD restartable send/receive over WAN Thread-Index: AQHSkE8fXkuYo0TvTyGgnDeYS4b6VKF+hfcf Date: Tue, 28 Feb 2017 15:04:16 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> In-Reply-To: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [24.6.174.236] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2017 15:04:25 -0000 thanks for all the pointers for the compression algorithms. I ran a few te= sts to compare compression overhead. These are local zfs send/receive, and= no network is involved. time zfs send -v | | | zfs receive -= s Here are the performance results that I got. no compression: real 0m20.892s user 0m0.000s sys 0m5.587s xz -0: real 8m38.569s user 10m28.551s sys 0m6.866s pxz -0: real 4m38.448s user 10m55.863s sys 0m13.324s gzip: real 3m51.297s user 4m12.035s sys 0m4.696s lz4: real 0m29.912s user 0m16.543s sys 0m10.810s lz4 has the least overhead in terms of time. pxz/xz seem to be prohibitiv= e give the above results. Unless, there is something basic I'm missing? I was really hoping that compressed sends would be available, as that would= actively eliminate this overhead, given that we use lz4 as the compression= algorithm when writing to disks. ________________________________ From: Karli Sj=F6berg [karli.sjoberg@slu.se] Sent: Sunday, February 26, 2017 8:41 AM To: Gary Palmer Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org Subject: Re: FreeBSD restartable send/receive over WAN Den 26 feb. 2017 4:16 em skrev Gary Palmer : > > On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote: > > The compression that we use on our ZFS filesystems is lz4. So, if I ha= ve to pipe it through a compression algorithm, that'd be uncompressing and = compressing it 4 times. > > > > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network) -> = uncompress (gzip) -> zfs recv (compress) -> disk (lz4) > > > > isn't this quite expensive? We have to transfer multi terabyte files o= n a WAN link. I'm also of the understanding that gzip by itself is single-= threaded, so that'd peg one of the CPUs to 100%. there might be other comp= ression algorithms that can be used, but sending the ZFS as it is compresse= d on the filesystem is something that would be optimal, and would reduce th= e overhead of the additional [de]compressions that are taking place? > > Without going into the efficiency part of your message: > > archivers/pigz: Parallel GZIP > archivers/pbzip2: Parallel BZIP2 > archivers/pixz: Parallel, indexing version of XZ > archivers/pxz: Parallel LZMA compressor using liblzma Also worth mentioning is, obviously: archivers/lz4 :) /K > > Regards, > > Gary > > > > > ________________________________________ > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on be= half of Jeremy Faulkner [gldisater@gmail.com] > > Sent: Saturday, February 25, 2017 4:03 PM > > To: freebsd-fs@freebsd.org > > Subject: Re: FreeBSD restartable send/receive over WAN > > > > Pipe it through a compressor > > > > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: > > > Hi, > > > > > > I just tried restartable send/receive in 10.3 and it works like a cha= rm. I was wondering if compressed send has made its way into FreeBSD? I c= hecked 10.3 and 11.0-RELEASE, and I don't see the -c/--compressed option. = Any pointers? > > > > > > Regards, > > > Shiva > > > > > > > > > ________________________________________ > > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on = behalf of Adam Nowacki [nowakpl@platinum.linux.pl] > > > Sent: Thursday, February 16, 2017 10:41 AM > > > To: freebsd-fs@freebsd.org > > > Subject: Re: FreeBSD restartable send/receive over WAN > > > > > > On 2017-02-16 19:22, Shiva Bhanujan wrote: > > >> Hello, > > >> > > >> I was wondering if restartable send/receive is available in FreeBSD?= We're running 10.2 and have a requirement of sending and receiving ZFS sn= apshots over a WAN link. The snapshots could be more than a few terabytes. > > >> > > >> Can somebody please give me pointers, and if this feature is or isn'= t available in FreeBSD? > > > > > > FreeBSD 10.3 and later. > > > > > > _______________________________________________ > > > freebsd-fs@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > _______________________________________________ > > > freebsd-fs@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Feb 28 19:45:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5938BCF1C58 for ; Tue, 28 Feb 2017 19:45:04 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2064BF2B; Tue, 28 Feb 2017 19:45:02 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1cinhj-0004z7-1B; Tue, 28 Feb 2017 20:44:59 +0100 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: =?utf-8?Q?Karli_Sj=C3=B6berg?= , "Gary Palmer" , "Shiva Bhanujan" Cc: "freebsd-fs@freebsd.org" , "Jeremy Faulkner" Subject: Re: FreeBSD restartable send/receive over WAN References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> Date: Tue, 28 Feb 2017 20:44:58 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: Quoted-Printable From: "Ronald Klop" Message-ID: In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> User-Agent: Opera Mail/12.16 (FreeBSD) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: -0.2 X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED, BAYES_50 autolearn=disabled version=3.4.0 X-Scan-Signature: 326ae57122e7e55f093438bd1fe32346 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2017 19:45:04 -0000 On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan = wrote: > thanks for all the pointers for the compression algorithms. I ran a f= ew = > tests to compare compression overhead. These are local zfs = > send/receive, and no network is involved. > > time zfs send -v | | | zfs = > receive -s > > Here are the performance results that I got. > > no compression: > real 0m20.892s > user 0m0.000s > sys 0m5.587s > > xz -0: > real 8m38.569s > user 10m28.551s > sys 0m6.866s > > pxz -0: > real 4m38.448s > user 10m55.863s > sys 0m13.324s > > gzip: > real 3m51.297s > user 4m12.035s > sys 0m4.696s > > lz4: > real 0m29.912s > user 0m16.543s > sys 0m10.810s > > > lz4 has the least overhead in terms of time. pxz/xz seem to be = > prohibitive give the above results. Unless, there is something basic = = > I'm missing? > > I was really hoping that compressed sends would be available, as that = = > would actively eliminate this overhead, given that we use lz4 as the = > compression algorithm when writing to disks. Why don't you test this with a network in between? That would give much = = more valuable numbers to compare for your use case. The number above say nothing about the gain vs the bottleneck you are = trying to remove. Ronald. > > > ________________________________ > From: Karli Sj=C3=B6berg [karli.sjoberg@slu.se] > Sent: Sunday, February 26, 2017 8:41 AM > To: Gary Palmer > Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org > Subject: Re: FreeBSD restartable send/receive over WAN > > > Den 26 feb. 2017 4:16 em skrev Gary Palmer : >> >> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote: >> > The compression that we use on our ZFS filesystems is lz4. So, if = I = >> have to pipe it through a compression algorithm, that'd be = >> uncompressing and compressing it 4 times. >> > >> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)= = >> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4) >> > >> > isn't this quite expensive? We have to transfer multi terabyte fil= es = >> on a WAN link. I'm also of the understanding that gzip by itself is = = >> single-threaded, so that'd peg one of the CPUs to 100%. there might = be = >> other compression algorithms that can be used, but sending the ZFS as= = >> it is compressed on the filesystem is something that would be optimal= , = >> and would reduce the overhead of the additional [de]compressions that= = >> are taking place? >> >> Without going into the efficiency part of your message: >> >> archivers/pigz: Parallel GZIP >> archivers/pbzip2: Parallel BZIP2 >> archivers/pixz: Parallel, indexing version of XZ >> archivers/pxz: Parallel LZMA compressor using liblzma > > Also worth mentioning is, obviously: > archivers/lz4 > > :) > > /K > >> >> Regards, >> >> Gary >> >> > >> > ________________________________________ >> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] o= n = >> behalf of Jeremy Faulkner [gldisater@gmail.com] >> > Sent: Saturday, February 25, 2017 4:03 PM >> > To: freebsd-fs@freebsd.org >> > Subject: Re: FreeBSD restartable send/receive over WAN >> > >> > Pipe it through a compressor >> > >> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: >> > > Hi, >> > > >> > > I just tried restartable send/receive in 10.3 and it works like a= = >> charm. I was wondering if compressed send has made its way into = >> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the = >> -c/--compressed option. Any pointers? >> > > >> > > Regards, >> > > Shiva >> > > >> > > >> > > ________________________________________ >> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org]= = >> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl] >> > > Sent: Thursday, February 16, 2017 10:41 AM >> > > To: freebsd-fs@freebsd.org >> > > Subject: Re: FreeBSD restartable send/receive over WAN >> > > >> > > On 2017-02-16 19:22, Shiva Bhanujan wrote: >> > >> Hello, >> > >> >> > >> I was wondering if restartable send/receive is available in = >> FreeBSD? We're running 10.2 and have a requirement of sending and = >> receiving ZFS snapshots over a WAN link. The snapshots could be more= = >> than a few terabytes. >> > >> >> > >> Can somebody please give me pointers, and if this feature is or = = >> isn't available in FreeBSD? >> > > >> > > FreeBSD 10.3 and later. >> > > >> > > _______________________________________________ >> > > freebsd-fs@freebsd.org mailing list >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > > To unsubscribe, send any mail to = >> "freebsd-fs-unsubscribe@freebsd.org" >> > > _______________________________________________ >> > > freebsd-fs@freebsd.org mailing list >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > > To unsubscribe, send any mail to = >> "freebsd-fs-unsubscribe@freebsd.org" >> > > >> > _______________________________________________ >> > freebsd-fs@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.or= g" >> > _______________________________________________ >> > freebsd-fs@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.or= g" >> > >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Mar 1 23:06:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A79BDCF44C8 for ; Wed, 1 Mar 2017 23:06:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 970A7C95 for ; Wed, 1 Mar 2017 23:06:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v21N67NB058227 for ; Wed, 1 Mar 2017 23:06:07 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 217440] FAT32 formatted USB stick with files written by PS4 - Invalid argument, unable to list directory contents Date: Wed, 01 Mar 2017 23:06:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Mar 2017 23:06:07 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217440 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 2 00:09:39 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34B80CF4A26 for ; Thu, 2 Mar 2017 00:09:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 24C6EB36 for ; Thu, 2 Mar 2017 00:09:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v2209cew051152 for ; Thu, 2 Mar 2017 00:09:39 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 194513] zfs recv hangs in state kmem arena Date: Thu, 02 Mar 2017 00:09:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: daveb@spectralogic.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Mar 2017 00:09:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194513 Dave Baukus changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |daveb@spectralogic.com --- Comment #21 from Dave Baukus --- (In reply to Steven Hartland from comment #10) Is it possible that the referenced change does not have enough of a positive effect with respect to zfs receive pending in kmem_arena because zfs receiv= e is attempting to allocate SPA_MAXBLOCKSIZE (16M) and the fragmentation check is looking for fragmentation below zfs_max_recordsize (1M) ? /* * Above limits know nothing about real level of KVA fragmentation. * Start aggressive reclamation if too little sequential KVA left. */ if (vmem_size(heap_arena, VMEM_MAXFREE) < zfs_max_recordsize) return (1); --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 2 12:12:27 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A217CF151B for ; Thu, 2 Mar 2017 12:12:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 52E4BB25 for ; Thu, 2 Mar 2017 12:12:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v22CCQ7v001759 for ; Thu, 2 Mar 2017 12:12:27 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 168158] [zfs] incorrect parsing of sharenfs options in zfs (fsshare.c) Date: Thu, 02 Mar 2017 12:12:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: max@mober.at X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Mar 2017 12:12:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D168158 max@mober.at changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |max@mober.at --- Comment #4 from max@mober.at --- FreeBSD 11.0-STABLE, and still the same. Will the patch by eborisch be included? If I want to include it, do I have to rebuild the whole kernel, or is it en= ough to rebuild zfs.ko? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 2 18:06:51 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8A7DCF47BB for ; Thu, 2 Mar 2017 18:06:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C83C5EFD for ; Thu, 2 Mar 2017 18:06:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v22I6pj8088933 for ; Thu, 2 Mar 2017 18:06:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 195565] [zfs] zpool import always dumps core Date: Thu, 02 Mar 2017 18:06:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: la5lbtyi@aon.at X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Overcome By Events X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Mar 2017 18:06:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D195565 Martin Birgmeier changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |Overcome By Events Status|New |Closed --- Comment #3 from Martin Birgmeier --- This does not seem to occur any more on 11.0. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 2 18:29:33 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 63932CF60AA for ; Thu, 2 Mar 2017 18:29:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 533AC1A5 for ; Thu, 2 Mar 2017 18:29:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v22ITXte055764 for ; Thu, 2 Mar 2017 18:29:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 168158] [zfs] incorrect parsing of sharenfs options in zfs (fsshare.c) Date: Thu, 02 Mar 2017 18:29:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: la5lbtyi@aon.at X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Mar 2017 18:29:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D168158 Martin Birgmeier changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |la5lbtyi@aon.at --- Comment #5 from Martin Birgmeier --- Now that this probably starts moving again, may I humbly suggest to use my patch instead for the reasons cited in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D176449#c7 -- Martin --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 2 18:39:12 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D3D4CF62D2 for ; Thu, 2 Mar 2017 18:39:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6D054A89 for ; Thu, 2 Mar 2017 18:39:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v22IdCIY075292 for ; Thu, 2 Mar 2017 18:39:12 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 216964] boot1.efi: can't boot from ZFS on 4kn HDD Date: Thu, 02 Mar 2017 18:39:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: tsoome@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Mar 2017 18:39:12 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D216964 Toomas Soome changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tsoome@freebsd.org --- Comment #5 from Toomas Soome --- (In reply to Sergey Kozlov from comment #4) The hint is on the screenshot; error 4 is EFI_BAD_BUFFER_SIZE; and indeed, = the read size is 2048, thats half the sector. So the solution is that we need b= oot1 to use buffer if the sector size is >512B. The similar problem most likely will hit us in loader.efi. The UFS is working because the ufs is using 4k data blocks - that is, all r= eads are at least 4k. Hm, I think... probably the best solution is to make sure zfs code does pro= vide at least sector size buffer for reads. It is quite easy to do with 12-curre= nt, for 11, we would need to backport some updates. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Mar 2 19:46:44 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2AB3CF5412 for ; Thu, 2 Mar 2017 19:46:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C23D0163 for ; Thu, 2 Mar 2017 19:46:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v22Jkicn037855 for ; Thu, 2 Mar 2017 19:46:44 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 216964] boot1.efi: can't boot from ZFS on 4kn HDD Date: Thu, 02 Mar 2017 19:46:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: tsoome@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Mar 2017 19:46:44 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D216964 Toomas Soome changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |In Progress --- Comment #6 from Toomas Soome --- Please check if the patch from https://reviews.freebsd.org/D9870 will help. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Mar 3 00:35:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A30CDCF6F5E for ; Fri, 3 Mar 2017 00:35:04 +0000 (UTC) (envelope-from Shiva.Bhanujan@Quorum.net) Received: from mail.quorumlabs.com (mail.quorum.net [64.74.133.216]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "mail.quorumlabs.com", Issuer "mail.quorumlabs.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 90B93198B; Fri, 3 Mar 2017 00:35:03 +0000 (UTC) (envelope-from Shiva.Bhanujan@Quorum.net) Received: from QLEXC01.Quorum.local ([fe80::79eb:fde8:ed8e:5648]) by QLEXC01.Quorum.local ([fe80::79eb:fde8:ed8e:5648%14]) with mapi id 14.02.0318.001; Thu, 2 Mar 2017 16:34:57 -0800 From: Shiva Bhanujan To: Ronald Klop , =?iso-8859-1?Q?Karli_Sj=F6berg?= , Gary Palmer CC: "freebsd-fs@freebsd.org" , Jeremy Faulkner Subject: RE: FreeBSD restartable send/receive over WAN Thread-Topic: FreeBSD restartable send/receive over WAN Thread-Index: AQHSkE8fXkuYo0TvTyGgnDeYS4b6VKF+hfcfgADVzwCAAu69hQ== Date: Fri, 3 Mar 2017 00:34:57 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.20.7.75] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2017 00:35:04 -0000 I ran the same set of tests between 2 FreeBSD instances, connected on a 1G = LAN. The the comms was setup using mbuffer. Here's the basic command.=0A= =0A= time zfs send -v | | mbuffer -O :8099 -b 1024 -m = 128M -P 10 -q -l /tmp/mbuffer.log=0A= mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | | = zfs recv =0A= =0A= Here are the results that I got.=0A= =0A= no compression:=0A= real 3m18.591s=0A= user 0m0.390s=0A= sys 0m8.177s=0A= =0A= xz -0:=0A= real 7m26.349s=0A= user 7m6.436s=0A= sys 0m8.471s=0A= =0A= pxz -0:=0A= real 2m28.470s=0A= user 6m44.168s=0A= sys 0m12.002s=0A= =0A= gzip:=0A= real 3m12.482s=0A= user 3m8.260s=0A= sys 0m4.974s=0A= =0A= lz4:=0A= real 1m58.363s=0A= user 0m10.000s=0A= sys 0m8.708s=0A= =0A= =0A= =0A= lz4 still comes out best. Is there anything else that I might be missing i= n my tests? don't have a real setup at this time to test WAN connections, = but I'd like to think that these results can be extrapolated to a WAN link = also. =0A= =0A= =0A= ________________________________________=0A= From: Ronald Klop [ronald-lists@klop.ws]=0A= Sent: Tuesday, February 28, 2017 11:44 AM=0A= To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan=0A= Cc: freebsd-fs@freebsd.org; Jeremy Faulkner=0A= Subject: Re: FreeBSD restartable send/receive over WAN=0A= =0A= On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan=0A= wrote:=0A= =0A= > thanks for all the pointers for the compression algorithms. I ran a few= =0A= > tests to compare compression overhead. These are local zfs=0A= > send/receive, and no network is involved.=0A= >=0A= > time zfs send -v | | | zfs=0A= > receive -s =0A= >=0A= > Here are the performance results that I got.=0A= >=0A= > no compression:=0A= > real 0m20.892s=0A= > user 0m0.000s=0A= > sys 0m5.587s=0A= >=0A= > xz -0:=0A= > real 8m38.569s=0A= > user 10m28.551s=0A= > sys 0m6.866s=0A= >=0A= > pxz -0:=0A= > real 4m38.448s=0A= > user 10m55.863s=0A= > sys 0m13.324s=0A= >=0A= > gzip:=0A= > real 3m51.297s=0A= > user 4m12.035s=0A= > sys 0m4.696s=0A= >=0A= > lz4:=0A= > real 0m29.912s=0A= > user 0m16.543s=0A= > sys 0m10.810s=0A= >=0A= >=0A= > lz4 has the least overhead in terms of time. pxz/xz seem to be=0A= > prohibitive give the above results. Unless, there is something basic=0A= > I'm missing?=0A= >=0A= > I was really hoping that compressed sends would be available, as that=0A= > would actively eliminate this overhead, given that we use lz4 as the=0A= > compression algorithm when writing to disks.=0A= =0A= =0A= Why don't you test this with a network in between? That would give much=0A= more valuable numbers to compare for your use case.=0A= The number above say nothing about the gain vs the bottleneck you are=0A= trying to remove.=0A= =0A= Ronald.=0A= =0A= =0A= =0A= =0A= >=0A= >=0A= > ________________________________=0A= > From: Karli Sj=F6berg [karli.sjoberg@slu.se]=0A= > Sent: Sunday, February 26, 2017 8:41 AM=0A= > To: Gary Palmer=0A= > Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org=0A= > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >=0A= >=0A= > Den 26 feb. 2017 4:16 em skrev Gary Palmer :=0A= >>=0A= >> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote:=0A= >> > The compression that we use on our ZFS filesystems is lz4. So, if I= =0A= >> have to pipe it through a compression algorithm, that'd be=0A= >> uncompressing and compressing it 4 times.=0A= >> >=0A= >> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)=0A= >> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4)=0A= >> >=0A= >> > isn't this quite expensive? We have to transfer multi terabyte files= =0A= >> on a WAN link. I'm also of the understanding that gzip by itself is=0A= >> single-threaded, so that'd peg one of the CPUs to 100%. there might be= =0A= >> other compression algorithms that can be used, but sending the ZFS as=0A= >> it is compressed on the filesystem is something that would be optimal,= =0A= >> and would reduce the overhead of the additional [de]compressions that=0A= >> are taking place?=0A= >>=0A= >> Without going into the efficiency part of your message:=0A= >>=0A= >> archivers/pigz: Parallel GZIP=0A= >> archivers/pbzip2: Parallel BZIP2=0A= >> archivers/pixz: Parallel, indexing version of XZ=0A= >> archivers/pxz: Parallel LZMA compressor using liblzma=0A= >=0A= > Also worth mentioning is, obviously:=0A= > archivers/lz4=0A= >=0A= > :)=0A= >=0A= > /K=0A= >=0A= >>=0A= >> Regards,=0A= >>=0A= >> Gary=0A= >>=0A= >> >=0A= >> > ________________________________________=0A= >> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on= =0A= >> behalf of Jeremy Faulkner [gldisater@gmail.com]=0A= >> > Sent: Saturday, February 25, 2017 4:03 PM=0A= >> > To: freebsd-fs@freebsd.org=0A= >> > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >> >=0A= >> > Pipe it through a compressor=0A= >> >=0A= >> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:=0A= >> > > Hi,=0A= >> > >=0A= >> > > I just tried restartable send/receive in 10.3 and it works like a=0A= >> charm. I was wondering if compressed send has made its way into=0A= >> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the=0A= >> -c/--compressed option. Any pointers?=0A= >> > >=0A= >> > > Regards,=0A= >> > > Shiva=0A= >> > >=0A= >> > >=0A= >> > > ________________________________________=0A= >> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org]=0A= >> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl]=0A= >> > > Sent: Thursday, February 16, 2017 10:41 AM=0A= >> > > To: freebsd-fs@freebsd.org=0A= >> > > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >> > >=0A= >> > > On 2017-02-16 19:22, Shiva Bhanujan wrote:=0A= >> > >> Hello,=0A= >> > >>=0A= >> > >> I was wondering if restartable send/receive is available in=0A= >> FreeBSD? We're running 10.2 and have a requirement of sending and=0A= >> receiving ZFS snapshots over a WAN link. The snapshots could be more=0A= >> than a few terabytes.=0A= >> > >>=0A= >> > >> Can somebody please give me pointers, and if this feature is or=0A= >> isn't available in FreeBSD?=0A= >> > >=0A= >> > > FreeBSD 10.3 and later.=0A= >> > >=0A= >> > > _______________________________________________=0A= >> > > freebsd-fs@freebsd.org mailing list=0A= >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > > To unsubscribe, send any mail to=0A= >> "freebsd-fs-unsubscribe@freebsd.org"=0A= >> > > _______________________________________________=0A= >> > > freebsd-fs@freebsd.org mailing list=0A= >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > > To unsubscribe, send any mail to=0A= >> "freebsd-fs-unsubscribe@freebsd.org"=0A= >> > >=0A= >> > _______________________________________________=0A= >> > freebsd-fs@freebsd.org mailing list=0A= >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >> > _______________________________________________=0A= >> > freebsd-fs@freebsd.org mailing list=0A= >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >> >=0A= >> _______________________________________________=0A= >> freebsd-fs@freebsd.org mailing list=0A= >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= > _______________________________________________=0A= > freebsd-fs@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= From owner-freebsd-fs@freebsd.org Fri Mar 3 04:20:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76473CF4BBF for ; Fri, 3 Mar 2017 04:20:38 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: from mail-io0-x230.google.com (mail-io0-x230.google.com [IPv6:2607:f8b0:4001:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B5E21445; Fri, 3 Mar 2017 04:20:38 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: by mail-io0-x230.google.com with SMTP id l7so67711736ioe.3; Thu, 02 Mar 2017 20:20:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=BZB1YtrzNAQpvFLBo4K3aQQPy/aI84xgyflTXlcT9J0=; b=IlWZ0wCyoo5ZDBjtOMhDU5Qr4VsmEnnomtVXgmWEbbHpVROLA6Tjxqn2cZ4wZ8lDNK 0isd+8ZRZGhFuhL2bRj5bg5HTgbXzm8iSMszNVWVMuoHcdAiFinfvC9PJWIZ66c5tlYy Vx+hZYpDfCslsU2/G9dPYOmN9WwsRG4Bvno7ZiyDy3ErIz3uzMMwowq+0gxqQm5MqAvf M19QiQkycj+XpnA0CNo1u0IMEBLRNTELa2AoHfLb/yy9Eeml4xbXLnxVMAZ/FDlowQGb VbEAvuso/kMK+zPeptylhmaKREXFN3gA/qzWKHFWfFcq4eTj8ZQ7bnElI+3pNHbfb9TK dKSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=BZB1YtrzNAQpvFLBo4K3aQQPy/aI84xgyflTXlcT9J0=; b=j9zonD2ftl6zbw3NJGaVJtUHWhIAZav5tDm1tGIsVEj+K046Lprao7biJYKGN+78hz bRFrYbsz5B8AeSWpIqdWkXabQE3T1QqEas8NJWzo9IcnTalNFf5hLA8Sbjsr45NIk4Z4 3lGyP3xegKJkr5CKpG/0RV9316d8A6OCuX4jLz5YTPLI1sy+qNrld5HSM3xLRua00oFP B1HyKQImM1SowFImeImSM7HYcXffehCMPIVNIR7aU+izs0j4nFu1mAfdw+u0s0VWeQ9u UTJdmPsS4bEvOC7x1Fygj57rOzLeR1SXNPQbtDNBJdfHbecc0M0zjlubE2mH+i5xBQSZ a2Dw== X-Gm-Message-State: AMke39kpV80MEG4MkcKXN1H/IjLSZ4R60aocmCu8KjXAQZwcuB1BsTWdyYwJ0aqMum2slc9qL0bcWtsWSJW7hQ== X-Received: by 10.107.183.20 with SMTP id h20mr1133720iof.18.1488514837530; Thu, 02 Mar 2017 20:20:37 -0800 (PST) MIME-Version: 1.0 References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> From: "Eric A. Borisch" Date: Fri, 03 Mar 2017 04:20:26 +0000 Message-ID: Subject: Re: FreeBSD restartable send/receive over WAN To: Gary Palmer , =?UTF-8?Q?Karli_Sj=C3=B6berg?= , Ronald Klop , Shiva Bhanujan Cc: Jeremy Faulkner , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2017 04:20:38 -0000 You can rate limit with mbuffer if you want experimental results for different bandwidths. (If you know what you can expect to get on your WAN, that would be a key point to test.) There are very likely crossover points in your "rankings" as a function of BW, depending on the compressibility of your data stream (and each compressor's ability to exploit it.) (Also with different machines/CPU.) Or you can measure the constituent parts to create a model and do the math across simulated link BW. You'll need throughputs and compression ratios for each compressor. Mbuffer without the -q should report the transferred (compressed) size. Decompression rate is likely faster than compression and therefore of little impact in the grand scheme. -Eric On Thu, Mar 2, 2017 at 6:35 PM Shiva Bhanujan wrote: > I ran the same set of tests between 2 FreeBSD instances, connected on a 1= G > LAN. The the comms was setup using mbuffer. Here's the basic command. > > time zfs send -v | | mbuffer -O :8099 -b 1024 -= m > 128M -P 10 -q -l /tmp/mbuffer.log > mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | = | > zfs recv > > Here are the results that I got. > > no compression: > real 3m18.591s > user 0m0.390s > sys 0m8.177s > > xz -0: > real 7m26.349s > user 7m6.436s > sys 0m8.471s > > pxz -0: > real 2m28.470s > user 6m44.168s > sys 0m12.002s > > gzip: > real 3m12.482s > user 3m8.260s > sys 0m4.974s > > lz4: > real 1m58.363s > user 0m10.000s > sys 0m8.708s > > > > lz4 still comes out best. Is there anything else that I might be missing > in my tests? don't have a real setup at this time to test WAN connection= s, > but I'd like to think that these results can be extrapolated to a WAN lin= k > also. > > > ________________________________________ > From: Ronald Klop [ronald-lists@klop.ws] > Sent: Tuesday, February 28, 2017 11:44 AM > To: Karli Sj=C3=B6berg; Gary Palmer; Shiva Bhanujan > Cc: freebsd-fs@freebsd.org; Jeremy Faulkner > Subject: Re: FreeBSD restartable send/receive over WAN > > On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan > wrote: > > > thanks for all the pointers for the compression algorithms. I ran a fe= w > > tests to compare compression overhead. These are local zfs > > send/receive, and no network is involved. > > > > time zfs send -v | | | zfs > > receive -s > > > > Here are the performance results that I got. > > > > no compression: > > real 0m20.892s > > user 0m0.000s > > sys 0m5.587s > > > > xz -0: > > real 8m38.569s > > user 10m28.551s > > sys 0m6.866s > > > > pxz -0: > > real 4m38.448s > > user 10m55.863s > > sys 0m13.324s > > > > gzip: > > real 3m51.297s > > user 4m12.035s > > sys 0m4.696s > > > > lz4: > > real 0m29.912s > > user 0m16.543s > > sys 0m10.810s > > > > > > lz4 has the least overhead in terms of time. pxz/xz seem to be > > prohibitive give the above results. Unless, there is something basic > > I'm missing? > > > > I was really hoping that compressed sends would be available, as that > > would actively eliminate this overhead, given that we use lz4 as the > > compression algorithm when writing to disks. > > > Why don't you test this with a network in between? That would give much > more valuable numbers to compare for your use case. > The number above say nothing about the gain vs the bottleneck you are > trying to remove. > > Ronald. > > > > > > > > > > ________________________________ > > From: Karli Sj=C3=B6berg [karli.sjoberg@slu.se] > > Sent: Sunday, February 26, 2017 8:41 AM > > To: Gary Palmer > > Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org > > Subject: Re: FreeBSD restartable send/receive over WAN > > > > > > Den 26 feb. 2017 4:16 em skrev Gary Palmer : > >> > >> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote: > >> > The compression that we use on our ZFS filesystems is lz4. So, if I > >> have to pipe it through a compression algorithm, that'd be > >> uncompressing and compressing it 4 times. > >> > > >> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network) > >> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4) > >> > > >> > isn't this quite expensive? We have to transfer multi terabyte file= s > >> on a WAN link. I'm also of the understanding that gzip by itself is > >> single-threaded, so that'd peg one of the CPUs to 100%. there might b= e > >> other compression algorithms that can be used, but sending the ZFS as > >> it is compressed on the filesystem is something that would be optimal, > >> and would reduce the overhead of the additional [de]compressions that > >> are taking place? > >> > >> Without going into the efficiency part of your message: > >> > >> archivers/pigz: Parallel GZIP > >> archivers/pbzip2: Parallel BZIP2 > >> archivers/pixz: Parallel, indexing version of XZ > >> archivers/pxz: Parallel LZMA compressor using liblzma > > > > Also worth mentioning is, obviously: > > archivers/lz4 > > > > :) > > > > /K > > > >> > >> Regards, > >> > >> Gary > >> > >> > > >> > ________________________________________ > >> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on > >> behalf of Jeremy Faulkner [gldisater@gmail.com] > >> > Sent: Saturday, February 25, 2017 4:03 PM > >> > To: freebsd-fs@freebsd.org > >> > Subject: Re: FreeBSD restartable send/receive over WAN > >> > > >> > Pipe it through a compressor > >> > > >> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: > >> > > Hi, > >> > > > >> > > I just tried restartable send/receive in 10.3 and it works like a > >> charm. I was wondering if compressed send has made its way into > >> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the > >> -c/--compressed option. Any pointers? > >> > > > >> > > Regards, > >> > > Shiva > >> > > > >> > > > >> > > ________________________________________ > >> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] > >> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl] > >> > > Sent: Thursday, February 16, 2017 10:41 AM > >> > > To: freebsd-fs@freebsd.org > >> > > Subject: Re: FreeBSD restartable send/receive over WAN > >> > > > >> > > On 2017-02-16 19:22, Shiva Bhanujan wrote: > >> > >> Hello, > >> > >> > >> > >> I was wondering if restartable send/receive is available in > >> FreeBSD? We're running 10.2 and have a requirement of sending and > >> receiving ZFS snapshots over a WAN link. The snapshots could be more > >> than a few terabytes. > >> > >> > >> > >> Can somebody please give me pointers, and if this feature is or > >> isn't available in FreeBSD? > >> > > > >> > > FreeBSD 10.3 and later. > >> > > > >> > > _______________________________________________ > >> > > freebsd-fs@freebsd.org mailing list > >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> > > To unsubscribe, send any mail to > >> "freebsd-fs-unsubscribe@freebsd.org" > >> > > _______________________________________________ > >> > > freebsd-fs@freebsd.org mailing list > >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> > > To unsubscribe, send any mail to > >> "freebsd-fs-unsubscribe@freebsd.org" > >> > > > >> > _______________________________________________ > >> > freebsd-fs@freebsd.org mailing list > >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org= " > >> > _______________________________________________ > >> > freebsd-fs@freebsd.org mailing list > >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org= " > >> > > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Fri Mar 3 08:55:02 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9E2F3CF6354 for ; Fri, 3 Mar 2017 08:55:02 +0000 (UTC) (envelope-from matthew@FreeBSD.org) Received: from smtp.infracaninophile.co.uk (smtp.infracaninophile.co.uk [IPv6:2001:8b0:151:1:c4ea:bd49:619b:6cb3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.infracaninophile.co.uk", Issuer "infracaninophile.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4817214B2 for ; Fri, 3 Mar 2017 08:55:02 +0000 (UTC) (envelope-from matthew@FreeBSD.org) Received: from liminal.local (unknown [IPv6:2001:8b0:151:1:1c1d:86a1:a200:b700]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: m.seaman@infracaninophile.co.uk) by smtp.infracaninophile.co.uk (Postfix) with ESMTPSA id 1E31DDA95 for ; Fri, 3 Mar 2017 08:54:59 +0000 (UTC) Authentication-Results: smtp.infracaninophile.co.uk; dmarc=none header.from=FreeBSD.org Authentication-Results: smtp.infracaninophile.co.uk/1E31DDA95; dkim=none; dkim-atps=neutral Subject: Re: FreeBSD restartable send/receive over WAN To: freebsd-fs@freebsd.org References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> From: Matthew Seaman Message-ID: Date: Fri, 3 Mar 2017 08:54:58 +0000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Xw4cGOax3qTBV1jbnD20AnRnOHHoa5BLd" X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,RDNS_NONE, SPF_SOFTFAIL autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on smtp.infracaninophile.co.uk X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2017 08:55:02 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Xw4cGOax3qTBV1jbnD20AnRnOHHoa5BLd Content-Type: multipart/mixed; boundary="ixmpcKQ2u5Ur6Nsm5AJB6VuUNd57gn5RA"; protected-headers="v1" From: Matthew Seaman To: freebsd-fs@freebsd.org Message-ID: Subject: Re: FreeBSD restartable send/receive over WAN References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> --ixmpcKQ2u5Ur6Nsm5AJB6VuUNd57gn5RA Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 03/03/2017 00:34, Shiva Bhanujan wrote: > I ran the same set of tests between 2 FreeBSD instances, connected > on a 1G LAN. The the comms was setup using mbuffer. Here's the basic co= mmand. >=20 > time zfs send -v | | mbuffer -O :8099 -b 1024= -m 128M -P 10 -q -l /tmp/mbuffer.log > mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | | zfs recv >=20 > Here are the results that I got. >=20 > no compression: > real 3m18.591s > user 0m0.390s > sys 0m8.177s >=20 > xz -0: > real 7m26.349s > user 7m6.436s > sys 0m8.471s >=20 > pxz -0: > real 2m28.470s > user 6m44.168s > sys 0m12.002s >=20 > gzip: > real 3m12.482s > user 3m8.260s > sys 0m4.974s >=20 > lz4: > real 1m58.363s > user 0m10.000s > sys 0m8.708s >=20 >=20 >=20 > lz4 still comes out best. Is there anything else that I might be > missing in my tests? don't have a real setup at this time to test > WAN connections, but I'd like to think that these results can be > extrapolated to a WAN link also. >=20 lz4 has the very interesting property that it will detect when it is taking too long to compress any particular block[*] and instead just pass the block through uncompressed. It trades off compression for speed= =2E The results you're seeing here on a reasonably high-bandwidth, low latency connection show that the time spent compressing the data is a significant part of the total, so lz4's short-cutting has a big benefit. If you were trying to transmit over a lower bandwidth connection, you'ld find that the share of elapsed time due to doing the compression was quite a bit less, and in that case a compression algorithm that tried harder and achieved better actual compression would give a faster overall result by sending less data over the wire. You would also not necessarily need to parallelize the compression over so many cores in order to keep a small pipe continually filled up. Or, in other words, you need to do your tests under the conditions you will be using in production in order to get completely reliable results. Cheers, Matthew --ixmpcKQ2u5Ur6Nsm5AJB6VuUNd57gn5RA-- --Xw4cGOax3qTBV1jbnD20AnRnOHHoa5BLd Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQJ8BAEBCgBmBQJYuS9iXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ2NTNBNjhCOTEzQTRFNkNGM0UxRTEzMjZC QjIzQUY1MThFMUE0MDEzAAoJELsjr1GOGkATflwQAI9UqV7n0Ucrcok3lyZlvEW+ /M2fMG99bfqct3VF/hoW+xb6jYQeaZF3O+rJCNj1ZeYqbnVtyjKnhSYXZGBA3oG1 uk6oC00ifMtTrf5P3+BbgUSKTaRhjnyfFu+Ukf0qyFMieSNHbPwxF5Y3FwFrn6eu tU9jKzUTZyRzL4vy5JsPQ8abo0WSQLSeGlob90upq4/RtyIAxGeLte5NGU9kLs14 kMLDess+O7SRqYfN8/aZ6QhBjFFEgm5jn8yH+go9SLLEzWUjlukCp2jLmk5xCoka nUpec4S0ix1n3gPG3C+pzwOWw9jvkMJGiI25ltUZLxhACmDixCv/JQPZ72udCmQd bxY3LrT81IYW1gNm3OeVwWLNmTpGlgEbLrbU9GPyCzBs7/7pG94Ems8Y7Q6zAz1w kl5L4Oc+pqeBc9sPbx37rKYiFcEizaOaujWszm92NpvNE1Y/oPKu2rsbSXl1gqZA muZ6I4cbmUJHmP4VK148YEsOEURLO8nnW+pJo+lcDxoH3b9vhb0nWk/xY6HVxgr/ kapM8wkwqShPr9WeB0OjvrIE7nQagHUTbFz0vyt/d2+biRCnyzZbKzjlQjSSha6L Xa8HWC0t1omsxqMxv2yANyFpPTfHRhl1Uc/DLP+i2lrnbF/95rwXK2eUnmLJ12nQ gz3aDpBlFPXlV50wVgAV =pKS+ -----END PGP SIGNATURE----- --Xw4cGOax3qTBV1jbnD20AnRnOHHoa5BLd-- From owner-freebsd-fs@freebsd.org Fri Mar 3 14:48:56 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EA134CF5F17 for ; Fri, 3 Mar 2017 14:48:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D99511B17 for ; Fri, 3 Mar 2017 14:48:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v23Emurg043771 for ; Fri, 3 Mar 2017 14:48:56 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 216964] boot1.efi: can't boot from ZFS on 4kn HDD Date: Fri, 03 Mar 2017 14:48:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kozlov.sergey.404@gmail.com X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2017 14:48:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D216964 --- Comment #7 from Sergey Kozlov --- (In reply to Toomas Soome from comment #6) The patch worked, details are added to the review comments. Thank you! --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Mar 3 15:49:55 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9393FCF6572 for ; Fri, 3 Mar 2017 15:49:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 835181867 for ; Fri, 3 Mar 2017 15:49:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v23FntY5014697 for ; Fri, 3 Mar 2017 15:49:55 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 216964] boot1.efi: can't boot from ZFS on 4kn HDD Date: Fri, 03 Mar 2017 15:49:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: tsoome@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2017 15:49:55 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D216964 --- Comment #8 from Toomas Soome --- (In reply to Sergey Kozlov from comment #7) After I did check on efipart.c (used by loader), I found that it already do= es address the large sector issue on its own, so that part is also covered; it really does make the life a lot simpler. Thanks for feedback and I'll seek = to have the patch commited. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Mar 4 17:02:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C7B7CCF99D4 for ; Sat, 4 Mar 2017 17:02:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B72A21C00 for ; Sat, 4 Mar 2017 17:02:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v24H23UR037312 for ; Sat, 4 Mar 2017 17:02:03 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 217440] FAT32 formatted USB stick with files written by PS4 - Invalid argument, unable to list directory contents Date: Sat, 04 Mar 2017 17:02:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Mar 2017 17:02:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217440 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cem@freebsd.org --- Comment #2 from Conrad Meyer --- How large is the USB stick? Is there any chance you could share an image of the USB stick to aid debugging / reproduction? Thanks. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Mar 4 17:15:18 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90396CF9CCE for ; Sat, 4 Mar 2017 17:15:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7F84912C5 for ; Sat, 4 Mar 2017 17:15:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v24HFIVI071302 for ; Sat, 4 Mar 2017 17:15:18 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 217440] FAT32 formatted USB stick with files written by PS4 - Invalid argument, unable to list directory contents Date: Sat, 04 Mar 2017 17:15:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Mar 2017 17:15:18 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217440 --- Comment #3 from Conrad Meyer --- Well, ok, these are the relevant dirents: 00000040 42 69 00 6f 00 6e 00 22 21 00 00 0f 00 72 ff ff |Bi.o.n."!....r= ..| 00000050 ff ff ff ff ff ff ff ff ff ff 00 00 ff ff ff ff |..............= ..| 00000060 01 41 00 6c 00 69 00 65 00 6e 00 0f 00 72 5f 00 |.A.l.i.e.n...r= _.| 00000070 20 00 49 00 73 00 6f 00 6c 00 00 00 61 00 74 00 | .I.s.o.l...a.= t.| It's a two-part LFN, in reverse order. The bytes from 0x41-0x4a are "ion\U2122\0". \U2122 is "Unicode Character 'TRADE MARK SIGN' (U+2122)." = The 8.3 encoding is in a 3rd dirent: 00000080 41 4c 49 45 4e 5f 7e 31 20 20 20 10 00 31 5d 19 |ALIEN_~1 ..1= ].| 00000090 61 4a 61 4a 00 00 c2 19 61 4a 06 00 00 00 00 00 |aJaJ....aJ....= ..| "ALIEN_~1. ". --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Mar 4 18:02:29 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D341BCF9D66 for ; Sat, 4 Mar 2017 18:02:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B45CF1DFB for ; Sat, 4 Mar 2017 18:02:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v24I2TEU059126 for ; Sat, 4 Mar 2017 18:02:29 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 217440] FAT32 formatted USB stick with files written by PS4 - Invalid argument, unable to list directory contents Date: Sat, 04 Mar 2017 18:02:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Mar 2017 18:02:29 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217440 --- Comment #4 from Conrad Meyer --- It looks like you may be able to use mount_msdosfs -o shortnames (-s) or -o nowin95 (-9) as a workaround to access your file, for now. $ dd if=3D/dev/zero of=3D./testfs bs=3D1m count=3D64 $ sudo mdconfig -a -t vnode -f testfs md1 $ sudo newfs_msdos -c1 -F32 /dev/md1 /dev/md1: 129022 sectors in 129022 FAT32 clusters (512 bytes/cluster) BytesPerSec=3D512 SecPerClust=3D1 ResSectors=3D32 FATs=3D2 Media=3D0xf0 Sec= PerTrack=3D63 Heads=3D16 HiddenSecs=3D0 HugeSectors=3D131072 FATsecs=3D1008 RootCluster= =3D2 FSInfo=3D1 Backup=3D2 $ mkdir testdir $ sudo mount_msdosfs /dev/md1 ./testdir $ touch "testdir/test=E2=84=A2" touch: testdir/test=E2=84=A2: Invalid argument $ sudo dtrace -n "fbt:::return /arg1=3D=3DEINVAL/ { stack(); }" -c "touch 'testdir/test=E2=84=A2'" = =E2=8F=8E dtrace: description 'fbt:::return ' matched 30012 probes touch: 'testdir/test=E2=84=A2': No such file or directory <<< diffe= rent error?=20 Only happens under dtrace dtrace: pid 12937 exited with status 1 CPU ID FUNCTION:NAME 3 56556 _vhold:return kernel`cache_lookup+0xba7 kernel`vfs_cache_lookup+0xac kernel`VOP_LOOKUP_APV+0x87 kernel`lookup+0x711 kernel`namei+0x59d kernel`vn_open_cred+0x21c kernel`kern_openat+0x25f kernel`amd64_syscall+0x51e kernel`0xffffffff80fc867b $ sudo dtrace -n "fbt:::return /arg1=3D=3DEINVAL/ { @[stack()] =3D count();= }" -c "touch 'testdir/234test=E2=84=A2'" dtrace: description 'fbt:::return ' matched 30012 probes touch: 'testdir/234test=E2=84=A2': No such file or directory dtrace: pid 12964 exited with status 1 kernel`cache_lookup+0xba7 kernel`vfs_cache_lookup+0xac kernel`VOP_LOOKUP_APV+0x87 kernel`lookup+0x711 kernel`namei+0x59d kernel`vn_open_cred+0x21c kernel`kern_openat+0x25f kernel`amd64_syscall+0x51e kernel`0xffffffff80fc867b 1 kernel`vn_open_cred+0x10f kernel`kern_openat+0x25f kernel`amd64_syscall+0x51e kernel`0xffffffff80fc867b 1 ... (irrelevant frames elided) Ok, this also seems to work: $ sudo mount_msdosfs -L en_US.UTF-8 /dev/md1 ./testdir $ touch 'testdir/234test=E2=84=A2' $ ls testdir 234test=E2=84=A2 Does mounting your USB stick with -L .UTF-8 work? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Mar 4 18:07:16 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1EBBCF9DFF for ; Sat, 4 Mar 2017 18:07:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8BA3E1EC0 for ; Sat, 4 Mar 2017 18:07:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v24I7Gv7048602 for ; Sat, 4 Mar 2017 18:07:16 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 217440] FAT32 formatted USB stick with files written by PS4 - Invalid argument, unable to list directory contents Date: Sat, 04 Mar 2017 18:07:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Mar 2017 18:07:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217440 --- Comment #5 from Conrad Meyer --- Ok, I umounted and remounted without -L en-US.UTF-8, and reproduced your is= sue: $ ls ./testdir ls: 234test?: Invalid argument So -L .UTF-8 should work around the issue. Why a UTF-8 encoding isn't= the default, I don't know. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Mar 4 20:08:50 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D501CF941E for ; Sat, 4 Mar 2017 20:08:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3CA6E1F52 for ; Sat, 4 Mar 2017 20:08:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v24K8oKc013127 for ; Sat, 4 Mar 2017 20:08:50 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 217440] FAT32 formatted USB stick with files written by PS4 - Invalid argument, unable to list directory contents Date: Sat, 04 Mar 2017 20:08:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Mar 2017 20:08:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217440 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ache@FreeBSD.org --- Comment #6 from Conrad Meyer --- I think I see why ls(1)/fts(3) shows "Invalid". The directory entries are readable: $ echo ./testdir/* ./testdir/123t=EF=BF=BDst ./testdir/abc? $ echo $? 0 However, that '?' is an actual question mark symbol, because =E2=84=A2 can = not be represented in iso-8859-1, only win-1252 and unicode. So you cannot access that file by its directory entry: $ ls ./testdir/abc? ls: ./testdir/abc?: Invalid argument $ truss stat ./testdir/abc? ... lstat("./testdir/abc?",0x7fffffffe298) ERR#22 'Invalid argument' I think that EINVAL return is bogus. lstat() misses should return ENOENT. = But that doesn't help you very much. Maybe VOP_READDIR should prefer 8.3 names if LFN names do not convert into cs_local. Although, lookup of 8.3 names doesn't work. So maybe not. It seems cs_local should default to UTF-8 or user's locale, not ISO-8859-1.= Or if it must be a 8-bit character set, Win-1252 may be a better choice. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Mar 4 21:17:16 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C02CDCF5468 for ; Sat, 4 Mar 2017 21:17:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AF94E10DD for ; Sat, 4 Mar 2017 21:17:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v24LHGmQ089125 for ; Sat, 4 Mar 2017 21:17:16 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 217440] FAT32 formatted USB stick with files written by PS4 - Invalid argument, unable to list directory contents Date: Sat, 04 Mar 2017 21:17:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: erik@nordstroem.no X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Mar 2017 21:17:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217440 --- Comment #7 from Erik Nordstr=C3=B8m --- Thanks for looking into this and for your findings thus far Conrad. Would y= ou like an image of the USB stick still or was the excerpts in my original post sufficient? The original USB stick is several hundreds of MB in size but if desired I c= an produce a much smaller image for repro and debug containing only a single s= mall file (a single screen shot instead of multiple MP4 videos) written to an otherwise empty USB stick which would have the same directory structure and file names. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Mar 4 22:41:33 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4FB51CF9DEE for ; Sat, 4 Mar 2017 22:41:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3F15815EB for ; Sat, 4 Mar 2017 22:41:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v24MfXbu099495 for ; Sat, 4 Mar 2017 22:41:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 217440] FAT32 formatted USB stick with files written by PS4 - Invalid argument, unable to list directory contents Date: Sat, 04 Mar 2017 22:41:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Mar 2017 22:41:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217440 --- Comment #8 from Conrad Meyer --- (In reply to Erik Nordstr=C3=B8m from comment #7) Hi Erik, No, we don't need the USB stick image any more. I can reproduce it with a small in-memory filesystem by mounting with and without "-L en_US.UTF-8". Did you see comment #5? Can you try mounting with "mount_msdosfs -L no_NO.UTF-8" or similar and see if that allows you to access your file?=20 Thanks. --=20 You are receiving this mail because: You are the assignee for the bug.=