From owner-freebsd-fs@freebsd.org Sun Mar 29 18:23:40 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C17352A652A for ; Sun, 29 Mar 2020 18:23:40 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48r3r06dR0z3G1V for ; Sun, 29 Mar 2020 18:23:28 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 3EA6040007 for ; Sun, 29 Mar 2020 20:23:14 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 29D8E40004; Sun, 29 Mar 2020 20:23:13 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, AWL, HTML_MESSAGE autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:6519:594e:bae9:86c7] (unknown [IPv6:2001:9b1:28ff:d901:6519:594e:bae9:86c7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id AD53740005; Sun, 29 Mar 2020 20:23:12 +0200 (CEST) From: Peter Eriksson Message-Id: <982F9A21-FF1C-4DAB-98B3-610D70714ED3@lysator.liu.se> Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS/NFS hickups and some tools to monitor stuff... Date: Sun, 29 Mar 2020 20:23:12 +0200 In-Reply-To: Cc: "PK1048.COM" To: FreeBSD Filesystems References: <66AB88C0-12E8-48A0-9CD7-75B30C15123A@pk1048.com> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48r3r06dR0z3G1V X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 130.236.254.3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-5.63 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[3.254.236.130.list.dnswl.org : 127.0.11.2]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-3.13)[ip: (-8.11), ipnet: 130.236.0.0/16(-4.17), asn: 2843(-3.34), country: SE(-0.03)] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Mar 2020 18:23:41 -0000 >>=20 >> Mostly home directories. No VM image files. About 20000 filesystems = per server with around 100 snapshots per filesystem. Around 150-180M = files/directories per server. >=20 > Wow. By =E2=80=9Cfilesystems=E2=80=9D I assume you mean zfs datasaets = and not zpools? >=20 Yes, ZFS datasets. Our AD right now contains ~130000 users (not all are = active thankfully, typically around 3000-5000 are online at the same = time (90% via SMB)) spread out over a number of servers). > Why so many? In the early days of zfs, due to the lack of per-user = quotas, there were sites with one zfs dataset per user (home directory) = so that quotas could be enforced. This raised serious performance = issues, mostly ay boot time due to the need to mount so many different = datasets. Per- ... > But, this may be the underlying source of the performance issues. Yeah. I=E2=80=99m aware of the user quotas which is nice. However they = only solve part of the problem(s): 1. With the GDPR laws and the =E2=80=9Cright to be forgotten=E2=80=9D we = need to be able to delete a user's files when they leave their = employment here and/or when students stop studying. This together with = snapshots would make this a much harder operation (if we just have one = big dataset for all users then we can=E2=80=99t just delete a specific = user=E2=80=99s dataset (and snapshots). Basically we would have to = delete the snapshot for *all users* then=E2=80=A6 2. Users that write a lot of data every day - this uses up a lot of = =E2=80=9Cquota=E2=80=9D and not =E2=80=9Crefquota=E2=80=9D. The =E2=80=9Cu= ser quota=E2=80=9D just counts the =E2=80=9Crefquota=E2=80=9D=E2=80=A6 = And then even if we find that user and makes them stop writing, their = old data will live on in the snapshots=E2=80=A6 Right now we are trying out a scheme where we give each user dataset a = =E2=80=9Cuserquota=E2=80=9D of X, a =E2=80=9Crefquota=E2=80=9D of X+1G = and a =E2=80=9Cquota=E2=80=9D of 3*X. This will hopefully lessen the = problem of =E2=80=9Cslow servers" when a user is filling up their ref = quotas since they=E2=80=99ll run into their user quotas before the ref = quotas=E2=80=A6 I=E2=80=99ve also modified the =E2=80=9Czfs snap=E2=80=9D command to = avoid taking snapshots on near-full datasets. We=E2=80=99ll see if this = makes things better. (I=E2=80=99ve also modified it to have a =E2=80=9Czfs clean=E2=80=9D = command to parallelise the snapshot deletion a bit (and make it possible = to be smart on which snapshots to remove. We used to do this via a = Python script but that is not nearly as efficient as doing it directly = in the =E2=80=9Czfs=E2=80=9D command. We normally set a user property = like =E2=80=9Cse.liu.it:expires=E2=80=9D to a date when a snapshot = should expire, and then the =E2=80=9Czfs clean=E2=80=9D command can look = for that property and delete just those that have expired. The idea is = to keep hourly snapshots for 2 days, daily snapshots for 2 weeks and = weekly snapshots for 3 months (or so, we=E2=80=99ve been testing = differents times here) - any older is on the backup server so users = easily can recover their own files via the Windows =E2=80=9CPrevious = Versions=E2=80=9D feature (or via the .zfs/snapshot=E2=80=9D directory = for Unix users).=20 >>> Maybe I=E2=80=99m asking eh obvious, but what is performance like = natively on the server for these operations? >>=20 >> Normal response times: >>=20 >> This is from a small Intel NUC running OmniOS so NFS 4.0: >>=20 >> $ ./pfst -v /mnt/filur01 >> [pfst, version 1.7 - Peter Eriksson ] >> 2020-03-28 12:19:10 [2114 =C2=B5s]: /mnt/filur01: = mkdir("t-omnibus-821-1=E2=80=9D) >=20 > You misunderstood my question. When you are seeing the performance = issue via NFS, do you also see a performance issue directly on the NFS = server? The tests I ran did not indicate the same performance issues directly on = the server (I ran the same test program locally). Well, except for = =E2=80=9Czfs=E2=80=9D-commands being slow though.=20 > If the SMB clients are all (or mostly) Windows 8 or newer, the = Microsoft CIFS/SMB client stack has lots of caching to make poor server = performance feel good. That caching by the client may be masking = comparable performance issues via SMB. Testing directly on the server = will remove the network fie share layer from the discussion, or focus = the discussion there. >=20 >> Mkdir & rmdir takes about the same amount of time here. (0.6 - 1ms). >=20 > Do reads/writes from/to existing files show the same degradation? = Especially reads? Didn=E2=80=99t test that at the time unfortunately. And right now things = are running pretty OK.. >>> What does the disk %busy look like on the disks that make up the = vdev=E2=80=99s? (iostat -x) >>=20 >> Don=E2=80=99t have those numbers (when we were seeing problems) = unfortunately but if I remember correctly fairly busy during the = resilver (not surprising). >>=20 >> Current status (right now): >>=20 >> # iostat -x 10 |egrep -v pass >> extended device statistics =20 >> device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t = qlen %b =20 >> nvd0 0 0 0.0 0.0 0 0 0 0 = 0 0=20 >> da0 3 55 31.1 1129.4 10 1 87 3 = 0 13=20 >> da1 4 53 31.5 1109.1 10 1 86 3 = 0 13=20 >> da2 5 51 41.9 1082.4 9 1 87 3 = 0 14 >=20 > If this is during typical activity, you are already using 13% of your = capacity. I also don=E2=80=99t like the 80ms per operation times. The spinning rust drives are HGST He10 (10TB SAS 7200rpm) drives on Dell = HBA330 controllers (LSI SAS3008). We also use HP server with their own = Smart H241 HBAs and are seeing similar latencies there.=20 = https://www.storagereview.com/review/hgst-ultrastar-he10-10tb-enterprise-h= ard-drive-review = > What does zpool list show (fragmentation)? 22-27% fragmentation with 50-53% cap (108T size) on the 3 biggest = servers. >> At least for the resilver problem. >=20 > There are tunings you can apply to make the resilver even more = background than it usually is. I don=E2=80=99t have them off the top of = my head. Yeah, I tried those. Didn=E2=80=99t make much difference though=E2=80=A6 > I have mamnaged zfs server with hundreds of thousands of snapshots = with no performance penalty, except for snapshot management functions = (zfs list -t snapshot, which would take many, many minutes to complete), = so just the presence of snapshots should not hurt. Be aware that = destroying snapshots in the order in which they were _created_ is much = faster. In other words, always destroy the oldest snapshot first and = work your way forward. Yeah, I know. That=E2=80=99s what we are doing. - Peter From owner-freebsd-fs@freebsd.org Sun Mar 29 19:16:22 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9CEC82A7C5D for ; Sun, 29 Mar 2020 19:16:22 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48r50q5lqmz44xB for ; Sun, 29 Mar 2020 19:16:11 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 07FDA40007 for ; Sun, 29 Mar 2020 21:16:02 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id E17A240005; Sun, 29 Mar 2020 21:16:01 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, AWL, HTML_MESSAGE autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:6519:594e:bae9:86c7] (unknown [IPv6:2001:9b1:28ff:d901:6519:594e:bae9:86c7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 0D55A40004; Sun, 29 Mar 2020 21:16:01 +0200 (CEST) From: Peter Eriksson Message-Id: <7790DD37-4F95-409E-9E33-1A330B1B49C8@lysator.liu.se> Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS/NFS hickups and some tools to monitor stuff... Date: Sun, 29 Mar 2020 21:16:00 +0200 In-Reply-To: Cc: "PK1048.COM" To: FreeBSD Filesystems References: <66AB88C0-12E8-48A0-9CD7-75B30C15123A@pk1048.com> <982F9A21-FF1C-4DAB-98B3-610D70714ED3@lysator.liu.se> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48r50q5lqmz44xB X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 130.236.254.3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-5.69 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[3.254.236.130.list.dnswl.org : 127.0.11.2]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-3.19)[ip: (-8.28), ipnet: 130.236.0.0/16(-4.23), asn: 2843(-3.39), country: SE(-0.03)] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Mar 2020 19:16:23 -0000 > I thought that snapshot deletion was single threaded within a zpool, = since TXGs are zpool-wide, not per dataset. So you may not be able to = destroy snapshot in parallel. Yeah, I thought so too but decided to try it anyway. It sometimes goes = faster and since I decoupled the "read snapshots to delete" and the = =E2=80=9Cdo the deletion=E2=80=9D into separate threads now it doesn=E2=80= =99t have to read all snapshots to delete first and then delete them = all, but can interleave the jobs. Basically what my code now does is: for all datasets (recursively) collect_snapshots_to_delete if more than a (configurable limit) is queued start a deletion worker (configurable limit) So it can continue gathering snapshots to delete while deleting a batch. = And it doesn=E2=80=99t have to wait for the reading all snapshots in all = dataset before starting to delete stuff. So if it (for some reason) is = slow then atleast it will have deleted _some_ snapshots until we = terminate the =E2=80=9Cclean=E2=80=9D command I did some tests on the speed with different number of =E2=80=9Cworker=E2=80= =9D threads and I actually did see some speed improvements (cut the time = in half in some cases). But it varies a lot I guess - if all metadata is = in the ARC then it normally is pretty quick anyway. I=E2=80=99ve been thinking of also adding separate read workers so if = one dataset takes a long time to read it=E2=80=99s snapshots then others = could continue but it=E2=80=99s a bit harder to code in a good way :-) What we do now is (simplified): # Create hourly snapshots that expire in 2 days: zfs snap -r -E =E2=80=9Cse.liu.it:expires=E2=80=9D -e 2d = "DATA/staff@${DATETIME}" # Clean expired snapshots (10 workers, atleast 500 snapshots per = delete) zfs clean -r -E =E2=80=9Cse.liu.it:expires=E2=80=9D -P10 -L500 = -e DATA/staff I have my patch available at GitHub ( = https://github.com/ptrrkssn/freebsd-stuff = ) if it would be of = interest.=20 (At first I modified the =E2=80=9Czfs destroy=E2=80=9D command but since = I always feel nervous about using that one since a slip of the finger = could have catastrophic consequences so I decided to create a separate one that = only works on snapshots and nothing else). > I expect zpool/zfs commands to be very slow when large zfs operations = are in flight. The fact that you are not seeing the issue locally means = the issue is not directly with the zpool/dataset but somehow with the = interaction between NFS Client <-> NFS Server <-> ZFS dataset =E2=80=A6 = NFS does not have to be sync, but can you force the NFS client to always = use sync writes? That might better leverage the SLOG. Since our use case = is VMs and VirtualBox does sync writes, we get the benefit of the SLOG. >=20 >>> If this is during typical activity, you are already using 13% of = your capacity. I also don=E2=80=99t like the 80ms per operation times. >>=20 >> The spinning rust drives are HGST He10 (10TB SAS 7200rpm) drives on = Dell HBA330 controllers (LSI SAS3008). We also use HP server with their = own Smart H241 HBAs and are seeing similar latencies there. >=20 > That should be Ok, but I have heard some reports of issues with the HP = Smart 4xx series controllers with FreeBSD. Why are you seeing higher = disk latency with SAS than we are with SATA? I assume you checked logs = for device communication errors and retries? Yeah, no errors. The HP H241 HBAs are not as well supported as the = SAS3008 ones, but they work OK. At least if you force them into = =E2=80=9CHBA=E2=80=9D mode (changeable from BIOS. Until we did that they = had their problems yes=E2=80=A6 also there were some firmware issues on = certain releases) Anyway, I we are going to expand the RAM in the servers from 256GB to = 512GB (or 768GB). A test I did on our test server seems to indicate that = the metadata set fits much better with more RAM so everything is much = faster. (Now I=E2=80=99d also like to see persistent L2ARC support (it would be = great to have the metadata cached on faster SSDs and have it survive a = reboot) - but that won=E2=80=99t happen until the switch to OpenZFS = (FreeBSD 13 hopefully) so=E2=80=A6 - Peter From owner-freebsd-fs@freebsd.org Sun Mar 29 20:42:02 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0BAF82628E8 for ; Sun, 29 Mar 2020 20:42:02 +0000 (UTC) (envelope-from freebsd@sysctl.cz) Received: from wes1-so2.wedos.net (wes1-so2.wedos.net [46.28.106.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48r6vj6rB5z4d85 for ; Sun, 29 Mar 2020 20:41:52 +0000 (UTC) (envelope-from freebsd@sysctl.cz) Received: from webmail.wedos.net (wes1-wm3.wedos.net [46.28.106.84]) by wes1-so2.wedos.net (Postfix) with ESMTPSA id 48r6vS1LF1zrt; Sun, 29 Mar 2020 22:41:40 +0200 (CEST) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sun, 29 Mar 2020 22:41:39 +0200 From: freebsd@sysctl.cz To: Martin Simmons Cc: freebsd-fs@freebsd.org Subject: Re: extattr In-Reply-To: <202003161217.02GCHKGL025219@higson.cam.lispworks.com> References: <202003161217.02GCHKGL025219@higson.cam.lispworks.com> Message-ID: <4b1176d902e26746cd6f0aca7aaa98d6@sysctl.cz> X-Sender: freebsd@sysctl.cz User-Agent: Roundcube Webmail/1.2.4 X-Rspamd-Queue-Id: 48r6vj6rB5z4d85 X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@sysctl.cz has no SPF policy when checking 46.28.106.16) smtp.mailfrom=freebsd@sysctl.cz X-Spamd-Result: default: False [4.08 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; IP_SCORE(1.25)[ipnet: 46.28.104.0/21(2.35), asn: 197019(3.82), country: CZ(0.09)]; TO_DN_SOME(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sysctl.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.93)[0.931,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_NO_DN(0.00)[]; NEURAL_SPAM_LONG(1.00)[0.997,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:197019, ipnet:46.28.104.0/21, country:CZ]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Mar 2020 20:42:02 -0000 Dne 2020-03-16 13:17, Martin Simmons napsal: >>>>>> On Sun, 15 Mar 2020 18:21:34 +0100, freebsd said: >> >> Hi, i am trying make port for signal desktop application >> but i have problems in test. I patched this npm library for bsd. >> I am not sure with actual test, because bsd have maybe other >> implementation and also a bit other results. >> Do you have some skills in xattr ? >> >> https://github.com/LinusU/fs-xattr/pull/37 > > Maybe look at the perl File::ExtAttr module for ideas? > > https://metacpan.org/pod/File::ExtAttr > > In particular, the OS detection: > > https://metacpan.org/source/RICHDAWE/File-ExtAttr-1.09/extattr_os.h > > __Martin Thank you for your links, but i have next update but i have still issue with test. Problem on FreeBSD with xattr is that has sometimes other behavior than on linux os. And therefore some test failed. http://www.reactivated.net/patches/mono/1.1.8/extended-attr-transparency.patch From owner-freebsd-fs@freebsd.org Sun Mar 29 21:26:30 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BC2B2264F25 for ; Sun, 29 Mar 2020 21:26:30 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48r7tx3VMcz3x03 for ; Sun, 29 Mar 2020 21:26:16 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 5333440002 for ; Sun, 29 Mar 2020 23:26:04 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 3244140004; Sun, 29 Mar 2020 23:26:04 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:6519:594e:bae9:86c7] (unknown [IPv6:2001:9b1:28ff:d901:6519:594e:bae9:86c7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 7A02640002 for ; Sun, 29 Mar 2020 23:26:03 +0200 (CEST) From: Peter Eriksson Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: ZFS ARC Metadata "sizing" for datasets&snapshots Date: Sun, 29 Mar 2020 23:26:03 +0200 References: <66AB88C0-12E8-48A0-9CD7-75B30C15123A@pk1048.com> <982F9A21-FF1C-4DAB-98B3-610D70714ED3@lysator.liu.se> To: FreeBSD Filesystems In-Reply-To: Message-Id: <25DADCD5-085B-4E85-B8DA-B3115472CB2D@lysator.liu.se> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48r7tx3VMcz3x03 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-4.13 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; MV_CASE(0.50)[]; IP_SCORE(-1.83)[ip: (-7.27), ipnet: 2001:6b0::/32(-1.04), asn: 1653(-0.85), country: EU(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Mar 2020 21:26:30 -0000 Hmm.. I wonder if someone knows how much ZFS ARC metadata one should = expect for a certain server. Just for fun, I did some tests. On a test server with 512GB RAM and arc_max set to 384GB, arc_meta_limit = set to 256GB, 12866 filesystems (datasets) and 430610 snapshots (spread = out over those filesystems), and an uptime of 1 day doing nothing = basically (taking some snapshots) I get these numbers: anon_size: 1.0 M arc_max: 412.3 G arc_meta_limit: 274.9 G arc_meta_max: 33.5 G arc_meta_used: 33.5 G compressed_size: 9.5 G data_size: 7.4 G hdr_size: 462.3 M metadata_size: 30.2 G mru_size: 16.7 G other_size: 2.8 G overhead_size: 28.2 G size: 40.9 G uncompressed_size: 45.9 G Ie ARC metadata_size at 30GB / arc_meta_used at 33.5G.=20 Doing a =E2=80=9Czfs list -t all=E2=80=9D takes ~100s and =E2=80=9Czfs = list=E2=80=9D takes ~3s. On production server (just booted) with just 256GB RAM doing =E2=80=9Czfs = list=E2=80=9D takes 0.4s but =E2=80=9Czfs list -t all=E2=80=9D is taking = a long time=E2=80=A6 anon_size: 3.3 M arc_max: 103.1 G arc_meta_limit: 51.5 G arc_meta_max: 2.2 G arc_meta_used: 2.2 G compressed_size: 1.3 G data_size: 2.7 G hdr_size: 17.9 M metadata_size: 2.0 G mru_size: 3.3 G other_size: 180.5 M overhead_size: 3.4 G size: 4.9 G uncompressed_size: 3.4 G The =E2=80=9Czfs list -t all=E2=80=9D took 2542 seconds (42 minutes) - = 131256 datasets+snapshots (1600 filesystems). That is ~50 = snapshots/filesystems per second. After that command has executed metadata_size has increased with 5GB and = a new =E2=80=9Czfs list -t all=E2=80=9D just takes 37 seconds. anon_size: 5.1 M arc_max: 103.1 G arc_meta_limit: 51.5 G arc_meta_max: 7.9 G arc_meta_used: 7.8 G compressed_size: 2.5 G data_size: 1.5 G hdr_size: 98.0 M metadata_size: 7.1 G mru_size: 7.3 G other_size: 660.9 M overhead_size: 6.1 G size: 9.4 G uncompressed_size: 8.2 G So... perhaps ~40KB (5G/131256) per dataset/snapshot on average. (Yes, = oversimplificated but anyway) Hmm.. Perhaps one should regularly do a =E2=80=9Czfs list -t all = >/dev/null=E2=80=9D just to prime the ARC metadata cache (and keep it = primed) :-) - Peter From owner-freebsd-fs@freebsd.org Tue Mar 31 19:36:12 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8B2E5268EF2 for ; Tue, 31 Mar 2020 19:36:12 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48sKLc2x73z4ZBV for ; Tue, 31 Mar 2020 19:35:52 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:a961:74c3:55c1:ed45] ([IPv6:2607:f3e0:0:4:a961:74c3:55c1:ed45]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 02VJWrWX031527 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Tue, 31 Mar 2020 15:32:53 -0400 (EDT) (envelope-from mike@sentex.net) To: freebsd-fs From: mike tancsa Subject: zfs promote Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWibkB DQRcsMzkAQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4 axtKRSG1t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1 qzAJweEtRdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6c Lm0EiHPOl5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5 o9KKu4O7gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQAB iQE8BBgBCAAmFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAlywzOQCGwwFCQHhM4AACgkQeVOE Fl5WrMhmjQf/dBCjAVn1J0GzSsHiLvSAQz1cchbdy8LD0Tnpzjgp5KLU7sNojbI8vqt4yKAi cayI88j8+xxNXPMWM4pHELuUuVHS5XTpHa/wwulUtI5w/zyKlUDsIvqTPZLUEwH7DfNBueVM WyNaIjV2kxSmM8rNMC+RkgyfbjGLCkmWsMRVuLIUYpl5D9WHmenUbiErlKU2KvEEXEg/aLKq 3m/AdM9RAYsP9O4l+sAZEfyYoNJzDhTZMzn/9Q0uFPLK9smDQh4WBTFaApveVJPHRKmHPoNF Xxj+yScYdQ4SKH34WnhNSELvnZQ3ulH5tpASmm0w+GxfZqSc8+QCwoKtBRDUxoE56A== Message-ID: Date: Tue, 31 Mar 2020 15:32:53 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 48sKLc2x73z4ZBV X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-2.71 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DMARC_NA(0.00)[sentex.net]; TO_DN_ALL(0.00)[]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; IP_SCORE(-1.71)[ipnet: 2607:f3e0::/32(-4.92), asn: 11647(-3.53), country: CA(-0.09)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Mar 2020 19:36:12 -0000 Hi,     While doing some VM tests, I was making heavy use of clones from my zfs server.  (clones are a super cool handy feature!)... However, when we went live, I accidentally used a clone as the production file system.  Now, I cannot delete those old snapshots.  It seems zfs promote is what I want, but I just want to make sure it will fully work the way I want it to. The history I did was zfs snapshot nfs3zroot/cyclenet@clean3 zfs clone nfs3zroot/cyclenet@clean3 nfs3zroot/cyclenetlive zfs set mountpoint=/cyclenet/live nfs3zroot/cyclenetlive If I do a zfs promote nfs3zroot/cyclenetlive I can then do a zfs destroy nfs3zroot/cyclenet@clean3 safely right ? i.e. nfs3zroot/cyclenetlive will then not have any historical dependencies ? Also, how much impact on the disk IO will the promote command have ? Is it long like a scrub, or quick like a snapshot ?     ---Mike From owner-freebsd-fs@freebsd.org Wed Apr 1 11:02:55 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 300152B84E4 for ; Wed, 1 Apr 2020 11:02:55 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 48sjw65S69z4SgT for ; Wed, 1 Apr 2020 11:02:45 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id 031B2aQb011854; Wed, 1 Apr 2020 12:02:36 +0100 (BST) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id 031B2aPf009260; Wed, 1 Apr 2020 12:02:36 +0100 Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id 031B2an5008896; Wed, 1 Apr 2020 12:02:36 +0100 Date: Wed, 1 Apr 2020 12:02:36 +0100 Message-Id: <202004011102.031B2an5008896@higson.cam.lispworks.com> From: Martin Simmons To: mike tancsa CC: freebsd-fs@freebsd.org In-reply-to: (message from mike tancsa on Tue, 31 Mar 2020 15:32:53 -0400) Subject: Re: zfs promote References: MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 48sjw65S69z4SgT X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of martin@lispworks.com has no SPF policy when checking 46.17.166.21) smtp.mailfrom=martin@lispworks.com X-Spamd-Result: default: False [-0.66 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.91)[-0.913,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.73)[-0.735,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[lispworks.com]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[21.166.17.46.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:51055, ipnet:46.17.166.0/24, country:GB]; IP_SCORE(-0.01)[country: GB(-0.07)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2020 11:02:55 -0000 >>>>> On Tue, 31 Mar 2020 15:32:53 -0400, mike tancsa said: > > Hi, > >     While doing some VM tests, I was making heavy use of clones from my > zfs server.  (clones are a super cool handy feature!)... However, when > we went live, I accidentally used a clone as the production file > system. I suggest giving some more details about your end goal. What was the intended file system structure (i.e. if you had not used a clone)? Were you intending to keep both nfs3zroot/cyclenet and nfs3zroot/cyclenetlive? If so, did you want them to be completely separate, without any sharing of space? >   Now, I cannot delete those old snapshots. Why do you want to delete the old snapshots? Is that multiple snapshots or just @clean3? >   It seems zfs promote > is what I want, but I just want to make sure it will fully work the way > I want it to. > > The history I did was > > zfs snapshot nfs3zroot/cyclenet@clean3 > zfs clone nfs3zroot/cyclenet@clean3 nfs3zroot/cyclenetlive > zfs set mountpoint=/cyclenet/live nfs3zroot/cyclenetlive > > If I do a zfs promote nfs3zroot/cyclenetlive > > I can then do a zfs destroy nfs3zroot/cyclenet@clean3 safely right ? > i.e. nfs3zroot/cyclenetlive will then not have any historical dependencies ? Not quite. Beware this from the documentation: "The clone parent-child dependency relationship is reversed, so that the origin file system becomes a clone of the specified file system." and "The snapshot that was cloned, and any snapshots previous to this snapshot, are now owned by the promoted clone." Therefore doing zfs promote nfs3zroot/cyclenetlive will cause nfs3zroot/cyclenet to become a clone of nfs3zroot/cyclenetlive@clean3. If you then want to destroy nfs3zroot/cyclenetlive@clean3, you will first need to destroy nfs3zroot/cyclenet. > Also, how much impact on the disk IO will the promote command have ? Is > it long like a scrub, or quick like a snapshot ? It is much more like a snapshot than a scrub. __Martin From owner-freebsd-fs@freebsd.org Wed Apr 1 14:10:16 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 538902BCE9C for ; Wed, 1 Apr 2020 14:10:16 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48sp4F69Vqz3C6p for ; Wed, 1 Apr 2020 14:10:05 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:2ca4:3b08:a9aa:a9d4] ([IPv6:2607:f3e0:0:4:2ca4:3b08:a9aa:a9d4]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 031E9tGd005394 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 1 Apr 2020 10:09:55 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: zfs promote To: Martin Simmons Cc: freebsd-fs@freebsd.org References: <202004011102.031B2an5008896@higson.cam.lispworks.com> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWibkB DQRcsMzkAQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4 axtKRSG1t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1 qzAJweEtRdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6c Lm0EiHPOl5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5 o9KKu4O7gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQAB iQE8BBgBCAAmFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAlywzOQCGwwFCQHhM4AACgkQeVOE Fl5WrMhmjQf/dBCjAVn1J0GzSsHiLvSAQz1cchbdy8LD0Tnpzjgp5KLU7sNojbI8vqt4yKAi cayI88j8+xxNXPMWM4pHELuUuVHS5XTpHa/wwulUtI5w/zyKlUDsIvqTPZLUEwH7DfNBueVM WyNaIjV2kxSmM8rNMC+RkgyfbjGLCkmWsMRVuLIUYpl5D9WHmenUbiErlKU2KvEEXEg/aLKq 3m/AdM9RAYsP9O4l+sAZEfyYoNJzDhTZMzn/9Q0uFPLK9smDQh4WBTFaApveVJPHRKmHPoNF Xxj+yScYdQ4SKH34WnhNSELvnZQ3ulH5tpASmm0w+GxfZqSc8+QCwoKtBRDUxoE56A== Message-ID: <42019eac-4211-8265-e73e-d3b418b870fe@sentex.net> Date: Wed, 1 Apr 2020 10:09:55 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: <202004011102.031B2an5008896@higson.cam.lispworks.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 48sp4F69Vqz3C6p X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-2.71 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; DMARC_NA(0.00)[sentex.net]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.71)[ipnet: 2607:f3e0::/32(-4.92), asn: 11647(-3.53), country: CA(-0.09)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2020 14:10:16 -0000 On 4/1/2020 7:02 AM, Martin Simmons wrote: >>>>>> On Tue, 31 Mar 2020 15:32:53 -0400, mike tancsa said: >> Hi, >> >>     While doing some VM tests, I was making heavy use of clones from my >> zfs server.  (clones are a super cool handy feature!)... However, when >> we went live, I accidentally used a clone as the production file >> system. > I suggest giving some more details about your end goal. > > What was the intended file system structure (i.e. if you had not used a > clone)? We use zrepl to manage replication / backups. It keeps a defined amount of local snapshots (a days worth), but has much more on the remote server.  As this file system changes a lot, I dont want to eatup local fast storage by keeping an increasingly older and divergent snapshot around on the main file server. > > Were you intending to keep both nfs3zroot/cyclenet and nfs3zroot/cyclenetlive? No, just nfs3zroot/cyclenetlive   Now, I cannot delete those old snapshots. > Why do you want to delete the old snapshots? Is that multiple snapshots or > just @clean3? The file system will diverge significantly over time from that original snapshot and will take up more and more space which I dont want locally. > Not quite. Beware this from the documentation: > > "The clone parent-child dependency relationship is reversed, so that the > origin file system becomes a clone of the specified file system." > > and > > "The snapshot that was cloned, and any snapshots previous to this snapshot, > are now owned by the promoted clone." > > Therefore doing zfs promote nfs3zroot/cyclenetlive will cause > nfs3zroot/cyclenet to become a clone of nfs3zroot/cyclenetlive@clean3. If you > then want to destroy nfs3zroot/cyclenetlive@clean3, you will first need to > destroy nfs3zroot/cyclenet. Hmm, I am trying to get my head around this. I should first simulate this in a test environment of course.  What you wrote however seems reversed, but perhaps my perspective is backwards to what is written ? ie Originally I did a zfs clone nfs3zroot/cyclenet@clean3 nfs3zroot/cyclenetlive so I read that as I made a clone called cyclenetlive of the snapshot nfs3zroot/cyclenet@clean3 ? So if promote reverses that, so nfs3zroot/cyclenet@clean3 becomes a clone of nfs3zroot/cyclenetlive ?  so I can then delete nfs3zroot/cyclenet@clean3 ? > > >> Also, how much impact on the disk IO will the promote command have ? Is >> it long like a scrub, or quick like a snapshot ? > It is much more like a snapshot than a scrub. Thank you for the help!     ---Mike > > __Martin > From owner-freebsd-fs@freebsd.org Wed Apr 1 16:47:38 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AC191279233 for ; Wed, 1 Apr 2020 16:47:38 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 48ssYt2QhVz3Fv6 for ; Wed, 1 Apr 2020 16:47:29 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id 031GlHHa022874; Wed, 1 Apr 2020 17:47:17 +0100 (BST) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id 031GlHPU011169; Wed, 1 Apr 2020 17:47:17 +0100 Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id 031GlHqa011166; Wed, 1 Apr 2020 17:47:17 +0100 Date: Wed, 1 Apr 2020 17:47:17 +0100 Message-Id: <202004011647.031GlHqa011166@higson.cam.lispworks.com> From: Martin Simmons To: mike tancsa CC: freebsd-fs@freebsd.org In-reply-to: <42019eac-4211-8265-e73e-d3b418b870fe@sentex.net> (message from mike tancsa on Wed, 1 Apr 2020 10:09:55 -0400) Subject: Re: zfs promote References: <202004011102.031B2an5008896@higson.cam.lispworks.com> <42019eac-4211-8265-e73e-d3b418b870fe@sentex.net> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 48ssYt2QhVz3Fv6 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of martin@lispworks.com has no SPF policy when checking 46.17.166.21) smtp.mailfrom=martin@lispworks.com X-Spamd-Result: default: False [-0.63 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.87)[-0.869,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.74)[-0.742,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[lispworks.com]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[21.166.17.46.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:51055, ipnet:46.17.166.0/24, country:GB]; IP_SCORE(-0.01)[country: GB(-0.07)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2020 16:47:39 -0000 >>>>> On Wed, 1 Apr 2020 10:09:55 -0400, mike tancsa said: > > On 4/1/2020 7:02 AM, Martin Simmons wrote: > > > > Were you intending to keep both nfs3zroot/cyclenet and nfs3zroot/cyclenetlive? > > No, just nfs3zroot/cyclenetlive OK, that simplifies things. > > Not quite. Beware this from the documentation: > > > > "The clone parent-child dependency relationship is reversed, so that the > > origin file system becomes a clone of the specified file system." > > > > and > > > > "The snapshot that was cloned, and any snapshots previous to this snapshot, > > are now owned by the promoted clone." > > > > Therefore doing zfs promote nfs3zroot/cyclenetlive will cause > > nfs3zroot/cyclenet to become a clone of nfs3zroot/cyclenetlive@clean3. If you > > then want to destroy nfs3zroot/cyclenetlive@clean3, you will first need to > > destroy nfs3zroot/cyclenet. > > Hmm, I am trying to get my head around this. I should first simulate > this in a test environment of course.  What you wrote however seems > reversed, but perhaps my perspective is backwards to what is written ? Yes, promote is a snapshot twister :-) > ie Originally I did a > > > zfs clone nfs3zroot/cyclenet@clean3 nfs3zroot/cyclenetlive > > so I read that as I made a clone called cyclenetlive of the snapshot > nfs3zroot/cyclenet@clean3 ? Correct. I.e. before promote, nfs3zroot/cyclenet and nfs3zroot/cyclenetlive share data with the @clean3 snapshot of nfs3zroot/cyclenet: nfs3zroot/cyclenet@clean3 ^ ^ | | | nfs3zroot/cyclenetlive | nfs3zroot/cyclenet > So if promote reverses that, so > nfs3zroot/cyclenet@clean3 becomes a clone of nfs3zroot/cyclenetlive ?  > so I can then delete nfs3zroot/cyclenet@clean3 ? No, that is confused, at least in the names of things: nfs3zroot/cyclenet@clean3 is a snapshot so it cannot itself be a clone. What happens is that the @clean3 snapshot is transferred from nfs3zroot/cyclenet to nfs3zroot/cyclenetlive, so it now called nfs3zroot/cyclenetlive@clean3. ("The snapshot that was cloned, and any snapshots previous to this snapshot, are now owned by the promoted clone.") In addition, nfs3zroot/cyclenet ("the origin file system") becomes a clone of nfs3zroot/cyclenetlive ("the specified file system"), joined at the transferred @clean3 snapshot. I.e. after promote, nfs3zroot/cyclenet and nfs3zroot/cyclenetlive share data with the @clean3 snapshot of nfs3zroot/cyclenetlive: nfs3zroot/cyclenetlive@clean3 ^ ^ | | | nfs3zroot/cyclenet | nfs3zroot/cyclenetlive That is why you would then need to destroy nfs3zroot/cyclenet before you can destroy nfs3zroot/cyclenetlive@clean3. __Martin From owner-freebsd-fs@freebsd.org Wed Apr 1 19:30:59 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1057627EC5F for ; Wed, 1 Apr 2020 19:30:59 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48sxBC0FHDz3LmQ for ; Wed, 1 Apr 2020 19:30:42 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:2ca4:3b08:a9aa:a9d4] ([IPv6:2607:f3e0:0:4:2ca4:3b08:a9aa:a9d4]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 031JUXIR028134 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 1 Apr 2020 15:30:33 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: zfs promote To: Martin Simmons Cc: freebsd-fs@freebsd.org References: <202004011102.031B2an5008896@higson.cam.lispworks.com> <42019eac-4211-8265-e73e-d3b418b870fe@sentex.net> <202004011647.031GlHqa011166@higson.cam.lispworks.com> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWibkB DQRcsMzkAQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4 axtKRSG1t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1 qzAJweEtRdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6c Lm0EiHPOl5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5 o9KKu4O7gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQAB iQE8BBgBCAAmFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAlywzOQCGwwFCQHhM4AACgkQeVOE Fl5WrMhmjQf/dBCjAVn1J0GzSsHiLvSAQz1cchbdy8LD0Tnpzjgp5KLU7sNojbI8vqt4yKAi cayI88j8+xxNXPMWM4pHELuUuVHS5XTpHa/wwulUtI5w/zyKlUDsIvqTPZLUEwH7DfNBueVM WyNaIjV2kxSmM8rNMC+RkgyfbjGLCkmWsMRVuLIUYpl5D9WHmenUbiErlKU2KvEEXEg/aLKq 3m/AdM9RAYsP9O4l+sAZEfyYoNJzDhTZMzn/9Q0uFPLK9smDQh4WBTFaApveVJPHRKmHPoNF Xxj+yScYdQ4SKH34WnhNSELvnZQ3ulH5tpASmm0w+GxfZqSc8+QCwoKtBRDUxoE56A== Message-ID: Date: Wed, 1 Apr 2020 15:30:33 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: <202004011647.031GlHqa011166@higson.cam.lispworks.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 48sxBC0FHDz3LmQ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-2.71 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sentex.net]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.71)[ipnet: 2607:f3e0::/32(-4.92), asn: 11647(-3.53), country: CA(-0.09)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2020 19:31:00 -0000 On 4/1/2020 12:47 PM, Martin Simmons wrote: > > I.e. before promote, nfs3zroot/cyclenet and nfs3zroot/cyclenetlive share data > with the @clean3 snapshot of nfs3zroot/cyclenet: > > nfs3zroot/cyclenet@clean3 > ^ ^ > | | > | nfs3zroot/cyclenetlive > | > nfs3zroot/cyclenet > > >> So if promote reverses that, so >> nfs3zroot/cyclenet@clean3 becomes a clone of nfs3zroot/cyclenetlive ?  >> so I can then delete nfs3zroot/cyclenet@clean3 ? > No, that is confused, at least in the names of things: > nfs3zroot/cyclenet@clean3 is a snapshot so it cannot itself be a clone. OK, I think I clarified things for myself with a minimal example of how it works how I got to the place I am at, and what the end result is 0(cage)# zfs create test1/base 0(cage)# touch /test1/base/first-file-on-base 0(cage)# touch /test1/base/second-file-on-base     0(cage)# zfs snapshot test1/base@clean 0(cage)# touch /test1/base/file-that-only-exists-on-base-and-not-clone 0(cage)# zfs clone test1/base@clean test1/clone-of-base 0(cage)# touch /test1/clone-of-base/file-only-on-clone 0(cage)# ls -l /test1/base/ total 6 drwxr-xr-x   2 root  wheel  uarch  5 Apr  1 15:26 . drwxrwxrwx  14 root  wheel  uarch 21 Apr  1 15:26 .. -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:26 file-that-only-exists-on-base-and-not-clone -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:25 first-file-on-base -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:26 second-file-on-base 0(cage)# ls -l /test1/clone-of-base/ total 4 drwxr-xr-x   2 root  wheel  uarch  5 Apr  1 15:26 . drwxrwxrwx  14 root  wheel  uarch 21 Apr  1 15:26 .. -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:26 file-only-on-clone -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:25 first-file-on-base -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:26 second-file-on-base 0(cage)# zfs list -t snapshot | grep clean test1/base@clean                                 19.4K      -  34.4K  - 0(cage)# zfs promote test1/clone-of-base 0(cage)# zfs list -t snapshot | grep clean test1/clone-of-base@clean                        20.9K      -  34.4K  - 0(cage)# zfs destroy -r test1/base 0(cage)# ls -l /test1/clone-of-base/ total 4 drwxr-xr-x   2 root  wheel  uarch  5 Apr  1 15:26 . drwxrwxrwx  13 root  wheel  uarch 20 Apr  1 15:27 .. -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:26 file-only-on-clone -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:25 first-file-on-base -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:26 second-file-on-base 0(cage)# zfs destroy test1/clone-of-base@clean 0(cage)# ls -l /test1/clone-of-base/ total 4 drwxr-xr-x   2 root  wheel  uarch  5 Apr  1 15:26 . drwxrwxrwx  13 root  wheel  uarch 20 Apr  1 15:27 .. -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:26 file-only-on-clone -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:25 first-file-on-base -rw-r--r--   1 root  wheel  uarch  0 Apr  1 15:26 second-file-on-base 0(cage)# Thanks for helping me work this through. I will do a zfs send|zfs recv onto another pool to be on the safe side, but I think I know what I need to do now!     ---Mike From owner-freebsd-fs@freebsd.org Fri Apr 3 13:23:36 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8472B2A2A4A for ; Fri, 3 Apr 2020 13:23:36 +0000 (UTC) (envelope-from artem@artem.ru) Received: from smtp48.i.mail.ru (smtp48.i.mail.ru [94.100.177.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48v0xZ097Dz3yrb for ; Fri, 3 Apr 2020 13:23:29 +0000 (UTC) (envelope-from artem@artem.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Date:Message-ID:Subject:From:To; bh=ubM0AY1IAAhBU180OZZkhoCamMbdZifWoDBJICTxcNE=; b=ZK80RZl/fe290xDef09BXpKo4/0318K4Pk8kw2RFzt7LM9pSUrzR7NpZUUXzDEoxx7lVUZ8VlMm3KPXYAYz+SuaworO42m1iakPbcIWGNgtTBJ5BGwNRpcDKbqsg5vIO1CZucRaL1qL9wIUGu/k6spkF6NeZfqBUu/BDu3wmTWM=; Received: by smtp48.i.mail.ru with esmtpa (envelope-from ) id 1jKMI2-00024z-ND for freebsd-fs@freebsd.org; Fri, 03 Apr 2020 16:23:19 +0300 To: freebsd-fs@freebsd.org From: Artem Kuchin Subject: gpart bootcode Operation not permitted Message-ID: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> Date: Fri, 3 Apr 2020 16:23:06 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-7564579A: 646B95376F6C166E X-77F55803: 0A44E481635329DB0E1AA8A03B392317D32E5E48865217365060145B739F5F5CE273474A718DA2B0F688BCB05C26794D9108A83D692E4A16388EF0F6F7107F3D6B026A3B79F8ADA4A199016549F85F2C X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE74F7EF37B1F033A66EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063748663CFF90615FA98638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FC3A028CFD5D5825711FEADAF9316B34AA174398CFDF5B27DE389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C07E7E81EEA8A9722B8941B15DA834481FCF19DD082D7633A0E7DDDDC251EA7DABA471835C12D1D977725E5C173C3A84C3CA5A41EBD8A3A0199FA2833FD35BB23DF004C906525384303BDABC7E18AA350CD8FC6C240DEA76428AA50765F790063752BC29AF30EF825DD81D268191BDAD3DBD4B6F7A4D31EC0B8E76B049FB1BA93ED81D268191BDAD3D78DA827A17800CE7AC4C3442B3CEE4CDEC76A7562686271E8729DE7A884B61D135872C767BF85DA227C277FBC8AE2E8BAEB924C2B054B06E75ECD9A6C639B01B4E70A05D1297E1BBC6867C52282FAC85D9B7C4F32B44FF57285124B2A10EEC6C00306258E7E6ABB4E4A6367B16DE6309 X-D57D3AED: Y8kq8+OzVozcFQziTi/Zi1xwo7H2ZNxGP5qz8aO2mjTJzjHGC4ogvVuzB3zfVUBtENeZ6b5av1fnCBE34JUDkWdM6QxE+Ga5d8voMtmXfSqfHEUSbqgzgTqYQ0PEDyTA X-Mailru-Sender: 0E9E14D9EC491FBA79C5613A73A5E7B2BB84FC407925F834A98F03896E6AAA4086AB2B85BF7603978A4382C47DA47812C77752E0C033A69E376A1339FE8876DF1FC4F5A70058821069EB1F849E6DBC830DA7A0AF5A3A8387 X-Mras: Ok X-Rspamd-Queue-Id: 48v0xZ097Dz3yrb X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mail.ru header.s=mail2 header.b=ZK80RZl/; dmarc=none; spf=none (mx1.freebsd.org: domain of artem@artem.ru has no SPF policy when checking 94.100.177.108) smtp.mailfrom=artem@artem.ru X-Spamd-Result: default: False [-2.32 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.989,0]; R_DKIM_ALLOW(-0.20)[mail.ru:s=mail2]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.996,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[artem.ru]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(0.06)[ipnet: 94.100.176.0/20(0.06), asn: 47764(0.24), country: RU(0.01)]; DKIM_TRACE(0.00)[mail.ru:+]; R_SPF_NA(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; RCVD_IN_DNSWL_LOW(-0.10)[108.177.100.94.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:47764, ipnet:94.100.176.0/20, country:RU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 13:23:36 -0000 I am lost with this problem. I have gmirror: # gmirror status        Name    Status  Components mirror/boot  COMPLETE  ada0p1 (ACTIVE)                        ada1p1 (ACTIVE) mirror/swap  COMPLETE  ada0p2 (ACTIVE)                        ada1p2 (ACTIVE) mirror/root  COMPLETE  ada0p3 (ACTIVE)                        ada1p3 (ACTIVE) # gpart show =>       34  976773101  ada0  GPT  (466G)          34        128     1  freebsd-boot  (64K)         162    8388608     2  freebsd-swap  (4.0G)     8388770  968384365     3  freebsd-ufs  (462G) =>       34  976773101  ada1  GPT  (466G)          34        128     1  freebsd-boot  (64K)         162    8388608     2  freebsd-swap  (4.0G)     8388770  968384365     3  freebsd-ufs  (462G) And i want to update bootcode on both disks after freebsd updating from source. i Do sysctl -w kern.geom.debugflags=16 # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 gpart: /dev/ada0p1: Operation not permitted # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1 gpart: /dev/ada1p1: Operation not permitted #gpart list Geom name: ada0 modified: false state: OK fwheads: 16 fwsectors: 63 last: 976773134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada0p1    Mediasize: 65536 (64K)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 17408    Mode: r1w1e1    efimedia: HD(1,GPT,346dc1b6-ca13-11e2-a56f-001cc0cc7f7d,0x22,0x80)    rawuuid: 346dc1b6-ca13-11e2-a56f-001cc0cc7f7d    rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f    label: boot0    length: 65536    offset: 17408    type: freebsd-boot    index: 1    end: 161    start: 34 2. Name: ada0p2    Mediasize: 4294967296 (4.0G)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 82944    Mode: r1w1e1    efimedia: HD(2,GPT,3ebf609f-ca13-11e2-a56f-001cc0cc7f7d,0xa2,0x800000)    rawuuid: 3ebf609f-ca13-11e2-a56f-001cc0cc7f7d    rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b    label: swap0    length: 4294967296    offset: 82944    type: freebsd-swap    index: 2    end: 8388769    start: 162 3. Name: ada0p3    Mediasize: 495812794880 (462G)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 82944    Mode: r1w1e1    efimedia: HD(3,GPT,4797d671-ca13-11e2-a56f-001cc0cc7f7d,0x8000a2,0x39b85f6d)    rawuuid: 4797d671-ca13-11e2-a56f-001cc0cc7f7d    rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b    label: root0    length: 495812794880    offset: 4295050240    type: freebsd-ufs    index: 3    end: 976773134    start: 8388770 Consumers: 1. Name: ada0    Mediasize: 500107862016 (466G)    Sectorsize: 512    Mode: r3w3e6 Same for ada1 What's wrong? -- Artem From owner-freebsd-fs@freebsd.org Fri Apr 3 14:11:32 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 18D322A3E8D for ; Fri, 3 Apr 2020 14:11:32 +0000 (UTC) (envelope-from SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48v20m5gjfz4Hdh for ; Fri, 3 Apr 2020 14:11:19 +0000 (UTC) (envelope-from SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id B309A28416; Fri, 3 Apr 2020 16:11:11 +0200 (CEST) Received: from illbsd.quip.test (ip-62-24-92-232.net.upcbroadband.cz [62.24.92.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id E82832840C; Fri, 3 Apr 2020 16:11:10 +0200 (CEST) Subject: Re: gpart bootcode Operation not permitted To: Artem Kuchin , freebsd-fs@freebsd.org References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: Date: Fri, 3 Apr 2020 16:11:10 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 48v20m5gjfz4Hdh X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz has no SPF policy when checking 94.124.105.4) smtp.mailfrom=SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz X-Spamd-Result: default: False [4.02 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(0.83)[ip: (0.29), ipnet: 94.124.104.0/21(0.15), asn: 42000(3.62), country: CZ(0.09)]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.99)[0.993,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; NEURAL_SPAM_LONG(1.00)[0.998,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 14:11:32 -0000 Artem Kuchin wrote on 2020/04/03 15:23: > I am lost with this problem. > > I have gmirror: > > # gmirror status >        Name    Status  Components > mirror/boot  COMPLETE  ada0p1 (ACTIVE) >                        ada1p1 (ACTIVE) > mirror/swap  COMPLETE  ada0p2 (ACTIVE) >                        ada1p2 (ACTIVE) > mirror/root  COMPLETE  ada0p3 (ACTIVE) I you are mirroring boot partition why you need to write directly to individual partitions on both disk instead of mirror? I never used this setup. I mirrored swap, /root, /usr, /var but not boot partition. Then I write gptzfsboot to both ob them individually. Miroslav Lachman From owner-freebsd-fs@freebsd.org Fri Apr 3 14:17:51 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BDDB62A42F6 for ; Fri, 3 Apr 2020 14:17:51 +0000 (UTC) (envelope-from artem@artem.ru) Received: from smtp29.i.mail.ru (smtp29.i.mail.ru [94.100.177.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48v2831SQcz4Kwt for ; Fri, 3 Apr 2020 14:17:38 +0000 (UTC) (envelope-from artem@artem.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:To:Subject; bh=ZV+xekRHpLmO69M7vW21Zoq2Ul0NCLbKU3GB6yDWT7U=; b=ehj83T/H+TybcH2iHe1wMCHkVv3PGM85ArueaBiD9SGNGe0vTgBuk2CUCctnncKFKlntBDpQMZjL/lzxsnn1drhMik5q1JEhmk5imKQRoHRDioCzUpzuHFg2p6zlZ0xiSjqEuCYPByfauSRUIn2kC1mGvQ4pUWA7Ci5KfwDWlBM=; Received: by smtp29.i.mail.ru with esmtpa (envelope-from ) id 1jKN8S-0006pK-4V; Fri, 03 Apr 2020 17:17:28 +0300 Subject: Re: gpart bootcode Operation not permitted To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs@freebsd.org References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> From: Artem Kuchin Message-ID: Date: Fri, 3 Apr 2020 17:17:14 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-7564579A: 646B95376F6C166E X-77F55803: 0A44E481635329DB0E1AA8A03B392317D32E5E48865217365060145B739F5F5C7DB406C2FBDEF77FF688BCB05C26794DCA0A61522C56790B4BAC3423453CCB2CE340203CA89B3B105A4D0F3D38E65D12 X-7FA49CB5: 0D63561A33F958A5265FFCA4FECC78504DF87CF01701123F728E727DF262015E8941B15DA834481FA18204E546F3947C7AE820D2C17D0E56F6B57BC7E64490618DEB871D839B7333395957E7521B51C2545D4CF71C94A83E9FA2833FD35BB23D27C277FBC8AE2E8B8C7ADC89C2F0B2A5A471835C12D1D977C4224003CC8364767815B9869FA544D8D32BA5DBAC0009BE9E8FC8737B5C224955AC6754D1393D37D81D268191BDAD3DBD4B6F7A4D31EC0B8E76B049FB1BA93ED81D268191BDAD3D78DA827A17800CE7C8623B8F170C382FB3661434B16C20AC446828A5085A663B75ECD9A6C639B01B78DA827A17800CE7B2B7C64F398C7410731C566533BA786A40A5AABA2AD371193C9F3DD0FB1AF5EB82E77451A5C57BD33C9F3DD0FB1AF5EB4E70A05D1297E1BBCB5012B2E24CD356 X-D57D3AED: Y8kq8+OzVozcFQziTi/Zi1xwo7H2ZNxGP5qz8aO2mjTJzjHGC4ogvVuzB3zfVUBtENeZ6b5av1fnCBE34JUDkWdM6QxE+Ga5d8voMtmXfSr6mgrAjbDbTgD/ql6jfy+t X-Mailru-Sender: 0E9E14D9EC491FBA79C5613A73A5E7B28B5495C0316DDE1D2B58347D38E3BFFCEC1A5DC81E8A98CC8A4382C47DA47812C77752E0C033A69E376A1339FE8876DF1FC4F5A70058821069EB1F849E6DBC830DA7A0AF5A3A8387 X-Mras: Ok X-Rspamd-Queue-Id: 48v2831SQcz4Kwt X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mail.ru header.s=mail2 header.b=ehj83T/H; dmarc=none; spf=none (mx1.freebsd.org: domain of artem@artem.ru has no SPF policy when checking 94.100.177.89) smtp.mailfrom=artem@artem.ru X-Spamd-Result: default: False [-2.33 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; R_DKIM_ALLOW(-0.20)[mail.ru:s=mail2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.998,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[artem.ru]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[mail.ru:+]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; RCVD_IN_DNSWL_LOW(-0.10)[89.177.100.94.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:47764, ipnet:94.100.176.0/20, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.06)[ipnet: 94.100.176.0/20(0.06), asn: 47764(0.24), country: RU(0.01)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 14:17:51 -0000 03.04.2020 17:11, Miroslav Lachman пишет: > Artem Kuchin wrote on 2020/04/03 15:23: >> I am lost with this problem. >> >> I have gmirror: >> >> # gmirror status >>         Name    Status  Components >> mirror/boot  COMPLETE  ada0p1 (ACTIVE) >>                         ada1p1 (ACTIVE) >> mirror/swap  COMPLETE  ada0p2 (ACTIVE) >>                         ada1p2 (ACTIVE) >> mirror/root  COMPLETE  ada0p3 (ACTIVE) > > I you are mirroring boot partition why you need to write directly to > individual partitions on both disk instead of mirror? > > I never used this setup. I mirrored swap, /root, /usr, /var but not > boot partition. Then I write gptzfsboot to both ob them individually. > > Miroslav Lachman > I don't know :) It is a historical setup. I even would say that mirroring swap is a bad idea, isn't it? But i need to have a working boot code on both disks because in case of failure i need to boot from any disk. Protective MBR is not mirrored anyway, so, need to install it separatelly on each disk for sure. Installing boot code into boot partion maybe can be done on mirror partition (or whatever it is calleed mirror/boot) but then how do i address it in the gpart boocode command ? Artem From owner-freebsd-fs@freebsd.org Fri Apr 3 14:42:53 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BDA692A4D82 for ; Fri, 3 Apr 2020 14:42:53 +0000 (UTC) (envelope-from SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48v2ht5pxyz4V2b for ; Fri, 3 Apr 2020 14:42:38 +0000 (UTC) (envelope-from SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 1219928417; Fri, 3 Apr 2020 16:42:33 +0200 (CEST) Received: from illbsd.quip.test (ip-62-24-92-232.net.upcbroadband.cz [62.24.92.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 254A02840C; Fri, 3 Apr 2020 16:42:32 +0200 (CEST) Subject: Re: gpart bootcode Operation not permitted To: Artem Kuchin , freebsd-fs@freebsd.org References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <48ba23fa-13dc-4a74-579c-2028479f302a@quip.cz> Date: Fri, 3 Apr 2020 16:42:32 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 48v2ht5pxyz4V2b X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz has no SPF policy when checking 94.124.105.4) smtp.mailfrom=SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz X-Spamd-Result: default: False [4.02 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(0.83)[ip: (0.29), ipnet: 94.124.104.0/21(0.14), asn: 42000(3.62), country: CZ(0.09)]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.99)[0.992,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; SPAMHAUS_AUTHBL_RECEIVED_FAIL(0.00)[232.92.24.62.khpj7ygk5idzvmvt5x4ziurxhy.authbl.dq.spamhaus.net:query timed out]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; NEURAL_SPAM_LONG(1.00)[0.998,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_FAIL(0.00)[232.92.24.62.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net:query timed out] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 14:42:53 -0000 Artem Kuchin wrote on 2020/04/03 16:17: > 03.04.2020 17:11, Miroslav Lachman пишет: >> Artem Kuchin wrote on 2020/04/03 15:23: >>> I am lost with this problem. >>> >>> I have gmirror: >>> >>> # gmirror status >>>         Name    Status  Components >>> mirror/boot  COMPLETE  ada0p1 (ACTIVE) >>>                         ada1p1 (ACTIVE) >>> mirror/swap  COMPLETE  ada0p2 (ACTIVE) >>>                         ada1p2 (ACTIVE) >>> mirror/root  COMPLETE  ada0p3 (ACTIVE) >> >> I you are mirroring boot partition why you need to write directly to >> individual partitions on both disk instead of mirror? >> >> I never used this setup. I mirrored swap, /root, /usr, /var but not >> boot partition. Then I write gptzfsboot to both ob them individually. >> >> Miroslav Lachman >> > I don't know :) It is a historical setup. I even would say that > mirroring swap is a bad idea, isn't it? It depends on what you need to protect by mirroring. If you need it just to protect saved data (desktop usage) then you don't need to mirror swap. If you need to stay up and running if some HDD failed (servers) then you need to mirror swap too. Or the system will panic. > But i need to have a working boot code on both disks because in case of > failure i need to boot from any disk. Yes you need to update boot code on each disk but as it is normally not modified you don't need to mirror it you just need to update it after system update / upgrade. > Protective MBR is not mirrored anyway, so, need to install it > separatelly on each disk for sure. Installing boot code into > > boot partion maybe can be done on mirror partition (or whatever it is > calleed mirror/boot) but then how do i address it I think you can use "gmirror stop" on boot partition, then "gmirror clear" and then update both individual boot partitions by gpart. Miroslav Lachman From owner-freebsd-fs@freebsd.org Fri Apr 3 14:54:37 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4A3F42A554E for ; Fri, 3 Apr 2020 14:54:37 +0000 (UTC) (envelope-from bob@eager.cx) Received: from kipling.tavi.co.uk (kipling.tavi.co.uk [81.187.145.130]) by mx1.freebsd.org (Postfix) with ESMTP id 48v2yR4D58z4Z31 for ; Fri, 3 Apr 2020 14:54:22 +0000 (UTC) (envelope-from bob@eager.cx) Received: from kipling.tavi.co.uk (localhost [127.0.0.1]) by kipling.tavi.co.uk (Postfix) with ESMTP id 2E3573BA2A for ; Fri, 3 Apr 2020 15:54:13 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=eager.cx; h=date:from:to :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=selector1; bh=oeBUF3X 18yDOsZu0ux8IAbXGs4c=; b=VawVXqzZq+peVAtpNBAAMgvq2qIjzc9FF/Qr+qp DeV6dQ/IBATwU1l/xzX14asGyhEE1aGkikQT3XCYg9UmolNkWR+KQkSi1RkB9/uG rr9QO/93Gy0eOVXC0SZpYWt0/r829zPtetnkUUY51ZF7agR1muChMKrUHoxgX0gB 4Hf8= Received: from raksha.tavi.co.uk (raksha.tavi.co.uk [81.187.145.139]) (Authenticated sender: rde@tavi.co.uk) by kipling.tavi.co.uk (Postfix) with ESMTPA id EBE923B7E1 for ; Fri, 3 Apr 2020 15:54:12 +0100 (BST) Date: Fri, 3 Apr 2020 15:54:12 +0100 From: Bob Eager To: freebsd-fs@freebsd.org Subject: Re: gpart bootcode Operation not permitted Message-ID: <20200403155412.4f4b1f49@raksha.tavi.co.uk> In-Reply-To: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; amd64-portbld-freebsd12.0) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAMFBMVEUwXjFLc0vD0cS7y7zw9PDZ4tkWSRaVrZZ+m39qi2tXfVj////7+/utwK4IPggAOAAJUUA7AAABKklEQVQ4jWPYjQMwDFYJp0NKEKCNJmEf9h8CsimXiL2e33s3/e7F7K2Cs3f3dCMkQkMKj4YuCY3K3iR+e7fMaiSjvkX0/5cFGrWpe2uLzOpaExUVqMS/8PX/Re5ey960OLBTZpFA8+IlSBKPQ92zNyUUBsosN58uIY0k8f+/ONCoYytkVuhWzVwNkYiYbqk5M3NmOVBi41YZ8RsGF7shEtFb5KJ3r969CyixM7OTPeFUxG2IxLO8/9/SvqXlc+/x3h295YzLlj2nIRJQj//nRvc5TEIal8RsXBLVuCQwIgoq/u80DomP6HEOk/iOS+IJLonZOCT+ReOQ+Lkbh0QKLonbOCR+7MYhsRqHBJrVcIl/1TgklqKLQyQ+tGKIgyQOqXpjig94diZRAgAXmDX6jyWafAAAAABJRU5ErkJggg====== MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 48v2yR4D58z4Z31 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=eager.cx header.s=selector1 header.b=VawVXqzZ; dmarc=pass (policy=none) header.from=eager.cx; spf=pass (mx1.freebsd.org: domain of bob@eager.cx designates 81.187.145.130 as permitted sender) smtp.mailfrom=bob@eager.cx X-Spamd-Result: default: False [-6.49 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[eager.cx:s=selector1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:kipling.tavi.co.uk]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[eager.cx:+]; DMARC_POLICY_ALLOW(-0.50)[eager.cx,none]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-3.59)[ip: (-9.50), ipnet: 81.187.0.0/16(-4.70), asn: 20712(-3.67), country: GB(-0.07)]; ASN(0.00)[asn:20712, ipnet:81.187.0.0/16, country:GB] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 14:54:37 -0000 I think you need to do this before starting the mirror. So, boot from CD or USB, without loading geom_mirror. Update the boot code. Boot normally. (I don't think the debug flags have been needed for a while; I never set them in that situation) On Fri, 3 Apr 2020 16:23:06 +0300 Artem Kuchin wrote: > I am lost with this problem. >=20 > I have gmirror: >=20 > # gmirror status > =A0=A0=A0=A0=A0=A0 Name=A0=A0=A0 Status=A0 Components > mirror/boot=A0 COMPLETE=A0 ada0p1 (ACTIVE) > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ada1p= 1 (ACTIVE) > mirror/swap=A0 COMPLETE=A0 ada0p2 (ACTIVE) > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ada1p= 2 (ACTIVE) > mirror/root=A0 COMPLETE=A0 ada0p3 (ACTIVE) > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ada1p= 3 (ACTIVE) >=20 >=20 > # gpart show > =3D>=A0=A0=A0=A0=A0=A0 34=A0 976773101=A0 ada0=A0 GPT=A0 (466G) =20 > =A0=A0=A0=A0=A0=A0=A0=A0 34=A0=A0=A0=A0=A0=A0=A0 128=A0=A0=A0=A0 1=A0 fr= eebsd-boot=A0 (64K) > =A0=A0=A0=A0=A0=A0=A0 162=A0=A0=A0 8388608=A0=A0=A0=A0 2=A0 freebsd-swap= =A0 (4.0G) > =A0=A0=A0 8388770=A0 968384365=A0=A0=A0=A0 3=A0 freebsd-ufs=A0 (462G) >=20 > =3D>=A0=A0=A0=A0=A0=A0 34=A0 976773101=A0 ada1=A0 GPT=A0 (466G) =20 > =A0=A0=A0=A0=A0=A0=A0=A0 34=A0=A0=A0=A0=A0=A0=A0 128=A0=A0=A0=A0 1=A0 fr= eebsd-boot=A0 (64K) > =A0=A0=A0=A0=A0=A0=A0 162=A0=A0=A0 8388608=A0=A0=A0=A0 2=A0 freebsd-swap= =A0 (4.0G) > =A0=A0=A0 8388770=A0 968384365=A0=A0=A0=A0 3=A0 freebsd-ufs=A0 (462G) >=20 >=20 > And i want to update bootcode on both disks after freebsd updating > from source. >=20 >=20 > i Do >=20 > sysctl -w kern.geom.debugflags=3D16 >=20 > # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 > gpart: /dev/ada0p1: Operation not permitted >=20 >=20 > # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1 > gpart: /dev/ada1p1: Operation not permitted >=20 >=20 > #gpart list >=20 > Geom name: ada0 > modified: false > state: OK > fwheads: 16 > fwsectors: 63 > last: 976773134 > first: 34 > entries: 128 > scheme: GPT > Providers: > 1. Name: ada0p1 > =A0=A0 Mediasize: 65536 (64K) > =A0=A0 Sectorsize: 512 > =A0=A0 Stripesize: 0 > =A0=A0 Stripeoffset: 17408 > =A0=A0 Mode: r1w1e1 > =A0=A0 efimedia: HD(1,GPT,346dc1b6-ca13-11e2-a56f-001cc0cc7f7d,0x22,0x80) > =A0=A0 rawuuid: 346dc1b6-ca13-11e2-a56f-001cc0cc7f7d > =A0=A0 rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f > =A0=A0 label: boot0 > =A0=A0 length: 65536 > =A0=A0 offset: 17408 > =A0=A0 type: freebsd-boot > =A0=A0 index: 1 > =A0=A0 end: 161 > =A0=A0 start: 34 > 2. Name: ada0p2 > =A0=A0 Mediasize: 4294967296 (4.0G) > =A0=A0 Sectorsize: 512 > =A0=A0 Stripesize: 0 > =A0=A0 Stripeoffset: 82944 > =A0=A0 Mode: r1w1e1 > =A0=A0 efimedia: > HD(2,GPT,3ebf609f-ca13-11e2-a56f-001cc0cc7f7d,0xa2,0x800000) rawuuid: > 3ebf609f-ca13-11e2-a56f-001cc0cc7f7d rawtype: > 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: swap0 > =A0=A0 length: 4294967296 > =A0=A0 offset: 82944 > =A0=A0 type: freebsd-swap > =A0=A0 index: 2 > =A0=A0 end: 8388769 > =A0=A0 start: 162 > 3. Name: ada0p3 > =A0=A0 Mediasize: 495812794880 (462G) > =A0=A0 Sectorsize: 512 > =A0=A0 Stripesize: 0 > =A0=A0 Stripeoffset: 82944 > =A0=A0 Mode: r1w1e1 > =A0=A0 efimedia:=20 > HD(3,GPT,4797d671-ca13-11e2-a56f-001cc0cc7f7d,0x8000a2,0x39b85f6d) > =A0=A0 rawuuid: 4797d671-ca13-11e2-a56f-001cc0cc7f7d > =A0=A0 rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b > =A0=A0 label: root0 > =A0=A0 length: 495812794880 > =A0=A0 offset: 4295050240 > =A0=A0 type: freebsd-ufs > =A0=A0 index: 3 > =A0=A0 end: 976773134 > =A0=A0 start: 8388770 > Consumers: > 1. Name: ada0 > =A0=A0 Mediasize: 500107862016 (466G) > =A0=A0 Sectorsize: 512 > =A0=A0 Mode: r3w3e6 >=20 > Same for ada1 >=20 > What's wrong? >=20 >=20 > -- >=20 > Artem >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 Bob From owner-freebsd-fs@freebsd.org Fri Apr 3 19:24:05 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 30F8E2657FB for ; Fri, 3 Apr 2020 19:24:05 +0000 (UTC) (envelope-from artem@artem.ru) Received: from fallback19.mail.ru (fallback19.mail.ru [185.5.136.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48v8xS29xTz4Kdf for ; Fri, 3 Apr 2020 19:23:55 +0000 (UTC) (envelope-from artem@artem.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:To:Subject; bh=faxuzCnqM3Gvun4WEMUIcS/gUcISH6LT2+Y8abqhmbo=; b=Vq6Vv/5L5jPDM95T5JHASsXnB/4XgxeBrYwyWFKGFtfCJDWzRRX51dQfuMFzzv0z0bAyv4+O/3xFjgSAkQLRtsvBb4Q1nAdDclONdX6IAvc7EyiWoLXitqQ9jN75xzzolCq5YsRH4yMWaWi7qDMOhX08jMphDneL0hiN0xqTGXI=; Received: from [10.161.64.52] (port=42148 helo=smtp44.i.mail.ru) by fallback19.m.smailru.net with esmtp (envelope-from ) id 1jKRur-0005Sr-FM for freebsd-fs@freebsd.org; Fri, 03 Apr 2020 22:23:45 +0300 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:To:Subject; bh=faxuzCnqM3Gvun4WEMUIcS/gUcISH6LT2+Y8abqhmbo=; b=Vq6Vv/5L5jPDM95T5JHASsXnB/4XgxeBrYwyWFKGFtfCJDWzRRX51dQfuMFzzv0z0bAyv4+O/3xFjgSAkQLRtsvBb4Q1nAdDclONdX6IAvc7EyiWoLXitqQ9jN75xzzolCq5YsRH4yMWaWi7qDMOhX08jMphDneL0hiN0xqTGXI=; Received: by smtp44.i.mail.ru with esmtpa (envelope-from ) id 1jKRup-0001Fk-Sb; Fri, 03 Apr 2020 22:23:44 +0300 Subject: Re: gpart bootcode Operation not permitted To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs@freebsd.org References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> <48ba23fa-13dc-4a74-579c-2028479f302a@quip.cz> From: Artem Kuchin Message-ID: <1605a04a-ad2c-5b2c-445f-fb8ccd7211d6@artem.ru> Date: Fri, 3 Apr 2020 22:23:44 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: <48ba23fa-13dc-4a74-579c-2028479f302a@quip.cz> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-7564579A: 646B95376F6C166E X-77F55803: 0A44E481635329DB0E1AA8A03B392317D32E5E48865217365060145B739F5F5C0F1B420F94771604F688BCB05C26794D0B69F6E0BD4E66D1687D7042622DFE379CF5E81C89F28AE6428EA7336302C8F6 X-7FA49CB5: 0D63561A33F958A55087A50933991B3E83EB63B4176FF930D88B2F9C13F1D7088941B15DA834481FA18204E546F3947C2FFDA4F57982C5F4F6B57BC7E64490618DEB871D839B7333395957E7521B51C2545D4CF71C94A83E9FA2833FD35BB23D27C277FBC8AE2E8B974A882099E279BDA471835C12D1D977C4224003CC8364767815B9869FA544D8D32BA5DBAC0009BE9E8FC8737B5C2249E66CFA920896299B76E601842F6C81A12EF20D2F80756B5F012D6517FE479FCD76E601842F6C81A127C277FBC8AE2E8BD41DDD0A7AC51B863AA81AA40904B5D99449624AB7ADAF3726B9191E2D567F0E725E5C173C3A84C34B08FA16E56A400835872C767BF85DA2F004C906525384306FED454B719173D6462275124DF8B9C9DE2850DD75B2526BE5BFE6E7EFDEDCD789D4C264860C145E X-D57D3AED: Y8kq8+OzVozcFQziTi/Zi1xwo7H2ZNxGP5qz8aO2mjTJzjHGC4ogvVuzB3zfVUBtENeZ6b5av1fnCBE34JUDkWdM6QxE+Ga5d8voMtmXfSqNNzAPQfPzgrRFuIOw5Q1i X-Mailru-Sender: 332320C0CE44B500CFE390B0004BF5C8BB322AC33489D4C159C8969F7C4701D2F1A7E6939DF55CF20A1FD29A504278DEE66B5C1DBFD5D09D2FFF0A5F0DFA254CD0701747CC0EF98689F635B1FCB23A66AE208404248635DF X-Mras: Ok X-7564579A: EEAE043A70213CC8 X-77F55803: 669901E4625912A97F9F52485CB584D7271FD7DF62800FDCFBFA32D92148CD71191528D1B8DD0848C47E5BB426707DFE85F47BC90E736BA8 X-7FA49CB5: 0D63561A33F958A5DE58E560561A628FD478082534706F1A56B969DDDAAA761E8941B15DA834481FA18204E546F3947C2FFDA4F57982C5F4F6B57BC7E64490618DEB871D839B7333395957E7521B51C2545D4CF71C94A83E9FA2833FD35BB23D27C277FBC8AE2E8B974A882099E279BDA471835C12D1D977C4224003CC8364767815B9869FA544D8D32BA5DBAC0009BE9E8FC8737B5C2249E66CFA920896299B76E601842F6C81A12EF20D2F80756B5F012D6517FE479FCD76E601842F6C81A127C277FBC8AE2E8BD41DDD0A7AC51B863AA81AA40904B5D99449624AB7ADAF3726B9191E2D567F0E725E5C173C3A84C34B08FA16E56A400835872C767BF85DA2F004C906525384306FED454B719173D6462275124DF8B9C9DE2850DD75B2526BE5BFE6E7EFDEDCD789D4C264860C145E X-D57D3AED: Y8kq8+OzVozcFQziTi/Zi1xwo7H2ZNxGP5qz8aO2mjTJzjHGC4ogvVuzB3zfVUBtENeZ6b5av1fnCBE34JUDkWdM6QxE+Ga5d8voMtmXfSqNNzAPQfPzgpuf5RTkiCWD X-Mailru-MI: 800 X-Mailru-Sender: A5480F10D64C9005265567A7F4CD75190B3DF64513278CD4191528D1B8DD0848DE5AA3C1F4620B2D8A4382C47DA47812C77752E0C033A69E376A1339FE8876DF1FC4F5A70058821069EB1F849E6DBC830DA7A0AF5A3A8387 X-Mras: Ok X-Rspamd-Queue-Id: 48v8xS29xTz4Kdf X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mail.ru header.s=mail2 header.b=Vq6Vv/5L; dkim=pass header.d=mail.ru header.s=mail2 header.b=Vq6Vv/5L; dmarc=none; spf=none (mx1.freebsd.org: domain of artem@artem.ru has no SPF policy when checking 185.5.136.251) smtp.mailfrom=artem@artem.ru X-Spamd-Result: default: False [0.32 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.09)[-0.087,0]; R_DKIM_ALLOW(-0.20)[mail.ru:s=mail2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[artem.ru]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[mail.ru:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[251.136.5.185.list.dnswl.org : 127.0.5.0]; NEURAL_SPAM_LONG(0.01)[0.006,0]; R_SPF_NA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[251.136.5.185.rep.mailspike.net : 127.0.0.17]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:47764, ipnet:185.5.136.0/22, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.70)[ip: (3.21), ipnet: 185.5.136.0/22(0.06), asn: 47764(0.24), country: RU(0.01)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 19:24:05 -0000 03.04.2020 17:42, Miroslav Lachman пишет: > > I think you can use "gmirror stop" on boot partition, then "gmirror > clear" and then update both individual boot partitions by gpart. > > I am afraid to do it. Gmirror stops mirroring, okay. What what gmirror clear does? What metadata is cleared and what it is used for? How to restart mirroring after that and how to make sure that mirror is 100% complete? Is it possible to exclude only boot partition from mirroring? Artem From owner-freebsd-fs@freebsd.org Fri Apr 3 19:49:10 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 61201266517 for ; Fri, 3 Apr 2020 19:49:10 +0000 (UTC) (envelope-from artem@artem.ru) Received: from fallback9.mail.ru (fallback9.mail.ru [94.100.178.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48v9VP2MFmz4TmR for ; Fri, 3 Apr 2020 19:48:59 +0000 (UTC) (envelope-from artem@artem.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:To:Subject; bh=KMBMjTpTyL6Fkx152CEK68slXknCpOXl+bUDNtfNmCY=; b=l04eOnZ6sa2WJp9yHa/07e4OcFKiCSzYtzrT5lTKt5NrKGMzuLw9WFZLNtCoDigGWZEvqSbnALAlWVDMT5kCXfN81MBjyaI4JZw+whOITv+j+LMbcRy1MlKNnujtKxvfAqfr1q2fCDZB8wD4F7tU2zCoMgz37DRSjprCtJmL3sY=; Received: from [10.161.64.55] (port=52960 helo=smtp47.i.mail.ru) by fallback9.m.smailru.net with esmtp (envelope-from ) id 1jKSJ5-0002sg-KR for freebsd-fs@freebsd.org; Fri, 03 Apr 2020 22:48:47 +0300 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:To:Subject; bh=KMBMjTpTyL6Fkx152CEK68slXknCpOXl+bUDNtfNmCY=; b=l04eOnZ6sa2WJp9yHa/07e4OcFKiCSzYtzrT5lTKt5NrKGMzuLw9WFZLNtCoDigGWZEvqSbnALAlWVDMT5kCXfN81MBjyaI4JZw+whOITv+j+LMbcRy1MlKNnujtKxvfAqfr1q2fCDZB8wD4F7tU2zCoMgz37DRSjprCtJmL3sY=; Received: by smtp47.i.mail.ru with esmtpa (envelope-from ) id 1jKSJ4-0006NV-Ls; Fri, 03 Apr 2020 22:48:46 +0300 Subject: Re: gpart bootcode Operation not permitted To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs@freebsd.org References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> <48ba23fa-13dc-4a74-579c-2028479f302a@quip.cz> <1605a04a-ad2c-5b2c-445f-fb8ccd7211d6@artem.ru> <159cf2f5-1898-019e-9f02-29750ff7fa7e@quip.cz> From: Artem Kuchin Message-ID: Date: Fri, 3 Apr 2020 22:48:47 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: <159cf2f5-1898-019e-9f02-29750ff7fa7e@quip.cz> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-7564579A: 646B95376F6C166E X-77F55803: 0A44E481635329DB0E1AA8A03B392317D32E5E48865217365060145B739F5F5CD6A7815655207F63F688BCB05C26794DEBCE1206A28A6B7ECED3CCF2C74F91C8114FCC263523FC11A8ABD66E6EFF508F X-7FA49CB5: 0D63561A33F958A52D81D746AFD1661F3C42E79DB4FE5043063BDCFD0BAA14CE8941B15DA834481FA18204E546F3947CD2DCF9CF1F528DBCF6B57BC7E64490618DEB871D839B7333395957E7521B51C2545D4CF71C94A83E9FA2833FD35BB23D27C277FBC8AE2E8B3A703B70628EAD7BA471835C12D1D977C4224003CC8364767815B9869FA544D8D32BA5DBAC0009BE9E8FC8737B5C224937E7452263E0972376E601842F6C81A12EF20D2F80756B5F012D6517FE479FCD76E601842F6C81A127C277FBC8AE2E8B8FA608DA6CD568A33AA81AA40904B5D99449624AB7ADAF3726B9191E2D567F0E725E5C173C3A84C34B08FA16E56A400835872C767BF85DA2F004C906525384306FED454B719173D6462275124DF8B9C9DE2850DD75B2526BE5BFE6E7EFDEDCD789D4C264860C145E X-D57D3AED: Y8kq8+OzVozcFQziTi/Zi1xwo7H2ZNxGP5qz8aO2mjTJzjHGC4ogvVuzB3zfVUBtENeZ6b5av1fnCBE34JUDkWdM6QxE+Ga5d8voMtmXfSr0/d6/f3e+FZf1Wbpd8t+6 X-Mailru-Sender: 332320C0CE44B500CFE390B0004BF5C85CBE817E35978DC38D01CEDDF71F5EDAC491D0F6EAF1B3220A1FD29A504278DEE66B5C1DBFD5D09D2FFF0A5F0DFA254CD0701747CC0EF98689F635B1FCB23A66AE208404248635DF X-Mras: Ok X-7564579A: EEAE043A70213CC8 X-77F55803: E8DB3678F13EF3E07F9F52485CB584D7271FD7DF62800FDC7A8E4B582222F5915FC857BB8C580187B64AECE9756F5AA6EF7B3EE531A61F57 X-7FA49CB5: 0D63561A33F958A539AF5AD87C13C640EF0D3CAA6C5881B742E5CD0B2061B87E8941B15DA834481FA18204E546F3947C062BEEFFB5F8EA3EF6B57BC7E64490618DEB871D839B7333395957E7521B51C2545D4CF71C94A83E9FA2833FD35BB23D27C277FBC8AE2E8B974A882099E279BDA471835C12D1D977C4224003CC8364767815B9869FA544D8D32BA5DBAC0009BE9E8FC8737B5C224937E7452263E0972376E601842F6C81A12EF20D2F80756B5F012D6517FE479FCD76E601842F6C81A127C277FBC8AE2E8B8FA608DA6CD568A33AA81AA40904B5D99449624AB7ADAF3726B9191E2D567F0E725E5C173C3A84C34B08FA16E56A400835872C767BF85DA2F004C906525384306FED454B719173D6462275124DF8B9C9DE2850DD75B2526BE5BFE6E7EFDEDCD789D4C264860C145E X-D57D3AED: Y8kq8+OzVozcFQziTi/Zi1xwo7H2ZNxGP5qz8aO2mjTJzjHGC4ogvVuzB3zfVUBtENeZ6b5av1fnCBE34JUDkWdM6QxE+Ga5d8voMtmXfSr0/d6/f3e+FZzUdJfQ/bjz X-Mailru-MI: 800 X-Mailru-Sender: A5480F10D64C9005EB82917A46D952765BA779D13F6CAE915FC857BB8C580187E64F4A3047C6771E8A4382C47DA47812C77752E0C033A69E376A1339FE8876DF1FC4F5A70058821069EB1F849E6DBC830DA7A0AF5A3A8387 X-Mras: Ok X-Rspamd-Queue-Id: 48v9VP2MFmz4TmR X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mail.ru header.s=mail2 header.b=l04eOnZ6; dkim=pass header.d=mail.ru header.s=mail2 header.b=l04eOnZ6; dmarc=none; spf=none (mx1.freebsd.org: domain of artem@artem.ru has no SPF policy when checking 94.100.178.49) smtp.mailfrom=artem@artem.ru X-Spamd-Result: default: False [1.01 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[mail.ru:s=mail2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[artem.ru]; NEURAL_SPAM_MEDIUM(0.19)[0.192,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[mail.ru:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[49.178.100.94.list.dnswl.org : 127.0.5.0]; NEURAL_SPAM_LONG(0.31)[0.306,0]; R_SPF_NA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[49.178.100.94.rep.mailspike.net : 127.0.0.17]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:47764, ipnet:94.100.176.0/20, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.81)[ip: (3.75), ipnet: 94.100.176.0/20(0.06), asn: 47764(0.24), country: RU(0.01)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 19:49:10 -0000 03.04.2020 22:44, Miroslav Lachman пишет: > Artem Kuchin wrote on 2020/04/03 21:23: > > gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 > gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1 > But why i cannot then just install it into mirror/boot? It does now even show it # gpart show mirror/boot gpart: No such geom: mirror/boot. # gpart show boot gpart: No such geom: boot. but geom mirror list Geom name: boot State: COMPLETE Components: 2 Balance: load Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ID: 3273908325 Type: AUTOMATIC Providers: 1. Name: mirror/boot    Mediasize: 65024 (64K)    Sectorsize: 512    Mode: r0w0e0 Consumers: 1. Name: ada0p1    Mediasize: 65536 (64K)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 17408    Mode: r1w1e1    State: ACTIVE    Priority: 0    Flags: NONE    GenID: 0    SyncID: 1    ID: 1407712434 2. Name: ada1p1    Mediasize: 65536 (64K)    Sectorsize: 512    Stripesize: 0    Stripeoffset: 17408    Mode: r1w1e1    State: ACTIVE    Priority: 1    Flags: NONE    GenID: 0    SyncID: 1    ID: 1997874466 From owner-freebsd-fs@freebsd.org Fri Apr 3 19:54:22 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C32BF266877 for ; Fri, 3 Apr 2020 19:54:22 +0000 (UTC) (envelope-from SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48v9cL0R7sz4WVd for ; Fri, 3 Apr 2020 19:54:09 +0000 (UTC) (envelope-from SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id CFF8D28416; Fri, 3 Apr 2020 21:44:54 +0200 (CEST) Received: from illbsd.quip.test (ip-62-24-92-232.net.upcbroadband.cz [62.24.92.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id D11212840C; Fri, 3 Apr 2020 21:44:52 +0200 (CEST) Subject: Re: gpart bootcode Operation not permitted To: Artem Kuchin , freebsd-fs@freebsd.org References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> <48ba23fa-13dc-4a74-579c-2028479f302a@quip.cz> <1605a04a-ad2c-5b2c-445f-fb8ccd7211d6@artem.ru> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <159cf2f5-1898-019e-9f02-29750ff7fa7e@quip.cz> Date: Fri, 3 Apr 2020 21:44:52 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: <1605a04a-ad2c-5b2c-445f-fb8ccd7211d6@artem.ru> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 48v9cL0R7sz4WVd X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz has no SPF policy when checking 94.124.105.4) smtp.mailfrom=SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz X-Spamd-Result: default: False [4.01 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(0.83)[ip: (0.29), ipnet: 94.124.104.0/21(0.14), asn: 42000(3.62), country: CZ(0.09)]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.99)[0.989,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; NEURAL_SPAM_LONG(1.00)[0.996,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 19:54:23 -0000 Artem Kuchin wrote on 2020/04/03 21:23: > 03.04.2020 17:42, Miroslav Lachman пишет: >> >> I think you can use "gmirror stop" on boot partition, then "gmirror >> clear" and then update both individual boot partitions by gpart. >> >> > > I am afraid to do it. Gmirror stops mirroring, okay. What what gmirror > clear does? What metadata is cleared and what it is used for? > > How to restart mirroring after that and how to make sure that mirror is > 100% complete? > > Is it possible to exclude only boot partition from mirroring? Of course that's what I am suggesting - just split up mirroring of boot partition and keep mirroring on swap and root! Gmirror clear deletes the very last sector on the given partition where metadata of gmirror are stored. It does not touch the data on the partitions. You don't need to worry. You have 3 mirrors. They are visible in /dev/mirror or by command "gmirror status": boot swap root As the "man gmirror" shows: gmirror stop [-fv] name ... You can run: gmirror stop -v boot It stops only the "boot" mirror. swap and root are still mirrored. Then you can clear metadata on each provider: gmirror clear ada0p1 gmirror clear ada1p1 Or you can use "gmirror destroy -v boot" istead of 3 command above. It should stop the "boot" mirror and then clear metadata on both providers. The you will have / your system will use 2 independent boot partitions: ada0p1 and ada1p1. The machine should be able to boot from any of those 2 disks because each of them has valid boot partitions / boot code. Gpart bootcode should work for them on each system upgrade. gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1 Kind regards Miroslav Lachman From owner-freebsd-fs@freebsd.org Fri Apr 3 21:09:08 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7C3962716E5 for ; Fri, 3 Apr 2020 21:09:08 +0000 (UTC) (envelope-from SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48vCGc2V2kz41sv for ; Fri, 3 Apr 2020 21:08:55 +0000 (UTC) (envelope-from SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 29E6728416; Fri, 3 Apr 2020 23:08:43 +0200 (CEST) Received: from illbsd.quip.test (ip-62-24-92-232.net.upcbroadband.cz [62.24.92.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 601E928422; Fri, 3 Apr 2020 23:08:42 +0200 (CEST) Subject: Re: gpart bootcode Operation not permitted To: Artem Kuchin , freebsd-fs@freebsd.org References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> <48ba23fa-13dc-4a74-579c-2028479f302a@quip.cz> <1605a04a-ad2c-5b2c-445f-fb8ccd7211d6@artem.ru> <159cf2f5-1898-019e-9f02-29750ff7fa7e@quip.cz> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <1127258c-8d0b-7f36-491c-ca8897ce6958@quip.cz> Date: Fri, 3 Apr 2020 23:08:42 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 48vCGc2V2kz41sv X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz has no SPF policy when checking 94.124.105.4) smtp.mailfrom=SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz X-Spamd-Result: default: False [4.02 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(0.83)[ip: (0.29), ipnet: 94.124.104.0/21(0.14), asn: 42000(3.62), country: CZ(0.09)]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.99)[0.993,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; NEURAL_SPAM_LONG(1.00)[0.997,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=T8al=5T=quip.cz=000.fbsd@elsa.codelab.cz]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 21:09:08 -0000 Artem Kuchin wrote on 2020/04/03 21:48: > 03.04.2020 22:44, Miroslav Lachman пишет: >> Artem Kuchin wrote on 2020/04/03 21:23: >> >> gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 >> gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1 >> > > But why i cannot then just install it into mirror/boot? > > It does now even show it > > # gpart show mirror/boot > gpart: No such geom: mirror/boot. > > # gpart show boot > gpart: No such geom: boot. gpart show does not work for unconfigured devices. If you put brand new empty HDD in to the machine (for example ada2) then gpart show ada2 will not work too until you create some scheme on it with "gpart create". Boot partition does not use scheme / filesystem. Miroslav Lachman From owner-freebsd-fs@freebsd.org Fri Apr 3 21:42:43 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A408C272D51 for ; Fri, 3 Apr 2020 21:42:43 +0000 (UTC) (envelope-from bob@eager.cx) Received: from kipling.tavi.co.uk (kipling.tavi.co.uk [81.187.145.130]) by mx1.freebsd.org (Postfix) with ESMTP id 48vD1B5bkmz4FWh for ; Fri, 3 Apr 2020 21:42:22 +0000 (UTC) (envelope-from bob@eager.cx) Received: from kipling.tavi.co.uk (localhost [127.0.0.1]) by kipling.tavi.co.uk (Postfix) with ESMTP id 77C6A3BEFC for ; Fri, 3 Apr 2020 22:42:17 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=eager.cx; h=date:from:to :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=selector1; bh=OtYCe83 jNt03On7FPZLRGhti7RI=; b=SgqIuHsL5bcMFZRZ3BrmUQGA9mww337VID/ORzT o5doBGP9NmMc4FZmkPwPEN7YtJBbw9Pfv9f7DmHsWYTNxNleKAOZgMFPzIqlYDmR NOhzwYEBlkG+u99twKlTobeoiIlEC6bNUeL1HVnkI4ACGumBw+jchcqktUJva6ZP I/Wk= Received: from raksha.tavi.co.uk (raksha.tavi.co.uk [81.187.145.139]) (Authenticated sender: rde@tavi.co.uk) by kipling.tavi.co.uk (Postfix) with ESMTPA id 3A78A3BA2A for ; Fri, 3 Apr 2020 22:42:17 +0100 (BST) Date: Fri, 3 Apr 2020 22:42:17 +0100 From: Bob Eager To: freebsd-fs@freebsd.org Subject: Re: gpart bootcode Operation not permitted Message-ID: <20200403224217.419b0342@raksha.tavi.co.uk> In-Reply-To: <1605a04a-ad2c-5b2c-445f-fb8ccd7211d6@artem.ru> References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> <48ba23fa-13dc-4a74-579c-2028479f302a@quip.cz> <1605a04a-ad2c-5b2c-445f-fb8ccd7211d6@artem.ru> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; amd64-portbld-freebsd12.0) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAMFBMVEUwXjFLc0vD0cS7y7zw9PDZ4tkWSRaVrZZ+m39qi2tXfVj////7+/utwK4IPggAOAAJUUA7AAABKklEQVQ4jWPYjQMwDFYJp0NKEKCNJmEf9h8CsimXiL2e33s3/e7F7K2Cs3f3dCMkQkMKj4YuCY3K3iR+e7fMaiSjvkX0/5cFGrWpe2uLzOpaExUVqMS/8PX/Re5ey960OLBTZpFA8+IlSBKPQ92zNyUUBsosN58uIY0k8f+/ONCoYytkVuhWzVwNkYiYbqk5M3NmOVBi41YZ8RsGF7shEtFb5KJ3r969CyixM7OTPeFUxG2IxLO8/9/SvqXlc+/x3h295YzLlj2nIRJQj//nRvc5TEIal8RsXBLVuCQwIgoq/u80DomP6HEOk/iOS+IJLonZOCT+ReOQ+Lkbh0QKLonbOCR+7MYhsRqHBJrVcIl/1TgklqKLQyQ+tGKIgyQOqXpjig94diZRAgAXmDX6jyWafAAAAABJRU5ErkJggg====== MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48vD1B5bkmz4FWh X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=eager.cx header.s=selector1 header.b=SgqIuHsL; dmarc=pass (policy=none) header.from=eager.cx; spf=pass (mx1.freebsd.org: domain of bob@eager.cx designates 81.187.145.130 as permitted sender) smtp.mailfrom=bob@eager.cx X-Spamd-Result: default: False [-6.52 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[eager.cx:s=selector1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:kipling.tavi.co.uk]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[eager.cx:+]; DMARC_POLICY_ALLOW(-0.50)[eager.cx,none]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-3.62)[ip: (-9.57), ipnet: 81.187.0.0/16(-4.74), asn: 20712(-3.70), country: GB(-0.07)]; ASN(0.00)[asn:20712, ipnet:81.187.0.0/16, country:GB] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 21:42:43 -0000 On Fri, 3 Apr 2020 22:23:44 +0300 Artem Kuchin wrote: > 03.04.2020 17:42, Miroslav Lachman ?????: > > > > I think you can use "gmirror stop" on boot partition, then "gmirror > > clear" and then update both individual boot partitions by gpart. > > > > > > I am afraid to do it. Gmirror stops mirroring, okay. What what > gmirror clear does? What metadata is cleared and what it is used for? > > How to restart mirroring after that and how to make sure that mirror > is 100% complete? > > Is it possible to exclude only boot partition from mirroring? As I said before, just boot from alternate media (CD, USB stick). The mirroring won't start. Do the boot code, then reboot normally. -- Bob