Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 10 Feb 2021 16:37:18 +0000
From:      Mike Clarke <jmc-freebsd2@milibyte.co.uk>
To:        freebsd-questions@freebsd.org
Subject:   Re: Bootenv containing several filesystems
Message-ID:  <2068084.5gTYqTD1LS@curlew>
In-Reply-To: <CADqw_gLg8uG8jXTTFe=ZODwyxDSov6os44_P7No+QOtuwbapQg@mail.gmail.com>
References:  <CADqw_gKG6ovTuN7bZvYy7PCydfCXH4M2fw68YLmLvZhxi-g2xw@mail.gmail.com> <2476830.FrFBg55ix7@curlew> <CADqw_gLg8uG8jXTTFe=ZODwyxDSov6os44_P7No+QOtuwbapQg@mail.gmail.com>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
On Tuesday, 9 February 2021 19:10:34 GMT Michael Schuster wrote:
> On Tue, Feb 9, 2021 at 5:30 PM Mike Clarke <jmc-freebsd2@milibyte.co.uk> 
> wrote:
> > On Tuesday, 9 February 2021 09:53:27 GMT Matthew Seaman wrote:
> > > There's an important difference between beadm and bectl which seems
> > > relevant here.  beadm defaults to accepting a tree of ZFSes as a boot
> > > environment, whereas bectl only applies to the ZFS at the top level of
> > > the boot environment unless you use the -r flag.
> > 
> > That probably accounts for a discrepancy that I always see between beadm
> > list
> > and bectl list for my BE which has child datasets:
> > 
> > curlew:/tmp% beadm list
> > BE        Active Mountpoint  Space Created
> > fbsd12.1y -      -            1.9G 2020-12-20 20:52
> > fbsd12.2a -      -          133.0M 2020-12-24 11:20
> > fbsd12.2b -      -           18.5M 2021-01-02 09:50
> > fbsd12.2c -      -           11.7M 2021-01-12 09:55
> > fbsd12.2d NR     /           39.4G 2021-02-05 10:46
> > curlew:/tmp% bectl list
> > BE        Active Mountpoint Space Created
> > fbsd12.1y -      -          61.3M 2020-12-20 20:52
> > fbsd12.2a -      -          6.97M 2020-12-24 11:20
> > fbsd12.2b -      -          2.80M 2021-01-02 09:50
> > fbsd12.2c -      -          5.91M 2021-01-12 09:55
> > fbsd12.2d NR     /          39.5G 2021-02-05 10:46
> 
> strangely, I don't see such a difference:
> 
> bectl:
> BE_20210205_121021_CURRENT14   -      -          81.6M 2021-02-05 12:10
> BE_20210205_181224_CURRENT14   -      -          49.9M 2021-02-05 18:12
> BE_20210206_102540_CURRENT14   -      -          153M  2021-02-06 10:25
> BE_20210206_175312_CURRENT14   NR     /          30.9G 2021-02-06 17:53
> BE_20210208_204901_CURRENT_14  -      -          31.9M 2021-02-08 20:49
> 
> beadm:
> BE_20210205_121021_CURRENT14   -      -           81.6M 2021-02-05 12:10
> BE_20210205_181224_CURRENT14   -      -           49.9M 2021-02-05 18:12
> BE_20210206_102540_CURRENT14   -      -          152.3M 2021-02-06 10:25
> BE_20210206_175312_CURRENT14   NR     /           30.9G 2021-02-06 17:53
> BE_20210208_204901_CURRENT_14  -      -           31.9M 2021-02-08 20:49
> 
> as you can see, the difference is negligable ...
> 
> is there some zpool or zfs property I need to set so that be(ctl|adm) (with
> appropriate options if need be) will create a recursive boot environment?

A possible explanation for the lack of any discrepancy with your system may 
be that there might have been relatively small changes between your BEs which 
were all created within a few days of each other. In my case there was a 
significant change between fbsd12.1y and fbsd12.2a when I upgraded from 12.1 
RELEASE to 12,2 RELEASE and there would have been a big change in the child 
dataset for /usr which I suspect bectl is not including in the total.

All my BEs were created with beadm which is always recursive.

Most of the options for the datasets in my BEs are default ones apart from 
noatime and compression on all datasets and noexec and  nosuid for a few 
datasets.

Here's the filesystems for a typical BE:

NAME                            USED  AVAIL  REFER  MOUNTPOINT
ssd/ROOT/fbsd12.2d             39.5G  75.6G  1.54G  /
ssd/ROOT/fbsd12.2d/usr         31.2G  75.6G  10.1G  /usr
ssd/ROOT/fbsd12.2d/usr/ports   8.45G  75.6G  6.84G  /usr/ports
ssd/ROOT/fbsd12.2d/usr/src     3.52G  75.6G  1.46G  /usr/src
ssd/ROOT/fbsd12.2d/var         6.50G  75.6G  1.63G  /var
ssd/ROOT/fbsd12.2d/var/db      3.13G  75.6G  1.93G  /var/db
ssd/ROOT/fbsd12.2d/var/db/pkg   776M  75.6G  83.6M  /var/db/pkg
ssd/ROOT/fbsd12.2d/var/empty    104K  75.6G    96K  /var/empty
ssd/ROOT/fbsd12.2d/var/tmp     1.03G  75.6G   128K  /var/tmp

Some datasets are created outside the BE to preserve data when switching 
between BEs.

NAME                      USED  AVAIL  REFER  MOUNTPOINT
home/DATA/var            2.87G  65.5G    31K  none
home/DATA/var/cache       867M  65.5G    31K  none
home/DATA/var/cache/pkg   867M  65.5G   866M  /var/cache/pkg
home/DATA/var/db         1.76G  65.5G    31K  none
home/DATA/var/db/mysql   1.76G  65.5G  1.01G  /var/db/mysql
home/DATA/var/log         275M   749M   201M  /var/log
home/DATA/var/mail         52K  65.5G    32K  /var/mail

I've been experimenting by switching between BEs created with beadm and bectl 
in an attempt to identify the reason for the discrepancies but I think that's 
just muddied the water instead of throwing light on things.

After creating two BEs with 'bectl -r' and booting into the second one after 
activating it with bectl I would have expected the space for the active BE, 
bectl-test2, to be about 37.7G but this space is still attributed to the 
previously active BE, fbsd12.2d.

The BEs are no longer displayed in date order by bectl, probably due to the 
difference in date format for the snapshots used by beadm and bectl.

curlew:/home/mike% bectl list
BE          Active Mountpoint Space Created
bectl-test  -      -          1.93M 2021-02-10 09:54
bectl-test2 NR     /          1.82G 2021-02-10 10:10
fbsd12.1y   -      -          61.3M 2020-12-20 20:52
fbsd12.2a   -      -          6.97M 2020-12-24 11:20
fbsd12.2b   -      -          2.85M 2021-01-02 09:50
fbsd12.2c   -      -          5.91M 2021-01-12 09:55
fbsd12.2d   -      -          37.7G 2021-02-05 10:46
curlew:/home/mike% beadm list
BE          Active Mountpoint  Space Created
fbsd12.1y   -      -            1.9G 2020-12-20 20:52
fbsd12.2a   -      -          189.7M 2020-12-24 11:20
fbsd12.2b   -      -           84.5M 2021-01-02 09:50
fbsd12.2c   -      -           11.7M 2021-01-12 09:55
fbsd12.2d   -      -           37.7G 2021-02-05 10:46
bectl-test  -      -            2.2M 2021-02-10 09:54
bectl-test2 NR     /            1.8G 2021-02-10 10:10

I then rebooted after using beadm to create and activate beadm-test. Both 
bectl and beadm then showed 39.5G for the space attributed to the active BE 
beadm-test.

curlew:/home/mike% bectl list
BE          Active Mountpoint Space Created
beadm-test  NR     /          39.5G 2021-02-10 10:20
bectl-test  -      -          924K  2021-02-10 09:54
bectl-test2 -      -          1.07M 2021-02-10 10:10
fbsd12.1y   -      -          61.3M 2020-12-20 20:52
fbsd12.2a   -      -          6.97M 2020-12-24 11:20
fbsd12.2b   -      -          2.85M 2021-01-02 09:50
fbsd12.2c   -      -          5.91M 2021-01-12 09:55
fbsd12.2d   -      -          1.18M 2021-02-05 10:46
curlew:/home/mike% beadm list
BE          Active Mountpoint  Space Created
fbsd12.1y   -      -            1.9G 2020-12-20 20:52
fbsd12.2a   -      -          189.7M 2020-12-24 11:20
fbsd12.2b   -      -           84.5M 2021-01-02 09:50
fbsd12.2c   -      -           11.7M 2021-01-12 09:55
fbsd12.2d   -      -            1.2M 2021-02-05 10:46
bectl-test  -      -            1.8M 2021-02-10 09:54
bectl-test2 -      -            2.1M 2021-02-10 10:10
beadm-test  NR     /           39.5G 2021-02-10 10:20

The next step was to use bectl to reactivate bectl-test2 and reboot which 
then left 37.7G attributed to the inactive beadm-test.

curlew:/home/mike% bectl list
BE          Active Mountpoint Space Created
beadm-test  -      -          37.7G 2021-02-10 10:20
bectl-test  -      -          924K  2021-02-10 09:54
bectl-test2 NR     /          1.82G 2021-02-10 10:10
fbsd12.1y   -      -          61.3M 2020-12-20 20:52
fbsd12.2a   -      -          6.97M 2020-12-24 11:20
fbsd12.2b   -      -          2.85M 2021-01-02 09:50
fbsd12.2c   -      -          5.91M 2021-01-12 09:55
fbsd12.2d   -      -          1.18M 2021-02-05 10:46
curlew:/home/mike% beadm list
BE          Active Mountpoint  Space Created
fbsd12.1y   -      -            1.9G 2020-12-20 20:52
fbsd12.2a   -      -          189.7M 2020-12-24 11:20
fbsd12.2b   -      -           84.5M 2021-01-02 09:50
fbsd12.2c   -      -           11.7M 2021-01-12 09:55
fbsd12.2d   -      -            1.2M 2021-02-05 10:46
bectl-test  -      -            1.8M 2021-02-10 09:54
bectl-test2 NR     /            1.8G 2021-02-10 10:10
beadm-test  -      -           37.7G 2021-02-10 10:20

I then used beadm to reactivate bectl-test2 by using 'beadm activate beadm-
test' followed by 'beadm activate bectl-test2' and rebooted with the result 
that both bectl and beadm then attributed 39.5G to the active BE, bectl-
test2. 

curlew:/home/mike% bectl list
BE          Active Mountpoint Space Created
beadm-test  -      -          1.45M 2021-02-10 10:20
bectl-test  -      -          924K  2021-02-10 09:54
bectl-test2 NR     /          39.5G 2021-02-10 10:10
fbsd12.1y   -      -          61.3M 2020-12-20 20:52
fbsd12.2a   -      -          6.97M 2020-12-24 11:20
fbsd12.2b   -      -          2.85M 2021-01-02 09:50
fbsd12.2c   -      -          5.91M 2021-01-12 09:55
fbsd12.2d   -      -          1.18M 2021-02-05 10:46
curlew:/home/mike% beadm list
BE          Active Mountpoint  Space Created
fbsd12.1y   -      -            1.9G 2020-12-20 20:52
fbsd12.2a   -      -          189.7M 2020-12-24 11:20
fbsd12.2b   -      -           84.5M 2021-01-02 09:50
fbsd12.2c   -      -           11.7M 2021-01-12 09:55
fbsd12.2d   -      -            1.2M 2021-02-05 10:46
bectl-test  -      -            1.8M 2021-02-10 09:54
bectl-test2 NR     /           39.5G 2021-02-10 10:10
beadm-test  -      -            2.5M 2021-02-10 10:20

Surprisingly using bectl to repeat the process with 'bectl activate beadm-
test' followed by 'bectl activate bectl-test2 resulted in 39.5G still being 
attributed to bectl-test2.

Throughout all these tests bectl consistently showed the size of fbsd12.1y to 
be 61.3M and beadm showed 1.9G.

So  appears to be a significant difference in the way beadm and bectl account 
for the space used by BEs with child datasets. Perhaps this only shows when 
the BEs are not all created with the same program. I'll probably start using 
bectl to create the next few BEs and see what happens to the space values 
after the older beadm BEs get destroyed.

Before long I'll be deleting fbsd12.1y and it will be interesting to see 
whether 1.9G or 61.3M get freed up if I use bectl to destroy it.

-- 
Mike Clarke





Want to link to this message? Use this URL: <http://docs.FreeBSD.org/cgi/mid.cgi?2068084.5gTYqTD1LS>