From owner-freebsd-arm@freebsd.org Thu May 6 20:42:58 2021 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E263162DACB for ; Thu, 6 May 2021 20:42:58 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fblrx3c1Sz53kw for ; Thu, 6 May 2021 20:42:57 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1620333775; bh=feMhn0EeqBcnX5TTpHcS2sWGiTeyIr4dpwQByoHWLAi=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=K4/yH29pstILmRy6mL7mEFcFO7hgkx2bCi2/Ezz8RzRLToC/07TIDlY1hx31paEkgnt/tY0bMhTOwTSq5PbU+OEgAEhXjsK4J7nFKPuivHmQK/cTK2A32VXzjfv36QcRAfJkxbDHxl/g7gFqM1Qj1/2wk+E/eLFNjkCZsTD/CSc1YWZxJhPz48d6wJF3MtKx1ggrliqblhjoeiN3SO4rGQmZfbTSF+ww0ZR+VjC17R+A/MZN37NcPTK6cLMpwRACaZWDAkWtBGkCslSoAZZIZnrbmxv1iTD2zv/F/I+u3O8wyVRC7qQhTkHByeeLjQ8P3YpeIoKqPuDBAoYRCTqj0Q== X-YMail-OSG: r.IGfCQVM1k7w3.zhKKqMb_VzM1X7K7nBMcUmv0_5IhKKKK2r5iGf8K5OOEonf4 wWsTToDeWhRXK08XUGdGMjbTPaLKFuk6.0p2VlLCvWspjF4d9Y20pKm_yH67aKCfIeABvjjVGS38 hLL62o4BJI.R8Ov1rdQMIq3gha4rxcCN80PRzohLzHoZ6zOlz66q_oR3siDv7zYjjy3TqDbFWQm7 tf6g2HmNUuurUh6IOKLBlzsaJtSGqPvcUNqCoX5zSNjpFHFrA9_fS5jpCk8P_b9yb9FrxXS7CG2R o5esT9v9zJPIYXUoxql5uZRIHX4FbYBZKQKAjx8Yg560W7hNJstxmYn__HdUmR4jBlvLE0H4hLp. 3QdkdnGrBOULUww_omBgjdqjOAfQBR6XiZHZu6H5BRqJLT3AXZy4HkpkKFFKgAkw6iskP9HRt_pf LTQoC6ZKCAGAknQwck9yRtoqswMTS7Ft4kr84vWw_zCRNzSd3VUo1csSrlIYJ5srkqKOyBe_hbsV 8RtwvdA.Y_arqmOXcXZJBmvveVuJqWCyPtt2kjuHhq.7VzFu938fohvy792Utn7zFjBXa.0Lr8vw 6Yl2eMrVOxtfvnprB5X_v1NStIk.0w_uhhK3qnA.gYReXdjT_pQStMygoLfyEWQfzZBK2IeOpePt hKhDqRS9u6MdWWEncDY7nmgc2Mpp9wGR7WnKa6DJPmQzJtCKozfaIDYxDqWb2NxLMzyDTOuStja0 FBodVuXEzuwdmzglWscEXEeranyECtTWvn3RXl2sH6gW4N6_B.S4h2reANmGqwtwIL31bUWTzBN7 rJ5Q2fX6zqyoBazswSIEgNgoIDwMsNJxmvlhAHrYlThicX.3.P8waVHwMB6XqXjPIb_rhtB8PZtt UUGvR3QJ0VXHNAH5uJ5APlKLaB4OQs23tEqv.PndiLge6sSAm2J_lLJRUctzq1Pdts4DeivXl5zP HYwwHcAuqOBAKTlZiRRQbgzMta4lkrIQQQToll6r.dUOyqz9yXo7bCvnzqIJ_KGmqQxV.5b_cw7S pMRYpm67I3FWc3q0ws4bnB4dP0AuUkvgzElHwlh10q_A7cJAS.K_eZY3Kg2mrOydHxSvdxJYYjbx mkrAYehpxlm3vGhFZczrhLb210qS4cd3uPKx8mFqTuyOZoOKlKuagIME9g27tCfODnK7Y8D_FvnB C6MmA0kT7f7FYOu8M99JJFx_vbqPXJHM_2CJWLlOOwXBtzCB5vLSEhHrrHSxva2x3qUnlLwpBvuP .EZN.QbPxfUbqMp2IGhdgMO8YF9lrq4tJyRKVOArJOvtGjvyCSoeq34FU0052Uy4RNgg9Vop6UJS VBWlG8UtIk3s5l_bPdFomuX6KNhAVngWXya4iDEInPtfH.cSb8omew06TrPnNvwhhnSXccr76xAZ 0SDSZMkSzkNFfSnIB2JAXFr2I7NZAjSK8916sXD2K_t3wsPwi9.TA2BrCsT9sFIY8Nvaq9.SSCJW UT8ycgN9QBaWx83oeSLDrPqUHgPYby1v3XOIMfWfLr7z3qzqv9gzHmonrZjZxsi9LW0w.DMG8wYc H7UonVQ8HOKuD2smK9NmyHndRoJQvSZLQijzY75Zvy2wxQlLxDAzHJZKe3jpzrbJ9jbhbN4VVoD_ 0pqwTFXkGWuTgPmM1IcOL9vA4FNpQVUSqtkX7iqekqK7n62Zw.BPv0hAEaaclUNMy77C5xrIIP6f _VEV7JyW2zjWRiLGi8hX9mPAosO.Epyi2djJzJhGxGfM4wUZlVXq7fc8CF87yDoEMq0nFnG0ppD3 Tjup_xu8hEgYZuYIdVo.ESRIX18_XOwWR7gviskZgyDnFVtzzeVMMf_Ea8J2vZBUhUJo8rn67Kws QNm8nr.Lo3xNhT0jL2d6nvKJmHoViXQD8Wlzw2lhspf_k1_ulhG1XDj5BnVqo8tuRcytJtlv0iJs 6mOgf57vjdIBpjYgAGX2InWJl3f9nkCqXjChVX1A4Za33YcmslHJ1ZGUDGB1c8PqFeegetKlEUa8 QDfSU2G88h9xb1TZmgp6Nflk31AGTlo6Fea8N7MpFU4WxASZBEQx_fDgpDD1e1W2MhsE_rUcg7wW lcu3RWKR2HKbnx0VSPc4G8aTwgQtarV8y4jnEAD8A84ABvGGeOkCPFZFk7e73DkIhrReqltS8jzs kwrxsQCQBYoLgViXemViylPDVWyyNSmAuaiu3l2wNrIq6Yw38SpGtYtllUyzqgdLXhM.ME3cxMkB n.DigdXU8VoCc7dSg26rsboIp3qNn5as4Ih8UkVcbPy4hpuCsvRvBK_YdF6Th8HQdpBRcQ.It4gb ilJMrICwFPMPBB4EL9BAwfiWqXHNcHNqu9ErxMMZFsg4yns7u9fJx2AsiCAUaebQmZzV8Bu_wR8x LiSKEtCvMyWr9AJfPvGawx7hxb0g9OSGorvOdDBdNVLYP8COsL8gm.9zUilCHNkzXKKuw7Y9xSVB fmTTVU7UJPMuhWIOEHkIL3Tw.BES68aCeYKyBz3taFlRl4psodqx_AOqtpOekeOlNqCaXu3XOl5_ O03PXqiN0EQScu2tUT04jP6TZtE9gbGnQCguwZhO5G900l3dwbE1df6.mHfw.KYVuooYgthN96ds bOzonaD5EVBZc6UW_je1Lv1jUFwDdEycWRvcAQmrwXmTs X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Thu, 6 May 2021 20:42:55 +0000 Received: by kubenode512.mail-prod1.omega.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 8471ef62991be17961ca2ad8cae418a6; Thu, 06 May 2021 20:42:53 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.80.0.2.43\)) Subject: Re: rpi4 zfs-on-root boot-to-usb3 [Example sequence that lead to booting zfs-on-root under GPT partitioning] From: Mark Millard In-Reply-To: <96F0BF83-D0FB-442F-B8A1-54194A724BD1@yahoo.com> Date: Thu, 6 May 2021 13:42:50 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <96F0BF83-D0FB-442F-B8A1-54194A724BD1@yahoo.com> To: tech-lists X-Mailer: Apple Mail (2.3654.80.0.2.43) X-Rspamd-Queue-Id: 4Fblrx3c1Sz53kw X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.99)[-0.992]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.65.147:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.65.147:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.147:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.147:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Porting FreeBSD to ARM processors List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 May 2021 20:42:58 -0000 On 2021-May-6, at 13:09, Mark Millard wrote: > On 2021-May-6, at 12:12, Mark Millard wrote: >=20 >> On 2021-May-6, at 05:55, Mark Millard wrote: >>=20 >>> On 2021-May-6, at 04:12, tech-lists wrote: >>>=20 >>>> How can zfs-on-root boot-to-usb3 on rpi4 be accomplished? >>>>=20 >>>> I've tried bsdinstall from a mmcsd-booted rpi4 but there seems to = be >>>> problems with it that I can't work around. What's really needed is = an >>>> installer, but these aren't made for arm64.aarch64 rpi4 from what I = can >>>> see (I'm no expert though, it's entirely feasible i've missed >>>> something). >>>>=20 >>>> Maybe one way of doing it would be to have a usb key (as ufs2) for = the >>>> system to boot on, then have /home /usr/obj and other larger dirs = on the >>>> usb3-zfs disk. >>>=20 >>> I used bsdinstall from booting a releng/13.0's release/13.0.0.0 >>> microsd card in a 8 GiBYte RPi4B to produce the: >>> . . . >>=20 >> Various details shown will just be my specific >> choices. (The RPi4B's that I have access to have >> the 2021-Apr-29 default(/critical) EEPROM image.) >>=20 >> Taking notes as I go (and readjusting as I >> progress and figure things out, eliminating >> failing attempts as well) . . . >>=20 >> Booting based on a microsd card with releng/13.0 's >> release/13.0.0 as its basis. The context has a >> working network with internet access. I'll note that setting up the microsd card context to have the correct timezone and time is something I presumed was already in place. But such is not the case for an RPI image from the servers. So I effectively presumed booting with /etc/rc.conf having, say, # ntpd_enable=3D"YES" ntpd_sync_on_start=3D"YES" ntpd_user=3D"root" already added (or a manual time set) and having done a: # tzsetup sequence with appropriate selections. >> # uname -apKU >> FreeBSD generic 13.0-RELEASE FreeBSD 13.0-RELEASE #0 = releng/13.0-n244733-ea31abc261f: Fri Apr 9 06:06:55 UTC 2021 = root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC = arm64 aarch64 1300139 1300139 >>=20 >> Plug in USB3 SSD. Ends up as da0. >>=20 >> # /bin/sh # Just for my familiarity >> # set -o vi >>=20 >> # mkdir -p /usr/freebsd-dist >> # cd /usr/freebsd-dist >> # fetch = http://ftp3.freebsd.org/pub/FreeBSD/releases/arm64/13.0-RELEASE/MANIFEST >> MANIFEST 782 B 6147 = kBps 00s >> # cd ~ >>=20 >> # bsdinstall >>=20 >> Continue with default keymap : Select >>=20 >> Enter hostname as ZFStest : OK >>=20 >> [*] base-dbg >> [*] kernel-dbg >> [ ] ports >> [ ] src >> [*] tests >> then: OK >>=20 >> (Note: I use git for src and ports.) >>=20 >> Main Site : OK >>=20 >> Auto (ZFS) : OK >>=20 >> Pool Name : Select >>=20 >> Enter name for zpool ztstp : OK >>=20 >> Swap Size : Select >>=20 >> Enter swap size 24g : OK >>=20 >> Proceed with Installation : Select >>=20 >> Stripe - No Redundancy : OK >>=20 >> [*] da0 : OK >>=20 >> Last Chance for da0 : YES >>=20 >> Downloads . . . >> Extracts . . . >>=20 >> New Password: . . . >> Retype New Password: . . . >>=20 >> genet0 : OK >>=20 >> configure IPv4 : YES >> configure DHCP : YES >> configure IPv6 : YES >> try SLAAC : YES >> Resolver Configuration : OK >>=20 >> time is UTC? : YES >>=20 >> America : OK >> United States of America : OK >> Pacific : OK >> Does PDT look reasonable? : Yes >> May 2021 6 : Set Date >> 11 07 00 : Set Time >>=20 >> [ ] local_unbound >> [*] sshd >> [ ] moused >> [ ] ntpdate >> [*] ntpd >> [*] powerd >> [*] dumpdev >> Then : OK >>=20 >> No hardening options enabled : OK >>=20 >> Add uses? : Yes >> . . . details omitted . . . >> OK ? yes >> Add another user? no >>=20 >> Handbook : OK >> [*] en : OK >>=20 >> Apply configuration and exit installer : OK >> open a shell : No >=20 > I suggested an inappropriate later stage if one > wants to try the same vintage of rpi-firmware > and u-boot as releng/13.0 itself uses. One > can copy over materials before the shutdown > by getting them from /boot/msdos/ . >=20 > Note that you might not want to copy over > /boot/msdos/EFI/... as the install will > already have such. >=20 > So, something like: >=20 > # mount -onoatime -tmsdosfs /dev/da0p1 /mnt > # cp -aRx /boot/msdos/[LRa-z]* /mnt/ > # umount /mnt >=20 > then do the shutdown and remove the microsd card. >=20 >> # shutdown -p now >=20 > The following is only if one did not copy from > /boot/msdosfs/ or one needs more recent > materials, such as U-Boot for *C0T revision > processors. >=20 >> At this point it still can not boot an RPi4B >> for lack of rpi firmware and U-Boot. >>=20 >> I have such available on other machine based >> on the latest ports instead of quarterly. There >> are RPi4B's in the world that need the more >> modern U-Boot compared to the quarterly that >> releng/13.0 is tied to by default. But you >> likely could install rpi-firmware and >> u-boot-rpi-arm64 on the microsd card and >> then copy over materials from there. >=20 > The above is dumb suggestion unless newer > material is needed. See before the shutdown > above. >=20 > The below is me getting more recent materials > (well, U-Boot) from another context. You might > not do similarly. >=20 >> In my context . . . >>=20 >> # gpart show -p da1 >> =3D> 40 468862048 da1 GPT (224G) >> 40 532480 da1p1 efi (260M) >> 532520 2008 - free - (1.0M) >> 534528 50331648 da1p2 freebsd-swap (24G) >> 50866176 417994752 da1p3 freebsd-zfs (199G) >> 468860928 1160 - free - (580K) >>=20 >> # mount -onoatime -tmsdosfs /dev/da1p1 /mnt >> # cp -aRx /usr/local/share/rpi-firmware/* /mnt/ >> # cp -aRx /mnt/config_arm64.txt /mnt/config.txt=20 >> # cp -aRx /usr/local/share/u-boot/u-boot-rpi-arm64/u-boot.bin /mnt/ >> # umount /mnt >=20 > (Note: It is possible to be more selective in > what to copy over. I did not complicate the > sequence with such handling.) >=20 >=20 > Back to the normal flow on the RPi4B given > appropriate RPi* materials copied over to the > msdos file system . . . >=20 >> Back to the RPi4B, no microsd card but plugging in the >> USB3 SSD and booting and logging in: >>=20 >> Dec 31 16:00:48 ZFStest login[1351]: ROOT LOGIN (root) ON ttyu0 >> FreeBSD 13.0-RELEASE (GENERIC) #0 releng/13.0-n244733-ea31abc261f: = Fri Apr 9 03:54:53 UTC 2021 >>=20 >> # gpart show -p >> =3D> 40 468862048 da0 GPT (224G) >> 40 532480 da0p1 efi (260M) >> 532520 2008 - free - (1.0M) >> 534528 50331648 da0p2 freebsd-swap (24G) >> 50866176 417994752 da0p3 freebsd-zfs (199G) >> 468860928 1160 - free - (580K) >>=20 >> # uname -apKU >> FreeBSD ZFStest 13.0-RELEASE FreeBSD 13.0-RELEASE #0 = releng/13.0-n244733-ea31abc261f: Fri Apr 9 03:54:53 UTC 2021 = root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC = arm64 aarch64 1300139 1300139 >>=20 >> I end up adding to /etc/rc.conf: >>=20 >> # >> ntpd_sync_on_start=3D"YES" >> ntpd_user=3D"root" >>=20 >> The first boot's time will be messed up for >> lack of the ntpd_sync_on_start=3D"YES" . >>=20 >> # shutdown -r now >>=20 >> After login: >>=20 >> # ls -Tld /etc/rc.conf >> -rw-r--r-- 1 root wheel 279 Dec 31 16:12:37 1969 /etc/rc.conf >> # touch /etc/rc.conf >>=20 >> There are other files around with such an odd timestamp. >>=20 >> # zpool list >> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP = HEALTH ALTROOT >> ztstp 199G 1.09G 198G - - 0% 0% 1.00x = ONLINE - >>=20 >> # zfs list >> NAME USED AVAIL REFER MOUNTPOINT >> ztstp 1.09G 192G 96K /ztstp >> ztstp/ROOT 1.08G 192G 96K none >> ztstp/ROOT/default 1.08G 192G 1.08G / >> ztstp/tmp 96K 192G 96K /tmp >> ztstp/usr 416K 192G 96K /usr >> ztstp/usr/home 128K 192G 128K /usr/home >> ztstp/usr/ports 96K 192G 96K /usr/ports >> ztstp/usr/src 96K 192G 96K /usr/src >> ztstp/var 680K 192G 96K /var >> ztstp/var/audit 96K 192G 96K /var/audit >> ztstp/var/crash 96K 192G 96K /var/crash >> ztstp/var/log 200K 192G 200K /var/log >> ztstp/var/mail 96K 192G 96K /var/mail >> ztstp/var/tmp 96K 192G 96K /var/tmp >>=20 >> # more /etc/sysctl.conf=20 >> # $FreeBSD$ >> # >> # This file is read when going to multi-user and its contents piped = thru >> # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for = details. >> # >>=20 >> # Uncomment this to prevent users from seeing information about = processes that >> # are being run under another UID. >> #security.bsd.see_other_uids=3D0 >> vfs.zfs.min_auto_ashift=3D12 >>=20 >> I'll note that in: >>=20 >> # more /boot/efi/config.txt=20 >> [all] >> arm_64bit=3D1 >> dtparam=3Daudio=3Don,i2c_arm=3Don,spi=3Don >> dtoverlay=3Dmmc >> dtoverlay=3Ddisable-bt >> device_tree_address=3D0x4000 >> kernel=3Du-boot.bin >>=20 >> [pi4] >> hdmi_safe=3D1 >> armstub=3Darmstub8-gic.bin >>=20 >> The hdmi)safe=3D1 line restricts the HDMI display >> resolution/scaling. Any of the following >> replacements for that line will avoid that but >> in some contexts one could end up in a "blind >> display" context instead, which is why hdmi_safe >> is enabled by default. >>=20 >> hdmi_safe=3D0 >> or: >> #hdmi_safe=3D1 >> or just delete the line. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)