Date: Sat, 2 Apr 2022 00:39:29 +0200 From: =?UTF-8?Q?=c5=81ukasz_Moska=c5=82a?= <lm@lukaszmoskala.pl> To: Mark Millard <marklmi@yahoo.com> Cc: freebsd-arm@freebsd.org Subject: Re: Booting rock64 from USB SSD Message-ID: <767ee719-287f-323d-ef18-4da3ef464d6c@lukaszmoskala.pl> In-Reply-To: <83A1232D-38B7-46E1-8991-043AAC36CE42@yahoo.com> References: <18284b32-6d1d-864a-63a9-21b5fe72deb1@lukaszmoskala.pl> <83A1232D-38B7-46E1-8991-043AAC36CE42@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
W dniu 1.04.2022 o 22:52, Mark Millard pisze: > On 2022-Apr-1, at 06:51, Łukasz Moskała <lm@lukaszmoskala.pl> wrote: > >> Hi everyone, >> >> I want to boot my rock64 from SSD (because all sd cards I could find are crap). >> >> I followed those instructions to flash u-boot to SPI: https://github.com/ayufan-rock64/linux-build/blob/master/recipes/flash-spi.md >> >> I was then able to boot FreeBSD from SSD (dd FreeBSD-13.0-RELEASE-arm64-aarch64-ROCK64.img to SSD, just like it would be to SD card), but ethernet wasn't working. Actually, it could receive packets but no outgoing packets were sent. Not even ARP packets - switch didn't detect any device on that port, but link was UP >> >> I assumed it was something to do with u-boot, so I erased u-boot from flash, then DD FreeBSD-13.0-RELEASE-arm64-aarch64-ROCK64.img to SD card, and it worked without problems. >> >> Do I understand correctly that this means that FreeBSD on rock64 uses custom u-boot? Can I somehow flash it to SPI? >> >> Possible workaround that I can think of would be to put /boot on sd card, install u-boot to sd card, then put / on SSD. > > My memory is that USB2 booting could be more normal, in that the > U-Boot from ports could access USB2 storage but not USB3 storage, > at least when I set this up long ago. I wanted USB3 use, thus the > below details. > > I have U-Boot on the /media/ device below. As I remember, it could not > deal with the USB3 port, which is the one I wanted to use. The > FreeBSD kernel was the first stage that could deal with the USB3 port > back when I set up the Rock64 that I have access to. > > I've not had to change the basic structure since I set it up. There > could be aspects that could be different these days and I'd not know. > Think of any comments as potentially being old information. > > Note: mmcsd0 here is actually an eMMC device instead of the microsd > card slot contents, leaving the microsd card slot free for other uses. > But I'd used a microsd card this way before using the eMMC device > (as I remember anyway). > > # gpart show -p > => 63 244277185 mmcsd0 MBR (116G) > 63 32705 - free - (16M) > 32768 102312 mmcsd0s1 fat32lba [active] (50M) > 135080 28760 - free - (14M) > 163840 241172480 mmcsd0s2 freebsd (115G) > 241336320 2940928 - free - (1.4G) > > => 0 241172480 mmcsd0s2 BSD (115G) > 0 230686720 mmcsd0s2a freebsd-ufs (110G) > 230686720 10485760 - free - (5.0G) > > => 40 1953525088 da0 GPT (932G) > 40 532480 da0p1 efi (260M) > 532520 2008 - free - (1.0M) > 534528 7340032 da0p2 freebsd-swap (3.5G) > 7874560 1048576 - free - (512M) > 8923136 23068672 da0p3 freebsd-swap (11G) > 31991808 2097152 - free - (1.0G) > 34088960 33554432 da0p4 freebsd-swap (16G) > 67643392 1740636160 da0p5 freebsd-ufs (830G) > 1808279552 4194304 da0p6 freebsd-swap (2.0G) > 1812473856 141051272 - free - (67G) > > For reference, from "gpart show -pl" : > > 67643392 1740636160 da0p5 Rock64root (830G) > > (The USB drive can boot other systems as well, with > widely varying amounts of RAM. Thus the efi partition > and the odd set of freebsd-swap partitions.) > > So, /media/ below is mmcsd0s1's fat32lba and /mnt/ is > mmcsd0s2a's freebsd-ufs. da0 is the USB3 SSD media, > which I do not give other details of here. (I manually > mounted these for this note.) > > # ls -Tld /media/*/*/* /mnt/* /mnt/etc/* > -r-xr-xr-x 1 root wheel 1243772 Jan 28 12:33:00 2022 /media/EFI/BOOT/bootaa64.efi > -r-xr-xr-x 1 root wheel 50618 Jan 28 12:32:28 2022 /media/dtb/rockchip/rk3328-rock64.dtb > -r--r--r-- 1 root wheel 6170 Feb 1 04:48:34 2020 /mnt/COPYRIGHT > drwxr-xr-x 23 root wheel 1536 Jan 28 15:26:41 2022 /mnt/boot > drwxr-xr-x 2 root wheel 512 Apr 26 14:39:22 2020 /mnt/etc > -rw-r--r-- 1 root wheel 37 Dec 31 16:00:18 2009 /mnt/etc/hostid > drwx------ 2 root wheel 33280 Nov 27 09:46:08 2019 /mnt/lost+found > > # ls -Tld /mnt/boot/dtb/overlays/rk3328-* > -r--r--r-- 1 root wheel 238 Jan 28 12:32:28 2022 /mnt/boot/dtb/overlays/rk3328-analog-sound.dtbo > -r--r--r-- 1 root wheel 1281 Jan 28 12:32:28 2022 /mnt/boot/dtb/overlays/rk3328-dwc3.dtbo > > (I make no use of rk3328-analog-sound.dtbo .) > > /mnt/boot/loader.conf has, in part, > > # A msdosfs /MNTPNT/dtb/rockchip/rk3328-rock64.dtb > # copy of the ufs /boot/dtb/rockchip/rk3328-rock64.dtb > # uses the intended DTB in u-boot and the kernel --and > # avoids needing to tell the kernel where to find a > # copy . . . > #rk3328_rock64_load="YES" > #rk3328_rock64_type="dtb" > #rk3328_rock64_name="/boot/dtb/rockchip/rk3328-rock64.dtb" > # > # rk3328 USB3-related: > fdt_overlays="rk3328-dwc3.dtbo" > # ucom is not automatically being loaded when umodem is loaded at boot. > ucom_load="YES" > umodem_load="YES" > # > vfs.root.mountfrom="ufs:/dev/gpt/Rock64root" > kern.cam.boot_delay=10000 > vfs.mountroot.timeout=10 > vfs.root_mount_always_wait=1 > > === > Mark Millard > marklmi at yahoo.com > Hi Mark, With some idea from what you did, I managed to do this: =================== BEGIN COMMAND LIST ============================ sudo pkg install u-boot-rock64 truncate -s 512M rock64-sd.img truncate -s 5G rock64-disk.img sudo mdconfig -a -t vnode -f FreeBSD-13.0-RELEASE-arm64-aarch64-ROCK64.img #returned md0 sudo mdconfig -a -t vnode -f rock64-sd.img #returned md1 sudo mdconfig -a -t vnode -f rock64-disk.img #returned md2 sudo gpart create -s gpt md1 sudo gpart add -t efi -b 32768 -s 50M -i 1 md1 sudo gpart add -t freebsd-ufs -i 2 md1 sudo newfs -O 2 -L bootfs /dev/md1p2 sudo dd if=/dev/md0p1 of=/dev/md1p1 bs=1M mkdir mp_old mp_new sudo mount -o ro /dev/md0p2 mp_old sudo mount -o rw /dev/md1p2 mp_new sudo rsync -rxav mp_old/boot/ mp_new/boot/ echo 'fdt_overlays="rk3328-dwc3.dtbo"' | sudo tee -a mp_new/boot/loader.conf echo 'vfs.root.mountfrom="ufs:/dev/ufs/rootfs"' | sudo tee -a mp_new/boot/loader.conf echo 'kern.cam.boot_delay=10000' | sudo tee -a mp_new/boot/loader.conf echo 'vfs.mountroot.timeout=10' | sudo tee -a mp_new/boot/loader.conf echo 'vfs.root_mount_always_wait=1' | sudo tee -a mp_new/boot/loader.conf sudo gpart create -s gpt md2 sudo gpart add -t freebsd-ufs -a 4096 -i 1 md2 sudo newfs -O 2 -t -L rootfs /dev/md2p1 mkdir mp_newroot sudo mount -o rw /dev/md2p1 mp_newroot sudo rsync -rxav --exclude=boot mp_old/ mp_newroot/ sudo rm mp_newroot/etc/fstab sudo mkdir mp_newroot/media/sd echo '/dev/ufs/rootfs / ufs rw 1 1' | sudo tee -a mp_newroot/etc/fstab echo '/dev/ufs/bootfs /media/sd ufs rw,noatime 1 2' | sudo tee -a mp_newroot/etc/fstab echo '/dev/msdosfs/EFI /media/sd/boot/efi msdosfs rw,noatime 0 0' | sudo tee -a mp_newroot/etc/fstab echo 'tmpfs /tmp tmpfs rw,mode=1777,size=50m 0 0' | sudo tee -a mp_newroot/etc/fstab cd mp_newroot sudo ln -sf media/sd/boot boot cd .. sudo umount mp_newroot sudo umount mp_new sudo umount mp_old sudo dd if=/usr/local/share/u-boot/u-boot-rock64/idbloader.img of=/dev/md1 seek=64 bs=512 conv=sync sudo dd if=/usr/local/share/u-boot/u-boot-rock64/u-boot.itb of=/dev/md1 seek=16384 bs=512 conv=sync sudo mdconfig -d -u 2 sudo mdconfig -d -u 1 sudo mdconfig -d -u 0 =================== END COMMAND LIST ============================ (my apologies if some lines wrap improperly) then, DD rock64-sd.img to sdcard and rock64-disk.img to disk. It would be possible to do that directly on disk and sdcard itself, but in my case, machine on which I ran these commands doesn't have sd card reader. Anyway, it successfully booted with SSD plugged into USB3 port, despite those messages: Root mount waiting for: usbus0 usbus2 usbus3 usbus4 CAM usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored) usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR after which da0 was detected and boot continued. Network is working, and everything seems okay (uptime is 6 minutes as of moment I'm writing this). I'm using official pine64 usb-sata adapter, but I suppose it doesn't really matter. I don't have any other adapters to check. I copied 2.5GiB file to it and it seems stable. I got a lot of "devmatch: Can't read linker hints file." That was fixed by kldxref -R /media/sd/boot root@generic:~ # df -h Filesystem Size Used Avail Capacity Mounted on /dev/ufs/rootfs 108G 4.2G 95G 4% / devfs 1.0K 1.0K 0B 100% /dev /dev/ufs/bootfs 432M 82M 316M 21% /media/sd /dev/msdosfs/EFI 50M 2.7M 47M 5% /media/sd/boot/efi tmpfs 50M 4.0K 50M 0% /tmp root@generic:~ # mount /dev/ufs/rootfs on / (ufs, local) devfs on /dev (devfs) /dev/ufs/bootfs on /media/sd (ufs, local, noatime) /dev/msdosfs/EFI on /media/sd/boot/efi (msdosfs, local, noatime) tmpfs on /tmp (tmpfs, local) root@generic:~ # gpart show -p => 40 1048496 mmcsd0 GPT (7.4G) [CORRUPT] 40 32728 - free - (16M) 32768 102400 mmcsd0p1 efi (50M) 135168 913368 mmcsd0p2 freebsd-ufs (446M) => 40 234441568 da0 GPT (112G) 40 4056 - free - (2.0M) 4096 234437512 da0p1 freebsd-ufs (112G) Thanks for your help, and I hope that whatever I made here will be usefull to someone. -- Łukasz Moskała
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?767ee719-287f-323d-ef18-4da3ef464d6c>