From owner-freebsd-questions@freebsd.org Mon Oct 1 13:59:05 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B737510BCF5F for ; Mon, 1 Oct 2018 13:59:05 +0000 (UTC) (envelope-from thor@irk.ru) Received: from mail.irk.ru (relay2.dsi.ru [195.206.40.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 418FB77315 for ; Mon, 1 Oct 2018 13:59:04 +0000 (UTC) (envelope-from thor@irk.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=irk.ru; s=dkim; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version: Date:Message-ID:From:References:To:Subject:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=SLRkY5lrLg9TCUT/6P5E1Pbjh66uhnY+xVZIK9UC7Vc=; b=RmDBRwHanwCIdQOa7wdjGZOhHX 0OPrNIvEyM6CZ65XvkeQXi0BMNhecjnbR0tQuRfLUB4FIsPvdIEDZwNKzZkD9Vl+hYdnllGQJR4eY EB4dTB1mpYmNHdsN5uNQowwdSS/i/TrHTuMnkNUccAixVepVY7cFLX7kEidFwg4YHvpQ=; Received: from [94.154.81.100] (helo=[192.168.1.130]) by mail.irk.ru with esmtpa (Exim 4.89 (FreeBSD)) (envelope-from ) id 1g6yXz-0001p5-3V for freebsd-questions@freebsd.org; Mon, 01 Oct 2018 21:47:39 +0800 Subject: Re: Concerns To: freebsd-questions@freebsd.org References: <20180929042332.GA45612@doctor.nl2k.ab.ca> From: thor Message-ID: Date: Mon, 1 Oct 2018 21:58:52 +0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <20180929042332.GA45612@doctor.nl2k.ab.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Oct 2018 13:59:06 -0000 Hello! I haven't understood anything but I have installed an encrypted RAID5 mfi some time ago, and there are some operators in my encryption routine that install the boot device via /boot/loader.conf. Maybe it helps. btw there was a problem that in early boot sequence the geli passphrase could not be entered. The following lines in /boot/loader.conf helped (I don't know why): ukbd_load="YES" kern.vty="sc" $ cat encode.sh #!/bin/sh # Install the encrypted volume # (Valid for FreeBSD-10.2-AMD. Check for other version please) # You should have 3 partotions: # mfid0p1 Boot manager # mfid0p2 A fresh minimal system 1.5GB. Do NOT install src! # mfid0p3 An empty partition spanning the rest of disc on /mnt # # We have NO swap. It's a separate problem. # After you boot you should delete enough files on mfid0p2 to # make a space for the new kernel you would surely compile. umount -f /mnt # Generate the geli key dd if=/dev/random of=/boot/key bs=64 count=1 # Init a geli partition geli init -b -s 4096 -K /boot/key /dev/mfid0p3 || exit # Attach the partition geli attach -k /boot/key /dev/mfid0p3 || exit # Format the partition newfs /dev/mfid0p3.eli # Enable the journal and soft updates # It's the last chance! After this point # we would need a live fs to do it! tunefs -A -n enable /dev/mfid0p3.eli tunefs -A -j enable /dev/mfid0p3.eli tunefs -p /dev/mfid0p3.eli # Only then we can mount the partitions and fill it. mount /dev/mfid0p3.eli /mnt sleep 10 cd / # Copy all plain files cp -p * /mnt # # Every dir in / except boot and mnt dirs # Check and recheck it! # (We would make a IF DIR operator but we are lazy) for i in bin dev etc lib libexec media proc rescue root sbin tmp usr var do cp -Rvp $i /mnt done sleep 10 mkdir /mnt/mnt mkdir /mnt/mnt/boot mount /dev/mfid0p2 /mnt/mnt/boot # Make a new fstab with our new encrypted partitions # mv /mnt/etc/fstab /mnt/etc/fstab.orig cat <>/mnt/etc/fstab # Device    Mountpoint    FStype    Options    Dump    Pass# /dev/mfid0p3.eli    /        ufs    rw,noatime    1    1 /dev/mfid0p2    /mnt/boot    ufs    rw,noatime    2    2 EEEOOOFFF ln -s /mnt/boot/boot /mnt/boot # loader.conf shows where is our kernel, root, etc # We also load the eli here cat <>/boot/loader.conf geom_eli_load="YES" geli_mfid0p3_keyfile0_load="YES" geli_mfid0p3_keyfile0_type="mfid0p3:geli_keyfile0" geli_mfid0p3_keyfile0_name="/boot/key" vfs.root.mountfrom="ufs:mfid0p3.eli" EEOOFF echo Reboot Now $su #dmesg ....... mfi0: 25667 (590706605s/0x0020/info) - Shutdown command received from host mfi0: 25668 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0060/1000/1f0c/1028) mfi0: 25669 (boot + 3s/0x0020/info) - Firmware version 1.22.52-1909 mfi0: 25670 (boot + 23s/0x0008/info) - Battery Present mfi0: 25671 (boot + 23s/0x0020/info) - Controller hardware revision ID (0x0) mfi0: 25672 (boot + 23s/0x0020/info) - Package version 6.3.3.0002 mfi0: 25673 (boot + 23s/0x0020/info) - Board Revision mfi0: 25674 (boot + 30s/0x0002/info) - Inserted: PD 04(e0xff/s4) mfi0: 25675 (boot + 30s/0x0002/info) - Inserted: PD 04(e0xff/s4) Info: enclPd=ffff, scsiType=0, portMap=04, sasAddr=1221000004000000,0000000000000000 mfi0: 25676 (boot + 30s/0x0002/WARN) - PD 04(e0xff/s4) is not a certified drive mfi0: 25677 (boot + 30s/0x0002/info) - Inserted: PD 05(e0xff/s5) mfi0: 25678 (boot + 30s/0x0002/info) - Inserted: PD 05(e0xff/s5) Info: enclPd=ffff, scsiType=0, portMap=05, sasAddr=1221000005000000,0000000000000000 mfi0: 25679 (boot + 30s/0x0002/WARN) - PD 05(e0xff/s5) is not a certified drive mfi0: 25680 (boot + 30s/0x0002/info) - Inserted: PD 06(e0xff/s6) mfi0: 25681 (boot + 30s/0x0002/info) - Inserted: PD 06(e0xff/s6) Info: enclPd=ffff, scsiType=0, portMap=06, sasAddr=1221000006000000,0000000000000000 mfi0: 25682 (boot + 30s/0x0002/WARN) - PD 06(e0xff/s6) is not a certified drive mfi0: 25683 (boot + 30s/0x0002/info) - Inserted: PD 07(e0xff/s7) mfi0: 25684 (boot + 30s/0x0002/info) - Inserted: PD 07(e0xff/s7) Info: enclPd=ffff, scsiType=0, portMap=07, sasAddr=1221000007000000,0000000000000000 mfi0: 25685 (boot + 30s/0x0002/WARN) - PD 07(e0xff/s7) is not a certified drive mfi0: 25686 (boot + 30s/0x0020/info) - Patrol Read resumed mfi0: 25687 (590801787s/0x0020/info) - Time established as 09/20/18 23:36:27; (31 seconds since power on) mfi0: 25688 (590801844s/0x0008/info) - Battery temperature is normal mfi0: 25689 (590801844s/0x0008/info) - Current capacity of the battery is above threshold mfi0: 25690 (590801844s/0x0008/info) - Battery started charging mfi0: 25691 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0060/1000/1f0c/1028) mfi0: 25692 (boot + 3s/0x0020/info) - Firmware version 1.22.52-1909 mfi0: 25693 (boot + 23s/0x0008/info) - Battery Present mfi0: 25694 (boot + 23s/0x0020/info) - Controller hardware revision ID (0x0) mfi0: 25695 (boot + 23s/0x0020/info) - Package version 6.3.3.0002 mfi0: 25696 (boot + 23s/0x0020/info) - Board Revision mfi0: 25697 (boot + 30s/0x0002/info) - Inserted: PD 04(e0xff/s4) .... mfid0 on mfi0 mfid0: 5721600MB (11717836800 sectors) RAID volume 'pink' is optimal mfi0: MFI_DCMD_PD_LIST_QUERY failed 2 mfi0: 25698 (boot + 30s/0x0002/info) - Inserted: PD 04(e0xff/s4) Info: enclPd=ffff, scsiType=0, portMap=04, sasAddr=1221000004000000,0000000000000000 mfi0: 25699 (boot + 30s/0x0002/WARN) - PD 04(e0xff/s4) is not a certified drive mfi0: 25700 (boot + 30s/0x0002/info) - Inserted: PD 05(e0xff/s5) mfi0: MFI_DCMD_PD_LIST_QUERY failed 2 mfi0: 25701 (boot + 30s/0x0002/info) - Inserted: PD 05(e0xff/s5) Info: enclPd=ffff, scsiType=0, portMap=05, sasAddr=1221000005000000,0000000000000000 mfi0: 25702 (boot + 30s/0x0002/WARN) - PD 05(e0xff/s5) is not a certified drive mfi0: 25703 (boot + 30s/0x0002/info) - Inserted: PD 06(e0xff/s6) mfi0: MFI_DCMD_PD_LIST_QUERY failed 2 mfi0: 25704 (boot + 30s/0x0002/info) - Inserted: PD 06(e0xff/s6) Info: enclPd=ffff, scsiType=0, portMap=06, sasAddr=1221000006000000,0000000000000000 mfi0: 25705 (boot + 30s/0x0002/WARN) - PD 06(e0xff/s6) is not a certified drive mfi0: 25706 (boot + 30s/0x0002/info) - Inserted: PD 07(e0xff/s7) mfi0: MFI_DCMD_PD_LIST_QUERY failed 2 mfi0: 25707 (boot + 30s/0x0002/info) - Inserted: PD 07(e0xff/s7) Info: enclPd=ffff, scsiType=0, portMap=07, sasAddr=1221000007000000,0000000000000000 mfi0: 25708 (boot + 30s/0x0002/WARN) - PD 07(e0xff/s7) is not a certified drive mfi0: 25709 (boot + 30s/0x0020/info) - Patrol Read resumed mfi0: 25710 (590802278s/0x0020/info) - Time established as 09/20/18 23:44:38; (31 seconds since power on) .... mfi0: 25711 (590802335s/0x0008/info) - Battery temperature is normal mfi0: 25712 (590802335s/0x0008/info) - Current capacity of the battery is above threshold mfi0: 25713 (590802335s/0x0008/info) - Battery started charging .... cryptosoft0: on motherboard GEOM_ELI: Device mfid0p3.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI:     Crypto: software mfi0: 25714 (590803765s/0x0008/info) - Battery charge complete On 09/29/18 12:23, The Doctor via freebsd-questions wrote: > I am restoring a full system from backup going from ZFS back to UFS. > > Found something in dmesg of interest > > > AVAGO MegaRAID SAS FreeBSD mrsas driver version: 06.712.04.00-fbsd > mfi0: port 0xf000-0xf0ff mem 0xfbe60000-0xfbe63fff,0xfbe00000-0xfbe3ffff irq 64 at device 0.0 numa-domain 1 on pci9 > mfi0: Using MSI > mfi0: Megaraid SAS driver Ver 4.23 > mfi0: FW MaxCmds = 1008, limiting to 128 > mfi0: MaxCmd = 1008, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c03f0 > mfi0: 17018 (591494719s/0x0020/info) - Shutdown command received from host > mfi0: 17019 (boot + 9s/0x0020/info) - Firmware initialization started (PCI ID 005b/1000/9276/1000) > mfi0: 17020 (boot + 9s/0x0020/info) - Firmware version 3.460.05-4565 > mfi0: 17021 (boot + 11s/0x0008/info) - Battery Present > mfi0: 17022 (boot + 11s/0x0020/info) - Package version 23.34.0-0005 > mfi0: 17023 (boot + 11s/0x0020/info) - Board Revision 001 > mfi0: 17024 (boot + 16s/0x0008/info) - Battery charge complete > mfi0: 17025 (boot + 16s/0x0008/info) - Battery temperature is normal > mfi0: 17026 (boot + 29s/0x0002/info) - Inserted: PD 09(e0xfc/s0) > mfi0: 17027 (boot + 29s/0x0002/info) - Inserted: PD 09(e0xfc/s0) Info: enclPd=fc, scsiType=0, portMap=01, sasAddr=5000c50096e12239,0000000000000000 > mfi0: 17028 (boot + 29s/0x0002/info) - Inserted: PD 0a(e0xfc/s1) > mfi0: 17029 (boot + 29s/0x0002/info) - Inserted: PD 0a(e0xfc/s1) Info: enclPd=fc, scsiType=0, portMap=00, sasAddr=5000c50084ca7425,0000000000000000 > mfi0: 17030 (591494782s/0x0020/info) - Time established as 09/29/18 0:06:22; (34 seconds since power on) > mfi0: 17031 (591494827s/0x0020/WARN) - Host driver needs to be upgraded to enable extended LD support > mfi0: 17032 (591494827s/0x0020/info) - Host driver is loaded and operational > mfid0 numa-domain 1 on mfi0 > mfid0: 3814912MB (7812939776 sectors) RAID volume (no label) is optimal > > Loader variables: > > Manual root filesystem specification: > : [options] > Mount using filesystem > and with the specified (optional) option list. > > eg. ufs:/dev/da0s1a > zfs:tank > cd9660:/dev/cd0 ro > (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /) > > ? List valid disk boot devices > . Yield 1 second (for background tasks) > Abort manual input > > mountroot> > List of GEOM managed disk devices: > cd0 gptid/da2fc326-c377-11e8-a335-0cc47aac511e ufsid/5baebbf017da70dd gptid/c680d0e9-c377-11e8-a335-0cc47aac511e msdosfs/EFISYS gptid/926bcda2-c377-11e8-a335-0cc47aac511e mfid0p3 mfid0p2 mfid0p1 mfid0 > > mountroot> random: unblocking device. > Trying to mount root from ufs:/dev/mfid0p2 []... > > How do fix the above so that the OS knows where the root is? > > > mfi0: 17033 (591505200s/0x0020/info) - Patrol Read started > mfi0: 17034 (591505200s/0x0001/info) - Consistency Check started on VD 00/0 > mfi0: Failed to get command > mfi0: 17035 (591505200s/0x0001/WARN) - Consistency Check started on an inconsistent VD 00/0 > > Anything to be concerned about? >