Date: Wed, 01 Oct 2014 23:50:26 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Encrypted (GELI) root on ZFS troubles Message-ID: <542CD992.7050509@denninger.net> In-Reply-To: <d4f3fecfcd8785c41e3658ff34123088@dweimer.net> References: <542C71C9.1050907@denninger.net> <d4f3fecfcd8785c41e3658ff34123088@dweimer.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms030201070209010405050509 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 10/1/2014 8:15 PM, dweimer wrote: > On 10/01/2014 4:27 pm, Karl Denninger wrote: >> So here's the fun part of what I'm trying to do (and getting frustrate= d >> with) >> >> I have set up a GPT disk with the following setup: >> >> =3D> 34 625142381 da2 GPT (298G) >> 34 6 - free - (3.0K) >> 40 1024 1 freebsd-boot (512K) >> 1064 4194304 2 freebsd-zfs [bootme] (2.0G) >> 4195368 134217728 3 freebsd-swap (64G) >> 138413096 486729312 4 freebsd-zfs (232G) >> 625142408 7 - free - (3.5K) >> >> Then on freebsd-boot I have written the bootloaders. >> >> The "bootme" filesystem has *only* the /boot directory copied over fro= m >> the rest of the system's root directory (that is, the kernel, loadable= s, >> /boot/loader.conf, etc); that pool is called "zboot" >> >> Partition 4 has the label "root0" on it, and thus shows up in /dev/gpt= =2E >> I have initialized that with geli, set the boot option flag (that is, >> prompt on boot) and created a pool called "root" on the resulting .eli= >> device and then put the system on that. That's all ok. >> >> Finally, I set the bootfs on that latter pool. There is no bootfs set= >> on /zboot: >> >> # zpool get bootfs zboot >> NAME PROPERTY VALUE SOURCE >> zboot bootfs - default >> >> It is set on the root pool to the proper filesystem: >> >> # zpool get bootfs root >> NAME PROPERTY VALUE SOURCE >> root bootfs root/R/10.1-CLEAN local >> >> The problem is that when the system boots geli "finds" the raw device >> (in this case /dev/da0p4), prompts for the password and attaches there= >> instead of in /dev/gpt. The gpt label is missing --- and equally bad >> the "root" pool does not appear to import at boot time either. >> >> As a result the system tries to mount root from /zboot (even though it= 's >> not been told to, and HAS been told where to mount off the root pool),= >> but there's no init in there (or anything else other than the boot >> filesystem itself) and as a result I get an immediate panic. >> >> If I boot off a different (working) zfs-based system the probe still >> finds the "prompt during boot" flag on that gpt partition and asks for= >> the password on the device. I can see the pool; zpool import shows it= : >> >> pool: root >> id: 17719633931604198170 >> state: ONLINE >> action: The pool can be imported using its name or numeric identifier= =2E >> config: >> >> root ONLINE >> da2p4.eli ONLINE >> >> Not so good. >> >> If I detach that the device reappears in /dev/gpt; I can then attach >> geli and import the pool in either location. Putting the cache file >> from the previous imported state in the zboot/boot/zfs directory doesn= 't >> help (nor does removing the cache file entirely) >> >> More-interestingly if I reboot the cloned system with the root pool >> imported it does come back up, even though the device is the base >> (da2p4.eli) rather than in the /dev/gpt directory. >> >> Anyone know what's going on here? And is there a way to have geli >> attach during boot-time off the /dev/gpt directory instead of on the >> base device partition name? > > On my work laptop (not turned on so I am going by memory on this), I > have a similar setup using a USB thumb drive for the boot volume. My > setup is as follows and works quite well, perhaps this will help you. > > Thumb Drive > da0 > > Disk Drive > ada0 > > da0 has a GPT table of > da0 GPT (8G) > 1 freebsd-boot (512k) -- /dev/gpt/usbboot > 2 freebsd-zfs (8G) -- /dev/gpt/usbzfs > > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 > > ada0 has a GPT table of > 1 freebsd-swap (8G) -- /dev/gpt/swap > 2 freebsd-zfs (222G) -- /dev/gpt/zroot > > > I used geli init -b /dev/gpt/zroot > when attached /dev/gpt/zroot.eli > > swap is auto encrypted at boot using the fstab line > /dev/gpt/swap.eli none none swap sw 0 0 > > I believe they devices only show up as /dev/gpt/... if the -l ... > option is used to set a label on the partition at creation time. > > 2 configured zpools > usbzfs > gpt/usbzfs > > zroot > gpt/zroot.eli > > zpool set bootfs=3Dusbzfs/boot usbzfs > zpool set bootfs=3Dzroot/ROOT/installation zroot (not sure if this does= > anything, I just set it) > > usbzfs/boot has a mountpoint of /zfsboot > > loader.conf: > zfs_load=3D"YES" > vfs.root.mountfrom=3D"zfs:zroot/ROOT/install" > > copied /boot to /zfsboot/boot > > zpool export usbzfs > > It will still boot after the zpool has been exported if the devices is > found, just doesn't get mounted, in my case this means I can remove > the USB thumb drive as soon as root is remounted from the geli > partition, after entering the password without causing any issues. > > I can send you the full gpt output and zpool status information > tomorrow morning when I am back in the office on my laptop if you > still need help getting yours working. > I got it working but..... 1. Having the kernel able to cross pools and examine the bootfs property would be fairly useful. I use beadm, but of course there are lots of risks associated with it and this sort of setup; if you forget to update the unencrypted boot area you could find yourself in pretty-serious trouble with an unbootable system -- if your emergency media is also out of date vis-a-vis zfs flags you've got trouble. 2. Not having the system look at the /dev/gpt entries (yes, I use "-l" when I create the partitions) is troublesome. Specifically, it's trouble when you have a lot of drives and an adapter that doesn't play nice with coherent unit numbering related to the cable point the disk is on -- which is true of most of the mps-driven adapters (and especially those with a SAS expander or three hooked to them.) If you've never pulled the wrong disk from a running machine by thinking the numbering is different than it actually is then this probably doesn't mean much to you :-) Not being able to be SURE which disk failed so you can make sure you yank the right one is a quite serious annoyance. --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ --------------ms030201070209010405050509 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDEwMDIwNDUwMjZaMCMGCSqGSIb3DQEJBDEW BBTF5mzHXYQAqGXi+DG80DkZhftJ7DBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAlJFOFI3J6zhK0IJzcwvmna6ykWmm 1SQiWLoRJNE6+fpqwPeIw/JMz+DmaK2tiR3Th1Ly14nXt33lHz0nseXnvp9+tk/Bt1Q6JF00 fdYcMurm7vtHphmN+hK5hz6E9iXxU+4+FIwofApQuyar8I8xuekXFCpc8dPBSHiFQLyEkfBR 5fDr/hyneh46Y0+4R+4aV7juMXsvoBp4r9mVzxYCja54L19muTl91qBRpdNlTrGAH9S+Eo7K 19cnir/NIbHYnC/wWHj07eqgT9dpXrWXTrzzGt0bA/PZAuap3l1qXJYPACaq9CQLqvqlGYm7 eQu0q4awkkffn80jjQgRd9FrCPFVpSCCYjpLFm+BYjchB6NsHF6WxMcP6TTZT4E6LHSWs1OI IIptkOLOLjaEEHY02PPDhTAcyEXqXs0ZmAI9bL2NXuxQz3Ht17B42VxfBGjGcLEokBvu/E9w 9PGyumMFn1HT6tiSkksNel2fDJVkFSqoEEMrkAB3U7rKORqnax9VR4qkWi5ahrQ94+HHUDi1 nt/o0rhEsoenVbENHahj2kAy7J/b+RmWhNv13TSkfpLKg2zeYfv6j1FSvxYky015wdTNxaiw spKBsCbfOMWwboWK3y7gFBg4zMl8sLXpCjFSEgh0Dkoq/ERei3tBeBE7As4kZeYbaJhITa6y ZqVfcmsAAAAAAAA= --------------ms030201070209010405050509--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?542CD992.7050509>