From owner-freebsd-stable@FreeBSD.ORG Wed Oct 1 21:28:33 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A34FAD86 for ; Wed, 1 Oct 2014 21:28:33 +0000 (UTC) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "NewFS.denninger.net", Issuer "NewFS.denninger.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D788C9B for ; Wed, 1 Oct 2014 21:28:32 +0000 (UTC) Received: from [192.168.1.40] (localhost [127.0.0.1]) by fs.denninger.net (8.14.9/8.14.8) with ESMTP id s91LSCvv063784 for ; Wed, 1 Oct 2014 16:28:13 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (TLS/SSL) [192.168.1.40] by Spamblock-sys (LOCAL/AUTH); Wed Oct 1 16:28:13 2014 Message-ID: <542C71C9.1050907@denninger.net> Date: Wed, 01 Oct 2014 16:27:37 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Encrypted (GELI) root on ZFS troubles Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms030609080300030206060509" X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 21:28:33 -0000 This is a cryptographically signed message in MIME format. --------------ms030609080300030206060509 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable So here's the fun part of what I'm trying to do (and getting frustrated with) I have set up a GPT disk with the following setup: =3D> 34 625142381 da2 GPT (298G) 34 6 - free - (3.0K) 40 1024 1 freebsd-boot (512K) 1064 4194304 2 freebsd-zfs [bootme] (2.0G) 4195368 134217728 3 freebsd-swap (64G) 138413096 486729312 4 freebsd-zfs (232G) 625142408 7 - free - (3.5K) Then on freebsd-boot I have written the bootloaders. The "bootme" filesystem has *only* the /boot directory copied over from the rest of the system's root directory (that is, the kernel, loadables, /boot/loader.conf, etc); that pool is called "zboot" Partition 4 has the label "root0" on it, and thus shows up in /dev/gpt.=20 I have initialized that with geli, set the boot option flag (that is, prompt on boot) and created a pool called "root" on the resulting .eli device and then put the system on that. That's all ok. Finally, I set the bootfs on that latter pool. There is no bootfs set on /zboot: # zpool get bootfs zboot NAME PROPERTY VALUE SOURCE zboot bootfs - default It is set on the root pool to the proper filesystem: # zpool get bootfs root NAME PROPERTY VALUE SOURCE root bootfs root/R/10.1-CLEAN local The problem is that when the system boots geli "finds" the raw device (in this case /dev/da0p4), prompts for the password and attaches there instead of in /dev/gpt. The gpt label is missing --- and equally bad the "root" pool does not appear to import at boot time either. As a result the system tries to mount root from /zboot (even though it's not been told to, and HAS been told where to mount off the root pool), but there's no init in there (or anything else other than the boot filesystem itself) and as a result I get an immediate panic. If I boot off a different (working) zfs-based system the probe still finds the "prompt during boot" flag on that gpt partition and asks for the password on the device. I can see the pool; zpool import shows it: pool: root id: 17719633931604198170 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: root ONLINE da2p4.eli ONLINE Not so good. If I detach that the device reappears in /dev/gpt; I can then attach geli and import the pool in either location. Putting the cache file from the previous imported state in the zboot/boot/zfs directory doesn't help (nor does removing the cache file entirely) More-interestingly if I reboot the cloned system with the root pool imported it does come back up, even though the device is the base (da2p4.eli) rather than in the /dev/gpt directory. Anyone know what's going on here? And is there a way to have geli attach during boot-time off the /dev/gpt directory instead of on the base device partition name? --=20 Karl Denninger karl@denninger.net /The Market Ticker/ --------------ms030609080300030206060509 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDEwMDEyMTI3MzdaMCMGCSqGSIb3DQEJBDEW BBQk1lT8Ph5bBHl8c4RwxDebUR4CCjBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIABMxWhMiIP+i3PF/RnzOXcS0CcBBZ hkvMKbeHO5aJEOpTvk+X1uClc/+VP2JskguhKYqmH0XE/EfIk8CZDBIOYw829N++0R5hLuYd qFP576qd80s6l9XfrwHqxMuca9GSd5T6KN83Z9eWDbeO4Y9asUfOfHbNtj+mp+7ipd8NFyLp zXyPKw0KF0qWNoCGqjx5eVovKBHULvuxvT/6GIDiwpuukSO93NFw5amsWMSowBf0sbW7Dgj6 UpltwaGoH+dP1KSW8nv1QHj4fnap9ba0xcZhwdmBegUcfTRr8HdTNJC+QprzgKgheIGUDQOd YBZ9BjiDr1dRqGKdckdwHeU2z6gIU8xyg67pOgZziURaGs7bqLdmM5jcwYgX22kf2cNNl5yX 5D65MmGI5/cUHw+YrcENjblKZakIIKHuvvlqS/SYnr8+GIc6RjLBodL/1g1cy7h1rEUz4hH5 1O2to8SnoHoo3k6GCHcJbqqYHFypiyQ4tB4yvuAnlRzJhqBrRFWA9Ep8RJMBoTgTVfQ9A7IT S7o06FztJ5vWSK9DL2pS38wEJnPrVbb3sk3jkiMBYAXPA7dv1qc7axlFUA7OT/LRoPxW2esh WwBPE+wrIRf0AbmHPJLM53xCgoNcI2uePg/ALf15zxeQ9/TfauVoVBZk+T6LBvMKhz9ORHf7 RMFN6xAAAAAAAAA= --------------ms030609080300030206060509--