Date: Sat, 15 Aug 2015 12:38:46 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-fs@freebsd.org Subject: Re: Panic in ZFS during zfs recv (while snapshots being destroyed) Message-ID: <55CF7926.1030901@denninger.net> In-Reply-To: <55BB443E.8040801@denninger.net> References: <55BB443E.8040801@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms060503000503080205070604 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Update: This /appears /to be related to attempting to send or receive a /cloned /snapshot. I use /beadm /to manage boot environments and the crashes have all come while send/recv-ing the root pool, which is the one where these clones get created. It is /not /consistent within a given snapshot when it crashes and a second attempt (which does a "recovery" send/receive) succeeds every time -- I've yet to have it panic twice sequentially. I surmise that the problem comes about when a file in the cloned snapshot is modified, but this is a guess at this point. I'm going to try to force replication of the problem on my test system. On 7/31/2015 04:47, Karl Denninger wrote: > I have an automated script that runs zfs send/recv copies to bring a > backup data set into congruence with the running copies nightly. The > source has automated snapshots running on a fairly frequent basis > through zfs-auto-snapshot. > > Recently I have started having a panic show up about once a week during= > the backup run, but it's inconsistent. It is in the same place, but I > cannot force it to repeat. > > The trap itself is a page fault in kernel mode in the zfs code at > zfs_unmount_snap(); here's the traceback from the kvm (sorry for the > image link but I don't have a better option right now.) > > I'll try to get a dump, this is a production machine with encrypted swa= p > so it's not normally turned on. > > Note that the pool that appears to be involved (the backup pool) has > passed a scrub and thus I would assume the on-disk structure is ok.....= > but that might be an unfair assumption. It is always occurring in the > same dataset although there are a half-dozen that are sync'd -- if this= > one (the first one) successfully completes during the run then all the > rest will as well (that is, whenever I restart the process it has alway= s > failed here.) The source pool is also clean and passes a scrub. > > traceback is at http://www.denninger.net/kvmimage.png; apologies for th= e > image traceback but this is coming from a remote KVM. > > I first saw this on 10.1-STABLE and it is still happening on FreeBSD > 10.2-PRERELEASE #9 r285890M, which I updated to in an attempt to see if= > the problem was something that had been addressed. > > --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms060503000503080205070604 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGXzCC BlswggRDoAMCAQICASkwDQYJKoZIhvcNAQELBQAwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEg U3lzdGVtcyBMTEMgQ0EwHhcNMTUwNDIxMDIyMTU5WhcNMjAwNDE5MDIyMTU5WjBaMQswCQYD VQQGEwJVUzEQMA4GA1UECBMHRmxvcmlkYTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEe MBwGA1UEAxMVS2FybCBEZW5uaW5nZXIgKE9DU1ApMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A MIICCgKCAgEAuYRY+EB2mGtZ3grlVO8TmnEvduVFA/IYXcCmNSOC1q+pTVjylsjcHKBcOPb9 TP1KLxdWP+Q1soSORGHlKw2/HcVzShDW5WPIKrvML+Ry0XvIvNBu9adTiCsA9nci4Cnf98XE hVpenER0qbJkBUOGT1rP4iAcfjet0lEgzPEnm+pAxv6fYSNp1WqIY9u0b1pkQiaWrt8hgNOc rJOiLbc8CeQ/DBP6rUiQjYNO9/aPNauEtHkNNfR9RgLSfGUdZuOCmJqnIla1HsrZhA5p69Bv /e832BKiNPaH5wF6btAiPpTr2sRhwQO8/IIxcRX1Vxd1yZbjYtJGw+9lwEcWRYAmoxkzKLPi S6Zo/6z5wgNpeK1H+zOioMoZIczgI8BlX1iHxqy/FAvm4PHPnC8s+BLnJLwr+jvMNHm82QwL J9hC5Ho8AnFU6TkCuq+P2V8/clJVqnBuvTUKhYMGSm4mUp+lAgR4L+lwIEqSeWVsxirIcE7Z OKkvI7k5x3WeE3+c6w74L6PfWVAd84xFlo9DKRdU9YbkFuFZPu21fi/LmE5brImB5P+jdqnK eWnVwRq+RBFLy4kehCzMXooitAwgP8l/JJa9VDiSyd/PAHaVGiat2vCdDh4b8cFL7SV6jPA4 k0MgGUA/6Et7wDmhZmCigggr9K6VQCx8jpKB3x1NlNNiaWECAwEAAaOB9DCB8TA3BggrBgEF BQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDALBgNVHQ8EBAMCBeAwLAYJYIZIAYb4QgENBB8W HU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBTFHJQt6cloXBdG1Pv1 o2YgH+7lWTAfBgNVHSMEGDAWgBQkcZudhX383d29sMqSlAOh+tNtNTAdBgNVHREEFjAUgRJr YXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAE9/dxi2YqjCYYhiybp4GKcm 7tBVa/GLW+qcHPcoT4dqmqghlLz8+iUH+HCJjRQATVGyMEnvISOKFVHC6aZIG+Sg7J8bfS4+ fjKDi9smRH2VPPx3bV8+yFYRNroMGHaPHZB/Xctmmvc+PZ9O2W7rExgrODtxIOB3Zs6wkYf+ ty+9r1KmTHlV+rRHI6timH1uiyFE3cPi1taAEBxf0851cJV8k40PGF8G48ewnq8SY9sCf5cv liXbpdgU+I4ND5BuTjg63WS32zuhLd1VSuH3ZC/QbcncMX5W3oLXmcQP5/5uTiBJy74kdPtG MSZ9rXwZPwNxP/8PXMSR7ViaFvjUkf4bJlyENFa2PGxLk4EUzOuO7t3brjMlQW1fuInfG+ko 3tVxko20Hp0tKGPe/9cOxBVBZeZH/VgpZn3cLculGzZjmdh2fqAQ6kv9Z9AVOG1+dq0c1zt8 2zm+Oi1pikGXkfz5UJq60psY6zbX25BuEZkthO/qiS4pxjxb7gQkS0rTEHTy+qv0l3QVL0wa NAT74Zaj7l5DEW3qdQQ0dtVieyvptg9CxkfQJE3JyBMb0zBj9Qhc5/hbTfhSlHzZMEbUuIyx h9vxqFAmGzfB1/WfOKkiNHChkpPW8ZeH9yPeDBKvrgZ96dREHFoVkDk7Vpw5lSM+tFOfdyLg xxhb/RZVUDeUMYIE4zCCBN8CAQEwgZYwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdGbG9y aWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBMTEMxHDAa BgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEgU3lzdGVt cyBMTEMgQ0ECASkwCQYFKw4DAhoFAKCCAiEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc BgkqhkiG9w0BCQUxDxcNMTUwODE1MTczODQ2WjAjBgkqhkiG9w0BCQQxFgQUsj301L0lWilS XmM0jDEJSHKqQh8wbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAEC MAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzAN BggqhkiG9w0DAgIBKDCBpwYJKwYBBAGCNxAEMYGZMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBAgEpMIGpBgsqhkiG9w0BCRACCzGBmaCBljCBkDELMAkGA1UE BhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQ Q3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqG SIb3DQEJARYTQ3VkYSBTeXN0ZW1zIExMQyBDQQIBKTANBgkqhkiG9w0BAQEFAASCAgAeJoPD 8I8TU6f4UIYY5o9vfFQyoUIY7wBCQWN6TzFNTmg4vVnG2MPDz6iGL1JvY0avvPS2fdpgby2G 0iGQTzuivKXW3tqIRfE3yDocV/9ruJPz92ji6B6Omj7uostdpafD/SdHKuV4ZdwmtmVByP8O 9d7jPD6S1uSw0gj8NpgCZ/Xx4pgaVN6WqIOr7F+6rwTTK+A2zydDsy94qTAfSoHThTd6XN/i 4g+Hl8kwPADBCHcfRIZy9v8Qm7MHcoVYUpeeY+530L9usmp5egLoB0culFAIf4Yh3//xLbed wqgXyZnX9b1xcJdKeKVyhxlDGDGayXBUOLHNy5Vr0bLWWHX7YlRn1HjJ5LOLiD9He99CAdTa P90cLcPF+VGVsO8dteGJvmd72e02QTFlxv/jgbXwtXfTPeZJxPq5pYAPY3q1GjqX1cqXCCAo XuRVpAl0VhFRajKZQYwKiMJEaUxYhZAAiNhppGMBZozXrc11t8J2Q1l4BsSNK43u+bQUzKn8 wu/J7qe6PC1QX6lJSMdYMfnvkpd+OiJCsQ7qnRzt2CsDr1Oyz1NYT2TBNTM2abD8m/RuJ/HQ +EzN0JosGP0Q70/668VslP9czOZ4xc8o4EXRHvOag9q4PzzW+zQ6IQ0eMHVHkgicVUEcL3aC XRu0KMO0pxxR6OA1OJIx/lmUNUuZCwAAAAAAAA== --------------ms060503000503080205070604--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55CF7926.1030901>