Date: Thu, 21 Jul 2016 08:04:52 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Panic on BETA1 in the ZFS subsystem Message-ID: <03cdf671-a7a8-12ac-3204-e5a1bf1ef062@denninger.net> In-Reply-To: <6cb46059-85c8-0c3b-7346-773647f1a962@FreeBSD.org> References: <8f44bc09-1237-44d0-fe7a-7eb9cf4fe85b@denninger.net> <54e5974c-312e-c33c-ab83-9e1148618ddc@FreeBSD.org> <97cf5283-683b-83fd-c484-18c14973b065@denninger.net> <c2f24b1e-be84-bcdd-ea0b-515cd2aca266@FreeBSD.org> <1f064549-fa72-fe9b-d66d-85923437bb9b@denninger.net> <6cb46059-85c8-0c3b-7346-773647f1a962@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms050401030908020605020501 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 7/21/2016 07:52, Andriy Gapon wrote: > On 21/07/2016 15:25, Karl Denninger wrote: >> The crash occurred during a backup script operating, which is (roughly= ) >> the following: >> >> zpool import -N backup (mount the pool to copy to) >> >> iterate over a list of zfs filesystems and... >> >> zfs rename fs@zfs-base fs@zfs-old >> zfs snapshot fs@zfs-base >> zfs send -RI fs@zfs-old fs@zfs-base | zfs receive -Fudv backup >> zfs destroy -vr fs@zfs-old >> >> The first filesystem to be done is the rootfs, that is when it panic'd= , >> and from the traceback it appears that the Zio's in there are from the= >> backup volume, so the answer to your question is "yes". > I think that what happened here was that a quite large number of TRIM > requests was queued by ZFS before it had a chance to learn that the > target vdev in the backup pool did not support TRIM. So, when the the > first request failed with ENOTSUP the vdev was marked as not supporting= > TRIM. After that all subsequent requests were failed without sending > them down the storage stack. But the way it is done means that all the= > requests were processed by the nested zio_execute() calls on the same > stack. And that lead to the stack overflow. > > Steve, do you think that this is a correct description of what happened= ? > > The state of the pools that you described below probably contributed to= > the avalanche of TRIMs that caused the problem. > The source for the backup a pool that is comprised entirely of SSDs (and thus supports TRIM), and the target is a pair of spinning rust devices (which of course do not support TRIM); the incremental receive to that pool does (of course) remove all the obsolete snapshots..... What I don't understand however, is why it has been running fine for a week or so, and why it immediately repeated the panic on a retry attempt -- or how to prevent it, at least at this point. I certainly do not want to leave the pool mounted when not in active backup use. --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms050401030908020605020501 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA3MjExMzA0NTJaME8GCSqGSIb3DQEJBDFCBEBo ShO8B3/jJPX4jjXe5L7XugO2oe155klyPvAql/DSTBDvDt9QbaOFpgvGF6UAOMp9/HwMSq0y febi9OPgVDwsMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAaN02kK0e dRDRNKeAoKvZXJLzFvXBVaP1H1AMmiScKHsQj9Kk6Pn0vbfrYaesQuS1DX6Nht9YbIWNz6fi IFOlO4oQAbutiq90YXtnn9XjU03tvzrkNnfJpCZ8/3QQV7Es/RcydGEPdLFVTQPdOtAHph5l l+gJwfUJ3wDV+H7B7hz1Q0Za8KiUuDwL76/m64vwg5Hk6o+47O5ASipDrPWW0znets6d1FcZ MGCnRMPHgXkrnXoL0rYswfhKARLDiqwy3IC32T3BmXbmr32B+i9B3FkiEM9f/buVVmKbhpU0 RhVu+FeMbn3sRcSpKVEp0L9o80DMvEysxP9PwIKjXXU27S9ux2sez15nDvXlf8q4asxIAOyb qscQCX0E6IeIJPbU8bjW5EV8s3AyOo7Op4PNdobhO5TC37jve4lWe+E+SHQDEzGf/c6RHvfv ZNRn3WXN0JKIKaY9DlrnFFWfYU6CGumrI0Vokdn6gku8ssq1SMy2JLqhDhD3vCTUn3AXJfZm owERFQ5IVyr43+UGMYSopzR2nrmbJhlt8FKbpgNLucvWFd9/Md0xvGIgOvv5yTAbM1rXtx7G Qwc/0EpZ3RBXaQ8KQdDCH31YwX25EWfEdXhIVZWwC7i4lKTKrcQlLLumR6d+PHfM1kjjtMjh qEKhufdqt2yj4Pv978xVKtanyHUAAAAAAAA= --------------ms050401030908020605020501--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?03cdf671-a7a8-12ac-3204-e5a1bf1ef062>