From owner-freebsd-fs@freebsd.org Mon Nov 20 15:37:37 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30150DEF05D for ; Mon, 20 Nov 2017 15:37:37 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [67.205.158.196]) by mx1.freebsd.org (Postfix) with ESMTP id 097DC2A5A for ; Mon, 20 Nov 2017 15:37:36 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 229B627461 for ; Mon, 20 Nov 2017 10:37:00 -0500 (EST) Received: from [192.168.10.23] (D13.Denninger.Net [192.168.10.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 544AF3A15CF for ; Mon, 20 Nov 2017 09:36:58 -0600 (CST) To: "freebsd-fs@freebsd.org" From: Karl Denninger Subject: MSDOS Filesystem question related to "read-only" files Message-ID: Date: Mon, 20 Nov 2017 09:36:56 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms040008050806070601010508" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 15:37:37 -0000 This is a cryptographically signed message in MIME format. --------------ms040008050806070601010508 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I'm running into an interesting issue here and wondering if there's a way to do this under FreeBSD. MSDOS filesystems have a "primitive" permission capability; specifically, they can have a "Read-only" attribute on a file.=C2=A0 It l= ooks like OpenBSD supports this from reading their man pages. FreeBSD doesn't appear to.=C2=A0 When you mount a msdos filesystem (e.g. = a USB stick) whoever owns the parent directory where you mount it gives you the permissions and "ownership" of files on said filesystem.=C2=A0 Al= l good so far.=C2=A0 But attempting to chmod a file to remove write permiss= ion "succeeds" (returns success) but does nothing. Is this capability simply not present on FreeBSD?=C2=A0 I'm interested in= using it as a means of "flagging" files on a USB stick in an application that I do not want to remove if the stick fills (basically, to "protect" them from being aged off) and it appears there's no way to do it, other than to use something unique in the filename that I would then have to pay attention to. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms040008050806070601010508 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTcxMTIwMTUzNjU2 WjBPBgkqhkiG9w0BCQQxQgRANLO9td2RXZD2/o7OutsQGU9PiDBULTHjsROoOYuUcam7C4l4 s4sxoiOqMHluj0SlMaf/w98RB9E8H5z+0nMCtjBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgA4l5JlCjwP5/Rqk5SsKRxv3Ln8Evwk33xhteqhrn+l/X8ItHHbm34m2DDT4osdKKWM HhJ9bphzIrbPeC2V92evV5Fhg4eIfPR00x6ec4YfPTHtrfRfkmfNXxST67W56OA4yZ+UE2m9 FZOHRKKpS5javGq9mNWI1mMiAmmS4+MzRKpauGXY4rZ33XuR8pzJpbvtUosTwi9XJBiHWTL/ dBTSHM2focZZ2rsIMDfGlBB/nlPAV9ForHWB7c5GU3H2eiNoaDMD2j/umxfO0vDyRxxgyQCe mV6p3tS3cFzmnH+p8M6F/dKwU9CQi9vttYSPGX8d2XJ+4MwHX8siVFhbj7NguFEeGePjGeDN F4mjXiG1LryeE6tV1fHTHjAdtStwSw40IL9lmPIaxW8R//1+7eSESHBrsN9LofQ4lBxtnJ5Q mG0hTu9/yspgTIE9bhtndoIA8XTYJ35Qho4LXap0YW6sEE3/K97UpIrcOgrXafgpKPnxtSs7 lC6yICw7AkoJ01OnjakZirT2bAGETNKQ8WfCYc35hz5NDOQUeDTF4pKv/RFwG67ymaXIVnyt Q6AqmaawtLakPDcMPGk8KxdkjvKRly1YRWHJ5GbDi6op40uLZe1y385QdCg9Et0jycdmb/wn IlW5eGeJSbYstxRcZfm504RRueIwJq/umrRvsAMNnQAAAAAAAA== --------------ms040008050806070601010508-- From owner-freebsd-fs@freebsd.org Mon Nov 20 18:55:02 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 339E9DF2D47 for ; Mon, 20 Nov 2017 18:55:02 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-io0-f182.google.com (mail-io0-f182.google.com [209.85.223.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 02B7A6809C for ; Mon, 20 Nov 2017 18:55:01 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-io0-f182.google.com with SMTP id z74so16777176iof.12 for ; Mon, 20 Nov 2017 10:55:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=JqDneKovIHFbVu8DE4KlGKfZ1OVmSD1hrR3K1aIV8H4=; b=Hz788Qb/r10sbxWofqxdPzzmNdGJo3wtJ0Q7n7qOTH5bD1juF3CUTMVipCpV5wBX19 UohI6ECKC0WpXC+5SCK3ZGdvb3YGPN85MvCzd66JWHQn0ycbC5UN0mntueuIvxLQXXIm 1H8NPcjflVA2hzVlnrW75f/5i0tcYaZZTvCtsmLK6pDchEG7b5czO/oXsj1LP75HNRcw VA9cl+0brrA6fgVg29k0y9jMw8Hi7l13A7SROrPqHBQNCfH/T2t35+tn+fHfpwow5Ttj igQXxkiDV/dBAVjCp9l/Hw8z88AeN1SdXn/2jWoN86OWCcx7NDst32wLn4FLVlGjf6GY 2jYw== X-Gm-Message-State: AJaThX4nNdUpe8sQ/eumP+mkFCjymjSD06pPx5OOXFDBy+3+HaJmq6xn AH10aMIka9uJ+XNnZ8v60GZ61g/a X-Google-Smtp-Source: AGs4zMYHDoPmRNWOnHRbjiKXSRakhjsAdXmjeUVFFpMN/2etvwCSvonJ+r0tEH9k2gzFJXAqlHiC4g== X-Received: by 10.107.38.79 with SMTP id m76mr15341708iom.187.1511202644274; Mon, 20 Nov 2017 10:30:44 -0800 (PST) Received: from mail-io0-f172.google.com (mail-io0-f172.google.com. [209.85.223.172]) by smtp.gmail.com with ESMTPSA id w96sm5151832ioe.76.2017.11.20.10.30.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Nov 2017 10:30:43 -0800 (PST) Received: by mail-io0-f172.google.com with SMTP id u42so16705076ioi.9 for ; Mon, 20 Nov 2017 10:30:43 -0800 (PST) X-Received: by 10.107.7.169 with SMTP id g41mr15504230ioi.38.1511202643349; Mon, 20 Nov 2017 10:30:43 -0800 (PST) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 10.2.165.150 with HTTP; Mon, 20 Nov 2017 10:30:42 -0800 (PST) In-Reply-To: References: From: Conrad Meyer Date: Mon, 20 Nov 2017 10:30:42 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: MSDOS Filesystem question related to "read-only" files To: Karl Denninger Cc: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 18:55:02 -0000 Hi Karl, In fact, msdosfs in FreeBSD should set the FAT READONLY attribute under two conditions: 1. The owner chmod's the file non-writeable (chmod u-w) (what you've described, I think). Or, 2. The super user or otherwise privileged user sets the "readonly" flag on the file via chflags(1). How have you determined that chmod u-w does nothing? What version of FreeBSD are you using? Best, Conrad On Mon, Nov 20, 2017 at 7:36 AM, Karl Denninger wrote: > I'm running into an interesting issue here and wondering if there's a > way to do this under FreeBSD. > > MSDOS filesystems have a "primitive" permission capability; > specifically, they can have a "Read-only" attribute on a file. It looks > like OpenBSD supports this from reading their man pages. > > FreeBSD doesn't appear to. When you mount a msdos filesystem (e.g. a > USB stick) whoever owns the parent directory where you mount it gives > you the permissions and "ownership" of files on said filesystem. All > good so far. But attempting to chmod a file to remove write permission > "succeeds" (returns success) but does nothing. > > Is this capability simply not present on FreeBSD? I'm interested in > using it as a means of "flagging" files on a USB stick in an application > that I do not want to remove if the stick fills (basically, to "protect" > them from being aged off) and it appears there's no way to do it, other > than to use something unique in the filename that I would then have to > pay attention to. > > -- > Karl Denninger > karl@denninger.net > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ From owner-freebsd-fs@freebsd.org Mon Nov 20 20:02:50 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E84A9DF43C6 for ; Mon, 20 Nov 2017 20:02:50 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [67.205.158.196]) by mx1.freebsd.org (Postfix) with ESMTP id C04206AE26 for ; Mon, 20 Nov 2017 20:02:50 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 7896A274BB for ; Mon, 20 Nov 2017 15:02:50 -0500 (EST) Received: from [192.168.10.23] (D13.Denninger.Net [192.168.10.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 6F7D73A1BCC for ; Mon, 20 Nov 2017 14:02:48 -0600 (CST) Subject: Re: MSDOS Filesystem question related to "read-only" files Cc: "freebsd-fs@freebsd.org" References: From: Karl Denninger Message-ID: Date: Mon, 20 Nov 2017 14:02:47 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms020103050803010300020803" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 20:02:51 -0000 This is a cryptographically signed message in MIME format. --------------ms020103050803010300020803 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable root@Test-MCP:/home/karl/HD-MCP # cd /mnt root@Test-MCP:/mnt # mount /dev/ufs/rootfs on / (ufs, local, noatime, soft-updates, nfsv4acls) devfs on /dev (devfs, local) tmpfs on /tmp (tmpfs, local) /dev/da0s1 on /mnt (msdosfs, local) root@Test-MCP:/mnt # ls -al total 408 drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Jan=C2=A0= 1=C2=A0 1980 . drwxr-xr-x=C2=A0 20 root=C2=A0=C2=A0 wheel=C2=A0=C2=A0=C2=A0=C2=A0 512 No= v 20 16:04 .. drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Nov 19 = 11:20 System Volume Information -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 127979 Nov 19 22:54 cam2-2017-11-19-22-54-47.jpg -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 271295 Nov 19 22:57 cam2-2017-11-19-22-57-20.jpg root@Test-MCP:/mnt # chmod u-w * root@Test-MCP:/mnt # ls -al total 408 drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Jan=C2=A0= 1=C2=A0 1980 . drwxr-xr-x=C2=A0 20 root=C2=A0=C2=A0 wheel=C2=A0=C2=A0=C2=A0=C2=A0 512 No= v 20 16:04 .. drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Nov 19 = 11:20 System Volume Information -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 127979 Nov 19 22:54 cam2-2017-11-19-22-54-47.jpg -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 271295 Nov 19 22:57 cam2-2017-11-19-22-57-20.jpg root@Test-MCP:/mnt # Nope.=C2=A0 The "w" is still there. No error returned from the "chmod" command (or if I call it from a C program) but the file mode is NOT changed whether I'm doing it as the superuser or as the owner of the file (and directory) On 11/20/2017 12:30, Conrad Meyer wrote: > Hi Karl, > > In fact, msdosfs in FreeBSD should set the FAT READONLY attribute > under two conditions: > > 1. The owner chmod's the file non-writeable (chmod u-w) (what you've > described, I think). Or, > 2. The super user or otherwise privileged user sets the "readonly" > flag on the file via chflags(1). > > How have you determined that chmod u-w does nothing? What version of > FreeBSD are you using? > > Best, > Conrad > > > On Mon, Nov 20, 2017 at 7:36 AM, Karl Denninger wr= ote: >> I'm running into an interesting issue here and wondering if there's a >> way to do this under FreeBSD. >> >> MSDOS filesystems have a "primitive" permission capability; >> specifically, they can have a "Read-only" attribute on a file. It loo= ks >> like OpenBSD supports this from reading their man pages. >> >> FreeBSD doesn't appear to. When you mount a msdos filesystem (e.g. a >> USB stick) whoever owns the parent directory where you mount it gives >> you the permissions and "ownership" of files on said filesystem. All >> good so far. But attempting to chmod a file to remove write permissio= n >> "succeeds" (returns success) but does nothing. >> >> Is this capability simply not present on FreeBSD? I'm interested in >> using it as a means of "flagging" files on a USB stick in an applicati= on >> that I do not want to remove if the stick fills (basically, to "protec= t" >> them from being aged off) and it appears there's no way to do it, othe= r >> than to use something unique in the filename that I would then have to= >> pay attention to. >> >> -- >> Karl Denninger >> karl@denninger.net >> /The Market Ticker/ >> /[S/MIME encrypted email preferred]/ --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms020103050803010300020803 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTcxMTIwMjAwMjQ3 WjBPBgkqhkiG9w0BCQQxQgRAjOLt2BDdvXSeFtrQY5H+qcXGLQq/KaYLrIx98AD6ocmV9OOa VS0Ev60ASwpdHQYkXVQTozCYFcgP2XT1dW1wNDBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgCFwle/WbLAZAyLTpvXtBMzJuiNjdJHBPGErVxDG4DF6K79fTjCDXLmR8avk/jWiOaF LHXoC9evkYE1ES/6r0UOrG2ohr3h1oDqx62ikCJ7A2MoPldCI3wV6sjc55GHwqWtAp6c94lC 121i0DAsaZzHXl2gWGVc7FZPMCb9zey44CCLKJMQZDGtx2eQetfawtG0xpvGsxTxh3bUrOqF o9+4f78PDcjhpgz2muMDbMJg0WKNeUJWNkJ0CL+F/K9pukXZFMSozXFx2jwDID409zftV9xs PdOJaIj/mNlRKTtjePVRev73zS74nZGEspLTQdPrJZabiz439fS3/y0/UupmsYCfSQs+f0de 4kb3+E1Cro0d5bm2ACR9H1boFBQtxXkF5/5nAQLlKFSNJ9fPXMgtaO0/6ffafWxQflFKga0V B9kbHmSOuqqm1yoFVjS3BuHcvxt6LufE9tMvHEBxiK4wTPSLU6AlAZyH4AmtoaZWUplERes1 tYfTYAYaQc5h0FXyhPEGtWT0QR5i41nzf56vnt+M86kWXQ70RYBELN8wFGZ5yNePH0nJLrUZ rVtdpRBPOtOFAxDeXgJsDvgbvBcVwCCYbqp6LZZgB7P+QNNLzOMF1QVeIdqLhhfE5LW6S5rS DZxiswDdnV//EPkBOM8jKtQuUrHKwoDEmelxOiuVZAAAAAAAAA== --------------ms020103050803010300020803-- From owner-freebsd-fs@freebsd.org Mon Nov 20 20:08:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C4A14DF4513 for ; Mon, 20 Nov 2017 20:08:14 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [67.205.158.196]) by mx1.freebsd.org (Postfix) with ESMTP id 9C5FB6B149 for ; Mon, 20 Nov 2017 20:08:14 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id C243627461 for ; Mon, 20 Nov 2017 15:08:14 -0500 (EST) Received: from [192.168.10.23] (D13.Denninger.Net [192.168.10.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id E1E053A1C00 for ; Mon, 20 Nov 2017 14:08:12 -0600 (CST) Subject: Re: MSDOS Filesystem question related to "read-only" files To: freebsd-fs@freebsd.org References: From: Karl Denninger Message-ID: <0e31a0cf-8a97-1d63-8ebb-9c61d22539cb@denninger.net> Date: Mon, 20 Nov 2017 14:08:12 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms000106040502070703060000" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 20:08:14 -0000 This is a cryptographically signed message in MIME format. --------------ms000106040502070703060000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Oh, my apologies -- I didn't include the version: FreeBSD 11.0-STABLE #0 r313159M: Fri Feb=C2=A0 3 09:58:35 CST 2017 On an RPI2, but I doubt that matters. On 11/20/2017 14:02, Karl Denninger wrote: > root@Test-MCP:/home/karl/HD-MCP # cd /mnt > root@Test-MCP:/mnt # mount > /dev/ufs/rootfs on / (ufs, local, noatime, soft-updates, nfsv4acls) > devfs on /dev (devfs, local) > tmpfs on /tmp (tmpfs, local) > /dev/da0s1 on /mnt (msdosfs, local) > root@Test-MCP:/mnt # ls -al > total 408 > drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Jan=C2= =A0 1=C2=A0 1980 . > drwxr-xr-x=C2=A0 20 root=C2=A0=C2=A0 wheel=C2=A0=C2=A0=C2=A0=C2=A0 512 = Nov 20 16:04 .. > drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Nov 1= 9 11:20 System Volume Information > -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 127979 Nov 19 22:54 > cam2-2017-11-19-22-54-47.jpg > -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 271295 Nov 19 22:57 > cam2-2017-11-19-22-57-20.jpg > root@Test-MCP:/mnt # chmod u-w * > root@Test-MCP:/mnt # ls -al > total 408 > drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Jan=C2= =A0 1=C2=A0 1980 . > drwxr-xr-x=C2=A0 20 root=C2=A0=C2=A0 wheel=C2=A0=C2=A0=C2=A0=C2=A0 512 = Nov 20 16:04 .. > drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Nov 1= 9 11:20 System Volume Information > -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 127979 Nov 19 22:54 > cam2-2017-11-19-22-54-47.jpg > -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 271295 Nov 19 22:57 > cam2-2017-11-19-22-57-20.jpg > root@Test-MCP:/mnt # > > Nope.=C2=A0 The "w" is still there. > > No error returned from the "chmod" command (or if I call it from a C > program) but the file mode is NOT changed whether I'm doing it as the > superuser or as the owner of the file (and directory) > > On 11/20/2017 12:30, Conrad Meyer wrote: >> Hi Karl, >> >> In fact, msdosfs in FreeBSD should set the FAT READONLY attribute >> under two conditions: >> >> 1. The owner chmod's the file non-writeable (chmod u-w) (what you've >> described, I think). Or, >> 2. The super user or otherwise privileged user sets the "readonly" >> flag on the file via chflags(1). >> >> How have you determined that chmod u-w does nothing? What version of >> FreeBSD are you using? >> >> Best, >> Conrad >> >> >> On Mon, Nov 20, 2017 at 7:36 AM, Karl Denninger w= rote: >>> I'm running into an interesting issue here and wondering if there's a= >>> way to do this under FreeBSD. >>> >>> MSDOS filesystems have a "primitive" permission capability; >>> specifically, they can have a "Read-only" attribute on a file. It lo= oks >>> like OpenBSD supports this from reading their man pages. >>> >>> FreeBSD doesn't appear to. When you mount a msdos filesystem (e.g. a= >>> USB stick) whoever owns the parent directory where you mount it gives= >>> you the permissions and "ownership" of files on said filesystem. All= >>> good so far. But attempting to chmod a file to remove write permissi= on >>> "succeeds" (returns success) but does nothing. >>> >>> Is this capability simply not present on FreeBSD? I'm interested in >>> using it as a means of "flagging" files on a USB stick in an applicat= ion >>> that I do not want to remove if the stick fills (basically, to "prote= ct" >>> them from being aged off) and it appears there's no way to do it, oth= er >>> than to use something unique in the filename that I would then have t= o >>> pay attention to. >>> >>> -- >>> Karl Denninger >>> karl@denninger.net >>> /The Market Ticker/ >>> /[S/MIME encrypted email preferred]/ --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms000106040502070703060000 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTcxMTIwMjAwODEy WjBPBgkqhkiG9w0BCQQxQgRAv2h7BCm/lFN96O06+ulTOMu4JkJrLsWf3wFCvpregV+Wlknk 9vKDyRCcd+H7l4yvQxx6HcqUsq7AXC1zjgofATBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgB7E6yiudP3VZPVycKjjSZiBi6/iZNWjtML/FADBnRywcDrDE5mns0aqqt47JYNLnil /Xn8IY7Uy4X3I7qopxzc4ZnxzPj+t7TTHQ3zb3nAyrVUl76sXgsKwhGp4nNfRhTC7U3C4Hfl mYFjQh1IA+Xcg7xK67bUlmiVng9Tx55lKizdxJojSbDD601n3SY/yEURAvqlAECGp3PdgyZk ePLjiaJhjkxqptnBaUI/C20BDf5cqMBmKLHTQuVWTY8vSkb5AL2jsdx3e+uyh+c4ohCrp7+c avKxfpVyGq7NEev+tQutL4VoJsO6djxybjYzqHQN+v5MoBUVCfxXdzVL1m2+9mgavLu11Fb3 59g2Yuw8KD+Tn8aOmycKYSp+ahpXl04scFXhbfF84EuVANKsJXkTj+1LJSeXHoKpHViipiEY +H6iqDmzJiiLKB77q9N5pV/knKb5O3uM0fEbLmbYJ71pjjd4Fcf3K9Mj9WAaynNMUnPDvjE1 w81LK8zRzoI8nvTr5t44M+s1A0ipRQykmxgMMmIb1gWK2I6RhTxFm9HbrLukzENXoan+R01t Sm+MuTqqx1p174W0L5faiyNCv6r0lnRaFPTlPyTpRvxuvOoA3K4zLIZ3O5DlsivKOUjsAlI2 SPg+O6aZb41aoPjJvVBmd3tnTUV4Uc3BvLQckJOI9QAAAAAAAA== --------------ms000106040502070703060000-- From owner-freebsd-fs@freebsd.org Mon Nov 20 20:22:18 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 795E6DF48BB for ; Mon, 20 Nov 2017 20:22:18 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [67.205.158.196]) by mx1.freebsd.org (Postfix) with ESMTP id 5001E6B97F for ; Mon, 20 Nov 2017 20:22:17 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id E0A1B27461 for ; Mon, 20 Nov 2017 15:22:17 -0500 (EST) Received: from [192.168.10.23] (D13.Denninger.Net [192.168.10.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 2C79C3A1C5F for ; Mon, 20 Nov 2017 14:22:15 -0600 (CST) Subject: Re: MSDOS Filesystem question related to "read-only" files To: freebsd-fs@freebsd.org References: <0e31a0cf-8a97-1d63-8ebb-9c61d22539cb@denninger.net> From: Karl Denninger Message-ID: <501b5bca-a4b0-7bd9-c1bd-09de0394e136@denninger.net> Date: Mon, 20 Nov 2017 14:22:15 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <0e31a0cf-8a97-1d63-8ebb-9c61d22539cb@denninger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms070700060304010605090001" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 20:22:18 -0000 This is a cryptographically signed message in MIME format. --------------ms070700060304010605090001 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Oh, here's another interesting anomaly -- MSDOS filesystems return "0" for the free file count! (gdb) print fsbuf $3 =3D {f_version =3D 537068824, f_type =3D 50, f_flags =3D 4096, f_bsize= =3D 4096, =C2=A0 f_iosize =3D 4096, f_blocks =3D 1949442, f_bfree =3D 1949436, f_ba= vail =3D 1949436, =C2=A0 f_files =3D 0, f_ffree =3D 0, f_syncwrites =3D 3, f_asyncwrites =3D= 5, =C2=A0 f_syncreads =3D 1907, f_asyncreads =3D 0, f_spare =3D 0xbf3fd628, = f_namemax =3D 255, =C2=A0 f_owner =3D 0, f_fsid =3D {val =3D 0xbf3fd680}, f_charspare =3D 0x= bf3fd688 "", =C2=A0 f_fstypename =3D 0xbf3fd6d8 "msdosfs", =C2=A0 f_mntfromname =3D 0xbf3fd6e8 "/dev/da0s1", f_mntonname =3D 0xbf3fd= 740 "/mnt"} Needless to say that can cause some interesting issues if you're coding to avoid potential inode exhaustion and the target is a MSDOS filesystem..... FAT filesystems DO have a maximum file count, incidentall= y. On 11/20/2017 14:08, Karl Denninger wrote: > Oh, my apologies -- I didn't include the version: > > FreeBSD 11.0-STABLE #0 r313159M: Fri Feb=C2=A0 3 09:58:35 CST 2017 > > On an RPI2, but I doubt that matters. > > > On 11/20/2017 14:02, Karl Denninger wrote: >> root@Test-MCP:/home/karl/HD-MCP # cd /mnt >> root@Test-MCP:/mnt # mount >> /dev/ufs/rootfs on / (ufs, local, noatime, soft-updates, nfsv4acls) >> devfs on /dev (devfs, local) >> tmpfs on /tmp (tmpfs, local) >> /dev/da0s1 on /mnt (msdosfs, local) >> root@Test-MCP:/mnt # ls -al >> total 408 >> drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Jan=C2= =A0 1=C2=A0 1980 . >> drwxr-xr-x=C2=A0 20 root=C2=A0=C2=A0 wheel=C2=A0=C2=A0=C2=A0=C2=A0 512= Nov 20 16:04 .. >> drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Nov = 19 11:20 System Volume Information >> -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 127979 Nov 19 22:54 >> cam2-2017-11-19-22-54-47.jpg >> -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 271295 Nov 19 22:57 >> cam2-2017-11-19-22-57-20.jpg >> root@Test-MCP:/mnt # chmod u-w * >> root@Test-MCP:/mnt # ls -al >> total 408 >> drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Jan=C2= =A0 1=C2=A0 1980 . >> drwxr-xr-x=C2=A0 20 root=C2=A0=C2=A0 wheel=C2=A0=C2=A0=C2=A0=C2=A0 512= Nov 20 16:04 .. >> drwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0=C2=A0=C2=A0 4096 Nov = 19 11:20 System Volume Information >> -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 127979 Nov 19 22:54 >> cam2-2017-11-19-22-54-47.jpg >> -rwxr-xr-x=C2=A0=C2=A0 1 hdmcp=C2=A0 wheel=C2=A0 271295 Nov 19 22:57 >> cam2-2017-11-19-22-57-20.jpg >> root@Test-MCP:/mnt # >> >> Nope.=C2=A0 The "w" is still there. >> >> No error returned from the "chmod" command (or if I call it from a C >> program) but the file mode is NOT changed whether I'm doing it as the >> superuser or as the owner of the file (and directory) >> >> On 11/20/2017 12:30, Conrad Meyer wrote: >>> Hi Karl, >>> >>> In fact, msdosfs in FreeBSD should set the FAT READONLY attribute >>> under two conditions: >>> >>> 1. The owner chmod's the file non-writeable (chmod u-w) (what you've >>> described, I think). Or, >>> 2. The super user or otherwise privileged user sets the "readonly" >>> flag on the file via chflags(1). >>> >>> How have you determined that chmod u-w does nothing? What version of= >>> FreeBSD are you using? >>> >>> Best, >>> Conrad >>> >>> >>> On Mon, Nov 20, 2017 at 7:36 AM, Karl Denninger = wrote: >>>> I'm running into an interesting issue here and wondering if there's = a >>>> way to do this under FreeBSD. >>>> >>>> MSDOS filesystems have a "primitive" permission capability; >>>> specifically, they can have a "Read-only" attribute on a file. It l= ooks >>>> like OpenBSD supports this from reading their man pages. >>>> >>>> FreeBSD doesn't appear to. When you mount a msdos filesystem (e.g. = a >>>> USB stick) whoever owns the parent directory where you mount it give= s >>>> you the permissions and "ownership" of files on said filesystem. Al= l >>>> good so far. But attempting to chmod a file to remove write permiss= ion >>>> "succeeds" (returns success) but does nothing. >>>> >>>> Is this capability simply not present on FreeBSD? I'm interested in= >>>> using it as a means of "flagging" files on a USB stick in an applica= tion >>>> that I do not want to remove if the stick fills (basically, to "prot= ect" >>>> them from being aged off) and it appears there's no way to do it, ot= her >>>> than to use something unique in the filename that I would then have = to >>>> pay attention to. >>>> >>>> -- >>>> Karl Denninger >>>> karl@denninger.net >>>> /The Market Ticker/ >>>> /[S/MIME encrypted email preferred]/ --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms070700060304010605090001 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTcxMTIwMjAyMjE1 WjBPBgkqhkiG9w0BCQQxQgRAguA8qDBxjXmA5ZE6HS0LKIjYsffD6HLnz/Z9Rhc88DZHbkaZ y8rj5SHNaAfYu6/I19W6INPwyyWXVY93c4G6oTBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgB9EqeHYI0TYkKUaXKqzTuwdRiH7udx7siiHuyDvqcE8q07vqfyRSA/zMRWj758utcH mO/ffnfqyBEYmuBNqUxodGpH/+sYHEb12o98h/n9u0/zCPO3gb8J8zMV7OezpiNFjEm9PWt6 8AuYSd4DDFYSBAAzHVZLUevTQA/ohlQYxQCgV2koUeYAODXrFF0e/G8Lg0id+0Gdsw3yg2yI tGUQTyzvVZTSuxqZCH5cQ7/hrlyIcXuVzC+2jNeejIA9cZ3t26DF9mbBYPGAoPq82aORRLQQ sby2uSnII3/qkOEitr+PYOGDG7PKwF9acEMeQN+eTbFT1EXNs0OIo/7t5jNuF8UkyMFR2+RE DB4injh0yEG9jLDy5IKtRpZNYObs78KsMi7d1JhW2UKH5tAWF7CyLuFdj1qmtMXjKuIOsNut KtMntZQakh76MssojUba+Dvvs0o0SSjwACmx0qG6XQEkz3Mk0RQ4zrpmxSMw41Rf7cQQq+6/ YpTk2lC4NPqA9RRf/4G0QpfdhxnAzUhMk6xpKvc3Tzpnl3B8MBVBKPMd9mmbCBeOPujxC0wQ 0lOl78RVtlqPj3ZsUL3KlMAhSYNQN8CsQTMOmQMreAMQeRQhLYG6h1d9xwm0OLaIXFxkCceP jfVVcTeYCUK8xp+hqe2PPka/mXFcg++zKqagTspXygAAAAAAAA== --------------ms070700060304010605090001-- From owner-freebsd-fs@freebsd.org Mon Nov 20 20:34:21 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C93CDDF4C8D for ; Mon, 20 Nov 2017 20:34:21 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.com) Received: from asp.reflexion.net (outbound-mail-210-101.reflexion.net [208.70.210.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5F6D96BF8C for ; Mon, 20 Nov 2017 20:34:20 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.com) Received: (qmail 21077 invoked from network); 20 Nov 2017 20:34:19 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 20 Nov 2017 20:34:19 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.3) with SMTP; Mon, 20 Nov 2017 15:34:19 -0500 (EST) Received: (qmail 14972 invoked from network); 20 Nov 2017 20:34:19 -0000 Received: from unknown (HELO mail.quorum.net) (64.74.133.216) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 20 Nov 2017 20:34:19 -0000 Received: from QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e]) by QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e%14]) with mapi id 14.03.0351.000; Mon, 20 Nov 2017 12:34:18 -0800 From: Shiva Bhanujan To: "freebsd-fs@freebsd.org" Subject: zio_done panic in 10.3 Thread-Topic: zio_done panic in 10.3 Thread-Index: AdNiPrZNqbG0j29/RtqtjwQQWT08Kg== Date: Mon, 20 Nov 2017 20:34:17 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.20.7.98] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 20:34:22 -0000 Hello,=0A= =0A= I'm getting a kernel panic in FreeBSD 10.3 p22 (r324806M). =0A= =0A= KDB: stack backtrace:=0A= #0 0xffffffff80982fe0 at kdb_backtrace+0x60=0A= #1 0xffffffff80945cb6 at vpanic+0x126=0A= #2 0xffffffff80945b83 at panic+0x43=0A= #3 0xffffffff80d4ac6b at trap_fatal+0x36b=0A= #4 0xffffffff80d4af6d at trap_pfault+0x2ed=0A= #5 0xffffffff80d4a5ea at trap+0x47a=0A= #6 0xffffffff80d305b2 at calltrap+0x8=0A= #7 0xffffffff8094db5d at _sx_xlock+0x5d=0A= #8 0xffffffff81a4ccec at zio_done+0x92c=0A= #9 0xffffffff81a486ea at zio_execute+0x10a=0A= #10 0xffffffff80993e15 at taskqueue_run_locked+0xe5=0A= #11 0xffffffff809948a8 at taskqueue_thread_loop+0xa8=0A= #12 0xffffffff8090f13a at fork_exit+0x9a=0A= #13 0xffffffff80d30aee at fork_trampoline+0xe=0A= =0A= We are doing send/receive of zfs snapshots by piping it through mbuffer and= netcat. I can't seem to relate send/receive to the above backtrace.=0A= =0A= The closest bug that I've found w/ a panic close to the above is as follows= :=0A= =0A= https://bugs.freebsd.org/bugzilla/show_bug.cgi?format=3Dmultiple&id=3D18196= 6=0A= =0A= I haven't been able to find any commits against this backtrace. Can anybod= y please point me to anything that might have addressed this issue? pleas= e let me know if there is any additional information that you might need.= =0A= =0A= Regards,=0A= Shiva=0A= From owner-freebsd-fs@freebsd.org Mon Nov 20 20:44:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5D732DF5086 for ; Mon, 20 Nov 2017 20:44:04 +0000 (UTC) (envelope-from rde@tavi.co.uk) Received: from kipling.tavi.co.uk (kipling.tavi.co.uk [81.187.145.130]) by mx1.freebsd.org (Postfix) with ESMTP id E72826C61C for ; Mon, 20 Nov 2017 20:44:03 +0000 (UTC) (envelope-from rde@tavi.co.uk) Received: from kipling.tavi.co.uk (localhost [127.0.0.1]) by kipling.tavi.co.uk (Postfix) with ESMTP id CCC1C892C3 for ; Mon, 20 Nov 2017 20:37:27 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=tavi.co.uk; h=date:from:to :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=selector1; bh=ypCuzWr 9hobHb8y5zG9uOiU2RPo=; b=KK7lCjRTODh6BsVx4DqMdQ1svQGGb1LOeCFPTMB AQPrMX48kd6p/kFdGzaKBfY7p1QgxAY8HEAoWlHa/sDWfCfECcoqtTYHYSd9fb8z UujscyozPHMKtGvWlKGljnQi3GZMtDGGRUNJLoIBAax6gdrHXbYvcxSpTXTiq0Zk pjgA= Received: from raksha.tavi.co.uk (raksha.tavi.co.uk [81.187.145.139]) (Authenticated sender: rde@tavi.co.uk) by kipling.tavi.co.uk (Postfix) with ESMTPA id 765F0892C0 for ; Mon, 20 Nov 2017 20:37:27 +0000 (GMT) Date: Mon, 20 Nov 2017 20:37:17 +0000 From: Bob Eager To: freebsd-fs@freebsd.org Subject: Re: MSDOS Filesystem question related to "read-only" files Message-ID: <20171120203706.5ffbfd28@raksha.tavi.co.uk> In-Reply-To: <501b5bca-a4b0-7bd9-c1bd-09de0394e136@denninger.net> References: <0e31a0cf-8a97-1d63-8ebb-9c61d22539cb@denninger.net> <501b5bca-a4b0-7bd9-c1bd-09de0394e136@denninger.net> X-Mailer: Claws Mail 3.15.0 (GTK+ 2.24.31; i386-portbld-freebsd11.0) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAMFBMVEUwXjFLc0vD0cS7y7zw9PDZ4tkWSRaVrZZ+m39qi2tXfVj////7+/utwK4IPggAOAAJUUA7AAABKklEQVQ4jWPYjQMwDFYJp0NKEKCNJmEf9h8CsimXiL2e33s3/e7F7K2Cs3f3dCMkQkMKj4YuCY3K3iR+e7fMaiSjvkX0/5cFGrWpe2uLzOpaExUVqMS/8PX/Re5ey960OLBTZpFA8+IlSBKPQ92zNyUUBsosN58uIY0k8f+/ONCoYytkVuhWzVwNkYiYbqk5M3NmOVBi41YZ8RsGF7shEtFb5KJ3r969CyixM7OTPeFUxG2IxLO8/9/SvqXlc+/x3h295YzLlj2nIRJQj//nRvc5TEIal8RsXBLVuCQwIgoq/u80DomP6HEOk/iOS+IJLonZOCT+ReOQ+Lkbh0QKLonbOCR+7MYhsRqHBJrVcIl/1TgklqKLQyQ+tGKIgyQOqXpjig94diZRAgAXmDX6jyWafAAAAABJRU5ErkJggg====== MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: base64 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 20:44:04 -0000 LS0tLS1CRUdJTiBQR1AgU0lHTkVEIE1FU1NBR0UtLS0tLQ0KSGFzaDogU0hBMjU2DQoNCk9uIE1v biwgMjAgTm92IDIwMTcgMTQ6MjI6MTUgLTA2MDANCkthcmwgRGVubmluZ2VyIDxrYXJsQGRlbm5p bmdlci5uZXQ+IHdyb3RlOg0KDQo+IE9oLCBoZXJlJ3MgYW5vdGhlciBpbnRlcmVzdGluZyBhbm9t YWx5IC0tIE1TRE9TIGZpbGVzeXN0ZW1zIHJldHVybiAiMCINCj4gZm9yIHRoZSBmcmVlIGZpbGUg Y291bnQhDQo+IA0KPiAoZ2RiKSBwcmludCBmc2J1Zg0KPiAkMyA9IHtmX3ZlcnNpb24gPSA1Mzcw Njg4MjQsIGZfdHlwZSA9IDUwLCBmX2ZsYWdzID0gNDA5NiwgZl9ic2l6ZSA9DQo+IDQwOTYsIGZf aW9zaXplID0gNDA5NiwgZl9ibG9ja3MgPSAxOTQ5NDQyLCBmX2JmcmVlID0gMTk0OTQzNiwNCj4g Zl9iYXZhaWwgPSAxOTQ5NDM2LA0KPiCgIGZfZmlsZXMgPSAwLCBmX2ZmcmVlID0gMCwgZl9zeW5j d3JpdGVzID0gMywgZl9hc3luY3dyaXRlcyA9IDUsDQo+IKAgZl9zeW5jcmVhZHMgPSAxOTA3LCBm X2FzeW5jcmVhZHMgPSAwLCBmX3NwYXJlID0gMHhiZjNmZDYyOCwNCj4gZl9uYW1lbWF4ID0gMjU1 LA0KPiCgIGZfb3duZXIgPSAwLCBmX2ZzaWQgPSB7dmFsID0gMHhiZjNmZDY4MH0sIGZfY2hhcnNw YXJlID0gMHhiZjNmZDY4OA0KPiAiIiwgZl9mc3R5cGVuYW1lID0gMHhiZjNmZDZkOCAibXNkb3Nm cyIsDQo+IKAgZl9tbnRmcm9tbmFtZSA9IDB4YmYzZmQ2ZTggIi9kZXYvZGEwczEiLCBmX21udG9u bmFtZSA9IDB4YmYzZmQ3NDANCj4gIi9tbnQifQ0KPiANCj4gTmVlZGxlc3MgdG8gc2F5IHRoYXQg Y2FuIGNhdXNlIHNvbWUgaW50ZXJlc3RpbmcgaXNzdWVzIGlmIHlvdSdyZQ0KPiBjb2RpbmcgdG8g YXZvaWQgcG90ZW50aWFsIGlub2RlIGV4aGF1c3Rpb24gYW5kIHRoZSB0YXJnZXQgaXMgYSBNU0RP Uw0KPiBmaWxlc3lzdGVtLi4uLi4gRkFUIGZpbGVzeXN0ZW1zIERPIGhhdmUgYSBtYXhpbXVtIGZp bGUgY291bnQsDQo+IGluY2lkZW50YWxseS4NCg0KQnV0IHN1cmVseSBvbmx5IGluIHRoZSByb290 IGRpcmVjdG9yeT8NCi0tLS0tQkVHSU4gUEdQIFNJR05BVFVSRS0tLS0tDQoNCmlRRXpCQUVCQ0FB ZEZpRUVWZ2RJMktlVmxkUEFoVVlhS0JkZjJhejhlNmdGQWxvVFBQMEFDZ2tRS0JkZjJhejgNCmU2 Z3lKd2Y5R2xJM2xuSlJmTzhjNVZGWmE0VHNVVG1pRElNV1Yrc2pUc20yMzFiQktxb0pQNzAvbTZ1 RlhSNEoNCkpjOFNyTEhrcXRtU1pMRWYxVzJnVmhaQlJBZzZCWVU1SFBpZ2F3VThaeEJVUHZMNXRF YThxUmpqUS9XbVJtYjcNCmNCQzl0RVI2S1lSUVRvTDdzS0ttdTBFMHh6czBLT3lGeDFacGZzVlQ4 c3NOd0Y3aHdBbHZSYTNCWHYxMUdLbm8NCmVaTXBWSTBBbkxEeVU4aThNQWxFRjZGa2t6RjJHdnl4 THBjbytJLzY0VnRFN2Y3OXhlSmM2NGQ0a0pSTnBMQ1INCklkKzNwandCQTUxaXpWcmRlaXVNMGVT S2M4bmlCWmk0WThwRStXd3VYWmR0WCtDN3VjY2xUWjRjZXFSUi9tVGINClFJWTNjM0hHRFBkaFp4 cStxR0NEMnlyVjk0V280Zz09DQo9bnM2Rw0KLS0tLS1FTkQgUEdQIFNJR05BVFVSRS0tLS0tDQo= From owner-freebsd-fs@freebsd.org Mon Nov 20 20:51:16 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE4FDDF5149 for ; Mon, 20 Nov 2017 20:51:16 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-io0-f181.google.com (mail-io0-f181.google.com [209.85.223.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A0A386C7B7 for ; Mon, 20 Nov 2017 20:51:16 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-io0-f181.google.com with SMTP id q101so17119279ioi.1 for ; Mon, 20 Nov 2017 12:51:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=7JAVkcMO7vWR5Xr7yVTsYXeQKagM25elts4aWwce1Fg=; b=dOTFsLrVGZAgDea9/knyHaJrxA8IpViwjdA0yfYwpzVfMSON01ppKdOmGRW66CEvjG bBlidUPixDWFVUdrcNs/rcFetYC02hLacpQNBLAtQjZEjizs64QDGk4HZKSwTmPqANFC O0lx5s4RtZp5d30sP7jv5OkEE4ViQXL7sXZQZIyDDH5paUJ8OUZ+KGuAHupMC6Y3tuBj ye0qMG/GwwN+xo+wiu6ux2Ui4AEloOXM3J0W7TwTKzehtdkHjAbdKSbx0BMY6vtU23NX OVdwpoop3hZ7sXrCYlWeEoR8bZkIFRi8C1wWStFbYIcw3pdVzFYVWmn4rjMd7QYDoVLu GaSg== X-Gm-Message-State: AJaThX5qGfPAejKwG1L+Ccu/MozkHjVCzOorBFgoQZaBeDbJHeBK7Bny VDiGetKgkrBJR57sQV2GyNeUfhu3 X-Google-Smtp-Source: AGs4zMZXij7uzLSYCWItqnMC0cWKGvyNctaaWy3PhbU4TPYJ5RVSU2NE2fkX73NNxTNzv0mO0pQlRg== X-Received: by 10.107.81.24 with SMTP id f24mr15045396iob.63.1511211070153; Mon, 20 Nov 2017 12:51:10 -0800 (PST) Received: from mail-io0-f173.google.com (mail-io0-f173.google.com. [209.85.223.173]) by smtp.gmail.com with ESMTPSA id 196sm4843682ioe.66.2017.11.20.12.51.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Nov 2017 12:51:10 -0800 (PST) Received: by mail-io0-f173.google.com with SMTP id i38so17142624iod.2 for ; Mon, 20 Nov 2017 12:51:10 -0800 (PST) X-Received: by 10.107.7.169 with SMTP id g41mr16016976ioi.38.1511211069886; Mon, 20 Nov 2017 12:51:09 -0800 (PST) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 10.2.165.150 with HTTP; Mon, 20 Nov 2017 12:51:09 -0800 (PST) In-Reply-To: References: From: Conrad Meyer Date: Mon, 20 Nov 2017 12:51:09 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: MSDOS Filesystem question related to "read-only" files To: Karl Denninger Cc: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 20:51:17 -0000 On Mon, Nov 20, 2017 at 12:02 PM, Karl Denninger wrote: > root@Test-MCP:/mnt # ls -al > ... > -rwxr-xr-x 1 hdmcp wheel 127979 Nov 19 22:54 > cam2-2017-11-19-22-54-47.jpg > ... > root@Test-MCP:/mnt # chmod u-w * > root@Test-MCP:/mnt # ls -al > ... > -rwxr-xr-x 1 hdmcp wheel 127979 Nov 19 22:54 > cam2-2017-11-19-22-54-47.jpg > ... > Nope. The "w" is still there. > > No error returned from the "chmod" command (or if I call it from a C > program) but the file mode is NOT changed whether I'm doing it as the > superuser or as the owner of the file (and directory) Indeed, msdosfs does not reflect the READONLY attribute back to userspace in the form of the mode. That's a bug that could be fixed pretty easily: --- a/sys/fs/msdosfs/msdosfs_vnops.c +++ b/sys/fs/msdosfs/msdosfs_vnops.c @@ -287,6 +287,8 @@ msdosfs_getattr(struct vop_getattr_args *ap) vap->va_fileid = fileid; mode = S_IRWXU|S_IRWXG|S_IRWXO; + if ((dep->de_Attributes & ATTR_READONLY) != 0) + mode &= ~(S_IWUSR|S_IWGRP|S_IWOTH); vap->va_mode = mode & (ap->a_vp->v_type == VDIR ? pmp->pm_dirmask : pmp->pm_mask); vap->va_uid = pmp->pm_uid; However, despite 'ls' showing 'w', it *does* set the READONLY attribute on the file. Try invoking ls(1) with '-o' and looking for the "rdonly" or "readonly" flag. Best, Conrad From owner-freebsd-fs@freebsd.org Mon Nov 20 21:33:54 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B594DF610C for ; Mon, 20 Nov 2017 21:33:54 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [67.205.158.196]) by mx1.freebsd.org (Postfix) with ESMTP id 034DD6E29A for ; Mon, 20 Nov 2017 21:33:53 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 1BFB6274BB for ; Mon, 20 Nov 2017 16:33:53 -0500 (EST) Received: from [192.168.10.23] (D13.Denninger.Net [192.168.10.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 225803A1E3F for ; Mon, 20 Nov 2017 15:33:51 -0600 (CST) Subject: Re: MSDOS Filesystem question related to "read-only" files To: freebsd-fs@freebsd.org References: <0e31a0cf-8a97-1d63-8ebb-9c61d22539cb@denninger.net> <501b5bca-a4b0-7bd9-c1bd-09de0394e136@denninger.net> <20171120203706.5ffbfd28@raksha.tavi.co.uk> From: Karl Denninger Message-ID: <6a5701bf-7bc0-48a4-ba54-b947e3fd3416@denninger.net> Date: Mon, 20 Nov 2017 15:33:51 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20171120203706.5ffbfd28@raksha.tavi.co.uk> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms000403000401050404090909" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Nov 2017 21:33:54 -0000 This is a cryptographically signed message in MIME format. --------------ms000403000401050404090909 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Language: en-US On 11/20/2017 14:37, Bob Eager wrote: > On Mon, 20 Nov 2017 14:22:15 -0600 > Karl Denninger wrote: > > > Oh, here's another interesting anomaly -- MSDOS filesystems return "0= " > > for the free file count! > > > (gdb) print fsbuf > > $3 =3D {f_version =3D 537068824, f_type =3D 50, f_flags =3D 4096, f_b= size =3D > > 4096, f_iosize =3D 4096, f_blocks =3D 1949442, f_bfree =3D 1949436, > > f_bavail =3D 1949436, > >=C2=A0=C2=A0 f_files =3D 0, f_ffree =3D 0, f_syncwrites =3D 3, f_async= writes =3D 5, > >=C2=A0=C2=A0 f_syncreads =3D 1907, f_asyncreads =3D 0, f_spare =3D 0xb= f3fd628, > > f_namemax =3D 255, > >=C2=A0=C2=A0 f_owner =3D 0, f_fsid =3D {val =3D 0xbf3fd680}, f_charspa= re =3D 0xbf3fd688 > > "", f_fstypename =3D 0xbf3fd6d8 "msdosfs", > >=C2=A0=C2=A0 f_mntfromname =3D 0xbf3fd6e8 "/dev/da0s1", f_mntonname =3D= 0xbf3fd740 > > "/mnt"} > > > Needless to say that can cause some interesting issues if you're > > coding to avoid potential inode exhaustion and the target is a MSDOS > > filesystem..... FAT filesystems DO have a maximum file count, > > incidentally. > > But surely only in the root directory? Uh, I pulled that on "." in the root directory (the mount point.)=C2=A0 W= rong is wrong, 'yanno, although insanely large numbers of files on FAT filesystems are bad news for performance reasons anyway. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms000403000401050404090909 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTcxMTIwMjEzMzUx WjBPBgkqhkiG9w0BCQQxQgRAej7B3o/xFpiwmLjarVQXlzYgEWs550uNdjCM28IfWy+/FLhE nzR5ArNn4Hp5PEAHl1p3d6oyzvTahtaNX0iB/zBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgBKOIiQIUjssmTmsf+6KhhzTKUutOugToEk/vTuVEhxVQDTkOPmUpNIQ4ELxzrXkmSU YlgcqTvKgXqkcbvU0jY2idf/7v/e6NsNmQgAxRbtdJYZgl9OKinBFYuu/uKVektXQAr+ETlS OfZsnsdk/Oou3wKEt2W2HeGETVohLnWvBkkIafFFesGssoqJcuibmVbjxtn84P8VklziMotI JQ1IKxtmMjed0ixMyFuSSyCVrfJWvJMNV1kiGg7dA6wN96zudrX84dGkBjlmDgQFEXSRUKFF BgliiXrV8VCc/lwoqfE6leVlA90xFmE242tAY+AYvil+BCpOeLQf7sZAnIiYnPGOW5dCXeTq v/AWzO8rVBBNuaRhWIL0g45MxdnFiR2PNuV/scHcuLTPPomyloifvJktAzd/dfslah4Y/KLn JqZOFos62Gwfnv9F6WdXLUivuNZCGFB/lQhvIaC2YWGQL8ftKQPnEtWdKMifLBQJ1VVEkZfw o45PXg7fyYJy2cH9/k/7ioQUgqNw645rc7vkKkzBmtoqMb54kDEYCH0FdOuSSWnsstcKNFIs +2H7dvwxhZfEooBBq1hqxQyFl4QVJ9t5kNYUj7hkv0DfZATk54KZtWtB7S0Xdn999BdS1x2J ibsjrqTWsECHBcd3YINIv6C9FHnj6TgmSqEZLRiEKQAAAAAAAA== --------------ms000403000401050404090909-- From owner-freebsd-fs@freebsd.org Tue Nov 21 09:49:11 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8743CDE923C for ; Tue, 21 Nov 2017 09:49:11 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f42.google.com (mail-lf0-f42.google.com [209.85.215.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0CBB5670F8 for ; Tue, 21 Nov 2017 09:49:10 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f42.google.com with SMTP id d10so2291968lfj.7 for ; Tue, 21 Nov 2017 01:49:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=dx1dDGhVOTVSSjJEL29QblZLuPTIf2QNp+1bl6pav8A=; b=KNQyqtxgl14yHefkvhgL7RC1mubdXZqJjpPm4+Q71NKbUE6Tt9wXHuWoMi+P3hqj7v sZPV8FQ1pQ+d+Ne6Jwp1MvKE4srvU2ie9tjunoJvHMnXhGISE8Pftv9LXiCKRN3quqyR MZJdKKSBJT4jmRVJNZR38AgpdYo14KmJn70KA694ZWXbtSAbosyV9sRHcCDzPMUNfSK7 ztWBjIvhtjS1ybaTMw5vGpnLvY0fm9R4ljNbeR8WAT22VE7NGZYjoyBQTPG8gNjyphrX xviCoTqdpPw4eUrann3tgGFkMIpGKW6eNQzzlK1bxB7tOs2CY/r9SzlA0dQRvA3TlX89 vfyA== X-Gm-Message-State: AJaThX4nBfsCIrXb3h7usNBK5Klvf0Thf5uUwCLxM+F5yySIlj8uBzo/ AsANH0L3tKVev9l46nMQIqKJh6stbvg= X-Google-Smtp-Source: AGs4zMYT1Sl4WVSiBQ7v7tI2DrqmVycUHrgROHrCZLUtJLkq/kdbTDIV3C7YKmAcVvmhnFjXt3ok0A== X-Received: by 10.25.87.147 with SMTP id l141mr197151lfb.111.1511257743133; Tue, 21 Nov 2017 01:49:03 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id 17sm3045911ljh.73.2017.11.21.01.49.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 Nov 2017 01:49:02 -0800 (PST) Subject: Re: zio_done panic in 10.3 To: Shiva Bhanujan , "freebsd-fs@freebsd.org" References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> From: Andriy Gapon Message-ID: <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> Date: Tue, 21 Nov 2017 11:49:01 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2017 09:49:11 -0000 On 20/11/2017 22:34, Shiva Bhanujan wrote: > Hello, > > I'm getting a kernel panic in FreeBSD 10.3 p22 (r324806M). > > KDB: stack backtrace: > #0 0xffffffff80982fe0 at kdb_backtrace+0x60 > #1 0xffffffff80945cb6 at vpanic+0x126 > #2 0xffffffff80945b83 at panic+0x43 > #3 0xffffffff80d4ac6b at trap_fatal+0x36b > #4 0xffffffff80d4af6d at trap_pfault+0x2ed > #5 0xffffffff80d4a5ea at trap+0x47a > #6 0xffffffff80d305b2 at calltrap+0x8 > #7 0xffffffff8094db5d at _sx_xlock+0x5d > #8 0xffffffff81a4ccec at zio_done+0x92c > #9 0xffffffff81a486ea at zio_execute+0x10a > #10 0xffffffff80993e15 at taskqueue_run_locked+0xe5 > #11 0xffffffff809948a8 at taskqueue_thread_loop+0xa8 > #12 0xffffffff8090f13a at fork_exit+0x9a > #13 0xffffffff80d30aee at fork_trampoline+0xe > > We are doing send/receive of zfs snapshots by piping it through mbuffer and netcat. I can't seem to relate send/receive to the above backtrace. > > The closest bug that I've found w/ a panic close to the above is as follows: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?format=multiple&id=181966 > > I haven't been able to find any commits against this backtrace. Can anybody please point me to anything that might have addressed this issue? please let me know if there is any additional information that you might need. Do you have a crash dump (vmcore) ? If not, could you please get one? -- Andriy Gapon From owner-freebsd-fs@freebsd.org Tue Nov 21 11:05:46 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 00C77DEAB79 for ; Tue, 21 Nov 2017 11:05:46 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail106.syd.optusnet.com.au (mail106.syd.optusnet.com.au [211.29.132.42]) by mx1.freebsd.org (Postfix) with ESMTP id 8A75D69478 for ; Tue, 21 Nov 2017 11:05:45 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.103] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail106.syd.optusnet.com.au (Postfix) with ESMTPS id 6E26A3C8584; Tue, 21 Nov 2017 22:05:36 +1100 (AEDT) Date: Tue, 21 Nov 2017 22:05:35 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Karl Denninger cc: "freebsd-fs@freebsd.org" Subject: Re: MSDOS Filesystem question related to "read-only" files In-Reply-To: Message-ID: <20171121213516.G1948@besplex.bde.org> References: MIME-Version: 1.0 X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=cK6QihWN c=1 sm=1 tr=0 a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=3GLR1-XzZKwA:10 a=nlC_4_pT8q9DhB4Ho9EA:9 a=wmKrrLTnj3UN5rPz8N8A:9 a=45ClL6m2LaAA:10 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2017 11:05:46 -0000 On Mon, 20 Nov 2017, Karl Denninger wrote: > I'm running into an interesting issue here and wondering if there's a > way to do this under FreeBSD. > > MSDOS filesystems have a "primitive" permission capability; > specifically, they can have a "Read-only" attribute on a file.=C2=A0 It l= ooks > like OpenBSD supports this from reading their man pages. FreeBSD used to support this as a permissions bit, but was changed to suppo= rt it as only an attribute (except for a buggy write-only affect on permission= s) See r254627 and my reply to the commit mail for r326031. > FreeBSD doesn't appear to.=C2=A0 When you mount a msdos filesystem (e.g. = a > USB stick) whoever owns the parent directory where you mount it gives > you the permissions and "ownership" of files on said filesystem.=C2=A0 Al= l > good so far.=C2=A0 r254627 changed it to do this. This is too simple. > But attempting to chmod a file to remove write permission > "succeeds" (returns success) but does nothing. It actually changes the attribute to ATTR_READONLY, but this is write only in current versions of FreeBSD. The change becomes active on reboot to som= e other OS's (including FreeBSD before r254627). r326031 unimproves this by reporting in struct stat that the file is read-only although it is still writeable. > Is this capability simply not present on FreeBSD?=C2=A0 I'm interested in > using it as a means of "flagging" files on a USB stick in an application > that I do not want to remove if the stick fills (basically, to "protect" > them from being aged off) and it appears there's no way to do it, other > than to use something unique in the filename that I would then have to > pay attention to. Immutable or nounlink flags would work better for this, but are unavailable for msdosfs. To prevent removal by rm -rf, there is nothing better than making the readonly attribute affect the permissions again, but a special application can handle this better by checking the attributes directly (the available ones depend on the file systems). Note that ordinary permissions don't affect root, so you have to be careful with rm -rf anyway. Perhaps the readonly attribute should be handled as an immutable flag. msdosfs also has a SYSTEM attribute which is even closer to immutability. MSDOS and Windows use HIDDEN | READONLY | SYSTEM for files that it doesn't want anyone to know about -- these are sort of immutable. But the SYSTEM attribute is currently treated even more simply than READONLY. It is purely write-only in the kernel, except applications can read it to use it for anything. It is used mainly by cp(1) to preserve it. You could put ffs on the USB stick and use its UF_READONLY as a hint. It doesn't affect permissions in ffs, so it doesn't prevent rm -rf, just like in msdosfs, but a application with special handling of it would work the same for both. ffs just doesn't need this because it has immutable flags that work better. Bruce From owner-freebsd-fs@freebsd.org Tue Nov 21 13:26:08 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 334BEDEECE2 for ; Tue, 21 Nov 2017 13:26:08 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.com) Received: from asp.reflexion.net (outbound-mail-210-101.reflexion.net [208.70.210.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D3E6B6E8B0 for ; Tue, 21 Nov 2017 13:26:06 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.com) Received: (qmail 28880 invoked from network); 21 Nov 2017 13:19:20 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 21 Nov 2017 13:19:20 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v8.40.3) with SMTP; Tue, 21 Nov 2017 08:19:20 -0500 (EST) Received: (qmail 18451 invoked from network); 21 Nov 2017 13:19:20 -0000 Received: from unknown (HELO mail.quorum.net) (64.74.133.216) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 21 Nov 2017 13:19:20 -0000 Received: from QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e]) by QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e%14]) with mapi id 14.03.0351.000; Tue, 21 Nov 2017 05:19:19 -0800 From: Shiva Bhanujan To: Andriy Gapon , "freebsd-fs@freebsd.org" Subject: RE: zio_done panic in 10.3 Thread-Topic: zio_done panic in 10.3 Thread-Index: AdNiPrZNqbG0j29/RtqtjwQQWT08KgAsk0+A//+0h/c= Date: Tue, 21 Nov 2017 13:19:17 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local> References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local>, <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> In-Reply-To: <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [50.201.188.3] Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2017 13:26:08 -0000 the vmcore file is =7E4G. can you please tell me how to deliver this to = you? From: Andriy Gapon =5Bavg=40FreeBSD.org=5D Sent: Tuesday, November 21, 2017 1:49 AM To: Shiva Bhanujan; freebsd-fs=40freebsd.org Subject: Re: zio_done panic in 10.3 On 20/11/2017 22:34, Shiva Bhanujan wrote: > Hello, >=20 > I'm getting a kernel panic in FreeBSD 10.3 p22 (r324806M).=20 >=20 > KDB: stack backtrace: > =230 0xffffffff80982fe0 at kdb_backtrace+0x60 > =231 0xffffffff80945cb6 at vpanic+0x126 > =232 0xffffffff80945b83 at panic+0x43 > =233 0xffffffff80d4ac6b at trap_fatal+0x36b > =234 0xffffffff80d4af6d at trap_pfault+0x2ed > =235 0xffffffff80d4a5ea at trap+0x47a > =236 0xffffffff80d305b2 at calltrap+0x8 > =237 0xffffffff8094db5d at _sx_xlock+0x5d > =238 0xffffffff81a4ccec at zio_done+0x92c > =239 0xffffffff81a486ea at zio_execute+0x10a > =2310 0xffffffff80993e15 at taskqueue_run_locked+0xe5 > =2311 0xffffffff809948a8 at taskqueue_thread_loop+0xa8 > =2312 0xffffffff8090f13a at fork_exit+0x9a > =2313 0xffffffff80d30aee at fork_trampoline+0xe >=20 > We are doing send/receive of zfs snapshots by piping it through mbuffer = and netcat. I can't seem to relate send/receive to the above backtrace. >=20 > The closest bug that I've found w/ a panic close to the above is as = follows: >=20 > = https://bugs.freebsd.org/bugzilla/show_bug.cgi?format=3Dmultiple&id=3D181966 >=20 > I haven't been able to find any commits against this backtrace. Can = anybody please point me to anything that might have addressed this issue? = please let me know if there is any additional information that you might = need. Do you have a crash dump (vmcore) ? If not, could you please get one? --=20 Andriy Gapon From owner-freebsd-fs@freebsd.org Tue Nov 21 17:09:56 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 608F9DF45D7 for ; Tue, 21 Nov 2017 17:09:56 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-io0-f169.google.com (mail-io0-f169.google.com [209.85.223.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 310E3766C2; Tue, 21 Nov 2017 17:09:55 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-io0-f169.google.com with SMTP id q64so7337856iof.13; Tue, 21 Nov 2017 09:09:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=HJFzPH8wjckvP6/8gOOStTX0Mx/HvHmVY+kjLTYN3SU=; b=eZN4zl0NZCbeKgpm6c+UZKqXBfaCUudyR4k7gg+i579u0TwJvi5eCaWKkQ/aZ6PdPa feJfSaCWAkceWHBmSBQ2kRa9ucaFiHiOVjoqHHWJ1wfe924V0U3vnF6hKN0YfLjU9jBA PW733crSft6y2ooim0s3qJTuKxcwU6JwLd8oSaVodW2mvjNXwhlOJWfMOwGZiYGUkyRG HpwIaqZXnRT9oDEotGRVTET9SoPS3cpYSuzHx2XJgD1LuEV9tsNQDkwIdCQdC5mYKFVE OJn88qtXCldE10qqwNs33mnw++QLVQzbvlsfts4E6OUpD+ftXn5Y/7/iygUloVU+8Wtm L4+A== X-Gm-Message-State: AJaThX5lWEchK9ShUsObUXJutahr/fboqnnu6zWLnG9fXeVlhldxZpKE fhy6Y5AEWgpGCAXx7IvweDAbcyRL X-Google-Smtp-Source: AGs4zManfgSrCuwS2TDY3F6ReAVsU60AXxWagj6iMrshucNWQdZdP0/nnM877kd8y/TELZCRuYCgpw== X-Received: by 10.107.25.18 with SMTP id 18mr19115258ioz.11.1511283842354; Tue, 21 Nov 2017 09:04:02 -0800 (PST) Received: from mail-it0-f43.google.com (mail-it0-f43.google.com. [209.85.214.43]) by smtp.gmail.com with ESMTPSA id g93sm2891053ioj.51.2017.11.21.09.04.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 Nov 2017 09:04:02 -0800 (PST) Received: by mail-it0-f43.google.com with SMTP id x28so2819123ita.0; Tue, 21 Nov 2017 09:04:02 -0800 (PST) X-Received: by 10.36.78.73 with SMTP id r70mr2940614ita.75.1511283842029; Tue, 21 Nov 2017 09:04:02 -0800 (PST) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 10.2.165.150 with HTTP; Tue, 21 Nov 2017 09:04:01 -0800 (PST) In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local> References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local> From: Conrad Meyer Date: Tue, 21 Nov 2017 09:04:01 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: zio_done panic in 10.3 To: Shiva Bhanujan Cc: Andriy Gapon , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2017 17:09:56 -0000 Have you tried compressing it with e.g. xz or zstd? On Tue, Nov 21, 2017 at 5:19 AM, Shiva Bhanujan wrote: > the vmcore file is ~4G. can you please tell me how to deliver this to you? > > > > From: Andriy Gapon [avg@FreeBSD.org] > Sent: Tuesday, November 21, 2017 1:49 AM > To: Shiva Bhanujan; freebsd-fs@freebsd.org > Subject: Re: zio_done panic in 10.3 > > > On 20/11/2017 22:34, Shiva Bhanujan wrote: >> Hello, >> >> I'm getting a kernel panic in FreeBSD 10.3 p22 (r324806M). >> >> KDB: stack backtrace: >> #0 0xffffffff80982fe0 at kdb_backtrace+0x60 >> #1 0xffffffff80945cb6 at vpanic+0x126 >> #2 0xffffffff80945b83 at panic+0x43 >> #3 0xffffffff80d4ac6b at trap_fatal+0x36b >> #4 0xffffffff80d4af6d at trap_pfault+0x2ed >> #5 0xffffffff80d4a5ea at trap+0x47a >> #6 0xffffffff80d305b2 at calltrap+0x8 >> #7 0xffffffff8094db5d at _sx_xlock+0x5d >> #8 0xffffffff81a4ccec at zio_done+0x92c >> #9 0xffffffff81a486ea at zio_execute+0x10a >> #10 0xffffffff80993e15 at taskqueue_run_locked+0xe5 >> #11 0xffffffff809948a8 at taskqueue_thread_loop+0xa8 >> #12 0xffffffff8090f13a at fork_exit+0x9a >> #13 0xffffffff80d30aee at fork_trampoline+0xe >> >> We are doing send/receive of zfs snapshots by piping it through mbuffer and netcat. I can't seem to relate send/receive to the above backtrace. >> >> The closest bug that I've found w/ a panic close to the above is as follows: >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?format=multiple&id=181966 >> >> I haven't been able to find any commits against this backtrace. Can anybody please point me to anything that might have addressed this issue? please let me know if there is any additional information that you might need. > > Do you have a crash dump (vmcore) ? > If not, could you please get one? > > -- > Andriy Gapon > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Nov 21 18:53:17 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1E07DF6BB2 for ; Tue, 21 Nov 2017 18:53:17 +0000 (UTC) (envelope-from daniel@versatushpc.com.br) Received: from mail-qk0-x229.google.com (mail-qk0-x229.google.com [IPv6:2607:f8b0:400d:c09::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C5817B910 for ; Tue, 21 Nov 2017 18:53:17 +0000 (UTC) (envelope-from daniel@versatushpc.com.br) Received: by mail-qk0-x229.google.com with SMTP id w125so13462031qkb.6 for ; Tue, 21 Nov 2017 10:53:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=versatushpc-com-br.20150623.gappssmtp.com; s=20150623; h=message-id:subject:from:to:date:mime-version :content-transfer-encoding; bh=rASIKlKoldpIca3QXVGAlG6BmFCTlm4xbvIzCWp26Fs=; b=c08kqJA3gueRI7j9wZylsZWcfj8ybAA2MjEbawQsESP69rrgbgIEEioTF+fjgGljIU JRR34adiNpLTo9/Q4p1OfLN9x0Sa6J02vmHSSzsE+5fo+QySVr4Gb7D5eCJ2OqAVxOv2 NUIEwnJpSJ0l+Y+xcCu/rQZXwSQLXAZr4jM3skpqCkm8m9sT1woTgwsC4eGEio23hWdY V8aarZA5T39DeFQXxP3mSvs3t2f00S1JQsmo8sgQPpEy9XJ4dPXERyR+LstSRco5csvY SPZ12cOuaMwP9aVfc8uRh6aE8/dtkHFao8BG2vBnqkEmWWqOe8WnxjDQ7p+JNAkVoCRi GnTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:date:mime-version :content-transfer-encoding; bh=rASIKlKoldpIca3QXVGAlG6BmFCTlm4xbvIzCWp26Fs=; b=uVKUOKkJZTdk3xZtbP1jbjaAdIJW1Jdm4p43C2zBRk54BlkDryCw2MXU8V32jwd3Hp 0FfID5JRs8Co9LoBUJtDGCNWemn8YvwvW39oHqMHqMVTt4Cn2jnRNroJF54X5z3SHgr0 wW6O5PmEhd1q0uV2t1tPFZofvoLeeI8fFmJgC6syqlHq5jVaGj1LvSAkr55r2bQiOd3F DjoxFSzb6ZytGqdIXawAsOf9/55vj3DcUbD3i7qWPbTN1NrSQB7CgJMr3By4AygKUPAp Y/0Bz09hI3Za9uQT15VKoOmYRQpuXCz3yPpevoPPdQCeKDtpl2B4x6U3Tcgq3ZwT8QBr 2MsQ== X-Gm-Message-State: AJaThX5k2vxINW2dx8bd0bvjeQiMb6moZAOkVqbGrDxImqx87NZzK0IZ tx66GWZ0cLCS/JHfp3iJeVNgv045Vls= X-Google-Smtp-Source: AGs4zMZEdHT3giClVBiRFrnSC2OJGSUVDwkv9Di83InvMnA36Xp8wpXYwD3ZkBQ03USF6CV98+NZeQ== X-Received: by 10.55.186.71 with SMTP id k68mr5805509qkf.58.1511290396152; Tue, 21 Nov 2017 10:53:16 -0800 (PST) Received: from dhilst-thinkpad ([179.228.138.121]) by smtp.gmail.com with ESMTPSA id 201sm9107215qkh.87.2017.11.21.10.53.13 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 10:53:14 -0800 (PST) Message-ID: <1511290391.2569.35.camel@versatushpc.com.br> Subject: Restricting zfs metadata view for non-root users. From: Daniel Hilst Selli To: freebsd-fs@freebsd.org Date: Tue, 21 Nov 2017 16:53:11 -0200 Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.24.6 (3.24.6-1.fc26) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2017 18:53:17 -0000 Hi everybody! I was testing zfs delegated administration [1]. I see that user without permissions couldn't read data from datasets but they still can read all the zfs metadata. Is this right? In my setup I have two users, foo and bar. They both have it's own datasets. As foo I can't read bar's snapshots, but he can list them. Is there a way to restrict metadata shown to one user? I don't want to expose snapshots from one user to another but still want they to be able to do their own backups by sending snapshots to this host. Regards! Daniel, [1]https://www.freebsd.org/doc/handbook/zfs-zfs-allow.html From owner-freebsd-fs@freebsd.org Tue Nov 21 19:30:56 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BA3A4DF76F4 for ; Tue, 21 Nov 2017 19:30:56 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.com) Received: from asp.reflexion.net (outbound-mail-210-143.reflexion.net [208.70.210.143]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A6537CF5E for ; Tue, 21 Nov 2017 19:30:55 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.com) Received: (qmail 28898 invoked from network); 21 Nov 2017 19:30:48 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 21 Nov 2017 19:30:48 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v8.40.3) with SMTP; Tue, 21 Nov 2017 14:30:48 -0500 (EST) Received: (qmail 9304 invoked from network); 21 Nov 2017 19:30:48 -0000 Received: from unknown (HELO mail.quorum.net) (64.74.133.216) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 21 Nov 2017 19:30:48 -0000 Received: from QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e]) by QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e%14]) with mapi id 14.03.0351.000; Tue, 21 Nov 2017 11:30:47 -0800 From: Shiva Bhanujan To: "cem@freebsd.org" CC: Andriy Gapon , "freebsd-fs@freebsd.org" , Shiva Bhanujan Subject: RE: zio_done panic in 10.3 Thread-Topic: zio_done panic in 10.3 Thread-Index: AdNiPrZNqbG0j29/RtqtjwQQWT08KgAsk0+A//+0h/eAAMUCgP//oow5 Date: Tue, 21 Nov 2017 19:30:46 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D636@QLEXC01.Quorum.local> References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.20.7.84] Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2017 19:30:56 -0000 it did get compressed to 0.5G - still too big to send via email. I did = send some more debug information by running kgdb on the core file to = Andriy, and I'm waiting for any analysis that he might provide. From: Conrad Meyer =5Bcem=40freebsd.org=5D Sent: Tuesday, November 21, 2017 9:04 AM To: Shiva Bhanujan Cc: Andriy Gapon; freebsd-fs=40freebsd.org Subject: Re: zio_done panic in 10.3 Have you tried compressing it with e.g. xz or zstd? On Tue, Nov 21, 2017 at 5:19 AM, Shiva Bhanujan wrote: > the vmcore file is =7E4G. can you please tell me how to deliver this to = you? > > > > From: Andriy Gapon =5Bavg=40FreeBSD.org=5D > Sent: Tuesday, November 21, 2017 1:49 AM > To: Shiva Bhanujan;=20 freebsd-fs=40freebsd.org > Subject: Re: zio_done panic in 10.3 > > > On 20/11/2017 22:34, Shiva Bhanujan wrote: >> Hello, >> >> I'm getting a kernel panic in FreeBSD 10.3 p22 (r324806M). >> >> KDB: stack backtrace: >> =230 0xffffffff80982fe0 at kdb_backtrace+0x60 >> =231 0xffffffff80945cb6 at vpanic+0x126 >> =232 0xffffffff80945b83 at panic+0x43 >> =233 0xffffffff80d4ac6b at trap_fatal+0x36b >> =234 0xffffffff80d4af6d at trap_pfault+0x2ed >> =235 0xffffffff80d4a5ea at trap+0x47a >> =236 0xffffffff80d305b2 at calltrap+0x8 >> =237 0xffffffff8094db5d at _sx_xlock+0x5d >> =238 0xffffffff81a4ccec at zio_done+0x92c >> =239 0xffffffff81a486ea at zio_execute+0x10a >> =2310 0xffffffff80993e15 at taskqueue_run_locked+0xe5 >> =2311 0xffffffff809948a8 at taskqueue_thread_loop+0xa8 >> =2312 0xffffffff8090f13a at fork_exit+0x9a >> =2313 0xffffffff80d30aee at fork_trampoline+0xe >> >> We are doing send/receive of zfs snapshots by piping it through mbuffer = and netcat. I can't seem to relate send/receive to the above backtrace. >> >> The closest bug that I've found w/ a panic close to the above is as = follows: >> >>=20 https://bugs.freebsd.org/bugzilla/show_bug.cgi?format=3Dmultiple&id=3D181966 >> >> I haven't been able to find any commits against this backtrace. Can = anybody please point me to anything that might have addressed this issue? = please let me know if there is any additional information that you might = need. > > Do you have a crash dump (vmcore) ? > If not, could you please get one? > > -- > Andriy Gapon > > > _______________________________________________ >=20 freebsd-fs=40freebsd.org mailing list >=20 https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to = =22freebsd-fs-unsubscribe=40freebsd.org=22 From owner-freebsd-fs@freebsd.org Tue Nov 21 20:52:27 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8A6BFD945BB for ; Tue, 21 Nov 2017 20:52:27 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f48.google.com (mail-lf0-f48.google.com [209.85.215.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1907C7FA19; Tue, 21 Nov 2017 20:52:26 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f48.google.com with SMTP id f134so15686342lfg.8; Tue, 21 Nov 2017 12:52:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=xC5r2kZ5PK/h/uUxzN6KswOYCN5NAmIe2mXwX0l9m2Q=; b=B930NR8qM3oyfzILeSeFpKri3flmLOwfbeTH88hnHWuldPsr1A7l6ExMbB9KbQRV/J dFaMZFWqjTRDh6hKd1feFJ/HZpb4MI2zow1e2bM5msHOPwl/rujDiyFxnAEymF5bVQhO JxegVFgVN5bElI0kXCEOh4vFFAFxBtkGiG+WxBA6b0I+vZs6MyojHcmVSlcYpRKH3QgJ rXGZTSLfjrhBJaK+W5if1sX8O8K+L1Ox62nBmZIkYD6S8+9MyZVfVBAAx44yUBgJqm8N fYyAJmmzLF3i0bmcmSMdo8qEvJQLIpiosD0XV8B+gnDd4tW5+r02wkFveZURtP/R/aiJ pmIA== X-Gm-Message-State: AJaThX7uT2FdjhQPAU4ACYyjKZyZ2Ltjb7ZTe25kSVqb5TP0MKXD4nar Q0XEY2GvVj3iaKRcFr7Gvx0qPwiea0g= X-Google-Smtp-Source: AGs4zMYO0id+br4hcCet8Gfl363LqNCYGpRmPwLesGk5sh/FREO+zUskPlXZ4ZGbnHD0mHFz7TqqeA== X-Received: by 10.46.32.230 with SMTP id g99mr4869013lji.147.1511297184676; Tue, 21 Nov 2017 12:46:24 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id h28sm3279395ljb.30.2017.11.21.12.46.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 Nov 2017 12:46:23 -0800 (PST) Subject: Re: zio_done panic in 10.3 To: Shiva Bhanujan , "cem@freebsd.org" Cc: "freebsd-fs@freebsd.org" References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D636@QLEXC01.Quorum.local> From: Andriy Gapon Message-ID: <41e2465d-e1b5-33ce-57b5-49bea6087d9a@FreeBSD.org> Date: Tue, 21 Nov 2017 22:46:22 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D636@QLEXC01.Quorum.local> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2017 20:52:27 -0000 On 21/11/2017 21:30, Shiva Bhanujan wrote: > it did get compressed to 0.5G - still too big to send via email. I did send some more debug information by running kgdb on the core file to Andriy, and I'm waiting for any analysis that he might provide. Yes, kgdb-over-email turned out to be a far more efficient compression :-) I already have an analysis based on the information provided by Shiva and by another user who has the same problem and contacted me privately. I am discussing possible ways to fix the problem with George Wilson who was very kind to double-check the analysis, complete it and suggest possible fixes. A short version is that dbuf_prefetch and dbuf_prefetch_indirect_done functions chain new zio-s under the same parent zio (a completion of one child zio may create another child zio). They do it using arc_read which can create either a logical zio in most cases or a vdev zio for a read from a cache device (2arc). zio_done() has a check for the completion of a parent zio's children but that check is not completely safe and can be broken by the pattern that dbuf_prefetch can create. So, under some specific circumstances the parent zio may complete and get destroyed while there is a child zio. I believe this problem to be rather rare, but there could be configurations and workloads where it's triggered more often. The problem does not happen if there are no cache devices. > From: Conrad Meyer [cem@freebsd.org] > > Sent: Tuesday, November 21, 2017 9:04 AM > > To: Shiva Bhanujan > > Cc: Andriy Gapon; freebsd-fs@freebsd.org > > Subject: Re: zio_done panic in 10.3 > > > > > > > > Have you tried compressing it with e.g. xz or zstd? > > -- Andriy Gapon From owner-freebsd-fs@freebsd.org Tue Nov 21 22:26:19 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7AB86DB9BDF for ; Tue, 21 Nov 2017 22:26:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6880D1F8D for ; Tue, 21 Nov 2017 22:26:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vALMQJ8r044953 for ; Tue, 21 Nov 2017 22:26:19 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Date: Tue, 21 Nov 2017 22:26:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.4-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2017 22:26:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Nov 22 14:40:20 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CD31DED32E for ; Wed, 22 Nov 2017 14:40:20 +0000 (UTC) (envelope-from youzhong@gmail.com) Received: from mail-ot0-x22d.google.com (mail-ot0-x22d.google.com [IPv6:2607:f8b0:4003:c0f::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23C707CB09; Wed, 22 Nov 2017 14:40:20 +0000 (UTC) (envelope-from youzhong@gmail.com) Received: by mail-ot0-x22d.google.com with SMTP id u10so13714863otc.12; Wed, 22 Nov 2017 06:40:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=BcjWCBm6ScrsfF3vaixEzLBBAkSFc67mfnhiq2FuJ4w=; b=kAHPh9+1r8ep9QwP8LSxrCjgGhRDgq0zfbgZqDdsk8fihtvbOZWbw+59QAHhF1mOHJ pxzU7IGzB27LQhyMmXAVaCU59bz99mTOAh/IgWacHLC8ERk1on+0crDSfdm8QPIbuyUD boWfHmlIs718mjaf9Y+ktUNXC/G061YG+bAVMFD5hGdPd61tBpDvRwNACtaAY4KpUwm5 LSQobqmVKhMj86VAIOf1kIGJaDnM54zcUwBc09qrsxQXQPmnrahtAe9QtsC/GO39jpl1 6MP+lrBI6V7rk31u/qBl2WzGENvRMfT1za6LAgrfRv4Wjji6tQ9jDee38s7L3LZZbSlG UHxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=BcjWCBm6ScrsfF3vaixEzLBBAkSFc67mfnhiq2FuJ4w=; b=V1eVwXXQ28c0VuaShJ1wPW8OXph2cjSYKPJj8rMhTNQJZU1gKg9VIHyzUZTLtbdAgH IDveUTA36kZ4zrDhjx8t9W+7JOYuJbNmEK7/tzP7KB8frrsd6UJhkc5VBQbzIWFFCPaG h0dLWSu8JEVa7fbP41IVBJaJm0DRtjW3u+BYsrAt3yX/K1j4rKnp2PsM851UlyOYw5aK bfaHdOgWe85AG0Dwu+1g8VVatSFhTypU9inM6DSto+vKXTvQgN4mTeyqsXDzTfFgHRCR z/v4mVXzbs2JfzqjNF+1BlQVAiUO5WEoGQEqEPUDGbq0hLZSWjCgBP/cYokpTk1QvwJG ekmw== X-Gm-Message-State: AJaThX78YJjqd3K2K/8MaLmzc2mFvEG5qEBSi5nMUrnl7LKHCN+qy1YB TXAv+Zd+6NidW2LIAsDCbnEnvn9QaayUhngMNFPzMA== X-Google-Smtp-Source: AGs4zMYpsag5BObbBd6Xlng0lBrq4seVmXvXyvcvshsgbel+WXicB69rGcafn4p4+9D0aUCKF20hSLmxYnUA2XGhTDE= X-Received: by 10.157.89.173 with SMTP id u45mr3075964oth.341.1511361619262; Wed, 22 Nov 2017 06:40:19 -0800 (PST) MIME-Version: 1.0 Received: by 10.74.140.73 with HTTP; Wed, 22 Nov 2017 06:40:18 -0800 (PST) In-Reply-To: <41e2465d-e1b5-33ce-57b5-49bea6087d9a@FreeBSD.org> References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D636@QLEXC01.Quorum.local> <41e2465d-e1b5-33ce-57b5-49bea6087d9a@FreeBSD.org> From: Youzhong Yang Date: Wed, 22 Nov 2017 09:40:18 -0500 Message-ID: Subject: Re: zio_done panic in 10.3 To: Andriy Gapon Cc: Shiva Bhanujan , "cem@freebsd.org" , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 14:40:20 -0000 Hi Andriy, This is nice! I am 100% sure it's exactly the same issue I experienced and then reported to illumos mailing list. In all the crash dumps zio->io_done = l2arc_read_done, so I thought the crash must be related to L2ARC. Once I set secondarycache=metadata, the frequency of crash went from one per 2 days down to one per week. I've been puzzled by what could have caused a zio being destroyed while there's still child zio. Your explanation definitely makes sense! By the way, is there a FreeBSD bug report or an illumos bug number tracking this issue? I would be more than happy to create one if needed, and also test your potential fix here in our environment. Thanks, --Youzhong On Tue, Nov 21, 2017 at 3:46 PM, Andriy Gapon wrote: > > > > On 21/11/2017 21:30, Shiva Bhanujan wrote: > > it did get compressed to 0.5G - still too big to send via email. I did > send some more debug information by running kgdb on the core file to > Andriy, and I'm waiting for any analysis that he might provide. > > Yes, kgdb-over-email turned out to be a far more efficient compression :-) > I already have an analysis based on the information provided by Shiva and > by > another user who has the same problem and contacted me privately. > I am discussing possible ways to fix the problem with George Wilson who > was very > kind to double-check the analysis, complete it and suggest possible fixes. > > A short version is that dbuf_prefetch and dbuf_prefetch_indirect_done > functions > chain new zio-s under the same parent zio (a completion of one child zio > may > create another child zio). They do it using arc_read which can create > either a > logical zio in most cases or a vdev zio for a read from a cache device > (2arc). > zio_done() has a check for the completion of a parent zio's children but > that > check is not completely safe and can be broken by the pattern that > dbuf_prefetch > can create. So, under some specific circumstances the parent zio may > complete > and get destroyed while there is a child zio. > > I believe this problem to be rather rare, but there could be > configurations and > workloads where it's triggered more often. > The problem does not happen if there are no cache devices. > > > From: Conrad Meyer [cem@freebsd.org] > > > > Sent: Tuesday, November 21, 2017 9:04 AM > > > > To: Shiva Bhanujan > > > > Cc: Andriy Gapon; freebsd-fs@freebsd.org > > > > Subject: Re: zio_done panic in 10.3 > > > > > > > > > > > > > > > > Have you tried compressing it with e.g. xz or zstd? > > > > -- > Andriy Gapon > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Wed Nov 22 15:29:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5837DEE371 for ; Wed, 22 Nov 2017 15:29:07 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f41.google.com (mail-lf0-f41.google.com [209.85.215.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4211D7E44B; Wed, 22 Nov 2017 15:29:06 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f41.google.com with SMTP id d10so7634912lfj.7; Wed, 22 Nov 2017 07:29:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=0ea+++xQTlndIlADwX0h1xxTZKV+6wlE2JejoctCa8k=; b=IzI4wLzTVdx3Fwhfz2Btt2e2Ab+MjadmG/7x9VZdCgRgGQgmvfJ+WFt3sy1jdCeqJt NV0a+RJbDjcjs07Qj/62IFm4owfA+Jh1cPAEAGGpYfskMYgi35tKm5iDVrnSeO6CNruD ZM9GwzQMHk8Mb6x2+QzPuygjK9eS+86QdnFbo3gsdLDTZwUEjnQLGsFyXp1WAPriKxtr gs7ROV6PcCIfdi0hZxG4ag0n5m58ZLp6zHWEzXFsOpKY+PZUQFnfyf7lDmp/ByKW0LXJ PURCdcwtIWs94flq22eFJs1Mft3NTIsAppf9TjHz7mme+xAJoIAHPGovkHS6ZezAoRkE uPAw== X-Gm-Message-State: AJaThX49K/laZrR54r7Jh4LQSUzGTsNnNpnRVDzZhadCiiYN82eycMz9 h9yjrDb6W3XM3qUbRErgpNZ0LY38pWw= X-Google-Smtp-Source: AGs4zMYjal7xoujmft9eM0Ej1jhNKRiq50gn3TmQEtBXk6K8JZpyuR7TOcKAJQsfeK5WiSGVtR5gnQ== X-Received: by 10.25.26.4 with SMTP id a4mr4589824lfa.105.1511364176211; Wed, 22 Nov 2017 07:22:56 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id 17sm3704514ljh.73.2017.11.22.07.22.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Nov 2017 07:22:55 -0800 (PST) Subject: Re: zio_done panic in 10.3 To: Youzhong Yang Cc: Shiva Bhanujan , "cem@freebsd.org" , "freebsd-fs@freebsd.org" References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D636@QLEXC01.Quorum.local> <41e2465d-e1b5-33ce-57b5-49bea6087d9a@FreeBSD.org> From: Andriy Gapon Message-ID: <78d712d9-dda3-0411-262e-bb64f9ab46eb@FreeBSD.org> Date: Wed, 22 Nov 2017 17:22:54 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 15:29:07 -0000 On 22/11/2017 16:40, Youzhong Yang wrote: > Hi Andriy, > > This is nice! I am 100% sure it's exactly the same issue I experienced and then > reported to illumos mailing list. In all the crash dumps zio->io_done = > l2arc_read_done, so I thought the crash must be related to L2ARC. Once I set > secondarycache=metadata, the frequency of crash went from one per 2 days down to > one per week. I've been puzzled by what could have caused a zio being destroyed > while there's still child zio. Your explanation definitely makes sense! Oh, I now recall seeing your report: https://illumos.topicbox.com/groups/zfs/Tccd8b4463865899e I remember that it raised my interest, but then I forgot about it and didn't correlate it with the latest reports. > By the way, is there a FreeBSD bug report or an illumos bug number tracking this > issue? I would be more than happy to create one if needed, and also test your > potential fix here in our environment. I am not aware of any existing bug report. It would be great if you could open one [ or two :-) ] If you open an illumos issue, please also add George Wilson as a watcher. I think that George is also interested in fixing this issue and he knows the relevant code better than me. Thank you! > On Tue, Nov 21, 2017 at 3:46 PM, Andriy Gapon > wrote: > > > > > On 21/11/2017 21:30, Shiva Bhanujan wrote: > > it did get compressed to 0.5G - still too big to send via email.  I did send some more debug information by running kgdb on the core file to Andriy, and I'm waiting for any analysis that he might provide. > > Yes, kgdb-over-email turned out to be a far more efficient compression :-) > I already have an analysis based on the information provided by Shiva and by > another user who has the same problem and contacted me privately. > I am discussing possible ways to fix the problem with George Wilson who was very > kind to double-check the analysis, complete it and suggest possible fixes. > > A short version is that dbuf_prefetch and dbuf_prefetch_indirect_done functions > chain new zio-s under the same parent zio (a completion of one child zio may > create another child zio).  They do it using arc_read which can create either a > logical zio in most cases or a vdev zio for a read from a cache device (2arc). > zio_done() has a check for the completion of a parent zio's children but that > check is not completely safe and can be broken by the pattern that dbuf_prefetch > can create.  So, under some specific circumstances the parent zio may complete > and get destroyed while there is a child zio. > > I believe this problem to be rather rare, but there could be configurations and > workloads where it's triggered more often. > The problem does not happen if there are no cache devices. > > > From: Conrad Meyer [cem@freebsd.org ] > > > > Sent: Tuesday, November 21, 2017 9:04 AM > > > > To: Shiva Bhanujan > > > > Cc: Andriy Gapon; freebsd-fs@freebsd.org > > > > Subject: Re: zio_done panic in 10.3 > > > > > > > > > > > > > > > > Have you tried compressing it with e.g. xz or zstd? > > > > -- > Andriy Gapon > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org > " > > -- Andriy Gapon From owner-freebsd-fs@freebsd.org Wed Nov 22 16:26:12 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8996BDEF37D for ; Wed, 22 Nov 2017 16:26:12 +0000 (UTC) (envelope-from youzhong@gmail.com) Received: from mail-ot0-x22b.google.com (mail-ot0-x22b.google.com [IPv6:2607:f8b0:4003:c0f::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4853D7FE1B; Wed, 22 Nov 2017 16:26:12 +0000 (UTC) (envelope-from youzhong@gmail.com) Received: by mail-ot0-x22b.google.com with SMTP id s12so14029067otc.0; Wed, 22 Nov 2017 08:26:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=stBy14Eiv7nPFQwDTpNGQbFHUo8wgciH6eSLBUTTAgc=; b=oSyP30+4+/eKhhddkPoK5NqOnFz3g/W7uD9Lh2dTMc2ZBaymnIW1y6TyudtPQDxne3 uk0LrCp2m0ReSVZoGbvUfQo0ajm4ixnuvH33k3Plt30DG3KPGrqLNdG4w72jcM9xHk7a RzXUY4LTf5YE9wkCg2neOLTSsj10Pr08NvNfm05Kq5octFZmCUEG6SH2PfR2jcErDEj/ s9XbfZ6pCU4j4s6sHsC9Kn4wyAu3wZx/jIhQ9f5xdVSB/9Io7zjVqFP7vz+LB7LXiQ0N YUmqQvmSCKrRU9dSo/+Sq0lsQtT2YAQ2e7qajAWzYJYVgUgdqHgJbXSYurvmiPcwfBtn ERlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=stBy14Eiv7nPFQwDTpNGQbFHUo8wgciH6eSLBUTTAgc=; b=onzOy/6touDePrnwIx3/YVp6TVJ8LttyZRJuazI/6Ggd3C+F8xXqSNSbiyzsYH1zh0 1eWsOjaU1pZkpfIWCP2UCPtPrERY342NzeASamfwBUoXBOqwkQrci+uYjdCxQPfFpXJ8 80dlIe4Gl2e2dCPREOLQVFJHg81UsF8LAADMenitD/F7zRbcPKCGHseyW1EpWTOd/qK2 K3q4aCVhh5dh1e0+KbMpkR55uX02Tm9Qa5YL10SjypF9IPdWO3VDU//xlLzyaaoMwKjz QfKYTaNBSBnUm11KqdqcXmvj+EfBoaQrDuy+L/LLBzOvTy3Bw3xslXq5yvwKLmPuGLq2 RoAQ== X-Gm-Message-State: AJaThX6LQL+dY6eRF4xoxB104hOQBjJIPVEzMPAd9STeIh/i5g0H5XEg l/CC5fVCRQkmIwyZQp8eIXX0kr3y/d5BhREMZZhtc7kh X-Google-Smtp-Source: AGs4zMaIl7+TR1trd002bmYx18ghVeVsgxsBSMwgGW8dbDpy1Egfg9Vzc0JVQrooCPKmWzMSpZ8zvNl0jKbTJFuOoBE= X-Received: by 10.157.89.173 with SMTP id u45mr3376243oth.341.1511367971248; Wed, 22 Nov 2017 08:26:11 -0800 (PST) MIME-Version: 1.0 Received: by 10.74.140.73 with HTTP; Wed, 22 Nov 2017 08:26:10 -0800 (PST) In-Reply-To: <78d712d9-dda3-0411-262e-bb64f9ab46eb@FreeBSD.org> References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D636@QLEXC01.Quorum.local> <41e2465d-e1b5-33ce-57b5-49bea6087d9a@FreeBSD.org> <78d712d9-dda3-0411-262e-bb64f9ab46eb@FreeBSD.org> From: Youzhong Yang Date: Wed, 22 Nov 2017 11:26:10 -0500 Message-ID: Subject: Re: zio_done panic in 10.3 To: Andriy Gapon Cc: Shiva Bhanujan , "cem@freebsd.org" , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 16:26:12 -0000 Thanks Andriy. Two bug reports filed: https://www.illumos.org/issues/8857 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=223803 On Wed, Nov 22, 2017 at 10:22 AM, Andriy Gapon wrote: > On 22/11/2017 16:40, Youzhong Yang wrote: > > Hi Andriy, > > > > This is nice! I am 100% sure it's exactly the same issue I experienced > and then > > reported to illumos mailing list. In all the crash dumps zio->io_done = > > l2arc_read_done, so I thought the crash must be related to L2ARC. Once I > set > > secondarycache=metadata, the frequency of crash went from one per 2 days > down to > > one per week. I've been puzzled by what could have caused a zio being > destroyed > > while there's still child zio. Your explanation definitely makes sense! > > Oh, I now recall seeing your report: > https://illumos.topicbox.com/groups/zfs/Tccd8b4463865899e > I remember that it raised my interest, but then I forgot about it and > didn't > correlate it with the latest reports. > > > By the way, is there a FreeBSD bug report or an illumos bug number > tracking this > > issue? I would be more than happy to create one if needed, and also test > your > > potential fix here in our environment. > > I am not aware of any existing bug report. > It would be great if you could open one [ or two :-) ] > If you open an illumos issue, please also add George Wilson as a watcher. > I think that George is also interested in fixing this issue and he knows > the > relevant code better than me. > > Thank you! > > > On Tue, Nov 21, 2017 at 3:46 PM, Andriy Gapon > > wrote: > > > > > > > > > > On 21/11/2017 21:30, Shiva Bhanujan wrote: > > > it did get compressed to 0.5G - still too big to send via email. > I did send some more debug information by running kgdb on the core file to > Andriy, and I'm waiting for any analysis that he might provide. > > > > Yes, kgdb-over-email turned out to be a far more efficient > compression :-) > > I already have an analysis based on the information provided by > Shiva and by > > another user who has the same problem and contacted me privately. > > I am discussing possible ways to fix the problem with George Wilson > who was very > > kind to double-check the analysis, complete it and suggest possible > fixes. > > > > A short version is that dbuf_prefetch and > dbuf_prefetch_indirect_done functions > > chain new zio-s under the same parent zio (a completion of one child > zio may > > create another child zio). They do it using arc_read which can > create either a > > logical zio in most cases or a vdev zio for a read from a cache > device (2arc). > > zio_done() has a check for the completion of a parent zio's children > but that > > check is not completely safe and can be broken by the pattern that > dbuf_prefetch > > can create. So, under some specific circumstances the parent zio > may complete > > and get destroyed while there is a child zio. > > > > I believe this problem to be rather rare, but there could be > configurations and > > workloads where it's triggered more often. > > The problem does not happen if there are no cache devices. > > > > > From: Conrad Meyer [cem@freebsd.org ] > > > > > > Sent: Tuesday, November 21, 2017 9:04 AM > > > > > > To: Shiva Bhanujan > > > > > > Cc: Andriy Gapon; freebsd-fs@freebsd.org freebsd-fs@freebsd.org> > > > > > > Subject: Re: zio_done panic in 10.3 > > > > > > > > > > > > > > > > > > > > > > > > Have you tried compressing it with e.g. xz or zstd? > > > > > > -- > > Andriy Gapon > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org > > " > > > > > > > -- > Andriy Gapon > From owner-freebsd-fs@freebsd.org Wed Nov 22 18:01:52 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34572DF158A for ; Wed, 22 Nov 2017 18:01:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 22B473E68 for ; Wed, 22 Nov 2017 18:01:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAMI1pBZ074132 for ; Wed, 22 Nov 2017 18:01:52 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Date: Wed, 22 Nov 2017 18:01:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.4-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mckusick@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 18:01:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 Kirk McKusick changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mckusick@FreeBSD.org --- Comment #1 from Kirk McKusick --- This report is perhaps the third or fourth about the slow sync problem. Unfortunately we have not been able to reproduce the slowdown. We believe it started with r230553. Note that I carefully use the word 'started' and not 'caused', because I believe that the change in r230553 is required to avoid loss of user data. But it is not clear what makes the sy= nc run for such a long time. No users who reported the slowdown have provided a useful insight into their configuration. What we need is a description of your filesystem. Ideal woul= d be a dump image that when restored demonstrates the problem. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Nov 22 18:41:47 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 424C6DF287F for ; Wed, 22 Nov 2017 18:41:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 30E2A6487B for ; Wed, 22 Nov 2017 18:41:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAMIfk4J046475 for ; Wed, 22 Nov 2017 18:41:47 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Date: Wed, 22 Nov 2017 18:41:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.4-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mi@ALDAN.algebra.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 18:41:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 --- Comment #2 from Mikhail T. --- I don=E2=80=99t know, what to say. I=E2=80=99ve been seeing this problem fo= r a while on all of the machines I manage (that=E2=80=99s about 8). Besides the tunefs output already included, I=E2=80=99m not sure, what to a= dd. While the mount command is waiting, it is in the biowr-state. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Nov 22 19:36:54 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C43DDF399C for ; Wed, 22 Nov 2017 19:36:54 +0000 (UTC) (envelope-from rde@tavi.co.uk) Received: from kipling.tavi.co.uk (kipling.tavi.co.uk [81.187.145.130]) by mx1.freebsd.org (Postfix) with ESMTP id EF847666B3 for ; Wed, 22 Nov 2017 19:36:53 +0000 (UTC) (envelope-from rde@tavi.co.uk) Received: from kipling.tavi.co.uk (localhost [127.0.0.1]) by kipling.tavi.co.uk (Postfix) with ESMTP id 39CDC892BE for ; Wed, 22 Nov 2017 19:36:45 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=tavi.co.uk; h=date:from:to :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=selector1; bh=xGaRTe6 8zMjMhUv0Ggemk2Z3eKU=; b=GM+Dg1DzVO5kMF/pjzl/Q9F13bl4bYx88r3yPzH 0yCw9/STtN4WUI0paSjr7mxQppc5uWXW+3l2uhnXW7apPdQ8z/VJpTVe0dK4y686 2Z6eNsdgQfd1JpgMORL5QrRxIP2DOyemupzcCxpLz3dPWB/YtnBRIc5HfijypXMd H3Lc= Received: from raksha.tavi.co.uk (raksha.tavi.co.uk [81.187.145.139]) (Authenticated sender: rde@tavi.co.uk) by kipling.tavi.co.uk (Postfix) with ESMTPA id 004EA892BB for ; Wed, 22 Nov 2017 19:36:44 +0000 (GMT) Date: Wed, 22 Nov 2017 19:36:44 +0000 From: Bob Eager To: freebsd-fs@freebsd.org Subject: Re: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Message-ID: <20171122193644.561060b0@raksha.tavi.co.uk> In-Reply-To: References: X-Mailer: Claws Mail 3.15.0 (GTK+ 2.24.31; i386-portbld-freebsd11.0) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAMFBMVEUwXjFLc0vD0cS7y7zw9PDZ4tkWSRaVrZZ+m39qi2tXfVj////7+/utwK4IPggAOAAJUUA7AAABKklEQVQ4jWPYjQMwDFYJp0NKEKCNJmEf9h8CsimXiL2e33s3/e7F7K2Cs3f3dCMkQkMKj4YuCY3K3iR+e7fMaiSjvkX0/5cFGrWpe2uLzOpaExUVqMS/8PX/Re5ey960OLBTZpFA8+IlSBKPQ92zNyUUBsosN58uIY0k8f+/ONCoYytkVuhWzVwNkYiYbqk5M3NmOVBi41YZ8RsGF7shEtFb5KJ3r969CyixM7OTPeFUxG2IxLO8/9/SvqXlc+/x3h295YzLlj2nIRJQj//nRvc5TEIal8RsXBLVuCQwIgoq/u80DomP6HEOk/iOS+IJLonZOCT+ReOQ+Lkbh0QKLonbOCR+7MYhsRqHBJrVcIl/1TgklqKLQyQ+tGKIgyQOqXpjig94diZRAgAXmDX6jyWafAAAAABJRU5ErkJggg====== MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 19:36:54 -0000 On Wed, 22 Nov 2017 18:41:47 +0000 bugzilla-noreply@freebsd.org wrote: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 >=20 > --- Comment #2 from Mikhail T. --- > I don?t know, what to say. I?ve been seeing this problem for a while > on all of the machines I manage (that?s about 8). >=20 > Besides the tunefs output already included, I?m not sure, what to > add. While the mount command is waiting, it is in the biowr-state. >=20 I don't know how long this particular 'long time' has been around, but it has seemed to take a pretty long time for 3-4 years for me! From owner-freebsd-fs@freebsd.org Wed Nov 22 21:17:47 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 601D6DF5F7C for ; Wed, 22 Nov 2017 21:17:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4E9C66A6BD for ; Wed, 22 Nov 2017 21:17:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAMLHlkN055907 for ; Wed, 22 Nov 2017 21:17:47 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Date: Wed, 22 Nov 2017 21:17:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.4-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mckusick@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 21:17:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 --- Comment #3 from Kirk McKusick --- (In reply to Mikhail T. from comment #2) Being in the biowr state is expected as the filesystem needs to sync out all the changes. The question is why there are so many of them. Randomly guessi= ng, perhaps it decides that there are a large number of access times that need = to be written. When upgrading to read-write, include the -o noatime option so there is no attempt to update access times. It would also be useful to collect the number of I/O (particularly writes) = that the downgrade command is doing (time -l if your shell does not otherwise privide the I/O statistics). You can also get an idea of the I/Os being done using iostat. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Nov 22 21:20:26 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1AE9DF602B for ; Wed, 22 Nov 2017 21:20:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CFA9E6A7AF for ; Wed, 22 Nov 2017 21:20:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAMLKQNr060043 for ; Wed, 22 Nov 2017 21:20:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Date: Wed, 22 Nov 2017 21:20:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.4-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: imp@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 21:20:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 Warner Losh changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |imp@FreeBSD.org --- Comment #4 from Warner Losh --- I think that r230553 counts as a while ago: It was made Wed Jan 25 20:54:09 2012 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Nov 22 21:42:46 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A81D9DF687A for ; Wed, 22 Nov 2017 21:42:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 963496B6FB for ; Wed, 22 Nov 2017 21:42:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAMLgj3L019844 for ; Wed, 22 Nov 2017 21:42:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223786] Remounting a UFS filesystem read-only takes very long time Date: Wed, 22 Nov 2017 21:42:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.4-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 21:42:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223786 Konstantin Belousov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kib@FreeBSD.org --- Comment #5 from Konstantin Belousov --- You may try the patch from https://reviews.freebsd.org/D13197, but I doubt = that it would change much for you. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 00:27:10 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 33BF6DF93CC for ; Fri, 24 Nov 2017 00:27:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 21F8A800FA for ; Fri, 24 Nov 2017 00:27:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAO0R9ux040086 for ; Fri, 24 Nov 2017 00:27:09 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 209571] ZFS and NVMe performing poorly. TRIM requests stall I/O activity Date: Fri, 24 Nov 2017 00:27:10 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: wheelcomplex@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 00:27:10 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209571 David NewHamlet changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wheelcomplex@gmail.com --- Comment #2 from David NewHamlet --- I believe that this issue still on FreeBSD-12-CURRENT with dmesg log: nvme0: async event occurred (log page id=3D0x2) nvme0: temperature above threshold and iostat -x 5 extended device statistics=20=20 device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b= =20=20 nvd0 0 1 0.6 153.6 2384 3287 0 2664 80 99 I am glad to test any patch for this in my test environment. ref: https://lists.freebsd.org/pipermail/freebsd-stable/2015-January/081621.html FreeBSD david-al.localdomain 12.0-CURRENT FreeBSD 12.0-CURRENT #11 5ea1f67b4ad(master)-dirty: Fri Nov 24 09:14:41 NZDT 2017=20=20=20=20 root@n550jk.localdomain:/tank/cross/obj/12.0-amd64.amd64/sys/INITZ-NODEBUG= =20 amd64 01:00.0 Non-Volatile memory controller: Intel Corporation Device f1a5 (rev = 03) nvme0: mem 0xdf200000-0xdf203fff irq 16 at device 0.0= on pci1 nvd0: NVMe namespace --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 05:28:20 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38D62DBA212 for ; Fri, 24 Nov 2017 05:28:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 266A366643 for ; Fri, 24 Nov 2017 05:28:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAO5SKLo042244 for ; Fri, 24 Nov 2017 05:28:20 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 212631] ZFS zpool scrub shows more than 100% complete Date: Fri, 24 Nov 2017 05:28:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: vermaden@interia.pl X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: version Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 05:28:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212631 vermaden@interia.pl changed: What |Removed |Added ---------------------------------------------------------------------------- Version|11.0-RC1 |CURRENT --- Comment #1 from vermaden@interia.pl --- I have the same problem on 12-CURRENT but on a different computer and diffe= rent pool. # uname -a FreeBSD bsd.local 12.0-CURRENT FreeBSD 12.0-CURRENT #0: Sat Nov 4 00:38:39= CET 2017 root@bsd.local:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG a= md64 # zpool status data pool: data state: ONLINE scan: scrub in progress since Wed Nov 22 22:30:57 2017 2.33T scanned out of 2.32T at 21.2M/s, (scan is slow, no estimated time) 0 repaired, 100.30% done config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/data.top.eli ONLINE 0 0 0 gpt/data.bottom.eli ONLINE 0 0 0 errors: No known data errors --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 11:01:26 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71A6FDDFBF9; Fri, 24 Nov 2017 11:01:26 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f46.google.com (mail-lf0-f46.google.com [209.85.215.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0E7776F69E; Fri, 24 Nov 2017 11:01:25 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f46.google.com with SMTP id x76so20389789lfb.6; Fri, 24 Nov 2017 03:01:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-language:content-transfer-encoding; bh=6FrtmZIx7kW/3Ko/v0TC6QdvxvoEkfXpbpwL8HYZisI=; b=IUMUkPNkPFQMj7F9F4b5udDFW+1cezZmOP+NH1KkcbfLZOkZOziUqv06AxNK7MG5Nv yuqsUqESHY5mlrZRTXMUJ+t8GiskiOl4i6IXtBvURrpFfNhloJLELkl/tfF6bMf8LftV fgOb11TEH+xjus3aLxEnZZpVmt3hxudDBqt8TIhmIN+Dq6H7aSkHPXmkSD2Vyd3CGWNO 8ud3wVFUO1NrNwUwt9bXAoT+Xn8q5VTeqHV+x7GTxkIpjeOR0zcoKXpbuDRxjetmzSqJ ffXDgPMoKYzyLVtf9xNNR0nxdO9EgBnKJxMfIXgoCGXGjlZvELWNr9bUKrJ1jSOf0L0u pyHQ== X-Gm-Message-State: AJaThX76Jl0HOS8bJlFhrqYcln5AUYqZqggcRmekAsvlBnJ2bwUhsAVH AFY0tTJRqKE41SfncnPFqUeJyxHzkFs= X-Google-Smtp-Source: AGs4zMY683jXmpsJMe3oL3xJpzNM1zW+ffcjrGJ3523EBi/7T/NgRHjLoysysdel4AG7cbyvWqi/ew== X-Received: by 10.46.68.195 with SMTP id b64mr8733440ljf.121.1511519433271; Fri, 24 Nov 2017 02:30:33 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id m95sm3694573lfi.76.2017.11.24.02.30.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Nov 2017 02:30:32 -0800 (PST) To: freebsd-fs@FreeBSD.org, freebsd-geom@FreeBSD.org From: Andriy Gapon Subject: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom Message-ID: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> Date: Fri, 24 Nov 2017 12:30:30 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 11:01:26 -0000 https://reviews.freebsd.org/D13224 Anyone interested is welcome to join the review. Thanks. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Nov 24 13:08:10 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D279DE50DA for ; Fri, 24 Nov 2017 13:08:10 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x232.google.com (mail-io0-x232.google.com [IPv6:2607:f8b0:4001:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C8B3C73C3F for ; Fri, 24 Nov 2017 13:08:09 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x232.google.com with SMTP id q101so29487591ioi.1 for ; Fri, 24 Nov 2017 05:08:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=wOUgQZRszG2Cc8EXcF4I4ct1fBYHQfUdMYXIWs+F+kM=; b=isUkA4ECI3JesPLAYxUKKYXeVMi2zrx/o0OP8GdV9QaT6lGbmir4UFVIyNYDzeWoK4 +Gh2PDDBVRyQhALliKCbNPQLYwW5g0o4avQoA84U/EKvj8J3LcVpDh5H8n2iVv+rwCd5 T+fNeh0gVujBdYv16TG9dlsP8zhzc4MEl+yoGn3W+a1c4UdBEQuocPLoe46snTPi73fw wTCbX1qijQhDWL/aQz2REYg8qYM77zxui9siy77UlEQPzKF2nGzNCbDj8mlmgbX2fAY/ wLWvj0afhHXsnq32X67GK7ksGpZ2PMec9riGRPI7fCJvhW1zTI+msQ2pYO78iaOaYVcv S3bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=wOUgQZRszG2Cc8EXcF4I4ct1fBYHQfUdMYXIWs+F+kM=; b=r7yLEAzNHtUXBPZjdsUXJpUvGyBHJPwGEL/uMxUcek4HRHfwVl89SFP+9vlTK9S7ro UIEnjCHr5961NiSAleI8o+SnIYymGlrGm/+kQltV+iB0+TIrWk4b+WRqIy56eIL6XIOW 0Xw+bzKRA1en+e+gb5LzXbSxIipkNLlc9gQQGuPEdYa1CFYzERfkmHtdcA/atodw/y8D InKzXYhUrXhw7Bk0JTk0E4gNlUxK4prxcVixPU5Bt8GanKh+rrOSlv+o8sZnG+2ot0np rxpXJh7S2HGrJ54IgbwEUwED284jn+BvooEssmo8F3Pn7LIEkIrKyu5wLes+3KlHa5J4 S5jQ== X-Gm-Message-State: AJaThX6QJaTSErvrlKg3uZhTb01vGgtvWE8zxqCHvYAt7rn6wZdLSAb0 9DVsyqGEtmaQkEsbZ7jurzwNEG7DlpKwPbeH2hROwQ== X-Google-Smtp-Source: AGs4zMY4MoBgdZFANsotjdt+Pxukf0cEindNxrZgT9zCGlGViUGX63RvX4ZZmzK0IC5ZT5xNwZUl0/rOkuCUehmtndY= X-Received: by 10.107.104.18 with SMTP id d18mr28887141ioc.136.1511528888799; Fri, 24 Nov 2017 05:08:08 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.108.204 with HTTP; Fri, 24 Nov 2017 05:08:08 -0800 (PST) X-Originating-IP: [2603:300b:6:5100:f964:7c3e:d2:aac5] In-Reply-To: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> From: Warner Losh Date: Fri, 24 Nov 2017 06:08:08 -0700 X-Google-Sender-Auth: tj7KN5Upw_bTsDr4uHtec8Eefns Message-ID: Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Andriy Gapon Cc: FreeBSD FS , freebsd-geom@freebsd.org, Scott Long Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 13:08:10 -0000 On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon wrote: > > https://reviews.freebsd.org/D13224 > > Anyone interested is welcome to join the review. > I think it's a really bad idea. It introduces a 'one-size-fits-all' notion of QoS that seems misguided. It conflates a shorter timeout with don't retry. And why is retrying bad? It seems more a notion of 'fail fast' or so other concept. There's so many other ways you'd want to use it. And it uses the same return code (EIO) to mean something new. It's generally meant 'The lower layers have retried this, and it failed, do not submit it again as it will not succeed' with 'I gave it a half-assed attempt, and that failed, but resubmission might work'. This breaks a number of assumptions in the BUF/BIO layer as well as parts of CAM even more than they are broken now. So let's step back a bit: what problem is it trying to solve? Warner From owner-freebsd-fs@freebsd.org Fri Nov 24 13:36:06 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5083DE5A9C for ; Fri, 24 Nov 2017 13:36:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B280974953 for ; Fri, 24 Nov 2017 13:36:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAODa6Zp058848 for ; Fri, 24 Nov 2017 13:36:06 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223839] System crashes after zpool import -X Date: Fri, 24 Nov 2017 13:36:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 13:36:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223839 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 13:39:31 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C439DE615C for ; Fri, 24 Nov 2017 13:39:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 69FE875024 for ; Fri, 24 Nov 2017 13:39:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAODdVMl063270 for ; Fri, 24 Nov 2017 13:39:31 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223808] zpool attach (and other commands?) fail with misleading error if a wiped disk had previously been used in a RAID array Date: Fri, 24 Nov 2017 13:39:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 13:39:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223808 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 13:41:27 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31195DE6395; Fri, 24 Nov 2017 13:41:27 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f54.google.com (mail-lf0-f54.google.com [209.85.215.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DE9487522F; Fri, 24 Nov 2017 13:41:26 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f54.google.com with SMTP id k66so25549237lfg.3; Fri, 24 Nov 2017 05:41:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=r36jXUEGMIy23uM2+d2siF/0w4/bQvR41bQZSuWZoOo=; b=Dfi8pWBslpSts5kHV2uMfnBKr0qLawEXf/LFbwNONjeNdHjS+GRcKi/JkswWoybJC0 /rPaKsOtIEtTpj7Do/u1aD6AX6tw567BrhkrD9adt5cwuoBNvZb54sD01vQkUPf7aS3m CVZ55F+MkfPL9bX+3pn5P/CfwP9ToD35ND8iUgT/J0KT37/OEhWDnRJu45k1wXq8Ugjx mlEonz7+DWAdBHZgzhbjc/MD7RfNd03TJBCy0uV7KGCUD75ldSU+vKr/DWrD8WJc1zEt F5OTJmnvp3OpZOIodt9e/cDvXa9ES/ig+dvFe/sPuzjnBhn11Y3KnAfY9eZWuu7nRx3W YSJQ== X-Gm-Message-State: AJaThX4pRU8wTx/muTap/6DsWa6erbTJirBRhltJXyn3PGXfz4wPPN6F 1Vddzi37QxkMPQ6Nh9h1dIs= X-Google-Smtp-Source: AGs4zMbnrPvRtz68h9+hnPjT0RKaFjc6RWusF10UJUlzr7pZDgsEUvOC4aqpTDJh8D/LuFrtWW/MAQ== X-Received: by 10.25.163.11 with SMTP id m11mr9390033lfe.179.1511530479138; Fri, 24 Nov 2017 05:34:39 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id s66sm4550021lje.40.2017.11.24.05.34.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Nov 2017 05:34:38 -0800 (PST) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Warner Losh Cc: FreeBSD FS , freebsd-geom@freebsd.org, Scott Long References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> From: Andriy Gapon Message-ID: <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> Date: Fri, 24 Nov 2017 15:34:36 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 13:41:27 -0000 On 24/11/2017 15:08, Warner Losh wrote: > > > On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon > wrote: > > > https://reviews.freebsd.org/D13224 > > Anyone interested is welcome to join the review. > > > I think it's a really bad idea. It introduces a 'one-size-fits-all' notion of > QoS that seems misguided. It conflates a shorter timeout with don't retry. And > why is retrying bad? It seems more a notion of 'fail fast' or so other concept. > There's so many other ways you'd want to use it. And it uses the same return > code (EIO) to mean something new. It's generally meant 'The lower layers have > retried this, and it failed, do not submit it again as it will not succeed' with > 'I gave it a half-assed attempt, and that failed, but resubmission might work'. > This breaks a number of assumptions in the BUF/BIO layer as well as parts of CAM > even more than they are broken now. > > So let's step back a bit: what problem is it trying to solve? A simple example. I have a mirror, I issue a read to one of its members. Let's assume there is some trouble with that particular block on that particular disk. The disk may spend a lot of time trying to read it and would still fail. With the current defaults I would wait 5x that time to finally get the error back. Then I go to another mirror member and get my data from there. IMO, this is not optimal. I'd rather pass BIO_NORETRY to the first read, get the error back sooner and try the other disk sooner. Only if I know that there are no other copies to try, then I would use the normal read with all the retrying. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Nov 24 14:12:28 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 370ABDE7643 for ; Fri, 24 Nov 2017 14:12:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2541C76865 for ; Fri, 24 Nov 2017 14:12:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAOECSme048650 for ; Fri, 24 Nov 2017 14:12:28 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223839] System crashes after zpool import -X Date: Fri, 24 Nov 2017 14:12:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 14:12:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223839 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |Not A Bug --- Comment #1 from Andriy Gapon --- Corrupted pools can cause crashes. zpool import -X is not magic command that can fix any pool. In fact, it's a dangerous extreme rewind that can make things even worse. It's not documented for that exact reason. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 14:49:26 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9EC12DE8182 for ; Fri, 24 Nov 2017 14:49:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 870C07766F for ; Fri, 24 Nov 2017 14:49:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAOEnQem026198 for ; Fri, 24 Nov 2017 14:49:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223808] zpool attach (and other commands?) fail with misleading error if a wiped disk had previously been used in a RAID array Date: Fri, 24 Nov 2017 14:49:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: stilezy@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 14:49:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223808 --- Comment #1 from Stilez --- As this could affect other situations, perhaps it's worth including a menti= on in the console messages during connection/recognition, if a disk being connected contains RAID metadata, as this could be helpful/worth exposing in other situations and the console connection messages are a logical place to= do so. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 14:58:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34698DE856C; Fri, 24 Nov 2017 14:58:05 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 046CA77CEF; Fri, 24 Nov 2017 14:58:04 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 498B620C6A; Fri, 24 Nov 2017 09:57:58 -0500 (EST) Received: from frontend1 ([10.202.2.160]) by compute6.internal (MEProxy); Fri, 24 Nov 2017 09:57:58 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsco.org; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=qVIuYEAsfpjDy6GFHeaqmYUXQPio2 uVChaJc9Aeq8Os=; b=v12oJ9CWdrq/ywPFKeZNkEVXl44tShaEcGNEDtUctXBOD 6GuZSe3fcFw0J69JaTrFf//o0WtjReDC5OEdf7fhCftRb/7wjUnYDsVv1AoB5oAN +h89BrWtYsC/e+hszmFr5+KN34hbdP/y0XAaBlOGMJoXPMziGMigoxm4l+ol9n87 p1iHdq9ABoD4jGFB/2BiHNfGBpRIfl+us8gWrUpDXLoJiQvkGQHwVWIhXfdRti7l uptZnVMxkoQ3qexTdGZo5YxTVPHG938ohoMbqiNslP8A2TficFtezumEq21E4zDG 9ItPZbW0BTlJ9YjLV2u2k+OOm7jMURWGTIdCJVNnA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=qVIuYE AsfpjDy6GFHeaqmYUXQPio2uVChaJc9Aeq8Os=; b=j6q8dcfIr5Qevxq0taQVzK 9EPL3kDNzXJ7jvrkSjOmhxxuS07mrLbq9iOh7jehiUnqooJSuTxMExb4W5HVWUKt BzKqvskYxR1m07E5wF3N75utkhyNDfhpGPCWyiHfrEaOUNmu1PwHkPRE14Otym2h rMu3yQoBhC/s9uEa6T/fCVNkWRDnLkorBFzcRtgS1U+3XsGUJfZwXGknPHia8N+4 WYoIemhfWKEVC/G2kaVnT30pXNPX/kPxN/OrIW/4ouxPt8eRwNblT9mz9ZLMtdli 6UMnssR2IBlHFhbjxHD/xwFFRu9PpwzzK/NWQiWwjvD4BuQ8ER6SO2bvySWFBo0g == X-ME-Sender: Received: from [192.168.0.103] (unknown [161.97.249.191]) by mail.messagingengine.com (Postfix) with ESMTPA id C09497F35E; Fri, 24 Nov 2017 09:57:57 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom From: Scott Long In-Reply-To: <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> Date: Fri, 24 Nov 2017 07:57:55 -0700 Cc: Warner Losh , FreeBSD FS , freebsd-geom@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> To: Andriy Gapon X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 14:58:05 -0000 > On Nov 24, 2017, at 6:34 AM, Andriy Gapon wrote: >=20 > On 24/11/2017 15:08, Warner Losh wrote: >>=20 >>=20 >> On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon > > wrote: >>=20 >>=20 >> https://reviews.freebsd.org/D13224 = >>=20 >> Anyone interested is welcome to join the review. >>=20 >>=20 >> I think it's a really bad idea. It introduces a 'one-size-fits-all' = notion of >> QoS that seems misguided. It conflates a shorter timeout with don't = retry. And >> why is retrying bad? It seems more a notion of 'fail fast' or so = other concept. >> There's so many other ways you'd want to use it. And it uses the same = return >> code (EIO) to mean something new. It's generally meant 'The lower = layers have >> retried this, and it failed, do not submit it again as it will not = succeed' with >> 'I gave it a half-assed attempt, and that failed, but resubmission = might work'. >> This breaks a number of assumptions in the BUF/BIO layer as well as = parts of CAM >> even more than they are broken now. >>=20 >> So let's step back a bit: what problem is it trying to solve? >=20 > A simple example. I have a mirror, I issue a read to one of its = members. Let's > assume there is some trouble with that particular block on that = particular disk. > The disk may spend a lot of time trying to read it and would still = fail. With > the current defaults I would wait 5x that time to finally get the = error back. > Then I go to another mirror member and get my data from there. There are many RAID stacks that already solve this problem by having a = policy of always reading all disk members for every transaction, and throwing = away the sub-transactions that arrive late. It=E2=80=99s not a policy that is = always desired, but it serves a useful purpose for low-latency needs. > IMO, this is not optimal. I'd rather pass BIO_NORETRY to the first = read, get > the error back sooner and try the other disk sooner. Only if I know = that there > are no other copies to try, then I would use the normal read with all = the retrying. >=20 I agree with Warner that what you are proposing is not correct. It = weakens the contract between the disk layer and the upper layers, making it less = clear who is responsible for retries and less clear what =E2=80=9CEIO=E2=80=9D means. = That contract is already weak due to poor design decisions in VFS-BIO and GEOM, and Warner and I are working on a plan to fix that. =20 Scott From owner-freebsd-fs@freebsd.org Fri Nov 24 16:33:58 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 529DEDEB554 for ; Fri, 24 Nov 2017 16:33:58 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x229.google.com (mail-it0-x229.google.com [IPv6:2607:f8b0:4001:c0b::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 059267AE6F for ; Fri, 24 Nov 2017 16:33:58 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x229.google.com with SMTP id y71so2400768ita.1 for ; Fri, 24 Nov 2017 08:33:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=0MVJoz49hDcnqmipn5h/XAsvMDtR8sTzp9kLvU5JS18=; b=oEuCnvZ2uOi0mqEEuCMF+Ea3KIQMW8xD4XIOzTDCGoKTFGxUdG8pKgCWcqLfEk63kB talGVqrxz368QnLhZqPCzYo+LUkeNRkAlTGomWkYu2Ub7QuPvtaoAKp40xt6ffwmQrI1 6bhPj6xeTqwjqqFGmnLbausFNdZ9KfBW/NiIgowQgVw8jOgUNPkDlucrsRZL4I9VIlqF zgkNAsNDKJ/7903BgsEKiaQm+8vo0GLzqp8Pl1pGYnwRWKHT2sha56QWI+/pcqLie5y7 N6jvMdc9N5PWnTQ47uUezW9hDMv0kGSpnrXcGiDVOg0A/icNn/iIc1VF9SD6mwKi6LK6 P0QQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=0MVJoz49hDcnqmipn5h/XAsvMDtR8sTzp9kLvU5JS18=; b=h9bc0+DlBBd6DrnQ+A2bn8RdX3V7QqJwAcJ84f2QNJW3f0k6ck7pGe8NqwNIW5Tsfi //RMKNclWdVjGqvlScW7jLVhI/D+0Nu8AMA26SNFeq9XN4n+a9ITdCVv3lJXieOxjVHO LKjeK0msiEEDGPlNSb+lVZaXodW+tL/rW2vIPTl7A5ADNyN0/7s7ijui+K0qJJTIQ6JX bRoT+TM6u6bXNx8znaB2QX7Lic4fiWl9g30eZb6MsFcjjbfw+zAdHOEu2UwJvjaGSYMh ni0nBSyxFSm+gKnxbipAkeYb/oDagiU45N6FC1O8KPZ1CGFF6rNubg3NK5jIP7BE8UIh B8XQ== X-Gm-Message-State: AJaThX5dasDbBsz4AXep+Guml56r6Bw6gU/23UMPun8lOo7BmrKFjK9+ gVsUZDIZy7HMZ/IZc6YG1+d2YI5jt71IY4e+N2RA/Q== X-Google-Smtp-Source: AGs4zMb2qcX4SYQ8dG9gG7QijUzFvld3wUlWcmNehzHGRemSK6Qjy7T0UBzw5z1gKtWtiDfacKGNpjOpyFOqluW63Yg= X-Received: by 10.36.77.143 with SMTP id l137mr17155000itb.50.1511541237108; Fri, 24 Nov 2017 08:33:57 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.108.204 with HTTP; Fri, 24 Nov 2017 08:33:56 -0800 (PST) X-Originating-IP: [2603:300b:6:5100:f964:7c3e:d2:aac5] In-Reply-To: <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> From: Warner Losh Date: Fri, 24 Nov 2017 09:33:56 -0700 X-Google-Sender-Auth: 9YkRcBZ2y_mwK-i8n9pyzcXRGJA Message-ID: Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Andriy Gapon Cc: FreeBSD FS , freebsd-geom@freebsd.org, Scott Long Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 16:33:58 -0000 On Fri, Nov 24, 2017 at 6:34 AM, Andriy Gapon wrote: > On 24/11/2017 15:08, Warner Losh wrote: > > > > > > On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon > > wrote: > > > > > > https://reviews.freebsd.org/D13224 D13224> > > > > Anyone interested is welcome to join the review. > > > > > > I think it's a really bad idea. It introduces a 'one-size-fits-all' > notion of > > QoS that seems misguided. It conflates a shorter timeout with don't > retry. And > > why is retrying bad? It seems more a notion of 'fail fast' or so other > concept. > > There's so many other ways you'd want to use it. And it uses the same > return > > code (EIO) to mean something new. It's generally meant 'The lower layers > have > > retried this, and it failed, do not submit it again as it will not > succeed' with > > 'I gave it a half-assed attempt, and that failed, but resubmission might > work'. > > This breaks a number of assumptions in the BUF/BIO layer as well as > parts of CAM > > even more than they are broken now. > > > > So let's step back a bit: what problem is it trying to solve? > > A simple example. I have a mirror, I issue a read to one of its members. > Let's > assume there is some trouble with that particular block on that particular > disk. > The disk may spend a lot of time trying to read it and would still fail. > With > the current defaults I would wait 5x that time to finally get the error > back. > Then I go to another mirror member and get my data from there. > IMO, this is not optimal. I'd rather pass BIO_NORETRY to the first read, > get > the error back sooner and try the other disk sooner. Only if I know that > there > are no other copies to try, then I would use the normal read with all the > retrying. > It sounds like you are optimizing the wrong thing and taking an overly simplistic view of quality of service. First, failing blocks on a disk is fairly rare. Do you really want to optimize for that case? Second, you're really saying 'If you can't read it fast, fail" since we only control the software side of read retry. There's new op codes being proposed that say 'read or fail within Xms' which is really what you want: if it's taking too long on disk A you want to move to disk B. The notion here was we'd return EAGAIN (or some other error) if it failed after Xms, and maybe do some emulation in software for drives that don't support this. You'd tweak this number to control performance. You're likely to get a much bigger performance win all the time by scheduling I/O to drives that have the best recent latency. Third, do you have numbers that show this is actually a win? This is a terrible thing from an architectural view. Absent numbers that show it's a big win, I'm very hesitant to say OK. Forth, there's a large number of places in the stack today that need to communicate their I/O is more urgent, and we don't have any good way to communicate even that simple concept down the stack. Finally, the only places that ZFS uses the TRYHARDER flag are for things like the super block if I'm reading the code right. It doesn't do it for normal I/O. There's no code to cope with what would happen if all the copies of a block couldn't be read with the NORETRY flag. One of them might contain the data. Warner From owner-freebsd-fs@freebsd.org Fri Nov 24 16:47:22 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1319DEB9A8 for ; Fri, 24 Nov 2017 16:47:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CF49F7B4AC for ; Fri, 24 Nov 2017 16:47:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAOGlMv0041498 for ; Fri, 24 Nov 2017 16:47:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223846] msdosfs does not reflect READONLY to user Date: Fri, 24 Nov 2017 16:47:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 16:47:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223846 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 17:04:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11523DEC274 for ; Fri, 24 Nov 2017 17:04:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F3B547BFDA for ; Fri, 24 Nov 2017 17:04:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAOH44sj089122 for ; Fri, 24 Nov 2017 17:04:04 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223846] msdosfs does not reflect READONLY to user Date: Fri, 24 Nov 2017 17:04:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: karl@denninger.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 17:04:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223846 --- Comment #2 from karl@denninger.net --- (In reply to Conrad Meyer from comment #1) I would argue it should; what's the point of a read-only flag if it's ignor= ed If you're going to have a permission system then IMHO it ought to be enforc= ed, such as it is and within the constraints of the underlying file system (e.g= . in the case of msdosfs the "owner" is effectively synthetic and there is no su= ch thing as "group" or "other".) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 17:10:26 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77B54DEC350 for ; Fri, 24 Nov 2017 17:10:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6596D7C0E5 for ; Fri, 24 Nov 2017 17:10:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAOHAQ1P097536 for ; Fri, 24 Nov 2017 17:10:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223846] msdosfs does not reflect READONLY to user Date: Fri, 24 Nov 2017 17:10:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 17:10:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223846 --- Comment #3 from Conrad Meyer --- Yeah, I agree with you. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 17:18:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C723DEC656; Fri, 24 Nov 2017 17:18:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f50.google.com (mail-lf0-f50.google.com [209.85.215.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AD0F97C6EF; Fri, 24 Nov 2017 17:18:04 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f50.google.com with SMTP id y2so25239532lfj.4; Fri, 24 Nov 2017 09:18:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=BiB9h//yrNXLyrWIgkH9wcDBFEPZO9r5royyXbkPszM=; b=rObH8Xv1smrxW3ti4DvJH4xYwdM24DS/JVGZok9eC8WL1tWhwBy2lxNUiaBUuSZJ4j USyifnq/NDQpM6XE1bPvKRSEO/fU9STVQ5XagK8nKirMo+AJpOhBP0sZA2Kgw+VvjrrF dRqzvo6oUmbHHZ+pVkrCyM6XQrzA0al/Nkv0y+2+ReH8APnRD9Q9y66acRDps4hXCMaj 0QhJDg4D856AUFIRXg7b4i/dvJiKzyKP7jsrmzBYzHrAxuRWJ7FMEFUtx2Fqo4duGaaQ qB2vpUdINYr4OrTQ+n0r4xj/+JBe9wqG/HLPx3PuK6C5myL7Sh8dLACDreW1KD3wB5PD /KRA== X-Gm-Message-State: AJaThX494WCGv8gs3ts/QhzCqwOTeGMcO2Nl+iivO22aERVKhVID3hsD 7LgLjRs+f+X3ypJTJq3Sby5fOHYgvDc= X-Google-Smtp-Source: AGs4zMb416hW/35IIKGKmxO+LzubmkdxPtdvJXKkuLDu0Uvk6va7LTAbJxT+uYVIqnY6E+XvCnWX0Q== X-Received: by 10.46.15.25 with SMTP id 25mr2779272ljp.119.1511543881502; Fri, 24 Nov 2017 09:18:01 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id e74sm1404384ljf.43.2017.11.24.09.17.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Nov 2017 09:18:00 -0800 (PST) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Scott Long Cc: Warner Losh , FreeBSD FS , freebsd-geom@freebsd.org References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> From: Andriy Gapon Message-ID: Date: Fri, 24 Nov 2017 19:17:58 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 17:18:05 -0000 On 24/11/2017 16:57, Scott Long wrote: > > >> On Nov 24, 2017, at 6:34 AM, Andriy Gapon wrote: >> >> On 24/11/2017 15:08, Warner Losh wrote: >>> >>> >>> On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon >> > wrote: >>> >>> >>> https://reviews.freebsd.org/D13224 >>> >>> Anyone interested is welcome to join the review. >>> >>> >>> I think it's a really bad idea. It introduces a 'one-size-fits-all' notion of >>> QoS that seems misguided. It conflates a shorter timeout with don't retry. And >>> why is retrying bad? It seems more a notion of 'fail fast' or so other concept. >>> There's so many other ways you'd want to use it. And it uses the same return >>> code (EIO) to mean something new. It's generally meant 'The lower layers have >>> retried this, and it failed, do not submit it again as it will not succeed' with >>> 'I gave it a half-assed attempt, and that failed, but resubmission might work'. >>> This breaks a number of assumptions in the BUF/BIO layer as well as parts of CAM >>> even more than they are broken now. >>> >>> So let's step back a bit: what problem is it trying to solve? >> >> A simple example. I have a mirror, I issue a read to one of its members. Let's >> assume there is some trouble with that particular block on that particular disk. >> The disk may spend a lot of time trying to read it and would still fail. With >> the current defaults I would wait 5x that time to finally get the error back. >> Then I go to another mirror member and get my data from there. > > There are many RAID stacks that already solve this problem by having a policy > of always reading all disk members for every transaction, and throwing away the > sub-transactions that arrive late. It’s not a policy that is always desired, but it > serves a useful purpose for low-latency needs. That's another possible and useful strategy. >> IMO, this is not optimal. I'd rather pass BIO_NORETRY to the first read, get >> the error back sooner and try the other disk sooner. Only if I know that there >> are no other copies to try, then I would use the normal read with all the retrying. >> > > I agree with Warner that what you are proposing is not correct. It weakens the > contract between the disk layer and the upper layers, making it less clear who is > responsible for retries and less clear what “EIO” means. That contract is already > weak due to poor design decisions in VFS-BIO and GEOM, and Warner and I > are working on a plan to fix that. Well... I do realize now that there is some problem in this area, both you and Warner mentioned it. But knowing that it exists is not the same as knowing what it is :-) I understand that it could be rather complex and not easy to describe in a short email... But then, this flag is optional, it's off by default and no one is forced to used it. If it's used only by ZFS, then it would not be horrible. Unless it makes things very hard for the infrastructure. But I am circling back to not knowing what problem(s) you and Warner are planning to fix. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Nov 24 17:21:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6ECE4DEC737; Fri, 24 Nov 2017 17:21:01 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f44.google.com (mail-lf0-f44.google.com [209.85.215.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EEA307C82E; Fri, 24 Nov 2017 17:21:00 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f44.google.com with SMTP id k66so26187653lfg.3; Fri, 24 Nov 2017 09:21:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=DDjvUMqXfUPMiy37wIeC40DYy9cXSNrEzbE7WjarZrI=; b=gNyJjZr6bI99/qbi+V8XV4IA1T1S2+/fRUqb39AZDQCgg3PX7D9gN2ysLvQtwLZ8H5 Yo7Gsdk1wkVEYO+y2LIqqkrKyj+zXF6gn6cC6Ur4JEHvNvdhgpvFlFWZoAWcm/yHbm1L Mce63ad4OhogFX0zKcswSZgMd6KnYhT3d1CKey2eKYlHZ4+BnOHBC3d1kcFB8xdpgbWh tjAtXW8h8AtAf4r6HBVG7kKkvyNiuIqVkpAdFSKYTWa9SXhhRijAg8wrcVa7exz/u/A+ 3IKDqrTJH3I0FJWsspvQt8M35TRFxkHwiJCGPG66fezK8SHENVOnMe2zNJ+5aWwxBT6Y ySmg== X-Gm-Message-State: AJaThX7zGQlMJydDAZghpL0gzV4NGfjTRXsdMzD9x5or3fOnDNnc9pSd oxYHDrdgStmvNNdh6t87syk= X-Google-Smtp-Source: AGs4zMYf29dJUpe8orSQznnn8j7bzE3GZVUW/OlfKvye3SHBc7zy493mcDJyGGVSAlryNcXXTU3M2A== X-Received: by 10.46.117.28 with SMTP id q28mr10136527ljc.14.1511544052915; Fri, 24 Nov 2017 09:20:52 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id u17sm3771525lfi.97.2017.11.24.09.20.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Nov 2017 09:20:52 -0800 (PST) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Warner Losh Cc: FreeBSD FS , freebsd-geom@freebsd.org, Scott Long References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> From: Andriy Gapon Message-ID: Date: Fri, 24 Nov 2017 19:20:51 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 17:21:01 -0000 On 24/11/2017 18:33, Warner Losh wrote: > > > On Fri, Nov 24, 2017 at 6:34 AM, Andriy Gapon > wrote: > > On 24/11/2017 15:08, Warner Losh wrote: > > > > > > On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon > > >> wrote: > > > > > >     https://reviews.freebsd.org/D13224 > > > > > >     Anyone interested is welcome to join the review. > > > > > > I think it's a really bad idea. It introduces a 'one-size-fits-all' notion of > > QoS that seems misguided. It conflates a shorter timeout with don't retry. And > > why is retrying bad? It seems more a notion of 'fail fast' or so other concept. > > There's so many other ways you'd want to use it. And it uses the same return > > code (EIO) to mean something new. It's generally meant 'The lower layers have > > retried this, and it failed, do not submit it again as it will not succeed' with > > 'I gave it a half-assed attempt, and that failed, but resubmission might work'. > > This breaks a number of assumptions in the BUF/BIO layer as well as parts of CAM > > even more than they are broken now. > > > > So let's step back a bit: what problem is it trying to solve? > > A simple example.  I have a mirror, I issue a read to one of its members.  Let's > assume there is some trouble with that particular block on that particular disk. >  The disk may spend a lot of time trying to read it and would still fail.  With > the current defaults I would wait 5x that time to finally get the error back. > Then I go to another mirror member and get my data from there. > IMO, this is not optimal.  I'd rather pass BIO_NORETRY to the first read, get > the error back sooner and try the other disk sooner.  Only if I know that there > are no other copies to try, then I would use the normal read with all the > retrying. > > > It sounds like you are optimizing the wrong thing and taking an overly > simplistic view of quality of service. > First, failing blocks on a disk is fairly rare. Do you really want to optimize > for that case? If it can be done without any harm to the sunny day scenario, then why not? I think that 'robustness' is the word here, not 'optimization'. > Second, you're really saying 'If you can't read it fast, fail" since we only > control the software side of read retry. Am I? That's not what I wanted to say, really. I just wanted to say, if this I/O fails, don't retry it, leave it to me. This is very simple, simplistic as you say, but I like simple. > There's new op codes being proposed > that say 'read or fail within Xms' which is really what you want: if it's taking > too long on disk A you want to move to disk B. The notion here was we'd return > EAGAIN (or some other error) if it failed after Xms, and maybe do some emulation > in software for drives that don't support this. You'd tweak this number to > control performance. You're likely to get a much bigger performance win all the > time by scheduling I/O to drives that have the best recent latency. ZFS already does some latency based decisions. The things that you describe are very interesting, but they are for the future. > Third, do you have numbers that show this is actually a win? I do not have any numbers right now. What kind of numbers would you like? What kind of scenarios? > This is a terrible > thing from an architectural view. You have said this several times, but unfortunately you haven't explained it yet. > Absent numbers that show it's a big win, I'm > very hesitant to say OK. > > Forth, there's a large number of places in the stack today that need to > communicate their I/O is more urgent, and we don't have any good way to > communicate even that simple concept down the stack. That's unfortunately, but my proposal has quite little to do with I/O scheduling, priorities, etc. > Finally, the only places that ZFS uses the TRYHARDER flag are for things like > the super block if I'm reading the code right. It doesn't do it for normal I/O. Right. But for normal I/O there is ZIO_FLAG_IO_RETRY which is honored in the same way as ZIO_FLAG_TRYHARD. > There's no code to cope with what would happen if all the copies of a block > couldn't be read with the NORETRY flag. One of them might contain the data. ZFS is not that fragile :) see ZIO_FLAG_IO_RETRY above. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Nov 24 17:32:49 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF593DECBF9 for ; Fri, 24 Nov 2017 17:32:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CD0757D225 for ; Fri, 24 Nov 2017 17:32:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAOHWnXl054344 for ; Fri, 24 Nov 2017 17:32:49 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223808] zpool attach (and other commands?) fail with misleading error if a wiped disk had previously been used in a RAID array Date: Fri, 24 Nov 2017 17:32:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 17:32:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223808 --- Comment #2 from Andriy Gapon --- Doesn't g_raid print a message when it claims a disk? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Nov 24 17:20:19 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1FDE3DEC70C for ; Fri, 24 Nov 2017 17:20:19 +0000 (UTC) (envelope-from pfevre@secuserve.com) Received: from fx307.security-mail.net (mxout.security-mail.net [85.31.212.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 872ED7C7D5 for ; Fri, 24 Nov 2017 17:20:16 +0000 (UTC) (envelope-from pfevre@secuserve.com) Received: from localhost (localhost [127.0.0.1]) by fx307.security-mail.net (Postfix) with ESMTP id 404C4A9A099 for ; Fri, 24 Nov 2017 18:10:51 +0100 (CET) Received: from fx307 (localhost [127.0.0.1]) by fx307.security-mail.net (Postfix) with ESMTP id 255DCA9A08A; Fri, 24 Nov 2017 18:10:51 +0100 (CET) Received: from cg92.security-mail.net (cg92.security-mail.net [172.17.230.92]) by fx307.security-mail.net (Postfix) with ESMTPS id B8527A9A096; Fri, 24 Nov 2017 18:10:50 +0100 (CET) Received: from cg92.security-mail.net (cg92.security-mail.net [172.17.230.111]) by cg92.security-mail.net (Postfix) with SMTP id B2F183842E; Fri, 24 Nov 2017 18:10:49 +0100 (CET) Received: by backupnext (sSMTP sendmail emulation); Fri, 24 Nov 2017 18:10:49 +0100 Received: from [193.248.54.89] (account pfevre@secuserve.com HELO [192.168.9.36]) by next.optimails.com (CommuniGate Pro SMTP 6.2c3) with ESMTPSA id 84764015; Fri, 24 Nov 2017 18:10:49 +0100 X-Virus-Scanned: E-securemail, by Secumail Secumail-id: <36f0.5a18529a.b7e31.0>, <17c36.5a185299.b238c.0> To: freebsd-fs@freebsd.org, none From: pierre Fevre Subject: Strange ZFS object sa_magic corrupted leading to kernel panic. Message-ID: Date: Fri, 24 Nov 2017 18:10:49 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 Content-Language: fr altermime_out: done X-Mailman-Approved-At: Fri, 24 Nov 2017 18:16:55 +0000 Content-Type: text/plain; format=flowed Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 17:20:19 -0000 Hi all, I'm having an unpredictable kernel panic on a Freebsd 10.4 server with a=20 2.6To=20Dataset. I managed to get a screenshot of the kernel Panic: Basically I got this: panic: solaris assert: sa.sa_magic =3D=3D (0x584530ab =3D=3D 0x2f505a),= file zfs_vfsops.c, line 610 KBD: stack backtrace: assfail3+0x2f zfs_space_delta_cb+0x107 dmu_objset_userquota_get_ids+0x36f First of all here are the dataset property: PROPERTY=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 VALUE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 SOURCE type=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 filesystem=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 - creation=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Tue Sep 26 17:30 2017=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 - used=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4.25T=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - available=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 2.14T=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 - referenced=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 3.02T=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 - compressratio=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1.17x=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= - mounted=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 yes=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - quota=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 none=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default reservation=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 non= e=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 default recordsize=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 128K=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 default mountpoint=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 /mnt local sharenfs=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 off=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default checksum=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 on=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default compression=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 on= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 inherited from zdata atime=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 off=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 inherited from zdata devices=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 on=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default exec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 on=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default setuid=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 on=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default readonly=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 off=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default jailed=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 off=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default snapdir=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 hidden=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 default aclmode=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 discard=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 default aclinherit=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 restricted=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default canmount=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 on=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default xattr=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 off=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 temporary copies=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default version=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 5=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - utf8only=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 off=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - normalization=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 none=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 - casesensitivity=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sensitive=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - vscan=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 off=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default nbmand=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 off=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default sharesmb=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 off=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default refquota=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 none=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default refreservation=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 none=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= default primarycache=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 all=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 default secondarycache=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 all=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 default usedbysnapshots=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1.23T=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - usedbydataset=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3.02T=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= - usedbychildren=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1.60G=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - usedbyrefreservation=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - logbias=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 latency=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 default dedup=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 off=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default mlslabel=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - sync=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 standard=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default refcompressratio=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1.15x=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - written=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - logicalused=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4.9= 1T=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 - logicalreferenced=C2=A0=C2=A0=C2=A0=C2=A0 3.49T=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - volmode=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 default=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 default filesystem_limit=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 none=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default snapshot_limit=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 none=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= default filesystem_count=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 none=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 default snapshot_count=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 none=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= default redundant_metadata=C2=A0=C2=A0=C2=A0 most=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 local I've started some investigation with ZDB and I found some specific=20 object=20corrupted: =C2=A0=C2=A0=C2=A0 1=C2=B0) First test: *zdb -vvbcc -e -AAA -LG -d NAS/Bac= kup/dataset=20 *after=20a long listing it ends with zdb 100% cpu loop with: ... 37236412 1 128K 3.00K 8K 3.00K 100.00 ZFS plain file 37236422 1 128K 5.00K 8K 5.00K 100.00 ZFS plain file 37236552 1 128K 37.0K 16K 37.0K 100.00 ZFS plain file 37236656 1 128K 12.5K 16K 12.5K 100.00 ZFS plain file 37236660 1 128K 12.5K 16K 12.5K 100.00 ZFS plain file 37236668 1 128K 12.5K 16K 12.5K 100.00 ZFS plain file ZFS_DBGMSG(zdb): spa=3DNAS async request task2 ^C =C2=A0=C2=A0=C2=A0 2=C2=B0) Then on that specific object with *zdb -vvbcc = -LG -AAA -e -dddd=20 NAS/Backup/dataset=2037236668* Dataset NAS/Backup/dataset [ZPL], ID 385164, cr_txg 182709, 2.81T, 9596843 = objects, rootbp DVA[0]=3D<1:190a69bf0000:3000> DVA[1]=3D<0:6f571c41000:3000= > [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P = birth375667L/3375667P fill=9596843 cksum=11d9ece7b8:69145d8bcf8:1447eb6a460= 63:2b83132cd77e1a Object lvl iblk dblk dsize lsize %full type 37236668 1 128K 12.5K 16K 12.5K 100.00 ZFS plain file (K=3Di= nherit) (Z=3Dinherit) 264 bonus ZFS znode dnode flags: USED_BYTES USERUSED_ACCOUNTED dnode maxblkid: 0 path /mailbox/account/INBOX.mdir/101610-S_______-20170930142954= -288 uid 1002 gid 1002 atime Sat Sep 30 16:29:54 2017 mtime Sat Sep 30 16:29:54 2017 ctime Sun Oct 1 10:33:16 2017 crtime Sat Sep 30 16:29:54 2017 gen 12273136 mode 100660 size 12708 parent 19189512 links 1 pflags 40800000004 xattr 0 rdev 0x0000000000000000 Indirect blocks: 0 L0 1:1c49fe739000:6000 3200L/1a00P F=3D1 B!59816/2159816 segment [0000000000000000, 0000000000003200) size 12.5K ZFS_DBGMSG(zdb): spa=3DNAS async request task2 =C2=A0=C2=A0=C2=A0 3=C2=B0) I finally removed the bogus object 37236668 Bu= t still I've got=20 the=20same error on *zdb -vvbcc -e -AAA -LG -d NAS/Backup/dataset* 37236660 1 128K 12.5K 16K 12.5K 100.00=20 ZFS plain file 37236668 1 128K 12.5K 16K 12.5K 100.00 ZFS plain file ZFS_DBGMSG(zdb): spa=3DNAS async request task2 So my final thought is that the problem is coming from the freespace=20 map.=20I'm actually investigating with *zdb -vvvmm -AAA -LG zdata *it ends= =20 also=20with endless zdb loop at 100 %cpu with: ... [ 53] F range: 027ae20000-027be20000 size: 1000000 [ 54] F range: 027be20000-027bfff000 size: 1df000 [ 55] FREE: txg 12262024, pass 1 [ 56] F range: 0278e00000-0278e20000 size: 020000 [ 57] FREE: txg 12262025, pass 1 [ 58] F range: 0278340000-0278360000 size: 020000 ZFS_DBGMSG(zdb): ^C I'm starting to get out of idea, and I don't even know if my=20 investigations=20are pointing to something.... This dataset is an Old dataset created in zfs v3 and upgraded in v5 and=20 has=20been send and received several time for the testing purpose. Thanks for any lead on a solution. Otherwhise I'm good for 2.6To rsync=20 and=20a huge service downtime. Regards, Pierre PIERRE F=C3=A8VRE SECUSERVE LYON =C2=A0=20 =20 Retrouvez nos nouveaut=C3=A9s sur notre BLOG Messaging as a Servi= ce=20=20=20=20=20 =C2=A0=20 From owner-freebsd-fs@freebsd.org Fri Nov 24 19:56:21 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 01AA7DF013A for ; Fri, 24 Nov 2017 19:56:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E452B335A for ; Fri, 24 Nov 2017 19:56:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vAOJuK2I021639 for ; Fri, 24 Nov 2017 19:56:20 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 223846] msdosfs does not reflect READONLY to user Date: Fri, 24 Nov 2017 19:56:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 19:56:21 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223846 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |patch --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Nov 25 08:22:40 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 796F0DBBDB8 for ; Sat, 25 Nov 2017 08:22:40 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f42.google.com (mail-lf0-f42.google.com [209.85.215.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 04ED7799C6 for ; Sat, 25 Nov 2017 08:22:39 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f42.google.com with SMTP id w23so27570371lfd.11 for ; Sat, 25 Nov 2017 00:22:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=3OA1nhV5idHQmMn9bWEBW9cBXcBXQuIfEjt4Ff/GIdY=; b=uaLIZw0fR8hHmz+cxjJ7hGHjQNOIcZRjUYrKh5NvE79k03wktE1no3ttF6uCDIH1yO 8K9J9JlG48cVhJXDzVuNYvJkuFMS3wdy7CWQ095CScSTMW6SBdMdPUEI7+wuOfQdPa7/ w7wnOXF73HsxpNJr+dDzLnI6MiIG5S74TWzZjk1X7J69oPQRrQpUmk9QO9zDgwa1n5Lj qjgv8AsjRm2B7VTx+bJ/ugbEhZldor4JdqVlmqGX+YP5i5u5gquZnw7KXbs2Ml8K1nRq icDlVFuE5fvpcjNH9pO3g/K3LprK63A5xfaQWPtGXw3lMKZ+GZxgt5Q7b7i0yYh8KASt vNvg== X-Gm-Message-State: AJaThX7M6UD1G0KOLJOoOi59eW23JlMjaV86yPGpfeaSHZGQJPDBhrDr AHVOalWDPStpRafRbi92erMEqNBbb1o= X-Google-Smtp-Source: AGs4zMZWr1oYr44ngyDp9M2WmtGu/i2Sf5gcDBRppEpD17pigeC6eL9LIRj2TwT0Whs3IWU/q2AJdg== X-Received: by 10.46.86.149 with SMTP id k21mr10015800lje.105.1511598152003; Sat, 25 Nov 2017 00:22:32 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id o28sm4018529lfc.50.2017.11.25.00.22.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 25 Nov 2017 00:22:31 -0800 (PST) Subject: Re: Strange ZFS object sa_magic corrupted leading to kernel panic. To: pierre Fevre , freebsd-fs@freebsd.org, none References: From: Andriy Gapon Message-ID: Date: Sat, 25 Nov 2017 10:22:30 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 08:22:40 -0000 On 24/11/2017 19:10, pierre Fevre wrote: > I'm having an unpredictable kernel panic on a Freebsd 10.4 server with a 2.6To > Dataset. > I managed to get a screenshot of the kernel Panic: Basically I got this: > >    panic: solaris assert: sa.sa_magic == (0x584530ab == 0x2f505a), file > zfs_vfsops.c, line 610 >    KBD: stack backtrace: >    assfail3+0x2f >    zfs_space_delta_cb+0x107 >    dmu_objset_userquota_get_ids+0x36f Please see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216586 It is likely that you have the same problem: $ date -j -f %s $((0x584530ab)) Mon 5 Dec 2016 11:17:31 EET It's a known and understood issue, but no one so far has been motivated enough to pursue it further. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Sat Nov 25 10:54:06 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2DE1DDE418; Sat, 25 Nov 2017 10:54:06 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 74B507D2CE; Sat, 25 Nov 2017 10:54:06 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 3DBAD20C14; Sat, 25 Nov 2017 05:54:04 -0500 (EST) Received: from frontend1 ([10.202.2.160]) by compute6.internal (MEProxy); Sat, 25 Nov 2017 05:54:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsco.org; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=RaB/AKEW6ssBr1WgIVTDCxOVmtpaK 56Amk4OLGmTdd0=; b=hktOnAyGXDhIs2jROcS4re2WNoJh4EeJcnIG3biKqwep0 akgP6d18WTEK6J1ICq9cpd2J+xSeYABCXdEmdICogmastpBYdhIKhtfzvndsl79D i5EmDeyE93bqaV04cBchYRjRnZETgmhl93xVqXNR+pHLdmkJFNnCCqcDR10JHNKd sW4nmpmR4WvxZgVG7LkbmFiaQRBshJ/11h3azhVaDWOz4j+npO8EkjMMqVwGMFoz O0r8uzyyxCywUzYnmpMcFubBfZZqzMvrMltmhIS722Ke6+buQ3DeHJVqz5eiyu8N p4rAYrqIiGUwhoJzNuD35XfIldWe74sR5ZzBnOFvA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=RaB/AK EW6ssBr1WgIVTDCxOVmtpaK56Amk4OLGmTdd0=; b=m6bVt2S+eF+rrbG6orqHnO zLcu2MALeeuZEucMSts3+TTTGB/11L6qVJz04n5Hzhy36weGuekbGzE7peUo/W5v wkAa4wi5FhjOTO0BIJGwKS5raiEIdaPRTsl5aSmvzrLrio76NFtRRClWs+1+yesY +QW8EfoEgQ3Sh+3je5TIy/j5sC4GHZj0e4DBQ5UT3LttTzXU4ZU1NcRc499IsvFq c/ulQSKhAl+o+AedGUD43cBWkuaZl0s57+wqEvEbfQ6uzieN7QeVYEXoYpHlpWBq zAXov7tTEaWXhk9icxs1hduwtOYNUXsHBluVO5bAZB6ccqa9uCjhU4Pb3LVUXdKQ == X-ME-Sender: Received: from [192.168.0.106] (unknown [161.97.249.191]) by mail.messagingengine.com (Postfix) with ESMTPA id BDD0E7E6EE; Sat, 25 Nov 2017 05:54:03 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom From: Scott Long In-Reply-To: Date: Sat, 25 Nov 2017 03:54:01 -0700 Cc: Warner Losh , FreeBSD FS , freebsd-geom@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> To: Andriy Gapon X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 10:54:06 -0000 > On Nov 24, 2017, at 10:17 AM, Andriy Gapon wrote: >=20 >=20 >>> IMO, this is not optimal. I'd rather pass BIO_NORETRY to the first = read, get >>> the error back sooner and try the other disk sooner. Only if I know = that there >>> are no other copies to try, then I would use the normal read with = all the retrying. >>>=20 >>=20 >> I agree with Warner that what you are proposing is not correct. It = weakens the >> contract between the disk layer and the upper layers, making it less = clear who is >> responsible for retries and less clear what =E2=80=9CEIO=E2=80=9D = means. That contract is already >> weak due to poor design decisions in VFS-BIO and GEOM, and Warner and = I >> are working on a plan to fix that. >=20 > Well... I do realize now that there is some problem in this area, = both you and > Warner mentioned it. But knowing that it exists is not the same as = knowing what > it is :-) > I understand that it could be rather complex and not easy to describe = in a short > email=E2=80=A6 >=20 There are too many questions to ask, I will do my best to keep the = conversation logical. First, how do you propose to distinguish between EIO due to a = lengthy set of timeouts, vs EIO due to an immediate error returned by the disk = hardware? CAM has an extensive table-driven error recovery protocol who=E2=80=99s = purpose is to decide whether or not to do retries based on hardware state information = that is not made available to the upper layers. Do you have a test case that = demonstrates the problem that you=E2=80=99re trying to solve? Maybe the error = recovery table is wrong and you=E2=80=99re encountering a case that should not be retried. If = that=E2=80=99s what=E2=80=99s going on, we should fix CAM instead of inventing a new work-around. Second, what about disk subsystems that do retries internally, out of = the control of the FreeBSD driver? This would include most hardware RAID = controllers. Should what you are proposing only work for a subset of the kinds of = storage systems that are available and in common use? Third, let=E2=80=99s say that you run out of alternate copies to try, = and as you stated originally, that will force you to retry the copies that had returned = EIO. How will you know when you can retry? How will you know how many times you will retry? How will you know that a retry is even possible? Should = the retries be able to be canceled? Why is overloading EIO so bad? brelse() will call bdirty() when a = BIO_WRITE command has failed with EIO. Calling bdirty() has the effect of = retrying the I/O. This disregards the fact that disk drivers only return EIO when = they=E2=80=99ve decided that the I/O cannot be retried. It has no termination condition for the = retries, and will endlessly retry I/O in vain; I=E2=80=99ve seen this quite = frequently. It also disregards the fact that I/O marked as B_PAGING can=E2=80=99t be retried in this = fashion, and will trigger a panic. Because we pretend that EIO can be retried, we are = left with a system that is very fragile when I/O actually does fail. Instead of = adding more special cases and blurred lines, I want to go back to enforcing = strict contracts between the layers and force the core parts of the system to = respect those contracts and handle errors properly, instead of just retrying and hoping for the best. > But then, this flag is optional, it's off by default and no one is = forced to > used it. If it's used only by ZFS, then it would not be horrible. > Unless it makes things very hard for the infrastructure. > But I am circling back to not knowing what problem(s) you and Warner = are > planning to fix. >=20 Saying that a feature is optional means nothing; while consumers of the = API might be able to ignore it, the producers of the API cannot ignore it. = It is these producers who are sick right now and should be fixed, instead of creating new ways to get even more sick. Scott From owner-freebsd-fs@freebsd.org Sat Nov 25 11:37:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B0F0DDF02E; Sat, 25 Nov 2017 11:37:35 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 0C77E7E234; Sat, 25 Nov 2017 11:37:35 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.55.3]) by phk.freebsd.dk (Postfix) with ESMTP id 3507927347; Sat, 25 Nov 2017 11:37:27 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.15.2/8.15.2) with ESMTP id vAPBbAan030380; Sat, 25 Nov 2017 11:37:11 GMT (envelope-from phk@phk.freebsd.dk) To: Scott Long cc: Andriy Gapon , FreeBSD FS , Warner Losh , freebsd-geom@freebsd.org Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom In-reply-to: From: "Poul-Henning Kamp" References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-ID: <30378.1511609830.1@critter.freebsd.dk> Content-Transfer-Encoding: quoted-printable Date: Sat, 25 Nov 2017 11:37:10 +0000 Message-ID: <30379.1511609830@critter.freebsd.dk> X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 11:37:35 -0000 -------- In message , Scott Long w= rites: > Why is overloading EIO so bad? brelse() will call bdirty() when a BIO_W= RITE > command has failed with EIO. Calling bdirty() has the effect of retryin= g the I/O. > This disregards the fact that disk drivers only return EIO when they=E2=80= =99ve decided > that the I/O cannot be retried. It has no termination condition for the= retries, and > will endlessly retry I/O in vain; I=E2=80=99ve seen this quite frequentl= y. The really annoying thing about this particular class of errors, is that if we propagated them up to the filesystems, very often things could be relocated to different blocks and we would avoid the unnecessary filesystem corruption. The real fundamental deficiency is that we do not have a way to say "give = up if this bio cannot be completed in X time" which is what people actually w= ant. That is suprisingly hard to provide, there are far too many corner-cases for me to enumerate them all, but let me just give one example: Imagine you issue a deadlined write to a RAID5 thing. Thee component writes happen smoothly, but the last two fail the deadline, with no way to predict how long time it will take before they complete or fail. * Does the bio write transaction fail ? * Does the bio write transaction time out ? * Do you attempt to complete the write to the RAID5 ? * Where do you store a copy of the data if you do ? * What happens next time a read happens on this bio's extent ? Then for an encore, imagine it was a read bio: Three DMAs go smoothly, two are outstanding and you don't know if/when they will complete/fail. * If you fail or time out the bio, how do you "taint" the space being read into until the two remaining DMAs are outstanding? * What if that space is mapped into userland ? * What if that space is being executed ? * What if one of the two outstanding DMAs later return garbage ? My conclusion back when I did GEOM, was that the only way to do something like this sanely, is to have a special GEOM do it for you, which always allocates a temp-space: allocate temp buffer if (write) copy write data to temp buffer issue bio downwards on temp buffer if timeout park temp buffer until biodone return(timeout) if (read) copy temp buffer to read space return (ok/error) -- = Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe = Never attribute to malice what can adequately be explained by incompetence= . From owner-freebsd-fs@freebsd.org Sat Nov 25 16:25:06 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2BD92DEA369 for ; Sat, 25 Nov 2017 16:25:06 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E0E626517D for ; Sat, 25 Nov 2017 16:25:05 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x236.google.com with SMTP id g73so32152947ioj.8 for ; Sat, 25 Nov 2017 08:25:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=INwbRnB4Pq35fT3wBMoGQsVmSwWa8Uzwez6+ZA+Wong=; b=o4Lmv1sLgXpR/W5FxIK2W45MqgEm3eW1Q5ufZLaYxdVob4ZMzks09gm/T4uP9N9Czp CIHLWgrcFtzMHk72QQ30gTLcb7ueVSnkl93kNn9yZ2SiivsEjgC4hZ+Wd1xdGYVwPt69 Bxa9DOJWxg0j9GZK0CuG1hXsqoepbmoFe3tW9Os9vcifNd74bt1jhLuESf4WAYLmZq4t iGVRmCO+CDylOKHMKdtfxC6brOQpXAwYYgE+7CmtDg1WF3dAqOKduG47bgQroIouiM9+ 7Xv11ziBUg9z51DMiF9yVBGvAXXp2xqr3QCeRv0kPRTBGC8q/VWbv80T1JOh8gOU2BGl X59A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=INwbRnB4Pq35fT3wBMoGQsVmSwWa8Uzwez6+ZA+Wong=; b=aTSKjLksQ5vHkhUo00gK1W0iTQu0EX8o2vz7h/JBYJtqfOSW4Kz57jajdn7XtmZ/ls UtVY9fbfWum8CCY0j4I7JcNal6pHT/YB4uENHI1kIP14Xtn3JGNoNaEl0sq1WdXbxSW6 eUzNa/MfpJ+/VtsC0Nz37d9OwRa8p1hcYkzmsfcU8Xb3EW7QyQPsl088HOToLMkfypT6 XSSRlkeK7AWRXHUtehcxQDVnseGvB73WNFNrJqhvb/pQyNM5v7w+Lg/wgzV7DHRSiucu aAg47NK4Sv10tW7jlf0pyGI9bUgsxxs9/XF0kLsWUKCJH+xIZfY+UwWOhElZNWOlCPa0 HX1Q== X-Gm-Message-State: AJaThX6blUDrZbR7SIhgiUGXkHFtREatn/P+o5h6G4NEp2T2H1rn+KXz FZzlZT4DKt/b6/DiiYabPEXjvtKRqFjl51VvZqo+0A== X-Google-Smtp-Source: AGs4zMZe+RePIe6UOmoCphMhiNYDZUz/+iu/mFebQM7KQ1V+YUeenNjnfy6CxK0Ko06+ny1hO6AXLOlxM8WMxivNo20= X-Received: by 10.107.81.24 with SMTP id f24mr35218615iob.63.1511627105118; Sat, 25 Nov 2017 08:25:05 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.108.204 with HTTP; Sat, 25 Nov 2017 08:25:04 -0800 (PST) X-Originating-IP: [2603:300b:6:5100:9579:bb73:7b7f:aadd] In-Reply-To: References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> From: Warner Losh Date: Sat, 25 Nov 2017 09:25:04 -0700 X-Google-Sender-Auth: dkGLRA2lZh9ar81kx7rluRa6pgo Message-ID: Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Andriy Gapon Cc: Scott Long , FreeBSD FS , freebsd-geom@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 16:25:06 -0000 On Fri, Nov 24, 2017 at 10:17 AM, Andriy Gapon wrote: > On 24/11/2017 16:57, Scott Long wrote: > > > > > >> On Nov 24, 2017, at 6:34 AM, Andriy Gapon wrote: > >> > >> On 24/11/2017 15:08, Warner Losh wrote: > >>> > >>> > >>> On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon >>> > wrote: > >>> > >>> > >>> https://reviews.freebsd.org/D13224 D13224> > >>> > >>> Anyone interested is welcome to join the review. > >>> > >>> > >>> I think it's a really bad idea. It introduces a 'one-size-fits-all' > notion of > >>> QoS that seems misguided. It conflates a shorter timeout with don't > retry. And > >>> why is retrying bad? It seems more a notion of 'fail fast' or so othe= r > concept. > >>> There's so many other ways you'd want to use it. And it uses the same > return > >>> code (EIO) to mean something new. It's generally meant 'The lower > layers have > >>> retried this, and it failed, do not submit it again as it will not > succeed' with > >>> 'I gave it a half-assed attempt, and that failed, but resubmission > might work'. > >>> This breaks a number of assumptions in the BUF/BIO layer as well as > parts of CAM > >>> even more than they are broken now. > >>> > >>> So let's step back a bit: what problem is it trying to solve? > >> > >> A simple example. I have a mirror, I issue a read to one of its > members. Let's > >> assume there is some trouble with that particular block on that > particular disk. > >> The disk may spend a lot of time trying to read it and would still > fail. With > >> the current defaults I would wait 5x that time to finally get the erro= r > back. > >> Then I go to another mirror member and get my data from there. > > > > There are many RAID stacks that already solve this problem by having a > policy > > of always reading all disk members for every transaction, and throwing > away the > > sub-transactions that arrive late. It=E2=80=99s not a policy that is a= lways > desired, but it > > serves a useful purpose for low-latency needs. > > That's another possible and useful strategy. > > >> IMO, this is not optimal. I'd rather pass BIO_NORETRY to the first > read, get > >> the error back sooner and try the other disk sooner. Only if I know > that there > >> are no other copies to try, then I would use the normal read with all > the retrying. > >> > > > > I agree with Warner that what you are proposing is not correct. It > weakens the > > contract between the disk layer and the upper layers, making it less > clear who is > > responsible for retries and less clear what =E2=80=9CEIO=E2=80=9D means= . That contract > is already > > weak due to poor design decisions in VFS-BIO and GEOM, and Warner and I > > are working on a plan to fix that. > > Well... I do realize now that there is some problem in this area, both > you and > Warner mentioned it. But knowing that it exists is not the same as > knowing what > it is :-) > I understand that it could be rather complex and not easy to describe in = a > short > email... > > But then, this flag is optional, it's off by default and no one is forced > to > used it. If it's used only by ZFS, then it would not be horrible. > Except that it isn't the same flag as what Solaris has (its B_FAILFAST does something different: it isn't about limiting retries but about failing ALL the queued I/O for a unit, not just trying one retry), and the problems that it solves are quite rare. And if you return a different errno, then the EIO contract is still fulfilled. > Unless it makes things very hard for the infrastructure. > But I am circling back to not knowing what problem(s) you and Warner are > planning to fix. > The middle layers of the I/O system are a bit fragile in the face of I/O errors. We're fixing that. Of course, you still haven't articulated why this approach would be better, nor show any numbers as to how it makes things better. Warner From owner-freebsd-fs@freebsd.org Sat Nov 25 16:36:31 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35785DEA70E for ; Sat, 25 Nov 2017 16:36:31 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x232.google.com (mail-io0-x232.google.com [IPv6:2607:f8b0:4001:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EA7286579E for ; Sat, 25 Nov 2017 16:36:30 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x232.google.com with SMTP id x63so32167206ioe.6 for ; Sat, 25 Nov 2017 08:36:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=PAGMPg1oviQBjanQMuXwwRvreJHA+xXSsbF+rVnmABo=; b=jwHO36iyQQyGYD/VJxZLY+Y1Slru/PtgkR7iCGVoSnfmXTMd/DE5Dwk3ExlWJ77/n8 6QXOTuIUWi1p1iP4G5bpPHqQj1DmBiDwAokIONmXoTlM+njgNK8eYlevnbbow9B0pc+U 5aGyyipRR0UZqSRLKebh0+42VMwnqNRS9oecjKD5mMSpQesXBDI+Xh3Mkg6yiPyE5RjQ /OaifUrOwdVeethDdE86HE1hdhGckNlYrA9euWMPn699PJ7l7Hk3dfFp+Q9Dok5fT9Oe UwzfJnK2OURLd9z4p1E1CeA8DIO64YpH4Zm31e+ruacOonWrF0yL/7f5ldJvcov9Z1G+ rnlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=PAGMPg1oviQBjanQMuXwwRvreJHA+xXSsbF+rVnmABo=; b=koKu0zDFs8pDtoILrcCq1z0JMBr0F6sURKAb3dFfHxUhuaVu0cOr80h5PGitOagOn7 Y47LftTn9P2QP7UnBEmRfB6YjLUGSGiyHAiocG/GGLiseM7UH6z6UhbrMc4PqyagyrGV 40OvGpt+oKCTssibdY5nVykELa4ckTl3rqD3tde2pDln+h2cF+ChaebYvE7I36RnjowV a6ms1dmjjzgNtgtP1jodhlXhi1SFolsQE8cGftDNJBlWwYj4IZSROTlk64p+qI6VNvXd FiOS0Y9e3uYWLFf4q9kC+ihkkuTxOoVbo0ExwytsBrNf22zqYvNjVyOftOS6VsRofg5k GTrQ== X-Gm-Message-State: AJaThX4+KavuFDOtdAQsOz4qBHDTknAuGIWIqhVqFsEbfsUysaPuclaV YJbgqzfS2CtFnnfVyC4xWlUtqmYco92Eo2amOhQHQQ== X-Google-Smtp-Source: AGs4zMaOq0NE2XAnPb2aSMAenvQYIra9r9I9OqDGjqHPMAzZj/hOPvKqt8rkqyfWTC0aR1H3wfmt/f6FieY5jmzAZpU= X-Received: by 10.107.30.81 with SMTP id e78mr22143577ioe.130.1511627790118; Sat, 25 Nov 2017 08:36:30 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.108.204 with HTTP; Sat, 25 Nov 2017 08:36:29 -0800 (PST) X-Originating-IP: [2603:300b:6:5100:9579:bb73:7b7f:aadd] In-Reply-To: References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> From: Warner Losh Date: Sat, 25 Nov 2017 09:36:29 -0700 X-Google-Sender-Auth: cPV8WY30lXU33pTYiZKqTUQRXtA Message-ID: Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Andriy Gapon Cc: FreeBSD FS , freebsd-geom@freebsd.org, Scott Long Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 16:36:31 -0000 On Fri, Nov 24, 2017 at 10:20 AM, Andriy Gapon wrote: > On 24/11/2017 18:33, Warner Losh wrote: > > > > > > On Fri, Nov 24, 2017 at 6:34 AM, Andriy Gapon > > wrote: > > > > On 24/11/2017 15:08, Warner Losh wrote: > > > > > > > > > On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon > > > >> wrote: > > > > > > > > > https://reviews.freebsd.org/D13224 > > D13224 > > > > > > > > > Anyone interested is welcome to join the review. > > > > > > > > > I think it's a really bad idea. It introduces a > 'one-size-fits-all' notion of > > > QoS that seems misguided. It conflates a shorter timeout with > don't retry. And > > > why is retrying bad? It seems more a notion of 'fail fast' or so > other concept. > > > There's so many other ways you'd want to use it. And it uses the > same return > > > code (EIO) to mean something new. It's generally meant 'The lower > layers have > > > retried this, and it failed, do not submit it again as it will not > succeed' with > > > 'I gave it a half-assed attempt, and that failed, but resubmission > might work'. > > > This breaks a number of assumptions in the BUF/BIO layer as well > as parts of CAM > > > even more than they are broken now. > > > > > > So let's step back a bit: what problem is it trying to solve? > > > > A simple example. I have a mirror, I issue a read to one of its > members. Let's > > assume there is some trouble with that particular block on that > particular disk. > > The disk may spend a lot of time trying to read it and would still > fail. With > > the current defaults I would wait 5x that time to finally get the > error back. > > Then I go to another mirror member and get my data from there. > > IMO, this is not optimal. I'd rather pass BIO_NORETRY to the first > read, get > > the error back sooner and try the other disk sooner. Only if I know > that there > > are no other copies to try, then I would use the normal read with > all the > > retrying. > > > > > > It sounds like you are optimizing the wrong thing and taking an overly > > simplistic view of quality of service. > > First, failing blocks on a disk is fairly rare. Do you really want to > optimize > > for that case? > > If it can be done without any harm to the sunny day scenario, then why not? > I think that 'robustness' is the word here, not 'optimization'. I fail to see how it is a robustness issue. You've not made that case. You want the I/O to fail fast so you can give another disk a shot sooner. That's optimization. > Second, you're really saying 'If you can't read it fast, fail" since we > only > > control the software side of read retry. > > Am I? > That's not what I wanted to say, really. I just wanted to say, if this I/O > fails, don't retry it, leave it to me. > This is very simple, simplistic as you say, but I like simple. Right. Simple doesn't make it right. In fact, simple often makes it wrong. We have big issues with the nvd device today because it's mindlessly queues all the trim requests to the NVMe device w/o collapsing them, resulting in horrible performance. > There's new op codes being proposed > > that say 'read or fail within Xms' which is really what you want: if > it's taking > > too long on disk A you want to move to disk B. The notion here was we'd > return > > EAGAIN (or some other error) if it failed after Xms, and maybe do some > emulation > > in software for drives that don't support this. You'd tweak this number > to > > control performance. You're likely to get a much bigger performance win > all the > > time by scheduling I/O to drives that have the best recent latency. > > ZFS already does some latency based decisions. > The things that you describe are very interesting, but they are for the > future. > > > Third, do you have numbers that show this is actually a win? > > I do not have any numbers right now. > What kind of numbers would you like? What kind of scenarios? The usual kind. How is latency for I/O improved when you have a disk with a few failing sectors that take a long time to read (which isn't a given: some sectors fail fast). What happens when you have a failed disk? etc. How does this compare with the current system. Basically, how do you know this will really make things better and isn't some kind of 'feel good' thing about 'doing something clever' about the problem that may actually make things worse. > This is a terrible > > thing from an architectural view. > > You have said this several times, but unfortunately you haven't explained > it yet. I have explained it. You weren't listening. 1. It breaks the EIO contract that's currently in place. 2. It presumes to know what kind of retries should be done at the upper layers where today we have a system that's more black and white. You don't know the same info the low layers have to know whether to try another drive, or just retry this one. 3. It assumes that retries are the source of latency in the system. they aren't necessarily. 4. It assumes retries are necessarily slow: they may be, they might not be. All depends on the drive (SSDs repeated I/O are often faster than actual I/O). 5. It's just one bit when you really need more complex nuances to get good QoE out of the I/O system. Retries is an incidental detail that's not that important, while latency is what you care most about minimizing. You wouldn't care if I tried to read the data 20 times if it got the result faster than going to a different drive. 6. It's putting the wrong kind of specific hints into the mix. > Absent numbers that show it's a big win, I'm > > very hesitant to say OK. > > > > Forth, there's a large number of places in the stack today that need to > > communicate their I/O is more urgent, and we don't have any good way to > > communicate even that simple concept down the stack. > > That's unfortunately, but my proposal has quite little to do with I/O > scheduling, priorities, etc. Except it does. It dictates error recovery policy which is I/O scheduling. > Finally, the only places that ZFS uses the TRYHARDER flag are for things > like > > the super block if I'm reading the code right. It doesn't do it for > normal I/O. > > Right. But for normal I/O there is ZIO_FLAG_IO_RETRY which is honored in > the > same way as ZIO_FLAG_TRYHARD. > > > There's no code to cope with what would happen if all the copies of a > block > > couldn't be read with the NORETRY flag. One of them might contain the > data. > > ZFS is not that fragile :) see ZIO_FLAG_IO_RETRY above. > Except TRYHARD in ZFS means 'don't fail ****OTHER**** I/O in the queue when an I/O fails' It doesn't control retries at all in Solaris. It's a different concept entirely, and one badly thought out. Warner From owner-freebsd-fs@freebsd.org Sat Nov 25 16:58:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49CC2DEADA2; Sat, 25 Nov 2017 16:58:38 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f43.google.com (mail-lf0-f43.google.com [209.85.215.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EF135662E9; Sat, 25 Nov 2017 16:58:37 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f43.google.com with SMTP id g35so28372631lfi.13; Sat, 25 Nov 2017 08:58:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=sNmBbqSvUdx+P0oBQz+HZ0eKlahQ3p0XcaiUMbppq94=; b=ZbYvg6jV7ktmRiS8wYCaHCHERogGTOY8Npky+qYT3+jp5cvvlWMDTohvMf13T44Fdv k856mLl06msqEevycDCZqvaFy0z5P3D6LMuQ2Rn7DVPWeGsye+F4EWW6zh3Y7gxBsger JFKzBWv/XiRArbdOYW5TR8jZK1Lx1D5qXkXR76XQ58heVIThtOfG7cUcC5zx/5TQeIwQ cp4kxE1gfRKxSKmH/RSi+9hnLBUtUa/xjZEWYEwIFnLTo7pv+BDKliFtJ1YR/B+RCb9j dzz236S/DUb0Eq3E4iKj0A33npuswtiw1G1Gs4F+6Nd3eCdYW2xuzSt1y8Q3X52P2V5P gf6w== X-Gm-Message-State: AJaThX4KWIf7Tpc0YteQA1eyDsSBQ5luze6tAgx0c7ZK5PjfjH7VPL1F Ad3nYuZRatRAfec5FodPk03zwUJoS3g= X-Google-Smtp-Source: AGs4zMaPazeEKorA20LgfmuxaRUD4LZnRTIi/eMwruMrxb9Wv325OBtJQl3mgpdxVZyMEiz1uxJGig== X-Received: by 10.25.20.77 with SMTP id k74mr9606078lfi.80.1511629110179; Sat, 25 Nov 2017 08:58:30 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id v12sm5027560ljd.15.2017.11.25.08.58.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 25 Nov 2017 08:58:29 -0800 (PST) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Warner Losh Cc: Scott Long , FreeBSD FS , freebsd-geom@freebsd.org References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> From: Andriy Gapon Message-ID: <27c9395f-5b3c-a062-3aee-de591770af0b@FreeBSD.org> Date: Sat, 25 Nov 2017 18:58:27 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 16:58:38 -0000 On 25/11/2017 18:25, Warner Losh wrote: > > > On Fri, Nov 24, 2017 at 10:17 AM, Andriy Gapon > wrote: > > On 24/11/2017 16:57, Scott Long wrote: > > > > > >> On Nov 24, 2017, at 6:34 AM, Andriy Gapon wrote: > >> > >> On 24/11/2017 15:08, Warner Losh wrote: > >>> > >>> > >>> On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon > >>> >> wrote: > >>> > >>> > >>>    https://reviews.freebsd.org/D13224 > > > >>> > >>>    Anyone interested is welcome to join the review. > >>> > >>> > >>> I think it's a really bad idea. It introduces a 'one-size-fits-all' > notion of > >>> QoS that seems misguided. It conflates a shorter timeout with don't > retry. And > >>> why is retrying bad? It seems more a notion of 'fail fast' or so other > concept. > >>> There's so many other ways you'd want to use it. And it uses the same return > >>> code (EIO) to mean something new. It's generally meant 'The lower layers > have > >>> retried this, and it failed, do not submit it again as it will not > succeed' with > >>> 'I gave it a half-assed attempt, and that failed, but resubmission might > work'. > >>> This breaks a number of assumptions in the BUF/BIO layer as well as > parts of CAM > >>> even more than they are broken now. > >>> > >>> So let's step back a bit: what problem is it trying to solve? > >> > >> A simple example.  I have a mirror, I issue a read to one of its > members.  Let's > >> assume there is some trouble with that particular block on that > particular disk. > >> The disk may spend a lot of time trying to read it and would still fail.  > With > >> the current defaults I would wait 5x that time to finally get the error back. > >> Then I go to another mirror member and get my data from there. > > > > There are many RAID stacks that already solve this problem by having a policy > > of always reading all disk members for every transaction, and throwing > away the > > sub-transactions that arrive late.  It’s not a policy that is always > desired, but it > > serves a useful purpose for low-latency needs. > > That's another possible and useful strategy. > > >> IMO, this is not optimal.  I'd rather pass BIO_NORETRY to the first read, get > >> the error back sooner and try the other disk sooner.  Only if I know that there > >> are no other copies to try, then I would use the normal read with all the retrying. > >> > > > > I agree with Warner that what you are proposing is not correct.  It weakens the > > contract between the disk layer and the upper layers, making it less clear who is > > responsible for retries and less clear what “EIO” means.  That contract is already > > weak due to poor design decisions in VFS-BIO and GEOM, and Warner and I > > are working on a plan to fix that. > > Well...  I do realize now that there is some problem in this area, both you and > Warner mentioned it.  But knowing that it exists is not the same as knowing what > it is :-) > I understand that it could be rather complex and not easy to describe in a short > email... > > But then, this flag is optional, it's off by default and no one is forced to > used it.  If it's used only by ZFS, then it would not be horrible. > > > Except that it isn't the same flag as what Solaris has (its B_FAILFAST does > something different: it isn't about limiting retries but about failing ALL the > queued I/O for a unit, not just trying one retry), and the problems that it > solves are quite rare. And if you return a different errno, then the EIO > contract is still fulfilled.  Yes, it isn't the same. I think that illumos flag does even more. > Unless it makes things very hard for the infrastructure. > But I am circling back to not knowing what problem(s) you and Warner are > planning to fix. > > > The middle layers of the I/O system are a bit fragile in the face of I/O errors. > We're fixing that. What are the middle layers? > Of course, you still haven't articulated why this approach would be better Better than what? > nor > show any numbers as to how it makes things better. By now, I have. See my reply to Scott's email. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Sat Nov 25 17:36:31 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9894ADEBB29; Sat, 25 Nov 2017 17:36:31 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f45.google.com (mail-lf0-f45.google.com [209.85.215.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 173D96767F; Sat, 25 Nov 2017 17:36:30 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f45.google.com with SMTP id k66so28461669lfg.3; Sat, 25 Nov 2017 09:36:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=iOl1D78pVkm1p2WO2iSswIdicfhFfzoRB2Bc9x8J0XA=; b=FwAhgIqykPRVE9Q778D+OPfmyn95E5zd8SIsartEGdWg1Ssj9VpFs/Gvcwq5N5rI9H 8jCfpIwC2dfgsmCoT7YdNA79XbZLsWf5M6TxrkJ5wTl7EiCNWYVFQHmxM/tN758mxqGU CXD6NL6lpvPFgTIjKy8/3ruVgDirqO4wOwg+LFRlaXftY4mRdtYkpTaTZGIuE3sk1fRS tMSOrAqrkxaNCqFBIDnxdKwu19Mqw2DTfK9+fcU3dNtKwo/tyRcsZVseXfphZ/t0lpFo uTO7GWHARKCOSkjmUCmBoJwIxNjPyzyNYRU+oq2OoJexdlVuqRKjQrMZexOnAD8HI3ZU Aplg== X-Gm-Message-State: AJaThX4VvLGCfaJtIBGAuRmop0Jb1Xk7CKTLNuZkpUJ67LBcdxotKHfj 520NpS6rN+tiPce5ICUf/OhRAutEV/c= X-Google-Smtp-Source: AGs4zMbBpiuwlbgmOFur1itv4JJ+4S3jA/+BP4yojPXM0feB1c+MAnucw6G9lWk0Z6UcVuGQhWCJQQ== X-Received: by 10.46.84.86 with SMTP id y22mr4011842ljd.89.1511631382512; Sat, 25 Nov 2017 09:36:22 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id f66sm1845239lfl.72.2017.11.25.09.36.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 25 Nov 2017 09:36:20 -0800 (PST) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Scott Long Cc: Warner Losh , FreeBSD FS , freebsd-geom@freebsd.org References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> From: Andriy Gapon Message-ID: <33101e6c-0c74-34b7-ee92-f9c4a11685d5@FreeBSD.org> Date: Sat, 25 Nov 2017 19:36:19 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 17:36:31 -0000 On 25/11/2017 12:54, Scott Long wrote: > > >> On Nov 24, 2017, at 10:17 AM, Andriy Gapon wrote: >> >> >>>> IMO, this is not optimal. I'd rather pass BIO_NORETRY to the first read, get >>>> the error back sooner and try the other disk sooner. Only if I know that there >>>> are no other copies to try, then I would use the normal read with all the retrying. >>>> >>> >>> I agree with Warner that what you are proposing is not correct. It weakens the >>> contract between the disk layer and the upper layers, making it less clear who is >>> responsible for retries and less clear what “EIO” means. That contract is already >>> weak due to poor design decisions in VFS-BIO and GEOM, and Warner and I >>> are working on a plan to fix that. >> >> Well... I do realize now that there is some problem in this area, both you and >> Warner mentioned it. But knowing that it exists is not the same as knowing what >> it is :-) >> I understand that it could be rather complex and not easy to describe in a short >> email… >> > > There are too many questions to ask, I will do my best to keep the conversation > logical. First, how do you propose to distinguish between EIO due to a lengthy > set of timeouts, vs EIO due to an immediate error returned by the disk hardware? At what layer / component? If I am the issuer of the request then I know how I issued that request and what kind of request it was. If I am an intermediate layer, then what do I care. > CAM has an extensive table-driven error recovery protocol who’s purpose is to > decide whether or not to do retries based on hardware state information that is > not made available to the upper layers. Do you have a test case that demonstrates > the problem that you’re trying to solve? Maybe the error recovery table is wrong > and you’re encountering a case that should not be retried. If that’s what’s going on, > we should fix CAM instead of inventing a new work-around. Let's assume that I am talking about the case of not being able to read an HDD sector that is gone bad. Here is a real world example: Jun 16 10:40:18 trant kernel: ahcich0: NCQ error, slot = 20, port = -1 Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 00 58 62 40 2c 00 00 08 00 00 Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): RES: 41 40 68 58 62 40 2c 00 00 00 00 Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): Retrying command Jun 16 10:40:20 trant kernel: ahcich0: NCQ error, slot = 22, port = -1 Jun 16 10:40:20 trant kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 00 58 62 40 2c 00 00 08 00 00 Jun 16 10:40:20 trant kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error Jun 16 10:40:20 trant kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jun 16 10:40:20 trant kernel: (ada0:ahcich0:0:0:0): RES: 41 40 68 58 62 40 2c 00 00 00 00 Jun 16 10:40:20 trant kernel: (ada0:ahcich0:0:0:0): Retrying command Jun 16 10:40:22 trant kernel: ahcich0: NCQ error, slot = 24, port = -1 Jun 16 10:40:22 trant kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 00 58 62 40 2c 00 00 08 00 00 Jun 16 10:40:22 trant kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error Jun 16 10:40:22 trant kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jun 16 10:40:22 trant kernel: (ada0:ahcich0:0:0:0): RES: 41 40 68 58 62 40 2c 00 00 00 00 Jun 16 10:40:22 trant kernel: (ada0:ahcich0:0:0:0): Retrying command Jun 16 10:40:25 trant kernel: ahcich0: NCQ error, slot = 26, port = -1 Jun 16 10:40:25 trant kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 00 58 62 40 2c 00 00 08 00 00 Jun 16 10:40:25 trant kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error Jun 16 10:40:25 trant kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jun 16 10:40:25 trant kernel: (ada0:ahcich0:0:0:0): RES: 41 40 68 58 62 40 2c 00 00 00 00 Jun 16 10:40:25 trant kernel: (ada0:ahcich0:0:0:0): Retrying command Jun 16 10:40:27 trant kernel: ahcich0: NCQ error, slot = 28, port = -1 Jun 16 10:40:27 trant kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 00 58 62 40 2c 00 00 08 00 00 Jun 16 10:40:27 trant kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error Jun 16 10:40:27 trant kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC ) Jun 16 10:40:27 trant kernel: (ada0:ahcich0:0:0:0): RES: 41 40 68 58 62 40 2c 00 00 00 00 Jun 16 10:40:27 trant kernel: (ada0:ahcich0:0:0:0): Error 5, Retries exhausted I do not see anything wrong in what CAM / ahci /ata_da did here. They did what I would expect them to do. They tried very hard to get data that I told them I need. Timestamp of the first error is Jun 16 10:40:18. Timestamp of the last error is Jun 16 10:40:27. So, it took additional 9 seconds to finally produce EIO. That disk is a part of a ZFS mirror. If the request was failed after the first attempt, then ZFS would be able to get the data from a good disk much sooner. And don't take me wrong, I do NOT want CAM or GEOM to make that decision by itself. I want ZFS to be able to tell the lower layers when they should try as hard as they normally do and when they should report an I/O error as soon as it happens without any retries. > Second, what about disk subsystems that do retries internally, out of the control > of the FreeBSD driver? This would include most hardware RAID controllers. > Should what you are proposing only work for a subset of the kinds of storage > systems that are available and in common use? Yes. I do not worry about things that are beyond my control. Those subsystems would behave as they do now. So, nothing would get worse. > Third, let’s say that you run out of alternate copies to try, and as you stated > originally, that will force you to retry the copies that had returned EIO. How > will you know when you can retry? How will you know how many times you > will retry? How will you know that a retry is even possible? I am continuing to use ZFS as an example. It already has all the logic built in. If all vdev zio-s (requests to mirror members as an example) fail, then their parent zio (a logical read from the mirror) will be retried (by ZFS) and when ZFS retries it sets a special flag (ZIO_FLAG_IO_RETRY) on that zio and on its future child zio-s. Essentially, my answer is you have to program it correctly, there is no magic. > Should the retries > be able to be canceled? I think that this is an orthogonal question. I do not have any answer and I am not ready to discuss this at all. > Why is overloading EIO so bad? brelse() will call bdirty() when a BIO_WRITE > command has failed with EIO. Calling bdirty() has the effect of retrying the I/O. > This disregards the fact that disk drivers only return EIO when they’ve decided > that the I/O cannot be retried. It has no termination condition for the retries, and > will endlessly retry I/O in vain; I’ve seen this quite frequently. It also disregards > the fact that I/O marked as B_PAGING can’t be retried in this fashion, and will > trigger a panic. Because we pretend that EIO can be retried, we are left with > a system that is very fragile when I/O actually does fail. Instead of adding > more special cases and blurred lines, I want to go back to enforcing strict > contracts between the layers and force the core parts of the system to respect > those contracts and handle errors properly, instead of just retrying and > hoping for the best. So, I suggest that the buffer layer (all the b* functions) does not use the proposed flag. Any problems that exist in it should be resolved first. ZFS does not use that layer. >> But then, this flag is optional, it's off by default and no one is forced to >> used it. If it's used only by ZFS, then it would not be horrible. >> Unless it makes things very hard for the infrastructure. >> But I am circling back to not knowing what problem(s) you and Warner are >> planning to fix. >> > > Saying that a feature is optional means nothing; while consumers of the API > might be able to ignore it, the producers of the API cannot ignore it. It is > these producers who are sick right now and should be fixed, instead of > creating new ways to get even more sick. I completely agree. But which producers of the API do you mean specifically? So far, you mentioned only the consumer level problems with the b-layer. Having said all of the above, I must admit one thing. When I proposed BIO_NORETRY I had only the simplest GEOM topology in mind: ZFS -> [partition] -> disk. Now that I start to think about more complex topologies I am not really sure how the flag should be handled by geom-s with complex internal behavior. If that can be implemented reasonably and clearly, if the flag will create a big mess. E.g., things like gmirrors on top of gmirrors and so on. Maybe the flag, if it ever accepted, should never be propagated automatically. Only geom-s that are aware of it should propagate or request it. That should be safer. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Sat Nov 25 17:38:22 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0C28DEBBEC for ; Sat, 25 Nov 2017 17:38:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x235.google.com (mail-io0-x235.google.com [IPv6:2607:f8b0:4001:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB21B678DD for ; Sat, 25 Nov 2017 17:38:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x235.google.com with SMTP id v21so32360521ioi.4 for ; Sat, 25 Nov 2017 09:38:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=3iyg3cV4DFZHCSkAutindZTnEutdchf5thqOLOgiRDE=; b=sBaoG7R6xwKCU2ZqOf+FmtFLSmDy+IGCKl1rYjgYcQVPD8SX74GnlHXJjXAihqhzxB qYO1Nmusy0hdZBTlqRmtawdcgBucaQkU0Oi4tbIfNvRJu/E2quYUNAGNTvPTvGyMqzqF 0o22Xp9QXupfqN2fY2yqRsZsYLnvRU0yDtOe3kzqC5BNYKWHxtKVL84kJ6Cdtxc0IUaw b2zgXXFmsZ6mo28JPH6uzln2ufZRyYjV2DKL4MjiLRhTsiM1opHMTqoHN+68wIuDXMnW Wb6C83vs2nNGaHHQlk6LuX+AFdQqubx+QyjY6BScJsQNCu/TLG5x/xZaaZgF4FG3KN/4 Wzdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=3iyg3cV4DFZHCSkAutindZTnEutdchf5thqOLOgiRDE=; b=K1MFaE0NLX8M0gARULypF+BPvaO9cadwCkp0Gf3GMb+8SWHx3AXQOiBf2MskJNXWvx FrobNwh9eqd+LUJY4IlzVLBrgLjongvx/3p+QrneBwcZ7LRELI84dBD4e2SGWxgnC+eQ GQZcLm17mkEvpWHHDxn/GLRc87xVcYrI2IpfbcaQu7xA1Nvvict1H/FcOTdG7xUbEXdM 8AcHYjzLtz2cCdF/brSskvr0vusHL/O+hUo5hjh+TwNeWUrQkQBYZKNUCvvbQdiTkOHi qmTewmQEdRtiprY7y80idvVDiyAVs5a/t8EndjWfwr7oAd6fmcKIx9KUMFf73A9YeHwW B+Lw== X-Gm-Message-State: AJaThX77kn4ORaek2gaTn+obi8SeTJZKg+KufBEH6zOrQGePLPIfqJUh 2e0AvenHmXLjQyEEb22NLuvTfE87MuHiWsGNvWRmuQ== X-Google-Smtp-Source: AGs4zMZpYS3c6FiEoS1YalzMszvrxFrWG+Ow/PBeOV+/wjBX9x9Kdxe/x6Qd2CetfGxNr8hrU8xe+Vh7WeKktp0Fnq4= X-Received: by 10.107.104.18 with SMTP id d18mr33320248ioc.136.1511631501988; Sat, 25 Nov 2017 09:38:21 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.108.204 with HTTP; Sat, 25 Nov 2017 09:38:21 -0800 (PST) X-Originating-IP: [2603:300b:6:5100:9579:bb73:7b7f:aadd] In-Reply-To: <27c9395f-5b3c-a062-3aee-de591770af0b@FreeBSD.org> References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> <27c9395f-5b3c-a062-3aee-de591770af0b@FreeBSD.org> From: Warner Losh Date: Sat, 25 Nov 2017 10:38:21 -0700 X-Google-Sender-Auth: hRNqrbEqFn4VgCb2ByZQrwSf-oE Message-ID: Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Andriy Gapon Cc: Scott Long , FreeBSD FS , freebsd-geom@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 17:38:23 -0000 On Sat, Nov 25, 2017 at 9:58 AM, Andriy Gapon wrote: > On 25/11/2017 18:25, Warner Losh wrote: > > > > > > On Fri, Nov 24, 2017 at 10:17 AM, Andriy Gapon > > wrote: > > > > On 24/11/2017 16:57, Scott Long wrote: > > > > > > > > >> On Nov 24, 2017, at 6:34 AM, Andriy Gapon > wrote: > > >> > > >> On 24/11/2017 15:08, Warner Losh wrote: > > >>> > > >>> > > >>> On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon > > > >>> >> wrote: > > >>> > > >>> > > >>> https://reviews.freebsd.org/D13224 > > D13224 > > > > > >>> > > >>> Anyone interested is welcome to join the review. > > >>> > > >>> > > >>> I think it's a really bad idea. It introduces a > 'one-size-fits-all' > > notion of > > >>> QoS that seems misguided. It conflates a shorter timeout with > don't > > retry. And > > >>> why is retrying bad? It seems more a notion of 'fail fast' or s= o > other > > concept. > > >>> There's so many other ways you'd want to use it. And it uses th= e > same return > > >>> code (EIO) to mean something new. It's generally meant 'The > lower layers > > have > > >>> retried this, and it failed, do not submit it again as it will > not > > succeed' with > > >>> 'I gave it a half-assed attempt, and that failed, but > resubmission might > > work'. > > >>> This breaks a number of assumptions in the BUF/BIO layer as wel= l > as > > parts of CAM > > >>> even more than they are broken now. > > >>> > > >>> So let's step back a bit: what problem is it trying to solve? > > >> > > >> A simple example. I have a mirror, I issue a read to one of its > > members. Let's > > >> assume there is some trouble with that particular block on that > > particular disk. > > >> The disk may spend a lot of time trying to read it and would > still fail. > > With > > >> the current defaults I would wait 5x that time to finally get th= e > error back. > > >> Then I go to another mirror member and get my data from there. > > > > > > There are many RAID stacks that already solve this problem by > having a policy > > > of always reading all disk members for every transaction, and > throwing > > away the > > > sub-transactions that arrive late. It=E2=80=99s not a policy tha= t is > always > > desired, but it > > > serves a useful purpose for low-latency needs. > > > > That's another possible and useful strategy. > > > > >> IMO, this is not optimal. I'd rather pass BIO_NORETRY to the > first read, get > > >> the error back sooner and try the other disk sooner. Only if I > know that there > > >> are no other copies to try, then I would use the normal read wit= h > all the retrying. > > >> > > > > > > I agree with Warner that what you are proposing is not correct. > It weakens the > > > contract between the disk layer and the upper layers, making it > less clear who is > > > responsible for retries and less clear what =E2=80=9CEIO=E2=80=9D= means. That > contract is already > > > weak due to poor design decisions in VFS-BIO and GEOM, and Warner > and I > > > are working on a plan to fix that. > > > > Well... I do realize now that there is some problem in this area, > both you and > > Warner mentioned it. But knowing that it exists is not the same as > knowing what > > it is :-) > > I understand that it could be rather complex and not easy to > describe in a short > > email... > > > > But then, this flag is optional, it's off by default and no one is > forced to > > used it. If it's used only by ZFS, then it would not be horrible. > > > > > > Except that it isn't the same flag as what Solaris has (its B_FAILFAST > does > > something different: it isn't about limiting retries but about failing > ALL the > > queued I/O for a unit, not just trying one retry), and the problems tha= t > it > > solves are quite rare. And if you return a different errno, then the EI= O > > contract is still fulfilled. > > Yes, it isn't the same. > I think that illumos flag does even more. Since it isn't the same, and there's not other systems that do a similar thing, that ups the burden of proof that this is a good idea. > Unless it makes things very hard for the infrastructure. > > But I am circling back to not knowing what problem(s) you and Warne= r > are > > planning to fix. > > > > > > The middle layers of the I/O system are a bit fragile in the face of I/= O > errors. > > We're fixing that. > > What are the middle layers? The buffer cache and lower layers of the UFS code is where the problems chiefly lie. > Of course, you still haven't articulated why this approach would be bette= r > > Better than what? Well, anything? > > nor > > show any numbers as to how it makes things better. > > By now, I have. See my reply to Scott's email. I just checked my email, I've seen no such reply. I checked it before I replied. Maybe it's just delayed. Warner From owner-freebsd-fs@freebsd.org Sat Nov 25 17:41:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB5D7DEBD5E; Sat, 25 Nov 2017 17:41:03 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f42.google.com (mail-lf0-f42.google.com [209.85.215.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A68067AB8; Sat, 25 Nov 2017 17:41:03 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f42.google.com with SMTP id o41so28431705lfi.2; Sat, 25 Nov 2017 09:41:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=XPeLSjy+VQezLXaWyLqVGxRiCz8i9BQr86mV3Lj2YAY=; b=P5aLhmuYvRBS2YzstZB/VysjO8tDdJJbkA4wc5dEAqR37aBMeuaP2jmTHwBSSXeZoZ 0/2t++942PSBvM4SrNJidCkQSPuCaHdvGBF5xAkDIaPTmKg6msEv9PINiWbmNcYkLE3i vmXz5BwDSRT/pGxmmHsim4Zlm6/ifAOlBZAJRArWDncGHwtgtTrM/BjaFwrPg+9suF2q gUxNmXczOtc90V3bTXcDD6x8RCsqpVPSX2NIw8eolApVVfBsl/wF+oKJF8hpowlPprof E341dmmpia2l/OkYqRHuWxAPA/S3nnZCG8TfSC2fGc6RsaWZ8/3BeIyYuYb9VLIXjrs3 J0ug== X-Gm-Message-State: AJaThX5VQgIz99QNQimDK5dBVUgDFPyYvPs1xAHfwmF285C9KdPkF+Db 8WvrI088SaihgKke3mGbprI= X-Google-Smtp-Source: AGs4zMaKaiPdOiBetpyS/MEYxVNA+8qROI/0uCcumggNbeOONXIBzJLKs7oDi2qoq0qJD4SHl4FwIg== X-Received: by 10.46.99.211 with SMTP id s80mr11250056lje.7.1511631660987; Sat, 25 Nov 2017 09:41:00 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id m26sm5121410ljb.61.2017.11.25.09.40.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 25 Nov 2017 09:41:00 -0800 (PST) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Warner Losh Cc: FreeBSD FS , freebsd-geom@freebsd.org, Scott Long References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> From: Andriy Gapon Message-ID: <9f23f97d-3614-e4d2-62fe-99723c5e3879@FreeBSD.org> Date: Sat, 25 Nov 2017 19:40:59 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 17:41:04 -0000 Before anything else, I would like to say that I got an impression that we speak from so different angles that we either don't understand each other's words or, even worse, misinterpret them. On 25/11/2017 18:36, Warner Losh wrote: > > > On Fri, Nov 24, 2017 at 10:20 AM, Andriy Gapon > wrote: > > On 24/11/2017 18:33, Warner Losh wrote: > > > > > > On Fri, Nov 24, 2017 at 6:34 AM, Andriy Gapon > > >> wrote: > > > >     On 24/11/2017 15:08, Warner Losh wrote: > >     > > >     > > >     > On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon > > >     > >>> wrote: > >     > > >     > > >     >     https://reviews.freebsd.org/D13224 > >      > > >     >> > >     > > >     >     Anyone interested is welcome to join the review. > >     > > >     > > >     > I think it's a really bad idea. It introduces a 'one-size-fits-all' > notion of > >     > QoS that seems misguided. It conflates a shorter timeout with don't > retry. And > >     > why is retrying bad? It seems more a notion of 'fail fast' or so > other concept. > >     > There's so many other ways you'd want to use it. And it uses the > same return > >     > code (EIO) to mean something new. It's generally meant 'The lower > layers have > >     > retried this, and it failed, do not submit it again as it will not > succeed' with > >     > 'I gave it a half-assed attempt, and that failed, but resubmission > might work'. > >     > This breaks a number of assumptions in the BUF/BIO layer as well as > parts of CAM > >     > even more than they are broken now. > >     > > >     > So let's step back a bit: what problem is it trying to solve? > > > >     A simple example.  I have a mirror, I issue a read to one of its > members.  Let's > >     assume there is some trouble with that particular block on that > particular disk. > >      The disk may spend a lot of time trying to read it and would still > fail.  With > >     the current defaults I would wait 5x that time to finally get the > error back. > >     Then I go to another mirror member and get my data from there. > >     IMO, this is not optimal.  I'd rather pass BIO_NORETRY to the first > read, get > >     the error back sooner and try the other disk sooner.  Only if I know > that there > >     are no other copies to try, then I would use the normal read with all the > >     retrying. > > > > > > It sounds like you are optimizing the wrong thing and taking an overly > > simplistic view of quality of service. > > First, failing blocks on a disk is fairly rare. Do you really want to optimize > > for that case? > > If it can be done without any harm to the sunny day scenario, then why not? > I think that 'robustness' is the word here, not 'optimization'. > > > I fail to see how it is a robustness issue. You've not made that case. You want > the I/O to fail fast so you can give another disk a shot sooner. That's > optimization. Then you can call a protection against denial-of-service an optimization too. You want to do things faster, right? > > Second, you're really saying 'If you can't read it fast, fail" since we only > > control the software side of read retry. > > Am I? > That's not what I wanted to say, really.  I just wanted to say, if this I/O > fails, don't retry it, leave it to me. > This is very simple, simplistic as you say, but I like simple. > > > Right. Simple doesn't make it right. In fact, simple often makes it wrong. I agree. The same applies to complex well. Let's stop at this. > We > have big issues with the nvd device today because it's mindlessly queues all the > trim requests to the NVMe device w/o collapsing them, resulting in horrible > performance. > > > There's new op codes being proposed > > that say 'read or fail within Xms' which is really what you want: if it's taking > > too long on disk A you want to move to disk B. The notion here was we'd return > > EAGAIN (or some other error) if it failed after Xms, and maybe do some emulation > > in software for drives that don't support this. You'd tweak this number to > > control performance. You're likely to get a much bigger performance win all the > > time by scheduling I/O to drives that have the best recent latency. > > ZFS already does some latency based decisions. > The things that you describe are very interesting, but they are for the future. > > > Third, do you have numbers that show this is actually a win? > > I do not have any numbers right now. > What kind of numbers would you like?  What kind of scenarios? > > > The usual kind. How is latency for I/O improved when you have a disk with a few > failing sectors that take a long time to read (which isn't a given: some sectors > fail fast). Today I gave an example of how four retries added about 9 seconds of additional delay. I think that that is significant. > What happens when you have a failed disk? etc. How does this compare > with the current system. I haven't done such an experiment. I guess it depends on how exactly the disk fails. There is a big difference between a disk dropping a link and a disk turning into a black hole. > Basically, how do you know this will really make things better and isn't some > kind of 'feel good' thing about 'doing something clever' about the problem that > may actually make things worse. > > > This is a terrible > > thing from an architectural view. > > You have said this several times, but unfortunately you haven't explained it > yet. > > > I have explained it. You weren't listening. This is the first time I see the below list or anything like it. > 1. It breaks the EIO contract that's currently in place. This needs further explanation. > 2. It presumes to know what kind of retries should be done at the upper layers > where today we have a system that's more black and white. I don't understand this argument. If your upper level code does not know how to do retries, then it should not concern itself with that and should not use the flag. > You don't know the > same info the low layers have to know whether to try another drive, or just > retry this one. Eh? Either we have different definitions of upper and lower layers or I don't understand how lower layers (e.g. CAM) can know about another drive. > 3. It assumes that retries are the source of latency in the system. they aren't > necessarily. I am not assuming that at all for the general case. > 4. It assumes retries are necessarily slow: they may be, they might not be. All > depends on the drive (SSDs repeated I/O are often faster than actual I/O). Of course. But X plus epsilon is always greater than X. And we know than in many practical cases epsilon can be rather large. > 5. It's just one bit when you really need more complex nuances to get good QoE > out of the I/O system. Retries is an incidental detail that's not that > important, while latency is what you care most about minimizing. You wouldn't > care if I tried to read the data 20 times if it got the result faster than going > to a different drive. That's a good point. But then again, it's the upper layers that have a better chance of predicting this kind of thing. That is, if I know that my backup storage is extremely slow, then I will allow the fast primary storage do all retries it wants to do. It's not CAM nor scsi_da nor a specific SIM that can make those decisions. It's an issuer of the I/O request [or an intermediate geom that encapsulates that knowledge and effectively acts as an issuer of I/O-s to the lower geoms]. > 6. It's putting the wrong kind of specific hints into the mix. This needs further explanation. > > Absent numbers that show it's a big win, I'm > > very hesitant to say OK. > > > > Forth, there's a large number of places in the stack today that need to > > communicate their I/O is more urgent, and we don't have any good way to > > communicate even that simple concept down the stack. > > That's unfortunately, but my proposal has quite little to do with I/O > scheduling, priorities, etc. > > > Except it does. It dictates error recovery policy which is I/O scheduling. > > > Finally, the only places that ZFS uses the TRYHARDER flag are for things like > > the super block if I'm reading the code right. It doesn't do it for normal I/O. > > Right.  But for normal I/O there is ZIO_FLAG_IO_RETRY which is honored in the > same way as ZIO_FLAG_TRYHARD. > > > There's no code to cope with what would happen if all the copies of a block > > couldn't be read with the NORETRY flag. One of them might contain the data. > > ZFS is not that fragile :) see ZIO_FLAG_IO_RETRY above. > > > Except TRYHARD in ZFS means 'don't fail ****OTHER**** I/O in the queue when an > I/O fails' It doesn't control retries at all in Solaris. It's a different > concept entirely, and one badly thought out. I think that it does control retries. And it does even more. My understanding is that bio-s with B_FAILFAST can be failed immediately in the situation roughly equivalent to a CAM devq (or simq) being frozen. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Sat Nov 25 17:50:08 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE8B2DEBFAC; Sat, 25 Nov 2017 17:50:08 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f43.google.com (mail-lf0-f43.google.com [209.85.215.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6AC8C67DE5; Sat, 25 Nov 2017 17:50:08 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f43.google.com with SMTP id x68so28511429lff.0; Sat, 25 Nov 2017 09:50:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=5ApCUC5DjZ1viDCnCrzu033PbzMQh+JO2M+WEA6YFh8=; b=CquXPToyf3XEhqCcFIHTuvpNPrIFqKCBFokh83IxQ+oYN3aSuSdvXrSOHqZFFQ84vz vhu8Qjz8JZoSka/xW6NTMm5ECJKoABu1aaHXhgjJqgcp3rByR/DgtG30q8O9RX7msDqq g4IxSrPcWYCMoQMA0gs8OANCF/PfAxxCjaYiAdRk+Plp0zYLU7NLFwgTEnzRddWj9gtO t0iSe/1/no3htvTsYhNsjvOqt2a2gXCfAX8aKDwO/Nq7RU4KOjWEv2GXSbtlfAt3Rc0P 5nxbcVEZzaf3uI04FEwySbDCnxaOce4fW/LRQ++dZCVhVjtLXMckjLh2bYT4lMQwlqGk 8Ggw== X-Gm-Message-State: AJaThX7sPSpEO7PZLcrL+gIQxz0jcCZMwX4PaulyjEzVMvq9IhRng6Bd 2RqYuI2wnS8U4nnsCkpfjxWlvMbVagw= X-Google-Smtp-Source: AGs4zMZZPtEaOC5dWapQhfIZTZBuOiIG6iLS8nMAw2SiAkHolrWh3Xnzl0XQnyoZVX4HQjvhBQqODQ== X-Received: by 10.46.77.148 with SMTP id c20mr13420749ljd.156.1511632205845; Sat, 25 Nov 2017 09:50:05 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id a9sm4184731lfg.12.2017.11.25.09.50.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 25 Nov 2017 09:50:05 -0800 (PST) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Warner Losh Cc: Scott Long , FreeBSD FS , freebsd-geom@freebsd.org References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> <27c9395f-5b3c-a062-3aee-de591770af0b@FreeBSD.org> From: Andriy Gapon Message-ID: Date: Sat, 25 Nov 2017 19:50:03 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 17:50:09 -0000 On 25/11/2017 19:38, Warner Losh wrote: > > > On Sat, Nov 25, 2017 at 9:58 AM, Andriy Gapon > wrote: > > On 25/11/2017 18:25, Warner Losh wrote: > > > > > > On Fri, Nov 24, 2017 at 10:17 AM, Andriy Gapon > > >> wrote: > > > >     On 24/11/2017 16:57, Scott Long wrote: > >     > > >     > > >     >> On Nov 24, 2017, at 6:34 AM, Andriy Gapon wrote: > >     >> > >     >> On 24/11/2017 15:08, Warner Losh wrote: > >     >>> > >     >>> > >     >>> On Fri, Nov 24, 2017 at 3:30 AM, Andriy Gapon > >     > > >     >>> >>> wrote: > >     >>> > >     >>> > >     >>>    https://reviews.freebsd.org/D13224 > >      > > >     >> > >     >>> > >     >>>    Anyone interested is welcome to join the review. > >     >>> > >     >>> > >     >>> I think it's a really bad idea. It introduces a 'one-size-fits-all' > >     notion of > >     >>> QoS that seems misguided. It conflates a shorter timeout with don't > >     retry. And > >     >>> why is retrying bad? It seems more a notion of 'fail fast' or so other > >     concept. > >     >>> There's so many other ways you'd want to use it. And it uses the > same return > >     >>> code (EIO) to mean something new. It's generally meant 'The lower > layers > >     have > >     >>> retried this, and it failed, do not submit it again as it will not > >     succeed' with > >     >>> 'I gave it a half-assed attempt, and that failed, but resubmission > might > >     work'. > >     >>> This breaks a number of assumptions in the BUF/BIO layer as well as > >     parts of CAM > >     >>> even more than they are broken now. > >     >>> > >     >>> So let's step back a bit: what problem is it trying to solve? > >     >> > >     >> A simple example.  I have a mirror, I issue a read to one of its > >     members.  Let's > >     >> assume there is some trouble with that particular block on that > >     particular disk. > >     >> The disk may spend a lot of time trying to read it and would still > fail.  > >     With > >     >> the current defaults I would wait 5x that time to finally get the > error back. > >     >> Then I go to another mirror member and get my data from there. > >     > > >     > There are many RAID stacks that already solve this problem by having > a policy > >     > of always reading all disk members for every transaction, and throwing > >     away the > >     > sub-transactions that arrive late.  It’s not a policy that is always > >     desired, but it > >     > serves a useful purpose for low-latency needs. > > > >     That's another possible and useful strategy. > > > >     >> IMO, this is not optimal.  I'd rather pass BIO_NORETRY to the first > read, get > >     >> the error back sooner and try the other disk sooner.  Only if I > know that there > >     >> are no other copies to try, then I would use the normal read with > all the retrying. > >     >> > >     > > >     > I agree with Warner that what you are proposing is not correct.  It > weakens the > >     > contract between the disk layer and the upper layers, making it less > clear who is > >     > responsible for retries and less clear what “EIO” means.  That > contract is already > >     > weak due to poor design decisions in VFS-BIO and GEOM, and Warner and I > >     > are working on a plan to fix that. > > > >     Well...  I do realize now that there is some problem in this area, > both you and > >     Warner mentioned it.  But knowing that it exists is not the same as > knowing what > >     it is :-) > >     I understand that it could be rather complex and not easy to describe > in a short > >     email... > > > >     But then, this flag is optional, it's off by default and no one is > forced to > >     used it.  If it's used only by ZFS, then it would not be horrible. > > > > > > Except that it isn't the same flag as what Solaris has (its B_FAILFAST does > > something different: it isn't about limiting retries but about failing ALL the > > queued I/O for a unit, not just trying one retry), and the problems that it > > solves are quite rare. And if you return a different errno, then the EIO > > contract is still fulfilled.  > > Yes, it isn't the same. > I think that illumos flag does even more. > > > Since it isn't the same, and there's not other systems that do a similar thing, > that ups the burden of proof that this is a good idea. > > >     Unless it makes things very hard for the infrastructure. > >     But I am circling back to not knowing what problem(s) you and Warner are > >     planning to fix. > > > > > > The middle layers of the I/O system are a bit fragile in the face of I/O errors. > > We're fixing that. > > What are the middle layers? > > > The buffer cache and lower layers of the UFS code is where the problems chiefly lie. Those are the upper layers from the point of view of GEOM and things below it. If they don't set that flag on the bio, then it is not going to magically appear in their I/O path. > > Of course, you still haven't articulated why this approach would be better > > Better than what? > > > Well, anything? I think that I have described how it is better than what we have now, which I think is a part of 'anything'. > > > nor > > show any numbers as to how it makes things better. > > By now, I have.  See my reply to Scott's email. > > > I just checked my email, I've seen no such reply. I checked it before I > replied.  Maybe it's just delayed. Sorry, my mistake. I thought that I sent that email in the morning, but I didn't. I have just sent it. Apologies again. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Sat Nov 25 17:57:40 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 206DDDEC22C for ; Sat, 25 Nov 2017 17:57:40 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x236.google.com (mail-it0-x236.google.com [IPv6:2607:f8b0:4001:c0b::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D5D0F681C6 for ; Sat, 25 Nov 2017 17:57:39 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x236.google.com with SMTP id b5so16490570itc.3 for ; Sat, 25 Nov 2017 09:57:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=sDDg+58CGPKx0+tTc1ED8m6HEe5BrIsIIQB/kA3ChwU=; b=qPFfmzjRNBbP3DmUvRWgC5jMjFTpdNySHNEs8H/xdoUucHPhTJiEXi40KmJeeH77Ey sY1c6XJ+8SS4Z0oqTJQ9XQn1y9K6v09yaqrPtqKmfWVtforMcMiq7TYqaT9DLUuNZA2A WiGcKFBDw7GXRV9Z3D6RsIJ4XMyMU7TZNOuYra1Q8c9dnWl5m3DqvlaeK1OeayjCg8jB wIXzdVHAxvnLU05sVIu+j+ECf9iI8/rotfDVUVQOtJmW77SwHCZBIYDUAwBkj7yosMCR 56TfIrZhYVrxHbWJQFjg/YMXIHP/MXKrfEAcRxZ78HvVBH5c6Vqc1OvzZKJVarY8JVdX zAsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=sDDg+58CGPKx0+tTc1ED8m6HEe5BrIsIIQB/kA3ChwU=; b=SWumRy3BbW9Nn2/eWGFSRujfdSqZ5cdFwU5QDGq8EX53mcIHBQyLhl9O+peJKJBvAe tR+r6hC150KdOjG+7i0SeFdudg2S5Tujn4gD6hRh/EK7Pg7FwU6+hRzHiU8lXT1iFBoa lAlCcvF01/jxNJVM2qcrLxhRv0y/aYdp1K49XqzRDBAGS2DKY+FrcugqTzTYk/v1CdOe N1Fg8v0mtfb5SjzOA4aEW8XfnMI8/AuzlOEHJbn7X0cIlMzXQkamZo0E5drn+OGExK9X mZMqbV+gfxnAWIca3C5Ul+E+AGtwlVsVpgtyzNgGFp92JS+cFXtIYLLpQUZKA057++4Y kEMw== X-Gm-Message-State: AJaThX5TN2OPyB3SrEQDcQAqGlafDqf4XjWJeA3th4XIpIzWxzHtiE30 U00Z4HISjbKY9v7xp/X0pNnd+fu396UNgmsb48G+/Q== X-Google-Smtp-Source: AGs4zMYazlz2WrPFSx5PeIGxX4EhK1qMgTHVpcFmS/KlPnGtbEi0uWZuo3DIO5vwYgUNvIHh2OtB3xdssoaEXn1dta8= X-Received: by 10.36.164.13 with SMTP id z13mr22202600ite.115.1511632659185; Sat, 25 Nov 2017 09:57:39 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.108.204 with HTTP; Sat, 25 Nov 2017 09:57:38 -0800 (PST) X-Originating-IP: [2603:300b:6:5100:9579:bb73:7b7f:aadd] In-Reply-To: <33101e6c-0c74-34b7-ee92-f9c4a11685d5@FreeBSD.org> References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> <33101e6c-0c74-34b7-ee92-f9c4a11685d5@FreeBSD.org> From: Warner Losh Date: Sat, 25 Nov 2017 10:57:38 -0700 X-Google-Sender-Auth: 6c28FYqH0hmmAfPKJFmQ73oXmro Message-ID: Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Andriy Gapon Cc: Scott Long , FreeBSD FS , freebsd-geom@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 17:57:40 -0000 On Sat, Nov 25, 2017 at 10:36 AM, Andriy Gapon wrote: > > Timestamp of the first error is Jun 16 10:40:18. > Timestamp of the last error is Jun 16 10:40:27. > So, it took additional 9 seconds to finally produce EIO. > That disk is a part of a ZFS mirror. If the request was failed after the > first > attempt, then ZFS would be able to get the data from a good disk much > sooner. > > And don't take me wrong, I do NOT want CAM or GEOM to make that decision by > itself. I want ZFS to be able to tell the lower layers when they should > try as > hard as they normally do and when they should report an I/O error as soon > as it > happens without any retries. Let's walk through this. You see that it takes a long time to fail an I/O. Perfectly reasonable observation. There's two reasons for this. One is that the disks take a while to make an attempt to get the data. The second is that the system has a global policy that's biased towards 'recover the data' over 'fail fast'. These can be fixed by reducing the timeouts, or lowing the read-retry count for a given drive or globally as a policy decision made by the system administrator. It may be perfectly reasonable to ask the lower layers to 'fail fast' and have either a hard or a soft deadline on the I/O for a subset of I/O. A hard deadline would return ETIMEDOUT or something when it's passed and cancel the I/O. This gives better determinism in the system, but some systems can't cancel just 1 I/O (like SATA drives), so we have to flush the whole queue. If we get a lot of these, performance suffers. However, for some class of drives, you know that if it doesn't succeed in 1s after you submit it to the drive, it's unlikely to complete successfully and it's worth the performance hit on a drive that's already acting up. You could have a soft timeout, which says 'don't do any additional action after X time has elapsed and you get word about this I/O. This is similar to the hard timeout, but just stops retrying after the deadline has passed. This scenario is better on the other users of the drive, assuming that the read-recovery operations aren't starving them. It's also easier to implement, but has worse worst case performance characteristics. You aren't asking to limit retries. You're really asking to the I/O subsystem to limit, where it can, the amount of time on an I/O so you can try another one. You're means to doing this is to tell it not to retry. That's the wrong means. It shouldn't be listed in the API that it's a 'NO RETRY' request. It should be a QoS request flag: fail fast. Part of why I'm being so difficult is that you don't understand this and are proposing a horrible API. It should have a different name. The other reason is that I absolutely do not want to overload EIO. You must return a different error back up the stack. You've show no interest in this past, which is also a needless argument. We've given good reasons, and you've poopooed them with bad arguments. Also, this isn't the data I asked for. I know things can fail slowly. I was asking for how it would improve systems running like this. As in "I implemented it, and was able to fail over to this other drive faster" or something like that. Actual drive failure scenarios vary widely, and optimizing for this one failure is unwise. It may be the right optimization, but it may not. There's lots of tricky edges in this space. Warner From owner-freebsd-fs@freebsd.org Sat Nov 25 22:17:51 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7375DF2032 for ; Sat, 25 Nov 2017 22:17:51 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com [IPv6:2607:f8b0:4001:c0b::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 983C871CD7 for ; Sat, 25 Nov 2017 22:17:51 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x22c.google.com with SMTP id 72so5201272itl.5 for ; Sat, 25 Nov 2017 14:17:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=LJmZtGwOy8guPaxwjyXIT3tK6dOEY/LFRWx1tpnpJSQ=; b=RghMCEjfdPhoF8iirse/K/MLeeQwCyBBUBDY4GIH4TSvyo6PeJvihzYn5hso1rNlLS +tnqaKeLnRgMLGXoyhERUjT7jVHpi6mWQqbaE5BG1d/iz/Yi8lFIvSpFjp83KXBLDvzl mfGLNV228BEqzpy7IuGp+mk3G1IzPGrVULVFy5xZMNEBfNY7ukdLtdMcOIivnrJUHBQF MMZqu50cjK8yirLEsSf2SNIid288qKfqZ50lTqRa1G2qg8DaOkrlnfwGIFfhaPM7nWCk 9uvOhbf6/QzBkEBHFWCbW+pcDUuYoB53Zj157D7Up9UZff18kpkmcbCaYG620pYhofaI IJkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=LJmZtGwOy8guPaxwjyXIT3tK6dOEY/LFRWx1tpnpJSQ=; b=g1IIKCAxqoHUdQR3yGRo00MvBZyR01MpZ3pA0zYL9SOO7rKQ9Ng1+0T6EskOPB578j +/wbZfQFYDG2dnRa3mYLz/Be8tNkhGpiQtLC+h3/YBfx7kleW4iGc6qF431bltes/s6i kKWEr6n1+yN3mvgYjuICcITj7EbNY/mQ3kxtMgIKpvLldHVt3jEMWVyzmfE7VYfPxmV0 iJAuczPABHq2rIiLf1VTB3kI+Pra8Q9wvfYUI6+wPReuB5c2pReA8a88qEfwSAQQE6OB 8X3Kfef0e6IMzxzk6+vc4svupBbC5IFhpvewy38cRzSt8XUtJm8Mjd95flx2bZOrDj7G Vwtg== X-Gm-Message-State: AJaThX6ehPRyj3Mg8i2foY7qR3lqq69qUl3OTf8lYgrpioUgBhLXY5p5 f3kZxgU5R/BGSU985SLEwV8p5fmLb1k+vI8eJoRCpw== X-Google-Smtp-Source: AGs4zMbF8Ht5E9L9iEvEwyz0HA3vAeq7KZT0urdLQtXp+Ntp5RarLahEh59YUo6SC+4j4DZaU+ZV3bjeOtF98AIV5Mw= X-Received: by 10.36.164.13 with SMTP id z13mr22853940ite.115.1511648270873; Sat, 25 Nov 2017 14:17:50 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.108.204 with HTTP; Sat, 25 Nov 2017 14:17:49 -0800 (PST) X-Originating-IP: [50.253.99.174] In-Reply-To: <9f23f97d-3614-e4d2-62fe-99723c5e3879@FreeBSD.org> References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <9f23f97d-3614-e4d2-62fe-99723c5e3879@FreeBSD.org> From: Warner Losh Date: Sat, 25 Nov 2017 15:17:49 -0700 X-Google-Sender-Auth: OFWKq_KlfodVn8Jxgh-5DhOWhSE Message-ID: Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Andriy Gapon Cc: FreeBSD FS , freebsd-geom@freebsd.org, Scott Long Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 22:17:51 -0000 On Sat, Nov 25, 2017 at 10:40 AM, Andriy Gapon wrote: > > Before anything else, I would like to say that I got an impression that we > speak > from so different angles that we either don't understand each other's > words or, > even worse, misinterpret them. I understand what you are suggesting. Don't take my disagreement with your proposal as willful misinterpretation. You are proposing something that's a quick hack. Maybe a useful one, but it's still problematical because it has the upper layers telling the lower layers what to do (don't do your retry), rather than what service to provide (I prefer a fast error exit to over every effort to recover the data). And it also does it by overloading the meaning of EIO, which has real problems which you've not been open to listening, I assume due to your narrow use case apparently blinding you to the bigger picture issues with that route. However, there's a way forward which I think that will solve these objections. First, designate that I/O that fails due to short-circuiting the normal recovery process, return ETIMEDOUT. The I/O stack currently doesn't use this at all (it was introduced for the network side of things). This is a general catch-all for an I/O that we complete before the lower layers have given it the maximum amount of effort to recover the data, at the user request. Next, don't use a flag. Instead add a 32-bit field that is call bio_qos for quality of service hints and another 32-bit field for bio_qos_param. This allows us to pass down specific quality of service desires from the filesystem to the lower layers. The parameter will be unused in your proposal. BIO_QOS_FAIL_EARLY may be a good name for a value to set it to (at the moment, just use 1). We'll assign the other QOS values later for other things. It would allow us to implement the other sorts of QoS things I talked about as well. As for B_FAILFAST, it's quite unlike what you're proposing, except in one incidental detail. It's a complicated state machine that the sd driver in solaris implemented. It's an entire protocol. When the device gets errors, it goes into this failfast state machine. The state machine makes a determination that the errors are indicators the device is GONE, at least for the moment, and it will fail I/Os in various ways from there. Any new I/Os that are submitted will be failed (there's conditional behavior here: depending on a global setting it's either all I/O or just B_FAILFAST I/O). ZFS appears to set this bit for its discovery code only, when a device not being there would significantly delay things. Anyway, when the device returns (basically an I/O gets through or maybe some other event happens), the driver exists this mode and returns to normal operation. It appears to be designed not for the use case that you described, but rather for a drive that's failing all over the place so that any pending I/Os get out of the way quickly. Your use case is only superficially similar to that use case, so the Solaris / Illumos experiences are mildly interesting, but due to the differences not a strong argument for doing this. This facility in Illumos is interesting, but would require significantly more retooling of the lower I/O layers in FreeBSD to implement fully. Plus Illumos (or maybe just Solaris) a daemon that looks at failures to manage them at a higher level, which might make for a better user experience for FreeBSD, so that's something that needs to be weighed as well. We've known for some time that HDD retry algorithms take a long time. Same is true of some SSD or NVMe algorithms, but not all. The other objection I have to 'noretry' naming is that it bakes the current observed HDD behavior and recovery into the API. This is undesirable as other storage technologies have retry mechanisms that happen quite quickly (and sometimes in the drive itself). The cutoff between fast and slow recovery is device specific, as are the methods used. For example, there's new proposals out in NVMe (and maybe T10/T13 land) to have new types of READ commands that specify the quality of service you expect, including providing some sort of deadline hint to clip how much effort is expended in trying to recover the data. It would be nice to design a mechanism that allows us to start using these commands when drives are available with them, and possibly using timeouts to allow for a faster abort. Most of your HDD I/O will complete within maybe ~150ms, with a long tail out to maybe as long as ~400ms. It might be desirable to set a policy that says 'don't let any I/Os remain in the device longer than a second' and use this mechanism to enforce that. Or don't let any I/Os last more than 20x the most recent median I/O time. A single bit is insufficiently expressive to allow these sorts of things, which is another reason for my objection to your proposal. With the QOS fields being independent, the clone routines just copies them and makes no judgement value on them. So, those are my problems with your proposal, and also some hopefully useful ways to move forward. I've chatted with others for years about introducing QoS things into the I/O stack, so I know most of the above won't be too contentious (though ETIMEDOUT I haven't socialized, so that may be an area of concern for people). Warner