From owner-freebsd-fs@freebsd.org Wed Feb 26 17:09:53 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7297C25DBD6 for ; Wed, 26 Feb 2020 17:09:53 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48SMjq3Rlkz4Bqw for ; Wed, 26 Feb 2020 17:09:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id CB1FF43173; Wed, 26 Feb 2020 18:09:40 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6B3PXpfRAY1q; Wed, 26 Feb 2020 18:09:40 +0100 (CET) Received: from [192.168.101.70] (unknown [192.168.101.70]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 38D9543172 for ; Wed, 26 Feb 2020 18:09:40 +0100 (CET) To: FreeBSD Filesystems From: Willem Jan Withagen Subject: ZFS pools in "trouble" Message-ID: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> Date: Wed, 26 Feb 2020 18:09:40 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48SMjq3Rlkz4Bqw X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 2001:4cb8:90:ffff::3 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.38 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[digiware.nl]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(-3.08)[ip: (-9.52), ipnet: 2001:4cb8::/29(-4.67), asn: 28878(-1.26), country: NL(0.03)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:2001:4cb8::/29, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Feb 2020 17:09:53 -0000 Hi, I'm using my pools in perhaps a rather awkward way as underlying storage for my ceph cluster: 1 disk per pool, with log and cache on SSD For one reason or another one of the servers has crashed ad does not really want to read several of the pools: ---- pool: osd_2 state: UNAVAIL Assertion failed: (reason == ZPOOL_STATUS_OK), file /usr/src/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c, line 5098. Abort (core dumped) ---- The code there is like: ---- default: /* * The remaining errors can't actually be generated, yet. */ assert(reason == ZPOOL_STATUS_OK); ---- And this on already 3 disks. Running: FreeBSD 12.1-STABLE (GENERIC) #0 r355208M: Fri Nov 29 10:43:47 CET 2019 Now this is a test cluster, so no harm there in matters of data loss. And the ceph cluster probably can rebuild everything if I do not lose too many disk. But the problem also lies in the fact that not all disk are recognized by the kernel, and not all disk end up mounted. So I need to remove a pool first to get more disks online. Is there anything I can do the get them back online? Or is this a lost cause? --WjW From owner-freebsd-fs@freebsd.org Wed Feb 26 20:26:42 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 044BC243D52 for ; Wed, 26 Feb 2020 20:26:42 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48SS4t2FMHz3Htl for ; Wed, 26 Feb 2020 20:26:37 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 284BE4000F for ; Wed, 26 Feb 2020 21:26:33 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 0D09C40010; Wed, 26 Feb 2020 21:26:32 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [192.168.1.132] (h-201-140.A785.priv.bahnhof.se [98.128.201.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 888D14000F for ; Wed, 26 Feb 2020 21:26:32 +0100 (CET) From: Peter Eriksson Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS pools in "trouble" Date: Wed, 26 Feb 2020 21:26:32 +0100 References: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> To: FreeBSD Filesystems In-Reply-To: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> Message-Id: <2A6BA1B1-35C7-42E3-BF03-7BF95B7825D8@lysator.liu.se> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48SS4t2FMHz3Htl X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-4.06 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; MV_CASE(0.50)[]; IP_SCORE(-1.76)[ip: (-6.90), ipnet: 2001:6b0::/32(-1.04), asn: 1653(-0.83), country: EU(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Feb 2020 20:26:42 -0000 What type of hardware are you using? Server, controllers and disks? - Peter > On 26 Feb 2020, at 18:09, Willem Jan Withagen wrote: >=20 > Hi, >=20 > I'm using my pools in perhaps a rather awkward way as underlying = storage for my ceph cluster: > 1 disk per pool, with log and cache on SSD >=20 > For one reason or another one of the servers has crashed ad does not = really want to read several of the pools: > ---- > pool: osd_2 > state: UNAVAIL > Assertion failed: (reason =3D=3D ZPOOL_STATUS_OK), file = /usr/src/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c, line 5098. > Abort (core dumped) > ---- >=20 > The code there is like: > ---- > default: > /* > * The remaining errors can't actually be generated, = yet. > */ > assert(reason =3D=3D ZPOOL_STATUS_OK); >=20 > ---- > And this on already 3 disks. > Running: > FreeBSD 12.1-STABLE (GENERIC) #0 r355208M: Fri Nov 29 10:43:47 CET = 2019 >=20 > Now this is a test cluster, so no harm there in matters of data loss. > And the ceph cluster probably can rebuild everything if I do not lose = too many disk. >=20 > But the problem also lies in the fact that not all disk are recognized = by the kernel, and not all disk end up mounted. So I need to remove a = pool first to get more disks online. >=20 > Is there anything I can do the get them back online? > Or is this a lost cause? >=20 > --WjW > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Thu Feb 27 08:12:12 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 65511258AA7 for ; Thu, 27 Feb 2020 08:12:12 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f174.google.com (mail-lj1-f174.google.com [209.85.208.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48Slkv32Nqz450n for ; Thu, 27 Feb 2020 08:12:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f174.google.com with SMTP id r19so2309677ljg.3 for ; Thu, 27 Feb 2020 00:12:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=MikcrL0O4lb2IKGpKNKUuQX9GGaB9FVva3m4rHt/QfM=; b=RMby96Mc0xkr0kd3dh0kuMWN0a+6zunI+2II7pu+f0mc2o4WgETylXgjVWA6kvdAP+ j5BMjHNVW+Zg8XDGm2yeQxw3r1caWJYG+KO81h9QMboK7qcD0il1XXsSuNOkVbFiQMcO xDC+AqFUv8G7brMbVfez0gBuA97HI16bQubiFQuKaql/DBiU72Ja7pYGAdnAc6CFjovh aEWGD13zkILmKMez7zTqtZ20FZFeUbIYkw79OS5wcxWBPQ4dtFVbKHCjnW5Hu/LWhpCt vX2lZ/UqA/iWJOx+kia0TSwEZD3nimF10vDT7p/pUvpurUp1ggJrn2jAaMm5RdHyaqI5 xgxg== X-Gm-Message-State: ANhLgQ2a0FFKpgnUnId6Hi+4uEEuWOpd9rgskwTC7R+lt3WaQy1l8UTs /pUeYpP6fJo5XLZmk9N3SoiD6Yxe X-Google-Smtp-Source: ADFU+vvpcwSe8jKkojB546aFMysdG7zSAX6W/QFBC/tO+2zgVfxZJ9aaU8utimMbYbYZ0zCb/Q91BA== X-Received: by 2002:a05:651c:1a2:: with SMTP id c2mr2102842ljn.79.1582791122457; Thu, 27 Feb 2020 00:12:02 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id v5sm2664492ljk.67.2020.02.27.00.11.59 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 27 Feb 2020 00:12:01 -0800 (PST) Subject: Re: ZFS pools in "trouble" To: Willem Jan Withagen , FreeBSD Filesystems References: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <91e1cd09-b6b8-f107-537f-ae2755aba087@FreeBSD.org> Date: Thu, 27 Feb 2020 10:11:59 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <71e1f22a-1261-67d9-e41d-0f326bf81469@digiware.nl> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 48Slkv32Nqz450n X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.174 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.02 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[174.208.85.209.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-1.02)[ip: (-0.39), ipnet: 209.85.128.0/17(-2.99), asn: 15169(-1.67), country: US(-0.05)]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 08:12:12 -0000 On 26/02/2020 19:09, Willem Jan Withagen wrote: > Hi, > > I'm using my pools in perhaps a rather awkward way as underlying storage for my > ceph cluster: >     1 disk per pool, with log and cache on SSD > > For one reason or another one of the servers has crashed ad does not really want > to read several of the pools: > ---- >   pool: osd_2 >  state: UNAVAIL > Assertion failed: (reason == ZPOOL_STATUS_OK), file > /usr/src/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c, line 5098. > Abort (core dumped) > ---- > > The code there is like: > ---- >         default: >                 /* >                  * The remaining errors can't actually be generated, yet. >                  */ >                 assert(reason == ZPOOL_STATUS_OK); > > ---- > And this on already 3 disks. > Running: > FreeBSD 12.1-STABLE (GENERIC) #0 r355208M: Fri Nov 29 10:43:47 CET 2019 > > Now this is a test cluster, so no harm there in matters of data loss. > And the ceph cluster probably can rebuild everything if I do not lose too many > disk. > > But the problem also lies in the fact that not all disk are recognized by the > kernel, and not all disk end up mounted. So I need to remove a pool first to get > more disks online. > > Is there anything I can do the get them back online? > Or is this a lost cause? Depends on what 'reason' is. I mean the value of the variable. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Thu Feb 27 17:41:59 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7F323248A11 for ; Thu, 27 Feb 2020 17:41:59 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com [IPv6:2607:f8b0:4864:20::32e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48T0NQ2pNsz4R5F for ; Thu, 27 Feb 2020 17:41:58 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: by mail-ot1-x32e.google.com with SMTP id j20so3748366otq.3 for ; Thu, 27 Feb 2020 09:41:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=pg+5Nwmu9EKhQUYJvyfh/RBvbcvxFXOJgPlbtdwQg7M=; b=Toy/DC3rQNdHJJ56A5F4Tz6AwowShL3rQ6B2a3s1CTD9qsLoW7hbfs3RUTAHt8gXdI nKCuBADAY51L0YY9kWrPBXqulXrNdIxH0dpKcJIzNwDK6LiZz2vGLRd4lr6Y1Yk8I7WO isFBtMzaFvJAi9NB0ncmd6wMj2NDsNZfmXXOAC5f8Td/qAF+sE/hK7wazVfcJSVUWhk8 q23L5YWXgh4vS/pzSZAPZLLHRV7ky024pY87xshv5i70W+9kWE10H8HAnbspd5a+nZp/ zgi+V5c3WWNMv3qKdz5uEf6VU074nMlo+F9ZNDYJHxFu8tyVImbe3Qv0Sg9fWN3rr2PA jafA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=pg+5Nwmu9EKhQUYJvyfh/RBvbcvxFXOJgPlbtdwQg7M=; b=suZGlPnTWO6G/Y8Ri2RYzLYrFTLq+F4NA7LW/Ph0YHJATm1GoSIsHn/RTUWL24jTOc HEJHbzEZqurou/v17PnhqAodvkFs4tMGvwA6aUONk9evouHnvqibD0j+qL+ip7JPaROB /fykQ1VmWh1l6YRyX0RWa/mO8SJ0A8ZjCQmcpMcY4Yg0kjTSmhzZnEBib6FBnCkgGTDi vjJqt8YWfWXbtPvnurKljYU3Nz8NGrzh9EmvtSdk4HgbH2Uja0xAXX0jCVAI3C5ObFfJ zoWs/XHdGfEvQYuJtuCcL1B44uNFsrC3qX9XKkGb8WzFJQH6f4+6azTjRXTEUvnEl5AY y3PQ== X-Gm-Message-State: APjAAAWg8SpsvVxJ72t66kGv/InAxiC03H12LC95Lu2QZ9ITMgN1CmFN XhFonYlFTPEj3e07/cahv6Y5n6G1HN19Nby6gyYJXd3Y X-Google-Smtp-Source: APXvYqw/KBLThOc8qyZmbJz5pDRIIe2Zj2BXksxM87b6uH1w8RskNzlpRM1JdE31Gezx3VwXtLaMsw/okbmItmx2/2k= X-Received: by 2002:a05:6830:1643:: with SMTP id h3mr73408otr.70.1582825316676; Thu, 27 Feb 2020 09:41:56 -0800 (PST) MIME-Version: 1.0 From: Luoqi Chen Date: Thu, 27 Feb 2020 09:41:47 -0800 Message-ID: Subject: Linux could write to read only files on FreeBSD NFS server To: freebsd-fs@freebsd.org X-Rspamd-Queue-Id: 48T0NQ2pNsz4R5F X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Toy/DC3r; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luoqichen@gmail.com designates 2607:f8b0:4864:20::32e as permitted sender) smtp.mailfrom=luoqichen@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(0.00)[ip: (-8.03), ipnet: 2607:f8b0::/32(-1.88), asn: 15169(-1.67), country: US(-0.05)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[e.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 17:41:59 -0000 Hi, This was actually a pretty old problem, I noticed it a few years back and have been monitoring it when I upgrade the os on either the linux or the freebsd side -- it's still present between the latest centos and freebsd as of today. I meant to look into this issue myself, but had never found time for it, that's why I'm writing to this list, maybe it's a known problem or someone's willing to spend some time on it. And here you go, the problem, % cat ~/rotest.sh #!/bin/sh cp /dev/null x getfacl x chmod -w x getfacl x echo aaa >> x echo status $? cat x rm -f x % sh ~/rotest.sh # file: x # owner: luoqi # group: wheel user::rw- group::r-- other::r-- # file: x # owner: luoqi # group: wheel user::r-- group::r-- other::r-- status 0 aaa The script was run on a centos inside a directory nfs mounted from a freebsd. The append would fail for a centos/centos or a freebsd/freebsd combo. It's very easy to reproduce, it doesn't depend on any specific centos or freebsd version, nor on nfs version 3 or 4, nor on underlying file system ffs or zfs. -luoqi From owner-freebsd-fs@freebsd.org Thu Feb 27 18:40:24 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8C3B024A29B for ; Thu, 27 Feb 2020 18:40:24 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: from mail-il1-x133.google.com (mail-il1-x133.google.com [IPv6:2607:f8b0:4864:20::133]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48T1gp3z3Dz4Pc4 for ; Thu, 27 Feb 2020 18:40:22 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: by mail-il1-x133.google.com with SMTP id s18so508557iln.0 for ; Thu, 27 Feb 2020 10:40:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=JQadNLR+TXJhWRO0EgYjVHOgJDrhG8L13dWsumrJWpY=; b=kJC8G1eafEDHt8Hx2HRe6v3GN2ty98mRXISSR28uA36DupTSER3OJFyERoS7zSIdny b7L2EzuDYId0xSdG1tHVha8hPcwOD5ayWw1lziQq1v5iD4Eb0wansE8M4Iaix5TBfbkk mDkCDJj3qrQ1fUecHQYb/td1MhzOY3D6kyWK8rUse0VlGempkoUsZJ/zFNZ1ImCDeQBh mTyq99abpHVRogLkwejA1oom9b+rndlQ+vWSJ4Ro6ytt7KdcR4sHOo5uDu4bTl5KUIV1 IDZml08LORjVx2fWqfDg/yjXWtkPbsKXrLHa3CS3Jkfj2Z+/B3k9mxEGIKoOCFQ+9QcI bjeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JQadNLR+TXJhWRO0EgYjVHOgJDrhG8L13dWsumrJWpY=; b=SlKhqFUn4x1q2XqC3S7061emYwOcJTPdId+2tJ6dWt/ff4QCRjKNVE4Fmo+ESFmOVB sRiNM4RNb1OyYQxee387kJSv+QSxDtaYhokTWIMQZViIIfnVwAczEYWQy7Pf5t5PbD5/ oSVVtER8gSprRqKppy2S2kom2c3oe8QuI7JnSSa1NJXss9CSuuum3iI/MBQdtxG3mJi6 xYdwUc1v1OhzPUuWt0YMcSr35RuTQ9RNPRFjoedR62mgdQxhfVKaA+F0OjPnKGanC+7A 6qaLZZPa5oSwTckjyuJw7CEix5bbX7DERwcjIijm6KfFNRT2PU0qhDvdJsOMWwsQ7VtF yhPw== X-Gm-Message-State: APjAAAXaxIMGRtYGAvvNg5xy2AncMP4yPE4j5J3divabuTTAUU3JsLyI sq2FK1mgZqyF5USM0lLdlB/s2gRbJyAhsII4BzQ= X-Google-Smtp-Source: APXvYqwpBXkdUjnGtLMhzj7477g6SptOVtOAy6HAM4KTT6IXBkgTrYjIDKv9d2dF5PQHvebQFxFVR2kBfUnPnXqKUnc= X-Received: by 2002:a92:8547:: with SMTP id f68mr726685ilh.26.1582828819689; Thu, 27 Feb 2020 10:40:19 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: jdelisle Date: Thu, 27 Feb 2020 12:40:08 -0600 Message-ID: Subject: Re: Linux could write to read only files on FreeBSD NFS server To: Luoqi Chen Cc: freebsd-fs X-Rspamd-Queue-Id: 48T1gp3z3Dz4Pc4 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=kJC8G1ea; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdelisle@gmail.com designates 2607:f8b0:4864:20::133 as permitted sender) smtp.mailfrom=jdelisle@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-8.43), ipnet: 2607:f8b0::/32(-1.88), asn: 15169(-1.67), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[3.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 18:40:24 -0000 I feel like this is missing a lot of important information needed to answer your question. What's your NFS mount command? What user are you running that script as? What do the permissions look like on the NFS server side? What does your NFS export look like? etc.. On Thu, Feb 27, 2020 at 11:42 AM Luoqi Chen wrote: > Hi, > > This was actually a pretty old problem, I noticed it a few years back and > have been monitoring it when I upgrade the os on either the linux or the > freebsd side -- it's still present between the latest centos and freebsd as > of today. I meant to look into this issue myself, but had never found time > for it, that's why I'm writing to this list, maybe it's a known problem or > someone's willing to spend some time on it. > > And here you go, the problem, > > % cat ~/rotest.sh > #!/bin/sh > cp /dev/null x > getfacl x > chmod -w x > getfacl x > echo aaa >> x > echo status $? > cat x > rm -f x > % sh ~/rotest.sh > # file: x > # owner: luoqi > # group: wheel > user::rw- > group::r-- > other::r-- > > # file: x > # owner: luoqi > # group: wheel > user::r-- > group::r-- > other::r-- > > status 0 > aaa > > The script was run on a centos inside a directory nfs mounted from a > freebsd. The append would fail for a centos/centos or a freebsd/freebsd > combo. It's very easy to reproduce, it doesn't depend on any specific > centos or freebsd version, nor on nfs version 3 or 4, nor on underlying > file system ffs or zfs. > > -luoqi > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Thu Feb 27 19:19:51 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 69DF724B676 for ; Thu, 27 Feb 2020 19:19:51 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48T2YK4LTxz3Ccn for ; Thu, 27 Feb 2020 19:19:49 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: by mail-ot1-x32f.google.com with SMTP id x97so256538ota.6 for ; Thu, 27 Feb 2020 11:19:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ET/QvkHNySft7xt3vrsLaCFezEg2seeGzpfLiXbmpXY=; b=H7GEUApFrrGM7TnBxczkk+mAgPuUnWb9WDDE1xmdYTlUk1pd6pmFwRvl7GcPVkdB55 IM80VeHkjDquRCD/hIwXZGCnqKUXaq+Yw19D8QCZSpUCR0cIkTmuhtaMDQGhtUahJs0O azYOVf5Vqlok8aWN9H2PSwV9IyLcwEdwECMjLq/cuP2FfzgfjgAM/geiZxvdwQx+1fzh 2DJe5TLv4zfeChfclQt9Kh+nrP0Lmxsaw3d44aXqVAg5EtYQrI0KS8mv5F6m5HUNhtdd RV8gF20fffwKeJiEfYUbSdxImyRU7wxP2VMBLRk0fq9ncxtazFAZvM/Mm9qVWl/APJTm fdVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ET/QvkHNySft7xt3vrsLaCFezEg2seeGzpfLiXbmpXY=; b=SYjnbplkgilzyGgs8ym4Jm+peAXPumX/2XorRcGKKR6YPWjWbXcxsGvLSyvErZVFhQ /bRSqNV1J8RZwQycQ9jhl+zrBGfxajLJ9W9SSp7fFnFNNSMwBTHRtIIAigIXR4fNVB3u tlqO869qPOz8laPD2XDhVV1nI7XTH7sNEnjNpmfJoPCDTaWE9C1S0pzVDij8qmk61L9C SiFXJ2aGBWsf96PHwFRI2fm7+2jt/pqYBw3hZwSZdODT1bkPOUqMXStFM04QBGg7VdLs JnuQPTklo41p/E3R+7dUWYOFpUrI92xL9XEjAwuKLS76yapDZ/AD7zl6WdG/jWG/nF80 OOwg== X-Gm-Message-State: APjAAAUH27pwOk38tG2/rde7ovf9MGIbiy86fyaQusVVtSf7S9cuq747 ImwvNIB6bcSQZlU/mPS0QKaFq3EwORGLvXAdZp4= X-Google-Smtp-Source: APXvYqy//+FWpXsKlXw35dc70Z3ss7CDWR0N77vIG21agEBVemqgLRbxu8tqtwpGQKfO1aZ6oqlQGrtuNW/08XJhWq0= X-Received: by 2002:a9d:5e82:: with SMTP id f2mr351663otl.240.1582831188121; Thu, 27 Feb 2020 11:19:48 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Luoqi Chen Date: Thu, 27 Feb 2020 11:19:38 -0800 Message-ID: Subject: Re: Linux could write to read only files on FreeBSD NFS server To: jdelisle Cc: freebsd-fs X-Rspamd-Queue-Id: 48T2YK4LTxz3Ccn X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=H7GEUApF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luoqichen@gmail.com designates 2607:f8b0:4864:20::32f as permitted sender) smtp.mailfrom=luoqichen@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; URI_COUNT_ODD(1.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-7.72), ipnet: 2607:f8b0::/32(-1.88), asn: 15169(-1.67), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[f.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 19:19:51 -0000 I gather you haven't tried the script -- I didn't bother to include more details because they didn't seem to matter, this is not some corner case scenario. My guess is this is something fundamental, like, linux expects the server side to enforce the access control while freebsd assumes that the client has already done the check. On Thu, Feb 27, 2020 at 10:40 AM jdelisle wrote: > I feel like this is missing a lot of important information needed to > answer your question. > > What's your NFS mount command? What user are you running that script as? > What do the permissions look like on the NFS server side? What does your > NFS export look like? etc.. > > On Thu, Feb 27, 2020 at 11:42 AM Luoqi Chen wrote: > >> Hi, >> >> This was actually a pretty old problem, I noticed it a few years back and >> have been monitoring it when I upgrade the os on either the linux or the >> freebsd side -- it's still present between the latest centos and freebsd >> as >> of today. I meant to look into this issue myself, but had never found time >> for it, that's why I'm writing to this list, maybe it's a known problem or >> someone's willing to spend some time on it. >> >> And here you go, the problem, >> >> % cat ~/rotest.sh >> #!/bin/sh >> cp /dev/null x >> getfacl x >> chmod -w x >> getfacl x >> echo aaa >> x >> echo status $? >> cat x >> rm -f x >> % sh ~/rotest.sh >> # file: x >> # owner: luoqi >> # group: wheel >> user::rw- >> group::r-- >> other::r-- >> >> # file: x >> # owner: luoqi >> # group: wheel >> user::r-- >> group::r-- >> other::r-- >> >> status 0 >> aaa >> >> The script was run on a centos inside a directory nfs mounted from a >> freebsd. The append would fail for a centos/centos or a freebsd/freebsd >> combo. It's very easy to reproduce, it doesn't depend on any specific >> centos or freebsd version, nor on nfs version 3 or 4, nor on underlying >> file system ffs or zfs. >> >> -luoqi >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> > From owner-freebsd-fs@freebsd.org Thu Feb 27 20:05:38 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6069D24CD2D for ; Thu, 27 Feb 2020 20:05:38 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48T3Z93tbdz41LB for ; Thu, 27 Feb 2020 20:05:37 +0000 (UTC) (envelope-from jdelisle@gmail.com) Received: by mail-io1-xd31.google.com with SMTP id z190so882431iof.1 for ; Thu, 27 Feb 2020 12:05:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zNXXzwYqVNC4gjJwwfxMyyZka/P+L4gUW8JZii2H0Ks=; b=naRbe16s4PlUqgRphPVtM8YjVQyCUBs0BLOLCLWNsyEGP7uwgjCNR7zd214v/WYjcF H2sAjPzOwI2QWORvGzzPFbmp7ONMlpQNWb34+ZXyyoqZ4m1lXXuj1tAMnu2cnqJlfafI x8anMIZ7zTZBGoO22EjgsDuvf51vUb9mGlq9AupOxeWtdrQwJb7EzWQnrU6gR9ij7xkq S5j2lDSq02yalU8bq1MXG84dIxp4ckhSF1kRvbtXaDu2vYaTzDdUGkFO34vgKP0E5gHz neUhcjFhmz4p+rYOs8MMiFENXPbLZspF1haSCjt/7iamByl/RlbaP242A59n9xiHvePN Txyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zNXXzwYqVNC4gjJwwfxMyyZka/P+L4gUW8JZii2H0Ks=; b=btlfPcTMtE92WupsUv4zPepfB9uCOODPfknzIIRv7AC08FrJlum41whis1CFIJTt1X AHcL1doJhV+EppRjcJt+/Bsv9WbaRmSmeGQByF3d3K+ssevqhvOa5SRv0naXWtuM1y5b ybMn1O3uuFQji11mI7xVgxAD037iki/x1BE6Pgb6QnE4HOFi7dBTKy/tHuM+qaTJ13KT R1ke3kN9QULLUZRRWdV8E7b6SwfZ9p/x+dJczSjSN01HX01IpO4sygNZdSIwGkLcEfJ/ im2C5UVXl4t0EGhKWQL6Mxzn46LXnPqten7ssHfY7KRj9aK30DFbTNk6E7b7UmlN46sE i4oA== X-Gm-Message-State: APjAAAUP97tDSy59hqvXk9PW8eWrYrMVBXibqLOMy+g2ArwGkyhqhx/r 5ojyWTYSXWO5j+1YmzwRtAmawkyj8KmZlqOd33qDfp44 X-Google-Smtp-Source: APXvYqyUdWTCNnrqoifS1DmHEZbHOMsufCsLqrfCYCLWLbwRB2LJBkrfIzsOLFJrmA/unK/4BE2L6rVWiRuZ/pJgRRU= X-Received: by 2002:a05:6638:501:: with SMTP id i1mr437391jar.25.1582833936085; Thu, 27 Feb 2020 12:05:36 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: jdelisle Date: Thu, 27 Feb 2020 14:05:24 -0600 Message-ID: Subject: Re: Linux could write to read only files on FreeBSD NFS server To: Luoqi Chen Cc: freebsd-fs X-Rspamd-Queue-Id: 48T3Z93tbdz41LB X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=naRbe16s; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdelisle@gmail.com designates 2607:f8b0:4864:20::d31 as permitted sender) smtp.mailfrom=jdelisle@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; URI_COUNT_ODD(1.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-6.53), ipnet: 2607:f8b0::/32(-1.88), asn: 15169(-1.67), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 20:05:38 -0000 Without the requested info running your script is a waste of time. This already strikes me as a case of user-error/ RTFM, but I'm giving you the benefit of the doubt and asking for info that's absolutely necessary to even begin looking at this. On Thu, Feb 27, 2020 at 1:19 PM Luoqi Chen wrote: > I gather you haven't tried the script -- I didn't bother to include more > details because they didn't seem to matter, this is not some corner case > scenario. My guess is this is something fundamental, like, linux expects > the server side to enforce the access control while freebsd assumes that > the client has already done the check. > > On Thu, Feb 27, 2020 at 10:40 AM jdelisle wrote: > >> I feel like this is missing a lot of important information needed to >> answer your question. >> >> What's your NFS mount command? What user are you running that script >> as? What do the permissions look like on the NFS server side? What does >> your NFS export look like? etc.. >> >> On Thu, Feb 27, 2020 at 11:42 AM Luoqi Chen wrote: >> >>> Hi, >>> >>> This was actually a pretty old problem, I noticed it a few years back and >>> have been monitoring it when I upgrade the os on either the linux or the >>> freebsd side -- it's still present between the latest centos and freebsd >>> as >>> of today. I meant to look into this issue myself, but had never found >>> time >>> for it, that's why I'm writing to this list, maybe it's a known problem >>> or >>> someone's willing to spend some time on it. >>> >>> And here you go, the problem, >>> >>> % cat ~/rotest.sh >>> #!/bin/sh >>> cp /dev/null x >>> getfacl x >>> chmod -w x >>> getfacl x >>> echo aaa >> x >>> echo status $? >>> cat x >>> rm -f x >>> % sh ~/rotest.sh >>> # file: x >>> # owner: luoqi >>> # group: wheel >>> user::rw- >>> group::r-- >>> other::r-- >>> >>> # file: x >>> # owner: luoqi >>> # group: wheel >>> user::r-- >>> group::r-- >>> other::r-- >>> >>> status 0 >>> aaa >>> >>> The script was run on a centos inside a directory nfs mounted from a >>> freebsd. The append would fail for a centos/centos or a freebsd/freebsd >>> combo. It's very easy to reproduce, it doesn't depend on any specific >>> centos or freebsd version, nor on nfs version 3 or 4, nor on underlying >>> file system ffs or zfs. >>> >>> -luoqi >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> >> From owner-freebsd-fs@freebsd.org Thu Feb 27 21:03:29 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3031C24EB09 for ; Thu, 27 Feb 2020 21:03:29 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48T4rv589lz3FMF for ; Thu, 27 Feb 2020 21:03:26 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id E4C9940010 for ; Thu, 27 Feb 2020 22:03:21 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id D1C7640012; Thu, 27 Feb 2020 22:03:21 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [192.168.1.132] (h-201-140.A785.priv.bahnhof.se [98.128.201.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id D7B9940010 for ; Thu, 27 Feb 2020 22:03:20 +0100 (CET) From: Peter Eriksson Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: Linux could write to read only files on FreeBSD NFS server Date: Thu, 27 Feb 2020 22:03:20 +0100 References: To: FreeBSD Filesystems In-Reply-To: Message-Id: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48T4rv589lz3FMF X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-4.06 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; MV_CASE(0.50)[]; IP_SCORE(-1.76)[ip: (-6.90), ipnet: 2001:6b0::/32(-1.04), asn: 1653(-0.83), country: EU(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 21:03:29 -0000 I can verify that this indeed seems to be the case - the file owner can = always write to files, no matter the permissions set. Tested both locally (on ZFS) and over NFS (from the same directory). - Peter > $ touch x > $ getfacl x > # file: x > # owner: peter86 > # group: employee-liu.se > user::rw- > group::r-- > other::r-- >=20 > $ acltool list-access x > # file: x > # owner: peter86 > # group: employee-liu.se > owner@:rw-p--aARWcCos:-------:allow > group@:r-----a-R-c--s:-------:allow > everyone@:r-----a-R-c--s:=E2=80=94=E2=80=94=E2=80=94:allow= >=20 > $ chmod -w x > $ getfacl x > # file: x > # owner: peter86 > # group: employee-liu.se > user::r-- > group::r-- > other::r-- >=20 > $ acltool list-access x > # file: x > # owner: peter86 > # group: employee-liu.se > owner@:r-----aARWcCos:-------:allow > group@:r-----a-R-c--s:-------:allow > everyone@:r-----a-R-c--s:-------:allow >=20 > $ echo aaa >>x >=20 > $ cat x > aaa Not even a =E2=80=9Cdeny=E2=80=9D ACL stops writing :-). I does stop me = from reading the file contents afterwards though :-) > $ acltool edit owner@:ALL::deny x > $ acltool lac x > # file: x > # owner: peter86 > # group: employee-liu.se > owner@:rwxpd-aARWcCos:-------:deny > owner@:r-----a-R-cC-s:-------:allow > group@:r-----a-R-c--s:-------:allow > everyone@:r-----a-R-c--s:-------:allow > $ echo foo >>x > $ cat x > cat: x: Permission denied > $ acltool edit -- -owner@:ALL::deny x > $ acltool lac x > # file: x > # owner: peter86 > # group: employee-liu.se > owner@:r-----a-R-cC-s:-------:allow > group@:r-----a-R-c--s:-------:allow > everyone@:r-----a-R-c--s:-------:allow > $ cat x > aaa > foo Btw =E2=80=9Cacltool" is a small tool I=E2=80=99ve been creating because = I find the current crop of NFSv4/ZFS-ACL handling tools a bit=E2=80=A6 = =E2=80=9Clacking=E2=80=9D.=20 The current =E2=80=9CWork-in-progress=E2=80=9D can be downloaded from https://github.com/ptrrkssn/acltool Compiles and works on FreeBSD, Solaris & Linux (only over NFSv4 for = Linux) sadly. No POSIX-ACL-support (atleast not yet). - Peter > On 27 Feb 2020, at 18:41, Luoqi Chen wrote: >=20 > Hi, >=20 > This was actually a pretty old problem, I noticed it a few years back = and > have been monitoring it when I upgrade the os on either the linux or = the > freebsd side -- it's still present between the latest centos and = freebsd as > of today. I meant to look into this issue myself, but had never found = time > for it, that's why I'm writing to this list, maybe it's a known = problem or > someone's willing to spend some time on it. >=20 > And here you go, the problem, >=20 > % cat ~/rotest.sh > #!/bin/sh > cp /dev/null x > getfacl x > chmod -w x > getfacl x > echo aaa >> x > echo status $? > cat x > rm -f x > % sh ~/rotest.sh > # file: x > # owner: luoqi > # group: wheel > user::rw- > group::r-- > other::r-- >=20 > # file: x > # owner: luoqi > # group: wheel > user::r-- > group::r-- > other::r-- >=20 > status 0 > aaa >=20 > The script was run on a centos inside a directory nfs mounted from a > freebsd. The append would fail for a centos/centos or a = freebsd/freebsd > combo. It's very easy to reproduce, it doesn't depend on any specific > centos or freebsd version, nor on nfs version 3 or 4, nor on = underlying > file system ffs or zfs. >=20 > -luoqi > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Thu Feb 27 22:11:55 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A54C0250FC1 for ; Thu, 27 Feb 2020 22:11:55 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48T6Mr4nBzz499X for ; Thu, 27 Feb 2020 22:11:52 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 4D66C40014 for ; Thu, 27 Feb 2020 23:11:49 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 3A22040012; Thu, 27 Feb 2020 23:11:49 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [192.168.1.132] (h-201-140.A785.priv.bahnhof.se [98.128.201.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 8B89140010; Thu, 27 Feb 2020 23:11:48 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: Linux could write to read only files on FreeBSD NFS server From: Peter Eriksson In-Reply-To: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> Date: Thu, 27 Feb 2020 23:11:48 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> To: FreeBSD Filesystems X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48T6Mr4nBzz499X X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-4.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-1.80)[ip: (-7.10), ipnet: 2001:6b0::/32(-1.04), asn: 1653(-0.83), country: EU(-0.01)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 22:11:55 -0000 I see that this was a bit unclear, writing to the protected file was via = NFS from a Linux (CentOS 7) client. I verified the ACLs and the file = content both via NFSv4 and locally on the FreeBSD server. Writing from an OmniOS (OpenSolaris) client fails. As does a local = write. As it should... Also, it=E2=80=99s not related to ACLs (atleast not directly). Using = =E2=80=9Cchmod=E2=80=9D gives the same effect: > $ rm -f x > $ touch x > $ chmod 000 x > $ ls -l x > ---------- 1 peter86 employee-liu.se 0 27 feb 22.46 x > $ echo foo >x > $ cat x > cat: x: Permission denied > $ chmod 600 x > cat x > foo Rick: Looking at a tcpdump capture of the NFS traffic from the Linux client it = seems to be doing: 1. Client -> Server: V4 Procedure: COMPOUND (1) SEQUENCE (53) PUTFH (22) OPEN (18)=20 share_access: OPEN4_SHARE_ACCESS_WRITE open type: OPEN4_NOCREATE ACCESS (3) Check: RD MD XT XE GETATTR (9) 2. Server -> Client: V4 Procedure: COMPOUND (1) Status: NFS4_OK Operations (5): SEQUENCE (53) Status: NFS4_OK PUTFH Status: NFS4_OK OPEN Status: NFS4_OK ACCESS [Access Denied] Status: NFS4_OK GETATTR Status: NFS4_OK 3. Client -> Server: V4 Procedure: COMPOUND (1) Tag: Operations: SEQUENCE (53)=20 PUTFH (22) WRITE (38) Stable: FILE_SYNC4 GETATTR (9) 4. Server -> Client V4 Procedure: COMPOUND Tag: Operations: SEQUENCE (53) Status: NFS4_OK PUTFH Status: NFS4_OK WRITE Status: NFS4_OK Committed: FILE_SYNC4 GETATTR Status: NFS4_OK (According to Wireshark) Looks like Linux ignores the Access Denied in packet 2 and just forges = ahead, and FreeBSD happily accepts the WRITE in packet 3=E2=80=A6 - Peter > On 27 Feb 2020, at 22:03, Peter Eriksson wrote: >=20 > I can verify that this indeed seems to be the case - the file owner = can always write to files, no matter the permissions set. >=20 > Tested both locally (on ZFS) and over NFS (from the same directory). From owner-freebsd-fs@freebsd.org Thu Feb 27 22:37:59 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5402A251BD0 for ; Thu, 27 Feb 2020 22:37:59 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660079.outbound.protection.outlook.com [40.107.66.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48T6xv5hxWz47xm for ; Thu, 27 Feb 2020 22:37:55 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ih7cp4NsF4swsGTteu0odpRieZJiH499gdDE//v5L+yn4aB3/4K7yaoyoBHV5m0CjygO3RLtZMPjxHXnjdpvmoSchUIKrn/Ntgvh+wbbb+N0pnCYGL7tAdbEiTy1AE551x/k/XTZPXQAaCijGDoe9jLeDJw3VsDx95LnnlEomvY/4dt5Q+i/3E52jN3tX6s0vd3xs76G2TPBvua7t6sD5pCxg2VOwREKBlUSQ11+3NmQvZKnt+tKZqDly641TajyCsO/zO+o1D+7oco+irCHF0tD8izJSKngmfPK29NpgXb+WVi/rNzN2x+Z8pY6vECbEXSVmJqh8jVcAAp0V/XWQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lbgUN0QdCFB16PeK+T1uZzK080Ztvyp0S1glxcNibio=; b=jxw4cyj5pD387OKr1ba+ZQpW+uS7/tLzSsvCJdYB9PsyZlhUwLS22D/3qBO75Mv88+S8eAyUe5sfuAJKBp/YchIlf6IlPGOY4atYBglPqUsI8yAFmspuMqM7Vk/Wg3/24qmihHypANEHe59qUcw9lrC2IEQVA3nvV9lcJRa/KR84aSTOInjsnuVJGGX4M1rvBYyMO5dh1UvhRWhtYcvTfydziD6Eu4W55OZsw2f1Yg6yfIzTJ94ffGOgUo1iS1VdLEqYZCCfHXCc1DrA+dTGUXjaPKZZDE2YK/7SAjpZtrt67vwsflN+ZypnYKF7Rd22cJYH37t9YYps4LtN/pMveA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YTBPR01MB3374.CANPRD01.PROD.OUTLOOK.COM (10.255.46.82) by YTBPR01MB3536.CANPRD01.PROD.OUTLOOK.COM (10.255.47.77) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2772.14; Thu, 27 Feb 2020 22:37:53 +0000 Received: from YTBPR01MB3374.CANPRD01.PROD.OUTLOOK.COM ([fe80::a50d:6237:4074:f9c4]) by YTBPR01MB3374.CANPRD01.PROD.OUTLOOK.COM ([fe80::a50d:6237:4074:f9c4%6]) with mapi id 15.20.2772.012; Thu, 27 Feb 2020 22:37:43 +0000 From: Rick Macklem To: Peter Eriksson , FreeBSD Filesystems Subject: Re: Linux could write to read only files on FreeBSD NFS server Thread-Topic: Linux could write to read only files on FreeBSD NFS server Thread-Index: AQHV7ZU6To48aBpgW0mPQBk2wdIrMagvh4MAgAATIQCAAALPjQ== Date: Thu, 27 Feb 2020 22:37:42 +0000 Message-ID: References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se>, <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> In-Reply-To: <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f20e488a-f2ac-4c3a-5329-08d7bbd5a8c8 x-ms-traffictypediagnostic: YTBPR01MB3536: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 03264AEA72 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(376002)(396003)(366004)(346002)(136003)(199004)(189003)(55016002)(52536014)(91956017)(76116006)(86362001)(7696005)(53546011)(66476007)(9686003)(66556008)(64756008)(66946007)(81166006)(81156014)(8936002)(478600001)(71200400001)(66446008)(5660300002)(786003)(8676002)(6506007)(110136005)(26005)(186003)(2906002)(296002)(316002)(33656002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTBPR01MB3536; H:YTBPR01MB3374.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Twx4gBnwmOywpHfBz+/XpIl4j4C0/ZtduG3q4v7xQPUY8erpxY7xgndxZEFEum5RV0pdOpfdt7ro57Y0zF5fCNzoCHkyrNEJH1fUM2+NuIBEdyN665tHKCO8iLwzqwFpeQYtNJckSZr0BViefcy3F0VOy5UrUCyIVe1cQDhJUIoI2iR1gkZlbXNVX/U2X0aNzG7akCo3Y55Hwvl9ho5XAvr7c9DnuzK7whVrFOrH3IM2JXHIFL1Y1pbGZsdOfRKgMCtwl2evcJbm8TeJccAr92W9w60pCjZQcbH31QQkRGEC4BA0u0krIHvM+NwDWITto08UTFLkBqLJe1BYq7HxxpH4KwQILyyorrP0yQ4+U+LSbxJwww+nNgBwxCo8FHCILj4bxGHGWxqS+Oq2DyOVnqT5oWYgbAAYYIuOKfkk8FqyA27Se4JFk/aTH7i/zgQJ x-ms-exchange-antispam-messagedata: t5uUmGu6igQTjFXJzV3b0eIJT/Xeh4q5wBgCsnU4/16FzbPfArhMZzMYu0gkYUxE0qwDn6O2b7cB4I/4FBRtc0ANOCz+PnD9Vqrj6ySsb1YGN0t8SCFwGx8qlC0c2Lbe52y3ZW0TIO6USEQOEq2umg== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: f20e488a-f2ac-4c3a-5329-08d7bbd5a8c8 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Feb 2020 22:37:42.9873 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: DlLJ09wZAhWrKSh+YH5r8N/TycxXwSKbpdLti4+sGTfnRi+MRHf8xnRf9OCX5EnwT+2tMyQhO3iY/UcHhgRpdg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTBPR01MB3536 X-Rspamd-Queue-Id: 48T6xv5hxWz47xm X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.79 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.70 / 15.00]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[79.66.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.40)[ipnet: 40.64.0.0/10(-3.83), asn: 8075(-3.12), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 22:37:59 -0000 Peter Eriksson wrote:=0A= >I see that this was a bit unclear, writing to the protected file was via N= FS from a Linux (CentOS >7) client. I verified the ACLs and the file conten= t both via NFSv4 and locally on the FreeBSD >server.=0A= >=0A= >Writing from an OmniOS (OpenSolaris) client fails. As does a local write. = As it should...=0A= >=0A= >Also, it=92s not related to ACLs (atleast not directly). Using =93chmod=94= gives the same effect:=0A= >=0A= >> $ rm -f x=0A= >> $ touch x=0A= >> $ chmod 000 x=0A= >> $ ls -l x=0A= >> ---------- 1 peter86 employee-liu.se 0 27 feb 22.46 x=0A= >> $ echo foo >x=0A= >> $ cat x=0A= >> cat: x: Permission denied=0A= >> $ chmod 600 x=0A= >> cat x=0A= >> foo=0A= >=0A= >Rick:=0A= >Looking at a tcpdump capture of the NFS traffic from the Linux client it s= eems to be doing:=0A= >=0A= >1. Client -> Server:=0A= >=0A= >V4 Procedure: COMPOUND (1)=0A= > SEQUENCE (53)=0A= > PUTFH (22)=0A= > OPEN (18)=0A= > share_access: OPEN4_SHARE_ACCESS_WRITE=0A= > open type: OPEN4_NOCREATE=0A= > ACCESS (3)=0A= > Check: RD MD XT XE=0A= > GETATTR (9)=0A= >=0A= >=0A= >2. Server -> Client:=0A= >=0A= >V4 Procedure: COMPOUND (1)=0A= >Status: NFS4_OK=0A= >Operations (5):=0A= > SEQUENCE (53)=0A= > Status: NFS4_OK=0A= > PUTFH=0A= > Status: NFS4_OK=0A= > OPEN=0A= > Status: NFS4_OK=0A= > ACCESS [Access Denied]=0A= Yep, this should tell Linux to fail, so I'd call this a Linux=0A= client bug. (Without looking at the spec., I'm pretty=0A= sure Access is supposed to return NFS_OK and the=0A= kinds of access allowed and not fail with NFSERR_ACESS.=0A= =0A= > Status: NFS4_OK=0A= > GETATTR=0A= > Status: NFS4_OK=0A= >=0A= >=0A= >3. Client -> Server:=0A= >=0A= >V4 Procedure: COMPOUND (1)=0A= >Tag: =0A= >Operations:=0A= > SEQUENCE (53)=0A= > PUTFH (22)=0A= > WRITE (38)=0A= > Stable: FILE_SYNC4=0A= > GETATTR (9)=0A= >=0A= >=0A= >4. Server -> Client=0A= >=0A= >V4 Procedure: COMPOUND=0A= >Tag: =0A= >Operations:=0A= > SEQUENCE (53)=0A= > Status: NFS4_OK=0A= > PUTFH=0A= > Status: NFS4_OK=0A= > WRITE=0A= > Status: NFS4_OK=0A= > Committed: FILE_SYNC4=0A= > GETATTR=0A= > Status: NFS4_OK=0A= Yep. NFS servers normally/always allow the owner of a=0A= file to do read/write irrespective of permissions.=0A= Why?=0A= Well, a POSIX file system only checks permissions upon=0A= open(2). Many POSIX apps then change permissions but continue=0A= to do I/O as allowed by the open(2).=0A= =0A= NFS is not POSIX compliant and does not do a POSIX=0A= open(2). (NFSv3 has no open and NFSv4 has an open=0A= that is basically a Windoze open/lock, or whatever they=0A= call it.)=0A= =0A= If an NFS server does not allow the owner access for I/O,=0A= then all those POSIX apps. break and users do not like=0A= the "NFS is not POSIX complaint" answer for why.=0A= =0A= The Linux folks might argue that the NFSv4 Open should=0A= fail, however I'd argue that it is not a POSIX open and=0A= might not be performed at POSIX open time by the client.=0A= (With delegations enabled, the Open does not even need=0A= to be done.)=0A= =0A= The "owner has access" has been standard practice for NFS=0A= servers for decades.=0A= =0A= rick=0A= =0A= (According to Wireshark)=0A= =0A= =0A= Looks like Linux ignores the Access Denied in packet 2 and just forges ahea= d, and FreeBSD happily accepts the WRITE in packet 3=85=0A= =0A= - Peter=0A= =0A= =0A= > On 27 Feb 2020, at 22:03, Peter Eriksson wrote:=0A= >=0A= > I can verify that this indeed seems to be the case - the file owner can a= lways write to files, no matter the permissions set.=0A= >=0A= > Tested both locally (on ZFS) and over NFS (from the same directory).=0A= =0A= From owner-freebsd-fs@freebsd.org Thu Feb 27 22:59:08 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CDC182521FA for ; Thu, 27 Feb 2020 22:59:08 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48T7QM0lSxz45mR for ; Thu, 27 Feb 2020 22:59:06 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: by mail-ot1-x332.google.com with SMTP id b3so851274otp.4 for ; Thu, 27 Feb 2020 14:59:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=eaZs2SLxn1JqqfMHH3vq6RRx+abSQ6+cqjIjzK3s7rU=; b=RG2/iMPfgLjYgIO7hU1kvVSPe3iKwoti6mNdihj27WsWTmUtsmJb2YKQRUe8i/hr4v 5LCe5xfCVn5gZIGRm02T1NCaAVpIke4t8V5lOFka/LjhX2TykhN5UHBlsS3TdvSAzhSn bBvfTVTQw7w2nExrnxweARAEiDbxm+47fK/P24w1VdpajhzU6en8Hh2XARULDqJ8d1Lm P+M+cqGJS5o1vQpf/5itbnBUg7E3fIROACoBYGxHVS4df+XdRvTQ4Z3nK5DLQS1kc/6Q QuJSWeXNUy9lyxiz2ncMiHGvbR7wpcxUe165UK4y4Kn6GZeyo3XJA/377Xg4a+vwKHsE nUnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=eaZs2SLxn1JqqfMHH3vq6RRx+abSQ6+cqjIjzK3s7rU=; b=WOOwr/Sf7Bp+0uPYh9kH/4rGpcyxqu21zPa7p14id5kg9ai7K6AahLEL6Z71pr22/D FhvbPHfljgj9PTFT6/9lHxzlAXnz70kEO7hz/HgpURQuDEiloXqI5tfsxXJFGO0Od+q0 mrkuR5UrciPYQ/b48LwisA88S8FJTiOWVZWwJsJwNpDfud2CGp/E1z9dxPwM37SDjmqF 5J+MLmT6xsYQoQK2/OWANZkCYrRvQgEy1VuEcoJ2rVbxUoZrzO5HYnF+vTDD9fBOklvN v/4DZpSjmWnuo3GO0+naONEgYbaV8L3xB9xO0cUuZkxn/chXJYB3DX9zETDDe8Ra5/Zu zOxA== X-Gm-Message-State: APjAAAWiFgSPmjIPzKkdqwmnqn0DQIjJh3BRx1W1WtY3NBuj1/D9gNLK ujl1aXQ6htrVPflI9PAmvaBWnF3+cM8xJatMsCrtZA== X-Google-Smtp-Source: APXvYqwc1Rh2Fnf0atID5vHsLrN/HxLRgHUCYkTlG9Gw9l+GVowddlHIQ9DWn+kMEj4Vb1Y1lFx2ixN6FbMpziQNCC4= X-Received: by 2002:a9d:3df6:: with SMTP id l109mr978477otc.284.1582844345551; Thu, 27 Feb 2020 14:59:05 -0800 (PST) MIME-Version: 1.0 References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> In-Reply-To: From: Luoqi Chen Date: Thu, 27 Feb 2020 14:58:55 -0800 Message-ID: Subject: Re: Linux could write to read only files on FreeBSD NFS server To: Rick Macklem Cc: Peter Eriksson , FreeBSD Filesystems X-Rspamd-Queue-Id: 48T7QM0lSxz45mR X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=RG2/iMPf; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luoqichen@gmail.com designates 2607:f8b0:4864:20::332 as permitted sender) smtp.mailfrom=luoqichen@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[5]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-8.51), ipnet: 2607:f8b0::/32(-1.88), asn: 15169(-1.67), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 22:59:08 -0000 One more piece of information that might help: this behavior started somewhere between centos 5 and 6, kernel 2.6.18 and 2.6.32, i.e., the same script would fail on 2.6.18. Timing wise I believe it coincided with the introduction of nfsv4. Even if this is a linux bug, given its dominant position, we don't have much of a choice but to try to be compatible. Does anyone have say access to a netapp and see how it behaves? -luoqi On Thu, Feb 27, 2020 at 2:38 PM Rick Macklem wrote: > Peter Eriksson wrote: > >I see that this was a bit unclear, writing to the protected file was via > NFS from a Linux (CentOS >7) client. I verified the ACLs and the file > content both via NFSv4 and locally on the FreeBSD >server. > > > >Writing from an OmniOS (OpenSolaris) client fails. As does a local write= . > As it should... > > > >Also, it=E2=80=99s not related to ACLs (atleast not directly). Using =E2= =80=9Cchmod=E2=80=9D > gives the same effect: > > > >> $ rm -f x > >> $ touch x > >> $ chmod 000 x > >> $ ls -l x > >> ---------- 1 peter86 employee-liu.se 0 27 feb 22.46 x > >> $ echo foo >x > >> $ cat x > >> cat: x: Permission denied > >> $ chmod 600 x > >> cat x > >> foo > > > >Rick: > >Looking at a tcpdump capture of the NFS traffic from the Linux client it > seems to be doing: > > > >1. Client -> Server: > > > >V4 Procedure: COMPOUND (1) > > SEQUENCE (53) > > PUTFH (22) > > OPEN (18) > > share_access: OPEN4_SHARE_ACCESS_WRITE > > open type: OPEN4_NOCREATE > > ACCESS (3) > > Check: RD MD XT XE > > GETATTR (9) > > > > > >2. Server -> Client: > > > >V4 Procedure: COMPOUND (1) > >Status: NFS4_OK > >Operations (5): > > SEQUENCE (53) > > Status: NFS4_OK > > PUTFH > > Status: NFS4_OK > > OPEN > > Status: NFS4_OK > > ACCESS [Access Denied] > Yep, this should tell Linux to fail, so I'd call this a Linux > client bug. (Without looking at the spec., I'm pretty > sure Access is supposed to return NFS_OK and the > kinds of access allowed and not fail with NFSERR_ACESS. > > > Status: NFS4_OK > > GETATTR > > Status: NFS4_OK > > > > > >3. Client -> Server: > > > >V4 Procedure: COMPOUND (1) > >Tag: > >Operations: > > SEQUENCE (53) > > PUTFH (22) > > WRITE (38) > > Stable: FILE_SYNC4 > > GETATTR (9) > > > > > >4. Server -> Client > > > >V4 Procedure: COMPOUND > >Tag: > >Operations: > > SEQUENCE (53) > > Status: NFS4_OK > > PUTFH > > Status: NFS4_OK > > WRITE > > Status: NFS4_OK > > Committed: FILE_SYNC4 > > GETATTR > > Status: NFS4_OK > Yep. NFS servers normally/always allow the owner of a > file to do read/write irrespective of permissions. > Why? > Well, a POSIX file system only checks permissions upon > open(2). Many POSIX apps then change permissions but continue > to do I/O as allowed by the open(2). > > NFS is not POSIX compliant and does not do a POSIX > open(2). (NFSv3 has no open and NFSv4 has an open > that is basically a Windoze open/lock, or whatever they > call it.) > > If an NFS server does not allow the owner access for I/O, > then all those POSIX apps. break and users do not like > the "NFS is not POSIX complaint" answer for why. > > The Linux folks might argue that the NFSv4 Open should > fail, however I'd argue that it is not a POSIX open and > might not be performed at POSIX open time by the client. > (With delegations enabled, the Open does not even need > to be done.) > > The "owner has access" has been standard practice for NFS > servers for decades. > > rick > > (According to Wireshark) > > > Looks like Linux ignores the Access Denied in packet 2 and just forges > ahead, and FreeBSD happily accepts the WRITE in packet 3=E2=80=A6 > > - Peter > > > > On 27 Feb 2020, at 22:03, Peter Eriksson wrote: > > > > I can verify that this indeed seems to be the case - the file owner can > always write to files, no matter the permissions set. > > > > Tested both locally (on ZFS) and over NFS (from the same directory). > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Thu Feb 27 23:11:33 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3956C2528E9 for ; Thu, 27 Feb 2020 23:11:33 +0000 (UTC) (envelope-from alan@peak.org) Received: from filter05.peak.org (filter05.peak.org [69.59.194.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.redcondor.net", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48T7hf0kmbz4Hry for ; Thu, 27 Feb 2020 23:11:29 +0000 (UTC) (envelope-from alan@peak.org) Received: from zmail-mta02.peak.org ([207.55.16.112]) by filter05.peak.org ({27dbf508-291b-4a6b-93f5-d568f05dc56a}) via TCP (outbound) with ESMTPS id 20200227231124796_0000 for ; Thu, 27 Feb 2020 15:11:24 -0800 X-RC-FROM: X-RC-RCPT: Received: from localhost (localhost [127.0.0.1]) by zmail-mta02.peak.org (Postfix) with ESMTP id EB6E34C4ED for ; Thu, 27 Feb 2020 15:11:18 -0800 (PST) Received: from zmail-mta02.peak.org ([127.0.0.1]) by localhost (zmail-mta02.peak.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id cnKO_QK8sH5T for ; Thu, 27 Feb 2020 15:11:18 -0800 (PST) Received: from mailproxy-lb-05.peak.org (mailproxy-lb-05.peak.org [207.55.17.95]) by zmail-mta02.peak.org (Postfix) with ESMTP id D97F14C4E9 for ; Thu, 27 Feb 2020 15:11:18 -0800 (PST) Subject: Re: Linux could write to read only files on FreeBSD NFS server To: freebsd-fs@freebsd.org References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> From: Alan Batie Message-ID: Date: Thu, 27 Feb 2020 15:10:54 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms060005040609020000000502" X-MAG-OUTBOUND: peakinternet.redcondor.net@207.55.16/22 X-Rspamd-Queue-Id: 48T7hf0kmbz4Hry X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=peak.org; spf=pass (mx1.freebsd.org: domain of alan@peak.org designates 69.59.194.81 as permitted sender) smtp.mailfrom=alan@peak.org X-Spamd-Result: default: False [-5.01 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; RCVD_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_ATTACHMENT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_TLS_LAST(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[peak.org,none]; IP_SCORE(-0.01)[country: US(-0.05)]; RCVD_IN_DNSWL_LOW(-0.10)[81.194.59.69.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:395796, ipnet:69.59.194.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 23:11:33 -0000 This is a cryptographically signed message in MIME format. --------------ms060005040609020000000502 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2/27/20 2:58 PM, Luoqi Chen wrote: > One more piece of information that might help: this behavior started > somewhere between centos 5 and 6, kernel 2.6.18 and 2.6.32, i.e., the s= ame > script would fail on 2.6.18. Timing wise I believe it coincided with th= e > introduction of nfsv4. >=20 > Even if this is a linux bug, given its dominant position, we don't have= > much of a choice but to try to be compatible. Does anyone have say acce= ss > to a netapp and see how it behaves? Is this what you mean? [101] $ df . Filesystem 1K-blocks Used Available Use% Mounted on filer01-cvo.peak.org:/vol/admin 167772160 73704064 94068096 44% /filer01/cvo-admin= [102] $ rm -f x [103] $ touch x [104] $ chmod 000 x [105] $ ls -l x ----------. 1 alan wheel 0 Feb 27 15:01 x [106] $ echo foo > x -bash: x: Permission denied [107] $ chmod 600 x [108] $ cat x [109] $ cat /etc/redhat-release CentOS release 6.10 (Final) This works the same way on a truenas server: [122] $ rm x [123] $ df . Filesystem 1K-blocks Used Available Use% Mounted on tnas01-cvo.fs10g.peak.org:/mnt/zdata/nfs/admin 78257431296 54539008 78202892288 1% /tnas01-cvo/ad= min [124] $ touch x [125] $ chmod 000 x [126] $ ls -l x ----------. 1 alan wheel 0 Feb 27 15:05 x [127] $ echo foo > x -bash: x: Permission denied [128] $ chmod 600 x [129] $ cat x [130] $ However it also does the same on a native FreeBSD 11 server: [116] $ uname -a FreeBSD zbackups02.peak.org 11.3-RELEASE-p3 FreeBSD 11.3-RELEASE-p3 #0: Mon Aug 19 21:08:43 UTC 2019 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 [105] $ cat /etc/redhat-release CentOS release 6.10 (Final) [106] $ df . Filesystem 1K-blocks Used Available Use% Mounted on zbackups02.peak.org:/zbackups/zmail03-admin 5039303296 91682304 4947620992 2% /zbackups [107] $ touch x [108] $ chmod 0 x [109] $ ls -l x ----------. 1 alan root 0 Feb 27 15:08 x [110] $ echo foo > x -bash: x: Permission denied [111] $ chmod 600 x [112] $ cat x --------------ms060005040609020000000502 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC C2swggTgMIIDyKADAgECAhBUdCKrz7BUVHpHGYaNdgQcMA0GCSqGSIb3DQEBCwUAMIGNMQsw CQYDVQQGEwJJVDEQMA4GA1UECAwHQmVyZ2FtbzEZMBcGA1UEBwwQUG9udGUgU2FuIFBpZXRy bzEjMCEGA1UECgwaQWN0YWxpcyBTLnAuQS4vMDMzNTg1MjA5NjcxLDAqBgNVBAMMI0FjdGFs aXMgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIENBIEcyMB4XDTE5MTIxMTE5MDcxNloXDTIwMTIx MTE5MDcxNlowGDEWMBQGA1UEAwwNYWxhbkBwZWFrLm9yZzCCASIwDQYJKoZIhvcNAQEBBQAD ggEPADCCAQoCggEBAJ1Gw0aKu1wno1Vb1MKlI+soIv5Ph03B7gUcg350uVwjm527faMnnV1D TZ415jn4Q5MHjIS5xjFUVJwM0DGOm+aNr0tFPOEL8Y8t2w5KCs7D4ikYderuv57WTwMgjDDY mOI9cUqs+npoFBcFTzx+RunErd8d22EEq61H7Ypyi+ltb4rZweE7KnaS5kgRovJXg8ii90ze dytd96JlTx8+oripPBaG+6RTlZxrQusbvSZpwjEv8xYa3Eh45Z2tBc1xcHNzvaDhprP01OA3 Yx4lIpSxIcAD23vtgjGhU9zycLqbutVpfaLrq3EwzGA7d6Xx97jsrXpCSHYd0TX2OkQKyPcC AwEAAaOCAa4wggGqMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUa/KNnmjBJQQfUTRX9hZc lOpNaRowfgYIKwYBBQUHAQEEcjBwMDsGCCsGAQUFBzAChi9odHRwOi8vY2FjZXJ0LmFjdGFs aXMuaXQvY2VydHMvYWN0YWxpcy1hdXRjbGlnMjAxBggrBgEFBQcwAYYlaHR0cDovL29jc3Aw OS5hY3RhbGlzLml0L1ZBL0FVVEhDTC1HMjAYBgNVHREEETAPgQ1hbGFuQHBlYWsub3JnMEcG A1UdIARAMD4wPAYGK4EfARgBMDIwMAYIKwYBBQUHAgEWJGh0dHBzOi8vd3d3LmFjdGFsaXMu aXQvYXJlYS1kb3dubG9hZDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwSAYDVR0f BEEwPzA9oDugOYY3aHR0cDovL2NybDA5LmFjdGFsaXMuaXQvUmVwb3NpdG9yeS9BVVRIQ0wt RzIvZ2V0TGFzdENSTDAdBgNVHQ4EFgQU4WAWRmEM5pxYuaRqT1VOZP+87GwwDgYDVR0PAQH/ BAQDAgWgMA0GCSqGSIb3DQEBCwUAA4IBAQAFrqbTZz6FiybHjpkR8a/eVtKR6ZTBnFE4f73m 0WC6mqryp4UKkTGl3Id/M1s54aH5Qof+x08Jlb7BgOh1VkjJNIPLWDY6qdZj/idf7DcwGMWN vXevh2QsMjJahnWOuedx+VPJybHfSLnc0iO6xXBrgbbNX6BdTidWbcaT/skBfBygtCy9KVm/ +5CR5NqVDxpmgrWlJsUkYMXaO0jvbCcEvY9LQ9nyMPPK/ttQU9XNNsarBC8cKcX95iL7rgTw AQ1r39pDjyZLC1+bMZqjUleJbpRiiBv6iXb4rFVkXg1R5LFOCX5n1ZtKk8loEFptGtaF+LCw lTtITblJ2dlelQZ3MIIGgzCCBGugAwIBAgIQT94QS+2VW96LrWWHzEFe4zANBgkqhkiG9w0B AQsFADBrMQswCQYDVQQGEwJJVDEOMAwGA1UEBwwFTWlsYW4xIzAhBgNVBAoMGkFjdGFsaXMg Uy5wLkEuLzAzMzU4NTIwOTY3MScwJQYDVQQDDB5BY3RhbGlzIEF1dGhlbnRpY2F0aW9uIFJv b3QgQ0EwHhcNMTkwOTIwMDcxMjA1WhcNMzAwOTIyMTEyMjAyWjCBjTELMAkGA1UEBhMCSVQx EDAOBgNVBAgMB0JlcmdhbW8xGTAXBgNVBAcMEFBvbnRlIFNhbiBQaWV0cm8xIzAhBgNVBAoM GkFjdGFsaXMgUy5wLkEuLzAzMzU4NTIwOTY3MSwwKgYDVQQDDCNBY3RhbGlzIENsaWVudCBB dXRoZW50aWNhdGlvbiBDQSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALdo c3rZPNQv+9xnyj3OlHz/iRnO2hpj8xlHkCdYKNwnRabAT6J0RA11A3ZkQiEZEw66B99ES7Ez v9IRBYmIwsr720lUptObF5L3yVzl3nzaittXwWsq+CQoDEci1cKkWF5SiO22+Np2Epu2HFxk w5nXMnZibrqnC6hUGsFogTDUUVRIuLlublwWYFhpqvDaCh//ucRgRW3+rTU1nBoT1XHkXrLs Cteefjoh+o01tNTWvGi4+3OyABidGPXuoYh7UbYX1u0sG1O8rO92t5zV7/Cr/Vza9EbySh6D rCqsY333sNxikKzFyBwebZv43t1xJyMVE/CRt7BLJOyHxd1Yq0sCAwEAAaOCAf4wggH6MA8G A1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUUtiIOsifeGbtifN7OHCUyQICNtAwQQYIKwYB BQUHAQEENTAzMDEGCCsGAQUFBzABhiVodHRwOi8vb2NzcDA1LmFjdGFsaXMuaXQvVkEvQVVU SC1ST09UMEUGA1UdIAQ+MDwwOgYEVR0gADAyMDAGCCsGAQUFBwIBFiRodHRwczovL3d3dy5h Y3RhbGlzLml0L2FyZWEtZG93bmxvYWQwJwYDVR0lBCAwHgYIKwYBBQUHAwIGCCsGAQUFBwME BggrBgEFBQcDCTCB4wYDVR0fBIHbMIHYMIGWoIGToIGQhoGNbGRhcDovL2xkYXAwNS5hY3Rh bGlzLml0L2NuJTNkQWN0YWxpcyUyMEF1dGhlbnRpY2F0aW9uJTIwUm9vdCUyMENBLG8lM2RB Y3RhbGlzJTIwUy5wLkEuJTJmMDMzNTg1MjA5NjcsYyUzZElUP2NlcnRpZmljYXRlUmV2b2Nh dGlvbkxpc3Q7YmluYXJ5MD2gO6A5hjdodHRwOi8vY3JsMDUuYWN0YWxpcy5pdC9SZXBvc2l0 b3J5L0FVVEgtUk9PVC9nZXRMYXN0Q1JMMB0GA1UdDgQWBBRr8o2eaMElBB9RNFf2FlyU6k1p GjAOBgNVHQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQELBQADggIBAGBEuhmiq3L7DkGaRMG6FTm9 na4v3ya3KW+xkhFvSZgPinqeBi5qfV+dCL/BCuO/JMH9mgI5z57DnYiLQC3CIHnEtalcTfhG PleRgjRMuFQLAeYM5UAZiiPT+D8S7faZ0CZ3glRLw51QTGQJZSC+bN7mgoiBG/HmGahvLWjl kjNZ6o6AmVC3HIV1mGowamiYNEVDmen+SAdJW9uhwP+xFFZodZ0lYJQ6FHg+3pSDVx6YdM94 n9e9tlMnXKB+CY92WmPXbUOMCUjYUmTsxEu9lJEusHv+eehThrO6HiVrkHvEathHnkhphpYm SlG2KOIwfwtqJjJ9C+EMCOcDDa1ndhUTVFMMTAZmyWLRGg0U0O9hzwPA520ZL0Q0iZI7E6Kl OmaQZQX+LORMK4V6hVW9qzPZhgjw2SYux8N8vAWA/3d4ky+j1uVIzk0qRXJ0iD+B1uTyOjEx 15fmm+mowp7ycOhNUxi4d8ycqb+QkPBbZtM+zCi7eWa9hOI6I2V3mZ9bFKUqonWcqfZhvy2D EZhzJLYQ0Zw5ztrR7+fmDjuHFBG07eQcMBOUT46qL7J3ncneUooyCvpNTAlxSzE3xEc96lDd 4v38Lnl3BsuIxH9p/xb2LBGNxgR12QjFVj33wX25fyE47PUPTRt+2wBJv5oNsjatNjS4w20C CoLfVtGgVPUrMYIEFzCCBBMCAQEwgaIwgY0xCzAJBgNVBAYTAklUMRAwDgYDVQQIDAdCZXJn YW1vMRkwFwYDVQQHDBBQb250ZSBTYW4gUGlldHJvMSMwIQYDVQQKDBpBY3RhbGlzIFMucC5B Li8wMzM1ODUyMDk2NzEsMCoGA1UEAwwjQWN0YWxpcyBDbGllbnQgQXV0aGVudGljYXRpb24g Q0EgRzICEFR0IqvPsFRUekcZho12BBwwDQYJYIZIAWUDBAIBBQCgggJFMBgGCSqGSIb3DQEJ AzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIwMDIyNzIzMTA1NVowLwYJKoZIhvcN AQkEMSIEIPWQqRk2cXGMMmXFumzdEETY/E3Q8Q0BwiozODlfcHOGMGwGCSqGSIb3DQEJDzFf MF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgIC AIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgbMGCSsGAQQBgjcQ BDGBpTCBojCBjTELMAkGA1UEBhMCSVQxEDAOBgNVBAgMB0JlcmdhbW8xGTAXBgNVBAcMEFBv bnRlIFNhbiBQaWV0cm8xIzAhBgNVBAoMGkFjdGFsaXMgUy5wLkEuLzAzMzU4NTIwOTY3MSww KgYDVQQDDCNBY3RhbGlzIENsaWVudCBBdXRoZW50aWNhdGlvbiBDQSBHMgIQVHQiq8+wVFR6 RxmGjXYEHDCBtQYLKoZIhvcNAQkQAgsxgaWggaIwgY0xCzAJBgNVBAYTAklUMRAwDgYDVQQI DAdCZXJnYW1vMRkwFwYDVQQHDBBQb250ZSBTYW4gUGlldHJvMSMwIQYDVQQKDBpBY3RhbGlz IFMucC5BLi8wMzM1ODUyMDk2NzEsMCoGA1UEAwwjQWN0YWxpcyBDbGllbnQgQXV0aGVudGlj YXRpb24gQ0EgRzICEFR0IqvPsFRUekcZho12BBwwDQYJKoZIhvcNAQEBBQAEggEAY914fxzK +iDZj80Z/rRB+/qheo+o++keO+VC2GJ4/91CSI79xzW0HazWNA+wsFsxKzw4+YtKJP4UP39q s3Bdg4tGCuep9C5G+vJup0yafZC/Q2Zev790cBK76/oHHFU6I6OUBVf16FUpHbff0C77Vuly HCeUz1fNjplZMffUw5JjaJRjcW8cqV/hnZ0XcqIAzsM9961e4j7eVp5mxW3TWqPSoRAIx3DT JrQ768boGe5RZIhu02NgNsia/E8NAoXeAp6jfuHBDcqhLf8830IOKF6pNFS/BuCF9/G4qMw3 5izcOZpaqdjuldi5Q+NAKtUWXHhD0Cz0PUKnaBihhnBlSgAAAAAAAA== --------------ms060005040609020000000502-- From owner-freebsd-fs@freebsd.org Thu Feb 27 23:31:26 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 07DE1253149 for ; Thu, 27 Feb 2020 23:31:26 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: from mail-oi1-x243.google.com (mail-oi1-x243.google.com [IPv6:2607:f8b0:4864:20::243]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48T87c3KVKz3PS9 for ; Thu, 27 Feb 2020 23:31:23 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: by mail-oi1-x243.google.com with SMTP id b18so1095965oie.2 for ; Thu, 27 Feb 2020 15:31:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QyvokkXoz/9RMuZPzHKaCSvHRiMCBau0/ZiFJWYqKtE=; b=A/FzFOD+ZTgyxj/jN514JP4b1XWP8XQzsSjYqpUosSoHyJhTpkdAyAKtEOqPhzMgkc WxuR/Zu1RErSyPep3JCKrnl1Qb04WVhAMdw9FNbonXJ6KZ6tMuwph8szi4nQaRNt/1as +n6xzkrVtW49/tofHJ1kkvBxBGQTlQ1Fqaq3FHsySOR5iHSp8bFinWLPkZg1wVUc8qfB CJrCUUlJPWb4nmjHRNIiflo/kttw9eN9NNiU+m+JHIxf8vCqRRIjuzjChhg4C+lgUVVA zahSSeZE5SepdiPjjnhpVwSr8aI28JPd/GKE7JqjmbtLhgpXvvzOfEAj9WUbqNIFzRCa YtqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QyvokkXoz/9RMuZPzHKaCSvHRiMCBau0/ZiFJWYqKtE=; b=HL4rhFHCY2R7aNhs0lB64nXmgEoWqUw+Ep/s9HuYtft4AqjmybKu8sKDnHbsbu+idh RcLhs1sxpQWKjx7qoTs2FaxowcMzVnj1s4yIwFp13snKc2BzT4rom0e7Zwi1TFeoj3Mh /hT5o6cT7UQ3rAQyEWL0maZ8WeCvmEJs3vAH4pssXzRA/6z7s31I9DLNm/Vd/jwfKnkm MDJO0i4TFU5zJuh6MF0v7yLZoFZ5aFSKVveYttiUCUQnhjnd/Oc8c9wtQ8iW77x35F/0 fweTg/DqPumanv+vQRE7cloU6lcdr3b2aj64vk2Q6cZezqdKBt5BigkQ8DUxt1pzjX83 +/HQ== X-Gm-Message-State: APjAAAVe2TAidoIWZhRFbsUrOAkKh4YZfiGdhRJ/h9JvfYkBJCUTZxQK FeAF6fDh31LBFvgQd2ueJxPuveFHVdRxhnNadEQ= X-Google-Smtp-Source: APXvYqxZmrSTMXDhh61GZdfFLDzml1ZLxh6m5NSGDwxyFnKahOnWlwJ/aXpmcCuULCv04e1hYeQbSwbv2xpLy9mRxj8= X-Received: by 2002:a05:6808:104:: with SMTP id b4mr1051624oie.169.1582846282001; Thu, 27 Feb 2020 15:31:22 -0800 (PST) MIME-Version: 1.0 References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> In-Reply-To: From: Luoqi Chen Date: Thu, 27 Feb 2020 15:31:12 -0800 Message-ID: Subject: Re: Linux could write to read only files on FreeBSD NFS server To: Alan Batie Cc: freebsd-fs X-Rspamd-Queue-Id: 48T87c3KVKz3PS9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=A/FzFOD+; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luoqichen@gmail.com designates 2607:f8b0:4864:20::243 as permitted sender) smtp.mailfrom=luoqichen@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[3.4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; IP_SCORE(0.00)[ip: (2.36), ipnet: 2607:f8b0::/32(-1.88), asn: 15169(-1.67), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 23:31:26 -0000 On Thu, Feb 27, 2020 at 3:12 PM Alan Batie wrote: > On 2/27/20 2:58 PM, Luoqi Chen wrote: > > One more piece of information that might help: this behavior started > > somewhere between centos 5 and 6, kernel 2.6.18 and 2.6.32, i.e., the > same > > script would fail on 2.6.18. Timing wise I believe it coincided with the > > introduction of nfsv4. > > > > Even if this is a linux bug, given its dominant position, we don't have > > much of a choice but to try to be compatible. Does anyone have say access > > to a netapp and see how it behaves? > > Is this what you mean? > > [101] $ df . > Filesystem 1K-blocks Used Available Use% Mounted on > filer01-cvo.peak.org:/vol/admin > 167772160 73704064 94068096 44% /filer01/cvo-admin > [102] $ rm -f x > [103] $ touch x > [104] $ chmod 000 x > [105] $ ls -l x > ----------. 1 alan wheel 0 Feb 27 15:01 x > [106] $ echo foo > x > -bash: x: Permission denied > [107] $ chmod 600 x > [108] $ cat x > [109] $ cat /etc/redhat-release > CentOS release 6.10 (Final) > > > This works the same way on a truenas server: > > [122] $ rm x > [123] $ df . > Filesystem 1K-blocks Used Available Use% Mounted on > tnas01-cvo.fs10g.peak.org:/mnt/zdata/nfs/admin > 78257431296 54539008 78202892288 1% > /tnas01-cvo/admin > [124] $ touch x > [125] $ chmod 000 x > [126] $ ls -l x > ----------. 1 alan wheel 0 Feb 27 15:05 x > [127] $ echo foo > x > -bash: x: Permission denied > [128] $ chmod 600 x > [129] $ cat x > [130] $ > > However it also does the same on a native FreeBSD 11 server: > > [116] $ uname -a > FreeBSD zbackups02.peak.org 11.3-RELEASE-p3 FreeBSD 11.3-RELEASE-p3 #0: > Mon Aug 19 21:08:43 UTC 2019 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 > > [105] $ cat /etc/redhat-release > CentOS release 6.10 (Final) > [106] $ df . > Filesystem 1K-blocks Used Available Use% Mounted on > zbackups02.peak.org:/zbackups/zmail03-admin > 5039303296 91682304 4947620992 2% /zbackups > [107] $ touch x > [108] $ chmod 0 x > [109] $ ls -l x > ----------. 1 alan root 0 Feb 27 15:08 x > [110] $ echo foo > x > -bash: x: Permission denied > [111] $ chmod 600 x > [112] $ cat x > Yes, except to leave the read bit(s) on. I can confirm that the write would fail on centos 6/7/8 if all bits are cleared. From owner-freebsd-fs@freebsd.org Thu Feb 27 23:40:13 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5BD9C253429 for ; Thu, 27 Feb 2020 23:40:13 +0000 (UTC) (envelope-from alan@peak.org) Received: from filter05.peak.org (filter05.peak.org [69.59.194.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.redcondor.net", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48T8Km0fr7z4G8v for ; Thu, 27 Feb 2020 23:40:11 +0000 (UTC) (envelope-from alan@peak.org) Received: from zmail-mta02.peak.org ([207.55.16.112]) by filter05.peak.org ({27dbf508-291b-4a6b-93f5-d568f05dc56a}) via TCP (outbound) with ESMTPS id 20200227233945445_0000; Thu, 27 Feb 2020 15:39:45 -0800 X-RC-FROM: Received: from localhost (localhost [127.0.0.1]) by zmail-mta02.peak.org (Postfix) with ESMTP id 80B8D4C4D6; Thu, 27 Feb 2020 15:39:38 -0800 (PST) Received: from zmail-mta02.peak.org ([127.0.0.1]) by localhost (zmail-mta02.peak.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id CpLB07yCmYxT; Thu, 27 Feb 2020 15:39:38 -0800 (PST) Received: from mailproxy-lb-05.peak.org (mailproxy-lb-05.peak.org [207.55.17.95]) by zmail-mta02.peak.org (Postfix) with ESMTP id 667C94C4B2; Thu, 27 Feb 2020 15:39:38 -0800 (PST) Subject: Re: Linux could write to read only files on FreeBSD NFS server To: Luoqi Chen Cc: freebsd-fs References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> From: Alan Batie Message-ID: <751dcbf4-9cd6-0d6f-a9d9-38615966cf9d@peak.org> Date: Thu, 27 Feb 2020 15:39:37 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms060509080102040905050203" X-MAG-OUTBOUND: peakinternet.redcondor.net@207.55.16/22 X-Rspamd-Queue-Id: 48T8Km0fr7z4G8v X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=peak.org; spf=pass (mx1.freebsd.org: domain of alan@peak.org designates 69.59.194.81 as permitted sender) smtp.mailfrom=alan@peak.org X-Spamd-Result: default: False [-5.01 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; RCVD_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[peak.org,none]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_IN_DNSWL_LOW(-0.10)[81.194.59.69.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:395796, ipnet:69.59.194.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-0.01)[country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 23:40:13 -0000 This is a cryptographically signed message in MIME format. --------------ms060509080102040905050203 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2/27/20 3:31 PM, Luoqi Chen wrote: > Yes, except to leave the read bit(s) on. I can confirm that the write w= ould > fail on centos 6/7/8 if all bits are cleared. >=20 That makes no difference: [117] $ sh -x ~/nfst + cat /etc/redhat-release CentOS release 6.10 (Final) + df . Filesystem 1K-blocks Used Available Use% Mounted on zbackups02.peak.org:/zbackups/zmail03-admin 5027215872 91682304 4935533568 2% /zbackups + rm -f x + touch x + chmod 444 x + ls -l x -r--r--r--. 1 alan root 0 Feb 27 15:38 x + echo foo /home/alan/nfst: line 9: x: Permission denied + cat x [118] $ --------------ms060509080102040905050203 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC C2swggTgMIIDyKADAgECAhBUdCKrz7BUVHpHGYaNdgQcMA0GCSqGSIb3DQEBCwUAMIGNMQsw CQYDVQQGEwJJVDEQMA4GA1UECAwHQmVyZ2FtbzEZMBcGA1UEBwwQUG9udGUgU2FuIFBpZXRy bzEjMCEGA1UECgwaQWN0YWxpcyBTLnAuQS4vMDMzNTg1MjA5NjcxLDAqBgNVBAMMI0FjdGFs aXMgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIENBIEcyMB4XDTE5MTIxMTE5MDcxNloXDTIwMTIx MTE5MDcxNlowGDEWMBQGA1UEAwwNYWxhbkBwZWFrLm9yZzCCASIwDQYJKoZIhvcNAQEBBQAD ggEPADCCAQoCggEBAJ1Gw0aKu1wno1Vb1MKlI+soIv5Ph03B7gUcg350uVwjm527faMnnV1D TZ415jn4Q5MHjIS5xjFUVJwM0DGOm+aNr0tFPOEL8Y8t2w5KCs7D4ikYderuv57WTwMgjDDY mOI9cUqs+npoFBcFTzx+RunErd8d22EEq61H7Ypyi+ltb4rZweE7KnaS5kgRovJXg8ii90ze dytd96JlTx8+oripPBaG+6RTlZxrQusbvSZpwjEv8xYa3Eh45Z2tBc1xcHNzvaDhprP01OA3 Yx4lIpSxIcAD23vtgjGhU9zycLqbutVpfaLrq3EwzGA7d6Xx97jsrXpCSHYd0TX2OkQKyPcC AwEAAaOCAa4wggGqMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUa/KNnmjBJQQfUTRX9hZc lOpNaRowfgYIKwYBBQUHAQEEcjBwMDsGCCsGAQUFBzAChi9odHRwOi8vY2FjZXJ0LmFjdGFs aXMuaXQvY2VydHMvYWN0YWxpcy1hdXRjbGlnMjAxBggrBgEFBQcwAYYlaHR0cDovL29jc3Aw OS5hY3RhbGlzLml0L1ZBL0FVVEhDTC1HMjAYBgNVHREEETAPgQ1hbGFuQHBlYWsub3JnMEcG A1UdIARAMD4wPAYGK4EfARgBMDIwMAYIKwYBBQUHAgEWJGh0dHBzOi8vd3d3LmFjdGFsaXMu aXQvYXJlYS1kb3dubG9hZDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwSAYDVR0f BEEwPzA9oDugOYY3aHR0cDovL2NybDA5LmFjdGFsaXMuaXQvUmVwb3NpdG9yeS9BVVRIQ0wt RzIvZ2V0TGFzdENSTDAdBgNVHQ4EFgQU4WAWRmEM5pxYuaRqT1VOZP+87GwwDgYDVR0PAQH/ BAQDAgWgMA0GCSqGSIb3DQEBCwUAA4IBAQAFrqbTZz6FiybHjpkR8a/eVtKR6ZTBnFE4f73m 0WC6mqryp4UKkTGl3Id/M1s54aH5Qof+x08Jlb7BgOh1VkjJNIPLWDY6qdZj/idf7DcwGMWN vXevh2QsMjJahnWOuedx+VPJybHfSLnc0iO6xXBrgbbNX6BdTidWbcaT/skBfBygtCy9KVm/ +5CR5NqVDxpmgrWlJsUkYMXaO0jvbCcEvY9LQ9nyMPPK/ttQU9XNNsarBC8cKcX95iL7rgTw AQ1r39pDjyZLC1+bMZqjUleJbpRiiBv6iXb4rFVkXg1R5LFOCX5n1ZtKk8loEFptGtaF+LCw lTtITblJ2dlelQZ3MIIGgzCCBGugAwIBAgIQT94QS+2VW96LrWWHzEFe4zANBgkqhkiG9w0B AQsFADBrMQswCQYDVQQGEwJJVDEOMAwGA1UEBwwFTWlsYW4xIzAhBgNVBAoMGkFjdGFsaXMg Uy5wLkEuLzAzMzU4NTIwOTY3MScwJQYDVQQDDB5BY3RhbGlzIEF1dGhlbnRpY2F0aW9uIFJv b3QgQ0EwHhcNMTkwOTIwMDcxMjA1WhcNMzAwOTIyMTEyMjAyWjCBjTELMAkGA1UEBhMCSVQx EDAOBgNVBAgMB0JlcmdhbW8xGTAXBgNVBAcMEFBvbnRlIFNhbiBQaWV0cm8xIzAhBgNVBAoM GkFjdGFsaXMgUy5wLkEuLzAzMzU4NTIwOTY3MSwwKgYDVQQDDCNBY3RhbGlzIENsaWVudCBB dXRoZW50aWNhdGlvbiBDQSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALdo c3rZPNQv+9xnyj3OlHz/iRnO2hpj8xlHkCdYKNwnRabAT6J0RA11A3ZkQiEZEw66B99ES7Ez v9IRBYmIwsr720lUptObF5L3yVzl3nzaittXwWsq+CQoDEci1cKkWF5SiO22+Np2Epu2HFxk w5nXMnZibrqnC6hUGsFogTDUUVRIuLlublwWYFhpqvDaCh//ucRgRW3+rTU1nBoT1XHkXrLs Cteefjoh+o01tNTWvGi4+3OyABidGPXuoYh7UbYX1u0sG1O8rO92t5zV7/Cr/Vza9EbySh6D rCqsY333sNxikKzFyBwebZv43t1xJyMVE/CRt7BLJOyHxd1Yq0sCAwEAAaOCAf4wggH6MA8G A1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUUtiIOsifeGbtifN7OHCUyQICNtAwQQYIKwYB BQUHAQEENTAzMDEGCCsGAQUFBzABhiVodHRwOi8vb2NzcDA1LmFjdGFsaXMuaXQvVkEvQVVU SC1ST09UMEUGA1UdIAQ+MDwwOgYEVR0gADAyMDAGCCsGAQUFBwIBFiRodHRwczovL3d3dy5h Y3RhbGlzLml0L2FyZWEtZG93bmxvYWQwJwYDVR0lBCAwHgYIKwYBBQUHAwIGCCsGAQUFBwME BggrBgEFBQcDCTCB4wYDVR0fBIHbMIHYMIGWoIGToIGQhoGNbGRhcDovL2xkYXAwNS5hY3Rh bGlzLml0L2NuJTNkQWN0YWxpcyUyMEF1dGhlbnRpY2F0aW9uJTIwUm9vdCUyMENBLG8lM2RB Y3RhbGlzJTIwUy5wLkEuJTJmMDMzNTg1MjA5NjcsYyUzZElUP2NlcnRpZmljYXRlUmV2b2Nh dGlvbkxpc3Q7YmluYXJ5MD2gO6A5hjdodHRwOi8vY3JsMDUuYWN0YWxpcy5pdC9SZXBvc2l0 b3J5L0FVVEgtUk9PVC9nZXRMYXN0Q1JMMB0GA1UdDgQWBBRr8o2eaMElBB9RNFf2FlyU6k1p GjAOBgNVHQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQELBQADggIBAGBEuhmiq3L7DkGaRMG6FTm9 na4v3ya3KW+xkhFvSZgPinqeBi5qfV+dCL/BCuO/JMH9mgI5z57DnYiLQC3CIHnEtalcTfhG PleRgjRMuFQLAeYM5UAZiiPT+D8S7faZ0CZ3glRLw51QTGQJZSC+bN7mgoiBG/HmGahvLWjl kjNZ6o6AmVC3HIV1mGowamiYNEVDmen+SAdJW9uhwP+xFFZodZ0lYJQ6FHg+3pSDVx6YdM94 n9e9tlMnXKB+CY92WmPXbUOMCUjYUmTsxEu9lJEusHv+eehThrO6HiVrkHvEathHnkhphpYm SlG2KOIwfwtqJjJ9C+EMCOcDDa1ndhUTVFMMTAZmyWLRGg0U0O9hzwPA520ZL0Q0iZI7E6Kl OmaQZQX+LORMK4V6hVW9qzPZhgjw2SYux8N8vAWA/3d4ky+j1uVIzk0qRXJ0iD+B1uTyOjEx 15fmm+mowp7ycOhNUxi4d8ycqb+QkPBbZtM+zCi7eWa9hOI6I2V3mZ9bFKUqonWcqfZhvy2D EZhzJLYQ0Zw5ztrR7+fmDjuHFBG07eQcMBOUT46qL7J3ncneUooyCvpNTAlxSzE3xEc96lDd 4v38Lnl3BsuIxH9p/xb2LBGNxgR12QjFVj33wX25fyE47PUPTRt+2wBJv5oNsjatNjS4w20C CoLfVtGgVPUrMYIEFzCCBBMCAQEwgaIwgY0xCzAJBgNVBAYTAklUMRAwDgYDVQQIDAdCZXJn YW1vMRkwFwYDVQQHDBBQb250ZSBTYW4gUGlldHJvMSMwIQYDVQQKDBpBY3RhbGlzIFMucC5B Li8wMzM1ODUyMDk2NzEsMCoGA1UEAwwjQWN0YWxpcyBDbGllbnQgQXV0aGVudGljYXRpb24g Q0EgRzICEFR0IqvPsFRUekcZho12BBwwDQYJYIZIAWUDBAIBBQCgggJFMBgGCSqGSIb3DQEJ AzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIwMDIyNzIzMzkzN1owLwYJKoZIhvcN AQkEMSIEIOb9GrkE7v51PZYefSwh8KNJ3Jy6yWDuQ+ypPugHqhWKMGwGCSqGSIb3DQEJDzFf MF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgIC AIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgbMGCSsGAQQBgjcQ BDGBpTCBojCBjTELMAkGA1UEBhMCSVQxEDAOBgNVBAgMB0JlcmdhbW8xGTAXBgNVBAcMEFBv bnRlIFNhbiBQaWV0cm8xIzAhBgNVBAoMGkFjdGFsaXMgUy5wLkEuLzAzMzU4NTIwOTY3MSww KgYDVQQDDCNBY3RhbGlzIENsaWVudCBBdXRoZW50aWNhdGlvbiBDQSBHMgIQVHQiq8+wVFR6 RxmGjXYEHDCBtQYLKoZIhvcNAQkQAgsxgaWggaIwgY0xCzAJBgNVBAYTAklUMRAwDgYDVQQI DAdCZXJnYW1vMRkwFwYDVQQHDBBQb250ZSBTYW4gUGlldHJvMSMwIQYDVQQKDBpBY3RhbGlz IFMucC5BLi8wMzM1ODUyMDk2NzEsMCoGA1UEAwwjQWN0YWxpcyBDbGllbnQgQXV0aGVudGlj YXRpb24gQ0EgRzICEFR0IqvPsFRUekcZho12BBwwDQYJKoZIhvcNAQEBBQAEggEAdO9J3Vn6 j8HsF5azy9PPGPiqzBPg5cozfNnn+FKLNKEh1Lmm1CHdR71kBA1x7XGIDk/PU1DcN4BjOR+E hPzcMUN6zBtFV8XFp0DjT4KnxVRCDNxnQkDgZ+63jygEaa0Pc62TBeH2kvHS6wfs+2X44hJa hWrn/jikmFrUWGSAWnZkc+duV9t3nC42885Ku2xC4dNYeUuVfXHM7dIz3zV5GVt3/aS3geya BJkCUgwLL3n0+5N8CDnrFGKJFfJykChjCfjSAFfEXhS31fIwrXvHxNq15farHKFhs+m3Pzxp FRi5O9vW2PcmWfOEcpLUp5qdHMdNbCyerq4a0hntvZYNyAAAAAAAAA== --------------ms060509080102040905050203-- From owner-freebsd-fs@freebsd.org Fri Feb 28 01:30:52 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CDF5A255D77 for ; Fri, 28 Feb 2020 01:30:52 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: from mail-oi1-x230.google.com (mail-oi1-x230.google.com [IPv6:2607:f8b0:4864:20::230]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48TBnR4QCdz3NbS for ; Fri, 28 Feb 2020 01:30:51 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: by mail-oi1-x230.google.com with SMTP id l136so1351172oig.1 for ; Thu, 27 Feb 2020 17:30:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=c8W4YZ/x9GuAyneela/kaArm60/T+KjyY7VMBsmWnHo=; b=R5UyD5XZdLWn2iPclsi9aNU8erqEpijX5nFavSXqWOh07wphmxeF1R8d5WtuZLwigw 207iD+pFetvIiCUq4wf8BKezQvi3IaW7tZuFfwyjPJOIPlyMYLnpykYyxt4UbVW1Pftc IW3pyTL8mfRQ5SZl24X5tifGgV6ZOskziE/zhNpNR+fOjgib2LiwrBPmtg1wHVvcSovl IZ11itAj8SsbpdYhQYM66Pi08+IHx9prXH+odLPXX0UlQJVNrmF3sg92Q4Bg+tQSspeg fSBY6wyqOs0J6u+9Z13ohg2cf8UwMYajWeYqLP5CuCeY9a3hr1WHeepkhH21OnMk20MW GPwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=c8W4YZ/x9GuAyneela/kaArm60/T+KjyY7VMBsmWnHo=; b=AmbmUS8oit5Q8oxjrosjy68Oh46bVZvm83wSFh3bTntin3YED6/ERw/MHPrFizsQms 80jD2x9SMMfWELq+N75lVUDOTqvC/+lXuuRYqofWr+WG+/xFamFmfu8azbiY1rvCPXC2 pAOzkHxCvM1TDg+R70M2tF4ALua0lVVYGorXNZv77mnKK5vT2PP6R8ubmXSIEMa7e1a8 JmMRPYoRg0bBSPpDy6ytHx8Xg92s2E8M3MtKIwuAUYfFtz5n1LG40QAI2ldOZBLYCohg FZKo//gXYsdNVGtDxcp7cgwr6gzBZiTvNMSoQjxMHhObSwwHV8x3dq4uwvgd6OSt6c6D jFvw== X-Gm-Message-State: APjAAAXsCuwafHrRmIUegxpOnLax2javir512zlMLOKJ3mgF2IecbwKa qcKKiGZpulQBZOdc8CEZ3H8juoTvNis+aGkCwzoK4nyaROI= X-Google-Smtp-Source: APXvYqz5ncBQTOGn7eqN95zBfK4TRMPBwWviFqLn6hhfirac71z9zKagDkLcOz514yXKhxSomLXL5PNInuiSmpdhHLo= X-Received: by 2002:aca:af09:: with SMTP id y9mr1353400oie.101.1582853449872; Thu, 27 Feb 2020 17:30:49 -0800 (PST) MIME-Version: 1.0 References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> <751dcbf4-9cd6-0d6f-a9d9-38615966cf9d@peak.org> In-Reply-To: <751dcbf4-9cd6-0d6f-a9d9-38615966cf9d@peak.org> From: Luoqi Chen Date: Thu, 27 Feb 2020 17:30:39 -0800 Message-ID: Subject: Re: Linux could write to read only files on FreeBSD NFS server To: Alan Batie Cc: freebsd-fs X-Rspamd-Queue-Id: 48TBnR4QCdz3NbS X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=R5UyD5XZ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luoqichen@gmail.com designates 2607:f8b0:4864:20::230 as permitted sender) smtp.mailfrom=luoqichen@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[0.3.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; IP_SCORE(0.00)[ip: (-7.98), ipnet: 2607:f8b0::/32(-1.88), asn: 15169(-1.67), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2020 01:30:52 -0000 On Thu, Feb 27, 2020 at 3:40 PM Alan Batie wrote: > On 2/27/20 3:31 PM, Luoqi Chen wrote: > > > Yes, except to leave the read bit(s) on. I can confirm that the write > would > > fail on centos 6/7/8 if all bits are cleared. > > > > That makes no difference: > > [117] $ sh -x ~/nfst > + cat /etc/redhat-release > CentOS release 6.10 (Final) > + df . > Filesystem 1K-blocks Used Available Use% Mounted on > zbackups02.peak.org:/zbackups/zmail03-admin > 5027215872 91682304 4935533568 2% /zbackups > + rm -f x > + touch x > + chmod 444 x > + ls -l x > -r--r--r--. 1 alan root 0 Feb 27 15:38 x > + echo foo > /home/alan/nfst: line 9: x: Permission denied > + cat x > [118] $ > > Sorry, my mistake, the read bit didn't matter, I mistook the error message from `cat x` as the echo failure, which I didn't get but you did, c1n7-154> sh -x rotest.sh + cat /etc/redhat-release CentOS release 6.5 (Final) + uname -a Linux c1n7 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux + mount ++ df . ++ cut '-d ' -f1 ++ tail -1 + grep c1n5:/data c1n5:/data on /net/c1n5/data type nfs (rw,nosuid,nodev,intr,sloppy,vers=4,addr=10.201.2.5,clientaddr=10.201.2.7) + rm -f x + touch x + ls -l x -rw-r--r--. 1 luoqi wheel 0 Feb 27 17:17 x + chmod 0444 x + ls -l x -r--r--r--. 1 luoqi wheel 0 Feb 27 17:17 x + echo aaa + cat x aaa + rm -f x + touch x + ls -l x -rw-r--r--. 1 luoqi wheel 0 Feb 27 17:17 x + chmod 0 x + ls -l x ----------. 1 luoqi wheel 0 Feb 27 17:17 x + echo bbb + cat x cat: x: Permission denied Another linux machine with the latest centos, same outcome, c1n14-314> sh -x rotest.sh + cat /etc/redhat-release CentOS Linux release 8.1.1911 (Core) + uname -a Linux c1n14 4.18.0-147.el8.x86_64 #1 SMP Wed Dec 4 21:51:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux + mount ++ df . ++ tail -1 ++ cut '-d ' -f1 + grep c1n5:/data c1n5:/data on /net/c1n5/data type nfs4 (rw,nosuid,nodev,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.201.2.14,local_lock=none,addr=10.201.2.5) + rm -f x + touch x + ls -l x -rw-r--r--. 1 luoqi wheel 0 Feb 27 17:15 x + chmod 0444 x + ls -l x -r--r--r--. 1 luoqi wheel 0 Feb 27 17:15 x + echo aaa + cat x aaa + rm -f x + touch x + ls -l x -rw-r--r--. 1 luoqi wheel 0 Feb 27 17:15 x + chmod 0 x + ls -l x ----------. 1 luoqi wheel 0 Feb 27 17:15 x + echo bbb + cat x cat: x: Permission denied The freebsd server is running, c1n5-45> uname -a FreeBSD c1n5 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC amd64 From owner-freebsd-fs@freebsd.org Fri Feb 28 11:12:12 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1AE312606B4 for ; Fri, 28 Feb 2020 11:12:12 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 48TRh94JcRz4shC for ; Fri, 28 Feb 2020 11:12:09 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id 01SBC4Ne040985; Fri, 28 Feb 2020 11:12:04 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id 01SBC4aN017681; Fri, 28 Feb 2020 11:12:04 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id 01SBC4ui017677; Fri, 28 Feb 2020 11:12:04 GMT Date: Fri, 28 Feb 2020 11:12:04 GMT Message-Id: <202002281112.01SBC4ui017677@higson.cam.lispworks.com> From: Martin Simmons To: Alan Batie CC: luoqi.chen@gmail.com, freebsd-fs@freebsd.org In-reply-to: <751dcbf4-9cd6-0d6f-a9d9-38615966cf9d@peak.org> (message from Alan Batie on Thu, 27 Feb 2020 15:39:37 -0800) Subject: Re: Linux could write to read only files on FreeBSD NFS server References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> <751dcbf4-9cd6-0d6f-a9d9-38615966cf9d@peak.org> X-Rspamd-Queue-Id: 48TRh94JcRz4shC X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of martin@lispworks.com has no SPF policy when checking 46.17.166.21) smtp.mailfrom=martin@lispworks.com X-Spamd-Result: default: False [-0.54 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.65)[-0.652,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.87)[-0.869,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[lispworks.com]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[21.166.17.46.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:51055, ipnet:46.17.166.0/24, country:GB]; FREEMAIL_CC(0.00)[gmail.com]; IP_SCORE(-0.01)[country: GB(-0.07)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2020 11:12:12 -0000 >>>>> On Thu, 27 Feb 2020 15:39:37 -0800, Alan Batie said: > > That makes no difference: > > [117] $ sh -x ~/nfst > + cat /etc/redhat-release > CentOS release 6.10 (Final) > + df . > Filesystem 1K-blocks Used Available Use% Mounted on > zbackups02.peak.org:/zbackups/zmail03-admin > 5027215872 91682304 4935533568 2% /zbackups > + rm -f x > + touch x > + chmod 444 x > + ls -l x > -r--r--r--. 1 alan root 0 Feb 27 15:38 x > + echo foo > /home/alan/nfst: line 9: x: Permission denied > + cat x > [118] $ Is this mounted with nfs version 3? I'm mounting with version 3 and also get Permission denied from CentOS release 6.10 (Final). __Martin From owner-freebsd-fs@freebsd.org Fri Feb 28 11:13:53 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3E0C9260753 for ; Fri, 28 Feb 2020 11:13:53 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id 48TRk737RZz3DLq for ; Fri, 28 Feb 2020 11:13:50 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id 01SBDmnn041223; Fri, 28 Feb 2020 11:13:48 GMT (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id 01SBDlZA017701; Fri, 28 Feb 2020 11:13:47 GMT Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id 01SBDlsl017697; Fri, 28 Feb 2020 11:13:47 GMT Date: Fri, 28 Feb 2020 11:13:47 GMT Message-Id: <202002281113.01SBDlsl017697@higson.cam.lispworks.com> From: Martin Simmons To: Luoqi Chen CC: rmacklem@uoguelph.ca, freebsd-fs@freebsd.org In-reply-to: (message from Luoqi Chen on Thu, 27 Feb 2020 14:58:55 -0800) Subject: Re: Linux could write to read only files on FreeBSD NFS server References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> X-Rspamd-Queue-Id: 48TRk737RZz3DLq X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of martin@lispworks.com has no SPF policy when checking 46.17.166.21) smtp.mailfrom=martin@lispworks.com X-Spamd-Result: default: False [-0.12 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.66)[-0.664,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.44)[-0.440,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[lispworks.com]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[21.166.17.46.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:51055, ipnet:46.17.166.0/24, country:GB]; IP_SCORE(-0.01)[country: GB(-0.07)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2020 11:13:53 -0000 >>>>> On Thu, 27 Feb 2020 14:58:55 -0800, Luoqi Chen said: > > One more piece of information that might help: this behavior started > somewhere between centos 5 and 6, kernel 2.6.18 and 2.6.32, i.e., the same > script would fail on 2.6.18. Timing wise I believe it coincided with the > introduction of nfsv4. Have you tried mounting it with nfsv3 recently? I can't repeat it with that version (I don't run nfsv4 at all). __Martin From owner-freebsd-fs@freebsd.org Fri Feb 28 16:43:03 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9E259266EB5 for ; Fri, 28 Feb 2020 16:43:03 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: from mail-ot1-x344.google.com (mail-ot1-x344.google.com [IPv6:2607:f8b0:4864:20::344]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48Tb1y2B19z4Jlq for ; Fri, 28 Feb 2020 16:43:02 +0000 (UTC) (envelope-from luoqi.chen@gmail.com) Received: by mail-ot1-x344.google.com with SMTP id r16so3180899otd.2 for ; Fri, 28 Feb 2020 08:43:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=E7XVqW8E3Ia5039Hy/nF6/v3hyHoBuseDNsgFyuNrzs=; b=BPxsXROlaGe1KUFrokbEIxPUNDP7IH5am63D+mtIDxtC/XJ2gCYT/Ers8PEfgLIpCC 0+WV/R1SctfX+UdIjZ8xdZItrN5h274QcPVyWCYWEu1ln4ByBQcOByWvu4R/0Ta3s1hp eyVgFzBkXOZPR395zN3y9xXKHgHPYYWxzbZpHP0o2M1BmlKDKtx3RZgtsl0T0CmgI8h7 fvjKiSEj2pKzKjIdSNMFWxauBV+iJhdnvX9/C2Bl+KY/xWVUE7UmwQjZthKMUZBCwKDP rtwH7dzC0zb27S6EvSo+J7KH+eJKtTae0/5fdSalxgiawBNlhdqbd/LLUQfbItXCiPH+ f+Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=E7XVqW8E3Ia5039Hy/nF6/v3hyHoBuseDNsgFyuNrzs=; b=DFsMczxq4lSk4WPKC2/dqmHFPmGdIgTPIjk5MQW5/qQnL3wOBNAh3aD8wlclyt3psp CvDG+H5QzbdjZPi3g8gmsZDicDf5kDG6h3ggAuMboKYB37WfOS+MhhxHeAc0LrjvmCh6 cVJOjWwh5WErI8BDexTZukhC0HF3YA0PVWmRdhpJP6s/kN+ruQzbe6U/dNbUpivw/9Tu RhWkxuZhE7+T+QQB35r3guIsvgUDHM0g7r1dvU8/jmd3YUjiDkik/RIOjdcim2KWebkl 7u2E2ujofset3k+y4u8gInAlcpxy8dDDuutgnnds5fVof9FxUq1IyFTLPRnWzRdwIIPY dbng== X-Gm-Message-State: APjAAAXFWdU26VlV4YoXAiyIA8ZezuE7Ux7UmRyjS2Ir1x9xO4tjwV3n VTXcSKfjORKVWTQwvn1TPd946FIB+fIWc1KgIMs= X-Google-Smtp-Source: APXvYqwXU2eIEOG2g1SM+C6Lj6f2Oh2m2mAtiEoW8IpFciXFOElaLX6I+rUV5yLmeH1IV4gi/hRstFPJxWZh4IZY+xM= X-Received: by 2002:a9d:5e82:: with SMTP id f2mr3968240otl.240.1582908181376; Fri, 28 Feb 2020 08:43:01 -0800 (PST) MIME-Version: 1.0 References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> <202002281113.01SBDlsl017697@higson.cam.lispworks.com> In-Reply-To: <202002281113.01SBDlsl017697@higson.cam.lispworks.com> From: Luoqi Chen Date: Fri, 28 Feb 2020 08:42:52 -0800 Message-ID: Subject: Re: Linux could write to read only files on FreeBSD NFS server To: Martin Simmons Cc: Rick Macklem , freebsd-fs X-Rspamd-Queue-Id: 48Tb1y2B19z4Jlq X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=BPxsXROl; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luoqichen@gmail.com designates 2607:f8b0:4864:20::344 as permitted sender) smtp.mailfrom=luoqichen@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[4.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (2.94), ipnet: 2607:f8b0::/32(-1.88), asn: 15169(-1.67), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2020 16:43:03 -0000 On Fri, Feb 28, 2020 at 3:13 AM Martin Simmons wrote: > >>>>> On Thu, 27 Feb 2020 14:58:55 -0800, Luoqi Chen said: > > > > One more piece of information that might help: this behavior started > > somewhere between centos 5 and 6, kernel 2.6.18 and 2.6.32, i.e., the > same > > script would fail on 2.6.18. Timing wise I believe it coincided with the > > introduction of nfsv4. > > Have you tried mounting it with nfsv3 recently? I can't repeat it with > that > version (I don't run nfsv4 at all). Looks like I'm getting senile... The script works correctly with nfsv3 mounts. This is a nfsv4 specific problem.