From owner-freebsd-fs@freebsd.org Sat Mar 13 16:29:42 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9EA7D57B779 for ; Sat, 13 Mar 2021 16:29:42 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "unimail.tu-dortmund.de", Issuer "DFN-Verein Global Issuing CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DySnd6WCGz53n3 for ; Sat, 13 Mar 2021 16:29:41 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from [192.168.111.102] (p2e513b89.dip0.t-ipconnect.de [46.81.59.137]) (authenticated bits=0) by unimail.uni-dortmund.de (8.16.1/8.16.1) with ESMTPSA id 12DGTdLd004960 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT); Sat, 13 Mar 2021 17:29:39 +0100 (CET) To: Konstantin Belousov Cc: freebsd-fs@freebsd.org References: <49130618-349a-bfc7-6d26-0c3763904dc5@tu-dortmund.de> From: Alexander Lochmann Subject: Re: [RFC] Understanding the locking of struct buf Message-ID: <48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de> Date: Sat, 13 Mar 2021 17:29:39 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: de-DE Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4DySnd6WCGz53n3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of alexander.lochmann@tu-dortmund.de designates 129.217.128.51 as permitted sender) smtp.mailfrom=alexander.lochmann@tu-dortmund.de X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:129.217.128.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tu-dortmund.de]; ARC_NA(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[129.217.128.51:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[129.217.128.51:from]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:680, ipnet:129.217.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-fs]; RECEIVED_SPAMHAUS_PBL(0.00)[46.81.59.137:received] X-Mailman-Approved-At: Sat, 13 Mar 2021 18:22:46 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Mar 2021 16:29:42 -0000 On 13.03.21 13:59, Konstantin Belousov wrote: > What consistency? If you are talking about multithreading memory model > as expected by the FreeBSD kernel, look at atomic(9). It has assumptions > like atomicity of naturally-aligned word-sized integer accesses written > out explicitly. > Yeah, I think that's what I meant. Thx! In sys/sys/buf.h, it says "All fields are protected by the buffer lock except those marked [...]". How does this fit together with atomic(9)? If I had to implement some extension to vfs_bio.c, for example, I would follow those locking rules without knowing that I may access some fields without any lock. Shouldn't that particular piece of documentation be updated? For example: b_bcount /* w: b_lock, r: no lock needed */ >> In FreeBSD, do I have to use lock X in any case except Y and Z? >> Or is it the other way round: Do I need no lock at all except for case X >> and Y? > I do not understand this question(?). > I'm sry. Maybe my question is just nuts.>> >>> Are you reporting a bug or just asking about LK_KERNPROC. Lockmgr >>> (kern_lock.c)is careful enough to handle LK_KERNPROC owner specially. In >>> particular, it does not report unlock of such lock to witness. >> First of all, I want to understand how things in FreeBSD work. >> >From what I understand now: When setting up an asynchronous IO request, >> buf.b_lock is taken by thread X. Later on LK_KERNPROC is used to hand >> over the ownership to the kernel. The lock itself is still acquired. > The lock is acquired in the sense that nobody else can take the buffer' > lock until the call to lockmgr(LK_RELEASE). So can we safely assume, for our analysis, that the b_lock is still acquired during bufdone and the other related fns? But from this point, there > is no thread owning the lock. Consider that the lock was converted to > the 1-counting semaphore.I'm not sure. What do you mean bei 1-counting semaphore? > >> The completion of the IO job is performed in the kernel's context, which >> releases buf.b_lock in brelse(). > The completion for async IO is performed in the context of some other > thread, typically either geom io up thread, or direct completion thread > of some disk driver. This is why this somewhat strange LK_KERNPROC > business is needed at all. > > For sync IO, that thread only signals original thread that io completed. > In this case, no LK_KERNPROC trick is performed. > That is clear to me. >> So there is no explicit lock call in the latter context, is it? > No lock call, but there is an unlock. Are you talking about BUF_UNLOCK()? > Assuming you wrote the stack bottom-up, this is exactly what I wrote above: > xpt_done_td is CAM IO completion thread, and it performs actions after hw > informed that io request (bio) was completed. > Thx. >> allocbuf > When brelse() notes that buffer was marked as 'no cache', it demolishes > the buffer right after async io finishes. Perhaps this is the case that > you observed. Yeah, but maybe our approach is just inaccurate. Due to the WITNESS_UNLOCK() call in __lockmgr_disown(), we assume the b_lock has already been released. - Alex -- Technische Universität Dortmund Alexander Lochmann PGP key: 0xBC3EF6FD Otto-Hahn-Str. 16 phone: +49.231.7556141 D-44227 Dortmund fax: +49.231.7556116 http://ess.cs.tu-dortmund.de/Staff/al