Date: Sun, 14 Mar 2021 10:35:24 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: Alexander Lochmann <alexander.lochmann@tu-dortmund.de> Cc: freebsd-fs@freebsd.org Subject: Re: [RFC] Understanding the locking of struct buf Message-ID: <YE3KzAiZwcSZZLWS@kib.kiev.ua> In-Reply-To: <48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de> References: <49130618-349a-bfc7-6d26-0c3763904dc5@tu-dortmund.de> <YEwx0WdSuJpzhONL@kib.kiev.ua> <c49dc72a-da0f-a089-7e93-e4e54d0c03eb@tu-dortmund.de> <YEy3JET6Lx7BkJP3@kib.kiev.ua> <48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Mar 13, 2021 at 05:29:39PM +0100, Alexander Lochmann wrote: > > > On 13.03.21 13:59, Konstantin Belousov wrote: > > What consistency? If you are talking about multithreading memory model > > as expected by the FreeBSD kernel, look at atomic(9). It has assumptions > > like atomicity of naturally-aligned word-sized integer accesses written > > out explicitly. > > > Yeah, I think that's what I meant. Thx! > > In sys/sys/buf.h, it says "All fields are protected by the buffer lock > except those marked [...]". How does this fit together with atomic(9)? > If I had to implement some extension to vfs_bio.c, for example, I would > follow those locking rules without knowing that I may access some fields > without any lock. You cannot write any working code that uses buffer cache, without studying both buffer cache code itself, and examples using it from filesystems. Short (even the whole-screen short) herald comment cannot explain all the nuances that grown in the 40+ years codebase. > Shouldn't that particular piece of documentation be updated? > For example: b_bcount /* w: b_lock, r: no lock needed */ Do you understand the purpose of the locking? What do you intent to do with the read value of b_bcount, if you do not own the buffer? What is you next action with that value? > >> In FreeBSD, do I have to use lock X in any case except Y and Z? > >> Or is it the other way round: Do I need no lock at all except for case X > >> and Y? > > I do not understand this question(?). > > > I'm sry. Maybe my question is just nuts.>> > >>> Are you reporting a bug or just asking about LK_KERNPROC. Lockmgr > >>> (kern_lock.c)is careful enough to handle LK_KERNPROC owner specially. In > >>> particular, it does not report unlock of such lock to witness. > >> First of all, I want to understand how things in FreeBSD work. > >> >From what I understand now: When setting up an asynchronous IO request, > >> buf.b_lock is taken by thread X. Later on LK_KERNPROC is used to hand > >> over the ownership to the kernel. The lock itself is still acquired. > > The lock is acquired in the sense that nobody else can take the buffer' > > lock until the call to lockmgr(LK_RELEASE). > So can we safely assume, for our analysis, that the b_lock is still > acquired during bufdone and the other related fns? LK_KERNPROC buffer is logically owned by the io completion thread(s). > But from this point, there > > is no thread owning the lock. Consider that the lock was converted to > > the 1-counting semaphore.I'm not sure. What do you mean bei 1-counting semaphore? > > > >> The completion of the IO job is performed in the kernel's context, which > >> releases buf.b_lock in brelse(). > > The completion for async IO is performed in the context of some other > > thread, typically either geom io up thread, or direct completion thread > > of some disk driver. This is why this somewhat strange LK_KERNPROC > > business is needed at all. > > > > For sync IO, that thread only signals original thread that io completed. > > In this case, no LK_KERNPROC trick is performed. > > > That is clear to me. > >> So there is no explicit lock call in the latter context, is it? > > No lock call, but there is an unlock. > Are you talking about BUF_UNLOCK()? > > > Assuming you wrote the stack bottom-up, this is exactly what I wrote above: > > xpt_done_td is CAM IO completion thread, and it performs actions after hw > > informed that io request (bio) was completed. > > > Thx. > >> allocbuf > > When brelse() notes that buffer was marked as 'no cache', it demolishes > > the buffer right after async io finishes. Perhaps this is the case that > > you observed. > > Yeah, but maybe our approach is just inaccurate. Due to the > WITNESS_UNLOCK() call in __lockmgr_disown(), we assume the b_lock has > already been released. > > - Alex > > -- > Technische Universität Dortmund > Alexander Lochmann PGP key: 0xBC3EF6FD > Otto-Hahn-Str. 16 phone: +49.231.7556141 > D-44227 Dortmund fax: +49.231.7556116 > http://ess.cs.tu-dortmund.de/Staff/al
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YE3KzAiZwcSZZLWS>