From owner-freebsd-smp Sun Apr 22 0:11: 7 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gratis.grondar.za (grouter.grondar.za [196.7.18.65]) by hub.freebsd.org (Postfix) with ESMTP id EC0CD37B423 for ; Sun, 22 Apr 2001 00:11:00 -0700 (PDT) (envelope-from mark@grondar.za) Received: from grondar.za (gratis.grondar.za [196.7.18.133]) by gratis.grondar.za (8.11.3/8.11.3) with ESMTP id f3M7Asw38713 for ; Sun, 22 Apr 2001 09:10:56 +0200 (SAST) (envelope-from mark@grondar.za) Message-Id: <200104220710.f3M7Asw38713@gratis.grondar.za> Cc: smp@FreeBSD.ORG Subject: Re: Please review - header cleanups References: In-Reply-To: ; from Bruce Evans "Sun, 22 Apr 2001 11:57:19 +1000." Date: Sun, 22 Apr 2001 09:12:25 +0200 From: Mark Murray Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Said Bruce Evans : > > > LINT compiles after adding includes of (and sometimes > > > ) to "only" about 50 .c files. Many more probably depend > > > on the pollution in :-(. About 1/2 of the 50 really should > > I tried cleaning up and didn't find a good way. There is > also pollution related to curproc in . includes > to get curproc defined so that the inlines in > compile. Compiling these inlines only when they are used and not > including shows that many places have come to depend on > for its side effect of declaring curproc. What a mess. > > So making a seems to be on the cards. > > The mess for is much older and messier than for . > Now, is sort of an extension of , but most places > that include it are for its (intentional) side effect of including the > old lock interface, . The old lock interface will be going > away, so we shouldn't move the include of to *.c. OTOH, > the entanglement of makes it difficult to include > in the right places (if any). I think the next step should > be to include instead of in *.h. Damn. :-). I went and put lockmgr into .c's locally. NP - I fix. > > How bad does a sound? > > > > How bad does any <*/*_macros.h> sound (for both macros and inline > > code) where the header contains approximately _only_ executable > > stuff as opposed to "pure" declarations? > > I don't see how this would help. .c files can't include *_macros.h > without knowing too many implementation details, and .h files can't > include them without getting lots of pollution (since lots of interfaces > are needed to implement complicated inline functions or to call > complicated macros like mtx_lock()) (unless *_macros.h provide > alternative non-polluting interfaces for _everything_ relevant, which > I think would be too much work). Yuk. OK, thanks! M -- Mark Murray Warning: this .sig is umop ap!sdn To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Apr 22 4:16:55 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gratis.grondar.za (grouter.grondar.za [196.7.18.65]) by hub.freebsd.org (Postfix) with ESMTP id 8BFB637B422 for ; Sun, 22 Apr 2001 04:16:48 -0700 (PDT) (envelope-from mark@grondar.za) Received: from grondar.za (gratis.grondar.za [196.7.18.133]) by gratis.grondar.za (8.11.3/8.11.3) with ESMTP id f3MBGfw40313 for ; Sun, 22 Apr 2001 13:16:44 +0200 (SAST) (envelope-from mark@grondar.za) Message-Id: <200104221116.f3MBGfw40313@gratis.grondar.za> To: smp@FreeBSD.ORG Subject: Re: Include file cleanup, take #2 References: <200104212147.f3LLlMw34206@gratis.grondar.za> In-Reply-To: <200104212147.f3LLlMw34206@gratis.grondar.za> ; from Mark Murray "Sat, 21 Apr 2001 23:48:57 +0200." Date: Sun, 22 Apr 2001 13:18:11 +0200 From: Mark Murray Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I said: > Please review and comment. > > http://prople.freebsd.org/~mark/patches/sys.SYS_MUTEX.diff.0 Please ignore this. I entrenched with it. I will post a new patch shortly that fixes this. Thanks! M -- Mark Murray Warning: this .sig is umop ap!sdn To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Apr 22 8: 9:56 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gratis.grondar.za (grouter.grondar.za [196.7.18.65]) by hub.freebsd.org (Postfix) with ESMTP id 83F1337B423 for ; Sun, 22 Apr 2001 08:09:50 -0700 (PDT) (envelope-from mark@grondar.za) Received: from grondar.za (gratis.grondar.za [196.7.18.133]) by gratis.grondar.za (8.11.3/8.11.3) with ESMTP id f3MF9iw41961 for ; Sun, 22 Apr 2001 17:09:46 +0200 (SAST) (envelope-from mark@grondar.za) Message-Id: <200104221509.f3MF9iw41961@gratis.grondar.za> To: smp@freebsd.org Subject: Kernel include file cleanup take #3. Date: Sun, 22 Apr 2001 17:11:25 +0200 From: Mark Murray Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi I made a bit of a faux pas with my last effort. The aim of this patch is to 1) Reduce the number of "#include " and "#include " in other include files. 2) Reduce the dependancy of .c files on lockmgr by making lockmgr a subinclude of sys/lock.h (lockmgr is deprecated and is being removed slowly). Please review. Thanks! http://people.freebsd.org/~markm/patches/sys.SYS_MUTEX.diff.1 M -- Mark Murray Warning: this .sig is umop ap!sdn To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Apr 22 22: 9:53 2001 Delivered-To: freebsd-smp@freebsd.org Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by hub.freebsd.org (Postfix) with ESMTP id 4478837B423 for ; Sun, 22 Apr 2001 22:09:49 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from bde.zeta.org.au (bde.zeta.org.au [203.2.228.102]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id PAA27843; Mon, 23 Apr 2001 15:09:29 +1000 Date: Mon, 23 Apr 2001 15:07:07 +1000 (EST) From: Bruce Evans X-Sender: bde@besplex.bde.org To: Mark Murray Cc: smp@FreeBSD.ORG Subject: Re: Kernel include file cleanup take #3. In-Reply-To: <200104221509.f3MF9iw41961@gratis.grondar.za> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Sun, 22 Apr 2001, Mark Murray wrote: > The aim of this patch is to > > 1) Reduce the number of "#include " and "#include > " in other include files. > > 2) Reduce the dependancy of .c files on lockmgr by making lockmgr > a subinclude of sys/lock.h (lockmgr is deprecated and is being > removed slowly). > > Please review. > > Thanks! > > http://people.freebsd.org/~markm/patches/sys.SYS_MUTEX.diff.1 I'm almost (:-) happy with this version. The includes of in and are too hard to move without moving the inline functions. was already a subinclude of , and _SYS_LOCKMGR_H_ is still misspelled (was missing SYS_ and MGR, now missing SYS_ ...). You actually made a subinclude of other headers that need it, and one that doesn't ( doesn't need it directly, but it includes which does). doesn't need it any more (but *.c might depend on it being there). Some of the includes of it are commented on too verbosely. Some of the new includes seem to be a bit more disordered than necessary. When inserting in unsorted include lists, I normally insert just before the first order reversal (not counting param.h or systm.h). Please change the copyright owner to yourself. FreeBSD, Inc. still doesn't exist in a form suitable for holding copyrights. I'm not sure if the names for the new types headers are right. Perhaps they should have a leading underscore to inhibit direct inclusion, or not have the "_types" suffix, to make them less verbose and not hint that they are limited to types. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Apr 22 23: 2:59 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gratis.grondar.za (grouter.grondar.za [196.7.18.65]) by hub.freebsd.org (Postfix) with ESMTP id 3877537B42C for ; Sun, 22 Apr 2001 23:02:51 -0700 (PDT) (envelope-from mark@grondar.za) Received: from grondar.za (gratis.grondar.za [196.7.18.133]) by gratis.grondar.za (8.11.3/8.11.3) with ESMTP id f3N62Tw49163; Mon, 23 Apr 2001 08:02:35 +0200 (SAST) (envelope-from mark@grondar.za) Message-Id: <200104230602.f3N62Tw49163@gratis.grondar.za> To: Bruce Evans Cc: smp@FreeBSD.ORG Subject: Re: Kernel include file cleanup take #3. References: In-Reply-To: ; from Bruce Evans "Mon, 23 Apr 2001 15:07:07 +1000." Date: Mon, 23 Apr 2001 08:04:05 +0200 From: Mark Murray Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > http://people.freebsd.org/~markm/patches/sys.SYS_MUTEX.diff.1 > > I'm almost (:-) happy with this version. The includes of > in and are too hard to move without moving > the inline functions. Right. > was already a subinclude of , and > _SYS_LOCKMGR_H_ is still misspelled (was missing SYS_ and MGR, now > missing SYS_ ...). D'uh. Ok fixed. > You actually made a subinclude of > other headers that need it, and one that doesn't ( doesn't > need it directly, but it includes which does). OK. Removed it (plus some other junk) from eventhandler. > doesn't need it any more (but *.c might depend on it > being there). Some of the includes of it are commented on too > verbosely. Empirically, a lot of .c's need it now. (particularly now that I've knocked sys/lockmgr.h out of them). > Some of the new includes seem to be a bit more disordered than necessary. > When inserting in unsorted include lists, I normally insert just before > the first order reversal (not counting param.h or systm.h). Noted. I'll fix that. There may be other ordering issues. This one will take a bit of time. > Please change the copyright owner to yourself. FreeBSD, Inc. still > doesn't exist in a form suitable for holding copyrights. OK. Done. For such small files I'd prefer them to be owned by the project, but I don't care enough to be religious about it. > I'm not sure if the names for the new types headers are right. Perhaps > they should have a leading underscore to inhibit direct inclusion, or > not have the "_types" suffix, to make them less verbose and not hint > that they are limited to types. How about _mutex.h and _lock.h? M -- Mark Murray Warning: this .sig is umop ap!sdn To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Apr 23 19:31:22 2001 Delivered-To: freebsd-smp@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id C03EF37B424 for ; Mon, 23 Apr 2001 19:31:19 -0700 (PDT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (robert@fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.11.3/8.11.3) with SMTP id f3O2Vof08031 for ; Mon, 23 Apr 2001 22:31:51 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Mon, 23 Apr 2001 22:31:50 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: smp@FreeBSD.org Subject: sysctl's and mutexes Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Right now, there are effectively two types of data sysctl's: ones where sysctl understands (to some extent) the semantics of what is being handled/stored (int, string, fixed-size opaque), and ones where it doesn't (user-provided handler). As the kernel becomes increasingly fine-grained, many if not all of the values tweakable using sysctl will need to have some sort of synchronization primitive that protects their consistency and validity. Right now, to implement acquisition of appropriate synchronization for any writable (and many readable if it is desirable to read values set frequently within the kernel) sysctl's, it is necessary to write a handler for each and every sysctl, which essentially consists of a wrapper in the form of: mtx_lock(&this_sysctls_mutex); ... mtx_unlock(&this_sysctls_mutex); Perhaps in some cases an appropriate sxlock. This seems like a bit of a waste, in that it requires almost all sysctl's to move from being normal well-defined types to being handlers, etc. I've been considering what would be involved in allowing an additional flag/field that would indicating to the sysctl system that an appropriate mutex would need to be acquired when performing operations on the node in question. For example: mtx_t jail_set_hostname_allowed_mtx; int jail_set_hostname_allowed = 1; SYSCTL_INT(_jail, OID_AUTO, set_hostname_allowed, CTLFLAG_RW|CTLFLAG_MTX, &jail_set_hostname_allowed, &jail_set_hostname_allowed_mtx, 0, "Processes in jail can set their hostnames"); This is a poor example, but you get the picture. Such a change would allow us to retain the simple declaration of sysctl's that are effectively statically compiled variables being exported to userland, while adapting to the fine-grained kernel code. This came up in particular for the jailNG code, where each jail instantiated introduces a fair number of sysctl's, all of which are expected to be accessed while holding the per-jail mutex, which is known when the sysctl is dynamically added. I may be off-base here, but it's just a thought. :-) Robert N M Watson FreeBSD Core Team, TrustedBSD Project robert@fledge.watson.org NAI Labs, Safeport Network Services To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 24 3:37:40 2001 Delivered-To: freebsd-smp@freebsd.org Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by hub.freebsd.org (Postfix) with ESMTP id A03D937B422 for ; Tue, 24 Apr 2001 03:37:35 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from bde.zeta.org.au (bde.zeta.org.au [203.2.228.102]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id UAA21195; Tue, 24 Apr 2001 20:37:16 +1000 Date: Tue, 24 Apr 2001 20:34:27 +1000 (EST) From: Bruce Evans X-Sender: bde@besplex.bde.org To: Mark Murray Cc: smp@FreeBSD.ORG Subject: Re: Kernel include file cleanup take #3. In-Reply-To: <200104230602.f3N62Tw49163@gratis.grondar.za> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Mon, 23 Apr 2001, Mark Murray wrote: > > > http://people.freebsd.org/~markm/patches/sys.SYS_MUTEX.diff.1 > > You actually made a subinclude of > > other headers that need it, and one that doesn't ( doesn't > > need it directly, but it includes which does). > > OK. Removed it (plus some other junk) from eventhandler. From conf? > > Please change the copyright owner to yourself. FreeBSD, Inc. still > > doesn't exist in a form suitable for holding copyrights. > > OK. Done. For such small files I'd prefer them to be owned by the project, > but I don't care enough to be religious about it. Actually, the copyrights from the original files would be better. > > I'm not sure if the names for the new types headers are right. Perhaps > > they should have a leading underscore to inhibit direct inclusion, or > > not have the "_types" suffix, to make them less verbose and not hint > > that they are limited to types. > > How about _mutex.h and _lock.h? OK. The pfind() changes gave about another .c files that need but don't include it. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 24 4: 9:56 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gratis.grondar.za (grouter.grondar.za [196.7.18.65]) by hub.freebsd.org (Postfix) with ESMTP id 8F5E237B422 for ; Tue, 24 Apr 2001 04:09:49 -0700 (PDT) (envelope-from mark@grondar.za) Received: from grondar.za (gratis.grondar.za [196.7.18.133]) by gratis.grondar.za (8.11.3/8.11.3) with ESMTP id f3OB9Uw63269; Tue, 24 Apr 2001 13:09:32 +0200 (SAST) (envelope-from mark@grondar.za) Message-Id: <200104241109.f3OB9Uw63269@gratis.grondar.za> To: Bruce Evans Cc: smp@FreeBSD.ORG Subject: Re: Kernel include file cleanup take #3. References: In-Reply-To: ; from Bruce Evans "Tue, 24 Apr 2001 20:34:27 +1000." Date: Tue, 24 Apr 2001 13:11:12 +0200 From: Mark Murray Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > On Mon, 23 Apr 2001, Mark Murray wrote: > > > > > http://people.freebsd.org/~markm/patches/sys.SYS_MUTEX.diff.1 > > > > You actually made a subinclude of > > > other headers that need it, and one that doesn't ( doesn't > > > need it directly, but it includes which does). > > > > OK. Removed it (plus some other junk) from eventhandler. > > From conf? Eventhandler now contains all it needs, and conf.h includes eventhandler.h. conf.h no longer contains lock junk. > > OK. Done. For such small files I'd prefer them to be owned by the project, > > but I don't care enough to be religious about it. > > Actually, the copyrights from the original files would be better. That works for me. > > > I'm not sure if the names for the new types headers are right. Perhaps > > > they should have a leading underscore to inhibit direct inclusion, or > > > not have the "_types" suffix, to make them less verbose and not hint > > > that they are limited to types. > > > > How about _mutex.h and _lock.h? > > OK. Roger that. Done. > The pfind() changes gave about another .c files that need > but don't include it. Bleagh. I'll fix that. M -- Mark Murray Warning: this .sig is umop ap!sdn To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 24 10:11:47 2001 Delivered-To: freebsd-smp@freebsd.org Received: from meow.osd.bsdi.com (meow.osd.bsdi.com [204.216.28.88]) by hub.freebsd.org (Postfix) with ESMTP id 04CAA37B423; Tue, 24 Apr 2001 10:11:45 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by meow.osd.bsdi.com (8.11.2/8.11.2) with ESMTP id f3OHBZG67714; Tue, 24 Apr 2001 10:11:35 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: Date: Tue, 24 Apr 2001 10:10:56 -0700 (PDT) From: John Baldwin To: Robert Watson Subject: RE: sysctl's and mutexes Cc: smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 24-Apr-01 Robert Watson wrote: > > Right now, there are effectively two types of data sysctl's: ones where > sysctl understands (to some extent) the semantics of what is being > handled/stored (int, string, fixed-size opaque), and ones where it doesn't > (user-provided handler). As the kernel becomes increasingly fine-grained, > many if not all of the values tweakable using sysctl will need to have > some sort of synchronization primitive that protects their consistency and > validity. Right now, to implement acquisition of appropriate > synchronization for any writable (and many readable if it is desirable to > read values set frequently within the kernel) sysctl's, it is necessary to > write a handler for each and every sysctl, which essentially consists of a > wrapper in the form of: > > mtx_lock(&this_sysctls_mutex); > > ... > > mtx_unlock(&this_sysctls_mutex); > > Perhaps in some cases an appropriate sxlock. This seems like a bit of a > waste, in that it requires almost all sysctl's to move from being normal > well-defined types to being handlers, etc. I've been considering what > would be involved in allowing an additional flag/field that would > indicating to the sysctl system that an appropriate mutex would need to be > acquired when performing operations on the node in question. For example: > > mtx_t jail_set_hostname_allowed_mtx; s/mtx_t/struct mtx/ :-P > int jail_set_hostname_allowed = 1; > SYSCTL_INT(_jail, OID_AUTO, set_hostname_allowed, CTLFLAG_RW|CTLFLAG_MTX, > &jail_set_hostname_allowed, &jail_set_hostname_allowed_mtx, 0, > "Processes in jail can set their hostnames"); It might very well be desirable to add a new parameter for each sysctl that is a pointer to a mutex. However, an appropriate SYSINIT() or some such during startup needs to initialize that mutex before that sysctl is used. If need be, we can also add a flag to determine if the sysctl should initialize the mutex itself or not. Ideally then, we would create a new SYSINIT on the fly to initialize the mutex in question. I think though, that requiring explicit initialization of each mutex shouldn't be too hard. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 24 10:25:42 2001 Delivered-To: freebsd-smp@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id 3CE2637B423; Tue, 24 Apr 2001 10:25:36 -0700 (PDT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (robert@fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.11.3/8.11.3) with SMTP id f3OHQ5f18377; Tue, 24 Apr 2001 13:26:06 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Tue, 24 Apr 2001 13:26:05 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: John Baldwin Cc: smp@FreeBSD.org Subject: RE: sysctl's and mutexes In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tue, 24 Apr 2001, John Baldwin wrote: > > int jail_set_hostname_allowed = 1; > > SYSCTL_INT(_jail, OID_AUTO, set_hostname_allowed, CTLFLAG_RW|CTLFLAG_MTX, > > &jail_set_hostname_allowed, &jail_set_hostname_allowed_mtx, 0, > > "Processes in jail can set their hostnames"); > > It might very well be desirable to add a new parameter for each sysctl > that is a pointer to a mutex. However, an appropriate SYSINIT() or some > such during startup needs to initialize that mutex before that sysctl is > used. If need be, we can also add a flag to determine if the sysctl > should initialize the mutex itself or not. Ideally then, we would > create a new SYSINIT on the fly to initialize the mutex in question. I > think though, that requiring explicit initialization of each mutex > shouldn't be too hard. I don't have strong feelings on the exact nature of how mutices are bound to sysctl's, as long as it's possible to do the following: 1) It should be possible to use an existing initialized mutex, especially with regards to dynamically allocated sysctl's. 2) It should be possible for several sysctl's to share the same mutex. Presumably if the SYSCTL code is initializing the mutex, there are questions about the relative orderings of initialization of the mutex and first use of the protected variable in kernel: the code in question needs to know if SYSCTL stuff runs first so it can determine if the mutex should be used to protect access to the variable or not... (and all the other general boot-time ordering issues). Robert N M Watson FreeBSD Core Team, TrustedBSD Project robert@fledge.watson.org NAI Labs, Safeport Network Services To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 24 10:46:54 2001 Delivered-To: freebsd-smp@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id BAEDA37B423; Tue, 24 Apr 2001 10:46:47 -0700 (PDT) (envelope-from arr@watson.org) Received: from localhost (arr@localhost) by fledge.watson.org (8.11.3/8.11.3) with SMTP id f3OHlIw18732; Tue, 24 Apr 2001 13:47:18 -0400 (EDT) (envelope-from arr@watson.org) Date: Tue, 24 Apr 2001 13:47:17 -0400 (EDT) From: "Andrew R. Reiter" To: Robert Watson Cc: John Baldwin , smp@FreeBSD.ORG Subject: RE: sysctl's and mutexes In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tue, 24 Apr 2001, Robert Watson wrote: > On Tue, 24 Apr 2001, John Baldwin wrote: > > > > int jail_set_hostname_allowed = 1; > > > SYSCTL_INT(_jail, OID_AUTO, set_hostname_allowed, CTLFLAG_RW|CTLFLAG_MTX, > > > &jail_set_hostname_allowed, &jail_set_hostname_allowed_mtx, 0, > > > "Processes in jail can set their hostnames"); > > > > It might very well be desirable to add a new parameter for each sysctl > > that is a pointer to a mutex. However, an appropriate SYSINIT() or some > > such during startup needs to initialize that mutex before that sysctl is > > used. If need be, we can also add a flag to determine if the sysctl > > should initialize the mutex itself or not. Ideally then, we would > > create a new SYSINIT on the fly to initialize the mutex in question. I > > think though, that requiring explicit initialization of each mutex > > shouldn't be too hard. > > I don't have strong feelings on the exact nature of how mutices are bound > to sysctl's, as long as it's possible to do the following: > > 1) It should be possible to use an existing initialized mutex, especially > with regards to dynamically allocated sysctl's. > > 2) It should be possible for several sysctl's to share the same mutex. As in being able to say that for (and this might be a bad example) kern.timecounter.* mibs, could all share a mutex which is really "bound" to kern.timecounter in genera. Or do you mean just more generically the idea that multiple sysctl's can share a mutex and who/what shares a mutex is something to be decided? > > Robert N M Watson FreeBSD Core Team, TrustedBSD Project > robert@fledge.watson.org NAI Labs, Safeport Network Services > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message > *-------------................................................. | Andrew R. Reiter | arr@fledge.watson.org | "It requires a very unusual mind | to undertake the analysis of the obvious" -- A.N. Whitehead To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 24 10:58:50 2001 Delivered-To: freebsd-smp@freebsd.org Received: from mx.databus.com (p101-44.acedsl.com [160.79.101.44]) by hub.freebsd.org (Postfix) with ESMTP id B6F1F37B423; Tue, 24 Apr 2001 10:58:47 -0700 (PDT) (envelope-from barney@mx.databus.com) Received: (from barney@localhost) by mx.databus.com (8.11.3/8.11.3) id f3OHwjJ10357; Tue, 24 Apr 2001 13:58:45 -0400 (EDT) (envelope-from barney) Date: Tue, 24 Apr 2001 13:58:45 -0400 From: Barney Wolff To: "Andrew R. Reiter" Cc: Robert Watson , John Baldwin , smp@FreeBSD.ORG Subject: Re: sysctl's and mutexes Message-ID: <20010424135845.A10320@mx.databus.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: ; from arr@watson.org on Tue, Apr 24, 2001 at 01:47:17PM -0400 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Pardon an outsider's question, but what exactly are these mutex's supposed to protect? Would a reader of a sysctl value have to acquire a read lock in order to read a non-atomic value? Is the rate at which these values are set and/or read so high as to justify more than a single mutex for the lot? Are there any operations that take long enough that anything other than a spinlock is justified? Sorry if these are dumb questions - I'm just a KISS sort of guy. Barney Wolff On Tue, Apr 24, 2001 at 01:47:17PM -0400, Andrew R. Reiter wrote: > > As in being able to say that for (and this might be a bad example) > kern.timecounter.* mibs, could all share a mutex which is really "bound" > to kern.timecounter in genera. Or do you mean just more generically the > idea that multiple sysctl's can share a mutex and who/what shares a mutex > is something to be decided? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 24 11:45:59 2001 Delivered-To: freebsd-smp@freebsd.org Received: from meow.osd.bsdi.com (meow.osd.bsdi.com [204.216.28.88]) by hub.freebsd.org (Postfix) with ESMTP id DD88437B43C; Tue, 24 Apr 2001 11:45:51 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by meow.osd.bsdi.com (8.11.2/8.11.2) with ESMTP id f3OIjjG70725; Tue, 24 Apr 2001 11:45:45 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: Date: Tue, 24 Apr 2001 11:45:07 -0700 (PDT) From: John Baldwin To: Robert Watson Subject: RE: sysctl's and mutexes Cc: smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 24-Apr-01 Robert Watson wrote: > On Tue, 24 Apr 2001, John Baldwin wrote: > >> > int jail_set_hostname_allowed = 1; >> > SYSCTL_INT(_jail, OID_AUTO, set_hostname_allowed, CTLFLAG_RW|CTLFLAG_MTX, >> > &jail_set_hostname_allowed, &jail_set_hostname_allowed_mtx, 0, >> > "Processes in jail can set their hostnames"); >> >> It might very well be desirable to add a new parameter for each sysctl >> that is a pointer to a mutex. However, an appropriate SYSINIT() or some >> such during startup needs to initialize that mutex before that sysctl is >> used. If need be, we can also add a flag to determine if the sysctl >> should initialize the mutex itself or not. Ideally then, we would >> create a new SYSINIT on the fly to initialize the mutex in question. I >> think though, that requiring explicit initialization of each mutex >> shouldn't be too hard. > > I don't have strong feelings on the exact nature of how mutices are bound > to sysctl's, as long as it's possible to do the following: > > 1) It should be possible to use an existing initialized mutex, especially > with regards to dynamically allocated sysctl's. > > 2) It should be possible for several sysctl's to share the same mutex. Exactly. > Presumably if the SYSCTL code is initializing the mutex, there are > questions about the relative orderings of initialization of the mutex and > first use of the protected variable in kernel: the code in question needs > to know if SYSCTL stuff runs first so it can determine if the mutex should > be used to protect access to the variable or not... (and all the other > general boot-time ordering issues). Right, which is why I think requriring explicit initialization will probably prove to be saner in the long run for most sysctl's. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 24 11:46:11 2001 Delivered-To: freebsd-smp@freebsd.org Received: from meow.osd.bsdi.com (meow.osd.bsdi.com [204.216.28.88]) by hub.freebsd.org (Postfix) with ESMTP id 480E737B423; Tue, 24 Apr 2001 11:46:07 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by meow.osd.bsdi.com (8.11.2/8.11.2) with ESMTP id f3OIjkG70729; Tue, 24 Apr 2001 11:45:50 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20010424135845.A10320@mx.databus.com> Date: Tue, 24 Apr 2001 11:45:07 -0700 (PDT) From: John Baldwin To: Barney Wolff Subject: Re: sysctl's and mutexes Cc: smp@FreeBSD.org, Robert Watson , "Andrew R. Reiter" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 24-Apr-01 Barney Wolff wrote: > Pardon an outsider's question, but what exactly are these mutex's > supposed to protect? Would a reader of a sysctl value have to > acquire a read lock in order to read a non-atomic value? > > Is the rate at which these values are set and/or read so high > as to justify more than a single mutex for the lot? Are there > any operations that take long enough that anything other than > a spinlock is justified? > > Sorry if these are dumb questions - I'm just a KISS sort of guy. If a user is using a sysctl to read/write a variable that we protect with a given lock inside the kernel, we need to use that same lock to protect the data in the sysctl handler. Granted, for some read-only sysctl's, it could be perfectly fine to not grab a lock while performing the sysctl. The trick though is that you always need to use the same locks to protect a given piece of data. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Apr 25 13:59:59 2001 Delivered-To: freebsd-smp@freebsd.org Received: from meow.osd.bsdi.com (meow.osd.bsdi.com [204.216.28.88]) by hub.freebsd.org (Postfix) with ESMTP id 1063637B423; Wed, 25 Apr 2001 13:59:47 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by meow.osd.bsdi.com (8.11.2/8.11.2) with ESMTP id f3PKxZG15010; Wed, 25 Apr 2001 13:59:35 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 Date: Wed, 25 Apr 2001 13:58:55 -0700 (PDT) From: John Baldwin To: smp@FreeBSD.org Subject: SMP Patch Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org [ bcc'd to -hackers, followups to -smp please ] The last part of the alpha SMP support has actually grown into a rather large patch that does several things: - It splits the per-process portions of hardclock() and statclock() off into hardclock_process() and statclock_process() respectively. hardclock() and statclock() call the *_process() functions for the current process so that UP systems will run as before. For SMP systems, it is simply necessary to ensure that all other processors execute the *_process() functions when the main clock functions are triggered on one CPU by an interrupt. For the alpha 4100, clock interrupts are delievered in a staggered broadcast fashion, so we simply call hardclock/statclock on the boot CPU and call the *_process() functions on the secondaries. For x86, we call statclock and hardclock as usual and then call forward_hardclock/statclock in the MD code to send an IPI to cause the AP's to execute forwared_hardclock/statclock which then call the *_process() functions. - forward_signal() and forward_roundrobin() have been reworked to be MI and to involve less hackery. Now the cpu doing the forward sets any flags, etc. and sends a very simple IPI_AST to the other cpu(s). AST IPIs now just basically return so that they can execute ast() and don't bother with setting the astpending or needresched flags themselves. This also removes the loop in forward_signal() as sched_lock closes the race condition that the loop worked around. - need_resched(), resched_wanted() and clear_resched() have been changed to take a process to act on rather than assuming curproc so that they can be used to implement forward_roundrobin() as described above. - Various other SMP variables have been moved to a MI subr_smp.c and a new header sys/smp.h declares MI SMP variables and API's. The IPI API's from machine/ipl.h have moved to machine/smp.h which is included by sys/smp.h. - The globaldata_register() and globaldata_find() functions as well as the SLIST of globaldata structures has become MI and moved into subr_smp.c. Also, the globaldata list is only available if SMP support is compiled in. The patch can be found at http://www.FreeBSD.org/~jhb/patches/smp.patch Please review. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 27 4:26:22 2001 Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id 5A81E37B422; Fri, 27 Apr 2001 04:26:20 -0700 (PDT) (envelope-from bright@fw.wintelcom.net) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id f3RBQKt23581; Fri, 27 Apr 2001 04:26:20 -0700 (PDT) Date: Fri, 27 Apr 2001 04:26:19 -0700 From: Alfred Perlstein To: smp@freebsd.org Cc: dillon@freebsd.org, jhb@freebsd.org Subject: that vm diff now makes it into single user mode. Message-ID: <20010427042619.W18676@fw.wintelcom.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i X-all-your-base: are belong to us. Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org If you're booting diskless you get to mounting/using the md0 disk and you panic because I haven't made ufs_readwrite safe. http://people.freebsd.org/~alfred/vm.diff I'm having fun but a bit concerned, I have a lot of places where locks are "optional", meaning that callers can have a lock or not mostly depending on if the function being called will block or not. Getting into UFS is starting to get somewhat hairy as there seems like there's going to be a lock of lock/unlock or places where the lock is held for a long period. Anyhow, if you want to check it out, comment on what's going on with it, let me know. If you'd like to take a subsystem and work out how to vm safe it let me know. Just let me know, ok? :) -- -Alfred Perlstein - [alfred@freebsd.org] Instead of asking why a piece of software is using "1970s technology," start asking why software is ignoring 30 years of accumulated wisdom. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 27 8:21:18 2001 Delivered-To: freebsd-smp@freebsd.org Received: from munin.odin-corporation.com (munin.odin-corporation.com [216.233.173.18]) by hub.freebsd.org (Postfix) with ESMTP id B4E3037B424 for ; Fri, 27 Apr 2001 08:21:06 -0700 (PDT) (envelope-from lars@odin-corporation.com) Received: from odin-corporation.com (localhost [127.0.0.1]) by munin.odin-corporation.com (8.11.1/8.11.1) with ESMTP id f3RFKtf98975 for ; Fri, 27 Apr 2001 10:20:55 -0500 (CDT) (envelope-from lars@odin-corporation.com) Message-ID: <3AE98E56.1E582376@odin-corporation.com> Date: Fri, 27 Apr 2001 10:20:54 -0500 From: Lars Fredriksenndow_rect Organization: Odin Corporation X-Mailer: Mozilla 4.76 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: no, en MIME-Version: 1.0 To: smp@freebsd.org Subject: interrupt problems with one CPU SMP system?? Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, Due to a bad processor, I have a SMP kernel running with one processor on a p3b-ds MB. This works just fine for a while, but eventually ahc0 and tx0 will report that interupts aren't working. Keyboard seems to work fine. The ahc0 problem may be related to the fact that ahc0 ends up routed to irq2 , tx0 is on irq5. Anyone run into this before? Lars To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 27 10:57:21 2001 Delivered-To: freebsd-smp@freebsd.org Received: from earth.backplane.com (earth-nat-cw.backplane.com [208.161.114.67]) by hub.freebsd.org (Postfix) with ESMTP id 1CE7837B422; Fri, 27 Apr 2001 10:57:19 -0700 (PDT) (envelope-from dillon@earth.backplane.com) Received: (from dillon@localhost) by earth.backplane.com (8.11.2/8.11.2) id f3RHvDW09866; Fri, 27 Apr 2001 10:57:13 -0700 (PDT) (envelope-from dillon) Date: Fri, 27 Apr 2001 10:57:13 -0700 (PDT) From: Matt Dillon Message-Id: <200104271757.f3RHvDW09866@earth.backplane.com> To: Alfred Perlstein Cc: smp@FreeBSD.ORG, jhb@FreeBSD.ORG Subject: Re: that vm diff now makes it into single user mode. References: <20010427042619.W18676@fw.wintelcom.net> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :If you're booting diskless you get to mounting/using the md0 disk :and you panic because I haven't made ufs_readwrite safe. : :http://people.freebsd.org/~alfred/vm.diff : :I'm having fun but a bit concerned, I have a lot of places where :locks are "optional", meaning that callers can have a lock or not :mostly depending on if the function being called will block or not. : :Getting into UFS is starting to get somewhat hairy as there seems :like there's going to be a lock of lock/unlock or places where the :lock is held for a long period. : :Anyhow, if you want to check it out, comment on what's going on :with it, let me know. If you'd like to take a subsystem and work :out how to vm safe it let me know. Just let me know, ok? :) : :-- :-Alfred Perlstein - [alfred@freebsd.org] There are about fifty thousand places where the VM system assumes a contextual lock. Basically what it comes down to is: * VM Pages cannot be busied from an interrupt, but can be unbusied * VM Objects cannot be manipulated from an interrupt * Mainline kernel code assumes type-stability for VM pages (easy) * Mainline kernel code assumes that if it looks a VM page up and the page is not busy, that it can busy it without any locking (since interrupts cannot busy new pages, only unbusy them). * Mainline kernel code assumes that it can manipulate VM objects without any locking if it does not block. The only place where we have 'real' locking in regards to supporting SMP under -stable are VM maps. i.e. the vm_map_lock() stuff existed in -stable originally to help out with kmap locking (since interrupts could manipulate kmap's), and was extended to support SMP in -stable due to the fact that VM maps could be shared amoungst rforked processes. Unfortunately, there is no yardstick to determine where you need VM locking -- the splvm() stuff in stable is only there for interrupt interlocks, not for process<->process interlocks, so you can't simply put a mutex in where splvm() was before. Under -current you are going to have to wind up using what will effectively be a giant lock for the VM system. I recommend starting with that and making sure you have a stable -current system, then you can start moving the locks inward. I do not recommend trying to immediately implement a mutex on major VM structures (like VM objects)... you are almost guarenteed to create dozens of deadlock situations. Make it work first, then start implementing the finer-grained locks. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 27 11: 2: 5 2001 Delivered-To: freebsd-smp@freebsd.org Received: from smtp10.phx.gblx.net (smtp10.phx.gblx.net [206.165.6.140]) by hub.freebsd.org (Postfix) with ESMTP id A2CB937B423; Fri, 27 Apr 2001 11:02:02 -0700 (PDT) (envelope-from tlambert@usr07.primenet.com) Received: (from daemon@localhost) by smtp10.phx.gblx.net (8.9.3/8.9.3) id LAA47280; Fri, 27 Apr 2001 11:02:01 -0700 Received: from usr07.primenet.com(206.165.6.207) via SMTP by smtp10.phx.gblx.net, id smtpdfX8Taa; Fri Apr 27 11:01:54 2001 Received: (from tlambert@localhost) by usr07.primenet.com (8.8.5/8.8.5) id LAA02107; Fri, 27 Apr 2001 11:03:09 -0700 (MST) From: Terry Lambert Message-Id: <200104271803.LAA02107@usr07.primenet.com> Subject: Re: that vm diff now makes it into single user mode. To: bright@wintelcom.net (Alfred Perlstein) Date: Fri, 27 Apr 2001 18:03:08 +0000 (GMT) Cc: smp@FreeBSD.ORG, dillon@FreeBSD.ORG, jhb@FreeBSD.ORG In-Reply-To: <20010427042619.W18676@fw.wintelcom.net> from "Alfred Perlstein" at Apr 27, 2001 04:26:19 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > Getting into UFS is starting to get somewhat hairy as there seems > like there's going to be a lock of lock/unlock or places where the > lock is held for a long period. FWIW: The correact approach to this is to lock the vnode, and treat everything lower than that as "opaque". If you can lock the vnode, you can make any call you want on it. Part of the problem here os some of the earlier "clean up" work that damaged the VFS interface. Really, the locks need to be asserted in the VOP_* macro, prior to calldown, and the locks need to be reentrant for the time being, for the same VFS. The easiest way to do this is to realize that there is no tail call optimization in the VFS code, _anywhere_, and just say "VOP_* calls from outside the VFS must lock the vnode, and VOP_* calls from withing the VFS must not, since the vnode will already be locked, UNLESS it is an internally acquired vnode that didn't come down from outside the VFS code". A gross approximation of this would be to put locks in all the VOP_ calls, and then pass a "do_lock" flag to all of them which is always "1" for code above the VFS layer (e.g. /sys/kerne/vfs_*.c), and _only occasionally_ "1" for calls from within the VFS layer (mostly "rename", "lookup", "delete", and other directory entry manipulation code in UFS, proper). You will need to "punt" on multithreading each VFS itself, until some later date (the only thing you will lose when doing this is the ability go into the same directory simultaneously on more than one processor). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 27 11:24:53 2001 Delivered-To: freebsd-smp@freebsd.org Received: from earth.backplane.com (earth-nat-cw.backplane.com [208.161.114.67]) by hub.freebsd.org (Postfix) with ESMTP id 1833B37B423; Fri, 27 Apr 2001 11:24:49 -0700 (PDT) (envelope-from dillon@earth.backplane.com) Received: (from dillon@localhost) by earth.backplane.com (8.11.2/8.11.2) id f3RIOlW10289; Fri, 27 Apr 2001 11:24:47 -0700 (PDT) (envelope-from dillon) Date: Fri, 27 Apr 2001 11:24:47 -0700 (PDT) From: Matt Dillon Message-Id: <200104271824.f3RIOlW10289@earth.backplane.com> To: Terry Lambert Cc: bright@wintelcom.net (Alfred Perlstein), smp@FreeBSD.ORG, jhb@FreeBSD.ORG Subject: Re: that vm diff now makes it into single user mode. References: <200104271803.LAA02107@usr07.primenet.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :FWIW: : :The correact approach to this is to lock the vnode, and treat :everything lower than that as "opaque". If you can lock the :vnode, you can make any call you want on it. : :Part of the problem here os some of the earlier "clean up" :work that damaged the VFS interface. : :Really, the locks need to be asserted in the VOP_* macro, :prior to calldown, and the locks need to be reentrant for :the time being, for the same VFS. : : Terry Lambert : terry@lambert.org Eeek! This isn't possible with our code base, Terry. What we really need to do is separate out read/write/truncate operations and make them functions of the VM object rather then the vnode. direct API direct API | | V V vnode -+--> VM object ---> filesystem (read/write/truncate) | +-----------------> filesystem (other ops) The problem we face now is that the VM system mostly bypasses the VNode... it bypasses the vnode entirely for anything that does not require actual I/O, so you have a lock reversal situation where VOP operations lock the vnode and then might lock the VM object, and VM operations (would have to) lock the VM object and then lock the VNode. When you add vm_map's and the buffer cache into the fray it gets even worse. The solution is to funnel all VMIO operations: read, write, and truncate/extend, through the VM object always. Now a VNODE op can lock the vnode and, if a read/write/truncate, then lock the VM object. A direct VMIO operation can bypass the vnode and lock the VM object directly (i.e. never need to lock the vnode). This allows us to separate meta filesystem operations from reads, writes, and truncate/extend operations, which in turn allows us to optimize reads, writes, and truncate/extend operations through the VM Object (which is well suited for such optimizations) rather then having to build such optimizations into the VNode (which is not well suited for such operations). This also allows the VM Object to serve as the cache for all I/O operations rather then having to go deep into the filesystem code and then access the VM Page cache indirectly through the buffer cache. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 27 12:47:52 2001 Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id 668E937B629; Fri, 27 Apr 2001 12:47:44 -0700 (PDT) (envelope-from bright@fw.wintelcom.net) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id f3RJlil06287; Fri, 27 Apr 2001 12:47:44 -0700 (PDT) Date: Fri, 27 Apr 2001 12:47:43 -0700 From: Alfred Perlstein To: Matt Dillon Cc: smp@FreeBSD.ORG, jhb@FreeBSD.ORG Subject: Re: that vm diff now makes it into single user mode. Message-ID: <20010427124743.E18676@fw.wintelcom.net> References: <20010427042619.W18676@fw.wintelcom.net> <200104271757.f3RHvDW09866@earth.backplane.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200104271757.f3RHvDW09866@earth.backplane.com>; from dillon@earth.backplane.com on Fri, Apr 27, 2001 at 10:57:13AM -0700 X-all-your-base: are belong to us. Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org * Matt Dillon [010427 10:57] wrote: > > There are about fifty thousand places where the VM system assumes a > contextual lock. Basically what it comes down to is: > > * VM Pages cannot be busied from an interrupt, but can be unbusied > > * VM Objects cannot be manipulated from an interrupt > > * Mainline kernel code assumes type-stability for VM pages (easy) > > * Mainline kernel code assumes that if it looks a VM page up and the > page is not busy, that it can busy it without any locking (since > interrupts cannot busy new pages, only unbusy them). > > * Mainline kernel code assumes that it can manipulate VM objects without > any locking if it does not block. > > The only place where we have 'real' locking in regards to supporting SMP > under -stable are VM maps. i.e. the vm_map_lock() stuff existed in > -stable originally to help out with kmap locking (since interrupts could > manipulate kmap's), and was extended to support SMP in -stable due to > the fact that VM maps could be shared amoungst rforked processes. > > Unfortunately, there is no yardstick to determine where you need VM > locking -- the splvm() stuff in stable is only there for interrupt > interlocks, not for process<->process interlocks, so you can't simply > put a mutex in where splvm() was before. > > Under -current you are going to have to wind up using what will > effectively be a giant lock for the VM system. I recommend starting > with that and making sure you have a stable -current system, then > you can start moving the locks inward. I do not recommend trying to > immediately implement a mutex on major VM structures (like VM objects)... > you are almost guarenteed to create dozens of deadlock situations. Make > it work first, then start implementing the finer-grained locks. Yes, this is what I was going for, the points you made above really clarify a lot of things, I had the general idea that this is how things worked but wasn't sure about several more points, perhaps you have the time to answer: When do vm related changes propogate to struct buf? Can is happen at interrupt time? Is it done when asking for access/lock to a buf? (that would be ideal) Or will I need to do locking under vfs_bio to make sure that b->b_flags and b->b_pages are in sync wrt to vm? I am going for that giant vm lock. I've also removed the atomic ops from vm_object and vm_page flags/refcounting. My guess is that this may gain us some performance, but I'll have to see, right now anything is better than giant and so far some cool stuff runs without the Giant lock: obreak getpagesize sbrk sstk mmap ovadvise munmap mprotect madvise mincore mmap mlock munlock minherit msync mlockall munlockall and.. vm_fault Using a combination of your and Doug Ambrisko's diskless stuff as well a dusty old p100 I've got a box here that boots in under 20 seconds, really makes the compile/test cycle amazingly tolerable. -- -Alfred Perlstein - [alfred@freebsd.org] http://www.egr.unlv.edu/~slumos/on-netbsd.html To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 27 13:30:35 2001 Delivered-To: freebsd-smp@freebsd.org Received: from gratis.grondar.za (grouter.grondar.za [196.7.18.65]) by hub.freebsd.org (Postfix) with ESMTP id 62F4E37B422 for ; Fri, 27 Apr 2001 13:30:29 -0700 (PDT) (envelope-from mark@grondar.za) Received: from grondar.za (gratis.grondar.za [196.7.18.133]) by gratis.grondar.za (8.11.3/8.11.3) with ESMTP id f3RKULp03621 for ; Fri, 27 Apr 2001 22:30:24 +0200 (SAST) (envelope-from mark@grondar.za) Message-Id: <200104272030.f3RKULp03621@gratis.grondar.za> To: smp@freebsd.org Subject: SMPng header cleanups commit candidate 1 Date: Fri, 27 Apr 2001 22:32:00 +0200 From: Mark Murray Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi With feedback from BDE, I have got the SMPng headers to a state where they may be close to commit-ready. Please review: http://people.freebsd.org/~markm/patches/sys.SYS_MUTEX.diff.2 The primary aim is to disentangle the sys/lock.h and sys/mutex.h spaghetti. Other aims are further de-spaghetti lockmgr; sort headers where practical. Thanks! M -- Mark Murray Warning: this .sig is umop ap!sdn To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Apr 28 20:32:52 2001 Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id 04A6F37B423; Sat, 28 Apr 2001 20:32:51 -0700 (PDT) (envelope-from bright@fw.wintelcom.net) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id f3T3Wl722285; Sat, 28 Apr 2001 20:32:47 -0700 (PDT) Date: Sat, 28 Apr 2001 20:32:47 -0700 From: Alfred Perlstein To: smp@FreeBSD.ORG Cc: dillon@FreeBSD.ORG, jhb@FreeBSD.ORG Subject: Re: that vm diff now makes it into single user mode. Message-ID: <20010428203247.A18676@fw.wintelcom.net> References: <20010427042619.W18676@fw.wintelcom.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20010427042619.W18676@fw.wintelcom.net>; from bright@wintelcom.net on Fri, Apr 27, 2001 at 04:26:19AM -0700 X-all-your-base: are belong to us. Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org * Alfred Perlstein [010427 04:26] wrote: > If you're booting diskless you get to mounting/using the md0 disk > and you panic because I haven't made ufs_readwrite safe. > > http://people.freebsd.org/~alfred/vm.diff I'm now building world on my p100 over NFS with the latest version of this patch. I'm pretty sure UFS isn't going to work all that well but the changes required to fix it should be minor. Review (of code and locking methods, but not style) and fixes would be appreciated. -- -Alfred Perlstein - [alfred@freebsd.org] Instead of asking why a piece of software is using "1970s technology," start asking why software is ignoring 30 years of accumulated wisdom. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Apr 28 23:10:42 2001 Delivered-To: freebsd-smp@freebsd.org Received: from cr66388-a.rchrd1.on.wave.home.com (cr66388-a.rchrd1.on.wave.home.com [24.114.165.24]) by hub.freebsd.org (Postfix) with ESMTP id CDFEE37B424; Sat, 28 Apr 2001 23:10:29 -0700 (PDT) (envelope-from jburkholder0829@home.com) Received: from k7.rchrd1.on.wave.home.com (k7.rchrd1.on.wave.home.com [10.0.0.253]) by cr66388-a.rchrd1.on.wave.home.com (Postfix) with ESMTP id 33925334; Sun, 29 Apr 2001 02:10:04 -0400 (EDT) Received: from k7.rchrd1.on.wave.home.com (localhost [127.0.0.1]) by k7.rchrd1.on.wave.home.com (Postfix) with ESMTP id 6B821223A; Sun, 29 Apr 2001 02:13:03 -0400 (EDT) X-Mailer: exmh version 2.3.1 01/18/2001 with nmh-1.0.4 To: Alfred Perlstein Cc: smp@FreeBSD.ORG, dillon@FreeBSD.ORG, jhb@FreeBSD.ORG Subject: Re: that vm diff now makes it into single user mode. In-Reply-To: Message from Alfred Perlstein of "Sat, 28 Apr 2001 20:32:47 PDT." <20010428203247.A18676@fw.wintelcom.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 29 Apr 2001 02:13:03 -0400 From: Jake Burkholder Message-Id: <20010429061303.6B821223A@k7.rchrd1.on.wave.home.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > * Alfred Perlstein [010427 04:26] wrote: > > If you're booting diskless you get to mounting/using the md0 disk > > and you panic because I haven't made ufs_readwrite safe. > > > > http://people.freebsd.org/~alfred/vm.diff > > I'm now building world on my p100 over NFS with the latest version > of this patch. I'm pretty sure UFS isn't going to work all that > well but the changes required to fix it should be minor. > > Review (of code and locking methods, but not style) and fixes would > be appreciated. Ok. I've just looked at what you've changed by following the patch. I started to point out what I thought were bugs, but I guess its just stuff you haven't got to yet (sendfile). i386/i386/vm_machdep.c: the mtx_trylock in vm_page_zero_idle is unnecessary, the lock is already held. This whole thing needs to be non-blocking if its going to be called from the idle loop, but I'm not sure that that's still possible. Its currently commented out. kern/sysv_shm.c: shm_deallocate_segment calls vm_object_deallocate and is called outside of vm_mtx. Does vm_object_deallocate need vm_mtx? vfs_bio.c: I don't see why vfs_vmio_release needs to unlock the vm_mtx, wakeup doesn't reschedule and brelvp doesn't sleep. I'd rather it not release the lock if it doesn't have to. Should the second mtx_lock after vm_object_pip_wakeupn be an unlock? Probably a typo. vm_mmap.c: munmap will return with the vm_mtx held if the protection check fails. vm_pageout.c: vm_daemon() needs an mtx_lock(vm_mtxp) at the bottom of the infinite loop. The following functions have a comment about needing the vm_mtx, but don't have the corresponding assertions: MA_OWNED: vfs_page_set_valid, swp_pager_meta_build, swapout_procs MA_NOTOWNED: brelse, vfs_unbusy_pages, vfs_busy_pages You might consider this style, but I'm not crazy about this kind of thing: gotlock = mtx_owned(vm_mtxp); if (!gotlock) mtx_lock(vm_mtxp); ... if (!gotlock) mtx_unlock(vm_mtxp); I'd rather you just recurse on vm_mtx and make sure that the function doesn't change the recursion level overall. I've been screwed more then once by this same thing with the old got_mplock flag in i386/i386/trap.c. About page faults not needing Giant, have you tried not acquiring Giant in trap where the page faults come in? I've pushed Giant down far enough in trap() for this to happe for i386 at least, I think Doug's done the same for alpha. Jake To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message