From owner-freebsd-arch@FreeBSD.ORG Fri Jan 15 01:51:05 2010 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8ECB1065670; Fri, 15 Jan 2010 01:51:05 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail03.syd.optusnet.com.au (mail03.syd.optusnet.com.au [211.29.132.184]) by mx1.freebsd.org (Postfix) with ESMTP id 271048FC13; Fri, 15 Jan 2010 01:51:04 +0000 (UTC) Received: from c220-239-227-214.carlnfd1.nsw.optusnet.com.au (c220-239-227-214.carlnfd1.nsw.optusnet.com.au [220.239.227.214]) by mail03.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id o0F1otJu012208 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 15 Jan 2010 12:50:58 +1100 Date: Fri, 15 Jan 2010 12:50:55 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: "M. Warner Losh" In-Reply-To: <20100114.135930.80200584442733547.imp@bsdimp.com> Message-ID: <20100115114822.O63406@delplex.bde.org> References: <20100114.105622.457034909117828677.imp@bsdimp.com> <4B4F7810.2080003@FreeBSD.org> <86625798-F339-4863-8F97-63B5232A6CF7@freebsd.org> <20100114.135930.80200584442733547.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: dougb@FreeBSD.org, svn-src-all@FreeBSD.org, src-committers@FreeBSD.org, rwatson@FreeBSD.org, freebsd-arch@FreeBSD.org, svn-src-head@FreeBSD.org Subject: Re: INCLUDE_CONFIG_FILE in GENERIC X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2010 01:51:05 -0000 On Thu, 14 Jan 2010, M. Warner Losh wrote: > In message: <86625798-F339-4863-8F97-63B5232A6CF7@freebsd.org> > "Robert N. M. Watson" writes: > : I agree. I see two kinds of users: > : > : - Desktop/server users who want their system to work without any > : special tuning or magic, and likely feel the comments they put in > : configuration files are important > : - Users who care a about a few hundred bytes of comments due to the > : nature of their special embedded hardware target environment "Stripping" comments has nothing to do with saving space. It is because it is technically difficult, and not implemented, to not strip them, except in the old limited code that just preserves the unprocessed top-level-only config file. > : As such, I see a reasonable "default" -- i.e., i386/amd64 GENERIC -- > : be to fully preserve the configuration and its comments. For the Your code to implement this is welcome :-). Even GENERIC is not quite complete, since it is missing the implicit include of DEFAULTS. OTOH, completing the config by merging with DEFAULTS, as I think the processed version must do, may give an unusable config by repeating things in DEFAULTS. The merge should probably have no processing at all, with include files concatenated and not replacing include directives, and DEFAULTS not included. > I think you are confusing two different things. > > The embedded folks omit this entirely. That's not the issue at all > here. Space is not the concern. We're putting it into GENERIC. > > Why are there two ways? Because there are two points in the process > one can squirt the data out of config. Let me explain (sorry for the > length, but it is necessary to clear up the confusion): > > The first way is "preserve the FILE the user used to create the > kernel" on the assumption that the FILE is what is important. No, it is on the non-assumption that the include directive didn't exist when the INCLUDE_CONFIG_FILE option was implemented. > This > will preserve the comments, but assumes that every single included > file from that file is recoverable in a trivial manner (eg, from cvs, > svn, p4, release ISOs, etc). This assumption is false, so this model became just broken when the include directive was implemented. History: INCLUDE_CONFIG_FILE option: 1996 include directive: 2001 processed output and -C: 2007 The -C option just preserves the breakage at its 2001-2007 level. > This model works well for the casual > user. Casual users presumably didn't notice the problem (until comments were lost in 2007) since they use GENERIC as is or with minor editing, and GENERIC doesn't use the include directive. Non-casual users don't notice since they know what is in their config files and don't use the INCLUDE_CONFIG_FILE option. (I use a combination of 3 levels of include files and 2 levels of symlinks to handle about 6 configurations times 2 arches times 4 FreeBSD versions.) > In this model, we're therefore able to preserve the kernel > config by copying the file, verbatim, into the kernel. I think this > is the model that best fits most user's needs, since they EITHER take > GENERIC and hack on it (in which case we preserve all that), OR they > include GENERIC and opt in/out of things based on that default. > Either way, this is the best way for users of releases to preserve the > data they need to recreate the kernel without losing data that's > important to the user, but not config (comments, spacing, order, etc). This may be enough, but it is a historical accident that config "works" this way. > The other way of preserving the config file is to say "I want > EVERYTHING applied, and I want a copy of that!" To get that, you have > to walk through config's internal data structures. By the time we get > to *THAT* point in the processing, comments are but a distant memory. Getting output from cpp has the same problems: - you can get fully preprocessed output fairly easily - cpp normally discards comments early, so you can't get them easily. I think gnu cpp now has an option to not discard them. - so you usually get only the fully preprocessed output - but sometimes you don't want so much preprocessing. None might be best. I don't know of any option in gnu cpp to get none. - the problem is harder in cpp than in config due to ifdefed out C code normally being completely removed, while config doesn't even have ifdef. Teaching cpp to preserve (only some) ifdefed out code, and presenting the results nicely (need to preserve the original ifdef structure) would be difficult. Except it might not be too difficult for either cpp or config to just concatenate all the files used. > Those are discarded early, along with the variations in spacing, blank > lines. By this point in the process, even the original ordering is > long gone, as is the origin of each of the remaining items in config's > internal data structures. However, the resulting config file that we > generate here is GUARANTEED to recreate the same kernel from the same > sources modulo any silly time/date/path stuff we encode. There's a I checked what happens with repeating DEFAULTS. Repeating it using "include" results in many warnings but the same generated files. > If space isn't an issue, we could save BOTH. That would be a bigger > patch, since we'd have to alter config to extract both kinds of data. > All I did was to move the -C option from an obscure src.conf variable > to be a full-fledged kernel option, leaving the rest of the > infrastructure intact. The advantage is that we cover both bases and > could export both views as a sysctl (right now we overload the two > different views with one sysctl). ISTR a long discussion about the -C option when it was implemented. Why didn't you complain back then? :-). I looked at my old mail about this. I wasn't involved with the initial discussion, but imp was :-). My mail was after the initial commit. I reported the following problems: - DEFAULTS wasn't processed - several ordering problems. Ones that reordered directives of the form: options FOO=bar nooptions FOO options FOO=baz made the generated file unusable. - many formatting problems in the generated file, making it a bad example. Bruce