From owner-freebsd-www Tue Apr 15 02:26:16 1997 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.5/8.8.5) id CAA28711 for www-outgoing; Tue, 15 Apr 1997 02:26:16 -0700 (PDT) Received: from pillar.elsevier.co.uk (root@pillar.elsevier.co.uk [193.131.222.35]) by freefall.freebsd.org (8.8.5/8.8.5) with ESMTP id CAA28704 for ; Tue, 15 Apr 1997 02:26:12 -0700 (PDT) Received: from snowdon.elsevier.co.uk (snowdon.elsevier.co.uk [193.131.197.164]) by pillar.elsevier.co.uk (8.8.5/8.8.5) with ESMTP id KAA16878 for ; Tue, 15 Apr 1997 10:23:35 +0100 (BST) Received: from cadair.elsevier.co.uk by snowdon.elsevier.co.uk with SMTP (PP); Tue, 15 Apr 1997 10:25:53 +0100 Received: from tees.elsevier.co.uk (tees.elsevier.co.uk [193.131.197.60]) by cadair.elsevier.co.uk (8.8.5/8.8.5) with ESMTP id KAA26819; Tue, 15 Apr 1997 10:25:44 +0100 (BST) Received: (from dpr@localhost) by tees.elsevier.co.uk (8.8.5/8.8.5) id KAA02194; Tue, 15 Apr 1997 10:25:44 +0100 (BST) To: cracauer@wavehh.hanse.de (Martin Cracauer) Cc: paul@originat.demon.co.uk (Paul Richards), www@freebsd.org Subject: Re: Comments on new search page.. References: <9704142319.AA25408@wavehh.hanse.de> From: Paul Richards Date: 15 Apr 1997 10:25:42 +0100 In-Reply-To: cracauer@wavehh.hanse.de's message of Tue, 15 Apr 1997 01:19:39 +0200 (MET DST) Message-ID: <57g1wsbpt5.fsf@tees.elsevier.co.uk> Lines: 236 X-Mailer: Gnus v5.4.37/Emacs 19.30 Sender: owner-freebsd-www@freebsd.org X-Loop: FreeBSD.org Precedence: bulk cracauer@wavehh.hanse.de (Martin Cracauer) writes: Getting into the detail like this makes it all sound much more complicated than it is when in fact it's a very straightforward setup. Is there consensus to make a start on this? We're not fundamentally disagreeing and the detail can fall out from an initial implementation. The existing /www/data area should be able to continue as it does while we build the new environment around it. i.e. we start with a /www/src /www/tool/{bin,lib} /www/build areas and start populating them since at the end of it all there will still be a /www/data with most of what is currently in it still there, only the cgi scripts and some other bits will migrate out. As a first phase we can start on /www/build without doing any of the rest as long as all the requisite tools make it into the main tree or ports so that in a FreeBSD environment you have everything you need. I'd phase it like this: 1) Develop /www/build This is the development of a build environment that relies on all the tools being available and takes data from /www/data and builds a site in /www/content. 2) Develop /www/tool This phase works on developing a bootstrap stage where you can start from nothing and build all your tools from /www/src into /www/tool from scratch without depending on anything from the local environment. I'll start working on /www/build as soon as John puts SP in FreeBSD somewhere if no-one raises any objections? > Let's look over the three ways to adress the problem again: > 1) We require mirrors to have all WWW-building tools up-to-date. This is a requirement but I think we agree on that but not on how it's achieved (although actually I think we do for the most part, see below). > 2) We build in www/ with relative program targets like > ../../src/usr.bin/sgmlfmt/sgmlfmt. We perform a 'make', but no > 'make install' in these program directories. If the program > needs additional steps after 'make all' to be operational in this > directory, an additional target 'make for-www' can be added, which > sets things up, but does not copy anything out of the src/ build > directory. I don't like the idea of running programs from their compilation area for many reasons, see later. > 3) We build all the tools from src/ and install them to a non-default > target where only the WWW build process will find and use > them. Every such tool then has two install targets, 'make install' > will install to where the FreeBSD release expects it and 'make > www-install' will install to wherever the WWW tools tree is. > (Or two copies of the program are in the CVS tree, see below). You seem to be suggesting that we utilise the normal /usr/src build mechanism and add new targets for /www. This isn't really acceptable either since /www isn't something that should be integrated into the OS. > 2) und 3) come in two variations > a) For all programs that are needed for the WWW build process we have > copies in src/ and www/ so the src/ is never needed. That way, > people that modify these tools don't have to care whether they > affect the general FreeBSD release. > b) We keep the tools in src/. We have to trust those who work on WWW > and make modifications that they don't break the src/ tree. There should never be any changes made to /usr/src purely because of /www needs. The base OS should not have to be modified in any way. > In both cases will tools that are *only* for www and never for a > release be added to www/ Agreed, I'm actually beginning to think that they could go in ports or we have something akin to the contrib area but the other way around i.e. /www/src/groff would be a wrapper that uses /usr/src/.../groff in the same way that /usr/src has wrappers for the contrib area. Both the contrib and ports mechanisms offer solutions to utilising the main /usr/src area without having to actually make /www specific changes to the main tree. > > We should think a bit about this. As far as possible it would be nice > > to not have to duplicate source trees but we also need to detach the > > www area from dependance on FreeBSD itself because the needs of www > > will move much faster than those of FreeBSD. It's essential that we > > have Perl5 for instance. > > I doubt that will be case for much more than perl. For any other tool > I can think of, either WWW or the normal FreeBSD system can use > whatever that other needs. The disadvantages of maintaining two copies > are far greater, IMHO. For a FreeBSD site that would be true but for non-FreeBSD sites you'd need to have standalone make build environments. Not an immediate goal (we should aim for a self-contained FreeBSD based system first) but the solution should not make a portable system impossible by requiring FreeBSD specific resources (such as the main source tree). > I think you didn't really understand what I'm proposing. I didn't entirely though I still disagree with a few things. > I want a WWW mirror to check out the whole www/ directory of the CVS > tree and parts of src/ (using a CVS module, this is trivial). Agreed. > The WWW build process then uses the programs it relies on from the > src/ directory, i.e. ../../src/usr.bin/smglfmt/sgmlfmt or > ../../src/share/mk/whtever.mk. Programs from this src. tree are not > installed to the base system. > > That way, we transport versions of the tools along with the www pages > and keep them in sync, but we don;t touch a user's machine. We had a misunderstanding here. My example of /tool was too specific. Although that's what we do in work in this situation I was intending to do as you suggested and have the tools under /www. My goal is definately to be able to tar up www, ship it to a mirror site and be able to type make to have a mirror built (obviously some local configuration may be necessary). > I agree that perl is a problem, but perl5 is required anyway for some > cgi stuff (I think), we could require that mirrors have > /usr/local/bin/perl == perl5. Have to think over it. No, I'm still not in favour of this. Any www code should not refer to anything outside of /www. As I said before, it's bad programming practice to have cgi scripts that use things like 'ls' so there shouln't ever be any reason to have to run system binaries. It's simply not possible to rely on the mirror sites having a setup that's totally compatible with the www system. The ports build system offers us a lot here. We don't have to require that mirrors take a complete /www/tool area, the ports mechanism can be used to build those parts that the local site doesn't have but this mechanism must first do the checking to see if the local environment is compatible and build the replacements if necessary. What this mechanism does during this process is setup the www environment appropriately. This can be done with symlinks from /www/tool/bin so that the scripts can always reference /www/tool/bin regardless of whether the local binaries are used or not. This is icing though, initially I'd recommend building all the tools we need into /www/tool/bin. > Yes, but this build environment needs to be set up. writeable Space > must be there, in an absolutly addressed location and since it can > vary (we can't push mirros to use a fixed location), you need to have > a build rocedure that will accept some way to tell it where to find > the stuff. But that build environment is within the /www area, this is internal to the product and has nothing to do with the local site. The local site would have to decide where www/ is to be put but they wouldn't have any control over what's under www/. > In my solution, the tool location is always fixed and you don't need > an extra place besides the CVS checkout tree and the target for the > built www pages. This is where there seems to be some confusion. Forget the /tool bit, it should have been /www/tool for this project since there's no requirement to share the tools amongst different projects. This is part of the way towards what you say since all the tools are now within /www/tool. I disagree though with running tools from the directories that they are built into for a number of reasons. 1) It's untidy, I don't like the idea of running programs from their object directory so there has to be a make install step. 2) You may not want to distribute the sources of the tools. I can see a need to distribute the "site build" environment, that is all the pre-built tools and all the site data so that mirrors can build the site but not have to build all the tools. 3) You can't rebuild the tools without clobbering the build environment. You should be able to make changes to a tool and compile it without having it appear in the runtime environment. This is why there's a make install step after all, otherwise we could run FreeBSD out of /usr/{src.obj}. 4) I think using a cvs distribution mechanism is in the back of your mind. I don't the the web site should rely on that. It'd be a nice option for mirrors but you shouldn't require that they mirror cvs areas in order to build a mirror web site. > > I think you may be confused by my use of the word build. In my example > > the build area is where the scripts used to build the site are > > placed. These are, for example, perl scripts for converting man pages > > into html. These never get installed on the web server (the web server > > is coceptually a different environment even if it's the same > > hardware). > > OK, but I imagine that the usual way to set up a mirror i to run the > build process on the same machine that is the http server. > > We need this for example to take care of local specialities like CGIs > to run locally and local additions. Not really. You set up the configuration for your target, you don't have to build the site on the machine that has the web server. In work we build the site inside the firewall and distribute the built site to the web server when it's all checked and ready. You should never build a site into a live area. There should always be a make install (or for this bit of the process a make publish) step that puts the built content live. Incidentally, all this talk of make. I suspect that the site build mechanism is more likely to be a perl script, quite possibly with a nice tk front end. > > It's akin to the two steps of building the FreeBSD documentation. You > > first have to build the tools that are needed to "build" the > > documents. > > The tools need to be built somehow, but I think they don't need to be > installed anywhere. See above. -- Dr Paul Richards. [p.richards@elsevier.co.uk] Originative Solutions Ltd. [paul@originat.demon.co.uk] Phone: 0370 462071 (Mobile), +44 (0)1865 843155 (Elsevier)