Date: 15 Apr 1997 10:25:42 +0100 From: Paul Richards <p.richards@elsevier.co.uk> To: cracauer@wavehh.hanse.de (Martin Cracauer) Cc: paul@originat.demon.co.uk (Paul Richards), www@freebsd.org Subject: Re: Comments on new search page.. Message-ID: <57g1wsbpt5.fsf@tees.elsevier.co.uk> In-Reply-To: cracauer@wavehh.hanse.de's message of Tue, 15 Apr 1997 01:19:39 %2B0200 (MET DST) References: <9704142319.AA25408@wavehh.hanse.de>
next in thread | previous in thread | raw e-mail | index | archive | help
cracauer@wavehh.hanse.de (Martin Cracauer) writes:
Getting into the detail like this makes it all sound much more
complicated than it is when in fact it's a very straightforward setup.
Is there consensus to make a start on this? We're not fundamentally
disagreeing and the detail can fall out from an initial
implementation.
The existing /www/data area should be able to continue as it does
while we build the new environment around it. i.e. we start with a
/www/src
/www/tool/{bin,lib}
/www/build
areas and start populating them since at the end of it all there will
still be a /www/data with most of what is currently in it still there,
only the cgi scripts and some other bits will migrate out.
As a first phase we can start on /www/build without doing any of the
rest as long as all the requisite tools make it into the main tree or
ports so that in a FreeBSD environment you have everything you need.
I'd phase it like this:
1) Develop /www/build
This is the development of a build environment that relies on
all the tools being available and takes data from /www/data and builds
a site in /www/content.
2) Develop /www/tool
This phase works on developing a bootstrap stage where you can
start from nothing and build all your tools from /www/src into
/www/tool from scratch without depending on anything from the local
environment.
I'll start working on /www/build as soon as John puts SP in FreeBSD
somewhere if no-one raises any objections?
> Let's look over the three ways to adress the problem again:
> 1) We require mirrors to have all WWW-building tools up-to-date.
This is a requirement but I think we agree on that but not on how it's
achieved (although actually I think we do for the most part, see below).
> 2) We build in www/ with relative program targets like
> ../../src/usr.bin/sgmlfmt/sgmlfmt. We perform a 'make', but no
> 'make install' in these program directories. If the program
> needs additional steps after 'make all' to be operational in this
> directory, an additional target 'make for-www' can be added, which
> sets things up, but does not copy anything out of the src/ build
> directory.
I don't like the idea of running programs from their compilation area
for many reasons, see later.
> 3) We build all the tools from src/ and install them to a non-default
> target where only the WWW build process will find and use
> them. Every such tool then has two install targets, 'make install'
> will install to where the FreeBSD release expects it and 'make
> www-install' will install to wherever the WWW tools tree is.
> (Or two copies of the program are in the CVS tree, see below).
You seem to be suggesting that we utilise the normal /usr/src build
mechanism and add new targets for /www. This isn't really acceptable
either since /www isn't something that should be integrated into the
OS.
> 2) und 3) come in two variations
> a) For all programs that are needed for the WWW build process we have
> copies in src/ and www/ so the src/ is never needed. That way,
> people that modify these tools don't have to care whether they
> affect the general FreeBSD release.
> b) We keep the tools in src/. We have to trust those who work on WWW
> and make modifications that they don't break the src/ tree.
There should never be any changes made to /usr/src purely because of
/www needs. The base OS should not have to be modified in any way.
> In both cases will tools that are *only* for www and never for a
> release be added to www/
Agreed, I'm actually beginning to think that they could go in ports or
we have something akin to the contrib area but the other way around
i.e. /www/src/groff would be a wrapper that uses /usr/src/.../groff in
the same way that /usr/src has wrappers for the contrib area.
Both the contrib and ports mechanisms offer solutions to utilising the
main /usr/src area without having to actually make /www specific
changes to the main tree.
> > We should think a bit about this. As far as possible it would be nice
> > to not have to duplicate source trees but we also need to detach the
> > www area from dependance on FreeBSD itself because the needs of www
> > will move much faster than those of FreeBSD. It's essential that we
> > have Perl5 for instance.
>
> I doubt that will be case for much more than perl. For any other tool
> I can think of, either WWW or the normal FreeBSD system can use
> whatever that other needs. The disadvantages of maintaining two copies
> are far greater, IMHO.
For a FreeBSD site that would be true but for non-FreeBSD sites you'd
need to have standalone make build environments. Not an immediate goal
(we should aim for a self-contained FreeBSD based system first) but
the solution should not make a portable system impossible by requiring
FreeBSD specific resources (such as the main source tree).
> I think you didn't really understand what I'm proposing.
I didn't entirely though I still disagree with a few things.
> I want a WWW mirror to check out the whole www/ directory of the CVS
> tree and parts of src/ (using a CVS module, this is trivial).
Agreed.
> The WWW build process then uses the programs it relies on from the
> src/ directory, i.e. ../../src/usr.bin/smglfmt/sgmlfmt or
> ../../src/share/mk/whtever.mk. Programs from this src. tree are not
> installed to the base system.
>
> That way, we transport versions of the tools along with the www pages
> and keep them in sync, but we don;t touch a user's machine.
We had a misunderstanding here. My example of /tool was too
specific. Although that's what we do in work in this situation I was
intending to do as you suggested and have the tools under /www. My
goal is definately to be able to tar up www, ship it to a mirror site
and be able to type make to have a mirror built (obviously some local
configuration may be necessary).
> I agree that perl is a problem, but perl5 is required anyway for some
> cgi stuff (I think), we could require that mirrors have
> /usr/local/bin/perl == perl5. Have to think over it.
No, I'm still not in favour of this. Any www code should not refer to
anything outside of /www. As I said before, it's bad programming
practice to have cgi scripts that use things like 'ls' so there
shouln't ever be any reason to have to run system binaries. It's
simply not possible to rely on the mirror sites having a setup that's
totally compatible with the www system.
The ports build system offers us a lot here. We don't have to require
that mirrors take a complete /www/tool area, the ports mechanism can
be used to build those parts that the local site doesn't have but this
mechanism must first do the checking to see if the local environment
is compatible and build the replacements if necessary. What this
mechanism does during this process is setup the www environment
appropriately. This can be done with symlinks from /www/tool/bin so
that the scripts can always reference /www/tool/bin regardless of
whether the local binaries are used or not.
This is icing though, initially I'd recommend building all the tools
we need into /www/tool/bin.
> Yes, but this build environment needs to be set up. writeable Space
> must be there, in an absolutly addressed location and since it can
> vary (we can't push mirros to use a fixed location), you need to have
> a build rocedure that will accept some way to tell it where to find
> the stuff.
But that build environment is within the /www area, this is internal
to the product and has nothing to do with the local site. The local
site would have to decide where www/ is to be put but they wouldn't
have any control over what's under www/.
> In my solution, the tool location is always fixed and you don't need
> an extra place besides the CVS checkout tree and the target for the
> built www pages.
This is where there seems to be some confusion. Forget the /tool bit,
it should have been /www/tool for this project since there's no
requirement to share the tools amongst different projects. This is
part of the way towards what you say since all the tools are now
within /www/tool. I disagree though with running tools from the
directories that they are built into for a number of reasons.
1) It's untidy, I don't like the idea of running programs from their
object directory so there has to be a make install step.
2) You may not want to distribute the sources of the tools. I can see
a need to distribute the "site build" environment, that is all the
pre-built tools and all the site data so that mirrors can build the
site but not have to build all the tools.
3) You can't rebuild the tools without clobbering the build
environment. You should be able to make changes to a tool and
compile it without having it appear in the runtime environment. This
is why there's a make install step after all, otherwise we could run
FreeBSD out of /usr/{src.obj}.
4) I think using a cvs distribution mechanism is in the back of your
mind. I don't the the web site should rely on that. It'd be a nice
option for mirrors but you shouldn't require that they mirror cvs
areas in order to build a mirror web site.
> > I think you may be confused by my use of the word build. In my example
> > the build area is where the scripts used to build the site are
> > placed. These are, for example, perl scripts for converting man pages
> > into html. These never get installed on the web server (the web server
> > is coceptually a different environment even if it's the same
> > hardware).
>
> OK, but I imagine that the usual way to set up a mirror i to run the
> build process on the same machine that is the http server.
>
> We need this for example to take care of local specialities like CGIs
> to run locally and local additions.
Not really. You set up the configuration for your target, you don't
have to build the site on the machine that has the web server. In
work we build the site inside the firewall and distribute the built
site to the web server when it's all checked and ready. You should
never build a site into a live area. There should always be a make
install (or for this bit of the process a make publish) step that puts
the built content live.
Incidentally, all this talk of make. I suspect that the site build
mechanism is more likely to be a perl script, quite possibly with a
nice tk front end.
> > It's akin to the two steps of building the FreeBSD documentation. You
> > first have to build the tools that are needed to "build" the
> > documents.
>
> The tools need to be built somehow, but I think they don't need to be
> installed anywhere.
See above.
--
Dr Paul Richards. [p.richards@elsevier.co.uk]
Originative Solutions Ltd. [paul@originat.demon.co.uk]
Phone: 0370 462071 (Mobile), +44 (0)1865 843155 (Elsevier)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?57g1wsbpt5.fsf>