Date: Tue, 28 Aug 2001 11:16:20 +0300 From: Maxim Sobolev <sobomax@FreeBSD.org> To: mi@aldan.algebra.com Cc: ports@FreeBSD.org Subject: Re: RFC: on the merits of post-build testing Message-ID: <3B8B5354.EE77877D@FreeBSD.org> References: <200108272211.f7RMB1k22861@misha.privatelabs.com>
next in thread | previous in thread | raw e-mail | index | archive | help
mi@aldan.algebra.com wrote: > Hello! > > Some of the software packages ported to FreeBSD (ported software or > software) come with own suite of post-build tests. Some ports try to use > those tests to check the results of the build prior to installation. > While it seems like a Good Idea(TM) to yours truly, some others > (violently) object to it. In this letter, I'll try to summarize and > illustrate why I like it, why you may dislike it, and why you might be > wrong disliking it :-) > > The benefit is rather obvious: it helps catching bugs. > > Bugs of different nature -- of the ported software, of the port, of the > local configuration (as in "over optimization" requested by the user > through the CFLAGS setting, for example). I look forward for the new > version of the compiler, for another example :-) > > Why some dislike it: > > * it is a waste of the CPU time > > well, so is checking for the result of the malloc() :-) So is not > compiling your kernel with -fomit-frame-pointer. I think, that majority > of those concerned about the CPU time will use the precompiled packages. > Testing time is, usually, only a fraction of the build time. > > * all such detectable errors _could_ be detected prior to > committing the port > > I wish. Take the graphics/lcms port, for example. The lcms author does > development on WinNT x86. The current port compiles on FreeBSD-Alpha, > but does not pass its own tests. If those weren't turned on by the port > maintainer, the author's numerous wrong assumptions about the sizes of > ints and longs would've been mysteriously crashing programs, which use > liblcms on the Alphas... Although I could've tested the thing on beast > prior to committing, how many people actually do such a thing? For > ports, that require other ports, this is not even possible, unless you > happen to have root-access to beast. The tests ran fine for me on > -current and -stable. They also ran fine on bento... > > * all such detectable errors _should_ be detected prior to > committing the port > > Well, sometimes, it is the users' CFLAGS, that the maintainer has no > control of. Sometimes it is some interaction with some other port, > the amount of memory, screen resolution -- there can be plenty of > situations, when the software builds, but does not work. I'm sure > everyone here has a story or two. > > * it is not going to guarantee anything anyway > > No. Neither is FreeBSD (nor any other software I know of) guaranteed to > work -- at all. Yet, some times it works. The testing will catch the > errors, the software's author(s) deemed likely to happen. > > * having to maintain the tests as well is an unneeded burden on > the port maintainer > > Yes, it is a burden, although not unbearable. No, it is needed, IMO. > > * this might be a good idea for some ports, but not the others, > therefore no general decision is possible/desirable > > Personally, I don't understand this, but even if you do, I'm not saying > we make such testing a requirement. IMO, it should be encouraged, > however. > > Assuming, your read it all up to here, here is what I propose: > > . add handling of the ``test'' target if defined in the port's > Makefile by the bsd.ports.mk; > > . make sure the test target is exercised automaticly after the > build and/or before the install; > > . allow the testing to be turned off with something like > NOPORTTESTS=YES > to satisfy (some of) my opponents; just like NOPORTDOCS, the > flag should not be set by default; > > . encourage the existing and new ports to check the ported > software for the bundled tests and create the above described > test-target whenever possible; if some of the tests fail due > to problems with the tests themselves (Tcl and Tk tests are > pretty horrible at that, for example), they should be patched > or taken out of the sequence (by patching the provided > test-scripts). > > Comments anyone? As a fairly fresh committer, I'm not sure what kind of > quorum/consensus is needed for something like this to make it through. That are a very interesting arguments, but the real point is that in 99.9% of cases running tests wouldn't cause anything but useless waste of CPU time. From the remainder 0.09% is associated with people adding unsupported optimisation levels into CFLAGS (they deserve punishment for that anyway) and 0.01% with people running strange hardware (i.e. Alphas and faulty x86). Therefore, I'm still convinced that this micro-optimisation would not bring us anything but additional maintenance headache, thus diverting valuable developers' resources from the really important problems affecting much wider auditory of users. -Maxim To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3B8B5354.EE77877D>