Date: Mon, 27 Aug 2001 18:11:00 -0400 (EDT) From: mi@aldan.algebra.com To: ports@freebsd.org Subject: RFC: on the merits of post-build testing Message-ID: <200108272211.f7RMB1k22861@misha.privatelabs.com>
next in thread | raw e-mail | index | archive | help
Hello! Some of the software packages ported to FreeBSD (ported software or software) come with own suite of post-build tests. Some ports try to use those tests to check the results of the build prior to installation. While it seems like a Good Idea(TM) to yours truly, some others (violently) object to it. In this letter, I'll try to summarize and illustrate why I like it, why you may dislike it, and why you might be wrong disliking it :-) The benefit is rather obvious: it helps catching bugs. Bugs of different nature -- of the ported software, of the port, of the local configuration (as in "over optimization" requested by the user through the CFLAGS setting, for example). I look forward for the new version of the compiler, for another example :-) Why some dislike it: * it is a waste of the CPU time well, so is checking for the result of the malloc() :-) So is not compiling your kernel with -fomit-frame-pointer. I think, that majority of those concerned about the CPU time will use the precompiled packages. Testing time is, usually, only a fraction of the build time. * all such detectable errors _could_ be detected prior to committing the port I wish. Take the graphics/lcms port, for example. The lcms author does development on WinNT x86. The current port compiles on FreeBSD-Alpha, but does not pass its own tests. If those weren't turned on by the port maintainer, the author's numerous wrong assumptions about the sizes of ints and longs would've been mysteriously crashing programs, which use liblcms on the Alphas... Although I could've tested the thing on beast prior to committing, how many people actually do such a thing? For ports, that require other ports, this is not even possible, unless you happen to have root-access to beast. The tests ran fine for me on -current and -stable. They also ran fine on bento... * all such detectable errors _should_ be detected prior to committing the port Well, sometimes, it is the users' CFLAGS, that the maintainer has no control of. Sometimes it is some interaction with some other port, the amount of memory, screen resolution -- there can be plenty of situations, when the software builds, but does not work. I'm sure everyone here has a story or two. * it is not going to guarantee anything anyway No. Neither is FreeBSD (nor any other software I know of) guaranteed to work -- at all. Yet, some times it works. The testing will catch the errors, the software's author(s) deemed likely to happen. * having to maintain the tests as well is an unneeded burden on the port maintainer Yes, it is a burden, although not unbearable. No, it is needed, IMO. * this might be a good idea for some ports, but not the others, therefore no general decision is possible/desirable Personally, I don't understand this, but even if you do, I'm not saying we make such testing a requirement. IMO, it should be encouraged, however. Assuming, your read it all up to here, here is what I propose: . add handling of the ``test'' target if defined in the port's Makefile by the bsd.ports.mk; . make sure the test target is exercised automaticly after the build and/or before the install; . allow the testing to be turned off with something like NOPORTTESTS=YES to satisfy (some of) my opponents; just like NOPORTDOCS, the flag should not be set by default; . encourage the existing and new ports to check the ported software for the bundled tests and create the above described test-target whenever possible; if some of the tests fail due to problems with the tests themselves (Tcl and Tk tests are pretty horrible at that, for example), they should be patched or taken out of the sequence (by patching the provided test-scripts). Comments anyone? As a fairly fresh committer, I'm not sure what kind of quorum/consensus is needed for something like this to make it through. -mi To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200108272211.f7RMB1k22861>