From owner-freebsd-arch@FreeBSD.ORG Tue Oct 2 14:19:56 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5139106566C; Tue, 2 Oct 2012 14:19:56 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6E7048FC12; Tue, 2 Oct 2012 14:19:56 +0000 (UTC) Received: by oagn9 with SMTP id n9so6298531oag.13 for ; Tue, 02 Oct 2012 07:19:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=8qvwGc8uFPh6HD2amh6kNVgUUS/wI4OOAWBrKkyGk98=; b=CmMMZxxkO/bTiMulW8aiK8tWOwyKE//g0TKx9TaWN+CFs62js3fF5kCcsBUctod2KU F7Kb22ti53Fx5jx3FsYhjchShaO8JRo4I5m3PNJEaAjtcQsgUWS+z6nEtzF7+mE5j6+p yPGJpfOLuGBjcZtstm532ODKnDDe1GOPtkn/4NRS2xm8x2gSqeS3RrxWYAUGjuCL+vjH n40YjTVdXDaDsiSPWGZR/OpcVyfbTZYb0L5uJntXMi4fOeUEZjknWdd49oNID4dodW1V dP64oScFp6KWIBh49tk9vtZpgcWPAMubhOYU3J/o62tdDegWZtVzq+xhUv6nCkdk+OSd wzmw== MIME-Version: 1.0 Received: by 10.60.170.241 with SMTP id ap17mr14742956oec.4.1349187595757; Tue, 02 Oct 2012 07:19:55 -0700 (PDT) Received: by 10.76.142.201 with HTTP; Tue, 2 Oct 2012 07:19:55 -0700 (PDT) In-Reply-To: <20121002000030.54CEE58093@chaos.jnpr.net> References: <20121001223100.E7D0D58093@chaos.jnpr.net> <20121002000030.54CEE58093@chaos.jnpr.net> Date: Tue, 2 Oct 2012 07:19:55 -0700 Message-ID: From: Garrett Cooper To: "Simon J. Gerraty" Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org, freebsd-arch@freebsd.org Subject: Re: [CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Oct 2012 14:19:56 -0000 On Mon, Oct 1, 2012 at 5:00 PM, Simon J. Gerraty wrote: >>> Not to mention the fact that bsd.prog.mk goes from being relatively >>> simple, to unspeakably hard to read, and all for rather limited = >>return. > > This btw I think is the more important issue. > I was looking at bsd.prog.mk in netbsd the other day. > It has no business being that complex. Ok. Even though I originally thought that my changes were a bit hacky for not modifying bsd.man.mk, bsd.nls.mk, etc, looking back the way they were resolved was a little elegant in its simplicity (albeit not optimized). ... >>It order to do this, I need to be able to build multiple programs so = >>things are as transparent. On the flipside, bsd.prog[s].mk can live on = > > We put the test cases in a subdir of the lib/prog > This has multiple benefits, and eliminates any impact on the normal > build of said libs/progs. Hmmm... that's one of the 3 approaches I provided, but it turned out to be annoying to make this work at first inspection because of how things were done in our build system. Just for review (even though it's OT), the three approaches I presented were as follows... Note: in all 3 examples, clock.c is the source code and t_clock.c is the relevant unittest code. 1. Test programs live with the sources (this was the requested approach), e.g. lib/libc/gen/... .../clock.c .../t_clock.c 2. Test programs live in subdirs: lib/libc/gen/... .../clock.c .../tests/... .../t_clock.c 3. Test programs completely decoupled from the source tree: lib/libc/gen/... .../clock.c tests/lib/libc/gen/... .../t_clock.c A hybrid approach (1.+3. / 2.+3.) should probably be used anyhow for stress tests and the like that really have no business living in one particular directory in the source tree (sort of achieving what tools/regression does today, but hopefully in a less messy manner as tools/regression appears to have grown organically minus any single sane order). > Also a number of our teams find it necessary to create mock classes etc to > adequately test their libs. Keeping all this in a tests/ subdir makes > it easy, to keep things neat & up to date, and ensure that the tests > pass. Trying to do all that in the same dir as the lib would be ugly. This (consolidating everything under a single directory) is the way that was requested by the beforementioned parties. >>> FWIW we use progs.mk (as bsd.progs.mk) from >>> ftp://ftp.netbsd.org/pub/NetBSD/misc/sjg/mk-*.tar.gz=20 >>> It isn't ideal, but it certainly avoids a lot of churn and complexity >>> for what is essentially a corner case. >> >>This requires pulling bmake into FreeBSD proper in order for things to = >>function the last I checked as bsd.prog.mk depends upon bmake directives = > > This is already happening ;-) I wish I had known that. I pinged marcel/obrien about this some weeks ago and didn't receive a response. Originally bsd.progs.mk was going to be a home grown effort that was going to be killed off and replaced with NetBSD's build infrastructure, but I didn't get a reply and had to put this (my work to replace bsd.prog.mk) plan in motion given requests I was provided above for resolving the source/unittest code consolidation effort. >>Ideally however, I would like doing this compared to running a custom = >>build infrastructure, but (as you probably know) this requires = >>rototilling the current FreeBSD build system to a large degree = > > Actually building FreeBSD-current using bmake requires only modest changes. Yes for the most part, and I agree that bmake is definitely more advanced than FreeBSD pmake so I consider it a very welcome change. What are the plans for importing bmake into FreeBSD (this should probably be a different thread)? ... > I know, but it is a very useful thing to be able to do. > You can add knobs to control such things of course. I agree that it's a good thing (in theory) -- it'd just help to discuss this with Julio first because I know of a couple cases where this would be unusable given how bsd.test.mk is coded: 1. The "it works for me on my machine!" certification: It encourages environment tainting, which could be a really, really bad thing; I've seen developers take interesting shortcuts when testing their code (me included sometimes :)..); I've seen hardcoded paths, harcoded "paths" for named semaphores, things that "just work" because of someone's host environment, feature specific assumptions (developer X was doing testing on feature Y and things broke when he/she committed the testcase to head), etc where the "it works for me on my machine!" certification would be true more often than not. These same individuals would more likely than not execute things taking shortcuts, but I want to avoid creating a system where developers cut corners and commit too early because working within the confines of the system is not conducive to getting work done. 2. Failure in a subdirectory results in lost coverage: a/... .../b/... <- tests live here .../c/d/... <- tests live here If I execute make test from a/ (one of my goals), then it should iterate down a/b, then a/c/d and run the tests in a DFS style (BFS if invoked with -j > 1 -- AIIIEEEE). If a/b fails to execute the make process will bomb out because of how it executes shell statements. However, if I let a/b pass, then developers/testers will accidentally overlook failures, unless output is captured from the run and errors are searched for in the end. This isn't a super big problem if the tests are deterministic enough that fixing one test permits the others to run, but the lost test coverage in that period of time (or others if it transiently fails) could disguise other bugs. This would be fixed by changing bsd.test.mk to use atf-run properly, but that's work (not hard) that still needs to be done. (there are some other cases, but they're eluding me now) > Yes, but making one test.mk handle potentially lots of frameworks > will get rather ugly quite quickly. Understood and I agree. I wasn't advocating having a single, monolithic bsd.test.mk with all the test frameworks integrated into it (that would be silly and unmaintainable). > Better to add per-framework .mk files, and perhaps have them include a > bsd.test.mk which only handles high level logic rather than the details > of the frameworks. > > That's laregly why we didn't use bsd.test.mk Hmmmm... My goal was to make ATF a "one ring to rule them all" infrastructure so that way everything could be unified under ATF in one way, shape or form (even if it's just reporting), because lord knows we don't need a lua, unittest/nose, etc wrapper for reporting results. All of the jobs I've worked at write custom wrappers around testing infrastructures in order to present the data in a pretty format (HTML), but that's a largely wasted effort (unless someone REALLY wants to learn CSS/HTML/JS, which is an ongoing pain thanks to browser incompatibilities), but I realize that this is also more effort than I could get to right away, and it would have to be done as needed instead. Thanks! -Garrett