Date: Mon, 19 Nov 2012 08:13:33 -0800 From: "Simon J. Gerraty" <sjg@juniper.net> To: Poul-Henning Kamp <phk@phk.freebsd.dk> Cc: Garrett Cooper <yanegomi@gmail.com>, George Neville-Neil <gnn@freebsd.org>, "freebsd-arch@FreeBSD.org Arch" <freebsd-arch@freebsd.org>, Matthew Fleming <mdf@freebsd.org>, Marcel Moolenaar <marcel@xcllnt.net> Subject: Re: [RFC] test layout/standardization for FreeBSD Message-ID: <20121119161333.6A37C58094@chaos.jnpr.net> In-Reply-To: <31881.1353312581@critter.freebsd.dk> References: <CAGH67wSoU08N8=40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com> <7099.1352886181@critter.freebsd.dk> <F3302426-CA5D-4B61-8F3A-081395B393D9@xcllnt.net> <31881.1353312581@critter.freebsd.dk>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 19 Nov 2012 08:09:41 +0000, "Poul-Henning Kamp" writes: >Some tests are not 100% deterministic no matter what. That is >an important distinction. Yes. It would probably be worth defining some meta tags to identify things like that. Even if the outcome were nothing more than a warning to the user "hey the results here can vary - don't panic" Still, to me, tests of that nature start to sound more like system tests than unit-tests. >>Also, the estimated duration for tests is very platform specific. > >Only if CPU or I/O bound. Tests for things like dynamic route-expiry, >and other timeout driven parts of the system will have long realtime >clock, but short cpu-clock. Again, that sounds more like a system test than unit-test. For a unit-test you would want to mock the environment so that your timer events or whatever are fully controlled (and compressed). >I fully agree with your "crawl before we run" approach, but testing >is a quite mature, if underappreciated, discipline, and we would do >well to learn from others mistakes. Agreed. One of the common misstakes it seems is bluring the boundaries between unit, function and system tests. System tests are needed for sure, but simple unit-tests (of the bits that can be simply unit-tested) are far more valuable for establishing good overal test coverage. >All I'm asking at this point, is that we make a space for storing >this metadata in a machine-readable format. Sounds reasonable, though exactly what and how, is to be determined, and my hope is that such info should not be needed until we start pushing unit-tests into system test territory. Thanks --sjg
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20121119161333.6A37C58094>