Date: Thu, 27 Mar 2014 10:47:56 +0900 From: Julio Merino <jmmv@freebsd.org> To: "Kilner, Peter" <Peter.Kilner@emc.com> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: ATF Test Cases Message-ID: <CAFY7cWASGfetQLLr_0Zce%2B-Z-1EdtQdW-gPQEezz9YZ7ZSA7sg@mail.gmail.com> In-Reply-To: <DB777D43-945C-4C0D-A3A8-66DD5C9B214E@isilon.com> References: <DB777D43-945C-4C0D-A3A8-66DD5C9B214E@isilon.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Mar 27, 2014 at 7:26 AM, Kilner, Peter <Peter.Kilner@emc.com> wrote= : > Hello, Hi Peter, Alan already answered this pretty well, but let me throw in my 2 cents: > I hope I am posting this question to the correct list. > > Is there a way to have a single ATF test case report multiple pass/fail r= esults? Currently I have seen that one can include a number of test condit= ions in a single test case (for example many aft_checks). However the test= case will only report one pass/fail result for that case. That's by design: one test case should represent one (and only one) scenario. Therefore, there can only be one meaningful result. > I would like to build a test case that will iterate though many configura= tions and would like to see a pass/fail for each config. However this is d= ifficult to implement with multiply test cases because of the for loops tha= t I am using. It sounds like you actually should split your test case into smaller ones and name each test case accordingly. Your email doesn't provide a lot of details, but I guess that if you are testing more than one configuration file is because each configuration file is exposing a specific behavior of the program under test. Consider this: "test_configuration" sounds like a really broad test case name and provides no information about what is wrong when the test fails. But if you consider "test_configuration__missing_hostname", "test_configuration__invalid_syntax", "test_configuration__duplicate_host_stanzas" as separate test cases, then when one fails it becomes obvious what specific case is broken. Doing this needn't be hard. If you are using atf-sh, my suggestion is to put all of the test case body in a separate function. Then, with a loop, you dynamically define small test case bodies (using eval) that call the auxiliary function with the right parameters. If you are using atf-c, you can do something similar, maybe even abusing ma= cros. But basically: just move the code to an auxiliary function and have all the individual test cases call it as a one-liner with the right arguments. See the following for some more ideas: https://wiki.freebsd.org/TestSuite/Structure#Test_programs_vs._test_case= s Please let me know if anything is not clear or lacks detail.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFY7cWASGfetQLLr_0Zce%2B-Z-1EdtQdW-gPQEezz9YZ7ZSA7sg>