Date: Wed, 26 Mar 2014 22:00:28 -0600 From: Alan Somers <asomers@freebsd.org> To: Julio Merino <jmmv@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, "Kilner, Peter" <Peter.Kilner@emc.com> Subject: Re: ATF Test Cases Message-ID: <CAOtMX2i4dQqkkD5-jNNuzTFdqOhLcVdD0nVLSekNjM9vfnQ6_w@mail.gmail.com> In-Reply-To: <CAFY7cWASGfetQLLr_0Zce%2B-Z-1EdtQdW-gPQEezz9YZ7ZSA7sg@mail.gmail.com> References: <DB777D43-945C-4C0D-A3A8-66DD5C9B214E@isilon.com> <CAFY7cWASGfetQLLr_0Zce%2B-Z-1EdtQdW-gPQEezz9YZ7ZSA7sg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Mar 26, 2014 at 7:47 PM, Julio Merino <jmmv@freebsd.org> wrote: > On Thu, Mar 27, 2014 at 7:26 AM, Kilner, Peter <Peter.Kilner@emc.com> wro= te: >> Hello, > > Hi Peter, > > Alan already answered this pretty well, but let me throw in my 2 cents: > >> I hope I am posting this question to the correct list. >> >> Is there a way to have a single ATF test case report multiple pass/fail = results? Currently I have seen that one can include a number of test condi= tions in a single test case (for example many aft_checks). However the tes= t case will only report one pass/fail result for that case. > > That's by design: one test case should represent one (and only one) > scenario. Therefore, there can only be one meaningful result. > >> I would like to build a test case that will iterate though many configur= ations and would like to see a pass/fail for each config. However this is = difficult to implement with multiply test cases because of the for loops th= at I am using. > > It sounds like you actually should split your test case into smaller > ones and name each test case accordingly. Your email doesn't provide > a lot of details, but I guess that if you are testing more than one > configuration file is because each configuration file is exposing a > specific behavior of the program under test. > > Consider this: "test_configuration" sounds like a really broad test > case name and provides no information about what is wrong when the > test fails. But if you consider > "test_configuration__missing_hostname", > "test_configuration__invalid_syntax", > "test_configuration__duplicate_host_stanzas" as separate test cases, > then when one fails it becomes obvious what specific case is broken. > > Doing this needn't be hard. > > If you are using atf-sh, my suggestion is to put all of the test case > body in a separate function. Then, with a loop, you dynamically > define small test case bodies (using eval) that call the auxiliary > function with the right parameters. I never thought about using eval to dynamically define testcases. Can you please give an example? > > If you are using atf-c, you can do something similar, maybe even abusing = macros. > > But basically: just move the code to an auxiliary function and have > all the individual test cases call it as a one-liner with the right > arguments. > > See the following for some more ideas: > > https://wiki.freebsd.org/TestSuite/Structure#Test_programs_vs._test_ca= ses > > Please let me know if anything is not clear or lacks detail. > _______________________________________________ > freebsd-testing@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-testing > To unsubscribe, send any mail to "freebsd-testing-unsubscribe@freebsd.org= "
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2i4dQqkkD5-jNNuzTFdqOhLcVdD0nVLSekNjM9vfnQ6_w>