Date: Fri, 30 Aug 2019 09:55:34 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Ed Maste <emaste@freebsd.org> Cc: Li-Wen Hsu <lwhsu@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org>, fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190830065534.GC71821@kib.kiev.ua> In-Reply-To: <CAPyFy2BNrFfZ1PHaLesW%2Bu7YmbhC7mtiZ%2BBsYrqYE0J7KdagaA@mail.gmail.com> References: <CAKBkRUwKKPKwRvUs00ja0%2BG9vCBB1pKhv6zBS-F-hb=pqMzSxQ@mail.gmail.com> <20190829114057.GZ71821@kib.kiev.ua> <CAPyFy2BNrFfZ1PHaLesW%2Bu7YmbhC7mtiZ%2BBsYrqYE0J7KdagaA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Aug 29, 2019 at 07:35:57PM -0400, Ed Maste wrote: > On Thu, 29 Aug 2019 at 07:41, Konstantin Belousov <kostikbel@gmail.com> wrote: > > > > More, I know that tests are of very low quality, which means that > > brokeness of the tests is not an indicator of anything until root cause > > is identified. > > "Low quality" needs clarification here. I can think of many attributes > of a test that might lead someone to claim tests are low quality: > > - The test result is not consistent (e.g., a "flaky test") > - The test does not actually test what it claims to This. In fact, for code base like OS which provides an API/ABI to the large corpus of user code, it does not matter what exactly test claims to test. We are interested in any behaviour change. But very unfortunate and deeply engraved wart in many tests I saw is that they depend on unspecified or non-guaranteed API details to claim success. So detected behaviour changes often appears to be irrelevant. When I was (forced to) look into test failures, it was 50 vs. 50 % of test bugs vs. some legitimately catched issues. > - The test does as it claims, but there is no value in the result > - Test coverage overall is insufficient (i.e., not an issue with a > specific test) > - The test has excessive requirements (run time, memory usage, etc.) > - The test is difficult to maintain This is too. My main complain is that to debug a test case, I must strip all atf* to be able to examine it under a debugger.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190830065534.GC71821>