Date: Fri, 30 Aug 2019 21:54:41 -0400 From: Ed Maste <emaste@freebsd.org> To: Konstantin Belousov <kostikbel@gmail.com> Cc: Li-Wen Hsu <lwhsu@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org>, fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <CAPyFy2CXVMEXZx0u9b0p_CVaD_gu8Gyzvud3KwEDciq83a2_8A@mail.gmail.com> In-Reply-To: <20190830065534.GC71821@kib.kiev.ua> References: <CAKBkRUwKKPKwRvUs00ja0%2BG9vCBB1pKhv6zBS-F-hb=pqMzSxQ@mail.gmail.com> <20190829114057.GZ71821@kib.kiev.ua> <CAPyFy2BNrFfZ1PHaLesW%2Bu7YmbhC7mtiZ%2BBsYrqYE0J7KdagaA@mail.gmail.com> <20190830065534.GC71821@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 30 Aug 2019 at 02:56, Konstantin Belousov <kostikbel@gmail.com> wrote: > > When I was (forced to) look into test failures, it was 50 vs. 50 % > of test bugs vs. some legitimately catched issues. Certainly if 50% of reported failures are actually test problems that's much too high. But independent of that, this still suggests the tests were responsible for reporting a good number of issues in advance of developers or end users. > > - The test is difficult to maintain > This is too. My main complain is that to debug a test case, I must strip > all atf* to be able to examine it under a debugger. Yes, this is my biggest complaint about our current test setup. But this impacts the tests' friendliness to developers, but not their efficacy in reporting regressions.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPyFy2CXVMEXZx0u9b0p_CVaD_gu8Gyzvud3KwEDciq83a2_8A>