Date: Thu, 26 Aug 1999 10:49:50 -0400 From: Brian McGovern <bmcgover@cisco.com> To: hackers@freebsd.org Subject: Looking for good QA tests... Message-ID: <199908261449.KAA02505@bmcgover-pc.cisco.com>
next in thread | raw e-mail | index | archive | help
Hi everyone... Back in June, at Usenix, Jordan and I talked about doing some more formalized QA on FreeBSD releases. Unfortunately, this has yet to go really far, as I was trying to complete some commitments, and have some infrastructure in place before I started making this endeavor overly public, as I'm also trying to keep the amount of time I have to put in to it to a moderate pace. However, I'm now at the point where I'd like to start collecting materials to do this. By "materials", I mean both test scenarios and code for performing these tests. While I now have your mind spinning, I'd like to lay down some limits. Firstly, my current resources for testing is relatively small. I have four machines that I can use most of the time. Therefore, we can't do "everything, all the time", so tests should be limited to cover as broad a range of items as we have time for. My estimation is we will get approximately 48 hours per release candidate as we move towards Jordan cutting a release. In between, we might get a bit more slack, but not a lot. Secondly, these tests can not take a lot of person hours to perform. _I_ can give approximately 1-2 man hours per test cycle. Therefore, these tests need to be automated, have good reporting (especially if they fail), and be easy to set up and run. Lastly, these tests should be tests, and not benchmarks. I can't stress this enough. Knowing how fasts my disks are is useless compared to a bug in the network interface that causes the machine to panic. Now, as more people sign on to both write code/test plans, as well as do the testing, we might be able to expand the types of tests we can run. My plans are to wrap up whatever I'm given in the form of code in to a port/package set, and have it distributed so that anyone with a spare machine can come onboard and help out. Therefore, I'm asking people to put their minds to work, and talk about valid tests that we could run to catch things that might slip through the cracks. I'm hoping that some of the long-timers can point out areas that have been missed before, and others to point out areas where they have seen local problems. I am hoping that there are also enough interested people out there with coding skills (this may be an area where heavy-duty skills are not required) can take some of these ideas, and convert them in to test code that operates within the limits above. I look forward to hearing your comments. -Brian To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199908261449.KAA02505>