Date: Fri, 29 Mar 2002 15:34:49 -0800 From: Terry Lambert <tlambert2@mindspring.com> To: Pete Ehlke <pde@ehlke.net> Cc: chat@freebsd.org Subject: Re: qmail (Was: Maintaining Access Control Lists ) Message-ID: <3CA4FA19.F72624A0@mindspring.com> References: <20020328203704.GA760@lpt.ens.fr> <p05101544b8c96484e42d@[10.0.1.8]> <20020329081349.GA1737@lpt.ens.fr> <20020329044824.B12348@ehlke.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Pete Ehlke wrote: > Note for QA folks: when the developer says "That's not a bug, you're using > it wrong", you are dealing with the worst sort of arrogant prima donna. I've spent a lot of time at one employer or another fixing things that weren't problems because a Q/A person tried to use the product to fulfill a role for which it was never designed, and for which it was never intended to be marketed. When doing Q/A, you need to make sure that what you are testing falls within the set of use cases for the product, and isn't some oddball configuration which can never happen unless you give the end users more access to your box than they can normally achieve using any combination of the permissable UIs (GUI, CLI, initial configuration, "wizards", etc.). You also have to realize that discovering a boundary condition isn't really enough to prevent shipping the product: "Eventually, a software comapny has to ship software." -- Greg Haerr Your best developer in any group of developers can probably tell you dozens of degenerate cases that will cause your product to perform badly and/or to fail outright. Some public tests (e.g. "Polygraph") are *designed* to adaptively find the worst case performance scenario for any product, and then exercise it. Then you end up with assinine things like "random page replacement" to become unpredictable to these tests, and get better numbers, which are only better numbers on the benchmark, since they throw out things that would optimize real world use of the product, like locality of reference. Some Q/A people also feel that they have failed at their job, unless they have crashed the product. So they come up with tests that can crash the product in 24 hours: using something like an "Avalanche" or other purpose-built test equipment, over a gigabit network, they crash a product after a little over 24 hours. In the real world, assuming that the problem is stress related, rather than overload boundary related (a bad assumption), this failure may happen once every 3 months, given real world load generating capacity over a couple of OC3's is a hell of a lot less than a colocated box built to do nothing but generate load. So while it's true that it's stupid for someone to claim that something is not a bug, in the case of manual configurations at a root prompt, perhaps it's not a bug. And for boundary conditions under unrealistic load on unrealistic configurations, it may be a bug... but just because it is severity 1, doesn't mean that it's priority 1: if a user will never see it, it has zero priority, no matter what it does to the box. In fact, if people take one lesson from this thread, it should be this: severity is not the same things as importance, is not the same thing as priority. There's a process called "requirements tracking"; it keeps people from implementing things customers aren't asking for, and it keeps people from testing things in ways that are not harmonious with how customers will actually use the product. Dan was wrong to claim that it wasn't a bug (I can see his reasons for wanting to make this claim, given his "posted reward" thing), but it was equally wrong to expect the code to work in that configuration for which it was never designed: I never expect my car to work in the water, or my boat to work on the freeway, even though both are vehicles... At some point, you have to cut your losses, and say that "the user is attempting to use the product for a purpose for which it was never designed, Sorry". -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-chat" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3CA4FA19.F72624A0>