Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 20 Dec 1999 19:08:47 -0800
From:      Jamie Norwood <mistwolf@mushhaven.net>
To:        Jason Young <doogie@staff.accessus.net>
Cc:        freebsd-qa@FreeBSD.ORG, "'jkh@freebsd.org'" <jkh@FreeBSD.ORG>
Subject:   Re: After 3.4 finally goes out the door
Message-ID:  <19991220190847.A59425@mushhaven.net>
In-Reply-To: <ABD44D466F85D311A69900A0C900DB6BC567@staff.accessus.net>
References:  <ABD44D466F85D311A69900A0C900DB6BC567@staff.accessus.net>

next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Dec 20, 1999 at 08:57:14PM -0600, Jason Young wrote:
> 
> Well, I was trying to avoid the 'someone oughta' and 'you really should'
> type of rambling. :) I simply propose that if we could just gain some more
> time to QA the "real" release, then we'd get some truly useful work done.
> 
> As for a standardized testing methodology or procedure? IMHO, that would be
> counterproductive. Everybody's got a different usage pattern, and different
> hardware, and trying to force people into the same mold will be hard and
> ultimately bad and wrong.
> 
> For instance, if you had some script to follow in sysinstall - "test this,
> and then this, and then this, and test this" - would the sysinstall bugs
> have been found? 

Not if a full-featured script was written. Beyond, not having a script
does not mean a bug .will. be found. I simply mean for people who don't
have the knowledge or time to play with trying hard to break it, having
a set set of things everyone should look at will get the broader range
of hardware out there.

> If end-users with different hardware and configurations and
> usage patterns hadn't tested moused, would we have shipped with roller wheel
> support? No, it's people doing what they normally do, which is quite likely
> to be outside of what the developer tested. This is a Good Thing, and we
> need people doing their own unexpected, possibly weird, sometimes nonobvious
> stuff. The developers aren't total morons and the obvious stuff obviously
> works fine.

Are the two, though, mutually exclusive? I don't think a script precludes
people doing their own thing so much as helps get people who don't HAVE 
their own thing involved and useful.

> The developers already have and already run the various regression tests
> that are available. A technically savvy QA volunteer might want to run
> those, and certainly that's fine, but I don't think we're short of dedicated
> and technical savvy developers familiar with the OS and its internals. What
> we need, IMHO, is a bunch of people putting it in real world situations,
> testing it under their various application and user loads, and seeing how it
> does.

But we couldn't/shouldn't be putting this on production machines, and the
only machine I really run FreeBSD on is one I cannot reboot once or twice
a day for a week every three months. Which means I don't have much I can
really test in a real-world situation.

> We had this in the form of the 3.4-RELEASE QA volunteer team, and everybody
> did a great job with the time and resources available. I hope that jkh and
> friends can see this, and see that just a little bit of foresight we can
> make sure that the next CD that ships holds a truly solid and kickass
> release.

I agree we did great, but I definately feel we could do better. Certain
things, like sysinstall, do need to be tested by as many people as possible.
Moused, when they were putting wheel support in, would have been great to
send out a list that said 'OK, if you have a wheel mouse, do X, Y, and Z
and tell us if it works/fails and what hardware you run on'.

Jamie
> 
> > -----Original Message-----
> > From: Jamie Norwood [mailto:mistwolf@mushhaven.net]
> > Sent: Monday, December 20, 1999 8:00 PM
> > To: Jason Young
> > Cc: 'Colin'; freebsd-qa@FreeBSD.ORG; 'jkh@freebsd.org'
> > Subject: Re: After 3.4 finally goes out the door
> > 
> > 
> > I agree with this totally. I also feel someone should step forward
> > and coordinate; I didn't do more than install and see if it worked
> > as I am not familier with how things should be tested. Also, I feel
> > that some things should be made available such as CVSUP scripts
> > and such to make sure the people involved really are testing the
> > right things. Could we maybe get a 'branch' named 'qa' that would show
> > rc or qa in the uname output? That would also make things 
> > easier as we 
> > could then test that tree for the cd, and bug fixes that are essential
> > could then be made to that tree only, and normal developement could
> > continue on the -STABLE/-RELEASE tree.
> > 
> > Jamie
> > 
> > 
> > On Mon, Dec 20, 1999 at 07:50:12PM -0600, Jason Young wrote:
> > > 
> > > I think it's a good idea to keep the list around, but I 
> > don't think that the
> > > QA volunteers were really used to their full potential.
> > > 
> > > I tested when I could (early in the testing cycle), and I 
> > feel that most of
> > > my efforts were for naught, because as usual, a large 
> > percentage of the
> > > commits in the testing period were very last minute (after 
> > most or all of
> > > the QA builds).
> > > 
> > > Yeah, it's a free software project and nobody's paid, and 
> > what gets out the
> > > door is still a fine piece of work (and major kudos to jkh 
> > and the rest of
> > > the guys on that). But, I don't think much meaningful 
> > testing of the real
> > > release was accomplished due to all the really really late 
> > changes. The last
> > > minute "oh I really need such and such MFC'd" trips are 
> > killers (sysinstall,
> > > moused, you won't have to rack your brain too hard for examples).
> > > 
> > > I assume that the schedule for 3.5-RELEASE is pretty much 
> > set already. Would
> > > it be possible to back most of that schedule up about a 
> > week, and in the
> > > week before the CD is cut, put the "pretty-much-final, 
> > we-really-mean-it,
> > > won't touch it unless the release is definitely fscked" up 
> > for general FTP?
> > > If the schedules are set well in advance, nobody will be hurting or
> > > complaining of the loss of one week's time in a three or 
> > four month -STABLE
> > > release cycle. The QA volunteers need a set release to 
> > assure its quality,
> > > ya know? This is no extra work or major change for anyone.
> > > 
> > > I think that if we could get the "frozen" release candidate 
> > out there in
> > > front of people for at least a week or so, we could muchly 
> > reduce the size
> > > of our ERRATA files.
> > > 
> > > > -----Original Message-----
> > > > From: Colin [mailto:cwass99@home.com]
> > > > Sent: Monday, December 20, 1999 7:18 PM
> > > > To: freebsd-qa@FreeBSD.ORG
> > > > Subject: After 3.4 finally goes out the door
> > > > 
> > > > 
> > > >      I realize we got off to a rather slow start, but all 
> > > > things considered
> > > > better than many expected.  My question is, is there any 
> > > > intent for this list
> > > > to survive the release of 3.4, as basically a home for 
> > people with the
> > > > time/energy/hardware to talk about testing?
> > > >      I think it would be a marvelous idea, assuming 
> > > > sufficient interest of
> > > > course, to continue this with 4.0-RC when that starts the 
> > > > long arduous path
> > > > from -CURRENT ;)
> > > > 
> > > > Cheers,
> > > > Colin
> > > > 
> > > > 
> > > > To Unsubscribe: send mail to majordomo@FreeBSD.org
> > > > with "unsubscribe freebsd-qa" in the body of the message
> > > > 
> > 
> > 
> > To Unsubscribe: send mail to majordomo@FreeBSD.org
> > with "unsubscribe freebsd-qa" in the body of the message
> > 


To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-qa" in the body of the message




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?19991220190847.A59425>