Date: Thu, 21 Jul 2005 12:00:31 +0100 (BST) From: Robert Watson <rwatson@FreeBSD.org> To: Alexey Yakimovich <aiy@ferens.net> Cc: freebsd-stable@freebsd.org Subject: Re: Quality of FreeBSD Message-ID: <20050721113927.T97888@fledge.watson.org> In-Reply-To: <1121917413.4895.47.camel@localhost.localdomain> References: <1121917413.4895.47.camel@localhost.localdomain>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 20 Jul 2005, Alexey Yakimovich wrote: > My advice to FreeBSD release engineering team: > - do more testing; > - have it tested with hardware what was published in "Hardware Notes"; > - do not release it for production if it is not in production quality; > - reread again what was written by yourself regarding 4.4 release > quality. > I wish to say more. > > This mail was written because I like FreeBSD and I want to continue > using it. And wouldn't mind to wait longer for real production quality > releases instead of start using something else. And please, I know, it's > open source project. While I agree more testing always helps, and that there are some fairly concete ways we can work to improve testing, there are also some practical realities to how software testing happens, especially for complex software products running on diverse hardware. I have a question for you though: Have you tried, and do you plan to try, our 6.0 test releases before 6.0-RELEASE goes out the door? Specifically, on the hardware you know you're having problems with 5.4 on? The way hardware gets tested is that people who have the hardware run the software on it under a variety of loads, and see if it works. Since a volunteer project of a couple of hundred developers can't buy all known past and future hardware, we have to rely on hardware vendors, software resellers, and FreeBSD users to do some of the testing. In order for that testing to affect a release, it must happen before the release goes out the door, rather than afterwards. And it has to happen sufficiently in advance of the release that someone can do something about the results of failed testing. If hardware isn't tested before the releasee, then inevitably people with that untested hardware are more likely to experience problems. This means that the best way to help us support your hardware is to run our test releases with useful workloads, and then provide feedback if/when they don't work. I realize you're providing feedback now on the 5.x branch, but what you may or may not know is that in the 6.x branch, we have a significant update to the ATA code that may get merged to 5.x, if it proves to be as much better as we hope. This means that we need you to test the future code, not the current code, in order to fix the problems you are experiencing. 90% of useful FreeBSD testing happens when large FreeBSD consumers take release of FreeBSD and deploy them in their testbeds and real-world environments, and find the bugs through the application of high levels of load and obscure hardware configurations. This is why later FreeBSD releases along a -STABLE branch are typically much more stable than earlier ones -- the code has run on millions of machines for untold amounts of load, instead of the thousand or so with a very selected load it's likely to run on during development. This is how all software vendors work, really -- be it Microsoft, or Apple, old-style UNIX vendors, or any of the Linux vendors. Some set of users sits on the bleeding edge and shakes out the early problems, and then the rest of the user base suffers through the later versions to shake out more subtle problems that gradually get resolved. The FreeBSD Project is working on moving towards a more formal testing regimen. This change will help shake out software bugs relating to workload -- i.e., IP stack bugs, file system bugs, etc. But the chances of it having a significant impact on broad hardware testing is very low. So if you have non-production instances of your production hardware, and can reproduce the workloads of your production environment on that hardware, what we would love you to do is run 6-CURRENT on it and tell us if that works better. If it does, then it's a question of back-porting the functionality (if possible) to 5.x. If it doesn't, then we can fix the problem in the active development tree, then merge as makes sense. 4.x became a great success after a quite shaking 3.x release branch, and after some bumps early in 4.x. It got there because of a lot of testing and improvement resulting from production experience. If you didn't have problems with 3.x and 4.x, it's because someone else got there first. The reason I suggest waiting for BETA2 is that BETA2 will have cleaned up support for running 5.x applications. Specifically, there are one or two system calls that have changed in 6.x, and require COMPAT_FREEBSD5 to be compiled into the kernel, which it wasn't in BETA1. Likewise, a number of library version bumps and compatibility pieces will be in BETA2. This will make it easier to test 5.x application workloads on a 6.x install. We take the concerns you've expressed seriously, and you should know that every FreeBSD developer I've talked with in the last few years has been talking about how to improve 5.x stability. The challenge has been to integrate the agressive feature set improvement in 5.x with our stability goals. Much of that improvement has happened for 6.x, and I think you'll find that you're much happier with the general level of testing and support there. This was possible because people running 5.x have provided us a lot of detailed feedback and bug reporting. 6.x is much less agressive in terms of feature set, and cleans up many of the architectural changes that made 5.x such a feature-rich releasee. Your feedback on 6.x sooner rather than latter will improve the quality of the 6.x release, and we'd appreciate it greatly if you could help us test it! Robert N M Watson
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050721113927.T97888>