Date: Wed, 29 Dec 2010 03:32:47 -0800 From: Garrett Cooper <yanegomi@gmail.com> To: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> Cc: "freebsd-arch@freebsd.org" <freebsd-arch@freebsd.org> Subject: Re: Schedule for releases Message-ID: <AANLkTinW9sBpR2VrZUCQw-DXpTHqDG787pT6ALoXiU1D@mail.gmail.com> In-Reply-To: <20101227131140.H6126@maildrop.int.zabbadoz.net> References: <DB4D8AC7-25D6-4901-BBF9-77BEB956840B@cederstrand.dk> <201012220852.oBM8q2Qi039123@lurza.secnetix.de> <20101222123834.GN23098@acme.spoerlein.net> <20101227131140.H6126@maildrop.int.zabbadoz.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Dec 27, 2010, at 6:47 AM, "Bjoern A. Zeeb"
<bzeeb-lists@lists.zabbadoz.net> wrote:
> On Wed, 22 Dec 2010, Ulrich Sp=F6rlein wrote:
>
> Hi,
>
>> I think this is the core "problem". Statistics[1] show, that most
>> developers run some form of -CURRENT and
> ...
>> [1] I just made this statistic up.
>
> and I think you are just plain wrong here. Seriously I would bet that
> >75% of the developers do not run some sort of head for their
> day-to-day work. They might use it for compile (and boot and maybe
> sometimes even some more) testing, they might run it in a VM, or a lab
> machine but not on their servers, not on their notebooks and not on
> their desktops they work with daily (and neither would I expect most
> consumers of FreeBSD unfortunately).
Maybe I'm just crazy but I run head on all of my boxes and netboot
releng whenever necessary..,
> I am still not convinced that whatever development model people and
> companies use (and I heard of in here) is better than to just devel
> on HEAD and if it works there merge it and backport it to your release
> branch for QA and shipping.
Yeah, I don't develop that way though except at home with my own
personal code; then again I don't hack stuff as much in that respect
as other folks do in my group.
> This might be a bit more work in the QA cycle but it saves you PYs on
> updating branches every couple of years and in addition allows you to
> start to cut a product from any branch {HEAD..your oldest} in theory
> whenever you want rather than being unable to deliver for half a year
> or a year. Our naming scheme doesn't matter much in that case. You
> lay your tags and start your branches yourself.
Depends. Some folks don't lay your own tags and whatever and try to
leverage as much from the project's ports and packaging infrastructure
so we don't have to reroll packages (this caveat fails though if the
base system is touched, ie bzip2, OpenSSH, OpenSSL, tcpdump, zlib,
etc). Some things are worth contributing back to releases and other
things may not be.
> Can you do what you ask us to do? Develop on HEAD and keep your own
> stuff going all the way down to 8, 7 and 6?
Parallel development after writing automated tests for non-device
driver specific items is the only way to scale here, for all parties
:/.
> In addition if you work on HEAD, you find problems as they occur and
> not years later when developers have long moved on to other projects
> and it's a pain for them to task swap back to whatever for a branch
> that indeed is only barely supported anymore.
This problem isn't going to go away unless the statement above remains
true and developers in companies are competent in the areas needing
merging and competent in FreeBSD.
> I know it won't help you with the backporting of new stuff like
> drivers to kind of unsupported release branches the way you would like
> to but the last paragraph applies there as well. It might be a lot
> easier when just done along initially.
Sometimes though (take mfi for example), backporting new driver
support completely breaks existing ABI compatibility for the driver --
for example -- because structures change. It would be nice if there
was a balance better struck for backwards compatibility vs new
hardware support, but often times it seems more difficult in areas
than it should be.
> Having fixed a bug of mine with one of the last 10 commits that
> happened for RELENG_6, I know the pain. I have also previously agreed
> with people to not commit changes anymore for the sake that they'd
> never get the testing they needed - those patches are avail though,
> in GNATS, ...
>
>
> I know that HEAD is broken once in a while but I think we got a lot
> better than it sometimes used to be (and similar things apply to stable
> branches). I have cut images from stable branches in the past; I have
> given up on that a while ago; I am actually only running HEAD these
> days (home and production) for the sake of eating that dogfood but
> also to get all the advantages. I do my own tweaks to it as well, like
> you would do to your products but I try to keep them to a minimum like
> I would expect you to as well.
>
>
> I would love to hear from people who previously hit the pitfalls and
> decided not to go with HEAD again. Why didn't you do it? Do you
> regret it? Will you change "next time"?
The only time that I really regretted running HEAD for an extended
period of time was when the VM changes were first being done for amd64
this year, when it broke the nvidia driver on my machine to the point
that I couldn't work with my desktop machine on real work I needed to
complete; I just had to sit back, probe stability periodically and
finally I jumped in July when the dust settled, but it was painful for
multiple reasons doing that when other bugfixes/enhancements were
being made to the system that could have made my life a lot easier.
> We still lack the parts that would tell us something in the last week
> or last 24 hours caused a regression that made my TCP/NFS/ZFS/UFS/<you
> name it> n% slower. Kris had been doing a good job in the past but as
> time shows we need more people, different setups, ...
I think you hit a good point. We cannot abandon the wealth of soak
testing in a large user base, but in order to ensure that we're doing
a good job for the FreeBSD community we need to ensure that we're
meeting at least the minimum bar for testing (whatever that might be)
to ensure that whatever we're pushing back is reasonably ok. I think
everyone on here can agree with that point.
> It's not only "compiles", "boots", but also the formerly in this thread
> mentioned "works correctly" and in addition to that the "works well as
> expected" or "works better than before" - hopefully;).
I usually do:
1. Compile.
2. Load.
3. Functional smoke test.
4. More focused probing.
Your testing may vary.
> I think this is a community problem as well. We need to have those
> things show up like quarterly status reports, like nanog gets the
> weekly CIDR reports, like there used to be "Internet monthly
> reports" (or do they still exist?), like tinderbox emails come in, ...
Quarterly is too long (unless you're referring to the performance
results). We need tests running a lot more frequently to keep up with
the rate of development in the project.
> If you do your daily/weekly/monthly regression tests for your
> products you can catch that. If you run HEAD, you'll also catch it
> timely to avoid binary searches of multiple quarters or years.
> Some of you have the infrastructure and I can understand that you cannot
> share (most of) it but you could run it on plain FreeBSD as well and
> show us the reports? Consider to do that regularly (it doesn't have to
> be daily but maybe (bi-)weekly or monthly). "Budget" for it in terms of
> infrastructure and employee time. It'll probably save you time (and
> with that money) in the end and help us to improve the solid
> foundation you are building your products on.
Sad part of this statement too is that some companies (like mine) have
a separation of development and QA, and due to the way that QA's
staffed they often times do manual testing in place of automated
testing, which isn't acceptable in the project. Also, I don't know if
it's necessarily a good idea to post results as there could be a large
degree of fud in them based on what features were added between
releases X, Y, and Z, and what modifications were made between those
releases, for what little it might be worth.
Thanks,
-Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTinW9sBpR2VrZUCQw-DXpTHqDG787pT6ALoXiU1D>