From owner-freebsd-ports@freebsd.org Fri Dec 16 10:08:57 2016 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 187DCC77B04 for ; Fri, 16 Dec 2016 10:08:57 +0000 (UTC) (envelope-from mailinglists@toco-domains.de) Received: from toco-domains.de (mail.toco-domains.de [176.9.39.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 980F7163A; Fri, 16 Dec 2016 10:08:56 +0000 (UTC) (envelope-from mailinglists@toco-domains.de) Received: from [0.0.0.0] (mail.toco-domains.de [IPv6:2a01:4f8:150:50a5::6]) by toco-domains.de (Postfix) with ESMTPA id E8A0B1AAF07A; Fri, 16 Dec 2016 11:08:53 +0100 (CET) Subject: Re: (In)Stability of the Quarterly Branch To: David Demelier , Matthew Seaman References: <3e7f94efc6428181a289742d7dd627df@acheronmedia.com> <20161215170154.0ca2017914c0bb032516b413@gmail.com> Cc: freebsd-ports@freebsd.org From: Torsten Zuehlsdorff Message-ID: <7e071aa9-4fd8-9e6e-07f1-d0fa6632087a@toco-domains.de> Date: Fri, 16 Dec 2016 11:08:53 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2016 10:08:57 -0000 On 16.12.2016 08:24, David Demelier wrote: > 2016-12-15 17:25 GMT+01:00 Matthew Seaman : >> On 2016/12/15 16:01, Olivier Duchateau wrote: >>>> The problem is that there are no tests in FreeBSD ports. All source >>>> based systems I've tested: pkgsrc, FreeBSD ports, OpenBSD, Gentoo; >>>> FreeBSD is the one that have the most instability. Not to mention >>>> committers that commit without testing the port, just look at >>>> www/redmine to get your point of view on that issue. >> >>> Are your serious when you said, there're no tests on FreeBSD ports. I >>> can tell you Xfce ports are tested with FreeBSD i386 9.3 and amd64 >>> 11.0 machines (on real hardware, no virtualization), and on poudriere >>> with Gtk+ 3.20 (port version is not not in ports tree, it's defaut >>> toolkits for the next stable release 4.14). >>> >>> For the LXQt desktop is the same thing (tested with official ports >>> tree Qt5 and which one in plasma5 branch (on KDE repository). >>> >>> I'm also working on the Pantheon desktop (desktop environment of >>> Elementary OS, I use Vala 0.30.2 and Vala 0.34.4, in order to test >>> stability of applications. >>> >>> I use also OpenBSD macppc, it's piece of shit. WebKit browers are >>> broken, Xfce components crash often, stable branch is outdated, fix >>> are not propagated in stable branch. Personally I prefer the FreeBSD >>> scheme, because I'm sure it's quite stable. >> >> Most port committers will run compile tests any time they update a port: >> the better ones will test compilation on all supported FreeBSD versions >> and all hardware architectures they have access to (ie. generally i386 >> and amd64). > > I'm not talking about being sure that the port builds, but that the > software works. This is a next step that is too often forgotten. For > example I remember several years ago having a problem with > audio/mumble. The port was building fine, the window opened fine but > it was impossible to speak because there was a problem regarding the > CELT libraries IIRC. I really support this, but from a committer perspective its quite impossible. Often enough we just have no idea how the port needs to work - so we must trust the maintainer. But too often there is none. >> Additionally the package build cluster will rebuild any modified ports >> within a few days for all of the OS versions and architectures the >> project tries to provide ports for: that's yet another level of >> validating the coding of the port itself. >> >> However, I believe the OP's point is that *we do not routinely run the >> software's own built-in regression tests for the packages we succeed in >> building*. This is something that is slowly coming. For instance, you >> can run 'make test' for many python, ruby or perl packages and see those >> tests being run. TEST_DEPENDS is pretty much standardized as the way to >> install dependencies required for testing nowadays. >> > > Yes, I fully understand the requirements of such tests. I just would > like that maintainers test the port by building it and *by running* > them. This is time consuming for sure, but if the maintainers have no > time, then just keep an old but fully working version :-) There are two problems with this: 1) Many ports have no maintainer 2) Even more ports have no tests at all I'm a big fan for testing. I test every software i wrote and in every firm i worked i had the lowest bug count. But if you try to teach other people to write tests too you hit a hard wall... So we need to address multiple problems. I'm currently working on an project to improve the management of the ports-tree. Hopefully i can free some time for all so we can test the runtime much better. Greetings, Torsten