From owner-freebsd-stable@FreeBSD.ORG Tue Jan 1 06:51:53 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DC50C84D for ; Tue, 1 Jan 2013 06:51:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 314868FC08 for ; Tue, 1 Jan 2013 06:51:52 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id r016pj28042728; Tue, 1 Jan 2013 08:51:45 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.3 kib.kiev.ua r016pj28042728 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id r016pj6w042727; Tue, 1 Jan 2013 08:51:45 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 1 Jan 2013 08:51:45 +0200 From: Konstantin Belousov To: Dominic Fandrey Subject: Re: Post 9.1 stable file system problems Message-ID: <20130101065145.GT82219@kib.kiev.ua> References: <50E225DF.3090004@bsdforen.de> <50E23283.8010407@bsdforen.de> <50E23647.6000309@bsdforen.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="sW3EvpYZ43JhLhGX" Content-Disposition: inline In-Reply-To: <50E23647.6000309@bsdforen.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: FreeBSD , Chris Rees X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jan 2013 06:51:53 -0000 --sW3EvpYZ43JhLhGX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 01, 2013 at 02:05:11AM +0100, Dominic Fandrey wrote: > On 01/01/2013 01:49, Dominic Fandrey wrote: > > On 01/01/2013 01:29, Chris Rees wrote: > >> On 1 Jan 2013 00:01, "Dominic Fandrey" wrote: > >>> > >>> I have a Tinderbox that I just updated to the current RELENG_9. > >>> Following the update build times for packages have increased by a > >>> factor between 5 and 20. I.e. I have packages that used to build in > >>> 5 minutes and now take an hour. > >>> > >>> I'm suspecting the file system ever since I saw that the majority of = CPU > >>> load was caused by ls when I looked at top (more than 2 minutes of CPU > >>> time were counted that moment). The majority of the time most of the = CPU > >>> load is caused by bsdtar, pkg_add, qmake-qt4, etc. Without exception > >>> tools that access a lot of files. > >>> > >>> The file system on which packages are built is nullfs mounted from > >>> an async mounted UFS. I turned async off, to no avail. > >>> > >>> /usr/src/UPDATING says that there were nullfs optimisations. So I > >>> think this is where the problem originates. I might hack the tinderbo= x to > >>> use 'ln -s' or set it up for NFS to verify this. > >> > >> Is your kernel newer than the Jail? The converse causes problems. > >=20 > > I ran makeJail for all jails after updating. > >=20 > > I also seem to have similar problems when building in the host-system. > > The unzip for openjdk-7 has just passed the 11 minutes CPU time mark. > > On my notebook it takes less than 10 seconds. >=20 > Just set WRKOBJDIRPREFIX to a tmpfs on the Tinderbox host system > and the extract takes less than a second. Originally WRKOBJDIRPREFIX > also pointed to a nullfs mount. >=20 > Afterwards I pointed WRKOBJDIRPREFIX to a UFS file system (without > nullfs involvement). The entire make extract took 20s. >=20 > So still faster by at least factor 30 than running it on a nullfs mount > (I eventually SIGINTed so I don't know how long it would've run). Start providing some useful debugging information ? At least dmesg, mount -v and sysctl kern.maxvnodes, 'sysctl vfs | grep vnodes' outputs. What is shown when you press ^T while slow process runs on nullfs ? Was the ^C reaction by terminating the process instant ? --sW3EvpYZ43JhLhGX Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ4oeBAAoJEJDCuSvBvK1BbsgQAIFBrA1SGNAHFMtgQluz18eI 71W8Lp4BnRWvulu4jo62IBQV22LOjtxO273YNgbuyVHogv1GBbtkJlXdWkK1Yels CcacdYTU7tcEnRjQmlBhy3cVSYYd7CJtauvtyUjxufVDYFJdWpSLt0eK78uDKuBQ M8ITFTouSa0CDHiMaLnRztn4IJSCE31EN0oqdPXw3zcFWki1Ut4biwVbVc6Va2h4 Yyog/QR0HwJnvGBO3fh94MHO57OJRxO6z3scB45RQFj/C1j+qJ1+ahmNhkT/hjDU gcIcQQ82LTLr2IyCewk03U6pfK3MbXhxRxlDTRZHOy3Q7ADKnPildj+6RxTJIP9Z y7jldyecQ+1xr48io8YxqinHNpbCI2/Navhd6IwWFcIEc77Zr/0YnpUEZjpC/0xL hgqocy4KRrtJ7HEymImL/dkiDGlkD+HOF9AMuQLLrKQkS/0qS6GPuio0LGouBIQv Rk5KgMncrksOgGwUntJboIp9QE531+vA5PyFzzkdCp//rckAOOqqFr4lx4F/U7gI vAtPlB8hqh12cgNY09hF0i2op1Lf7cJr26RySFDJFcgat3h/QCvuwVDyHLpu7Vp+ MTHi9ZUuk1h8uXN3C3MFUWe9k+DmzJvbFTJDzfD8Fqu7J4/zgJtKDcKuVHBuAnYQ 2uJ/uU/WatywVLfVpFHv =UwcB -----END PGP SIGNATURE----- --sW3EvpYZ43JhLhGX--