Date: Sat, 5 Jan 2013 17:09:18 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: Dominic Fandrey <kamikaze@bsdforen.de> Cc: FreeBSD <freebsd-stable@freebsd.org>, Chris Rees <utisoft@gmail.com> Subject: Re: Post 9.1 stable file system problems Message-ID: <20130105150918.GR82219@kib.kiev.ua> In-Reply-To: <20130101155806.GU82219@kib.kiev.ua> References: <50E225DF.3090004@bsdforen.de> <CADLo838mUdr96zQw2bTPUFWwUNoF=Zb4akEL6FfasQDOW5tN8A@mail.gmail.com> <50E23283.8010407@bsdforen.de> <50E23647.6000309@bsdforen.de> <20130101065145.GT82219@kib.kiev.ua> <50E2E720.3040803@bsdforen.de> <20130101155806.GU82219@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
--3w1+XHTy8jVCESz1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 01, 2013 at 05:58:06PM +0200, Konstantin Belousov wrote: > On Tue, Jan 01, 2013 at 02:39:44PM +0100, Dominic Fandrey wrote: > > On 01/01/2013 07:51, Konstantin Belousov wrote: > > > On Tue, Jan 01, 2013 at 02:05:11AM +0100, Dominic Fandrey wrote: > > >> On 01/01/2013 01:49, Dominic Fandrey wrote: > > >>> On 01/01/2013 01:29, Chris Rees wrote: > > >>>> On 1 Jan 2013 00:01, "Dominic Fandrey" <kamikaze@bsdforen.de> wrot= e: > > >>>>> > > >>>>> I have a Tinderbox that I just updated to the current RELENG_9. > > >>>>> Following the update build times for packages have increased by a > > >>>>> factor between 5 and 20. I.e. I have packages that used to build = in > > >>>>> 5 minutes and now take an hour. > > >>>>> > > >>>>> I'm suspecting the file system ever since I saw that the majority= of CPU > > >>>>> load was caused by ls when I looked at top (more than 2 minutes o= f CPU > > >>>>> time were counted that moment). The majority of the time most of = the CPU > > >>>>> load is caused by bsdtar, pkg_add, qmake-qt4, etc. Without except= ion > > >>>>> tools that access a lot of files. > > >>>>> > > >>>>> The file system on which packages are built is nullfs mounted from > > >>>>> an async mounted UFS. I turned async off, to no avail. > > >>>>> > > >>>>> /usr/src/UPDATING says that there were nullfs optimisations. So I > > >>>>> think this is where the problem originates. I might hack the tind= erbox to > > >>>>> use 'ln -s' or set it up for NFS to verify this. > > >>>> > > >>>> Is your kernel newer than the Jail? The converse causes problems. > > >>> > > >>> I ran makeJail for all jails after updating. > Did you rebuild your modules together with the new kernel ? >=20 > > >>> > > >>> I also seem to have similar problems when building in the host-syst= em. > > >>> The unzip for openjdk-7 has just passed the 11 minutes CPU time mar= k. > > >>> On my notebook it takes less than 10 seconds. > > >> > > >> Just set WRKOBJDIRPREFIX to a tmpfs on the Tinderbox host system > > >> and the extract takes less than a second. Originally WRKOBJDIRPREFIX > > >> also pointed to a nullfs mount. > > >> > > >> Afterwards I pointed WRKOBJDIRPREFIX to a UFS file system (without > > >> nullfs involvement). The entire make extract took 20s. > > >> > > >> So still faster by at least factor 30 than running it on a nullfs mo= unt > > >> (I eventually SIGINTed so I don't know how long it would've run). > > >=20 > > > Start providing some useful debugging information ? > >=20 > > That one might be interesting. It's all system time: > >=20 > > # time -lh make extract > > =3D=3D=3D> License GPLv2 accepted by the user > > =3D=3D=3D> Found saved configuration for openjdk-7.9.05_1 > > =3D=3D=3D> Extracting for openjdk-7.9.05_2 > > =3D> SHA256 Checksum OK for openjdk-7u6-fcs-src-b24-09_aug_2012.zip. > > =3D> SHA256 Checksum OK for apache-ant-1.8.4-bin.zip. > > =3D=3D=3D> openjdk-7.9.05_2 depends on file: /usr/local/bin/unzip - f= ound > > ^Ctime: command terminated abnormally > > 4m29.30s real 3.03s user 4m22.55s sys > > 5008 maximum resident set size > > 135 average shared memory size > > 2932 average unshared data size > > 127 average unshared stack size > > 7772 page reclaims > > 0 page faults > > 0 swaps > > 19 block input operations > > 101 block output operations > > 0 messages sent > > 0 messages received > > 41 signals received > > 1597 voluntary context switches > > 16590 involuntary context switches >=20 > Ok, from your mount -v output, are the three nullfs mounts the only > nullfs mount ever used ? >=20 > Is it only unzip which demostrates the silly behaviour ? Or does it > happen with any program ? E.g., does ls(1) or sha1 on the nullfs mount > also slow ? >=20 > Could you try some low-tech profiling on the slow program. For instance, > you could run ktrace/kdump -R to see which syscalls are slow. >=20 > Most darkly part of your report for me, is that I also use nullfs-backed > jails both on HEAD and stable/9, with bigger scale, and I do not have > an issue. I just did > pooma32% time unzip -q /usr/local/arch/freebsd/distfiles/openjdk-7u6-fcs-= src-b24-09_aug_2012.zip > unzip -q 3.25s user 23.77s system 78% cpu 34.482 total > over nullfs mount of > /usr/home on /usr/sfw/local8/opt/pooma32/usr/home (nullfs, local). >=20 > Please try the following patch, which changes nullfs behaviour to be > non-cached by default. You could turn on the caching with the 'mount -t > nullfs -o cache from to' mounting command. I am interested if use/non-use > of -o cache makes a difference for you. Ping. Any update ? --3w1+XHTy8jVCESz1 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ6EIdAAoJEJDCuSvBvK1BLmEP/1HBfqeXHoyEw4s46y25dzUH lPryrvLTKIaQF/+OKMVtQ/4IRVBdrWDoT8vVZa9zYYp0LSv75KZjiNhrmcsnxqUK TQ1ne6LxlRPrxSliC0osqLjohZzBbt0wZg+QcsRQGmgv2Pq/IZprXPRHEPnCl/+e VjlighkkZjENs1fj6LY+XlFQnQO3s9vRouIaF1H35aQ8f/CWl3By32/h0qoIYCnP iHQsqSyjoshOTrNzj9Kg1UU1zcVNRkTnWLwebh57aq8wzrhMGB4uSatig6AL+Qa3 pk/x+b7dQaFX5pGRbUisJSXQG3Mui60oxBhYMXWhmVzFSv6JWqs8C2/sDVmZpDtF cgpS8ivOJUZpm+nrGkhegqRog5+9cvlE9hckUSWm514DlV362xIRUmE0800M8Imc Ny92I1qsJHVmH+UaTVH+g+O7jQ8lzLbyvKkxvTcHc7shrcqhodyzz+BwJIq+rxZr VVhRP7r7GKDxNSfmITlcOjx2n+yV/HvPio9KLAx46MXlDClT9NRvxNMsKlCMiSt2 +sKDPf+dSw4xf8soKXdLe9198JM8yO9N7Kwg+CdOQ4EYZ6vTsz6FwfCNPyxHN7RN LE1fNcCxtuWVKtscukuohsqc+8RqxMV17eXC0ORxGhBKjuTxFoftbogohNSgR7L+ kaD4223z733KXKRUdYve =6z4/ -----END PGP SIGNATURE----- --3w1+XHTy8jVCESz1--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130105150918.GR82219>