Date: Sat, 12 Jan 2013 23:13:39 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: "Simon L. B. Nielsen" <simon@FreeBSD.org> Cc: freebsd-hubs@freebsd.org, John Marshall <john.marshall@riverwillow.com.au> Subject: Re: SLOW rsync from bit0.us-west Message-ID: <20130112211339.GM2561@kib.kiev.ua> In-Reply-To: <CAC8HS2EBKhS2t8hpGZnmwzh06fKAd1d69fdHz7VU=CxZDk=iEg@mail.gmail.com> References: <50F0F2CE.9050605@riverwillow.com.au> <CAC8HS2E8KB67U_m_gWznXMZNHRCZNB%2BNe_u5BGsCnrh%2BiYxAHg@mail.gmail.com> <CAC8HS2EBKhS2t8hpGZnmwzh06fKAd1d69fdHz7VU=CxZDk=iEg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Fm6XlJPzS/ViAqAU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 12, 2013 at 07:58:18PM +0000, Simon L. B. Nielsen wrote: > On 12 January 2013 15:37, Simon L. B. Nielsen <simon@freebsd.org> wrote: > > On 12 January 2013 05:21, John Marshall > > <john.marshall@riverwillow.com.au> wrote: > >> I have been updating the gnats collection on our mirror via rsync since > >> the CVSup updates stopped but noticed this morning that the updates > >> appeared to be stalled. I killed off jobs that appeared to be stuck > >> downloading incremental file list (after about 20 minutes) but then > >> thought to run rsync manually with -vvv. What I saw then was short > >> bursts of messages like 'recv_file_name(bin/nnnn)' interspersed with > >> 15-20 seconds of nothing. tcpdump showed just a handful of packets eve= ry > >> 15 seconds or so. The update eventually finished after more than 2 hou= rs. > > > > The server which runs bit0.us-west (which is also svn0.us-west), is > > totally bogged down. > > > > CPU: 3.1% user, 0.0% nice, 96.0% system, 0.0% interrupt, 1.0% idle > > > > As it has just been updated to the latest stable/9 I'm a bit > > suspicious of that... > > > > Thanks for letting us know. I will start poking people. >=20 > Please have another look and see if things aren't better now. >=20 > It looks like we are stressing nullfs more than people had before - at > least with way more vnode / files so the server was spending all of > its time doing vnode lookups. What are your sysctl kerne.numvnodes and kern.maxvnodes ? How did you determined that lookups were the cause ? The deal with nullfs is that I MFCed a set of changes recently which allow to keep nullfs vnodes cached. As a consequence, the nullfs vnodes could be (usefully) locked shared not, which provides up to 100% increase in the speed of the metadata-intensive operations on nullfs, as measured by Peter Holm. The drawback of the change is that nullfs vnodes are cached, which effectively doubles the amount of vnodes the system operates on. I added the nocache nullfs mount option to return to the pre-cached behaviour, but this is not propagated to the stable/9 yet. Supposedly, you could use the option if increasing kern.maxvnodes is not possible. --Fm6XlJPzS/ViAqAU Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ8dIDAAoJEJDCuSvBvK1B+4kP/iNFjo8uREzRh2WyDcPzcc1c gWOHdqoZ8w/x/8DTSyFt0V3e1YNNASS7VnSA0rEhei20wEt49TL80WtgGMaeh4TD tpYJGcbMUdH0Zr/RommXUysN851AAQoj6ZKgEfnvvY7+sm8TUqO1aJa48dGf8Lgm NHXdjppY124CvzM6rflGwRZncJ7xttNsfOw3v/uyhNkZUwIcsVCHFkoGYCAUQL6S VZq51Ad6CrMotzXmHcOV2gUsauk4vzouOHX6SByiJI6S3nfMqRfajAkzSljfKJsB pFHgANO3mAdXiIaiT5A5rpbYqmpA6V7FSrlrgzdO46lIimY2Ou+NkPWs47P54Ea8 kqXPSir+BhVX2EfbnRMx4j5bTAH3uFv2Hn/o3Datt5CdyFvR0giGqgMOZZxRn43r P2obaQ9r5h+zTsUXzNIVTu3JYQaQR1skDib54rFsxPvn/BNOZyIM0Rmmv/w/pont S+i/GC+aoHvdq1iNjDEm2A4nyLKicKxBai1s6qMYc/n2KS46Scd1x+/DG+2w4Iw9 XnlAPfZJ2iYhdMBrxs8qh1ivodiMzXUrDJC8UQYF1KGq0h+xoMcAXbbUIH5FuMnh /9Ut4r1FMUWZpfdpvpznJigZA4TEPkwSn123RaqZIX0fp7lM54W7VfRWlBaW6mZS 5pll70eiLJk/i3KFfFXT =fwrS -----END PGP SIGNATURE----- --Fm6XlJPzS/ViAqAU--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130112211339.GM2561>