Date: Sat, 16 Mar 2013 20:25:29 -0500 From: "John Mehr" <jcm@visi.com> To: <freebsd-stable@freebsd.org> Subject: Re: svn - but smaller? Message-ID: <web-12358364@mailback4.g2host.com> In-Reply-To: <20130315001321.K32142@sola.nimnet.asn.au> References: <web-11636850@mailback4.g2host.com> <513E2DA5.70200@mac.com> <web-12282796@mailback4.g2host.com> <op.wts7cnaeg7njmm@michael-think> <web-11149903@mailback3.g2host.com> <20130313152150.E32142@sola.nimnet.asn.au> <web-11167474@mailback3.g2host.com> <20130315001321.K32142@sola.nimnet.asn.au>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, 17 Mar 2013 02:14:30 +1100 (EST)
Ian Smith <smithi@nimnet.asn.au> wrote:
> On Wed, 13 Mar 2013 21:11:28 -0500, John Mehr wrote:
> > On Wed, 13 Mar 2013 16:29:45 +1100 (EST)
> > Ian Smith <smithi@nimnet.asn.au> wrote:
> svn0.us-west.freebsd.org as per example, but it's the
>closest to here
> anyway; pings avg 217.5ms stddev 0.4ms, us-east avg
>269.3ms sd 9.5ms.
> fastest_cvsup used to find cvs mirrors in Australia at
>around 55-70ms,
> so for another while I thought RTT might be the issue -
>but it's not.
I'm seeing ~110ms pings to svn0.us-west.freebsd.org.
>
> > I'll definitely put in a verbosity level that does
>exactly what you suggest.
> > Sorry about that.
>
> Not at all, and thanks! But neither, I now find, is
>logging the extra
> time issue; as mentioned elsewhere, running at -v0 makes
>next to no
> difference, 12:48 west vs 12:54 east on 4 do-nothing
>runs, 700KB/s link,
> not running in script(1) - though I think that matters
>not, or little.
>
> So I began watching top more closely, also running
>iostat -w10, noticing
> that svnup was pegging CPU at 100% with ~75% _system_?
>time for nearly 9
> minutes of the run, doing little or no disk I/O and a
>max of 130KB/s in,
> 22.6KB/s out, no stress there, growing to ~20MB
>resident, then doing a
> flurry of disk I/O at about 3-5MB/s for the last 3m40s
>odd. IOW, svnup
> here is mostly heavily CPU-bound, which I don't
>understand, but then I
> know nothing about what level of computation the
>protocol demands.
Except for the time duration of the run, the behavior
you're describing is normal. Using the svn protocol, the
program runs in three phases:
1) It issues a series of get-dir commands to the server to
get a list of all the files/directories in the repository
and stores the structure in memory. There is some disk
activity during this as it creates the directory tree (it
uses one lstat per directory and creates the directory if
it doesn't exist). On my development machine, this phase
takes about 25% of the total time of a do-nothing run.
2) Because the svn protocol doesn't include MD5 signatures
and the last-author/committed date/committed revision
fields in the results of the get-dir commands
<gripe>although they *are* included in the http protocol
responses</gripe> it has to issue a get-file request for
each file to get this information. This phase takes about
50% of the total time of a do-nothing run.
3) The downloaded MD5 signatures are for versions of the
files that do not have their $FreeBSD$ tags expanded, so
each file on the local machine must be loaded into memory
and have their $FreeBSD: ... $ tags collapsed and MD5
summed to see if there's a match. If there's no match,
the file on the server is downloaded. This phase takes up
the remaining 25% and is I/O intensive.
This three step process is further complicated because of
the huge time penalty that occurs when sending commands
one-at-a-time. To get around this, as many commands that
can fit in 32KB are grouped together and sent in bulk.
> I've done some more controlled tests since, running both
>iostat -w10 and
> netstat -w10 -dn before, during and until a bit after
>the run, attached,
> hoping these might help pinpoint any bottlenecks? This
>was running -v1
> default verbosity, hence the (not at all heavy) tty
>output numbers.
>
> Just sometimes, running slower hardware can have
>advantages! On your
> times, yours likely has more than 500% of my single-core
>P3-M's 1133MHz
> performance, which would tend to mask this issue pretty
>well for you :)
I do all of my development and test runs on a nettop with
an AMD E-350 (dual core, 1.6GHz) processor, and with the
program only utilizing one core I'd be willing to bet the
two chips are in roughly the same league. Memory and hard
drives probably aren't, and with 512MB of memory, how does
your swap utilization look?
As soon as I can get a replacement CMOS battery, I'll be
dusting off my old Athlon 2700+ system to run tests on it.
> John Baldwin has best described when c{,v}sup deletes
>(only in-tree)
> files, covered in section 'delete' in csup(1). I think
>this should
> probably be default behaviour, even if svn proper would
>do otherwise.
Will do. Setting up both a default of deleting only files
that appear in previous checkouts plus giving each user
the option of skipping specific directories from any
pruning is on the to-do list. Thanks!
>
> cheers, Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?web-12358364>