Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 2 Jan 2016 21:07:15 +1100 (EST)
From:      Bruce Evans <brde@optusnet.com.au>
To:        Allan Jude <allanjude@freebsd.org>
Cc:        "Jonathan T. Looney" <jtl@freebsd.org>, src-committers@freebsd.org,  svn-src-all@freebsd.org, svn-src-head@freebsd.org
Subject:   Re: svn commit: r292955 - head/lib/libmd
Message-ID:  <20160102210313.M934@besplex.bde.org>
In-Reply-To: <56876DF1.4030807@freebsd.org>
References:  <201512301804.tBUI4oGp065466@repo.freebsd.org> <20151231115651.R995@besplex.bde.org> <20151231143314.Y1520@besplex.bde.org> <5684D606.3080609@freebsd.org> <D2AADE36.4E8D4%jlooney@juniper.net> <56857911.5010205@freebsd.org> <56876DF1.4030807@freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 2 Jan 2016, Allan Jude wrote:

> On 2015-12-31 13:50, Allan Jude wrote:
>> On 2015-12-31 13:32, Jonathan T. Looney wrote:
>>> On 12/31/15, 2:15 AM, "Allan Jude" <allanjude@freebsd.org> wrote:
>>>
>>>> It seems these problems also slow things down, a lot:
>>>>
>>>> # time md5 /media/md5test/bigdata
>>>> MD5 (/media/md5test/bigdata) = 6afad0bf5d8318093e943229be05be67
>>>> 4.310u 3.476s 0:07.79 99.8%     20+167k 0+0io 0pf+0w
>>>> # time env LD_PRELOAD=/usr/obj/media/svn/md5/head/tmp/lib/libmd.so
>>>> /usr/obj/media/svn/md5/head/sbin/md5/md5 /media/md5test/bigdata
>>>> MD5 (/media/md5test/bigdata) = 6afad0bf5d8318093e943229be05be67
>>>> 4.133u 0.354s 0:04.49 99.7%     20+167k 1+0io 0pf+0w
>>>>
>>>> (file is fully cached in ZFS ARC, dd reads it at 11GB/s)
>>>>
>>>> Will investigate more tomorrow.
>>>
>>> md5 will be slower than dd due to the extra processing it needs to do to
>>> generate the hash. I suspect that explains the difference you're seeing
>>> between those utilities.
>>
>> Sorry, you missed my point here.
>>
>> I replaced MDXFile() with the implementation included in my earlier
>> email. Using the newer libmd with that code, cut the time to md5 the
>> SAME data down a lot. I need to do a more scientific test on a box that
>> isn't doing other stuff still though.
>>
>> The comment about dd doing 11GB/s, was just to clarify that I wasn't
>> reading the file from disk, which would introduce other variables.
>
> I found the cause of my bogus benchmark, the world on my test machine
> was just old enough to be missing jmg@'s bufsize patch.
>
> Now the difference is about 1 second on a 2GB file, so ignore my
> foolishness.

That patch is surprisingly new.

The main slowness that I complained about was for the other path in md5
that must be used for special files.  That uses stdio so it suffers from
stdio trusting st_blksize.  But st_blksize is rarely as small as the old
size BUFSIZ in MDXFile.

Bruce



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160102210313.M934>