From owner-freebsd-questions@FreeBSD.ORG Thu Mar 13 16:55:36 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 41C13C75 for ; Thu, 13 Mar 2014 16:55:36 +0000 (UTC) Received: from wp376.webpack.hosteurope.de (wp376.webpack.hosteurope.de [IPv6:2a01:488:42::50ed:8591]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0337F1F4 for ; Thu, 13 Mar 2014 16:55:36 +0000 (UTC) Received: from tmo-101-127.customers.d1-online.com ([80.187.101.127] helo=dijkstra-old.hb22.cruwe.de); authenticated by wp376.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) id 1WO8uq-0005cf-W6; Thu, 13 Mar 2014 17:55:33 +0100 Date: Thu, 13 Mar 2014 17:55:26 +0100 From: "Christopher J. Ruwe" To: Subject: cannot witness rsync delta-algorithm Message-ID: <20140313175526.1ea13c4f@dijkstra-old.hb22.cruwe.de> X-Mailer: Claws Mail 3.9.3 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-bounce-key: webpack.hosteurope.de;cjr@cruwe.de;1394729736;8095cd45; X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Mar 2014 16:55:36 -0000 Having read the rsync delta transfer paper included with the rsync documentation (Andrew Tridgell, Paul Mackerras: The Rsync Algorithm), I wanted to witness this in practice.=20 (rsync should use rolling MD5 checksums on source and target file chunks to determine if the data is present on remote target without having to transfer the data.) I do not succeed. I have setup a file 'here' using dd and /dev/zero: [cjr@dijkstra:~/rsync-exp]$ dd if=3D/dev/zero of=3Dhere bs=3D1M count=3D100 100+0 records in 100+0 records out 104857600 bytes transferred in 0.101729 secs (1030755108 bytes/sec) I used 100 MB to ensure that changing delta-window-size would not be an issue. I then used rsync to copy 'here' to 'there'. Afterwards, I changed the last MB of here: [cjr@dijkstra:~/rsync-exp]$ rsync -vh --inplace here there sent 104.88M bytes received 35 bytes 69.92M bytes/sec total size is 104.86M speedup is 1.00 [cjr@dijkstra-old:cb-krb/rsync-exp]$ diff here there =20 here and there obviously have equal content. Then, I changed the last 1 MB of here: [cjr@dijkstra:~/rsync-exp]$ dd if=3D/dev/random of=3Dhere bs=3D1M count=3D1 seek=3D99 1+0 records in 1+0 records out 1048576 bytes transferred in 0.026790 secs (39140716 bytes/sec) 'here' and 'there' differ: [cjr@dijkstra:~/rsync-exp]$ diff here there Files here and there differ =46rom what I understand from rsync workings, I would expect another rsync from 'here' to 'there' only to copy some MBs due to possibly different window sizes, but significantly less than 100MB. Yet, it does not. Why? [cjr@dijkstra:~/rsync-exp]$ rsync -vhc --inplace here there sent 104.88M bytes received 35 bytes 69.92M bytes/sec total size is 104.86M speedup is 1.00 [cjr@dijkstra:~/rsync-exp]$ diff here there =20 'here' has been copied as a whole to 'there'. The '-W' flag has not been set! My rsync has been compiled thus: [cjr@dijkstra:~]$ pkg info rsync rsync-3.1.0_1 Name : rsync Version : 3.1.0_1 Installed on : Mon Mar 10 21:23:56 CET 2014 Origin : net/rsync Architecture : freebsd:10:x86:64 Prefix : /usr/local Categories : net ipv6 Licenses : GPLv3 Maintainer : ehaupt@FreeBSD.org WWW : http://rsync.samba.org/ Comment : Network file distribution/synchronization utility Options : ACL : off ATIMES : off DOCS : on FLAGS : off ICONV : off POPT_PORT : off RENAMED : off RENFL : on SSH : on TIMELIMIT : off ZLIB_BASE : on [...] Can anybody explain this behaviour? Thanks and cheers, --=20 Christopher TZ: GMT + 1h GnuPG/GPG: 0xE8DE2C14 =20 FreeBSD 10.0-STABLE #0 r262291+cf0fa48(stable/10): Sat Feb 22 02:01:26 CET 2014 cjr@dijkstra.hb22.cruwe.de:/usr/obj/usr/src/sys/GENERIC=20 =20 Punctuation matters: "Lets eat Grandma or Lets eat, Grandma" - Punctuation saves lives. "A panda eats shoots and leaves" or "A panda eats, shoots, and leaves" - Punctuation teaches proper biology. "With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead." (RFC 1925)