From owner-freebsd-bugs@FreeBSD.ORG Thu Dec 29 11:30:09 2005 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5EEE716A41F for ; Thu, 29 Dec 2005 11:30:09 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2229A43D5E for ; Thu, 29 Dec 2005 11:30:05 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id jBTBU5hA088959 for ; Thu, 29 Dec 2005 11:30:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id jBTBU51u088958; Thu, 29 Dec 2005 11:30:05 GMT (envelope-from gnats) Resent-Date: Thu, 29 Dec 2005 11:30:05 GMT Resent-Message-Id: <200512291130.jBTBU51u088958@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Forrest W Christian Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8DF8016A41F for ; Thu, 29 Dec 2005 11:20:12 +0000 (GMT) (envelope-from fwc@mt.net) Received: from mt.net (mail.MT.net [206.127.64.140]) by mx1.FreeBSD.org (Postfix) with SMTP id C141C43D5E for ; Thu, 29 Dec 2005 11:20:11 +0000 (GMT) (envelope-from fwc@mt.net) Received: (qmail 79783 invoked from network); 29 Dec 2005 11:20:09 -0000 Received: from unknown (HELO backup.mt.net) (206.127.64.132) by mail.mt.net with SMTP; 29 Dec 2005 11:20:09 -0000 Received: by backup.mt.net (sSMTP sendmail emulation); Thu, 29 Dec 2005 11:20:03 +0000 Message-Id: <20051229112011.C141C43D5E@mx1.FreeBSD.org> Date: Thu, 29 Dec 2005 11:20:03 +0000 From: Forrest W Christian To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: Subject: bin/91049: Patch to make dumps more rsync-friendly X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Forrest W Christian List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Dec 2005 11:30:09 -0000 >Number: 91049 >Category: bin >Synopsis: Patch to make dumps more rsync-friendly >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Thu Dec 29 11:30:04 GMT 2005 >Closed-Date: >Last-Modified: >Originator: Forrest Christian >Release: FreeBSD 6.0-RELEASE i386 >Organization: Montana Internet Corporation >Environment: System: FreeBSD backup.mt.net 6.0-RELEASE FreeBSD 6.0-RELEASE #6: Tue Dec 13 00:11:49 UTC 2005 root@backup.mt.net:/usr/src/sys/i386/compile/BACKUP i386 >Description: When using dump to generate level 0 dumps which are then rsync'ed to a remote machine, the fact that the dump date is stored with each header (inode) record makes rsync significantly less efficient than necessary. This also applies to inode access times when they are not important data to retain. When implementing an offsite backup solution of this type, these dates in particular are not important, especially if it prevents effective offsite backups. >How-To-Repeat: generate a level 0 dump. rsync to remote location. Generate another level 0 dump, rsync again and note the large amount of data being transferred just because of the date change. >Fix: Patch to add -r (don't store dump dates) and -R (don't store dump dates and access times) options is attached. This is against the 6.0-release version of the dump source (it this is the current revision of the code besides a minor manual change). --- orig/dump.8 Thu Dec 29 10:25:32 2005 +++ ./dump.8 Thu Dec 29 11:09:58 2005 @@ -38,7 +38,7 @@ .Nd file system backup .Sh SYNOPSIS .Nm -.Op Fl 0123456789acLnSu +.Op Fl 0123456789acLnrRSu .Op Fl B Ar records .Op Fl b Ar blocksize .Op Fl C Ar cachesize @@ -268,6 +268,19 @@ prompts for a new tape. It is recommended to be a bit conservative on this option. The default tape length is 2300 feet. +.It Fl r +Be rsync-friendly. Normally dump stores the date of the current +and prior dump in numerous places throughout the dump. This causes +problems when rsync or another incremental file transfer program is +used to update a remote copy of a level 0 dump, since the date changes +for each dump. This option sets both dates to the epoch, permitting +rsync to be much more efficient when transferring a dump file. +.It Fl R +Be even more rsync-friendly. This option disables the storage of the +actual inode access time (storing it instead as the epoch). +This option also sets -r. This permits rsync to be even more efficient +when transferring dumps generated from filesystems with numerous files +which are not changing other than their access times. .It Fl T Ar date Use the specified date as the starting time for the dump instead of the time determined from looking in --- orig/dump.h Thu Dec 29 10:25:32 2005 +++ ./dump.h Thu Dec 29 10:47:01 2005 @@ -76,6 +76,8 @@ int unlimited; /* if set, write to end of medium */ int cachesize; /* size of block cache in bytes */ +int rsync_friendly; /* be friendly with rsync */ +int rsync_morefriendly; /* be even more friendly with rsync */ int notify; /* notify operator flag */ int blockswritten; /* number of blocks written on current tape */ int tapeno; /* current tape number */ --- orig/main.c Thu Dec 29 10:25:32 2005 +++ ./main.c Thu Dec 29 11:07:05 2005 @@ -117,13 +117,15 @@ if (TP_BSIZE / DEV_BSIZE == 0 || TP_BSIZE % DEV_BSIZE != 0) quit("TP_BSIZE must be a multiple of DEV_BSIZE\n"); level = '0'; + rsync_friendly = 0; + rsync_morefriendly = 0; if (argc < 2) usage(); obsolete(&argc, &argv); while ((ch = getopt(argc, argv, - "0123456789aB:b:C:cD:d:f:h:LnP:Ss:T:uWw")) != -1) + "0123456789aB:b:C:cD:d:f:h:LnP:RrSs:T:uWw")) != -1) switch (ch) { /* dump level */ case '0': case '1': case '2': case '3': case '4': @@ -189,6 +191,15 @@ popenout = optarg; break; + case 'r': /*store slightly less data to be friendly with rsync*/ + rsync_friendly = 1; + break; + + case 'R': /*store even less data to be even more friendly with rsync*/ + rsync_friendly = 1; + rsync_morefriendly = 1; + break; + case 'S': /* exit after estimating # of tapes */ just_estimate = 1; break; @@ -236,6 +247,10 @@ (void)fprintf(stderr, "\n"); exit(X_STARTUP); } + if (rsync_friendly && (level>'0')) { + (void)fprintf(stderr, "rsync friendly options should only be used with level 0 dumps.\n"); + exit(X_STARTUP); + } if (Tflag && uflag) { (void)fprintf(stderr, "You cannot use the T and u flags together.\n"); @@ -383,6 +398,11 @@ spcl.c_level = level - '0'; spcl.c_type = TS_TAPE; + if (rsync_friendly) { /*don't store real dump times*/ + spcl.c_date = 0; + spcl.c_ddate = 0; + } + if (spcl.c_date == 0) { tmsg = "the epoch\n"; } else { @@ -391,7 +411,7 @@ } msg("Date of this level %c dump: %s", level, tmsg); - if (!Tflag) + if ((!Tflag)&&(!rsync_friendly)) getdumptime(); /* /etc/dumpdates snarfed */ if (spcl.c_ddate == 0) { tmsg = "the epoch\n"; --- orig/traverse.c Thu Dec 29 10:25:32 2005 +++ ./traverse.c Thu Dec 29 11:05:01 2005 @@ -682,6 +682,10 @@ { int32_t sum, cnt, *lp; + if (rsync_morefriendly) { + spcl.c_atime=0; + spcl.c_atimensec=0; + } spcl.c_inumber = ino; spcl.c_magic = FS_UFS2_MAGIC; spcl.c_checksum = 0; >Release-Note: >Audit-Trail: >Unformatted: