Date: Thu, 29 Dec 2005 11:20:03 +0000 From: Forrest W Christian <fwc@mt.net> To: FreeBSD-gnats-submit@FreeBSD.org Subject: bin/91049: Patch to make dumps more rsync-friendly Message-ID: <20051229112011.C141C43D5E@mx1.FreeBSD.org> Resent-Message-ID: <200512291130.jBTBU51u088958@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 91049
>Category: bin
>Synopsis: Patch to make dumps more rsync-friendly
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: change-request
>Submitter-Id: current-users
>Arrival-Date: Thu Dec 29 11:30:04 GMT 2005
>Closed-Date:
>Last-Modified:
>Originator: Forrest Christian
>Release: FreeBSD 6.0-RELEASE i386
>Organization:
Montana Internet Corporation
>Environment:
System: FreeBSD backup.mt.net 6.0-RELEASE FreeBSD 6.0-RELEASE #6: Tue Dec 13 00:11:49 UTC 2005 root@backup.mt.net:/usr/src/sys/i386/compile/BACKUP i386
>Description:
When using dump to generate level 0 dumps which are then rsync'ed to a remote
machine, the fact that the dump date is stored with each header (inode) record makes
rsync significantly less efficient than necessary. This also applies to inode access
times when they are not important data to retain. When implementing an offsite backup
solution of this type, these dates in particular are not important, especially if it
prevents effective offsite backups.
>How-To-Repeat:
generate a level 0 dump. rsync to remote location. Generate another level 0 dump,
rsync again and note the large amount of data being transferred just because of the
date change.
>Fix:
Patch to add -r (don't store dump dates) and -R (don't store dump dates and access times)
options is attached. This is against the 6.0-release version of the dump source (it
this is the current revision of the code besides a minor manual change).
--- orig/dump.8 Thu Dec 29 10:25:32 2005
+++ ./dump.8 Thu Dec 29 11:09:58 2005
@@ -38,7 +38,7 @@
.Nd file system backup
.Sh SYNOPSIS
.Nm
-.Op Fl 0123456789acLnSu
+.Op Fl 0123456789acLnrRSu
.Op Fl B Ar records
.Op Fl b Ar blocksize
.Op Fl C Ar cachesize
@@ -268,6 +268,19 @@
prompts for a new tape.
It is recommended to be a bit conservative on this option.
The default tape length is 2300 feet.
+.It Fl r
+Be rsync-friendly. Normally dump stores the date of the current
+and prior dump in numerous places throughout the dump. This causes
+problems when rsync or another incremental file transfer program is
+used to update a remote copy of a level 0 dump, since the date changes
+for each dump. This option sets both dates to the epoch, permitting
+rsync to be much more efficient when transferring a dump file.
+.It Fl R
+Be even more rsync-friendly. This option disables the storage of the
+actual inode access time (storing it instead as the epoch).
+This option also sets -r. This permits rsync to be even more efficient
+when transferring dumps generated from filesystems with numerous files
+which are not changing other than their access times.
.It Fl T Ar date
Use the specified date as the starting time for the dump
instead of the time determined from looking in
--- orig/dump.h Thu Dec 29 10:25:32 2005
+++ ./dump.h Thu Dec 29 10:47:01 2005
@@ -76,6 +76,8 @@
int unlimited; /* if set, write to end of medium */
int cachesize; /* size of block cache in bytes */
+int rsync_friendly; /* be friendly with rsync */
+int rsync_morefriendly; /* be even more friendly with rsync */
int notify; /* notify operator flag */
int blockswritten; /* number of blocks written on current tape */
int tapeno; /* current tape number */
--- orig/main.c Thu Dec 29 10:25:32 2005
+++ ./main.c Thu Dec 29 11:07:05 2005
@@ -117,13 +117,15 @@
if (TP_BSIZE / DEV_BSIZE == 0 || TP_BSIZE % DEV_BSIZE != 0)
quit("TP_BSIZE must be a multiple of DEV_BSIZE\n");
level = '0';
+ rsync_friendly = 0;
+ rsync_morefriendly = 0;
if (argc < 2)
usage();
obsolete(&argc, &argv);
while ((ch = getopt(argc, argv,
- "0123456789aB:b:C:cD:d:f:h:LnP:Ss:T:uWw")) != -1)
+ "0123456789aB:b:C:cD:d:f:h:LnP:RrSs:T:uWw")) != -1)
switch (ch) {
/* dump level */
case '0': case '1': case '2': case '3': case '4':
@@ -189,6 +191,15 @@
popenout = optarg;
break;
+ case 'r': /*store slightly less data to be friendly with rsync*/
+ rsync_friendly = 1;
+ break;
+
+ case 'R': /*store even less data to be even more friendly with rsync*/
+ rsync_friendly = 1;
+ rsync_morefriendly = 1;
+ break;
+
case 'S': /* exit after estimating # of tapes */
just_estimate = 1;
break;
@@ -236,6 +247,10 @@
(void)fprintf(stderr, "\n");
exit(X_STARTUP);
}
+ if (rsync_friendly && (level>'0')) {
+ (void)fprintf(stderr, "rsync friendly options should only be used with level 0 dumps.\n");
+ exit(X_STARTUP);
+ }
if (Tflag && uflag) {
(void)fprintf(stderr,
"You cannot use the T and u flags together.\n");
@@ -383,6 +398,11 @@
spcl.c_level = level - '0';
spcl.c_type = TS_TAPE;
+ if (rsync_friendly) { /*don't store real dump times*/
+ spcl.c_date = 0;
+ spcl.c_ddate = 0;
+ }
+
if (spcl.c_date == 0) {
tmsg = "the epoch\n";
} else {
@@ -391,7 +411,7 @@
}
msg("Date of this level %c dump: %s", level, tmsg);
- if (!Tflag)
+ if ((!Tflag)&&(!rsync_friendly))
getdumptime(); /* /etc/dumpdates snarfed */
if (spcl.c_ddate == 0) {
tmsg = "the epoch\n";
--- orig/traverse.c Thu Dec 29 10:25:32 2005
+++ ./traverse.c Thu Dec 29 11:05:01 2005
@@ -682,6 +682,10 @@
{
int32_t sum, cnt, *lp;
+ if (rsync_morefriendly) {
+ spcl.c_atime=0;
+ spcl.c_atimensec=0;
+ }
spcl.c_inumber = ino;
spcl.c_magic = FS_UFS2_MAGIC;
spcl.c_checksum = 0;
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20051229112011.C141C43D5E>