From owner-freebsd-isp Thu Jan 22 01:23:08 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id BAA27908 for freebsd-isp-outgoing; Thu, 22 Jan 1998 01:23:08 -0800 (PST) (envelope-from owner-freebsd-isp@FreeBSD.ORG) Received: from ren.dtir.qld.gov.au (firewall-user@ns.dtir.qld.gov.au [203.108.138.66]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id BAA27902 for ; Thu, 22 Jan 1998 01:23:04 -0800 (PST) (envelope-from syssgm@dtir.qld.gov.au) Received: by ren.dtir.qld.gov.au; id TAA01963; Thu, 22 Jan 1998 19:38:39 +1000 (EST) Received: from ogre.dtir.qld.gov.au(167.123.8.3) by ren.dtir.qld.gov.au via smap (3.2) id xma001961; Thu, 22 Jan 98 19:38:26 +1000 Received: from localhost.dtir.qld.gov.au (localhost.dtir.qld.gov.au [127.0.0.1]) by ogre.dtir.qld.gov.au (8.8.7/8.8.7) with SMTP id TAA01819; Thu, 22 Jan 1998 19:18:16 +1000 (EST) Message-Id: <199801220918.TAA01819@ogre.dtir.qld.gov.au> X-Authentication-Warning: ogre.dtir.qld.gov.au: localhost.dtir.qld.gov.au [127.0.0.1] didn't use HELO protocol To: Chuck , chrw cc: freebsd-isp@FreeBSD.ORG, syssgm@dtir.qld.gov.au Subject: Re: recovering tar archieve / ssh References: In-Reply-To: from Chuck at "Thu, 08 Jan 1998 22:19:49 +0000" Date: Thu, 22 Jan 1998 19:18:15 +1000 From: Stephen McKay Sender: owner-freebsd-isp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org G'day! I'm resurrecting an old thread because I didn't see any conclusion and because so many things need answering. On Thursday, 8th January 1998, Chuck wrote: > >On Thu, 8 Jan 1998, chrw wrote: > >> Ive been using tar thru an ssh pipe to the fbsd streamer host, like >> >> tar cvfpl - * | ssh -c blowfish molly 'cat > /dev/nrst0' >> >> everything seemed to work fine, but today i tried doing a desaster >> recovery test.. It start extracting fine, but after 1 or 2 mins tar >> terminates with "tar: Unexpected EOF on archive file" :( Someone suggested adding '-e none' to prevent '~.' being recognised as an excape. This is unnecessary since ssh only allocates pseudo ttys for interactive connections and (from the manual): "If no pseudo tty has been allocated, the session is trans- parent and can be used to reliably transfer binary data." Someone suggested -B on tar to reblock data passing through a pipe. This is unnecessary since FreeBSD use gnu tar and it always sets -B when '-' is used as the file name. Strangely, no one suggested using 'dd' instead of 'cat'. cat will use a small block size. Using dd, you can specify a large blocksize. 32KB works well for me on both QIC and Exabyte tapes. Use the same block size on tar also unless you know the difference between fixed and variable block size tape drives. Once you are happy with 'dd', then you can try 'team', from the ports collection. This will give you much better throughput and I use it all the time. Use something like: tar clbf 64 - . | ssh -c blowfish molly 'team 32k 8 > /dev/nrst0' In this example, team will write in 32KB blocks and will use 8 cooperating processes (so the total amount of buffering is 256KB) to smooth the erratic data flow over the network. I even use it for local tape writing as it can smooth over the speed dips from tar itself when it hits a patch of many small files. >> the setup is, im doing backup from several sgi servers, onto the freebsd >> streamer host. I cant see this should be a problem. Have you tried reading the tape on the same machine that has the tape drive? This will remove doubts about the network and ssh. What errors (if any) were in the system logs of the machine with the tape drive? >> Anyway, EVEN if it is a tape error, i cannot rest assure that my backups >> are good untill i have to verified every single tape i got :( >> >> Maybe tar can verify each archieve it makes automatically, in the >> future..? Ill go check the man page. Yes, you should be reading back your tapes. Gnu tar has a --compare flag that might be useful, but some files would have changed since the backup. I am happy to just read the tape to verify that there are no media errors. Of course, I have restored some files occasionally to test that my scripts are not totally broken and to be sure I've used the norewind device when I should be. >Sounds familiar. I have a FreeBSD 2.2.2 PC, Tomato, with a TR4 (SCSI) >drive named nrst0. I run dump on it whenever I feel like it, and have >restored files from way down the tape. It goes from one end to the other, >then turns around and goes the other way. I don't know how many tracks >there are --it takes quite a few end-to-end trips-- but it works. > >But, when I run dump over the (thinwire) Ethernet from my other >FreeBSD 2.2.2 PC, Daisy, to the same nrst0 via rmt, first, it hiccups >about every two seconds, and averages about 50 kBps. *** AND *** it >only goes to the end of the tape once and does not turn around. It says >that Tomato's operator should mount a fresh tape. This in spite of dump's >-a option. I had strange failures remote dumping from FreeBSD 2.1.5 box to Solaris 2.5 so I gave up on rmt. Since then I've only used rsh or ssh as the data transport. Dump to stdout then pipe it to team on the remote box using rsh or ssh and things should work better and a lot faster. >So I've taken to running dump from Daisy to a BIG file on Tomato, and >then writing that file to tape using dd. Very fast. For my home FreeBSD boxes, I use dump, rsh and team all the time and have no problems, and it is fast (well as fast as the tape drives can handle anyway). > ABHOR SECRECY -- DEFEND PRIVACY Wise words. Oops, is that the US secret service kicking in my door? No sir, I had no intention of encrypting my grocery list! Ouch! Ouch! Stephen.