From owner-freebsd-questions@FreeBSD.ORG Mon Nov 7 17:29:41 2005 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 738ED16A41F for ; Mon, 7 Nov 2005 17:29:41 +0000 (GMT) (envelope-from kirk@strauser.com) Received: from kanga.honeypot.net (kanga.honeypot.net [208.162.254.122]) by mx1.FreeBSD.org (Postfix) with ESMTP id 24BBB43D6B for ; Mon, 7 Nov 2005 17:29:38 +0000 (GMT) (envelope-from kirk@strauser.com) Received: from localhost (localhost [127.0.0.1]) by kanga.honeypot.net (Postfix) with ESMTP id 522F222240A for ; Mon, 7 Nov 2005 11:29:37 -0600 (CST) Received: from kanga.honeypot.net ([127.0.0.1]) by localhost (kanga.honeypot.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 97838-08 for ; Mon, 7 Nov 2005 11:29:36 -0600 (CST) Received: from janus.daycos.com (janus.daycos.com [204.26.70.77]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by kanga.honeypot.net (Postfix) with ESMTP id 83FF8222408 for ; Mon, 7 Nov 2005 11:29:36 -0600 (CST) From: Kirk Strauser To: freebsd-questions@freebsd.org Date: Mon, 7 Nov 2005 11:29:32 -0600 User-Agent: KMail/1.8.2 References: <200511040956.19087.kirk@strauser.com> <200511041129.17912.kirk@strauser.com> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2694721.7iXZAHlP67"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200511071129.34262.kirk@strauser.com> X-Virus-Scanned: amavisd-new at honeypot.net Subject: Re: Fast diff command for large files? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Nov 2005 17:29:41 -0000 --nextPart2694721.7iXZAHlP67 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Monday 07 November 2005 10:40, francisco@natserv.net wrote: > I had the same setup a while back. > A few suggestions. Thanks for the tips; unfortunately, any fix that involves touching the=20 =46oxPro code is basically impossible. It's not that we *can't*, but that= =20 the sole FoxPro programmer at our company is completely occupied with other= =20 projects. > What type of system is this? In particular do any record can be modified > or are only recent records changed? Nope - every line in each table is subject to change. Here's how our current system works: 1) Copy each FoxPro table file (and associated memo file if one exists) to = a=20 Unix server via Samba. 2) Run my modified version of the "xbase" program to convert each table to = a=20 tab-delimited file that can be loaded into PostgreSQL using the "copy=20 table" command. These files are named "foo.dump", "bar.dump", etc. 3) If "foo.dump-old" exists: a) Using Andrew's algorithm, get the difference between foo.dump-old and foo.dump. Write these out as a set of "delete from ..." commands and a "copy table" command. Pipe this relatively tiny file into the "psql" command to upload the modifications. Otherwise: b) Use the psql command to upload foo.dump 4) "mv foo.dump foo.dump-old" 5) Profit! I've already cut the runtime in half. The next big step is going to be=20 getting our Windows admin to install rsync on the fileserver so that we can= =20 minimize the time spent in step one. With the exception of the space=20 required by keeping the old version of the dump files (step 4), this is=20 exceeding all of our performance expectations by a wide margin. Even better, step 3a cuts the time that the PostgreSQL server has to spend= =20 committing the new data by several orders of magnitude. The net effect is= =20 that our web visitors don't see a noticeable slowdown during the import=20 stage. =20 =2D-=20 Kirk Strauser --nextPart2694721.7iXZAHlP67 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iD8DBQBDb47+5sRg+Y0CpvERAn3HAJ48eDr6BzIr4ynASeXtd4EQPKRiLACdFfe1 VMB6s+iELhud7pAvWPhqRBU= =7EEU -----END PGP SIGNATURE----- --nextPart2694721.7iXZAHlP67--