From owner-freebsd-arch@FreeBSD.ORG Mon Apr 4 15:46:10 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 61C8C16A4CE; Mon, 4 Apr 2005 15:46:10 +0000 (GMT) Received: from shrike.submonkey.net (cpc4-cdif3-6-1-cust116.cdif.cable.ntl.com [82.23.41.116]) by mx1.FreeBSD.org (Postfix) with ESMTP id CE39543D1D; Mon, 4 Apr 2005 15:46:09 +0000 (GMT) (envelope-from setantae@submonkey.net) Received: from setantae by shrike.submonkey.net with local (Exim 4.50 (FreeBSD)) id 1DITly-0001k8-B1; Mon, 04 Apr 2005 16:45:54 +0100 Date: Mon, 4 Apr 2005 16:45:54 +0100 From: Ceri Davies To: John Baldwin Message-ID: <20050404154554.GG19136@submonkey.net> Mail-Followup-To: Ceri Davies , John Baldwin , Max Laier , Dan Nelson , Garance A Drosihn , Colin Percival , lists@mhoerich.de, freebsd-arch@FreeBSD.org References: <424B3AAB.6090200@wadham.ox.ac.uk> <424DC747.4020604@wadham.ox.ac.uk> <200504020126.16738.max@love2party.net> <4880d4c8fa4f5a350a0072ab1574ecc9@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="rwEMma7ioTxnRzrJ" Content-Disposition: inline In-Reply-To: <4880d4c8fa4f5a350a0072ab1574ecc9@FreeBSD.org> X-PGP: finger ceri@FreeBSD.org User-Agent: Mutt/1.5.9i Sender: Ceri Davies cc: Dan Nelson cc: Colin Percival cc: lists@mhoerich.de cc: freebsd-arch@FreeBSD.org cc: Max Laier cc: Garance A Drosihn Subject: Re: Adding bsdiff to the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Apr 2005 15:46:10 -0000 --rwEMma7ioTxnRzrJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Apr 02, 2005 at 03:16:02PM -0500, John Baldwin wrote: >=20 > On Apr 1, 2005, at 6:26 PM, Max Laier wrote: >=20 > >On Saturday 02 April 2005 00:12, Colin Percival wrote: > >>>>In the last episode (Apr 01), Mario Hoerich said: > >>>>> Not that it's important, but the names probably aren't the best > >>>>> possible choice, as 'bsdiff' seems to suggest 'BSD licensed diff'. > >> > >>No, it would be "BSD licensed iff". :-) > >> > >>>At 9:28 AM -0600 4/1/05, Dan Nelson wrote: > >>>>Yes, that's what I assumed this thread was about for the first > >>>>couple posts. bdiff/bpatch sound like better names. What's the > >>>>'s' stand for? > >> > >>Err... nothing. Or rather, I'm not sure what it stands for. I was > >>looking for a name for a diff tool which worked on "binary software"=20 > >>(or > >>more generally, files with lots of "byte-substitutions"), and which=20 > >>uses > >>"bytewise subtraction" as part of its encoding process... (I'm sure=20 > >>you > >>can think of other possible meanings of "bs", as well.) > > > >Though it's "*B*inary *S*mall diff" ... and I like that name! >=20 > Heh, that's what I thought the "bs" stood for at first as well given=20 > that that is bs{diff,patch}'s claim to fame. I am starting to think it's "BikeShed Diff" ;-) Ceri --=20 Only two things are infinite, the universe and human stupidity, and I'm not sure about the former. -- Einstein (attrib.) --rwEMma7ioTxnRzrJ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (FreeBSD) iD8DBQFCUWEyocfcwTS3JF8RAlK9AKCLzXNeRt9e0wxoeDYh1MIOYtYvWACdEUNZ Ef5eEMJVG7K/7zvPnIDZLoU= =SfLj -----END PGP SIGNATURE----- --rwEMma7ioTxnRzrJ-- From owner-freebsd-arch@FreeBSD.ORG Thu Apr 7 06:50:08 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AD0A016A4CE for ; Thu, 7 Apr 2005 06:50:08 +0000 (GMT) Received: from hirsch.in-berlin.de (hirsch.in-berlin.de [192.109.42.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id 53FB743D45 for ; Thu, 7 Apr 2005 06:50:07 +0000 (GMT) (envelope-from wagner@luthien.iceflower.in-berlin.de) X-Envelope-From: wagner@luthien.iceflower.in-berlin.de Received: from hirsch.in-berlin.de (localhost [127.0.0.1]) j376o3dM017889; Thu, 7 Apr 2005 08:50:03 +0200 Received: (from uucp@localhost)j376o3jn017882; Thu, 7 Apr 2005 08:50:03 +0200 Received: (from wagner@localhost) by luthien.iceflower.in-berlin.de (8.13.3/8.13.3) id j376imwB027984; Thu, 7 Apr 2005 08:44:48 +0200 (CEST) (envelope-from wagner) Date: Thu, 7 Apr 2005 08:44:48 +0200 (CEST) Message-Id: <200504070644.j376imwB027984@luthien.iceflower.in-berlin.de> From: Olaf Wagner To: Colin Percival In-Reply-To: <424BD4FB.1050304@wadham.ox.ac.uk> X-Newsgroups: luthien.freebsd.arch User-Agent: tin/1.6.2-20030910 ("Pabbay") (UNIX) (FreeBSD/5.4-PRERELEASE (i386)) cc: freebsd-arch@freebsd.org Subject: Re: Adding bsdiff to the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Apr 2005 06:50:08 -0000 In article <424BD4FB.1050304@wadham.ox.ac.uk> you wrote: > Alexey Dokuchaev wrote: > I'll conceed that portsnap is not yet used by the majority of our user > base; but I think that is largely because portsnap is still quite new, > and thus relatively unknown. At present portsnap is the only mechanism > available by which most users can securely maintain an up-to-date copy > of the FreeBSD ports tree; it also provides some other advantages over > cvsup (reduced bandwidth and ports INDEX/INDEX-5/INDEX-6 files). Since Just out of interest: how does it do that? I've not tested it yet, but what intelligence or knowledge does it use to be so much more efficient (1/10) than CVSup? (I myself haven't found anything as efficient as CVSup yet, at least for replicating CVS repositories...) > portsnap and its dependencies will not significantly bloat the base > system -- portsnap + bsdiff weigh in at a combined 54kB -- I think it > is a sufficiently useful tool to justify inclusion. -- /\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ || Olaf Wagner | wagner(at)luthien.in-berlin.de (priv.)| || Cranachstrasse 7 | wagner(at)elego.de (business) | || D-12157 Berlin | phone: +49 30 85 60 26 70 | || Germany / Deutschland | fax: +49 30 85 58 01 88 | \///////////////////////////////////////////////////////////////// From owner-freebsd-arch@FreeBSD.ORG Thu Apr 7 07:36:17 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 26C2B16A4CE for ; Thu, 7 Apr 2005 07:36:17 +0000 (GMT) Received: from pd3mo2so.prod.shaw.ca (shawidc-mo1.cg.shawcable.net [24.71.223.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8320E43D54 for ; Thu, 7 Apr 2005 07:36:16 +0000 (GMT) (envelope-from colin.percival@wadham.ox.ac.uk) Received: from pd5mr8so.prod.shaw.ca (pd5mr8so-qfe3.prod.shaw.ca [10.0.141.184]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0IEK00K0LFSG6EE0@l-daemon> for freebsd-arch@freebsd.org; Thu, 07 Apr 2005 01:36:16 -0600 (MDT) Received: from pn2ml5so.prod.shaw.ca ([10.0.121.149]) by pd5mr8so.prod.shaw.ca (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0IEK00E44FSG2MF0@pd5mr8so.prod.shaw.ca> for freebsd-arch@freebsd.org; Thu, 07 Apr 2005 01:36:16 -0600 (MDT) Received: from [192.168.0.60] (S0106006067227a4a.vc.shawcable.net [24.87.209.6]) by l-daemon (iPlanet Messaging Server 5.2 HotFix 1.18 (built Jul 28 2003)) with ESMTP id <0IEK0020VFSF6F@l-daemon> for freebsd-arch@freebsd.org; Thu, 07 Apr 2005 01:36:16 -0600 (MDT) Date: Thu, 07 Apr 2005 00:36:09 -0700 From: Colin Percival In-reply-to: <200504070644.j376imwB027984@luthien.iceflower.in-berlin.de> To: Olaf Wagner Message-id: <4254E2E9.2090504@wadham.ox.ac.uk> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Accept-Language: en-us, en X-Enigmail-Version: 0.90.1.0 X-Enigmail-Supports: pgp-inline, pgp-mime References: <200504070644.j376imwB027984@luthien.iceflower.in-berlin.de> User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050406) cc: freebsd-arch@freebsd.org Subject: Re: Adding bsdiff to the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Apr 2005 07:36:17 -0000 Olaf Wagner wrote: > In article <424BD4FB.1050304@wadham.ox.ac.uk> you wrote: >>At present portsnap is the only mechanism >>available by which most users can securely maintain an up-to-date copy >>of the FreeBSD ports tree; it also provides some other advantages over >>cvsup (reduced bandwidth and ports INDEX/INDEX-5/INDEX-6 files). > > Just out of interest: how does it do that? I've not tested it yet, > but what intelligence or knowledge does it use to be so much more > efficient (1/10) than CVSup? (I myself haven't found anything as > efficient as CVSup yet, at least for replicating CVS repositories...) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Exactly. CVSup is a tool for replicating CVS repositories; portsnap is a tool for checking out the latest version of all the files in the repository. CVSup is solving a very difficult problem; portsnap is solving a very simple problem -- so it's not all that surprising that portsnap can be a bit more efficient. The reason portsnap is more efficient lies in how portsnap and CVSup determine which files need to be updated. The ports tree contains roughly 71000 files, and the first thing the CVSup client does is list all of these files and send that list to the server. In contrast, portsnap has an index file -- containing, roughly speaking, that same list -- and the portsnap client merely sends the sha256 hash of this index file to the server, which responds with either "I recognize that index -- here's a patch which will turn it into the latest index" or "I don't recognize that -- here's the new index". Because these indices have no user-serviceable parts (in fact, mucking about with the files in /usr/local/portsnap at all is strongly discouraged), there is a very good chance that the portsnap server will have a useful patch. As a result, while CVSup uses (in this initial stage) bandwidth which is proportional to the number of files in the ports tree, portsnap uses bandwidth proportional to the number of files which have been modified, which is typically around 1% of the tree per day. When it comes to the actual distribution of patches to files in the tree, portsnap is also marginally more efficient than CVSup, due to differences in how they encode the patches, but the real gains come in the process of identifying which files need to be updated. Colin Percival PS. CVSup's inefficiency in dealing with large trees containing a small number of updated files isn't only relevant in the context of updating a ports tree; it is even more notable when tracking the security branches of the src tree. In the paper in which I introduced FreeBSD Update, I gave an example of where FreeBSD Update -- which distributes binary updates to the base system -- used less than half of the bandwidth needed by CVSup for the task of applying the corresponding updates to the src tree. From owner-freebsd-arch@FreeBSD.ORG Thu Apr 7 19:47:43 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E40BE16A4CE; Thu, 7 Apr 2005 19:47:43 +0000 (GMT) Received: from cheer.mahoroba.org (gw4.mahoroba.org [218.45.22.175]) by mx1.FreeBSD.org (Postfix) with ESMTP id CD4D343D1F; Thu, 7 Apr 2005 19:47:40 +0000 (GMT) (envelope-from ume@mahoroba.org) Received: from lyrics.mahoroba.org (ume@lyrics.mahoroba.org [IPv6:3ffe:501:185b:8010:280:88ff:fe03:4841]) (user=ume mech=CRAM-MD5 bits=0)j37JlUIn081073 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 8 Apr 2005 04:47:31 +0900 (JST) (envelope-from ume@mahoroba.org) Date: Fri, 08 Apr 2005 04:47:29 +0900 Message-ID: From: Hajimu UMEMOTO To: arch@FreeBSD.org User-Agent: Wanderlust/2.15.1 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.7 (=?ISO-8859-4?Q?Sanj=F2?=) APEL/10.6 Emacs/22.0.50 (i386-unknown-freebsd5.4) MULE/5.0 (SAKAKI) X-Operating-System: FreeBSD 5.4-PRERELEASE X-PGP-Key: http://www.imasy.or.jp/~ume/publickey.asc X-PGP-Fingerprint: 1F00 0B9E 2164 70FC 6DC5 BF5F 04E9 F086 BF90 71FE Organization: Internet Mutual Aid Society, YOKOHAMA MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-Greylist: Sender succeded SMTP AUTH authentication, not delayed by milter-greylist-2.0b5 (cheer.mahoroba.org [IPv6:3ffe:501:185b:8010::1]); Fri, 08 Apr 2005 04:47:31 +0900 (JST) X-Virus-Scanned: by amavisd-new X-Virus-Status: Clean X-Spam-Status: No, score=-5.7 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on cheer.mahoroba.org cc: Hajimu UMEMOTO Subject: [CFR] thread-safe getserv*(3) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Apr 2005 19:47:44 -0000 Hi, I've added getserv{byname,byport,ent}_r for internal use within libc, and made getserv{byname,byport,ent} thread-safe by using them. You can get my patch from: http://www.imasy.or.jp/~ume/FreeBSD/getserv-thread-safe-6-current-20050408.diff Please review it. Sincerely, -- Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan ume@mahoroba.org ume@{,jp.}FreeBSD.org http://www.imasy.org/~ume/ From owner-freebsd-arch@FreeBSD.ORG Fri Apr 8 01:39:21 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 276F516A4CE for ; Fri, 8 Apr 2005 01:39:21 +0000 (GMT) Received: from mailgate1.nau.edu (mailgate1.nau.edu [134.114.96.58]) by mx1.FreeBSD.org (Postfix) with ESMTP id EBD5243D58 for ; Fri, 8 Apr 2005 01:39:20 +0000 (GMT) (envelope-from Earl.Duque@NAU.EDU) Received: from CONVERSION-DAEMON.mailgate1.nau.edu by mailgate1.nau.edu (PMDF V6.2-X17 #30770) id <0IEL00F01TXJY6@mailgate1.nau.edu> for freebsd-arch@freebsd.org; Thu, 07 Apr 2005 18:39:20 -0700 (MST) Received: from [134.114.107.1] (client255.vpn.nau.edu [134.114.107.1]) by mailgate1.nau.edu (PMDF V6.2-X17 #30770)freebsd-arch@freebsd.org; Thu, 07 Apr 2005 18:39:19 -0700 (MST) Date: Thu, 07 Apr 2005 18:39:11 -0700 From: Earl Duque To: freebsd-arch@freebsd.org Message-id: <4255E0BF.5030106@nau.edu> Organization: Northern Arizona University MIME-version: 1.0 Content-type: text/plain; charset=us-ascii; format=flowed Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.1) Gecko/20040707 Subject: Removing T/TCP and replacing it with something simpler X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Earl.Duque@NAU.EDU List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Apr 2005 01:39:21 -0000 From owner-freebsd-arch@FreeBSD.ORG Fri Apr 8 15:11:38 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B111016A4CE for ; Fri, 8 Apr 2005 15:11:38 +0000 (GMT) Received: from blake.polstra.com (blake.polstra.com [64.81.189.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1C17743D1D for ; Fri, 8 Apr 2005 15:11:38 +0000 (GMT) (envelope-from jdp@polstra.com) Received: from strings.polstra.com (dsl081-189-067.sea1.dsl.speakeasy.net [64.81.189.67]) by blake.polstra.com (8.13.1/8.13.1) with ESMTP id j38FBbrt032904 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 8 Apr 2005 08:11:37 -0700 (PDT) (envelope-from jdp@strings.polstra.com) Received: (from jdp@localhost) by strings.polstra.com (8.13.1/8.13.1/Submit) id j38FBa4B048590; Fri, 8 Apr 2005 08:11:36 -0700 (PDT) (envelope-from jdp) Message-ID: X-Mailer: XFMail 1.5.5 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <4254E2E9.2090504@wadham.ox.ac.uk> Date: Fri, 08 Apr 2005 08:11:36 -0700 (PDT) From: John Polstra To: Colin Percival X-Bogosity: No, tests=bogofilter, spamicity=0.450984, version=0.14.5 cc: freebsd-arch@freebsd.org Subject: Re: Adding bsdiff to the base system X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Apr 2005 15:11:38 -0000 On 07-Apr-2005 Colin Percival wrote: > The reason portsnap is more efficient lies in how portsnap and CVSup > determine which files need to be updated. The ports tree contains > roughly 71000 files, and the first thing the CVSup client does is list > all of these files and send that list to the server. > > In contrast, portsnap has an index file -- containing, roughly speaking, > that same list -- and the portsnap client merely sends the sha256 hash of > this index file to the server, which responds with either "I recognize > that index -- here's a patch which will turn it into the latest index" > or "I don't recognize that -- here's the new index". Nice! John