From owner-freebsd-ports@FreeBSD.ORG Fri Dec 3 21:32:21 2010 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7E093106564A; Fri, 3 Dec 2010 21:32:21 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id D5F378FC12; Fri, 3 Dec 2010 21:32:20 +0000 (UTC) Received: by fxm16 with SMTP id 16so7799568fxm.13 for ; Fri, 03 Dec 2010 13:32:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:newsgroups:to:cc:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=/OspNhyKNsPGguZRDTbyk6sUkIrbrlHJvGi4hwtrj+A=; b=g0Tbw+KnDRcMi8jnSgFzDSCmY95nHK0UKeYgQPf+j5kiqfqlZDgVomDvO0vAm9OECe ARtRYGjSt/uLnTGqihHY237j0/FfyV9RtSqPoM1Q3T1kbgYODn6upkE6d4KExN40QJFk NsYwuXbTETgDBWTXYtn3WVhfYhKE8Ei7rHVWE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:newsgroups:to:cc :subject:references:in-reply-to:content-type :content-transfer-encoding; b=klTYTrqgb02HKD7oNOPug+V0v3L6hnNPT5ZEhfc91TvvQnNXansrN1It2sL/h1zxc3 4tFJAcb6AGSsM2Mm0dwXaNm1yXi1QkUxAL7mskGRchuYcqwxDar5VcE9gEYLf3JscJVS twovy5+KJRUlq5qlVdN1+SHpEnSaFfLRALdz4= Received: by 10.223.96.66 with SMTP id g2mr2657360fan.61.1291410475960; Fri, 03 Dec 2010 13:07:55 -0800 (PST) Received: from limbo.lan ([193.33.173.33]) by mx.google.com with ESMTPS id n1sm785590fam.40.2010.12.03.13.07.53 (version=SSLv3 cipher=RC4-MD5); Fri, 03 Dec 2010 13:07:54 -0800 (PST) Message-ID: <4CF95C27.4030507@gmail.com> Date: Fri, 03 Dec 2010 23:07:51 +0200 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; uk-UA; rv:1.9.2.12) Gecko/20101029 Thunderbird/3.1.6 MIME-Version: 1.0 Newsgroups: gmane.os.freebsd.devel.ports To: Julien Laffaye References: <4CF38D7F.6070206@gmx.de> <4CF3F16E.3020501@DataIX.net> <4CF439F1.6050703@gmx.de> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: Matthias Andree , freebsd-ports@freebsd.org Subject: Re: packages compressed with xz X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Dec 2010 21:32:21 -0000 30.11.2010 04:40, Julien Laffaye wrote: > You can specify limits during compression, so the question is should we do that > so that hosts with N MB of RAM can decompress packages? Do we retain the > compression ratio over bzip2 if we limit compression memory to 512 MB so that > decompression would be possible with, say, 128 MB? > > According to xz(1), in its default mode (-6), xz uses ~100MiB for > compression and ~10MiB for decompression. > That seems to be acceptable. You possibly miss something about compression/decompression. The designated memory size is not directly affected only by compression mode. When decompressing you will need memory for: 1. Data history. 2. Dictionary. 3. Some indexes. And those ones are all empty at start. So say, if you are compressing something really huge trying to use 4G of memory you end using that much memory between 2G - 3G of source data. And we will need 512MB to decompress that hunk of data. Are the packages _that_ large? I think that in worst possible case we need to double the size of package for comfort decompression. If the lower point is 64Mb then 32Mb package compressed with any compression strategy would be decompressed without hitting a swap. We'll need someone to do actual testing but look at this one: # ls -la ImageMagick-6.6.5-10.tar.xz -rw-r--r-- 1 root wheel 6316324 21 ноя 23:52 ImageMagick-6.6.5-10.tar.xz # time xz -dt ImageMagick-6.6.5-10.tar.xz 0.860u 0.036s 0:01.02 87.2% 68+1492k 56+0io 1pf+0w -- Sphinx of black quartz judge my vow.