Date: Sat, 19 Jun 2010 12:44:47 +0200 From: Matthias Andree <mandree@FreeBSD.org> To: Lasse Collin <lasse.collin@tukaani.org> Cc: ports@freebsd.org, Christian Weisgerber <naddy@freebsd.org>, portmgr@freebsd.org Subject: Re: FreeBSD ports USE_XZ critical issue on low-RAM computers Message-ID: <4C1C9F9F.8090808@FreeBSD.org> In-Reply-To: <201006190855.05061.lasse.collin@tukaani.org> References: <4C1BA4D4.9000205@FreeBSD.org> <201006190855.05061.lasse.collin@tukaani.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigCA9B8E4EA4DB9230D4C3BC5F Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Greetings, first my thanks to Lasse for the prompt reply. Lasse Collin wrote on Jun 19: > On 2010-06-18 Matthias Andree wrote: > =20 >> I've just had xz break my devel/libtool22 FreeBSD port build on a low >> memory computer (128 MB). >> >> Reason is that xz by default caps memory use at ~40% of physical RAM >> (this is documented), and skips decompressing files if that doesn't >> suffice. >> =20 > A snapshot of XZ Utils newer than the 4.999.9beta release has different= > default limit (I know that an official release would be nice etc. but > there isn't one now.): > > - If 40 % of RAM is at least 80 MiB, 40 % of RAM is used as > the limit. > - If 80 % of RAM is less than 80 MiB, 80 % of RAM is used as > the limit. > - Otherwise 80 MiB is used as the limit. > > The above avoids the problem on most systems since 80 MiB is enough for= > all typical .xz files. > > The limit was a problem on Gentoo too. There's still a problem with the= > above default limit if you have 16 MiB RAM (that's a real-world > example), so Gentoo put XZ_OPT=3D--memory=3Dmax to their build system. = They > too think it is better to let the system be slow and swap very heavily > for an hour or two than refuse decompression in a critical script. It is /not the application's responsibility/ to sidestep an inadequate system configuration. <rant>Just because noobs can use Unix today thanks to GNOME and KDE does not mean we should target command-line utility defaults for them, or turn the average command line utility like xz into a sissy.</rant> It might be useful if build systems (such as Gentoo's system around emerge, or FreeBSD port, or pkgsrc that originated around NetBSD) could know that for low memory computer they may want to download the bigger gz instead because it's for /them/ orders of magnitude faster to decompress than .bz2 or .xz which might cause swap disk thrashing. Decompression is *not* a facultative job. If I need the uncompressed contents, I need them, period. I don't decompress the file for the fun of it out of a certain mood, but because I need to. We have system facilities for limiting resources, including those that limit virtual memory. Don't work against or around them. Remember that if you play too nicely, others will starve you. There's no guarantee that the "40% of RAM" will be in RAM unless you use privileged operations (think mlock/mlockall). If there are tons of other applications allocating memory the OS might decide to swap 15% of RAM of xz's address space out to disk. Oops. If I want to avoid thrashing, I can tune my swap configuration, or memory limits properly. I can abort an ongoing decompression (SIGINT) if it swaps too much. Point to counter your defaults: On some of my systems, xz wouldn't even get the default 40% of RAM because I often impose 300 MB Virtual Memory limits on computers with 1 GB of RAM. >> - This feature is, in my perception, INADEQUATE during decompression. >> If I have a .xz file (downloaded from the Internet) that needs 90 MB >> RAM to decompress, then I need to use those 90 MB no matter if >> that's nice or not, it's just critical. >> >> I am proposing to Lasse Collin to drop memory capping functionality >> in xz/lzma in decompress mode, and in lzmadec/xzdec. >> =20 > Naturally the limiter functionality won't be removed, but a different > default value can be considered, including no limit by default. Please do that. For decompression, -M max should be the default. For compression, it's less critical because service is degraded, not denied, but I'd still think -M max would be the better default. I can always put "export XZ_OPT=3D-3" in /etc/profile.d/local.sh or wherever it belongs on the OS of the day. I still think utilities and applications should /not/ impose arbitrarily lower limits by default though. I see that a feature to limit memory for compression or decompression can be useful, particularly if I compress for other systems (think compressing on my workstation for an embedded system). I propose to put the essence of this discussion into the manpage and have the memory limiter warnings point to the --memory option in the manpage as a usability improvement. > Would you find no limit OK after xz allocated and used 1 GiB of memory > without a warning after you tried to decompress a relatively big file > you just downloaded on a slightly older system with 512 MiB RAM? I gues= s Yes, I absolutely want that. > if it is a critical file decompressed by a critical script, you don't > mind it swapping a couple of hours, because you just want it done no > matter how long it takes. But in normal command line use some people > would prefer to get an error first so that it is possible to consider > e.g. using another system to do the decompression (possibly > recompressing with lower settings or with another tool) instead of just= > overriding the limit. They can use ulimit(1) -- or whatever their favourite login shell offers -- to set soft memory limits. Use the system facilities, but don't duplicate the standard operating system knobs in applications. > One possibility could be to make the limit for decompression e.g. max(8= 0 > MiB, 40 % of RAM), since all typical files will decompress with 80 MiB > (you need to use advanced options to create files needing more). That > way also systems with less than 128 MiB RAM would decompress all typica= l > files by default, possibly slowly with heavy swapping, and systems with= > more RAM would be protected from unexpected memory usage of very rarely= > occurring .xz files. You're overcomplicating matters, and that's giving you headaches: > Determining a good limit has been quite a bit of a problem for me. > Obviously a DoS protection mechanism shouldn't be a DoS itself. Agreed. You are facing those difficulties because you are mixing up the machinery (the --memory feature per se - which does not belong in the application because the OS already has it) with the rules/policies (such as the default setting for memory). You are overcomplicating matters with trying to be smart. > Disabling the limiter completely by default doesn't seem like an option= , > because it would only change who will be annoyed. Comments are very > welcome. Thanks. It is a necessity to change it. Applications do not have the freedom to decide if they are in the mood to accept or refuce a crucial operation= =2E If I tell my Unix utility to decompress, I expect it to do exactly that. If I want it to ask questions, I add --interactive on the command line. Feel free to pop up a million of obnoxious "do you really want" requestors in your MS Windows GUI application, but xz is for Unix, and it should behave the Unix way: do what you're told to, and try as hard as you can. Don't I find xz a very useful tool, but the -M default of "if this, 40%, of that, 80%, or if foo, then X MB" for decompression is a major glitch and should be remedied and changed for -M max at least for decompression.= Best regards Matthias --=20 Matthias Andree (ports/ committer) --------------enigCA9B8E4EA4DB9230D4C3BC5F Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkwcn6MACgkQvmGDOQUufZWf6QCdGkhL/HJl92GkFOCEWrAucPoU 7j8An0uKtg70I2VX3Yu2h8ylo6cfRwmk =e/0h -----END PGP SIGNATURE----- --------------enigCA9B8E4EA4DB9230D4C3BC5F--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C1C9F9F.8090808>