From owner-freebsd-bugs Fri Oct 18 5:50:10 2002 Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 075AE37B401 for ; Fri, 18 Oct 2002 05:50:07 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9B13643E9C for ; Fri, 18 Oct 2002 05:50:06 -0700 (PDT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.6/8.12.6) with ESMTP id g9ICo6x3059757 for ; Fri, 18 Oct 2002 05:50:06 -0700 (PDT) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.6/8.12.6/Submit) id g9ICo6dV059756; Fri, 18 Oct 2002 05:50:06 -0700 (PDT) Date: Fri, 18 Oct 2002 05:50:06 -0700 (PDT) Message-Id: <200210181250.g9ICo6dV059756@freefall.freebsd.org> To: freebsd-bugs@FreeBSD.org Cc: From: Peter Pentchev Subject: Re: misc/44195: globbing/argument limits Reply-To: Peter Pentchev Sender: owner-freebsd-bugs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org The following reply was made to PR misc/44195; it has been noted by GNATS. From: Peter Pentchev To: abc@anchorageinternet.org Cc: bug-followup@freebsd.org, "Kerr, Greg" , "Choudhury, Raj" Subject: Re: misc/44195: globbing/argument limits Date: Fri, 18 Oct 2002 15:48:13 +0300 On Fri, Oct 18, 2002 at 11:57:36AM +0000, abc@anchorageinternet.org wrote: > > > >Number: 44195 > > > >Category: misc > > > >Synopsis: globbing/argument limits > > > >Originator: Joe Public > > > >Release: i386 FreeBSD 4.7-RELEASE > > > >Organization: > > > no org > > > >Environment: > > > ^^^^^^^^^^^^^^^^^^^^^^^^ > > > >Description: > > > argument limits painful to users in days of 100GB drives. > > > >How-To-Repeat: > > > try a command and give it a few thousand arguments, > >=20 > > It is not a matter of how many arguments you give to a command, it is > > simply a matter of how *long* the command line becomes. Lugging around > > a multimegabyte command line buffer through shells, execv() system calls > > and such would be a *major* strain on your system. >=20 > i would've assumed the command line stays in place in memory, > and only a pointer is passed around - and checking the exec(3) > manpage seems to show this is the case in fact. Not when exec(3) invokes the execve(2) system call, as stated in the very first paragraph of the exec(3) manual page. The execve(2) system call needs to copy the arguments to kernel space to examine them, and then to build a single command line for the new process. > > > like in file modifying command a folder with 6000 files. > > > find(1) is too slow, and combining it with xargs is a kludge. > >=20 > > If you mean that 'find -exec' is too slow, then I would argue that using > > -exec is the kludge, when xargs(1) is available. I am pretty sure that > > the find(1) and xargs(1) utilities were actually developed together, > > with a common goal in mind, that goal being *exactly* processing of > > multiple files in one go. >=20 > ok - interesting - i appreciate you explaining this. > it should be in a the FAQ or something. > =20 > > The -exec primary to find(1) is extremely inefficient when dealing with > > many files - it spawns a new process for each file it finds, which, as > > you note, is too slow. The xargs utility will do a much better job; I > > would be very interested in what exactly do you consider to be a kludge > > about it. >=20 > i consider it to be a kludge when you have to: >=20 > find -s "$I" ! -type d | xargs tar rvf "$I.tar" && \ > && gzip -f9 "$I.tar" && mv "$I.tar.gz" "$I.tgz"=20 >=20 > just to create a sorted tar/gzip archive > of a directory tree. hehe - as i look at > it - i say to myself "this is bullshit" :). As noted in my response to your other PR, this particular use of find(1) and tar(1) may be optimized :) Besides, the only "kludge" in that example is the need to update the tarball incrementally using tar's 'r' command instead of 'c'; I, personally, would not consider that too large a price to pay for being able to process the whole list of files at all. > i mean - UNIX hackers have got to be smarter > than to demand all that from a user just to > accomplish such a minor ordinary task. Yep, see both above and below :) > also, when you do something like: >=20 > find / \! \( -path \*/bin/\* -or -path \*/lib/\* \ > -or -path \*/libexec/\* -or -path /usr/games/\* \ > -or -path \*/sbin/\* -or -path /boot/\* \ > -or -path /dev/\* -or -path /modules/\* \ > -or -path /proc/\* -or -path /root/\* \) \ > -type f -exec chown root:wheel {} \;\ > -exec chmod 0644 {} \; >=20 > ie, something a find(1) with 2 -exec's, xargs fails, > and you are forced to double or triple (or more) the code > it takes - according to the number of -exec's you need to > perform - i consider this to be a kludge as well. If you need to execute multiple commands, there are several things you might do. The simplest is to create a small shell script, and use xargs(1) to execute it; the shell script runs chown, chmod, or whatever, on all its arguments. Another way would be capturing find(1)'s output into a file, then running xargs(1) as many times as needed, redirecting its input to read this file; something like: find / \! \( ... \) > filelist xargs chown root:wheel < filelist xargs chmod 0644 < filelist Still another way might avoid the temporary file altogether, with some creative file descriptor hackery. I *think* I have done this before, but right now, I cannot remember the proper incantations to make the shell duplicate find(1)'s output to a new file descriptor, run xargs from fd 1's output, then run another copy of xargs, making it read from the file descriptor that find(1)'s output was duplicated to. I know it is possible, it is just that I cannot remember how to do it :) > > PS. This will very probably be my last post on this subject, and nobody > > should be surprised if this PR is closed very soon; what with the recent > > mailing list "activity", it scores big on my troll indicator. I could > > be wrong, of course, but I'm just stating my opinion here. >=20 > ok - well - no trolling - just installed 4.7, and hit > a point of frustration with things that have been > bugging me over the years - that don't see to > get fixed or improve. >=20 > i truly value the effort you made to try to explain, > though as stated, i still don't see the problem > in fixing things. Apologies for the above paragraph of mine and my attitude in somewhat summarily dismissing your other PR's at first; there has been quite a bit of trolling on the various FreeBSD lists recently, and there have been a couple of bogus PR's filed in the process, so I was a bit trigger-happy there. G'luck, Peter --=20 Peter Pentchev roam@ringlet.net roam@FreeBSD.org PGP key: http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 This would easier understand fewer had omitted. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-bugs" in the body of the message