From owner-freebsd-bugs Fri Oct 18 6:50:10 2002 Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EB9C537B401 for ; Fri, 18 Oct 2002 06:50:05 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4F86D43E3B for ; Fri, 18 Oct 2002 06:50:05 -0700 (PDT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.6/8.12.6) with ESMTP id g9IDo5x3023665 for ; Fri, 18 Oct 2002 06:50:05 -0700 (PDT) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.6/8.12.6/Submit) id g9IDo4e7023662; Fri, 18 Oct 2002 06:50:04 -0700 (PDT) Date: Fri, 18 Oct 2002 06:50:04 -0700 (PDT) Message-Id: <200210181350.g9IDo4e7023662@freefall.freebsd.org> To: freebsd-bugs@FreeBSD.org Cc: From: abc@anchorageinternet.org Subject: Re: misc/44195: globbing/argument limits Reply-To: abc@anchorageinternet.org Sender: owner-freebsd-bugs@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org The following reply was made to PR misc/44195; it has been noted by GNATS. From: abc@anchorageinternet.org To: Peter Pentchev Cc: Subject: Re: misc/44195: globbing/argument limits Date: Fri, 18 Oct 2002 13:46:20 GMT ok - thanks - you were very helpful. i put this stuff in little scripts so on my web site so hopefully people can see some simple techniques and avoid bugging you guys like i did :) i did check around *quite a bit* for the things you answered, and failed to find answers as good as the ones you provided. i was getting grumpy with some things that were frustrating and you made me a happy FBSD user once again :) ps. i leave the following in this email so i have a copy in my sent mail and can study your answers more. nothing new follows. thank you very much. > On Fri, Oct 18, 2002 at 11:57:36AM +0000, abc@anchorageinternet.org wrote: > > > > >Number: 44195 > > > > >Category: misc > > > > >Synopsis: globbing/argument limits > > > > >Originator: Joe Public > > > > >Release: i386 FreeBSD 4.7-RELEASE > > > > >Organization: > > > > no org > > > > >Environment: > > > > ^^^^^^^^^^^^^^^^^^^^^^^^ > > > > >Description: > > > > argument limits painful to users in days of 100GB drives. > > > > >How-To-Repeat: > > > > try a command and give it a few thousand arguments, > > > > > > It is not a matter of how many arguments you give to a command, it is > > > simply a matter of how *long* the command line becomes. Lugging around > > > a multimegabyte command line buffer through shells, execv() system calls > > > and such would be a *major* strain on your system. > > > > i would've assumed the command line stays in place in memory, > > and only a pointer is passed around - and checking the exec(3) > > manpage seems to show this is the case in fact. > > Not when exec(3) invokes the execve(2) system call, as stated in the > very first paragraph of the exec(3) manual page. The execve(2) system > call needs to copy the arguments to kernel space to examine them, and > then to build a single command line for the new process. > > > > > like in file modifying command a folder with 6000 files. > > > > find(1) is too slow, and combining it with xargs is a kludge. > > > > > > If you mean that 'find -exec' is too slow, then I would argue that using > > > -exec is the kludge, when xargs(1) is available. I am pretty sure that > > > the find(1) and xargs(1) utilities were actually developed together, > > > with a common goal in mind, that goal being *exactly* processing of > > > multiple files in one go. > > > > ok - interesting - i appreciate you explaining this. > > it should be in a the FAQ or something. > > > > > The -exec primary to find(1) is extremely inefficient when dealing with > > > many files - it spawns a new process for each file it finds, which, as > > > you note, is too slow. The xargs utility will do a much better job; I > > > would be very interested in what exactly do you consider to be a kludge > > > about it. > > > > i consider it to be a kludge when you have to: > > > > find -s "$I" ! -type d | xargs tar rvf "$I.tar" && \ > > && gzip -f9 "$I.tar" && mv "$I.tar.gz" "$I.tgz" > > > > just to create a sorted tar/gzip archive > > of a directory tree. hehe - as i look at > > it - i say to myself "this is bullshit" :). > > As noted in my response to your other PR, this particular use of find(1) > and tar(1) may be optimized :) Besides, the only "kludge" in that > example is the need to update the tarball incrementally using tar's 'r' > command instead of 'c'; I, personally, would not consider that too large > a price to pay for being able to process the whole list of files at all. > > > i mean - UNIX hackers have got to be smarter > > than to demand all that from a user just to > > accomplish such a minor ordinary task. > > Yep, see both above and below :) > > > also, when you do something like: > > > > find / \! \( -path \*/bin/\* -or -path \*/lib/\* \ > > -or -path \*/libexec/\* -or -path /usr/games/\* \ > > -or -path \*/sbin/\* -or -path /boot/\* \ > > -or -path /dev/\* -or -path /modules/\* \ > > -or -path /proc/\* -or -path /root/\* \) \ > > -type f -exec chown root:wheel {} \;\ > > -exec chmod 0644 {} \; > > > > ie, something a find(1) with 2 -exec's, xargs fails, > > and you are forced to double or triple (or more) the code > > it takes - according to the number of -exec's you need to > > perform - i consider this to be a kludge as well. > > If you need to execute multiple commands, there are several things you > might do. > > The simplest is to create a small shell script, and use xargs(1) to > execute it; the shell script runs chown, chmod, or whatever, on all its > arguments. > > Another way would be capturing find(1)'s output into a file, then > running xargs(1) as many times as needed, redirecting its input to > read this file; something like: > > find / \! \( ... \) > filelist > xargs chown root:wheel < filelist > xargs chmod 0644 < filelist > > Still another way might avoid the temporary file altogether, with some > creative file descriptor hackery. I *think* I have done this before, > but right now, I cannot remember the proper incantations to make the > shell duplicate find(1)'s output to a new file descriptor, run xargs > from fd 1's output, then run another copy of xargs, making it read from > the file descriptor that find(1)'s output was duplicated to. I know it > is possible, it is just that I cannot remember how to do it :) > > > > PS. This will very probably be my last post on this subject, and nobody > > > should be surprised if this PR is closed very soon; what with the recent > > > mailing list "activity", it scores big on my troll indicator. I could > > > be wrong, of course, but I'm just stating my opinion here. > > > > ok - well - no trolling - just installed 4.7, and hit > > a point of frustration with things that have been > > bugging me over the years - that don't see to > > get fixed or improve. > > > > i truly value the effort you made to try to explain, > > though as stated, i still don't see the problem > > in fixing things. > > Apologies for the above paragraph of mine and my attitude in somewhat > summarily dismissing your other PR's at first; there has been quite a > bit of trolling on the various FreeBSD lists recently, and there have > been a couple of bogus PR's filed in the process, so I was a bit > trigger-happy there. > > G'luck, > Peter > > -- > Peter Pentchev roam@ringlet.net roam@FreeBSD.org > PGP key: http://people.FreeBSD.org/~roam/roam.key.asc > Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 > This would easier understand fewer had omitted. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-bugs" in the body of the message