From owner-freebsd-current Fri May 10 16:41:57 2002 Delivered-To: freebsd-current@freebsd.org Received: from avocet.prod.itd.earthlink.net (avocet.mail.pas.earthlink.net [207.217.120.50]) by hub.freebsd.org (Postfix) with ESMTP id C688E37B405 for ; Fri, 10 May 2002 16:41:40 -0700 (PDT) Received: from dialup-209.247.140.188.dial1.sanjose1.level3.net ([209.247.140.188] helo=mindspring.com) by avocet.prod.itd.earthlink.net with esmtp (Exim 3.33 #2) id 176K0j-0001Wq-00; Fri, 10 May 2002 16:41:18 -0700 Message-ID: <3CDC5A7D.40FC75EE@mindspring.com> Date: Fri, 10 May 2002 16:40:45 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Mikhail Teterin Cc: Mikhail Teterin , current@FreeBSD.org Subject: Re: does the order of .a files matter? References: <200205101233.g4ACXctb041093@corbulon.video-collage.com> <200205101343.01636.mi+mx@aldan.algebra.com> <3CDC2F2C.2CB9C818@mindspring.com> <200205101838.35989.mi+mx@aldan.algebra.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-current@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Mikhail Teterin wrote: > = Most linkers don't do what you want, which is make up for programmer > = incompetence by doing an automatic topological sort on all symbol > = dependencies, regardless of where or in what type of file the symbol > = is defined, because most linkers treat archives and libraries very > = differently than lists of object files. > = > = This is the technically correct thing for them to do. > > Doing what you explain by incompetence is no different from listing the > object files (.o) on the linker's command line in an arbitrary order -- > say, alphabetically, as done by most of the FreeBSD's own Makefiles. Not > out of incompetence, but rather for neatness sake. The fact, that the > linker has no problems in this case shows the existing inconsitency. You are wrong. If you had listed the .o files seperately on the command line, then the link would have succeeded, had you not exceeded the command line length limit. An archive is not a macro expansion for a list of the object files it contains. > = This only works intra-archive, not inter-archive. For inter-archive, > = you are expected to order the archives in the proper order, and the > > This expectation is not even documented anywhere... Yes, it is. You need to read the "ld" and "ar" and "ranlib" man pages, paying heavy attention to just what a static archive is and isn't. > = > Tsort is ALREADY incorporated into ld in some shape, because object > = > files on command line or within one .a CAN be misordered. > ************ I saw this. It's irrelevent because you said "or within one .a". It's not true for the "within one .a", either (that's what ranlib is for). The command line linking is not handled by a topological sort. It's handled by the fact that when you list objects on the command line, you are forcing their incorporation into the resulting linker output. When you specify archives on the command line, you are NOT; you are saying "pull the objects in this thing in, if you see that they resolve unresolved externals *known to you at the time you encounter the archive, in order, on the command line*". Very different. > = Within one .a, they are only permitted to be misordered if "ranlib" > = has been run on the archive (see the quotation of the ranlib manual > = page, above). > > Your attempt to dodge explaining the other case -- that of misordered > object files _on command line_ has been logged. I'm not attempting to dodge anything. Your expectations of an archive being treated as a list of the objects archived in it, is wrong. > = Within multiple .a's, they are handled differently, because linking > = against a .a file does not necessarily pull in all of the object files > = in the archive. *This is intentional; it is by design*. > > Design of what? Of linker? Of the interaction of archive files and the linker. > If so, the design is inconsistent (broken?), In your opinion. > for it handles the same entities (object files) differently depending on > how they are given to it (in a single .a, on command line, in multiple > .a files). Your opinion disagrees with the documentation. At the command line, type "info ld"; if you read the documentation, you will find: The linker will search an archive only once, at the location where it is specified on the command line. If the archive defines a symbol which was undefined in some object which appeared before the archive on the command line, the linker will include the appropriate file(s) from the archive. However, an undefined symbol in an object appearing later on the command line will not cause the linker to search the archive again. See the `-(' option for a way to force the linker to search archives multiple times. You may list the same archive multiple times on the command line. This type of archive searching is standard for Unix linkers. However, if you are using `ld' on AIX, note that it is different from the behaviour of the AIX linker. > = It's my understanding that you are making libraries in the first place > = in order to get around command line length limitations, and have > = settled upon archives, rather than incremental linking using "ld -r -o > = A.o ${A_OBJS}". > > Not quite. The software's original build is broken up into dozens of > interdependent libraries. They had no problems linking together on > neither Solaris, nor AIX, nor HP-UX, nor NT. NT and AIX, I understand. If you didn't have a problem on Solaris, I'd have to guess that you used the C++ compiler, rather than the standard SunSoft ANSI C compiler. Solaris does a number of things that it shouldn't, and it leads to programmers making bad assumptions about things like static declaration of template class constructor instances. On occasion, rather than generating link time errors, you end up with run time errors, instead, which is really bad, if your testing doesn't test each and every code path in your code. > = If this is the case, it would probably be a good idea to choose > = which objects go into which library carefully, to avoid ending > = up with undefined symbols. > > I'm not (yet) developing this software. I'm just trying to port it! Once again, reading the documentation... `-( ARCHIVES -)' `--start-group ARCHIVES --end-group' The ARCHIVES should be a list of archive files. They may be either explicit file names, or `-l' options. The specified archives are searched repeatedly until no new undefined references are created. Normally, an archive is searched only once in the order that it is specified on the command line. If a symbol in that archive is needed to resolve an undefined symbol referred to by an object in an archive that appears later on the command line, the linker would not be able to resolve that reference. By grouping the archives, they all be searched repeatedly until all possible references are resolved. Using this option has a significant performance cost. It is best ***************************************************************** to use it only when there are unavoidable circular references ************************************************************* between two or more archives. **************************** `--whole-archive' For each archive mentioned on the command line after the `--whole-archive' option, include every object file in the archive in the link, rather than searching the archive for the required object files. This is normally used to turn an archive file into a shared library, forcing every object to be included in the resulting shared library. This option may be used more than once. Two notes when using this option from gcc: First, gcc doesn't know about this option, so you have to use `-Wl,-whole-archive'. Second, don't forget to use `-Wl,-no-whole-archive' after your ****************************************************** list of archives, because gcc will add its own list of archives to ****************************************************************** your link and you may not want this flag to affect those as well. **************************************************************** Circular references are bad. Dragging everything in is bad. > = Alternately, if you insist on using ".a" files directly, as if they > = were normal object files, someone has already posted that you should > = probably use the "--whole-archive" argument, so that the archive > = contents aren't pulled in *only* if they define symbols which are in > = the current undefined symbol list, but to pull them in unconditionally > = (i.e. "treat them as a list of object files instead of an archive"). > > That suggestion I missed. But it looks like --whole-archive will suck in > everything, including the really unneeded objects. Yes. It's for "unavoidable circular references" ... or for "avoidable circular references that the programmer chooses not to avoid". > Perhaps, the linkers on the commercial OSes I listed do use some smarter > equivalent of ``--whole-archive'' by default. IMO, it makes sense, since > those, who know, THEIR libraries are organized properly, and care for > the speed of linking can always add --no-whole-archive (or equivalent) > to THEIR makefiles -- speeding up their ``build world''. AIX and NT implicitly do the "--start-group ... --end-group" by default. AIX does it because of the way it implements system calls (it's a side effect of the implemetnation of entrypoint resolution). NT seems to do it to make source code harder to port from NT to UNIX. The Solaris stuff is "different". I won't go into how it really works in detail, since the user visible effects are similar to what you want, even though it can cause C++ programs to explode at runtime when they should explode at compile time, instead. > I'm now readying my horse and armor for the trip to figure out, why is > this sort of gymnastics only needed with the OSF toolchain (actually, I > may be wrong here, since I did not _personally_ verify this thing links > on the other systems without the hoop-jumping). The short answer is "This is how UNIX is supposed to work". OSF, SVR4, Linux, FreeBSD, Ultrix, Huerikon, Unisys, NCR, Arete, Pyramid, Altos, Microport, and about 140 other UNIX distributions linkers *all* work the same way (not SCO Xenix; SCO Xenix uses a Microsoft toolchain). > =Libraries are not object files, and they are not lists of object files. > > The documentation suggests otherwise. From ar(1): > > The GNU ar program creates, modifies, and extracts from > archives. An archive is a single file holding a collec- > tion of other files in a structure that makes it possible > to retrieve the original individual files (called members > of the archive). > > In fact, Debian's .deb files are compressed ar-balls, just like > FreeBSD's .tgz packages are compressed tar-balls (less confusingly > named, though). You need to read the "info" documentation. It is significantly more detailed than the man pages. > = A true library implies that you have done the dependency ordering > = necessary to avoid the problems you are seeing because you are using > = libraries and *have not done that*. > > I wish, you could point me to some documentation on that. "info ld" > = In other words, *do not use "--whole-archive", please* > > It seems, --whole-archive is likely to bloat the executable with > the stuff, that is genuinly unneeded, so, no, I'm not going to use > that. If the only reason you are using archives is because you are trying to overcome the command line limitations, you are going to be doomed to pulling in all the objects, anyway, since that's what it wants in reality. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message