Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 10 May 2002 16:40:45 -0700
From:      Terry Lambert <tlambert2@mindspring.com>
To:        Mikhail Teterin <mi+mx@aldan.algebra.com>
Cc:        Mikhail Teterin <mi@corbulon.video-collage.com>, current@FreeBSD.org
Subject:   Re: does the order of .a files matter?
Message-ID:  <3CDC5A7D.40FC75EE@mindspring.com>
References:  <200205101233.g4ACXctb041093@corbulon.video-collage.com> <200205101343.01636.mi%2Bmx@aldan.algebra.com> <3CDC2F2C.2CB9C818@mindspring.com> <200205101838.35989.mi%2Bmx@aldan.algebra.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Mikhail Teterin wrote:
> = Most linkers don't do what you want, which is make up for programmer
> = incompetence by doing an automatic topological sort on all symbol
> = dependencies, regardless of where or in what type of file the symbol
> = is defined, because most linkers treat archives and libraries very
> = differently than lists of object files.
> =
> = This is the technically correct thing for them to do.
> 
> Doing what you explain by incompetence is no different from listing the
> object files (.o) on the linker's command line in an arbitrary order --
> say, alphabetically, as done by most of the FreeBSD's own Makefiles. Not
> out of incompetence, but rather for neatness sake. The fact, that the
> linker has no problems in this case shows the existing inconsitency.

You are wrong.

If you had listed the .o files seperately on the command line,
then the link would have succeeded, had you not exceeded the
command line length limit.

An archive is not a macro expansion for a list of the object
files it contains.


> = This only works intra-archive, not inter-archive. For inter-archive,
> = you are expected to order the archives in the proper order, and the
> 
> This expectation is not even documented anywhere...

Yes, it is.  You need to read the "ld" and "ar" and "ranlib" man
pages, paying heavy attention to just what a static archive is
and isn't.


> = > Tsort is ALREADY incorporated into ld in some shape, because object
> = > files on command line or within one .a CAN be misordered.
>              ************

I saw this.  It's irrelevent because you said "or within one .a".

It's not true for the "within one .a", either (that's what ranlib
is for).

The command line linking is not handled by a topological sort.
It's handled by the fact that when you list objects on the command
line, you are forcing their incorporation into the resulting linker
output.  When you specify archives on the command line, you are NOT;
you are saying "pull the objects in this thing in, if you see that
they resolve unresolved externals *known to you at the time you
encounter the archive, in order, on the command line*".

Very different.


> = Within one .a, they are only permitted to be misordered if "ranlib"
> = has been run on the archive (see the quotation of the ranlib manual
> = page, above).
> 
> Your attempt to dodge explaining the other case -- that of misordered
> object files _on command line_ has been logged.

I'm not attempting to dodge anything.  Your expectations of an
archive being treated as a list of the objects archived in it,
is wrong.



> = Within multiple .a's, they are handled differently, because linking
> = against a .a file does not necessarily pull in all of the object files
> = in the archive. *This is intentional; it is by design*.
> 
> Design of what? Of linker?

Of the interaction of archive files and the linker.

> If so, the design is inconsistent (broken?),

In your opinion.

> for it handles the same entities (object files) differently depending on
> how they are given to it (in a single .a, on command line, in multiple
> .a files).

Your opinion disagrees with the documentation.  At the command
line, type "info ld"; if you read the documentation, you will find:

     The linker will search an archive only once, at the location where
     it is specified on the command line.  If the archive defines a
     symbol which was undefined in some object which appeared before
     the archive on the command line, the linker will include the
     appropriate file(s) from the archive.  However, an undefined
     symbol in an object appearing later on the command line will not
     cause the linker to search the archive again.

     See the `-(' option for a way to force the linker to search
     archives multiple times.

     You may list the same archive multiple times on the command line.

     This type of archive searching is standard for Unix linkers.
     However, if you are using `ld' on AIX, note that it is different
     from the behaviour of the AIX linker.


> = It's my understanding that you are making libraries in the first place
> = in order to get around command line length limitations, and have
> = settled upon archives, rather than incremental linking using "ld -r -o
> = A.o ${A_OBJS}".
> 
> Not quite. The software's original build is broken up into dozens of
> interdependent libraries. They had no problems linking together on
> neither Solaris, nor AIX, nor HP-UX, nor NT.

NT and AIX, I understand.

If you didn't have a problem on Solaris, I'd have to guess that
you used the C++ compiler, rather than the standard SunSoft ANSI
C compiler.

Solaris does a number of things that it shouldn't, and it leads
to programmers making bad assumptions about things like static
declaration of template class constructor instances.

On occasion, rather than generating link time errors, you end up
with run time errors, instead, which is really bad, if your testing
doesn't test each and every code path in your code.


> = If this is the case, it would probably be a good idea to choose
> = which objects go into which library carefully, to avoid ending
> = up with undefined symbols.
> 
> I'm not (yet) developing this software. I'm just trying to port it!

Once again, reading the documentation...

  `-( ARCHIVES -)'
  `--start-group ARCHIVES --end-group'

     The ARCHIVES should be a list of archive files.  They may be
     either explicit file names, or `-l' options.

     The specified archives are searched repeatedly until no new
     undefined references are created.  Normally, an archive is
     searched only once in the order that it is specified on the
     command line.  If a symbol in that archive is needed to resolve an
     undefined symbol referred to by an object in an archive that
     appears later on the command line, the linker would not be able to
     resolve that reference.  By grouping the archives, they all be
     searched repeatedly until all possible references are resolved.

     Using this option has a significant performance cost.  It is best
     *****************************************************************
     to use it only when there are unavoidable circular references
     *************************************************************
     between two or more archives.
     ****************************

  `--whole-archive'

     For each archive mentioned on the command line after the
     `--whole-archive' option, include every object file in the archive
     in the link, rather than searching the archive for the required
     object files.  This is normally used to turn an archive file into
     a shared library, forcing every object to be included in the
     resulting shared library.  This option may be used more than once.

     Two notes when using this option from gcc: First, gcc doesn't know
     about this option, so you have to use `-Wl,-whole-archive'.
     Second, don't forget to use `-Wl,-no-whole-archive' after your
             ******************************************************
     list of archives, because gcc will add its own list of archives to
     ******************************************************************
     your link and you may not want this flag to affect those as well.
     ****************************************************************

Circular references are bad.  Dragging everything in is bad.


> = Alternately, if you insist on using ".a" files directly, as if they
> = were normal object files, someone has already posted that you should
> = probably use the "--whole-archive" argument, so that the archive
> = contents aren't pulled in *only* if they define symbols which are in
> = the current undefined symbol list, but to pull them in unconditionally
> = (i.e. "treat them as a list of object files instead of an archive").
> 
> That suggestion I missed. But it looks like --whole-archive will suck in
> everything, including the really unneeded objects.

Yes.  It's for "unavoidable circular references" ... or for
"avoidable circular references that the programmer chooses not
to avoid".


> Perhaps, the linkers on the commercial OSes I listed do use some smarter
> equivalent of ``--whole-archive'' by default. IMO, it makes sense, since
> those, who know, THEIR libraries are organized properly, and care for
> the speed of linking can always add --no-whole-archive (or equivalent)
> to THEIR makefiles -- speeding up their ``build world''.

AIX and NT implicitly do the "--start-group ... --end-group" by default.
AIX does it because of the way it implements system calls (it's a side
effect of the implemetnation of entrypoint resolution).  NT seems to do
it to make source code harder to port from NT to UNIX.

The Solaris stuff is "different".  I won't go into how it really works
in detail, since the user visible effects are similar to what you
want, even though it can cause C++ programs to explode at runtime
when they should explode at compile time, instead.


> I'm now readying my horse and armor for the trip to figure out, why is
> this sort of gymnastics only needed with the OSF toolchain (actually, I
> may be wrong here, since I did not _personally_ verify this thing links
> on the other systems without the hoop-jumping).

The short answer is "This is how UNIX is supposed to work".  OSF, SVR4,
Linux, FreeBSD, Ultrix, Huerikon, Unisys, NCR, Arete, Pyramid, Altos,
Microport, and about 140 other UNIX distributions linkers *all* work
the same way (not SCO Xenix; SCO Xenix uses a Microsoft toolchain).


> =Libraries are not object files, and they are not lists of object files.
> 
> The documentation suggests otherwise. From ar(1):
> 
>         The GNU ar program creates, modifies,  and  extracts  from
>         archives.   An  archive is a single file holding a collec-
>         tion of other files in a structure that makes it  possible
>         to  retrieve the original individual files (called members
>         of the archive).
> 
> In fact, Debian's .deb files are compressed ar-balls, just like
> FreeBSD's .tgz packages are compressed tar-balls (less confusingly
> named, though).

You need to read the "info" documentation.  It is significantly
more detailed than the man pages.


> = A true library implies that you have done the dependency ordering
> = necessary to avoid the problems you are seeing because you are using
> = libraries and *have not done that*.
> 
> I wish, you could point me to some documentation on that.

"info ld"


> = In other words, *do not use "--whole-archive", please*
> 
> It seems, --whole-archive is likely to bloat the executable with
> the stuff, that is genuinly unneeded, so, no, I'm not going to use
> that.

If the only reason you are using archives is because you are
trying to overcome the command line limitations, you are going
to be doomed to pulling in all the objects, anyway, since that's
what it wants in reality.

-- Terry

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-current" in the body of the message




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3CDC5A7D.40FC75EE>