#46 [rlibtool]: ar command deduction
Closed 2 weeks ago by RJVB. Opened 2 years ago by RJVB.

Determining which command to execute to as the library archiver (ar) when using the LLVM toolchain remains as problematic as it was (at least) 2 years ago.

For example:

rlibtool  --tag=CC   --mode=link ccache /opt/local/bin/clang-mp-8.0  -Ofast -march=native -g -flto=thin -m64  -Werror=unknown-warning-option -Werror=invalid-command-line-argument -I/opt/local/var/lnxports/build/_opt_local_linux-ports_multimedia_VLC/VLC/work/vlc-3.0.12/contrib/include -Wall -Wextra -Wsign-compare -Wundef -Wpointer-arith -Wvolatile-register-var -Wformat -Wformat-security -Wbad-function-cast -Wwrite-strings -Wmissing-prototypes -Werror-implicit-function-declaration -Winit-self -pipe -fvisibility=hidden -O3 -fno-math-errno -funsafe-math-optimizations -funroll-loops -fomit-frame-pointer -fstack-protector-strong -no-undefined -static -L/opt/local/lib -Wl,--enable-new-dtags -Wl,-rpath,/opt/local/lib -Wl,-rpath,/opt/local/libexec/qt512/lib -Wl,-rpath,/opt/local/x86_64-linux-gnu -Ofast -march=native -g -flto=thin -Wl,-rpath=/opt/local/libexec/vlc3/lib -L/opt/local/var/lnxports/build/_opt_local_linux-ports_multimedia_VLC/VLC/work/vlc-3.0.12/contrib/lib -Wl,-z,defs -o libcompat.la -rpath /opt/local/libexec/vlc3/lib/vlc dummy.lo strlcpy.lo strnstr.lo  -lm 
rlibtool: lconf: {.name="libtool"}.
rlibtool: fdcwd: {.fdcwd=AT_FDCWD, .realpath="/opt/local/var/lnxports/build/_opt_local_linux-ports_multimedia_VLC/VLC/work/vlc-3.0.12/compat"}.
rlibtool: lconf: fstatat(AT_FDCWD,".",...) = 0 {.st_dev = 59, .st_ino = 2946022}.
rlibtool: lconf: openat(AT_FDCWD,"libtool",O_RDONLY,0) = -1 [ENOENT].
rlibtool: lconf: openat(AT_FDCWD,"../",O_DIRECTORY,0) = 3.
rlibtool: lconf: fstat(3,...) = 0 {.st_dev = 59, .st_ino = 2940448}.
rlibtool: lconf: openat(3,"libtool",O_RDONLY,0) = 4.
rlibtool: lconf: found "/opt/local/var/lnxports/build/_opt_local_linux-ports_multimedia_VLC/VLC/work/vlc-3.0.12/libtool".
rlibtool: link: /opt/local/bin/clang-mp-ar crs .libs/libcompat.a .libs/dummy.o .libs/strlcpy.o .libs/strnstr.o
rlibtool: exec error upon slbt_exec_link_create_archive(), line 1380: (see child process error messages).
rlibtool: < returned to > slbt_exec_link(), line 1995.
make[3]: *** [libcompat.la] Error 2

This is no actual child error message: the command /opt/local/bin/clang-mp-ar does not exist.

NB:
- clang is the C language compiler; all the other, non-compiler LLVM tools are called llvm-XXXX, so llvm-ar, llvm-nm or llvm-ranlib.
- the user-visible LLVM toolchain commands almost always have the version suffixed, possible with an additional infix. In general, if the compiler is called clang-X-Y then the archiver will be called llvm-ar-X-Y (or clangXYZ -> llvm-arXYZ).

Indeed, the actual libtool script from the example above (where X=MP and Y=8.0) has this line:

AR="/opt/local/bin/llvm-ar-mp-8.0"

Remember, here CC=/opt/local/bin/clang-mp-8.0.

The current deduction made by rlibtool is thus three times wrong:
- it uses clang as the prefix instead of llvm
- the version suffix is discarded
- the remaining suffix is inserted between the prefix and the tool name, instead of appended to the tool name.

My guess is that this probably extends to the companion commands too (nm and ranlib). It beats me why r/slibtool obtain certain information from the libtool script, but not this (I do remember this came up before but not if ever got a convincing argument why it should NOT be done).


It beats me why r/slibtool obtain certain information from the libtool script

My understanding is that after slibtoolize is completed the need for rlibtool will be entirely removed as long as the autoreconf step is done. Additionally there are plans to internalize the required ar logic which should make this issue moot. :)

It beats me why r/slibtool obtain certain information from the libtool script

My understanding is that after slibtoolize is completed the need for rlibtool will be entirely removed as long as the autoreconf step is done. Additionally there are plans to internalize the required ar logic which should make this issue moot. :)

Yep on both accounts.

Regarding archive merging and archive creation:

For archive creation, I'm not opposed to retrieving the values of AR and RANLIB from the project's generated libtool for those who do not (or cannot) use (the forthcoming) slibtooize+autoreconf.

For archive merging, however, there's no safe way to implement this via temporary directories, and llvm-ar does not support MRI scripts in the first place (at least as far as I can remember) These indeed are the main two reasons why archive merging should be implemented inside of slibtool (with libperk and libelfbites as the backend libraries).

Additionally there are plans to internalize the required ar logic which should make this issue moot. :)

If the logic is correct ...

I can't say if the use of clang always obliges to use llvm-{ar,nm,ranlib} (a priori the latter is integrated in llvm-ar) under normal circumstances. A priori not, I'd say. The only exception is when LTO is used; GNU ar doesn't recognise LLVM bitcode objects, and this case also requires the use of the matching llvm-ar as the bitcode is version specific. In my experience one thus needs to set the AR and NM env. variables when running configure ... which is also a reason why I'm hammering on getting those settings from the libtool script. They're there because I specified them...

libelfbites? Why does that sound like something Linux-specific?

libelfbites? Why does that sound like something Linux-specific?

For the purpose of archive merging, and for as long as the .a archive follows one of the commonly used formats, the binary format of the objects contained in the archive should not matter. If the objects are of a binary format that's not natively supported by slibtool then the merging operation will be followed by an invocation of RANLIB, in which case a setting of RANLIB inside of the generated gnu libtool script will be respected by rlibtool.

To follow up:

  • in-memory archive merging has now been implemented and integrated.

  • nonetheless, the above issue is valid; to be specific, in rlibtool mode, the generation of individual archives (ar -crs) should always derive the AR variable from the located libtool script, and only resort to heuristics when that variable was either not found or was not set.

  • thanks for catching and reporting this!

You're welcome!

I was just telling myself the other day that I should look at r/slibtool to see how it had progressed, thanks for the reminder :)

Should this ticket be closed or does your 2nd point above mean it's too early for that?

You're welcome!

I was just telling myself the other day that I should look at r/slibtool to see how it had progressed, thanks for the reminder :)

Should this ticket be closed or does your 2nd point above mean it's too early for that?

For future reference, will copy below my closing comment for the other llvm-ar issue:-)

  • internal archive merging has now been both implemented and integrated (see 89caff5, and then the minor fix in the following commit c1e423b).

  • in rlibtool mode, AR and RANLIB are now derived from the located libtool (d29f9c6); specifying --ar=llvm=ar on the command line (again, in rlibtool mode) is thus no longer needed.

  • Closing this for now as fixed, but please test and confirm!

Metadata Update from @midipix:
- Issue status updated to: Closed (was: Open)

2 weeks ago
  • please test and confirm!

Working on that, which means I'm going to have to find an appropriate test product. A test build of a project I had been testing in the past failed in the install phase due to a missing static library (that should be missing because "not asked for").

The main interest for me remains the drop-in-libtool-replacement functionality in particular on Mac where libtool can represent a serious bottleneck due to the gatekeeper daemon.

And I already stumbled across a number of problems there, like the use of unknown linker options -soname (instead of -install_name) and --undefined. The missing static archive thing was also on Mac.

A test build of a project I had been testing in the past failed in the install phase due to a missing static library (that should be missing because "not asked for").

Can you post the associated build log? And were you using the rlibtool symlink or just slibtool? With rlibtool you should not be seeing that problem, as it should detect whether static libraries had been disabled. This particular aspect has been working well since the much earlier versions, and the log should be able to shed some light on the issue.

And I already stumbled across a number of problems there, like the use of unknown linker options -soname (instead of -install_name) and --undefined. The missing static archive thing was also on Mac.

I believe we have an open issue for those as well. I would like to get to that soon, but do need someone on the other end of the wire for the purpose of testing ... (some people on irc have also mentioned PureDarwin in a VM as a testing option, if you happen to have some experience with that please let us know ...)

Can you post the associated build log?

That would be way too long I'm afraid. Idem for another project in which I run into a similar issue where a libfoo.a.slibtool.deps file is requested but the presence of a Iibfoo.a.disabled seems to confirm that the fact was detected correctly that I don't want static libraries. BTW, that project built once completely with MAKEFLAGS=LIBTOOL=/path/to/rlibtool but now fails in a way that suggests the linker is invoked without pulling in libfoo at all.
I was about to ask you if you have a test project I could use that's a bit simpler and doesn't require a ton of dependencies.

I want to do some more testing myself on Linux to be certain that none of the changes I had to make broke anything.

BTW, I see there's an optional quiet operation (all that trace info can cause a significant performance hit) but I can't seem to figure out how to activate it (other than through a patch ;) ).

And were you using the rlibtool symlink or just slibtool?

rlibtool, as always in the past.

PureDarwin in a VM as a testing option, if you happen to have some experience with that please let us know ...)

Sadly I don't, but you can run the full OS in VirtualBox nowadays. You can even do that legally if you put an old/dead Apple device under the computer running the VM (you'd be "running on Apple hardware") ;)

You could of course also set up a fork on github and use their CI functionality!

I was about to ask you if you have a test project I could use that's a bit simpler and doesn't require a ton of dependencies.

That actually is a great idea. Can you try building pkgconf then? It only depends on the system library, has a clean autotools-based build system, and appears to correctly build (and cross-build) with the current rlibtool. It also has static libraries disabled by default, I believe.

It looks like that project is a little bit too simple - the build completes just fine (on Mac, which was my main concern).

I think that the problems I've seen are with projects that use an internal static library, although that also doesn't fit with everything I've seen.

(reopening so the comment actually gets posted!)

And since I reopened it: one of the AR-related changes I need to make is replacing *aarg++ = "-Wl,--whole-archive"; with *aarg++ = "-Wl,-force_load,%s"; where %s is the path to the archive at hand. I tried using the lib variable but that's a .la file which obviously cannot be used here. Is the path to the actual archive already available at this point?

I think we can just use asprintf here to allocate the appropriate buffer and accept that little bit of mem leaking as I doubt slibtool will continue to run for long after the final commandline has been constructed and executed. So:

asprintf(aargc, "-Wl,-force_load,%s", pathToArchive);
aargc++;

Metadata Update from @RJVB:
- Issue status updated to: Open (was: Closed)

2 weeks ago

Metadata Update from @RJVB:
- Issue status updated to: Closed (was: Open)

2 weeks ago

Login to comment on this ticket.

Metadata