#47 Library path priority plight
Closed 8 months ago by midipix. Opened 3 years ago by mjo.

(A follow-up to what we discussed on IRC)

During linking, slibtool converts any libtool archive (*.la) arguments into -L<path> -l<library> flags. For example, when building eclib,

rlibtool --tag=CXX --mode=link ... -o solve_conic ../libsrc/libec.la

produces the linker command,

x86_64-pc-linux-gnu-g++ ... -L../libsrc/.libs -lec -o .libs/solve_conic

Compare with GNU libtool, which transforms the *.la file argument into a *.so (on Linux, anyway):

x86_64-pc-linux-gnu-g++ ... ../libsrc/.libs/libec.so -o .libs/solve_conic

The slibtool approach is more elegant, because it avoids encoding the (platform-specific) shared library extension on the command-line, but it's not equivalent in all cases to what was intended. Or more to the point: it's not equivalent to what GNU libtool does.

Specifically, -L<path> -l<library> will use the library search path to find <library>, and the priority given to <path> depends on where in the command-line -L<path> appears. If the user has other -L flags in his LDFLAGS -- or if the build system puts them there -- then it's possible that another copy of <library> will be picked up from one of those locations that has a higher precedence than <path>.

For a concrete example, the ax_boost_base.m4 macro from the autoconf-archive will add the system's library location to the command-line explicitly as e.g. -L/usr/lib64. When that happens, the transformed command gives priority to /usr/lib64/<library> over the copy in <path> that was just built. This can result in build failures if the existing, installed library has dependencies that are incompatible with the one just built. And presumably it could cause test failures in the package regardless.

Suggested fix: if possible, slibtool could rearrange the -L flags to ensure that -L<path> is given priority over any other -L flags before passing them to the linker.


Metadata Update from @midipix:
- Issue assigned to midipix

9 months ago

Thanks for the detailed and insightful report!

In a world not yet ideal this could (and probably should) be addressed on slibtool's side. But before we take the plunge -- ax_boost_base.m4, seriously? Hard-coding a system library path before everything else is wrong on multiple levels and should definitely be fixed upstream... Nonetheless:

  • the first possible solution is to do exactly as you have suggested, however that might get a little bit tricky due to the underlying assumption, if I understand it correctly, that the package being built will -L it's newly built libraries using a path relative to the current working directly. That is almost always true, and implementation should not be too hard given that slibtool already finalizes the argument vector as the last step before executing the link command.

  • alternatively, and more surgically, we could move "up" all -L arguments that refer to locations associated with the current build, specifically based on their .libs/ suffix.

Any thoughts?

  • the first possible solution is to do exactly as you have suggested, however that might get a little bit tricky due to the underlying assumption, if I understand it correctly, that the package being built will -L it's newly built libraries using a path relative to the current working directly. That is almost always true, and implementation should not be too hard given that slibtool already finalizes the argument vector as the last step before executing the link command.

  • alternatively, and more surgically, we could move "up" all -L arguments that refer to locations associated with the current build, specifically based on their .libs/ suffix.

The plot thickens, because ...

  • you cannot move an -L argument "up" to a point where it would precede an existing -l argument since that would change linking semantics and is thus not permitted.

  • similarly, you cannot move "up" the -l argument together with the -L argument, again since that would change linking semantics and will likely cause the build to fail (as libraries need to be specified after the object files that depend on them).

Is it possible that this is just a ax_boost_base bug after all? We haven't seen any other instance where the current logic led to undesired results, and so I wonder whether the only real solution is for ax_boost_base.m4 to be fixed.

Trying to be smart and reordering flags to be seems like it could lead to unforeseen consequences.

Take this xmlsec build issue I ran into the other day. https://github.com/lsh123/xmlsec/pull/756

As described in the PR description the program was trying to link some system libraries with -Wl,-Bstatic and others with -Wl,-Bdynamic, but the former doesn't work if only dynamic libraries are installed which went unnoticed because GNU libtool discarded the intent by being clever and ordering all of the -Wl flags at the end of linker command.

While the issue described here is different I wonder if perhaps its better if this was fixed in boost.

The recent batch of commits entailed several enhancements and refinements in slbt_adjust_linker_argument() and slbt_exec_link_finalize_argument_vector which come to ensure selection of the correct variant of an internal dependency (.a or .so) while retaining relative directory semantics. This is is not the issue reported here, but I'm mentioning this since our original discussion might have actually been motivated by a manifestation of the .a vs. .so in disguise.

As a general rule, build-project -L arguments should always precede install-library (or "system") -L arguments. Arguably, violating this principle is a bug that should be fixed irrespective of the presence or absence of negative side effects. Even more so, and since the order of -l arguments with respect to -L arguments is significant, each -l argument pertaining to a newly built library must be preceded by a corresponding -L argument. Here, too, some linkers would search for a library in a folder specified after the corresponding -l argument, which is something that slibtool now (as of commit c5c351d) attempts to prevent (the linker might still do the wrong thing, though: given -lfoo -L/path/to/libfoo -lbar, slibtool will keep -the -L intact (since followed by another -l argument), and the linker might still search for libfoo.so (and libfoo.a) in it even though it shouldn't.

My suggestion is that we close this issue for now, and revisit or reopen it in the future should the need for that arise.

Hm I must've missed these emails three weeks ago. The issue in this case is 100% that the boost macro is garbage. I only reported it here because it's a case where GNU libtool manages to succeed but slibtool does not.

Naturally, reordering the linker flags does have some potential to break things (changing the existing behavior is the whole point), but my guess was that giving preference to .libs/ would be less likely to hurt than allowing the earlier paths to override it. But who knows.

In any case, I'm fine with closing the issue.

Hm I must've missed these emails three weeks ago. The issue in this case is 100% that the boost macro is garbage. I only reported it here because it's a case where GNU libtool manages to succeed but slibtool does not.

Naturally, reordering the linker flags does have some potential to break things (changing the existing behavior is the whole point), but my guess was that giving preference to .libs/ would be less likely to hurt than allowing the earlier paths to override it. But who knows.

In any case, I'm fine with closing the issue.

Thanks for following up, sounds good! And for what it's worth, the gnu libtool testsuite actually has a test entitled uninstalled libraries have priority, so I guess we are keeping the spirit after all, ... cheers! :wine_glass:

PS. please make sure to test everything with slibtool-0.5.36, best ever, which is expected in a day or two.

Metadata Update from @midipix:
- Issue status updated to: Closed (was: Open)

8 months ago

Login to comment on this ticket.

Metadata