FFI: Linking Against Shared Libraries
Every decent implementation of Common Lisp has a foreign function interface. When we need to call functions defined in external libraries and following a calling convention of C, we tell our Lisp system (1) where to find the library, (2) how to find the function, (3) what types of arguments it expects. Then we call it.
There is a common portability layer for many FFI implementations: CFFI. I'll use it for my example here, but the problem I'm thinking of doesn't depend on it. It does depend, however, on OS dlopen facility that we find in the POSIX world.
What we see in many Lisp libraries that use CFFI:
(define-foreign-library frobnicator (cffi-features:unix "/usr/lib/libfrobnicate.so")) (load-foreign-library 'frobnicator)
How many problems do you see in this code? Zero? Then delay publishing your libraries which use FFI, until you finish reading this entry.
Two things are wrong with "/usr/lib/libfrobnicator.so":
Hardcoded library path. When a developer had to specify a foreign library to load, he probably looked up its location on his own system and added it to define-foreign-library, without thinking that it could have another location on the user's machine.
Misguided attempts to get it right are likely to add something like "/usr/lib/x86_64-linux-gnu/libfrobnicator.so" to the picture, possibly conditionalized on cffi-features:x86-64. It remains wrong. The only right thing to do is to omit the directory and rely on dlopen() to figure it out.
Underlying native FFI may be incapable of doing it this way, doing a user-level library search before dlopen. Then native FFI is wrong: complain to your vendor about your Common Lisp implementation.
Developer's library used at runtime. When a shared library is installed on a typical unix-like system, we see something like libfrobnicate.so.1.2.3, with libfrobnicate.so.1.2 and libfrobnicate.so.1 as symlinks, and maybe libfrobnicate.so symlink as well. How should we decide which name to use in define-foreign-library?
If it's a Linux distribution with a package manager, an important clue can be acquired by examining packages where this files and symlinks belong. A version-less libfrobnicate.so is usually in a separate package, with its name ending in -dev or -devel, together with include files which make it possible to compile programs depending on this library. It's not really different on most Unix-like systems other than Linux, even if it's not that obvious from package layouts.
For ELF-based systems, each shared library has a soname that serves as a key when a dynamic loader searches for program or library dependencies; there is an entry in the library header where the soname is stored. The soname of our libfrobnicate.so is probably "libfrobnicate.so.1" (no "/usr/lib/" or other paths; a major version follows the suffix, subversions omitted).
When we build a C program, the linker's command line includes something like -lfrobnicate (no suffix, no directory). The linker searchs some directories (depending on its configuration and command line arguments) for libfrobnicate.so, then it reads a soname from the file (libfrobnicate.so) itself, and this soname ends up in a generated executable. Thus "libfrobnicate.so" is never used for loading the program after it was compiled; only its soname ("libfrobnicate.so.1") is seen and used by the dynamic loader.
If we use a bare soname with no paths in our foreign library reference, we maximize the ability for OS to do the right thing on a user's machine. Unless we know for sure that OS is incapable of doing the right thing anyway, our code should be like this:
(define-foreign-library frobnicator (cffi-features:unix "libfrobnicate.so.1")) (load-foreign-library 'frobnicator)
These is a common objection to this practice: "I don't care on the exact major version". Well, the reason to have major versions in sonames is allowing library vendors to make any incompatible change to the library interface. Are you sure that your code will survive any incompatible change? Then I would like to see your code and learn to provide such robustness.
Several major versions can have enough in common to allow Lisp code to remain the same. Fortunately, CFFI allows a list of alternatives instead of a single name:
(define-foreign-library frobnicator (cffi-features:unix (:or "libfrobnicate.so.2" "libfrobnicate.so.1")))
More alternatives can be added easily when the old code is verified to work with newer and newer major versions of the library. That's the only way to do it reliably. Remember that for an incompatible major upgrade, getting an instant failure is our best hope: sometimes, the interface change will cause our code to do the wrong thing silently, or worse, to work most of the time and fail spontaneously.
There is a class of situations where it can be right to use something other than soname, and even to hardcode library path. Some shared libraries are not supposed to be integrated with the rest of the system: linker won't normally see it when looking for -lname, and dynamic loader won't find it where expected. It's often the case for modules loaded into scripting languages, such as Perl, Python or TCL. If you call such a library from Common Lisp (and it may be actually useful), feel free to hardcode something like "/usr/lib/perl/5.14/auto/Encode/Encode.so" into your FFI code.
Most of the time, linking against unversioned name of a shared library seems to work, as long as you have it installed (that's why this practice has become so widespread). There are some exceptions, however. It may be interesting for Linux users to take a look at libncurses.so or libc.so under their /usr hierarchy (in /usr/lib, /usr/lib32 or /usr/lib/x86_64-linux-gnu): these files are not shared libraries at all. They contain linker directives, so the linker has a way to know where to find real libraries.
Dynamic loader doesn't understand linker directives, and it shouldn't (I've read a suggestion on #lisp channel that it really should, or it's broken). Yet another thing that may go wrong with library upgrades is just that: one day, an unversioned name that was a symbolic link may become a text file with linker directives. C programmers won't notice any difference, even if it's done within the same major version of a library. Common Lisp code will continue to work only if you follow a good practice of using soname for dynamic linking.