Library path in gcc

Sang -

There are three stages of making your C code run: compilation, linking, and runtime.

gcc is mostly responsible for the compilation, while when linking, it uses a linker program (e.g., ld) with some additional flags.

  1. In compilation, gcc requires only header (.h) files to ensure that external function calls have correct parameters matched with the definition.
  2. Next, at the linking stage, the linker ld connects/links all compiled objects and libraries to an executable file by wrapping some appropriate headers after combining the objects. This stage produces the executable file or linkable library. In windows, it is PE (Portable Executables format: EXE, DLL, or OCX), in Linux, it is ELF (Executable and Linkable format: no extension, .so, .dynlib).
  3. Runtime is when you request the OS to execute the executable file.

There are two types of libraries: static library (in Linux .a, in Window .lib) and dynamic library ( .so, .dynlib, dll, .ocx). The static libraries are copied and embedded directly into the executable file. In opposite, the dynamic libraries are not. The dynamic libraries are supposed to exist in the system where the executable file is executed at runtime.

So, where does gcc look for the header files and the dynamic libraries?

Check the configuration

In compilation

For c: echo | gcc -x c -E -Wp,-v - >/dev/null
For c++: echo | gcc -x c++ -E -Wp,-v - >/dev/null

These are the outputs on my PC.

For C

ignoring nonexistent directory "/usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/../../../../x86_64-pc-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/include
 /usr/local/include
 /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/include-fixed
 /usr/include
End of search list.
echo | gcc -x c -E -Wp,-v - >/dev/null

For c++

ignoring nonexistent directory "/usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/../../../../x86_64-pc-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/include
 /usr/local/include
 /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/include-fixed
 /usr/include
End of search list.
echo | gcc -x c++ -E -Wp,-v - >/dev/null

In linking

With ld command

To list ld's search directories:

ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012

The output from my PC.

In Arch Linux:

SEARCH_DIR("/usr/x86_64-pc-linux-gnu/lib64")
SEARCH_DIR("/usr/lib")
SEARCH_DIR("/usr/local/lib")
SEARCH_DIR("/usr/x86_64-pc-linux-gnu/lib")
ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012

In Ubuntu:

SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu")
SEARCH_DIR("=/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/lib/x86_64-linux-gnu64")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib")
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")
SEARCH_DIR("=/usr/x86_64-linux-gnu/lib64")
SEARCH_DIR("=/usr/x86_64-linux-gnu/lib")
ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012

To debug ld, use the LD_DEBUG environment variable. See help with LD_DEBUG=help ld.

Valid options for the LD_DEBUG environment variable are:

  libs        display library search paths
  reloc       display relocation processing
  files       display progress for input file
  symbols     display symbol table processing
  bindings    display information about symbol binding
  versions    display version dependencies
  scopes      display scope information
  all         all previous options combined
  statistics  display relocation statistics
  unused      determined unused DSOs
  help        display this help message and exit

To direct the debugging output into a file instead of standard output
a filename can be specified using the LD_DEBUG_OUTPUT environment variable.
LD_DEBUG=help ld

The most useful value to debug library path searching is LD_DEBUG=libs ld foo. Sample output on my PC:

     96426:	find library=libbfd-2.38.so [0]; searching
     96426:	 search cache=/etc/ld.so.cache
     96426:	  trying file=/usr/lib/libbfd-2.38.so
     96426:	
     96426:	find library=libctf.so.0 [0]; searching
     96426:	 search cache=/etc/ld.so.cache
     96426:	  trying file=/usr/lib/libctf.so.0
     96426:	
     96426:	find library=libc.so.6 [0]; searching
     96426:	 search cache=/etc/ld.so.cache
     96426:	  trying file=/usr/lib/libc.so.6
     96426:	
     96426:	find library=libz.so.1 [0]; searching
     96426:	 search cache=/etc/ld.so.cache
     96426:	  trying file=/usr/lib/libz.so.1
     96426:	
     96426:	
     96426:	calling init: /lib64/ld-linux-x86-64.so.2
     96426:	
     96426:	
     96426:	calling init: /usr/lib/libc.so.6
     96426:	
     96426:	
     96426:	calling init: /usr/lib/libz.so.1
     96426:	
     96426:	
     96426:	calling init: /usr/lib/libbfd-2.38.so
     96426:	
     96426:	
     96426:	calling init: /usr/lib/libctf.so.0
     96426:	
     96426:	
     96426:	initialize program: ld
     96426:	
     96426:	
     96426:	transferring control: ld
     96426:	
ld: /home/transang/foo/dqrdc2.o: undefined reference to symbol 'memmove@@GLIBC_2.2.5'
ld: /usr/lib/libc.so.6: error adding symbols: DSO missing from command line
     96426:	
     96426:	calling fini: ld [0]
     96426:	
     96426:	
     96426:	calling fini: /usr/lib/libctf.so.0 [0]
     96426:	
     96426:	
     96426:	calling fini: /usr/lib/libbfd-2.38.so [0]
     96426:	
     96426:	
     96426:	calling fini: /usr/lib/libz.so.1 [0]
     96426:
LD_DEBUG=libs ld foo

With GCC command

gcc wraps some flags when calling ld. Therefore, the directories list is different. To list them:

gcc -print-search-dirs | sed '/^lib/b 1;d;:1;s,/[^/.][^/]*/\.\./,/,;t 1;s,:[^=]*=,:;,;s,;,;  ,g' | tr \; \\012 | tr : \\012

The output from my PC.

In Arch Linux:

libraries

  /usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/
/usr/x86_64-pc-linux-gnu/lib/x86_64-pc-linux-gnu/12.1.0/
/usr/x86_64-pc-linux-gnu/lib/
/usr/lib/x86_64-pc-linux-gnu/12.1.0/
/usr/lib/
/lib/x86_64-pc-linux-gnu/12.1.0/
/lib/
/usr/lib/x86_64-pc-linux-gnu/12.1.0/
/usr/lib/
/usr/x86_64-pc-linux-gnu/lib/
/usr/lib/
/lib/
/usr/lib/
gcc -print-search-dirs | sed '/^lib/b 1;d;:1;s,/[^/.][^/]*/\.\./,/,;t 1;s,:[^=]*=,:;,;s,;,; ,g' | tr \; \\012 | tr : \\012

In Ubuntu:

libraries

  /usr/lib/gcc/x86_64-linux-gnu/7/
/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu/7/
/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu/
/usr/x86_64-linux-gnu/lib/
/usr/lib/x86_64-linux-gnu/7/
/usr/lib/x86_64-linux-gnu/
/usr/lib/
/lib/x86_64-linux-gnu/7/
/lib/x86_64-linux-gnu/
/lib/
/usr/lib/x86_64-linux-gnu/7/
/usr/lib/x86_64-linux-gnu/
/usr/lib/
/usr/x86_64-linux-gnu/lib/
/usr/lib/
/lib/
/usr/lib/
gcc -print-search-dirs | sed '/^lib/b 1;d;:1;s,/[^/.][^/]*/\.\./,/,;t 1;s,:[^=]*=,:;,;s,;,; ,g' | tr \; \\012 | tr : \\012
  • My gcc's build information with gcc --verbose.

In Arch Linux:

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/12.1.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-bootstrap --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --with-linker-hash-style=gnu --with-system-zlib --enable-__cxa_atexit --enable-cet=auto --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-default-ssp --enable-gnu-indirect-function --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-multilib --enable-plugin --enable-shared --enable-threads=posix --disable-libssp --disable-libstdcxx-pch --disable-werror --with-build-config=bootstrap-lto --enable-link-serialization=1
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.1.0 (GCC)
gcc --verbose

In Ubuntu:

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.4.0-1ubuntu1~18.04.1' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
gcc --verbose

Note: gcc looks for the libraries' names from left to right and stops finding when it matches the first library with the searching term.

To pass additional flags to the underlining ld command when invoking gcc. Use -Wl,<command separated flags>.  The following command options have the same effect:

gcc -Wl,flag1,-flag2,flag3,flag4
# same as
gcc -Wl,flag1,-flag2 -Wl,flag3,flag4
# same as
ld flag1 -flag2 flag3 flag4
  • You also can show the library search directories list by adding the verbose flag -v when linking.
    For example: gcc -v foo.o bar.o -o foo

In runtime

There are many tools to analyze ELF files, namely: ldd (Linux only), otool (Mac only), objdump (Linux only), nm (POSIX), readelf.

ldd command (Linux only)

To figure out which libraries are linked with a program and the libraries' full path, use ldd. For example, ldd foo. ldd is the most useful command because it recursively looks for required dynamic libraries and outputs their paths in the running system, while other commands only look for direct dependencies.

Sample output on my PC.

	linux-vdso.so.1 (0x00007fff08bf3000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbc85170000)
	libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007fbc85110000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbc84f1f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fbc852e8000)
	libopenblas.so.0 => /lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007fbc82d3a000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbc82d17000)
	libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007fbc82a77000)
	libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007fbc82a2b000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fbc82a11000)

Similar to ld command, LD_DEBUG environment variable can be used to debug ldd. In fact, setting LD_DEBUG also outputs the debug information when executing a command, e.g., LD_DEBUG=all ./foo.

otool command (Mac)

Use otool -L foo. Sample output:

foo:
	mylib.so (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

objdump command (Linux)

Use objdump -p foo | grep NEEDED. Sample output:

  NEEDED               libm.so.6
  NEEDED               libmvec.so.1
  NEEDED               libc.so.6
objdump -p foo | grep NEEDED

readelf command

Use readelf -d foo | grep NEEDED. Sample output:

 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libmvec.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
readelf -d foo | grep NEEDED

How to add custom directories as a library search directory

Custom library paths, which are added to the compilation/linking, by the following method have higher priority than the default library search directories. In other words, if a library is found in a customized directory, it will be selected rather than the library in system default.

Compilation

Method 1: use CPATH (c or c++), C_INCLUDE_PATH (c only), CPLUS_INCLUDE_PATH (c++ only). For example: CPATH=/home/transang/my_libs_headers gcc -c foo.c -o foo.o.

Method 2: use -I<dir_path> to gcc when compiling. For example: gcc -I/home/transang/my_libs_headers -c foo.c -o foo.o.

Linking

Method 1: To add custom directories to the library linking search directories list, use LIBRARY_PATH environment variable. For example: LIBRARY_PATH=./mylib:/home/transang/my_libs gcc foo.o bar.o -o foo.

Method 2: add the flag -L<dir_path> to gcc when linking. For example: gcc -L/home/transang/my_libs -L./mylib foo.o bar.o -o foo.

From GCC man page:

-Ldir           Add directory dir to the list of directories to be searched for -l.

Method 3: add the flag -rpath-link <dir_path> to ld. For example: ld -rpath-link /home/transang/my_libs or gcc -Wl,-rpath-link,/home/transang/my_libs.

Method 4: add the flag -rpath <dir_path> to ld. For example: ld -rpath /home/transang/my_libs or gcc -Wl,-rpath,/home/transang/my_libs. Note: this affects the library searching path in runtime .

Note 1: LIBRARY_PATH environment variable's value does not affect the results of ld --verbose and gcc -print-search-dirs commands.

Note 2: method 3 and 4 do not work when I tried but the ld manpage says it works so I leave them for reference.

The linker uses the following search paths to locate required shared libraries.
1. Any directories specified by -rpath-link options.
2. Any directories specified by -rpath options. The difference between -rpath and -rpath-link is that directories specified by -rpath options are included in the executable and used at runtime, whereas the -rpath-link option is only effective at link time.
3. On an ELF system, if the -rpath and rpath-link options were not used, search the contents of the environment variable LD_RUN_PATH.
4. On SunOS, if the -rpath option was not used, search any directories specified using -L options.
5. For a native linker, the contents of the environment variable LD_LIBRARY_PATH.
6. The default directories, normally `/lib' and `/usr/lib'.

Runtime

For dynamic libraries, regardless of what you have configured for library searching paths. At runtime, the OS uses an independent configuration.

To add customized directories, use LD_LIBRARY_PATH environment variable when executing the executable file. For e.g. LD_LIBRARY_PATH=./mylib:/home/transang/my_libs ./foo.

How to add dynamic libraries when linking

So far, I have introduced a way to figure out the current configuration and modify it to add more directories for library searching.

To link a library, add -l<lib_name> flag to the gcc command when linking. If the lib_name does not start with :, gcc will look for a library named lib<lib_name>.so. Otherwise, the file name lib_name will be searched.

For example: with -lfoo, gcc looks for libfoo.so file. With -l:foo.so, gcc looks for foo.so file (Look at it precisely, -l: requires specifying library name extension, -l adds lib prefix and extension suffix to the library name).

Clear environment

In some cases, you may want a clean build that isolates from the running machine. You can use the following command:

LDFLAGS= LIBRARY_PATH= LD_LIBRARY_PATH= CPATH= C_INCLUDE_PATH= CPLUS_INCLUDE_PATH= CPPFLAGS= CFLAGS= gcc <command>

Moreover, the system library header searching path is configured in /etc/ld.so.conf.

# cat /etc/ld.so.conf
include ld.so.conf.d/*.conf
cat /etc/ld.so.conf

By default, /etc/ld.so.conf refers to the content of all files in the directory /etc/ld.so.conf.d. You can look into the directory to control the configuration.

total 24
drwxr-xr-x  2 root root 4096 Jun  4 15:55 .
drwxr-xr-x 90 root root 4096 Jun 14 14:00 ..
-rw-r--r--  1 root root   66 Apr  4 08:20 cuda.conf
-rw-r--r--  1 root root   21 May 24 07:30 fakeroot.conf
-rw-r--r--  1 root root   17 Nov 27  2021 openmpi.conf
-rw-r--r--  1 root root   16 May 27 06:09 R.conf
ls -al /etc/ld.so.conf.d/
/opt/cuda/lib64
/opt/cuda/nvvm/lib64
/opt/cuda/extras/CUPTI/lib64
cat /etc/ld.so.conf.d/cuda.conf
/usr/lib/libfakeroot
cat /etc/ld.so.conf.d/fakeroot.conf
/usr/lib/openmpi
cat /etc/ld.so.conf.d/openmpi.conf
/usr/lib/R/lib
cat /etc/ld.so.conf.d/R.conf

Embedding library paths for runtime at linking

rpath/runpath can be configured at the linking stage to add library paths to the ELF file header which will be used to scan for dynamic libraries at runtime. See dynamic array tags table here: DT_RPATH (value 15) and DT_RUNPATH (value 29). More about rpath at its wiki page.

Compare RPATH and RUNPATH

runpath and rpath are nearly similar with runpath being newer and gradually replacing rpath.

Search order with LD_LIBRARY_PATH

The search order is as follows, from high to low:

  • rpath
  • LD_LIBRARY_PATH
  • runpath

Nested dependencies

DT_RUNPATH is used only for direct dependencies, while DT_RPATH is used for nested dependencies (ref).

Set rpath with ld

Use ld -rpath <path> or gcc -Wl,-rpath,<path>.
For example: gcc main.o -o main -l:mylib.so -Wl,-rpath,/home/transang/mylib -L.. Then at runtime, there is no need to specify LD_LIBRARY_PATH, just use ./main.

Note: the correct flag to embed the search path to the ELF file is -rpath, not -rpath-link.

By default, ld/ gcc use RPATH. To use RUNPATH, use --enable-new-dtags. For example: gcc main.o -o main -l:mylib.so -Wl,-rpath,./libs,--enable-new-dtags -L. To explicitly use RPATH, use --disable-new-dtags.
Although, RUNPATH is considered newer, setting it as default caused many compilations failed (ref). So that if you want to use -rpath, it is better to also explicitly specify its usage with --disable-new-dtags / --enable-new-dtags.

Set runpath with patchelf

Use patchelf --set-rpath <path>. For example:

gcc main.o -o main -l:mylib.so -L.
patchelf --set-rpath . main
./main

By default patchelf use RUNPATH. patchelf also clears existing RPATH/RUNPATH headers.

Get configured rpath/runpath

Use one of the following commands

  • objdump -x foo | grep RPATH
  • readelf -d foo | grep RPATH
  • objdump -x foo | grep RUNPATH
  • readelf -d foo | grep RUNPATH

Sample outputs:

  RUNPATH              /home/transang/mylib
objdump -x main | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [/home/transang/mylib]
readelf -d main | grep RUNPATH
 0x000000000000000f (RPATH)              Library rpath: [/home/transang/mylib]
readelf -d main | grep RPATH
  RPATH                /home/transang/mylib
objdump -x main | grep RPATH

Relative search path

rpath/runpath supports the following path patterns:

  • Absolute paths: starting with /.
  • Paths relative to the current working dir: starting with ./.
  • Paths relative to the executable file: the $ORIGIN variable is expanded at runtime to the absolute path of the directory containing the executable file.

For example, you can use patchelf --set-rpath '$ORIGIN/libs' main to add the libs directory next to main to the library search path.

Security concern with the setuid/setgid flag

For files with the setuid/setgid flag set (via chmod a+s <file> command), LD_LIBRARY_PATH is ignored, any rpath/runpath containing $ORIGIN expansion are also ignored.

Also note that ldd does not consider the setuid flag checking in its output.