CUDA/Cudnn/Driver/GCC/TensorFlow/Python version compatibility charts

Running (training) legacy machine learning models, especially models written for TensorFlow v1, is not a trivial task mostly due to the version incompatibility issue. This post will show the compatibility table with references to official pages.

The general flow of the compatibility resolving process is

• TensorFlow → Python
• TensorFlow → Cudnn/Cuda → NVIDIA driver/GCC

Quick resolution for Tensorflow version 1 user

You can use TensorFlow version 1, by installing exactly the following versions of the required components:

• tensorflow==1.15 (latest v1).
• tensorflow-gpu==1.15
• Python 3.7 (the latest version supported by TF 1.15).
• CUDA 10.0 (required by TF 1.15). Note: install a different minor version CUDA 10.2 does not work.
• NVIDIA driver 410.48 (comes with CUDA 10.0 Toolkit) or NVIDIA 470 (I installed via apt and confirmed that it works).
• Cudnn 7.4 (required by TF 1.15).
• tensorflow-probability==0.7 (my project's need).
• tensorflow-datasets==3.2.1 (my project's need).

Let's solve these compatibility requirements one by one.

TensorFlow compatibility

Official page with the compatibility chart

Linux

CPU (Linux)

GPU (Linux)

Note that the minor CUDA version is required. For example, if you want to run TensorFlow 1.15, you must install CUDA 10.0. Installing CUDA 10.2 does not work with the following errors:

2022-06-21 21:10:57.123081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.695
pciBusID: 0000:b3:00.0
2022-06-21 21:10:57.123186: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64
2022-06-21 21:10:57.123237: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64
2022-06-21 21:10:57.123279: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64
2022-06-21 21:10:57.123320: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64
2022-06-21 21:10:57.123362: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64
2022-06-21 21:10:57.123402: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64
2022-06-21 21:10:57.123443: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64
2022-06-21 21:10:57.123452: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

macOS

CPU (macOS)

GPU (macOS)

CUDA → NVIDIA Driver compatibility

Official compatibility chart.

This compatibility requirement is generally less strict. The requirement is usually a least version requirement. However, it is recommended to install the nearest, requirement-met, and available driver.

Using the driver that comes with the CUDA ToolKit is the best choice because its compatibility is well tested.

Official page that lists links to download CUDA Toolkit, including old versions.

Cudnn/CUDA compatibility

The compatibility requirement is most flexible. NVIDIA has separated cudnn builds for each cuda version. Here is the official link that lists all builds. Note that you might need to register an account or login to download.

However, TensorFlow does support specific cudnn versions. You have to look at the first section to select the working version of cudnn to install.

Nvidia driver → GCC compatibility

This requirement is relatively less strict. There are several ways to install an nvidia driver.

• (Recommended) Install the driver that comes with the CUDA Toolkit. This is recommended because its compatibility with CUDA is well-tested.
• (Recommended) Using OS package manager: pacman, apt, etc. Recently, they work very well. Especially, the package manager handles the burden of maintaining GCC version compatibility. Old driver version might not available in this way, but usually, it works even with old CUDA versions.
• Install the nvidia driver by a runfile. It is difficult to download an old driver officially. The official page only provides the latest version.

If you install the driver packed with CUDA Toolkit, you have to meet the GCC version requirement. See the next section for this.

CUDA → GCC compatibility

By default, the CUDA installer will abort if the GCC version is too high. You can enforce the installation but any unexpected issue may occur without any warranty.

The document for each CUDA version's requirement is fragmented. There is no official page that shows a compatibility chart. The main page only lists the requirement for the latest CUDA version.

Fortunately, the community builds a compatibility chart here.

Release notes for CUDA Toolkit (CTK):

• latest: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html
• 11.5.1: https://docs.nvidia.com/cuda/archive/11.5.1/
• 11.4.2: https://docs.nvidia.com/cuda/archive/11.4.2/
• 11.4.1: https://docs.nvidia.com/cuda/archive/11.4.1/
• 11.4.0: https://docs.nvidia.com/cuda/archive/11.4.0/
• 11.3: https://docs.nvidia.com/cuda/archive/11.3.0/index.html
• 11.2: https://docs.nvidia.com/cuda/archive/11.2.2/index.html
• 11.1: https://docs.nvidia.com/cuda/archive/11.1.1/index.html
• 11.0: https://docs.nvidia.com/cuda/archive/11.0/cuda-toolkit-release-notes/index.html
• 10.2: https://developer.download.nvidia.com/compute/cuda/10.2/Prod/docs/sidebar/CUDA_Toolkit_Release_Notes.pdf
• 10.1: https://developer.download.nvidia.com/compute/cuda/10.1/Prod/docs/sidebar/CUDA_Toolkit_Release_Notes.pdf
• 10.0: https://developer.download.nvidia.com/compute/cuda/10.0/Prod/docs/sidebar/CUDA_Toolkit_Release_Notes.pdf
• 9.2: https://developer.download.nvidia.com/compute/cuda/9.2/Prod/docs/sidebar/CUDA_Toolkit_Release_Notes.pdf
• 9.1: https://developer.download.nvidia.com/compute/cuda/9.1/Prod/docs/sidebar/CUDA_Toolkit_Release_Notes.pdf
• 9.0: https://developer.download.nvidia.com/compute/cuda/9.0/Prod/docs/sidebar/CUDA_Toolkit_Release_Notes.pdf
• 8.0: https://developer.nvidia.com/compute/cuda/8.0/Prod2/docs/sidebar/CUDA_Toolkit_Release_Notes-pdf
• 7.5: http://developer.download.nvidia.com/compute/cuda/7.5/Prod/docs/sidebar/CUDA_Toolkit_Release_Notes.pdf
• 7.0: http://developer.download.nvidia.com/compute/cuda/7_0/Prod/doc/CUDA_Toolkit_Release_Notes.pdf
• 6.5: http://developer.download.nvidia.com/compute/cuda/6_5/rel/docs/CUDA_Toolkit_Release_Notes.pdf
• 6.0: http://developer.download.nvidia.com/compute/cuda/6_0/rel/docs/CUDA_Toolkit_Release_Notes.pdf
• 5.5: http://developer.download.nvidia.com/compute/cuda/5_5/rel/docs/CUDA_Toolkit_Release_Notes.pdf

How to manage GCC versions manually

update-alternatives is a handy tool to manage multiple GCC versions in Ubuntu. For CentOS, you can use scl instead.

See the sample command lines below:

# sudo apt install -yq gcc-7 g++-7
# sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-7 7
# sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 7


# sudo update-alternatives --config gccgcc
There are 4 choices for the alternative gcc (providing /usr/bin/gcc).

  Selection    Path             Priority   Status
------------------------------------------------------------
  0            /usr/bin/gcc-10   10        auto mode
  1            /usr/bin/gcc-10   10        manual mode
  2            /usr/bin/gcc-7    7         manual mode
* 3            /usr/bin/gcc-8    8         manual mode
  4            /usr/bin/gcc-9    9         manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/bin/gcc-7 to provide /usr/bin/gcc (gcc) in manual mode


# sudo update-alternatives --config g++
There are 4 choices for the alternative g++ (providing /usr/bin/g++).

  Selection    Path             Priority   Status
------------------------------------------------------------
  0            /usr/bin/g++-10   10        auto mode
  1            /usr/bin/g++-10   10        manual mode
  2            /usr/bin/g++-7    7         manual mode
* 3            /usr/bin/g++-8    8         manual mode
  4            /usr/bin/g++-9    9         manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/bin/g++-7 to provide /usr/bin/g++ (g++) in manual mode

# gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
# sudo yum install centos-release-scl
# sudo yum install devtoolset-9
# scl enable devtoolset-9 bash
# gcc --version
gcc (GCC) 9.1.1 20190605 (Red Hat 9.1.1-2)

Other compatibilities

For other packages such as TensorFlow Probability, TensorFlow Datasets. The best way now is to look at their Releases pages on GitHub (TF Probability, TF Datasets) to find the compatible version. Looking at the release date is a nice trick to make it quicker.

For example, for TensorFlow 1.15, the compatible versions for TensorFlow Probability is 0.7, for TensorFlow Datasets is 3.2.1.