[FIXED] How to fix "NVRM: API mismatch" between client version and kernel module version when installing Nvidia drivers for a GTX 560 Ti in Ubuntu 20.04?

Issue

I have installed nvidia-driver-390 after adding a GTX 560 Ti on an Intel Core i5 12600K PC running Kubuntu 20.04 LTS.

After rebooting I get the following error:

$ dmesg|grep -i nvrm -A3
[  113.647054] NVRM: API mismatch: the client has the version 460.91.03, but
               NVRM: this kernel module has the version 390.144.  Please
               NVRM: make sure that this kernel module and all NVIDIA driver
               NVRM: components have the same version.

How do I get that client version set to 390.144?

Newer drivers such as nvidia-driver-460 do not seem to support this video card. Also I get a warning:

$ ubuntu-drivers devices
WARNING:root:_pkg_get_support nvidia-driver-390: package has invalid Support Legacyheader, cannot determine support level
== /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0 ==
modalias : pci:v000010DEd00001200sv00001043sd000083ACbc03sc00i00
vendor   : NVIDIA Corporation
model    : GF114 [GeForce GTX 560 Ti]
driver   : nvidia-340 - distro non-free
driver   : nvidia-driver-390 - distro non-free recommended
driver   : xserver-xorg-video-nouveau - distro free builtin

I run a newer kernel in order to have networking with the Intel 2.5G Ethernet IGC driver. I can boot into kernel 5.4.0.91-generic, but then I won't have a network.

$ uname -a
Linux Kairos 5.13.0-22-generic #22~20.04.1-Ubuntu SMP Tue Nov 9 15:07:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Furthermore, dkms gives:

$ dkms status
nvidia, 390.144, 5.13.0-22-generic, x86_64: installed
nvidia, 390.144, 5.4.0-91-generic, x86_64: built

While /var/lib/dkms/ shows:

$ tree /var/lib/dkms/
/var/lib/dkms/
├── dkms_dbversion
└── nvidia
    ├── 390.144
    │   ├── 5.13.0-22-generic
    │   │   └── x86_64
    │   │       ├── log
    │   │       │   └── make.log
    │   │       └── module
    │   │           ├── nvidia-drm.ko
    │   │           ├── nvidia.ko
    │   │           └── nvidia-modeset.ko
    │   ├── 5.4.0-91-generic
    │   │   └── x86_64
    │   │       ├── log
    │   │       │   └── make.log
    │   │       └── module
    │   │           ├── nvidia-drm.ko
    │   │           ├── nvidia.ko
    │   │           ├── nvidia-modeset.ko
    │   │           └── nvidia-uvm.ko
    │   └── source -> /usr/src/nvidia-390.144
    └── kernel-5.13.0-22-generic-x86_64 -> 390.144/5.13.0-22-generic/x86_64

12 directories, 10 files

And modinfo nvidia returns:

filename:       /lib/modules/5.13.0-22-generic/updates/dkms/nvidia.ko
alias:          char-major-195-*
version:        390.144
supported:      external
license:        NVIDIA
srcversion:     D635B75826DE88984609590
alias:          pci:v000010DEd00000E00sv*sd*bc04sc80i00*
alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
depends:        ipmi_msghandler
retpoline:      Y
name:           nvidia
vermagic:       5.13.0-22-generic SMP mod_unload modversions 
parm:           NVreg_Mobile:int
parm:           NVreg_ResmanDebugLevel:int
parm:           NVreg_RmLogonRC:int
parm:           NVreg_ModifyDeviceFiles:int
parm:           NVreg_DeviceFileUID:int
parm:           NVreg_DeviceFileGID:int
parm:           NVreg_DeviceFileMode:int
parm:           NVreg_UpdateMemoryTypes:int
parm:           NVreg_InitializeSystemMemoryAllocations:int
parm:           NVreg_UsePageAttributeTable:int
parm:           NVreg_MapRegistersEarly:int
parm:           NVreg_RegisterForACPIEvents:int
parm:           NVreg_CheckPCIConfigSpace:int
parm:           NVreg_EnablePCIeGen3:int
parm:           NVreg_EnableMSI:int
parm:           NVreg_TCEBypassMode:int
parm:           NVreg_UseThreadedInterrupts:int
parm:           NVreg_EnableStreamMemOPs:int
parm:           NVreg_EnableBacklightHandler:int
parm:           NVreg_RestrictProfilingToAdminUsers:int
parm:           NVreg_EnableUserNUMAManagement:int
parm:           NVreg_EnableIBMNPURelaxedOrderingMode:int
parm:           NVreg_MemoryPoolSize:int
parm:           NVreg_KMallocHeapMaxSize:int
parm:           NVreg_VMallocHeapMaxSize:int
parm:           NVreg_IgnoreMMIOCheck:int
parm:           NVreg_RegistryDwords:charp
parm:           NVreg_RegistryDwordsPerDevice:charp
parm:           NVreg_RmMsg:charp
parm:           NVreg_AssignGpus:charp

I have found many posts about how to upgrade the driver to the latest version, currently 495, however it does not solve my problem and that version does not support my video card. Only version 390 works, but then I get that API mismatch error.

Solution

I found the solution and now Kubuntu starts up normally as expected.

Before starting, I wanted to confirm this version mismatch between loaded kernel module and newly installed standalone module files. User ladar showed in his answer that the following will confirm this issue

$ cat /proc/driver/nvidia/version
$ cat /sys/module/nvidia/version

First, I purged all drivers and libraries

$ sudo apt purge nvidia-* && sudo apt purge libnvidia-*

Then I checked if there weren't any nvidia drivers installed. The following did not return anything, as it should

$ dpkg -l nvidia-*|grep ^ii

Then I downloaded the correct version of drivers for my Geforce GTX 560 Ti from https://www.nvidia.com/Download/driverResults.aspx/177153/en-us. You should select your driver from https://www.nvidia.com/Download/index.aspx?lang=en-us.

After this I rebooted into a terminal

$ sudo init 3

I logged in on the command line and went to the directory where I just downloaded the driver, then changed permissions and ran the installation file (it may ask some configuration questions), followed by a reboot

$ cd $HOME/Downloads/Nvidia/
$ chmod a+x NVIDIA-Linux-x86_64-390.144.run
$ sudo ./NVIDIA-Linux-x86_64-390.144.run 
$ sudo reboot

I now had my normal SSDM login screen and after logging in my dual screens were both in the correct resolution.

Answered By - León van de Pavert

Answer Checked By - David Goodson (PHPFixing Volunteer)