Installer freezes node on updating_container_ld_cache

[reverson]

Whenever I run this on an ubuntu node the node stops responding to ssh never becomes available for gpu workloads.

This is what I see in the logs
`kubectl logs nvidia-driver-installer-6vnpd -n kube-system -c nvidia-driver-installer -f

• NVIDIA_DRIVER_VERSION=384.111
• NVIDIA_DRIVER_DOWNLOAD_URL_DEFAULT=https://us.download.nvidia.com/tesla/384.111/NVIDIA-Linux-x86_64-384.111.run
• NVIDIA_DRIVER_DOWNLOAD_URL=https://us.download.nvidia.com/tesla/384.111/NVIDIA-Linux-x86_64-384.111.run
• NVIDIA_INSTALL_DIR_HOST=/home/kube/bin/nvidia
• NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia
  ++ basename https://us.download.nvidia.com/tesla/384.111/NVIDIA-Linux-x86_64-384.111.run
• NVIDIA_INSTALLER_RUNFILE=NVIDIA-Linux-x86_64-384.111.run
• ROOT_MOUNT_DIR=/root
• set +x
  Downloading kernel sources...
  Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
  Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
  Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
  Get:4 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [81.2 kB]
  Get:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
  Get:6 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [652 kB]
  Get:7 http://archive.ubuntu.com/ubuntu xenial/universe Sources [9802 kB]
  Get:8 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
  Get:9 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [450 kB]
  Get:10 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [3735 B]
  Get:11 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
  Get:12 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
  Get:13 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
  Get:14 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
  Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [256 kB]
  Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1028 kB]
  Get:17 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]
  Get:18 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [819 kB]
  Get:19 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [18.8 kB]
  Get:20 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [5157 B]
  Get:21 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [8080 B]
  Fetched 25.3 MB in 12s (1955 kB/s)
  Reading package lists...
  Reading package lists...
  Building dependency tree...
  Reading state information...
  The following additional packages will be installed:
  linux-headers-4.4.0-127
  The following NEW packages will be installed:
  linux-headers-4.4.0-127 linux-headers-4.4.0-127-generic
  0 upgraded, 2 newly installed, 0 to remove and 60 not upgraded.
  Need to get 10.8 MB of archives.
  After this operation, 78.4 MB of additional disk space will be used.
  Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 linux-headers-4.4.0-127 all 4.4.0-127.153 [9950 kB]
  Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 linux-headers-4.4.0-127-generic amd64 4.4.0-127.153 [816 kB]
  debconf: delaying package configuration, since apt-utils is not installed
  Fetched 10.8 MB in 3s (3240 kB/s)
  Selecting previously unselected package linux-headers-4.4.0-127.
  (Reading database ... 9482 files and directories currently installed.)
  Preparing to unpack .../linux-headers-4.4.0-127_4.4.0-127.153_all.deb ...
  Unpacking linux-headers-4.4.0-127 (4.4.0-127.153) ...
  Selecting previously unselected package linux-headers-4.4.0-127-generic.
  Preparing to unpack .../linux-headers-4.4.0-127-generic_4.4.0-127.153_amd64.deb ...
  Unpacking linux-headers-4.4.0-127-generic (4.4.0-127.153) ...
  Setting up linux-headers-4.4.0-127 (4.4.0-127.153) ...
  Setting up linux-headers-4.4.0-127-generic (4.4.0-127.153) ...
  Downloading kernel sources... DONE.
  Configuring installation directories...
  /usr/local/nvidia /
  Updating container's ld cache...
  `
  At this point in another window if I ssh to that node it just hangs.

[rohitagarwal003]

The Ubuntu installation is only tested on GKE. Even there it's not officially supported yet.

I am going to close this but see the comments on #57 (the issue is probably an older kernel or not using overlay2)