NVIDIA vGPU 6.3
Hi All
Its time to update your NVIDIA TESLA M6, M10, M60, P4, P6, P40, P100, V100 with vGPU 6.3
The brand “GRID” is now vGPU (Virtual GPU Software), in my previously articles I wrote GRID and now its branded vGPU.
NVIDIA have released new drivers for NVIDIA vGPU 6.3 for October 2018.
NVIDIA vGPU 6.3 now support VMware Horizon 7.6 and Citrix Virtual Apps & Desktop 7 1808 (aka XenApp/XenDesktop) + Citrix Hypervisor (aka XenServer) 7.6
Important:
- NVIDIA vGPU 6.3 is supported with VMware Horizon 7.6, 7.5, 7.4, 7.3, 7.2, 7.1, 7.0, 6.2
- NVIDIA vGPU 6.3 is only supported with Citrix Virtual Apps & Desktops (aka XenDesktop) 7.15, 7.17, 7.18, 7 1808 in HDX 3D Pro mode
- If you are a customer using XenServer 7.2, 7.3, 7.4 its no longer supported with NVIDIA vGPU 6.3 and should plan upgrading to XenServer 7.5 or 7.6.
- Customers using Citrix XenServer 7.0, 7.1 is supported with NVIDIA vGPU 6.0
- Customer using Citrix XenServer 7.2, 7.3 is supported with NVIDIA vGPU 6.0 & 6.1
- Customer using Citrix XenServer 7.4 is supported with NVIDIA vGPU 6.0 & 6.1 & 6.2
- Customers using Citrix XenServer 7.5 is supported with NVIDIA vGPU 6.2
This release includes the following software:
- NVIDIA vGPU Manager version 390.94 for the Citrix XenServer, VMware vSphere
- NVIDIA Windows driver version 392.05
- NVIDIA Linux driver version 390.96
New in this Release:
- Inclusion of the name and version of the licensed product in logged license acquisition events
- Miscellaneous bug fixes
- Support for Citrix Hypervisor (aka) XenServer 7.6
- Support for Citrix Virtual Apps and Desktops (formerly Citrix XenDesktop) 7 1808
- Support for VMware Horizon 7.6
Other important notes about NVIDIA vGPU 6.3
- Citrix XenServer 7.2, 7.3, 7.4 are no longer supported. Citrix XenServer 7.1 and XenServer 7.0 is supported
Supported NVIDIA GPUs with vGPU 6.3
- Tesla M6
- Tesla M10
- Tesla M60
- Tesla P4
- Tesla P6
- Tesla P40
- Tesla P100 PCIe 16 GB
- Tesla P100 SXM2 16 GB
- Tesla P100 PCIe 12GB
- Tesla V100 SXM2
- Tesla V100 SXM2 32GB
- Tesla V100 PCIe
- Tesla V100 32GB
- Tesla V100 FHHL
Supported Hypervisors with NVIDIA vGPU 6.3
- Citrix XenServer
Citrix XenServer 7.0,7.1, 7.5, 7.6 (supported with Tesla M6, M10, M60, P4, P6, P40, P100,V100) - VMware vSphere
VMware vSphere 6.7 (supported with Tesla M6, M10, M60, P4, P6, P40, P100,V100)
VMware vSphere 6.5 (supported with Tesla M6, M10, M60, P4, P6, P40, P100,V100)
Vmware vSphere 6.0 update3, update 2, update 1, RTM b2494585 (supported with Tesla M6, M10, M60, P4, P6, P40, P100,V100)
VMware vSphere 5.5 (supported with Tesla M6,M10,M60) - Microsoft Hyper-V 2016
Microsoft Windows Server 2016 with Hyper-V role (supported with Tesla M6, M10, M60, P4, P6, P40, P100)
note: Microsoft Windows Server with Hyper-V role supports GPU pass-through over Microsoft Virtual PCI bus. This bus is supported through paravirtualized drivers. - Linux with KVM
H3C, Sangfor (supported with Tesla M6, M60) - Red Hat Enterprise Linux with KVM
Red Hat Enterprise Linux with KVM 7.2, 7.3 (supported with Tesla M6, M10, M60, P4, P6, P40, P100, V100)
Red Hat Enterprise Linux with KVM 7.0, 7.1 (supported with Tesla M6,M10,M60) - Nutanix AHV
Nutanix AOS Hypervisor (AHV) 5.5, 5.6, 5.8 (supported with Tesla M10, M60, P40)
Supported Hypervisor with migration of vGPU across hypervisors
Above is only supported with Citrix XenServer 7.6
This release of NVIDIA vGPU software provides support for the following NVIDIA GPUs on Citrix XenServer, running on validated server hardware platform
- Tesla M6
- Tesla M10
- Tesla M60
- Tesla P4
- Tesla P6
- Tesla P40
- Tesla P100 PCIe 16 GB (XenMotion with vGPU is not supported.)
- Tesla P100 SXM2 16 GB (XenMotion with vGPU is not supported.)
- Tesla P100 PCIe 12GB (XenMotion with vGPU is not supported.)
- Tesla V100 SXM2 (XenMotion with vGPU is not supported.)
- Tesla V100 SXM2 32GB (XenMotion with vGPU is not supported.)
- Tesla V100 PCIe (XenMotion with vGPU is not supported.)
- Tesla V100 PCIe 32GB (XenMotion with vGPU is not supported.)
- Tesla V100 FHHL (XenMotion with vGPU is not supported.)
Whats new in NVIDIA vGPU 6.3– 390.94-392.05-390.96
NVIDIA have released a new version of GRID 6.3 – 390.94-392.05-390.96 for NVIDIA GRID (Tesla M6, M10, M60, P4, P6, P40, P100,V100 platform)
Included in this release is
- NVIDIA GRID Virtual GPU Manager versions 390.94 for Citrix XenServer 7.0
- NVIDIA GRID Virtual GPU Manager versions 390.94 for Citrix XenServer 7.1
- NVIDIA GRID Virtual GPU Manager versions 390.94 for Citrix XenServer 7.5
- NVIDIA GRID Virtual GPU Manager versions 390.94 for Citrix XenServer 7.6
- NVIDIA GRID Virtual GPU Manager version 390.94 for VMware vSphere 5.5 Hypervisor (ESXi)
- NVIDIA GRID Virtual GPU Manager version 390.94 for VMware vSphere 6.0 Hypervisor (ESXi)
- NVIDIA GRID Virtual GPU Manager version 390.94 for VMware vSphere 6.5 Hypervisor (ESXi)
- NVIDIA GRID Virtual GPU Manager version 390.94 for VMware vSphere 6.7 Hypervisor (ESXi)
- NVIDIA GRID Virtual GPU Manager version 390.94 for Nutanix AHV 5.5, 5.6, 5.8 RTM
- NVIDIA GRID Virtual GPU Manager version 390.94 for Huawei UVP version RC520
- NVIDIA Windows drivers for vGPU version 392.05
- NVIDIA Linux drivers for vGPU version 390.96
Important:
The GRID vGPU Manager and Windows guest VM drivers must be installed together. Older VM drivers will not function correctly with this release of GRID vGPU Manager. Similarly, older GRID vGPU Managers will not function correctly with this release of Windows guest drivers
Windows Guest OS support in NVIDIA vGPU 6.3 – 392.05
GRID vGPU 392.05 supports following Windows release as a guest OS
- Microsoft Windows 7 (32/64bit)
- Microsoft Windows 8 (32/64bit)
- Microsoft Windows 8.1 (32/64bit)
- Microsoft Windows 10 (32/64bit) (1507, 1511, 1607, 1703, 1709)
- Microsoft Windows Server 2008R2
- Microsoft Windows Server 2012 R2
- Microsoft Windows Server 2016 (1607, 1709)
Linux Guest OS support in NVIDIA vGPU 6.3 – 390.96
GRID vGPU 390.96 supports following Linux distributions as a guest OS only on supported Tesla GPUs
- Red Hat Enterprise Linux 7.0-7.5
- CentOS 7.0-7.5
- Ubuntu 16.04 LTS
- Ubuntu 14.04 LTS
Important driver notes to NVIDIA vGPU 6.3
- In pass-through mode, GPUs based on the Pascal architecture support only 64-bit guest operating systems. No 32-bit guest operating systems are supported in pass-through mode for these GPUs.
- ESXi 5.5 is only supported with Pass-through and not vGPU
- ESXi 6.0 Update 3 is required for pass-through mode on GPUs based on the Pascal architecture.
- Windows 7 and Windows Server 2008 R2 are not supported in pass-through mode on GPUs based on the Pascal architecture.
- Only Tesla M6 is supported as the primary display device in a bare-metal deployment.
- Red Hat Enterprise Linux with KVM 7.0 and 7.1 are supported only on Tesla M6, Tesla M10, and Tesla M60 GPUs.
- Red Hat Enterprise Linux with KVM supports Windows guest operating systems only under specific Red Hat subscription programs. For details, see Certified guest operating systems for Red Hat Enterprise Linux with KVM.
- Windows 7, Windows Server 2008 R2, 32-bit Windows 10, and 32-bit Windows 8.1 are supported only on Tesla M6, Tesla M10, and Tesla M60 GPUs.
Guide – Update existing NVIDIA vGPU Manager (Hypervisor)
Citrix Hypervisor (aka XenServer)
NVIDIA vGPU Manager 390.94 for Citrix XenServer 7 & 7.1
If you have a NVIDIA M6, M10, M60, P4, P6, P40, P100 vGPU manager installed in Citrix XenServer. Upgrade with one of below methodology:
Methodology 1 – the manual way “No GUI”
Upgrading an existing installation of the NVIDIA vGPU driver on Citrix XenServer 7, use the rpm -U command to upgrade:
If you have NVIDIA TESLA M6 / M10 / M60 / P4 / P6 / P40 / P100 / V100
[root@localhost ~]# rpm -Uv NVIDIA-vGPU-xenserver-7.0-390.94.x86_64.rpm (#if you have for XenServer 7)
[root@localhost ~]# rpm -Uv NVIDIA-vGPU-xenserver-7.1-390.94.x86_64.rpm (#if you have for XenServer 7.1)
Preparing packages for installation…
The recommendation from NVIDIA is to shutdown all VMs using a GPU. The machine does continue to work during the update, but since you need to reboot the XenServer itself, it’s better to gracefully shutdown the VMs. So after your VMs have been shutdown and you upgraded the NVIDIA driver, you can reboot your host.
[root@localhost ~]# xe host-disable
[root@localhost ~]# xe host-reboot
Methodology 2 – the “GUI” way
Select Install Update… from the Tools menu
Click Next after going through the instructions on the Before You Start section
Click Add on the Select Update section and open NVIDIA’s XenServer Supplemental Pack ISO
If you have NVIDIA M6/M10/M60/P4/P6/P40/P100/V100 select following file:
“NVIDIA-vGPU-xenserver-7.0-390.94.x86_64.iso ” (#if you have XenServer 7)
“NVIDIA-vGPU-xenserver-7.1-390.94.x86_64.iso ” (#if you have XenServer 7.1)
Click Next on the Select Update section
In the Select Servers section select all the XenServer hosts on which the Supplemental Pack should be installed on and click Next
Click Next on the Upload section once the Supplemental Pack has been uploaded to all the XenServer hosts
Getting Started
Click Next on the Prechecks section
Click Install Update on the Update Mode section
Click Finish on the Install Update section
After the XenServer platform has rebooted, verify that the vGPU package installed and loaded correctly by checking for the NVIDIA kernel driver in the list of kernel loaded modules.
Validate from putty or XenCenter CLI
run lsmod | grep nvidia
Verify that the NVIDIA kernel driver can successfully communicate with the vGPU physical GPUs in your system by running the nvidia-smi command, which should produce a listing of the GPUs in your platform:
Check driver version is 390.94, if it is then your host is ready for GPU awesomeness and make your VM rock.
NVIDIA vGPU Manager 390.94 for Citrix XenServer 7.5 or 7.6
If you have a NVIDIA vGPU M6, M10, M60, P4, P6, P40, P100 vGPU manager installed in Citrix XenServer. Upgrade with one of below methodology:
Methodology 1 – the manual way “No GUI”
Upgrading an existing installation of the NVIDIA driver on Citrix XenServer 7.6, use the rpm -U command to upgrade:
If you have NVIDIA TESLA M6 / M10 / M60 / P4 / P6 / P40 / P100 / V100
[root@localhost ~]# rpm -Uv NVIDIA-vGPU-xenserver-7.6-390.94x86_64.rpm
Preparing packages for installation…
The recommendation from NVIDIA is to shutdown all VMs using a GPU. The machine does continue to work during the update, but since you need to reboot the XenServer itself, it’s better to gracefully shutdown the VMs. So after your VMs have been shutdown and you upgraded the NVIDIA driver, you can reboot your host.
[root@localhost ~]# xe host-disable
[root@localhost ~]# xe host-reboot
Methodology 2 – the “GUI” way
Select Install Update… from the Tools menu
Click Next after going through the instructions on the Before You Start section
Click Add on the Select Update section and open NVIDIA’s XenServer Supplemental Pack ISO
If you have NVIDIA GRID M6/ M10/M60/P4/P6/P40/P100/V100 select following file:
“NVIDIA-vGPU-xenserver-7.5-390.94.x86_64.iso ” if XenServer 7.5
“NVIDIA-vGPU-xenserver-7.6-390.94.x86_64.iso ” if XenServer 7.6
Click Next on the Select Update section
In the Select Servers section select all the XenServer hosts on which the Supplemental Pack should be installed on and click Next
Click Next on the Upload section once the Supplemental Pack has been uploaded to all the XenServer hosts
Getting Started
Click Next on the Prechecks section
Click Install Update on the Update Mode section
Click Finish on the Install Update section
After the XenServer platform has rebooted, verify that the GRID package installed and loaded correctly by checking for the NVIDIA kernel driver in the list of kernel loaded modules.
Validate from putty or XenCenter CLI
run lsmod | grep nvidia
Verify that the NVIDIA kernel driver can successfully communicate with the GRID physical GPUs in your system by running the nvidia-smi command, which should produce a listing of the GPUs in your platform:
Check driver version is 390.94, if it is then your host is ready for GPU awesomeness and make your VM rock.
GRID vGPU Manager 390.94 for VMware vSphere 6.0
To update the NVIDIA GPU VIB, you must uninstall the currently installed VIB and install the new VIB.
To uninstall the currently installed VIB:
- Stop all virtual machines using 3D acceleration.
- Place the ESXi host into Maintenance mode.
- Open a command prompt on the ESXi host.
- Stop the xorg service by running the command:/etc/init.d/xorg stop
- Remove the NVIDIA VMkernel driver by running the command:vmkload_mod -u nvidia
- Identify the NVIDIA VIB name by running this command:esxcli software vib list | grep NVIDIA
- Remove the VIB by running the command:esxcli software vib remove -n nameofNVIDIAVIBYou can now install a new NVIDIA GPU VIB
- Use the esxcli command to install the vGPU Manager package:
If you have NVIDIA GRID TESLA M6 / M10 / M60 / P4 / P6 / P40 / P100 / v100 select following file:
[root@lesxi ~] esxcli software vib install -v /NVIDIA-vGPU-VMware_ESXi_6.0_Host_Driver_390.94-1OEM.600.0.0.2494585.vib
After the ESXi host has rebooted, verify that the GRID package installed and loaded correctly by checking for the NVIDIA kernel driver in the list of kernel loaded modules.
[root@lesxi ~]# vmkload_mod -l | grep nvidia
Preparing packages for installation…
Validate
run nvidia-smi
Verify that the NVIDIA kernel driver can successfully communicate with the GRID physical GPUs in your system by running the nvidia-smi command, which should produce a listing of the GPUs in your platform:
Check driver version is 390.94 if it is then your host is ready for GPU awesomeness and make your VM rock.
GRID vGPU Manager 390.94 for VMware vSphere 6.5
To update the NVIDIA GPU VIB, you must uninstall the currently installed VIB and install the new VIB.
To uninstall the currently installed VIB:
- Stop all virtual machines using 3D acceleration.
- Place the ESXi host into Maintenance mode.
- Open a command prompt on the ESXi host.
- Stop the xorg service by running the command:/etc/init.d/xorg stop
- Remove the NVIDIA VMkernel driver by running the command:vmkload_mod -u nvidia
- Identify the NVIDIA VIB name by running this command:esxcli software vib list | grep NVIDIA
- Remove the VIB by running the command:esxcli software vib remove -n nameofNVIDIAVIBYou can now install a new NVIDIA GPU VIB
- Use the esxcli command to install the vGPU Manager package:
If you have NVIDIA GRID TESLA M6 / M10 / M60 / P4 / P6 / P40 / P100 / v100 select following file:
[root@lesxi ~] esxcli software vib install -v /NVIDIA-vGPU-VMware_ESXi_6.5_Host_Driver_390.94-1OEM.650.0.0.2494585.vib
After the ESXi host has rebooted, verify that the GRID package installed and loaded correctly by checking for the NVIDIA kernel driver in the list of kernel loaded modules.
[root@lesxi ~]# vmkload_mod -l | grep nvidia
Preparing packages for installation…
Validate
run nvidia-smi
Verify that the NVIDIA kernel driver can successfully communicate with the GRID physical GPUs in your system by running the nvidia-smi command, which should produce a listing of the GPUs in your platform:
Check driver version is 390.94 if it is then your host is ready for GPU awesomeness and make your VM rock.
GRID vGPU Manager 390.94 for VMware vSphere 6.7
To update the NVIDIA GPU VIB, you must uninstall the currently installed VIB and install the new VIB.
To uninstall the currently installed VIB:
- Stop all virtual machines using 3D acceleration.
- Place the ESXi host into Maintenance mode.
- Open a command prompt on the ESXi host.
- Stop the xorg service by running the command:/etc/init.d/xorg stop
- Remove the NVIDIA VMkernel driver by running the command:vmkload_mod -u nvidia
- Identify the NVIDIA VIB name by running this command:esxcli software vib list | grep NVIDIA
- Remove the VIB by running the command:esxcli software vib remove -n nameofNVIDIAVIBYou can now install a new NVIDIA GPU VIB
- Use the esxcli command to install the vGPU Manager package:
If you have NVIDIA TESLA M6 / M10 / M60 / P4 / P6 / P40 / P100 / v100 select following file:
[root@lesxi ~] esxcli software vib install -v /NVIDIA-vGPU-VMware_ESXi_6.7_Host_Driver_390.94-1OEM.650.0.0.2494585.vib
After the ESXi host has rebooted, verify that the GRID package installed and loaded correctly by checking for the NVIDIA kernel driver in the list of kernel loaded modules.
[root@lesxi ~]# vmkload_mod -l | grep nvidia
Preparing packages for installation…
Validate
run nvidia-smi
Verify that the NVIDIA kernel driver can successfully communicate with the NVIDIA physical GPUs in your system by running the nvidia-smi command, which should produce a listing of the GPUs in your platform:
Check driver version is 390.94 if it is then your host is ready for GPU awesomeness and make your VM rock.
Update existing NVIDIA vGPU Driver for (Virtual Machine)
When the hypervisor NVIDIA vGPU manager is updated, next is updating the Virtual Machines vGPU driver.
- 392.05_grid_win8_win7_32bit_international.exe
- 392.05_grid_win8_win7_server2012R2_server2008R2_64bit_international.exe
- 392.05_grid_win10_32bit_international.exe
- 392.05_grid_win10_server2016_64bit_international.exe
- NVIDIA-Linux-x86_64-390.96-grid.run (only available with M10/M6/M60)
The vGPU driver for Windows 7, 8, 8.1, 10 is available with NVIDIA GRID vGPU download. This is available for both M6/M10/M60/P4/P6/P40/P100/V100
Update your Golden Images and reprovisioning the new virtual machines with updated vGPU drivers, if you have stateless machines update vGPU drivers on each.
#HINT – Express upgrade of drivers is the recommended option according to the setup. If you use the “Custom” option, you will have the option to do a “clean” installation. The downside of the “clean installation” is that it will remove all profiles and custom settings. The pro of using the clean installation option is that it will reinstall the complete driver, meaning that there will be no old driver files left on the system. I most of the time recommends using a “Clean” installation to keep it vanilla 🙂
#HINT (Citrix XenDesktop 7.12/7.13/7.14/7.15/7.16/7.17/7.18/7 1808 customers)
The NVIDIA vGPU API provides direct access to the frame buffer of the GPU, providing the fastest possible frame rate for a smooth and interactive user experience. If you install NVIDIA drivers before you install a VDA with HDX 3D Pro, NVIDIA vGPU is enabled by default.
To enable NVIDIA vGPU on a VM, disable Microsoft Basic Display Adapter from the Device Manager. Run the following command and then restart the VDA: NVFBCEnable.exe -enable -noreset
If you install NVIDIA drivers after you install a VDA with HDX 3D Pro, NVIDIA vGPU is disabled. Enable NVIDIA vGPU by using the NVFBCEnable tool provided by NVIDIA.
To disable NVIDIA vGPU, run the following command and then restart the VDA: NVFBCEnable.exe -disable -noreset
Source
NVIDIA Virtual GPU Software Documentation
https://docs.nvidia.com/grid/index.html
NVIDIA Virtual GPU Software Supported Products
NVIDIA Virtual GPU Software Quick Start Guide
NVIDIA Tesla M6/M10/M60/P4/P6/P40/P100/V100 – sources
vGPU vGPU Manager + Drivers are only available to customers and NVIDIA NPN partners for M6/M10/M60/P4/P6/P40/P100
Download if you are a NPN partner
Download if you are a GRID M6, M10, M60, P4, P6, P40, P100, V100 customer
