This page explains how to install GenAI on VMware Tanzu Platform for Cloud Foundry with an existing Tanzu Application Service deployment on VMware vSphere, so that apps hosted on Tanzu Application Service can use AI models running on NVIDIA vGPUs.

Caution

GenAI on Tanzu Platform is currently in beta and is intended for evaluation and test purposes only. Do not use in a production environment. This document applies to GenAI on Tanzu Platform v0.6.0 and may not apply to later versions.

Prerequisites

  • A Tanzu Application Service environment with:

    • Tanzu Operations Manager v3.0.24 or later.
    • Tanzu Application Service v4.0.x or later.
    • Internet access from BOSH compilation and Tanzu Application Service worker VMs, while GenAI on Tanzu Platform installs.
      • The VMs need online access to download models and components.
      • Once installation is complete, models and data stay within your Tanzu Application Service network, and Tanzu foundation VMs make no outbound calls to external third-party APIs or services.
      • Future support for air-gapped installation is under evaluation.
  • Hardware:
    • VMware ESXi 8.0.2 host machine.
    • NVIDIA Telsa T4 Graphics card, physically plugged into the VMware ESXi host.
  • NVIDIA Delegated License Service (DLS, on-premises) for NVIDIA License System v3.2.
  • Software, to install as described in the following sections.
    • NVIDIA GPU vSphere Installation Bundle (VIB) for ESXi, NVD-AIE-800_550.54.16-1OEM.800.1.0.20613240_23471877.zip.
    • One of the following vGPU drivers, to install on your VMware vSphere ESXi host as described in the following sections.
      • NVIDIA AI Enterprise (NVAIe) v5.0 vGPU software (requires NVIDIA license).
      • NVIDIA vGPU Software (requires NVIDIA license).
      • Ubuntu Guest OS driver.
    • GenAI on Tanzu Platform v0.6.0 or higher.
    • A BOSH stemcell with EFI support, downloaded from BOSH Stemcells. Stemcells that support EFI are available for:

Step 1: Set Up Your ESXi Host and Install a vGPU Driver

There are three options for the vGPU driver that’s installed on your ESXi host, which GenAI on Tanzu Platform includes when it creates vGPU-enabled VMs:

Each option has pros and cons.

  • NVIDIA NVAIe performs the best, by supporting the “-C compute-optimized” profile when assigning virtual slices of the GPU.
  • The Ubuntu OSS driver binds VMs to their underlying hosts, which prevents things like vMotion and ESXi host maintenance, but it does not require additional licensing from NVIDIA and is useful for test systems or POCs.

To install a NVIDIA NVAIe or NVIDIA vGPU driver on your ESXi host:

  1. Validate NVIDIA support for your ESXi version, vCenter version, and vGPU card by checking the support matrix for your vGPU driver:

  2. Download the appropriate version of the NVAIe or NVIDIA vGPU software from the NVIDIA License portal.

  3. Extract the downloaded zip file and find the guest driver.

  4. Upload the guest driver to a local file server where GenAI on Tanzu Platform can access it when it creates VMs. This can be either:

    • On Tanzu Application Service through a static file buildpack.
    • On an existing customer file server.

  5. Install the NVIDIA GPU vSphere Installation Bundle (VIB) on your ESXi Host, as described in Installing and configuring the NVIDIA VIB on ESXi in the Broadcom Support knowledge base.

    Note

    It is critical that the Guest Driver and ESXi host driver / VIB come from the same NVIDIA software release.

  6. Follow the NVIDIA License System Quick Start Guide to set up an NVIDIA license server, either in the cloud or on-premises:

Step 2: Install the GenAI on Tanzu Platform Tile

Download the GenAI on Tanzu Platform tile from Broadcom Support. After you have the GenAI on Tanzu Platform tile downloaded, upload the tile to your Tanzu Operations Manager. After the tile has been uploaded and added for installation, you’ll need to configure a couple of required properties, as follows.

Assign AZs and Networks

Configure the AZ for the tile to use where your GPUs are located. This can be an existing AZ within your deployment or a new one added with ESXi hosts that have installed GPUs. VMs deployed by the GenAI on VMware Tanzu Platform tile are new workloads and have no correlation to Deigo or isolation segments.

Settings tab in Tanzu Operations Manager to assign AZs and Networks.

Database Config

Configure a backing postgresql database. You may wish to use the VMware Postgres for Tanzu Application Service tile to provide the database.

Settings tab in Tanzu Operations Manager to configure database.

Step 3: (Optional) Configure Virtual GPUs (vGPUs)

Important

Only fill these fields out if you are using NVIDIA VGPU.

To configure vGPU support, click on the Virtual GPUs (vGPU) tab on the left of the page. Then enter required information as follows.

Nvidia license server address

This is the address to the NVIDIA license server. In this example, we are using a NVIDIA NLS appliance running on-premises vs. the NVIDIA CLS cloud service.

Nvidia license server client token

Get this token from your NVIDIA license server of choice. It is needed to register BOSH deployed VMs and successfully activate the NVIDIA GPU drivers.

Nvidia drivers URL

In this example, the guest driver is hosted on a TrueNAS fileserver by using the “webdav” protocol. The example string is “http://webdav:[email protected]:8080/webserver/nvidia-linux-grid-550_550.54.15_amd64.deb”. You should be able to host this file on any webserver running on your premises. If you do not have a fileserver to use, you can even host this on Tanzu Application Service itself by using the static file buildpack.

SHA256 of the Nvidia driver

For security, provide the SHA-256 hash value of the NVIDIA driver file. For example, to obtain this on MacOS, run a command and paste in output resembling:

$ shasum -a 256 ./nvidia-linux-grid-550_550.54.15_amd64.deb
cad102a736f6b0b62b7e75d874afb12d84a3cc35dc6f653269439b3033b358e6

Step 4: Deploy models

The GenAI on Tanzu Platform for Cloud Foundry tile is now configured and ready for usage on vSphere. Refer to the following tutorials and how-to guides to learn more about how to deploy and consume AI models on Tanzu Platform.

check-circle-line exclamation-circle-line close-line
Scroll to top icon