When you deploy a deep learning VM in vSphere IaaS control plane by using kubectl or directly on a vSphere cluster, you must fill in custom VM properties.

For information about deep learning VM images in VMware Private AI Foundation with NVIDIA, see About Deep Learning VM Images in VMware Private AI Foundation with NVIDIA.

OVF Properties of Deep Learning VMs

When you deploy a deep learning VM, you must fill in custom VM properties to automate the configuration of the Linux operating system, the deployment of the vGPU guest driver, and the deployment and configuration of NGC containers for the DL workloads.

The latest deep learning VM image has the following OVF properties:

Category Parameter Label in the vSphere Client Description
Base OS Properties instance-id Instance ID Required. A unique instance ID for the VM instance.

An instance ID uniquely identifies an instance. When an instance ID changes, cloud-init treats the instance as a new instance and runs the cloud-init process to again.
hostname Hostname Required. The host name of the appliance.
seedfrom URL to seed instance data from Optional. An URL to pull the value for the user-data parameter and metadata from.
public-keys SSH public key If provided, the instance populates the default user's SSH authorized_keys with this value.
user-data Encoded user-data

A set of scripts or other metadata that is inserted into the VM at provisioning time.

This property is the actual contents of the cloud-init script. This value must be base64 encoded.

password Default user's password Required. The password for the default vmware user account.

vGPU Driver Installation

 vgpu-license vGPU license Required. The NVIDIA vGPU client configuration token. The token is saved in the /etc/nvidia/ClientConfigToken/client_configuration_token.tok file.
nvidia-portal-api-key NVIDIA Portal API key

Required in a connected environment. The API key you downloaded from the NVIDIA Licensing Portal. The key is required for vGPU guest driver installation.

vgpu-host-driver-version vGPU host driver version Install directly this version of the vGPU guest driver.
vgpu-url URL for air-gapped vGPU downloads

Required in a disconnected environment. The URL to download the vGPU guest driver from. For information on the required configuration of the local Web server, see Preparing VMware Cloud Foundation for Private AI Workload Deployment.
DL Workload Automation registry-uri Registry URI Required in a disconnected environment or if you plan to use a private container registry to avoid downloading images from the Internet. The URI of a private container registry with the deep learning workload container images.

Required if you are referring to a private registry in user-data or image-oneliner.
registry-user Registry username Required if you are using a private container registry that requires basic authentication.
registry-passwd Registry password Required if you are using a private container registry that requires basic authentication.
registry-2-uri Secondary registry URI Required if you are using a second private container registry that is based on Docker and requires basic authentication.

For example, when deploying a deep learning VM with the NVIDIA RAG DL workload pre-installed, a pgvector image is downloaded from Docker Hub. You can use the registry-2- parameters to work around a pull rate limit for docker.io.

registry-2-user Secondary registry username Required if you are using a second private container registry.
registry-2-passwd Secondary registry password Required if you are using a second private container registry.
image-oneliner Encoded one-line command A one-line bash command that is run at VM provisioning. This value must be base64 encoded.

You can use this property to specify the DL workload container you want to deploy, such as PyTorch or TensorFlow. See Deep Learning Workloads in VMware Private AI Foundation with NVIDIA.

Caution: Avoid using both user-data and image-oneliner.
docker-compose-uri Encoded Docker compose file

Required if you need a Docker compose file to start the DL workload container. The contents of the docker-compose.yaml file that will be inserted into the virtual machine at provisioning after the virtual machine is started with GPU enabled. This value must be base64 encoded.
config-json Encoded config.json The contents of a configuration file for adding the following details:

This value must be base64 encoded.

conda-environment-install Conda Environment Install A comma-separated list of Conda environments to be automatically installed after VM deployment is complete.

Available environments: pytorch2.3_py3.12, pytorch1.13.1_py3.10, tf2.16.1_py3.12, and tf1.15.5_py3.7.

Deep Learning Workloads in VMware Private AI Foundation with NVIDIA

You can provision a deep learning virtual machine with a supported deep learning (DL) workload in addition to its embedded components. The DL workloads are downloaded from the NVIDIA NGC catalog and are GPU-optimized and validated by NVIDIA and VMware by Broadcom.

For an overview of the deep learning VM images, see About Deep Learning VM Images in VMware Private AI Foundation with NVIDIA.

CUDA Sample

You can use a deep learning VM with running CUDA samples to explore vector addition, gravitational n-body simulation, or other examples on a VM. See the CUDA Samples page.

After the deep learning VM is launched, it runs a CUDA sample workload to test the vGPU guest driver. You can examine the test output in the /var/log/dl.log file.

Table 1. CUDA Sample Container Image
Component Description
Container image 
nvcr.io/nvidia/k8s/cuda-sample:ngc_image_tag
For example:
nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8

For information on the CUDA Sample container images that are supported for deep learning VMs, see VMware Deep Learning VM Release Notes.
Required inputs To deploy a CUDA Sample workload, you must set the OVF properties for the deep learning virtual machine in the following way:
  • Use one of the following properties that are specific for the CUDA Sample image.
    • Cloud-init script. Encode it in base64 format. 
      #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    set_proxy "http" "https" "socks5"
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    DEFAULT_REG_URI="nvcr.io"
    REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

    if [[ -z "$REGISTRY_URI_PATH" ]]; then
      # If REGISTRY_URI_PATH is null or empty, use the default value
      REGISTRY_URI_PATH=$DEFAULT_REG_URI
      echo "REGISTRY_URI_PATH was empty. Using default: $REGISTRY_URI_PATH"
    fi
    
    # If REGISTRY_URI_PATH contains '/', extract the URI part
    if [[ $REGISTRY_URI_PATH == *"/"* ]]; then
      REGISTRY_URI=$(echo "$REGISTRY_URI_PATH" | cut -d'/' -f1)
    else
      REGISTRY_URI=$REGISTRY_URI_PATH
    fi
  
    REGISTRY_USERNAME=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    REGISTRY_PASSWORD=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    if [[ -n "$REGISTRY_USERNAME" && -n "$REGISTRY_PASSWORD" ]]; then
      docker login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_URI
    else
      echo "Warning: the registry's username and password are invalid, Skipping Docker login."
    fi
    
    deploy_dcgm_exporter

    echo "Info: running the vectoradd CUDA container"
    CUDA_SAMPLE_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/cuda-sample"
    CUDA_SAMPLE_VERSION="ngc_image_tag"
    docker run -d $CUDA_SAMPLE_IMAGE:$CUDA_SAMPLE_VERSION

- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi
  
      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker

      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="3.2.5-3.1.8-ubuntu22.04"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }

      For example, for vectoradd-cuda11.7.1-ubi8, provide the following script in base64 format: 

      I2Nsb3VkLWNvbmZpZwp3cml0ZV9maWxlczoKLSBwYXRoOiAvb3B0L2Rsdm0vZGxfYXBwLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBzZXQgLWV1CiAgICBzb3VyY2UgL29wdC9kbHZtL3V0aWxzLnNoCiAgICBzZXRfcHJveHkgImh0dHAiICJodHRwcyIgInNvY2tzNSIKICAgIHRyYXAgJ2Vycm9yX2V4aXQgIlVuZXhwZWN0ZWQgZXJyb3Igb2NjdXJzIGF0IGRsIHdvcmtsb2FkIicgRVJSCiAgICBERUZBVUxUX1JFR19VUkk9Im52Y3IuaW8iCiAgICBSRUdJU1RSWV9VUklfUEFUSD0kKGdyZXAgcmVnaXN0cnktdXJpIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKCiAgICBpZiBbWyAteiAiJFJFR0lTVFJZX1VSSV9QQVRIIiBdXTsgdGhlbgogICAgICAjIElmIFJFR0lTVFJZX1VSSV9QQVRIIGlzIG51bGwgb3IgZW1wdHksIHVzZSB0aGUgZGVmYXVsdCB2YWx1ZQogICAgICBSRUdJU1RSWV9VUklfUEFUSD0kREVGQVVMVF9SRUdfVVJJCiAgICAgIGVjaG8gIlJFR0lTVFJZX1VSSV9QQVRIIHdhcyBlbXB0eS4gVXNpbmcgZGVmYXVsdDogJFJFR0lTVFJZX1VSSV9QQVRIIgogICAgZmkKICAgIAogICAgIyBJZiBSRUdJU1RSWV9VUklfUEFUSCBjb250YWlucyAnLycsIGV4dHJhY3QgdGhlIFVSSSBwYXJ0CiAgICBpZiBbWyAkUkVHSVNUUllfVVJJX1BBVEggPT0gKiIvIiogXV07IHRoZW4KICAgICAgUkVHSVNUUllfVVJJPSQoZWNobyAiJFJFR0lTVFJZX1VSSV9QQVRIIiB8IGN1dCAtZCcvJyAtZjEpCiAgICBlbHNlCiAgICAgIFJFR0lTVFJZX1VSST0kUkVHSVNUUllfVVJJX1BBVEgKICAgIGZpCiAgCiAgICBSRUdJU1RSWV9VU0VSTkFNRT0kKGdyZXAgcmVnaXN0cnktdXNlciAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCiAgICBSRUdJU1RSWV9QQVNTV09SRD0kKGdyZXAgcmVnaXN0cnktcGFzc3dkIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKICAgIGlmIFtbIC1uICIkUkVHSVNUUllfVVNFUk5BTUUiICYmIC1uICIkUkVHSVNUUllfUEFTU1dPUkQiIF1dOyB0aGVuCiAgICAgIGRvY2tlciBsb2dpbiAtdSAkUkVHSVNUUllfVVNFUk5BTUUgLXAgJFJFR0lTVFJZX1BBU1NXT1JEICRSRUdJU1RSWV9VUkkKICAgIGVsc2UKICAgICAgZWNobyAiV2FybmluZzogdGhlIHJlZ2lzdHJ5J3MgdXNlcm5hbWUgYW5kIHBhc3N3b3JkIGFyZSBpbnZhbGlkLCBTa2lwcGluZyBEb2NrZXIgbG9naW4uIgogICAgZmkKICAgIAogICAgZGVwbG95X2RjZ21fZXhwb3J0ZXIKCiAgICBlY2hvICJJbmZvOiBydW5uaW5nIHRoZSB2ZWN0b3JhZGQgQ1VEQSBjb250YWluZXIiCiAgICBDVURBX1NBTVBMRV9JTUFHRT0iJFJFR0lTVFJZX1VSSV9QQVRIL252aWRpYS9rOHMvY3VkYS1zYW1wbGUiCiAgICBDVURBX1NBTVBMRV9WRVJTSU9OPSJ2ZWN0b3JhZGQtY3VkYTExLjcuMS11Ymk4IgogICAgZG9ja2VyIHJ1biAtZCAkQ1VEQV9TQU1QTEVfSU1BR0U6JENVREFfU0FNUExFX1ZFUlNJT04KCi0gcGF0aDogL29wdC9kbHZtL3V0aWxzLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBlcnJvcl9leGl0KCkgewogICAgICBlY2hvICJFcnJvcjogJDEiID4mMgogICAgICB2bXRvb2xzZCAtLWNtZCAiaW5mby1zZXQgZ3Vlc3RpbmZvLnZtc2VydmljZS5ib290c3RyYXAuY29uZGl0aW9uIGZhbHNlLCBETFdvcmtsb2FkRmFpbHVyZSwgJDEiCiAgICAgIGV4aXQgMQogICAgfQoKICAgIGNoZWNrX3Byb3RvY29sKCkgewogICAgICBsb2NhbCBwcm94eV91cmw9JDEKICAgICAgc2hpZnQKICAgICAgbG9jYWwgc3VwcG9ydGVkX3Byb3RvY29scz0oIiRAIikKICAgICAgaWYgW1sgLW4gIiR7cHJveHlfdXJsfSIgXV07IHRoZW4KICAgICAgICBsb2NhbCBwcm90b2NvbD0kKGVjaG8gIiR7cHJveHlfdXJsfSIgfCBhd2sgLUYgJzovLycgJ3tpZiAoTkYgPiAxKSBwcmludCAkMTsgZWxzZSBwcmludCAiIn0nKQogICAgICAgIGlmIFsgLXogIiRwcm90b2NvbCIgXTsgdGhlbgogICAgICAgICAgZWNobyAiTm8gc3BlY2lmaWMgcHJvdG9jb2wgcHJvdmlkZWQuIFNraXBwaW5nIHByb3RvY29sIGNoZWNrLiIKICAgICAgICAgIHJldHVybiAwCiAgICAgICAgZmkKICAgICAgICBsb2NhbCBwcm90b2NvbF9pbmNsdWRlZD1mYWxzZQogICAgICAgIGZvciB2YXIgaW4gIiR7c3VwcG9ydGVkX3Byb3RvY29sc1tAXX0iOyBkbwogICAgICAgICAgaWYgW1sgIiR7cHJvdG9jb2x9IiA9PSAiJHt2YXJ9IiBdXTsgdGhlbgogICAgICAgICAgICBwcm90b2NvbF9pbmNsdWRlZD10cnVlCiAgICAgICAgICAgIGJyZWFrCiAgICAgICAgICBmaQogICAgICAgIGRvbmUKICAgICAgICBpZiBbWyAiJHtwcm90b2NvbF9pbmNsdWRlZH0iID09IGZhbHNlIF1dOyB0aGVuCiAgICAgICAgICBlcnJvcl9leGl0ICJVbnN1cHBvcnRlZCBwcm90b2NvbDogJHtwcm90b2NvbH0uIFN1cHBvcnRlZCBwcm90b2NvbHMgYXJlOiAke3N1cHBvcnRlZF9wcm90b2NvbHNbKl19IgogICAgICAgIGZpCiAgICAgIGZpCiAgICB9CgogICAgIyAkQDogbGlzdCBvZiBzdXBwb3J0ZWQgcHJvdG9jb2xzCiAgICBzZXRfcHJveHkoKSB7CiAgICAgIGxvY2FsIHN1cHBvcnRlZF9wcm90b2NvbHM9KCIkQCIpCgogICAgICBDT05GSUdfSlNPTl9CQVNFNjQ9JChncmVwICdjb25maWctanNvbicgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQogICAgICBDT05GSUdfSlNPTj0kKGVjaG8gJHtDT05GSUdfSlNPTl9CQVNFNjR9IHwgYmFzZTY0IC0tZGVjb2RlKQoKICAgICAgSFRUUF9QUk9YWV9VUkw9JChlY2hvICIke0NPTkZJR19KU09OfSIgfCBqcSAtciAnLmh0dHBfcHJveHkgLy8gZW1wdHknKQogICAgICBIVFRQU19QUk9YWV9VUkw9JChlY2hvICIke0NPTkZJR19KU09OfSIgfCBqcSAtciAnLmh0dHBzX3Byb3h5IC8vIGVtcHR5JykKICAgICAgaWYgW1sgJD8gLW5lIDAgfHwgKC16ICIke0hUVFBfUFJPWFlfVVJMfSIgJiYgLXogIiR7SFRUUFNfUFJPWFlfVVJMfSIpIF1dOyB0aGVuCiAgICAgICAgZWNobyAiSW5mbzogVGhlIGNvbmZpZy1qc29uIHdhcyBwYXJzZWQsIGJ1dCBubyBwcm94eSBzZXR0aW5ncyB3ZXJlIGZvdW5kLiIKICAgICAgICByZXR1cm4gMAogICAgICBmaQogIAogICAgICBjaGVja19wcm90b2NvbCAiJHtIVFRQX1BST1hZX1VSTH0iICIke3N1cHBvcnRlZF9wcm90b2NvbHNbQF19IgogICAgICBjaGVja19wcm90b2NvbCAiJHtIVFRQU19QUk9YWV9VUkx9IiAiJHtzdXBwb3J0ZWRfcHJvdG9jb2xzW0BdfSIKCiAgICAgIGlmICEgZ3JlcCAtcSAnaHR0cF9wcm94eScgL2V0Yy9lbnZpcm9ubWVudDsgdGhlbgogICAgICAgIHN1ZG8gYmFzaCAtYyAnZWNobyAiZXhwb3J0IGh0dHBfcHJveHk9JHtIVFRQX1BST1hZX1VSTH0KICAgICAgICBleHBvcnQgaHR0cHNfcHJveHk9JHtIVFRQU19QUk9YWV9VUkx9CiAgICAgICAgZXhwb3J0IEhUVFBfUFJPWFk9JHtIVFRQX1BST1hZX1VSTH0KICAgICAgICBleHBvcnQgSFRUUFNfUFJPWFk9JHtIVFRQU19QUk9YWV9VUkx9CiAgICAgICAgZXhwb3J0IG5vX3Byb3h5PWxvY2FsaG9zdCwxMjcuMC4wLjEiID4+IC9ldGMvZW52aXJvbm1lbnQnCiAgICAgICAgc291cmNlIC9ldGMvZW52aXJvbm1lbnQKICAgICAgZmkKICAgICAgCiAgICAgICMgQ29uZmlndXJlIERvY2tlciB0byB1c2UgYSBwcm94eQogICAgICBzdWRvIG1rZGlyIC1wIC9ldGMvc3lzdGVtZC9zeXN0ZW0vZG9ja2VyLnNlcnZpY2UuZAogICAgICBzdWRvIGJhc2ggLWMgJ2VjaG8gIltTZXJ2aWNlXQogICAgICBFbnZpcm9ubWVudD1cIkhUVFBfUFJPWFk9JHtIVFRQX1BST1hZX1VSTH1cIgogICAgICBFbnZpcm9ubWVudD1cIkhUVFBTX1BST1hZPSR7SFRUUFNfUFJPWFlfVVJMfVwiCiAgICAgIEVudmlyb25tZW50PVwiTk9fUFJPWFk9bG9jYWxob3N0LDEyNy4wLjAuMVwiIiA+IC9ldGMvc3lzdGVtZC9zeXN0ZW0vZG9ja2VyLnNlcnZpY2UuZC9wcm94eS5jb25mJwogICAgICBzdWRvIHN5c3RlbWN0bCBkYWVtb24tcmVsb2FkCiAgICAgIHN1ZG8gc3lzdGVtY3RsIHJlc3RhcnQgZG9ja2VyCgogICAgICBlY2hvICJJbmZvOiBkb2NrZXIgYW5kIHN5c3RlbSBlbnZpcm9ubWVudCBhcmUgbm93IGNvbmZpZ3VyZWQgdG8gdXNlIHRoZSBwcm94eSBzZXR0aW5ncyIKICAgIH0KCiAgICBkZXBsb3lfZGNnbV9leHBvcnRlcigpIHsKICAgICAgQ09ORklHX0pTT05fQkFTRTY0PSQoZ3JlcCAnY29uZmlnLWpzb24nIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKICAgICAgQ09ORklHX0pTT049JChlY2hvICR7Q09ORklHX0pTT05fQkFTRTY0fSB8IGJhc2U2NCAtLWRlY29kZSkKICAgICAgRENHTV9FWFBPUlRfUFVCTElDPSQoZWNobyAiJHtDT05GSUdfSlNPTn0iIHwganEgLXIgJy5leHBvcnRfZGNnbV90b19wdWJsaWMgLy8gZW1wdHknKQoKICAgICAgRENHTV9FWFBPUlRFUl9JTUFHRT0iJFJFR0lTVFJZX1VSSV9QQVRIL252aWRpYS9rOHMvZGNnbS1leHBvcnRlciIKICAgICAgRENHTV9FWFBPUlRFUl9WRVJTSU9OPSIzLjIuNS0zLjEuOC11YnVudHUyMi4wNCIKICAgICAgaWYgWyAteiAiJHtEQ0dNX0VYUE9SVF9QVUJMSUN9IiBdIHx8IFsgIiR7RENHTV9FWFBPUlRfUFVCTElDfSIgIT0gInRydWUiIF07IHRoZW4KICAgICAgICBlY2hvICJJbmZvOiBsYXVuY2hpbmcgRENHTSBFeHBvcnRlciB0byBjb2xsZWN0IHZHUFUgbWV0cmljcywgbGlzdGVuaW5nIG9ubHkgb24gbG9jYWxob3N0ICgxMjcuMC4wLjE6OTQwMCkiCiAgICAgICAgZG9ja2VyIHJ1biAtZCAtLWdwdXMgYWxsIC0tY2FwLWFkZCBTWVNfQURNSU4gLXAgMTI3LjAuMC4xOjk0MDA6OTQwMCAkRENHTV9FWFBPUlRFUl9JTUFHRTokRENHTV9FWFBPUlRFUl9WRVJTSU9OCiAgICAgIGVsc2UKICAgICAgICBlY2hvICJJbmZvOiBsYXVuY2hpbmcgRENHTSBFeHBvcnRlciB0byBjb2xsZWN0IHZHUFUgbWV0cmljcywgZXhwb3NlZCBvbiBhbGwgbmV0d29yayBpbnRlcmZhY2VzICgwLjAuMC4wOjk0MDApIgogICAgICAgIGRvY2tlciBydW4gLWQgLS1ncHVzIGFsbCAtLWNhcC1hZGQgU1lTX0FETUlOIC1wIDk0MDA6OTQwMCAkRENHTV9FWFBPUlRFUl9JTUFHRTokRENHTV9FWFBPUlRFUl9WRVJTSU9OCiAgICAgIGZpCiAgICB9

      which corresponds to the following script in plain-text format:

      #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    set_proxy "http" "https" "socks5"
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    DEFAULT_REG_URI="nvcr.io"
    REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

    if [[ -z "$REGISTRY_URI_PATH" ]]; then
      # If REGISTRY_URI_PATH is null or empty, use the default value
      REGISTRY_URI_PATH=$DEFAULT_REG_URI
      echo "REGISTRY_URI_PATH was empty. Using default: $REGISTRY_URI_PATH"
    fi
    
    # If REGISTRY_URI_PATH contains '/', extract the URI part
    if [[ $REGISTRY_URI_PATH == *"/"* ]]; then
      REGISTRY_URI=$(echo "$REGISTRY_URI_PATH" | cut -d'/' -f1)
    else
      REGISTRY_URI=$REGISTRY_URI_PATH
    fi
  
    REGISTRY_USERNAME=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    REGISTRY_PASSWORD=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    if [[ -n "$REGISTRY_USERNAME" && -n "$REGISTRY_PASSWORD" ]]; then
      docker login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_URI
    else
      echo "Warning: the registry's username and password are invalid, Skipping Docker login."
    fi
    
    deploy_dcgm_exporter

    echo "Info: running the vectoradd CUDA container"
    CUDA_SAMPLE_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/cuda-sample"
    CUDA_SAMPLE_VERSION="vectoradd-cuda11.7.1-ubi8"
    docker run -d $CUDA_SAMPLE_IMAGE:$CUDA_SAMPLE_VERSION

- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi
  
      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker

      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="3.2.5-3.1.8-ubuntu22.04"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }
    • Image one-liner. Encode it in base64 format
      docker run -d nvcr.io/nvidia/k8s/cuda-sample:ngc_image_tag

      For example, for vectoradd-cuda11.7.1-ubi8, provide the following script in base64 format:

      ZG9ja2VyIHJ1biAtZCBudmNyLmlvL252aWRpYS9rOHMvY3VkYS1zYW1wbGU6dmVjdG9yYWRkLWN1ZGExMS43LjEtdWJpOA==

      which corresponds to the following script in plain-text format:

      docker run -d nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8
  • Enter the vGPU guest driver installation properties, such as vgpu-license and nvidia-portal-api-key.
  • Provide values for the properties required for a disconnected environment as needed.

See OVF Properties of Deep Learning VMs.
Output
  • Installation logs for the vGPU guest driver in /var/log/vgpu-install.log.

    To verify that the vGPU guest driver is installed, and the license is allocated, run the following command:

    nvidia-smi -q |grep -i license
  • Cloud-init script logs in /var/log/dl.log.

PyTorch

You can use a deep learning VM with a PyTorch library to explore conversational AI, NLP, and other types of AI models, on a VM. See the PyTorch page.

After the deep learning VM is launched, it starts a JupyterLab instance with PyTorch packages installed and configured.

Table 2. PyTorch Container Image
Component Description
Container image 
nvcr.io/nvidia/pytorch-pb24h1:ngc_image_tag
For example: 
nvcr.io/nvidia/pytorch-pb24h1:24.03.02-py3

For information on the PyTorch container images that are supported for deep learning VMs, see VMware Deep Learning VM Release Notes.
Required inputs To deploy a PyTorch workload, you must set the OVF properties for the deep learning virtual machine in the following way:
  • Use one of the following properties that are specific for the PyTorch image.
    • Cloud-init script. Encode it in base64 format.
      #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    set_proxy "http" "https" "socks5"

    DEFAULT_REG_URI="nvcr.io"
    REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

    if [[ -z "$REGISTRY_URI_PATH" ]]; then
      # If REGISTRY_URI_PATH is null or empty, use the default value
      REGISTRY_URI_PATH=$DEFAULT_REG_URI
      echo "REGISTRY_URI_PATH was empty. Using default: $REGISTRY_URI_PATH"
    fi
    
    # If REGISTRY_URI_PATH contains '/', extract the URI part
    if [[ $REGISTRY_URI_PATH == *"/"* ]]; then
      REGISTRY_URI=$(echo "$REGISTRY_URI_PATH" | cut -d'/' -f1)
    else
      REGISTRY_URI=$REGISTRY_URI_PATH
    fi
  
    REGISTRY_USERNAME=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    REGISTRY_PASSWORD=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    if [[ -n "$REGISTRY_USERNAME" && -n "$REGISTRY_PASSWORD" ]]; then
      docker login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_URI
    else
      echo "Warning: the registry's username and password are invalid, Skipping Docker login."
    fi

    deploy_dcgm_exporter

    CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
    enableJupyterAuth=$(echo "${CONFIG_JSON}" | jq -r '.enable_jupyter_auth // empty')

    if [ -z "${enableJupyterAuth}" ] || [ "${enableJupyterAuth}" == true ]; then
      # Generate a random jupyter token
      TOKEN=$(python3 -c "import secrets; print(secrets.token_hex(32))")
      # Set the token to guestinfo
      vmtoolsd --cmd "info-set guestinfo.dlworkload.jupyterlab.token $TOKEN"
      echo "Info: JupyterLab notebook access token, $TOKEN"
    else
      TOKEN=""
    fi

    echo "Info: running the PyTorch container"
    PYTORCH_IMAGE="$REGISTRY_URI_PATH/nvidia/pytorch-pb24h1"
    PYTORCH_VERSION="ngc_image_tag"
    docker run -d --gpus all -p 8888:8888 $PYTORCH_IMAGE:$PYTORCH_VERSION /usr/local/bin/jupyter lab --allow-root --ip=* --port=8888 --no-browser --NotebookApp.token="$TOKEN" --NotebookApp.allow_origin="*" --notebook-dir=/workspace

- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi

      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker

      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="3.2.5-3.1.8-ubuntu22.04"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }

      For example, for pytorch-pb24h1:24.03.02-py3, provide the following script in base 64 format: 

      I2Nsb3VkLWNvbmZpZwp3cml0ZV9maWxlczoKLSBwYXRoOiAvb3B0L2Rsdm0vZGxfYXBwLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBzZXQgLWV1CiAgICBzb3VyY2UgL29wdC9kbHZtL3V0aWxzLnNoCiAgICB0cmFwICdlcnJvcl9leGl0ICJVbmV4cGVjdGVkIGVycm9yIG9jY3VycyBhdCBkbCB3b3JrbG9hZCInIEVSUgogICAgc2V0X3Byb3h5ICJodHRwIiAiaHR0cHMiICJzb2NrczUiCgogICAgREVGQVVMVF9SRUdfVVJJPSJudmNyLmlvIgogICAgUkVHSVNUUllfVVJJX1BBVEg9JChncmVwIHJlZ2lzdHJ5LXVyaSAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCgogICAgaWYgW1sgLXogIiRSRUdJU1RSWV9VUklfUEFUSCIgXV07IHRoZW4KICAgICAgIyBJZiBSRUdJU1RSWV9VUklfUEFUSCBpcyBudWxsIG9yIGVtcHR5LCB1c2UgdGhlIGRlZmF1bHQgdmFsdWUKICAgICAgUkVHSVNUUllfVVJJX1BBVEg9JERFRkFVTFRfUkVHX1VSSQogICAgICBlY2hvICJSRUdJU1RSWV9VUklfUEFUSCB3YXMgZW1wdHkuIFVzaW5nIGRlZmF1bHQ6ICRSRUdJU1RSWV9VUklfUEFUSCIKICAgIGZpCiAgICAKICAgICMgSWYgUkVHSVNUUllfVVJJX1BBVEggY29udGFpbnMgJy8nLCBleHRyYWN0IHRoZSBVUkkgcGFydAogICAgaWYgW1sgJFJFR0lTVFJZX1VSSV9QQVRIID09ICoiLyIqIF1dOyB0aGVuCiAgICAgIFJFR0lTVFJZX1VSST0kKGVjaG8gIiRSRUdJU1RSWV9VUklfUEFUSCIgfCBjdXQgLWQnLycgLWYxKQogICAgZWxzZQogICAgICBSRUdJU1RSWV9VUkk9JFJFR0lTVFJZX1VSSV9QQVRICiAgICBmaQogIAogICAgUkVHSVNUUllfVVNFUk5BTUU9JChncmVwIHJlZ2lzdHJ5LXVzZXIgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQogICAgUkVHSVNUUllfUEFTU1dPUkQ9JChncmVwIHJlZ2lzdHJ5LXBhc3N3ZCAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCiAgICBpZiBbWyAtbiAiJFJFR0lTVFJZX1VTRVJOQU1FIiAmJiAtbiAiJFJFR0lTVFJZX1BBU1NXT1JEIiBdXTsgdGhlbgogICAgICBkb2NrZXIgbG9naW4gLXUgJFJFR0lTVFJZX1VTRVJOQU1FIC1wICRSRUdJU1RSWV9QQVNTV09SRCAkUkVHSVNUUllfVVJJCiAgICBlbHNlCiAgICAgIGVjaG8gIldhcm5pbmc6IHRoZSByZWdpc3RyeSdzIHVzZXJuYW1lIGFuZCBwYXNzd29yZCBhcmUgaW52YWxpZCwgU2tpcHBpbmcgRG9ja2VyIGxvZ2luLiIKICAgIGZpCgogICAgZG9ja2VyIHJ1biAtZCAtLWdwdXMgYWxsIC1wIDg4ODg6ODg4OCAkUkVHSVNUUllfVVJJX1BBVEgvbnZpZGlhL3B5dG9yY2g6MjMuMTAtcHkzIC91c3IvbG9jYWwvYmluL2p1cHl0ZXIgbGFiIC0tYWxsb3ctcm9vdCAtLWlwPSogLS1wb3J0PTg4ODggLS1uby1icm93c2VyIC0tTm90ZWJvb2tBcHAudG9rZW49JycgLS1Ob3RlYm9va0FwcC5hbGxvd19vcmlnaW49JyonIC0tbm90ZWJvb2stZGlyPS93b3Jrc3BhY2UKCi0gcGF0aDogL29wdC9kbHZtL3V0aWxzLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBlcnJvcl9leGl0KCkgewogICAgICBlY2hvICJFcnJvcjogJDEiID4mMgogICAgICB2bXRvb2xzZCAtLWNtZCAiaW5mby1zZXQgZ3Vlc3RpbmZvLnZtc2VydmljZS5ib290c3RyYXAuY29uZGl0aW9uIGZhbHNlLCBETFdvcmtsb2FkRmFpbHVyZSwgJDEiCiAgICAgIGV4aXQgMQogICAgfQoKICAgIGNoZWNrX3Byb3RvY29sKCkgewogICAgICBsb2NhbCBwcm94eV91cmw9JDEKICAgICAgc2hpZnQKICAgICAgbG9jYWwgc3VwcG9ydGVkX3Byb3RvY29scz0oIiRAIikKICAgICAgaWYgW1sgLW4gIiR7cHJveHlfdXJsfSIgXV07IHRoZW4KICAgICAgICBsb2NhbCBwcm90b2NvbD0kKGVjaG8gIiR7cHJveHlfdXJsfSIgfCBhd2sgLUYgJzovLycgJ3tpZiAoTkYgPiAxKSBwcmludCAkMTsgZWxzZSBwcmludCAiIn0nKQogICAgICAgIGlmIFsgLXogIiRwcm90b2NvbCIgXTsgdGhlbgogICAgICAgICAgZWNobyAiTm8gc3BlY2lmaWMgcHJvdG9jb2wgcHJvdmlkZWQuIFNraXBwaW5nIHByb3RvY29sIGNoZWNrLiIKICAgICAgICAgIHJldHVybiAwCiAgICAgICAgZmkKICAgICAgICBsb2NhbCBwcm90b2NvbF9pbmNsdWRlZD1mYWxzZQogICAgICAgIGZvciB2YXIgaW4gIiR7c3VwcG9ydGVkX3Byb3RvY29sc1tAXX0iOyBkbwogICAgICAgICAgaWYgW1sgIiR7cHJvdG9jb2x9IiA9PSAiJHt2YXJ9IiBdXTsgdGhlbgogICAgICAgICAgICBwcm90b2NvbF9pbmNsdWRlZD10cnVlCiAgICAgICAgICAgIGJyZWFrCiAgICAgICAgICBmaQogICAgICAgIGRvbmUKICAgICAgICBpZiBbWyAiJHtwcm90b2NvbF9pbmNsdWRlZH0iID09IGZhbHNlIF1dOyB0aGVuCiAgICAgICAgICBlcnJvcl9leGl0ICJVbnN1cHBvcnRlZCBwcm90b2NvbDogJHtwcm90b2NvbH0uIFN1cHBvcnRlZCBwcm90b2NvbHMgYXJlOiAke3N1cHBvcnRlZF9wcm90b2NvbHNbKl19IgogICAgICAgIGZpCiAgICAgIGZpCiAgICB9CgogICAgIyAkQDogbGlzdCBvZiBzdXBwb3J0ZWQgcHJvdG9jb2xzCiAgICBzZXRfcHJveHkoKSB7CiAgICAgIGxvY2FsIHN1cHBvcnRlZF9wcm90b2NvbHM9KCIkQCIpCgogICAgICBDT05GSUdfSlNPTl9CQVNFNjQ9JChncmVwICdjb25maWctanNvbicgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQogICAgICBDT05GSUdfSlNPTj0kKGVjaG8gJHtDT05GSUdfSlNPTl9CQVNFNjR9IHwgYmFzZTY0IC0tZGVjb2RlKQoKICAgICAgSFRUUF9QUk9YWV9VUkw9JChlY2hvICIke0NPTkZJR19KU09OfSIgfCBqcSAtciAnLmh0dHBfcHJveHkgLy8gZW1wdHknKQogICAgICBIVFRQU19QUk9YWV9VUkw9JChlY2hvICIke0NPTkZJR19KU09OfSIgfCBqcSAtciAnLmh0dHBzX3Byb3h5IC8vIGVtcHR5JykKICAgICAgaWYgW1sgJD8gLW5lIDAgfHwgKC16ICIke0hUVFBfUFJPWFlfVVJMfSIgJiYgLXogIiR7SFRUUFNfUFJPWFlfVVJMfSIpIF1dOyB0aGVuCiAgICAgICAgZWNobyAiSW5mbzogVGhlIGNvbmZpZy1qc29uIHdhcyBwYXJzZWQsIGJ1dCBubyBwcm94eSBzZXR0aW5ncyB3ZXJlIGZvdW5kLiIKICAgICAgICByZXR1cm4gMAogICAgICBmaQoKICAgICAgY2hlY2tfcHJvdG9jb2wgIiR7SFRUUF9QUk9YWV9VUkx9IiAiJHtzdXBwb3J0ZWRfcHJvdG9jb2xzW0BdfSIKICAgICAgY2hlY2tfcHJvdG9jb2wgIiR7SFRUUFNfUFJPWFlfVVJMfSIgIiR7c3VwcG9ydGVkX3Byb3RvY29sc1tAXX0iCgogICAgICBpZiAhIGdyZXAgLXEgJ2h0dHBfcHJveHknIC9ldGMvZW52aXJvbm1lbnQ7IHRoZW4KICAgICAgICBlY2hvICJleHBvcnQgaHR0cF9wcm94eT0ke0hUVFBfUFJPWFlfVVJMfQogICAgICAgIGV4cG9ydCBodHRwc19wcm94eT0ke0hUVFBTX1BST1hZX1VSTH0KICAgICAgICBleHBvcnQgSFRUUF9QUk9YWT0ke0hUVFBfUFJPWFlfVVJMfQogICAgICAgIGV4cG9ydCBIVFRQU19QUk9YWT0ke0hUVFBTX1BST1hZX1VSTH0KICAgICAgICBleHBvcnQgbm9fcHJveHk9bG9jYWxob3N0LDEyNy4wLjAuMSIgPj4gL2V0Yy9lbnZpcm9ubWVudAogICAgICAgIHNvdXJjZSAvZXRjL2Vudmlyb25tZW50CiAgICAgIGZpCiAgICAgIAogICAgICAjIENvbmZpZ3VyZSBEb2NrZXIgdG8gdXNlIGEgcHJveHkKICAgICAgbWtkaXIgLXAgL2V0Yy9zeXN0ZW1kL3N5c3RlbS9kb2NrZXIuc2VydmljZS5kCiAgICAgIGVjaG8gIltTZXJ2aWNlXQogICAgICBFbnZpcm9ubWVudD1cIkhUVFBfUFJPWFk9JHtIVFRQX1BST1hZX1VSTH1cIgogICAgICBFbnZpcm9ubWVudD1cIkhUVFBTX1BST1hZPSR7SFRUUFNfUFJPWFlfVVJMfVwiCiAgICAgIEVudmlyb25tZW50PVwiTk9fUFJPWFk9bG9jYWxob3N0LDEyNy4wLjAuMVwiIiA+IC9ldGMvc3lzdGVtZC9zeXN0ZW0vZG9ja2VyLnNlcnZpY2UuZC9wcm94eS5jb25mCiAgICAgIHN5c3RlbWN0bCBkYWVtb24tcmVsb2FkCiAgICAgIHN5c3RlbWN0bCByZXN0YXJ0IGRvY2tlcgoKICAgICAgZWNobyAiSW5mbzogZG9ja2VyIGFuZCBzeXN0ZW0gZW52aXJvbm1lbnQgYXJlIG5vdyBjb25maWd1cmVkIHRvIHVzZSB0aGUgcHJveHkgc2V0dGluZ3MiCiAgICB9

      which corresponds to the following script in plain-text format.

      #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    set_proxy "http" "https" "socks5"

    DEFAULT_REG_URI="nvcr.io"
    REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

    if [[ -z "$REGISTRY_URI_PATH" ]]; then
      # If REGISTRY_URI_PATH is null or empty, use the default value
      REGISTRY_URI_PATH=$DEFAULT_REG_URI
      echo "REGISTRY_URI_PATH was empty. Using default: $REGISTRY_URI_PATH"
    fi
    
    # If REGISTRY_URI_PATH contains '/', extract the URI part
    if [[ $REGISTRY_URI_PATH == *"/"* ]]; then
      REGISTRY_URI=$(echo "$REGISTRY_URI_PATH" | cut -d'/' -f1)
    else
      REGISTRY_URI=$REGISTRY_URI_PATH
    fi
  
    REGISTRY_USERNAME=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    REGISTRY_PASSWORD=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    if [[ -n "$REGISTRY_USERNAME" && -n "$REGISTRY_PASSWORD" ]]; then
      docker login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_URI
    else
      echo "Warning: the registry's username and password are invalid, Skipping Docker login."
    fi

    deploy_dcgm_exporter

    CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
    enableJupyterAuth=$(echo "${CONFIG_JSON}" | jq -r '.enable_jupyter_auth // empty')

    if [ -z "${enableJupyterAuth}" ] || [ "${enableJupyterAuth}" == true ]; then
      # Generate a random jupyter token
      TOKEN=$(python3 -c "import secrets; print(secrets.token_hex(32))")
      # Set the token to guestinfo
      vmtoolsd --cmd "info-set guestinfo.dlworkload.jupyterlab.token $TOKEN"
      echo "Info: JupyterLab notebook access token, $TOKEN"
    else
      TOKEN=""
    fi

    echo "Info: running the PyTorch container"
    PYTORCH_IMAGE="$REGISTRY_URI_PATH/nvidia/pytorch-pb24h1"
    PYTORCH_VERSION="24.03.02-py3"
    docker run -d --gpus all -p 8888:8888 $PYTORCH_IMAGE:$PYTORCH_VERSION /usr/local/bin/jupyter lab --allow-root --ip=* --port=8888 --no-browser --NotebookApp.token="$TOKEN" --NotebookApp.allow_origin="*" --notebook-dir=/workspace

- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi

      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker

      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="3.2.5-3.1.8-ubuntu22.04"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }
    • Image one-liner. Encode it in base64 format.
      docker run -d -p 8888:8888 nvcr.io/nvidia/pytorch-pb24h1:ngc_image_tag /usr/local/bin/jupyter lab --allow-root --ip=* --port=8888 --no-browser --NotebookApp.token='' --NotebookApp.allow_origin='*' --notebook-dir=/workspace

      For example, for pytorch-pb24h1:24.03.02-py3, provide the following script in base 64 format: 

      ZG9ja2VyIHJ1biAtZCAtcCA4ODg4Ojg4ODggbnZjci5pby9udmlkaWEvcHl0b3JjaC1wYjI0aDE6MjQuMDMuMDItcHkzIC91c3IvbG9jYWwvYmluL2p1cHl0ZXIgbGFiIC0tYWxsb3ctcm9vdCAtLWlwPSogLS1wb3J0PTg4ODggLS1uby1icm93c2VyIC0tTm90ZWJvb2tBcHAudG9rZW49JycgLS1Ob3RlYm9va0FwcC5hbGxvd19vcmlnaW49JyonIC0tbm90ZWJvb2stZGlyPS93b3Jrc3BhY2U=

      which corresponds to the following script in plain-text format:

      docker run -d -p 8888:8888 nvcr.io/nvidia/pytorch-pb24h1:24.03.02-py3 /usr/local/bin/jupyter lab --allow-root --ip=* --port=8888 --no-browser --NotebookApp.token='' --NotebookApp.allow_origin='*' --notebook-dir=/workspace
  • Enter the vGPU guest driver installation properties, such as vgpu-license and nvidia-portal-api-key.
  • Provide values for the properties required for a disconnected environment as needed.

See OVF Properties of Deep Learning VMs.
Output
  • Installation logs for the vGPU guest driver in /var/log/vgpu-install.log.

    To verify that the vGPU guest driver is installed, run the nvidia-smi command.

  • Cloud-init script logs in /var/log/dl.log.
  • PyTorch container.

    To verify that the PyTorch container is running, run the sudo docker ps -a and sudo docker logs container_id command.

  • JupyterLab instance that you can access at http://dl_vm_ip:8888

    In the terminal of JupyterLab, verify that the following functionality is available in the notebook:

    • To verify that JupyterLab can access the vGPU resource, run nvidia-smi.
    • To verify that the PyTorch related packages are installed, run pip show.

TensorFlow

You can use a deep learning VM with a TensorFlow library to explore conversational AI, NLP, and other types of AI models, on a VM. See the TensorFlow page.

After the deep learning VM is launched, it starts a JupyterLab instance with TensorFlow packages installed and configured.

Table 3. TensorFlow Container Image
Component Description
Container image 
nvcr.io/nvidia/tensorflow-pb24h1:ngc_image_tag

For example: 

nvcr.io/nvidia/tensorflow-pb24h1:24.03.02-tf2-py3

For information on the TensorFlow container images that are supported for deep learning VMs, see VMware Deep Learning VM Release Notes.
Required inputs To deploy a TensorFlow workload, you must set the OVF properties for the deep learning virtual machine in the following way:
  • Use one of the following properties that are specific for the TensorFlow image.
    • Cloud-init script. Encode it in base64 format. 
      #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    set_proxy "http" "https" "socks5"
    
    DEFAULT_REG_URI="nvcr.io"
    REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

    if [[ -z "$REGISTRY_URI_PATH" ]]; then
      # If REGISTRY_URI_PATH is null or empty, use the default value
      REGISTRY_URI_PATH=$DEFAULT_REG_URI
      echo "REGISTRY_URI_PATH was empty. Using default: $REGISTRY_URI_PATH"
    fi
    
    # If REGISTRY_URI_PATH contains '/', extract the URI part
    if [[ $REGISTRY_URI_PATH == *"/"* ]]; then
      REGISTRY_URI=$(echo "$REGISTRY_URI_PATH" | cut -d'/' -f1)
    else
      REGISTRY_URI=$REGISTRY_URI_PATH
    fi
  
    REGISTRY_USERNAME=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    REGISTRY_PASSWORD=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    if [[ -n "$REGISTRY_USERNAME" && -n "$REGISTRY_PASSWORD" ]]; then
      docker login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_URI
    else
      echo "Warning: the registry's username and password are invalid, Skipping Docker login."
    fi

    deploy_dcgm_exporter

    CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
    enableJupyterAuth=$(echo "${CONFIG_JSON}" | jq -r '.enable_jupyter_auth // empty')

    if [ -z "${enableJupyterAuth}" ] || [ "${enableJupyterAuth}" == true ]; then
      # Generate a random jupyter token
      TOKEN=$(python3 -c "import secrets; print(secrets.token_hex(32))")
      # Set the token to guestinfo
      vmtoolsd --cmd "info-set guestinfo.dlworkload.jupyterlab.token $TOKEN"
      echo "Info: JupyterLab notebook access token, $TOKEN"
    else
      TOKEN=""
    fi

    echo "Info: running the Tensorflow container"    
    TENSORFLOW_IMAGE="$REGISTRY_URI_PATH/nvidia/tensorflow-pb24h1"
    TENSORFLOW_VERSION="ngc_image_tag"
    docker run -d --gpus all -p 8888:8888 $TENSORFLOW_IMAGE:$TENSORFLOW_VERSION /usr/local/bin/jupyter lab --allow-root --ip=* --port=8888 --no-browser --NotebookApp.token="$TOKEN" --NotebookApp.allow_origin="*" --notebook-dir=/workspace
    
- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi

      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker

      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="3.2.5-3.1.8-ubuntu22.04"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }

      For example, for tensorflow-pb24h1:24.03.02-tf2-py3, provide the following script in base64 format:

      I2Nsb3VkLWNvbmZpZwp3cml0ZV9maWxlczoKLSBwYXRoOiAvb3B0L2Rsdm0vZGxfYXBwLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBzZXQgLWV1CiAgICBzb3VyY2UgL29wdC9kbHZtL3V0aWxzLnNoCiAgICB0cmFwICdlcnJvcl9leGl0ICJVbmV4cGVjdGVkIGVycm9yIG9jY3VycyBhdCBkbCB3b3JrbG9hZCInIEVSUgogICAgc2V0X3Byb3h5ICJodHRwIiAiaHR0cHMiICJzb2NrczUiCiAgICAKICAgIERFRkFVTFRfUkVHX1VSST0ibnZjci5pbyIKICAgIFJFR0lTVFJZX1VSSV9QQVRIPSQoZ3JlcCByZWdpc3RyeS11cmkgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQoKICAgIGlmIFtbIC16ICIkUkVHSVNUUllfVVJJX1BBVEgiIF1dOyB0aGVuCiAgICAgICMgSWYgUkVHSVNUUllfVVJJX1BBVEggaXMgbnVsbCBvciBlbXB0eSwgdXNlIHRoZSBkZWZhdWx0IHZhbHVlCiAgICAgIFJFR0lTVFJZX1VSSV9QQVRIPSRERUZBVUxUX1JFR19VUkkKICAgICAgZWNobyAiUkVHSVNUUllfVVJJX1BBVEggd2FzIGVtcHR5LiBVc2luZyBkZWZhdWx0OiAkUkVHSVNUUllfVVJJX1BBVEgiCiAgICBmaQogICAgCiAgICAjIElmIFJFR0lTVFJZX1VSSV9QQVRIIGNvbnRhaW5zICcvJywgZXh0cmFjdCB0aGUgVVJJIHBhcnQKICAgIGlmIFtbICRSRUdJU1RSWV9VUklfUEFUSCA9PSAqIi8iKiBdXTsgdGhlbgogICAgICBSRUdJU1RSWV9VUkk9JChlY2hvICIkUkVHSVNUUllfVVJJX1BBVEgiIHwgY3V0IC1kJy8nIC1mMSkKICAgIGVsc2UKICAgICAgUkVHSVNUUllfVVJJPSRSRUdJU1RSWV9VUklfUEFUSAogICAgZmkKICAKICAgIFJFR0lTVFJZX1VTRVJOQU1FPSQoZ3JlcCByZWdpc3RyeS11c2VyIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKICAgIFJFR0lTVFJZX1BBU1NXT1JEPSQoZ3JlcCByZWdpc3RyeS1wYXNzd2QgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQogICAgaWYgW1sgLW4gIiRSRUdJU1RSWV9VU0VSTkFNRSIgJiYgLW4gIiRSRUdJU1RSWV9QQVNTV09SRCIgXV07IHRoZW4KICAgICAgZG9ja2VyIGxvZ2luIC11ICRSRUdJU1RSWV9VU0VSTkFNRSAtcCAkUkVHSVNUUllfUEFTU1dPUkQgJFJFR0lTVFJZX1VSSQogICAgZWxzZQogICAgICBlY2hvICJXYXJuaW5nOiB0aGUgcmVnaXN0cnkncyB1c2VybmFtZSBhbmQgcGFzc3dvcmQgYXJlIGludmFsaWQsIFNraXBwaW5nIERvY2tlciBsb2dpbi4iCiAgICBmaQoKICAgIGRlcGxveV9kY2dtX2V4cG9ydGVyCgogICAgQ09ORklHX0pTT05fQkFTRTY0PSQoZ3JlcCAnY29uZmlnLWpzb24nIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKICAgIENPTkZJR19KU09OPSQoZWNobyAke0NPTkZJR19KU09OX0JBU0U2NH0gfCBiYXNlNjQgLS1kZWNvZGUpCiAgICBlbmFibGVKdXB5dGVyQXV0aD0kKGVjaG8gIiR7Q09ORklHX0pTT059IiB8IGpxIC1yICcuZW5hYmxlX2p1cHl0ZXJfYXV0aCAvLyBlbXB0eScpCgogICAgaWYgWyAteiAiJHtlbmFibGVKdXB5dGVyQXV0aH0iIF0gfHwgWyAiJHtlbmFibGVKdXB5dGVyQXV0aH0iID09IHRydWUgXTsgdGhlbgogICAgICAjIEdlbmVyYXRlIGEgcmFuZG9tIGp1cHl0ZXIgdG9rZW4KICAgICAgVE9LRU49JChweXRob24zIC1jICJpbXBvcnQgc2VjcmV0czsgcHJpbnQoc2VjcmV0cy50b2tlbl9oZXgoMzIpKSIpCiAgICAgICMgU2V0IHRoZSB0b2tlbiB0byBndWVzdGluZm8KICAgICAgdm10b29sc2QgLS1jbWQgImluZm8tc2V0IGd1ZXN0aW5mby5kbHdvcmtsb2FkLmp1cHl0ZXJsYWIudG9rZW4gJFRPS0VOIgogICAgICBlY2hvICJJbmZvOiBKdXB5dGVyTGFiIG5vdGVib29rIGFjY2VzcyB0b2tlbiwgJFRPS0VOIgogICAgZWxzZQogICAgICBUT0tFTj0iIgogICAgZmkKCiAgICBlY2hvICJJbmZvOiBydW5uaW5nIHRoZSBUZW5zb3JmbG93IGNvbnRhaW5lciIgICAgCiAgICBURU5TT1JGTE9XX0lNQUdFPSIkUkVHSVNUUllfVVJJX1BBVEgvbnZpZGlhL3RlbnNvcmZsb3ctcGIyNGgxIgogICAgVEVOU09SRkxPV19WRVJTSU9OPSIyNC4wMy4wMi10ZjItcHkzIgogICAgZG9ja2VyIHJ1biAtZCAtLWdwdXMgYWxsIC1wIDg4ODg6ODg4OCAkVEVOU09SRkxPV19JTUFHRTokVEVOU09SRkxPV19WRVJTSU9OIC91c3IvbG9jYWwvYmluL2p1cHl0ZXIgbGFiIC0tYWxsb3ctcm9vdCAtLWlwPSogLS1wb3J0PTg4ODggLS1uby1icm93c2VyIC0tTm90ZWJvb2tBcHAudG9rZW49IiRUT0tFTiIgLS1Ob3RlYm9va0FwcC5hbGxvd19vcmlnaW49IioiIC0tbm90ZWJvb2stZGlyPS93b3Jrc3BhY2UKICAgIAotIHBhdGg6IC9vcHQvZGx2bS91dGlscy5zaAogIHBlcm1pc3Npb25zOiAnMDc1NScKICBjb250ZW50OiB8CiAgICAjIS9iaW4vYmFzaAogICAgZXJyb3JfZXhpdCgpIHsKICAgICAgZWNobyAiRXJyb3I6ICQxIiA+JjIKICAgICAgdm10b29sc2QgLS1jbWQgImluZm8tc2V0IGd1ZXN0aW5mby52bXNlcnZpY2UuYm9vdHN0cmFwLmNvbmRpdGlvbiBmYWxzZSwgRExXb3JrbG9hZEZhaWx1cmUsICQxIgogICAgICBleGl0IDEKICAgIH0KCiAgICBjaGVja19wcm90b2NvbCgpIHsKICAgICAgbG9jYWwgcHJveHlfdXJsPSQxCiAgICAgIHNoaWZ0CiAgICAgIGxvY2FsIHN1cHBvcnRlZF9wcm90b2NvbHM9KCIkQCIpCiAgICAgIGlmIFtbIC1uICIke3Byb3h5X3VybH0iIF1dOyB0aGVuCiAgICAgICAgbG9jYWwgcHJvdG9jb2w9JChlY2hvICIke3Byb3h5X3VybH0iIHwgYXdrIC1GICc6Ly8nICd7aWYgKE5GID4gMSkgcHJpbnQgJDE7IGVsc2UgcHJpbnQgIiJ9JykKICAgICAgICBpZiBbIC16ICIkcHJvdG9jb2wiIF07IHRoZW4KICAgICAgICAgIGVjaG8gIk5vIHNwZWNpZmljIHByb3RvY29sIHByb3ZpZGVkLiBTa2lwcGluZyBwcm90b2NvbCBjaGVjay4iCiAgICAgICAgICByZXR1cm4gMAogICAgICAgIGZpCiAgICAgICAgbG9jYWwgcHJvdG9jb2xfaW5jbHVkZWQ9ZmFsc2UKICAgICAgICBmb3IgdmFyIGluICIke3N1cHBvcnRlZF9wcm90b2NvbHNbQF19IjsgZG8KICAgICAgICAgIGlmIFtbICIke3Byb3RvY29sfSIgPT0gIiR7dmFyfSIgXV07IHRoZW4KICAgICAgICAgICAgcHJvdG9jb2xfaW5jbHVkZWQ9dHJ1ZQogICAgICAgICAgICBicmVhawogICAgICAgICAgZmkKICAgICAgICBkb25lCiAgICAgICAgaWYgW1sgIiR7cHJvdG9jb2xfaW5jbHVkZWR9IiA9PSBmYWxzZSBdXTsgdGhlbgogICAgICAgICAgZXJyb3JfZXhpdCAiVW5zdXBwb3J0ZWQgcHJvdG9jb2w6ICR7cHJvdG9jb2x9LiBTdXBwb3J0ZWQgcHJvdG9jb2xzIGFyZTogJHtzdXBwb3J0ZWRfcHJvdG9jb2xzWypdfSIKICAgICAgICBmaQogICAgICBmaQogICAgfQoKICAgICMgJEA6IGxpc3Qgb2Ygc3VwcG9ydGVkIHByb3RvY29scwogICAgc2V0X3Byb3h5KCkgewogICAgICBsb2NhbCBzdXBwb3J0ZWRfcHJvdG9jb2xzPSgiJEAiKQoKICAgICAgQ09ORklHX0pTT05fQkFTRTY0PSQoZ3JlcCAnY29uZmlnLWpzb24nIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKICAgICAgQ09ORklHX0pTT049JChlY2hvICR7Q09ORklHX0pTT05fQkFTRTY0fSB8IGJhc2U2NCAtLWRlY29kZSkKCiAgICAgIEhUVFBfUFJPWFlfVVJMPSQoZWNobyAiJHtDT05GSUdfSlNPTn0iIHwganEgLXIgJy5odHRwX3Byb3h5IC8vIGVtcHR5JykKICAgICAgSFRUUFNfUFJPWFlfVVJMPSQoZWNobyAiJHtDT05GSUdfSlNPTn0iIHwganEgLXIgJy5odHRwc19wcm94eSAvLyBlbXB0eScpCiAgICAgIGlmIFtbICQ/IC1uZSAwIHx8ICgteiAiJHtIVFRQX1BST1hZX1VSTH0iICYmIC16ICIke0hUVFBTX1BST1hZX1VSTH0iKSBdXTsgdGhlbgogICAgICAgIGVjaG8gIkluZm86IFRoZSBjb25maWctanNvbiB3YXMgcGFyc2VkLCBidXQgbm8gcHJveHkgc2V0dGluZ3Mgd2VyZSBmb3VuZC4iCiAgICAgICAgcmV0dXJuIDAKICAgICAgZmkKCiAgICAgIGNoZWNrX3Byb3RvY29sICIke0hUVFBfUFJPWFlfVVJMfSIgIiR7c3VwcG9ydGVkX3Byb3RvY29sc1tAXX0iCiAgICAgIGNoZWNrX3Byb3RvY29sICIke0hUVFBTX1BST1hZX1VSTH0iICIke3N1cHBvcnRlZF9wcm90b2NvbHNbQF19IgoKICAgICAgaWYgISBncmVwIC1xICdodHRwX3Byb3h5JyAvZXRjL2Vudmlyb25tZW50OyB0aGVuCiAgICAgICAgc3VkbyBiYXNoIC1jICdlY2hvICJleHBvcnQgaHR0cF9wcm94eT0ke0hUVFBfUFJPWFlfVVJMfQogICAgICAgIGV4cG9ydCBodHRwc19wcm94eT0ke0hUVFBTX1BST1hZX1VSTH0KICAgICAgICBleHBvcnQgSFRUUF9QUk9YWT0ke0hUVFBfUFJPWFlfVVJMfQogICAgICAgIGV4cG9ydCBIVFRQU19QUk9YWT0ke0hUVFBTX1BST1hZX1VSTH0KICAgICAgICBleHBvcnQgbm9fcHJveHk9bG9jYWxob3N0LDEyNy4wLjAuMSIgPj4gL2V0Yy9lbnZpcm9ubWVudCcKICAgICAgICBzb3VyY2UgL2V0Yy9lbnZpcm9ubWVudAogICAgICBmaQogICAgICAKICAgICAgIyBDb25maWd1cmUgRG9ja2VyIHRvIHVzZSBhIHByb3h5CiAgICAgIHN1ZG8gbWtkaXIgLXAgL2V0Yy9zeXN0ZW1kL3N5c3RlbS9kb2NrZXIuc2VydmljZS5kCiAgICAgIHN1ZG8gYmFzaCAtYyAnZWNobyAiW1NlcnZpY2VdCiAgICAgIEVudmlyb25tZW50PVwiSFRUUF9QUk9YWT0ke0hUVFBfUFJPWFlfVVJMfVwiCiAgICAgIEVudmlyb25tZW50PVwiSFRUUFNfUFJPWFk9JHtIVFRQU19QUk9YWV9VUkx9XCIKICAgICAgRW52aXJvbm1lbnQ9XCJOT19QUk9YWT1sb2NhbGhvc3QsMTI3LjAuMC4xXCIiID4gL2V0Yy9zeXN0ZW1kL3N5c3RlbS9kb2NrZXIuc2VydmljZS5kL3Byb3h5LmNvbmYnCiAgICAgIHN1ZG8gc3lzdGVtY3RsIGRhZW1vbi1yZWxvYWQKICAgICAgc3VkbyBzeXN0ZW1jdGwgcmVzdGFydCBkb2NrZXIKCiAgICAgIGVjaG8gIkluZm86IGRvY2tlciBhbmQgc3lzdGVtIGVudmlyb25tZW50IGFyZSBub3cgY29uZmlndXJlZCB0byB1c2UgdGhlIHByb3h5IHNldHRpbmdzIgogICAgfQoKICAgIGRlcGxveV9kY2dtX2V4cG9ydGVyKCkgewogICAgICBDT05GSUdfSlNPTl9CQVNFNjQ9JChncmVwICdjb25maWctanNvbicgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQogICAgICBDT05GSUdfSlNPTj0kKGVjaG8gJHtDT05GSUdfSlNPTl9CQVNFNjR9IHwgYmFzZTY0IC0tZGVjb2RlKQogICAgICBEQ0dNX0VYUE9SVF9QVUJMSUM9JChlY2hvICIke0NPTkZJR19KU09OfSIgfCBqcSAtciAnLmV4cG9ydF9kY2dtX3RvX3B1YmxpYyAvLyBlbXB0eScpCgogICAgICBEQ0dNX0VYUE9SVEVSX0lNQUdFPSIkUkVHSVNUUllfVVJJX1BBVEgvbnZpZGlhL2s4cy9kY2dtLWV4cG9ydGVyIgogICAgICBEQ0dNX0VYUE9SVEVSX1ZFUlNJT049IjMuMi41LTMuMS44LXVidW50dTIyLjA0IgogICAgICBpZiBbIC16ICIke0RDR01fRVhQT1JUX1BVQkxJQ30iIF0gfHwgWyAiJHtEQ0dNX0VYUE9SVF9QVUJMSUN9IiAhPSAidHJ1ZSIgXTsgdGhlbgogICAgICAgIGVjaG8gIkluZm86IGxhdW5jaGluZyBEQ0dNIEV4cG9ydGVyIHRvIGNvbGxlY3QgdkdQVSBtZXRyaWNzLCBsaXN0ZW5pbmcgb25seSBvbiBsb2NhbGhvc3QgKDEyNy4wLjAuMTo5NDAwKSIKICAgICAgICBkb2NrZXIgcnVuIC1kIC0tZ3B1cyBhbGwgLS1jYXAtYWRkIFNZU19BRE1JTiAtcCAxMjcuMC4wLjE6OTQwMDo5NDAwICREQ0dNX0VYUE9SVEVSX0lNQUdFOiREQ0dNX0VYUE9SVEVSX1ZFUlNJT04KICAgICAgZWxzZQogICAgICAgIGVjaG8gIkluZm86IGxhdW5jaGluZyBEQ0dNIEV4cG9ydGVyIHRvIGNvbGxlY3QgdkdQVSBtZXRyaWNzLCBleHBvc2VkIG9uIGFsbCBuZXR3b3JrIGludGVyZmFjZXMgKDAuMC4wLjA6OTQwMCkiCiAgICAgICAgZG9ja2VyIHJ1biAtZCAtLWdwdXMgYWxsIC0tY2FwLWFkZCBTWVNfQURNSU4gLXAgOTQwMDo5NDAwICREQ0dNX0VYUE9SVEVSX0lNQUdFOiREQ0dNX0VYUE9SVEVSX1ZFUlNJT04KICAgICAgZmkKICAgIH0=

      which corresponds to the following script in plain-text format:

      #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    set_proxy "http" "https" "socks5"
    
    DEFAULT_REG_URI="nvcr.io"
    REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

    if [[ -z "$REGISTRY_URI_PATH" ]]; then
      # If REGISTRY_URI_PATH is null or empty, use the default value
      REGISTRY_URI_PATH=$DEFAULT_REG_URI
      echo "REGISTRY_URI_PATH was empty. Using default: $REGISTRY_URI_PATH"
    fi
    
    # If REGISTRY_URI_PATH contains '/', extract the URI part
    if [[ $REGISTRY_URI_PATH == *"/"* ]]; then
      REGISTRY_URI=$(echo "$REGISTRY_URI_PATH" | cut -d'/' -f1)
    else
      REGISTRY_URI=$REGISTRY_URI_PATH
    fi
  
    REGISTRY_USERNAME=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    REGISTRY_PASSWORD=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    if [[ -n "$REGISTRY_USERNAME" && -n "$REGISTRY_PASSWORD" ]]; then
      docker login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_URI
    else
      echo "Warning: the registry's username and password are invalid, Skipping Docker login."
    fi

    deploy_dcgm_exporter

    CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
    enableJupyterAuth=$(echo "${CONFIG_JSON}" | jq -r '.enable_jupyter_auth // empty')

    if [ -z "${enableJupyterAuth}" ] || [ "${enableJupyterAuth}" == true ]; then
      # Generate a random jupyter token
      TOKEN=$(python3 -c "import secrets; print(secrets.token_hex(32))")
      # Set the token to guestinfo
      vmtoolsd --cmd "info-set guestinfo.dlworkload.jupyterlab.token $TOKEN"
      echo "Info: JupyterLab notebook access token, $TOKEN"
    else
      TOKEN=""
    fi

    echo "Info: running the Tensorflow container"    
    TENSORFLOW_IMAGE="$REGISTRY_URI_PATH/nvidia/tensorflow-pb24h1"
    TENSORFLOW_VERSION="24.03.02-tf2-py3"
    docker run -d --gpus all -p 8888:8888 $TENSORFLOW_IMAGE:$TENSORFLOW_VERSION /usr/local/bin/jupyter lab --allow-root --ip=* --port=8888 --no-browser --NotebookApp.token="$TOKEN" --NotebookApp.allow_origin="*" --notebook-dir=/workspace
    
- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi

      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker

      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="3.2.5-3.1.8-ubuntu22.04"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }
    • Image one-liner. Encode it in base64 format.
      docker run -d -p 8888:8888 nvcr.io/nvidia/tensorflow-pb24h1:ngc_image_tag /usr/local/bin/jupyter lab --allow-root --ip=* --port=8888 --no-browser --NotebookApp.token='' --NotebookApp.allow_origin='*' --notebook-dir=/workspace

      For example, for tensorflow-pb24h1:24.03.02, provide the following script in base64 format:

      ZG9ja2VyIHJ1biAtZCAtcCA4ODg4Ojg4ODggbnZjci5pby9udmlkaWEvdGVuc29yZmxvdy1wYjI0aDE6MjQuMDMuMDItdGYyLXB5MyAvdXNyL2xvY2FsL2Jpbi9qdXB5dGVyIGxhYiAtLWFsbG93LXJvb3QgLS1pcD0qIC0tcG9ydD04ODg4IC0tbm8tYnJvd3NlciAtLU5vdGVib29rQXBwLnRva2VuPScnIC0tTm90ZWJvb2tBcHAuYWxsb3dfb3JpZ2luPScqJyAtLW5vdGVib29rLWRpcj0vd29ya3NwYWNl

      which corresponds to the following script in plain-text format:

      docker run -d -p 8888:8888 nvcr.io/nvidia/tensorflow-pb24h1:24.03.02-tf2-py3 /usr/local/bin/jupyter lab --allow-root --ip=* --port=8888 --no-browser --NotebookApp.token='' --NotebookApp.allow_origin='*' --notebook-dir=/workspace
  • Enter the vGPU guest driver installation properties, such as vgpu-license and nvidia-portal-api-key.
  • Provide values for the properties required for a disconnected environment as needed.

See OVF Properties of Deep Learning VMs.
Output
  • Installation logs for the vGPU guest driver in /var/log/vgpu-install.log.

    To verify that the vGPU guest driver is installed, log in to the VM over SSH and run the nvidia-smi command.

  • Cloud-init script logs in /var/log/dl.log.
  • TensorFlow container.

    To verify that the TensorFlow container is running, run the sudo docker ps -a and sudo docker logs container_id commands.

  • JupyterLab instance that you can access at http://dl_vm_ip:8888.

    In the terminal of JupyterLab, verify that the following functionality is available in the notebook:

    • To verify that JupyterLab can access the vGPU resource, run nvidia-smi.
    • To verify that the TensorFlow related packages are installed, run pip show.

DCGM Exporter

You can use a deep learning VM with a Data Center GPU Manager (DCGM) exporter to monitor the health of and get metrics from GPUs used by a DL workload, using NVIDIA DCGM, Prometheus, and Grafana.

See the DCGM Exporter page.

In a deep learning VM, you run the DCGM Exporter container together with a DL workload that performs AI operations. After the deep learning VM is started, DCGM Exporter is ready to collect vGPU metrics and export the data to another application for further monitoring and visualization. You can run the monitored DL workload as a part of the cloud-init process or from the command line after the virtual machine is started.

Table 4. DCGM Exporter Container Image
Component Description
Container image 
nvcr.io/nvidia/k8s/dcgm-exporter:ngc_image_tag

For example:

nvcr.io/nvidia/k8s/dcgm-exporter:3.2.5-3.1.8-ubuntu22.04

For information on the DCGM Exporter container images that are supported for deep learning VMs, see VMware Deep Learning VM Release Notes.
Required inputs To deploy a DCGM Exporter workload, you must set the OVF properties for the deep learning virtual machine in the following way:
  • Use one of the following properties that are specific for the DCGM Exporter image.
    • Cloud-init script. Encode it in base64 format.
      #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    set_proxy "http" "https" "socks5"
    
    DEFAULT_REG_URI="nvcr.io"
    REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

    if [[ -z "$REGISTRY_URI_PATH" ]]; then
      # If REGISTRY_URI_PATH is null or empty, use the default value
      REGISTRY_URI_PATH=$DEFAULT_REG_URI
      echo "REGISTRY_URI_PATH was empty. Using default: $REGISTRY_URI_PATH"
    fi
    
    # If REGISTRY_URI_PATH contains '/', extract the URI part
    if [[ $REGISTRY_URI_PATH == *"/"* ]]; then
      REGISTRY_URI=$(echo "$REGISTRY_URI_PATH" | cut -d'/' -f1)
    else
      REGISTRY_URI=$REGISTRY_URI_PATH
    fi
  
    REGISTRY_USERNAME=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    REGISTRY_PASSWORD=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    if [[ -n "$REGISTRY_USERNAME" && -n "$REGISTRY_PASSWORD" ]]; then
      docker login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_URI
    else
      echo "Warning: the registry's username and password are invalid, Skipping Docker login."
    fi

    echo "Info: running the DCGM Export container"
    deploy_dcgm_exporter

- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi

      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker


      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="ngc_image_tag"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }

      For example, for a deep learning VM with a pre-installed a dcgm-exporter:3.2.5-3.1.8-ubuntu22.04 DCGM Exporter instance, provide the following script in base64 format

      I2Nsb3VkLWNvbmZpZwp3cml0ZV9maWxlczoKLSBwYXRoOiAvb3B0L2Rsdm0vZGxfYXBwLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBzZXQgLWV1CiAgICBzb3VyY2UgL29wdC9kbHZtL3V0aWxzLnNoCiAgICB0cmFwICdlcnJvcl9leGl0ICJVbmV4cGVjdGVkIGVycm9yIG9jY3VycyBhdCBkbCB3b3JrbG9hZCInIEVSUgogICAgc2V0X3Byb3h5ICJodHRwIiAiaHR0cHMiICJzb2NrczUiCiAgICAKICAgIERFRkFVTFRfUkVHX1VSST0ibnZjci5pbyIKICAgIFJFR0lTVFJZX1VSSV9QQVRIPSQoZ3JlcCByZWdpc3RyeS11cmkgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQoKICAgIGlmIFtbIC16ICIkUkVHSVNUUllfVVJJX1BBVEgiIF1dOyB0aGVuCiAgICAgICMgSWYgUkVHSVNUUllfVVJJX1BBVEggaXMgbnVsbCBvciBlbXB0eSwgdXNlIHRoZSBkZWZhdWx0IHZhbHVlCiAgICAgIFJFR0lTVFJZX1VSSV9QQVRIPSRERUZBVUxUX1JFR19VUkkKICAgICAgZWNobyAiUkVHSVNUUllfVVJJX1BBVEggd2FzIGVtcHR5LiBVc2luZyBkZWZhdWx0OiAkUkVHSVNUUllfVVJJX1BBVEgiCiAgICBmaQogICAgCiAgICAjIElmIFJFR0lTVFJZX1VSSV9QQVRIIGNvbnRhaW5zICcvJywgZXh0cmFjdCB0aGUgVVJJIHBhcnQKICAgIGlmIFtbICRSRUdJU1RSWV9VUklfUEFUSCA9PSAqIi8iKiBdXTsgdGhlbgogICAgICBSRUdJU1RSWV9VUkk9JChlY2hvICIkUkVHSVNUUllfVVJJX1BBVEgiIHwgY3V0IC1kJy8nIC1mMSkKICAgIGVsc2UKICAgICAgUkVHSVNUUllfVVJJPSRSRUdJU1RSWV9VUklfUEFUSAogICAgZmkKICAKICAgIFJFR0lTVFJZX1VTRVJOQU1FPSQoZ3JlcCByZWdpc3RyeS11c2VyIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKICAgIFJFR0lTVFJZX1BBU1NXT1JEPSQoZ3JlcCByZWdpc3RyeS1wYXNzd2QgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQogICAgaWYgW1sgLW4gIiRSRUdJU1RSWV9VU0VSTkFNRSIgJiYgLW4gIiRSRUdJU1RSWV9QQVNTV09SRCIgXV07IHRoZW4KICAgICAgZG9ja2VyIGxvZ2luIC11ICRSRUdJU1RSWV9VU0VSTkFNRSAtcCAkUkVHSVNUUllfUEFTU1dPUkQgJFJFR0lTVFJZX1VSSQogICAgZWxzZQogICAgICBlY2hvICJXYXJuaW5nOiB0aGUgcmVnaXN0cnkncyB1c2VybmFtZSBhbmQgcGFzc3dvcmQgYXJlIGludmFsaWQsIFNraXBwaW5nIERvY2tlciBsb2dpbi4iCiAgICBmaQoKICAgIGVjaG8gIkluZm86IHJ1bm5pbmcgdGhlIERDR00gRXhwb3J0IGNvbnRhaW5lciIKICAgIGRlcGxveV9kY2dtX2V4cG9ydGVyCgotIHBhdGg6IC9vcHQvZGx2bS91dGlscy5zaAogIHBlcm1pc3Npb25zOiAnMDc1NScKICBjb250ZW50OiB8CiAgICAjIS9iaW4vYmFzaAogICAgZXJyb3JfZXhpdCgpIHsKICAgICAgZWNobyAiRXJyb3I6ICQxIiA+JjIKICAgICAgdm10b29sc2QgLS1jbWQgImluZm8tc2V0IGd1ZXN0aW5mby52bXNlcnZpY2UuYm9vdHN0cmFwLmNvbmRpdGlvbiBmYWxzZSwgRExXb3JrbG9hZEZhaWx1cmUsICQxIgogICAgICBleGl0IDEKICAgIH0KCiAgICBjaGVja19wcm90b2NvbCgpIHsKICAgICAgbG9jYWwgcHJveHlfdXJsPSQxCiAgICAgIHNoaWZ0CiAgICAgIGxvY2FsIHN1cHBvcnRlZF9wcm90b2NvbHM9KCIkQCIpCiAgICAgIGlmIFtbIC1uICIke3Byb3h5X3VybH0iIF1dOyB0aGVuCiAgICAgICAgbG9jYWwgcHJvdG9jb2w9JChlY2hvICIke3Byb3h5X3VybH0iIHwgYXdrIC1GICc6Ly8nICd7aWYgKE5GID4gMSkgcHJpbnQgJDE7IGVsc2UgcHJpbnQgIiJ9JykKICAgICAgICBpZiBbIC16ICIkcHJvdG9jb2wiIF07IHRoZW4KICAgICAgICAgIGVjaG8gIk5vIHNwZWNpZmljIHByb3RvY29sIHByb3ZpZGVkLiBTa2lwcGluZyBwcm90b2NvbCBjaGVjay4iCiAgICAgICAgICByZXR1cm4gMAogICAgICAgIGZpCiAgICAgICAgbG9jYWwgcHJvdG9jb2xfaW5jbHVkZWQ9ZmFsc2UKICAgICAgICBmb3IgdmFyIGluICIke3N1cHBvcnRlZF9wcm90b2NvbHNbQF19IjsgZG8KICAgICAgICAgIGlmIFtbICIke3Byb3RvY29sfSIgPT0gIiR7dmFyfSIgXV07IHRoZW4KICAgICAgICAgICAgcHJvdG9jb2xfaW5jbHVkZWQ9dHJ1ZQogICAgICAgICAgICBicmVhawogICAgICAgICAgZmkKICAgICAgICBkb25lCiAgICAgICAgaWYgW1sgIiR7cHJvdG9jb2xfaW5jbHVkZWR9IiA9PSBmYWxzZSBdXTsgdGhlbgogICAgICAgICAgZXJyb3JfZXhpdCAiVW5zdXBwb3J0ZWQgcHJvdG9jb2w6ICR7cHJvdG9jb2x9LiBTdXBwb3J0ZWQgcHJvdG9jb2xzIGFyZTogJHtzdXBwb3J0ZWRfcHJvdG9jb2xzWypdfSIKICAgICAgICBmaQogICAgICBmaQogICAgfQoKICAgICMgJEA6IGxpc3Qgb2Ygc3VwcG9ydGVkIHByb3RvY29scwogICAgc2V0X3Byb3h5KCkgewogICAgICBsb2NhbCBzdXBwb3J0ZWRfcHJvdG9jb2xzPSgiJEAiKQoKICAgICAgQ09ORklHX0pTT05fQkFTRTY0PSQoZ3JlcCAnY29uZmlnLWpzb24nIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKICAgICAgQ09ORklHX0pTT049JChlY2hvICR7Q09ORklHX0pTT05fQkFTRTY0fSB8IGJhc2U2NCAtLWRlY29kZSkKCiAgICAgIEhUVFBfUFJPWFlfVVJMPSQoZWNobyAiJHtDT05GSUdfSlNPTn0iIHwganEgLXIgJy5odHRwX3Byb3h5IC8vIGVtcHR5JykKICAgICAgSFRUUFNfUFJPWFlfVVJMPSQoZWNobyAiJHtDT05GSUdfSlNPTn0iIHwganEgLXIgJy5odHRwc19wcm94eSAvLyBlbXB0eScpCiAgICAgIGlmIFtbICQ/IC1uZSAwIHx8ICgteiAiJHtIVFRQX1BST1hZX1VSTH0iICYmIC16ICIke0hUVFBTX1BST1hZX1VSTH0iKSBdXTsgdGhlbgogICAgICAgIGVjaG8gIkluZm86IFRoZSBjb25maWctanNvbiB3YXMgcGFyc2VkLCBidXQgbm8gcHJveHkgc2V0dGluZ3Mgd2VyZSBmb3VuZC4iCiAgICAgICAgcmV0dXJuIDAKICAgICAgZmkKCiAgICAgIGNoZWNrX3Byb3RvY29sICIke0hUVFBfUFJPWFlfVVJMfSIgIiR7c3VwcG9ydGVkX3Byb3RvY29sc1tAXX0iCiAgICAgIGNoZWNrX3Byb3RvY29sICIke0hUVFBTX1BST1hZX1VSTH0iICIke3N1cHBvcnRlZF9wcm90b2NvbHNbQF19IgoKICAgICAgaWYgISBncmVwIC1xICdodHRwX3Byb3h5JyAvZXRjL2Vudmlyb25tZW50OyB0aGVuCiAgICAgICAgc3VkbyBiYXNoIC1jICdlY2hvICJleHBvcnQgaHR0cF9wcm94eT0ke0hUVFBfUFJPWFlfVVJMfQogICAgICAgIGV4cG9ydCBodHRwc19wcm94eT0ke0hUVFBTX1BST1hZX1VSTH0KICAgICAgICBleHBvcnQgSFRUUF9QUk9YWT0ke0hUVFBfUFJPWFlfVVJMfQogICAgICAgIGV4cG9ydCBIVFRQU19QUk9YWT0ke0hUVFBTX1BST1hZX1VSTH0KICAgICAgICBleHBvcnQgbm9fcHJveHk9bG9jYWxob3N0LDEyNy4wLjAuMSIgPj4gL2V0Yy9lbnZpcm9ubWVudCcKICAgICAgICBzb3VyY2UgL2V0Yy9lbnZpcm9ubWVudAogICAgICBmaQogICAgICAKICAgICAgIyBDb25maWd1cmUgRG9ja2VyIHRvIHVzZSBhIHByb3h5CiAgICAgIHN1ZG8gbWtkaXIgLXAgL2V0Yy9zeXN0ZW1kL3N5c3RlbS9kb2NrZXIuc2VydmljZS5kCiAgICAgIHN1ZG8gYmFzaCAtYyAnZWNobyAiW1NlcnZpY2VdCiAgICAgIEVudmlyb25tZW50PVwiSFRUUF9QUk9YWT0ke0hUVFBfUFJPWFlfVVJMfVwiCiAgICAgIEVudmlyb25tZW50PVwiSFRUUFNfUFJPWFk9JHtIVFRQU19QUk9YWV9VUkx9XCIKICAgICAgRW52aXJvbm1lbnQ9XCJOT19QUk9YWT1sb2NhbGhvc3QsMTI3LjAuMC4xXCIiID4gL2V0Yy9zeXN0ZW1kL3N5c3RlbS9kb2NrZXIuc2VydmljZS5kL3Byb3h5LmNvbmYnCiAgICAgIHN1ZG8gc3lzdGVtY3RsIGRhZW1vbi1yZWxvYWQKICAgICAgc3VkbyBzeXN0ZW1jdGwgcmVzdGFydCBkb2NrZXIKCgogICAgICBlY2hvICJJbmZvOiBkb2NrZXIgYW5kIHN5c3RlbSBlbnZpcm9ubWVudCBhcmUgbm93IGNvbmZpZ3VyZWQgdG8gdXNlIHRoZSBwcm94eSBzZXR0aW5ncyIKICAgIH0KCiAgICBkZXBsb3lfZGNnbV9leHBvcnRlcigpIHsKICAgICAgQ09ORklHX0pTT05fQkFTRTY0PSQoZ3JlcCAnY29uZmlnLWpzb24nIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKICAgICAgQ09ORklHX0pTT049JChlY2hvICR7Q09ORklHX0pTT05fQkFTRTY0fSB8IGJhc2U2NCAtLWRlY29kZSkKICAgICAgRENHTV9FWFBPUlRfUFVCTElDPSQoZWNobyAiJHtDT05GSUdfSlNPTn0iIHwganEgLXIgJy5leHBvcnRfZGNnbV90b19wdWJsaWMgLy8gZW1wdHknKQoKICAgICAgRENHTV9FWFBPUlRFUl9JTUFHRT0iJFJFR0lTVFJZX1VSSV9QQVRIL252aWRpYS9rOHMvZGNnbS1leHBvcnRlciIKICAgICAgRENHTV9FWFBPUlRFUl9WRVJTSU9OPSIzLjIuNS0zLjEuOC11YnVudHUyMi4wNCIKICAgICAgaWYgWyAteiAiJHtEQ0dNX0VYUE9SVF9QVUJMSUN9IiBdIHx8IFsgIiR7RENHTV9FWFBPUlRfUFVCTElDfSIgIT0gInRydWUiIF07IHRoZW4KICAgICAgICBlY2hvICJJbmZvOiBsYXVuY2hpbmcgRENHTSBFeHBvcnRlciB0byBjb2xsZWN0IHZHUFUgbWV0cmljcywgbGlzdGVuaW5nIG9ubHkgb24gbG9jYWxob3N0ICgxMjcuMC4wLjE6OTQwMCkiCiAgICAgICAgZG9ja2VyIHJ1biAtZCAtLWdwdXMgYWxsIC0tY2FwLWFkZCBTWVNfQURNSU4gLXAgMTI3LjAuMC4xOjk0MDA6OTQwMCAkRENHTV9FWFBPUlRFUl9JTUFHRTokRENHTV9FWFBPUlRFUl9WRVJTSU9OCiAgICAgIGVsc2UKICAgICAgICBlY2hvICJJbmZvOiBsYXVuY2hpbmcgRENHTSBFeHBvcnRlciB0byBjb2xsZWN0IHZHUFUgbWV0cmljcywgZXhwb3NlZCBvbiBhbGwgbmV0d29yayBpbnRlcmZhY2VzICgwLjAuMC4wOjk0MDApIgogICAgICAgIGRvY2tlciBydW4gLWQgLS1ncHVzIGFsbCAtLWNhcC1hZGQgU1lTX0FETUlOIC1wIDk0MDA6OTQwMCAkRENHTV9FWFBPUlRFUl9JTUFHRTokRENHTV9FWFBPUlRFUl9WRVJTSU9OCiAgICAgIGZpCiAgICB9
      which corresponds to the following script in plain-text format: 
      #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    set_proxy "http" "https" "socks5"
    
    DEFAULT_REG_URI="nvcr.io"
    REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

    if [[ -z "$REGISTRY_URI_PATH" ]]; then
      # If REGISTRY_URI_PATH is null or empty, use the default value
      REGISTRY_URI_PATH=$DEFAULT_REG_URI
      echo "REGISTRY_URI_PATH was empty. Using default: $REGISTRY_URI_PATH"
    fi
    
    # If REGISTRY_URI_PATH contains '/', extract the URI part
    if [[ $REGISTRY_URI_PATH == *"/"* ]]; then
      REGISTRY_URI=$(echo "$REGISTRY_URI_PATH" | cut -d'/' -f1)
    else
      REGISTRY_URI=$REGISTRY_URI_PATH
    fi
  
    REGISTRY_USERNAME=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    REGISTRY_PASSWORD=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    if [[ -n "$REGISTRY_USERNAME" && -n "$REGISTRY_PASSWORD" ]]; then
      docker login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_URI
    else
      echo "Warning: the registry's username and password are invalid, Skipping Docker login."
    fi

    echo "Info: running the DCGM Export container"
    deploy_dcgm_exporter

- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi

      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker


      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="3.2.5-3.1.8-ubuntu22.04"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }
      Note: You can also add the instructions for running the DL workload whose GPU performance you want to measure with DCGM Exporter to the cloud-init script.
    • Image one-liner. Encode it in base64 format. 
      docker run -d --gpus all --cap-add SYS_ADMIN --rm -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:ngc_image_tag-ubuntu22.04

      For example, for dcgm-exporter:3.2.5-3.1.8-ubuntu22.04, provide the following script in base64 format:

      ZG9ja2VyIHJ1biAtZCAtLWdwdXMgYWxsIC0tY2FwLWFkZCBTWVNfQURNSU4gLS1ybSAtcCA5NDAwOjk0MDAgbnZjci5pby9udmlkaWEvazhzL2RjZ20tZXhwb3J0ZXI6My4yLjUtMy4xLjgtdWJ1bnR1MjIuMDQ=

      which corresponds to the following script in plain-text format:

      docker run -d --gpus all --cap-add SYS_ADMIN --rm -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:3.2.5-3.1.8-ubuntu22.04
  • Enter the vGPU guest driver installation properties, such as vgpu-license and nvidia-portal-api-key.
  • Provide values for the properties required for a disconnected environment as needed.

See OVF Properties of Deep Learning VMs.
Output
  • Installation logs for the vGPU guest driver in /var/log/vgpu-install.log.

    To verify that the vGPU guest driver is installed, log in to the VM over SSH and run the nvidia-smi command.

  • Cloud-init script logs in /var/log/dl.log.
  • DCGM Exporter that you can access at http://dl_vm_ip:9400.

Next, in the deep learning VM, you run a DL workload, and visualize the data on another virtual machine by using Prometheus at http://visualization_vm_ip:9090 and Grafana at http://visualization_vm_ip:3000.

Run a DL Workload on the Deep Leaning VM

Run the DL workload you want to collect vGPU metrics for and export the data to another application for further monitoring and visualization.

  1. Log in to the deep learning VM as vmware over SSH.
  2. Run the container for the DL workload, pulling it from the NVIDIA NGC catalog or from a local container registry.

    For example, to run the following command to run the tensorflow-pb24h1:24.03.02-tf2-py3 image from NVIDIA NGC:

    docker run -d --gpus all -p 8888:8888 nvcr.io/nvidia/tensorflow-pb24h1:24.03.02-tf2-py3 /usr/local/bin/jupyter lab --allow-root --ip=* --port=8888 --no-browser --NotebookApp.token="$TOKEN" --NotebookApp.allow_origin="*" --notebook-dir=/workspace
  3. Start using the DL workload for AI development.

Install Prometheus and Grafana

You can visualize and monitor the vGPU metrics from the DCGM Exporter virtual machine on a virtual machine running Prometheus and Grafana.

  1. Create a visualization VM with Docker Community Engine installed.
  2. Connect to the VM over SSH and create a YAML file for Prometheus.
    $ cat > prometheus.yml << EOF
global:
  scrape_interval: 15s
  external_labels:
    monitor: 'codelab-monitor'
scrape_configs:
  - job_name: 'dcgm'
    scrape_interval: 5s
    metrics_path: /metrics
    static_configs:
      - targets: [dl_vm_with_dcgm_exporter_ip:9400']
EOF
  3. Create a data path.
    $ mkdir grafana_data prometheus_data && chmod 777 grafana_data prometheus_data
  4. Create a Docker compose file to install Prometheus and Grafana.
    $ cat > compose.yaml << EOF
services:
  prometheus:
    image: prom/prometheus:v2.47.2
    container_name: "prometheus0"
    restart: always
    ports:
      - "9090:9090"
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
      - "./prometheus_data:/prometheus"
  grafana:
    image: grafana/grafana:10.2.0-ubuntu
    container_name: "grafana0"
    ports:
      - "3000:3000"
    restart: always
    volumes:
      - "./grafana_data:/var/lib/grafana"
EOF
  5. Start the Prometheus and Grafana containers.
    $ sudo docker compose up -d

View vGPU Metrics in Prometheus

You can access Prometheus at http://visualization-vm-ip:9090. You can view the following vGPU information in the Prometheus UI:

Information UI Section
Raw vGPU metrics from the deep learning VM Status > Target

To view the raw vGPU metrics from the deep learning VM, click the endpoint entry.
Graph expressions
  1. On the main navigation bar, click the Graph tab.
  2. Enter an expression and click Execute

For more information on using Prometheus, see the Prometheus documentation.

Visualize Metrics in Grafana

Set Prometheus as a data source for Grafana and visualize the vGPU metrics from the deep learning VM in a dashboard.

  1. Access Grafana at http://visualization-vm-ip:3000 by using the default user name admin and password admin.
  2. Add Prometheus as the first data source, connecting to visualization-vm-ip on port 9090.
  3. Create a dashboard with the vGPU metrics.

For more information on configuring a dashboard using a Prometheus data source, see the Grafana documentation.

Triton Inference Server

You can use a deep learning VM with a Triton Inference Server for loading a model repository and receive inference requests.

See the Triton Inference Server page.

Table 5. Triton Inference Server Container Image
Component Description
Container image 
nvcr.io/nvidia/tritonserver-pb24h1:ngc_image_tag

For example: 

nvcr.io/nvidia/tritonserver-pb24h1:24.03.02-py3

For information on the Triton Inference Server container images that are supported for deep learning VMs, see VMware Deep Learning VM Release Notes.
Required inputs To deploy a Triton Inference Server workload, you must set the OVF properties for the deep learning virtual machine in the following way:
  • Use one of the following properties that are specific for the Triton Inference Server image.
    • Cloud-init script. Encode it in base64 format.
      #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    set_proxy "http" "https" "socks5"

    DEFAULT_REG_URI="nvcr.io"
    REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

    if [[ -z "$REGISTRY_URI_PATH" ]]; then
      # If REGISTRY_URI_PATH is null or empty, use the default value
      REGISTRY_URI_PATH=$DEFAULT_REG_URI
      echo "REGISTRY_URI_PATH was empty. Using default: $REGISTRY_URI_PATH"
    fi
    
    # If REGISTRY_URI_PATH contains '/', extract the URI part
    if [[ $REGISTRY_URI_PATH == *"/"* ]]; then
      REGISTRY_URI=$(echo "$REGISTRY_URI_PATH" | cut -d'/' -f1)
    else
      REGISTRY_URI=$REGISTRY_URI_PATH
    fi
  
    REGISTRY_USERNAME=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    REGISTRY_PASSWORD=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    if [[ -n "$REGISTRY_USERNAME" && -n "$REGISTRY_PASSWORD" ]]; then
      docker login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_URI
    else
      echo "Warning: the registry's username and password are invalid, Skipping Docker login."
    fi

    deploy_dcgm_exporter

    echo "Info: running the Triton Inference Server container"
    TRITON_IMAGE="$REGISTRY_URI_PATH/nvidia/tritonserver-pb24h1"
    TRITON_VERSION="24.03.02-py3"
    docker run -d --gpus all -p 8000:8000 -p 8001:8001 -p 8002:8002 -v /home/vmware/model_repository:/models $TRITON_IMAGE:$TRITON_VERSION tritonserver --model-repository=/models --model-control-mode=poll
    
- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi

      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker

      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="3.2.5-3.1.8-ubuntu22.04"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }

      For example, for tritonserver:23.10-py3, provide the following script in base64 format: 

      I2Nsb3VkLWNvbmZpZwp3cml0ZV9maWxlczoKLSBwYXRoOiAvb3B0L2Rsdm0vZGxfYXBwLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBzZXQgLWV1CiAgICBzb3VyY2UgL29wdC9kbHZtL3V0aWxzLnNoCiAgICB0cmFwICdlcnJvcl9leGl0ICJVbmV4cGVjdGVkIGVycm9yIG9jY3VycyBhdCBkbCB3b3JrbG9hZCInIEVSUgogICAgc2V0X3Byb3h5ICJodHRwIiAiaHR0cHMiICJzb2NrczUiCgogICAgREVGQVVMVF9SRUdfVVJJPSJudmNyLmlvIgogICAgUkVHSVNUUllfVVJJX1BBVEg9JChncmVwIHJlZ2lzdHJ5LXVyaSAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCgogICAgaWYgW1sgLXogIiRSRUdJU1RSWV9VUklfUEFUSCIgXV07IHRoZW4KICAgICAgIyBJZiBSRUdJU1RSWV9VUklfUEFUSCBpcyBudWxsIG9yIGVtcHR5LCB1c2UgdGhlIGRlZmF1bHQgdmFsdWUKICAgICAgUkVHSVNUUllfVVJJX1BBVEg9JERFRkFVTFRfUkVHX1VSSQogICAgICBlY2hvICJSRUdJU1RSWV9VUklfUEFUSCB3YXMgZW1wdHkuIFVzaW5nIGRlZmF1bHQ6ICRSRUdJU1RSWV9VUklfUEFUSCIKICAgIGZpCiAgICAKICAgICMgSWYgUkVHSVNUUllfVVJJX1BBVEggY29udGFpbnMgJy8nLCBleHRyYWN0IHRoZSBVUkkgcGFydAogICAgaWYgW1sgJFJFR0lTVFJZX1VSSV9QQVRIID09ICoiLyIqIF1dOyB0aGVuCiAgICAgIFJFR0lTVFJZX1VSST0kKGVjaG8gIiRSRUdJU1RSWV9VUklfUEFUSCIgfCBjdXQgLWQnLycgLWYxKQogICAgZWxzZQogICAgICBSRUdJU1RSWV9VUkk9JFJFR0lTVFJZX1VSSV9QQVRICiAgICBmaQogIAogICAgUkVHSVNUUllfVVNFUk5BTUU9JChncmVwIHJlZ2lzdHJ5LXVzZXIgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQogICAgUkVHSVNUUllfUEFTU1dPUkQ9JChncmVwIHJlZ2lzdHJ5LXBhc3N3ZCAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCiAgICBpZiBbWyAtbiAiJFJFR0lTVFJZX1VTRVJOQU1FIiAmJiAtbiAiJFJFR0lTVFJZX1BBU1NXT1JEIiBdXTsgdGhlbgogICAgICBkb2NrZXIgbG9naW4gLXUgJFJFR0lTVFJZX1VTRVJOQU1FIC1wICRSRUdJU1RSWV9QQVNTV09SRCAkUkVHSVNUUllfVVJJCiAgICBlbHNlCiAgICAgIGVjaG8gIldhcm5pbmc6IHRoZSByZWdpc3RyeSdzIHVzZXJuYW1lIGFuZCBwYXNzd29yZCBhcmUgaW52YWxpZCwgU2tpcHBpbmcgRG9ja2VyIGxvZ2luLiIKICAgIGZpCgogICAgZGVwbG95X2RjZ21fZXhwb3J0ZXIKCiAgICBlY2hvICJJbmZvOiBydW5uaW5nIHRoZSBUcml0b24gSW5mZXJlbmNlIFNlcnZlciBjb250YWluZXIiCiAgICBUUklUT05fSU1BR0U9IiRSRUdJU1RSWV9VUklfUEFUSC9udmlkaWEvdHJpdG9uc2VydmVyLXBiMjRoMSIKICAgIFRSSVRPTl9WRVJTSU9OPSIyNC4wMy4wMi1weTMiCiAgICBkb2NrZXIgcnVuIC1kIC0tZ3B1cyBhbGwgLXAgODAwMDo4MDAwIC1wIDgwMDE6ODAwMSAtcCA4MDAyOjgwMDIgLXYgL2hvbWUvdm13YXJlL21vZGVsX3JlcG9zaXRvcnk6L21vZGVscyAkVFJJVE9OX0lNQUdFOiRUUklUT05fVkVSU0lPTiB0cml0b25zZXJ2ZXIgLS1tb2RlbC1yZXBvc2l0b3J5PS9tb2RlbHMgLS1tb2RlbC1jb250cm9sLW1vZGU9cG9sbAogICAgCi0gcGF0aDogL29wdC9kbHZtL3V0aWxzLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBlcnJvcl9leGl0KCkgewogICAgICBlY2hvICJFcnJvcjogJDEiID4mMgogICAgICB2bXRvb2xzZCAtLWNtZCAiaW5mby1zZXQgZ3Vlc3RpbmZvLnZtc2VydmljZS5ib290c3RyYXAuY29uZGl0aW9uIGZhbHNlLCBETFdvcmtsb2FkRmFpbHVyZSwgJDEiCiAgICAgIGV4aXQgMQogICAgfQoKICAgIGNoZWNrX3Byb3RvY29sKCkgewogICAgICBsb2NhbCBwcm94eV91cmw9JDEKICAgICAgc2hpZnQKICAgICAgbG9jYWwgc3VwcG9ydGVkX3Byb3RvY29scz0oIiRAIikKICAgICAgaWYgW1sgLW4gIiR7cHJveHlfdXJsfSIgXV07IHRoZW4KICAgICAgICBsb2NhbCBwcm90b2NvbD0kKGVjaG8gIiR7cHJveHlfdXJsfSIgfCBhd2sgLUYgJzovLycgJ3tpZiAoTkYgPiAxKSBwcmludCAkMTsgZWxzZSBwcmludCAiIn0nKQogICAgICAgIGlmIFsgLXogIiRwcm90b2NvbCIgXTsgdGhlbgogICAgICAgICAgZWNobyAiTm8gc3BlY2lmaWMgcHJvdG9jb2wgcHJvdmlkZWQuIFNraXBwaW5nIHByb3RvY29sIGNoZWNrLiIKICAgICAgICAgIHJldHVybiAwCiAgICAgICAgZmkKICAgICAgICBsb2NhbCBwcm90b2NvbF9pbmNsdWRlZD1mYWxzZQogICAgICAgIGZvciB2YXIgaW4gIiR7c3VwcG9ydGVkX3Byb3RvY29sc1tAXX0iOyBkbwogICAgICAgICAgaWYgW1sgIiR7cHJvdG9jb2x9IiA9PSAiJHt2YXJ9IiBdXTsgdGhlbgogICAgICAgICAgICBwcm90b2NvbF9pbmNsdWRlZD10cnVlCiAgICAgICAgICAgIGJyZWFrCiAgICAgICAgICBmaQogICAgICAgIGRvbmUKICAgICAgICBpZiBbWyAiJHtwcm90b2NvbF9pbmNsdWRlZH0iID09IGZhbHNlIF1dOyB0aGVuCiAgICAgICAgICBlcnJvcl9leGl0ICJVbnN1cHBvcnRlZCBwcm90b2NvbDogJHtwcm90b2NvbH0uIFN1cHBvcnRlZCBwcm90b2NvbHMgYXJlOiAke3N1cHBvcnRlZF9wcm90b2NvbHNbKl19IgogICAgICAgIGZpCiAgICAgIGZpCiAgICB9CgogICAgIyAkQDogbGlzdCBvZiBzdXBwb3J0ZWQgcHJvdG9jb2xzCiAgICBzZXRfcHJveHkoKSB7CiAgICAgIGxvY2FsIHN1cHBvcnRlZF9wcm90b2NvbHM9KCIkQCIpCgogICAgICBDT05GSUdfSlNPTl9CQVNFNjQ9JChncmVwICdjb25maWctanNvbicgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQogICAgICBDT05GSUdfSlNPTj0kKGVjaG8gJHtDT05GSUdfSlNPTl9CQVNFNjR9IHwgYmFzZTY0IC0tZGVjb2RlKQoKICAgICAgSFRUUF9QUk9YWV9VUkw9JChlY2hvICIke0NPTkZJR19KU09OfSIgfCBqcSAtciAnLmh0dHBfcHJveHkgLy8gZW1wdHknKQogICAgICBIVFRQU19QUk9YWV9VUkw9JChlY2hvICIke0NPTkZJR19KU09OfSIgfCBqcSAtciAnLmh0dHBzX3Byb3h5IC8vIGVtcHR5JykKICAgICAgaWYgW1sgJD8gLW5lIDAgfHwgKC16ICIke0hUVFBfUFJPWFlfVVJMfSIgJiYgLXogIiR7SFRUUFNfUFJPWFlfVVJMfSIpIF1dOyB0aGVuCiAgICAgICAgZWNobyAiSW5mbzogVGhlIGNvbmZpZy1qc29uIHdhcyBwYXJzZWQsIGJ1dCBubyBwcm94eSBzZXR0aW5ncyB3ZXJlIGZvdW5kLiIKICAgICAgICByZXR1cm4gMAogICAgICBmaQoKICAgICAgY2hlY2tfcHJvdG9jb2wgIiR7SFRUUF9QUk9YWV9VUkx9IiAiJHtzdXBwb3J0ZWRfcHJvdG9jb2xzW0BdfSIKICAgICAgY2hlY2tfcHJvdG9jb2wgIiR7SFRUUFNfUFJPWFlfVVJMfSIgIiR7c3VwcG9ydGVkX3Byb3RvY29sc1tAXX0iCgogICAgICBpZiAhIGdyZXAgLXEgJ2h0dHBfcHJveHknIC9ldGMvZW52aXJvbm1lbnQ7IHRoZW4KICAgICAgICBzdWRvIGJhc2ggLWMgJ2VjaG8gImV4cG9ydCBodHRwX3Byb3h5PSR7SFRUUF9QUk9YWV9VUkx9CiAgICAgICAgZXhwb3J0IGh0dHBzX3Byb3h5PSR7SFRUUFNfUFJPWFlfVVJMfQogICAgICAgIGV4cG9ydCBIVFRQX1BST1hZPSR7SFRUUF9QUk9YWV9VUkx9CiAgICAgICAgZXhwb3J0IEhUVFBTX1BST1hZPSR7SFRUUFNfUFJPWFlfVVJMfQogICAgICAgIGV4cG9ydCBub19wcm94eT1sb2NhbGhvc3QsMTI3LjAuMC4xIiA+PiAvZXRjL2Vudmlyb25tZW50JwogICAgICAgIHNvdXJjZSAvZXRjL2Vudmlyb25tZW50CiAgICAgIGZpCiAgICAgIAogICAgICAjIENvbmZpZ3VyZSBEb2NrZXIgdG8gdXNlIGEgcHJveHkKICAgICAgc3VkbyBta2RpciAtcCAvZXRjL3N5c3RlbWQvc3lzdGVtL2RvY2tlci5zZXJ2aWNlLmQKICAgICAgc3VkbyBiYXNoIC1jICdlY2hvICJbU2VydmljZV0KICAgICAgRW52aXJvbm1lbnQ9XCJIVFRQX1BST1hZPSR7SFRUUF9QUk9YWV9VUkx9XCIKICAgICAgRW52aXJvbm1lbnQ9XCJIVFRQU19QUk9YWT0ke0hUVFBTX1BST1hZX1VSTH1cIgogICAgICBFbnZpcm9ubWVudD1cIk5PX1BST1hZPWxvY2FsaG9zdCwxMjcuMC4wLjFcIiIgPiAvZXRjL3N5c3RlbWQvc3lzdGVtL2RvY2tlci5zZXJ2aWNlLmQvcHJveHkuY29uZicKICAgICAgc3VkbyBzeXN0ZW1jdGwgZGFlbW9uLXJlbG9hZAogICAgICBzdWRvIHN5c3RlbWN0bCByZXN0YXJ0IGRvY2tlcgoKICAgICAgZWNobyAiSW5mbzogZG9ja2VyIGFuZCBzeXN0ZW0gZW52aXJvbm1lbnQgYXJlIG5vdyBjb25maWd1cmVkIHRvIHVzZSB0aGUgcHJveHkgc2V0dGluZ3MiCiAgICB9CgogICAgZGVwbG95X2RjZ21fZXhwb3J0ZXIoKSB7CiAgICAgIENPTkZJR19KU09OX0JBU0U2ND0kKGdyZXAgJ2NvbmZpZy1qc29uJyAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCiAgICAgIENPTkZJR19KU09OPSQoZWNobyAke0NPTkZJR19KU09OX0JBU0U2NH0gfCBiYXNlNjQgLS1kZWNvZGUpCiAgICAgIERDR01fRVhQT1JUX1BVQkxJQz0kKGVjaG8gIiR7Q09ORklHX0pTT059IiB8IGpxIC1yICcuZXhwb3J0X2RjZ21fdG9fcHVibGljIC8vIGVtcHR5JykKCiAgICAgIERDR01fRVhQT1JURVJfSU1BR0U9IiRSRUdJU1RSWV9VUklfUEFUSC9udmlkaWEvazhzL2RjZ20tZXhwb3J0ZXIiCiAgICAgIERDR01fRVhQT1JURVJfVkVSU0lPTj0iMy4yLjUtMy4xLjgtdWJ1bnR1MjIuMDQiCiAgICAgIGlmIFsgLXogIiR7RENHTV9FWFBPUlRfUFVCTElDfSIgXSB8fCBbICIke0RDR01fRVhQT1JUX1BVQkxJQ30iICE9ICJ0cnVlIiBdOyB0aGVuCiAgICAgICAgZWNobyAiSW5mbzogbGF1bmNoaW5nIERDR00gRXhwb3J0ZXIgdG8gY29sbGVjdCB2R1BVIG1ldHJpY3MsIGxpc3RlbmluZyBvbmx5IG9uIGxvY2FsaG9zdCAoMTI3LjAuMC4xOjk0MDApIgogICAgICAgIGRvY2tlciBydW4gLWQgLS1ncHVzIGFsbCAtLWNhcC1hZGQgU1lTX0FETUlOIC1wIDEyNy4wLjAuMTo5NDAwOjk0MDAgJERDR01fRVhQT1JURVJfSU1BR0U6JERDR01fRVhQT1JURVJfVkVSU0lPTgogICAgICBlbHNlCiAgICAgICAgZWNobyAiSW5mbzogbGF1bmNoaW5nIERDR00gRXhwb3J0ZXIgdG8gY29sbGVjdCB2R1BVIG1ldHJpY3MsIGV4cG9zZWQgb24gYWxsIG5ldHdvcmsgaW50ZXJmYWNlcyAoMC4wLjAuMDo5NDAwKSIKICAgICAgICBkb2NrZXIgcnVuIC1kIC0tZ3B1cyBhbGwgLS1jYXAtYWRkIFNZU19BRE1JTiAtcCA5NDAwOjk0MDAgJERDR01fRVhQT1JURVJfSU1BR0U6JERDR01fRVhQT1JURVJfVkVSU0lPTgogICAgICBmaQogICAgfQ==

      which corresponds to the following script in plain-text format:

      #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    set_proxy "http" "https" "socks5"

    DEFAULT_REG_URI="nvcr.io"
    REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

    if [[ -z "$REGISTRY_URI_PATH" ]]; then
      # If REGISTRY_URI_PATH is null or empty, use the default value
      REGISTRY_URI_PATH=$DEFAULT_REG_URI
      echo "REGISTRY_URI_PATH was empty. Using default: $REGISTRY_URI_PATH"
    fi
    
    # If REGISTRY_URI_PATH contains '/', extract the URI part
    if [[ $REGISTRY_URI_PATH == *"/"* ]]; then
      REGISTRY_URI=$(echo "$REGISTRY_URI_PATH" | cut -d'/' -f1)
    else
      REGISTRY_URI=$REGISTRY_URI_PATH
    fi
  
    REGISTRY_USERNAME=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    REGISTRY_PASSWORD=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    if [[ -n "$REGISTRY_USERNAME" && -n "$REGISTRY_PASSWORD" ]]; then
      docker login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_URI
    else
      echo "Warning: the registry's username and password are invalid, Skipping Docker login."
    fi

    deploy_dcgm_exporter

    echo "Info: running the Triton Inference Server container"
    TRITON_IMAGE="$REGISTRY_URI_PATH/nvidia/tritonserver-pb24h1"
    TRITON_VERSION="24.03.02-py3"
    docker run -d --gpus all -p 8000:8000 -p 8001:8001 -p 8002:8002 -v /home/vmware/model_repository:/models $TRITON_IMAGE:$TRITON_VERSION tritonserver --model-repository=/models --model-control-mode=poll
    
- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi

      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker

      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="3.2.5-3.1.8-ubuntu22.04"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }
    • Image one-liner encoded in base64 format
      docker run -d --gpus all --rm -p8000:8000 -p8001:8001 -p8002:8002 -v /home/vmware/model_repository:/models nvcr.io/nvidia/tritonserver-pb24h1:ngc_image_tag tritonserver --model-repository=/models --model-control-mode=poll

      For example, for tritonserver:24.03.02-py3, provide the following script in base 64 format:

      ZG9ja2VyIHJ1biAtZCAtLWdwdXMgYWxsIC0tcm0gLXA4MDAwOjgwMDAgLXA4MDAxOjgwMDEgLXA4MDAyOjgwMDIgLXYgL2hvbWUvdm13YXJlL21vZGVsX3JlcG9zaXRvcnk6L21vZGVscyBudmNyLmlvL252aWRpYS90cml0b25zZXJ2ZXItcGIyNGgxOjI0LjAzLjAyLXB5MyB0cml0b25zZXJ2ZXIgLS1tb2RlbC1yZXBvc2l0b3J5PS9tb2RlbHMgLS1tb2RlbC1jb250cm9sLW1vZGU9cG9sbA==

      which corresponds to the following script in plain-text format: 

      docker run -d --gpus all --rm -p8000:8000 -p8001:8001 -p8002:8002 -v /home/vmware/model_repository:/models nvcr.io/nvidia/tritonserver-pb24h1:24.03.02-py3 tritonserver --model-repository=/models --model-control-mode=poll
  • Enter the vGPU guest driver installation properties, such as vgpu-license and nvidia-portal-api-key.
  • Provide values for the properties required for a disconnected environment as needed.

See OVF Properties of Deep Learning VMs.
Output
  • Installation logs for the vGPU guest driver in /var/log/vgpu-install.log.

    To verify that the vGPU guest driver is installed, log in to the VM over SSH and run the nvidia-smi command.

  • Cloud-init script logs in /var/log/dl.log.
  • Triton Inference Server container.

    To verify that the Triton Inference Server container is running, run the docker ps -a and docker logs container_id commands.

The model repository for the Triton Inference Server is in /home/vmware/model_repository. Initially, the model repository is empty and the initial log of the Triton Inference Server instance shows that no model is loaded.

Create a Model Repository

To load your model for model inference, perform these steps:

  1. Create the model repository for your model.

    See the NVIDIA Triton Inference Server Model Repository documentation .

  2. Copy the model repository to /home/vmware/model_repository so that the Triton Inference Server can load it.
    cp -r path_to_your_created_model_repository/* /home/vmware/model_repository/

Send Model Inference Requests

  1. Verify that the Triton Inference Server is healthy and models are ready by running this command in the deep learning VM console.
    curl -v localhost:8000/v2/simple_sequence
  2. Send a request to the model by running this command on the deep learning VM.
    curl -v localhost:8000/v2/models/simple_sequence

For more information on using the Triton Inference Server, see NVIDIA Triton Inference Server Model Repository documentation.

NVIDIA RAG

You can use a deep learning VM to build Retrieval Augmented Generation (RAG) solutions with an Llama2 model.

See the NVIDIA RAG Applications Docker Compose documentation (requires specific account permissions).

Table 6. NVIDIA RAG Container Image
Component Description
Container images and models 
docker-compose-nim-ms.yaml
rag-app-multiturn-chatbot/docker-compose.yaml
in the NVIDIA sample RAG pipeline.

For information on the NVIDIA RAG container applications that are supported for deep learning VMs, see VMware Deep Learning VM Release Notes.
Required inputs To deploy an NVIDIA RAG workload, you must set the OVF properties for the deep learning virtual machine in the following way:
  • Enter a cloud-init script. Encode it in base64 format.

    For example, for version 24.08 of NVIDIA RAG, provide the following script: 

    I2Nsb3VkLWNvbmZpZwp3cml0ZV9maWxlczoKLSBwYXRoOiAvb3B0L2Rsdm0vZGxfYXBwLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBzZXQgLWV1CiAgICBzb3VyY2UgL29wdC9kbHZtL3V0aWxzLnNoCiAgICB0cmFwICdlcnJvcl9leGl0ICJVbmV4cGVjdGVkIGVycm9yIG9jY3VycyBhdCBkbCB3b3JrbG9hZCInIEVSUgogICAgc2V0X3Byb3h5ICJodHRwIiAiaHR0cHMiCiAgICAKICAgIHN1ZG8gbWtkaXIgLXAgL29wdC9kYXRhLwogICAgc3VkbyBjaG93biB2bXdhcmU6dm13YXJlIC9vcHQvZGF0YQogICAgc3VkbyBjaG1vZCAtUiA3NzUgL29wdC9kYXRhCiAgICBjZCAvb3B0L2RhdGEvCgogICAgY2F0IDw8RU9GID4gL29wdC9kYXRhL2NvbmZpZy5qc29uCiAgICB7CiAgICAgICJfY29tbWVudF8xIjogIlRoaXMgcHJvdmlkZXMgZGVmYXVsdCBzdXBwb3J0IGZvciBSQUcgdjI0LjA4OiBsbGFtYTMtOGItaW5zdHJ1Y3QgbW9kZWwiLAogICAgICAiX2NvbW1lbnRfMiI6ICJVcGRhdGUgbGxtX21zX2dwdV9pZDogc3BlY2lmaWVzIHRoZSBHUFUgZGV2aWNlIElEIHRvIG1ha2UgYXZhaWxhYmxlIHRvIHRoZSBpbmZlcmVuY2Ugc2VydmVyIHdoZW4gdXNpbmcgbXVsdGlwbGUgR1BVIiwKICAgICAgIl9jb21tZW50XzMiOiAiVXBkYXRlIGVtYmVkZGluZ19tc19ncHVfaWQ6IHNwZWNpZmllcyB0aGUgR1BVIElEIHVzZWQgZm9yIGVtYmVkZGluZyBtb2RlbCBwcm9jZXNzaW5nIHdoZW4gdXNpbmcgbXVsdGlwbGUgR1BVIiwKICAgICAgInJhZyI6IHsKICAgICAgICAib3JnX25hbWUiOiAibnZpZGlhIiwKICAgICAgICAib3JnX3RlYW1fbmFtZSI6ICJhaXdvcmtmbG93cyIsCiAgICAgICAgInJhZ19uYW1lIjogImFpLWNoYXRib3QtZG9ja2VyLXdvcmtmbG93IiwKICAgICAgICAicmFnX3ZlcnNpb24iOiAiMjQuMDgiLAogICAgICAgICJyYWdfYXBwIjogInJhZy1hcHAtbXVsdGl0dXJuLWNoYXRib3QiLAogICAgICAgICJuaW1fbW9kZWxfcHJvZmlsZSI6ICJhdXRvIiwKICAgICAgICAibGxtX21zX2dwdV9pZCI6ICIwIiwKICAgICAgICAiZW1iZWRkaW5nX21zX2dwdV9pZCI6ICIwIiwKICAgICAgICAibW9kZWxfZGlyZWN0b3J5IjogIm1vZGVsLWNhY2hlIiwKICAgICAgICAibmdjX2NsaV92ZXJzaW9uIjogIjMuNDEuMiIKICAgICAgfQogICAgfQogICAgRU9GCgogICAgQ09ORklHX0pTT049JChjYXQgIi9vcHQvZGF0YS9jb25maWcuanNvbiIpCiAgICByZXF1aXJlZF92YXJzPSgiT1JHX05BTUUiICJPUkdfVEVBTV9OQU1FIiAiUkFHX05BTUUiICJSQUdfVkVSU0lPTiIgIlJBR19BUFAiICJOSU1fTU9ERUxfUFJPRklMRSIgIkxMTV9NU19HUFVfSUQiICJFTUJFRERJTkdfTVNfR1BVX0lEIiAiTU9ERUxfRElSRUNUT1JZIiAiTkdDX0NMSV9WRVJTSU9OIikKCiAgICAjIEV4dHJhY3QgcmFnIHZhbHVlcyBmcm9tIC9vcHQvZGF0YS9jb25maWcuanNvbgogICAgZm9yIGluZGV4IGluICIkeyFyZXF1aXJlZF92YXJzW0BdfSI7IGRvCiAgICAgIGtleT0iJHtyZXF1aXJlZF92YXJzWyRpbmRleF19IgogICAgICBqcV9xdWVyeT0iLnJhZy4ke2tleSwsfSB8IHNlbGVjdCAoLiE9bnVsbCkiCiAgICAgIHZhbHVlPSQoZWNobyAiJHtDT05GSUdfSlNPTn0iIHwganEgLXIgIiR7anFfcXVlcnl9IikKICAgICAgaWYgW1sgLXogIiR7dmFsdWV9IiBdXTsgdGhlbiAKICAgICAgICBlcnJvcl9leGl0ICIke2tleX0gaXMgcmVxdWlyZWQgYnV0IG5vdCBzZXQuIgogICAgICBlbHNlCiAgICAgICAgZXZhbCAke2tleX09XCIiJHt2YWx1ZX0iXCIKICAgICAgZmkKICAgIGRvbmUKCiAgICAjIFJlYWQgcGFyYW1ldGVycyBmcm9tIGNvbmZpZy1qc29uIHRvIGNvbm5lY3QgRFNNIFBHVmVjdG9yIG9uIFJBRwogICAgQ09ORklHX0pTT05fQkFTRTY0PSQoZ3JlcCAnY29uZmlnLWpzb24nIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKICAgIENPTkZJR19KU09OX1BHVkVDVE9SPSQoZWNobyAiJHtDT05GSUdfSlNPTl9CQVNFNjR9IiB8IGJhc2U2NCAtZCkKICAgIFBHVkVDVE9SX1ZBTFVFPSQoZWNobyAke0NPTkZJR19KU09OX1BHVkVDVE9SfSB8IGpxIC1yICcucmFnLnBndmVjdG9yJykKICAgIGlmIFtbIC1uICIke1BHVkVDVE9SX1ZBTFVFfSIgJiYgIiR7UEdWRUNUT1JfVkFMVUV9IiAhPSAibnVsbCIgXV07IHRoZW4KICAgICAgZWNobyAiSW5mbzogZXh0cmFjdCBEU00gUEdWZWN0b3IgcGFyYW1ldGVycyBmcm9tIGNvbmZpZy1qc29uIGluIFhNTCIKICAgICAgUE9TVEdSRVNfVVNFUj0kKGVjaG8gJHtQR1ZFQ1RPUl9WQUxVRX0gfCBhd2sgLUZbOkAvXSAne3ByaW50ICQ0fScpCiAgICAgIFBPU1RHUkVTX1BBU1NXT1JEPSQoZWNobyAke1BHVkVDVE9SX1ZBTFVFfSB8IGF3ayAtRls6QC9dICd7cHJpbnQgJDV9JykKICAgICAgUE9TVEdSRVNfSE9TVF9JUD0kKGVjaG8gJHtQR1ZFQ1RPUl9WQUxVRX0gfCBhd2sgLUZbOkAvXSAne3ByaW50ICQ2fScpCiAgICAgIFBPU1RHUkVTX1BPUlRfTlVNQkVSPSQoZWNobyAke1BHVkVDVE9SX1ZBTFVFfSB8IGF3ayAtRls6QC9dICd7cHJpbnQgJDd9JykKICAgICAgUE9TVEdSRVNfREI9JChlY2hvICR7UEdWRUNUT1JfVkFMVUV9IHwgYXdrIC1GWzpAL10gJ3twcmludCAkOH0nKQoKICAgICAgZm9yIHZhciBpbiBQT1NUR1JFU19VU0VSIFBPU1RHUkVTX1BBU1NXT1JEIFBPU1RHUkVTX0hPU1RfSVAgUE9TVEdSRVNfUE9SVF9OVU1CRVIgUE9TVEdSRVNfREI7IGRvCiAgICAgICAgaWYgWyAteiAiJHshdmFyfSIgXTsgdGhlbgogICAgICAgICAgZXJyb3JfZXhpdCAiJHt2YXJ9IGlzIG5vdCBzZXQuIgogICAgICAgIGZpCiAgICAgIGRvbmUKICAgIGZpCgogICAgZ3B1X2luZm89JChudmlkaWEtc21pIC1MKQogICAgZWNobyAiSW5mbzogdGhlIGRldGVjdGVkIEdQVSBpbmZvLCAkZ3B1X2luZm8iCiAgICBpZiBbWyAke05JTV9NT0RFTF9QUk9GSUxFfSA9PSAiYXV0byIgXV07IHRoZW4gCiAgICAgIGNhc2UgIiR7Z3B1X2luZm99IiBpbgogICAgICAgICpBMTAwKikKICAgICAgICAgIE5JTV9NT0RFTF9QUk9GSUxFPSI3NTEzODJkZjQyNzJlYWZjODNmNTQxZjM2NGQ2MWIzNWFlZDljY2U4YzdiMGM4NjkyNjljZWE1YTM2NmNkMDhjIgogICAgICAgICAgZWNobyAiSW5mbzogR1BVIHR5cGUgQTEwMCBkZXRlY3RlZC4gU2V0dGluZyB0ZW5zb3JydF9sbG0tQTEwMC1mcDE2LXRwMS10aHJvdWdocHV0IGFzIHRoZSBkZWZhdWx0IE5JTSBtb2RlbCBwcm9maWxlLiIKICAgICAgICAgIDs7CiAgICAgICAgKkgxMDAqKQogICAgICAgICAgTklNX01PREVMX1BST0ZJTEU9ImNiNTJjYmM3M2E2YTcxMzkyMDk0MzgwZjkyMGEzNTQ4ZjI3YzVmY2M5ZGFiMDJhOThkYzFiY2IzYmU5Y2Y4ZDEiCiAgICAgICAgICBlY2hvICJJbmZvOiBHUFUgdHlwZSBIMTAwIGRldGVjdGVkLiBTZXR0aW5nIHRlbnNvcnJ0X2xsbS1IMTAwLWZwMTYtdHAxLXRocm91Z2hwdXQgYXMgdGhlIGRlZmF1bHQgTklNIG1vZGVsIHByb2ZpbGUuIgogICAgICAgICAgOzsKICAgICAgICAqTDQwUyopCiAgICAgICAgICBOSU1fTU9ERUxfUFJPRklMRT0iZDhkZDhhZjgyZTAwMzVkN2NhNTBiOTk0ZDg1YTM3NDBkYmQ4NGRkYjRlZDMzMGUzMGM1MDllMDQxYmE3OWY4MCIKICAgICAgICAgIGVjaG8gIkluZm86IEdQVSB0eXBlIEw0MFMgZGV0ZWN0ZWQuIFNldHRpbmcgdGVuc29ycnRfbGxtLUw0MFMtZnAxNi10cDEtdGhyb3VnaHB1dCBhcyB0aGUgZGVmYXVsdCBOSU0gbW9kZWwgcHJvZmlsZS4iCiAgICAgICAgICA7OwogICAgICAgICopCiAgICAgICAgICBOSU1fTU9ERUxfUFJPRklMRT0iODgzNWMzMTc1MmZiYzY3ZWY2NThiMjBhOWY3OGUwNTY5MTRmZGVmMDY2MDIwNmQ4MmYyNTJkNjJmZDk2MDY0ZCIKICAgICAgICAgIGVjaG8gIkluZm86IE5vIHN1cHBvcnRlZCBHUFUgdHlwZSBkZXRlY3RlZCAoQTEwMCwgSDEwMCwgTDQwUykuIFNldHRpbmcgdmxsbSBhcyB0aGUgZGVmYXVsdCBOSU0gbW9kZWwgcHJvZmlsZS4iCiAgICAgICAgICA7OwogICAgICBlc2FjCiAgICBlbHNlCiAgICAgIGVjaG8gIkluZm86IHVzaW5nIHRoZSBOSU0gbW9kZWwgcHJvZmlsZSBwcm92aWRlZCBieSB0aGUgdXNlciwgJE5JTV9NT0RFTF9QUk9GSUxFIgogICAgZmkKCiAgICBSQUdfVVJJPSIke09SR19OQU1FfS8ke09SR19URUFNX05BTUV9LyR7UkFHX05BTUV9OiR7UkFHX1ZFUlNJT059IgogICAgUkFHX0ZPTERFUj0iJHtSQUdfTkFNRX1fdiR7UkFHX1ZFUlNJT059IgogICAgTkdDX0NMSV9VUkw9Imh0dHBzOi8vYXBpLm5nYy5udmlkaWEuY29tL3YyL3Jlc291cmNlcy9udmlkaWEvbmdjLWFwcHMvbmdjX2NsaS92ZXJzaW9ucy8ke05HQ19DTElfVkVSU0lPTn0vZmlsZXMvbmdjY2xpX2xpbnV4LnppcCIKCiAgICBpZiBbICEgLWYgLmluaXRpYWxpemUgXTsgdGhlbgogICAgICAjIGNsZWFuIHVwCiAgICAgIHJtIC1yZiBjb21wb3NlLmVudiBuZ2MqICR7UkFHX05BTUV9KiAke01PREVMX0RJUkVDVE9SWX0qIC5pbml0aWFsaXplCgogICAgICAjIGluc3RhbGwgbmdjLWNsaQogICAgICB3Z2V0IC0tY29udGVudC1kaXNwb3NpdGlvbiAke05HQ19DTElfVVJMfSAtTyBuZ2NjbGlfbGludXguemlwICYmIHVuemlwIC1xIG5nY2NsaV9saW51eC56aXAKICAgICAgZXhwb3J0IFBBVEg9YHB3ZGAvbmdjLWNsaToke1BBVEh9CgogICAgICBBUElLRVk9IiIKICAgICAgREVGQVVMVF9SRUdfVVJJPSJudmNyLmlvIgoKICAgICAgUkVHSVNUUllfVVJJX1BBVEg9JChncmVwIHJlZ2lzdHJ5LXVyaSAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCiAgICAgIGlmIFtbIC16ICIke1JFR0lTVFJZX1VSSV9QQVRIfSIgXV07IHRoZW4KICAgICAgICBSRUdJU1RSWV9VUklfUEFUSD0ke0RFRkFVTFRfUkVHX1VSSX0KICAgICAgICBlY2hvICJJbmZvOiByZWdpc3RyeSB1cmkgd2FzIGVtcHR5LiBVc2luZyBkZWZhdWx0OiAke1JFR0lTVFJZX1VSSV9QQVRIfSIKICAgICAgZmkKCiAgICAgIGlmIFtbICIkKGdyZXAgcmVnaXN0cnktdXJpIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykiID09ICoiJHtERUZBVUxUX1JFR19VUkl9IiogXV07IHRoZW4KICAgICAgICBBUElLRVk9JChncmVwIHJlZ2lzdHJ5LXBhc3N3ZCAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCiAgICAgIGZpCgogICAgICBpZiBbIC16ICIke0FQSUtFWX0iIF07IHRoZW4KICAgICAgICAgIGVycm9yX2V4aXQgIk5vIEFQSUtFWSBmb3VuZCIKICAgICAgZmkKCiAgICAgICMgY29uZmlnIG5nYy1jbGkKICAgICAgbWtkaXIgLXAgfi8ubmdjCgogICAgICBjYXQgPDwgRU9GID4gfi8ubmdjL2NvbmZpZwogICAgICBbQ1VSUkVOVF0KICAgICAgYXBpa2V5ID0gJHtBUElLRVl9CiAgICAgIGZvcm1hdF90eXBlID0gYXNjaWkKICAgICAgb3JnID0gJHtPUkdfTkFNRX0KICAgICAgdGVhbSA9ICR7T1JHX1RFQU1fTkFNRX0KICAgICAgYWNlID0gbm8tYWNlCiAgICBFT0YKICAgICAgCiAgICAgICMgRXh0cmFjdCByZWdpc3RyeSBVUkkgaWYgcGF0aCBjb250YWlucyAnLycKICAgICAgaWYgW1sgJHtSRUdJU1RSWV9VUklfUEFUSH0gPT0gKiIvIiogXV07IHRoZW4KICAgICAgICBSRUdJU1RSWV9VUkk9JChlY2hvICIke1JFR0lTVFJZX1VSSV9QQVRIfSIgfCBjdXQgLWQnLycgLWYxKQogICAgICBlbHNlCiAgICAgICAgUkVHSVNUUllfVVJJPSR7UkVHSVNUUllfVVJJX1BBVEh9CiAgICAgIGZpCgogICAgICBSRUdJU1RSWV9VU0VSPSQoZ3JlcCByZWdpc3RyeS11c2VyIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKCiAgICAgICMgRG9ja2VyIGxvZ2luIGlmIGNyZWRlbnRpYWxzIGFyZSBwcm92aWRlZAogICAgICBpZiBbWyAtbiAiJHtSRUdJU1RSWV9VU0VSfSIgJiYgLW4gIiR7QVBJS0VZfSIgXV07IHRoZW4KICAgICAgICBkb2NrZXIgbG9naW4gLXUgJHtSRUdJU1RSWV9VU0VSfSAtcCAke0FQSUtFWX0gJHtSRUdJU1RSWV9VUkl9CiAgICAgIGVsc2UKICAgICAgICBlY2hvICJXYXJuaW5nOiB0aGUgJHtSRUdJU1RSWV9VUkl9IHJlZ2lzdHJ5J3MgdXNlcm5hbWUgYW5kIHBhc3N3b3JkIGFyZSBpbnZhbGlkLCBTa2lwcGluZyBEb2NrZXIgbG9naW4uIgogICAgICBmaQoKICAgICAgIyBEb2NrZXJIdWIgbG9naW4gZm9yIGdlbmVyYWwgY29tcG9uZW50cwogICAgICBET0NLRVJIVUJfVVJJPSQoZ3JlcCByZWdpc3RyeS0yLXVyaSAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCiAgICAgIERPQ0tFUkhVQl9VU0VSTkFNRT0kKGdyZXAgcmVnaXN0cnktMi11c2VyIC9vcHQvZGx2bS9vdmYtZW52LnhtbCB8IHNlZCAtbiAncy8uKm9lOnZhbHVlPSJcKFteIl0qXCkuKi9cMS9wJykKICAgICAgRE9DS0VSSFVCX1BBU1NXT1JEPSQoZ3JlcCByZWdpc3RyeS0yLXBhc3N3ZCAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCgogICAgICBET0NLRVJIVUJfVVJJPSR7RE9DS0VSSFVCX1VSSTotZG9ja2VyLmlvfQogICAgICBpZiBbWyAtbiAiJHtET0NLRVJIVUJfVVNFUk5BTUV9IiAmJiAtbiAiJHtET0NLRVJIVUJfUEFTU1dPUkR9IiBdXTsgdGhlbgogICAgICAgIGRvY2tlciBsb2dpbiAtdSAke0RPQ0tFUkhVQl9VU0VSTkFNRX0gLXAgJHtET0NLRVJIVUJfUEFTU1dPUkR9ICR7RE9DS0VSSFVCX1VSSX0KICAgICAgZWxzZQogICAgICAgIGVjaG8gIldhcm5pbmc6ICR7RE9DS0VSSFVCX1VSSX0gbm90IGxvZ2dlZCBpbiIKICAgICAgZmkKCiAgICAgICMgRG93bmxvYWQgUkFHIGZpbGVzCiAgICAgIG5nYyByZWdpc3RyeSByZXNvdXJjZSBkb3dubG9hZC12ZXJzaW9uICR7UkFHX1VSSX0KCiAgICAgIG1rZGlyIC1wIC9vcHQvZGF0YS8ke01PREVMX0RJUkVDVE9SWX0KCiAgICAgICMgVXBkYXRlIHRoZSBkb2NrZXItY29tcG9zZSBZQU1MIGZpbGVzIHRvIGNvcnJlY3QgdGhlIGlzc3VlIHdpdGggR1BVIGZyZWUvbm9uLWZyZWUgc3RhdHVzIHJlcG9ydGluZwogICAgICAvdXNyL2Jpbi9weXRob24zIC1jICJpbXBvcnQgeWFtbCwganNvbiwgc3lzOyBwcmludChqc29uLmR1bXBzKHlhbWwuc2FmZV9sb2FkKHN5cy5zdGRpbi5yZWFkKCkpKSkiIDwgIiR7UkFHX0ZPTERFUn0vZG9ja2VyLWNvbXBvc2UtbmltLW1zLnlhbWwiPiBkb2NrZXItY29tcG9zZS1uaW0tbXMuanNvbgogICAgICBqcSAtLWFyZyBwcm9maWxlICIke05JTV9NT0RFTF9QUk9GSUxFfSIgXAogICAgICAgICAnLnNlcnZpY2VzLiJuZW1vbGxtLWluZmVyZW5jZSIuZW52aXJvbm1lbnQuTklNX01BTklGRVNUX0FMTE9XX1VOU0FGRSA9ICIxIiB8CiAgICAgICAgICAuc2VydmljZXMuIm5lbW9sbG0taW5mZXJlbmNlIi5lbnZpcm9ubWVudC5OSU1fTU9ERUxfUFJPRklMRSA9ICRwcm9maWxlIHwKICAgICAgICAgIC5zZXJ2aWNlcy4ibmVtb2xsbS1pbmZlcmVuY2UiLmRlcGxveS5yZXNvdXJjZXMucmVzZXJ2YXRpb25zLmRldmljZXNbMF0uZGV2aWNlX2lkcyA9IFsiJHtMTE1fTVNfR1BVX0lEOi0wfSJdIHwKICAgICAgICAgIGRlbCguc2VydmljZXMuIm5lbW9sbG0taW5mZXJlbmNlIi5kZXBsb3kucmVzb3VyY2VzLnJlc2VydmF0aW9ucy5kZXZpY2VzWzBdLmNvdW50KScgZG9ja2VyLWNvbXBvc2UtbmltLW1zLmpzb24gPiB0ZW1wLmpzb24gJiYgbXYgdGVtcC5qc29uIGRvY2tlci1jb21wb3NlLW5pbS1tcy5qc29uCiAgICAgIC91c3IvYmluL3B5dGhvbjMgLWMgImltcG9ydCB5YW1sLCBqc29uLCBzeXM7IHByaW50KHlhbWwuc2FmZV9kdW1wKGpzb24ubG9hZChzeXMuc3RkaW4pLCBkZWZhdWx0X2Zsb3dfc3R5bGU9RmFsc2UsIHNvcnRfa2V5cz1GYWxzZSkpIiA8IGRvY2tlci1jb21wb3NlLW5pbS1tcy5qc29uID4gIiR7UkFHX0ZPTERFUn0vZG9ja2VyLWNvbXBvc2UtbmltLW1zLnlhbWwiCiAgICAgIHJtIC1yZiBkb2NrZXItY29tcG9zZS1uaW0tbXMuanNvbgoKICAgICAgIyBVcGRhdGUgZG9ja2VyLWNvbXBvc2UgWUFNTCBmaWxlcyB0byBjb25maWcgUEdWZWN0b3IgYXMgdGhlIGRlZmF1bHQgZGF0YWJzZQogICAgICAvdXNyL2Jpbi9weXRob24zIC1jICJpbXBvcnQgeWFtbCwganNvbiwgc3lzOyBwcmludChqc29uLmR1bXBzKHlhbWwuc2FmZV9sb2FkKHN5cy5zdGRpbi5yZWFkKCkpKSkiIDwgIiR7UkFHX0ZPTERFUn0vJHtSQUdfQVBQfS9kb2NrZXItY29tcG9zZS55YW1sIj4gcmFnLWFwcC1tdWx0aXR1cm4tY2hhdGJvdC5qc29uCiAgICAgIGpxICcuc2VydmljZXMuImNoYWluLXNlcnZlciIuZW52aXJvbm1lbnQuQVBQX1ZFQ1RPUlNUT1JFX05BTUUgPSAicGd2ZWN0b3IiIHwKICAgICAgICAgLnNlcnZpY2VzLiJjaGFpbi1zZXJ2ZXIiLmVudmlyb25tZW50LkFQUF9WRUNUT1JTVE9SRV9VUkwgPSAiJHtQT1NUR1JFU19IT1NUX0lQOi1wZ3ZlY3Rvcn06JHtQT1NUR1JFU19QT1JUX05VTUJFUjotNTQzMn0iIHwKICAgICAgICAgLnNlcnZpY2VzLiJjaGFpbi1zZXJ2ZXIiLmVudmlyb25tZW50LlBPU1RHUkVTX1BBU1NXT1JEID0gIiR7UE9TVEdSRVNfUEFTU1dPUkQ6LXBhc3N3b3JkfSIgfAogICAgICAgICAuc2VydmljZXMuImNoYWluLXNlcnZlciIuZW52aXJvbm1lbnQuUE9TVEdSRVNfVVNFUiA9ICIke1BPU1RHUkVTX1VTRVI6LXBvc3RncmVzfSIgfAogICAgICAgICAuc2VydmljZXMuImNoYWluLXNlcnZlciIuZW52aXJvbm1lbnQuUE9TVEdSRVNfREIgPSAiJHtQT1NUR1JFU19EQjotYXBpfSInIHJhZy1hcHAtbXVsdGl0dXJuLWNoYXRib3QuanNvbiA+IHRlbXAuanNvbiAmJiBtdiB0ZW1wLmpzb24gcmFnLWFwcC1tdWx0aXR1cm4tY2hhdGJvdC5qc29uCiAgICAgIC91c3IvYmluL3B5dGhvbjMgLWMgImltcG9ydCB5YW1sLCBqc29uLCBzeXM7IHByaW50KHlhbWwuc2FmZV9kdW1wKGpzb24ubG9hZChzeXMuc3RkaW4pLCBkZWZhdWx0X2Zsb3dfc3R5bGU9RmFsc2UsIHNvcnRfa2V5cz1GYWxzZSkpIiA8IHJhZy1hcHAtbXVsdGl0dXJuLWNoYXRib3QuanNvbiA+ICIke1JBR19GT0xERVJ9LyR7UkFHX0FQUH0vZG9ja2VyLWNvbXBvc2UueWFtbCIKICAgICAgcm0gLXJmIHJhZy1hcHAtbXVsdGl0dXJuLWNoYXRib3QuanNvbgoKICAgICAgIyBjb25maWcgY29tcG9zZS5lbnYKICAgICAgY2F0IDw8IEVPRiA+IGNvbXBvc2UuZW52CiAgICAgIGV4cG9ydCBNT0RFTF9ESVJFQ1RPUlk9Ii9vcHQvZGF0YS8ke01PREVMX0RJUkVDVE9SWX0iCiAgICAgIGV4cG9ydCBOR0NfQVBJX0tFWT0ke0FQSUtFWX0KICAgICAgZXhwb3J0IFVTRVJJRD0kKGlkIC11KQogICAgICBleHBvcnQgTExNX01TX0dQVV9JRD0ke0xMTV9NU19HUFVfSUR9CiAgICAgIGV4cG9ydCBFTUJFRERJTkdfTVNfR1BVX0lEPSR7RU1CRURESU5HX01TX0dQVV9JRH0KICAgIEVPRgoKICAgICAgaWYgW1sgLW4gIiR7UEdWRUNUT1JfVkFMVUV9IiAmJiAiJHtQR1ZFQ1RPUl9WQUxVRX0iICE9ICJudWxsIiBdXTsgdGhlbiAKICAgICAgICBjYXQgPDwgRU9GID4+IGNvbXBvc2UuZW52CiAgICAgICAgZXhwb3J0IFBPU1RHUkVTX0hPU1RfSVA9IiR7UE9TVEdSRVNfSE9TVF9JUH0iCiAgICAgICAgZXhwb3J0IFBPU1RHUkVTX1BPUlRfTlVNQkVSPSIke1BPU1RHUkVTX1BPUlRfTlVNQkVSfSIKICAgICAgICBleHBvcnQgUE9TVEdSRVNfUEFTU1dPUkQ9IiR7UE9TVEdSRVNfUEFTU1dPUkR9IgogICAgICAgIGV4cG9ydCBQT1NUR1JFU19VU0VSPSIke1BPU1RHUkVTX1VTRVJ9IgogICAgICAgIGV4cG9ydCBQT1NUR1JFU19EQj0iJHtQT1NUR1JFU19EQn0iCiAgICBFT0YKICAgICAgZmkKICAgIAogICAgICB0b3VjaCAuaW5pdGlhbGl6ZQoKICAgICAgZGVwbG95X2RjZ21fZXhwb3J0ZXIKICAgIGZpCgogICAgIyBzdGFydCBOR0MgUkFHCiAgICBlY2hvICJJbmZvOiBydW5uaW5nIHRoZSBSQUcgYXBwbGljYXRpb24iCiAgICBzb3VyY2UgY29tcG9zZS5lbnYKICAgIGlmIFsgLXogIiR7UEdWRUNUT1JfVkFMVUV9IiBdIHx8IFsgIiR7UEdWRUNUT1JfVkFMVUV9IiA9ICJudWxsIiBdOyB0aGVuIAogICAgICBlY2hvICJJbmZvOiBydW5uaW5nIHRoZSBwZ3ZlY3RvciBjb250YWluZXIgYXMgdGhlIFZlY3RvciBEYXRhYmFzZSIKICAgICAgZG9ja2VyIGNvbXBvc2UgLWYgJHtSQUdfRk9MREVSfS8ke1JBR19BUFB9L2RvY2tlci1jb21wb3NlLnlhbWwgLS1wcm9maWxlIGxvY2FsLW5pbSAtLXByb2ZpbGUgcGd2ZWN0b3IgdXAgLWQKICAgIGVsc2UKICAgICAgZWNobyAiSW5mbzogdXNpbmcgdGhlIHByb3ZpZGVkIERTTSBQR1ZlY3RvciBhcyB0aGUgVmVjdG9yIERhdGFiYXNlIgogICAgICBkb2NrZXIgY29tcG9zZSAtZiAke1JBR19GT0xERVJ9LyR7UkFHX0FQUH0vZG9ja2VyLWNvbXBvc2UueWFtbCAtLXByb2ZpbGUgbG9jYWwtbmltIHVwIC1kCiAgICBmaQogICAgCi0gcGF0aDogL29wdC9kbHZtL3V0aWxzLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBlcnJvcl9leGl0KCkgewogICAgICBlY2hvICJFcnJvcjogJDEiID4mMgogICAgICB2bXRvb2xzZCAtLWNtZCAiaW5mby1zZXQgZ3Vlc3RpbmZvLnZtc2VydmljZS5ib290c3RyYXAuY29uZGl0aW9uIGZhbHNlLCBETFdvcmtsb2FkRmFpbHVyZSwgJDEiCiAgICAgIGV4aXQgMQogICAgfQoKICAgIGNoZWNrX3Byb3RvY29sKCkgewogICAgICBsb2NhbCBwcm94eV91cmw9JDEKICAgICAgc2hpZnQKICAgICAgbG9jYWwgc3VwcG9ydGVkX3Byb3RvY29scz0oIiRAIikKICAgICAgaWYgW1sgLW4gIiR7cHJveHlfdXJsfSIgXV07IHRoZW4KICAgICAgICBsb2NhbCBwcm90b2NvbD0kKGVjaG8gIiR7cHJveHlfdXJsfSIgfCBhd2sgLUYgJzovLycgJ3tpZiAoTkYgPiAxKSBwcmludCAkMTsgZWxzZSBwcmludCAiIn0nKQogICAgICAgIGlmIFsgLXogIiRwcm90b2NvbCIgXTsgdGhlbgogICAgICAgICAgZWNobyAiTm8gc3BlY2lmaWMgcHJvdG9jb2wgcHJvdmlkZWQuIFNraXBwaW5nIHByb3RvY29sIGNoZWNrLiIKICAgICAgICAgIHJldHVybiAwCiAgICAgICAgZmkKICAgICAgICBsb2NhbCBwcm90b2NvbF9pbmNsdWRlZD1mYWxzZQogICAgICAgIGZvciB2YXIgaW4gIiR7c3VwcG9ydGVkX3Byb3RvY29sc1tAXX0iOyBkbwogICAgICAgICAgaWYgW1sgIiR7cHJvdG9jb2x9IiA9PSAiJHt2YXJ9IiBdXTsgdGhlbgogICAgICAgICAgICBwcm90b2NvbF9pbmNsdWRlZD10cnVlCiAgICAgICAgICAgIGJyZWFrCiAgICAgICAgICBmaQogICAgICAgIGRvbmUKICAgICAgICBpZiBbWyAiJHtwcm90b2NvbF9pbmNsdWRlZH0iID09IGZhbHNlIF1dOyB0aGVuCiAgICAgICAgICBlcnJvcl9leGl0ICJVbnN1cHBvcnRlZCBwcm90b2NvbDogJHtwcm90b2NvbH0uIFN1cHBvcnRlZCBwcm90b2NvbHMgYXJlOiAke3N1cHBvcnRlZF9wcm90b2NvbHNbKl19IgogICAgICAgIGZpCiAgICAgIGZpCiAgICB9CgogICAgIyAkQDogbGlzdCBvZiBzdXBwb3J0ZWQgcHJvdG9jb2xzCiAgICBzZXRfcHJveHkoKSB7CiAgICAgIGxvY2FsIHN1cHBvcnRlZF9wcm90b2NvbHM9KCIkQCIpCgogICAgICBDT05GSUdfSlNPTl9CQVNFNjQ9JChncmVwICdjb25maWctanNvbicgL29wdC9kbHZtL292Zi1lbnYueG1sIHwgc2VkIC1uICdzLy4qb2U6dmFsdWU9IlwoW14iXSpcKS4qL1wxL3AnKQogICAgICBDT05GSUdfSlNPTj0kKGVjaG8gJHtDT05GSUdfSlNPTl9CQVNFNjR9IHwgYmFzZTY0IC0tZGVjb2RlKQoKICAgICAgSFRUUF9QUk9YWV9VUkw9JChlY2hvICIke0NPTkZJR19KU09OfSIgfCBqcSAtciAnLmh0dHBfcHJveHkgLy8gZW1wdHknKQogICAgICBIVFRQU19QUk9YWV9VUkw9JChlY2hvICIke0NPTkZJR19KU09OfSIgfCBqcSAtciAnLmh0dHBzX3Byb3h5IC8vIGVtcHR5JykKICAgICAgaWYgW1sgJD8gLW5lIDAgfHwgKC16ICIke0hUVFBfUFJPWFlfVVJMfSIgJiYgLXogIiR7SFRUUFNfUFJPWFlfVVJMfSIpIF1dOyB0aGVuCiAgICAgICAgZWNobyAiSW5mbzogVGhlIGNvbmZpZy1qc29uIHdhcyBwYXJzZWQsIGJ1dCBubyBwcm94eSBzZXR0aW5ncyB3ZXJlIGZvdW5kLiIKICAgICAgICByZXR1cm4gMAogICAgICBmaQoKICAgICAgY2hlY2tfcHJvdG9jb2wgIiR7SFRUUF9QUk9YWV9VUkx9IiAiJHtzdXBwb3J0ZWRfcHJvdG9jb2xzW0BdfSIKICAgICAgY2hlY2tfcHJvdG9jb2wgIiR7SFRUUFNfUFJPWFlfVVJMfSIgIiR7c3VwcG9ydGVkX3Byb3RvY29sc1tAXX0iCgogICAgICBpZiAhIGdyZXAgLXEgJ2h0dHBfcHJveHknIC9ldGMvZW52aXJvbm1lbnQ7IHRoZW4KICAgICAgICBzdWRvIGJhc2ggLWMgJ2VjaG8gImV4cG9ydCBodHRwX3Byb3h5PSR7SFRUUF9QUk9YWV9VUkx9CiAgICAgICAgZXhwb3J0IGh0dHBzX3Byb3h5PSR7SFRUUFNfUFJPWFlfVVJMfQogICAgICAgIGV4cG9ydCBIVFRQX1BST1hZPSR7SFRUUF9QUk9YWV9VUkx9CiAgICAgICAgZXhwb3J0IEhUVFBTX1BST1hZPSR7SFRUUFNfUFJPWFlfVVJMfQogICAgICAgIGV4cG9ydCBub19wcm94eT1sb2NhbGhvc3QsMTI3LjAuMC4xIiA+PiAvZXRjL2Vudmlyb25tZW50JwogICAgICAgIHNvdXJjZSAvZXRjL2Vudmlyb25tZW50CiAgICAgIGZpCiAgICAgIAogICAgICAjIENvbmZpZ3VyZSBEb2NrZXIgdG8gdXNlIGEgcHJveHkKICAgICAgc3VkbyBta2RpciAtcCAvZXRjL3N5c3RlbWQvc3lzdGVtL2RvY2tlci5zZXJ2aWNlLmQKICAgICAgc3VkbyBiYXNoIC1jICdlY2hvICJbU2VydmljZV0KICAgICAgRW52aXJvbm1lbnQ9XCJIVFRQX1BST1hZPSR7SFRUUF9QUk9YWV9VUkx9XCIKICAgICAgRW52aXJvbm1lbnQ9XCJIVFRQU19QUk9YWT0ke0hUVFBTX1BST1hZX1VSTH1cIgogICAgICBFbnZpcm9ubWVudD1cIk5PX1BST1hZPWxvY2FsaG9zdCwxMjcuMC4wLjFcIiIgPiAvZXRjL3N5c3RlbWQvc3lzdGVtL2RvY2tlci5zZXJ2aWNlLmQvcHJveHkuY29uZicKICAgICAgc3VkbyBzeXN0ZW1jdGwgZGFlbW9uLXJlbG9hZAogICAgICBzdWRvIHN5c3RlbWN0bCByZXN0YXJ0IGRvY2tlcgoKICAgICAgZWNobyAiSW5mbzogZG9ja2VyIGFuZCBzeXN0ZW0gZW52aXJvbm1lbnQgYXJlIG5vdyBjb25maWd1cmVkIHRvIHVzZSB0aGUgcHJveHkgc2V0dGluZ3MiCiAgICB9CgogICAgZGVwbG95X2RjZ21fZXhwb3J0ZXIoKSB7CiAgICAgIENPTkZJR19KU09OX0JBU0U2ND0kKGdyZXAgJ2NvbmZpZy1qc29uJyAvb3B0L2Rsdm0vb3ZmLWVudi54bWwgfCBzZWQgLW4gJ3MvLipvZTp2YWx1ZT0iXChbXiJdKlwpLiovXDEvcCcpCiAgICAgIENPTkZJR19KU09OPSQoZWNobyAke0NPTkZJR19KU09OX0JBU0U2NH0gfCBiYXNlNjQgLS1kZWNvZGUpCiAgICAgIERDR01fRVhQT1JUX1BVQkxJQz0kKGVjaG8gIiR7Q09ORklHX0pTT059IiB8IGpxIC1yICcuZXhwb3J0X2RjZ21fdG9fcHVibGljIC8vIGVtcHR5JykKCiAgICAgIERDR01fRVhQT1JURVJfSU1BR0U9IiRSRUdJU1RSWV9VUklfUEFUSC9udmlkaWEvazhzL2RjZ20tZXhwb3J0ZXIiCiAgICAgIERDR01fRVhQT1JURVJfVkVSU0lPTj0iMy4yLjUtMy4xLjgtdWJ1bnR1MjIuMDQiCiAgICAgIGlmIFsgLXogIiR7RENHTV9FWFBPUlRfUFVCTElDfSIgXSB8fCBbICIke0RDR01fRVhQT1JUX1BVQkxJQ30iICE9ICJ0cnVlIiBdOyB0aGVuCiAgICAgICAgZWNobyAiSW5mbzogbGF1bmNoaW5nIERDR00gRXhwb3J0ZXIgdG8gY29sbGVjdCB2R1BVIG1ldHJpY3MsIGxpc3RlbmluZyBvbmx5IG9uIGxvY2FsaG9zdCAoMTI3LjAuMC4xOjk0MDApIgogICAgICAgIGRvY2tlciBydW4gLWQgLS1ncHVzIGFsbCAtLWNhcC1hZGQgU1lTX0FETUlOIC1wIDEyNy4wLjAuMTo5NDAwOjk0MDAgJERDR01fRVhQT1JURVJfSU1BR0U6JERDR01fRVhQT1JURVJfVkVSU0lPTgogICAgICBlbHNlCiAgICAgICAgZWNobyAiSW5mbzogbGF1bmNoaW5nIERDR00gRXhwb3J0ZXIgdG8gY29sbGVjdCB2R1BVIG1ldHJpY3MsIGV4cG9zZWQgb24gYWxsIG5ldHdvcmsgaW50ZXJmYWNlcyAoMC4wLjAuMDo5NDAwKSIKICAgICAgICBkb2NrZXIgcnVuIC1kIC0tZ3B1cyBhbGwgLS1jYXAtYWRkIFNZU19BRE1JTiAtcCA5NDAwOjk0MDAgJERDR01fRVhQT1JURVJfSU1BR0U6JERDR01fRVhQT1JURVJfVkVSU0lPTgogICAgICBmaQogICAgfQ==

    which corresponds to the following script in plain-text format:

    #cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    set -eu
    source /opt/dlvm/utils.sh
    trap 'error_exit "Unexpected error occurs at dl workload"' ERR
    set_proxy "http" "https"
    
    sudo mkdir -p /opt/data/
    sudo chown vmware:vmware /opt/data
    sudo chmod -R 775 /opt/data
    cd /opt/data/

    cat <<EOF > /opt/data/config.json
    {
      "_comment_1": "This provides default support for RAG v24.08: llama3-8b-instruct model",
      "_comment_2": "Update llm_ms_gpu_id: specifies the GPU device ID to make available to the inference server when using multiple GPU",
      "_comment_3": "Update embedding_ms_gpu_id: specifies the GPU ID used for embedding model processing when using multiple GPU",
      "rag": {
        "org_name": "nvidia",
        "org_team_name": "aiworkflows",
        "rag_name": "ai-chatbot-docker-workflow",
        "rag_version": "24.08",
        "rag_app": "rag-app-multiturn-chatbot",
        "nim_model_profile": "auto",
        "llm_ms_gpu_id": "0",
        "embedding_ms_gpu_id": "0",
        "model_directory": "model-cache",
        "ngc_cli_version": "3.41.2"
      }
    }
    EOF

    CONFIG_JSON=$(cat "/opt/data/config.json")
    required_vars=("ORG_NAME" "ORG_TEAM_NAME" "RAG_NAME" "RAG_VERSION" "RAG_APP" "NIM_MODEL_PROFILE" "LLM_MS_GPU_ID" "EMBEDDING_MS_GPU_ID" "MODEL_DIRECTORY" "NGC_CLI_VERSION")

    # Extract rag values from /opt/data/config.json
    for index in "${!required_vars[@]}"; do
      key="${required_vars[$index]}"
      jq_query=".rag.${key,,} | select (.!=null)"
      value=$(echo "${CONFIG_JSON}" | jq -r "${jq_query}")
      if [[ -z "${value}" ]]; then 
        error_exit "${key} is required but not set."
      else
        eval ${key}=\""${value}"\"
      fi
    done

    # Read parameters from config-json to connect DSM PGVector on RAG
    CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
    CONFIG_JSON_PGVECTOR=$(echo "${CONFIG_JSON_BASE64}" | base64 -d)
    PGVECTOR_VALUE=$(echo ${CONFIG_JSON_PGVECTOR} | jq -r '.rag.pgvector')
    if [[ -n "${PGVECTOR_VALUE}" && "${PGVECTOR_VALUE}" != "null" ]]; then
      echo "Info: extract DSM PGVector parameters from config-json in XML"
      POSTGRES_USER=$(echo ${PGVECTOR_VALUE} | awk -F[:@/] '{print $4}')
      POSTGRES_PASSWORD=$(echo ${PGVECTOR_VALUE} | awk -F[:@/] '{print $5}')
      POSTGRES_HOST_IP=$(echo ${PGVECTOR_VALUE} | awk -F[:@/] '{print $6}')
      POSTGRES_PORT_NUMBER=$(echo ${PGVECTOR_VALUE} | awk -F[:@/] '{print $7}')
      POSTGRES_DB=$(echo ${PGVECTOR_VALUE} | awk -F[:@/] '{print $8}')

      for var in POSTGRES_USER POSTGRES_PASSWORD POSTGRES_HOST_IP POSTGRES_PORT_NUMBER POSTGRES_DB; do
        if [ -z "${!var}" ]; then
          error_exit "${var} is not set."
        fi
      done
    fi

    gpu_info=$(nvidia-smi -L)
    echo "Info: the detected GPU info, $gpu_info"
    if [[ ${NIM_MODEL_PROFILE} == "auto" ]]; then 
      case "${gpu_info}" in
        *A100*)
          NIM_MODEL_PROFILE="751382df4272eafc83f541f364d61b35aed9cce8c7b0c869269cea5a366cd08c"
          echo "Info: GPU type A100 detected. Setting tensorrt_llm-A100-fp16-tp1-throughput as the default NIM model profile."
          ;;
        *H100*)
          NIM_MODEL_PROFILE="cb52cbc73a6a71392094380f920a3548f27c5fcc9dab02a98dc1bcb3be9cf8d1"
          echo "Info: GPU type H100 detected. Setting tensorrt_llm-H100-fp16-tp1-throughput as the default NIM model profile."
          ;;
        *L40S*)
          NIM_MODEL_PROFILE="d8dd8af82e0035d7ca50b994d85a3740dbd84ddb4ed330e30c509e041ba79f80"
          echo "Info: GPU type L40S detected. Setting tensorrt_llm-L40S-fp16-tp1-throughput as the default NIM model profile."
          ;;
        *)
          NIM_MODEL_PROFILE="8835c31752fbc67ef658b20a9f78e056914fdef0660206d82f252d62fd96064d"
          echo "Info: No supported GPU type detected (A100, H100, L40S). Setting vllm as the default NIM model profile."
          ;;
      esac
    else
      echo "Info: using the NIM model profile provided by the user, $NIM_MODEL_PROFILE"
    fi

    RAG_URI="${ORG_NAME}/${ORG_TEAM_NAME}/${RAG_NAME}:${RAG_VERSION}"
    RAG_FOLDER="${RAG_NAME}_v${RAG_VERSION}"
    NGC_CLI_URL="https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/${NGC_CLI_VERSION}/files/ngccli_linux.zip"

    if [ ! -f .initialize ]; then
      # clean up
      rm -rf compose.env ngc* ${RAG_NAME}* ${MODEL_DIRECTORY}* .initialize

      # install ngc-cli
      wget --content-disposition ${NGC_CLI_URL} -O ngccli_linux.zip && unzip -q ngccli_linux.zip
      export PATH=`pwd`/ngc-cli:${PATH}

      APIKEY=""
      DEFAULT_REG_URI="nvcr.io"

      REGISTRY_URI_PATH=$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      if [[ -z "${REGISTRY_URI_PATH}" ]]; then
        REGISTRY_URI_PATH=${DEFAULT_REG_URI}
        echo "Info: registry uri was empty. Using default: ${REGISTRY_URI_PATH}"
      fi

      if [[ "$(grep registry-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')" == *"${DEFAULT_REG_URI}"* ]]; then
        APIKEY=$(grep registry-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      fi

      if [ -z "${APIKEY}" ]; then
          error_exit "No APIKEY found"
      fi

      # config ngc-cli
      mkdir -p ~/.ngc

      cat << EOF > ~/.ngc/config
      [CURRENT]
      apikey = ${APIKEY}
      format_type = ascii
      org = ${ORG_NAME}
      team = ${ORG_TEAM_NAME}
      ace = no-ace
    EOF
      
      # Extract registry URI if path contains '/'
      if [[ ${REGISTRY_URI_PATH} == *"/"* ]]; then
        REGISTRY_URI=$(echo "${REGISTRY_URI_PATH}" | cut -d'/' -f1)
      else
        REGISTRY_URI=${REGISTRY_URI_PATH}
      fi

      REGISTRY_USER=$(grep registry-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

      # Docker login if credentials are provided
      if [[ -n "${REGISTRY_USER}" && -n "${APIKEY}" ]]; then
        docker login -u ${REGISTRY_USER} -p ${APIKEY} ${REGISTRY_URI}
      else
        echo "Warning: the ${REGISTRY_URI} registry's username and password are invalid, Skipping Docker login."
      fi

      # DockerHub login for general components
      DOCKERHUB_URI=$(grep registry-2-uri /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      DOCKERHUB_USERNAME=$(grep registry-2-user /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      DOCKERHUB_PASSWORD=$(grep registry-2-passwd /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')

      DOCKERHUB_URI=${DOCKERHUB_URI:-docker.io}
      if [[ -n "${DOCKERHUB_USERNAME}" && -n "${DOCKERHUB_PASSWORD}" ]]; then
        docker login -u ${DOCKERHUB_USERNAME} -p ${DOCKERHUB_PASSWORD} ${DOCKERHUB_URI}
      else
        echo "Warning: ${DOCKERHUB_URI} not logged in"
      fi

      # Download RAG files
      ngc registry resource download-version ${RAG_URI}

      mkdir -p /opt/data/${MODEL_DIRECTORY}

      # Update the docker-compose YAML files to correct the issue with GPU free/non-free status reporting
      /usr/bin/python3 -c "import yaml, json, sys; print(json.dumps(yaml.safe_load(sys.stdin.read())))" < "${RAG_FOLDER}/docker-compose-nim-ms.yaml"> docker-compose-nim-ms.json
      jq --arg profile "${NIM_MODEL_PROFILE}" \
         '.services."nemollm-inference".environment.NIM_MANIFEST_ALLOW_UNSAFE = "1" |
          .services."nemollm-inference".environment.NIM_MODEL_PROFILE = $profile |
          .services."nemollm-inference".deploy.resources.reservations.devices[0].device_ids = ["${LLM_MS_GPU_ID:-0}"] |
          del(.services."nemollm-inference".deploy.resources.reservations.devices[0].count)' docker-compose-nim-ms.json > temp.json && mv temp.json docker-compose-nim-ms.json
      /usr/bin/python3 -c "import yaml, json, sys; print(yaml.safe_dump(json.load(sys.stdin), default_flow_style=False, sort_keys=False))" < docker-compose-nim-ms.json > "${RAG_FOLDER}/docker-compose-nim-ms.yaml"
      rm -rf docker-compose-nim-ms.json

      # Update docker-compose YAML files to config PGVector as the default databse
      /usr/bin/python3 -c "import yaml, json, sys; print(json.dumps(yaml.safe_load(sys.stdin.read())))" < "${RAG_FOLDER}/${RAG_APP}/docker-compose.yaml"> rag-app-multiturn-chatbot.json
      jq '.services."chain-server".environment.APP_VECTORSTORE_NAME = "pgvector" |
         .services."chain-server".environment.APP_VECTORSTORE_URL = "${POSTGRES_HOST_IP:-pgvector}:${POSTGRES_PORT_NUMBER:-5432}" |
         .services."chain-server".environment.POSTGRES_PASSWORD = "${POSTGRES_PASSWORD:-password}" |
         .services."chain-server".environment.POSTGRES_USER = "${POSTGRES_USER:-postgres}" |
         .services."chain-server".environment.POSTGRES_DB = "${POSTGRES_DB:-api}"' rag-app-multiturn-chatbot.json > temp.json && mv temp.json rag-app-multiturn-chatbot.json
      /usr/bin/python3 -c "import yaml, json, sys; print(yaml.safe_dump(json.load(sys.stdin), default_flow_style=False, sort_keys=False))" < rag-app-multiturn-chatbot.json > "${RAG_FOLDER}/${RAG_APP}/docker-compose.yaml"
      rm -rf rag-app-multiturn-chatbot.json

      # config compose.env
      cat << EOF > compose.env
      export MODEL_DIRECTORY="/opt/data/${MODEL_DIRECTORY}"
      export NGC_API_KEY=${APIKEY}
      export USERID=$(id -u)
      export LLM_MS_GPU_ID=${LLM_MS_GPU_ID}
      export EMBEDDING_MS_GPU_ID=${EMBEDDING_MS_GPU_ID}
    EOF

      if [[ -n "${PGVECTOR_VALUE}" && "${PGVECTOR_VALUE}" != "null" ]]; then 
        cat << EOF >> compose.env
        export POSTGRES_HOST_IP="${POSTGRES_HOST_IP}"
        export POSTGRES_PORT_NUMBER="${POSTGRES_PORT_NUMBER}"
        export POSTGRES_PASSWORD="${POSTGRES_PASSWORD}"
        export POSTGRES_USER="${POSTGRES_USER}"
        export POSTGRES_DB="${POSTGRES_DB}"
    EOF
      fi
    
      touch .initialize

      deploy_dcgm_exporter
    fi

    # start NGC RAG
    echo "Info: running the RAG application"
    source compose.env
    if [ -z "${PGVECTOR_VALUE}" ] || [ "${PGVECTOR_VALUE}" = "null" ]; then 
      echo "Info: running the pgvector container as the Vector Database"
      docker compose -f ${RAG_FOLDER}/${RAG_APP}/docker-compose.yaml --profile local-nim --profile pgvector up -d
    else
      echo "Info: using the provided DSM PGVector as the Vector Database"
      docker compose -f ${RAG_FOLDER}/${RAG_APP}/docker-compose.yaml --profile local-nim up -d
    fi
    
- path: /opt/dlvm/utils.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    error_exit() {
      echo "Error: $1" >&2
      vmtoolsd --cmd "info-set guestinfo.vmservice.bootstrap.condition false, DLWorkloadFailure, $1"
      exit 1
    }

    check_protocol() {
      local proxy_url=$1
      shift
      local supported_protocols=("$@")
      if [[ -n "${proxy_url}" ]]; then
        local protocol=$(echo "${proxy_url}" | awk -F '://' '{if (NF > 1) print $1; else print ""}')
        if [ -z "$protocol" ]; then
          echo "No specific protocol provided. Skipping protocol check."
          return 0
        fi
        local protocol_included=false
        for var in "${supported_protocols[@]}"; do
          if [[ "${protocol}" == "${var}" ]]; then
            protocol_included=true
            break
          fi
        done
        if [[ "${protocol_included}" == false ]]; then
          error_exit "Unsupported protocol: ${protocol}. Supported protocols are: ${supported_protocols[*]}"
        fi
      fi
    }

    # $@: list of supported protocols
    set_proxy() {
      local supported_protocols=("$@")

      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)

      HTTP_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.http_proxy // empty')
      HTTPS_PROXY_URL=$(echo "${CONFIG_JSON}" | jq -r '.https_proxy // empty')
      if [[ $? -ne 0 || (-z "${HTTP_PROXY_URL}" && -z "${HTTPS_PROXY_URL}") ]]; then
        echo "Info: The config-json was parsed, but no proxy settings were found."
        return 0
      fi

      check_protocol "${HTTP_PROXY_URL}" "${supported_protocols[@]}"
      check_protocol "${HTTPS_PROXY_URL}" "${supported_protocols[@]}"

      if ! grep -q 'http_proxy' /etc/environment; then
        sudo bash -c 'echo "export http_proxy=${HTTP_PROXY_URL}
        export https_proxy=${HTTPS_PROXY_URL}
        export HTTP_PROXY=${HTTP_PROXY_URL}
        export HTTPS_PROXY=${HTTPS_PROXY_URL}
        export no_proxy=localhost,127.0.0.1" >> /etc/environment'
        source /etc/environment
      fi
      
      # Configure Docker to use a proxy
      sudo mkdir -p /etc/systemd/system/docker.service.d
      sudo bash -c 'echo "[Service]
      Environment=\"HTTP_PROXY=${HTTP_PROXY_URL}\"
      Environment=\"HTTPS_PROXY=${HTTPS_PROXY_URL}\"
      Environment=\"NO_PROXY=localhost,127.0.0.1\"" > /etc/systemd/system/docker.service.d/proxy.conf'
      sudo systemctl daemon-reload
      sudo systemctl restart docker

      echo "Info: docker and system environment are now configured to use the proxy settings"
    }

    deploy_dcgm_exporter() {
      CONFIG_JSON_BASE64=$(grep 'config-json' /opt/dlvm/ovf-env.xml | sed -n 's/.*oe:value="\([^"]*\).*/\1/p')
      CONFIG_JSON=$(echo ${CONFIG_JSON_BASE64} | base64 --decode)
      DCGM_EXPORT_PUBLIC=$(echo "${CONFIG_JSON}" | jq -r '.export_dcgm_to_public // empty')

      DCGM_EXPORTER_IMAGE="$REGISTRY_URI_PATH/nvidia/k8s/dcgm-exporter"
      DCGM_EXPORTER_VERSION="3.2.5-3.1.8-ubuntu22.04"
      if [ -z "${DCGM_EXPORT_PUBLIC}" ] || [ "${DCGM_EXPORT_PUBLIC}" != "true" ]; then
        echo "Info: launching DCGM Exporter to collect vGPU metrics, listening only on localhost (127.0.0.1:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 127.0.0.1:9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      else
        echo "Info: launching DCGM Exporter to collect vGPU metrics, exposed on all network interfaces (0.0.0.0:9400)"
        docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 $DCGM_EXPORTER_IMAGE:$DCGM_EXPORTER_VERSION
      fi
    }
  • Enter the vGPU guest driver installation properties, such as vgpu-license and nvidia-portal-api-key.
  • Provide values for the properties required for a disconnected environment as needed.

See OVF Properties of Deep Learning VMs.
Output
  • Installation logs for the vGPU guest driver in /var/log/vgpu-install.log.

    To verify that the vGPU guest driver is installed, log in to the VM over SSH and run the nvidia-smi command.

  • Cloud-init script logs in /var/log/dl.log.

    To track deployment progress, run tail -f /var/log/dl.log .

  • Sample chatbot Web application that you can access at http://dl_vm_ip:3001

    You can upload your own knowledge base.

Assign a Static IP Address to a Deep Learning VM in VMware Private AI Foundation with NVIDIA

By default, the deep learning VM images are configured with DHCP address assignment. If you want to deploy a deep learning VM with a static IP address directly on a vSphere cluster, you must add additional code to the cloud-init section.

On vSphere with Tanzu, IP address assignment is determined by the network configuration for the Supervisor in NSX.

Procedure

  1. Create a cloud-init script in plain-text format for the DL workload you plan to use.
    See Deep Learning Workloads in VMware Private AI Foundation with NVIDIA.
  2. Add the following additional code to the cloud-init script. 
    #cloud-config
<instructions_for_your_DL_workload>

manage_etc_hosts: true
 
write_files:
  - path: /etc/netplan/50-cloud-init.yaml
    permissions: '0600'
    content: |
      network:
        version: 2
        renderer: networkd
        ethernets:
          ens33:
            dhcp4: false # disable DHCP4
            addresses: [x.x.x.x/x]  # Set the static IP address and mask
            routes:
                - to: default
                  via: x.x.x.x # Configure gateway
            nameservers:
              addresses: [x.x.x.x, x.x.x.x] # Provide the DNS server address. Separate mulitple DNS server addresses with commas.
 
runcmd:
  - netplan apply
  3. Encode the resulting cloud-init script in base64 format.
  4. Set the resulting cloud-init script in base64 format as a value to the user-data OVF parameter of the deep learning VM image.

Example: Assigning a Static IP Address to a CUDA Sample Workload

For an example deep learning VM with a CUDA Sample DL workload:

Deep Learning VM Element Example Value
DL workload image nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8
IP address 10.199.118.245
Subnet prefix /25
Gateway 10.199.118.253
DNS servers
  • 10.142.7.1
  • 10.132.7.1

you provide the following cloud-init code: 

I2Nsb3VkLWNvbmZpZwp3cml0ZV9maWxlczoKLSBwYXRoOiAvb3B0L2Rsdm0vZGxfYXBwLnNoCiAgcGVybWlzc2lvbnM6ICcwNzU1JwogIGNvbnRlbnQ6IHwKICAgICMhL2Jpbi9iYXNoCiAgICBkb2NrZXIgcnVuIC1kIG52Y3IuaW8vbnZpZGlhL2s4cy9jdWRhLXNhbXBsZTp2ZWN0b3JhZGQtY3VkYTExLjcuMS11Ymk4CgptYW5hZ2VfZXRjX2hvc3RzOiB0cnVlCiAKd3JpdGVfZmlsZXM6CiAgLSBwYXRoOiAvZXRjL25ldHBsYW4vNTAtY2xvdWQtaW5pdC55YW1sCiAgICBwZXJtaXNzaW9uczogJzA2MDAnCiAgICBjb250ZW50OiB8CiAgICAgIG5ldHdvcms6CiAgICAgICAgdmVyc2lvbjogMgogICAgICAgIHJlbmRlcmVyOiBuZXR3b3JrZAogICAgICAgIGV0aGVybmV0czoKICAgICAgICAgIGVuczMzOgogICAgICAgICAgICBkaGNwNDogZmFsc2UgIyBkaXNhYmxlIERIQ1A0CiAgICAgICAgICAgIGFkZHJlc3NlczogWzEwLjE5OS4xMTguMjQ1LzI1XSAgIyBTZXQgdGhlIHN0YXRpYyBJUCBhZGRyZXNzIGFuZCBtYXNrCiAgICAgICAgICAgIHJvdXRlczoKICAgICAgICAgICAgICAgIC0gdG86IGRlZmF1bHQKICAgICAgICAgICAgICAgICAgdmlhOiAxMC4xOTkuMTE4LjI1MyAjIENvbmZpZ3VyZSBnYXRld2F5CiAgICAgICAgICAgIG5hbWVzZXJ2ZXJzOgogICAgICAgICAgICAgIGFkZHJlc3NlczogWzEwLjE0Mi43LjEsIDEwLjEzMi43LjFdICMgUHJvdmlkZSB0aGUgRE5TIHNlcnZlciBhZGRyZXNzLiBTZXBhcmF0ZSBtdWxpdHBsZSBETlMgc2VydmVyIGFkZHJlc3NlcyB3aXRoIGNvbW1hcy4KIApydW5jbWQ6CiAgLSBuZXRwbGFuIGFwcGx5

which corresponds to the following script in plain-text format:

#cloud-config
write_files:
- path: /opt/dlvm/dl_app.sh
  permissions: '0755'
  content: |
    #!/bin/bash
    docker run -d nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8

manage_etc_hosts: true
 
write_files:
  - path: /etc/netplan/50-cloud-init.yaml
    permissions: '0600'
    content: |
      network:
        version: 2
        renderer: networkd
        ethernets:
          ens33:
            dhcp4: false # disable DHCP4
            addresses: [10.199.118.245/25]  # Set the static IP address and mask
            routes:
                - to: default
                  via: 10.199.118.253 # Configure gateway
            nameservers:
              addresses: [10.142.7.1, 10.132.7.1] # Provide the DNS server address. Separate mulitple DNS server addresses with commas.
 
runcmd:
  - netplan apply

Configure a Deep Learning VM with a Proxy Server

To connect your deep learning VM to the Internet in a disconnected environment where Internet access is over a proxy server, you must provide the proxy server details in the config.json file in the virtual machine.

Procedure

  1. Create a JSON file with the properties for proxy server.
    Proxy server that does not require authentication 
    {  
  "http_proxy": "protocol://ip-address-or-fqdn:port",
  "https_proxy": "protocol://ip-address-or-fqdn:port"
}
    Proxy server that requires authentication 
    {  
  "http_proxy": "protocol://username:password@ip-address-or-fqdn:port",
  "https_proxy": "protocol://username:password@ip-address-or-fqdn:port"
}

    where:

    • protocol is the communication protocol used by the proxy server, such as http or https.
    • username and password are the credentials for authentication to the proxy server. If the proxy server does not require authentication, skip these parameters.
    • ip-address-or-fqdn: The IP address or host name of the proxy server.
    • port: The port number on which the proxy server is listening for incoming requests.
  2. Encode the resulting JSON code in base64 format.
  3. When you deploy the the deep learning VM image, add the encoded value to the config-json OVF property.