Using BOSH VM Extensions

This topic describes how to configure Kubernetes clusters with BOSH VM extensions using VMware Tanzu Kubernetes Grid Integrated Edition (TKGI).

Overview

BOSH VM extensions are VM configurations stored in the BOSH cloud config. BOSH VM extensions allow you to specify IaaS-specific configurations for your VMs such as custom security group and load balancer configurations.

TKGI supports configuring Kubernetes clusters using BOSH VM extensions. You can use BOSH VM extensions to configure TKGI-provisioned Linux and Windows clusters.

For information about BOSH VM Extensions, see VM Extensions Block in Usage in the Cloud Foundry BOSH documentation.

Warning: Configure VM Extensions only if you are already familiar with BOSH VM Extensions. If you use VM extensions, you might accidentally override more settings than you intend. For example, if you use a VM extension to add tags, the default tags are removed from all instance groups.

Create a Cluster Using VM Extensions

To create a new Kubernetes cluster configured with VM extensions:

Create a VM extensions configuration file. For more information, see Create a Cluster Configuration File for BOSH VM Extensions below.
To create a cluster using a VM extensions file:
```
tkgi create-cluster CLUSTER-NAME --config-file CONFIG-FILENAME
```
Where:
- CLUSTER-NAME is the name of the cluster to create.
- CONFIG-FILENAME is the name of the VM extension configuration file created above.

Configure a Cluster Using VM Extensions

To configure an existing Kubernetes cluster with VM extensions:

If you are updating a cluster that uses a public cloud CSI driver, see Limitations on Using a Public Cloud CSI Driver in Release Notes for additional requirements.
Create a VM extensions configuration file. For more information, see Create a Cluster Configuration File for BOSH VM Extensions below.
To modify a cluster using a VM extensions file:
```
tkgi update-cluster CLUSTER-NAME --config-file CONFIG-FILENAME
```
Where:
- CLUSTER-NAME is the name of the cluster to modify.
- CONFIG-FILENAME is the name of the VM extension configuration file created above.

WARNING: Update the configuration file only on a TKGI cluster that has been upgraded to the current TKGI version. For more information, see Tasks Supported Following a TKGI Control Plane Upgrade in About Tanzu Kubernetes Grid Integrated Edition Upgrades.

Create a Cluster Configuration File for BOSH VM Extensions

To create a VM extensions configuration file in JSON format:

Create a new configuration file containing the following content:

vSphere VM Extension Configuration File Template:

{
    "instance_groups":[
      {
        "name":"master",
        "vm_extension":{
          "vmx_options":{
            "disk.enableUUID": "1"
          },
          "nsxt":{ 
            "ns_groups":[NSX-NS-GROUPS] 
          }
        }
      },
      {
        "name":"WORKER-NAME",
        "vm_extension":{
          "vmx_options":{
            "disk.enableUUID": "1"
          }
        }
      },  	
      {
        "name":"NODE-POOL-NAME",
        "vm_extension":{
          "vmx_options":{
            "disk.enableUUID": "1"
          }
        }
      }
   ]
}

Public Cloud VM Extension Configuration File Template:

{
    "instance_groups":[
      {
        "name":"master",
        "vm_extension":{
          "vmx_options":{
            "disk.enableUUID": "1"
          }
        }
      },
      {
        "name":"WORKER-NAME",
        "vm_extension":{
          "vmx_options":{
            "disk.enableUUID": "1"
          }
        }
      }
   ]
}

Where:

NSX-NS-GROUPS (vSphere with VMware NSX Only) is a comma-separated list of NS Group names that the instances belong to.
NODE-POOL-NAME (vSphere Only) is the instance group name for the node pool VM extensions. The node pool must be configured in the cluster’s compute profile. For Linux clusters, the node pool name must be prefixed worker- and for Windows Workers windows-worker-. For example: worker-tiny-1 and windows-worker-tiny-1.
WORKER-NAME is worker for Linux clusters and windows-worker for Windows Worker clusters.

Update the vm_extension and vmx_options parameters of each instance group with the custom BOSH VM extensions and VMX options to apply to the Kubernetes cluster.

Warning: Volumes will not attach to your nodes if you do not include the disk.enableUUID vmx_options parameter in your configuration.

For example:

{
    "instance_groups":[
      {
        "name":"master",
        "vm_extension":{
          "cpu_hot_add_enabled": "true",
          "nsxt":{ 
            "ns_groups":["master-2"] 
          },
          "vmx_options":{
            "disk.enableUUID": "1"
          }
        }
      },
      {
        "name":"windows-worker",
        "vm_extension":{
          "cpu_hot_add_enabled": "true",
          "vmx_options":{
            "disk.enableUUID": "1"
          }
        }
      },  	
      {
        "name":"windows-worker-tiny-1",
        "vm_extension":{
          "cpu_hot_add_enabled": "true",
          "vmx_options":{
            "ctkEnabled": "TRUE",
            "disk.enableUUID": "1"
          }
        }
      }
   ]
}

The supported BOSH VM extensions are specific to each IaaS. For the names of the cloud properties you can use in your VM extension configurations, see the following topics in Usage in the Cloud Foundry BOSH documentation:

For AWS, see: VM Types / VM extensions.
For Azure, see: VM Types / VM extensions.
For GCP, see: VM Types / VM extensions.
For vSphere, see: VM Types / VM extensions.

Remove BOSH VM Extensions From a Cluster

To remove a VM extensions instance group from a cluster:

Create a VM extension configuration file containing the following:
```
{
    "instance_groups":[
      {
        "name":"INSTANCE-GROUP",
        "vm_extension":{
        }
      }
   ]
}
```
Where:
- INSTANCE-GROUP is the name of the instance group VM extensions to be removed. Specify either master, worker, windows-worker or a node pool VM extension instance group name.
To remove the VM extensions instance group using the CLI:
```
tkgi update-cluster CLUSTER-NAME --config-file CONFIG-FILENAME
```
Where:
- CLUSTER-NAME is the name of the cluster to remove VM extensions from.
- CONFIG-FILENAME is the name of the VM extension configuration file created above.