This topic describes how to configure Kubernetes clusters with BOSH VM extensions using VMware Tanzu Kubernetes Grid Integrated Edition (TKGI).
BOSH VM extensions are VM configurations stored in the BOSH cloud config. BOSH VM extensions allow you to specify IaaS-specific configurations for your VMs such as custom security group and load balancer configurations.
TKGI supports configuring Kubernetes clusters using BOSH VM extensions. You can use BOSH VM extensions to configure TKGI-provisioned Linux and Windows clusters.
For information about BOSH VM Extensions, see VM Extensions Block in Usage in the Cloud Foundry BOSH documentation.
Warning: Configure VM Extensions only if you are already familiar with BOSH VM Extensions. If you use VM extensions, you might accidentally override more settings than you intend. For example, if you use a VM extension to add tags, the default tags are removed from all instance groups.
To create a new Kubernetes cluster configured with VM extensions:
To create a cluster using a VM extensions file:
tkgi create-cluster CLUSTER-NAME --config-file CONFIG-FILENAME
Where:
CLUSTER-NAME
is the name of the cluster to create.CONFIG-FILENAME
is the name of the VM extension configuration file created above.To configure an existing Kubernetes cluster with VM extensions:
To modify a cluster using a VM extensions file:
tkgi update-cluster CLUSTER-NAME --config-file CONFIG-FILENAME
Where:
CLUSTER-NAME
is the name of the cluster to modify.CONFIG-FILENAME
is the name of the VM extension configuration file created above.WARNING: Update the configuration file only on a TKGI cluster that has been upgraded to the current TKGI version. For more information, see Tasks Supported Following a TKGI Control Plane Upgrade in About Tanzu Kubernetes Grid Integrated Edition Upgrades.
To create a VM extensions configuration file in JSON format:
Create a new configuration file containing the following content:
vSphere VM Extension Configuration File Template:
{
"instance_groups":[
{
"name":"master",
"vm_extension":{
"vmx_options":{
"disk.enableUUID": "1"
},
"nsxt":{
"ns_groups":[NSX-NS-GROUPS]
}
}
},
{
"name":"WORKER-NAME",
"vm_extension":{
"vmx_options":{
"disk.enableUUID": "1"
}
}
},
{
"name":"NODE-POOL-NAME",
"vm_extension":{
"vmx_options":{
"disk.enableUUID": "1"
}
}
}
]
}
Public Cloud VM Extension Configuration File Template:
{
"instance_groups":[
{
"name":"master",
"vm_extension":{
"vmx_options":{
"disk.enableUUID": "1"
}
}
},
{
"name":"WORKER-NAME",
"vm_extension":{
"vmx_options":{
"disk.enableUUID": "1"
}
}
}
]
}
Where:
NSX-NS-GROUPS
(vSphere with VMware NSX Only) is a comma-separated list of NS Group names that the instances belong to.NODE-POOL-NAME
(vSphere Only) is the instance group name for the node pool VM extensions. The node pool must be configured in the cluster’s compute profile. For Linux clusters, the node pool name must be prefixed worker-
and for Windows Workers windows-worker-
. For example: worker-tiny-1
and windows-worker-tiny-1
.WORKER-NAME
is worker
for Linux clusters and windows-worker
for Windows Worker clusters.Update the vm_extension
and vmx_options
parameters of each instance group with the custom BOSH VM extensions and VMX options to apply to the Kubernetes cluster.
Warning: Volumes will not attach to your nodes if you do not include the disk.enableUUID
vmx_options
parameter in your configuration.
For example:
{
"instance_groups":[
{
"name":"master",
"vm_extension":{
"cpu_hot_add_enabled": "true",
"nsxt":{
"ns_groups":["master-2"]
},
"vmx_options":{
"disk.enableUUID": "1"
}
}
},
{
"name":"windows-worker",
"vm_extension":{
"cpu_hot_add_enabled": "true",
"vmx_options":{
"disk.enableUUID": "1"
}
}
},
{
"name":"windows-worker-tiny-1",
"vm_extension":{
"cpu_hot_add_enabled": "true",
"vmx_options":{
"ctkEnabled": "TRUE",
"disk.enableUUID": "1"
}
}
}
]
}
The supported BOSH VM extensions are specific to each IaaS. For the names of the cloud properties you can use in your VM extension configurations, see the following topics in Usage in the Cloud Foundry BOSH documentation:
To remove a VM extensions instance group from a cluster:
Create a VM extension configuration file containing the following:
{
"instance_groups":[
{
"name":"INSTANCE-GROUP",
"vm_extension":{
}
}
]
}
Where:
INSTANCE-GROUP
is the name of the instance group VM extensions to be removed. Specify either master
, worker
, windows-worker
or a node pool VM extension instance group name.To remove the VM extensions instance group using the CLI:
tkgi update-cluster CLUSTER-NAME --config-file CONFIG-FILENAME
Where:
CLUSTER-NAME
is the name of the cluster to remove VM extensions from.CONFIG-FILENAME
is the name of the VM extension configuration file created above.WARNING: Update the configuration file only on a TKGI cluster that has been upgraded to the current TKGI version. For more information, see Tasks Supported Following a TKGI Control Plane Upgrade in About Tanzu Kubernetes Grid Integrated Edition Upgrades.