Windows Custom Machine Images

This topic describes how to create a Windows custom machine image that Tanzu Kubernetes Grid (TKG) can use to create and manage workload clusters that run Windows-based workloads on Windows worker nodes.

If you are building an application based on legacy Windows components (such as .NET Framework) and want to deploy Windows containers on vSphere, follow this procedure to build a Windows custom machine image. It is divided into the following sections:

Windows Image Prerequisites lists the environment and files you need to successfully build and use a Windows image.
Build a Windows Image shows you how to use Kubernetes Image Builder and Docker to customize and build a Windows image.
Use a Windows Image for a Workload Cluster shows you how to deploy a workload cluster that uses your Windows image as a template.

Note
VMware does not support TKG workload clusters with Windows worker nodes in proxied or air-gapped environments.

Windows Image Prerequisites

To build a custom machine image, you need:

vSphere 7 or greater.
A Linux workstation.

Note
Due to an issue with the open-source packer utility used by Kubernetes Image Builder, you cannot build a Windows machine image on a MacOS machine.
Docker Desktop. To install, see the Docker product download page. Use apt instead of snap to install Docker from a CLI.
The Tanzu CLI. To install, see Install the Tanzu CLI in Install the Tanzu CLI and Kubernetes CLI for Use with Standalone Management Clusters.
A TKG v2.3 standalone management cluster.
- Create a management cluster by following the procedure in Deploy Management Clusters with the Installer Interface.
A recent (newer than April 2021) Windows Server 2019 ISO image. Download through your Microsoft Developer Network (MSDN) or Volume Licensing (VL) account. The use of evaluation media is not supported or recommended.
The latest VMware Tools Windows ISO image. Download from VMware Tools.
A datastore on your vCenter that can accommodate your custom Windows VM template, which can have a starting size greater than 10GB (thin provisioned).

Build a Windows Image

Import the Ubuntu or Photon OVA image into vCenter to use as a template for your custom image by following these steps:
1. Go to the Broadcom Support Portal and log in with your VMware customer credentials.
2. Go to the Tanzu Kubernetes Grid downloads page.
3. In the version drop-down, select 2.3.1.
4. Locate Ubuntu 2004 Kubernetes v1.26.8 OVA or Photon v3 Kubernetes v1.26.8 OVA, and click the download button.
5. Log in to vCenter.
6. Right-click your host or cluster and click Deploy OVF Template… to import the image.
7. Right-click the imported image, hover over Template, and click Convert to Template.
8. To ensure the template is ready to use, select your host or cluster, select the VMs tab, then select VM Templates to see the template listed.
Import the Windows Server 2019 ISO and the VMware Tools Windows ISO images into your datastore by following these steps:
1. Log in to vCenter.
2. Navigate to your datastore and click the Files tab.
3. Open the iso folder or create one if none exists by clicking New Folder.
4. Upload both ISO files to the iso folder.

Create a YAML file named builder.yaml with the following configuration:

apiVersion: v1
kind: Namespace
metadata:
 name: imagebuilder
---
apiVersion: v1
kind: Service
metadata:
 name: imagebuilder-wrs
 namespace: imagebuilder
spec:
 selector:
   app: image-builder-resource-kit
 type: NodePort
 ports:
 - port: 3000
   targetPort: 3000
   nodePort: 30008
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: image-builder-resource-kit
 namespace: imagebuilder
spec:
 selector:
   matchLabels:
     app: image-builder-resource-kit
 template:
   metadata:
     labels:
       app: image-builder-resource-kit
   spec:
     nodeSelector:
       kubernetes.io/os: linux
     containers:
     - name: windows-imagebuilder-resourcekit
       image: projects.registry.vmware.com/tkg/windows-resource-bundle:v1.26.8_vmware.2-tkg.1
       imagePullPolicy: Always
       ports:
         - containerPort: 3000

The windows-resource-bundle container image has the Kubernetes Windows binaries that Image Builder needs to build a Windows OVA.

Connect the Kubernetes CLI to your management cluster by running:
```
kubectl config use-context MY-MGMT-CLUSTER-admin@MY-MGMT-CLUSTER
```
Where MY-MGMT-CLUSTER is the name of your management cluster.
Apply the builder.yaml configuration file.
```
kubectl apply -f ./builder.yaml
```
This deploys a container that allows your machine to retrieve the files it needs from an HTTP location during the image build process. The output is similar to:
```
namespace/imagebuilder created
service/imagebuilder-wrs created
deployment.apps/image-builder-resource-kit created
```

To ensure the container is running, run:

kubectl get pods -n imagebuilder

The output is similar to:

NAME                                          READY   STATUS    RESTARTS   AGE
image-builder-resource-kit-756ccdc4ff-rcxpq   1/1     Running   0          4m16s

Retrieve a control plane IP address for the management cluster:
1. List the cluster’s nodes, with wide output:
```
kubectl get nodes -o wide
```
2. From the output, record the INTERNAL-IP value of the node with ROLE listed as control-plane.

Create a JSON file in an empty folder named windows.json with the following configuration:

{
  "additional_executables_destination_path": "C:\\ProgramData\\Temp",
  "additional_executables_list": "http://CONTROLPLANE-IP:30008/files/antrea-windows/antrea-windows-advanced.zip",
  "additional_executables": "true",
  "additional_prepull_images": "mcr.microsoft.com/windows/servercore:ltsc2019",
  "build_version": "windows-2019-kube-v1.26.8",
  "cloudbase_init_url": "http://CONTROLPLANE-IP:30008/files/cloudbase_init/CloudbaseInitSetup_1_1_4_x64.msi",
  "cluster": "VSPHERE-CLUSTER-NAME",
  "containerd_sha256_windows": "2e0332aa57ebcb6c839a8ec807780d662973a15754573630bea249760cdccf2a",
  "containerd_url": "http://CONTROLPLANE-IP:30008/files/containerd/cri-containerd-v1.6.18+vmware.1.windows-amd64.tar",
  "containerd_version": "v1.6.18",
  "convert_to_template": "true",
  "create_snapshot": "false",
  "datacenter": "DATACENTER-NAME",
  "datastore": "DATASTORE-NAME",
  "disable_hypervisor": "false",
  "disk_size": "40960",
  "goss_inspect_mode": "true",
  "insecure_connection": "true",
  "kubernetes_base_url": "http://CONTROLPLANE-IP:30008/files/kubernetes/",
  "kubernetes_semver": "v1.26.8+vmware.2",
  "kubernetes_series": "v1.26.8",
  "linked_clone": "false",
  "load_additional_components": "true",
  "netbios_host_name_compatibility": "false",
  "network": "NETWORK",
  "nssm_url": "http://CONTROLPLANE-IP:30008/files/nssm/nssm.exe",
  "os_iso_path": "OS-ISO-PATH",
  "password": "VCENTER-PASSWORD",
  "pause_image": "mcr.microsoft.com/oss/kubernetes/pause:3.6",
  "prepull": "false",
  "resource_pool": "",
  "runtime": "containerd",
  "template": "",
  "unattend_timezone": "WINDOWS-TIMEZONE",
  "username": "VCENTER-USERNAME",
  "vcenter_server": "VCENTER-IP",
  "vmtools_iso_path": "VMTOOLS-ISO-PATH",
  "windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups",
  "windows_updates_kbs": "",
  "wins_url": ""
}

Where:

WINDOWS-TIMEZONE is the Windows timezone for your environment. For example, GMT Standard Time.
VSPHERE-CLUSTER-NAME is the name of your vSphere cluster. For example, VSAN-Cluster.
VCENTER-PASSWORD is your vCenter password.
VCENTER-USERNAME is your vCenter username.
DATASTORE-NAME is the name of your datastore as it appears in vCenter. For example vsanDatastore.
DATACENTER-NAME is the name of your data center as it appears in vCenter. For example VSAN-DC.
VMTOOLS-ISO-PATH is the path to the VMware Tools ISO file structured like this: [DATASTORE-NAME] iso/VMware-tools-windows-11.2.5-17337674.iso.
NETWORK the name of a network or port group as it appears in vCenter Menu > Networking > Networks. For example, VM Network.
OS-ISO-PATH is the path to the Windows Server 2019 ISO file structured like this: [DATASTORE-NAME] iso/en-us_windows_server_2019_updated_aug_2022_x64_dvd_a6431a28.iso.
VCENTER-IP is the IP address or FQDN of your vCenter server.
CONTROLPLANE-IP is the management cluster IP address retrieved above.
CONTAINERD-URL and CONTAINERD-SHA are the containerd.path and containerd.sha256 values retrieved above.

Query the Windows resource bundle on the control plane’s nodePort endpoint and verify the existent file path:

curl http://CONTROLPLANE-IP:30008
Add the XML file that contains the Windows settings by following these steps:
1. Go to the autounattend.xml file on Broadcom Communities Sample Exchange.
2. Select Download.
3. If you are using the Windows Server 2019 evaluation version, remove <ProductKey>...</ProductKey>.
4. Name the file autounattend.xml.
5. Save the file in the same folder as the windows.json file.

From the folder containing your windows.json and autounattend.xml file, build the custom image by running:

docker run -it --rm --mount type=bind,source=$(pwd)/windows.json,target=/windows.json --mount type=bind,source=$(pwd)/autounattend.xml,target=/home/imagebuilder/packer/ova/windows/windows-2019/autounattend.xml -e PACKER_VAR_FILES="/windows.json" -e IB_OVFTOOL=1 -e IB_OVFTOOL_ARGS='--skipManifestCheck' -e PACKER_FLAGS='-force -on-error=ask' -t projects.registry.vmware.com/tkg/image-builder:v0.1.14_vmware.1 build-node-ova-vsphere-windows-2019

If you see an error or the build hangs, reference the Packer logs on your workstation to troubleshoot issues in the windows.json configuration file. Additionally you may add -e PACKER_LOG=1 to the command line above to receive more verbose logging on your console.

To ensure the Windows image is ready to use, select your host or cluster in vCenter, select the VMs tab, then select VM Templates to see the Windows image listed.

Use a Windows Image for a Workload Cluster

With this procedure, you create a configuration file for your Windows workload cluster, reference the Windows image in the configuration file, then use the Tanzu CLI to create the workload cluster.

Copy your management cluster configuration file and save it with a new name by following the procedure in vSphere with Standalone Management Cluster Configuration Files.
In the new configuration file, add or modify the following:
```
IS_WINDOWS_WORKLOAD_CLUSTER: "true"
```
Deploy a workload cluster as described in Create Workload Clusters.