Windows Custom Machine Images

If you are building an application based on legacy Windows components (such as .NET Framework) and want to deploy Windows containers on vSphere, follow this procedure to build a Windows custom machine image. It is divided into the following sections:

Windows Image Prerequisites lists the environment and files you need to successfully build and use a Windows image.
Build a Windows Image shows you how to use Kubernetes Image Builder and Docker to customize and build a Windows image.
Use a Windows Image for a Workload Cluster shows you how to deploy a workload cluster that uses your Windows image as a template.

Note
Windows workload clusters running on Tanzu Kubernetes Grid v1.6 cannot be upgraded to v2.1. For a workaround, see the TKG v2.1 Release Notes.

Windows Image Prerequisites

To build a custom machine image, you need:

vSphere 6.7 Update 3 or greater.
A Linux workstation.
Docker Desktop. To install, see the Docker product download page. Use apt instead of snap to install Docker from a CLI.
The Tanzu CLI. To install, see Download and Unpack the Tanzu CLI and kubectl in Install the Tanzu CLI and Other Tools for Use with Standalone Management Clusters.
A TKG v2.1 standalone management cluster.
- Create a management cluster by following the procedure in Deploy Management Clusters with the Installer Interface.
A recent (newer than April 2021) Windows Server 2019 ISO image. Download through your Microsoft Developer Network (MSDN) or Volume Licensing (VL) account. The use of evaluation media is not supported or recommended.
The latest VMware Tools Windows ISO image. Download from VMware Tools.
A datastore on your vCenter that can accommodate your custom Windows VM template, which can have a starting size greater than 10GB (thin provisioned).

Build a Windows Image

Import the Ubuntu or Photon OVA image into vCenter to use as a template for your custom image by following these steps:
1. Go to https://customerconnect.vmware.com and log in with your VMware Customer Connect credentials.
2. Go to the Tanzu Kubernetes Grid downloads page.
3. In the VMware Tanzu Kubernetes Grid row, click Go to Downloads.
4. In the Select Version drop-down, select 2.1.
5. Under Product Downloads, scroll to the section labeled Kubernetes OVAs for VMware Tanzu Kubernetes Grid 2.1, locate Ubuntu 2004 Kubernetes v1.24.10 OVA or Photon v3 Kubernetes v1.24.10 OVA, and click Download Now.
6. Log in to vCenter.
7. Right-click your host or cluster and click Deploy OVF Template… to import the image.
8. Right-click the imported image, hover over Template, and click Convert to Template.
9. To ensure the template is ready to use, select your host or cluster, select the VMs tab, then select VM Templates to see the template listed.
Import the Windows Server 2019 ISO and the VMware Tools Windows ISO images into your datastore by following these steps:
1. Log in to vCenter.
2. Navigate to your datastore and click the Files tab.
3. Open the iso folder or create one if none exists by clicking New Folder.
4. Upload both ISO files to the iso folder.

Create a YAML file named builder.yaml with the following configuration:

apiVersion: v1
kind: Namespace
metadata:
 name: imagebuilder
---
apiVersion: v1
kind: Service
metadata:
 name: imagebuilder-wrs
 namespace: imagebuilder
spec:
 selector:
   app: image-builder-resource-kit
 type: NodePort
 ports:
 - port: 3000
   targetPort: 3000
   nodePort: 30008
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: image-builder-resource-kit
 namespace: imagebuilder
spec:
 selector:
   matchLabels:
     app: image-builder-resource-kit
 template:
   metadata:
     labels:
       app: image-builder-resource-kit
   spec:
     nodeSelector:
       kubernetes.io/os: linux
     containers:
     - name: windows-imagebuilder-resourcekit
       image: projects.registry.vmware.com/tkg/windows-resource-bundle:v1.24.10_vmware.1-tkg.2
       imagePullPolicy: Always
       ports:
         - containerPort: 3000

Connect the Kubernetes CLI to your management cluster by running:
```
kubectl config use-context MY-MGMT-CLUSTER-admin@MY-MGMT-CLUSTER
```
Where MY-MGMT-CLUSTER is the name of your management cluster.
Apply the builder.yaml configuration file.
```
kubectl apply -f ./builder.yaml
```
This deploys a container that allows your machine to retrieve the files it needs from an HTTP location during the image build process. The output is similar to:
```
namespace/imagebuilder created
service/imagebuilder-wrs created
deployment.apps/image-builder-resource-kit created
```

To ensure the container is running, run:

kubectl get pods -n imagebuilder

The output is similar to:

NAME                                          READY   STATUS    RESTARTS   AGE
image-builder-resource-kit-756ccdc4ff-rcxpq   1/1     Running   0          4m16s

Retrieve a control plane IP address for the management cluster:
1. List the cluster’s nodes, with wide output:
```
kubectl get nodes -o wide
```
2. From the output, record the INTERNAL-IP value of the node with ROLE listed as control-plane.

Create a JSON file in an empty folder named windows.json with the following configuration:

{
  "additional_executables_destination_path": "C:\\ProgramData\\Temp",
  "additional_executables_list": "http://CONTROLPLANE-IP:30008/files/antrea-windows/antrea-windows-advanced.zip,http://CONTROLPLANE-IP:30008/files/kubernetes/kube-proxy.exe",
  "additional_executables": "true",
  "additional_prepull_images": "mcr.microsoft.com/windows/servercore:ltsc2019",
  "build_version": "windows-2019-kube-v1.24.10",
  "cloudbase_init_url": "http://CONTROLPLANE-IP:30008/files/cloudbase_init/CloudbaseInitSetup_1_1_4_x64.msi",
  "cluster": "VSPHERE-CLUSTER-NAME",
  "containerd_sha256_windows": "d29f5276584e869a5933db668fd6f17b7417c48ac04dd1c2a2c7f412f948f89c",
  "containerd_url": "http://CONTROLPLANE-IP:30008/files/containerd/cri-containerd-v1.6.6+vmware.3.windows-amd64.tar",
  "containerd_version": "v1.6.6",
  "convert_to_template": "true",
  "create_snapshot": "false",
  "datacenter": "DATACENTER-NAME",
  "datastore": "DATASTORE-NAME",
  "disable_hypervisor": "false",
  "disk_size": "40960",
  "goss_inspect_mode": "true",
  "goss_url": "http://CONTROLPLANE-IP:30008/files/goss/goss-alpha-windows-amd64.exe",
  "insecure_connection": "true",
  "kubernetes_base_url": "http://CONTROLPLANE-IP:30008/files/kubernetes/",
  "kubernetes_semver": "v1.24.10+vmware.1",
  "kubernetes_series": "v1.24.10",
  "linked_clone": "false",
  "load_additional_components": "true",
  "netbios_host_name_compatibility": "false",
  "network": "NETWORK",
  "nssm_url": "http://CONTROLPLANE-IP:30008/files/nssm/nssm.exe",
  "os_iso_path": "OS-ISO-PATH",
  "password": "VCENTER-PASSWORD",
  "pause_image": "mcr.microsoft.com/oss/kubernetes/pause:3.6",
  "prepull": "false",
  "resource_pool": "",
  "runtime": "containerd",
  "template": "",
  "unattend_timezone": "WINDOWS-TIMEZONE",
  "username": "VCENTER-USERNAME",
  "vcenter_server": "VCENTER-IP",
  "vmtools_iso_path": "VMTOOLS-ISO-PATH",
  "windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups",
  "windows_updates_kbs": "",
  "wins_url": "http://CONTROLPLANE-IP:30008/files/wins/wins.exe",
  "wins_version": "v0.4.11"
}

Where:

WINDOWS-TIMEZONE is the Windows timezone for your environment. For example, GMT Standard Time.
VSPHERE-CLUSTER-NAME is the name of your vSphere cluster. For example, VSAN-Cluster.
VCENTER-PASSWORD is your vCenter password.
VCENTER-USERNAME is your vCenter username.
DATASTORE-NAME is the name of your datastore as it appears in vCenter. For example vsanDatastore.
DATACENTER-NAME is the name of your data center as it appears in vCenter. For example VSAN-DC.
VMTOOLS-ISO-PATH is the path to the VMware Tools ISO file structured like this: [DATASTORE-NAME] iso/VMware-tools-windows-11.2.5-17337674.iso.
NETWORK the name of a network or port group as it appears in vCenter Menu > Networking > Networks. For example, VM Network.
OS-ISO-PATH is the path to the Windows Server 2019 ISO file structured like this: [DATASTORE-NAME] iso/en-us_windows_server_2019_updated_aug_2022_x64_dvd_a6431a28.iso.
VCENTER-IP is the IP address or FQDN of your vCenter server.
CONTROLPLANE-IP is the management cluster IP address retrieved above.
CONTAINERD-URL and CONTAINERD-SHA are the containerd.path and containerd.sha256 values retrieved above.

Query the Windows resource bundle on the control plane’s nodePort endpoint and verify the existent file path:

curl http://CONTROLPLANE-IP:30008
Add the XML file that contains the Windows settings by following these steps:
1. Go to the autounattend.xml file on VMware {code} Sample Exchange.
2. Select Download.
3. If you are using the Windows Server 2019 evaluation version, remove <ProductKey>...</ProductKey>.
4. Name the file autounattend.xml.
5. Save the file in the same folder as the windows.json file.

From the folder containing your windows.json and autounattend.xml file, build the custom image by running:

docker run -it --rm --mount type=bind,source=$(pwd)/windows.json,target=/windows.json --mount type=bind,source=$(pwd)/autounattend.xml,target=/home/imagebuilder/packer/ova/windows/windows-2019/autounattend.xml -e PACKER_VAR_FILES="/windows.json" -e IB_OVFTOOL=1 -e IB_OVFTOOL_ARGS='--skipManifestCheck' -e PACKER_FLAGS='-force -on-error=ask' -t projects.registry.vmware.com/tkg/image-builder:v0.1.13_vmware.2 build-node-ova-vsphere-windows-2019

If you see an error or the build hangs, reference the Packer logs on your workstation to troubleshoot issues in the windows.json configuration file. Additionally you may add -e PACKER_LOG=1 to the command line above to receive more verbose logging on your console.

To ensure the Windows image is ready to use, select your host or cluster in vCenter, select the VMs tab, then select VM Templates to see the Windows image listed.

Use a Windows Image for a Workload Cluster

With this procedure, you create a configuration file for your Windows workload cluster, reference the Windows image in the configuration file, then use the Tanzu CLI to create the workload cluster.

Copy your management cluster configuration file and save it with a new name by following the procedure in vSphere with Standalone Management Cluster Configuration Files.
In the new configuration file, add or modify the following:
```
IS_WINDOWS_WORKLOAD_CLUSTER: "true"
```
Deploy a workload cluster as described in Create Workload Clusters.