If you are building an application based on legacy Windows components (such as .NET Framework) and want to deploy Windows containers on vSphere, follow this procedure to build a Windows custom machine image. It is divided into the following sections:
Note: Windows workload clusters running on Tanzu Kubernetes Grid v1.4 cannot be upgraded to v1.5. For a workaround, see the TKG v1.5 Release Notes.
To build a custom machine image, you need:
kubectl in Install the Tanzu CLI and Other Tools.Import the Ubuntu or Photon image into vCenter to use as a template for your custom image by following these steps:
Create a management cluster by following the procedure in Deploy Management Clusters with the Installer Interface. When prompted, use the Ubuntu or Photon image template you added in the previous step. For more information, see How Base OS Image Choices are Generated.
Import the Windows Server 2019 ISO and the VMware Tools Windows ISO images into your datastore by following these steps:
iso folder or create one if none exists by clicking New Folder.iso folder.Create a YAML file named builder.yaml with the following configuration:
apiVersion: v1
kind: Namespace
metadata:
name: imagebuilder
---
apiVersion: v1
kind: Service
metadata:
name: imagebuilder-wrs
namespace: imagebuilder
spec:
selector:
app: image-builder-resource-kit
type: NodePort
ports:
- port: 3000
targetPort: 3000
nodePort: 30008
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: image-builder-resource-kit
namespace: imagebuilder
spec:
selector:
matchLabels:
app: image-builder-resource-kit
template:
metadata:
labels:
app: image-builder-resource-kit
spec:
nodeSelector:
kubernetes.io/os: linux
containers:
- name: windows-imagebuilder-resourcekit
image: projects.registry.vmware.com/tkg/windows-resource-bundle:v1.22.9_vmware.1-tkg.1
imagePullPolicy: Always
ports:
- containerPort: 3000
Connect the Kubernetes CLI to your management cluster by running:
kubectl config use-context MY-MGMT-CLUSTER-admin@MY-MGMT-CLUSTER
Where MY-MGMT-CLUSTER is the name of your management cluster.
Apply the builder.yaml configuration file.
kubectl apply -f ./builder.yaml
This deploys a container that allows your machine to retrieve the files it needs from an HTTP location during the image build process. The output is similar to:
namespace/imagebuilder created
service/imagebuilder-wrs created
deployment.apps/image-builder-resource-kit created
To ensure the container is running, run:
kubectl get pods -n imagebuilder
The output is similar to:
NAME READY STATUS RESTARTS AGE
image-builder-resource-kit-756ccdc4ff-rcxpq 1/1 Running 0 4m16s
Retrieve a control plane IP address for the management cluster:
Set the kubectl context to the management cluster:
kubectl config use-context MGMT-CLUSTER-NAME-admin@MGMT-CLUSTER-NAME
Where MGMT-CLUSTER-NAME is the name of the cluster.
List the cluster’s nodes, with wide output:
kubectl get nodes -o wide
From the output, record the INTERNAL-IP value of the node with ROLE listed as control-plane,master.
Retrieve the containerd component’s URL and SHA:
Query the control plane’s nodePort endpoint:
curl http://CONTROLPLANE-IP:30008
From the output, record the containerd.path and containerd.sha256 values. The containerd.path value ends with something like containerd/cri-containerd-v1.5.9+vmware.2.windows-amd64.tar.
Create a JSON file in an empty folder named windows.json with the following configuration:
{
"unattend_timezone": "WINDOWS-TIMEZONE",
"windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups",
"windows_updates_kbs": "",
"kubernetes_semver": "v1.22.9",
"cluster": "VSPHERE-CLUSTER-NAME",
"template": "",
"password": "VCENTER-PASSWORD",
"folder": "",
"runtime": "containerd",
"username": "VCENTER-USERNAME",
"datastore": "DATASTORE-NAME",
"datacenter": "DATACENTER-NAME",
"convert_to_template": "true",
"vmtools_iso_path": "VMTOOLS-ISO-PATH",
"insecure_connection": "true",
"disable_hypervisor": "false",
"network": "NETWORK",
"linked_clone": "false",
"os_iso_path": "OS-ISO-PATH",
"resource_pool": "",
"vcenter_server": "VCENTER-IP",
"create_snapshot": "false",
"netbios_host_name_compatibility": "false",
"nssm_url": "https://upstreamartifacts.azureedge.net/nssm/nssm.exe",
"kubernetes_base_url": "http://CONTROLPLANE-IP:30008/files/kubernetes/",
"containerd_url": "CONTAINERD-URL",
"containerd_sha256_windows": "CONTAINERD-SHA",
"pause_image": "mcr.microsoft.com/oss/kubernetes/pause:3.5",
"prepull": "false",
"additional_prepull_images": "mcr.microsoft.com/windows/servercore:ltsc2019",
"additional_download_files": "",
"additional_executables": "true",
"additional_executables_destination_path": "c:/k/antrea/",
"additional_executables_list": "http://CONTROLPLANE-IP:30008/files/antrea-windows/antrea-windows-advanced.zip",
"load_additional_components": "true"
}
Where:
WINDOWS-TIMEZONE is the Windows timezone for your environment. For example, GMT Standard Time.VSPHERE-CLUSTER-NAME is the name of your vSphere cluster. For example, VSAN-Cluster.VCENTER-PASSWORD is your vCenter password.VCENTER-USERNAME is your vCenter username.DATASTORE-NAME is the name of your datastore as it appears in vCenter. For example vsanDatastore.DATACENTER-NAME is the name of your data center as it appears in vCenter. For example VSAN-DC.VMTOOLS-ISO-PATH is the path to the VMware Tools ISO file structured like this: [DATASTORE-NAME] iso/VMware-tools-windows-11.2.5-17337674.iso.NETWORK the name of a network or port group as it appears in vCenter Menu > Networking > Networks. For example, VM Network.OS-ISO-PATH is the path to the Windows Server 2019 ISO file structured like this: [DATASTORE-NAME] iso/en-us_windows_server_2019_updated_aug_2021_x64_dvd_a6431a28.iso.VCENTER-IP is the IP address or FQDN of your vCenter server.CONTROLPLANE-IP is the management cluster IP address retrieved above.CONTAINERD-URL and CONTAINERD-SHA are the containerd.path and containerd.sha256 values retrieved above.Add the XML file that contains the Windows settings by following these steps:
autounattend.xml file on VMware {code} Sample Exchange.<ProductKey>...</ProductKey>.autounattend.xml.windows.json file.From the folder containing your windows.json and autounattend.xml file, build the custom image by running:
docker run -it --rm --mount type=bind,source=$(pwd)/windows.json,target=/windows.json --mount type=bind,source=$(pwd)/autounattend.xml,target=/home/imagebuilder/packer/ova/windows/windows-2019/autounattend.xml -e PACKER_VAR_FILES="/windows.json" -e IB_OVFTOOL=1 -e IB_OVFTOOL_ARGS='--skipManifestCheck' -e PACKER_FLAGS='-force -on-error=ask' -t projects.registry.vmware.com/tkg/image-builder:v0.1.11_vmware.3 build-node-ova-vsphere-windows-2019
If you see an error or the build hangs, reference the Packer logs on your workstation to troubleshoot issues in the windows.json configuration file. Additionally you may add -e PACKER_LOG=1 to the command line above to receive more verbose logging on your console.
To ensure the Windows image is ready to use, select your host or cluster in vCenter, select the VMs tab, then select VM Templates to see the Windows image listed.
With this procedure, you create a configuration file for your Windows workload cluster, reference the Windows image in the configuration file, then use the Tanzu CLI to create the workload cluster.
Copy your management cluster configuration file and save it with a new name by following the procedure in Create a Tanzu Kubernetes Cluster Configuration File.
In the new configuration file, add or modify the following:
IS_WINDOWS_WORKLOAD_CLUSTER: "true"
VSPHERE_WINDOWS_TEMPLATE: windows-2019-kube-v1.22.9
ENABLE_MHC: "false"
Deploy a workload cluster as described in Deploy Tanzu Kubernetes Clusters to vSphere.
