Windows Custom Machine Images

This topic describes how to create a Windows custom machine image that Tanzu Kubernetes Grid (TKG) can use to create and manage workload clusters that run Windows-based workloads on Windows worker nodes.

If you are building an application based on legacy Windows components (such as .NET Framework) and want to deploy Windows containers on vSphere, follow this procedure to build a Windows custom machine image. It is divided into the following sections:

Note

VMware does not support TKG workload clusters with Windows worker nodes in proxied or air-gapped environments.

Windows Image Prerequisites

To build a custom machine image, you need:

  • vSphere 7 or greater.
  • A Linux workstation.

    Note

    Due to an issue with the open-source packer utility used by Kubernetes Image Builder, you cannot build a Windows machine image on a MacOS machine.

  • Docker Desktop. To install, see the Docker product download page. Use apt instead of snap to install Docker from a CLI.
  • The Tanzu CLI. To install, see Install the Tanzu CLI in Install the Tanzu CLI and Kubernetes CLI for Use with Standalone Management Clusters.
  • A TKG v2.4 standalone management cluster.
  • A recent (newer than April 2021) Windows Server 2019 ISO image. Download through your Microsoft Developer Network (MSDN) or Volume Licensing (VL) account. The use of evaluation media is not supported or recommended.
  • The latest VMware Tools Windows ISO image. Download from VMware Tools.
  • A datastore on your vCenter that can accommodate your custom Windows VM template, which can have a starting size greater than 10GB (thin provisioned).

Build a Windows Image

  1. Import the Ubuntu or Photon OVA image into vCenter to use as a template for your custom image by following these steps:

    1. Go to the Broadcom Support Portal and log in with your VMware customer credentials.
    2. Go to the Tanzu Kubernetes Grid downloads page.
    3. In the version drop-down, select 2.4.1.
    4. Locate Ubuntu 2004 Kubernetes v1.27.5 OVA or Photon v3 Kubernetes v1.27.5 OVA, and click Download Now.
    5. Log in to vCenter.
    6. Right-click your host or cluster and click Deploy OVF Template… to import the image.
    7. Right-click the imported image, hover over Template, and click Convert to Template.
    8. To ensure the template is ready to use, select your host or cluster, select the VMs tab, then select VM Templates to see the template listed.
  2. Import the Windows Server 2019 ISO and the VMware Tools Windows ISO images into your datastore by following these steps:

    1. Log in to vCenter.
    2. Navigate to your datastore and click the Files tab.
    3. Open the iso folder or create one if none exists by clicking New Folder.
    4. Upload both ISO files to the iso folder.
  3. Create a YAML file named builder.yaml with the following configuration:

    apiVersion: v1
    kind: Namespace
    metadata:
     name: imagebuilder
    ---
    apiVersion: v1
    kind: Service
    metadata:
     name: imagebuilder-wrs
     namespace: imagebuilder
    spec:
     selector:
       app: image-builder-resource-kit
     type: NodePort
     ports:
     - port: 3000
       targetPort: 3000
       nodePort: 30008
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
     name: image-builder-resource-kit
     namespace: imagebuilder
    spec:
     selector:
       matchLabels:
         app: image-builder-resource-kit
     template:
       metadata:
         labels:
           app: image-builder-resource-kit
       spec:
         nodeSelector:
           kubernetes.io/os: linux
         containers:
         - name: windows-imagebuilder-resourcekit
           image: projects.registry.vmware.com/tkg/windows-resource-bundle:v1.27.5_vmware.1-tkg.1
           imagePullPolicy: Always
           ports:
             - containerPort: 3000
    

    The windows-resource-bundle container image has the Kubernetes Windows binaries that Image Builder needs to build a Windows OVA.

  4. Connect the Kubernetes CLI to your management cluster by running:

    kubectl config use-context MY-MGMT-CLUSTER-admin@MY-MGMT-CLUSTER
    

    Where MY-MGMT-CLUSTER is the name of your management cluster.

  5. Apply the builder.yaml configuration file.

    kubectl apply -f ./builder.yaml
    

    This deploys a container that allows your machine to retrieve the files it needs from an HTTP location during the image build process. The output is similar to:

    namespace/imagebuilder created
    service/imagebuilder-wrs created
    deployment.apps/image-builder-resource-kit created
    
  6. To ensure the container is running, run:

    kubectl get pods -n imagebuilder
    

    The output is similar to:

    NAME                                          READY   STATUS    RESTARTS   AGE
    image-builder-resource-kit-756ccdc4ff-rcxpq   1/1     Running   0          4m16s
    
  7. Retrieve a control plane IP address for the management cluster:

    1. List the cluster’s nodes, with wide output:

      kubectl get nodes -o wide
      
    2. From the output, record the INTERNAL-IP value of the node with ROLE listed as control-plane.

  8. Create a JSON file in an empty folder named windows.json with the following configuration:

    {
      "additional_executables_destination_path": "C:\\ProgramData\\Temp",
      "additional_executables_list": "http://CONTROLPLANE-IP:30008/files/antrea-windows/antrea-windows-advanced.zip",
      "additional_executables": "true",
      "additional_prepull_images": "mcr.microsoft.com/windows/servercore:ltsc2019",
      "build_version": "windows-2019-kube-v1.27.5",
      "cloudbase_init_url": "http://CONTROLPLANE-IP:30008/files/cloudbase_init/CloudbaseInitSetup_1_1_4_x64.msi",
      "cluster": "VSPHERE-CLUSTER-NAME",
      "containerd_sha256_windows": "2e0332aa57ebcb6c839a8ec807780d662973a15754573630bea249760cdccf2a",
      "containerd_url": "http://CONTROLPLANE-IP:30008/files/containerd/cri-containerd-v1.6.18+vmware.1.windows-amd64.tar",
      "containerd_version": "v1.6.18",
      "convert_to_template": "true",
      "create_snapshot": "false",
      "datacenter": "DATACENTER-NAME",
      "datastore": "DATASTORE-NAME",
      "disable_hypervisor": "false",
      "disk_size": "40960",
      "goss_inspect_mode": "true",
      "insecure_connection": "true",
      "kubernetes_base_url": "http://CONTROLPLANE-IP:30008/files/kubernetes/",
      "kubernetes_semver": "v1.27.5+vmware.1",
      "kubernetes_series": "v1.27.5",
      "linked_clone": "false",
      "load_additional_components": "true",
      "netbios_host_name_compatibility": "false",
      "network": "NETWORK",
      "nssm_url": "http://CONTROLPLANE-IP:30008/files/nssm/nssm.exe",
      "os_iso_path": "OS-ISO-PATH",
      "password": "VCENTER-PASSWORD",
      "pause_image": "mcr.microsoft.com/oss/kubernetes/pause:3.8",
      "prepull": "false",
      "resource_pool": "",
      "runtime": "containerd",
      "template": "",
      "unattend_timezone": "WINDOWS-TIMEZONE",
      "username": "VCENTER-USERNAME",
      "vcenter_server": "VCENTER-IP",
      "vmtools_iso_path": "VMTOOLS-ISO-PATH",
      "windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups",
      "windows_updates_kbs": "",
      "wins_url": ""
    }
    

    Where:

    • WINDOWS-TIMEZONE is the Windows timezone for your environment. For example, GMT Standard Time.
    • VSPHERE-CLUSTER-NAME is the name of your vSphere cluster. For example, VSAN-Cluster.
    • VCENTER-PASSWORD is your vCenter password.
    • VCENTER-USERNAME is your vCenter username.
    • DATASTORE-NAME is the name of your datastore as it appears in vCenter. For example vsanDatastore.
    • DATACENTER-NAME is the name of your data center as it appears in vCenter. For example VSAN-DC.
    • VMTOOLS-ISO-PATH is the path to the VMware Tools ISO file structured like this: [DATASTORE-NAME] iso/VMware-tools-windows-11.2.5-17337674.iso.
    • NETWORK the name of a network or port group as it appears in vCenter Menu > Networking > Networks. For example, VM Network.
    • OS-ISO-PATH is the path to the Windows Server 2019 ISO file structured like this: [DATASTORE-NAME] iso/en-us_windows_server_2019_updated_aug_2022_x64_dvd_a6431a28.iso.
    • VCENTER-IP is the IP address or FQDN of your vCenter server.
    • CONTROLPLANE-IP is the management cluster IP address retrieved above.
    • CONTAINERD-URL and CONTAINERD-SHA are the containerd.path and containerd.sha256 values retrieved above.
  9. Query the Windows resource bundle on the control plane’s nodePort endpoint and verify the existent file path:

    curl http://CONTROLPLANE-IP:30008

  10. Add the XML file that contains the Windows settings by following these steps:

    1. Go to the autounattend.xml file on Broadcom Communities Sample Exchange.
    2. Select Download.
    3. If you are using the Windows Server 2019 evaluation version, remove <ProductKey>...</ProductKey>.
    4. Name the file autounattend.xml.
    5. Save the file in the same folder as the windows.json file.
  11. From the folder containing your windows.json and autounattend.xml file, build the custom image by running:

    docker run -it --rm --mount type=bind,source=$(pwd)/windows.json,target=/windows.json --mount type=bind,source=$(pwd)/autounattend.xml,target=/home/imagebuilder/packer/ova/windows/windows-2019/autounattend.xml -e PACKER_VAR_FILES="/windows.json" -e IB_OVFTOOL=1 -e IB_OVFTOOL_ARGS='--skipManifestCheck' -e PACKER_FLAGS='-force -on-error=ask' -t projects.registry.vmware.com/tkg/image-builder:v0.1.14_vmware.1 build-node-ova-vsphere-windows-2019
    

    If you see an error or the build hangs, reference the Packer logs on your workstation to troubleshoot issues in the windows.json configuration file. Additionally you may add -e PACKER_LOG=1 to the command line above to receive more verbose logging on your console.

  12. To ensure the Windows image is ready to use, select your host or cluster in vCenter, select the VMs tab, then select VM Templates to see the Windows image listed.

Use a Windows Image for a Workload Cluster

With this procedure, you create a configuration file for your Windows workload cluster, reference the Windows image in the configuration file, then use the Tanzu CLI to create the workload cluster.

  1. Copy your management cluster configuration file and save it with a new name by following the procedure in vSphere with Standalone Management Cluster Configuration Files.

  2. In the new configuration file, add or modify the following:

    IS_WINDOWS_WORKLOAD_CLUSTER: "true"
    
  3. Deploy a workload cluster as described in Create Workload Clusters.

check-circle-line exclamation-circle-line close-line
Scroll to top icon