Windows Custom Machine Images

If you are building an application based on legacy Windows components (such as .NET Framework) and want to deploy Windows containers on vSphere, follow this procedure to build a Windows custom machine image. It is divided into the following sections:

Windows Image Prerequisites

To build a custom machine image, you need:

  • vSphere 6.7 Update 3 or greater.
  • A macOS or Linux workstation.
  • Docker Desktop. To install, see the Docker product download page.
  • The Tanzu CLI. To install, see Download and Unpack the Tanzu CLI and kubectl in Install the Tanzu CLI and Other Tools.
  • Tanzu Kubernetes Grid v1.5.x or greater.
  • A recent (newer than April 2021) Windows Server 2019 ISO image. Download through your Microsoft Developer Network (MSDN) or Volume Licensing (VL) account. The use of evaluation media is not supported or recommended.
  • The latest VMware Tools Windows ISO image. Download from VMware Tools.
  • A datastore on your vCenter that can accommodate your custom Windows VM template, which can have a starting size greater than 10GB (thin provisioned).

Build a Windows Image

  1. Import the Ubuntu 2004 Kubernetes v1.22.8 OVA image into vCenter to use as a template for your custom image by following these steps:

    1. Go to https://my.vmware.com and log in with your My VMware credentials.
    2. Go to the Tanzu Kubernetes Grid downloads page.
    3. In the VMware Tanzu Kubernetes Grid row, click Go to Downloads.
    4. In the Select Version drop-down, select 1.5.3.
    5. Under Product Downloads, scroll to the section labeled Kubernetes OVAs for VMware Tanzu Kubernetes Grid 1.5.3, locate Ubuntu 2004 Kubernetes v1.22.8 OVA, and click Download Now.
    6. Log in to vCenter.
    7. Right-click your host or cluster and click Deploy OVF Template… to import the image.
    8. Right-click the imported image, hover over Template, and click Convert to Template.
    9. To ensure the template is ready to use, select your host or cluster, select the VMs tab, then select VM Templates to see the template listed.
  2. Create a management cluster by following the procedure in Deploy Management Clusters with the Installer Interface. When prompted, use the Ubuntu 2004 Kubernetes v1.22.8 OVA image template you added in the previous step. For more information, see How Base OS Image Choices are Generated.

  3. Import the Windows Server 2019 ISO and the VMware Tools Windows ISO images into your datastore by following these steps:

    1. Log in to vCenter.
    2. Navigate to your datastore and click the Files tab.
    3. Open the iso folder or create one if none exists by clicking New Folder.
    4. Upload both ISO files to the iso folder.
  4. Create a YAML file named builder.yaml with the following configuration:

    apiVersion: v1
    kind: Namespace
    metadata:
     name: imagebuilder
    ---
    apiVersion: v1
    kind: Service
    metadata:
     name: imagebuilder-wrs
     namespace: imagebuilder
    spec:
     selector:
       app: image-builder-resource-kit
     type: NodePort
     ports:
     - port: 3000
       targetPort: 3000
       nodePort: 30008
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
     name: image-builder-resource-kit
     namespace: imagebuilder
    spec:
     selector:
       matchLabels:
         app: image-builder-resource-kit
     template:
       metadata:
         labels:
           app: image-builder-resource-kit
       spec:
         nodeSelector:
           kubernetes.io/os: linux
         containers:
         - name: windows-imagebuilder-resourcekit
           image: projects.registry.vmware.com/tkg/windows-resource-bundle:v1.22.8_vmware.1-tkg.1
           imagePullPolicy: Always
           ports:
             - containerPort: 3000
    
  5. Connect the Kubernetes CLI to your management cluster by running:

    kubectl config use-context MY-MGMT-CLUSTER-admin@MY-MGMT-CLUSTER
    

    Where MY-MGMT-CLUSTER is the name of your management cluster.

  6. Apply the builder.yaml configuration file.

    kubectl apply -f ./builder.yaml
    

    This deploys a container that allows your machine to retrieve the files it needs from an HTTP location during the image build process. The output is similar to:

    namespace/imagebuilder created
    service/imagebuilder-wrs created
    deployment.apps/image-builder-resource-kit created
    
  7. To ensure the container is running, run:

    kubectl get pods -n imagebuilder
    

    The output is similar to:

    NAME                                          READY   STATUS    RESTARTS   AGE
    image-builder-resource-kit-756ccdc4ff-rcxpq   1/1     Running   0          4m16s
    
  8. Retrieve a control plane IP address for the management cluster:

    1. Set the kubectl context to the management cluster:

      kubectl config use-context MGMT-CLUSTER-NAME-admin@MGMT-CLUSTER-NAME
      

      Where MGMT-CLUSTER-NAME is the name of the cluster.

    2. List the cluster’s nodes, with wide output:

      kubectl get nodes -o wide
      
    3. From the output, record the INTERNAL-IP value of the node with ROLE listed as control-plane,master.

  9. Retrieve the containerd component’s URL and SHA:

    1. Query the control plane’s nodePort endpoint:

      curl http://CONTROLPLANE-IP:30008
      
    2. From the output, record the containerd.path and containerd.sha256 values. The containerd.path value ends with something like containerd/cri-containerd-v1.5.9+vmware.2.windows-amd64.tar.

  10. Create a JSON file in an empty folder named windows.json with the following configuration:

    {
     "unattend_timezone": "WINDOWS-TIMEZONE",
     "windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups",
     "windows_updates_kbs": "",
     "kubernetes_semver": "v1.22.8",
     "cluster": "VSPHERE-CLUSTER-NAME",
     "template": "",
     "password": "VCENTER-PASSWORD",
     "folder": "",
     "runtime": "containerd",
     "username": "VCENTER-USERNAME",
     "datastore": "DATASTORE-NAME",
     "datacenter": "DATACENTER-NAME",
     "convert_to_template": "true",
     "vmtools_iso_path": "VMTOOLS-ISO-PATH",
     "insecure_connection": "true",
     "disable_hypervisor": "false",
     "network": "NETWORK",
     "linked_clone": "false",
     "os_iso_path": "OS-ISO-PATH",
     "resource_pool": "",
     "vcenter_server": "VCENTER-IP",
     "create_snapshot": "false",
     "netbios_host_name_compatibility": "false",
     "kubernetes_base_url": "http://CONTROLPLANE-IP:30008/files/kubernetes/",
     "containerd_url": "CONTAINERD-URL",
     "containerd_sha256_windows": "CONTAINERD-SHA",
     "pause_image": "mcr.microsoft.com/oss/kubernetes/pause:3.5",
     "prepull": "false",
     "additional_prepull_images": "mcr.microsoft.com/windows/servercore:ltsc2019",
     "additional_download_files": "",
     "additional_executables": "true",
     "additional_executables_destination_path": "c:/k/antrea/",
     "additional_executables_list": "http://CONTROLPLANE-IP:30008/files/antrea-windows/antrea-windows-advanced.zip",
     "load_additional_components": "true"
    }
    

    Where:

    • WINDOWS-TIMEZONE is the Windows timezone for your environment. For example, GMT Standard Time.
    • VSPHERE-CLUSTER-NAME is the name of your vSphere cluster. For example, VSAN-Cluster.
    • VCENTER-PASSWORD is your vCenter password.
    • VCENTER-USERNAME is your vCenter username.
    • DATASTORE-NAME is the name of your datastore as it appears in vCenter. For example vsanDatastore.
    • DATACENTER-NAME is the name of your data center as it appears in vCenter. For example VSAN-DC.
    • VMTOOLS-ISO-PATH is the path to the VMware Tools ISO file structured like this: [DATASTORE-NAME] iso/VMware-tools-windows-11.2.5-17337674.iso.
    • NETWORK the name of a network or port group as it appears in vCenter Menu > Networking > Networks. For example, VM Network.
    • OS-ISO-PATH is the path to the Windows Server 2019 ISO file structured like this: [DATASTORE-NAME] iso/en-us_windows_server_2019_updated_aug_2021_x64_dvd_a6431a28.iso.
    • VCENTER-IP is the IP address or FQDN of your vCenter server.
    • CONTROLPLANE-IP is the management cluster IP address retrieved above.
    • CONTAINERD-URL and CONTAINERD-SHA are the containerd.path and containerd.sha256 values retrieved above.
  11. Add the XML file that contains the Windows settings by following these steps:

    1. Go to the autounattend.xml file on VMware {code} Sample Exchange.
    2. Select Download.
    3. If you are using the Windows Server 2019 evaluation version, remove <ProductKey>...</ProductKey>.
    4. Name the file autounattend.xml.
    5. Save the file in the same folder as the windows.json file.
  12. From the folder containing your windows.json and autounattend.xml file, build the custom image by running:

    docker run -it --rm --mount type=bind,source=$(pwd)/windows.json,target=/windows.json --mount type=bind,source=$(pwd)/autounattend.xml,target=/home/imagebuilder/packer/ova/windows/windows-2019/autounattend.xml -e PACKER_VAR_FILES="/windows.json" -e IB_OVFTOOL=1 -e IB_OVFTOOL_ARGS='--skipManifestCheck' -e PACKER_FLAGS='-force -on-error=ask' -t projects.registry.vmware.com/tkg/image-builder:v0.1.11_vmware.3 build-node-ova-vsphere-windows-2019
    

    If you see an error or the build hangs, reference the Packer logs on your workstation to troubleshoot issues in the windows.json configuration file. Additionally you may add -e PACKER_LOG=1 to the command line above to receive more verbose logging on your console.

  13. To ensure the Windows image is ready to use, select your host or cluster in vCenter, select the VMs tab, then select VM Templates to see the Windows image listed.

Use a Windows Image for a Workload Cluster

With this procedure, you create a configuration file for your Windows workload cluster, reference the Windows image in the configuration file, then use the Tanzu CLI to create the workload cluster.

  1. Copy your management cluster configuration file and save it with a new name by following the procedure in Create a Tanzu Kubernetes Cluster Configuration File.

  2. In the new configuration file, add or modify the following:

    IS_WINDOWS_WORKLOAD_CLUSTER: "true"
    VSPHERE_WINDOWS_TEMPLATE: windows-2019-kube-v1.22.8
    ENABLE_MHC: "false"
    
  3. Deploy a workload cluster as described in Deploy Tanzu Kubernetes Clusters to vSphere.

check-circle-line exclamation-circle-line close-line
Scroll to top icon