Checkout how to provision TKG clusters with Windows node pools by using the Cluster v1beta1 API based on Cluster Class.

Before you provision TKG clusters with Windows node pools, consider the following prerequisites and requirements:

  • Create a local Content Library containing Windows licensing information and custom TKr together with a Photon or Ubuntu TKr for the control plane. For further information, see https://github.com/vmware-tanzu/vsphere-tanzu-kubernetes-grid-image-builder. For information about creating a local Content Library that contains a TKr, see Create a Local Content Library (for Air-Gapped Cluster Provisioning).
  • If you are planning to use Active Directory:
    • Make sure that the DNS server for vCenter Server can correctly resolve the Active Directory DNS SRV records. Use the Domain Controllers as the DNS server
    • Verify that an NTP server is configured and synchronization with the Active Directory Domain Controllers. Use the Domain Controllers as the NTP server.
  • Windows Server 2022 is supported starting from TKr 1.31.
  • Windows node pools are only supported for TKG clusters created by using the v1beta1 API and Cluster Class.
  • To use Windows node pools for existing TKG clusters, upgrade the TKG Service to 3.2. The clusters must be based on builtin-generic-v3.2.0 Cluster Class, which is available starting from the TKG Service 3.2.
  • All containers to be deployed to Windows nodes must be built from a Windows Server 2022 base. Older Windows versions are not supported.
  • Clusters with Windows node pools only support Antrea as the CNI. Calico is not supported.
  • When configuring proxies, note that Windows system services do not support proxy servers using HTTPS. If set, TKG will only configure containerd and kubelet with the proxy server but not the system. In addition, for noProxy, Windows system services do not support CIDR blocks. If they are specified, TKG will read the CIDR mask and replace one or more octets with a wild-card, which will only fully align on /24, /16 and /8 sized subnets.

Once you upgrade the TKG Service to version 3.2 and TKr to version 1.31, existing TKG clusters created by using Cluster Class are automatically updated to use the builtin-generic-v3.2.0 Cluster Class in the spec.topology.class of the cluster custom resource.

To add Windows node pools to a TKG cluster, use the run.tanzu.vmware.com/resolve-os-image: os-type=windows annotation in the node pool definition.

Following is an example definition for a TKG cluster deployed with Windows node pools:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: wc-win-2
  namespace: win-ns1
spec:
  clusterNetwork:
    services:
      cidrBlocks: ["198.51.100.0/12"]
    pods:
      cidrBlocks: ["192.0.2.0/16"]
    serviceDomain: "cluster.local"
  topology:
    class: builtin-generic-v3.2.0 # <=== Mandatory Settings
    version: v1.30.1---vmware.2-fips-vkr.2
    controlPlane:
      replicas: 1
    workers:
      machineDeployments:
        - class: node-pool
          name: node-pool-photon
          replicas: 1
        - class: node-pool
          name: node-pool-windows
          metadata:
            annotations:
              run.tanzu.vmware.com/resolve-os-image: os-type=windows #<=== Mandatory Settings
          replicas: 1
          variables:
            overrides:
            - name: vmClass
              value: best-effort-large # Use a larger footprint for Windows node
    variables:
      - name: vmClass
        value: best-effort-medium
      - name: storageClass
        value: wcpglobalstorageprofile

Applying a Secondary Volume for containerd and kubelet

Windows uses different paths for containerd and kubelet compared to Linux. To use a secondary volume or containerd and kubelet, make sure to use the following paths:

  • Kubelet: c:\var\lib\kubelet
  • ContainerD: c:\programdata\containerd

These must be set as a variable override for the Windows nodepool, and not on the cluster scoped variables.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: pnid-md-cc-ppwp-35ns2-c4
  namespace: pnid-md-cc-ppwp-35ns2
spec:
  clusterNetwork:
    pods:
    cidrBlocks:
    - 192.0.2.0/16
    serviceDomain: cluster.local
    services:
    cidrBlocks:
    - 198.51.100.0/12
  topology:
    class: builtin-generic-v3.2.0
    controlPlane:
    metadata:
    annotations:
    run.tanzu.vmware.com/resolve-os-image: os-name=photon
    replicas: 1
    variables:
    - name: vmClass
    value: best-effort-small
    - name: storageClass
    value: vsansp
    version: v1.31.0---vmware.3-fips-vkr.1
    workers:
    machineDeployments:
    - class: node-pool
    metadata:
    annotations:
            run.tanzu.vmware.com/resolve-os-image: os-type=windows
        name: np1-worker
    	replicas: 2
    	variables:
      	overrides:
      	- name: volumes
        	value:
        	- name: kubelet-dir
          	mountPath: C:\var\lib\kubelet
          	capacity: 15Gi
          	storageClass: vsansp
        	- name: containerd-dir
          	mountPath: C:\ProgramData\containerd
          	capacity: 25Gi
          	storageClass: vsansp
      	- name: vmClass
        	value: best-effort-medium