This topic describes how to use the tool Stembuild to create a Windows stemcell for BOSH, for use by Tanzu Kubernetes Grid Integrated Edition (TKGI) on vSphere.



Overview of Stembuild

A BOSH stemcell is a versioned operating system image.

You must create a BOSH stemcell for Windows before you can deploy Windows workers in Kubernetes clusters using Tanzu Kubernetes Grid Integrated Edition on vSphere.

Stembuild is a binary that you use to build BOSH stemcells for Windows Server 2019.

Stembuild creates a BOSH stemcell from a base Windows image. The Stembuild CLI has two commands, construct and package, which you run against a Windows Server 2019 VM. These commands are used to create a stemcell in Construct and Package the BOSH Stemcell below.



Overview of Windows Stemcell Creation

To create a Windows stemcell for vSphere, you create a base Windows VM from a volume-licensed ISO and subsequently maintain that base template with all Windows-recommended security updates, but without the BOSH dependencies.

The Windows VM with security updates serves as the base for all future stemcells produced from clones of that base VM. This enables you to build new stemcells without having to run Windows updates from scratch each time. You can also use a “snapshot” feature to maintain an updated Windows image that does not contain the BOSH dependencies.

VMware recommends installing any available critical updates and then rebuilding the stemcell from a clone of the original VM.

The BOSH stemcell that you create in this topic is based on Windows Server 2019. If you already have a BOSH stemcell for Windows on vSphere, see Monthly Stemcell Upgrades below.

For more information, see Best Practices for Stembuild for Tanzu Application Service & Tanzu Kubernetes Grid Integrated Edition in VMware Tanzu Tech Tutorials.

To construct, package and upload a BOSH Stemcell for Windows to TKGI, complete the following:

  1. Prerequisites
  2. Create and Configure a Base VM for the BOSH Stemcell
  3. Construct and Package the BOSH Stemcell
  4. Update Ops Manager With the Updated Stemcell



Prerequisites

Before you create a BOSH Windows stemcell for Tanzu Kubernetes Grid Integrated Edition on vSphere, you must have:

  • A vSphere environment. To ensure the VM hardware used by the stemcell is compatible with your deployment environment’s ESXi/ESX host and vCenter Server versions, see ESXi/ESX hosts and compatible virtual machine hardware versions list (2007240) in the VMware Knowledge Base.

  • An ISO for a Windows Server 2019 Server Core installation, build number: 17763, from Microsoft Developer Network (MSDN) or Microsoft Volume Licensing Service Center (VLSC). The Windows Server 2019 ISO must be a clean, base ISO file. You can use an evaluation copy for testing, but VMware does not recommend an evaluation copy for production because the licensing expires. For more information, see the Windows Server documentation or the Microsoft Volume Licensing Service Center website.

    Note: A clean ISO file has no custom scripts or tooling. For example, the ISO must have no logging or antivirus tools installed.

  • Download the following from Stemcells (Windows) on Broadcom Support:

    • A Windows stemcell
    • A stembuild command line interface (CLI) from a 2019.x release

    Refer to Product Snapshot in Release Notes for the compatible version of each to download.

  • Microsoft Local Group Policy Object Utility (LGPO) downloaded to the same folder as your stembuild CLI.

  • The minimum vCenter user permissions required to use stembuild for vSphere stemcells, specifically:

    • VirtualMachine.GuestOperations.Modify
    • VirtualMachine.GuestOperations.Execute
    • VirtualMachine.GuestOperations.Query
    • VirtualMachine.Config.AddRemoveDevice
    • VirtualMachine.Interact.SetCDMedia
    • VApp.Export
    • System.Anonymous*
    • System.Read*
    • System.View*

Permissions marked with an * are generated upon creating a new user in vCenter and cannot be set within the vCenter UI.



Create and Configure a Base VM for the BOSH Stemcell

Before using Stembuild to create a stemcell, you need to create a Windows Server 2019 VM and update the VM with the latest Windows updates.

Warning: VMware recommends that you build your stemcell with kubernetes-csi-proxy v1.1.3 or later to avoid the Kubernetes CVE: CVE-2023-3893. For more information, see CVE-2023-3893: Insufficient input sanitization on kubernetes-csi-proxy leads to privilege escalation #119594 in the Kubernetes GitHub repository.

To create and update a base Windows VM, follow these procedures in the TAS for VMs [Windows] documentation, in order:

  1. Create a Base VM for the BOSH Stemcell
  2. Configure the Base VM
  3. Expose Ethernet Adapter Information on Worker Node VMs below
  4. Clone the Base VM


Expose Ethernet Adapter Information on Worker Node VMs

By default, VMware Tools prevents worker node VM Ethernet adapter information from being exposed on Windows clusters. You must ensure this Ethernet adapter information is exposed on your Windows base OS image to avoid the oVM.Guest.Net Error on Windows Clusters issue on your Windows clusters.

To ensure Ethernet adapter information is exposed on worker node VMs:

  1. Verify that Ethernet adapter information on worker node VMs is already exposed:

    1. Open a command line window in your Windows base OS environment.
    2. Run ipconfig.
    3. Review the return for any vEthernet Ethernet adapters.
    4. If a vEthernet Ethernet adapter is present, Ethernet adapter information on worker node VMs is not exposed in this environment.

  2. If Ethernet adapter information is not exposed on worker node VMs:

    1. Copy the tools configuration from C:\ProgramData\VMware\VMware Tools\tools.conf.example to C:\ProgramData\VMware\VMware Tools\tools.conf.
    2. Edit the new copy of tools.conf.

    3. Add the following to the configuration:

      exclude-nics=""
    4. Save your changes.



Construct and Package the BOSH Stemcell

To create, configure, and package a BOSH Stemcell, follow these procedures, in order:

  1. Construct the BOSH Stemcell, in the TAS for VMs [Windows] documentation.
  2. Remove Hidden Devices, below.
  3. Package the BOSH Stemcell, in the TAS for VMs [Windows] documentation.


Remove Hidden Devices

To ensure your BOSH Windows stemcell can work properly, confirm the stemcell does not have any hidden devices:

  1. Open the vSphere Management console.
  2. Confirm a network adapter is not assigned to the target VM.
  3. Power on the target VM.
  4. Log into the target VM.
  5. Start PowerShell.

  6. Confirm the VM has hidden devices:

    Get-PnpDevice -Class net | ? Status -eq Unknown

  7. If there are hidden devices, clean up those devices:

    $Devs = Get-PnpDevice -Class net | ? Status -eq Unknown
ForEach ($Dev in $Devs) {
    Write-Host "Removing $($Dev.FriendlyName)" -ForegroundColor Cyan
    $RemoveKey = "HKLM:\SYSTEM\CurrentControlSet\Enum\$($Dev.InstanceId)"
    Get-Item $RemoveKey | Select-Object -ExpandProperty Property | %{ Remove-ItemProperty -Path $RemoveKey -Name $_ -Verbose }
}

  8. Power off the target VM.

Note: The ovs-windows job requires a VM with a net-adapter named Ethernet0. Remove hidden devices from the target VM to ensure the Ethernet0 net-adapter name is not taken before the ovs-windows job starts.



Update Ops Manager With the Updated Stemcell

To update Ops Manager with the new BOSH Windows stemcell:

  1. Open Ops Manager.

  2. Navigate to the Stemcell Library.

  3. Replace the existing stemcell in the Ops Manager stemcell library with your new updated stemcell.

  4. Deploy the TKGI tile.

  5. gMSA: If your Windows worker clusters authenticate users via a group Managed Service Account (gMSA) in Microsoft AD, manually reboot its worker nodes as described in the known issue GMSA authentication failures after stemcell upgrade.



Monthly Stemcell Upgrades

Microsoft typically releases Windows updates with security patches on the second Tuesday of each month.

After each Microsoft Windows security update, update your BOSH stemcell by following these procedures, in order:

  1. Configure the Base VM in the TAS for VMs [Windows] documentation.

  2. Construct and Package the BOSH Stemcell, above.

  3. Update Ops Manager With the Updated Stemcell, above.



Known Issues

For known issues with stemcell creation, see Known Issues in the TAS for VMs [Windows] documentation.

check-circle-line exclamation-circle-line close-line
Scroll to top icon