This topic explains how to build and use custom VM images as the basis for cluster nodes deployed by Tanzu Kubernetes Grid on Azure. You use custom VM images instead of the default images available on the Azure Marketplace.

Prerequisites

Before you begin this procedure, you must have:

Build an Image with Kubernetes Image Builder

To build a Tanzu Kubernetes Grid image for Azure on your local workstation:

  1. Clone the Kubernetes Image Builder tool.

    git clone https://github.com/kubernetes-sigs/image-builder.git
    
  2. Set local environment variables to access, and target your Azure account.

    export AZURE_SUBSCRIPTION_ID=<your-subscription-id>
    export AZURE_TENANT_ID=<your-tenant-id>
    export AZURE_CLIENT_ID=<your-client-id>
    export AZURE_CLIENT_SECRET=<your-client-secret>
    export AZURE_LOCATION="westus2"
    export AZURE_ENVIRONMENT="AzurePublicCloud"
    
  3. Log into Azure, and set your subscription.

    az cloud set --name AzureCloud
    
    az login --service-principal --username $AZURE_CLIENT_ID --password $AZURE_CLIENT_SECRET --tenant $AZURE_TENANT_ID
    
    az account set --subscription $AZURE_SUBSCRIPTION_ID
    
  4. Determine the Image Builder configuration version that you want to build from.

    • Search the VMware {code} Sample Exchange for TKG Image Builder to list the available versions.
    • Each version corresponds to the Kubernetes version that Image Builder uses. For example, image-builder-1.19.1-cfg.zip builds a Kubernetes v1.19.1 image.
    • If you need to create a management cluster, which you must do when you first install Tanzu Kubernetes Grid, choose the default Kubernetes version of your Tanzu Kubernetes Grid version. For example, in Tanzu Kubernetes Grid v1.2.0, the default Kubernetes version is v1.19.1.
  5. Download the configuration code zip file, and unpack its contents into the image-builder/images/capi directory. After extracting the contents, you may need to move the extracted files up one level, so that build_image.sh, apply_goss.sh and other scripts are directly under /images/capi.

  6. From the image-builder/images/capi directory, run build-image.sh to create your image:

    ./build-image.sh
    
  7. Parse and record the values listed in the output. You reference these values later in the BOM file as name, resourceGroup, subscriptionID, gallery, and version:

    • ManagedImageName: name
    • ManagedImageSharedImageGalleryId: /subscriptions/subscriptionID/resourceGroups/cluster-api-images/providers/Microsoft.Compute/galleries/gallery/images/capi-ubuntu-1804/versions/version

    For example, from the following build-image.sh output:

    ManagedImageName: capi-1602021415
    ManagedImageSharedImageGalleryId: /subscriptions/881e14ab-d120-412d-b8ad-8f555fd86943/resourceGroups/cluster-api-images/providers/Microsoft.Compute/galleries/ClusterAPI/images/capi-ubuntu-1804/versions/0.3.1602021415
    

    You can retrieve the following values:

    • name: capi-1602021415
    • resourceGroup: cluster-api-images
    • subscriptionID: 881e14ab-d120-412d-b8ad-8f555fd86943
    • gallery: ClusterAPI
    • version: 0.3.1602021415

Troubleshooting

In some cases of component versioning, Goss tests fail.

If the Goss tests fail:

  1. Edit the images/capi/packer/config/goss-args.json file in Image Builder directory to set "goss_inspect_mode": "true".
  2. Run build-image.sh again.
  3. After the image is deployed, SSH into the VM manually, and run the tests directly on the VM.

References

See:

Use a Custom VM Image

How you configure Tanzu Kubernetes Grid to use a custom VM image for the nodes it creates on Azure depends on whether the image resides in a Shared Image Gallery or just in a resource group for your own account:

  • Shared Image Gallery images:

    • Can be created with the Build an Image with Kubernetes Image Builder procedure above, which saves the image to the gallery ClusterAPI.
    • Can be created by others and saved to a Shared Image Gallery.
    • Have no limit on simultaneous deployments.
    • Can be configured in your cluster configuration file or in a version-specific Bill of Materials (BoM) file.
  • Managed images:

    • Reside within your own subscription and have not been saved to an Azure Shared Image Gallery.
    • Support a maximum of 20 simultaneous deployments.
    • Are typically used for development and testing only.

The sections below explain how to use VM images from each of these sources.

Edit the Cluster Configuration to Specify a Shared Image

This procedure configures your local Tanzu Kubernetes Grid cluster configuration file to specify to a custom image in a Shared Image Gallery, such as one created by the Build an Image with Kubernetes Image Builder procedure above.

The cluster configuration file is ~/.tkg/config.yaml by default, or a file passed to the tkg CLI --config option.

  1. In the cluster configuration file, add a top-level azure-image: definition block with a property corresponding to the Kubernetes version, such as v1.18.8+vmware.1.

  2. Under the Kubernetes version property, set values for the subproperties resourceGroup, name, subscriptionID, version, and gallery. If you created the image as above, you recorded these values from the build-image.sh output. For example:

    azure-image:
        v1.18.8+vmware.1:
            resourceGroup: cluster-api-images
            name: capi-1602021415
            subscriptionID: 881e14ab-d120-412d-b8ad-8f555fd86943
            version: 0.3.1602021415
            gallery: ClusterAPI
    

For more information, see Using Shared Image Gallery (Recommended) in The Cluster API Provider Azure Book.

Edit the Cluster Configuration to Specify a Managed Image

This procedure configures your local Tanzu Kubernetes Grid cluster configuration file to specify to a custom managed image.

A managed image is an image in your own resource group and managed through your subscription that has not been saved to a Shared Image Gallery. Managed images support a maximum of 20 simultaneous deployments and are typically used for development and testing only.

The cluster configuration file is ~/.tkg/config.yaml by default, or a file passed to the tkg CLI --config option.

  1. In the cluster configuration file, add a top-level azure-image: definition block with a property corresponding to the Kubernetes version, such as v1.18.8+vmware.1.

  2. Under the Kubernetes version property, set the value of a subproperty id to the full resource ID of your image. For example:

    azure-image:
        v1.19.1+vmware.2:
            id: "/subscriptions/881e14ab-d120-412d-b8ad-8f555fd86943/resourceGroups/CLUSTER-API-IMAGES/providers/Microsoft.Compute/images/capi-1602526760"
    

You can run az image list to see your list of managed images with full resource id fields.

For more information, see Using Image ID in The Cluster API Provider Azure Book.

Edit a Bill of Materials to Specify a Shared Image

This procedure configures a Tanzu Kubernetes Grid Bill of Materials (BoM) YAML file to point to a custom image in a Shared Image Gallery, such as one created by the Build an Image with Kubernetes Image Builder procedure above.

Instead of this procedure, you can follow the Edit the Cluster Configuration to Specify a Shared Image procedure above to centralize your Tanzu Kubernetes Grid configurations in a single file and avoid having to track image customizations in separate BoM files.

  1. In ~/.tkg/bom, find the BOM file that matches your Image Builder configuration version, which is based on its Kubernetes version:

    • If a Kubernetes version is not the default for a Tanzu Kubernetes Grid version, then its BOM file is named for the Kubernetes version. For example, bom-1.18.8+vmware.1.yaml can use an image created with image-builder-1.18.8-cfg.
    • If a Kubernetes version is the default for a Tanzu Kubernetes Grid version, then its BOM file is named for the Tanzu Kubernetes Grid version. For example, bom-1.2.0+vmware.1.yaml can use an image created with image-builder-1.19.1-cfg because v1.19.1 is the default Kubernetes version in Tanzu Kubernetes Grid v1.2.0.
    • In all cases, the components.kubernetes.version field in a BOM file indicates its Kubernetes version.
  2. In the BOM file, replace the entire azure: block using the values recorded from build-image.sh output.

    • This modification replaces the publisher and sku values for a Marketplace image with gallery and name values for a Shared Image Gallery image.
    azure:
      resourceGroup: <resourceGroup>
      name: <name>
      subscriptionID: <subscriptionID>
      gallery: <gallery>
      version: <version>
    

What to Do Next

After you configure the BOM file with your custom VM image, you can deploy your management cluster or a workload cluster to Azure:

check-circle-line exclamation-circle-line close-line
Scroll to top icon