This topic describes how to use the Tanzu Kubernetes Grid CLI to deploy a management cluster to Amazon Elastic Compute Cloud (Amazon EC2) from a YAML configuration file.

Prerequisites

  • Ensure that you have met all of the requirements listed in Install the Tanzu Kubernetes Grid CLI and Deploy Management Clusters to Amazon EC2.
  • It is strongly recommended to use the Tanzu Kubernetes Grid installer interface rather than the CLI to deploy your first management cluster to Amazon EC2. When you deploy a management cluster by using the installer interface, it populates the config.yaml file for the management cluster with the required parameters. You can use the created config.yaml as a model for future deployments from the CLI.

Create the Cluster Configuration File

The tkg CLI uses and modifies settings in a cluster configuration file. This file defaults to ~/.tkg/config.yaml but can be specified with the --config option for relevant commands to support configuring multiple management clusters separately.

The cluster configuration file provides the base configuration for management clusters and Tanzu Kubernetes clusters. When you deploy a management cluster from the CLI, the tkg init command uses this configuration.

Also, the tkg config permissions aws command that you run to create a CloudFormation stack in your Amazon EC2 account retrieves the values of the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION variables from the cluster configuration file. Alternatively, it can read these variables from the AWS default credential provider chain. For more information, see Create an AWS CloudFormation Stack below.

To configure the cluster configuration file, follow the steps below:

  1. If this is the first time that you are running Tanzu Kubernetes Grid commands on your machine and you have not already deployed a management cluster to Amazon EC2 by using the Tanzu Kubernetes Grid installer interface, open a terminal and run the tkg get management-cluster command:

    tkg get management-cluster
    

    Running a tkg command for the first time creates the $HOME/.tkg folder that contains the cluster configuration file config.yaml and other configuration files.

  2. Open the .tkg/config.yaml file in a text editor:

    • If you have already deployed a management cluster to Amazon EC2 from the installer interface, review the variables that describe your previous deployment and update your configuration file if needed.

    • If you have not already deployed a management cluster to Amazon EC2 from the installer interface, copy and paste the following rows at the end of your configuration file:

      • New VPC: If you want to deploy a development management cluster with a single control plane node to a new VPC, add the variables listed below to the configuration file:

        AWS_ACCESS_KEY_ID:
        AWS_SECRET_ACCESS_KEY:
        AWS_REGION:
        AWS_NODE_AZ:
        AWS_PRIVATE_NODE_CIDR:
        AWS_PUBLIC_NODE_CIDR:
        AWS_SSH_KEY_NAME:
        AWS_VPC_CIDR:
        BASTION_HOST_ENABLED:
        SERVICE_CIDR:
        CLUSTER_CIDR:
        CONTROL_PLANE_MACHINE_TYPE:
        NODE_MACHINE_TYPE:
        

        If you want to deploy a production management cluster with three control plane nodes to a new VPC, add the following variables to the list of variables above:

        AWS_NODE_AZ_1:
        AWS_NODE_AZ_2:
        AWS_PRIVATE_NODE_CIDR_1:
        AWS_PRIVATE_NODE_CIDR_2:
        AWS_PUBLIC_NODE_CIDR_1:
        AWS_PUBLIC_NODE_CIDR_2:
        

        See the example configuration below for a production management cluster with three control plane nodes:

        AWS_ACCESS_KEY_ID: EXAMPLEWDRMY2A7QLK6UM
        AWS_SECRET_ACCESS_KEY: EXAMPLEghtYU/H6WUILO/dSqutLrIfxaz94mP6G
        AWS_REGION: us-west-2
        AWS_NODE_AZ: us-west-2a
        AWS_NODE_AZ_1: us-west-2b
        AWS_NODE_AZ_2: us-west-2c
        AWS_PRIVATE_NODE_CIDR: 10.0.0.0/24
        AWS_PRIVATE_NODE_CIDR_1: 10.0.2.0/24
        AWS_PRIVATE_NODE_CIDR_2: 10.0.4.0/24
        AWS_PUBLIC_NODE_CIDR: 10.0.1.0/24
        AWS_PUBLIC_NODE_CIDR_1: 10.0.3.0/24
        AWS_PUBLIC_NODE_CIDR_2: 10.0.5.0/24
        AWS_SSH_KEY_NAME: tkg
        AWS_VPC_CIDR: 10.0.0.0/16
        BASTION_HOST_ENABLED: "true"
        SERVICE_CIDR: 100.64.0.0/13
        CLUSTER_CIDR: 100.96.0.0/11
        CONTROL_PLANE_MACHINE_TYPE: m5.large
        NODE_MACHINE_TYPE: m5.large
        
      • Existing VPC: If you want to deploy a development management cluster with a single control plane node to an existing VPC, add the variables listed below to the configuration file:

        AWS_ACCESS_KEY_ID:
        AWS_SECRET_ACCESS_KEY:
        AWS_REGION:
        AWS_NODE_AZ:
        AWS_PRIVATE_SUBNET_ID:
        AWS_PUBLIC_SUBNET_ID:
        AWS_SSH_KEY_NAME:
        AWS_VPC_ID:
        BASTION_HOST_ENABLED:
        SERVICE_CIDR:
        CLUSTER_CIDR:
        CONTROL_PLANE_MACHINE_TYPE:
        NODE_MACHINE_TYPE:
        

        If you want to deploy a production management cluster with three control plane nodes to an existing VPC, add the following variables to the list of variables above:

        AWS_NODE_AZ_1:
        AWS_NODE_AZ_2:
        AWS_PRIVATE_SUBNET_ID_1:
        AWS_PRIVATE_SUBNET_ID_2:
        AWS_PUBLIC_SUBNET_ID_1:
        AWS_PUBLIC_SUBNET_ID_2:
        

        See the example configuration below for a production management cluster with three control plane nodes:

        AWS_ACCESS_KEY_ID: EXAMPLEWDRMY2A7QLK6UM
        AWS_SECRET_ACCESS_KEY: EXAMPLEghtYU/H6WUILO/dSqutLrIfxaz94mP6G
        AWS_REGION: us-west-2
        AWS_NODE_AZ: us-west-2a
        AWS_NODE_AZ_1: us-west-2b
        AWS_NODE_AZ_2: us-west-2c
        AWS_PRIVATE_SUBNET_ID: subnet-ID
        AWS_PRIVATE_SUBNET_ID_1: subnet-ID
        AWS_PRIVATE_SUBNET_ID_2: subnet-ID
        AWS_PUBLIC_SUBNET_ID: subnet-ID
        AWS_PUBLIC_SUBNET_ID_1: subnet-ID
        AWS_PUBLIC_SUBNET_ID_2: subnet-ID
        AWS_SSH_KEY_NAME: tkg
        AWS_VPC_ID: vpc-ID
        BASTION_HOST_ENABLED: "true"
        SERVICE_CIDR: 100.64.0.0/13
        CLUSTER_CIDR: 100.96.0.0/11
        CONTROL_PLANE_MACHINE_TYPE: m5.large
        NODE_MACHINE_TYPE: m5.large
        

    The table in Configuration Parameter Reference describes all of the configuration options that you can provide for deployment of a management cluster to Amazon EC2. Line order in the configuration file does not matter.

Configuration Parameter Reference

The table below describes all of the variables that you must set for deployment to Amazon EC2, along with some AWS-specific optional variables. To set them in a configuration file, leave a space between the colon (:) and the variable value. For example:

AWS_REGION: us-west-2

IMPORTANT:

  • As described in Configuring the Management Cluster, environment variables override values from a cluster configuration file. To use all settings from a config.yaml or other file, unset any conflicting environment variables before you deploy the management cluster from the CLI.
  • Tanzu Kubernetes Grid does not support IPv6 addresses. This is because upstream Kubernetes only provides alpha support for IPv6. Always provide IPv4 addresses in the procedures in this topic.

Parameter Value Description
AWS_ACCESS_KEY_ID Your AWS access key ID Enter the access key ID for your AWS account.
AWS_SECRET_ACCESS_KEY Your AWS secret access key Enter the secret access key for your AWS account.
AWS_REGION us-west-2, ap-northeast-2, etc. The name of the AWS region in which to deploy the management cluster. For example, us-west-2. You can also specify the us-gov-east and us-gov-west regions in AWS GovCloud. If you have already set a different region as an environment variable, for example, in Deploy Management Clusters to Amazon EC2, you must unset that environment variable.
AWS_NODE_AZ us-west-2a, ap-northeast-2b, etc. The name of the AWS availability zone in your chosen region that you want use as the availability zone for this management cluster. Availability zone names are the same as the AWS region name, with a single lower-case letter suffix, such as a, b, c. For example, us-west-2a. To deploy a prod management cluster with three control plane nodes, you must also set AWS_NODE_AZ_1 and AWS_NODE_AZ_2. The letter suffix in each of these availability zones must be unique. For example, us-west-2a, us-west-2b, and us-west-2c.
AWS_NODE_AZ_1 and AWS_NODE_AZ_2 us-west-2a, ap-northeast-2b, etc. Set these variables if you want to deploy a prod management cluster with three control plane nodes. Both availability zones must be in the same region as AWS_NODE_AZ. See AWS_NODE_AZ above for more information.
AWS_PRIVATE_NODE_CIDR 10.0.0.0/24 Set this variable if you set AWS_VPC_CIDR. If the recommended range of 10.0.0.0/24 is not available, enter a different IP range in CIDR format for private nodes to use. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ. To deploy a prod management cluster with three control plane nodes, you must also set AWS_PRIVATE_NODE_CIDR_1 and AWS_PRIVATE_NODE_CIDR_2.
AWS_PRIVATE_NODE_CIDR_1 10.0.2.0/24 If the recommended range of 10.0.2.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ_1. See AWS_PRIVATE_NODE_CIDR above for more information.
AWS_PRIVATE_NODE_CIDR_2 10.0.4.0/24 If the recommended range of 10.0.4.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ_2. See AWS_PRIVATE_NODE_CIDR above for more information.
AWS_PUBLIC_NODE_CIDR 10.0.1.0/24 Set this variable if you set AWS_VPC_CIDR. If the recommended range of 10.0.1.0/24 is not available, enter a different IP range in CIDR format for public nodes to use. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ. To deploy a prod management cluster with three control plane nodes, you must also set AWS_PUBLIC_NODE_CIDR_1 and AWS_PUBLIC_NODE_CIDR_2.
AWS_PUBLIC_NODE_CIDR_1 10.0.3.0/24 If the recommended range of 10.0.3.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ_1. See AWS_PUBLIC_NODE_CIDR above for more information.
AWS_PUBLIC_NODE_CIDR_2 10.0.5.0/24 If the recommended range of 10.0.5.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ_2. See AWS_PUBLIC_NODE_CIDR above for more information.
AWS_PRIVATE_SUBNET_ID Private subnet for the VPC If you set AWS_VPC_ID to use an existing VPC, enter the ID of a private subnet that already exists in AWS_NODE_AZ. This setting is optional. If you do not set it, tkg init identifies the private subnet automatically. To deploy a prod management cluster with three control plane nodes, you must also set AWS_PRIVATE_SUBNET_ID_1 and AWS_PRIVATE_SUBNET_ID_2.
AWS_PRIVATE_SUBNET_ID_1 Private subnet for the VPC Enter the ID of a private subnet that exists in AWS_NODE_AZ_1. If you do not set this variable, tkg init identifies the private subnet automatically. See AWS_PRIVATE_SUBNET_ID above for more information.
AWS_PRIVATE_SUBNET_ID_2 Private subnet for the VPC Enter the ID of a private subnet that exists in AWS_NODE_AZ_2. If you do not set this variable, tkg init identifies the private subnet automatically. See AWS_PRIVATE_SUBNET_ID above for more information.
AWS_PUBLIC_SUBNET_ID Public subnet for the VPC If you set AWS_VPC_ID to use an existing VPC, enter the ID of a public subnet that already exists in AWS_NODE_AZ. This setting is optional. If you do not set it, tkg init identifies the public subnet automatically. To deploy a prod management cluster with three control plane nodes, you must also set AWS_PUBLIC_SUBNET_ID_1 and AWS_PUBLIC_SUBNET_ID_2.
AWS_PUBLIC_SUBNET_ID_1 Public subnet for the VPC Enter the ID of a public subnet that exists in AWS_NODE_AZ_1. If you do not set this variable, tkg init identifies the public subnet automatically. See AWS_PUBLIC_SUBNET_ID above for more information.
AWS_PUBLIC_SUBNET_ID_2 Public subnet for the VPC Enter the ID of a public subnet that exists in AWS_NODE_AZ_2. If you do not set this variable, tkg init identifies the public subnet automatically. See AWS_PUBLIC_SUBNET_ID above for more information.
AWS_SSH_KEY_NAME Your SSH key name Enter the name of the SSH private key that you registered with your AWS account.
AWS_VPC_ID VPC ID name To use a VPC that already exists in your selected AWS region, enter the ID of the VPC and then set AWS_PUBLIC_SUBNET_ID and AWS_PRIVATE_SUBNET_ID. Set either AWS_VPC_ID or AWS_VPC_CIDR, but not both.
AWS_VPC_CIDR 10.0.0.0/16 If you want Tanzu Kubernetes Grid to create a new VPC in the selected region, set the AWS_VPC_CIDR, AWS_PUBLIC_NODE_CIDR, and AWS_PRIVATE_NODE_CIDR variables. If the recommended range of 10.0.0.0/16 is not available, enter a different IP range in CIDR format in AWS_VPC_CIDR for the management cluster to use. Set either AWS_VPC_CIDR or AWS_VPC_ID, but not both.
BASTION_HOST_ENABLED "true" or "false" This option is set to "true" in the global Tanzu Kubernetes Grid configuration. Specify "true" to deploy an AWS basion host or "false" to reuse an existing bastion host. If no bastion host exists in your availability zone(s) and you set AWS_VPC_ID to use an existing VPC, set BASTION_HOST_ENABLED to "true".
SERVICE_CIDR 100.64.0.0/13 The CIDR range to use for the Kubernetes services. The recommended range is 100.64.0.0/13. Change this value only if the recommended range is unavailable.
CLUSTER_CIDR 100.96.0.0/11 The CIDR range to use for pods. The recommended range is 100.96.0.0/11. Change this value only if the recommended range is unavailable.
CONTROL_PLANE_MACHINE_TYPE For example, t3.small An AWS instance type available in your specified region and availability zone(s) for the control plane node VMs, depending on the expected workloads that you will run in the cluster. Possible values may include i3.xlarge, r4.8xlarge, m5a.4xlarge, m5.large, and t3.small, but see Finding an Amazon EC2 instance type in the AWS documentation for actual choices. You can also specify or override this value by using the tkg init --size or --controlplane-size options.
NODE_MACHINE_TYPE For example, m5.large An AWS instance type available in your specified region and availability zone(s) for the worker node VMs, depending on the expected workloads that you will run in the cluster. Possible values may include i3.xlarge, r4.8xlarge, m5a.4xlarge, m5.large, and t3.small, but see Finding an Amazon EC2 instance type in the AWS documentation for actual choices. You can also specify or override this value by using the tkg init --size or --worker-size options.
ENABLE_MHC "true" or "false" Enables or disables the MachineHealthCheck controller, which provides node health monitoring and node auto-repair for Tanzu Kubernetes clusters. This option is enabled in the global Tanzu Kubernetes Grid configuration by default, for all Tanzu Kubernetes clusters. To disable MachineHealthCheck on the clusters that you deploy with this management cluster, set ENABLE_MHC to false. Set this variable only if you want to override your global configuration. You can enable or disable MachineHealthCheck on individual clusters after deployment by using the CLI. For instructions, see Configure Machine Health Checks for Tanzu Kubernetes Clusters.
MHC_UNKNOWN_STATUS_TIMEOUT For example, 10m Property of MachineHealthCheck. By default, if the Ready condition of a node remains Unknown for longer than 5m, MachineHealthCheck considers the machine unhealthy and recreates it. Set this variable if you want to change the default timeout.
MHC_FALSE_STATUS_TIMEOUT For example, 10m Property of MachineHealthCheck. By default, if the Ready condition of a node remains False for longer than 5m, MachineHealthCheck considers the machine unhealthy and recreates it. Set this variable if you want to change the default timeout.

Create an AWS CloudFormation Stack

Before deploying a management cluster to Amazon EC2 for the first time, you must create an AWS CloudFormation stack for Tanzu Kubernetes Grid in your AWS account. To create the CloudFormation stack, you run the tkg config permissions aws command from the CLI or enable the Automate creation of AWS CloudFormation Stack in the installer interface.

This CloudFormation stack provides the identity and access management (IAM) resources that Tanzu Kubernetes Grid needs to create management clusters and Tanzu Kubernetes clusters on Amazon EC2. The IAM resources are added to the control plane and node roles when they are created during cluster deployment.

You need to create only one CloudFormation stack per AWS account. The IAM resources that the CloudFormation stack provides are global, meaning they are not specific to any region. For more information about CloudFormation stacks, see Working with Stacks in the AWS documentation.

IMPORTANT: In Tanzu Kubernetes Grid v1.2 and later, the Automate creation of AWS CloudFormation Stack checkbox and the tkg config permissions aws command replace the clusterawsadm command line utility. For existing management and Tanzu Kubernetes clusters, initially deployed with v1.1.x or earlier, continue to use the CloudFormation stack that you created by running the clusterawsadm alpha bootstrap create-stack command. If you want to use the same AWS account for your existing clusters and Tanzu Kubernetes Grid v1.2 and later, both stacks must be present in the account. For more information, see Prepare to Upgrade Clusters on Amazon EC2.

Do one of the following:

  • If you have already created the CloudFormation stack for Tanzu Kubernetes Grid in your AWS account, proceed to Run the tkg init Command below.
  • If you have not already created the CloudFormation stack for Tanzu Kubernetes Grid in your AWS account:

    1. Ensure that the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION are set in your configuration file or set them in the AWS default credential provider chain. For more information, see Working with AWS Credentials in the AWS documentation.

    2. Run the following command:

      tkg config permissions aws
      

      To specify the path to your configuration file, use the --config flag. The default path is $HOME/.tkg/config.yaml. For more information about this command, run tkg config permissions aws --help.

Run the tkg init Command

After you have updated .tkg/config.yaml, or another cluster configuration file, you deploy a management cluster by running the tkg init command. The cluster configuration file provides the base configuration for management clusters and Tanzu Kubernetes clusters. You provide more precise configuration information for individual clusters by running the tkg init command with different options.

Warning: The tkg init command takes time to complete. While tkg init is running, do not run additional invocations of tkg init on the same bootstrap machine to deploy multiple management clusters, run tkg set management-cluster to change context, or edit ~/.kube-tkg/config.

To deploy a management cluster to Amazon EC2, you must at least specify the --infrastructure aws option to tkgi init:

tkg init --infrastructure aws

The table in CLI Options Reference describes additional command-line options for deploying a management cluster to Amazon EC2.

Monitoring Progress

When you run tkg init, you can follow the progress of the deployment of the management cluster in the terminal. For more detail, open the log file listed in the terminal output Logs of the command execution can also be found at....

Deployment of the management cluster can take several minutes. The first run of tkg init takes longer than subsequent runs because it has to pull the required Docker images into the image store on your bootstrap machine. Subsequent runs do not require this step, so are faster.

The first run of tkg init also adds settings to your configuration file.

If tkg init fails before the management cluster deploys to AWS, you should clean up artifacts on your bootstrap machine before you re-run tkg init. See the Troubleshooting Tips topic for details. If the machine on which you run tkg init shuts down or restarts before the local operations finish, the deployment will fail.

Validation Checks

When you run tkg init, the command performs several validation checks before deploying the management cluster. The command verifies that the target Amazon EC2 infrastructure meets the following requirements:

  • The AWS credentials that you provided are valid.
  • Cloud Formation stack exists.
  • Node Instance type is supported.
  • Region and AZ match.
  • AWS_AMI_ID is valid.

When these conditions are not met, the tkg init command fails.

CLI Options Reference

The table below describes command-line options that you can set for deployment to AWS.

For example, to create a highly available Amazon EC2 management cluster aws-mgmt-cluster in which all of the control plane and worker node VMs are t3.xlarge size:

tkg init --infrastructure aws --name aws-mgmt-cluster --plan prod --size t3.xlarge

Option Value Description
--infrastructure aws Required
--name Name for the management cluster Name must comply with DNS hostname requirements as outlined in RFC 952 and amended in RFC 1123
If you do not specify a name, a unique name is generated.
Scaling and Availability
--plan dev or prod dev ("development"), the default, deploys a management cluster with a single control plane node.
prod ("production") deploys a highly available management cluster with three control plane nodes.
Configuration Files
--config Local file system path to .yaml file, e.g. /path/to/file/my-config.yaml Configuration file to use or create, other than the default $HOME/.tkg/config.yaml. If the config file was not already created by hand or prior tkg init calls, it is created. All other files are created in the default folder.
This option, for example, lets you deploy multiple management clusters that share a VPC.
--kubeconfig Local file system path to .yaml file, e.g. /path/to/file/my-kubeconfig.yaml Kube configuration file or use, or create, other than the default $HOME/.kube-tkg/config.yaml.
This option lets you customize or share multiple kubeconfig files for multiple management clusters.
Nodes
--size An AWS instance type available in your specified region and availability zone(s). Possible values may include i3.xlarge, r4.8xlarge, m5a.4xlarge, m5.large, and t3.small, but see Finding an Amazon EC2 instance type in the AWS documentation for actual choices. Size for both control plane and worker node VMs.
--controlplane-size Size for control plane node VMs. Overrides the --size option and CONTROL_PLANE_MACHINE_TYPE parameter.
--worker-size Size for worker node VMs. Overrides the --size option and NODE_MACHINE_TYPE parameter.
Customer Experience Improvement Program
--ceip-participation true or false false opts out of the VMware Customer Experience Improvement Program. Default is true.
You can also opt in or out of the program after deploying the management cluster. For information, see Opt in or Out of the VMware CEIP and Customer Experience Improvement Program ("CEIP").

What to Do Next

  • For information about what happened during the deployment of the management cluster, how to connect kubectl to the management cluster, and how to create namespaces see Examine the Management Cluster Deployment.
  • For information about how to create namespaces in the management cluster, see Create Namespaces in the Management Cluster.
  • If you need to deploy more than one management cluster, on any or all of vSphere, Azure, and Amazon EC2, see Manage Your Management Clusters. This topic also provides information about how to add existing management clusters to your CLI instance, obtain credentials, scale and delete management clusters, and how to opt in or out of the CEIP.
check-circle-line exclamation-circle-line close-line
Scroll to top icon