This topic describes how to use the Tanzu Kubernetes Grid CLI to deploy a management cluster to Amazon Elastic Compute Cloud (Amazon EC2) from a YAML configuration file.
t3.large
or t3.xlarge
, see Amazon EC2 Instance Types.config.yaml
file for the management cluster with the required parameters. You can use the created config.yaml
as a model for future deployments from the CLI.The tkg
CLI uses and modifies settings in a cluster configuration file. This file defaults to ~/.tkg/config.yaml
but can be specified with the --config
option for relevant commands to support configuring multiple management clusters separately.
The cluster configuration file provides the base configuration for management clusters and Tanzu Kubernetes clusters. When you deploy a management cluster from the CLI, the tkg init
command uses this configuration.
Also, the tkg config permissions aws
command that you run to create a CloudFormation stack in your Amazon EC2 account retrieves the values of the AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, AWS_REGION
variables from the cluster configuration file. Alternatively, it can read these variables from the AWS default credential provider chain. For more information, see Create an AWS CloudFormation Stack below.
To configure the cluster configuration file, follow the steps below:
If this is the first time that you are running Tanzu Kubernetes Grid commands on your machine and you have not already deployed a management cluster to Amazon EC2 by using the Tanzu Kubernetes Grid installer interface, open a terminal and run the tkg get management-cluster
command:
tkg get management-cluster
Running a tkg
command for the first time creates the $HOME/.tkg
folder that contains the cluster configuration file config.yaml
and other configuration files.
Open the .tkg/config.yaml
file in a text editor:
If you have already deployed a management cluster to Amazon EC2 from the installer interface, review the variables that describe your previous deployment and update your configuration file if needed.
If you have not already deployed a management cluster to Amazon EC2 from the installer interface, copy and paste the following rows at the end of your configuration file:
New VPC: If you want to deploy a development management cluster with a single control plane node to a new VPC, add the variables listed below to the configuration file:
AWS_ACCESS_KEY_ID:
AWS_SECRET_ACCESS_KEY:
AWS_REGION:
AWS_NODE_AZ:
AWS_PRIVATE_NODE_CIDR:
AWS_PUBLIC_NODE_CIDR:
AWS_SSH_KEY_NAME:
AWS_VPC_CIDR:
BASTION_HOST_ENABLED:
SERVICE_CIDR:
CLUSTER_CIDR:
CONTROL_PLANE_MACHINE_TYPE:
NODE_MACHINE_TYPE:
If you want to deploy a production management cluster with three control plane nodes to a new VPC, add the following variables to the list of variables above:
AWS_NODE_AZ_1:
AWS_NODE_AZ_2:
AWS_PRIVATE_NODE_CIDR_1:
AWS_PRIVATE_NODE_CIDR_2:
AWS_PUBLIC_NODE_CIDR_1:
AWS_PUBLIC_NODE_CIDR_2:
See the example configuration below for a production management cluster with three control plane nodes:
AWS_ACCESS_KEY_ID: EXAMPLEWDRMY2A7QLK6UM
AWS_SECRET_ACCESS_KEY: EXAMPLEghtYU/H6WUILO/dSqutLrIfxaz94mP6G
AWS_REGION: us-west-2
AWS_NODE_AZ: us-west-2a
AWS_NODE_AZ_1: us-west-2b
AWS_NODE_AZ_2: us-west-2c
AWS_PRIVATE_NODE_CIDR: 10.0.0.0/24
AWS_PRIVATE_NODE_CIDR_1: 10.0.2.0/24
AWS_PRIVATE_NODE_CIDR_2: 10.0.4.0/24
AWS_PUBLIC_NODE_CIDR: 10.0.1.0/24
AWS_PUBLIC_NODE_CIDR_1: 10.0.3.0/24
AWS_PUBLIC_NODE_CIDR_2: 10.0.5.0/24
AWS_SSH_KEY_NAME: tkg
AWS_VPC_CIDR: 10.0.0.0/16
BASTION_HOST_ENABLED: "true"
SERVICE_CIDR: 100.64.0.0/13
CLUSTER_CIDR: 100.96.0.0/11
CONTROL_PLANE_MACHINE_TYPE: m5.large
NODE_MACHINE_TYPE: m5.large
Existing VPC: If you want to deploy a development management cluster with a single control plane node to an existing VPC, add the variables listed below to the configuration file:
AWS_ACCESS_KEY_ID:
AWS_SECRET_ACCESS_KEY:
AWS_REGION:
AWS_NODE_AZ:
AWS_PRIVATE_SUBNET_ID:
AWS_PUBLIC_SUBNET_ID:
AWS_SSH_KEY_NAME:
AWS_VPC_ID:
BASTION_HOST_ENABLED:
SERVICE_CIDR:
CLUSTER_CIDR:
CONTROL_PLANE_MACHINE_TYPE:
NODE_MACHINE_TYPE:
If you want to deploy a production management cluster with three control plane nodes to an existing VPC, add the following variables to the list of variables above:
AWS_NODE_AZ_1:
AWS_NODE_AZ_2:
AWS_PRIVATE_SUBNET_ID_1:
AWS_PRIVATE_SUBNET_ID_2:
AWS_PUBLIC_SUBNET_ID_1:
AWS_PUBLIC_SUBNET_ID_2:
See the example configuration below for a production management cluster with three control plane nodes:
AWS_ACCESS_KEY_ID: EXAMPLEWDRMY2A7QLK6UM
AWS_SECRET_ACCESS_KEY: EXAMPLEghtYU/H6WUILO/dSqutLrIfxaz94mP6G
AWS_REGION: us-west-2
AWS_NODE_AZ: us-west-2a
AWS_NODE_AZ_1: us-west-2b
AWS_NODE_AZ_2: us-west-2c
AWS_PRIVATE_SUBNET_ID: subnet-ID
AWS_PRIVATE_SUBNET_ID_1: subnet-ID
AWS_PRIVATE_SUBNET_ID_2: subnet-ID
AWS_PUBLIC_SUBNET_ID: subnet-ID
AWS_PUBLIC_SUBNET_ID_1: subnet-ID
AWS_PUBLIC_SUBNET_ID_2: subnet-ID
AWS_SSH_KEY_NAME: tkg
AWS_VPC_ID: vpc-ID
BASTION_HOST_ENABLED: "true"
SERVICE_CIDR: 100.64.0.0/13
CLUSTER_CIDR: 100.96.0.0/11
CONTROL_PLANE_MACHINE_TYPE: m5.large
NODE_MACHINE_TYPE: m5.large
The table in Configuration Parameter Reference describes all of the configuration options that you can provide for deployment of a management cluster to Amazon EC2. Line order in the configuration file does not matter.
The table below describes all of the variables that you must set for deployment to Amazon EC2, along with some AWS-specific optional variables. To set them in a configuration file, leave a space between the colon (:
) and the variable value. For example:
AWS_REGION: us-west-2
IMPORTANT:
config.yaml
or other file, unset any conflicting environment variables before you deploy the management cluster from the CLI.Parameter | Value | Description |
---|---|---|
AWS_ACCESS_KEY_ID |
Your AWS access key ID | Enter the access key ID for your AWS account. |
AWS_SECRET_ACCESS_KEY |
Your AWS secret access key | Enter the secret access key for your AWS account. |
AWS_REGION |
us-west-2 , ap-northeast-2 , etc. |
The name of the AWS region in which to deploy the management cluster. For example, us-west-2 . You can also specify the us-gov-east and us-gov-west regions in AWS GovCloud. If you have already set a different region as an environment variable, for example, in Deploy Management Clusters to Amazon EC2, you must unset that environment variable. |
AWS_NODE_AZ |
us-west-2a , ap-northeast-2b , etc. |
The name of the AWS availability zone in your chosen region that you want use as the availability zone for this management cluster. Availability zone names are the same as the AWS region name, with a single lower-case letter suffix, such as a , b , c . For example, us-west-2a . To deploy a prod management cluster with three control plane nodes, you must also set AWS_NODE_AZ_1 and AWS_NODE_AZ_2 . The letter suffix in each of these availability zones must be unique. For example, us-west-2a , us-west-2b , and us-west-2c . |
AWS_NODE_AZ_1 and AWS_NODE_AZ_2 |
us-west-2a , ap-northeast-2b , etc. |
Set these variables if you want to deploy a prod management cluster with three control plane nodes. Both availability zones must be in the same region as AWS_NODE_AZ . See AWS_NODE_AZ above for more information. |
AWS_PRIVATE_NODE_CIDR |
10.0.0.0/24 |
Set this variable if you set AWS_VPC_CIDR . If the recommended range of 10.0.0.0/24 is not available, enter a different IP range in CIDR format for private nodes to use. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ . To deploy a prod management cluster with three control plane nodes, you must also set AWS_PRIVATE_NODE_CIDR_1 and AWS_PRIVATE_NODE_CIDR_2 . |
AWS_PRIVATE_NODE_CIDR_1 |
10.0.2.0/24 |
If the recommended range of 10.0.2.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ_1 . See AWS_PRIVATE_NODE_CIDR above for more information. |
AWS_PRIVATE_NODE_CIDR_2 |
10.0.4.0/24 |
If the recommended range of 10.0.4.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ_2 . See AWS_PRIVATE_NODE_CIDR above for more information. |
AWS_PUBLIC_NODE_CIDR |
10.0.1.0/24 |
Set this variable if you set AWS_VPC_CIDR . If the recommended range of 10.0.1.0/24 is not available, enter a different IP range in CIDR format for public nodes to use. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ . To deploy a prod management cluster with three control plane nodes, you must also set AWS_PUBLIC_NODE_CIDR_1 and AWS_PUBLIC_NODE_CIDR_2 . |
AWS_PUBLIC_NODE_CIDR_1 |
10.0.3.0/24 |
If the recommended range of 10.0.3.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ_1 . See AWS_PUBLIC_NODE_CIDR above for more information. |
AWS_PUBLIC_NODE_CIDR_2 |
10.0.5.0/24 |
If the recommended range of 10.0.5.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in AWS_NODE_AZ_2 . See AWS_PUBLIC_NODE_CIDR above for more information. |
AWS_PRIVATE_SUBNET_ID |
Private subnet for the VPC | If you set AWS_VPC_ID to use an existing VPC, enter the ID of a private subnet that already exists in AWS_NODE_AZ . This setting is optional. If you do not set it, tkg init identifies the private subnet automatically. To deploy a prod management cluster with three control plane nodes, you must also set AWS_PRIVATE_SUBNET_ID_1 and AWS_PRIVATE_SUBNET_ID_2 . |
AWS_PRIVATE_SUBNET_ID_1 |
Private subnet for the VPC | Enter the ID of a private subnet that exists in AWS_NODE_AZ_1 . If you do not set this variable, tkg init identifies the private subnet automatically. See AWS_PRIVATE_SUBNET_ID above for more information. |
AWS_PRIVATE_SUBNET_ID_2 |
Private subnet for the VPC | Enter the ID of a private subnet that exists in AWS_NODE_AZ_2 . If you do not set this variable, tkg init identifies the private subnet automatically. See AWS_PRIVATE_SUBNET_ID above for more information. |
AWS_PUBLIC_SUBNET_ID |
Public subnet for the VPC | If you set AWS_VPC_ID to use an existing VPC, enter the ID of a public subnet that already exists in AWS_NODE_AZ . This setting is optional. If you do not set it, tkg init identifies the public subnet automatically. To deploy a prod management cluster with three control plane nodes, you must also set AWS_PUBLIC_SUBNET_ID_1 and AWS_PUBLIC_SUBNET_ID_2 . |
AWS_PUBLIC_SUBNET_ID_1 |
Public subnet for the VPC | Enter the ID of a public subnet that exists in AWS_NODE_AZ_1 . If you do not set this variable, tkg init identifies the public subnet automatically. See AWS_PUBLIC_SUBNET_ID above for more information. |
AWS_PUBLIC_SUBNET_ID_2 |
Public subnet for the VPC | Enter the ID of a public subnet that exists in AWS_NODE_AZ_2 . If you do not set this variable, tkg init identifies the public subnet automatically. See AWS_PUBLIC_SUBNET_ID above for more information. |
AWS_SSH_KEY_NAME |
Your SSH key name | Enter the name of the SSH private key that you registered with your AWS account. |
AWS_VPC_ID |
VPC ID name | To use a VPC that already exists in your selected AWS region, enter the ID of the VPC and then set AWS_PUBLIC_SUBNET_ID and AWS_PRIVATE_SUBNET_ID . Set either AWS_VPC_ID or AWS_VPC_CIDR , but not both. |
AWS_VPC_CIDR |
10.0.0.0/16 |
If you want Tanzu Kubernetes Grid to create a new VPC in the selected region, set the AWS_VPC_CIDR , AWS_PUBLIC_NODE_CIDR , and AWS_PRIVATE_NODE_CIDR variables. If the recommended range of 10.0.0.0/16 is not available, enter a different IP range in CIDR format in AWS_VPC_CIDR for the management cluster to use. Set either AWS_VPC_CIDR or AWS_VPC_ID , but not both. |
BASTION_HOST_ENABLED |
"true" or "false" |
This option is set to "true" in the global Tanzu Kubernetes Grid configuration. Specify "true" to deploy an AWS basion host or "false" to reuse an existing bastion host. If no bastion host exists in your availability zone(s) and you set AWS_VPC_ID to use an existing VPC, set BASTION_HOST_ENABLED to "true" . |
SERVICE_CIDR |
100.64.0.0/13 |
The CIDR range to use for the Kubernetes services. The recommended range is 100.64.0.0/13. Change this value only if the recommended range is unavailable. |
CLUSTER_CIDR |
100.96.0.0/11 |
The CIDR range to use for pods. The recommended range is 100.96.0.0/11. Change this value only if the recommended range is unavailable. |
CONTROL_PLANE_MACHINE_TYPE |
For example, t3.small |
An AWS instance type available in your specified region and availability zone(s) for the control plane node VMs, depending on the expected workloads that you will run in the cluster. Possible values may include i3.xlarge , r4.8xlarge , m5a.4xlarge , m5.large , and t3.small , but see Finding an Amazon EC2 instance type in the AWS documentation for actual choices. You can also specify or override this value by using the tkg init --size or --controlplane-size options. |
NODE_MACHINE_TYPE |
For example, m5.large |
An AWS instance type available in your specified region and availability zone(s) for the worker node VMs, depending on the expected workloads that you will run in the cluster. Possible values may include i3.xlarge , r4.8xlarge , m5a.4xlarge , m5.large , and t3.small , but see Finding an Amazon EC2 instance type in the AWS documentation for actual choices. You can also specify or override this value by using the tkg init --size or --worker-size options. |
ENABLE_MHC |
"true" or "false" |
Enables or disables the MachineHealthCheck controller, which provides node health monitoring and node auto-repair for Tanzu Kubernetes clusters. This option is enabled in the global Tanzu Kubernetes Grid configuration by default, for all Tanzu Kubernetes clusters. To disable MachineHealthCheck on the clusters that you deploy with this management cluster, set ENABLE_MHC to false . Set this variable only if you want to override your global configuration. You can enable or disable MachineHealthCheck on individual clusters after deployment by using the CLI. For instructions, see Configure Machine Health Checks for Tanzu Kubernetes Clusters. |
MHC_UNKNOWN_STATUS_TIMEOUT |
For example, 10m |
Property of MachineHealthCheck . By default, if the Ready condition of a node remains Unknown for longer than 5m , MachineHealthCheck considers the machine unhealthy and recreates it. Set this variable if you want to change the default timeout. |
MHC_FALSE_STATUS_TIMEOUT |
For example, 10m |
Property of MachineHealthCheck . By default, if the Ready condition of a node remains False for longer than 5m , MachineHealthCheck considers the machine unhealthy and recreates it. Set this variable if you want to change the default timeout. |
Before deploying a management cluster to Amazon EC2 for the first time, you must create an AWS CloudFormation stack for Tanzu Kubernetes Grid in your AWS account. To create the CloudFormation stack, you run the tkg config permissions aws
command from the CLI or enable the Automate creation of AWS CloudFormation Stack in the installer interface.
This CloudFormation stack provides the identity and access management (IAM) resources that Tanzu Kubernetes Grid needs to create management clusters and Tanzu Kubernetes clusters on Amazon EC2. The IAM resources are added to the control plane and node roles when they are created during cluster deployment.
You need to create only one CloudFormation stack per AWS account. The IAM resources that the CloudFormation stack provides are global, meaning they are not specific to any region. For more information about CloudFormation stacks, see Working with Stacks in the AWS documentation.
IMPORTANT: In Tanzu Kubernetes Grid v1.2 and later, the Automate creation of AWS CloudFormation Stack checkbox and the tkg config permissions aws
command replace the clusterawsadm
command line utility. For existing management and Tanzu Kubernetes clusters, initially deployed with v1.1.x or earlier, continue to use the CloudFormation stack that you created by running the clusterawsadm alpha bootstrap create-stack
command. If you want to use the same AWS account for your existing clusters and Tanzu Kubernetes Grid v1.2 and later, both stacks must be present in the account. For more information, see Prepare to Upgrade Clusters on Amazon EC2.
Do one of the following:
tkg init
Command below.If you have not already created the CloudFormation stack for Tanzu Kubernetes Grid in your AWS account:
Ensure that the AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, and AWS_REGION
are set in your configuration file or set them in the AWS default credential provider chain. For more information, see Working with AWS Credentials in the AWS documentation.
Run the following command:
tkg config permissions aws
To specify the path to your configuration file, use the --config
flag. The default path is $HOME/.tkg/config.yaml
. For more information about this command, run tkg config permissions aws --help
.
tkg init
CommandAfter you have updated .tkg/config.yaml
, or another cluster configuration file, you deploy a management cluster by running the tkg init
command. The cluster configuration file provides the base configuration for management clusters and Tanzu Kubernetes clusters. You provide more precise configuration information for individual clusters by running the tkg init
command with different options.
Warning: The tkg init
command takes time to complete. While tkg init
is running, do not run additional invocations of tkg init
on the same bootstrap machine to deploy multiple management clusters, run tkg set management-cluster
to change context, or edit ~/.kube-tkg/config
.
To deploy a management cluster to Amazon EC2, you must at least specify the --infrastructure aws
option to tkgi init
:
tkg init --infrastructure aws
The table in CLI Options Reference describes additional command-line options for deploying a management cluster to Amazon EC2.
Monitoring Progress
When you run tkg init
, you can follow the progress of the deployment of the management cluster in the terminal. For more detail, open the log file listed in the terminal output Logs of the command execution can also be found at....
Deployment of the management cluster can take several minutes. The first run of tkg init
takes longer than subsequent runs because it has to pull the required Docker images into the image store on your bootstrap machine. Subsequent runs do not require this step, so are faster.
The first run of tkg init
also adds settings to your configuration file.
If tkg init
fails before the management cluster deploys to AWS, you should clean up artifacts on your bootstrap machine before you re-run tkg init
. See the Troubleshooting Tips topic for details. If the machine on which you run tkg init
shuts down or restarts before the local operations finish, the deployment will fail.
Validation Checks
When you run tkg init
, the command performs several validation checks before deploying the management cluster. The command verifies that the target Amazon EC2 infrastructure meets the following requirements:
When these conditions are not met, the tkg init
command fails.
The table below describes command-line options that you can set for deployment to AWS.
For example, to create a highly available Amazon EC2 management cluster aws-mgmt-cluster
in which all of the control plane and worker node VMs are t3.xlarge
size:
tkg init --infrastructure aws --name aws-mgmt-cluster --plan prod --size t3.xlarge
Option | Value | Description |
---|---|---|
--infrastructure |
aws |
Required |
--name |
Name for the management cluster | Name must comply with DNS hostname requirements as outlined in RFC 952 and amended in RFC 1123 If you do not specify a name, a unique name is generated. |
Scaling and Availability | ||
--plan |
dev or prod |
dev ("development"), the default, deploys a management cluster with a single control plane node.prod ("production") deploys a highly available management cluster with three control plane nodes. |
Configuration Files | ||
--config |
Local file system path to .yaml file, e.g. /path/to/file/my-config.yaml |
Configuration file to use or create, other than the default $HOME/.tkg/config.yaml . If the config file was not already created by hand or prior tkg init calls, it is created. All other files are created in the default folder.This option, for example, lets you deploy multiple management clusters that share a VPC. |
--kubeconfig |
Local file system path to .yaml file, e.g. /path/to/file/my-kubeconfig.yaml |
Kube configuration file or use, or create, other than the default $HOME/.kube-tkg/config.yaml .This option lets you customize or share multiple kubeconfig files for multiple management clusters. |
Nodes | ||
--size |
An AWS instance type available in your specified region and availability zone(s). Possible values may include i3.xlarge , r4.8xlarge , m5a.4xlarge , m5.large , and t3.small , but see Finding an Amazon EC2 instance type in the AWS documentation for actual choices. |
Size for both control plane and worker node VMs. |
--controlplane-size |
Size for control plane node VMs. Overrides the --size option and CONTROL_PLANE_MACHINE_TYPE parameter. |
|
--worker-size |
Size for worker node VMs. Overrides the --size option and NODE_MACHINE_TYPE parameter. |
|
Customer Experience Improvement Program | ||
--ceip-participation |
true or false |
false opts out of the VMware Customer Experience Improvement Program. Default is true .You can also opt in or out of the program after deploying the management cluster. For information, see Opt in or Out of the VMware CEIP and Customer Experience Improvement Program ("CEIP"). |
kubectl
to the management cluster, and how to create namespaces see Examine the Management Cluster Deployment.