Deploy Management Clusters to Amazon EC2 with the CLI

This topic describes how to use the Tanzu Kubernetes Grid CLI to deploy a management cluster to Amazon Elastic Compute Cloud (Amazon EC2) from a YAML configuration file.

Prerequisites

Ensure that you have met all of the requirements listed in Install the Tanzu Kubernetes Grid CLI and Deploy Management Clusters to Amazon EC2.
- For information about the configurations of the different sizes of node instances, for example, t3.large or t3.xlarge, see Amazon EC2 Instance Types.
- For information about when to create a Virtual Private Cloud (VPC) and when to reuse an existing VPC, see Resource Usage in Your Amazon Web Services Account.
It is strongly recommended to use the Tanzu Kubernetes Grid installer interface rather than the CLI to deploy your first management cluster to Amazon EC2. When you deploy a management cluster by using the installer interface, it populates the config.yaml file for the management cluster with the required parameters. You can use the created config.yaml as a model for future deployments from the CLI.

Create the Cluster Configuration File

The tkg CLI uses and modifies settings in a cluster configuration file. This file defaults to ~/.tkg/config.yaml but can be specified with the --config option for relevant commands to support configuring multiple management clusters separately.

The cluster configuration file provides the base configuration for management clusters and Tanzu Kubernetes clusters. When you deploy a management cluster from the CLI, the tkg init command uses this configuration.

Also, the tkg config permissions aws command that you run to create a CloudFormation stack in your Amazon EC2 account retrieves the values of the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION variables from the cluster configuration file. Alternatively, it can read these variables from the AWS default credential provider chain. For more information, see Create an AWS CloudFormation Stack below.

To configure the cluster configuration file, follow the steps below:

If this is the first time that you are running Tanzu Kubernetes Grid commands on your machine and you have not already deployed a management cluster to Amazon EC2 by using the Tanzu Kubernetes Grid installer interface, open a terminal and run the tkg get management-cluster command:
```
tkg get management-cluster
```
Running a tkg command for the first time creates the $HOME/.tkg folder that contains the cluster configuration file config.yaml and other configuration files.

Open the .tkg/config.yaml file in a text editor:

If you have already deployed a management cluster to Amazon EC2 from the installer interface, review the variables that describe your previous deployment and update your configuration file if needed.

If you have not already deployed a management cluster to Amazon EC2 from the installer interface, copy and paste the following rows at the end of your configuration file:

New VPC: If you want to deploy a development management cluster with a single control plane node to a new VPC, add the variables listed below to the configuration file:

AWS_ACCESS_KEY_ID:
AWS_SECRET_ACCESS_KEY:
AWS_REGION:
AWS_NODE_AZ:
AWS_PRIVATE_NODE_CIDR:
AWS_PUBLIC_NODE_CIDR:
AWS_SSH_KEY_NAME:
AWS_VPC_CIDR:
BASTION_HOST_ENABLED:
SERVICE_CIDR:
CLUSTER_CIDR:
CONTROL_PLANE_MACHINE_TYPE:
NODE_MACHINE_TYPE:

If you want to deploy a production management cluster with three control plane nodes to a new VPC, add the following variables to the list of variables above:

AWS_NODE_AZ_1:
AWS_NODE_AZ_2:
AWS_PRIVATE_NODE_CIDR_1:
AWS_PRIVATE_NODE_CIDR_2:
AWS_PUBLIC_NODE_CIDR_1:
AWS_PUBLIC_NODE_CIDR_2:

See the example configuration below for a production management cluster with three control plane nodes:

AWS_ACCESS_KEY_ID: EXAMPLEWDRMY2A7QLK6UM
AWS_SECRET_ACCESS_KEY: EXAMPLEghtYU/H6WUILO/dSqutLrIfxaz94mP6G
AWS_REGION: us-west-2
AWS_NODE_AZ: us-west-2a
AWS_NODE_AZ_1: us-west-2b
AWS_NODE_AZ_2: us-west-2c
AWS_PRIVATE_NODE_CIDR: 10.0.0.0/24
AWS_PRIVATE_NODE_CIDR_1: 10.0.2.0/24
AWS_PRIVATE_NODE_CIDR_2: 10.0.4.0/24
AWS_PUBLIC_NODE_CIDR: 10.0.1.0/24
AWS_PUBLIC_NODE_CIDR_1: 10.0.3.0/24
AWS_PUBLIC_NODE_CIDR_2: 10.0.5.0/24
AWS_SSH_KEY_NAME: tkg
AWS_VPC_CIDR: 10.0.0.0/16
BASTION_HOST_ENABLED: "true"
SERVICE_CIDR: 100.64.0.0/13
CLUSTER_CIDR: 100.96.0.0/11
CONTROL_PLANE_MACHINE_TYPE: m5.large
NODE_MACHINE_TYPE: m5.large

Existing VPC: If you want to deploy a development management cluster with a single control plane node to an existing VPC, add the variables listed below to the configuration file:

AWS_ACCESS_KEY_ID:
AWS_SECRET_ACCESS_KEY:
AWS_REGION:
AWS_NODE_AZ:
AWS_PRIVATE_SUBNET_ID:
AWS_PUBLIC_SUBNET_ID:
AWS_SSH_KEY_NAME:
AWS_VPC_ID:
BASTION_HOST_ENABLED:
SERVICE_CIDR:
CLUSTER_CIDR:
CONTROL_PLANE_MACHINE_TYPE:
NODE_MACHINE_TYPE:

If you want to deploy a production management cluster with three control plane nodes to an existing VPC, add the following variables to the list of variables above:

AWS_NODE_AZ_1:
AWS_NODE_AZ_2:
AWS_PRIVATE_SUBNET_ID_1:
AWS_PRIVATE_SUBNET_ID_2:
AWS_PUBLIC_SUBNET_ID_1:
AWS_PUBLIC_SUBNET_ID_2:

See the example configuration below for a production management cluster with three control plane nodes:

AWS_ACCESS_KEY_ID: EXAMPLEWDRMY2A7QLK6UM
AWS_SECRET_ACCESS_KEY: EXAMPLEghtYU/H6WUILO/dSqutLrIfxaz94mP6G
AWS_REGION: us-west-2
AWS_NODE_AZ: us-west-2a
AWS_NODE_AZ_1: us-west-2b
AWS_NODE_AZ_2: us-west-2c
AWS_PRIVATE_SUBNET_ID: subnet-ID
AWS_PRIVATE_SUBNET_ID_1: subnet-ID
AWS_PRIVATE_SUBNET_ID_2: subnet-ID
AWS_PUBLIC_SUBNET_ID: subnet-ID
AWS_PUBLIC_SUBNET_ID_1: subnet-ID
AWS_PUBLIC_SUBNET_ID_2: subnet-ID
AWS_SSH_KEY_NAME: tkg
AWS_VPC_ID: vpc-ID
BASTION_HOST_ENABLED: "true"
SERVICE_CIDR: 100.64.0.0/13
CLUSTER_CIDR: 100.96.0.0/11
CONTROL_PLANE_MACHINE_TYPE: m5.large
NODE_MACHINE_TYPE: m5.large

The table in Configuration Parameter Reference describes all of the configuration options that you can provide for deployment of a management cluster to Amazon EC2. Line order in the configuration file does not matter.

Configuration Parameter Reference

The table below describes all of the variables that you must set for deployment to Amazon EC2, along with some AWS-specific optional variables. To set them in a configuration file, leave a space between the colon (:) and the variable value. For example:

AWS_REGION: us-west-2

IMPORTANT:

As described in Configuring the Management Cluster, environment variables override values from a cluster configuration file. To use all settings from a config.yaml or other file, unset any conflicting environment variables before you deploy the management cluster from the CLI.
Tanzu Kubernetes Grid does not support IPv6 addresses. This is because upstream Kubernetes only provides alpha support for IPv6. Always provide IPv4 addresses in the procedures in this topic.

Parameter	Value	Description
`AWS_ACCESS_KEY_ID`	Your AWS access key ID	Enter the access key ID for your AWS account.
`AWS_SECRET_ACCESS_KEY`	Your AWS secret access key	Enter the secret access key for your AWS account.
`AWS_REGION`	`us-west-2`, `ap-northeast-2`, etc.	The name of the AWS region in which to deploy the management cluster. For example, `us-west-2`. You can also specify the `us-gov-east` and `us-gov-west` regions in AWS GovCloud. If you have already set a different region as an environment variable, for example, in Deploy Management Clusters to Amazon EC2, you must unset that environment variable.
`AWS_NODE_AZ`	`us-west-2a`, `ap-northeast-2b`, etc.	The name of the AWS availability zone in your chosen region that you want use as the availability zone for this management cluster. Availability zone names are the same as the AWS region name, with a single lower-case letter suffix, such as `a`, `b`, `c`. For example, `us-west-2a`. To deploy a `prod` management cluster with three control plane nodes, you must also set `AWS_NODE_AZ_1` and `AWS_NODE_AZ_2`. The letter suffix in each of these availability zones must be unique. For example, `us-west-2a`, `us-west-2b`, and `us-west-2c`.
`AWS_NODE_AZ_1` and `AWS_NODE_AZ_2`	`us-west-2a`, `ap-northeast-2b`, etc.	Set these variables if you want to deploy a `prod` management cluster with three control plane nodes. Both availability zones must be in the same region as `AWS_NODE_AZ`. See `AWS_NODE_AZ` above for more information.
`AWS_PRIVATE_NODE_CIDR`	`10.0.0.0/24`	Set this variable if you set `AWS_VPC_CIDR`. If the recommended range of 10.0.0.0/24 is not available, enter a different IP range in CIDR format for private nodes to use. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in `AWS_NODE_AZ`. To deploy a `prod` management cluster with three control plane nodes, you must also set `AWS_PRIVATE_NODE_CIDR_1` and `AWS_PRIVATE_NODE_CIDR_2`.
`AWS_PRIVATE_NODE_CIDR_1`	`10.0.2.0/24`	If the recommended range of 10.0.2.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in `AWS_NODE_AZ_1`. See `AWS_PRIVATE_NODE_CIDR` above for more information.
`AWS_PRIVATE_NODE_CIDR_2`	`10.0.4.0/24`	If the recommended range of 10.0.4.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in `AWS_NODE_AZ_2`. See `AWS_PRIVATE_NODE_CIDR` above for more information.
`AWS_PUBLIC_NODE_CIDR`	`10.0.1.0/24`	Set this variable if you set `AWS_VPC_CIDR`. If the recommended range of 10.0.1.0/24 is not available, enter a different IP range in CIDR format for public nodes to use. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in `AWS_NODE_AZ`. To deploy a `prod` management cluster with three control plane nodes, you must also set `AWS_PUBLIC_NODE_CIDR_1` and `AWS_PUBLIC_NODE_CIDR_2`.
`AWS_PUBLIC_NODE_CIDR_1`	`10.0.3.0/24`	If the recommended range of 10.0.3.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in `AWS_NODE_AZ_1`. See `AWS_PUBLIC_NODE_CIDR` above for more information.
`AWS_PUBLIC_NODE_CIDR_2`	`10.0.5.0/24`	If the recommended range of 10.0.5.0/24 is not available, enter a different IP range in CIDR format. When Tanzu Kubernetes Grid deploys your management cluster, it creates this subnetwork in `AWS_NODE_AZ_2`. See `AWS_PUBLIC_NODE_CIDR` above for more information.
`AWS_PRIVATE_SUBNET_ID`	Private subnet for the VPC	If you set `AWS_VPC_ID` to use an existing VPC, enter the ID of a private subnet that already exists in `AWS_NODE_AZ`. This setting is optional. If you do not set it, `tkg init` identifies the private subnet automatically. To deploy a `prod` management cluster with three control plane nodes, you must also set `AWS_PRIVATE_SUBNET_ID_1` and `AWS_PRIVATE_SUBNET_ID_2`.
`AWS_PRIVATE_SUBNET_ID_1`	Private subnet for the VPC	Enter the ID of a private subnet that exists in `AWS_NODE_AZ_1`. If you do not set this variable, `tkg init` identifies the private subnet automatically. See `AWS_PRIVATE_SUBNET_ID` above for more information.
`AWS_PRIVATE_SUBNET_ID_2`	Private subnet for the VPC	Enter the ID of a private subnet that exists in `AWS_NODE_AZ_2`. If you do not set this variable, `tkg init` identifies the private subnet automatically. See `AWS_PRIVATE_SUBNET_ID` above for more information.
`AWS_PUBLIC_SUBNET_ID`	Public subnet for the VPC	If you set `AWS_VPC_ID` to use an existing VPC, enter the ID of a public subnet that already exists in `AWS_NODE_AZ`. This setting is optional. If you do not set it, `tkg init` identifies the public subnet automatically. To deploy a `prod` management cluster with three control plane nodes, you must also set `AWS_PUBLIC_SUBNET_ID_1` and `AWS_PUBLIC_SUBNET_ID_2`.
`AWS_PUBLIC_SUBNET_ID_1`	Public subnet for the VPC	Enter the ID of a public subnet that exists in `AWS_NODE_AZ_1`. If you do not set this variable, `tkg init` identifies the public subnet automatically. See `AWS_PUBLIC_SUBNET_ID` above for more information.
`AWS_PUBLIC_SUBNET_ID_2`	Public subnet for the VPC	Enter the ID of a public subnet that exists in `AWS_NODE_AZ_2`. If you do not set this variable, `tkg init` identifies the public subnet automatically. See `AWS_PUBLIC_SUBNET_ID` above for more information.
`AWS_SSH_KEY_NAME`	Your SSH key name	Enter the name of the SSH private key that you registered with your AWS account.
`AWS_VPC_ID`	VPC ID name	To use a VPC that already exists in your selected AWS region, enter the ID of the VPC and then set `AWS_PUBLIC_SUBNET_ID` and `AWS_PRIVATE_SUBNET_ID`. Set either `AWS_VPC_ID` or `AWS_VPC_CIDR`, but not both.
`AWS_VPC_CIDR`	`10.0.0.0/16`	If you want Tanzu Kubernetes Grid to create a new VPC in the selected region, set the `AWS_VPC_CIDR`, `AWS_PUBLIC_NODE_CIDR`, and `AWS_PRIVATE_NODE_CIDR` variables. If the recommended range of 10.0.0.0/16 is not available, enter a different IP range in CIDR format in `AWS_VPC_CIDR` for the management cluster to use. Set either `AWS_VPC_CIDR` or `AWS_VPC_ID`, but not both.
`BASTION_HOST_ENABLED`	`"true"` or `"false"`	This option is set to `"true"` in the global Tanzu Kubernetes Grid configuration. Specify `"true"` to deploy an AWS basion host or `"false"` to reuse an existing bastion host. If no bastion host exists in your availability zone(s) and you set `AWS_VPC_ID` to use an existing VPC, set `BASTION_HOST_ENABLED` to `"true"`.
`SERVICE_CIDR`	`100.64.0.0/13`	The CIDR range to use for the Kubernetes services. The recommended range is 100.64.0.0/13. Change this value only if the recommended range is unavailable.
`CLUSTER_CIDR`	`100.96.0.0/11`	The CIDR range to use for pods. The recommended range is 100.96.0.0/11. Change this value only if the recommended range is unavailable.
`CONTROL_PLANE_MACHINE_TYPE`	For example, `t3.small`	An AWS instance type available in your specified region and availability zone(s) for the control plane node VMs, depending on the expected workloads that you will run in the cluster. Possible values may include `i3.xlarge`, `r4.8xlarge`, `m5a.4xlarge`, `m5.large`, and `t3.small`, but see Finding an Amazon EC2 instance type in the AWS documentation for actual choices. You can also specify or override this value by using the `tkg init --size` or `--controlplane-size` options.
`NODE_MACHINE_TYPE`	For example, `m5.large`	An AWS instance type available in your specified region and availability zone(s) for the worker node VMs, depending on the expected workloads that you will run in the cluster. Possible values may include `i3.xlarge`, `r4.8xlarge`, `m5a.4xlarge`, `m5.large`, and `t3.small`, but see Finding an Amazon EC2 instance type in the AWS documentation for actual choices. You can also specify or override this value by using the `tkg init --size` or `--worker-size` options.
`ENABLE_MHC`	`"true"` or `"false"`	Enables or disables the `MachineHealthCheck` controller, which provides node health monitoring and node auto-repair for Tanzu Kubernetes clusters. This option is enabled in the global Tanzu Kubernetes Grid configuration by default, for all Tanzu Kubernetes clusters. To disable `MachineHealthCheck` on the clusters that you deploy with this management cluster, set `ENABLE_MHC` to `false`. Set this variable only if you want to override your global configuration. You can enable or disable `MachineHealthCheck` on individual clusters after deployment by using the CLI. For instructions, see Configure Machine Health Checks for Tanzu Kubernetes Clusters.
`MHC_UNKNOWN_STATUS_TIMEOUT`	For example, `10m`	Property of `MachineHealthCheck`. By default, if the `Ready` condition of a node remains `Unknown` for longer than `5m`, `MachineHealthCheck` considers the machine unhealthy and recreates it. Set this variable if you want to change the default timeout.
`MHC_FALSE_STATUS_TIMEOUT`	For example, `10m`	Property of `MachineHealthCheck`. By default, if the `Ready` condition of a node remains `False` for longer than `5m`, `MachineHealthCheck` considers the machine unhealthy and recreates it. Set this variable if you want to change the default timeout.

Create an AWS CloudFormation Stack

Before deploying a management cluster to Amazon EC2 for the first time, you must create an AWS CloudFormation stack for Tanzu Kubernetes Grid in your AWS account. To create the CloudFormation stack, you run the tkg config permissions aws command from the CLI or enable the Automate creation of AWS CloudFormation Stack in the installer interface.

This CloudFormation stack provides the identity and access management (IAM) resources that Tanzu Kubernetes Grid needs to create management clusters and Tanzu Kubernetes clusters on Amazon EC2. The IAM resources are added to the control plane and node roles when they are created during cluster deployment.

You need to create only one CloudFormation stack per AWS account. The IAM resources that the CloudFormation stack provides are global, meaning they are not specific to any region. For more information about CloudFormation stacks, see Working with Stacks in the AWS documentation.

IMPORTANT: In Tanzu Kubernetes Grid v1.2 and later, the Automate creation of AWS CloudFormation Stack checkbox and the tkg config permissions aws command replace the clusterawsadm command line utility. For existing management and Tanzu Kubernetes clusters, initially deployed with v1.1.x or earlier, continue to use the CloudFormation stack that you created by running the clusterawsadm alpha bootstrap create-stack command. If you want to use the same AWS account for your existing clusters and Tanzu Kubernetes Grid v1.2 and later, both stacks must be present in the account. For more information, see Prepare to Upgrade Clusters on Amazon EC2.

Do one of the following:

If you have already created the CloudFormation stack for Tanzu Kubernetes Grid in your AWS account, proceed to Run the tkg init Command below.
If you have not already created the CloudFormation stack for Tanzu Kubernetes Grid in your AWS account:
1. Ensure that the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION are set in your configuration file or set them in the AWS default credential provider chain. For more information, see Working with AWS Credentials in the AWS documentation.
2. Run the following command:
```
tkg config permissions aws
```
  To specify the path to your configuration file, use the --config flag. The default path is $HOME/.tkg/config.yaml. For more information about this command, run tkg config permissions aws --help.

Run the `tkg init` Command

After you have updated .tkg/config.yaml, or another cluster configuration file, you deploy a management cluster by running the tkg init command. The cluster configuration file provides the base configuration for management clusters and Tanzu Kubernetes clusters. You provide more precise configuration information for individual clusters by running the tkg init command with different options.

Warning: The tkg init command takes time to complete. While tkg init is running, do not run additional invocations of tkg init on the same bootstrap machine to deploy multiple management clusters, run tkg set management-cluster to change context, or edit ~/.kube-tkg/config.

To deploy a management cluster to Amazon EC2, you must at least specify the --infrastructure aws option to tkgi init:

tkg init --infrastructure aws

The table in CLI Options Reference describes additional command-line options for deploying a management cluster to Amazon EC2.

Monitoring Progress

When you run tkg init, you can follow the progress of the deployment of the management cluster in the terminal. For more detail, open the log file listed in the terminal output Logs of the command execution can also be found at....

Deployment of the management cluster can take several minutes. The first run of tkg init takes longer than subsequent runs because it has to pull the required Docker images into the image store on your bootstrap machine. Subsequent runs do not require this step, so are faster.

The first run of tkg init also adds settings to your configuration file.

If tkg init fails before the management cluster deploys to AWS, you should clean up artifacts on your bootstrap machine before you re-run tkg init. See the Troubleshooting Tips topic for details. If the machine on which you run tkg init shuts down or restarts before the local operations finish, the deployment will fail.

Validation Checks

When you run tkg init, the command performs several validation checks before deploying the management cluster. The command verifies that the target Amazon EC2 infrastructure meets the following requirements:

The AWS credentials that you provided are valid.
Cloud Formation stack exists.
Node Instance type is supported.
Region and AZ match.
AWS_AMI_ID is valid.

When these conditions are not met, the tkg init command fails.

CLI Options Reference

The table below describes command-line options that you can set for deployment to AWS.

For example, to create a highly available Amazon EC2 management cluster aws-mgmt-cluster in which all of the control plane and worker node VMs are t3.xlarge size:

tkg init --infrastructure aws --name aws-mgmt-cluster --plan prod --size t3.xlarge

Option	Value	Description
`--infrastructure`	`aws`	Required
`--name`	Name for the management cluster	Name must comply with DNS hostname requirements as outlined in RFC 952 and amended in RFC 1123 If you do not specify a name, a unique name is generated.
Scaling and Availability
`--plan`	`dev` or `prod`	`dev` ("development"), the default, deploys a management cluster with a single control plane node. `prod` ("production") deploys a highly available management cluster with three control plane nodes.
Configuration Files
`--config`	Local file system path to `.yaml` file, e.g. `/path/to/file/my-config.yaml`	Configuration file to use or create, other than the default `$HOME/.tkg/config.yaml`. If the config file was not already created by hand or prior `tkg init` calls, it is created. All other files are created in the default folder. This option, for example, lets you deploy multiple management clusters that share a VPC.
`--kubeconfig`	Local file system path to `.yaml` file, e.g. `/path/to/file/my-kubeconfig.yaml`	Kube configuration file or use, or create, other than the default `$HOME/.kube-tkg/config.yaml`. This option lets you customize or share multiple `kubeconfig` files for multiple management clusters.
Nodes
`--size`	An AWS instance type available in your specified region and availability zone(s). Possible values may include `i3.xlarge`, `r4.8xlarge`, `m5a.4xlarge`, `m5.large`, and `t3.small`, but see Finding an Amazon EC2 instance type in the AWS documentation for actual choices.	Size for both control plane and worker node VMs.
`--controlplane-size`		Size for control plane node VMs. Overrides the `--size` option and `CONTROL_PLANE_MACHINE_TYPE` parameter.
`--worker-size`		Size for worker node VMs. Overrides the `--size` option and `NODE_MACHINE_TYPE` parameter.
Customer Experience Improvement Program
`--ceip-participation`	`true` or `false`	`false` opts out of the VMware Customer Experience Improvement Program. Default is `true`. You can also opt in or out of the program after deploying the management cluster. For information, see Opt in or Out of the VMware CEIP and Customer Experience Improvement Program ("CEIP").

What to Do Next

For information about what happened during the deployment of the management cluster, how to connect kubectl to the management cluster, and how to create namespaces see Examine the Management Cluster Deployment.
For information about how to create namespaces in the management cluster, see Create Namespaces in the Management Cluster.
If you need to deploy more than one management cluster, on any or all of vSphere, Azure, and Amazon EC2, see Manage Your Management Clusters. This topic also provides information about how to add existing management clusters to your CLI instance, obtain credentials, scale and delete management clusters, and how to opt in or out of the CEIP.