Before you can use the Tanzu Kubernetes Grid CLI or installer interface to deploy a management cluster, you must prepare the machine on which you run the Tanzu Kubernetes Grid CLI and set up your Amazon Web Services Account (AWS) account.

If you are installing Tanzu Kubernetes Grid on VMware Cloud on AWS, you are installing to a vSphere environment. See Preparing VMware Cloud on AWS in Prepare a vSphere Management as a Service Infrastructure to prepare your environment, and Deploy Management Clusters to vSphere to deploy management clusters.

General Requirements

  • The Tanzu Kubernetes Grid CLI installed locally. See Install the Tanzu Kubernetes Grid CLI.
  • You have the access key and access key secret for an active AWS account.
  • Your AWS account must have Administrator privileges.
  • Your AWS account has sufficient resource quotas for the following. For more information, see Amazon VPC Quotas in the AWS documentation and Resource Usage in Your Amazon Web Services Account below:
    • Virtual Private Cloud (VPC) instances. By default, each management cluster that you deploy creates one VPC and one or three NAT gateways. The default NAT gateway quota is 5 instances per availability zone, per account.
    • Elastic IP (EIP) addresses. The default EIP quota is 5 EIP addresses per region, per account.
  • Traffic is allowed between your local bootstrap machine and port 6443 of all VMs in the clusters you create. Port 6443 is where the Kubernetes API is exposed.
  • Traffic is allowed between your local bootstrap machine and the image repositories listed in the management cluster Bill of Materials (BoM) file, over port 443, for TCP.*
    • The BoM file is under ~/.tkg/bom/ and its name includes the Tanzu Kubernetes Grid version, for example bom-1.2.0+vmware.1.yaml for v1.2.0.
    • Run a DNS lookup on all imageRepository values to find their CNAMEs, for example registry.tkg.vmware.run requires network access to registry.tkg.vmware.run.bintray.com.
  • The AWS CLI installed locally.
  • jq installed locally.

    The AWS CLI uses jq to process JSON when creating SSH key pairs. It is also used to prepare the environment or configuration variables when you deploy Tanzu Kubernetes Grid by using the CLI.

*Or see Deploying Tanzu Kubernetes Grid in an Internet-Restricted Environment for installing without external network access.

Resource Usage in Your Amazon Web Services Account

For each cluster that you create, Tanzu Kubernetes Grid provisions a set of resources in your AWS account.

For development management clusters that are not configured for high availability, Tanzu Kubernetes Grid provisions the following resources:

  • 3 VMs, including a control plane node, a worker node (to run the cluster agent extensions) and, by default, a bastion host. If you specify additional VMs in your node pool, those are provisioned as well.
  • 4 security groups, one for the load balancer and one for each of the initial VMs.
  • 1 private subnet and 1 public subnet in the specified availability zone.
  • 1 public and 1 private route table in the specified availability zone.
  • 1 classic load balancer.
  • 1 internet gateway.
  • 1 NAT gateway in the specified availability zone.
  • By default, 1 EIP, for the NAT gateway, when clusters are deployed in their own VPC. You can optionally share VPCs rather than creating new ones, such as a workload cluster sharing a VPC with its management cluster.

For production management clusters, which are configured for high availability, Tanzu Kubernetes Grid provisions the following resources to support distribution across three availability zones:

  • 3 control plane VMs
  • 3 private and public subnets
  • 3 private and public route tables
  • 3 NAT gateways
  • By default, 3 EIPs, one for each NAT gateway, for clusters deployed in their own VPC. You can optionally share VPCs rather than creating new ones, such as a workload cluster sharing a VPC with its management cluster.

AWS implements a set of default limits or quotas on these types of resources and allows you to modify the limits. Typically, the default limits are sufficient to get started creating clusters from Tanzu Kubernetes Grid. However, as you increase the number of clusters you are running or the workloads on your clusters, you will encroach on these limits. When you reach the limits imposed by AWS, any attempts to provision that type of resource fail. As a result, Tanzu Kubernetes Grid will be unable to create a new cluster, or you might be unable to create additional deployments on your existing clusters. Therefore, regularly assess the limits you have specified in AWS account and adjust them as necessary to fit your business needs.

For information about the sizes of cluster node instances, see Amazon EC2 Instance Types in the AWS documentation.

Virtual Private Clouds and NAT Gateway Limits

If you create a new Virtual Private Cloud (VPC) when you deploy a management cluster, Tanzu Kubernetes Grid also creates a dedicated NAT gateway for the management cluster or if you deploy a production management cluster, three NAT gateways, one in each of the availability zones. In this case, by default, Tanzu Kubernetes Grid creates a new VPC and one or three NAT gateways for each Tanzu Kubernetes cluster that you deploy from that management cluster. By default, AWS allows 5 NAT gateways per availability zone per account. Consequently, if you always create a new VPC for each cluster, you can create only 5 development clusters in a single availability zone. If you already have five NAT gateways in use, Tanzu Kubernetes Grid is unable to provision the necessary resources when you attempt to create a new cluster. If you do not want to change the default quotas, to create more than 5 development clusters in a given availability zone, you must share existing VPCs, and therefore their NAT gateways, between multiple clusters.

There are 3 possible scenarios regarding VPCs and NAT gateway usage when you deploy management clusters and Tanzu Kubernetes clusters.

  • Create a new VPC and NAT gateway(s) for every management cluster and Tanzu Kubernetes cluster

    If you deploy a management cluster and use the option to create a new VPC and if you make no modifications to the configuration when you deploy Tanzu Kubernetes clusters from that management cluster, the deployment of each of the Tanzu Kubernetes clusters also creates a VPC and one or three NAT gateways. In this scenario, you can deploy one development management cluster and up to 4 development Tanzu Kubernetes clusters, due to the default limit of 5 NAT gateways per availability zone.

  • Reuse a VPC and NAT gateway(s) that already exist in your availability zone(s)

    If a VPC already exists in the availability zone(s) in which you are deploying a management cluster, for example a VPC that you created manually or by using tools such as CloudFormation or Terraform, you can specify that the management cluster should use this VPC. In this case, all of the Tanzu Kubernetes clusters that you deploy from that management cluster also use the specified VPC and its NAT gateway(s).

    An existing VPC must be configured with the following networking:

    • Two subnets for development clusters or six subnets for production clusters
    • One NAT gateway for development clusters or three NAT gateways for production clusters
    • One internet gateway and corresponding routing tables
  • Create a new VPC and NAT gateway(s) for the management cluster and deploy Tanzu Kubernetes clusters that share that VPC and NAT gateway(s)

    If you are starting with an empty availability zone(s), you can deploy a management cluster and use the option to create a new VPC. If you want the Tanzu Kubernetes clusters to share a VPC that Tanzu Kubernetes Grid created, you must modify the cluster configuration when you deploy Tanzu Kubernetes clusters from this management cluster.

For information about how to deploy management clusters that either create or reuse a VPC, see Deploy Management Clusters to Amazon EC2 with the Installer Interface and Deploy Management Clusters to Amazon EC2 with the CLI.

For information about how to deploy Tanzu Kubernetes clusters that share a VPC that Tanzu Kubernetes Grid created when you deployed the management cluster, see Deploy a Cluster that Shares a VPC with the Management Cluster.

Register an SSH Public Key with Your AWS Account

To enable Tanzu Kubernetes Grid VMs to launch on Amazon EC2, you must provide the public key part of an SSH key pair to Amazon EC2 for every region in which you plan to deploy management clusters.

NOTE: AWS supports only RSA keys. The keys required by AWS are of a different format to those required by vSphere. You cannot use the same key pair for both vSphere and AWS deployments.

If you do not already have an SSH key pair, you can create one by performing the steps below:

  1. Set the following environment variables for your AWS account:

    • export AWS_ACCESS_KEY_ID=aws_access_key, where aws_access_key is your AWS access key.

    • export AWS_SECRET_ACCESS_KEY=aws_access_key_secret, where aws_access_key_secret is your AWS access key secret.

    • (Multi-factor authentication only) export AWS_SESSION_TOKEN=aws_session_token, where aws_session_token is your AWS session token. Set this variable if you use multi-factor authentication.

    • export AWS_REGION=aws_region, where aws_region is the AWS region in which you intend to deploy the cluster. For example, us-west-2.

      For the full list of AWS regions, see AWS Service Endpoints. In addition to the regular AWS regions, you can also specify the us-gov-east and us-gov-west regions in AWS GovCloud.

  2. For each region that you plan to use with Tanzu Kubernetes Grid, create a key pair named default and save it as default.pem.

    aws ec2 create-key-pair --key-name default --output json | jq .KeyMaterial -r > default.pem
    

    To create a key pair for a region that is not the default in your profile, or set locally as AWS_DEFAULT_REGION, include the --region option.

  3. Log in to your Amazon EC2 dashboard and go to Network & Security > Key Pairs to verify that the created key pair is registered with your account.

Tag AWS Resources

If both of the following are true, you must add the kubernetes.io/cluster/YOUR-CLUSTER-NAME=shared tag to the public subnet or subnets that you intend to use for the management cluster:

  • You deploy the management cluster to an existing VPC that was not created by Tanzu Kubernetes Grid.
  • You want to create services of type LoadBalancer in the management cluster.

Adding the kubernetes.io/cluster/YOUR-CLUSTER-NAME=shared tag to the public subnet or subnets enables you to create services of type LoadBalancer in the management cluster. To add this tag, follow the steps below:

  1. Gather the ID or IDs of the public subnet or subnets within your existing VPC that you want to use for the management cluster. To deploy a prod management cluster, you must provide three subnets.

  2. Create the required tag by running the following command:

    aws ec2 create-tags --resources YOUR-PUBLIC-SUBNET-ID-OR-IDS --tags Key=kubernetes.io/cluster/YOUR-CLUSTER-NAME,Value=shared
    

    Where:

    • YOUR-PUBLIC-SUBNET-ID-OR-IDS is the ID or IDs of the public subnet or subnets that you gathered in the previous step.
    • YOUR-CLUSTER-NAME is the name of the management cluster that you want to deploy.

    For example:

    aws ec2 create-tags --resources subnet-00bd5d8c88a5305c6 subnet-0b93f0fdbae3436e8 subnet-06b29d20291797698 --tags Key=kubernetes.io/cluster/my-management-cluster,Value=shared
    

If you want to use services of type LoadBalancer in a Tanzu Kubernetes cluster after you deploy the cluster to a VPC that was not created by Tanzu Kubernetes Grid, follow the tagging instructions in Deploy a Cluster to an Existing VPC and Add Subnet Tags (Amazon EC2).

What to Do Next

Your environment is now ready for you to deploy the management cluster to Amazon EC2.

check-circle-line exclamation-circle-line close-line
Scroll to top icon