Clusters on AWS

This topic describes ways of configuring Tanzu Kubernetes Grid (TKG) workload clusters to use features that are specific to AWS, and that are not entirely configurable in the cluster’s flat configuration file or Kubernetes-style object spec.

For information about how configure workload clusters on AWS using configuration files and object specs, see AWS Cluster Configuration Files.

Important

Tanzu Kubernetes Grid v2.4.x is the last version of TKG that supports the creation of TKG workload clusters on AWS. The ability to create TKG workload clusters on AWS will be removed in the Tanzu Kubernetes Grid v2.5 release.

Starting from now, VMware recommends that you use Tanzu Mission Control to create native AWS EKS clusters instead of creating new TKG workload clusters on AWS. For information about how to create native AWS EKS clusters with Tanzu Mission Control, see Managing the Lifecycle of AWS EKS Clusters in the Tanzu Mission Control documentation.

For more information, see Deprecation of TKG Management and Workload Clusters on AWS and Azure in the VMware Tanzu Kubernetes Grid v2.4 Release Notes.

Adding Subnet Tags for an Existing VPC

If you want to create services of type LoadBalancer in the cluster, you must add the kubernetes.io/cluster/YOUR-CLUSTER-NAME=shared tag to the public subnet or subnets that you intend to use for your workload cluster.

Adding the kubernetes.io/cluster/YOUR-CLUSTER-NAME=shared tag to the public subnet or subnets enables you to create services of type LoadBalancer after you deploy the cluster. To add this tag and then deploy the cluster, follow the steps below:

  1. Gather the ID or IDs of the public subnet or subnets within your VPC that you want to use for the cluster. To deploy a prod workload cluster, you must provide three subnets.

  2. Create the required tag by running the following command:

    aws ec2 create-tags --resources YOUR-PUBLIC-SUBNET-ID-OR-IDS --tags Key=kubernetes.io/cluster/YOUR-CLUSTER-NAME,Value=shared

    Where:

    • YOUR-PUBLIC-SUBNET-ID-OR-IDS is the ID or IDs of the public subnet or subnets that you gathered in the previous step.
    • YOUR-CLUSTER-NAME is the name of the workload cluster that you want to create.

    For example:

    aws ec2 create-tags --resources subnet-00bd5d8c88a5305c6 subnet-0b93f0fdbae3436e8 subnet-06b29d20291797698 --tags Key=kubernetes.io/cluster/my-cluster,Value=shared

  3. Configure the cluster.

  4. Create the cluster. For example:

    tanzu cluster create my-cluster -f my-cluster-config.yaml

Deploy a GPU-Enabled Cluster

To deploy a workload cluster that takes advantage of NVIDIA GPU-based VMs available on AWS:

  1. In the configuration file for the cluster, set NODE_MACHINE_TYPE, for worker nodes, to a GPU-compatible VM type, such as g4dn.8xlarge.

  2. Deploy the cluster with the cluster configuration file:

    tanzu cluster create MY-GPU-CLUSTER -f MY-GPU-CONFIG

  3. Install a GPU cluster policy and GPU operator on the cluster:

    1. Set the kubectl context to the cluster, if it is not already the current context.

    2. Download the required NVIDIA GPU resources from the Cluster API Provider AWS repository and save them to your current directory:

    3. Apply the cluster policy:

      kubectl apply clusterpolicy-crd.yaml

    4. Apply the GPU operator:

      kubectl apply gpu-operator-components.yaml

  4. Run kubectl get pods -A. You should see listings for gpu-operator- pods in the default namespace and nvidia- pods in the gpu-operator-resources namespace.

  5. To test the GPU-enabled cluster, run the CUDA VectorAdd vector addition test in the NVIDIA documentation.

  6. To test the GPU operator:

    1. Scale up the workload cluster’s worker node count:

      tanzu cluster scale MY-GPU-CLUSTER -w 2

    2. Run kubectl get pods -A again. You should see additional gpu-operator- and nvidia- pods listed for the added nodes.

Deploy Clusters on Different AWS Accounts

You can deploy workload clusters across multiple AWS accounts if your Kubernetes clusters are either internet-facing or deployed into peered VPCs via VPC peering or AWS Transit Gateway.

To prepare a secondary AWS account for deploying workload clusters, prepare the secondary account and set up trust relationships between it and the management cluster account as follows:

Create IAM Roles

First, you need set up IAM roles in the secondary AWS account.

To do this, use the tanzu mc permissions aws command. Follow the same process as described in the Create IAM Resources section of Deploy Management Clusters from a Configuration File.

Enable Cross-Account Trust

To enable a management cluster in one AWS account to deploy workload clusters into a secondary AWS account, you first need to set up a trust policy in the secondary account.

To do this, find the controllers.tkg.cloud.vmware.com created by tanzu mc permissions aws in the secondary account. Then edit the trust relationship as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Principal": {
        "AWS": "arn:aws:iam::MANAGEMENT-ACCOUNT-ID:root"
      },
    }
  ]
}

Where MANAGEMENT-ACCOUNT-ID is the AWS account ID where the management cluster is deployed.

Enable the Management Cluster to Assume IAM Role

After setting up the trust policy, enable the management cluster account’s control-plane.tkg.cloud.vmware.com IAM role to assume the controllers.tkg.cloud.vmware.com IAM role in the secondary account.

To do this, attach a new policy or add an inline policy as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::SECONDARY-ACCOUNT-ID:role/controllers.tkg.cloud.vmware.com"
    }
  ]
}

Where SECONDARY-ACCOUNT-ID is the AWS account ID of the secondary account.

Create an AWSClusterRoleIdentity Resource in the Management Cluster

To set up the new cross-account IAM role in the management cluster, you need to create a new AWSClusterRoleIdentity resource object in Kubernetes as follows:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
kind: AWSClusterRoleIdentity
metadata:
  name: IDENTITY-NAME
spec:
  allowedNamespaces: {}
  durationSeconds: 900
  roleARN: "arn:aws:iam::SECONDARY-ACCOUNT-ID:role/controllers.tkg.cloud.vmware.com"
  sourceIdentityRef:
    kind: AWSClusterControllerIdentity
    name: default

Where:

  • IDENTITY-NAME is anything to identify the resource. For example a name that represents a line of business’s dev account (LOB-dev)
  • SECONDARY-ACCOUNT-ID is the ID from the previous setup steps.

AWSClusterRoleIdentity resources are globally scoped. You can set the allowedNamespaces field to restrict which namespaces are allowed to manage clusters using the IAM role, by setting its value to either an explicit list of namespaces, or a selector. See the Secure Access to Identities in The Cluster API Book.

Deploy a Workload Cluster to the Secondary Account

After creating the AWSClusterRoleIdentity resource, you can use it to deploy a workload cluster to the secondary AWS account.

To do this, include the following line in your cluster configuration file, and run tanzu cluster create with any of the standard options:

AWS_IDENTITY_REF_NAME: IDENTITY-NAME

Where IDENTITY-NAME is the name of the AWSClusterRoleIdentity created in the previous step.

check-circle-line exclamation-circle-line close-line
Scroll to top icon