RSS
 

Create a Management Cluster Configuration File

This documentation includes configuration file templates that you can use to deploy management clusters to each of vSphere without a Supervisor, Amazon Web Services (AWS), and Azure. The templates include all of the options that are relevant to deploying management clusters on a given infrastructure provider. You can copy the templates and follow the instructions in this section to update them.

Consult the Tanzu CLI Configuration File Variable Reference for details about each setting. The sections below also contain links to other sections of this documentation to provide additional information.

Important

As described in Configuring the Management Cluster, environment variables override values from a cluster configuration file. To use all settings from a cluster configuration file, unset any conflicting environment variables before you deploy the management cluster from the CLI.

Support for IPv6 addresses in Tanzu Kubernetes Grid is limited; see Deploy Clusters on IPv6 (vSphere Only). If you are not deploying to an IPv6-only networking environment, all IP address settings in your configuration files must be IPv4.

Some parameters configure identical properties. For example, the SIZE property configures the same infrastructure settings as all of the control plane and worker node size and type properties for the different infrastructure providers, but at a more general level. In such cases, avoid setting conflicting or redundant properties.

Create the Configuration File

  1. Copy and paste the contents of the template for your infrastructure provider into a text editor.

    Copy a template from one of the following locations:

    For example, if you have already deployed a management cluster from the installer interface, you can save the file in the default location for cluster configurations, ~/.config/tanzu/tkg/clusterconfigs.

  2. Save the file with a .yaml extension and an appropriate name, for example aws-mgmt-cluster-config.yaml.

The subsequent sections describe how to update the settings that are common to all infrastructure providers as well as the settings that are specific to each of vSphere, AWS, and Azure.

Configure Basic Management Cluster Creation Information

The basic management cluster creation settings define the infrastructure on which to deploy the management cluster and other basic settings. They are common to all infrastructure providers.

  • For CLUSTER_PLAN specify whether you want to deploy a development cluster, which provides a single control plane node, or a production cluster, which provides a highly available management cluster with three control plane nodes. Specify dev or prod.

  • For INFRASTRUCTURE_PROVIDER, specify aws, azure, or vsphere.

    INFRASTRUCTURE_PROVIDER: aws

    INFRASTRUCTURE_PROVIDER: azure

    INFRASTRUCTURE_PROVIDER: vsphere

  • Optionally deactivate participation in the VMware Customer Experience Improvement Program (CEIP) by setting ENABLE_CEIP_PARTICIPATION to false. For information about the CEIP, see Manage Participation in CEIP and https://www.vmware.com/solutions/trustvmware/ceip.html.

  • Optionally deactivate audit logging by setting ENABLE_AUDIT_LOGGING to false. For information about audit logging, see Audit Logging.
  • If the recommended CIDR ranges of 100.64.0.0/13 and 100.96.0.0/11 are unavailable, update CLUSTER_CIDR for the cluster pod network and SERVICE_CIDR for the cluster service network.

For example:

#! ---------------------------------------------------------------------
#! Basic cluster creation configuration
#! ---------------------------------------------------------------------

CLUSTER_NAME: aws-mgmt-cluster
CLUSTER_PLAN: dev
INFRASTRUCTURE_PROVIDER: aws
ENABLE_CEIP_PARTICIPATION: true
ENABLE_AUDIT_LOGGING: true
CLUSTER_CIDR: 100.96.0.0/11
SERVICE_CIDR: 100.64.0.0/13

Configure Identity Management

Set IDENTITY_MANAGEMENT_TYPE to ldap or oidc. Set to none or omit to deactivate identity management. It is strongly recommended to enable identity management for production deployments.

IDENTITY_MANAGEMENT_TYPE: oidc

IDENTITY_MANAGEMENT_TYPE: ldap

OIDC

To configure OIDC, update the variables below. For information about how to configure the variables, see Identity Providers - OIDC in the Tanzu CLI Configuration File Variable Reference.

For example:

OIDC_IDENTITY_PROVIDER_CLIENT_ID: 0oa2i[...]NKst4x7
OIDC_IDENTITY_PROVIDER_CLIENT_SECRET: 331!b70[...]60c_a10-72b4
OIDC_IDENTITY_PROVIDER_GROUPS_CLAIM: groups
OIDC_IDENTITY_PROVIDER_ISSUER_URL: https://dev-[...].okta.com
OIDC_IDENTITY_PROVIDER_SCOPES: openid,groups,email
OIDC_IDENTITY_PROVIDER_USERNAME_CLAIM: email

LDAP

To configure LDAP, uncomment and update the LDAP_* variables with information about your LDAPS server. For information about how to configure the variables, see Identity Providers - LDAP in the Tanzu CLI Configuration File Variable Reference.

For example:

LDAP_BIND_DN: ""
LDAP_BIND_PASSWORD: ""
LDAP_GROUP_SEARCH_BASE_DN: dc=example,dc=com
LDAP_GROUP_SEARCH_FILTER: (objectClass=posixGroup)
LDAP_GROUP_SEARCH_GROUP_ATTRIBUTE: memberUid
LDAP_GROUP_SEARCH_NAME_ATTRIBUTE: cn
LDAP_GROUP_SEARCH_USER_ATTRIBUTE: uid
LDAP_HOST: ldaps.example.com:636
LDAP_ROOT_CA_DATA_B64: ""
LDAP_USER_SEARCH_BASE_DN: ou=people,dc=example,dc=com
LDAP_USER_SEARCH_FILTER: (objectClass=posixAccount)
LDAP_USER_SEARCH_NAME_ATTRIBUTE: uid
LDAP_USER_SEARCH_USERNAME: uid

Configure Proxies

To optionally send outgoing HTTP(S) traffic from the management cluster to a proxy, for example in an internet-restricted environment, uncomment and set the *_PROXY settings. The proxy settings are common to all infrastructure providers. You can choose to use one proxy for HTTP requests and another proxy for HTTPS requests or to use the same proxy for both HTTP and HTTPS requests. You cannot change the proxy after you deploy the cluster.

Note

On vSphere, traffic from cluster VMs to vCenter cannot be proxied. In a proxied vSphere environment, you need to either set VSPHERE_INSECURE to true, or else add the vCenter IP address or hostname to the TKG_NO_PROXY list.

  • TKG_HTTP_PROXY_ENABLED: Set this to true to configure a proxy.

  • TKG_PROXY_CA_CERT: Set this to the proxy server’s CA if its certificate is self-signed.

  • TKG_HTTP_PROXY: This is the URL of the proxy that handles HTTP requests. To set the URL, use the format below:

    PROTOCOL://USERNAME:PASSWORD@FQDN-OR-IP:PORT

    Where:

    • (Required) PROTOCOL: This must be http.
    • (Optional) USERNAME and PASSWORD: This is your HTTP proxy username and password. You must set USERNAME and PASSWORD if the proxy requires authentication.
    Note

    When deploying management clusters with CLI, the following non-alphanumeric characters cannot be used in passwords: # ` | / \ ? % { [ ] } " < > @ .

    For supported characters like !, use the character itself rather than the character code and put the password in double-quotes.

    • (Required) FQDN-OR-IP: This is the FQDN or IP address of your HTTP proxy.
    • (Required) PORT: This is the port number that your HTTP proxy uses.

    For example, http://user:password@myproxy.com:1234.

  • TKG_HTTPS_PROXY: This is the URL of the proxy that handles HTTPS requests. You can set TKG_HTTPS_PROXY to the same value as TKG_HTTP_PROXY or provide a different value. To set the value, use the URL format from the previous step, where:

    • (Required) PROTOCOL: This must be http.
    • (Optional) USERNAME and PASSWORD: This is your HTTPS proxy username and password. You must set USERNAME and PASSWORD if the proxy requires authentication.
    Note

    When deploying management clusters with CLI, the following non-alphanumeric characters cannot be used in passwords: # ` | / \ ? % { [ ] } " < > @ .

    For supported characters like !, use the character itself rather than the character code and put the password in double-quotes.

    • (Required) FQDN-OR-IP: This is the FQDN or IP address of your HTTPS proxy.
    • (Required) PORT: This is the port number that your HTTPS proxy uses.

    For example, http://user:password@myproxy.com:1234.

  • TKG_NO_PROXY: This sets one or more comma-separated network CIDRs or hostnames that must bypass the HTTP(S) proxy, for example to enable the management cluster to communicate directly with infrastructure that runs on the same network, behind the same proxy. Do not use spaces in the comma-separated list setting. For example, noproxy.yourdomain.com,192.168.0.0/24.

    On vSphere, this list must include:

    • Your vCenter IP address or hostname.
    • The CIDR of VSPHERE_NETWORK, which includes the IP address of your control plane endpoint. If you set VSPHERE_CONTROL_PLANE_ENDPOINT to an FQDN, also add that FQDN to the TKG_NO_PROXY list.

    Internally, Tanzu Kubernetes Grid appends localhost, 127.0.0.1, the values of CLUSTER_CIDR and SERVICE_CIDR, .svc, and .svc.cluster.local to the value that you set in TKG_NO_PROXY. It also appends your AWS VPC CIDR and 169.254.0.0/16 for deployments to AWS and your Azure VNET CIDR, 169.254.0.0/16, and 168.63.129.16 for deployments to Azure. For vSphere, you must manually add the CIDR of VSPHERE_NETWORK, which includes the IP address of your control plane endpoint, to TKG_NO_PROXY. If you set VSPHERE_CONTROL_PLANE_ENDPOINT to an FQDN, add both the FQDN and VSPHERE_NETWORK to TKG_NO_PROXY.

    Important

    If the cluster VMs need to communicate with external services and infrastructure endpoints in your Tanzu Kubernetes Grid environment, ensure that those endpoints are reachable by the proxies that you set above or add them to TKG_NO_PROXY. Depending on your environment configuration, this may include, but is not limited to:

    • Your OIDC or LDAP server
    • Harbor
    • VMware NSX
    • NSX Advanced Load Balancer
    • AWS VPC CIDRs that are external to the cluster

For example:

#! ---------------------------------------------------------------------
#! Proxy configuration
#! ---------------------------------------------------------------------

TKG_HTTP_PROXY_ENABLED: true
TKG_PROXY_CA_CERT: "LS0t[...]tLS0tLQ==""
TKG_HTTP_PROXY: "http://myproxy.com:1234"
TKG_HTTPS_PROXY: "http://myproxy.com:1234"
TKG_NO_PROXY: "noproxy.yourdomain.com,192.168.0.0/24"

Configure Node Settings

By default, all cluster nodes run Ubuntu v20.04, for all infrastructure providers. On vSphere you can optionally deploy clusters that run Photon OS on their nodes. On AWS, nodes can optionally run Amazon Linux 2. For the architecture, the default and only current choice is amd64. For the OS and version settings, see see Node Configuration in the Tanzu CLI Configuration File Variable Reference.

For example:

#! ---------------------------------------------------------------------
#! Node configuration
#! ---------------------------------------------------------------------

OS_NAME: "photon"
OS_VERSION: "3"
OS_ARCH: "amd64"

How you set node compute configuration and sizes depends on the infrastructure provider. For information, see Management Cluster Configuration for vSphere, Management Cluster Configuration for AWS, or Management Cluster Configuration for Microsoft Azure.

Configure Machine Health Checks

Optionally, update variables based on your deployment preferences and using the guidelines described in the Machine Health Checks section of Tanzu CLI Configuration File Variable Reference.

For example:

ENABLE_MHC:
ENABLE_MHC_CONTROL_PLANE: true
ENABLE_MHC_WORKER_NODE: true
MHC_UNKNOWN_STATUS_TIMEOUT: 10m
MHC_FALSE_STATUS_TIMEOUT: 20m

Configure a Private Image Registry

If you are deploying the management cluster in an Internet-restricted environment, uncomment and update the TKG_CUSTOM_IMAGE_REPOSITORY_* settings. These settings are common to all infrastructure providers. You do not need to configure the private image registry settings if:

  • You are deploying the management cluster in an Internet-restricted environment and you have set the TKG_CUSTOM_IMAGE_REPOSITORY_* variables by running the tanzu config set command, as described in Prepare an Internet-Restricted Environment. Environment variables set by running tanzu config set override values from a cluster configuration file.
  • If you are deploying the management cluster in an environment that has access to the external internet.

For example:

#! ---------------------------------------------------------------------
#! Image repository configuration
#! ---------------------------------------------------------------------

TKG_CUSTOM_IMAGE_REPOSITORY: "custom-image-repository.io/yourproject"
TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE: "LS0t[...]tLS0tLQ=="

Configure Antrea CNI

By default, clusters that you deploy with the Tanzu CLI provide in-cluster container networking with the Antrea container network interface (CNI).

You can optionally deactivate Source Network Address Translation (SNAT) for pod traffic, implement hybrid, noEncap, NetworkPolicyOnly traffic encapsulation modes, use proxies and network policies, and implement Traceflow.

For more information about Antrea, see the following resources:

To optionally configure these features on Antrea, uncomment and update the ANTREA_* variables. For example:

#! ---------------------------------------------------------------------
#! Antrea CNI configuration
#! ---------------------------------------------------------------------

ANTREA_NO_SNAT: true
ANTREA_NODEPORTLOCAL: true
ANTREA_NODEPORTLOCAL_ENABLED: true
ANTREA_NODEPORTLOCAL_PORTRANGE: 61000-62000
ANTREA_TRAFFIC_ENCAP_MODE: "encap"
ANTREA_PROXY: true
ANTREA_PROXY_ALL: true
ANTREA_PROXY_LOAD_BALANCER_IPS: false
ANTREA_PROXY_NODEPORT_ADDRS:
ANTREA_PROXY_SKIP_SERVICES: ""
ANTREA_POLICY: true
ANTREA_TRACEFLOW: true
ANTREA_DISABLE_UDP_TUNNEL_OFFLOAD: false
ANTREA_ENABLE_USAGE_REPORTING: false
ANTREA_EGRESS: true
ANTREA_EGRESS_EXCEPT_CIDRS: ""
ANTREA_FLOWEXPORTER: false
ANTREA_FLOWEXPORTER_COLLECTOR_ADDRESS: "flow-aggregator.flow-aggregator.svc:4739:tls"
ANTREA_FLOWEXPORTER_POLL_INTERVAL: "5s"
ANTREA_FLOWEXPORTER_ACTIVE_TIMEOUT: "5s"
ANTREA_FLOWEXPORTER_IDLE_TIMEOUT: "15s"
ANTREA_IPAM: false
ANTREA_KUBE_APISERVER_OVERRIDE: ""
ANTREA_MULTICAST: false
ANTREA_MULTICAST_INTERFACES: ""
ANTREA_NETWORKPOLICY_STATS: true
ANTREA_SERVICE_EXTERNALIP: true
ANTREA_TRANSPORT_INTERFACE: ""
ANTREA_TRANSPORT_INTERFACE_CIDRS: ""

What to Do Next

Continue to update the configuration file settings for vSphere, AWS, or Azure. For the configuration file settings that are specific to each infrastructure provider, see the corresponding topic:

check-circle-line exclamation-circle-line close-line
Scroll to top icon