Deploying VMware Tanzu for Kubernetes Operations on VMware Cloud on AWS Using Service Installer for VMware Tanzu

The following provides the steps for deploying Tanzu for Kubernetes Operations on VMware Cloud on AWS using Service Installer for VMware Tanzu.

This deployment references the design provided in VMware Tanzu for Kubernetes Operations on VMware Cloud on AWS Reference Design.

Network Design

The following diagram represents the network design required for deploying Tanzu for Kubernetes Operations on VMware Cloud on AWS using Service Installer for VMware Tanzu.

Network Design for TKO on VMC on AWS

Prerequisites

Before deploying Tanzu for Kubernetes Operations on VMware Cloud on AWS using Service Installer for VMware Tanzu, ensure the following:

  • A Software-Defined Data Center (SDDC) is running in VMC on an AWS instance.

  • A public IP address is reserved in VMC on AWS.

    A public IP address is required for bootstrapping VMs and VMware Tanzu Kubernetes Grid nodes to talk to external networks.

    For information about requesting a public IP address on VMware Cloud, see Request or Release a Public IP Address.

  • You have installed Service Installer for VMware Tanzu.

    For information on how to download and deploy VMware Service Installer for VMware Tanzu, see Service Installer for VMware Tanzu.

Overview of Steps

The following are the main steps for deploying VMware Tanzu for Kubernetes Operations on VMware Cloud on AWS using Service Installer for VMware Tanzu:

  1. Create a Network Segment
  2. Enable External Access
  3. Download NSX Advanced Load Balancer Controller
  4. Deploy Tanzu for Kubernetes Operations

Create a Network Segment

Create a network segment for Tanzu Kubernetes Grid. The Tanzu Kubernetes Grid management network segment is the only network that is configured manually. You attach the Service Installer for VMware Tanzu OVA to the Tanzu Kubernetes Grid management network. The installer deploys the Tanzu Kubernetes Grid management cluster to the network.

  1. Log in to the VMware Cloud Services portal.
  2. Navigate to your Organization.
  3. On the VMware Cloud service tile, click Launch Service.
  4. Go to Inventory.
  5. On the SDDC tile, click View Details.
  6. Go to Networking & Security > Network > Segments.

  7. Click Add Segment to create a new network segment with a unique subnet for the Tanzu Kubernetes Grid management network. Enter the following:

    • Subnets: Specify an IPv4 CIDR block for the segment.
      Make sure that the new subnet CIDR does not overlap with sddc-cgw-network-1 or any other existing segments.

    • Set DHCP Config: Configure DHCP settings.

      Note: Do not allocate all usable IPs under the DHCP range. The Tanzu Kubernetes Grid management control plane requires an IP address outside the DHCP scope.

  8. Click Save.

For additional product documentation, see Create or Modify a Network Segment.

Enable External Access

Service Installer for VMware Tanzu acts as a bootstrap machine. Set up Internet access for the Service Installer VM from the VMware Cloud portal. The Service Installer VM needs a connection to VMware Public Image Registry to pull the required container images for Tanzu Kubernetes Grid deployment.

  1. Create and configure an inventory group for the bootstrap VM.

    1. Go to Networking & Security > Inventory > Groups > Compute Groups.
    2. Click Add Group.
    3. Under Set Members, provide the IP address of the Service Installer OVA.

  2. Create a Firewall Rule.

    1. Go to Networking & Security > Security > Gateway Firewall > Compute Gateway.
    2. Click Add Rule to add a new rule.
    3. Enter the parameters for the new rule so that Service Installer OVA can communicate with DNS, NTP servers, and public networks.

    Optionally, to access the Service Installer over SSH from an external network, create NAT and add the required firewall rules.

    Note: This firewall rule is temporary. You can be delete the rule after you create your Tanzu for Kubernetes Operations environment.

For additional product documentation, see Add or Modify Compute Gateway Firewall Rules.

Download and Import NSX Advanced Load Balancer Controller and Kubernetes Templates

Download the NSX Advanced Load Balancer Controller and base Kubernetes images.

  1. Download and import the required Photon/Ubuntu Kubernetes base OVAs to vCenter. To download the images, go to VMware Tanzu Kubernetes Grid Download Product.

  2. After importing the images, convert the images to a template.

  3. Upload the NSX Advanced Load Balancer Controller OVA:
    1. Download the NSX Advanced Load Balancer (20.1.7) OVA from MarketPlace.
    2. Create a Content Library and upload the NSX Advanced Load Balancer Controller OVA.

Deploy Tanzu for Kubernetes Operations

  1. Log in to the Service Installer for VMware Tanzu VM over SSH.

    Enter ssh root@Service-Installer-IP.

  2. Configure and verify NTP.

    To configure and verify NTP on a Photon OS, see VMware KB-76088.

  3. Import a certificate and private key to the Service Installer for VMware Tanzu bootstrap VM using a copy utility such as SCP or WinSCP (for Windows).

    Note: Service Installer uses the certificate for NSX Advanced Load Balancer, Harbor, Prometheus, and Grafana. Ensure that the certificate and private key are in PEM format and are not encrypted. Encrypted certificate files are not supported. If you do not upload a certificate, Service Installer generates a self-signed certificate.

  4. Enter the following in a browser window to access the Service Installer user interface:

    http://<IP of Service Installer VM>:8888/

  5. On the VMC on AWS tile, click Deploy.

  6. On the Configure and Generate JSON tile, click Proceed.

    By default, the values entered in the GUI are saved in the vmc-tkgm.json file located at /opt/vmware/arcas/src.

    See the sample JSON file file for reference.

  7. Execute the following command to initiate the deployment:

    arcas --env vmc --file /path/to/vmc-tkgm.json --session --vmc_pre_configuration --avi_configuration --tkg_mgmt_configuration --shared_service_configuration --workload_preconfig --workload_deploy --deploy_extensions

    For /path/to/vmc-tkgm.json, enter the path to the JSON file that you created in the previous step.

  8. Use the following command to cleanup the deployment.

    arcas --env  vmc --file /path/to/vmc-tkgm.json --cleanup

    The following table describes the parameters of the command.

    Python CLI Command Parameter Description
    –session Establishes a session with CSP/VMC and performs pre-validation
    –vmc_pre_configuration Creates the required segments, firewalls rules, inventory groups, and services
    –avi_configuration Creates the resource pool and folders for NSX Advanced Load Balancer Controller
    Deploys Avi Control Plane, generates & replaces certs and performs initial configuration (DNS,NTP)
    –tkg_mgmt_configuration Configures the required networks in Avi, creates the cloud, SE group, and IPAM profile, and maps IPAM & SE group with Cloud
    Creates the resource pool and folders for TKG Mgmt Cluster
    Deploys the TKG management cluster
    Registers the TKG management cluster with TMC
    –shared_service_configuration Deploys Shared Service cluster(Makes use of TMC CLI)
    Adds required tags to the cluster
    Deploys Certmanager, Contour, and Harbor
    –workload_preconfig Creates the required network configuration in Avi, creates a new SE Groups for Workload Clusters
    Creates a new AKO config for Workload Clusters
    –workload_deploy Deploys a workload cluster (uses the Tanzu Mission Control CLI)
    Adds required tags to the cluster
    –deploy_extensions Deploys extensions (Prometheus, Grafana)
    –cleanup cleanup the deployment performed by SIVT and start from scratch
    –verbose Enables verbose logging

  9. Do the following to integrate with SaaS services such as Tanzu Mission Control, Tanzu Service Mesh, and Tanzu Observability. In the vmc-tkgm.json file,

    • to activate or deactivate Tanzu Mission Control, enter "tanzuObservabilityAvailability": "true/false".
    • to activate or deactivate Tanzu Service Mesh, enter "tkgWorkloadTsmIntegration": "true/false".
    • to activate or deactivate Tanzu Observability, enter "tanzuObservabilityAvailability": "true/false".

    Note: If you edit the JSON manually, ensure that the values you enter are in quotes.

  10. Activate or deactivate Tanzu Kubernetes Grid extensions. For example,

    • to activate or deactivate Prometheus and Grafana, enter "enableExtensions": "true/false".
    • to activate or deactivate Harbor, enter "enableHarborExtension": "true/false".

Note:

  • Tanzu Mission Control is required to activate Tanzu Service Mesh and Tanzu Observability.
  • If Tanzu Observability is activated, Prometheus and Grafana are not supported.
  • When Tanzu Mission Control is activated, only Photon is supported.

Sample Input File

Service Installer automatically generates the JSON file for deploying Tanzu Kubernetes Grid. The following following sample file is an example of an automatically generated JSON file.

Note: The sample file is also available in the Service Installer VM at the following location: /opt/vmware/arcas/src/vmc/vmc-tkgm.json.sample.

{
   "envSpec":{
      "sddcRefreshToken":"n06OXXXXXXXXXX",
      "orgName":"Tanzu",
      "sddcName":"Sver1",
      "sddcDatacenter":"SDDC-Datacenter",
      "sddcCluster":"Cluster-1",
      "sddcDatastore":"Workload-Datastore",
      "contentLibraryName":"SDDC-Local",
      "aviOvaName":"ALB-20.1.7-9154",
      "resourcePoolName":""
   },
   "marketplaceSpec":{
      "refreshToken":""
   },
   "envVariablesSpec":{
      "dnsServersIp":"",
      "searchDomains":"tanzu.lab",
      "ntpServersIp":""
   },
   "saasEndpoints":{
      "tmcDetails":{
         "tmcAvailability":"true",
         "tmcRefreshToken":"t9TfXXXXJuMCq3",
	 "tmcInstanceURL": "https://xxxx.tmc.com"
      },
      "tanzuObservabilityDetails":{
         "tanzuObservabilityAvailability":"true",
         "tanzuObservabilityUrl":"https://surf.wavefront.com",
         "tanzuObservabilityRefreshToken":"6777a3a8-XXXX-XXXX-XXXXX-797b20638660"
      }
   },
   "componentSpec":{
      "aviMgmtNetworkSpec":{
         "aviMgmtGatewayCidr":"11.12.13.14/24",
         "aviMgmtDhcpStartRange":"11.12.13.15",
         "aviMgmtDhcpEndRange":"11.12.13.28"
      },
      "aviComponentSpec":{
         "aviPasswordBase64":"Vk13YXJlMSE=",
         "aviBackupPassPhraseBase64":"Vk13YXJlMSE=",
         "enableAviHa":"false",
         "aviClusterIp":"",
         "aviClusterFqdn":"",
         "aviSize":"essentials",
         "aviCertPath":"",
         "aviCertKeyPath":""
      },
      "identityManagementSpec":{
         "identityManagementType":"oidc",
         "oidcSpec":{
            "oidcIssuerUrl":"https://xxx.xxx.com",
            "oidcClientId":"0xxxxxxx3a",
            "oidcClientSecret":"xxxxxxxxxxxxxxxxg",
            "oidcScopes":"email",
            "oidcUsernameClaim":"email",
            "oidcGroupsClaim":"group"
         },
         "ldapSpec":{
            "ldapEndpointIp":"",
            "ldapEndpointPort":"",
            "ldapBindPWBase64":"",
            "ldapBindDN":"",
            "ldapUserSearchBaseDN":"",
            "ldapUserSearchFilter":"",
            "ldapUserSearchUsername":"",
            "ldapGroupSearchBaseDN":"",
            "ldapGroupSearchFilter":"",
            "ldapGroupSearchUserAttr":"",
            "ldapGroupSearchGroupAttr":"",
            "ldapGroupSearchNameAttr":"",
            "ldapRootCAData":""
         }
      },
      "tkgClusterVipNetwork":{
         "tkgClusterVipNetworkGatewayCidr":"11.12.15.14/24",
         "tkgClusterVipDhcpStartRange":"11.12.15.15",
         "tkgClusterVipDhcpEndRange":"11.12.15.28",
         "tkgClusterVipIpStartRange":"11.12.15.30",
         "tkgClusterVipIpEndRange":"11.12.15.40"
      },
      "tkgMgmtSpec":{
         "tkgMgmtNetworkName":"TKG-Management",
         "tkgMgmtGatewayCidr":"11.12.16.14/24",
         "tkgMgmtClusterName":"mgmt-cluster",
         "tkgMgmtSize":"medium",
         "tkgMgmtDeploymentType":"dev",
         "tkgMgmtClusterCidr":"100.96.0.0/11",
         "tkgMgmtServiceCidr":"100.64.0.0/13",
         "tkgMgmtBaseOs":"photon",
         "tkgMgmtRbacUserRoleSpec":{
            "clusterAdminUsers":"",
            "adminUsers":"",
            "editUsers":"",
            "viewUsers":""
         },
         "tkgMgmtClusterGroupName":"default"
      },
      "tkgSharedServiceSpec":{
         "tkgSharedGatewayCidr":"11.12.17.14/24",
         "tkgSharedDhcpStartRange":"11.12.17.15",
         "tkgSharedDhcpEndRange":"11.12.17.30",
         "tkgSharedClusterName":"shared-cluster",
         "tkgSharedserviceSize":"large",
         "tkgSharedserviceDeploymentType":"dev",
         "tkgSharedserviceWorkerMachineCount":"1",
         "tkgSharedserviceClusterCidr":"100.96.0.0/11",
         "tkgSharedserviceServiceCidr":"100.64.0.0/13",
         "tkgSharedserviceBaseOs":"photon",
         "tkgSharedserviceKubeVersion":"v1.21.8",
         "tkgSharedserviceRbacUserRoleSpec":{
            "clusterAdminUsers":"[email protected]",
            "adminUsers":"[email protected]",
            "editUsers":"",
            "viewUsers":""
         },
         "tkgSharedserviceClusterGroupName":"default",
         "tkgSharedserviceEnableDataProtection":"false",
         "tkgSharedClusterCredential":"",
         "tkgSharedClusterBackupLocation":""
      },
      "tkgMgmtDataNetworkSpec":{
         "tkgMgmtDataGatewayCidr":"11.12.18.14/24",
         "tkgMgmtDataDhcpStartRange":"11.12.18.15",
         "tkgMgmtDataDhcpEndRange":"11.12.18.30",
         "tkgMgmtDataServiceStartRange":"11.12.18.31",
         "tkgMgmtDataServiceEndRange":"11.12.18.40"
      },
      "tkgWorkloadDataNetworkSpec":{
         "tkgWorkloadDataGatewayCidr":"11.12.19.14/24",
         "tkgWorkloadDataDhcpStartRange":"11.12.19.15",
         "tkgWorkloadDataDhcpEndRange":"11.12.19.30",
         "tkgWorkloadDataServiceStartRange":"11.12.19.31",
         "tkgWorkloadDataServiceEndRange":"11.12.19.40"
      },
      "tkgWorkloadSpec":{
         "tkgWorkloadGatewayCidr":"11.12.20.14/24",
         "tkgWorkloadDhcpStartRange":"11.12.20.15",
         "tkgWorkloadDhcpEndRange":"11.12.20.30",
         "tkgWorkloadClusterName":"workload-cluster",
         "tkgWorkloadSize":"large",
         "tkgWorkloadDeploymentType":"dev",
         "tkgWorkloadWorkerMachineCount":"1",
         "tkgWorkloadClusterCidr":"100.96.0.0/11",
         "tkgWorkloadServiceCidr":"100.64.0.0/13",
         "tkgWorkloadBaseOs":"photon",
         "tkgWorkloadKubeVersion":"v1.22.5",
         "tkgWorkloadTsmIntegration":"false",
         "namespaceExclusions":{
            "exactName":"",
            "startsWith":""
         },
         "tkgWorkloadRbacUserRoleSpec":{
            "clusterAdminUsers":"[email protected]",
            "adminUsers":"[email protected]",
            "editUsers":"",
            "viewUsers":""
         },
         "tkgWorkloadClusterGroupName":"default",
         "tkgWorkloadEnableDataProtection":"false",
         "tkgWorkloadClusterCredential":"",
         "tkgWorkloadClusterBackupLocation":""
      },
      "harborSpec":{
         "enableHarborExtension":"false",
         "harborFqdn":"harbor.xx.xx",
         "harborPasswordBase64":"cGFzc3dvcmQ=",
         "harborCertPath":"",
         "harborCertKeyPath":""
      }
   },
   "tanzuExtensions":{
      "enableExtensions":"false",
      "tkgClustersName":"",
      "logging":{
         "syslogEndpoint":{
            "enableSyslogEndpoint":"false",
            "syslogEndpointAddress":"",
            "syslogEndpointPort":"",
            "syslogEndpointMode":"",
            "syslogEndpointFormat":""
         },
         "httpEndpoint":{
            "enableHttpEndpoint":"false",
            "httpEndpointAddress":"",
            "httpEndpointPort":"",
            "httpEndpointUri":"",
            "httpEndpointHeaderKeyValue":"Authorization Bearer Axxxxxxxxxxxxxx"
         },
         "kafkaEndpoint":{
            "enableKafkaEndpoint":"false",
            "kafkaBrokerServiceName":"",
            "kafkaTopicName":""
         }
      },
      "monitoring":{
         "enableLoggingExtension":"false",
         "prometheusFqdn":"",
         "prometheusCertPath":"",
         "prometheusCertKeyPath":"",
         "grafanaFqdn":"",
         "grafanaCertPath":"",
         "grafanaCertKeyPath":"",
         "grafanaPasswordBase64":""
      }
   }
}
check-circle-line exclamation-circle-line close-line
Scroll to top icon