This topic describes how to configure BOSH Director for vSphere with NSX-T integration for VMware Tanzu Kubernetes Grid Integrated Edition (TKGI).

Prerequisites

Before you begin this procedure, ensure that you have successfully completed all preceding steps for installing Tanzu Kubernetes Grid Integrated Edition on vSphere with NSX-T, including:

How Ops Manager Accesses NSX-T Manager

To create, delete, and modify NSX-T networking resources, Ops Manager tiles, and APIs use a VMware NSX Manager account with the Enterprise Administrator role and permissions.

Users configure Ops Manager to authenticate to NSX Manager for different purposes in different tiles:

Step 1: Open the BOSH Director Tile

To configure BOSH Director:

  1. Log in to Ops Manager with the user name and password credentials that you set up in Configure Ops Manager for Tanzu Kubernetes Grid Integrated Edition.

  2. Click the BOSH Director for vSphere tile.

    Ops Manager UI before the TKGI tile has been imported

Step 2: Configure vCenter for Tanzu Kubernetes Grid Integrated Edition

To configure BOSH Director with your vCenter settings:

  1. Select vCenter Config in BOSH Director.

    TKGI tile vCenter Config tab default configuration

  2. Enter the following information:

    • Name: A name that you provide for your vCenter configuration. This field is used to identify the data center configuration in Ops Manager if you are configuring multiple data centers.
    • vCenter Host: The hostname of the vCenter that manages ESXi/vSphere.

      Note: The FQDN for the vCenter Server cannot contain uppercase letters.

    • vCenter Username: A vCenter user name with create and delete privileges for virtual machines (VMs) and folders.
    • vCenter Password: The password for the vCenter user specified above.
    • Datacenter Name: The name of the data center as it appears in vCenter.
    • Virtual Disk Type: The Virtual Disk Type to provision for all VMs. For guidance on selecting a virtual disk type, see vSphere Virtual Disk Types.
    • Ephemeral Datastore Names (comma delimited): The names of the datastores that store ephemeral VM disks deployed by Ops Manager.
    • Persistent Datastore Names (comma delimited): The names of the datastores that store persistent VM disks deployed by Ops Manager.

    Note: The vSphere datastore type must be Datastore. Tanzu Kubernetes Grid Integrated Edition does not support the use of vSphere Datastore Clusters with or without Storage DRS. For more information, see Datastores and Datastore Clusters in the vSphere documentation.

  3. For Networking, select NSX Networking.

    TKGI tile vCenter Config tab with NSX networking selected

  4. Configure NSX networking as follows:

    • NSX Mode: Select NSX-T to use NSX-T networking for dynamically created node networks.
    • NSX Address: Enter the FQDN or IP address of the NSX-T Manager, or the VIP address of the NSX-T Management Cluster.
    • NSX Authentication: Select the authentication mode, either Local User Authentication or Certificate Authentication.
    • NSX Username and NSX Password: If you selected Local User Authentication, enter the NSX-T Manager user name and password.
    • NSX Manager Principal Identity Certificate and NSX Manager Principal Identity Private Key: If you selected Certificate Authentication, enter the NSX Manager Principal Identity Certificate and Private Key. For more information, see Generating and Registering the NSX Manager Superuser Principal Identity Certificate and Key.
    • Use NSX-T Policy API: Select this option to use the NSX-T Policy API instead of the NSX-T Management API. For more information, see Considerations for Using the NSX-T Policy API with TKGI.
    • NSX CA Cert: Provide the CA certificate in PEM format that authenticates to the NSX server. Copy the contents of the NSX CA certificate that you generated in Generate and Register the NSX-T Management SSL Certificate and Private Key to this field.

  5. Configure the following folder names:

    • VM Folder: The vSphere data center folder where Ops Manager places VMs.
    • Template Folder: The vSphere data center folder where Ops Manager places VMs.
    • Disk path Folder: The vSphere datastore folder where Ops Manager creates attached disk images. You must not nest this folder.

    Note: After your initial deployment, you cannot edit the VM Folder, Template Folder, and Disk path Folder names.

    TKGI tile vCenter Config tab NSX networking NSX CA Cert and folder config

  6. Click Save.

    TKGI tile vCenter Config tab - Settings updated notification

Step 3: Configure BOSH Director

To configure BOSH Director settings:

  1. Select Director Config in BOSH Director.

    TKGI tile Director Config tab default configuration

  2. In the NTP Servers (comma delimited) field, enter your NTP server addresses.

    Note: The NTP server configuration only updates after VM recreation. Ensure that you select the Recreate VMs deployed by the BOSH Director check box if you modify the value of this field.

  3. Leave the JMX Provider IP Address field blank.

  4. Leave the Bosh HM Forwarder IP Address field blank.

  5. Select the Enable VM Resurrector Plugin to enable BOSH Resurrector functionality.

  6. Select Enable Post Deploy Scripts to run a post-deploy script after deployment. This script allows the job to execute additional commands against a deployment.

    Note: You must enable post-deploy scripts to install Tanzu Kubernetes Grid Integrated Edition.

  7. Select Recreate VMs deployed by the BOSH Director to force BOSH to recreate all BOSH-managed VMs on the next deploy. This process does not destroy any persistent disk data.

  8. For typical Tanzu Kubernetes Grid Integrated Edition deployments, the default settings for all other BOSH Director configuration parameters are suitable. Optionally you can apply additional configurations to BOSH Director. See Director Config Pane in Configuring BOSH Director on vSphere in the Ops Manager documentation for details.

    Note: If you need to be able to remotely access the BOSH Director VM using the BOSH CLI, and you are deploying Tanzu Kubernetes Grid Integrated Edition with NSX-T in a NAT topology, you must provide the Director Hostname for BOSH at the time of installation. See Director Config Pane in Configuring BOSH Director on vSphere in the Ops Manager documentation for details.

  9. Click Save.

    TKGI tile Director Config tab - Settings updated notification

Step 4: Create Availability Zones

On vSphere with NSX-T, operators define and create Availability Zones (AZs) using vCenter clusters and resource pools. Plans defined in the TKGI tile then use these AZs to enable high availability for TKGI clusters.

The Tanzu Kubernetes Grid Integrated Edition control plane also runs in one of the AZs.

For more information on AZs in TKGI, see Availability Zones in Tanzu Kubernetes Grid Integrated Edition Architecture.

To configure Availability Zones:

  1. Select Create Availability Zones in BOSH Director.

    TKGI tile Create Availability Zones tab default configuration

  2. Use the following steps to create one or more Availability Zones for Tanzu Kubernetes Grid Integrated Edition to use:

    1. Click Add and create the Tanzu Kubernetes Grid Integrated Edition Management AZ.
    2. Enter a unique Name for the Availability Zone, such as AZ-MGMT.
    3. Select the IaaS configuration (vSphere/vCenter).
    4. Enter the name of an existing vCenter Cluster to use as an Availability Zone, such as COMP-Cluster-1.
    5. Enter the name of the Tanzu Kubernetes Grid Integrated Edition Management Resource Pool in the vCenter cluster that you specified above, such as RP-MGMT-TKGI. The jobs running in this Availability Zone share the CPU and memory resources defined by the pool.
    6. Click Add Cluster and create at least one Tanzu Kubernetes Grid Integrated Edition Compute AZ.
    7. Specify the Cluster and the Resource Pool, such as RP-TKGI-AZ. Alternatively, specify the Cluster and the Host Group. See Using vSphere Host Group for more information.
    8. (Optional) If you are using a host group with vSAN Stretched Clusters, set the VM-Host Affinity Rule dropdown to SHOULD. This setting maintains high availability by letting TKGI restart VMs in another host group if their AZ fails. TKGI ignores this setting if the vSAN cluster has no host group configured.
    9. Add additional clusters as necessary. Click the trash icon to delete a cluster. The first cluster cannot be deleted.

    TKGI tile Create Availability Zones AZ-MGMT configuration

    TKGI tile Create Availability Zones AZ-COMP-1 configuration

    TKGI tile Create Availability Zones AZ-COMP-2 configuration

    TKGI tile Create Availability Zones PKS-AZ1-HostGroup1 configuration

  3. Click Save.

    TKGI tile Create Availability Zones - Settings updated notification

Step 5: Create Networks

You must configure and create BOSH Director networking.

Note: If you are using No-NAT with Virtual Switch (VSS/VDS) Topology, create the infrastructure network and optionally the TKGI network by following the instructions in Create Networks Pane in Configuring BOSH Director on vSphere in the Ops Manager documentation. While completing the steps in Create Networks Pane, do not create the services network. With TKGI on NSX-T, NSX-T manages the dynamically created networks.

To configure BOSH Director networking:

  1. Select Create Networks in BOSH Director.

    TKGI tile Create Networks tab default configuration

  2. Select Enable ICMP checks to enable ICMP on your networks. Ops Manager uses ICMP checks to confirm that components within your network are reachable.

  3. Click Add Network.

    TKGI tile Create Networks tab NET-MGMT-PKS network configuration

  4. Create the following network:

    • NET-MGMT-TKGI: Network for Ops Manager, BOSH Director, and Tanzu Kubernetes Grid Integrated Edition components. This network maps to the NSX logical switch created for the Tanzu Kubernetes Grid Integrated Edition Management Network. See Create Management Plane in Installing and Configuring NSX-T Data Center v3.0 for TKGI.

      Note: NSX-T automatically creates the service network to be used by the control plane and worker nodes (VMs) for Kubernetes clusters managed by Tanzu Kubernetes Grid Integrated Edition. Do not manually create this network.

    Use the following values as a guide when you define the network in BOSH. Replace the IP addresses with ranges you defined for the Create Management Plane in Installing and Configuring NSX-T Data Center v3.0 for TKGI. Reserve any IP addresses from the subnet that are already in use, such as the IP for Ops Manager and subnet gateway.

    Infrastructure
    Network    		 Field Configuration
    Name NET-MGMT-TKGI
    vSphere Network Name LS-MGMT-TKGI
    CIDR 10.0.0.0/24
    Reserved IP Ranges 10.0.0.1-10.0.0.2
    DNS 10.20.20.1
    Gateway 10.0.0.1

  5. Select the AZ-MGMT Availability Zone to use with the NET-MGMT-TKGI network.

    Note: Do not select the COMPUTE network at this point in the configuration. It will be configured at the end of the procedure.

  6. Click Save.

    TKGI tile Create Networks - Settings updated notification

Step 6: Assign AZs and Networks

To configure the AZs and the Network for BOSH Director:

  1. Select Assign AZs and Networks in BOSH Director.

    TKGI tile Assign AZs and Networks tab default configuration

  2. Use the drop-down menu to select a Singleton Availability Zone. The Ops Manager Director installs in this Availability Zone. For Tanzu Kubernetes Grid Integrated Edition, this will be the AZ-MGMT availability zone.

  3. Use the drop-down menu to select a Network for BOSH Director. BOSH Director runs on the Tanzu Kubernetes Grid Integrated Edition Management Plane network. Select the NST-MGTM-TKGI network.

  4. Click Save.

    TKGI tile Assign AZs and Networks - Settings updated notification

Step 7: Configure Security

To configure a BOSH Director certificate and password:

  1. Select Security in BOSH Director.

  2. In Trusted Certificates, enter a custom certificate authority (CA) certificate to insert into your organization’s certificate trust chain. This feature allows all BOSH-deployed components in your deployment to trust a custom root certificate.

    If you are using self-signed CAs for the infrastructure components (NSX-T, vCenter), you need to add every CA of every component your deployment might connect to. In other words, the bundle must include all certificates for any component that connects to or from BOSH.

    If you are using a private Docker registry, such as VMware Harbor, use this field to enter the certificate for the registry. See Integrating Harbor Registry with Tanzu Kubernetes Grid Integrated Edition for details.

  3. Choose Generate passwords or Use default BOSH password. Use the Generate passwords option for increased security.

  4. Click Save. To view your saved Director password, click the Credentials tab.

Step 8: Configure BOSH DNS

To configure BOSH Director DNS:

  1. Select BOSH DNS Config in BOSH Director.

  2. (Optional) In Excluded Recursors, enter a list of prohibited DNS recursor addresses.

  3. (Optional) In Recursor Timeout, enter a time limit for contacting the connected DNS recursors. This includes dialing, writing, and reading from the DNS recursor. If any of these actions exceeds the time limit you set, the action fails.

    Note: This time limit must include one of the Go parse duration time units. For example, entering 5s sets the timeout limit to five seconds. For more information about supported time units, see func ParseDuration in the Go Programming Language documentation.

  4. (Optional) In Handlers, enter a list of custom domain handlers in JSON format.
  5. Click Save.

Step 9: Configure Logging

To configure BOSH Director logging:

  1. Select Syslog in BOSH Director.

  2. (Optional) To send BOSH Director system logs to a remote server, select Yes.

  3. In the Address field, enter the IP address or DNS name for the remote server.

  4. In the Port field, enter the port number that the remote server listens on.

  5. In the Transport Protocol dropdown menu, select TCP or UDP. This selection determines which transport protocol is used to send the logs to the remote server.

  6. (Optional) Select the Enable TLS check box to send encrypted logs to remote server with TLS. After you select the check box, perform the following steps:

    1. Enter either the name or SHA1 fingerprint of the remote peer in Permitted Peer.
    2. Enter the SSL certificate for the remote server in SSL Certificate.

      Note: For an optimal security configuration, enable TLS encryption when you are forwarding logs. Logs can contain sensitive information, such as cloud provider credentials.

  7. (Optional) Enter an integer in Queue Size. This value specifies the number of log messages held in the buffer. The default value is 100,000.
  8. (Optional) Select the check box to Forward Debug Logs to an external source. This option is deselected by default. If you select it, you might generate a large amount of log data.
  9. (Optional) Enter configuration details for rsyslog in the Custom rsyslog Configuration field. This field requires the RainerScript syntax.
  10. Click Save Syslog Settings.

Step 10: Configure Resources

To configure BOSH Director resources:

  1. Select Resource Config in BOSH Director.

  2. Adjust any values as necessary for your deployment. Under the Instances, Persistent Disk Type, and VM Type fields, choose Automatic from the drop-down menu to allocate the recommended resources for the job. If the Persistent Disk Type field reads None, the job does not require persistent disk space.

    Note: Ops Manager requires a Director VM with at least 8 GB memory.

    Note: If you set a field to Automatic and the recommended resource allocation changes in a future version, Ops Manager automatically uses the updated recommended allocation.

  3. Click Save.

Step 11: (Optional) Add Custom VM Extensions

Use the Ops Manager API to add custom properties to your VMs such as associated security groups and load balancers.

For more information, see Managing Custom VM Extensions.

Step 12: Deploy BOSH

To deploy BOSH:

  1. Go to the Ops Manager Installation Dashboard.

    Ops Manger UI showing the Review Pending Changes button

  2. Click Review Pending Changes.

    Ops Manager UI Review Pending Changes view

  3. Click Apply Changes.

    Ops Manager UI Applying Changes progress view

  4. Confirm changes applied successfully.

    Ops Manager UI Apply Changes - Changes Applied notification

  5. Check BOSH VM. Log in to vCenter and check for the p-bosh VM deployment in the Tanzu Kubernetes Grid Integrated Edition Management resource pool.

    vCenter UI p-bosh VM deployment configuration

Step 13: Update Network Availability Zones

After successfully deploying BOSH, ensure that both the Management AZ and the Compute AZs appear in the Tanzu Kubernetes Grid Integrated Edition tile Plans.

To ensure that the Management AZ and the Compute AZs are included in the NET-MGMT-TKGI network you defined above:

  1. Return to the BOSH tile and click Create Networks.

    TKGI tile Create Networks configuration

  2. Edit the network (NET-MGMT-TKGI) and each COMPUTE AZ.

    TKGI tile Create Networks NET-MGMT-PKS network configuration

  3. Click Save.

    TKGI tile Create Networks - Settings updated notification

  4. Review pending changes, and click Apply Changes to redeploy BOSH.

Next Step

Generate and Register the NSX Manager Superuser Principal Identity Certificate and Key for Tanzu Kubernetes Grid Integrated Edition.

check-circle-line exclamation-circle-line close-line
Scroll to top icon