Deploying TAS for VMs with NSX-T Networking

This topic describes how to install VMware Tanzu Application Service for VMs (TAS for VMs) on vSphere with NSX-T internal networking, using the VMware NSX-T Container Plug-in for Ops Manager.

Overview

TAS for VMs uses a Container Network Interface (CNI) plugin to support secure and direct internal communication between containers. This plugin can be:

• The internal Silk plugin that comes packaged with TAS for VMs, or

• On vSphere, NSX-T Container Plug-in (NCP), which installs as the VMware NSX-T Container Plug-in for Ops Manager tile in Ops Manager.

• On vSphere, the NSX-T Container Plug-in (NCP), which is installed as the VMware NSX-T Container Plug-in for Ops Manager tile in Ops Manager.

Prerequisites

Before deploying TAS for VMs with NSX-T networking, you must have the following:

• An NSX-T environment with NSX-T components installed and configured. The NSX-T version must support the versions of NCP and TAS for VMs you intend to use. Verify the compatibility between NSX-T, NCP and TAS for VMs with the following documentation:

  • Product Interoperability Matrix: TAS for VMs, VMware NSX-T, and VMware NSX-T Container Plug-in for Ops Manager for supported version combinations.
  • VMware NSX-T Data Center Documentation. In particular, review the NSX Container Plug-in (NCP) Release Notes and NSX-T Data Center Installation Guide for the versions of NCP and NSX-T that you want to install.

• BOSH and Ops Manager installed and configured on vSphere. For more information, see Deploying Ops Manager on vSphere and Configuring BOSH Director on vSphere.

• The VMware NSX-T Container Plug-in for Ops Manager tile is downloaded from VMware Tanzu Network and imported to the Ops Manager Installation Dashboard. For information about downloading and importing VMware Tanzu products to the Installation Dashboard, see Add and Import Products in Adding and Deleting Products.

• The TAS for VMs tile is downloaded from VMware Tanzu Network and imported to the Ops Manager Installation Dashboard. The TAS for VMs tile must be in one of these states:

  • Configured but not currently deployed. You did not click Review Pending Changes, then Apply Changes on this version of TAS for VMs.
  • Deployed previously, with the Container network interface plugin field set to External in the Networking pane of the TAS for VMs tile.

  Note: Deploying TAS for VMs with its container network interface (CNI) set to Silk configures Diego Cells to use an internally-managed container network. Subsequently switching the CNI interface to External NSX-T leads to errors.

Architecture

The following diagram shows how to deploy an NSX-T machine to run TAS for VMs across multiple vSphere hardware clusters. NSX-T runs a Tier-0 (T0) router and multiple Tier-1 (T1) routers, each connecting to a network within Ops Manager. Each vSphere hardware column cluster corresponds to an Availability Zone in Ops Manager:

When a developer pushes an app to a new org for the first time, the NSX-T plugin triggers NSX-T to create a new T1 router and allocate an address range for the org, on demand.

Install and Configure TAS for VMs and NSX-T

Installing NSX-T to run with TAS for VMs requires:

1. Configure NSX-T to Integrate with TAS for VMs

2. Enable NSX-T Mode in the BOSH Director

3. Configure TAS for VMs for External Container Networking

4. Install and Configure the NSX-T Tile

Set Up NSX-T to Integrate with TAS for VMs

To set up NSX-T to integrate with TAS for VMs, complete these procedures:

• Configure Logical Switches

• Configure Routers

• Configure Load Balancer

Configure Logical Switches

To configure logical switches:

1. In vSphere, create logical network switches to correspond to the networks that Ops Manager uses.

   1. Log in to the NSX-T Manager Dashboard.
   2. Go to Advanced Networking & Security.
   3. Go to the Switching pane.

   4. For each of these networks:

      • Infrastructure (BOSH and Ops Manager, defined in the Assign AZs and Networks pane of the BOSH Director tile)
      • Deployment (TAS for VMs, defined in the Assign AZs and Networks pane of the TAS for VMs tile)
      • Services and Dynamic Services (marketplace services and on-demand services, also defined in the TAS for VMs tile)

      • Isolation Segment (optional, defined in the Assign AZs and Networks pane of the Isolation Segment tile) do the following:

        1. Click +ADD.
        2. Enter a name for the logical switch (such as TAS for VMs-Infrastructure, TAS for VMs-Deployment).

        3. Click ADD.

           

Configure Routers

Configure NAT Rules

1. Create T0 network address translation (NAT) rules to communicate with Ops Manager:

2. Create network address translation (NAT) rules to communicate with Ops Manager:

   1. Go to Networking.
   2. Go to the NAT pane.
   3. Select your T0 gateway.
   4. Choose ADD NAT RULE.
   5. Add a rule for destination NAT (DNAT) with:
      • The externally-specified destination IP address of incoming requests. If your Ops Manager has a DNS entry (for example, opsmgr.example.com), this is its IP address.
      • The Ops Manager internal network address.
      • Firewall Setting bypass - the packet bypasses firewall rules.
   6. Add the corresponding source NAT (SNAT) rule with:
      • The externally-specified destination IP address.
      • The Ops Manager internal network address.
      • Firewall Setting bypass - the packet bypasses firewall rules.
   7. Add a rule for source NAT (SNAT) for the infrastructure and deployment networks:
      • The externally-specified destination IP address.
      • The internal network address in CIDR notation.
      • Firewall Setting bypass - the packet bypasses firewall rules.

3. Create T1 routers for TAS for VMs, to connect from the T0 router. For each Ops Manager network, Infrastructure, Deployment, and so on, create a T1 router as follows:

4. In the NSX-T Manager UI, navigate to Advanced Networking & Security > Routing > Routers.

   1. Click +ADD > Tier-1 Router.

   2. Configure the router. Include the Edge Cluster and Edge Cluster Members; they are required to enable the Load Balancer. The Infrastructure network router configuration might look like the following diagram:

      

5. Create T1 router downlink ports for TAS for VMs. For each T1 router you created, add a New Router Port as follows, to allow traffic in and out:

   1. In the NSX-T Manager UI, select the T1 router.
   2. In Configuration > Router Ports, click +ADD to add a new router port.
   3. For Logical Switch, enter the name of the logical switch you defined for the network in Add New Logical Switch, above.

   4. For IP Address, use the first IP of the appropriate subnet. In this example, 192.168.1.0/24 is set aside for Infrastructure (Ops Manager and BOSH Director), and 192.168.2.0/24 for the Deployment, so 192.168.1.1 and 192.168.2.1 are used respectively.

      

6. Advertise the routes of the T1 routers to the T0 router, so the T0 router can correctly route incoming requests based on their destination IP address:

7. Create T1 gateways for TAS for VMs, to connect from the T0 gateway. For each Ops Manager network, Infrastructure, Deployment, and so on, create a T1 gateway as follows:

8. In the NSX-T Manager UI, navigate to Networking, then Tier-1 Gateways.

   1. Click ADD TIER-1 GATEWAY.
   2. Configure the gateway. Include the Edge Cluster as it is required to enable the Load Balancer. T

9. Advertise the routes of the T1 gateways to the T0 gateway, so the T0 gateway can correctly route incoming requests based on their destination IP address:

   1. Edit your T1 Gateway and navigate to Route Advertisement.
   2. Enable All Connected Segments & Service Ports.
   3. Enable All LB VIP Routes. (Required if Load Balancing service is configured.)

10. Allocate an IP block for TAS for VMs orgs.

    1. From the NSX-T Manager, navigate to Networking, then IP Address Pools, click "IP ADDRESS BLOCKS tab, and click ADD IP ADDRESS BLOCK.
    2. Enter a name (for example, TAS for VMs-container-ip-block). This IP block name is also used in the VMware NSX-T tile in the NCP section under IP Blocks of Container Networks.
    3. Enter a description, such as Subnets are allocated from this pool to each newly-created org.
    4. Enter a CIDR to allocate an address block large enough to accommodate all TAS for VMs apps. A /14 CIDR is large enough for ~1,000 Orgs with ~250 apps each. If you are planning such a large foundation, see VMware NSX-T TAS for VMs limits in the VMware documentation.

11. Create an external SNAT IP pool.

    1. Navigate to Networking, then IP Address Pools, click "IP ADDRESS POOLS tab, and click ADD IP ADDRESS POOL.
    2. Enter a name (such as external-ip-pool) and a description (such as IP pool that provides 1 public IP for each TAS for VMs Org). Later, you will enter this pool name on the VMware NSX-T tile in the NCP section under IP Pools used to provide External (NAT) IP Addresses to Org Networks.
    3. Set a subnet of externally routable ip addresses for future NAT ip addresses

Configure Segments

To configure segments:

1. In vSphere, create segments that correspond to the networks that Ops Manager uses.
   1. Log in to the NSX-T Manager Dashboard.
   2. Go to Networking.
   3. Go to the Segments pane.
   4. For each of these networks…
      • Infrastructure (BOSH and Ops Manager, defined in the Assign AZs and Networks pane of the BOSH Director tile)
      • Deployment (TAS for VMs, defined in the Assign AZs and Networks pane of the TAS for VMs tile)
      • Services and Dynamic Services (marketplace services and on-demand services, also defined in the TAS for VMs tile)
      • Isolation Segment (optional, defined in the Assign AZs and Networks pane of the Isolation Segment tile) …do the following:
        1. Click ADD SEGMENT.
        2. Enter a name for the segment
        3. Enter a Gateway to connect to.
        4. Click SAVE.

Configure Load Balancer

To configure a load balancer:

1. Create Active Monitors (health checks) for use by the virtual servers later on. In the NSX-T Manager UI, navigate to Networking, then Load Balancing, and click the Monitors tab.
   1. Create the health monitor for web load balancing:
   2. Click Add Active Monitor.
   3. Select HTTP.
      • Name: tas-web-monitor
      • Monitoring Port: 8080
      • Monitoring Port: 8080
   4. Configure Additional Properties:
      • HTTP Request URL: /health
      • HTTP Response Code: 200
   5. Click Save.
2. Create the health monitor for TCP load balancing:
   1. Click ADD ACTIVE MONITOR.
   2. Select HTTP.
      • Name: tas-tcp-monitor
      • Monitoring Port: 80
   3. Configure Additional Properties:
      • HTTP Request URL: /health
      • HTTP Response Code: 200
   4. Click Save.

3. Create the health monitor for SSH load balancing:

   1. Click ADD ACTIVE MONITOR.
   2. Select TCP:
      • Name: tas-ssh-monitor
      • Monitoring Port: 2222
   3. Click Save.

4. Create Server Pools (collections of VMs which handle traffic) for use by the virtual servers. In the NSX-T Manager UI, navigate to Networking, then Load Balancing, and click the Server Pools tab.

   1. Create the server pool for web load balancing:
   2. Click Add Server Pool to add a new pool.
      • Name: tas-web-pool
   3. Enter SNAT Translation: Automap
   4. Click Select Members:
      • Membership Type: Static
   5. Click Active Monitor Set:
      • Select tas-web-monitor
      • Click Apply
   6. Click Save.

5. Create the server pool for TCP load balancing:

   1. Click Add Server Pool to add a new pool.
      • Name: tas-tcp-pool
   2. Enter SNAT Translation: Disabled
   3. Click Select Members:
      • Membership Type: Static
   4. Click Active Monitor Set:
      • Select tas-tcp-monitor
      • Click Apply
   5. Click Save.

6. Create the server pool for SSH load balancing:

   1. Click ADD SERVER POOL to add a new pool.
      • Name: tas-ssh-pool
   2. Enter SNAT Translation: Disabled
   3. Click Select Members:
      • Membership Type: Static
   4. Click Active Monitor Set:
      • Select tas-ssh-monitor
      • Click Apply
   5. Click Save.

Create virtual servers:

1. In the NSX-T Manager UI, navigate to Advanced Networking & Security > Networking > Load Balancing > Virtual Servers.

2. Create the virtual server which forwards unencrypted web (HTTP) traffic to the foundation:

   Note: Foundations requiring end-to-end encryption should not enable the virtual server on port 80, or, if enabled, should configure it to redirect traffic to the encrypted port (443).

   1. Click +ADD.
   2. Enter General Properties:
      • Name: pas-web-vs
      • Application Type: Layer 4 (TCP)
      • Application Profile: nsx-default-lb-fast-tcp-profile
   3. Click Next.
   4. Enter Virtual Server Identifiers:
      • IP Address: use the address of the DNS record of *.system.YOUR-SYSTEM-DOMAIN.com
      • Port: 80,443
   5. Enter Server Pool and Rules:
      • Default Server Pool: pas-web-pool
   6. Click Next several times, then click Finish.
3. Create the virtual server which forwards traffic to apps with custom ports to the foundation:
   1. Click +ADD to add a new virtual server.
   2. Enter General Properties:
      • Name: pas-tcp-vs
      • Application Type: Layer 4 (TCP)
      • Application Profile: nsx-default-lb-fast-tcp-profile
   3. Click Next.
   4. Enter Virtual Server Identifiers:
      • IP Address: use the address of the DNS record of tcp.apps.YOUR-SYSTEM-DOMAIN.com
      • Port: use the same ports as configured in the TAS for VMs Tile > Networking > TCP Routing Ports, e.g. 1024-1123,5900
   5. Click Next.
   6. Enter Server Pool and Rules:
      • Default Server Pool: pas-tcp-pool
   7. Click Next, then click Finish.

4. Create the virtual server which forwards SSH traffic to the foundation:

   1. Click +ADD to add a new virtual server.
   2. Enter General Properties:
      • Name: pas-ssh-vs
      • Application Type: Layer 4 (TCP)
      • Application Profile: nsx-default-lb-fast-tcp-profile
   3. Click Next.
   4. Enter Virtual Server Identifiers:
      • IP Address: use the address of the DNS record of ssh.system.YOUR-SYSTEM-DOMAIN.com
      • Port: 2222
   5. Click Next.
   6. Enter Server Pool and Rules:
      • Default Server Pool: pas-ssh-pool
   7. Click Next, then click Finish.

5. Create the load balancer. In the NSX-T Manager UI, navigate to Networking, then Load Balancing, and click the Load Balancers tab.

   1. Click ADD LOAD BALANCER.
      1. Enter the fields:
         • Name: tas-lb
         • Load Balancer Size: Select Small unless you have a larger Foundation.
         • Attachment: t1-deployment Attach your load balancer to the Tier 1 gateway fronting your deployment instances.
      2. Click Save
      3. Click Yes when prompted Want to continue configuring this Load Balancer?
      4. Click Virtual Servers Set

6. Click Add Active Monitor.

7. Select HTTP:
   • Name: tas-web-monitor
   • Monitoring Port: 8080.
8. Configure Additional Properties:
   • HTTP Request URL: /health
   • HTTP Response Code: 200.
9. Click Save.

To create the health monitor for TCP load balancing:

1. Click Add Active Monitor.
2. Select HTTP:
   • Name: tas-tcp-monitor
   • Monitoring Port: 80.
3. Configure Additional Properties:
   • HTTP Request URL: /health
   • HTTP Response Code: 200
4. Click Save.

To create the health monitor for SSH load balancing:

1. Click Add Active Monitor.
2. Select TCP:
   • Name: tas-ssh-monitor
   • Monitoring Port: 2222.
3. Click Save.

To create Server Pools (collections of VMs which handle traffic) for use by the virtual servers:

1. In the NSX-T Manager UI, go to Networking, then Load Balancing.
2. Click the Server Pools tab.

To create the server pool for web load balancing:

1. Click Add Server Pool : Name: tas-web-pool
2. Enter SNAT Translation: Automap
3. Click Select Members: Membership Type: Static
4. Click Active Monitor Set: tas-web-monitor
5. Click Apply
6. Click Save.

To create the server pool for TCP load balancing:

1. Click Add Server Pool to add a new pool.

   • Name: tas-tcp-pool
     1. Enter SNAT Translation: Disabled
     2. Click Select Members: Membership Type: Static
     3. Click Active Monitor Set: tas-tcp-monitor
     4. Click Apply
     5. Click Save.

2. Create the server pool for SSH load balancing:

   1. Click Add Server Pool Name: tas-ssh-pool
   2. Enter SNAT Translation: Disabled
   3. Click Select Members: Membership Type: Static
   4. Click Active Monitor Set: tas-ssh-monitor
   5. Click Apply.
   6. Click Save.

To create the load balancer:

1. In the NSX-T Manager UI, go to Networking, then Load Balancing.
2. Click the Load Balancers tab.
3. Click Add Load Balancer.
   1. Enter the fields:
      • Name: tas-lb
      • Load Balancer Size: Select Small unless you have a larger Foundation.
      • Attachment: t1-deployment Attach your load balancer to the Tier 1 gateway fronting your deployment instances.
   2. Click Save.
   3. Click Yes, when prompted with, Want to continue configuring this Load Balancer?.

To create the virtual server that forwards unencrypted web (HTTP) traffic to the foundation.

Note: For foundations requiring end-to-end encryption, do not enable the virtual server on port 80. If it must be enabled, configure it to redirect traffic to the encrypted port (443).

1. Click Virtual Servers Set.
2. Click Add Virtual Server.
3. Select L4 TCP.
   • Name: tas-web-vs
   • Application Profile: default-tcp-lb-app-profile
   • IP Address: use the address of the DNS record of *.system.YOUR-SYSTEM-DOMAIN.com
   • Port: 80,443
   • Server Pool: tas-web-pool
4. Click Save.

To create the virtual server that forwards traffic to apps with custom tcp ports to the foundation:

1. Click Add Virtual Server.
2. Select L4 TCP.
   • Name: tas-tcp-vs
   • Application Profile: default-tcp-lb-app-profile
   • IP Address: Use the address of the DNS record of tcp.apps.YOUR-SYSTEM-DOMAIN.com
   • Port: Use the same ports as configured in the TAS for VMs Tile, then Networking, and TCP Routing Ports. For example: 1024-1123,5900
   • Server Pool: tas-tcp-pool
3. Click Save.

To create the virtual server that forwards SSH traffic to the foundation:

1. Click Add Virtual Server.
2. Select L4 TCP.
   • Name: tas-ssh-vs
   • Application Profile: default-tcp-lb-app-profile
   • IP Address: Use the address of the DNS record of ssh.system.YOUR-SYSTEM-DOMAIN.com.
   • Port: 2222
   • Server Pool: tas-ssh-pool
3. Click Save.

Enable NSX-T Mode in the BOSH Director

To enable NSX-T mode in the BOSH Director:

1. From the Ops Manager Installation Dashboard, open the BOSH Director tile.

2. In the vCenter Configs pane, click the pencil icon for the vCenter Config you want to edit.

3. Select NSX Networking below.

4. Configure BOSH Director authentication to the NSX Manager by following the NSX Networking instructions in the Step 2: Configure vCenter section of Configuring BOSH Director on vSphere.

5. Verify that the Use NSX-T Policy API option is selected.

Configure TAS for VMs for External Container Networking

To configure TAS for VMs for external container networking:

1. If you have not already done so, download the TAS for VMs tile from VMware Tanzu Network and import it to the Installation Dashboard.
   
   For instructions, see Add and Import Products.

2. Configure TAS for VMs, following the directions in Configuring TAS for VMs. When you configure Networking, select External under Container networking interface plugin.

   

3. Configure TAS for VMs, following the directions in Configuring TAS for VMs. When you configure Networking, select External under Container networking interface plugin.
   

4. Configure TAS for VMs to add router, diego_brain, and tcp_router instances to the corresponding NSX-T server pools upon deployment.

   1. Open the TAS for VMs tile > Resource Config pane.
   2. Click the arrow next to each job to reveal the NSX-T CONFIGURATION column.
   3. Under Logical Load Balancer, fill in the JSON server_pools list with the NSX-T Server Pool these instance should be added to upon deployment.
      • router -> tas-web-pool
      • diego_brain -> tas-ssh-pool
      • tcp_router -> tas-tcp-pool
   4. Click Save

Install and Configure the NSX-T Container Plug-In

To install and configure the tile:

1. If you have not already done so, download the VMware NSX-T Container Plug-in for Ops Manager tile from VMware Tanzu Network and import it to the Installation Dashboard. For instructions, see Add and Import Products.

   1. If you are using VMware Workspace ONE Access, formerly called VMware Identity Manager
   (vIDM), then select **Client Certificate Authentication**.
   1. Otherwise, select **Basic Authentication with Username and Password** and enter **NSX Manager Admin Username** and **Admin Password** credentials in the fields underneath.
   

   • NSX Manager CA Cert: Obtain this certificate from NSX-T Manager as follows:

     1. ssh into NSX-T Manager using the admin account that you created when you deployed NSX-T Manager.

     2. From the NSX-T Manager command line, run get certificate api to retrieve the certificate.

        

2. Open and configure the NCP (NSX-T Container Plugin) pane as follows:

   • TAS for VMs Foundation Name: If unsure, use TAS for VMs. If multiple foundations co-exist on the same NSX-T Manager, choose a unique string, such as TAS for VMs-beta. NCP creates artifacts, such as T1 gateways and prefixes their names with this string for easy identification.
   • Overlay Transport Zone: A uniquely identifying string for the Transport Zone that you chose when you created segments for each network. This can be the name of the transport zone if no other zones in NSX-T share the same name, or else the UUID for the transport zone.
   • Tier-0 Router: A uniquely identifying string for the T0 gateway. This can be the tag string that you gave the gateway in NSX-T Manager if no other T0 gateways in NSX-T share the same name, or else the UUID for the gateway.
   • IP Blocks of Container Networks: Use the same IP block created Configure Gateways.
   • Subnet Prefix of Container Networks: Subnet mask to set the address range size for apps in a single org. Defaults to 24. This number must be higher than the mask for all TAS for VMs orgs in the NSX-T Manager New IP Block pane, to define each org’s fraction of the total TAS for VMs address space.
   • IP Pools used to provide External (NAT) IP Addresses to Org Networks: Use the same IP Pool created iin Configure Gateways.
   • Enable NSX-T Policy API: Enable this checkbox to use the new Policy API.

3. In the NSX Node Agent pane, enable the Enable Debug Level of Logging for NSX Node Agent checkbox.

4. Click Save and return to the Installation Dashboard.

5. After you have configured both the TAS for VMs tile and the VMware NSX-T tile, click Review Pending Changes, then Apply Changes to deploy TAS for VMs with NSX-T networking.

Upgrade TAS for VMs with NSX-T Networking

After you have deployed TAS for VMs with NSX-T, you may need to upgrade either Ops Manager, TAS for VMs, the NSX-T Container Plug-in or NSX-T Data Center. If you upgrade one of these components, you may need to upgrade the other components as well.

For example, if you want to upgrade NSX-T Data Center, you may need to upgrade the NSX-T Container Plug-in first.

To upgrade TAS for VMs with NSX-T Networking:

1. Plan the upgrade by determining the compatibility of NCP, NSX-T and TAS for VMs by checking the following documentation:

   • See Product Interoperability Matrix: TAS for VMs, VMware NSX-T, and VMware NSX-T Container Plug-in for Ops Manager for supported version combinations.
   • See VMware NSX-T Data Center Documentation. In particular, review the NSX Container Plug-in (NCP) Release Notes and NSX-T Data Center Installation Guide for the versions of NCP and NSX-T that you want to install.

2. Download the desired version of VMware NSX-T Container Plug-in for Ops Manager tile from VMware Tanzu Network.

3. In Ops Manager, import the new version of the tile to the Installation Dashboard. For instructions, see Adding and Importing Products.

4. Click Review Pending Changes and review your changes.

5. Click Apply Changes.

6. Continue with the upgrade of Ops Manager, TAS for VMs, or NSX-T Data Center. For more information, see Upgrade NCP in a Ops Manager Environment in the VMware NSX-T Data Center documentation.