Enabling and configuring TCP routing in TAS for VMs

You can enable TCP routing in TAS for VMs. This article describes TCP routing and walks you through the steps of enabling and configuring it.

TCP routing overview

TCP routing enables developers to run apps that serve requests on non-HTTP TCP protocols. You can use TCP routing to comply with regulatory requirements to terminate TLS as close to apps as possible so that packets are not decrypted before reaching the app level.

TCP routing architecture

This diagram shows the layers of network address translation that occur in VMware Tanzu Application Service for VMs (TAS for VMs) in support of TCP routing.

Route Ports

The following is an example workflow that includes route ports, back end ports, and app ports:

A developer creates a TCP route for their app based on a TCP domain and a route port, and maps this route to one or more apps. For more information, see Create a Route in Configuring Routes and Domains.
Clients make requests to the route. DNS resolves the domain name to the load balancer.
The load balancer listens on the port and forwards requests for the domain to the TCP routers. The load balancer must listen on a range of ports to support multiple TCP route creation. Additionally, TAS for VMs must be configured with this range, so that the platform knows which ports can be reserved when developers create TCP routes.
The TCP router can be dynamically configured to listen on the port when the route is mapped to an app. The domain the request was originally sent to is no longer relevant to the routing of the request to the app. The TCP router keeps a dynamically updated record of the back ends for each route port. The back ends represent instances of an app mapped to the route. The TCP router chooses a back end using a round-robin load balancing algorithm for each new TCP connection from a client. Because the TCP router is protocol-agnostic, it does not recognize individual requests, only TCP connections. All client requests transit the same connection to the selected back end until the client or back end closes the connection. Each subsequent connection triggers the selection of a back end.
Because containers each have their own private network, the TCP router does not have direct access to app containers. When a container is created for an app instance, a port on the Diego Cell VM is randomly chosen and iptables are configured to forward requests for this port to the internal interface on the app container. The TCP router then receives a mapping of the route port to the Diego Cell IP and port.
By default, the Diego Cell only routes requests to port 8080, the app port, on the app container internal interface. The app port is the port on which apps must listen. Developers can use the Cloud Controller API to update the ports an app can receive requests on. For more information, see Configuring Apps to Listen on Custom Ports (Beta).

Prerequisites for Enabling TCP Routing

Note: If you have mutual TLS app identity verification enabled, app containers accept incoming communication only from the Gorouter. This deactivates TCP routing. For more information, see TLS to Apps and Other Back End Services in HTTP Routing.

Before enabling TCP routing, you must set up networking requirements. To set up networking requirements:

Choose a domain from which your developers are to create TCP routes for their apps. For example, create a domain which is similar to your app domain but prefixed by the TCP subdomain, such as tcp.APP-DOMAIN.com, where APP-DOMAIN is the name of your app domain.
Configure DNS to resolve this domain name to the IP address of a highly-available load balancer that can forward traffic for the domain to the TCP routers. For more information, see Domains in Configuring Routes and Domains. If you are operating an environment that does not require high availability, configure DNS to resolve the TCP domain name you have chosen directly to a single instance of the TCP router.
(Optional) Choose IP addresses for the TCP routers and configure your load balancer to forward requests for the domain you chose in the first step above to these addresses. Skip this step if you have configured DNS to resolve the TCP domain name to an instance of the TCP router. To configure your IP addresses for your TAS for VMs deployment, follow the procedure for enabling TCP routing in Configure Networking in Configuring TAS for VMs.
(Optional) Decide how many TCP routes you want to support. For each TCP route, you must reserve a port. Configure your load balancer to forward the range of ports to the TCP routers. Skip this step if you have configured DNS to resolve the TCP domain name to an instance of the TCP router. For more information about configuring port reservations, see Modify TCP Port Reservations below.
To configure your ports for your TAS for VMs deployment, enable TCP routing in TAS for VMs. For more information, see Configure networking in Configuring TAS for VMs.

Enable TCP routing

To enable TCP routing:

Go to the Ops Manager Installation Dashboard.
Click the VMware Tanzu Application Service for VMs tile.
Select Networking.
In TCP routing, click Allow.
For TCP router IPs, enter the IP addresses to assign to the TCP routers as multiple values as a comma-separated list or as a range. For example, 10.254.0.1, 10.254.0.2 or 10.254.0.1-10.254.0.2. These addresses must be within your subnet CIDR block. The addresses are the same IP addresses with which you configured your load balancer in Prerequisites for configuring TCP routing, unless you configured DNS to resolve the TCP domain name directly to an IP for the TCP router.
For TCP routing ports, enter one or more ports to which the load balancer forwards requests. To support multiple TCP routes, VMware recommends allocating multiple ports. Do one of the following:
- To allocate a single port or range of ports, enter a single port or a range of ports.
  Note: If you configured AWS for TAS for VMs manually, enter 1024-1123. This range of ports corresponds to the rules you created for -tcp-elb.
- To allocate a list of ports:
  1. Enter a single port.
  2. After deploying TAS for VMs, follow the procedure in Modify TCP port reservations below to add TCP routing ports through the cf CLI.
(Optional) For TCP request timeout, modify the default value of 300 seconds. This field determines when the TCP router closes idle connections from clients to apps that use TCP routes. You may want to increase this value to enable developers to push apps that require long-running idle connections with clients.
For AWS, Azure, or GCP Ops Manager deployments, add the name of your load balancer to the TCP Router field in the Resource Config pane of the TAS for VMs tile. For more information, see Configuring load balancing for TAS for VMs.

Configure TCP Routing After Deploying TAS for VMs

After you enable TCP routing and deploy TAS for VMs, you must add the TCP shared domain and configure org quotas to enable developers to create TCP routes. To do this, you must use the Cloud Foundry Command Line Interface (cf CLI) and have an admin user account.

For more information about the cf CLI, see Using the Cloud Foundry Command Line Interface (cf CLI).

Configure TAS for VMs with Your TCP Domain

After deploying TAS for VMs, you must configure TAS for VMs with the domain that you configured in Prerequisites for Enabling TCP Routing. This is the domain from which developers create TCP routes.

To configure TAS for VMs with your TCP domain:

List your router groups by running:
```
cf router-groups
```
You should see default-tcp as a response.
Create a shared domain and associate it with the default-tcp router group by running:
```
cf create-shared-domain tcp.APP-DOMAIN.com --router-group default-tcp
```
Where APP-DOMAIN is the name of your app domain.
Verify that TCP appears under type next to your TCP domain by running:
```
cf domains
```

Configure a Quota for TCP Routes

Since TCP route ports are a limited resource in some environments, quotas are configured to allow creation of zero TCP routes by default. After you deploy TAS for VMs, you can increase the maximum number of TCP routes for all orgs or for particular orgs and spaces. Because you reserve a route port for each TCP route, you manage the quota for route ports using the --reserved-route-ports cf CLI command option. For more information, see Creating and Modifying Quota Plans.

You can configure quotas for TCP routes in the following ways:

If you have a default quota that applies to all orgs, you can update it to configure the number of route ports that can be reserved by each org by running:
```
cf update-org-quota QUOTA --reserved-route-ports NUMBER-OF-ROUTE-PORTS
```
Where:
- QUOTA is the maximum number of TCP routes you want to allocate to all orgs.
- NUMBER-OF-ROUTE-PORTS is the number of route ports you want to allow each org to reserve.
To create a new quota that governs the number of route ports that can be created in a particular org:
1. Target the org for which you want to create the quota by running:
```
cf target -o ORG-NAME
```
  Where ORG-NAME is the name of the org for which you want to create the quota.
2. Run:
```
cf create-quota QUOTA --reserved-route-ports NUMBER-OF-ROUTE-PORTS
```
  Where:
  - QUOTA is the maximum number of TCP routes you want to allocate to particular orgs.
  - NUMBER-OF-ROUTE-PORTS is the number of route ports you want to allow each org to reserve.
To create a new quota that governs the number of route ports that can be created in a particular space:
1. Target the space for which you want to create the quota by running:
```
cf target -s SPACE-NAME
```
  Where SPACE-NAME is the name of the space for which you want to create the quota.
2. Run:
```
cf create-space-quota QUOTA --reserved-route-ports NUMBER-OF-ROUTE-PORTS
```
  Where:
  - QUOTA is the maximum number of TCP routes you want to allocate to a particular space.
  - NUMBER-OF-ROUTE-PORTS is the number of route ports you want to allow the space to reserve.

Create a TCP route

For information about creating a TCP route, see Create a TCP Route with a Port in Configuring Routes and Domains.

Modify TCP port reservations

After deploying TAS for VMs, you can modify the range of ports available for TCP routes using cf curl commands, as demonstrated with the commands below. These commands require you to have an admin user account with the routing.router_groups.read and routing.router_groups.write scopes.

In a terminal window, view the reservable_ports by running:
```
cf curl /routing/v1/router_groups
```
Record the guid from the output.
To configure a new port, run:
```
cf curl /routing/v1/router_groups/GUID
```
Where GUID is the GUID you recorded in the previous step.

To configure multiple ports, enter a comma-separated list of ports or port ranges by running:
```
cf curl \
-X PUT -d '{"reservable_ports":"PORTS-OR-PORT-RANGES"}' \
/routing/v1/router_groups/f7392031-a488-4890-8835-c4a038a3bded
```
Where PORTS-OR-PORT-RANGES is a comma-separated list of ports or port ranges. For example, "reservable_ports":"1024-1199,1234-1248,1312".

Important Do not enter reservable_ports that conflict with other TCP router instances or ephemeral port ranges. VMware recommends using port ranges within 1024-2047 and 18000-32767 on default installations. Check which ports are available on the TCP router VMs to verify that no additional ports are in use. For more information, see TCP router fails to configure routes when there is a port conflict with a local process in the Routing Release repository on GitHub.

Disable TCP Routing

To deactivate TCP routing:

Go to Ops Manager Installation Dashboard.
Click the VMware Tanzu Application Service for VMs tile.
Select Networking.
Under TCP routing, select Disable.
Remove the TCP routing domain by running:
```
cf delete-domain DOMAIN
```
Where DOMAIN is the name of your TCP routing domain.