Using Proxies with Tanzu Kubernetes Grid Integrated Edition on VMware NSX

This topic describes how HTTP/HTTPS proxies work in VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) with NSX, and how to set proxies globally.

To configure proxy settings specifically for individual TKGI clusters, see Configure Cluster Proxies.

Overview

If your environment includes HTTP proxies, you can configure Tanzu Kubernetes Grid Integrated Edition with NSX to use these proxies so that Tanzu Kubernetes Grid Integrated Edition-deployed Kubernetes control plane and worker nodes access public Internet services and other internal services through a proxy.

In addition, Tanzu Kubernetes Grid Integrated Edition proxy settings apply to the TKGI API instance. When an Tanzu Kubernetes Grid Integrated Edition operator creates a Kubernetes cluster, the TKGI API VM behind a proxy is able to manage NSX objects on the standard network.

You can also proxy outgoing HTTP/HTTPS traffic from Ops Manager and the BOSH Director so that all Tanzu Kubernetes Grid Integrated Edition components use the same proxy service.

The following diagram illustrates the network architecture:

TKGI Proxy Architecture

Enable TKGI API and Kubernetes Proxy

To configure a global HTTP proxy for all outgoing HTTP/HTTPS traffic from the Kubernetes cluster nodes and the TKGI API server, perform the following steps:

Navigate to Ops Manager and log in.
Click the Tanzu Kubernetes Grid Integrated Edition tile.
Click Networking.
Under HTTP/HTTPS proxy, select Enabled. When this option is enabled, you can proxy HTTP traffic, HTTPS traffic, or both.
(Optional) Configure Tanzu Kubernetes Grid Integrated Edition to use a proxy.

Production environments can deny direct access to public Internet services and between internal services by placing an HTTP or HTTPS proxy in the network path between Kubernetes nodes and those services.

Configure Tanzu Kubernetes Grid Integrated Edition to use your proxy and activate the following:
- TKGI API access to public Internet services and other internal services.
- Tanzu Kubernetes Grid Integrated Edition-deployed Kubernetes nodes access to public Internet services and other internal services.
- Tanzu Kubernetes Grid Integrated Edition Telemetry ability to forward Telemetry data to the CEIP and Telemetry program.
  Note: This setting does not set the proxy for running Kubernetes workloads or pods.
To complete your global proxy configuration for all outgoing HTTP/HTTPS traffic from your Kubernetes clusters, perform the following steps:
1. To proxy outgoing HTTP traffic, enter the URL of your HTTP proxy endpoint under HTTP Proxy URL. For example, http\://myproxy.com:1234.
2. (Optional) If your outgoing HTTP proxy uses basic authentication, enter the user name and password in the HTTP Proxy Credentials fields.
3. To proxy outgoing HTTPS traffic, enter the URL of your HTTP proxy endpoint under HTTPS Proxy URL. For example, http\://myproxy.com:1234.
  
  Note: Using an HTTPS connection to the proxy server is not supported. HTTP and HTTPS proxy options can only be configured with an HTTP connection to the proxy server. You cannot populate either of the proxy URL fields with an HTTPS URL. The proxy host and port can be different for HTTP and HTTPS traffic, but the proxy protocol must be HTTP.
4. (Optional) If your HTTPS proxy uses basic authentication, enter the user name and password in the HTTPS Proxy Credentials fields.
5. Under No Proxy, enter the comma-separated list of IP addresses that must bypass the proxy to allow for internal Tanzu Kubernetes Grid Integrated Edition communication.
  
  Include 127.0.0.1 and localhost in the No Proxy list.
  
  Also include the following in the No Proxy list:
  - Your Tanzu Kubernetes Grid Integrated Edition environment’s CIDRs, such as the service network CIDR where your Tanzu Kubernetes Grid Integrated Edition cluster is deployed, the deployment network CIDR, the node network IP block CIDR, and the pod network IP block CIDR.
  - The FQDN of any registry, such as the Harbor API FQDN, or component communicating with Tanzu Kubernetes Grid Integrated Edition, using a hostname instead of an IP address.
  - The IP addresses for your NSX Manager, vCenter Server, and all ESXi hosts, if you are upgrading and have an existing proxy configuration for reaching a Docker registry or other external services.
  - Any additional IP addresses or domain names that must bypass the proxy.
    
    The No Proxy property for vSphere accepts wildcard domains denoted by a prefixed \*. or ..
    
    For example:
```
127.0.0.1,localhost,
*.example1.com,
.example2.com,
example3.com,
198.51.100.0/24,
203.0.113.0/24,
192.0.2.0/24
```
Note: By default the 10.100.0.0/8 and 10.200.0.0/8 IP address ranges, .internal, .svc,.svc.cluster.local, .svc.cluster, and your Tanzu Kubernetes Grid Integrated Edition FQDN are not proxied. This allows internal Tanzu Kubernetes Grid Integrated Edition communication.

Do not use the _ character in the No Proxy field. Entering an underscore character in this field can cause upgrades to fail.

Because some jobs in the VMs accept \*. as a wildcard, while others only accept ., we recommend that you define a wildcard domain using both of them. For example, to denote example.com as a wildcard domain, add both \*.example.com and example.com to the No Proxy property.
Save the changes to the Tanzu Kubernetes Grid Integrated Edition tile.
Proceed with any remaining Tanzu Kubernetes Grid Integrated Edition tile configurations and deploy Tanzu Kubernetes Grid Integrated Edition. See Installing Tanzu Kubernetes Grid Integrated Edition on vSphere with NSX.

Enable Ops Manager and BOSH Proxy

To enable an HTTP proxy for outgoing HTTP/HTTPS traffic from Ops Manager and the BOSH Director, perform the following steps:

Log in to Ops Manager.
Select User Name > Settings in the upper right.
Click Proxy Settings.
Under HTTP Proxy, enter the FQDN or IP address of the HTTP proxy endpoint. For example, http://myproxy.com:80.
Under HTTPS Proxy, enter the FQDN or IP address of the HTTPS proxy endpoint. For example, http://myproxy.com:80.
Note: Using an HTTPS connection to the proxy server is not supported. Ops Manager and BOSH HTTP and HTTPS proxy options can be only configured with an HTTP connection to the proxy.
Under No Proxy, include the hosts that must bypass the proxy. This is required.

In addition to 127.0.0.1 and localhost, include the BOSH Director IP, Ops Manager IP, TKGI API VM IP, and the TKGI Database VM IP. If the TKGI Database is in HA mode (beta), enter all your database IPs in the No Proxy field.
```
127.0.0.1,localhost,BOSH-DIRECTOR-IP,TKGI-API-IP,OPS-MANAGER-IP,TKGI-DATABASE-IP
```
Note: Ops Manager does not allow the use of a CIDR range in the No Proxy field. You must specify each individual IP address to bypass the proxy.

The No Proxy field does not accept wildcard domain notation, such as .docker.io and .docker.com. You must specify the exact IP or FQDN to bypass the proxy, such as registry-1.docker.io.
Click Save.
Return to the Ops Manager Installation Dashboard and click Review Pending Changes.
Click Apply Changes to deploy Ops Manager and the BOSH Director with the updated proxy settings.