This topic describes how VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) administrators can create and delete network profiles for Kubernetes clusters provisioned by TKGI on vSphere with NSX‑T integration.
This topic also describes the use cases for when a TKGI administrator should use a network profile.
TKGI supports network profiles on TKGI on vSphere with NSX‑T only.
To work with TKGI network profiles you must be either a cluster manager or cluster administrator:
To create or delete a network profile, you must be a cluster administrator: pks.clusters.admin
.
To use a network profile, you must be a cluster manager: pks.clusters.manage
or a cluster administrator: pks.clusters.admin
.
Note: If a cluster manager, pks.clusters.manage
, attempts to create or delete a network profile, the following error occurs: “You do not have enough privileges to perform this action. Please contact the TKGI administrator.”
You can use network profiles to customize your TKGI Kubernetes clusters on vSphere with NSX‑T. For information on when to use network profiles, see Network Profile Use Cases below.
TKGI cluster administrators can administer network profiles in the following ways:
TKGI cluster administrators can also use network profiles in all the ways that a cluster manager can:
For information on managing network profiles, see Using and Managing Network Profiles.
The following is the basic structure of a network profile JSON configuration:
{
"name": "PROFILE-NAME",
"description": "PROFILE-DESCRIP",
"parameters": {
TOP-LEVEL-PARAMETERS,
"cni_configurations": {
"type": "nsxt",
"parameters": {
CNI-CONFIGURATIONS-PARAMETERS
}
}
}
}
Where:
PROFILE-NAME
is the internal name of the network profile.PROFILE-DESCRIP
is an internal description of the network profile.TOP-LEVEL-PARAMETERS
are one or more comma-delimited top-level parameters
in a Network Profile. For more information, see Top-Level Parameters below.CNI-CONFIGURATIONS-PARAMETERS
are one or more comma-delimited cni_configurations
parameters
in a Network Profile. For more information, see cni_configurations Parameters below.To create a network profile in TKGI:
Create a network profile configuration JSON file with the following content:
{
"name": "PROFILE-NAME",
"description": "PROFILE-DESCRIP",
"parameters": {
"cni_configurations": {
"type": "nsxt",
"parameters": {
}
}
}
}
Where:
PROFILE-NAME
is the internal name for your network profile.PROFILE-DESCRIP
is an internal description for your network profile.Edit the file to specify your network parameters. For information about the available network parameters, see Network Profile Parameters below.
Review your network profile configuration carefully. If you are modifying an existing cluster, ensure that you are modifying only parameters that support modification. For information on which network profile parameters are updateable in this version of TKGI, see the network profile parameters tagged Updatable
in the Network Profile Parameters tables below. You cannot modify any other network profile parameters on an existing cluster.
To create a network profile from your network profile configuration, run the following TKGI CLI command:
tkgi create-network-profile PATH-TO-YOUR-NETWORK-PROFILE-CONFIGURATION
Where PATH-TO-YOUR-NETWORK-PROFILE-CONFIGURATION
is the path to your network profile configuration file.
For example:
$ tkgi create-network-profile np-routable-pods.jsonNetwork profile example-network-profile successfully created
Cluster managers can create new clusters with your network profile and assign your network profile to existing clusters. For information on managing network profiles, see Using and Managing Network Profiles.
The following is an example of a complete network profile JSON configuration:
{
"name": "example-network-profile",
"description": "Example Network Profile with All Available Parameters -- FOR ILLUSTRATION PURPOSES ONLY",
"parameters": {
"lb_size": "large",
"pod_ip_block_ids": [
"ebe78a74-a5d5-4dde-ba76-9cf4067eee55",
"ebe78a74-a5d5-4dde-ba76-9cf4067eee56" ],
"pod_subnet_prefix": 27,
"pod_routable": true,
"fip_pool_ids": [
"e50e8f6e-1a7a-45dc-ad49-3a607baa7fa0",
"ebe78a74-a5d5-4dde-ba76-9cf4067eee55" ],
"t0_router_id": "5a7a82b2-37e2-4d73-9cb1-97a8329e1a90",
"master_vms_nsgroup_id": "9b8d535a-d3b6-4735-9fd0-56305c4a5293",
"node_ip_block_ids": [
"2250dc43-63c8-4bb8-b8cf-c6e12ccfb7de", "3d577e5c-dcaf-4921-9458-d12b0e1318e6" ],
"node_routable": true,
"node_subnet_prefix": 20,
"nodes_dns": [
"8.8.8.8", "192.168.115.1", "192.168.116.1" ],
"dns_lookup_mode": "API_INGRESS",
"ingress_prefix": "api",
"single_tier_topology": true,
"infrastructure_networks": [
"30.0.0.0/24",
"192.168.111.0/24",
"192.168.115.1" ],
"failover_mode": "PREEMPTIVE",
"cni_configurations": {
"type": "nsxt",
"parameters": {
"nsx_lb": false,
"nsx_ingress_controller": true,
"x_forwarded_for": "insert",
"ingress_ip": "192.168.160.212",
"log_settings": {
"log_level": "DEBUG",
"log_firewall_traffic": "ALL" },
"ingress_persistence_settings": {
"persistence_type": "cookie",
"persistence_timeout": 1 },
"max_l4_lb_service": 10,
"l4_persistence_type": "source_ip",
"l4_lb_algorithm": "weighted_round_robin",
"top_firewall_section_marker":"section-id",
"bottom_firewall_section_marker":"section-id",
"lb_http_request_header_size":60,
"lb_http_response_header_size":45,
"lb_http_response_timeout":30,
"connect_retry_timeout":30,
"enable_hostport": true,
"enable_nodelocaldns": true,
"client_ssl_profile": "example_ssl_profile_ID",
"lb_connection_multiplexing_enabled": true,
"lb_connection_multiplexing_number": 80
}
}
}
}
Note: This example network profile is for illustration purposes only. It is not intended to be used as a template for a network profile configuration.
To update an existing cluster’s network profile:
After you create a cluster, you can modify only specific network profile parameters. Ensure that you modify only parameters that support modification.
For information on which network profile parameters are updateable in this version of TKGI, see the network profile parameters tagged Updatable
in the Network Profile Parameters tables below. You cannot modify any other network profile parameters on an existing cluster.
For more information, see Update-Cluster Network Profile Validation Rules below.
To create a modified network profile configuration file:
name
field to a unique name.If you are updating the pod_ip_block_ids
field, reorder the IP Block IDs or add additional IP Block IDs.
For example, the following network profile has two pod_ip_block_ids
, the first is the original IP block used when creating the cluster, and the second is the new IP block to use for pods.
{
"description": "Example network profile for adding pod IP addresses to an existing cluster",
"name": "pod-ips-add",
"parameters": {
"pod_ip_block_ids": [
"ebe78a74-a5d5-4dde-ba76-9cf4067eee55",
"ebe78a74-a5d5-4dde-ba76-9cf4067eee56"
]
}
}
Note: Update only network profile properties that support being updated.
For more information on configuring a network profile, see Network Profile Parameters above.
Review and save the network profile configuration file.
To create a network profile from a configuration file:
Run the following TKGI CLI command:
tkgi create-network-profile PATH-TO-YOUR-NETWORK-PROFILE-CONFIGURATION
Where PATH-TO-YOUR-NETWORK-PROFILE-CONFIGURATION
is the path to your network profile configuration file.
To update a cluster with a modified network profile:
To apply the network profile created above to your cluster, run the following command::
tkgi update-cluster CLUSTER-NAME --network-profile NETWORK-PROFILE-NAME
Where:
CLUSTER-NAME
is the unique name of your cluster.NETWORK-PROFILE-NAME
is the name of the network profile you want to use for your cluster.TKGI validates the network profile before updating the cluster with the new network profile. For more information, see Update-Cluster Network Profile Validation Rules below.
TKGI uses strict validation rules before applying a network profile to a cluster with an existing network profile:
pod_ip_block_ids
field in the new network profile contains the same entries as the existing network profile, the entry passes validation.TKGI administrators can delete a network profile that is not in use.
To delete a network profile:
Run the following TKGI CLI command:
tkgi delete-network-profile NETWORK-PROFILE-NAME
Where NETWORK-PROFILE-NAME
is the name of the network profile you want to delete.
Note: You cannot delete a network profile that is in use.
The Top-Level Parameters and cni_configurations Parameters sections below describe the parameters you can add to a Network Profile.
TKGI supports the following top-level network profile parameters:
Parameter | Type | Description |
---|---|---|
name |
String | Name of the network profile. |
description |
String | Description of the network profile. |
parameters |
Map | One or more name-value pairs. |
cni_configurations |
Map | Map containing key-value pairs for configuring NCP (see table below). |
dns_lookup_mode |
String | DNS lookup mode. Values: “API” , “API_INGRESS” .For Kubernetes API LB: “API” .For Ingress controller: “API_INGRESS” . |
failover_mode |
String | If the preferred node fails and recovers, enable the node to preempt a peer as the active node. Values: “PREEMPTIVE” , “NON_PREEMPTIVE” .Default: “PREEMPTIVE” . |
fip_pool_ids |
String | Array of floating IP pool UUIDs defined in NSX-T. |
infrastructure_networks |
String | Array of IP addresses and subnets for Node Networks for use with a Shared Tier-1 topology in a Multi-Tier-0 environment. |
ingress_prefix |
String | Ingress controller hostname prefix for DNS lookup. If DNS mode is set to API_INGRESS , TKGI creates the cluster with ingress_prefix.hostname as the Kubernetes control plane FQDN. TKGI confirms that the ingress subdomain can be resolved as a subdomain prefix on the host before creating new clusters. |
lb_size |
String | Size of the NSX-T load balancer service. Values: “small” , “medium” , “large” .Default Value: “small” . |
master_vms_nsgroup_id |
String | Namespace Group UUID as defined in NSX-T. |
nodes_dns |
String Updatable |
Array (up to 3) of DNS server IP addresses for lookup of Kubernetes nodes and pods. |
pod_ip_block_ids |
String Updatable |
Array of Pod IP Block UUIDs. |
pod_routable |
Boolean | Make the Pods subnet routable. Values: true , false .Default Value: false . |
pod_subnet_prefix |
Integer | Size of the Pods IP Block subnet. |
single_tier_topology |
Boolean | Use a single Tier-1 Router per cluster (shared). Values: true , false .Default Value: true . |
t0_router_id |
String | Tenant Tier-0 Router UUID defined in NSX-T. |
Note: On an existing cluster, you can modify only the network profile parameters that are labeled Updatable.
TKGI supports the following cni_configurations
parameters:
Parameter | Type | Description |
---|---|---|
type |
String | Values: “nsxt” . |
parameters |
Map | One or more name-value pairs for NCP settings. |
bottom_firewall_section_marker |
String | UUID of the bottom section-id for the distributed firewall (DFW) section as defined in NSX-T.See also: top_firewall_section_marker below and Define DFW Section Markers. |
client_ssl_profile |
String Updatable |
The NSX-T client-side ssl profile to use, exposed by NCP as client_ssl_profile . Default: The default NCP client SSL profile. For more information, see client_ssl_profile below. |
connect_retry_timeout |
Integer | Configure HTTP LoadBalancer connection retry timeout. Example Value: 30 .See also: lb_http_response_timeout and persistence_timeout . |
enable_hostport |
Boolean Updatable |
Enable NCP support for Kubernetes Host Port. Values: true , false .Default Value: false . |
enable_nodelocaldns |
Boolean Updatable |
Enable NCP support for Kubernetes NodeLocal DNSCache. Values: true , false .Default Value: false . |
ingress_ip |
String | IP address to use for the ingress controller. |
ingress_persistence_settings |
String | Parameters for customizing Layer 7 persistence. See also: persistence_timeout and persistence_type |
l4_lb_algorithm |
String | Layer 4 load balancer behavior. Values: “round_robin” , “least_connection” ,“ip_hash” , “weighted_round_robin” .Default Value: “round_robin” . See also: l4_persistence_type and max_l4_lb_service . |
l4_persistence_type |
String | Connection stickiness based on source_ip .See also: l4_lb_algorithm and max_l4_lb_service . |
lb_connection_multiplexing_enabled |
Boolean Updatable |
Enable NSX-T load balancer TCP multiplexing. Values: true , false .Default: false . |
lb_connection_multiplexing_number |
Integer Updatable |
The maximum number of NSX-T load balancer TCP multiplexing connections. Default: 6 . |
lb_http_request_header_size |
Integer | Configure HTTP LoadBalancer request header size. Example Value: 60 . |
lb_http_response_header_size |
Integer | Configure HTTP LoadBalancer response header size. Example Value: 45 . |
lb_http_response_timeout |
Integer | Configure HTTP LoadBalancer response timeout. Example Value: 30 .See also: connect_retry_timeout and persistence_timeout . |
log_dropped_traffic |
Boolean | Log dropped firewall traffic. Values: true , false .Default: false . A log_settings parameter. See also: log_firewall_traffic , log_level , log_settings . |
log_firewall_traffic |
String | Log firewall traffic. Values: ALL , ALLOW , DENY .Default: DENY . A log_settings parameter. See also: log_dropped_traffic , log_level , log_settings . |
log_level |
String | Values: “INFO” , “WARNING” , “DEBUG” , “ERROR” , “CRITICAL” . A log_settings parameter. See also: log_dropped_traffic , log_firewall_traffic , log_settings . |
log_settings |
Map | Parameters for configuring NCP logging. See also: log_dropped_traffic , log_firewall_traffic , log_level . |
max_l4_lb_service |
Integer | Limit the maximum number of layer 4 virtual servers per cluster. Minimum Value: 1 .See also: l4_lb_algorithm and l4_persistence_type . |
nsx_ingress_controller |
Boolean | Use NSX-T layer 7 virtual server as the ingress controller for the Kubernetes cluster. Default is true. Values: true , false . |
nsx_lb |
Boolean | Use NSX-T layer 4 virtual server for each Kubernetes service of type LoadBalancer. Values: true , false .Default Value: true . |
persistence_timeout |
Integer | An ingress_persistence_settings parameter. Persistence timeout interval in seconds.See also: connect_retry_timeout and lb_http_response_timeout . |
persistence_type |
String | An ingress_persistence_settings parameter. Specify the ingress persistence type.Values: “cookie” , “source_ip” . Default Value: true . |
top_firewall_section_marker |
String | UUID of the top section-id for the distributed firewall (DFW) section as defined in NSX-T.See also: bottom_firewall_section_marker above and Define DFW Section Markers. |
x_forwarded_for |
String | Use the same source IP for calling clients. Values: “insert” , “replace” . |
Note: On an existing cluster, you can modify only the network profile parameters that are labeled Updatable.
The following describes commonly used network profile parameters:
The primary use case for updating the network profile client_ssl_profile
field is to configure which NSX-T client-side ssl profile is to be used by NCP.
You can change the client_ssl_profile
parameter to a valid NSX-T client-side ssl profile ID.
Configure client_ssl_profile
in the cni_configurations
parameters section of your network profile.
If client_ssl_profile
is blank or incorrect, the default NCP client SSL profile value is used instead:
nsx-default-client-ssl-profile
.default-balanced-client-ssl-profile
.Both of these default NCP client SSL profiles use balanced-level pre-defined ciphers.
The primary use case for updating the network profile enable_hostport
field is to activate or deactivate NCP support for Kubernetes Host Port.
You can change the enable_hostport
parameter to either true
or false
. Configure enable_hostport
in the cni_configurations
parameters section of your network profile.
The Kubernetes Host Port feature allows you to expose an application to be externally accessible through a single port from outside of your cluster. For more information on Host Port, see Pod Security Policies in the Kubernetes documentation.
The primary use case for updating the network profile enable_nodelocaldns
field is to activate or deactivate NCP support for Kubernetes NodeLocal DNSCache.
You can change the enable_nodelocaldns
parameter to either true
or false
. Configure enable_nodelocaldns
in the cni_configurations
parameters section of your network profile.
NodeLocal DNSCache improves cluster DNS performance by running a dns caching agent on cluster nodes as a DaemonSet. For more information on NodeLocal DNSCache, see Using NodeLocal DNSCache in Kubernetes clusters in the Kubernetes documentation.
The primary use case for updating the network profile lb_connection_multiplexing_enabled
field is to enable NSX-T load balancer TCP multiplexing.
You can change the lb_connection_multiplexing_enabled
parameter to either true
or false
.
Configure lb_connection_multiplexing_enabled
in the cni_configurations
parameters section of your network profile.
The primary use case for updating the network profile lb_connection_multiplexing_number
field is to configure the maximum number of NSX-T load balancer TCP multiplexing connections.
You can change the lb_connection_multiplexing_number
parameter to an integer value.
Configure lb_connection_multiplexing_number
in the cni_configurations
parameters section of your network profile.
The primary use case for updating the network profile nodes_dns
field is to update the DNS server configuration for a cluster.
You can configure the network profile nodes_dns
field to add, modify or remove IP addresses from a cluster DNS server configuration. For more information on the network profile nodes_dns
field and an example of a nodes_dns
configuration, see Specify Nodes DNS Servers.
Note: If you modify a DNS server configuration, do not exceed the maximum of three DNS server IP addresses.
The primary use case for updating the network profile pod_ip_block_ids
field is to add additional IP addresses for pods when a cluster is at or near the point exhausting all available public IP addresses for pods.
You can change the pod_ip_block_ids
parameter array as follows:
You cannot remove any of the IP Block IDs from an existing pod_ip_block_ids
parameter array. Do not create a copy of a network profile, remove pod_ip_block_ids
array values, and assign the new profile to a cluster that has the original profile assigned.
For more information on modifying a network profile pod_ip_block_ids
field, see Add Pod IPs in Customizing Pod Networks. For more information on the pod_ip_block_ids
field, see Network Profile Parameters above.
Network profiles let you customize configuration parameters for Kubernetes clusters provisioned by TKGI on vSphere with NSX‑T.
You can apply a network profile to a Kubernetes cluster for the following scenarios:
Topic | Description |
---|---|
Size a Load Balancer | Customize the size of the NSX-T load balancer service that is created when a Kubernetes cluster is provisioned. |
Customizing Pod Networks | Customize Kubernetes Pod Networks, including adding pod IP addresses, subnet size, and routability. |
Customize Node Networks | Customize Kubernetes Node Networks, including the IP addresses, subnet size, and routability. |
Customize Floating IP Pools | Specify a custom floating IP pool. |
Configure Bootstrap NSGroups | Specify an NSX-T Namespace Group where the Kubernetes control plane nodes will be added to during cluster creation. |
Configure Edge Router Selection | Specify the NSX-T Tier-0 router where Kubernetes node and Pod networks will be connected to. |
Specify Nodes DNS Servers | Specify one or more DNS servers for Kubernetes clusters. |
Configure DNS for Pre-Provisioned IPs | Configure DNS lookup of the Kubernetes API load balancer or ingress controller. |
Configure the TCP Layer 4 Load Balancer | Configure layer 4 TCP load balancer settings; use a third-party load balancer. |
Configure the HTTP/S Layer 7 Ingress Controller | Configure layer 7 HTTP/S ingress controller settings; use third-party ingress controller. |
Define DFW Section Markers | Configure top or bottom section markers for explicit DFW rule placement. |
Configure NCP Logging | Configure NCP logging. |
Dedicated Tier-1 Topology | Use dedicated Tier-1 routers, rather than a shared router, for each cluster’s Kube node, Namespace, and NSX-T load balancer. |