To ensure optimal Kubernetes cluster performance, you should follow certain best practices. This section highlights some of the key best practices.

Setting Adequate Quotas in OpenStack

For an OpenStack provider, set quotas that are large enough to accommodate a large cluster.

Table 1. Sample Commands
Command Description 
nova quota-update --key-pairs 500 
   --instances 500 
   --cores 4000 
   --ram 12288000 <tenant_ID>
 Set quotas for a 500-node cluster, where each node has 8 vCPUs and 24G RAM 
neutron quota-update --tenant-id <tenant_ID> 
   --pool 300 
   --port 1000 
   --loadbalancer 300 
   --floatingip 150
 Neutron command to allocate quota according to your network. Port number should be greater than instance plus load balancer number. 
cinder quota-update --volumes 500 
   --gigabytes 5000 <tenant_ID>
 Cinder command to allocate quota according to the number of persistent volumes that you want to create.

Best Practices for Creating Large Clusters

To create a large cluster, a best practice is to first create a small cluster, then scale it out. For example, to create a stable 500-node cluster, start by creating a 30-node cluster, then scale it out with a maximum of 30 nodes at a time until you reach 500 nodes.

Tips:
  • If your cluster is larger than 200 nodes, you might see RPC timeouts in the OpenStack service logs. If that occurs, increase the RPC timeout setting for those services. For example for a Nova service, increase the value of the rpc_response_timeout configuration option in the nova.conf file.
  • It may take time to refresh the status of created resources when scaling out a cluster. Add the --skip-refresh option to the vkube cluster scaleout command to decrease the deployment time. With this option, the scale out operation does not check the state of existing resources such as VMs or load balancers, and assumes that the resources are successfully created.

Managing High CPU Usage with an OpenStack Provider

If you are using VMware Integrated OpenStack deployed in compact mode as your OpenStack provider, you may notice high CPU usage on the controller or compute service VM’s. If so, increase the number of vCPU’s to 16 per VM.

Alternatives to Load Balancing with NSX-V Backing

When you create services in Kubernetes and you specify the type as LoadBalancer, NSX Edge load balancers are deployed for every service. The load balancer distributes the traffic to all Kubernetes worker nodes up to 32 members. If your Kubernetes cluster includes more that 32 worker nodes, use the Kubernetes Ingress resource instead.

Persistent Volume Claim Management

If you create many persistent volume claims and associated pods in parallel, you should use VMware Integrated OpenStack in HA mode. VMware Integrated OpenStack in compact mode may not provide enough capacity to handle the large number of incoming API requests.

If dynamic provisioning of persistent volumes fails even with VMware Integrated OpenStack in HA mode, check the OpenStack service logs to see if the failures are due to RPC timeouts and increase the RPC timeout setting for those services. For example for a Nova service, you can increase the value of the rpc_response_timeout configuration option in the nova.conf file.

Best Practice for Configuring an SDDC Provider with LDAP

When configuring an SDDC provider with LDAP, a best practice is to set the filters for the LDAP user and group ensure that each filter is returning fewer than 1000 users or groups. If the limit is exceeded, the query returns no result and reports an error.

Best Practices for Backup

If a cluster fails or the VMware Integrated OpenStack with Kubernetes dashboard become unresponsive, the following best practices will help to ensure that you can always restore your configuration.
  • After deploying VMware Integrated OpenStack with Kubernetes, create a snapshot of your VMware Integrated OpenStack with Kubernetes VM on the vCenter server.
  • When a provider or a cluster is added or deleted, create a backup.
  • After changing the configuration of an application running on a cluster, backup the configuration of the application.
  • Move backups from the VMware Integrated OpenStack with Kubernetes VM to external storage.
  • Regularly delete unneeded backup files to ensure adequate storage space on the VMware Integrated OpenStack with Kubernetes VM.